This is exactly why Fodor argues that psychology should be explained on its on level with symbols rather than appealing to neurology. But if you're interested in modeling symbols, there's much better options than ANNs (see nearly any programming language ever).
>ANNs are meant to model brain computation
But we don't really know how that works! So if you know if you're not modelling the low-level behaviour right, you can't assume that there's a correspondence of the higher level computation when you don't really know what that higher level computations are.
But we do have a hypothesis: that it is done by a large number of simple units with very high connectivity and in deep layers. This is what neural networks model.
Personally I was skeptical of this model of the brain, but they have achieved remarkable success in practice, as well as Nobel prizes. The neural networks people may have been onto something all along (I say that grudgingly).