It's basically https://en.wikipedia.org/wiki/Chinese_room – a classic in the philosophy of consciousness and AI.
It's was a genuine question though. I'm not saying LLM use neurons, are like neurons, or are meant to be like neurons. I'm saying if you can do the math, for a substrate, that doesn't mean much. The simple math that runs it isn't the secret sauce.