So I already rewrote and deleted this more times than I can count, and the daystar is coming up. I realize I got caught up in the weeds, and my core argument was left wanting. Sorry about that. Regrouping then ...

Anthropomorphism and Anthropodenial are two different forms of Anthropocentrism.

But the really interesting story to me is when you look at the LLM in its own right, to see what it's actually doing.

I'm not disputing the autoregressive framing. I fully admit I started it myself!

But once we're there, what I really wanted to say (just like Turing and Dijkstra did), is that the really interesting question isn't "is it really thinking?" , but what this kind of process is doing, is it useful, what can I do or play with it, and -relevant to this particular story- what can go (catastrophically) wrong.

see also: https://en.wikipedia.org/wiki/Anthropectomy