I only used "reason about" as shorthand for the computation that takes place between input text gets tokenized and output tokens gets translated back to text. There are a lot of layers in the middle where the tokens no longer represent text directly but rather a combination of the different tokens through the attention and MLP projections.

There’s enough experience with LLMs now and comparisons to our own thinking that we can say LLMs are thinking and doing so not so differently from us - at least for some aspects of reasoning.

Trying to pretend otherwise at this point doesn’t help us to utilize them or understand their limits. We’re better off accepting that intelligence isn’t unique to us as a species. Then we can see that modeling language as a tool for us as a vehicle for thinking and reasoning, which philosophers like Wittgenstein already landed upon before LLMs.

No decent modern LLM is going to start its reply with “I found the following vulnerabilities” when you ask it to find security vulnerabilities in a straightforward hello world.