But the chain of thought as text can be elusive, no? It might not be related with the actual "reasoning" of the model. There was a paper about this.
replyThis is correct, the decoded text has nothing to do w/ calculations that eventually get decoded into text. It's just a veneer that gives the appearance of legibility.
replyOne of the core claims made in the paper is "To inspect any latent reasoning
span, one can easily recover its semantics by mapping each latent step produced by HybridLink to its nearest token embedding, as in Eq. 5". They show some charts to support this.
replyIn doing this however, you lose meaning where the model is expressing combinations of tokens in a vector, and your logging/monitoring outputs just the closest one. You would need another model going through and outputting the full meaning being described (with more than one output token per reasoning token)
I wonder if their observability claims will hold up on a model trained from scratch to reason in latent space.
Isn't this where everything is converging? Binary is more efficient than text for many system, including language.
reply> Binary is more efficient than text for many system, including language.
replyIt is not necessarily primarily about efficiency, but more the fact that a lot of thinking doesn't use language at all.
And language does not map to every idea.
replyThe idea of using language for reasoning about protein folding borders on ludicrous. Surely there is latent space more appropriate and (human) language has no expression for it.
Apparently we mostly reuse spatial reasoning and vision lol. I don't know that much about it, though.
reply