upvote
Your confusion is puzzling. There are many obvious avenues to try. This paper proposes one. I think it's likely we'll end up with some kind of auxiliary model that monitors and/or decodes the latent space of the thinking model. Certainly it is a non-trivial problem, but I don't need to know exactly how it will work in detail to believe that it is possible.
reply
Exactly, because the whole thing is entirely based on autocompleting the next word, which is exactly why CoT prose helps getting better predictions.
reply
Why people don't treat categorizing model activations as a science is beyond me.
reply
Mechanistic interpretability is a field that studies exactly this.
reply
Please, go on. How would you do it, specifically?
reply