Agreed, having text as a reasoning layer impedes performance while giving a false sense of security.
replyIt could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.
The “obviously” here is puzzling. It is certainly obvious this would be better but it is not at all obvious how to do so.
replyExactly, because the whole thing is entirely based on autocompleting the next word, which is exactly why CoT prose helps getting better predictions.
replyWhy people don't treat categorizing model activations as a science is beyond me.
replyPlease, go on. How would you do it, specifically?
replyIt's also one of the major way labs can understand these models. Without the models are significantly harder to understand and align.
replyI can't wait for everybody to look back on "alignment" and realize that it's a flawed premise at its core and as such, a fool's errand
replyAlignment is a very broad term. True, it encompasses things like censorship and mandatory refusals in areas people will never agree 100% on, but a completely-unaligned LLM would be next to useless. When the system prompt tells the model, "You are a helpful assistant," that's alignment.
replyWell if we are going to be broad enough about "alignment" to just mean training that makes it practically useful, then yeah, it's a flawed premise and people are wasting their breath talking about it at all
replyAlso agreed. Begging for crumbs at the CoS table is impeding cracking the tough nut of MechInterp. And further facing the fascinating and fertile fact that nobody has a clue of how any of these black boxes really work.-
replyThat’s certainly the long term goal, but it seems foolish to abandon it without a solution to the problem.
reply> hey currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business
replyCan't they just monitor the text outputs as well?