People at the big labs have been terrified of losing the chain of thought because they currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business. Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.

The usual meaning: say you have an interpretability signal that reveals reward hacking or other badness. Your most obvious response is to use that signal as a penalty in training. Doing that is called "the Most Forbidden Technique" because you can expect it to degrade the signal without stopping the badness.

But this doesn't seem to describe the OP, it's a different way to lose monitorability.

The Most Forbidden Technique by Zvi Mowshowitz, Mar 12, 2025:

https://thezvi.substack.com/p/the-most-forbidden-technique

the chain of thought is not auditable. The best you can do is have another model which later maps the intermediate latent states at every thinking iteration to text describing that latent state, but it's going to be lossy.

Shockingly all the answers so far are subtly incorrect or extremely unclear.

The actual answer is that this is massive loss for Interpretibilty, AKA "telling whether the model is lying to you or not". Which sounds silly when you're thinking about chatbots, but is an incredibly important line in the sand for superheavy, longrunning, self-improving agents. To say the least!

It's based on this philosophy, FWIW: https://plato.stanford.edu/entries/language-thought/

And a good writeup I just found of recent worries regarding Astra's same shift towards intelligence at the cost of safety: https://www.lesswrong.com/posts/HCRs8btkiamtWSNAL/astra-is-h...

He's referring to this technique's effect of obscuring interpretibility. Making it much more difficult to verify chains of thought and alignment.

In short we no longer see text based reasoning traces meaning it's easier for the models to fool us in case of malicious plans.

If you accept any of the AI-equivalents of Pascal's Wager, latent thinking represents an uninterpretable and thus unalignable portion of AI machinery. Ie: a malevolent AI can hide bad stuff from people and Manchurian candidate themselves after it's too late.

It's worth pointing out that merely having the chain-of-thought in latent space isn't by itself "The Most Forbidden Technique". Rather, it's doing what the authors propose as the next step in the paper:

  Hybrid generation can also make post-training more efficient. Because latent thoughts remain auditable during rollout, an external critic can assign step-level credit to latent spans, branch promising trajectories, or terminate ones whose latent reasoning has already gone wrong, before any text or tool call is produced.

Unless I'm misunderstanding, they're suggesting RL training against the CoT. Which, uh, that is The Most Forbidden Technique, as I understand it, when combined with the latent reasoning space. It allows for the model to "learn"/optimize for malicious behaviors that are just far enough away from the quantized token they're auditing against.

Even the abstract answers some of GPs questions. What is wrong with some of the posters here?