They reference this paper which describes a method to decrypt reasoning traces (by sending the encrypted trace back to the model and asking it to transcribe it):
replydeleted
replyInteresting, but I suppose that's a hole that can be easily patched.
replypatched with gpt 6
replyYeah, with GPT 6 even OpenAI doesn't know what it's thinking.
reply[dead]
replyThey do store the reasoning locally. It's encrypted, though.
replyFew weeks ago there was a new paper out where researchers took the encrypted reasoning tokens and injected it into a new session with a week or model in the same family that they could reliably jailbreak. They would then ask the model to repeat its reasoning and the results were pretty consistent.
They used the LLM as a decryption oracle of sorts.
Reasoning works as long as there is a consistent latent space representation. Any kind of poison will just become part of the representation. There's evidence that even directly training on encrypted reasoning traces works, because the length is already a strong signal.
reply[flagged]
reply