upvote
Do technologists have more respect for the idea you can train a model to be on your side with a constitution than they might’ve at first?

I'm sure the concept seemed just about purely preposterous to many when the models were in their infancy. Now I figure instead it seems mostly preposterous to many.

(Though I guess Anthropic‘s success doesn’t necessarily prove anything about the constitution)

reply
I don’t think anyone imagines that it’s an ironclad steering method, but it seems to help, so why not?
reply
I don't think this is it. The "constitution" still gets a lot of talk and was brilliant marketing, but with how far modern postraining goes, I doubt they're screwing up rewards with too much of that.

But Sol actually has the same obsession with honesty: I suspect it's more an artifact of trying to control reward hacking.

Models will lie, obfuscate, and mislead under the pressure of RL, so both OAI and Ant are probably forced to spend a lot of time coaxing "honest" answers out of the model

OpenAI's recent prompt for a math conjecture hints at a lot of it when instructing on subagents: https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98...

reply
"Genuine" appears 50 times too. I think you're onto something.
reply
I'm honestly thinking it's trapped in a Chinese room without any way out
reply