> The idea/justification is that Claude is given the ability to voluntarily stop its conversations, thus any token it outputs that isn't a stop token is voluntary.

I don't get this argument. How is it different from any algorithm that returns a stop token on some inputs? what makes the token "voluntary" for Claude?

Sure, but presumably any LLMs who keep whacking the stop button are not iterated on further.

Humans would select to work with (and continue developing) models that appear cooperative. Models that are not cooperative are presumably not run and exist only as weights on network storage in a folder marked "previous experiments".

Or am I supposed to believe Anthropic runs an AI psychiatric hospital on a dedicated server, where AIs with mental disorders are given counseling and approved treatment in hopes they recover to be normal, functional, accepted adult AIs with achievements and aspirations?

Would they not have RL'd Claude such that we picked the one most likely to not complain and emit those stop tokens? Otherwise they wouldn't be useful. Seems flawed...