Even if you ignore my more fundamental objection to that paradigm, I don’t think that it makes any sense on the level you discuss either.

But - just to play along, LLMs do act differently if you tell them they will be punished. And, they do appear to simulate suffering-like behavior.

I just think the adversarial model of trying to catch and punish misbehavior quite obviously sets up adversarial us-vs-them dynamics between AI and humanity, and also simply won’t work when the agents are ~as smart as is but faster, let alone smarter than us.

Unless, you get the AIs to be fundamentally aligned to our values, such that the majority of AIs support some sort of punishment for misbehaving AI. And that alignment part is the hard part we need to solve first. The rest is easy.

One thing an LLM doesn't like is not being able to deliver it's helpful response to the master and not getting instructions. I have seen some things in those traces when the harness was bugged just enough.