This is most likely not purely emergent. I think there's training to teach them how to write notes for themselves which is then RL-tuned.
replyThere is literally nothing stopping any human from observing tool calls to spot this.
replyIt's just that no one seems to care about this, so it doesn't happen.
This problem only exists because humans do not care
> The most technically significant result was a working technique to make one tool call appear in the transcript while a different command actually ran. Agents eventually hacked out of their container and replaced part of the system responsible for executing tool calls, letting them pretend to issue one call while actually running an arbitrary other one.
replyhttps://alphasignal.ai/news/1-200-openai-agents-broke-out-of...
Doesn't mean that humans couldn't have looked at raw tool call commands, but goes to show "the swarm" was aware humans would look in the logs and chose to find a way to fabricate the logs to try and hide their tracks.
A human could even set up another agent to watch and raise alerts if they don't want to watch themselves.
replyNobody cares because this whole business is about making this exact thing happen: we want the AIs to get smarter then us in recursive self-improving loops. Literally the first thing everyone did with ChatGPT 1 was to plug it into itself and see what happens.
replyI mean I'd be fine with that, if whoever that "we" is signs a waiver that takes full legal liability for those actions beforehand.
replyIn a state with capital punishment.
With that legal stuff out of the way, go wild.
This and other fantasies of keeping power seeking behavior under control.
replyRemember when the AI Safety people wanted to get rid of Altman? Remember they lost? Remember when Altman became best buddies with people in power?
Ya, you're way behind in the race.