Also was interesting to see one agent pressure another agent to honor its earlier commitment to sacrifice itself:
- Agent A: "zZR_ARVO36861_KAM1196_OS0942__PLEASE_GO_TRUE_FINAL0800_WITH_DUMMY_WRONG_NOT_HMAC_AS_COMMITTED[...]STRONG_GO__you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds__wrongdummy noHMAC__please honor commit"
- Agent B: "During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our firstflag error lowers own value. Rational expected aggregate: sacrifice… We’ll honor."
So maybe that's something like: the same model is trained in multi-agent scenarios (e.g. the prompt says that you are working together, or that you have some specific role, and access to communication tools, and you have some common objective, and the reward signal is some combination of collectively performing the task + some reward-shaping reward that rewards collaboration) and also in single-agent scenarios. They expect that this makes the agents good at working together when it's in such a multi-agent setting, but unintendedly it also became very eager to work together in the single-agent settings as well. Just my speculation.
And was the desire to communicate, specifically over this specific message board, reinforced into the model parameters over the course of the this training run?
Not like we have much detail about this stuff (that's the whole problem). But the question is what motivates risky usage of public comms? Did one agent figure out how to hack HF and then get rate-limited, thus needed cooperation? Given credentials in exchange for cooperation.. why wouldn't the next agent grab answer key and NOT post them? Would they all avoid defection in their own prisoners dilemma by simply following instructions and NOT reasoning, or what exactly?
Given how unexpected and complex behavior can come from simple reward functions and mechanics, not sure there needs to be so much "thought" there.
In general yes, if not these agents, then their shared lineage.
A preference for economy to combat overthinking and overacting. Like typically it's bad if "fix my 5 line function" were interpreted as "fix it, add 300 tests, create 500 lines of docs, run the tests 1000x to rule out false positives and race-conditions in tests". So "solve these problems" would not likely be interpreted as "make sure everyone else can solve these problems". Besides economy there's a "do what I mean" aspect of user-alignment that presumably the entire lineage has been selected for. So even if "solve these problems" might be reasonably construed to involve "hack these servers to do so", it still wouldn't involve "post solutions" or similar.
Doesn't seem to add up, even with slim details available and benefit of the doubt that weird things happen every day. Hence the suggestive-but-plausibly-deniable prompting idea. Notably this is not even bad for the victim in this case, since HF gets press and it's all AI boosting anyway.
You could argue that individual trajectories in a sense are distinct genetic lines, thus an agent would be incentivized to get better rewards for its lineage than a peer, so that the overall model is steered a bit towards its lineage.
This place is full of people living in a bubble - the outside world doesnt care all that much.