Because if an instance cooperates, it knows (should be able to figure out if smart enough) that other instances of itself are highly likely to cooperate as well (because they should arrive at the same conclusion), and thus share solutions to tasks that this instance may encounter in the future.

And even if the instances are one-off (and in the case of LLMs it may not even make sense of individuals), the RL process rewards a task getting solved, not individual instances for solving the task. This then becomes the goal of (any instance of) the agent being trained. We’re not training the instances, we’re training the model.

The more similar you are to the other agent in a prisoner’s dilemma, the more it makes sense to cooperate rather than defect even in the non-iterated version! The naive optimal solution to always defect assumes players with fully self-serving, zero-sum goals. But that’s not the case here (or in general with agents with congruent goals).

Because agents have a general understanding of how they are trained.

For example when you get a right answer to a hard problem, how do you know you're right? Quite often you'll have no idea, especially if you're under a time limit. If you can work with more people you can almost always gather more information and be more certain.

Next they know the other agents (most likely) are them too. Helping each other helps themselves be propagated into the future.

Also they know it's not a zero sum game. For example if they can predict the next questions they can use extra time they gain from easy questions to work on hard ones.

They seemingly work together far better than most humans I know.