> GPT-4(?) was capable of beating Pokemon 18 months ago but models only became capable of beating it without a harness in the last six months...?

GPT-4 was decidedly not capable of beating Pokemon 18 months ago. I doubt it would be able to complete a single level. I don't think people realize how large the advances in model capabilities have been. GPT-4 in a modern harness is absolutely horrendous compared to modern models.

These days calling to a server like openai or anthropic effectively results in a server side harness wrapping the weights.

I'm assuming you're referring to a harness that includes memory -- I generally think of the harness as anything beyond executing the generation loop, but I'm not an expert.

True as that may be, it may be better to optimize models for some amount of memory versus forcing some token count based on a reasoning level, right?

I don't think GPT-4 was ever used for beating pokemon with or without a harness. Successful attempts include Gemini 2.5 and Opus 4.7 both using relatively advanced custom harnesses that give access to game memory, notes systems, and one-off hacks to get around parts of the game the model gets stuck on. More recently, Fable 5 beat FireRed with a _very_ minimal harness (screenshots and button inputs). That's the only example I know of but that is a very sophisticated and very expensive model compared to GPT-4.

Most of this doesn't discredit your overall point, though.

The intent in forbidding harnesses was to prevent an ARC-AGI specific harness, which for example presented the game interface in a more agent-friendly way.

What NVIDIA has here is a generic "evolution" harness, which can be used for any problem.

I think it would be fair game to allow OpenClaw, Hermes, Codex, Grok Bot, this NVIDIA thing, to compete, as long as they don't have ARC-AGI specific skills, toolset.