To quote the people who make it:
reply> ARC-AGI-3 is an interactive reasoning benchmark which challenges AI agents to explore novel environments, acquire goals on the fly, build adaptable world models, and learn continuously.
This harness does nothing to actually accomplish those goals.
It's a clever trick, sure, but you aren't allowed to use a calculator on your basic algebra tests in school for a reason.
I don't think we got continuous learning here, but we very specifically got interim goal setting and custom world models; the thinking traces demonstrate this round trip of building a world model, mental or coded, then stopping when reality doesn't correlate, then hypothesizing and creating a new model.
reply