undefined

points

[-]

Arc AGI are simple games, the hardness comes from the input being basically adversarial to LLM training. if you use an LLM scaffold that removes the adversarial part you are measuring something else.

the harness basically outsources the alien nature of what the LLM is asked to do to algorithms it writes. this would actually be impressive if you got it to do that for a much more complicated game than Arc.

with this harness the ARC AGI test becomes a test of whether or not the model can work out the transition rules in a very simple game.

by fnordpiglet1 hours ago|

parent|

[-]

A great deal of mathematics is transforming nonlinear problems into linear ones and solving them with linear techniques. Others are solving non linear problems through stochastic methods. In almost all cases most non trivial math is done by transforming a harder problem into a simpler one.

I get what you mean in terms of testing the model itself to see its improvement in some domain. However if you can transform the domain to be better adapted to the model and achieve the desired results, this is indeed an accomplishment because a whole domain of problems is shown to be practically feasible with this technique without expensive model improvements. Of course the benchmark still exists without the harness, but the harness also exists which allows these problems to be solved.

As noted elsewhere the models themselves were used to build the harness, which means the models can in fact score this scores without intervention but building a harness for themselves adapted to the domain and using it. Is this cheating by the goal posts you’re setting?

There’s a real tension between “I want to solve problems and this technique shows how to solve the problem domain,” and the “I want to measure how something performs unassisted with other techniques.” Fortunately it’s not a mutually exclusive situation. You can do both simultaneously, gain the benefit of the technique to transform the problem into something tractable and keep measuring using the benchmark.

by Tadpole91811 hours ago|

parent|

[-]

To quote the people who make it:

> ARC-AGI-3 is an interactive reasoning benchmark which challenges AI agents to explore novel environments, acquire goals on the fly, build adaptable world models, and learn continuously.

This harness does nothing to actually accomplish those goals.

It's a clever trick, sure, but you aren't allowed to use a calculator on your basic algebra tests in school for a reason.

by vessenes1 hours ago|

parent|

[-]

I don't think we got continuous learning here, but we very specifically got interim goal setting and custom world models; the thinking traces demonstrate this round trip of building a world model, mental or coded, then stopping when reality doesn't correlate, then hypothesizing and creating a new model.

by Tadpole91811 hours ago|

prev|

[-]

The point of Arc-AGI-3 is to measure model performance. We already know that models can one-shot and iterate on very rudimentary game implementations. And, naturally, once it effectively has a copy of the source code, it can use that to play the game better.

This harness is really moving the goalpost by defeating the entire point of the test. Instead of seeing the strength of a model's world view, its ability to internally derive and intuit rules, and its ability to keep track of game state over time, we're just letting the AI cheat. This is just the LLM equivalent of running a chess engine to the side.

And this harness would not work in a remotely complex game and relies on the fact that Arc-AGI-3 is a focused test that only made the games as complicated as they needed to be for current model performance.

by vessenes1 hours ago|

parent|

[-]

I think this is just too simplistic a take; Arc-AGI-1 was wide open to all models, harnesses, etc, and had quite a lot of innovative structures implemented by hobbyists. At the time, this was seen as a good thing (it was), because we don't know the best system architecture for all sorts of problems right now -> innovation is good.

The games are designed to allow assessment of a system. Knowing better systems to solve the games is a step forward. If any of the frontier labs could have one-shotted -3 in March with a custom harness, they would have done so.

by skeledrew1 hours ago|

parent|

prev|

[-]

Sounds like a distinction between sport and work. How useful is pure model performance if it's known that there are conditions in which even greater performance can be achieved on real tasks? How useful is it to know how fast/far a person can run if they can ride a bicycle or drive a vehicle?