I think it could indicate that "semi-private" dataset likely leaked to their training data.
replyA dataset being as popular as their's is will contaminate the data just by people discussing it and creating their own public test sets of similar problems.
replyStill, probably not that much compared to employees targeting it.
It says "Provider Adapter" so presumably they put some manual work in to make this work.
replyARC has their own writeup on the result, which offers some nuance. https://arcprize.org/blog/astra
replytl;dr it's 62% when apples-to-apples to other models, which is still notable.
ARC's harness is just straight up broken. No serious harness removes reasoning context between each step. Not only does this significantly lower performance over all reasoning LLMs, but it also increase cost as you destroy the cache on every turn. Tossing the oldest entry when context fills up instead of using compaction is equally bad with the same issues.
replyWoah, that is a crazy interesting read!
replyThe no-reasoning version scores 35% while the low reasoning one scores 17%? What?
replyI suspect this is "no reasoning set" which might be "default: medium" or perhaps some smart routing. I don't think it's literally "no reasoning".
replyIt's simulating the Dunning-Kruger effect.
replyLook at those costs!
replyRight?
replyBetween $18k-40k to run a benchmark.
But scored less on V2 and V1 ... too much overfitting?
replyhttps://mvakde.github.io/blog/44-on-arc-1/ makes a good case that all the performance on the arc agi tests is overfitting, based on the fact that v1 performance did not translate directly to v2 performance
replysaturated before (higher degree) AGI-2
reply