Those people haven't verified their results against the private set: https://arcprize.org/leaderboard
replyAstra also not verified using private set, but on "semi-private" set
replyif that is true then why is astra on the official ARC leaderboard now ?
replyARC leaderboard has results from semi-private data for frontier models, they have another competition for private data.
replyIt is described in their methodology: https://arcprize.org/policy
It makes sense, since once OpenAI API receive task, it is not private anymore but leaked to OpenAI.
Where are results for private data?
replyWhich LLMs participate on private set? Open weight LLMs only?
Yes, they run competitions once a year amongst open weight models
reply