These types of tests are kind of moot as agentic harnesses are taking over.
replyIMHO an Ai is the llm plus it's harness.
A good harness would allow the llm to investigate on a map.
Just like the llm can use a python script to figure out how many r's there are in strawberry.
These tests are simply not that predictable of performance of the llm.
The test here is not how close the state is to Africa, the test is coming up with a question that is hard for other AIs to answer.
replyAre you giving it your API for these other AIs to evaluate their responses? This 'test' seems perverse.
replyI don't understand the question.
replyThe other AIs don't see the question until they are asked to react.