Enough! I think it's a reasonably fair test of the models. Let's put it this way, I use the same process to resolve problems outside my peak expertise, and I find it works well.
replyI think the solution is easy enough that any competent colleague would come up with it. Some variant on it has probably even been published (hence why I'm not chomping at the bit to publish). I think it comes down to the models being extremely spiky, and the problems they're trying to solve being easily verifiable in code.
Have you ever told the AI your answer? I bet if you did it would train on it and know it going forward.
replyNah. And I have an enterprise plan, so I'm /relatively/ confident that they aren't going to do that.
replyWith my personal little test problem, that never made a difference.
replyHowever current frontier models got good enough at websearch and thinking through the problem, that they can finally piece together the answer from my own public writing (eg on stackoverflow). And recently someone wrote up an answer to the problem and put it on Arxiv.