Yeah I remember when the original post about this came out. Def not recent. Though I think their point survives in that they didn't exactly RL on this.
replyI don't think age of the puzzle even matters, all models have search capacities these days
reply> all models have search capacities these days
replyone would hope that they disable websearch and internet access (maybe all tools?) when doing generalization testing?
Is search part of the model or the harness?
replyMaybe that's a rhetorical question but just in case - the search would always be part of the harness. A model is only handling next-token prediction for a given input. That token may be something like [[web search]] to invoke a tool call but the actual call would be handled by the harness.
reply