I thought I was an OpenAI fanboy, but version 5.6 isn’t for me. Sol Ultra just keeps working and checking, and working and checking again, but it can’t even correct minor errors that aren’t a problem for 5.5 xhigh. I’ve rolled Codex back to 5.5 for now.
replyThe problem is they nerfed 5.5 about a month ago. The change was immediately visible to me: context compacting started to happen about 3x as frequently.
replyI think 5.6 is still not even close to 5.5 xhigh pre-nerf.
Yes, this is exactly why I said yesterday that OpenAI does benchmaxxing, seemingly quite a bit [1]. I got a flurry of downvotes for it at first, but I think people came around to it once they tried the model like you did.
replyUltimately I think the issue is that OpenAI is under tremendous pressure to perform, but GPT-6 is not ready yet, so they had to push GPT-5 to its limits, and the only way they could do it was with really heavy RLHF, which has its shortcomings. Like, it is super obvious that Sol, Terra and Luna are all heavily biased towards working on a problem relentlessly because that's what their reward functions emphasized. That pushes up their scores in some benchmarks but does not translate to actual intelligence and capability.