Evidence for g in LLMs: https://en.wikipedia.org/wiki/G_factor_(psychometrics)
replyUnsure, or as LLMs say, I’d gently push back on that. In my opinion, the jaggedness has increased. While this type of capability has improved, I still see longer running Astra agents making the assumptions that any employable person at any level wouldn't make. Also, generally, all models feel more robotic and less pleasant to talk to (subjective) while still lacking judgement.
replyI am very interested in benchmark correlations.
replyVals IOI, olympiad programming problems solved in C++, and WeirdML v3, unusual machine learning coding tasks also mirror StarSkirmish Bench.
I did a StarCraft: BW hillclimb experiment with GPT-6 Astra and Claude 5.5 Opus. Peter Gostev did a Chess hillclimb saw similar behavior as I did with GPT-6 Astra.
Might want to reach out to the author of the above benchmark. I think having a working group of multiple distinct benchmark owners might help quickly evaluate new model developments
replyIs this meaningful? Aren't frontier models MoE anyway? If you issue a StarCraft query they will probably just route to their StarCraft optimized model.
reply