I think it's safe to assume every single model is benchmaxxed. I mean why wouldn't you include training data from common benchmarks? Not only is a higher score good marketing, but these benchmarks are supposed to represent common tasks so training for those should in principle improve performance across a wider range of tasks.
replyThere's an entire section in the linked post about measuring benchmaxxing, so I guess that probably says something.
replyOut of all the tech CEOs out there, Zuck would never lie to me or manipulate me. I know this in my heart.
replyThe thing about Zuckerberg is that although he's deeply odd he has no ability to conceal this.
replySam Altman is much more dangerous because he is occasionally able to come across as human.
All of them are doing all the benchmaxing they can most likely. Dumb not to when so many people care about those numbers and don’t have the opportunity to try every model out personally.
replyMuse Spark was generally good. I'd be surprised if this model felt benchmaxxed.
reply