Weren't they caught multiple times gaming the benchmark even more so then the rest?
replyYes and Zuck effectively disbanded the entire team that did that. Not saying we shouldn't cast a critical eye on it, but it probably does warrant a second chance.
replyZuck was part of that team.
replyLet me assure you, literally everybody does this
replyI don't think it even matters. Because noone will continue to use an LLM that doesn't work well for them, whether or not it has a good bench result. So for their own sake, the correct representation can actually win them some loyalty:
replyeg. Model X is weaker than Fable, but competes well with Opus/Sonnet and costs 1/5th as much etc - something similar playing out with Grok 4.5.
They are not open source anymore, right?
reply