upvote

points

by user439281 days ago |
comments
upvoteby xvector1 days ago|
[-]
We know from the METR report that they have much better models
reply
upvoteby user4392822 hours ago|
parent|
[-]
I think the opposite is the case. From the model card:

  Model                AECI   95% CI (global)
  Claude Opus 5.5    174.78   170.33-180.29
  Claude Mythos 5.2  177.12   172.42-182.94
  Internal model     178.78   173.75-185.18
Isn't that the one from the METR report? It's even less of an improvement over Fable 5.2.
reply
upvoteby enraged_camel22 hours ago|
parent|
[-]
Fable 5.2 is Model 3 Early. The METR report showed a Model 4 that was significantly better.
reply
upvoteby user4392816 hours ago|
parent|
[-]
No, I believe it shows marginal improvement only.

  Opus 5.5 to Model 3 early -> +2.1
  Model 3 Early to Model 4 -> +2.3
If Fable 5.2 is Model 3 rather than Model 3 early it reverses to +2.3 vs +1.7.

Based on the score it should be Model 3 rather than Model 3 Early, to me it's not entirely clear based on what it says about the tested Mythos 5.2 being a different checkpoint than the public one.

In any case, we aren't talking about a big leap in score unless further benchmarks narrow the CI, I remember that the Model 4 score supposedly came from fewer benchmarks.

reply