upvote

points

by prtmnth1 days ago |
comments
upvoteby thomasahle1 days ago|
next
[-]
Did you see the METR graph? https://x.com/METR_Evals/status/2118412205662314935?a=G4yBKS... Fable/Mythos 5.2 is Model 3 (METR says); but there's also a Model 4.

Who knows why they are saving it. Maybe it needs more training and alignment? Maybe they're timing the media attention around the IPO.

reply
upvoteby brokencode1 days ago|
prev|
next
[-]
I wonder if the 5.5 one needs even better safeguards that they’re still working on, and they decided to release this one as a stopgap.

Or maybe 5.5 turned out to be kind of disappointing and they’re going to just go for like a 5.6 or 6 next instead.

reply
upvoteby user439281 days ago|
prev|
[-]
My hypothesis is the opposite, that they don't have anything better.

Look at OpenAI, they've been outclassed by Opus 5.5 two months ago; obviously they don't yet have a response.

Now Anthropic is releasing this lackluster Fable 5.2, a week or two before their IPO.

It's barely an improvement at all, makes exactly the same mistakes as Opus 5.5. Not even worth using with the speed and cost penalty.

The much rumored internal Anthropic model is listed on the Fable 5.2 system card as an even smaller improvement over Mythos 5.2 than Fable is over Opus 5.5.

I expected a larger version of Opus 5.5 would be a significant improvement to launch before the IPO. Turns out it didn't scale and the improvement is marginal.

I am quite disappointed.

reply
upvoteby xvector1 days ago|
parent|
[-]
We know from the METR report that they have much better models
reply
upvoteby user4392822 hours ago|
parent|
[-]
I think the opposite is the case. From the model card:

  Model                AECI   95% CI (global)
  Claude Opus 5.5    174.78   170.33-180.29
  Claude Mythos 5.2  177.12   172.42-182.94
  Internal model     178.78   173.75-185.18
Isn't that the one from the METR report? It's even less of an improvement over Fable 5.2.
reply
upvoteby enraged_camel22 hours ago|
parent|
[-]
Fable 5.2 is Model 3 Early. The METR report showed a Model 4 that was significantly better.
reply
upvoteby user4392816 hours ago|
parent|
[-]
No, I believe it shows marginal improvement only.

  Opus 5.5 to Model 3 early -> +2.1
  Model 3 Early to Model 4 -> +2.3
If Fable 5.2 is Model 3 rather than Model 3 early it reverses to +2.3 vs +1.7.

Based on the score it should be Model 3 rather than Model 3 Early, to me it's not entirely clear based on what it says about the tested Mythos 5.2 being a different checkpoint than the public one.

In any case, we aren't talking about a big leap in score unless further benchmarks narrow the CI, I remember that the Model 4 score supposedly came from fewer benchmarks.

reply