This guess/explanation predicts that most users won't match what the initial social media hype claims, doesn't discount user experience as simple habituation nor does it assume companies are lying when they say there have been no changes to the model itself (quantization included).
I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria.
Personally, I can detect a separation between Sol and Astra (but not as large as that between Opus and Fable). While they can solve most of the same problems, Astra takes less time, is less frustrating to talk to, is cleaner, notices more, spins wheels less and requires less corrections.
IMO this is 90% of it (as someone who has a bit of a different interaction style and runs these things less autonomously, and hasn't generally seen the claimed regressions). Day 1: throw new stuff at it that failed badly, exciting to see something make more progress! Day n: reality sets in that it still wasn't perfect the first time.
So it may be a widespread hallucination. But there's no evidence of that either.
If they have a "hey we're being benchmarked" mode, which is not hard to imagine, avoiding tripping it is going to be annoying and difficult to prove.
> "Demand for Astra is really unprecedented. We're pulling all the levers possible to sustain the demand, but I've not seen anything like it until now and we went through very steep growth before. Priority will always be to keep excellent service for existing users, but we might have to pause new Pro subscriptions for a bit if this continues."> We've made some improvements that improve usage on the long tail for power users of Astra when logged in with your ChatGPT account.
> No change in quality and a pure win that on the long tail can result in up to 3-4X less usage being drawn from the subscription.
https://x.com/thsottiaux/status/2096717905614524491 (https://xcancel.com/thsottiaux/status/2096717905614524491)
You can't fool everybody all of the time, but you can fool almost everybody most of the time.
But most of all, it's easy to fool yourself.
Point is, I really don't buy all the stories about a model suddenly being downgraded without at least a modicum of substance. People are grasping at straws in the noise.
Then after a few days you notice the prompts that it does badly on that the old ones did fine with and everyone is convinced there's a regression when it's just a different part of prompt space
It prioritises getting something working over making something good during the 1-shot phase and outputs maximum slop.
I’m working on hard things, it is very noticeable when it is hums through something and then falls over on something it should not
I can tell by analyzing my own prompts to look at when I get frustrated ;)
It can very well be Sol, no? What stops them from using cheaper model for some requests during "rush" hours or simply use cheaper model for every Nth request.
That would trigger a full prefill (context recompute) every Nth request because cached tokens aren't interchangeable between models, and that would require way more compute than just staying on Astra.
To avoid full recompute, you could prefill a cheaper model's context incrementally by always feeding it Astra's outputs in the background (and vice versa), but then that would require 1.5-2 more VRAM for each session + the complexity of keeping them in sync.
If the rumors are true that Astra is a looped transformer, a more practical approach would be to dynamically adjust the loop count during peak hours.
Alternative theory - it always seems amazing when it first comes out then the novelty wears off and we’re just meh about it. New model is a model is a model. I bought a PS5 Pro and was genuinely blown away by it at first…few weeks later I’m just like…eh it looks pretty good I guess? It’s still the same, I’m just used to it now and the wow factor along a new thing is going. Kinda like that.
Or they are just compute constrained so they have to serve a shittier version. Who knows?
I hate how opaque these companies are. It feels deceptive and evil.