undefined

points

[-]

At what point does this become false advertising, or even fraud? Imagine paying AWS for a high-performance server, but 5% of customers who order one quietly get a slower, cheaper server and still pay the full price. That's a serious claim and makes me skeptical Anthropic is doing something like that without more evidence.

by kuboble1 days ago|

parent|

[-]

Actually, as a person who rented a dedicated server or two (not from aws) this exact thing happens quite often.

For example a provider quietly underclocks your cpu or puts it is an energy saving plan hoping you won't notice.

by cvwright1 days ago|

parent|

[-]

Or your "dedicated" CPUs on your VM disappear for minutes at a time, or just process a tiny number of instructions

by apt-apt-apt-apt1 days ago|

parent|

prev|

[-]

I think they could easily reduce several parameters to slightly reduce cost (something like "it's the same model, a few less passes, but quality is still 95%+").

Or switch to a faster, cheaper cache mechanism or something that's almost as good, but much cheaper to run, in the name of improving efficiency and keeping up with demand. This switch to cheaper mechanisms is what it feels like to me.

by eggdaft1 days ago|

parent|

prev|

[-]

It's a problem for us. When they nerf the model our productivity really drops (due to the specific use case and process we have). We've started using OpenAI and will gradually use it more and more if this continues.

by cft1 days ago|

prev|

[-]

The degradation only happens to some accounts but not to all.

https://www.reddit.com/r/ClaudeCode/s/64DXRERLyU

It would be nice to know how they select. I max out my $200 subscription every week and I was selected. On Saturday, the Opus on my account that happened to be supervising a running live process, became unusable.

by tmpz221 days ago|

parent|

[-]

It would be trivial for them to opt-out accounts that are doing benchmarking from a planned regression.

I really really really do not trust companies on the verge of IPOing from goosing their numbers. Even if the lawyers are up their ass, the social pressures are more than enough to compel the behavior. For every good person willing to do the right thing even at the cost of a delayed career progression, there are 20 people willing to do anything not even for a dollar but for an "atta boy!" from the boss.

by Wowfunhappy1 days ago|

parent|

[-]

If it's a private benchmark, how would they know it's a benchmark?

by krackers1 days ago|

parent|

[-]

Look up eval awareness. There are all sorts of subtle signals an LLM could use to know if something is shaped like a benchmark or not. But that's not a bad thing, every animal behaves differently in play vs real-life.

by Wowfunhappy1 days ago|

parent|

[-]

Yes the model can do that, but it would not be "trivial" (as GGP had said) for Anthropic to opt out accounts performing private benchmarks from this theoretical model downgrade people think is happening.

The most plausible approach would be for Anthropic to train some separate classifiers to detect benchmarks. I'm not saying it's completely impossible but IMO this is conspiracy theory levels of effort.

by tmpz221 days ago|

parent|

prev|

[-]

In the context of public relations they would not need to ID every benchmark only the most credible or most public benchmarks many of which are openly available on github and whose problem-set could be fingerprinted.

You're absolutely right that it is a game of whack-a-mole and eventually we may have leading private + credible + popular benchmarks that ultimately dissuade the behavior, if it can be dissuaded.

by cft1 days ago|

parent|

prev|

[-]

That's a major reason to use local models. Or open weights models on a trusted cloud provider. Even if they are way less capable, they're more predictable