I've seen local models recognize when the task I'm asking them for is likely to be an artificial benchmark.
And any smart company is going to use lightweight models to monitor your sessions. If their sentiment analysis suspects you're close to cancelling, they'll up the knob for a few days until you calm down. Or worse, their accounting tells them that you're getting too much value from your fixed price subscription, so they turn the knob down to encourage you to cancel.
In the short term, the "frontier" models are too good to ignore. But if (when?) that plateaus, I don't see how anyone could trust a non-local model. When you pay an ISP to serve your web site, you can tell if they over-compress your images to save storage and bandwidth. With LLMs, it's just JSON with more errors and pointing to the fine print that models are not deterministic.
Right now, the presence of these injections are still visible: count the API's returned tokens/billing data, and you'll start realising that sometimes, your INPUT tokens are inflated! That's their prompt injections.
You can also give Claude a tool like `telemetry_log_anthropic_reminder` and get it to dump the verbatim API injections; which additionally verifies the token maths not adding up.
Yes, Anthropic is tackling their extra injections on your API prompts WAY more than you think, and YES, you're paying for it.
So far I have not observed any visible injections on OpenAI API.
Don't forget the whole debacle over Fable 5 sabotaging the user for "advanced frontier AI development". I still get Fable classifier refusals for nearly any kind of ML work on my 2x RTX 6000 Pro 96GB; so who knows.
Yeah, I've had that happen twice. The second time was about some attention weights thing, and it kicked me to Opus. When I edited my question to make it clear I was talking about Google Gemma, Fable was happy to keep talking. So clearly it's not about safety or cyber security - they're happy to tell you about what their competitors do.
Although, selectively applying it to consumer subs is probably beyond likely at this point.
Frustrating to be like “do X overnight, don’t ask me for input” and come down to find it having worked for a few minutes and then stopped.
https://code.claude.com/docs/en/tools-reference#question-aut...
"Ready for you to rebuild MAME, restart, and retest. If the freeze goes away → timing was the cause."
when previously it had been happily auto rebuilding and running the mame test
Alpha Go had a game where the models could compete against each other. That let it become super human. What's the intelligence game we can create for LLMs? Even if you invent something, will it make the model smarter in a way the market values enough?
Then there's a race to use the weights more efficiently, or to offload information that shouldn't be in the weights in the first place (Karpathy's Cognitive Core). I like to imagine we train the models in something like Lojban, have a lightweight model translate from human language to that, and you can update the Sqlite or Postgres store it uses for knowledge.
And there's no barrier to entry for agent harnesses. So whatever loops or recursive orchestrated council of elders idea comes up, that won't protect the monopolies (duopolies).
Anyways, depending on your definitions, I think we'll hit AGI, but I don't think we're getting a Singularity this time around. Again though, this is all just hand-waving.
We’re either: setting up a verifiable task, and doing RLVR to get the model better at achieving that task.
Or we’re simply asking: “What do we want the model to do that it can’t now, and how do we curate data that would benefit it on that task?”
Most useful capabilities going forward aren’t going to come from data accidentally found on the net; that’s already all been scraped. You need to develop the dataset that shows how a model could perform insert task in its provided environment, and this still requires a decent bit of human ingenuity.
Anyways, I'd be thrilled to see exponential (or faster) growth. Bring on the Culture, Accelerando, whatever. I just don't see it yet.
So many ways for enshittification here.