This is a factor in favor of stability/security of software, but there are many others against:
- software (code) changes all the time, so there are windows of opportunity during which a bug is exploitable; in addition to that, a bug may take a relatively long time to be fixed
- a model used for attack may be stronger than the model used for defense, both in terms of model quality and compute allocated
- with software complexity increasing (and team/companies behind projects getting bigger), the margin for mistakes grows thinner, and introducing misconfigurations or weaknesses becomes exponentially easier (with "exponentially", I mean literally, because the interdependence of the components, both technical and human)
And last but not least: in general, attackers are more skilled than defenders; in best case, defenders are well-trained. And the idea of having the population of potential skilled attackers growing is very unsettling.
I could see it going either way.
If it requires a lot of compute and trying, this is something that could be provided for common software.
So the whole thing is forcing the good guys to outspend on tokens to preemptively defend against the risk of the bad guys outspending them on tokens, rather than buying tokens to actually add features to the product etc.
So are they creating a market for the solution by helping create the problem? A kind of rent-seeking AI security-industrial complex!!
White hats are constrained by needing to pay for their own tokens, only using (expensive) vendors who meet governance and risk requirements etc. Black hats are free to take over accounts and steal services from wherever they can.
The path to substantial profitability for Anthropic is questionable. The Chinese LLMs threaten them by far the most of the three major US LLMs. The money for Anthropic is certainly not in $20-$200 subscriptions. And they don't have anywhere near the consumer potential that GPT does, in terms of unleashing an ad spigot. So how far will the API money scale while being undercut by China.
OpenAI has to fight with Google for the ad business, they're specifically building Gemini to focus on consumer + search. Anthropic's business looks cute next to Google's search ad business (which is entirely at risk in this inflection). Meta looks like the biggest potential loser right now, ad dollars will be sucked out of the rotting Facebook network (not Instagram) and redirected to the rapidly expanding, hyper rich context LLM interaction. Advertising on Facebook will feel like running dumb banner ads on Excite in a few years, compared to what GPT will know about its users.
People that think Chinese LLMs are a general threat, don't understand consumer destination services, which is what GPT's future is. China currently has nothing to threaten with in that realm. There is half a trillion dollars of advertising up for grabs.
This is just not true, building an effective advertising platform costs significant amounts of money, time and people.
Remember that you need to hire a sales force for this, and sales scales linearly rather than sub-linearly like engineering.
Additionally, you need to spend a lot of money dealing with fraud, fake and malicious ads.
Furthermore, you need to figure out where to put the ads and how to rank them.
Finally, advertising is a zero sum game (given that the internet has already killed lots of print & OOH advertising), so the only way to win is to better better/cheaper (preferably both) than Google/Meta/Amazon. Best of luck with that (although to be fair to OpenAI they did hire Fidji who knows a lot of this stuff from her time at Facebook).
They don't have a Sheryl Sandberg type figure, and she was also really important in selling FB ads to large advertisers.
Just looking at their leadership team I don't see anyone with a background in (successful) ads companies, so I'm pretty sceptical that they can build this out quickly enough to matter.
This is one reason
> and trying
and this is the other.
As a matter of basic logic, there will never be a time when it will be known that there are no bugs.
I’m in the “glorified spell checker” camp, although I don’t mean to reduce their impressive utility and belittle them in the way many people read that term and infer.
So I am not sure that an llm “justifies” anything. I mean that their “thinking” text talks about justifications but it is just a very advanced statistical regurgitation of the kind of text humans use. I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc).
What you really have is a model that tries the statistically most probable thing to say next and so on and what is really cool is how effective this is at generating a path that we can slap a narrative over afterwards that makes the whole thing feel motivated and consistent, like the model started off knowing how it was going to get to the destination.
Which is, under the hood, a completely different kind of “intelligence” as the supercomputer in War Games.
Likewise, the fact that LLMs are a stochastic autoregressive process (which is a class of systems every bit as rich as the ODEs used to model neurons) tells us nothing a priori.
If I give an LLM to compact its context window, so the context it carries can evolve iteratively over time as more and more things come in, is that enough?
Compacting the context is really a very, very interesting example here. The "next token predictor" is telling an external tool to change all "previous" tokens. So an LLM + a harness that allows compacting the context is no longer just a token predictor at all!
You don't need continuous learning to get interesting dynamics. You just need feedback loops.
They're not comparable.
LLMs are language model, and nowhere in their code you can find actual reasoning. Re-reinforcement is not magical process that builds conscience or emotions.
We are talking about probability built on statistics, with extea steps.
Stop humanizing LLMs.
You can't find actual reasoning in a brain either. (Note that you can't tell the difference between a conscious brain and a comatose brain by examining them.) This is the same as Leibniz's mill argument ... it's a fallacy of composition.
> Re-reinforcement is not magical process that builds conscience or emotions.
They aren't the result of magic at all, but we are nowhere near the point of identifying what processes do or don't produce consciousness (or a conscience) or can be characterized as having emotions.
> Stop humanizing LLMs.
That's a clearly dishonest mischaracterization of the GP.
I've read some of your other comments about LLMs and I find them unreasonably reductionistic, whereas I think the word "just" should be banned from ontological discussion, so I don't think further engagement would be beneficial and I won't be engaging in it. (And I'm actually quite conservative in ascribing cognitive traits to LLMs or other "AI".)
While an agent doesn't necessarily have to be powered by an LLM, most modern AI agents are.
You pointing at a human brain does not change that an AI agent is not intelligent and cannot think, we are still talking about probability built on statistics with extra steps.
I am not trying to be dishonest, we should stop making analogies between AI and actual thinking, because they are two entire different concepts.
Who developed these technologies used the words "thinking" and "reasoning", this does not mean they are actually thinking and reasoning. Somewhere you still have a processor calculating, with no empathy.
So, again: stop humanizing AI. This sentence shouldn't make you angry.
Humans forget stuff all the time anyway. Would you give them the same diagnosis?
Btw, what you describe about 'the most probably next token' would be true for a model that only went through pre-training where they only train on exactly that task.
But there's a lot of re-inforcement learning afterwards.
That just shifts the distribution of tokens produced. Ultimately they are still just next token predictors.
Like, even "reasoning" models basically work by generating more tokens at inference time, and using them to shift the distribution towards more useful outcomes (in some cases).
Just like humans produce one word after another when they talk, but they don't generally try to imitate other humans.
I think you mean task oriented, because they're still generally terrible at goal oriented activities except in those domains where the goal can be reduced to a familiar, explicitly practiced task or pattern.