So I guess it depends on how much the latest-greatest model motivates people, and my read on the current churn is that developers are extremely unloyal to brand at this point and will jump to whoever has the best model. And as long as the best model is running on programmable GPUs, that will be the dominant form.
Custom AI for things like facial recognition in cameras has existed for decades, before LLMs were a thing. I don't see that getting replaced. And on-device conversational intelligence might go that route as well, we'll have to wait and see. It's a lot of silicon to dedicated to a static non-changing thing. My money would be on programmable TPU-like things (Apple's NPU kind of stuff). It just seems more flexible to have an array of compute that you can load different models into, so you can update it, etc.
What about all the things you don't currently use an LLM for?
If a specialized chip can run a model 100 times faster, you can suddenly use it for a lot of things at sub-second latency. You can write "make white transparent and add a red outline to x.png" instead of the corresponding imagemagick invocation and perceive little to no latency difference. You can hook it up to your browser and have it yank out all advertisements live, or tell it to highlight anything that might interest you, again, live. There's probably thousands of latent use cases nobody has thought of that would be enabled by a truly fast LLM, even a mediocre one.
I don't think an on-device model needs to change much; it's already quite general in its capabilities.
Looking for a lamp in a specific style?
"Make a VR application set inside my apartment (based on all the photos of my apartment from my photos directory) with all lamps under $100 that fit Scandinavian interiors and could be delivered to my house before Friday. Place the lamp on the dining room table, allow us to: cycle through lamps, change time of day, and interact with all the lamps and furniture".
Five seconds later and bam you have this new piece of software that you'll use once and then dispose of.
AI: "I'm sorry, a security guardrail prevents me from performing this operation."
Your company will decide what makes economical sense.
If you can have only one AI processor in your laptop (because they're big and expensive), it's going to be a GPU. This AI processor needs to inference LLMs, audio processing, image generation, video generation, etc. This is on top of normal graphics processing requirements such as video games, playing videos, decoding, encoding, etc.
At the enterprise level, I can see some ASICs working once the market fully matures and improvements in architectures slow down drastically while demand for inference increases drastically. How far are we from this world? Maybe 5-10 years? It seems like model architectures are still changing rapidly and labs want fast experimentation that programmable GPUs offer.
GPUs will still dominate in general - just like how CPUs still dominate despite ASICs.
They are fast, but they're still programmable accelerators, not a model burned into the gates.
If they designed this right, it means that once they have a model, so long as they keep the hyperparameters fixed they can change the weights much faster than it takes to spin up a completely new chip, essentially at a cost of doing a minor revision.
It takes about 18 months to go through the design, verification, and manufacturing process if you move at breakneck pace. Design could probably be sped up.
About 18 months ago the top model was GPT-4o. Not great by today's standards, but still good enough for many tasks (certainly a big chunk of chatbot queries). The current SOTA covers far more use cases, but importantly at a level that surpasses many thresholds of utility.
Maybe, but what is the shelf-life of that 18 month decision? Barely good enough today, when it launches, starts to get worse and worse every month going forward. You have to recuperate that investment on your depreciating asset.
And you are competing against anyone with the foresight to use a TPU instead, and the benefit from any new paper that finds how to distill, quantize or whatever better so their solution gets algorithmic boosts while you are locked in.
One could also imagine hybrid models, where part of the model is burned into ASICs and part of the model exists in VRAM/HBM2 so it can be updated.
I don't have enough low-level knowledge to evaluate the technical or economic feasibility of the above ideas, however.
Sometimes you need speed; sometimes you need quality. There are very different use cases for each. For my own workflows, I sometimes want something very simple done ASAP; other workflows need "subjective" reasoning and careful crafting of responses.
A model will often come up with worse results given more cycles of compute, only because it will tailspin from second guesses, rethinking and literal flip-flopping on concepts.
--- edit: to those following the thread below... if you look at the comments from the account replying, it's pretty obviously a pro-China account and all replies are antagonistic against anything other than a total submission to the Chinese state. My responses are intentionally antagonistic as every point I've brought up is completely ignored in favor of insults, so yeah, I've been insulting back.
If you let a model spin too much, you get worse results... that's a fact, even for the best models. Nothing you've stated since actually refutes that and acting like a paid bot doesn't mean anything.
I mean, if you want to felate Xi Jinping, have at it... that's all on you bro. I don't have a horse in this race.
You're just an idiot who assumes everyone that doesn't agree with you is the same. I'm surprised you're able to reply with Xi's penis in your mouth... even if it is kind of spacious in there for his limited size. How is the poo bear these days? Does he like it when you gargle his testicles in your mouth?
Yesterday there's a news on a breakthrough for probabilistic computer with 1 million p-bits [1].
Since LLM is stochastic in nature, this type of new computer can be much better than ASIC for processing LLM data.
[1] Biggest Probabilistic Computer Turns Noise into Answers:
This is not like a bayes model or something were it's distributions all the way down.
Assuming such work was happening.
Many AI uses are not that volatile. I had a 20 minute conversation today with some company's AI phone assistant. It was extremely good and would have been very helpful if any of the dozen people it tried to route me to would have picked up their phone. That AI won't need to be upgraded for a very long time. There is no reason for it to have a cloud brain except to force a recurring revenue for the company selling it.
Hardcore gamers are constantly throwing down insane money on the latest hardware. The rest of us can get by for a couple years with whatever we bought when the last one broke. Yeah, it's not the latest, but it gets the job done. I wonder if AI has not already reached the point where a gen 10 CPU--uh, I mean a v3 AI model--will get the job done for the next year. If I really need the up-to-the-second latest abilities for a minute, I can fallback to a cloud brain @ 1M tokens/$. Why pay a monthly lease on a 5-year plan for a 4-door Ford Ranger as your daily commuter? Buy a Clio and rent an F-250 twice a year when you need the hauling/towing capabilities.
Imagine if Anthropic could give effectively unlimited access to Sonnet, for $20. Wouldn’t that be an appealing option for many users? I know I’d make a lot of use of it for agentic tasks, office work, summarization, etc; when right now I’d save quota for more important tasks.
It is a bit like saying "why would you hire someone with a doctorate when you could get unlimited high school grads". How appealing that sounds depends on your needs.
The ability levels of the cheap models are encroaching on the abilities of the frontier models faster than frontier models are expanding their abilities. If we haven't already, we will very soon reach a "good enough" state where having the "best" model matters less and less and less.
By analogy, if you buy a new computer, do you get the absolute fastest CPU available? Maybe, depending on your workload. But if you're 90% of the population, you get the cheapest one that has enough power to meet your expected workload, which is mid-range, not top of the line.
Many of the responses seem to say "but there is a market for cheap models".
There is a world where we have cheap models and we don't have ASICs powering them. Things like TPUs and NPUs, which are programmable, are likely to fill that role. They are optimized for inference while also allowing different (and updated) models to run on them.
Given two companies competing on the cheap end. First company goes TPU, second goes ASIC: who wins? My bet is on TPU since they can update their model, even if their hardware is slightly more expensive and slightly slower, since the optionality of new models beats the performance gap. That may not hold forever but given the pace and volatility of the current LLM market, I believe it will hold for some time.
Models like Kimi 3, GLM 5.2, or even Fable 5 for that matter are reasonable to burn to.ASIC because they are over the threshold of "good enough to be generally useful", something that will continue to be true in the future.
Most people do not need the latest model, they need a sufficient model. If I had Fable 5 on an ASIC, I imagine I would use that and ignore paying API rates for Fable 6.
Which is why my original post mentioned the volatility in models. We aren't just doing research on frontier, there is a huge amount of research on quantization, distillation, etc. that is changing the landscape at the low-end almost as much as it is changing on the frontier.
And it is also why I mention revealed preference. What feels sufficient / "good enough" today is a moving target. This isn't just a question of what you want, it is a question of what is economically viable for the entity that will be designing, manufacturing and marketing this ASIC of which you speak.
This is me personally. The calculus is different for other people. But I suspect Kimi 3 is pretty darn close to that tipping point for an awful lot of people.
The managers of the firm will.
They dont care that its faster unless it translates into the financials. They want lower costs, higher revenues - explain how it fits brudda.
There’s absolutely a place for a lifestyle subscription to a sonnet model that you could just use everywhere all the time.
[1] https://finance.yahoo.com/technology/ai/articles/google-plan...
A Fable 5 model running at 9,000 tokens/s on an ASIC rather than 150 tokens/s on electricity chugging Nvidia GPUs, or even giant SRAM Cerebras or Groq chips could be good enough to meet the majority of demand.
pretty stupid statement lmao
The incremental unlock of capability by ever increasing frontier model sizes will eventually reach diminishing returns.
I would argue tnference speed increases would actually unlock a different kind of more meaningful value for a wider audience.
640k was enough ... in 1981 ... almost fifty years later is 50,000 lower than a standard off the shelf PC now
The comment seems like the nerd equivalent of 6 7
The quote became famous because it's a failure to imagine that people would find new uses for computer memory if it became plentiful and cheap. In the quote Gates is not expecting people to come up with more demanding applications for computer memory (RAM) than the ones which were available in 1981.
Computer memory did in fact become plentiful and cheap after this, and computer programs became more complex and memory-hungry and today we wouldn't consider a 640kb an acceptable amount of ram for even the lowest-end device.
OP is repeating the quote about memory to indicate that they think that modern LLMs will be a similar resource. We should not expect that demand for LLM inference will stay flat, and that once everyone has cheap access to Fable level, we won't find new more demanding uses for it.
I get this reference!
Yes. The latest OpenAI and Anthropic models are terrible at planning roadtrips.
This is something I try to use them for frequently. They constantly get things completely wrong.
I’d say that about half of the stops they suggest fail to follow whatever filters I’ve asked for.
And this is one of the big things that seems to be missed in these discussions: There is no longer a universal linear trend of LLMs being 'better' each iteration. They are becoming more specialized, and ones that approach problems from a different angle (like Fable/Mythos) can appear breakthrough when first released, but we don't appear to be on a path that actually leads to general purpose hyper intelligence.
It was nonsense to draw in investment and justify an inflated valuation.
lmao
This is starting to look at a lot like Intel vs Arm from the last era.
The Fable & Mythos are starting to look like a giant Xeon, while the smaller lighter models are starting to look like a lot of tiny ARM chips which sip on power instead.
The risk is the same as what Intel had. There is a group who are pushing them to go bigger and with a resource no limit approach, who have a lot of dollars to push you that way.
Follow them and they lead you to a pile of money, but then you risk something like Apple Silicon happening.
Something which got better because of efficiency & continuous improvement, not neutered due to it.
The problem was that Intel stopped making things faster and better, and shifted to wearables and mobile chips instead of investing in their hard-tech strategy that had worked for decades.
Any value will come from the largest models, and those largest models are unlikely to ever run on consumer hardware within their window of relevancy.
Small models are good enough depending on your task. That's the point. A model you can run on your phone or laptop is an incredibly useful tool for a lot of problems even though it isn't the "best" theoretically possible model.
And firms will be kept in check with financials.
If your competitor starts using chinese models and delivers better earnings whilst you are spending more on american ones... hahaaha. Wait and see what happens.
You will be FIRED!
You can run a Moroccan chicken fragment on the cheap in homage to what you cannot run
There is already custom hardware see cerebras.
GPUs have a lot of slack there is at least one lab that had a (small 8b) model generate almost 3000 tokens per second on a MI300X for a talk, instead of the typical software stack that did maybe 100ish tokens per second.
High bandwidth flash storage is in the works, i.e hard drives with TBs of storage and over 1 TB per second of read speeds. Meaning that in a couple of years you may be able to buy a card with 40-90GBs of HBM and 4TB of HBF and run a 3T model locally at a reasonable speed for 10-20k as opposed to a cool mil.
There is no "may" here. You will see this.
It's always difficult to see it from the present, but we're not at some end stage in hardware development; we're still on the same curve our predecessors also couldn't see: they couldn't imagine that there would be high performance computers carried in our pockets, with staggering amounts of storage and compute, putting to shame the machines they filled rooms with.
It'll obviously be China, and they won't need the bleeding edge of lithography tech to make it happen. Every Chinese smartphone will have something like Sonnet 5, along your car (well, not those of us in the US, but we'll look longingly at pictures of them while we drive whatever the government decides we're allowed to drive in Fortress America).
Give it ten years and your smart litterbox from Temu will be running its own local model.
China won in cloud services? Nope, not even close. They had to clone AWS just to try to keep up.
China won in mobile? Nope. Although they're very competitive there.
China won in search? Nope. Baidu who?
China won in ecommerce? Nope. Their dominance is overwhelmingly domestic.
China won in software? Nope. Windows is US. Android is US. iOS is US. MacOS is US. Linux is US/Europe/Global. Look at the top 50 largest software companies.
China won in silicon? Nope. Look at the top 50 companies.
China won in .... India and Latin America can manufacture iPhones now.
But sure, China's the obvious winner this time. Good luck.
The comparison I've seen elsewhere is the old console systems with separate cartridges for games... I wouldn't want to be regularly swapping them, but if they came out in a form factor that didn't require me to shell out multiple-4 digit figures in upgrades just to use the next model, it'd easily be worth it for me.
I've already got a home lab, and it's specced to last, minus the GPU. I picked up a separate system for local llm experimentation, but I'm not likely to be upgrading it again. The value add is incredibly small compared to the cost. The real benefits are data privacy and never worrying about rate limits, and there's a price point beyond which an incremental improvement to the model doesn't justify upgrading the system.
Ultimately however, that just means that models will become dirt-cheap. The money will be made with applications built on top of the models.
As of today, that appears to be Google!
guess you are arguing that the models _now_ will
be durably useful enough to commit the time to
creating the ASIC?
Subjectively, it's hard to say if people will pay for "a model from 18 months ago, but REALLY FAST AND CHEAP"
The speed difference suggests some use cases that might narrow the performance gap. With a > 50x performance delta you have some headroom to play with.
You can do many many fast iterations of ye olde "Ralph loops." You can also jack up the reasoning/effort level. And you could probably do some combination of both, while still running really fast ie 10x the number of iterations at 2x reasoning/effort.
So I think a hypothetical "50x faster Opus 4.8, but burned into ASICs" could be pretty competitive against the frontier models from 2027, 2028, 2029, and maybe beyond?
What's fascinating is that we are pushing the state of the art of hardware at this point.
I don't understand how this works when the models are evolving so fast that your burned ASIC is outdated (or at least not top of the line) in a few weeks.
To me, it's like imagining if Sonnet 3 was burned into an ASIC 8 years ago and then never changed. It would still be revolutionary, and today we would have an entire ecosystem of tools and services built around it, likely surpassing some of our current workflows.
The frontier is a different beast, but it would likely mean competing on price.
I think that's the problem at the moment. A much better LLM that doesn't use my battery is 20ms and <10mb of data away.
Sure, an ASIC model will always be behind (12 months? 18 months?), and a hosted flagship will be significantly better. But as time marches on, would I use a flagship model from last year if it came on a PCI card and cost $1000? Without a shadow of a doubt.
Cerebras tries to get around this by keeping everything on cache SRAM as much as possible, which it burns directly to the chip wafer itself and physically places that SRAM directly next to the tiny compute unit that does the actual math.
An ideal setup (not sure how easy this is to achieve in practice), is the burn the weights of the model directly to the chip as a sort of ROM, the actual math operations as actual digital circuits, and have SRAM, or even something akin to naked registers to directly compute off inference batch data. Cuts out 2-3 layers of abstraction and indirection.
You still need some memory for the context, in flight answers, ... but not for the model weights and for the output of the intermediate layers. I found taalas demo here: https://chatjimmy.ai/
Scaling this up to 2.8 Trillion (350X increase), will certainly be challenging.
If I was younger and had the right background, I'd love to dive into attempting somethign like this
Last I saw they posted Deepseek R1 numbers in Feb of this year.
The challenge is rolling out a new one every 7-8 weeks as the weights change & cheap enough for a hyper scaler to afford to buy one and save enough on power over the next 8 weeks as a payoff.
In fact the superior models are irreducibly nothing but superior web services run from China.
Nobody needs "web services run from China" to use Chinese open weight models.
Likewise, for compute, is the ASIC somehow going to beat a systolic array? You can't have one circuit per weight: the die area and electrical fan-out would be insane. I'm not seeing how an ASIC specialized for a specific model would actually help much.
I mean, sure, we can build more specific accelerators, e.g. for softmax, but these work fine in the context of a programmable pipeline.
Yes, there are more exotic things out there, like optical matrix multiplication systems. Those are different. But above, aren't you talking about just doing conventional digital linear algebra, but with a model-specific set of circuits?
I can accept the idea of specializing a circuit for a specific model shape, but I'm not seeing a need to specialize a circuit for the weights inside the shape.