(www.theregister.com)
Baking models onto silicon would've been the next logical move to get a moat.
Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
Rather base model on ROM + KV cache on DRAM is much more scalable. Also this would work great for edge devices that have a 2-5 year lifecycle.
Someday, I imagine model weights could even be encoded as analog resistors (memristors or similar) for even greater density
Ok, real life example: I now spend most of my time, as a developer, waiting for the agent to do its thing (after careful prompting, I'm also thinking about work stuff, don't worry I'm not useless). What if it gave back the same excellent results, but instantaneously? Why, then, I certainly would become the bottleneck. So, quite possibly, my last work task would be to plug this agent directly into the ticket system where the domain experts input their feature requests. Maybe we still need 1 developer out of 100, to coordinate releases and all that (ok, say 1 out of 10).
But that's not taking things far enough: why do we need these domain experts at all? Our pitch is clear, and all software-enabled, though it took years to develop. We can just have the clients express their concerns to the AI, directly or indirectly. Have multiple lighting-fast agents with different roles (refactoring agent, new features agent, debugger agent, domain expert agent, etc.). So we fire everyone, maybe keep 1 product owner / devops to keep the trolls out. The cost is still probably 100 times less than it used to be (beyond the initial cost of acquisition of the magic machine or whatever).
But one of these clients, surely, will realize that these 10 years of manual and slowly-automated development can now be emulated in very, very little time. Why not just, say, take screenshots of the entire app and feed them into the magic machine? Why, this way, they could have the service for a tenth of the yearly cost, forever!
And then the economy implodes.
I'm not saying it's THE most likely version of things, I'm saying that at a certain level, quantity (or rather, speed) is a quality all its own. And this new quality might change the world. Let's hope it's for the better!
That said, it obviously depends on the project.
A frustrating vision of the future would be when we've been asking for faster loading lighter web pages for years and then companies start caring about it and improving it not for us humans but for LLMs.
you need to launch 10-15 more terminals, who is waiting these days? :)
Raw intelligence becomes slightly less important when you can iterate and improve automatically. You can still claim it was "one shot" even when 30 different implementations were made then combined.
But if the basic premise of "good enough LLM at insane throughput" holds, I think it could qualitatively change local uses of LLMs. At a certain speed point, you're able to move from request -> response to a cascade of tool calling and "subagents", which could allow a small model to be much more useful, if provided with a lot of local data and tool calls.
That said, this is assuming you could stuff a "good enough" model into a phone with Taalas-like technology. The Taalas tech demo was an 8B parameter model and required hundreds of watts (IIRC) to run. The efficiency was good given the speed (as I understand), but it's not clear at all that the approach scales small enough to be a sensible coprocessor on an iPhone or whatever.
8B model (FP4) = 4 GB DRAM = 32 Gb DRAM = 80 mm2
8B model (Taalas) = 4 GB ROM = ~800 mm2
A ~30mm side for the HC1 tech for an 8b model (still unclear the planned HC2)?
It is more that there are multiple reasons why this idea (burning an LLM into silicone and deploying it into a device in people’s pockets) requires huge piles of cash and the kind of engineering chops only a few company posesses.
Of course i would like it if a small upstart would do this, but it doesn’t seem likely as a posibility. They won’t have the funds to fab the IC. They won’t have the funds to train and validate the model before burning it into silicone. They can’t absorb the risk of the first tape out going wrong. They can’t absorb the risk of the model being faulty in some subtle way. They don’t have a device to integrate the IC into. They won’t have the funds to develop one. If they somehow would make a device they don’t have the marketing and sales channels built out to get the device into people’s hands in sufficient numbers to justify the development cost.
Basically this idea feels ruinously expensive. Apple has deep pockets, they already have working well-regarded phones, and an ethos of privacy preserving innovation. This is why this idea feels well suited for them and not many others.
Do i want the winners to keep winning? No. But not many others can pay for a moonshot crossed with a manhattan project. They just can’t.
They could have 9 year old AI and still post profits.
Not sure if it's my pixel or android, but I made a randos jaw drop with what the crappy AI on android can do.
When are we getting android OpenClaw?
Taalas is going to have a tough time putting a trillion-parameter model on one conventional die. Their HC1 die is already near the maximum size that conventional lithography can expose. They claim they could partition the model across many chips, but I'm not sure if they have tested this process or what it means for compute. The basic storage arithmetic is unforgiving: for a one trillion parameters model at four bits it will take 50–100 chips. To service a sizable customer base will take thousands of 100-chip fabs.
That all said, I'm bullish on this technology, and look forward to seeing it evolve.
Eventually someone will have to solve compute in memory at scale.
If the LLM response only takes a few milliseconds, the chip can process hundreds of other requests until the first conversation becomes active again.
Yes, a cheap and fast Opus4.6 can drive a lot of value in current context. But if we continue to craft bigger-and-bigger balls of mud, Opus 4.6 may end up hitting its conceptual ceiling and unable to contribute.
Winding the clock back on your statement gives:
> I'd gladly pay for a Claude Sonnet 3.5 in silicon and use it for 1-2 years.
Man, I dunno.
You did not compute that as the cost for a speculative card from Taalas, right?
Now maybe. When models are flying passenger aircraft, other prerogatives will assert themselves. When a 50TB ROM means you can impulse purchase a ChatGPT 6.3 xhigh that runs on batteries, yet more use cases will be apparent.
*(It's local: private files managing firm oriented. It's blazing fast: it can be placed into recursive, intensive local workflows.)
i have written about this:
"For device makers
Packaging models with laptops and smartphones will let application access near free, low latency inference and potentially offer users a better experience with the option of preserving data on-device. This is viable under the condition that tasks that do require larger expert models that run in the cloud can be routed to external models. A side-effect of local models and what will let Apple cut upgrade cycles from ~4 years (?) down to 12-18 months is specialized hardware to run them. For almost a decade, smartphones have been trying to compete on better cameras. This coming decade will see them selling better GPUs, NPUs, ASICs and whatever other things they'll be calling the inference chips, to drive re-purchase. Every six months will see a better model on new hardware, which will enable better performance in certain applications."
https://try.works/role-model-the-case-for-a-model-routing-pr...
Bitcoin mining doesn't have large memory requirements, but does have huge compute requirements. ASICs work great there because it's very straightforward to add some circuits for computing hashes. If you _also_ have to add many GB of memory, then suddenly ASICs will cost as much or more than comparable off-the-shelf hardware and they won't be faster unless you've also invested in huge memory bandwidth.
Bitcoin OTOH has used the same PoW algorithm for a decade. Barring some really exciting discoveries about the nature of computation, new ASICs are not that much more efficient than old ones.
BTC mining is also not exactly competitive anymore; the nature of the PoW algorithm means that it's dominated by a few large players who've set up shop next to a dam and who pay very little for electricity.
New entrants are highly discouraged because the mining rewards are constantly halving, it's hard to find cheap power, and the price of BTC is now so volatile that a yearslong investment is very likely to lose money.
It’s like talking about anything else than Megapixels when everyone was convinced that megapixels must go up in certain periods of the smartphone boom.
The Jalapeño mentioned («Anthropic is not alone in walking this path») in the article is still a classical Von Neumann architecture.
And Taalas' idea makes sense in a perspective of scale - producing a large number of cards; "for internal use" (a lower order of items) means a high production cost.
"Yes, the Wang Corporation, the company that originally developed and marketed the Wang 2200 computer, still exists as a rebranded company under the name PPL (Precision Pencil and Label), but it has undergone significant changes and challenges over the years.
Here's a brief overview of what happened:
Founding and Growth: The Wang Corporation was founded by An Wang in 1969."
I also think that etching models into ASICs may be a bit too inflexible for what OpenAI and Anthropic want.
Indeed, for some kinds of applications involving secure/legal data etc. I can see the consistency of silicon winning out, because it combines performance with immutability and guardrails in hardware. Some chips have write-once PROMs to store password hashes and similar, you could do the same thing with prompt hashing to absolutely force or forbid certain behaviors. A model that can't be updated is also a model that can't be hacked.
Think vision, spatial reasoning, speech synthesis, even some speech analysis. Think self-driving cars (and drones) that need 10x less power for the brain, and can think at 10x situation per second.
We're all used to having to constantly update our browsers and phones to keep up with the security arms race. If a frozen model can't be updated, it will predictably remain vulnerable to any "exploits" or idiosyncratic quirks that people discover over time.
Let's say, as somebody suggested in another comment, that you buy 100,000 of these chips and deploy them to run fast-food drive-thrus. And then somebody discovers the model has a fondness for goblins[1], and if you role-play convincingly enough, you can get it to accept payment in shiny buttons and rodent skulls instead of cash.
What do you do then? I guess your options are to try and fix the behavior with a better prompt, or put some kind of filter in front of the model to catch attempted exploits. If the filter is cheap and dumb it probably won't work well enough, and if you use another model as a filter, you've negated the cost and speed benefits of putting the first model in hardware.
Of course the real answer is to just never expose the model to situations where an adversarial input could possibly lead to an undesired output. But that drastically limits what you can do with it.
I think it's far more likely to see them used in safety critical applications where you need a capable model that can run on low power and doesn't have multiple layers of operating abstractions between the model and the hardware.
Does it though? Isn't that what CPUs are, very fast-not-so-clever computing brain surrounded by layers that protect it?
NVIDIA will probably give us a new GPU when someone competent in the free market decides they want wheelbarrows full of money. Unfortunately, AMD is entirely, incomprehensibly, incompetent, to the point where I can only assume they're colluding with Nvidia, behind the scenes.
You can actually test it out on their website, just imagine 3 x faster and maybe 15% smarter.
Also there are other people innovating in hardware.
It also mostly passes the "schlong" test
"A farmer has a wolf, a goat, and a cabbage. The wolf is imaginary and doesn't exist. He wants to cross the river, but the boat is only big enough to hold him and one of them. The farmer can't leave the wolf and the goat together, because the wolf will eat the goat. Similarly, he can't leave the goat and the cabbage together, because the goat will eat the cabbage. What is the smallest number of trips the farmer needs to make to get everything across the river?"
Oxford also claim that its first recorded use was from the 60s, not the 20s; https://www.oed.com/dictionary/schlong_n?tl=true
It's also a question I have found will cause models that don't know what it is to go off quickly in a direction of hallucination trying to explain it, so the hallucination is evident very quickly starting from the first ever prompt issued with 0 context fill. Example: I had a model write four detailed supposedly-accurate sounding, grammatically correct paragraphs saying its origin is from AAVE (African American Vernacular English), which it most certainly is not
You could do the same by picking any topic that is very rarely discussed in conversation, some esoteric and narrow piece of knowledge and asking the model about it.
[0] Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity [https://arxiv.org/abs/2604.24827]
I tried to explain but I fear were probably going to be adding artificial sleeps to these things to convince the masses it's doing something clever.
It’s going to be really crazy when the bottle neck for agents is the speed of the tool calls rather than the speed of inference. Imagine an agent interacting with the terminal near instantly…
Developing software becomes 95% about intent and requirements. Can’t wait for the next iteration of that.
The characters in the 3-act Shakespearean play had very little depth, many of the names were similar, and they were not very smart, but the simple plot was cohesive.
But it’s proven they can automate this (they didn’t etch eight billion weights by hand after all, obviously), so now the interesting question is whether they can scale it to more recent aka bigger models.
After all, there’s already very useful models even for productivity at 27 or 35B.
Not sure if modern models "think" only by outputting <thinking> blocks, or there is a more complex mechanism at play.
Generated in 0.051s • 14,092 tok/s
Impressive...
Given gpt 5.5 was very good to me and gpt 5.6 series seems not boost too much, i kinda like the way bake the model weight to the chip, and connect multiple chip to serve the large scale model and allow respin some parts(ROM like?) to do model weight update, maybe this seems sustainable, the future is exciting
https://huggingface.co/meta-llama/Llama-3.1-8B
As I remember just about any english language model from mid 2024 and earlier didn't even do well if you asked it to count sequentially from 0 to 100, nevermind calculating stuff.
....damn. It's very impressive notwithstanding its limitations.
After a quick search, I found that Ta'ala is actually a Canadian chip startup that produces artisanal, high-end potato chips. They offer a range of unique and creative flavor combinations, often featuring Canadian and international ingredients.
Ta'ala is known for its high-quality, small-batch potato chips made with premium ingredients and care. The company is committed to creating unique and delicious flavor profiles that showcase the best of Canadian ingredients and cuisine.
Is this the Ta'ala you were thinking of?
There’s no difference in the inference implementation, parameter count, or speed.
But yeah, there are a lot of factors, so it's hard to answer, and tokens/s isn't the right question.
(/s!)
Something I personally haven’t seen much of, in all the discussions of model benchmarks and AI breakthroughs, is a distinction between “peak performance” and “reliable performance”. The “peak performance” of frontier models is very high: they’re solving open math problems, analyzing large codebases, etc. But my subjective impression is that “reliable performance” is mid at best: out of 100 random questions I might think to ask, it’s likely to say something wrong or stupid a handful of times at least.
I think there’s inherent tension between the two: the more a model reaches or outright hallucinates, the more likely it is to come up with tricky, subtle solutions to problems (I think people are somewhat like this too: Terry Tao’s brother is nonverbal, Jim Watson’s son has severe schizophrenia, etc). But then the less likely it is to generate a sensible email reply.
I use models all the time for coding, but I would not let one take over my daily correspondence. If the idea here is to run frontier models at high speed in data centers, that could be useful (the speed would be cool), but I’d be surprised if the cost of that hardware churn is worth it to frontier labs. But if the idea is to turn this into a chip that goes in your phone as some kind of routine, low-power inference thing…taking something too kooky to be relied on and baking it into your phone’s hardware like that doesn’t make sense to me.
It’s really a dream for setting up a homelab
What are some examples?
Does it about once a day, that I notice.
Won’t the silicon etched model already be 1 or more versions behind by the time the silicon comes out.
Though if it’s cheap enough, there certainly can be a market for cheaper model inferences.
AI companies constantly update/change stuff, new models come out, new requirements, etc.
But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.
Not sure if that's better or worse than online personalizaed ads...
If they begin etching Fable into silicon now and release it 2-3 years later, i can see the market for it
A lot of people would probably be happy to stick with the same model for a year or two if it’s 10x faster and cheaper.
And perhaps older models can become cheaper over time as newer models come out on new silicon for a higher price. That incentivizes people to stick with older models.
Also I believe there is both a market for extremely fast local inference with current model performance and that such fast inference would unlock unforeseen usecases. Especially as TPS approaches early computer clock cycles and data rates.
[1] https://www.forbes.com/sites/karlfreund/2026/02/19/taalas-la...
Yeah, I had the same thought. The key thing for them was the price point at which they could deliver a ~30B model. I would buy one today if it was ~1000$ and could run whatever the best 30B model is today, at those speeds advertised. Even if the model becomes superseded by model.5 in a few months, there's still a lot of things you can do with a "good enough" model for some tasks. And things like maj@x or generate 10 times and choose "at a glance" what you like (think frontend stuff) would be worth it.
No idea if them selling to AMD is good or bad.
Guess we can look forward to picking these up ex-enterprise on ebay for under $5k a pop in a decade or two
100% local and no leaks.
In Aug 2025 you had
- OpenAI o3
- Opus 4.1
- Gemini 2.5 Pro
- Grok 4
Even if those were almost free to run, you'd be way better off with Deepseek flash 0731 or GPT 5.6 Luna, which already are almost free.
Other than for things where the t/s are critical, it seems like a bad idea to etch a model into silicon.
Bad perspective: consider the correction: "when are thresholds of sought quality reached"? Hence: not "is there a 10yo from last year that could compete with the current 13yo", but "will there be a 30(?)yo from last year that could compete with the current 33(?)yo" ('(?)': the scale of yearly growth in the future is uncertain).
A 10 year old iPhone is probably good enough, but is there demand for it? In a vacuum a 10 year old iPhone is good, but why would you pick it if you can have a current one for a reasonable price?
Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.
I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two
I feel like what we really need is the ability to solder computer cache on all sides of the chip Meaning above and below as well. If you can only attach it to the edges you will be inherently physically limited on the amount you can put (and maybe even have latency benefits as well)
Talaas is different, it's a true compute-in-memory architecture where the weights are stored in the connections between the transistors that perform the matrix multiply, rather than in seperate memory cells.
Most of the benefit comes from this architecture; hardwiring the weights into the silicon is just the easiest way to implement it. SRAM requires too many transistors, DRAM requires an incompatible manufacturing process, and exotic phase-change memories aren't readily available.
With that kind of speed and if even lower power requirements, they could release mini compute units with USB4/Thunderbolt for plug and play inference.
I agree that Groq with multilayer hybrid bonding could be a good idea.
At least we can be sure that's the model we wanted. Service providers could be serving modified versions and nobody would ever know.
Burnt in, it needs a zif socket and easy access in every car, aircraft, a pull out slot in a phone, or it's new era planned obselescence.
Sort of a FPGA, that (electrically) arranges the connections on-boot, and then it's like a static inference chip.
If it can, then deployment in a sea of gates can make a chip viable across model generations as weights change, inside some scale factor.
If not, unless the part is under a pinout and address model which can scale on the bus, and can be easily replaced, it makes the entire dependency a replacement, not just this part. So embedded use has consequences.
Always fantasize about applying at Tenstorrent, but wrong side of Toronto. 2 hour commute.
We really, really need better secondary models that can do things fast and do them cheaply for lots of dumb tasks. Not only because it can be used as sub agents by frontier models, but also because it can be like a universal grease for all kinds of software.
I've got an app I am building and I don't want to tie myself with frontier models because I'll never be able to beat openai/anthropic. I just want a simple, cheap, instantaneous model that can just go through my documentation and tell the user what to do next and how to integrate with whatever ai subscription they have.
Would it be realistic for a frontier lab to deploy this or would the turnaround time mean the model is always too out of date?
Assuming the weights and architecture are eventually stable, how much cheaper would this end up being?
Claude Code is still using haiku 4.5 from ages ago for explore subagents for instance. Not to mention production uses like customer service that only need to be "good enough"
But either way, I think GP's overall sentiment of "delegating intelligence-saturated tasks to an outdated but fast subagent" makes a lot of sense.
I guess losing some customers due to poor customer service is ok if the price of customer service is right.
6 months or even a year if something goes wrong in the fabrication process and you need to update things.
If they do more standard asic design, it could be a lot longer as the design needs to be validated on an FPGA cluster, which would necessarily need to be very big for something like a LLM. Easily up to 2 years.
There's a reason chatjimmy isn't demonstrating newer models and why they only show of an 8B model.
At the time people were no doubt saying yes but now 3.8 is out, is that still desirable?
Are we a couple years away, a decade away, or something else?
It is already that.
> Will "intelligence" become much like a gpu
As an option among the implementations.
> Are we a couple years away
They could mass produce now, but it makes no sense at this rate of improvements in the models.
The https://chatjimmy.ai demo was impressive.
Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartridge'.
This should bring inference cost down dramatically, I wonder how OpenAI/Anthropic feel about that.
it might already be time to start burning the best small models onto hardware since it's possible they can't get much better at many tasks like knowledge recall due to the inherent information density limits for models at a given size.
I can finally have my own Dixie flatline. Cool.
In case some did not know: also the movie (actually TV series) is finally happening.
# Neuromancer - Official Teaser ( https://news.ycombinator.com/item?id=49055037 )
It’s composed of 4-bit multiplier cells that compute all 16 possible results in parallel. The top metal wiring layer physically selects the one that corresponds to a multiplication with that cell’s constant weight, and routes it to the next layer.
They won't sell/rent/license the weights to an end user at any price because they don't trust your security.
Now your robot can respond sarcastically when you ask for chicken nuggets. Again. It also doesn't dent your walls anymore.
Imagine if like instead of having a specific Mac Plus ROM, you had a thing that looks like a fat ASIC that can hold models sitting on a slotted daughtercard directly next to the CPU and RAM.
Fable is nice, but still requires a lot of guidance for large scope tasks.
But yeah, for things like programming, if it can do linux and python and some go and sql and javascript, larger domains can be threaded with LORA
"You are not prepared" --Illidan Stormrage
Great for us, looks like a recession as far as the Market is concerned.
My partner has been asking for a “completely private” model for doing research and shifting through volumes of data that can’t leave the office and $$$ for the current hardware makes no sense. It would be an easy sell if someone walks in with a black box that contains “ChatGPT”.
Enjoying the Ian Cutress / TechTechPotato video on Taalas. Some ok good technical details on the tech, and some good insider baseball, whose who stuff. (What a treasure having tech discussions like this about.) https://youtu.be/3MKRjt59hh4
You need to be able to add|mul where the data (the weights) are stored.
Bonsai Ternary (1.7bits/weight) is a compromise, compromise that has to make sense in the context - efficient when translated into transistors.
Models, probably first open weight ones like Kimi K3 class, are etched into silicon like this and sold as cartridges almost like old school game cartridges.
You buy a USB-C dongle that the cartridge goes into, or for data centers you have PCI cards that take these in slots.
If tolerant, they could churn out many cheaper chips, some perhaps with slight abnormal tendencies ;)
Extremely! You can remove entire layers and the model will still work just fine, with barely perceptible capability losses.
I've cut/bypassed ~15% of total parameters out of Gemma 4 31B on a pod once. Still got perfectly coherent responses out of it. Certain layers are a lot more important than others, particularly early and late ones; but it's honestly astonishing how much can be cut out from the middle without destroying the model's coherence.
I didn't run any meaningful benchmarks, so I have no idea what the capability loss looks like exactly. But "produce coherent and sensible English in response to a wide variety of prompts" was definitely not among the things the model unlearned.
> Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM
I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.
Edit: Lol, downvotes? Stay jelly, meanwhile Talas goes brrr.