undefined

points

[-]

> Taalas needed a giant chip (6nm) for an 8B model.

You're phrasing it like it was kind of an inherent technical limitation with this kind of burning weights into silicon. Which is also not new, it goes back to the 1980s with fixed function digital signal processors and little linear regressions or hardware classifiers for industrial control systems, all are the same basic principle.

It's just usually not worth it to go super small process node, because most models people thought to turn into silicon were pretty small parameter sizes. We're talking 10-100 weight regression or at most 2-4k weight neural net, used in some instrument or factory equipment. You can do a decent MNIST OCR with a 4k weight neural net. For this, 180/130nm is fine.

Or you might think it's required with their special 4-bit as transistor thing (plausible). It's more that when you're experimenting and iterating, TSMC 6nm is their advertised path for rapid prototyping at cost for proof of concepts. And that's already in hot demand, while good luck if you're a startup trying to break in with 3/4nm as your first run.

by greenknight5 hours ago|

prev|

[-]

Nope. But we are hitting some pretty impressive levels with 128B models.

The other thing is, a lot of the time, model performance is improved with more 'thinking' time.

The thinking time is just more tokens... but instead of say 1000 tokens or 10,000 tokens worth of thinking its 1,000,000... how does that improve model performance? Could a 128B model hit levels of GPT Sol?

by cherioo5 hours ago|

parent|

[-]

Thinking generates a ton of tokens. These baked in chips tend to not have a lot of memory for context. I am not sure taalas supports Thinking at all.

The more problem like these they solve the more they will look like GPU.

by nextaccountic5 hours ago|

parent|

[-]

couldn't one just add some hundreds of GB of HBM?

by kimixa4 hours ago|

parent|

[-]

Yeah, but then there's the size of KV cache needing to be read through that HBM interface for each token, putting a hard limit on the tok/s based on the memory bandwidth.

On some models a large context can be a notable proportion of the size of the weights themselves.

For example, qwen 3.8 27b uses ~64kb/token for the kv cache - so for a 256k token context that's ~16gb of the kv cache for a ~54gb model (assuming 2 bytes-per-param/f16 for both).

So if the current non-baked-in chip is already memory bandwidth bound, as is often the case for current hardware and models, and the "only KV cache in HBM" chip has the same total memory bandwidth, it can only ever be (54/16)=~3.4x faster for the baked in-silicon model.

EDIT: I guess actually (54+16)/16=~4.3x faster, as the current implementation would need to read that KV cache too :)