upvote

points

by latchkey4 hours ago |
comments
upvoteby boredatoms25 minutes ago|
next
[-]
Power/cooling is the premium.

Can always build a bigger hall

reply
upvoteby latchkey2 minutes ago|
parent|
[-]
Exactly my point. Their architecture requires someone to invest the capex / opex to also build another hall.
reply
upvoteby p1esk4 hours ago|
prev|
next
[-]
The real question is what’s their perf/dollar vs nvidia?
reply
upvoteby zozbot2344 hours ago|
parent|
next
[-]
I guess it depends what you mean by "perf". If you optimize everything for the absolutely lowest latency given your power budget, your throughput is going to suck - and vice versa. Throughput is ultimately what matters when everything about AI is so clearly power-constrained, latency is a distraction. So TPU-like custom chips are likely the better choice.
reply
upvoteby p1esk4 hours ago|
parent|
next
[-]
By perf I mean how much does it cost to serve 1T model to 1M users at 50 tokens/sec.
reply
upvoteby zozbot2343 hours ago|
parent|
[-]
All 1T models are not equal. E.g. how many active parameters? what's the native quantization? how long is the max context? Also, it's quite likely that some smaller models in common use are even sub-1T. If your model is light enough, the lower throughput doesn't necessarily hurt you all that much and you can enjoy the lightning-fast speed.
reply
upvoteby p1esk3 hours ago|
parent|
next
[-]
Just pick some reasonable values. Also, keep in mind that this hardware must still be useful 3 years from now. What’s going to happen to cerebras in 3 years? What about nvidia? Which one is a safer bet?

On the other hand, competition is good - nvidia can’t have the whole pie forever.

reply
upvoteby zozbot2342 hours ago|
parent|
[-]
> Just pick some reasonable values.

And that's the point - what's "reasonable" depends on the hardware and is far from fixed. Some users here are saying that this model is "blazing fast" but a bit weaker than expected, and one might've guessed as much.

> On the other hand, competition is good - nvidia can’t have the whole pie forever.

Sure, but arguably the closest thing to competition for nVidia is TPUs and future custom ASICs that will likely save a lot on energy used per model inference, while not focusing all that much on being super fast.

reply
upvoteby fragmede3 hours ago|
parent|
prev|
[-]
> Throughput is ultimately what matters

I disagree. Yes it does matter, but because the popular interface is via chat, streaming the results of inference feels better to the squishy messy gross human operating the chat, even if it ends up taking longer. You can give all the benchmark results you want, humans aren't robots. They aren't data driven, they have feelings, and they're going to go with what feels better. That isn't true for all uses, but time to first byte is ridiculously important for human-computer interaction.

reply
upvoteby zozbot2343 hours ago|
parent|
[-]
You just have to change the "popular interface" to something else. Chat is OK for trivia or genuinely time-sensitive questions, everything else goes through via email or some sort of webmail-like interface where requests are submitted and replies come back asynchronously. (This is already how batch APIs work, but they only offer a 50% discount compared to interactive, which is not enough to really make a good case for them - especially not for agentic workloads.)
reply
upvoteby xnx4 hours ago|
parent|
prev|
next
[-]
Or Google TPUs.
reply
upvoteby latchkey4 hours ago|
parent|
[-]
TPUs don't have enough memory either, but they have really great interconnects, so they can build a nice high density cluster.

Compare the photos of a Cerebras deployment to a TPU deployment.

https://www.nextplatform.com/wp-content/uploads/2023/07/cere...

https://assets.bwbx.io/images/users/iqjWHBFdfxIU/iOLs2FEQxQv...

The difference is striking.

reply
upvoteby p1esk3 hours ago|
parent|
[-]
Oh wow the cabling in the first link is really sloppy!
reply
upvoteby latchkey4 hours ago|
parent|
prev|
[-]
Exactly. They won't ever tell you. It is never published.

Let's not forget that the CEO is an SEC felon who got caught trying to pull a fast one.

reply
upvoteby spwa44 hours ago|
prev|
[-]
Oh don't worry. Ever since the power issue started developing rack space is no longer at a premium. Or at least, it's no longer the limiting factor. Power is.
reply
upvoteby latchkey4 hours ago|
parent|
[-]
The dirty secret is that there is plenty of power. But, it isn't all in one place and it is often stranded in DC's that can't do the density needed for AI compute.

Training models needs everything in one DC, inference doesn't.

reply