Openrouter on nitro gets me there. Although the TPS is 5X less still.
I have tried GPT OSS 120b on cerebras and it flies. But it's mostly wrong.
The fact that Cerebras, which has a willing and major customer like OpenAI begging at their heels to deploy on their hardware, now posts about a 2.8T open weight model says a lot.
[0] https://blogs.nvidia.com/blog/gpus-openai-gpt-6-astra-ultraf...
For some reason though on openrouter the tps of 6.1 sol ultrafast is usually lower than opus 5.5 fast. Is there readings off as surely cerebras hardware is faster than low user density nvidia still?
They don't really show it here, but supposedly a Cerebras' chip is sub-divided into a 2D mesh of what they call PEs (cores). So data flow generally flows in two axis, and maintains consistent timings to its 4 neighboring cores. They show the layers flowing from left-to-right being streamed by PEs, but I wonder how true that ends up being: maybe the communications are easier to stream via diagonals, or some other shape that has better topology for data flow.
Then, even from die shots of CPUs and GPUs we often see that cores are tiled as squares, and those don't have the same cross-core connection and streaming requirement. Now, I assume that some parts of what we call a core can be moved around more freely while maintaining whatever power/heating/timing requirements we have, so perhaps it's easier to 'force' a square by moving those loose parts around. But why do we confine those to squares? Is super-core layout just something not worth investing time into, like how interconnects are timed, non-convex tilings, and distances to I/O?
And for streaming architectures like Cerebras, why aren't other types of tilings considered, something like a hexagon grid? This could have 3 inputs / 3 outputs, which could allow more complex data flow (?). Similarly, we could have 8 connections with longer diagonals in a square tiling, etc.
I guess, what are the tradeoffs in place such that so many chip designs seem to reuse square tilings both locally and globally?
Wouldn't a rectangular brick layout be topologically a hexagonal tiling
I could see that square packing is more simple, but by shifting elements per row I don't see why it couldn't propagate at odd angles.
they have never been able to keep up with Groq/Cerebras-level inference speeds for some reason.
What do they mean by “unlike a GPU batch”? It’s not a hardware limitation of GPUs right? It’s just the scaffolding to do this in software isn’t available right
A server that both holds KV cache in its own memory, and uses its own compute to perform attention operations. That way, you don't have to transfer cache entries server to server for every attention sweep - only the "requests" and "responses". Attention bandwidth stays server-local.
Wafers can have trillions of transistors, 10-16 times more than the 4 trillion of the WSE-3.
1 Terabyte of SRAM per wafer will be the upper limit for a few years [1].
Such a weird company with a terminally online on linkedin ceo, garbage devrel on X. Cant really be friends with CSPs coz everyone has their own chip, no consumer solution either, no neocloud would want to onboard them either coz its baggage no one wants to risk dev time/customer money with.
If something is roughly as intelligent as a normal human expert in a given area, but 100 times faster, that would be a valid example of super intelligence.
I'm sure that by some definitions like this we've already discovered narrow super intelligence.
For example - breadth of knowledge. Is knowledge a kind of intelligence? If so then current agents absolutely have a breadth of knowledge which is superhuman.
Tortious really does win the race, and there is no such thing as a free lunch.
Nobody needs this, and the quality of models they're deploying, or all llms for that matter, require heavy human review (if you care at all about maintainability).
One developer can barely keep up with the daily output of multiple llm sessions writing code at 50 tps.
Im convinced anyone confidently thinking 2k tps is a no brainer is probably not concerned about the quality of their work (many such cases).
There are diminishing returns, but half of the rate you mentioned would already help me keep focus.