upvote

points

by jrflo2 hours ago |
comments
upvoteby jaggederest2 hours ago|
next
[-]
https://taalas.com/ has done it already for a wildly obsolete model. 14000 tokens per second.

https://chatjimmy.ai/ is their interactive. Tiny context, very dumb, but absurdly fast. Imagine this as a tool call for claude code for trivial changes - the tool call from the harness takes longer than the execution.

reply
upvoteby ElijahLynn2 hours ago|
parent|
next
[-]
Wow! You weren't kidding,

I just tried it too and 14,098 tokens in .05 seconds, I barely blinked and it was done. There was no typing at all appearing on the screen. It just showed up.

https://chatjimmy.ai/chats/01dc66a4-4b1b-4dea-bb5f-926855e37...

reply
upvoteby WalterGR1 hours ago|
parent|
[-]
That link isn't bringing up your chat, FYI. It just shows the default new chat state.
reply
upvoteby christophilus1 hours ago|
parent|
prev|
next
[-]
Wow. This is absolutely wild. I didn't expect that.

If we get to anywhere near this speed for the equivalent of the current models... I don't even know what to think about that future.

reply
upvoteby iamjackg2 hours ago|
parent|
prev|
[-]
Holy crap, I was not prepared for how fast it responded. I just wrote "Just wanted to see how fast you are! Can you write me a quick story about a tiger who lives inside a block of cheese the size of a house?"

I pressed Enter, and the response was instant.

> Generated in 0.037s • 14,205 tok/s

This is unbelievable.

reply
upvoteby jcul2 hours ago|
parent|
next
[-]
It's crazy. Are they doing any precomputing as you type, I wonder if you paste a block of text is it the same speed.
reply
upvoteby nerdsniper2 hours ago|
parent|
next
[-]
I pasted and instantly hit enter on this prompt: "I generated a filter set using REW v5.31.3 using real-world sweep tone measurements from the room I'm listening in . How can I use it as my MacOS output equalizer so that my spotify music is adjusted for this room and speakers"

and it gave a very reasonable answer in non-perceptible time.

reply
upvoteby throwuxiytayq2 hours ago|
parent|
prev|
[-]
No, it really does take ~0.03s to generate the answer. Try your browser's developer tools and watch the requests.
reply
upvoteby 8cvor6j844qw_d62 hours ago|
parent|
prev|
next
[-]
I'd like to imagine the things that can be done with this speed and the current frontier models.
reply
upvoteby throwup2381 hours ago|
parent|
next
[-]
Fully interactive games where you can talk to every NPC by text or voice and have an LLM drive the story (with your own meta prompts to guide it, if you so wish). Maybe even have them generate assets on the fly too.

I’m still trying to figure out coding agents. I can’t even begin to imagine the things it would enable. Even the most mundane ideas like LLMs-in-HiFreq-trading have huge implications.

reply
upvoteby ElijahLynn2 hours ago|
parent|
prev|
next
[-]
Truth, it feels like we're in the dial-up age of LLMs right now. And this Jimmy AI is fiber.
reply
upvoteby Dig1t2 hours ago|
parent|
prev|
[-]
Seriously, if Fable or even Opus was this fast that would be a real game changer.
reply
upvoteby reducesuffering37 minutes ago|
parent|
[-]
RSI will be models better than Fable running faster than this, you won't even need a human in the loop to figure out what to do. The high level goal will be accomplished better than the human in an instant
reply
upvoteby kooi32 minutes ago|
parent|
prev|
next
[-]
"Stochastic gradient descent algorithm in Haskel"

"LMS algorithm in bash"

Just barfed it up lol.

Amazing.

reply
upvoteby baal80spam37 minutes ago|
parent|
prev|
next
[-]
Just wow, I made a similar request. Result: Generated in 0.042s • 14,201 tok/s

This is crazy.

reply
upvoteby HDThoreaun2 hours ago|
parent|
prev|
[-]
For what its worth the frontier lab models can surely be a lot faster if they wanted them to be but theyre supply constrained so theyre doing stuff like multi tenancy. Since you cant self host them no one outside the labs really knows speed as a solo tenant
reply
upvoteby ericd31 minutes ago|
parent|
[-]
You can kind of get a sense by running these things at home - I'm currently running Laguna. One interesting thing is that per stream doesn't actually slow down that much with multiple concurrents, because the bottleneck remains the memory bandwidth until pretty significant request depths, and then eventually you hit the GPU's limits. It's one of the big forces that pushes for centralization in this stuff, the fixed costs to run one are huge, the marginal costs of additional tenants, relatively small.
reply
upvoteby Yopolo2 hours ago|
prev|
next
[-]
And don't underestimate how much money Google, Microsoft, Amazon and Meta still have to spend on this tech.

Blocking Fable for sure made it very politicl a lot sooner than i expected it to happen.

and because China already has massive problems of getting access, they are pushing it on hardware too like what Huawai did without EUV.

It seems China is already able to do DUV a lot sooner than others expected.

reply
upvoteby re-thc2 hours ago|
parent|
[-]
> It seems China is already able to do DUV a lot sooner than others expected.

That's the media and in particular US KOLs of all sorts driving the wrong impression of China and other places. China and many other places for example have fast public transport that the US doesn't and can't even imagine today. They're not behind.

China's DUV still isn't that production grade (mass produce-able) so don't get that hyped up the wrong way (in a different direction).

The whole China-is-behind with tech and in particular semi wasn't that they can't. The truth is they spent decades in internal politics and corruption. That all got solved with the bans, so thank the bans! Jensen even said the bans were bad.

reply
upvoteby FuriouslyAdrift1 hours ago|
prev|
next
[-]
Yep it will be ASICs and DSPs all over again. Orders of magnitude changes.
reply
upvoteby monkeydust1 hours ago|
parent|
[-]
So which shovels companies are the ones to watch for burnt in silicon models ?
reply
upvoteby FuriouslyAdrift1 hours ago|
parent|
[-]
Imagine the price of a $9 million NVL72 dropped to about $100, used 4 orders of magnitude less power, was the size of ARM cpu, be bundled with pretty much any electronic device, and ran as fast a frontier AI is today.

That's about how disrupting DSPs were to the industries they arose out of (over a very long time frame).

How would that disrupt the industry?

reply
upvoteby vjvjvjvjghv1 hours ago|
prev|
next
[-]
Not an expert on this but wouldn’t this be possible with something similar to an FPGA?
reply
upvoteby jaggederest13 minutes ago|
parent|
next
[-]
My understanding is that for FPGA the issue is either it eats all your gates on internal memory if you interleave, or it takes forever to load everything between the SRAM on the board and the actual FPGA component over a bus, last time I looked into it.
reply
upvoteby kooi27 minutes ago|
parent|
prev|
[-]
Weights can be baked into silicone or programmed into hardware ala FPGA, but the context will always be dynamic.

High speed SRAM is where the $$$ is

reply
upvoteby coffeebeqn2 hours ago|
prev|
[-]
What does that mean though? Like some kind of a ROM memory ?
reply
upvoteby bob10291 hours ago|
parent|
[-]
Stacked ROM can, in theory, be a lot denser than anything that depends on a capacitor and refresh cycle.

I don't think it would be that difficult to manufacture compared to other process tech. HBM is really hard to do compared to other memory types.

reply