If llama 3 70b were available for $400 today, people would make it work. They would have 4 of them working side by side or 1 of them working with a GPU on another model.
replyTo me, it's like imagining if Sonnet 3 was burned into an ASIC 8 years ago and then never changed. It would still be revolutionary, and today we would have an entire ecosystem of tools and services built around it, likely surpassing some of our current workflows.
The frontier is a different beast, but it would likely mean competing on price.
I think you'd be better off getting 1% of a $40,000 card. Or maybe 10% of a $4,000 card is more realistic.
replyI think that's the problem at the moment. A much better LLM that doesn't use my battery is 20ms and <10mb of data away.
for many purposes they're good enough now. If I had an opus 4.8 class model on a box next to me that could produce tokens at rates like 5000/s, i don't know if i'd need a new one for a long time. I think we might be underestimating how powerful very very fast LLMs could be, since they could iterate on tons of small variations on tasks. paired with deterministic guardrails that gate "doneness", you could loop on tasks for a long time having the agent try different strategies until the goal was reached, in ways that are just impractical now (and very expensive)
replyExactly. A fast, cheap model, particularly with the right harness and loops, might take us a lot farther than we might guess.
replyI think it makes more sense if you think about an ASIC pipeline which is updated on an annual basis, like Apple's M or A-series chips.
replySure, an ASIC model will always be behind (12 months? 18 months?), and a hosted flagship will be significantly better. But as time marches on, would I use a flagship model from last year if it came on a PCI card and cost $1000? Without a shadow of a doubt.