upvote

points

by CamperBob214 hours ago |
comments
upvoteby reissbaker13 hours ago|
next
[-]
RTX 6000 Pro retails for $10k so an 8x is $80k before anything else in the computer, and long-context will have... pretty bad performance (20+ seconds of waiting before any tokens come out), but it's true it technically works.

I don't think cloud models are going away; the hardware for good perf is expensive and higher param count models will remain smarter for a looong time. Even if the hardware cost for kind-of-usable perf fell to only $10k, cloud ones will be way faster and you'd need a lot of tokens to break even.

reply
upvoteby zozbot23413 hours ago|
parent|
next
[-]
> I don't think cloud models are going away; the hardware for good perf is expensive

I think local AI will win in its niche by repurposing users' existing hardware, especially as cloud hardware itself gets increasingly bottlenecked in all sorts of ways and the price of cloud tokens rises. You don't have to care about "bad" performance when you've got dedicated hardware that runs your workloads 24/7. Time-critical work that also requires the latest and greatest model can stay on the cloud, but a vast amount of AI work just isn't that critical.

reply
upvoteby reissbaker9 hours ago|
parent|
next
[-]
Users do not have an existing $80k of hardware, are not going to buy $80k of hardware for worse performance than paying $100/month, and models are continuing to grow in size while memory grows in price.
reply
upvoteby zozbot2344 hours ago|
parent|
next
[-]
You said you need $80k in hardware for "good performance". I'm saying the local AI inference workflow will be a lot more flexible about performance than that, and can get away with something vastly cheaper and in line with what the user owns already.
reply
upvoteby otabdeveloper46 hours ago|
parent|
prev|
[-]
> paying $100/month

There will not ever be a monthly subscription for LLM tokens. The economics isn't there.

Local tokens will always be cheaper.

reply
upvoteby entrope1 hours ago|
parent|
[-]
What's the basis for saying local tokens will always be cheaper? As others have outlined, LLMs serving one user at a time are pretty expensive, but concurrent users become much more cost-effective (assuming there's enough RAM for the contexts). If "local" to you means ~10 hours daily use by a team of employees, the company still has to balance against cloud services that can amortize non-recurring costs over 24 hours of service per day.
reply
upvoteby zozbot23438 minutes ago|
parent|
[-]
Why wouldn't a team of employees be able to run AI workloads 24/7? Not all workloads are time sensitive.
reply
upvoteby ai_fry_ur_brain13 hours ago|
parent|
prev|
[-]
"I think"

Well your thinking is completely vibes based and not cemented in any reality I exist in.

reply
upvoteby CamperBob210 hours ago|
parent|
[-]
Other sites beckon.
reply
upvoteby otabdeveloper46 hours ago|
parent|
prev|
next
[-]
> higher param count models will remain smarter for a looong time

They're not smarter, they just know more stuff.

You probably don't need knowledge about Pokemon or the Diamond Sutra in your enterprise coding LLM.

The "smarts" comes from post-training, especially around tool use.

reply
upvoteby anon77256 hours ago|
parent|
[-]
If the smarts came from post-training, we could show significant gains by doing that post-training again for previous generations of models. But we know that isn’t happening - effective post training is necessary but not sufficient for model performance.
reply
upvoteby alfiedotwtf5 hours ago|
parent|
prev|
[-]
If 8 x RTX 6000 is getting you 20s before initial token, how are cloud vendors doing this?
reply
upvoteby zozbot23414 hours ago|
prev|
[-]
4-bit quantization is native for Kimi 2.x series.
reply
upvoteby CamperBob213 hours ago|
parent|
[-]
You're right, I was thinking of Qwen. K2.6 will run at UD-Q2_K_XL precision on 4x RTX6000 boards, but I have no idea if it's worthwhile.
reply