upvote

points

by sajithdilshan1 days ago |
comments
upvoteby 1122331 days ago|
next
[-]
Hard to guess, it can go either way. If you will need to be in a syndicate to use non-sterilized models, that mac makes sense. But if there is mandatory registration of personal cyberarms, you risk going to mines once they check you purchases. You could try to play normie and pretend you simply wanted to show off, by keeping your actual work on external disk, but that leaves traces on system. Counting on someone in the Gap renting you gray iron works as long as you can swap credits. Still, this gear is tiny. Put it in your e-car, with uplink, and leave it at uncle's farm. Discreet.
reply
upvoteby woah1 days ago|
parent|
next
[-]
It was a dark rainy night in Neo-Tokyo as Blake puffed on his vapor cartridge and watched the Mac dealers prowl below. Almost 15k Union Credits to get one of them to meet you in an e-cafe with a fully loaded M5, but man, the inference rush from one of those things was something else.
reply
upvoteby techmunky1 days ago|
parent|
[-]
superior zero latency local skooma
reply
upvoteby glitchc1 days ago|
parent|
prev|
next
[-]
I'm sold on "personal cyberarms" as a concept

Do they include footguns from pointer bugs?

reply
upvoteby Razengan1 days ago|
parent|
prev|
[-]
I gotta have some of what you had :)
reply
upvoteby kridsdale11 days ago|
parent|
prev|
[-]
I thought it was a fun bit of cyberpunk fiction. Those who downvoted him seem to have taken it at face value?

I appreciate the reference to RUSH: Red Barchetta in the final line.

reply
upvoteby nowittyusername1 days ago|
prev|
next
[-]
512 option isnt worth it imo, you get severe slowdowns when weights are that large. 256 is the sweet spot, you can run large open weight models at decent speeds for full private inference.
reply
upvoteby Octoth0rpe1 days ago|
parent|
next
[-]
a) we don't actually know what the prices will look like yet, b) what about same weights + huge context? or, same weights that you'd run on 128gb/256gb, but multiple models running for different tasks?
reply
upvoteby zamadatix1 days ago|
parent|
[-]
I think it's safe to start the conversation as about bad as the jump from 256 to 512 on the M3, which was a little more than double base to 256. If it's surprisingly different at launch then it can be a party, but there is no sense getting your hopes up for that at the moment.

Longer context also slows token prediction proportional to the context size. If it wasn't regularly referenced then there would be no need to keep it in RAM.

Usually the pitch for more memory is "I can run a massive model/context and get my answer in a while instead of next weekend from disk".

reply
upvoteby cma22 hours ago|
parent|
prev|
next
[-]
> you get severe slowdowns when weights are that large.

Not necessarily for MoE

reply
upvoteby throw0101c1 days ago|
parent|
prev|
[-]
> 512 option isnt worth it imo, you get severe slowdowns when weights are that large.

I think most people are getting 512 for running Chrome with a bunch of tabs open. /s

reply
upvoteby geodel1 days ago|
prev|
next
[-]
Agreed.

Specially since one can pay half right now to OpenAI and sign a 12 year iron clad contract for uninterrupted service delivery of OpenAI Pro.

reply
upvoteby jeffybefffy5192 hours ago|
parent|
next
[-]
And not have to worry about them training on your data even tho you ticked a box saying "dont do that".
reply
upvoteby Kurtz791 days ago|
parent|
prev|
next
[-]
I think we all expect the heavy subsidized subscriptions to end or significantly increase in price at some point, but it could be years from now and I'd rather spend a similar figure on an hypotetical Mac Studio M8 Ultra, or whatever more advanced competitor that will have likley appeared by that time.

A more apples-to-apples comparison would be with API cost in OpenRouter at the same tok/s rate for the same models that you can run locally, maybe.

reply
upvoteby qwytw1 days ago|
parent|
next
[-]
> heavy subsidized subscriptions to end

Is there evidence that's true though? I mean gross margins on subscriptions being negative since the API is seemingly very profitable (if the price is compared with the cost of serving very large open models).

As long as there is pressure from other providers serving cheaper models that are somewhat competitive without having to incur any of the R&D costs raising prices will be tricky.

reply
upvoteby BatFastard1 days ago|
parent|
prev|
[-]
>A more apples-to-apples comparison

Don't you mean an Apple to NVidea comparison?

reply
upvoteby vardump1 days ago|
parent|
prev|
next
[-]
I hope that was sarcasm.
reply
upvoteby cyclopeanutopia1 days ago|
parent|
[-]
"iron clad" :)
reply
upvoteby prmoustache1 days ago|
parent|
[-]
Censorship included.
reply
upvoteby patrickmcnamara1 days ago|
parent|
prev|
[-]
HN always has these completely contrived counterarguments. What is actually going to realistically happen that will prevent use of an LLM provider? Did you think that the OP literally meant the 12 years or maybe it was just to show how expensive using a Mac Mini as an alternative is?
reply
upvoteby geodel1 days ago|
parent|
next
[-]
> how expensive using a Mac Mini as an alternative is?

I think it goes without saying. And it is eminently evident over last couple of decades that from compute to storage to meals 3rd part providers have saved billions upon billions of dollars to enterprises and individuals alike by providing these essential services.

reply
upvoteby kridsdale11 days ago|
parent|
prev|
[-]
Mass revolts of the peasantry burning down data centers and cutting fiber lines.
reply
upvoteby geodel1 days ago|
parent|
[-]
Yes, it feels like that. Whereas frontier labs are pushing the frontier of human knowledge, selflessly working towards pulling humanity from dark ages. Ignorant peasants trying to burn the modern civilization down. Don't they know data centers and fiber lines are lifeline of modern economy?
reply
upvoteby ericmay1 days ago|
prev|
next
[-]
Just commenting here because you're discussing hardware: I thought the test results from the SSD published in this article [1] were pretty interesting. Maybe that's old news though.

[1] https://www.macworld.com/article/3238319/mac-studio-m5-max-r...

reply
upvoteby simonw1 days ago|
prev|
[-]
Yeah, anyone who thinks local AI is going to save them money is likely to be disappointed, at least if they want to run models that are even remotely capable.

Plenty of other reasons to get excited about local AI, but I don't think cost is one of them.

reply
upvoteby criddell1 days ago|
parent|
next
[-]
Maybe you are using a local model to go after some Millennium Prize problem and you don't want OpenAI to take your work and use it to win the prize for themselves? $15k might be a bargain.

And, yes, I know a current local model wasn't going to solve the Navier-Stokes problem, but I'm just using it as an example where privacy might be valuable.

reply
upvoteby simonw1 days ago|
parent|
next
[-]
Agreed, plenty of other reasons to get excited about local AI.
reply
upvoteby bel81 days ago|
parent|
prev|
next
[-]
I'll try that argument with my wife next time I want to buy a $15k mac.

It's a bold strategy cotton, lets see if it pays off for em.

reply
upvoteby auntienomen19 hours ago|
parent|
[-]
It will also reduce your heating bill.
reply
upvoteby Danox1 days ago|
parent|
prev|
[-]
[dead]
reply
upvoteby hgoel1 days ago|
parent|
prev|
next
[-]
Despite being on a site called Hacker News, we seem to often overlook the simple aspect of wanting local AI hardware to hack (not necessarily in the cybersecurity sense) with. I got my local AI hardware because it's an enjoyable hobby for me.
reply
upvoteby Danox1 days ago|
parent|
next
[-]
Yes it surprises me too…
reply
upvoteby ionwake1 days ago|
parent|
prev|
[-]
apparently if you ever point out HN starts for hackernews and thus expect related attitudes you get downvoted by shocked ( what I guess are zoomers and not bots ) that desperately opine the name is a random abberation doesn't mean anything and one should not deviate from our corporate overlods in any manner.
reply
upvoteby SXX1 days ago|
parent|
[-]
A lot of people commenting on how bad idea buying local hardware for inference is also miss the fact that even in 3 years that hardware gonna cost something.

Might be if RAM prices get much more reasonable its gonna be 1/3 of the price, but it's very much possible its gonna be half or more.

And if you're buing Mac Studio and not some AI-only locked down board it's possible to reuse it for other purposes.

reply
upvoteby matt-p1 days ago|
parent|
prev|
[-]
On a personal level maybe not yet, but for a medium business upwards it may make sense.
reply
upvoteby Danox1 days ago|
parent|
[-]
Having no debt and owning in the long run always works out better than a lifetime of renting, if you don’t have to, the massive rent letting these days, is very frustrating at some point don’t you have to draw the line?
reply