undefined

points

[-]

Apple’s article isn’t targeting people who bought multiple M3 Ultras before the RAMpocalypse - those folks can be safely assumed to already be on board with running large models locally.

by LoganDark2 hours ago|

parent|

[-]

I thought the article was about showing this specific model can be run locally, not convincing people about running models locally in the first place.

by unshavedyak7 hours ago|

prev|

[-]

Maybe you (or i?) missed the point - it wasn't that nobody has those machines, it was that at $40k for 4x the 256GB versions is a substantial price, and so the parent parent was suggesting that it would be compared against OpenRouter. Ie how long would it take someone to spend $40k worth of credits on OpenRouter?

I've had the same thought on self hosting for the past 2 years. Eg i could invest $10k into hardware or spend those on credits. I chose credits, because by the time i spend 10k i suspect the market will have dramatically shifted in requirements, compute, etc.

by nhecker5 hours ago|

parent|

[-]

> I chose credits, because by the time i spend 10k i suspect the market will have dramatically shifted in requirements, compute, etc.

I agree, at least for inference credits. But anecdotal counterpoint: about ten years ago I put together a system using about $1600 worth of new and used components. Including electricity that TCO has been something like $14/month. I'd be hard pressed to find 128GB of RAM, 12 cores of x86_84, and 40 TiB (raw) worth of disk, and unlimited ingress/egress traffic for rent at that rate. That's what, $0.02 an hour, rounding up?

Yes, a hyperscaler's rent-a-computer would get faster and be more energy efficient. Possibly more carbon efficient over it's lifetime, although I have no way to quantify that. But in raw dollars it would have costed me absurdly more to rent. On the other hand, my machine has sat cold and unused for multiple months when I didn't need it to be running, and I don't consume all of it's resources around the clock. So maybe I could have gotten away with a lesser amount of local compute in a Raspberry Pi or similar, and an equally small cloud box for $5 a month or something. I dunno. I don't even own a laptop or desktop, and try to use 'cloud' services whenever practical. (Edit: always giving priority to privacy in those circumstances; only data I don't care about goes up.)

There are tradeoffs. I don't use my local server for LLM inference, it's far to inefficient for that; so I'll happily pay OpenRouter per token as needed. So I do think that hosted compute for rent does have a purpose. I guess all that's to say that I don't see it in black and white; there are times to invest in local hw and times to ride on the optimized bleeding edge of what other companies are willing to host in big datacenters.

This turned into a bit more of a ramble than I intended.

by LoganDark5 hours ago|

parent|

prev|

[-]

The substantial price is one reason I'm disappointed that open models are still getting bigger and bigger, rather than more efficient in any way.

I just also think that plenty of people buy clusters of Mac Studios anyway.

by nhecker5 hours ago|

parent|

[-]

Haven't smallish (16-32 GB VRAM) open-weights models been getting better and more capable over the last few years? Seems like what you can run on 'a laptop' today beats the pants off of whatever you could run on that same laptop two years ago.

by LoganDark5 hours ago|

parent|

[-]

I guess. I found that Gemini 3.1 Pro and 4 Argon are by far the best at what I like to use LLMs for. Even 3.8 Flash doesn't compare, and I'd imagine something I could run on my local hardware (M4 Max 128GB) would be just as disappointing as the rest, at least for now, even if better than earlier models. One of the reasons I've been hoping to see diffusion language models eventually.