undefined

points

[-]

1tb would likely be ~$20k - given the current >$10k price tag of 256gb. Would you still be considering it at that price?

by jdcasale15 hours ago|

parent|

[-]

I'd consider a 1tb machine at 20k, but I'm not going to pick up a 256gb one at all. 1TB fits a frontier-ish model in memory without massive quantization, which is a very interesting capability for a non-rack piece of compute.

by epolanski14 hours ago|

parent|

[-]

But why...?

At that point just rent proper GPUs in the cloud, you'd have way more power and pay only what you use for.

by albrewer7 hours ago|

parent|

[-]

Because I spent $15k in AI costs last month doing real work at my day job; I'd like to do the same thing for myself but I don't like lighting cash on fire.

by petercooper15 hours ago|

parent|

prev|

[-]

More likely double that, even. I think you'd still see many buyers there. You can spend like $16k alone on a RTX 6000 PRO with a mere 96GB of VRAM now..

by tristor15 hours ago|

parent|

prev|

[-]

I would probably spend up to $30k if I could get 1TB of Unified Memory, because it would allow me a guarantee to run pretty much any local model I want, including >1T parameter models with reasonable quants. I wouldn't be surprised if 512GB is close to $20k when it becomes orderable in October. The justification is less about absolute price and more about price to what it enables. 512GB really doesn't enable much over 128GB for me, but 1TB would massively change things.

I have a bit of paranoia/anxiety about AI, but it's not what most people are concerned with. I understand the limits of these tools very well, and still find them extremely useful. What concerns me is that it's going to become difficult to impossible in the future to run local models which have near-SOTA capabilities in a way in which you can exercise full control of the model. I see the writing on the wall, and its more than worth it for me to invest early to ensure my own capabilities. I am very much not a fan of our "you'll own nothing and be happy" directionality for the world, and I am (at least currently) privileged to have the means to slow that decline for my own self.

by nowittyusername10 hours ago|

parent|

[-]

I suspect we will see more companies which will burn the weights right in to silicon arise. There is already at least one company out there that showed its possible so others will follow IMO. Basically you will see the rise of disposable weights like Nintendo cartridges back in the day. Use it for a time until the better model comes out and you get a new "chip". Though there's a caveat for this business model and that requires you to pump out lots of these chips on the cheap so you are beholden to the lithography companies and what they can produce for you. If you can do this at scale and doesn't require the latest state of the art nm architecture design you are golden...

by tristor10 hours ago|

parent|

[-]

> There is already at least one company out there that showed its possible so others will follow IMO.

Are your referring to Taalas / chatjimmy ?

I definitely think we'll see an ASIC-like approach in the future, especially for embedded small models where it may require minimal silicon area and can result in near-realtime performance. But at the frontier, I don't think this is a solved problem and will continue to have model weight churn that will advantage more flexible general-purpose hardware.

by f0cus1015 hours ago|

prev|

[-]

chaining an option?

by DennisP13 hours ago|

parent|

[-]

They say you can cluster up to four with a shared memory pool, and get three times the inference performance of a single machine.

by tristor15 hours ago|

parent|

prev|

[-]

RDMA is buggy and Thunderbolt only delivers 1/10th the throughput of native connectivity. 1TB of Unified Memory w/ 1.2TB/s of bandwidth with marginally ~$30k cost is a different story than 1TB of sorta Unified Memory w/ an effective 120GB/s of bandwidth with a marginally ~$40k cost + all the RDMA bugs.

by Lwerewolf14 hours ago|

parent|

[-]

You need latency for token parallelism, not bandwidth. Hence actual RDMA that bypasses the software TCP stack (ROCe or whatever).