From there any motherboard and CPU that can give you 4x slots of 16x Gen 5 PCIe slots positioned such that you can install 4x cards is probably fine. Ideally with a CPU capable of memory encryption.
My latest pairing is a ASUS Pro WS W890E-SAGE SE For $800 open box, and a Xeon 654 for $1300. My first build was an AMD EPYC rig for similar cost a few months ago using a Gigabyte ME03-PE0 and a AMD EPYC Siena 8224P for about 2k combined. I specifically wanted one amd and one intel machine for reproducible build diversity which most probably do not care about.
Do not get too attached to specific cpus and boards. Get what is available for cheap with the needed pcie bandwidth open box on ebay from reputable sellers to save thousands of dollars.
Be sure to get a power supply that can handle all 4 cards. I run Silverstone Hela 2500s at about $800. Do not go cheap on power supplies. It will end in pain. Can confirm.
Then 2-3k for 64gb in DDR5 RDIMMs. This is the most painful thing to purchase but it is what it is.
Not only do I recommend this setup, I plan to build more of them to share with my team.
If you are patient and buy used, a 10k budget can get you a very powerful local AI rig running 100% FOSS software/drivers even today.
Which is exactly why said power and utility will soon be BANNED under a myriad of false aegis'.
An economy that almost exclusively extracts ECONOMIC RENT from the working class, won't permit any "leakage" or autonomy.
Most of my peers do use OpenAI and Anthropic for development work, which pains me, but I will never tell someone else how to do their job if the end result meets my quality bar.
I lead core systems engineering building custom enclaves and operating systems and I do 100% of my work with hardware I own on my own property. When the internet goes down, I do not care and keep building.
It all seems hopelessly adhoc at the moment.
2xDGX Spark cluster is my current load, it is sufficient for a great deal many wonderful, wonderful things. It isn't fast as consumer AI, but I have nevertheless been able to do wonderful things with it for the past year'ish.. I'd get a couple more, but indeed they've tripled in price, so yeah .. things will just do for now, and it's fine. Pretty great, actually.
I think the way we control AI is we air gap it. My cluster is air-gapped.
i just calculated that it should be Per month (30 days) 380euro in my region, which is a lot after an initial 20k.
I have to say, no limits stopping sounds nice though.
$300/mo in the bay area for 1kwh running 24/7, at least until I get solar installed, then free.
> do you not fear that your hardware will be obsolete
If it never gets better than it is right now, I cannot think of any problems in software I cannot solve locally, so I am all set for a while.
Though also, the capabilities on my -existing- cards have gone UP dramatically in the past 6 months since as the local LLM community keeps optimizing.
Sure in 3 years all these cards will be a lot cheaper than I paid, but having them for the next 3 years is absolutely worth it for me.
This costs me less than a used car. Most people chase car and phone upgrades every 2 years. I tend to get something good enough and upgrade once a decade. Depends on your goals really.
In practice though, I can produce about 3B tokens/month of Qwen 3.8 Flash Next which is the model I crank almost 24/7. That would cost something like 1k/mo on OpenRouter. My setup costs about $300/mo in electricity, though getting solar soon to drop that to zero.
My entire 12u rack has roughly the same capability as an H200, though at maybe 15k all in with mostly used parts vs 50k for a used H200 checking todays ebay prices and that is -just- the card you still need to build a system for.
Consumer cards are a lot more power hungry than enterprise cards and they take up more space. I run my setup next to my dryer, running on 240v for 15% better power efficiency.
Also using transparent memory encryption on a Xeon and an EPYC for confidential and private inference guarantees no third party can offer me. Of course I can run abliterated models that do any security research I want without refusals.
Strata and Radiance are the state of the art ATM for running big MoE models on small consumer GPUs at high performance.
Just starting to dive in on local AI with my own 16gb card, and the struggling with evaluating what tradeoffs are worth it.
As for cost a 9700 is running at about 2k USD MSRP right now and the rated power is 300w.
It seems like it just shifts which third parties you rely on. I think local inference is cool and your setup sounds dope, but you still have third party hardware manufacturers, and third party models in the mix.
What is the advantage of this over OpenRouter, which lets me select for, jurisdiction, quantization, ZDR, etc.
note: My use case can't even come close to having this make sense financially, but I understand if yours does. Either way, that doesn't make the third party argument better.
When the internet goes down, my productivity is not impacted, and I have total privacy on my work at all times. Also no rate limiting etc, and I can burn tokens freely.
Sounds interesting, though a bit pricey (if I'm reading the prices correctly it's 20k just for GPUs). Fingers crossed that the prices will fall.
sure, in a very general sense you're dealing with third parties in both scenarios, but if you actually own the hardware and model weights, you can control your own destiny for the lifespan of that hardware.
no ongoing liabilities except the power bill and opportunity cost of dropping $20k on hardware.
He isn't using OAI or Anthropic models. He is using models that area available for absurdly cheap from completely independent providers in multiple jurisdictions.
I get the base level argument, but from a practical point of view there are so many different dependencies that shifting one isn't really doing much.
If you own your own hardware, you don’t have to depend on and trust third parties continually, just once when you bought it.
If the power goes out, internet goes down, I fire up my generator and keep building.
We are talking about digital sovereignty and ownership.
I own the means of my production.
I'm not the one claiming to be free of third party dependencies. Third parties built my hardware, they wrote the firmware in the hardware, they wrote the OS on the hardware, they designed the network stack that you might use to connect to a local server, they provide security updates to all of that software, they ensure that I have a stable power supply to run all this, etc.
I'm incredibly reliant on third parties, some of which happen to sell me tokens. I recognize that I live in a world where I will ALWAYS have a dependency on third parties, and rather enjoy the fact that I can use GLM flash on someone else's $20k computer for a few cents.