The RAM choice was because I have never regretted buying more RAM - it's practically always a better trade than a slightly faster CPU - and 96GB DIMMs were at a sweet spot compared to 128GB DIMMs.
That, and the ability to have big LLMs in memory, for some local inference, even if it's slow mixed CPU/GPU inference, or paged on demand. And if not for big LLMs, then to keep models cached for quick swapping.
I don't mean to judge, it's your money but to me it seems like an enormous waste. Just like spending $100k on a car when you can get one for $15k that does pretty much exactly the same job.
It's not so easy to get nice second-hand hardware here in Switzerland, and my HEDT is nice and quiet, doesn't need to be rack-mounted, plugs straight into the wall. I keep it in the basement next to the internet router anyway.
The "sensible" choice is to rent. It's the same with cars; most people these days lease (about 50% of new cars in CH, which will be a majority if you compare it with auto loan and cash purchase).
I think leasing might be okayish if you find a really good deal, but it's really not much different than buying new which is just a shit deal no matter how you turn it. A 1-4 year old car is pretty much new anyway, I don't see any reason to buy brand new.
That’s for everyone
Never really used it all, usually only about 40%, but it's one of those better to have than not need, and better than selling and re-buying a larger memory machine (when it's something you can't upgrade, like a Mac or certain other laptops)