That under-states the matter. It is a terrible time, price-wise, to build a NAS.
I'd almost rather have no AI whatsoever and have storage 1/10 the price of pre-AI times.
(If there were a magical choice between having AI and significantly more expensive storage, and having no AI and some program to dump that investment money into getting and somehow leveraging significantly more available storage, that is.)
I had a drive go bad in my 4 year old home-built NAS a few weeks ago, and it cost 2x what it originally did to buy the same capacity drive, and that was going to a grey-market importer on eBay, it was more on "reputable" sites.
Almost? ALMOST!?!?!
If you handed me a button that would make it like LLMs had never existed I'd be slamming that button so hard Sam Altman's clothes would spin around.
Return memory and storage prices to normal, undo the sloppification of ~everything, remove all these annoying "features" that are so useful they have to force them upon people, and make scammers actually have to put in a slight bit of effort, all at the "cost" of real human developers, artists, writers, etc. getting paid for their work....
If this is a hard decision for you, you are the problem.
Which is 100% what this has felt like to work with, all spring and summer.
So we're in a slop phase, it'll pass. The first few years of youtube gave no hints that we'd get stuff like Veritasium.
Ignore the slop and use the tools to create something you never thought you could do. That's what it's for!
They (probably) don't want to stop progress (especially unqualified like this) in general. They'd like a world where LLMs didn't come to exist.
And whether LLMs are progress at all remains to be proven.
I built a 24 TB HDD NAS in a 36 bay chassis for around $1000 all-in: mb, ram, rails, chassis, rack, disks, hba, nic
Awful Watts/TB but the plan is to run it and the GPU rig on solar.
The nice thing about the array being big is that I can just RAID 1 it instead of worrying about tinkering with RAID 5, and leave a drive as a hot spare.
PCIe 2.5G NIC for the uplink, and then it can serve over SMB or iSCSI. The main use case for this thing, incidentally, is it just is a caching proxy that holds Docker images, models off of Huggingface, and so on.
Can't you get that in a single disk that costs less than a kilodollar?
I fell in love with the silly idea of squeezing 10G NIC + 10G writes (to a 2x12 RAID 10 HDD array) on the thunderbolt bus of my 12 year old macbook pro.
Then I got distracted with the server mb and truenas and ZFS.
Seriously, if arts and creativity is what sets humans apart from other animals, then AI has almost completely displaced our capacity to even consider doing these activities ourselves. People reach for AI when they should be composing a birthday greeting themselves.
People cannot be relied upon to provide accurate birthdays but nobody would suffer the social faux pas of an incorrect birthday congrats note. Nor would they send it to a spam catching email, but rather are guaranteed to send it to a regularly checked address. They know the persons birthday after all.
You can then sell high quality birthday information correlated to contact information to ad agencies.
Fuck this current internet. So dystopian.
Like seriously, outside of some close friends and family, are you sitting there deliberating over a message for your coworker Steve in the Slack chat or a cousin you barely talk to outside of a birthday and Christmas card?
> or that people have almost completely lost the ability to consider making art
They haven't lost it, no, but there's a lot of financial incentive to stop paying people to do it.
Like I’m already giving up two full drives for redundancy (which saved my ass - I recently had two drives fail on me in quick succession — both SSDs from what looks like an identical batch) but then the advice is kinda saying I need to keep at least another drive worth of space free for the pool to perform well and not crap itself. That hurts with current prices for sure.
I'm not entirely sure, but it seems to me that free space (and the 20% reservation) is mostly a proxy for fragmentation, and you can therefore better look at fragmentation directly. That would mean that if you mostly store large files, there shouldn't be a lot of fragmentation even at high utilization. The whole "ZFS changes allocation algorithm from 80% usage on" is something of 10+ years distant past, and lots of things around the allocator have been improved. It's also something that probably isn't too different from the performance of other filesystems at high utilization, so it shouldn't be exaggerated.
Well, other filesystems can defragment.
ZFS will definitely degrade write performance gradually from 90% utilization, and will hit a stronger cliff at 95%+. Same with SSD, using a consumer SSD above 90%+ utilization would rapidly degrade its lifetime. The effect will be smaller for very large pools and very large files, but the effect is stil there.
I'm repeating this so that people who set up these drives know what is actually going to happen.
Maybe because your statement about SSDs is incorrect. Consumer SSDs have spare blocks too, and wear leveling prevents the scenario you're describing.
Instead the math can be different e.g. it's ok to lose some data but not all. Therefore you might prefer the unraid approach over zfs where losing more than parity doesn't kill the whole pool.
I don't want to lose 26TB just for a mirror, and I have a spare 8TB USB HDD. I'm torn between unraid and just Debian, and I'm torn between just two separate devices and one RAID 1 partition one RAID 0
[edit] also snapshots aren’t really workable for large files. Remux a movie file and now it occupies twice the space.
Does it work? Server can reboot and use TPM to unlock rootfs? What about /boot - encrypted and tamper proof? Resists evil maid attack?
Rabbit hole, I know, but so fascinating if you solved it all :)
A previous hoster once gave me someone else's drives without wiping them. I don't want random customers snooping around on my data if a similar mistake happens with my disks.
For home use, I run without disk encryption. If I ever need to do data recovery, it's not going to be possible with encrypted disks and one point of a centralized NAS is to have stable long term storage.
But in reality that will never happen and the only actual attack you need to be worried about is junkies breaking in and flogging the drives on facebook marketplace. For which, this level of security is fine.
For this you dont need TPM. Just a LUKS key in rootfs /etc. He said TPM :)
You have SATA or SAS to pick from. The CPU requirements for a storage server are not high. On a typical ATX board you have motherboard SATA and can put SAS controllers in the spare PCIe slots.
My first "NAS" was two 22TB hard drives in a ZFS pool on my motherboard SATA
I recommend staying away from SATA drives (huge consumer rush) and look for SAS drives on eBay, particularly HC520 Helium drives from HGST/WD. Need a SAS3008 PCIe adapter ($20) and a SFF-8643 splitter ($30). No backplane is required. Huge lots of decommissioned drives frequently appear. I bought Qty 10* of HC520 (12TB) SAS drives for $1000 about 3 weeks ago, avg age is about 2.5 years, still well within its rated lifetime.
Yea may be some of the stuff is fear mongering and cargo culting. I was told ECC is necessary for ZFS. When the article was written, it was cheap af (2024) to buy ECC RAM so not much consideration was given to it.
-- (*) I have no idea what to do with it. Anyone has any good idea for using 120TB space? I have about 40TB unused bandwidth in the datacenter, may be host a Debian mirror? Donate storage/bandwidth to Internet Archive? Please contact me, appreciate it.
That's cheap indeed. Enough headroom for some failing disks too. How is the noise and power usage? I didn't look at SAS drives at all, because my impression was that they're very noisy. I can place my NAS in a closed off room, but it's not too far away and I was afraid SAS drives would be audible through the wall. At the same time, the shucked drive I'm using presents itself as an WD Ultrastar, which comes very close to a SAS drive, and isn't very noisy.
I think they're same as SATA drives, just different interface. AFAIK they have the same physical dimensions and same internals.
Also something people forget for home nas or figuring out cost for a nas, the power draw of the entire system times your electricity price you then have to multiply this times 2x to 4x times if your climate requires cooling air conditioning of any kind
Is it not the case that if you don't have ECC memory, ZFS could end up writing a checksum that does not match the data if you get a bitflip in just the right (wrong) spot?
The myth, popularized by a notorious thread on the TrueNAS forums [1], is specifically that ZFS requires ECC RAM, and will do worse than other filesystems without it, because scrubbing will multiply a single bitflip into a failed pool.
A ZFS core developer says that that isn't the case [2]. Here's some more reasoning [3], also about many other myths.
[1] https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...
But everything is actually quite sensitive.
We’ve accepted lack of ECC because Intel decided it would be a product line differentiator, and serious customers who didn’t want random crashes or to lose data would buy chips with ECC.
It’s actually less of an issue these days because DDR5 has (by spec) some in-line ECC; won’t help with multi-bit errors but its an improvement on what came before.
AMD has been allowing ECC on lots of regular hardware for a long time.
People don't tend to buy ECC for desktop use because it costs significantly more (used server ram is/was often cheap... but it often doesn't work in desktop boards), and the performance specs are poor.
My home servers are mostly retired desktops, so they get my old desktop ram and I don't want to pay premium prices for jedec speed ecc ram on my desktops, thanks.
Since DDR5 doesn't include reporting on bit errors (afaik), it likely means much fewer single bit errors, but most experienced errors will be multi-bit. Although, I dunno what proportion of bit errors is on the ram chips and what's on the bus... there's no protection from bus errors.
If there were reporting, you could replace chips with high error rates, but without reporting you'll keep running them until they fail enough to notice.
Which means all copies of the data will be corrupted. This can be anything from an irrecoverable file to complete filesystem corruption.
But generally, yeah, not any more dangerous than any other filesystem, and ECC used to be cheap so it was a no-brainer, you should have backups anyway TBH if we're being honest about storage resiliency.
What has been debunked is the "scrub of death" issue, on a scrub a bad bit flip would cause an error, which would be copied over with good data -- well it was technically good before. It would be statistically difficult to have a fault on a read, then a clean read, then a second bit flip destroying the data.
[1]: https://arstechnica.com/civis/threads/ars-walkthrough-using-...
[2]: https://en.wikipedia.org/wiki/ZFS#2004%E2%80%932010:_Develop...
ZFS without ECC is no more risky than any other file system / software RAID without ECC.
As no one owes you an explanation, it would take you five seconds to Google this and discover:
1. It's been disproven, with one of the original ZFS developers chiming in.
2. The original source of the rumor was a forum post that somehow became canon.
...(after much longer) that it's a rabbit hole with nuance. https://news.ycombinator.com/item?id=18481910