Having that said I'm desperately trying to get REAL hardware to run that benchmark. With some successes ;)
Few months ago I got Hetzner machine from Kent Overstreet and I was able to finish 3 runs before machine died... Results: https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/
Currently I've got even more interesting machine with tons of disks and I'm running new set of benchmarks but it's really in its initial stage.
https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/ 2nd run in progress... one run on REAL hardware takes much more time than on GH runner so it's slow.
But this new hardware has also so many disks that the plan is to try also more complex, tiered cache topologies. I'm working on it.
I'm happy to answer any other questions, sources of every piece of this benchmark are freely available and I'm not saying they are 100% correct. I'm open to improvements.
I've been saying it for months, but eventually I'm going to move the automated builds off the 48 core monster and we'll be able to use that for automated perf testing too. The machine we just got has spindles for EC perf testing, but the Hetzner monster has very high end enterprise ssdd.
Also, just got done with the Rust for Linux conference, still not home but here's slides that still need reformatting: https://evilpiepirate.org/~kent/Kangrejos-2026-bcachefs.pdf
* 4 HDDs (for example dm-raid has read balancing optimized specifically for HDDs)
* 5 HDDs (classical raid should see no improvement but btrfs and bcachefs should balance the load)
* 4 SSDs
* 3 HDDs + 1 SSD no tiering
* 2 HDDs + 2 SSD no tiering
* 1 drive 10x larger than others (since how bcachefs and btrfs allocators work)
* nocowDisregard, I just saw you have some RAID10 tests in there so three SSDs won't be enough.
1. Dual Ext4 + external 32GB journal X4 pcie SSD (the prior winner of benchmark surveys)
2. Bare F2FS after a trim and SSD vendor software cache flush operation (it should be slower, but knowing how much slower on identical hardware could be interesting.)
3. DRBD across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)
4. CephFS across a 48U 100Gbps host rack (single X4 pcie data drive per host, OS on primary)
Best regards =3
5. a ZFS dRaid configuration. There could be very different characteristics there with it using slabs.
Speaking of slabs, MS ReFS of you feel adventurous!
I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?
If it's something as simple as a KVM hypervisor that only runs 1 test VM at a time (with no other load from anything else other than the basic systemd daemons, ssh daemon etc running on the hypervisor), the results could be very close to bare metal.
I can see it being very time consuming and annoying to do repeated manual bare metal OS installs and new partitioning/filesystem creation for such a large variety of tests.
The author does also say that performance isn't really the main thing but rather, data integrity:
Well don't do that then. There's lots of other options. Probably the simplest is a single bare metal install on a simple filesystem on one device. run the filesystems under test on other storage dedicated to testing.
You could also boot into a network install and use local storage exclusively for testing.
Also take a look at tests on real hardware. There are not many of them but there are some. I pointed to them in my first answer.
Real hardware is used in: https://bartosz.fenski.pl/modern-fs-benchmark/real-hw/ https://bartosz.fenski.pl/modern-fs-benchmark/sas-hdd/
But unfortunatelly it's much more limited number of actual runs. sas-hdd is still in progress so numbers for it should increase over time.
In this case, given that the author's own disclaimer (above) already disclaims the numeric readings, I'm not sure how it's possible to make any inference on "shapes and ratios" derived from the numeric readings.
I think you are sugarcoating the shit show that was bcachefs's history of involvement in the linux kernel. I mean, do I need to mention that the person was subjected to a code of conduct enforcement action due to his long history of abuse and unprofessional behavior?
I'm saying ZFS on another OS.
zfs is shunned. can't really get around that copyright issue. bcachefs just hit a setback, which i am hopeful will eventually be resolved.
Well except for Ubuntu, one of the most popular Linux distros supporting it.
But there's a LOT of FUD about it.
I could drop bricks on and cord pull those all day and they would not lose data. Which is a small ask for a filesystem IMO.
I learned about VDO today. [0][1] I'd never heard of it before, but I'm using Gentoo Linux and both the dm-vdo kernel module and the 'vdo' software provided by [1] are distributed by Gentoo... so this isn't some weirdo Red Hat thing. It looks like you manage and used these just like any other thin-provisioned LVM volume [2], but -like I said- I've not used this before, and have only just skimmed the docs, so it's possible that I'm missing something important.
[0] <https://docs.redhat.com/en/documentation/red_hat_enterprise_...>
[1] <https://github.com/dm-vdo/vdo>
[2] <https://docs.redhat.com/en/documentation/red_hat_enterprise_...>
You can mix devices of different sizes and types on bcachefs. You can have foreground and background devices to balance performance and also different compression settings for foreground and background transactions.
You can set replicas=N to the individual file or directory on bcachefs. For example files you can just re-download or re-build. Likewise you can set a higher number of copies to important files.
There's a NAS appliance called NASty and it shares publicly usage stats: https://nasty-telemetry.pages.dev/ Those numbers are NASty alone. There are more users on other systems.
Not everyone gets it though, that's for sure.
And, if you want to know if it's mature, I'd trust the user reports over the one liners :)
There is and have been many promising and exciting FS to replace the old boring ones, but for storage you not only want to avoid technical issues but also maintainer(s) drama...
The community infighting has sucked, but that's a thing that matters primarily for maintainers.
I think most users just want something that works.
That said I certainly hope that one day the technical advantage of bcachefs will be so overwhelming that maybe the decision to remove it will be overturned. And if big vendors make it their default FS the bus factor will disappear (even if unofficially you'd still be the sole maintainer, but no one cares about that in the enterprise world...)
To answer the original question, most people who care about their filesystem at all care about its stability. Not just "does it work now" but also "will it work and improve over time". Infighting puts the future at risk.
But you might want to check out the bus factor on btrfs too; when a maintainer says "but we've saved Facebook billions and billions of dollars!", calls for the other filesystem maintainer to be ejected from the community, then quits to join Anthropic a month later - that's not a vote of confidence.
I'd be very happy if people could just stop bringing up drama and us factors. We put it behind us a year ago, but it seems not everyone got the memo.
Btrfs regulars:
- 1 from Meta
- 1 from Oracle
- 4 from SuSe
- 2 from WDC
bcachefs:
- Kent Overstreet
So that leaves btrfs which always seemed complex and brittle to me (compared to zfs at least)
I get that real hardware costs (author mentions EUR 70 a month for a suitable server), but without at least a baseline snapshot comparison run between real hardware, both SSD and HDD, and the sparse file-backed loop devices, it's hard to take much away from this.
Sadly the AI apocalypse isn't making stuff like this easy to do as a hobby.
In fact they delivered the erasure coding for parity raid back in march this year.
The thing is that as soon as you seriously give a chance to Bcachefs you see how good it is. I can only tell you that mixing different device tiers and having a per-file/directory replication setting is a god send specially in these times where storage costs more than gold.
I'm pretty sure Bcachefs is amazing and better than Btrfs. I also think Zfs is amazing and better than Btrfs. Even so, I still use Btrfs because I know it is guaranteed to always be present on any Linux without any effort on my part.
> That might be the best thing happened to the project since now development can happen at its own pace without the clicky bait influencers.
A better approach might have been to just paused mainline merging instead of forcing being kicked out?
Eg "Hey Linus, Bcachefs is still in early development and I need to merge changes in a pace that is not compatible with Linux development process. So I'm going to pause for a while now and once it reaches maintenance status I will focus on submitting patches in a healthy pace that you can digest".
I used to think that about ReiserFS, too. It was in the mainline kernel, development was snappy, and it solved some performance problems. I used it all over the place.
Things then subsequently... changed. :-/
100%. My system is rock solid and the last thing I need is rolling the dice after every update on whether my system will boot. https://www.reddit.com/r/archlinux/comments/eywcp7/linux_551...
I'm impressed with bcachefs's accomplishments though, and if they ever reconcile with the kernel I'll surely give it a fair shake.
How large is large? I've deleted files with sizes of tens to hundreds of GBs and not seen that, and can probably whip up a test with a single-digit TB file if motivated.
Do you perhaps have 'discard=sync' in your mount options, or are using a kernel earlier than 6.2, which is the version -according to the docs- where async discard became the default?
Actual distro support, and doing it right with people actually communicating with each other, has always been a priority for the project.
Except RHEL. They don’t include it in their kernels.
Alma Linux started including it again though.
It can never be easy.
As for BTRFS I think its also pretty good. Its just that I have the impression its development is guided by the needs of its sponsors and sadly for us META doesn't need RAID5.
A lot of things were tried, people did try to mediate.
The particularly galling thing though was when I finally started looking - post split - comparing bcachefs PRs to other subsystems and especially XFS - I was being more conservative with what I considered a critical bugfix.
There was never a clear statement on what the issue was. What you guys got in public was about as much as I got.
All I can say is - going fast when you're stabilizing and getting bugfixes out the door is what you can and should be doing when you've invested in test coverage, test automation, keeping the codebase clean and asserted, and building up a community that works well together on testing and shaking things out.
I genuinely do not know what they were thinking.
If you are referring to why bcachefs was removed from the Linux kernel, here's a discussion on bcachefs being removed from the Linux kernel.
Instead it got kicked out because Kent constantly ignored the kernel's contribution rules and is unlikely it will ever be accepted back into the kernel.
And it went in when it did because Redhat was pushing for it and claiming to be supportive - but that never materialized. They wanted to get something for free without investing, or putting in the absolute bare minimum.
A _lot_ of people were saying publicly and privately "dear god yes we need something better than btrfs" - but no one from the existing kernel community was interested in stepping up.
Community's still growing, though. A lot of people have gotten active in making sure bcachefs actually works well for people end to end, and there's a hell of a lot more to shipping a filesystem than just writing kernel code.
Sounds strange, but there's more to life than sitting at a keyboard and monitor writing code and debugging it. Weird, right? I know. I've been doing that all my youth and then reality hit me. Can't say it was a bad thing.
The FS was marked experimental, so there is no urgency in fixing bugs or providing features in a certain cycle. Everyone using it knows what they got themselves into. You can still provide the DKMS module for faster fixes and features for anyone who wants to use BCacheFS more seriously for the time that the upstreaming process takes, but eventually it would have all been on mainline.
Asahi is taking a similar approach where they have their downstream kernel and push things upstream once they are mature.
That means the upstream kernel is not useful for running on that hardware now, but things are moving there eventually.
All this has been discussed to death, we don't need people armchair quarterbacking a year later. It's over, it's time to move on.
also tricks to make it easy to convert a root FS to ZFS now that Ubuntu Server 24.04 added native root-on-zfs support: https://github.com/pirate/zfsify
I do have a lot more pull requests to merge than I did before. I don't know if you want to count "Kent isn't reviewing PRs fast enough" as drama :)
I say max, as ZFS stores multiples of the physical block size up to the record size. So even with 128k record size, if your file fits in two physical blocks, it will write two physical blocks of data.
The `ashift` parameter[2] controls the physical block size, typically 512 bytes or 4k for HDDs. Though higher can be useful on SSDs (but less tested, have seen some bug reports with >4k block sizes).
Typically you'd only want to use such small records if you are tuning it for a specific workload, like if you have a database that writes 8k pages.
Compression acts on records, so by limiting the record size, you limit the effectiveness of the compression: 1.2 and 1.8 blocks worth of data both gets written as 2 physical blocks.
On the upside, record size is a dataset property, so you can have many different datasets with different record sizes on the same pool.
[1]: https://github.com/fenio/modern-fs-benchmark/blob/599ec72fe3...
[2]: https://openzfs.github.io/openzfs-docs/Performance%20and%20T...
I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? Can you confirm if any ofthe filesystems really came out of the other side in a usable state after scrubbing?
-----
> Trivial-op p99, idle (ms) # A trivial operation — one 4k write + fsync every 200ms (like a shell appending history or an editor updating its swap file) — run alone for 10s. p99 of the fsync completion
In fact, if it's OK for me to ask, are any of the metrics tht you used standard industry metrics? It looks like several of the tests are bypassing the kernel's page cache? -- which I worry may fall into the trap of "I modified the system to be unrepresentative of reality and then tested it".
----
> kernel 7.0.0-1012-azure
Can you confirm if you tested on a bare metal machine? were you the only tenant?
ZFS and btrfs were designed from the start to handle this, by using checksums on every piece of (meta)data and redundancy to return the same data as was stored to the kernel, and rewrite the bad data.
I've tested my own machines running ZFS by random writes out of band from the filesystem/kernel and it has always found and fixed them.
But if it helps, just some "initial gut feel observations" from me:
* It's definitely not possible to find issue with the the _sheer amount_ of results, but there's just far too much for a human to absorb, all presented at once
* Overall text size is quite small, and difficult to read
* The page doesn't make a strong statement of _what_ is under test: the first words are: "modern-fs-benchmark Multi-device CoW filesystems under workloads classic benchmarks skip" -- which defines the webpage in terms of what it is _not_, without stating what benchmarks are actually present.
* The first line of teh page contains run statistics that probably eithre want to b at the bottom, or just don't need to be in the webpage at all: "latest run 2026-09-18 18:50:45 UTC, kernel 7.0.0-1012-azure, 593 runs recorded · 145 trend points shown"
* A significant proportion of the free text is caveats. There's nothing wrong with being transparent about limitations, but they may be a sign that there might be alternative ways to present the data, or that the data may be flawed (depending on the caveat)
* Theres several categories that I think have been invented for the purpose of collation, but I don't think are defined on the page. I think "Overall Core" and "Core I/O" aren't explained, which means by definition it's impossible for a reader to understand the score table.
* And as we're all aware right now, current Claude models are currently struggling to write coherent English. There's several incoherent sentences on the page. It's a Claude issue.
Sometimes there are ways to make things easier without dumbing them down, but way too many people conflate the two; I get nervous when non engineers say "I've studied this, it should be easy".
At some point I'd like to get our own automated pts runs going, since Michael is not consistent with what hardware he tests on and he hasn't been consistent with getting them out.