Made me think of this:

I got a 10G ethernet network card for my NAS only to realize it has to overlap with my modem's supported bandwidths (IIRC 2.5G, 5G).

Knowing nothing about the space, I had assumed it would use max(node1, node2), but instead it negotiated a 1G link. So it was faster to use the mobo's built-in 2.5G port.

I have a backup node with a 40G NIC & a ZFS pool of just 8x HDDs set up as a pool of two RAIDZ1 vdevs striped together (i.e. 4x drives in raidz1-0 & 4x drives in raidz1-1 make up the "backup" pool). Restoring full backup images to another server I usually get ~11-12 Gbps over NFS, no flash cache or anything involved.

Honestly, outside of random access/small file access, my primary NVMe ZFS server isn't all that much faster in raw throughput - despite being 22x NVMe drives going direct to the CPU instead of 8 HDDs going through a SATA controller. I think it's easier to hit other bottlenecks with ZFS/network transfers well before the disk throughput itself. E.g., enabling jumbo frames for NFS did still give me a decent perf/efficiency bonus.

> Can you actually saturate the links with the spinning drives?

There can easily be a bottleneck depending on how the setup the sata/sas, but if you can get sustained sequential reads or writes, 16x drives at 6 Gbps sata should be able to saturate 2x 25 Gbps ethernet. The store link shows two expansion ports as well which should help get bandwidth to the point where 25 Gbps is useful.

Less likely with random reads/writes or mixed use.

You can fill it with SSDs, and SFP28 is so common the prices are cheap:

https://www.fs.com/c/25g-sfp28-3215

But no, spinning disks won't saturate it, even if you were doing 100% sequential reads.

(I originally said fill it with NVMe - I was wrong)

with the zil/slog on nvme yes -- you would want redundant power, UPS and a raid of nvme drives but with all that in place the data would get securely written to flash media before being flushed to spinning rust.