Hypura reads tensor weights from the GGUF file on NVMe into RAM/GPU memory pools, then compute happens entirely in RAM/GPU.
replyThere is no writing to SSDs on inference with this architecture.
Even if there was a ton of writing, I'm not sure where NVMe even comes in the picture, write durability is about the flash cells on SSDs, nothing to do with the interface, someone correct me if I'm wrong.
replyPeople talk about "SSD endurance", but enough parallel I/O on M1/M2 can make the NVMe controller choke, with very weird latncy spikes.
replyI had assumed heat generation on the controller if it's continuously reading. But maybe it's not actually bad.
replyJust pop a heatsink on it and call it good.
reply