You can improve that with speculative preload. I'm sure models could be designed and tuned around efficient SSD offloading to keep throughput pretty high.
replyIt would apply equally to GPU or RAM inference as those are also bandwidth constrained on decode, so people already try to optimize for it.
replysurely the supply of unified memory will rise to meet demand before this is needed
reply