You could always stream from SSD storage. Especially effective if you get a cheap old-gen HEDT with lots of PCIe slots to add NVMe storage to and reasonable overall PCIe bandwidth.
replyn-gram per-layer embeddings[1][2] might be it.
reply[1] https://sebastianraschka.com/llm-architecture-gallery/per-la...
[2]: See DS 4.1-Flash and Qwen-3.8-Next.
this is to offload VRAM to DRAM (for GP comment), and makes no difference for URAM
replyn-grams can be kept on SSD, no need to hold them in any kind of RAM (at least w/o batching)
replyNah, I’m streaming ngrams off NVMe on my Spark-alike right now. Works surprisingly well (except for when I accidentally bottlenecked it through my NAS)
replyinteresting, peer comment seems to indicate this is a possibility as well, will have to take a deeper look
replyYou can definitely offload n-gram embeddings to storage; they're very sparsely used (only a few KB fetched per token) so this is quite effective. Loading to DRAM only becomes necessary if they are a bottleneck to overall performance (which might happen if you're doing very wide batches and everything else uses super fast VRAM/HBM).
replyI was looking at the qwen-next-flash, and the weights would fill my OEM Spark on their own, before the n-gram. I'm unclear if offloading to disk can work here, is that what you are implying is possible?!
replyCheck out eugr’s TP=1 sparkrun recipe :)
replyIt’s an NVFP4 quant, but it fits, and is surprisingly capable.
Am I missing a joke? WTF is URAM?
replyunified memory, not sure if anyone uses URAM, I human hallucinated it
reply