this is to offload VRAM to DRAM (for GP comment), and makes no difference for URAM
replyYou can definitely offload n-gram embeddings to storage; they're very sparsely used (only a few KB fetched per token) so this is quite effective. Loading to DRAM only becomes necessary if they are a bottleneck to overall performance (which might happen if you're doing very wide batches and everything else uses super fast VRAM/HBM).
replyI was looking at the qwen-next-flash, and the weights would fill my OEM Spark on their own, before the n-gram. I'm unclear if offloading to disk can work here, is that what you are implying is possible?!
replyCheck out eugr’s TP=1 sparkrun recipe :)
replyIt’s an NVFP4 quant, but it fits, and is surprisingly capable.
do you have a HF link? HF search is not uncovering it for me
reply(or is it somewhere else)
Nah, I’m streaming ngrams off NVMe on my Spark-alike right now. Works surprisingly well (except for when I accidentally bottlenecked it through my NAS)
replyinteresting, peer comment seems to indicate this is a possibility as well, will have to take a deeper look
replyn-grams can be kept on SSD, no need to hold them in any kind of RAM (at least w/o batching)
replyAm I missing a joke? WTF is URAM?
replyunified memory, not sure if anyone uses URAM, I human hallucinated it
reply