Forgive the ignorance, but how are you running a 763B parameter model on 64GB of RAM?
replySSD weights streaming. The actual weights is 552B, then engram.
replywhich engine?
replyBy “quantitizing” it
replyThe smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.
reply