GLM 5.3 Flash on 128GB Ram would have to be 2-bit right?
replyI'm no expert but it seems the consensus is you get a suboptimal experience under 4-bit. Please correct me if I'm mistaken here
MoE, especially these Chinese ones trained on low-bits already is actually quite quantizable to lower bits, unlike smaller / dense models.
replyAlso Expert MoE offload (to SSD), which still doesn’t let me (64GB) run a flagship but if you have 96GB you can.
reply