Man I am having a hell of a time trying to optimize 3.8 over 3.6. I don’t have a particularly powerful setup but I can usually push 20-30tok/s on 3.6 and I can barely get to 10 on 3.8. Both unsloth same VRAM/RAM distribution more or less. My 3.6 is still producing consistently better results and faster
replyThat's interesting since both models are dense. I wonder if this is more of an optimization issue with 3.8 rather than something inherent to the architecture.
replyCould have sworn I read these were the same architectures the other day .... 3.6 and 3.8 at this size.
reply