When orgs/bencmarks claim 1% deviation, in most cases that means measuring perplexity loss on datasets like wikitext or c4. Even if the loss is calculated via KLD or similar, its not a good proxy for whats actually degradaing at the task level across an entire rollout.

And for MoEs, very small amounts of loss can mean you're flipped to entirely different experts (this is also a problem more broadly with numerical stability issues too).

It depends on model size I think, but yeah, from my understanding at ~30B and below Q6 or even Q4 will get you 95%+ of the way there