undefined

points

[-]

But quantized local models also dip in evals.

[-]

It doesn't matter if it's local or not, I just suggested local so people could test themselves.

But no, evals and benchmarks do not detect quantization, until around 4 bit quantization.

[-]

That is not my experience. I have issues where quantized models fail pretty early so I have evals to catch it.

And the second article seems to reinforce the idea. Yes, broad perplexity tests are not a good proxy for the tests used during training.

Quantizing anything is effectively changing the weights, even if you don’t quantize the weights themselves.