undefined

points

[-]

I would say they do compound until proven otherwise.

Having "Wait, bar is not true, so that won't work" is not necessarily a correction. In fact, the problem is: across a long text it is a correction of a single mistake, but we are talking about thousands here.

But yes, of course that was a rough estimate. But the problem is - we don't really know what we are measuring here. Maybe there's a 2,000,000x difference of intelligence between coding indexes 52 and 50. By some measure that just feels small because that's how we process it akin to audio db.

Regardless the point is KLD and whatever they came up with is not meaningful. And they did not publish comparisons on real benchmarks.

by maxbond11 hours ago|

parent|

[-]

Just at a sniff test level, don't you think that if the quantization resulted in anything like 2M% error in a pretty typical context length, it would be plain as day? You'd do an A/B test and one of them would look like standard generated text and one of them would veer into incoherence? If not - what on would 2M% error even mean then?

> Regardless the point is KLD and whatever they came up with is not meaningful.

I'm not saying you're wrong, I'm just saying this isn't a meaningful metric either, mostly because it is using a different type of error (divergence along a trajectory) than what was actually measured (divergence at a fixed point) and so can't be used for this purpose. It could establish an upper bound but going by your work that upper bound is so high it may as well be infinite. That's somewhat concerning but doesn't necessarily suggest it performs badly in a typical case (which is how I'd interpret an expectation of 2M% error).

by verdverm12 hours ago|

prev|

[-]

actually, bar is true

but wait, the models constantly go back and forth on these things in their thinking traces, so it is unclear which self correcting is actually correct

by maxbond11 hours ago|

parent|

[-]

It doesn't really matter to this argument, if they are self correcting at all, then we can't assume all errors will permanently injure the trajectory. It's not like dead reckoning or a similar process where there is never an opportunity to reassess. It's more like a long division problem; it is possible to correct errors using in band information, without the external reference dead reckoning would require. (The incidence of false positive self correction does matter to the question of whether the model is actually of comparable quality after the quantization, of course.)