upvote
98 can be bad. E.g. uptime or aviation safety.

98 can be good I.e. human test score or funds recovered from debt.

I think for AI tasks it is somewhere between but 2% losses will add up (or you need more loops to correct and hence cost)

reply
Dont worry - it only hallucinates security breaking bugs in 2% of the code it produces. Comparatively :-D

In all seriousness I have no idea how they came to that conclusion. We really need better standards here for evaluating these things but obviously that is difficult.

reply
I think they mean "98% of the average score on these old benchmarks which are no longer representative".
reply