The parallels with self-driving cars are interesting. We demand that agents don’t just produce better code than most humans (which the frontier models do), but that they perform much much much better.
To my surprise, none of them came close to producing an acceptable PR. Quality varied, but even the very best results were missing significant pieces.
My takeaway is that the human in the loop is still massively important for many things. And that makes it really hard to come up with a useful benchmark.