Thanks, that makes sense!
replyOne suggestion for the future: QA Bench would be even more valuable with a human benchmark, so you can clearly see how the agents perform compared to human QA testers. That would settle that discussion also :)
Haha amazing idea! FWIW in the past we have worked with external QA to label ground truth but it was so riddled with errors we decided to put 3 full-time SWEs on it instead.
reply