Yes we are (it's MIT licensed)! Feel free to email me at jeff@momentic.ai or file a pull request directly on https://github.com/momentic-ai/qa-bench
replyEverything has to be reproducible and runnable by the general public
Thanks, that makes sense!
replyOne suggestion for the future: QA Bench would be even more valuable with a human benchmark, so you can clearly see how the agents perform compared to human QA testers. That would settle that discussion also :)
Haha amazing idea! FWIW in the past we have worked with external QA to label ground truth but it was so riddled with errors we decided to put 3 full-time SWEs on it instead.
reply