Thanks for the comment - I’ve actually been working on something to generate evals from private repos, as I 100% agree that public benchmarks are either contaminated (as OpenAI found with SWE-bench Verified), not high quality (as OpenAI found with SWE-bench Pro), or in the case of “good” benchmarks (Cognition’s Frontier Code comes to mind), non-representative of a specific repo’s work.

Sharing some challenges / learnings from the journey so far:

* Tests alone are non-representative: an agent can pass tests but write the code in a way that’s subjectively worse than another test-passing result

* However, tests are still important and the best form of deterministic evaluation we have

* Creating executable environments to run the tests is challenging, especially when considering that this should be doable for arbitrary repos

* Selecting tasks that discriminate is an art - they need to be challenging, but not too challenging, represent the variety of work that’s done in the repo, and contain tests that failed prior to the change

* LLMs / agents are very powerful at judging code - but doing so in a way that is calibrated, consistent, and representative of codebase standards requires careful rubric creation and grading

* I’ve shifted towards using agents (from “static” LLM calls) to generate the rubrics, and to later grade the rubrics. This is powerful but even more non-deterministic