upvote
"Cost per pass" is distinct from "cost per successful task."

Cost per pass is simply the average cost per task divided by the pass rate.

If you draw a line from the origin to any harness on that chart, its slope (pass rate ÷ cost) represents passes per dollar. Cost per pass is the inverse of that slope, meaning the lowest cost per pass gives you the steepest line.

reply
In this case, I am not entirely sure how cost per pass is distinct from cost per successful task given that success is defined as pass/fail.

To illustrate the difference, the author of FrontierHarness included "Figure 3. Cost and pass rate of each harness with each model", in which we can see that Codex CLI has $0.937 cost per pass, but a $0.573 cost per task. Measuring cost per task also changes the winner to OpenCode

reply
We use cost per pass, not average cost per task, because we want to see the harness keep a high pass rate and low cost at the same time.
reply