I'm not looking to be convinced about this tool as much as I am genuinely looking for a way to eval these kinds of tools in general

I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another

not super pleased with it

the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for

one thing you could do is run it against FrontierHarness eval and see how it stacks up?

Yeah learn how to do even the basics of science, please. Control groups!! Confounders!!

Letting another LLM decide quality probably isn't the only meterstick you want, either.