> I'm confused. Who made this benchmark and why?

It's a thinly veiled advert for Atopile. I don't know if it's a company behind it or just someone with a lot of money to spend on advertising since their about page (and docs) both 404, and the "packages" link sends you to a login screen that tries to unlock 1password. I guess claude can make circuit boards but not functioning websites?

We spent a lot time evaluating which model is the best to use in our atopile agent harness and thought the benchmark and results are interesting enough to share.

Evaluating a model for specific use-cases like this that are bit more broad scope than a lot of typical microbenchmarks turned out to be quite the challenge.

Determining deterministically what a "good" electrical design is non-trivial and most electrical-engineers rely on good ol' intuition and decade long experience, so we tried to formalize it.

The only reason it was somewhat possible within a reasonable amount of effort is because atopile provides most of the complicated infrastructure for the benchmark: constraint solver, simulation, code-first electronics modeling, erc & drc checks, high level models of passive electronics components etc

Humans learn differently from LLMs. LLMs already contain most of the information they need to be able to do PCB layout from reading the entire internet, but they need this kind of fast feedback loop to learn how to extract and use that information in the right way. Humans learn to act at the same time they learn knowledge.

because human engineers are not reproducible, so having an accurate measure of the skills of one of them has a much lower utility return (those benchmarks are very expensive to run, and making them for humans will be at least as much expensive)