One thing I discovered was that the worker agent, having access to all the skills, would sometimes expand scope unnecessarily.

This led to the agent making the solution "better" than the initial request, which is what I want most of the time in my actual development (e.g. /tmp/frame-N.bmp instead of a single /tmp/frame.bmp).

I ended up testing a flow where the supervisor chooses the skill(s), and only injects the subset into the worker. Not sure I love it, but it made the worker execution cleaner.

For the verifier (not documented in the blog post), I used a fresh-worker context that would attempt to adversarially poke holes in the solution. This worked pretty well, but required increasing the timeout by 2-3x (thus invalidating the benchmark).

[0]https://github.com/jumploops/chum