I would be extremely leery to use an automated hill-climbing loop, repeating the same inputs, and burning tokens on the same task over and over. That’s not going to move my project forward, and I don’t have unlimited tokens to spend.
Worse, how can I be sure that these agents will follow deterministic definitions of friction and priorities for removing it? I can imagine “write less tests” would remove a lot of friction. I see this just adding another layer, where I now would be documenting the system that writes my documentation. I learned how that is likely to turn out from playing the telephone game as a child: poorly.
This mainly automates the process that you are doing by hand. If you have your own definition of friction and priorities, you can tell this to your agent and it will fit into the CLI loop. You can review any change that is suggested, and spot-check transcripts if you feel the need.
It's like most AI systems in that you can have it earn more autonomy over time, as you add guidance and guardrails.
- https://supabase.com/evals - https://nextjs.org/evals
What is a better analogy for how you think developer tools should think about agent experience: an RL environment (hillclimbing a task like you mention), or a test suite (to be used in CI/CD, etc) as referenced by Supabase/Vercel?
The test suite eval style is useful and mostly a complement. You do want regression tests on critical features.
Thanks for sharing.