That's really interesting. I like Pydantic AI a lot and wondered why all of the harnesses seem to be written in Javascript instead of it. What do you use it for headless, though? I haven't tried Hermes or similar yet, so don't have a handle on what you do with them.
It's the killer feature from me personally, often when wanting to troubleshoot for example things like Kubernetes workloads. LLMs are now really good at it, but we doesn't want them to like delete a pod.
Other harnesses like Codex have often on static rules, like the allow/deny of claude code, that can filter out based on regex. It's quite good already, but sometimes the model can find a way to write something that wasn't anticipated, or in a convoluted way.
After, I guess it's something that can be added in an open-source harness like Pi, and add like this new Jev model or something else equivalent
Personally, I’m not convinced that a cascade of LLM judges lets me sleep soundly at night.
My agents stay in their sandbox and escalating changes to that env or prod is solely in the (human) operator’s domain.
Obviously this is less convenient but I think it’s a necessary tradeoff for security and correctness.