1. Guardrails - deterministic, social intelligence, team alignment & accountability 2. Learn by doing 3. make it stupid easy for the agent to research and access data 4. DRY
Research supports this. Try picking up some ideas from my harness: https://github.com/rush86999/atom
As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the model, it fills the context with conflicting instructions, it reduces the ability of the agent to respond to novel situations (and really in real life most situations are going to be novel). If you want to follow a process or a checklist you probably shouldn't use an LLM, or you should use it for some sub-tasks in the checklist/process but something more deterministic to work through the list.
How do you handle security?
Both “internally” against e.g. data loss, I’m assuming via limiting the harness, and “externally”, i.e. stuff like prompt injection risks?
If you want outlier performance from these models it is best to just ask in the most high level prompt of the most minimal harness and let them loose.
Any extra information reduces their performance.
However, as often as these models output masterpieces, they also produce utter garbage so our current choice is for them to have a process to follow that can be reviewed by humans and LLMs.
Not always. In my case LLM goes to grafana mcp, pulls metrics/traces/cpu profiles. Figures out what is slow and proposes a solution.
I do this constantly.
As the upstream comment points you, you don't need to specify. Sota models are that good. And by being overprescriptive you can accidentally shut off branches that they would've taken, downgrading the quality of their work.
In a perfect world everything is covered by distributed tracing and the problems are only in your application code and the agent just needs to find the data
In reality the data is often missing or misleading. “Your observability sucks”? Yeah, but that’s life
You could start by asking your AI "help me add better observability to our stack"
The native app I'm building on top, which I hope people who are less technical (or not technical at all) will use, is even more interesting because it's not just supposed to shell out to the CLI for everything and needs its own state.
I find your description intriguing but I'd like to see it to make sure I understand it.
I will however say that this page alone does a pretty good job of illustrating what an agent harness might look like: https://docs.agno.com/tools/overview
* System prompt
* Tool calls
* Model definition
Everything else (guards / etc) can just exist as code abstractions between the agent layer and the tool layer.
Could you give an example of an accounting guardrail you created?
Usually I’m returning TSV as a default format and I add a `help-all` subcommand to list every available command at once when needed. Another thing that helps is adding just-in-time context-sensitive hints, such as: user has just run a list query with at least one result. Add a one-liner to the response explaining the command shape for getting the detail view of the first response.
In terms of skill files, I like to have my CLI generate them dynamically at runtime by walking their own current command tree and then feeding that through a text template.
Examples from a public project: https://github.com/radiusmethod/gitlab-kiosk/blob/main/skill...
As for an example: if our agent wants to book a journal entry to cash accounts for a client, it MUST provide receipt and directly link the transaction from the clients bank feed, if it attempts to do so without the requisite information we deny the tool call and ask the agent to escalate back to the client for proof of receipt.
Often times this results in the agent not doing the work and instead sending a message back to the client asking for proof of the transaction.
For humans on our platform there may be valid situations where we’d want to allow this, but for our agent this is a hard guardrail thus why it’s not just standard validation for any JE posting on our platform.
if our agent wants to book a journal entry to cash accounts for a client, it MUST provide receipt and directly link the transaction from the clients bank feed