We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents.
We paired this with skills initially, but found that the way folks built skills was often too prescriptive and limited to the authors own specific function in the company. A 2k line long skill suffers from the same gaps as we do, if an agent is just following a laundry list it’s less likely to reason about the request it’s doing.
So we instead asked ourselves: what if we just _let_ the agent reason about the work to be done and only provided the tools + guardrails to gather context and perform accounting work?
Turns out frontier models are GOOD at what they do, they outperformed our highly prescriptive skills and were able to work across a larger set of tasks even without instruction on how to do those tasks.
It’s a breath of fresh air from the decade of CRUD I’ve worked on, harness engineering is very neat.
1. Guardrails - deterministic, social intelligence, team alignment & accountability 2. Learn by doing 3. make it stupid easy for the agent to research and access data 4. DRY
Research supports this. Try picking up some ideas from my harness: https://github.com/rush86999/atom
As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the model, it fills the context with conflicting instructions, it reduces the ability of the agent to respond to novel situations (and really in real life most situations are going to be novel). If you want to follow a process or a checklist you probably shouldn't use an LLM, or you should use it for some sub-tasks in the checklist/process but something more deterministic to work through the list.
How do you handle security?
Both “internally” against e.g. data loss, I’m assuming via limiting the harness, and “externally”, i.e. stuff like prompt injection risks?
Not always. In my case LLM goes to grafana mcp, pulls metrics/traces/cpu profiles. Figures out what is slow and proposes a solution.
I do this constantly.
As the upstream comment points you, you don't need to specify. Sota models are that good. And by being overprescriptive you can accidentally shut off branches that they would've taken, downgrading the quality of their work.
The native app I'm building on top, which I hope people who are less technical (or not technical at all) will use, is even more interesting because it's not just supposed to shell out to the CLI for everything and needs its own state.
I find your description intriguing but I'd like to see it to make sure I understand it.
I will however say that this page alone does a pretty good job of illustrating what an agent harness might look like: https://docs.agno.com/tools/overview
* System prompt
* Tool calls
* Model definition
Everything else (guards / etc) can just exist as code abstractions between the agent layer and the tool layer.
Could you give an example of an accounting guardrail you created?
Usually I’m returning TSV as a default format and I add a `help-all` subcommand to list every available command at once when needed. Another thing that helps is adding just-in-time context-sensitive hints, such as: user has just run a list query with at least one result. Add a one-liner to the response explaining the command shape for getting the detail view of the first response.
In terms of skill files, I like to have my CLI generate them dynamically at runtime by walking their own current command tree and then feeding that through a text template.
Examples from a public project: https://github.com/radiusmethod/gitlab-kiosk/blob/main/skill...
As for an example: if our agent wants to book a journal entry to cash accounts for a client, it MUST provide receipt and directly link the transaction from the clients bank feed, if it attempts to do so without the requisite information we deny the tool call and ask the agent to escalate back to the client for proof of receipt.
Often times this results in the agent not doing the work and instead sending a message back to the client asking for proof of the transaction.
For humans on our platform there may be valid situations where we’d want to allow this, but for our agent this is a hard guardrail thus why it’s not just standard validation for any JE posting on our platform.
if our agent wants to book a journal entry to cash accounts for a client, it MUST provide receipt and directly link the transaction from the clients bank feed
When I say handoff, I mean:
* handoff from a terminal CLI to webui (on a phone)?
* handoff from one team member, to another?
* handoff from one communication modality, like writing a prompt in a TUI, to email?
* handoff from one model to another, or one provider (openrouter)( to another (llama.cpp)
I used to think that a PR would be a good place to centralize all this. Who cares what IDE, or developer, or location. But, now I feel like an agent harness might contain that better.
Why do I want handoff? I keep losing context of where my harness is running. Sometimes I am inside an isolated VM. Sometimes I'm on my laptop, sometimes I'm on my home machine with the big GPU for local models. If I could spin up a harness that could identify itself inside my tailscale network, then I could probably have a single web UI which allows me to keep all that context straight.
I'm tempted to experiment with Pi to configure such a thing. But, perhaps there are patterns out there already with a harness I have not considered.
Not sure how others do it, but opencode stores sessions in a sqlite db and you can extract them and share them as needed.
https://opencode.ai/docs/cli/#export
Pro-tip: Building your own extremely minimal harness takes about 15m and is both fun and enlightening. Agents are unsurprisingly quite good at it, but ask them to walk you through it step by step.
Let's assume handoff happens when one "agent" finishes its work on one task, i.e. "submit a PR".
At that point you want to exit the agent/clear context etc (any context the next actor needs should be in the handoff artifact).
And the orchestrator calls the next agent with the artifact.
Claude can do this with subagents. If you want to get more serious, I'd look at "durable workflows" and check out what the pi people have to say: https://earendil-works.github.io/absurd/ https://earendil-works.github.io/absurd/patterns/pi-ai-agent...
you should also look at dbos https://www.dbos.dev/
And then do a search for these terms on HN and get some idea of their shortcomings vs a 'real' orchestration tool like Airflow or Dagster
harness = chassis, model = engine, fuel = tokens, agent = car
I’m curious what y’all might think and whether that analogy carries more explanatory power
I guess you could say that tokens = hay, and agent = horse and cart, from there? Not sure how useful the hay part is but you could observe from the second that there are many different things you could harness a horse to (also a plough, or a coach, or just a saddle) based on your goal.
Saying the harness is like a car's chassis doesn't work as well for me because the chassis isn't as configurable as a climbing harness for as little work.
Getting deeper into the climbing analogy you can even swap out the harnesses themselves for wildly different climbs. Like using Claude Code with a bunch of agents for medical software (climbing K2 where that extra padding comes in super handy) and pi.dev with a local model for a respectable web project (sport route where you'll be back in a few hours and it's safe to be a little more exposed).
I'm glad your article made HN, and thank you for pi!
And Pi is the best harness because of the amazing extension system. You can build extensions that turn Pi into a stock trader, software factory, anything. I tried switching to another harness but none have extension functionality as good as Pi.
Even if there is a new harness or agent project, I tell Pi to dig into the codebase and then make me an extension that brings that functionality into Pi. I did it with Prime Intellect’s and Deepseek’s harnesses and those are built on Pi.
Don't get ahead of yourself. Harnesses are not exactly rocket science and will be a commodity.
The real value providers here are the hardware, then the LLM as a distant second, and at a much larger distance the harness.
Labs are now post-training models with Harness so that Harness now gets absorbed into the weights.
Solar goes all the way up => power is commodity.
Some hyperscaler goes bankrupt => hardware is commodity.
Models get real good => output is a commodity, no profitable problems to solve anymore.
Open source models get good => models are commodity.
I really though this comment was a satire ...
This entire forum is infested with shameless hype chasers and biological linkedin bots.
I’m thinking of how in cyberpunk, people are replacing their cybernetic enhancements all the time. You could alternatively bioengineer your own body towards the desired outcomes, but that’s more constrained by the trajectory your body has already taken, whereas the promise of cybernetic parts is that they are more independently replaceable. (Probably an illusion in practice, but I’m talking about the fictional ideal.)
As another analogy, monolithic software tends to quickly become hard to change significantly, whereas a plugin architecture tends to be more flexible and modular, and people can share and combine their various plugins.
They probably used an LLM to come up with this bizarre metaphor.
- They can already reason better than many humans and are still improving all the time
- Harnesses are improving all the time
- We're already exploring things like long term memory, long term goals, and other things that humans have which LLMs traditionally lack
- An AI agent can read and reason about every piece of AI research ever published, including looking for insights that humans may have missed. A team of humans could never do this even if they dedicated their whole lives to it.
- They can design and execute experiments on a mass scale to determine what does and doesn't work
- Large AI labs have more than sufficient resources and motivation to throw at the problem, and are in fact doing this.
If you provide what you'd consider AGI, we may not agree on that definition, but I and other skeptics could at least discuss with you whether A.) that seems reasonably achievable given LLMs inherent limitations and B.) whether any of what you'd listed is actually likely to get us there.
As it stands, neither is possible without knowing what you believe AGI to be, but for what it's worth, coming from someone who both does see LLMs as valuable tools but whose definition for AGI also contains, among other things, reliable self-assessment of factual uncertainty [1] and basic counting and grade school maths [0][2] without tools or eternally scaling training data, I have yet to read any evidence that LLMs can achieve my, rather strict, metric for AGI.
These models are amazing tools, their ability to leverage massive amounts of high quality training data to further sciences truly awe inspiring, but that does not mean intelligence, at least in my definition that requires some internals these models have never been proven to possess. It's nuts that solving Erdos problems can be done by a model which struggles to count or solve a sudoku without external tools, but that's where the technology has been for years now and no paper I have read has shown that LLMs can overcome that to any scalable degree. You can push further with training data, but the limitations remain, albeit less noticeable. Any externalities, be it tools (self-scripted or called by the model), external memory solutions of all shapes and sizes, etc. I personally also feel cannot be required for or lead towards AGI as intelligence may be better leveraged by such externalities, but should never require them, so much of your suggestion I feel shouldn't be considered even if one believes LLMs can yield intelligence. I will admit that I am very extreme here though, this is not a position held by everyone for good reason. At the end I will always point towards the "extraordinary claims require extraordinary proof" of it all and that LLMs, in the face of any doubt, should be viewed akin to how Stockfish can play better than any grandmaster, but that does not mean intelligence, at least in my world.
If your definition for intelligence does not require basic arithmetics or an understanding of ones own knowledge gaps, then maybe LLMs can achieve that, but I'd push back on that truly rising to the AGI moniker. Maybe a more comprehensive or even my definition of AGI is possible whilst keeping the autoregressive nature after all, but there is no evidence supporting that by itself and quite a few things that haven't even begun to be overcome before something of that magnitude could be honestly considered.
It's akin to "let's colonise Mars by 2020 or 2030 or 2040 for sure, then terraform it" proposals. If that were possible, wouldn't we see a lot of these methods applied on earth and in a moon base long before (as in, we'd have had a permanent moon base in the early 2000s)? Same with LLMs, if they can truly yield AGI, we'd see some of the major deficiencies dealt with long before. The fact that we neither are terraforming earth, nor have any permanent off world colonies, nor have solved some of the listed, inherent limitations with LLMs by their design, that's what informs my skepticism that both are reasonably achievable in the timelines some industry "experts" (read hype merchants) propose on the regular. You tend to see some progress, a path toward solving actionable problems long before full implementation, at least in the real world...
[0] https://logicalintelligence.com/blog/energy-based-model-sudo...
And no I came up with the metaphor all on my own, send me the chat of you getting the LLM to come up with it. Why not argue based on merit instead of strawman and ad hominem attacks?
Harnesses (and the concept of agents before them) presuppose competence in LLMs which simply doesn’t exist.
0. https://www.businessinsider.com/sam-altman-ai-utility-electr...
His idea of metering is predicated on the thing he’s selling being AGI, it is not, and all his predictions have turned to dust.
Also that isn’t how metaphors work - they illuminate by comparison, if the comparison is not close they are not useful.
I don’t believe in AGI, but that doesn’t mean I don’t find AI useful. I just understand that the correct harness can take them to the next level.
Then how do you explain the wild success at using them for development?
That doesn’t make them intelligent agents which think independently.
I have a system that entirely reverse engineers old arcade games. Creates semantic symbol mappings that were considered impossible just a couple years ago.
Granted, it took me a couple weeks to build the system.
From impossible to a couple weeks in just a couple years.
Would you like to see it or continue to pretend these things don't exist? Your call.
(It's finding the coolest stuff - the anti-tampering hacks they put into the old machines is fascinating.)
You can never tell if the goomba opinion of the forum will agree we have reached AGI (seen that happen on a few threads lately) or will readily call that a ludicrous proposition.
The words "once that settles" are doing historic levels of work here.
No human on earth has a clear idea whether model technology will settle tomorrow or 100 years from now.
There's every reason to expect architectural breakthroughs will keep being discovered and causing nuclear blasts of forward progress.
I do agree that harnesses are going to extend AI capabilities a lot in the next year, but after reading Pi's page I don't see anything that makes it particularly special in terms of functionality, other than being more provider-agnostic.
Many of my harnesses eventually turn into customized UIs around the chat interface.
The harness facilitates the work animal doing work for you.
Not climbing harnesses to keep you safe.
1. You can use the '/new-tool' and tell what kind of tool you want (including whether it should be task-scoped, workspace-scoped, or global), the model builds it, the harness runs validation and other tests until the tool is ready
2. The model decides that in such and such task, it would be helpful to have a tool like this, it can build a task-scoped tool.
In either scenario, the tool catalog is rebuilt, and the new tool is instantly available in the next turn.
What I can see is a world where we end up with a Chromium-shaped harness, a fully featured standard implementation everyone builds against, because doing every single thing yourself would be crazy.
The antithesis to Pi, if you will.
Also, having only a "standard implementation" makes no sense for a harness. A standard implementation would need to try to be as good as possible at all things. But you'd often want a specialised harness designed for exactly your use case.
Some standard solution will emerge, which will be amplified by models being trained specifically to work with it.
I primarily like how it manages sessions, and how agents can easily reference other sessions.
So if you do want to use it, use the Codex sub. Once you install it, run Pi and /login and you’ll get login with ChatGPT. From there, Pi can tweak it’s settings if you ask. Check out their extensions (or ask Pi) and that will take you most of the way there.
What hiccups were you having?
Not out of the box, but you can add agent sdk. I'm not sure how great the results will be though.
looking at the website. i can't really tell if they have benchmarks and measuremnts on how all that improves capablities over just using regular agent withtout all that
Don't understand what people see in them.
also i think its hard to build general harnesses if they were trained on specific harness architecture.
There’s evidence of harnesses making a smaller, weaker model perform better than SOTA and some benchmarks ban harnesses because it becomes too easy.
Well kind of, I wouldn't be surprised to see that some things marketed as agents are actually good old deterministic software.
It truly proves like there's a handful of thought leaders on Twitter that everybody follows blindly and start to copy down to the lexicon and parrot everywhere else.
Right now every tool is shipping some kind of AI agent, but I can’t help but feel that AI agents in large companies will eventually be some kind of internal app with internal MCPs, CLIs, APIs etc.
There might be different harnesses for different use cases that different people have different levels of access to.
This would make sense for the platform/infrastructure engineers who can build a modular harness that a person or team can get access to.
You could have agents team members use locally that have memory enabled for personalization and then agents that anyone can use to ask questions about company context, which wouldn’t personalize things.
the harness is what you take with you on a trip/task
whatever you take with you is not free (system prompt, tools, skills …)
some models are really good even if you bring almost no skills, tools or system prompt
the harness is the complement to the model
the better the model the more minimal the harness can be
harnesses like pi [0] and smol [1]are on the more minimal end of things
Anyway I've been building my own harness on top of pi- www.freepi.ai (it's based on Pi, but now I have an OpenAI compatible endpoint so I'm thinking of it more like free-api :-) ). Basically ad+training supported so I can offer completely free inference. It's really important to me that we don't have harnesses and intelligence trapped in a "have and have not" world. If we don't all have access to intelligence we will end up in a dark place.
Thats again where the visual of Steven Hawking and the wheelchair really stand out in my mind. It's not enough to have the raw intelligence, we need a really good wheelchair too.
FYI, the pic is Royal Robbins (https://en.wikipedia.org/wiki/Royal_Robbins), who was kind of a hero to some of my climbing buddies back in the 70's.
https://github.com/aaif-goose/goose
Full disclosure: I work at the LF, but not the AAIF.