When I say "benchmark the query engine using the foobar dataset and compare it to run 431", the agents go and run my special benchmark tool and use the different subcommands to compare results and so on.
I'm sure a new VCS would be a little less smooth sailing, but not by much.
I highly doubt that your tool is like this:
> git branch -vv | grep ': gone]'| grep -v "*" | awk '{ print $1; }' | xargs -r git branch -d
Or:
> ffmpeg -i main_course.mp4 -i reaction_cam.mov \ -filter_complex \ "[1:v]scale=480:270[pip_scaled]; \ [0:v][pip_scaled]overlay=W-w-20:20[pip_video]; \ [pip_video]drawtext=text='LIVE RECORDING':fontcolor=white:fontsize=24:box=1:boxcolor=black@0.6:x=30:y=30[final_video]; \ [0:a][1:a]amix=inputs=2:duration=first:dropout_transition=2[final_audio]" \ -map "[final_video]" -map "[final_audio]" \ -c:v libx264 -crf 21 -preset fast \ -c:a aac -b:a 192k \ output_production.mp4
LLMs generate these for breakfast.
The crazy thing to me is that this kind of “composition of small tools to create something bigger” is the biggest vindication of the Unix philosophy I can think of.
I have to wonder how much of that behavior was trained into the model and how much it is the secret herbs and spices they toss into the harnesses own system prompts.
There are work arounds though and I am creating what I call knowledge triggers for Pi that are similar to claude's "PreToolUse" so having the agent use oak all the time is not an issue in my opinion.
The challenge for oak is why? Considering how I actually want to slow agents down so I can ensure it is doing the right thing and because the massive bottle kneck is the LLM themselves, speed when measured in milliseconds or even seconds will not concern many.
I thought oak was more of, we know how to prompt inject context based on code that is stored in oak for example, but faster operations can help, but the use case is limited. The missing piece for better/correct code is context at the right time.
There's a limit of how many simultaneous instructions an agent can follow (the exact number depends on the specific model so instructions that are fine for one model may overwhelm another). If this keeps happening, consider trimming your instructions or even better, solving it at the harness level (like intercepting and rewriting ripgrep calls to use your thing, like rtk [0] does in agents that supports this)
Overall, never leave to an agent an instruction that must be followed at all times. For example, doing things in a git hook beats a multi-command workflow every time the agent commit, etc.
Is this state of things forever? I don't think so. Very soon models will become so better this will be a non-problem
Another option: when model invokes standard tool, rewrite the invocation to newfangled tool.
Bunch of ways of doing it:
(a) Invocation of standard tool returns error saying to use newfangled tool instead
(b) Invocation of standard tool returns message saying it has been dynamically rewritten to invoke newfangled tool, followed by newfangled tool output
(c) Invocation of standard tool in context is dynamically rewritten to invocation of newfangled tool, prior to execution
In case (c), the model ends up thinking it somehow knew about this new thing all along, even though it actually didn’t
Regardless, in any of these cases, the implementation for any of these above options may be vastly superior to the “naive” implementation for agents — but then the parent comment here is right that an engineer would need to justify their implementation to users, not just make a loud conjecture. It’s a non-trivial claim to say that a bespoke solution not present in tool-use training and accounting for context-rot would result in a better performing model. Moreover, justifying an agent-specific efficiency gain that humans wouldn’t benefit from makes the claim even more non-trivial. Using Sagan’s razor, it’s then reasonable for people to ask for a comparably non-trivial amount of evidence.
In the end, it became heavy overengineering: people no longer understood not only the repo itself, but also the extra layer describing it. Meanwhile, coding assistants are already quite good at reading codebases directly.
Sounds like a good optimization to me. VCS is a waste of tokens for sure. I’m intrigued to hear more.
The thing we’re trying to optimize is not whether an agent can remember the command. It’s the runtime shape of agent-driven development.
When an agent drives a VCS through a captured terminal, things that are tolerable for humans become direct costs: clone/setup time, worktree setup, full status output, huge diffs, branch cleanup, interactive prompts, shared-checkout mutation, repeated preflight checks. Those costs show up as wall time, bytes over the wire, transcript tokens, and recovery steps.
So the Oak bet is narrower than “agents can’t use git.” They can. The bet is that if you assume branch-per-agent workflows, lots of parallel sandboxes, large repos, and non-interactive command execution, the VCS interface should have different defaults if you want to optimize for shipping speed and efficiency of token usage. If you're already going fast enough and not running out of tokens - then using oak seems pretty silly.
People do not need to ditch git to try Oak out. One workflow we care about is letting agents work in Oak where the agent-specific costs matter, then exporting back to git for the human review, CI, release, or compliance workflows.
Totally agree this should be provable and benchmarked. The homepage has Oak vs Git numbers because we do not want “for agents” to just be vibes. We’re measuring transcript bytes, estimated tokens, tool calls, wall time, large diff/status behavior, and contention in agent-style workflows. We’re also working on the benchmarks repo in the open: https://oak.space/oak/benchmarks
The exciting part to me is that we can already improve on tokens and timing despite starting with the model-prior deficit you’re describing. If we can win on measured agent workflows while git still has the advantage of being deeply baked into the models, I’m incredibly bullish on where Oak can get to as the tool and the ecosystem matures.
Longer term, if Oak proves useful and sticks around, future frontier models will likely have more Oak examples in training data, which lowers the upfront learning tax for an extra boost.