The difference in tokens between the two also makes super curious. The system prompt can't be that different (I'd even bet Pi's shorter) and the 4 tools shouldn't make as much of a difference. I'm gonna have to try it.
Do we have any insight into whether it is actually spam and not useful info such as project or programming language specific context?
but even injected context that when I read it sounds useful can oversteer the model and make it second guess or take a more complicated route than it normally would
(you can see this when looking at traces with and without that injected context)
often harnesses also mention in their system prompt locations of markdown files that the model can consult if the model thinks they might help
that hint alone as part of the system prompt can be strong enough to make the model read in more tokens than would have been necessary
'spam' is maybe a harsh way to say it
unfortunately I don't see an easy way other than to invest time and tokens into finding out which parts of the added context (in system prompt, injected in turns etc etc) are actually helpful or harmful and when
I'm just doing the easiest thing I could think of: start from nothing or close to nothing
that seems to work better than what most harnesses are doing
turns out GPT 5.6 Sol is all you need
You should use --disallowed-tools to prune any tools not needed for the task. Note that this is also a perpetual game of whack-a-mole since they’re always adding new tools.
It's so unfortunate they don't let you use the subscription with other harnesses anymore - since even if I used OpenCode they'd still get a bunch of useful data from the API calls, meanwhile I could stretch their tier limits way further.
Your subscription is not subsidised, it is just closer to the actual cost of the model…
200$/month is a lot of money on Luna/DS4 flash, like really a lot and the results are much better than clowning on bloated CC.
It's absurd how you have more and more organizations encoding their processes on LLMs and "engineers" (charlatan coders) don't even bother optimizing the tool they use most.
I won't argue with the cost effectiveness, but the results are very much not better. Opus and Fable are in a different league than DS4 Flash. Even GPT Terra, which I really like overall, sometimes gets stuck in weird loops and starts to do stupid stuff once its context window fills up. Whereas I can more or less trust the big models to just Do The Thing™ on the first try.
With that said, you get way more value out of a GPT subscription than you do from Claude, partly because of the ability to use more efficient harnesses.
I’m not sure if they let you skip the cache write cost on the first turn. That would imply cross-user caching infrastructure or special casing the default system prompt to give you a discount. Maybe? Away from the computer but you could try a “hello” in a fresh session and see what was billed.
for coding agents 'shell' is often all you need (just make sure the environment has the necessary tools)
It is everything. My experience with Claude Code is that you have to decide when to compact to make it efficient. It defaults everything to 1M context and it will never keep it in check. It is strange how little cache reads you hit in smol, that may be a configuration issue.
I’m asking because I’ve been looking for agent harness comparison tools too. I’m interested in more than just the inputs and outputs—I also want the system prompts, traces, and tool calls. It’s useful to understand why Codex, for example, uses more tokens while Pi doesn’t.
Fewer tokens aren’t necessarily better if the agent skipped important checks. On the other hand, using more tokens could just mean it’s overthinking the process. Either way, seeing the full execution trace for the same task is really valuable.
I agree fewer tokens is not necessarily better but a bit counter-intuitively often the harness using fewer tokens is not only done faster but has better results
(that said: of course check the results, look at the full traces, agree!)
the uncached tokens are also from runs where smol finished a task below 1024 tokens (the minimum amount of tokens needed to activate caching) which is less tokens than other harnesses are using for their system prompt (!)
> GPT-5.6 and later models: Caching is available for prefixes containing at least 1,024 tokens. This is a strict minimum.
https://developers.openai.com/api/docs/guides/prompt-caching
so in this specific case the count of uncached tokens for smol makes it look worse than it actually is
that said: it does makes sense to add more tasks that are difficult enough to fill the context window to compare the harnesses for how well they deal with compaction
staying below compaction (or with compaction at fewer compactions) is not only cheaper and faster, it also helps the agent stay on track
also, are you using a tool to collect those metrics? what is it?
https://news.ycombinator.com/item?id=49006862
I'll have more about it in the next hours/days, you can follow me on twitter in the meantime (https://x.com/__tosh)
like creating a checksum of a file, merging csvs and so on, fixing a makefile pipeline
with known 'good' outcomes
all harnesses could reach the outcomes, only cost, time, number of tool uses and so on were different
(Claude Code failed once in 1 task but I think that was just an unfortunate outlier, the tasks aren't that difficult)
Be careful here. Remember these are non-deterministic models at the end of the day, and even with everything being "the same" you can have two runs where the same model, same harness, same tools can arrive at the same conclusion through a wildly different sequence of events.
I will add more tasks (esp longer ones) and think more about grading, the current tasks were easy to grade because the desired outcomes are well specced but I will also look into more open ended tasks and how to grade those
thank you!
the "spamming" is mostly compaction appending files, tool artifacts, images in its summary there is a known github issue for codex. only solution is periodic clean up but its also how a lot of the agentic orchestration is performed and able to work for days.
I will look into how token usage looks like for longer sessions and more complex tasks
re caching: the cache ratio for this bench looks 'bad' for smol because it often finishes a task before caching kicks in (caching starts at 1024 tokens)
thank you for flagging this