Update: let me elaborate more, I read the HN entry about the team of 5 Qwens being close to fable quality... so I embarked to add to my harness (https://github.com/alainnothere/privibe because we all have one right?) and once I had added the orchestrate tool I had to now measure somehow it it's able to improve quality, so I claude and then Qwen to run a experiment to call my harness passing the model to use and ask them to do something, 3 rounds I think and see what quality came out of it... and as per the grading... qwen and claude agreed that the quality of the response comming out the harness using the orchestrate was VERY CLOSE, with opus and fable providing more "in my opinion you should" which matches my experience on why they are better.
One more update: I can't get copilot models to work decently, it may be my way of working... but I can get Qwen to follow what I try to do... if Qwen get's me he can then fire copilot cli and get amazing results by using copilot... bonus points seems like copilot cli is cheaper with the AIC (cost of the requests) so you can make the copilot subscription longer
Smart agent to orchestrate cheaper agents for labour.
Dumber agents to be very explicit about asking smarter agents for requests that require deep information.
This is a thing - the SOTA models really light up the more information you can provide them, and humans are too lazy / incapable of doing that in every prompt.
Entirely depends on what you are trying to do.
Edit: you can also use a smart agent to ask another smart agent, obviously, which is good for personal workflows.
But there are definitely cases.
GPT Pro is an absolutely spectacular model, that almost cannot be used without 'prompt help'. Or rather, it can only be effectively leveraged by very intricate prompting, which requires the help of another agent.
I typically use Claude Code to write prompts for GPT Pro , and cut and paste the prompt in, and let GPT Pro work for 20 minutes, then take the response back to Claude Code and work with Opus to consider the results.
Opus is 'dumber' (nominally) then GPT Pro.
What's the best method to have a 'smart' agent just.. ask itslef?
You may use https://github.com/pingdotgg/t3code as a wrapper above claide/codex/pi/... harnesses, or a proxy below (forgot which one, as agent what theo uses for balancing multiple claude subs)
Asking Claude to make prompts is like a Math teacher stuck working at a cafe about Math problem, the just light up.
They are very good at at it.
* Orchestrator: Opus 5.5
* Feature lead: GPT 6.1 Sol
* Worker: Muse Spark 1.3 Contributor
GPT 6.1 Sol is surprisingly efficient too, I've been impressed with it.
My orchestration skill setup is https://github.com/beefsack/processed-beef/
Does anyone have any evidence that having the most powerful model being the orchestrator, and other sub agents being progressively less powerful, is optimal? Is this just common sense or it has been measured somehow?
For me it could be that this scheme may make the code itself poorly written, verbose or faulty
In my setup, the orchestrator also drives a lot of the design and planning, and the less capable models get inputs from the more capable models by responding back up the chain. The more capable models also review everything that is produced by the cheaper models.
I've also had systems that invert this and allow cheaper models to call out to more capable models when they need to, but in my observation I found them quite bad at knowing even when to call out and I find wrapping the cheap models a bit more reliable in that problems tend to get caught before they bubble up to the user.
I'd be interested to know what the difference in cost is in this setup on building a complex feature vs using Opus directly.
At this point if I could do anything with zero effort or time, I would get Opus 5.5 to make a grand scheme, handoff to the local model, and give it ability to reach out to Opus again.
That way you can do old school intent matching and rule-based bots that work super well, and only shell out for deeper work.
Claude will write some scripts to call the LLM API and use it like any other tool.
I haven't found a pressing need for anything more complex than that. My multi-agent setups tend to be a pile of bespoke scripts per project anyway, an MCP server is only gonna get in the way. But I suppose this is because I self-host the models, if you're using OpenRouter or whatever it may need more complexity than a curl call to an unauthenticated port
So with the gdrive app for my OS it just looks like a local file path. Had the local agent learn to parse it.
I still prompt in AI studio direct as needed; local agents see the log change but cannot push back to Gemini their result.
Few reasons:
AI Studio is available where ever via browser, no VS Code or my usual vim setup to miss. I can switch between projects there as an idea hits and the agent switches up the local repo path and session history.
AI studio can search web, etc, etc already so my agent is less complicated with addons in general
Also using AI Studio between local agent turns has been my "human in the middle" approach; to have an explicit gap in the process and check each turn.
I only have a 24GB 3090 most models also take longer locally and I never hit the free tier limits with Gemini :D
Though the latest Meta model is twice as fast at 30B for me as other similar size models. 1-2 hour code gen and QA of build errors is now 30-50 minutes to parse latest changes from Gemini and get a successful build. The Meta model is pretty great at "fill in the middle" from a well structured plan.
Overall this loop feels like a normal workday pace. I freelance so I have a lot more say over that.
Ps: as you won't use your proxy to distill or alter claudecode requests to anthropic you should be compliant with anthropic rules.
Both are, I think, valid options. Depends on the workflow you're looking for.
https://opencode.ai/docs/agents/
There are a lot of ways to do this if you stop using Big Ai harnesses that make it difficult to use models besides theirs.
How is this different than just telling the model you’re using to fill in the gaps itself?
If you’re not manually reviewing the intermediate hop, how does that add to the “understanding you”?
Why Qwen directing the others? For the things claude refuse, Qwen can make very specific questions and still get an answer without triggering the thing.
It's like, find the relevant scientific papers by using Google search and the scientific paper MCPs, summarizing stuff, etc. Then the big model can come in and actually think on the artifacts
I ask task x -> big model thinks and delegate to luna heavy dumb work.
My sense so far is it’s very much about problem domain “taste” and there is no one correct answer (that is token cost efficient).
Does the smartest agent need the full flavor of the context of the problem to make real progress (i.e what’s been tried, how you’ve influenced the approach) or does the smartest agent really need a crisp problem definition and context room to really work.
I too wonder the same...
The real-world provides us many examples of such big -> little, or smart -> not-as smart workflows. For instance, we don't typically ask juniors or mid-level SDEs to design solutions to exceedingly challenging problems for a reason. However there are _some_ areas that those with less experience can tackle that is relatively risk-free if their work is reviewed by a more experienced engineer, which leads me to a conclude that it's task dependent.
https://code.claude.com/docs/en/advisor
It allegedly will automatically judge when to call the stronger model, not sure how much instructions in prompts or AGENTS.md can steer it.
If we compare coding agents with Jev on vs off we get those numbers (metered with Memdoor): - read-heavy tasks (coding questions , logs tail etc..): -63% to -65% input tokens, 40-45% fewer model calls, on GLM 5.3 half the wall time. Reproduced a month apart on two models (DeepSeek V4.1 Flash, GLM 5.3). - a file edit: -23%
Two things match the post. Calibration is what makes it safe: I apply a decision at confidence >= 0.75 and fail below this threshold or on timeout or on an unknown tool family. The dollar saving scales with the price of the model being saved: on a $0.04/M model Jev costs what it saves so on a $0.38/M model it's negligible but on Claude model for example it can be outstanding.
Where it did not help: giving Jev more context about the repo. I found same thing as the post.
If you are interested i have put figures here : https://github.com/guregodevo/memdoor/blob/main/docs/feature...
People, please try to understand what you are building. Don't just build it as asked!
That's Hacker News for you. They have chronic tunnel vision and are unable to look at implications that go beyond the technical. Particular blind spot for techno-social consequences.
With an assemblage of models like Jev + Opus or Jev + Qwen, Jev would often do the right thing when deciding to escalate to another model - I was impressed that it would usually make the right call, especially if you gave it good awareness of what the other model capabilities were. "ESCALATE to opus if you need good vision awareness".
When Haiku 5.5 came out though, for my tasks, the cost differences disappeared almost immediately. Haiku was good enough / fast enough for so many things that introducing a decider layer didn't offer huge wins. Jev's "millisecond" level performance was nice but would get dwarfed by other latencies in my chain ( web browsing ), that it didn't make a measurable difference in my tasks.
Jev's confidence score was more reliable. Which means that they were able to use Jev despite it having lower accuracy, because of the confidence was below a cut off, they would escalate to an expensive model (Opus)?
Whereas Luna, even though it was more accurate than Jev, its confidence was way off, so they couldn't reliably know when to fall back?
- Mark Twain
Despite the title, the article is not making claims of, or demonstrating general usage improvements or general coding agent improvements
Why give all the context to an agent, when the context can also be curated, updated and made concise automatically for each new problem.
Not really a good analogy, but it feels like having an image you like but every time you view it, it runs it through another loop of jpg compression. Use it when it's most effective, otherwise let it preserve it's details.
wait till they found out about that nanosecond latency “conditional model”
turns out I’ve been an “RLIF” training expert my whole career
They propose:
Valentino → Female Child
Yet this is obviously a boy name, else it would be Valentina. Is this model going to return confidence values? No, they want decisions. This is an inherent problem in AI with reductive logic for these tasks.
Likewise, dates. Is 17/08/22:
- 20170822 (Chinese write the year first, and in speech and chat they will say '17' not '2017'.)
- 20220817 (Everywhere But North America.)
Is 04/06/2009:
- 20090604 (Non-North American)
- 20090406 (American)
Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.
It’s an interesting problem. And i have following thoughts to it?
- Are all cases even decidable?
- Shouldn’t one false positive rank higher than a hundred rank negative.
-Shouldnt’t the AI just abstain from decisions to avoid false positives.
- Is it smarter to seperate this in a lot of small sub problems and to define this as a rule set of small yes no decicions, why does this need to be solved agentically?
- You should definitely clean up deterministically first.
I feel like if you really want to solve this problem, you would approach it differently. This seems like a wrong approach and a bad article on top?
No, and I think undecideable is quite frequent.
> -Shouldnt’t the AI just abstain from decisions to avoid false positives.
Yes, and I think regulation should be quite strong in this regard.
> - Is it smarter to seperate this in a lot of small sub problems and to define this as a rule set of small yes no decicions, why does this need to be solved agentically?
For larger contexts I worry about agentic confabulation, but humans unquestionably look at the gestalt when doing matching. I don't want idiot clankers making enforced decisions because they have misunderstood one field.
> - You should definitely clean up deterministically first.
If you can't do field validation on entry then I'm not sure deterministic rules work better, they may be worse, although easier to reason about.
I personally wouldn’t lean towards LLMs on identity resolution. I would like give it the dataset and then ask me questions about specs? I really struggle to find the LLM part reasonable here. I mean potentially some typos, or nicknames, but those could be potentially resolved deterministically also.
I would probably have reviewer running that suggests new rules that I could implement. To me this seems more like compiler work than LLM work.
Cf Lies we believe about Names, Addresses and Dates.
Perhaps there’s something else in the post that you’re talking about but please do take the time to actually read what’s there, I’m only telling you what’s written in the text around the problem you think you’ve found.
The system doesn't return confidence values because that's not what they are designing. The models, however, do and to me it's interesting that Jev's measure of confidence is pretty well calibrated to its correctness.
>Without knowing the cultural context, the location, the document type, the document author, etc, you could be getting it wrong.
And? There's no perfect process, just ones that asymptotically approach correctness at exponentially increasing cost. They own the system and get to decide what works for them.
Rather than binary classification we should be locating document fragments in a high dimensional feature space, and with all those fragments coming to a confidence bounded guess.
Additionally, when someone submits documents that can't be recognised as valid, the response should not be INVALID, it should be UNDECIDED. In that sense, passing to Opus is great, but passing a tiny fragment of the problem is a bad way to do it.
Give me a decision model that can do both, more expensive than today’s decision models but still cheaper than full on frontier model, and it’d open up so many more use cases, IMO
I agree. What I’ve encountered though is even building enough cases for a calibration data set is hard. And so we farm out to smarter expensive models like Astra to label cases and use that as a calibration data set for the now cheaper LLM judge actually running as our judge
Is this standard ? Or am I just too lazy to manually label enough data to calibrate ?
I don't usually get excited about the high-level, "whoooa man" marketing-speak, but the parallel is striking.
I don't have the actual data but based on the graphs I estimate the expected calibration error to be 2-3% lower for Luna on both the tasks.
Its true that Luna seems to give lots of high confidence values but it's also correct more. Higher number of mistakes at higher confidence isn't itself undesirable if its pass rate is also high. It'd be good if they reported actual ECE.
>Most of Jev's mistakes are closer to 50% confidence, which is what you'd expect out of a well-calibrated model.
This statement is also incorrect
On a serious note: all the heat and light is about AI for automating coding tasks, but a significant amount of management is just theater about deliverables. So much of that could be replaced by better product delivery pipelines that can accurately generate the "when can I expect this release?" without a layer of management that considers sitting in zoom calls all day work.
But maybe I'm just a little bit too jaded.
That isn't obvious, unless you've been doing ML for a while.
These are the results of a little performance tuning/permutation testing run against context format and question wording we did. Obviously there is a lot of noise here but I was actually kinda surprised that there was a clear "best way to ask the question" even for relatively low N and low complexity/ambiguity of the task.
(This is a ~30s Ai-driven gif presentation of the results, the full grid shows up about 15s in): https://imgur.com/a/wGFZvGw
These models in particular have some pretty dubiously generalized and diverse training datasets. You can see this for yourself BTW by looking at the datasets SOTA Jev-like models are being trained against and evaluated on. No knock against the teams btw, I just think it's rather demystifying and makes me quite optimistic at how much better these models are going to get over the next few years: https://labs.baseten.co/articles/mammoth-1-and-elephant-1-fr...
Some problems are obviously hard known-unknowns (eg “solve Navier Stokes”). The problem is unknown-unknowns or knowledge gaps, where the model doesn’t know they need help.
I don't believe the answer is ever "how certain is the model?" It's "how much is at stake?" Or "how hard of a problem does this appear to be?" It's not about teaching the model to have uncertainty, but rather to be avoidant of things that are sufficiently important to get right.
If you say 4x lower, that changes the semantics for lower than x vs higher than x, which is absurd.
where does it say in the rulebook that human languages have to have perfectly symmetric pairs of words?
Hope my sarcasm is making it across.
GPT-6 Luna is a small LLM, not a decision model :P
It's what I use locally. For my workload, it was right up there with Jev.
Similarly, Grok has been routing to Opus internally for a little while now I think.
Any source on that?
And yes, that saves money since less tokens are sent to a remote LLM.