I have just re-analysed most common failed tool-calls and adjusted the tool so that it works. I have a manual repair step on failure that programmatically attempts to fix some things. On failure, the harness reports the error, the repaired function, and the result. Overall, seems to work fine. But it's very model-specific. Most commonly the model fails on shell commands where it hallucinates some programs. If it does it often enough, I just promote those to commands in the PATH. Over time, it has happened less.

> and on-device model combination right now

That would depend entirely on what your device is. This sounds likely not to be an issue with the harness, but the capabilities of the models you've tried.

I experience almost no tool call failure using my nothing-special harness and DSv4 Flash.

You can't afford the best model. What are your specs and what models + quants have you tried?

Qwen 3.6 35B A3B and Qwen 3.6 27B can both do reliable tool calls on Pi at Q4_K_M using llama.cpp

Pi.dev requires some plugins to work well. Using Qwen3.6-27B/35B locally at Q8, I was quite frustrated with failed tool calls and tried many things.

Ultimately this combo worked:

1. https://pi.dev/packages/pi-tool-guard —- corrects key name synonyms and common structure errors, so tool calls succeed automatically (e.g if the model hallucinates old_str instead of oldText). It also wraps top level oldText/newText in an edits array if the tool didn’t do it.

2. https://pi.dev/packages/@aboutlo/pi-smart-edit - white-space-tolerant edits, as Qwen would sometimes add a fifth space to a four space indent

Hashline edit tools didn’t work well for me at all, they confused the model and it still failed to edit correctly. Also line removals would invalidate the rest of the file requiring re-reads. I tried pi-hashline-edit-pro, though I see it now keeps a database of hashes to help keep them stable across edits. Regardless Qwen kept thinking that the hashline prefixes were part of the source.