I read through the skill file of graphify and ran some of the commands it was telling the agent to run. The vibe I got is there was a lot of structured information returned that the model (and harness) isn't set up to work with effectively.

It seems like a really interesting tool and maybe future models will know how to use these tools more effectively, but yeah I remain skeptical.

The more I think about it, I suspect that either the training data or the finetuning process (or both) will have to incorporate these tools to really get benefit from them.

I was demo'ing Gemini 4 Argon last night and watching it use codemode to basically generate structured docs about the codebase for itself

so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive

This has been a general trend in my experience. The more structure you give it the worse it performs. And it also creates more work for you.