I’m not sure what the methodology of these are, but they certainly don’t match what I experience. Maybe I need to look deeper for relevant benchmarks.
This happens to me more commonly in large projects (>100k LOC) and in those projects it seems to happen every few sessions. I feel this specific benchmark would be impacted by this more than the smaller contrived benchmarks.
To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myself.
With Fable I can start with a general ask like "I'm trying to do X, can you investigate and tell me what the shape would look like" and have it poke around and think, ask me questions with single-choice or multiple-choice answers, then break the task into small chunks, each of which becomes a ticket.
With Astra, it's like pulling teeth. It often does not understand what I'm trying to do, takes things literally, does not go above and beyond (i.e. infer intent), and stops way too short of the actual goal. I have to constantly prod it and it's frankly exhausting.
Otherwise I wouldn’t see others in the exact same codebase struggle and underutilize agents while others thrive using the exact same ones.