They ask self serving questions, underspecify their requests, omit crucial context that the agent is blatantly not going to have access to, or subtly misdirect the agent. They expect the agent to figure out everything: you'll never catch them write a prompt longer than one or two sentences. They never steer the agent or look at the CoT traces.
My boss being a particularly poor case: he apparently has the habit of arguing with the agent, as if it was a person, as if there was any merit to that. Starts being a dickhead with it, shouts at it, what have you. Was flabbergasted we don't.
On the more practical side, they have zero mental model of the harness they're using (Copilot Chat in VS Code). They're surprised when the cheap-ass Auto model, which is almost always some beyond-demented version of GPT, does stupid things. They have no concept of skills, zero understanding of what an MCP server is, haven't heard of lifecycle hooks, agent memory, the various fs scopes (session, workspace, user). No concept of how to have the agent inspect its own debug logs for higher accuracy action provenance.
This also snowballs. Having to give them a stock config is one thing, but even beyond that, you won't see them experimenting. The MCP you're using doesn't support some action? They'll never interrogate whether the underlying scoped OAuth token or bearer token does support it, and they'll never ask their agent to patch the functionality in. They'll not consider the various user flows it can perform on their behalf. They'll not string them together into end-to-end automated workflows unless you explain it to them this is possible, and even after that, they'll just kind of ignore it. They'll never build tooling, extend the harnessing, etc.
Whether this has more to do with age or just disinterest-induced lackluster adoption, up for opinion.
I've noticed the same effect, but the lines it always seems to fall on are if the person fails one (or heaven forbid both) of these: 1) are you curious about how your tools work and how to get better using them? 2) can you hold the mental map of both what you are solving and how your tools work in your head, and explain how information flows.
There is also a dash of: 3) are you willing to try something, even if it has a bit of a screwup risk, just to see what happens?
Why not just actually do the thing, instead?
I did give up on cheaper models, they required constant babysitting, and in those cases yes, the benefits indeed evaporated. The expensive models have been genuinely working wonders though, and were still able to justify themselves economically plenty, at least by my own measurements.
I think there's also one underappreciated and indirect way agents help with productivity: they counteract the attention span collapse of the past years. By being addictive themselves, they keep you engaged, and being engaged means being productive. Not even asking an agent to check something out feels too rich, and once you've asked, you're already one foot into the flow.
There's also definitely been some honeymoon effect going on for me, where I dived into more work more readily, just to see if the agent can figure things out on its own.
Busy does not always equal doing something useful.
Easy to pay a lot to the model vendor though eh?
Sure. So to clarify, no, I did not just spend it on busy procrastination, and I don't think it promotes that either. Would contradict my story anyhow, this was not some trick I was trying to play on you.
> Easy to pay a lot to the model vendor though eh?
Certainly, as I'm not the one paying. Though it's exactly corporate who really wants to have it both ways (who wouldn't?), and keeps trying to get me to use the crappy useless models because they are cheaper, despite them tanking productivity rather than helping it, so go figure.
There will definitely be a time when the hype dries up, and mgmt will start playing hardball. I'm confident that the value is there, and that I'll be able to demonstrate it to them that it is more than worth it. You can choose to not believe that, up to you. Maybe it really isn't true for your line of work, after all.
That said, the review burden is rough, that I can agree with. I outright felt compelled to evaluate whether the additional review burden did not outweigh the benefits, but at least for my tasks it did not. So grumpily, I simply live with that pain.
Maybe it helps if I mention that my line of work is DevOps and Operations. I have an ongoing suspicion that this area is better suited than average for agentic work. The codebases are relatively tiny, the languages and technologies used are very well represented in training data, and there's a decent amount of side chore. I can definitely imagine agents being a lot more frustrating to work with on proper, sizeable codebases, and the numbers simply no longer adding up. I don't have much of a first hand account with that.
My closest exposure is some personal toy projects, where getting the actual vision out there ended up requiring an inordinate number of turns (this is with a frontier model). In my estimation it was still worth it, but I definitely had to give it a back of the napkin calc.
As far as my work goes, it is of course not magic, creatively worded AWS docs will still trip it up (as they initially also do me). In those cases, my expertise is still required. But the well trodden is very well trodden, and I could cut out a lot of cruft, including a lot of organizational minutia, which I very much appreciate. I was able to burn through my backlog almost completely, for example.