Your workflow is probably closer to what most SWEs are actually doing.
The only way you're going to let an agent go off on its own to one-shot a patch is if your quality bar is merely "the code works."
Agents today can generate solid code even for relatively complex requirements. However, they don't always make the right trade-offs.
Just because something works doesn't mean it scales. It doesn't mean it can handle unexpected user input. It doesn't mean it's easily extensible.
Today engineers really just need to define those high-level technical requirements.
At least within our company, this is quickly becoming what it means to be a software engineer.