> I have to kick the hell out of its plans in ways that I don't think someone without a ton of real world experience could do

I think that’s the real point the article’s making. Unless you already know the ropes, you can’t tell if a plan is actually any good. I use LLMs all the time for work, but only because I get the system well enough to spot the nonsense or catch when a plan duplicates something that already exists. They really do keep rewriting the same functions over and over in the code.