I think this lesson is getting partially outdated. Yes, you need to be specific about what you want, and with earlier LLMs, you need to had both domain knowledge and some general software development experience to front-load various big and small choices about design, architecture and operational reality - what libraries to use, how system components communicate, how you handle auth and store secrets, etc. Otherwise the LLM would pull some random mix of ideas from its latent space, and give you something that's broken in really stupid ways.
Nowadays, it doesn't feel like that to me, not anymore. I still need some understanding to verify the proposals, but I found the last ~6 months of SOTA models to make good choices. Like, just yesterday I asked Claude to design me some simple service, and focused on explaining it the domain parts (nature of systems I want to integrate together, the purpose of that, and the user's priorities and use cases), and the design I got back had specific suggestions around security, authentication, deployment, failover, integration, behavioral impedance-matching between integrated systems, and more, that I all recognized as based on solid software engineering and ops practices, but deviating from it explicitly in every place where it would be wrong for this specific project. The model considered way more corner cases than I did, and I'm actually really impressed by it.
But then, I find greenfield development is easy with LLMs. Modifying existing systems, especially legacy ones, is where I need to babysit and micromanage models - because any misunderstanding or inaccuracy, which often comes from stale documentation or naming mistakes, tends to get amplified and confuse the agents. No matter how precisely you specify your epic, if the model will find something that contradicts your knowledge/intent, there are good chances it'll get confused and make subtle errors, and you won't realize until much later.
The way I see it: models are highly biased to treat everything they read as "ground truth", all of equal importance. There's no nuanced notion that some information may be stale, that there's a temporal and causal order to sources, and that some information may just be wrong.
And this compounds when you let your LLM write code and documentation over time.
True! hence the need for someone to review the final spec output and own it as their own output. I have also found LLM to be better at debugging and solving 'a' specific problem, which I believe is due to output's surface area to be reviewed is lesser in comparison.
I definitely need AI help for the discovery part… so it always starts with a simple “I need to do X”
This is btw why Epiq was developed, to keep the board as code, git-backed, distributed (via an event log mechanism), and with the ability to replay the board, to see what agents actually did:
At which point you might as well write the code yourself and get a deterministic result faster, better and cheaper.
Where you would have a point is if you'd say we have excellent tooling for wrangling code, but less tooling and tradition to write and manage specs.
Around February you could get away with very vague prompts to Claude. I feel like models have regressed since
I honestly feel like basically nobody knows anything about these models, it's all just vibes (and I'm no different).
> I honestly feel like basically nobody knows anything about these models, it's all just vibes
This, too. Since only providers know what they actually serve, what they change and what limits they impose.
There are some visible degradations though. E.g. Claude-ish.
As for a personal anecdote: around February I created a rather complex quiz web app for myself and friends with multiple question types, sync between screens, multiple media upload types, multiple scoring and timing types, MC inetrface etc. etc. etc. It took me a week or so in the evenings with rather vague prompts to make it.
Now Claude (and Codex) cannot reliably build a much simpler web app even with precise instructions while also maintaining the visual consistency.
But I will agree with you, it's a feeling, not a precise measurement.
Add to that:
- Of course, "labs" (quotes) are incentivized to throw coders under the bus and aim for the biggest possible market.-
- "Sharp, focused, brief, elegant, precise" editing, as would benefit the coding use-case, is actually token-saving, ergo, undesirable.-
The only thing that can stop this, would be the quality and functionality of the codeslop generated by these models to became so low that it actually interferes with the (alledged) recursive self-improvement of models (ie. models start to perform worse/degrade).-
Until such a time, we serfs, will eat what's on our plate, pay for it, and continue to kneel before the machine god overlords. And be glad for it.-
I feel like need much more precise instructions much earlier in the process now than when I was building in February.