People’s job is to partially reign in this volatile environment by creating processes with stable output.
That is to say I do agree that building reliable processes out of unreliable parts with feedback is the modus operandi. However AI cannot meaningfully handle feedback and learn. And that is a key unsolved problem.
If the Model makes repeated mistakes on the same subject matter, you can update your agent.md file, or you can add skills to deal with specific prompts, or you provide a better default harness.
The whole idea of coding agents is their harness makes a big difference vs a pure raw model.
> However AI cannot meaningfully handle feedback and learn
How do you think models are created? They are trained on feedback and learn.
Its not cheap but you can post train models. This is how custom models are mode, that deal with specific tasks more efficiently and accurately.
Example ... Composer? Its base Kimi v2.5 model that has been post-trained 2 weeks, to create Composer 2.5, what is a much better coding model.
Its literally trained to make less mistakes by feeding it correct data. Hell, a lot of the models you are using, are often the same base model, where v2.0 was the initial released model but the model keeps training, so when they release v2.1, its still the same model, but with more training time on feedback provided to v2.0.
LLM Models are not a cake you cook one time and they are done, and you start from zero again. If you have the money, and a powerful server setup, you can take a model like GLM 5.2 and post-train it, to reduce specific errors. Sure, you need a ton of money because its a large model.
But people have been doing this with 5M, 100M, 1B, 5B models for a long time already. To the point that some of the small models can do specific tasks, almost or better then some of the huge more general trained models.
That's all just prompting.
> How do you think models are created? They are trained on feedback and learn.
No one is post training models on a single mistake. At least I have not seen it. I also doubt it is effective. Post-training on a single failure will not meaningfully change the model. That even sidesteps the entire problem that you don't even have access to models if you use a provider like anthropic/openai
And telling someone not to repeat a mistake is… ?
I feel like this line of thinking is kind of an unfair comparison. I'm not saying LLMs are magical beings that can suddenly learn by themselves after getting something wrong, but your "person making mistake then being corrected" assumes you do tell the person about the mistake and tell them to avoid doing the same mistake in the future, but for the "LLM making mistake" example you then intentionally avoid letting the prompt being changed in response to the mistake, which would be the "then being corrected" part on the LLM side of the comparison.
Similarly, if you just let a person make a mistake and don't let them know about the mistake, they might keep making that same mistake over and over again.
If you update how you use the LLM as you discover what mistakes it does, just like you'd correct a person, then you can use an LLM and also the LLM can "be taught to not make that mistake again".
> How many p are in strawperry?
> There are 0 “p”s in strawperry.
And I can trigger the same mistake with various words even when adjusting the prompt many times. So I cannot teach chatgpt to correctly count characters.
please take a look at the error(s) made in the prior run. what could've been done better? create or modify an existing skill to emphasize this, or suggest additional language in AGENTS.md.
We’re not at the point where AI is capable of knowing what went wrong and self-aware enough to understand how it could reliably change its own behavior.
For months I’ve been trying to have the agents stop manually writing our auto-generated SQL migrations and run the command that generates them instead. SOTA models insist on occasionally getting it wrong.
Well this is the central bet of AI coding isn't it? We, the humans-in-the-loop, get better at knowing ahead of time which patterns AI will handle better than others, all the while the models actually get better.
The whole "soul.md" stuff today is a poor approximation to that. But I wonder whether it will grow into it, like chain of thought prompting grew into reasoning models.
This is what I was referring to:
https://www.cnbc.com/2025/08/19/sam-altman-on-gpt-6-people-w...
It's marketing speak, but the goal is clearly there, no idea how achievable.
These models are great tools, but putting them and people on the same level does a disservice to our species and also is simply incorrect to what we know these models to be and their capabilities/limitations.
[0] https://www.theguardian.com/technology/2026/jun/01/meta-ai-h...
At the same time, one should acknowledge that not all tasks are on the same level.