I don't think OP is claiming that prior to LLM coding everything in the software development world was super rigorous (I assume that's effectively what you mean with the "rose-tinted glasses" comment). But rigor was actually possible and in a deterministic way too, which is fundamentally impossible with LLMs. You can build all kinds of guardrails and processes around LLMs that make it somewhat approach rigor again, but it's still fundamentally based on a bunch of statistical probabilities instead of deterministic, repeatable results.
All of the methods I see to mitigate the fundamental and inherent issues of LLMs seem roughly equivalent to the kind of crap you see in astrology groups or palm reading etc. You need Venus and Mercury to be in alignment while Mars is retrograde if you want to be able to get the right results from your token predictor.
Any software engineering practice that had enough review and feedback to work with humans should work more or less the same with AI coders.
It’s when someone tries replacing an entire team or an entire process with a single prompt that they get in trouble.
Sure, but LLMs are non-deterministic in ways that no sane human ever would be. See the "Is it better to drive or walk to the carwash" scenario from a few months ago as one of many, many examples. Or a personal example I encountered just a week ago: I asked Claude (Opus 4.8 in case any of the "you aren't using the latest model that totally fixes that issue" types are interested) to convert a bunch of DB calls that currently use raw ADO.NET calls to use Dapper instead.
The projects in this repo were on .NET 4.8.1 and were still using the older format for the .csproj file instead of the newer (and far better) "SDK-style" format that Microsoft introduced a few years ago. It tried to use the dotnet CLI to add references to Dapper, even though the older format of .csproj doesn't work with that. The dotnet CLI returned errors about trying to add the package references for Dapper, which Claude completely ignored while continuing to try and convert the ADO.NET calls to Dapper. And at the end it tried building the project, which of course failed, and then it confidently informed me that the conversion had been completed successfully and that the build completed successfully and all tests were passing successfully, even though the output from the build it had done immediately prior clearly told the LLM otherwise.
A real human, despite being non-deterministic, would have caught the issue at multiple stages. They would have seen the error when trying to add the reference. If they ignored that then they would have seen the red squiggly lines all over the (deterministic) IDE telling them there was something wrong, along with autocomplete for Dapper calls not working. And if they continued to ignore those and managed to keep going anyways, they would have clearly seen that the build failed, with tons of errors specifically about references to Dapper failing to resolve. An LLM keeps going on its merry way in ways that effectively 0 humans would.
They also don't learn, so they never get less unpredictable. You can't give the senior robot the production keys and expect it won't delete prod.