There's definitely still AI/LLM integration happening, but is kept out of specific areas of the business.
The normalisation of inexplicable failures is, in my opinion, the consequence of programmers being unable to refuse their managers who have fallen for the LLM siren call, lest they lose their jobs.
If you randomly screw up customer orders (think of DoorDash or an online shop or Airbnb). They lose trust in you and you lose your business to the competition. Going happy go lucky and being irresponsible in the business can bankrupt most* businesses.
* well, of course except the criminal empires which are bailed out by our tax money.
So that is a strange choice for repeatable, understandable operations. Might as well use Jev.
Firstly, the transaction failed and notified you about the error - that's certainly intentional.
Second, there's failures that would be invisible to you as the customer, such as "instead moving $100 from account A to account B it credited account B but didn't debit account A, without generating an error". Those are mostly the systems I'm talking about being insulated from AI development. Without more detail about the exact problem you had it's hard to tell if it's a failure in customer facing systems or backend infra.
Third, I'm assuming you were able to reach out to the bank directly and resolve the issue by talking to a person (if the issue was really outside the norm), which isn't something you can assume will be possible with a lot of customer service ops these days (or you're going to be waiting hours/days for that callback).
Fourth, you can't really know the error rate of the bank's systems, or how common a given particular error is. It may be a known issue, or it may be a completely unreported one. Assuming it's an error with something on the backend/backbone of the bank's operations, it's running code that can be inspected, reviewed, understood, and fixed - sometimes by a very expensive COBOL consultant.
Manufacturing lines have tight tolerances. Science has 95% confidence intervals (or greater). HFT has fractional pennies to steamroll up. But “business” (broadly), leadership, macro decisions 3+ steps removed from the coal face can safely operate at wider tolerances.
I cringe whenever I see “xx.xx% growth” on a report as if the value in that hundredth of a percent place is going to sway anyone’s opinion one way or the other. It’s superfluous, wasteful and I would argue, harmful.
The U.S. Marines teach the “70% solution” which says that making a decision that is 70% correct now is better than making a 100% correct decision later.
The speed of your OODA loops is critically important, and cannot be overlooked or expensed in favor of determinism, predictability etc for its own sake. (After all “no plan survives first contact”)
The main argument for LLM-driven development is much simpler: "It will get better".
The current state of LLM coding is about a year old. Imagine if we dismissed human coding efforts after a year. Rust, Python2 -> Python3 transition, Python type checking, Windows, C++, … nothing of that was done in a year and emerged in perfection in the first year. Everything takes ages to mature into a usable product. LLM coding is still in the "throw mud at the wall and see what sticks" stage, give it some more years and see how it will develop and what approaches actually work at. For the time being, LLMs are just the most useful development tool in the history of development tools, that's a pretty solid start in such a short time.
Not unlike legacy code, where the original developers are long gone and no one knows when that system breaking bug is about to wake up.
Even in the best of cases tho, fully human written and reviewed code will still fail eventually.
So I would say we are not normalizing failures (yet) but rather normalizing legacy.
Compilers actually actively try to stay deterministic in their output.
Nice way of putting it.
I know because I've spent the better part of a year having them do exactly this as a precursor to rewrites in more perfomant tech stacks. Once this research is done, it's a fairly safe and mechanical translation job.
Say that with confidence now. You'll say with shock and incredulity later, "It can be fix? Right?!?!" (Cue some meme). Reproducing bugs is often hard for well designed systems. For randomly designed systems, it's a nightmare.
Fixing software is hard and fixing bad software is harder and fixing software where you didn't even think at the time how to make it maintainable is hardest.
Legacy systems are usually systems that were well designed at the time. That's why it's worth fixing them now.
--> And none of this is saying AI designed systems are bad. But if the only criteria is "code is deterministic by default, it works and we can fix problems later" then AI seems likely a terribleness accelerator.
Code written with AI can be fixed, if well designed, they are not mutually exclusive.
Also legacy systems were absolutely not always well designed, I've picked up a fair share of systems that were absolutely not well thought out, usually due to time pressure, or any other context specific reasons.
I do believe we are creating legacy systems faster now, and it's because teams are unable to keep up with the speed of changes, not because their system is not meant to be maintainable.
My point remains tho, using AI doesn't have to mean normalizing failures.
The point is that it boils down to writing the tests correctly, regardless of who is implementing the actual code. Hand-written code without test coverage has the same problems as AI generated code.
Or have them rig the tests so that they always pass.
https://ravimohan.blogspot.com/2007/04/learning-from-sudoku-...
Perhaps designing with assumptions that the entire stack is imperfect is the route to higher quality, along the lines of chaos monkey, which randomly kills processes on production boxes… which ensures resilience is real.
Even without AI our tech stack is so deep it’s hard to imagine every feature at every layer being provably correct.