If we continue from your "water comes from the tap" example this situation feels like someone who has no experience of plumbing deciding they're going to replace their bathroom without bothering to find out how any of it works. They'll possibly end up with a bathroom that superficially works but they're also liable to find problems down the line with leaks, things not draining properly, or water pressure issues.
Some of the people writing software purely via LLMs are even turning this into a business, which I guess would be analogous to deciding you're going to run sewage treatment for a small city without understanding anything about how that works. It'll mainly hurt when the problems are invisible, so food poisoning, or inlets backing up and overflowing into a nearby river.
I agree it's pretty bloody cheeky to run an agency like that though. The only ways out I can see are licensing or real liability.
But let he who has never done dodgy home improvements cast the first stone.
In the bathroom remodel analogy, the AI is the plumber that you hire to do the work for you. You tell it what you want, it figures out the details necessary to get it to you. If what you want isn't possible, or requires large changes, a good plumber will raise the issue to you and explain the constraints they are working under, ask for guidance, and offer recommendations.
Sometimes the AI does things in a weird or inefficient way, or neglects details that become important later, or misunderstands something important about the task resulting in an outcome other than what you wanted, but that's also the same as the plumber. This is why people usually want to be at home and keep an eye on the people working on their improvement projects or repairs, because they want to make sure that the guy they hired is replacing the broken faucet, not the functional but odd-looking one.
There is still value in knowing the underlying systems well enough to present a good design vision to the plumber up-front. It's still possible to have a competent executor produce a bad idea, which is how we get stuff like Groverhaus, the Doom Bathroom, and McMansion Hell, which are examples of "slop architecture" in exactly the same way as we are seeing "slop software".
People looking for self-contained, spot solutions to localized problems will have much more capacity to solve them locally, while people with in-depth expertise will be able to build systems that are even more reliable, performant, and scalable than what they were building before.
So we'll get the benefits of disintermediation for the basics, and much greater capacity to deal with effectively with complexity for the not-so-basics.
One of the things I've built in my current role is tooling to help understand current architecture, and to resolve the differences between things as-documented, as-implemented, and as-actually-behaving. It's used for a few things, including updating documentation, but it's been most impactful when onboarding new engineers. They can simply ask questions about how things work and why, and get explanations and deep dives wherever they need it.
Is the article claiming that?
> Not knowing all the details seems more like the default human condition, in the sense that most people don't know exactly where their water comes from or where their waste goes exactly. "Water comes from the tap, food comes from the store".
Sure. The ability to specialize, and on the flip-side not have to care about a lot of things, remains one of the biggest driving forces of human civilization. But in this article, I don't think the person worried about cognitive surrender is the "most people" in relation to food or water. They're the farmer or the water infrastructure engineer! They absolutely for sure do need to know those things. And all around us we see a lure of systems that invite them to surrender.
> Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.
Maybe not. But the might be cognitively surrendering when they use the LLM to summarize the day's news or write a message to a friend, or even plan dinner for the week. I'd say they are. I agree that it's not clear-cut.
> They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision.
No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.
> This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.
Yes. But they wouldn't usually outsource summarizing a letter from grandma, or the political agenda of a candidate/party in an upcoming election. That's surrender.
> It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, lib updates, server updates etc).
… or ones own abilities deteriorate.
> Tooling and processes that make that stuff more visible would close a lot of the gaps.
Not the problem of one's own skills though.
Fair. Maybe I "cognitively surrendered" a long time ago just as a consequence of working in large teams :(
I hit the breaking point of not being able to read all the code well before LLMs came along. Also people would leave and the documentation for their subsystems would be wrong or incomplete, so there were always mysteries to chase down (during an incident, naturally) even in code written by hand.
Gets to the point, you don't exactly read the MRs, you're mainly checking their approach against what you talked about earlier and making sure it doesn't look way more gnarly than it's supposed to.
Working with agents doesn't feel that different.
> No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.
It does bug me sometimes when people say "yeah we built the deck..." and they got a contractor to do it. But then we do say things like "Nolan made the film" even though he didn't hold a camera or build the sets. To me, one-shotting doesn't feel like building, but engaging in a loop and applying control to get the outcome you want, does. Regardless of the exact toolset.
I don't personally see super hard lines between using a compiler, a 4GL, a sql query, stitching together 10 libraries and a framework I didn't write, using an LLM or even directing a team.
> Not the problem of one's own skills though.
I read somewhere once we can't weld Saturn V engines anymore, or at least not economically. We would also probably have a lot of trouble finding people who could do all the engine calcs the way they did. It doesn't necessarily mean that rocket progress is stalled.
> Yes. But they wouldn't usually outsource summarizing a letter from grandma, or the political agenda of a candidate/party in an upcoming election. That's surrender.
Agree 100% on the letter from Grandma.
I do say that, too. But I'm not in the film industry. My understanding is that those who are, don't use words as crude as "made" to describe a role in the making of a film. I believe they say that he wrote, directed or produced a film.
We are not talking about some other field of expertise here, where we are outsiders – the production of food, the supply of water, or the making of movies – we're talking about our own field of expertise.
Oh, no, film insiders use ‘made’ constantly, or even cruder - attribute credit without using a verb at all. Saying it’s a “Nolan film” is super common. Here’s a VFX person saying “Nolan’s CGI”: https://youtu.be/AFLcP-t0-kI (I’m ex-film industry, and still film industry tangential, and found this clip being discussed on a film industry mailing list.)
Film people are far more likely to pay attention to who the director is than non-film people, and it’s very common for film people to talk in terms of directors ‘making’ the film or otherwise casually assign ownership. I think this is especially true for famous directors and good films, and maybe a bit less true for unknown directors and mediocre films.
They didn't read code before, they just wrote it? That makes no sense.