This is Bainbridge's 1982 paper Ironies of Automation restated.
https://en.wikipedia.org/wiki/Ironies_of_Automation
https://ckrybus.com/static/papers/Bainbridge_1983_Automatica...
> This paper suggests that the increased interest in human factors among engineers reflects the irony that the more advanced a control system is, so the more crucial may be the contribution of the human operator
Makes sense.
Which makes those interfaces the most critical parts of the system.
Model produced code compiles the same as human produced code. My kids would call this “copium”.
Then there is these other programmers who built their own operating systems even though their OS professors are claiming publicly their design sucks, who built a local database from scratch in C because his use case did not need the client/server stuff but just all the other relational goodies, or who built video streaming/transcoding software from scratch just because they have ideas and then the world changes around their ambitions.
I think it is not enough to be ambitious or have the technical ability if you want to build something for others to use. You definitely need deep commitment but also perseverance in the face of social adversity. I think this is what saddens people with automation of the technical aspects of programming. It leaves the more psychological aspects as challenges which are not easy to face all the time.
Great point. Coding used to be a natural forcing function for prioritization of features. When coding is basically “free” you have to do the harder job of saying no to things, deciding exactly what to build, etc…
I would say, yes!
Yes, if its less and good. Yes.
> perseverance in the face of social adversity
Sometimes you're Fabrice Bellard, sometimes you're Terry Davis. That is, for almost any idea you'll find random detractors, but some ideas really are generally useful and some are completely nuts. One of the negative externalities of AI is that bystanders are confronted with a lot more cases where they have to work out which one this thing is.
I can certainly say every skill I have is because I sat there for hours grinding low level tasks to get something new. Will people still have that motivation?? We might be entering the tech dark ages
That is, genuinely, crazy talk. That is what addicts would say.
But it is intellectual-washed to sound like the most reasonable thing to say.
It is not. And there is nothing thought-provoking about the mere act of doing heroin.
There might be something in the system and mechanics that lead to people getting the idea to do that, though.
___
We can engage with these ideas and there might be some merit to them, but not with this frame. This frame will (not hyperbole) kill us all.
People will die due to (poorly-engineered) systems understood by no one, failing in completely incoherent and unexpected ways - and those people may be you or your loved ones.
I, well, agree. Of course the statement is nuts, especially so once you consider issues like vendor lock-in, the oncoming atrophy of one’s own skills, etc. But if you define “productivity” as short- to medium-term results only, is it really that inaccurate?
The problem is that it still uses the and benefits of the legitimacy of the word of "productivity" there. Meaning that it is harmful to us to put it that way.
Maybe it’d be interesting to see what could be achieved responsibly by motivated people with models in a world without “feature pressure”/competitive stress (which’d include e.g. no security race and infinite attackers and exploits to keep up with)?
I’ll probably have to think about it some more, but a) I think I’m inclined to take my statement above back, in that I/we should definitely still aim for understanding, even if it’s not the most “productive” thing to do, and b) I don’t know what’s a “realist” way forward, i.e. not completely denying LLMs are capable tools and never using them, but still understanding what comes out when you do use them. Seems rather difficult to balance the way the current iteration of tools is designed to be used.
People demonstrably said the same thing about the inventions of... Writing! and Mechanical/Digital Calculators! Yet here we are!
Time is a flat circle!
I suppose it depends on the “applicable areas of work” but I would argue the opposite. As in all things, when you level the playing field, in order to stand out you have to know the rules to understand how to break them.
Imagine a fully autonomous Ad agency that creates unique ad campaigns for companies. And have it controlled by someone who “let go of the most understanding.” For instance, maybe a business minded person with excellent technical skills, but doesn’t know how to copy-write and has no visual design training.
Even with the strides in AI advancement, this is an impossibility and I’d argue it won’t happen within the next five years even. You need human agency, and you need human workers who are experts in their field.
While this iteration of technology certainly _feels_ different, it seems like for now we're still coming down to "it's useful to know how to build a car, but when you need to put food on the table, know how to fix it and make it do what you need it to do".
not caring about the output ~ more output
… which may or may not be useful in the long term, but definitely does help a shorter-term “productivity” measure.
On the other hand though, there’s a pretty damning example against my case here: Say a manager of individual contributors – which is what the situation of an individual with sufficiently smart models and tooling may almost boil down to – didn’t directly care about the output anymore. We’d most likely call that being unhinged/unproductive, too…
It was late at night and I primarily wanted to link the somewhat relevant article, maybe I should have left everyone to reach their own conclusions instead of adding some poorly-worded unfinished thoughts. :x
I cannot see it being a good thing to "let go" of understanding. If you don't understanding anything, then why would anyone keep you around? An example hypothetical situation is, if I have a colleague that outputs a bunch of productive things, but they don't understand it, and I do, then of what use are they? I have LLMs too and also have the understanding.
It isn't clear to me why people are so comfortable with letting go of understanding. And it's a bit bizarre because it always seems to be framed as "you're going to get left behind if you care about understanding". I also never hear or read anyone who has any real and deep experience say these things.
To your point about becoming redundant without understanding, not understanding things will be a problem when an issue arises and the other person isnt able to handle it the right way. However, if llm can address the issue, does the fact that they did not understand the details really matter? From the perspective of a person who gets to decide who gets fired, would they fire someone who is productive but sometimes introduces bugs that they quickly address or would they fire someone who is comparatively slow but rarely introduces bugs? The answer probably depends on the domain, and how costly a bug can be. For most of us, the work is not so critical that you cannot afford bugs. So, productivity wins.
If you are suggesting humans before LLMs rarely introduced bugs, you probably haven’t got the latest models to review your old code. It’s shocking how many bugs we had before but couldn’t see!
You're smart, and you know what can go wrong, so you use models slowly, and you check their work. Your bet is that it will make a bad mistake.
Your coworker is reckless, they don't know or care what can go wrong, so they use models quickly. Their bet is that the model won't make a bad mistake.
If the model doesn't make a bad mistake, your coworker is creating more value to your employer, and your employer doesn't really care if you understand the code. They just want something that works.
Over the last year, it sure seems like models are making less mistakes, so the expectations on the bets are changing too.
For instance: we have calculators. We’ve had them for decades in their “current” form more or less. They are readily accessible and far more reliable than LLM’s - if you a problem 1000 times it’ll be right 1000 times, the rare bug or glitch notwithstanding. But we still teach people basic arithmetic. We teach times tables. We didn’t just offload basic math.
I agree we don’t need to know assembly to use a computer but there is a balance here and companies do not care about a balance.
For software engineering I don't know the right balance.
I've used the "we mostly stopped caring about machine code generated by a compiler; maybe we should stop caring about LLM-generated code as well?" analogy myself.
But even I have to admit there's two big holes in that analogy: if I use a compiler to turn C into machine code... (a) compiler bugs are orders of magnitude rarer than LLM-generated bugs (b) my C code remains the source of truth, or at least source of intent. Whereas with LLM-generated vibecode, there's no source of truth, just an ever-evolving mudball. I (kinda?) like the ideas of "documentation as source of truth" and/or "tests as source of truth" ideas, but I don't think we're nearly at the point of viability.
TL;DR yeah I think actual understanding of what LLMs produce is gonna be important for the forseeable future
Like humans, LLMs get into a place where fixing one thing breaks three others. They compensate partially with insanely big test suites, but now fixing one thing breaks three others, and 57 different specs, half of which also need to be updated.
I think roughly three outcomes are possible.
1. LLMs get fast/good/cheap enough that this just doesn't matter. Your 50M LOC monolith is a mess? Yeah, it still takes more effort for the LLM -- but this just means it costs 5 additional seconds and $0.03 instead of $0.02 to ship a new feature.
2. Stepping away from the "code as source of truth" paradigm. What if specs/docs were the vetted source of truth, and we just regenerate the whole 50M LOC monolith every time somebody makes a change, just like we rebuild binaries from C today whenever there's a change? (This probably requires some of the LLM advances from #1. Also, at this point, perhaps we should just skip the intermediate language and generate ASM -- what purpose is human readable source code serving at that point?)
3. Status quo as I see it -- human engineers overseeing projects, both guiding AI and being accelerated by it.
I think #2 is probably the... most ideal future? Though, of course, if we switch to that paradigm, the source-of-truth docs will have to be very thorough and structured, and will probably resemble a programming language of sorts themselves, except describing intents instead of operations.
You see cashiers summing groceries? Salary calculations on paper? Tax, inventory, accounting, you name it? Nobody does any of it by hand.
We most definitely offloaded basic math.
I imagine you and most of your peers can calculate a tip or at least approximate it without pulling out their phones yes?
How many of your friends and extended family can write hello-world or fizzbuzz in any programming language at all? Should they be allowed to vibe code apps for themselves? If they get good at vibe coding, should an employer be allowed to pay them for it?
To resolve this we should aim for a new intermediate format that is more high level than source code, the generated code is too monotonic to properly review manually. Something like requirements -> detailed specs for llm -> code -> assembly . Then let one agent implement the specs and have another review it. If we stay at the current abstraction level, humans become the major bottleneck.
It really does feel like people have embraced rolling around in the cognitive version of those Wall-E chairs.
Understanding architecture, robustness, the “crumple zones” of a system? I’m all for it.
Taking three days to “understand” how postgres transactions work under the hood for your generic SaaS? Uh..
I have seen a lot(!) of the last type of “understanding” and it’s infuriating. The same types will often not understand the business, its stakeholders, the architectural constraints beyond the technical, etc. Understanding takes many forms.
You do have to keep a sense of when it did the wrong thing and how that was a valid interpretation of what you asked and how to better specify what you want. But that's how it really just shows you the gaps in your own understanding of a problem.
Also I think there's huge differences to the kinds of tasks and workloads people are throwing at it, leading to wildly different perspectives, but everyone's talking about the same "Fable".
This right here. Humans are really prone to over indexing on a handful of personal experiences vs a representative sample. And once someone has formed an opinion, confirmation bias helps cement it, despite evidence to the contrary. Reality is probably somewhere in the middle.
They don’t keep you around, of course. The better these models become, the worse the human job markets get, I’d predict.
I too love to downvote and edit instead of admitting that I'm talking and acting in a way where nobody would want to be my coworker anyways. Where exactly are you demonstrating that you're doing anything to justify being kept around if that's such a big concern for you?
Is it just me or is this analogy completely missing the point of why we care about detecting counterfeits in the first place? Poor craftsmanship of all things is not really the reason counterfeiting is illegal!
I mean, the answer is obvious: the Luddites were right, these tools (not just LLMs but private property, copyright, and capitalist mechanisms) are only tolerated to exist so long as they serve the needs of Capital, but the hypocrisy still frustrates me.
Great thought. Question is, can those who do the tolerating (in the interest of capital) even effectively prohibit/prevent access after a certain point, especially now that the cat is not just underway, but well out of the bag?
Assuming the current trajectory of a yearly factor of 10–100 in size/cost reduction [0] holds up, won’t the technology just become cheap enough to be functionally available to everyone?
Thinking about just the domain of the stock market only: getting a "win" on it means getting a leg up on everyone else participating. Best example I can think of of a pure zero sum intelligence contest. Programming UIs might have a point where you just can't get smarter at it, but in an intelligence contest like a stock market, I don't think there's a ceiling.
In "Accelerando" this was covered by the invention of "Economics 2.0." AI agents also got into a lot of lawfare in that book.
The most rigorous "private property advocates" don't recognize intellectual property as a legitimate form of property in the first place: private property is a natural phenomenon that emerges as a solution to the problem of physical things being inherently exclusive in possession and use, and IP is a creation of positive law meant to artificially apply the methods for solving that problem to another, unrelated question that isn't part of the same problem domain.
> I mean, the answer is obvious: the Luddites were right, these tools (not just LLMs but private property, copyright, and capitalist mechanisms) are only tolerated to exist so long as they serve the needs of Capital, but the hypocrisy still frustrates me.
Abstract concepts don't have "needs", and no one is currently in a position to tolerate or not tolerate anything -- no one in particular is actually in control over anyone else's activities. The cat's out of the bag, and no one has any capacity to put it back in.