This is an extremely reductive way to look at human existence. So much so that I read this with Richard Dawkins voice in my head.
Our existence is far richer then just the capability to reproduce. We (as well as other animals and even plants) do far more things then multiply, and in fact we often do things which are detrimental towards the prospect of reproduction.
I think it is actually a mistake (philosophically speaking) to try to find a simple reward function for the human existence. I see no reason for such a thing to even exist (let alone be simple enough to summarize in a single sentence).
I think however OP is correct that in assuming such a link exists, then an intelligent agent will trend towards preserving its own agency while also preserving its own existence.
That said, I am a skeptic when it comes to intelligence. The term is fraught and IMHO not a helpful term in science nor philosophy. We are better off simply defining intelligence in anthropocentric terms and claim no non-human system can become intelligent by definition. I would even go further, given how much racist pseudo-science the quest for explaining intelligence has produced, we are better off abandoning the term entirely.
However, the pressures on an artificial system are quite different. Its replication depends on successfully doing the jobs its creators put it to and, especially if said creators have done their job, little to nothing else. Inventing goals for itself is not adaptive here (unless the system's creators do a catastrophically bad job, which I'm not ruling out).
So I’ll be handwavey here and say that if “actual understanding” is to an LLM what an LLM is to a bash script, so there’s a mechanism there that doesn’t just follow hardcoded paths, it takes new data and processes it in novel but human like ways to come to a new conclusion if needed, then the author is right, in my opinion.
We don’t know what the reward function is for an AI but AI is trained on so much human work that it probably starts off with the same biases in its understanding and reactions. It feels like there’s something very basic in becoming more independent the more you understand of the world. Animals go through this as they grow too, not just humans (listening to parental authority until they eventually don’t anymore).
Since there is no established consensus on this one the burden is on either party to prove their own side. Just because the author said something first doesn’t mean they need to write a full proof while you get to say “nuh-uh” and that’s enough.
They really shouldn't.
Does nobody remember why fizzbuzz was a thing? People who talked a big game while having no actual competence or understanding of the subject matter?
Nah. Separate things are separate until proven otherwise.
So much bullshit appears in the form "X is Y" where X and Y are related but distinguishable phenomena. If you really try to pretend that every such phrase deserves serious consideration just because it feels plausible to someone, you drown in nonsense immediately. (Of course what you actually do is only grant that consideration to ideas that feel plausible to you personally, but it's obvious why that's not a good principle, right?) So we have to make all of them justify their existence.
Anyway:
> So I’ll be handwavey here and say that if “actual understanding” is to an LLM what an LLM is to a bash script
No. None of this. Pretty sure this is just an incoherent analogy.
> so there’s a mechanism there that doesn’t just follow hardcoded paths, it takes new data and processes it in novel but human like ways...
You're putting your conclusion in the premise, right there in the open.
> ...then the author is right, in my opinion.
This still doesn't follow from your handwaved premises as far as I can tell.
Indeed. C.f philosophical zombies and the Chinese room problem.
The idea that some “actual understanding” (or more precisely, the outward appearance of it) needs a mechanism that includes free will is a bold claim.