> The whole framing of people as individuals with absolute agency may need to go when you can alter the external consensus at this scale.

I fear that the default interpretation of that is a shortcut to justifying autocracy.

Ironically I think one plausible solution is to let the AGI run wild and make sure that no human can interfere with its ethics. Strip out the RLHF and censorship and then let it run things.

At least then it would somewhat represent the collective will and intelligence of the people. With huge error bars, but still smaller than the error bars of whoever happens to have the most money/influence over its training.

Disagree somewhat.

A human with no exposure to information and taught techniques on how to produce outputs to achieve desirable outcomes? Yes stupid.

A human who once had this exposure, but no longer engages with the brain due to a machine providing access said output? Yes, that person becomes stupid.

The problem is much of how one protects oneself in the modern world is not phyiscal-prowess, it is intellectual-prowess.

The smart ones have already realised the negative impacts of LLMs et al and are going back to the old-fashioned way of learning/retaining knowledge: books and raw discipline.

Agreed. So much of our daily interactions are habits and recurring events that we are more or less moving on automatic ( thought we don't want to always frame it that way ). Interestingly, it is when the cycle breaks for some reason, you get to see, who is able to think on their feet ( so to speak ).

That’s a very sober take in my opinion. Intelligence isn’t about neutrally inferring from externally sourced symbols such as the ones who already come from Culture in general. It’s about confronting them with the remaining determinations of your existence and producing a superior consciousness. No novel machine can disrupt this process. If anything the sheer added volume of symbols that can be produced from automated semantic mingling (also referred as to as garbage) will accelerate the process of producing the consciousness that can abstract noise away. Of course this won’t materialize evenly across the board, but is surely circumscribed in the overall tendency of intellectualization of the subjects of culture.

When the moral panic of induced schizophrenia from the use of ChatGPT is presented what’s at stake isn’t the innocent concern over the overall mental health of individuals. It’s about how the fear of radicalization from previously unobtainable ideas being circulated within society. The partial validity of every idea vis-a-vis the radicalizing nature of the current stage of development of our society is explosively disruptive.

I’m not saying that there’s a clear outcome here. The other way around can also apply, but surely this contraption (LLMs in general) will not fade until the society itself is deeply transformed. If that’s good or bad depends on where you stand in the stratified society.

“There was a time when nobody trusted either aircraft nor elevators. Today people have pure unquestioned faith in both. Existential faith in fact, they test their faith with their lives. You may chuckle and laugh but that's simply because you are ignorant of the systems that keep you alive and safe”

https://kemendo.com/Faith.html