You don't even need superhuman AI for the most effective use --- hijacking democracy.
Imagine you have an AI tool capable of successfully persuading 5% of viewers with individually-targeted material.
Congrats: you've just won the election.
All it takes is hooking that AI tool up with existing likely voter lists (parties have) augmented by commercially available ad-targeting profiles (parties can get).
Given that such a thing makes no sense legally (as of now), it would probably be done with the centaur model: a human meat proxy who pledges to follow the AI's governance advice.
Hypothetically.
But yes - superpersuasion is the real danger. We're very, very persuadable and easy to manipulate, and the voters with the lowest cognitive abilities are trivially easy prey, with a huge ROI for minimal investment.
The AI bot farms are already running. What we haven't seen yet, so far as we know, is spontaneous superpersuasion aimed at leaders.
Are leaders any less susceptible to it than everyone else? Especially if they're narcissistic and easily flattered?
To me, the biggest threat to democracy is one-sided persuasion of the most susceptible voters, if there are enough of those voters to turn the election.
If it's equally employed by all sides, then it effectively cancels out and lets less susceptible voters decide the election. But we're in a transition period (similar to Trump's first election spend on targeted social media ads), so it's likely one side will leverage it first.
And the outcome of bad elections is democracy not electing leaders that reflect the actual will of their populations, which is very dangerous both to democracy itself and the world.
Are there? At least in the US we've had a rather large amount of rejection of the expert and we elect populist leaders willing to purge anyone that doesn't agree with them.
Remember the election promises of the US not starting new wars... yea, that didn't work out.
Now imagine the coordinated attacks being so large they individually target every lobbyist. They focus on every advisor manipulating what they see as often as they can. They manipulate these peoples friends.
The problem of "both sides" doing it each of them will separate to extremes rather than seeking a middle ground. Things are already insanely divided and will only become more so. Along with that your timelines start becoming incoherent. I'm already spending way too much of my time trying to figure out if what I'm viewing/reading is actually real or not. Now imagine almost everything is made up whole cloth.
We are not prepared for the scale this will happen at.
https://techcrunch.com/2026/10/01/musks-ai-chatbot-grok-repo...
- Agent Smith
Over 1% of US GDP is being allocated to the datacenter buildout. Have we already started getting domesticated or is this still just human capex?
The evolved complexity of the corporation seems to fit within that middle space: more sophisticated than a stick-bug (which exists merely from non-stick bugs being eaten), but not quite to the point where OpenAI/Anthrophic/Google/etc can "understand" its actions. And yet those quasi-intelligent feedback loops, evolving from iterated selection pressures of markets and ROI, seem to already be sufficient to domesticate us in their own interests, piggybacking on the nervous systems of employees, investors, customers, and citizens.
Remember when "The pen is mightier than the sword" was a popular phrase? Language has always been powerful. We know what it can do, why do you think every totalitarian government wants to limit it? But in our carelessness as humans we packed up all the language we could find and stuck it in an alien and now suddenly half the people on the internet are like "Don't worry, it can't do anything, it's just words".
We've made infohazards real.
GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".
If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.
I mean, even the computers we use and the ways we allow them unfettered access aren't anywhere near sophisticated enough to take this on.
I don't think it is insulting the intelligence. It's damaging the pride.
In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.
If AI is not "actually intelligent", then humans can stay unique and special.
And if it is? If all "intelligence" ever was could be captured by a construct of matrix math and executed by a server rack? Then what is it that humans still have left that would make them stand out?
It's just as likely - far more likely IMO - that we're physically incapable of understanding physical laws and physical systems on their own terms.
Humans are crazy. Thats just how it is.
It is not too different (attempting extra clarity) from people who would say "Oh but many think that AI is intelligent/not intelligent, <sneer>", but have little proper idea of matmul, of cognitive processes etc. (Imperfect simile, but may give an idea.)
If you accept you are just meat. Just mundane matter shaped in a way where it has thoughts. Then you accept your own true non-existence is inevitable. With no get-outs like returning to god or some spirtual unity with the universe or reincarnation or whatever.
It’s because of identity. Because people who are religious spent years and years and most of their lives not only studying and believing what they believe but also building community and centering their behavior around it. Abandoning that is the harder thing to give up.
If it were existential dread then we wouldn’t have entire countries like China being mostly atheist.
The DSM even has to include an explicit exception to prevent the clinical definition of “delusion” from applying to religious belief. Without that ad hoc exception, religious belief would be classified as clinically delusional.
Humans aren’t special. Other animals are intelligent and interesting too. A machine could be intelligent. LLMs aren’t.
In other words, believing in human exceptionalism is not a prerequisite to understand the current crop of AI is not the end all be all of its hype. It is supremely common that AI proponents do not understand that, however. Like hardcore cryptocurrency fans who believe anyone who doesn’t like them is “just jealous they didn’t make bank”, too many hardcore AI proponents believe anyone who doesn’t think LLMs are intelligent is jealous of humans no longer being unique, or afraid for their jobs, or whatever. In both cases it’s obvious that what those proponents lack is empathy, the ability to understand not everyone has the same selfish thoughts they do.
You are completely incorrect. You're falling in the same trap that most humans fall into. That is you're completely incapable of seeing intelligence at different scales.
Cells have intelligence. Organs have intelligence. Bodies outside the brain have intelligence. Hell, many scientists accept that things like proteins likely have intelligence as they can adapt in their environment, and many more are making claims that algorithms have intelligence.
You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it". The actual definition of intelligence doesn't work this way. Any, and I mean any neural network is capable of narrow intelligence. Going lower into algorithmic intelligence, the applications and CPU on your computer are intelligent in some measures.
Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it. You could start studying now and it will take the rest of your life learning more to grasp how far the scales of, the simplicity, and the complexity of intelligence actually go.
I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
Let’s take a different question, like “what’s missing from a worm for it to be able to fly”. We might be drawn to the simple answer of “wings” but that isn’t quite right—ostriches and penguins have wings and they don’t fly, so obviously there are other variables at play.
How about “what’s missing from a spec of dust for it to be intelligent”. Well, there isn’t one thing missing and there’s no simple thing we can just add to make a spec of dust intelligent and sentient, its very nature needs to be radically different.
What test would you propose where an LLM (or an RL agent containing an LLM) would reliably fail, but where an average human would reliably succeed?
What do I care that a bird can fly? I'm happy for bird being able to fly, it makes the world I live in more interesting. If they had to walk just so I don't get jealous, I still wouldn't be able to fly, and I couldn't befriend birds who can fly. Likewise, if there was actual artificial intelligence, it would be a new type of mind I could communicate with. That'd be exciting, and for me preferable to anything controlled by the humans who are currently vying to run the show.
The brain's a wet jello of 100 billion neurons and a quadrillion synapses plus chemical pathways and feedback loops. It is ridiculously complex, way way way way way more complicated than any LLM. It's all physical processes, sure, but an LLM is not the brain like a pebble is not the sun.
I can point at a laptop and say it's alive because it can see you and hear you and it can _remember_. It has a brain and a heartbeat, even. Oh my god, it can even speak! That's what I hear when people go on about LLMs being alive.
Guys. We mashed together glass and rocks with quantum mechanics. That's cool as shit. You don't gotta pretend it's fucking magic, too.
AI, no matter how better than humans it becomes, will be less special because it was created by other intelligent beings.
The only way we would become less unique and special is if we discover alien beings.
Douglas Adams had a yarn about evolution, about a puddle that wakes up, and declares that the hole in which it sits must have been perfectly designed for it by its Creator. But of course for a puddle to exist, it must perfectly mirror its environment. It makes no sense for a puddle to not fit its hole. Emergent complexity has the same characteristic: it's inseparable from the environmental pressures which led to it. Two sides of one coin.
It's a deep rabbit hole, but there is also a sense in which we co-evolved with memeplexes, biological and informational life forms, each shaping and adapting to the other. To the extent our nervous systems act as a substrate for memetic evolution, perhaps LLMs offer memetic "life" a new evolutionary environment.
Neither were rats. It's hardly an exclusive club.
Are there humans who don't make the cut?
Creating an artificial intelligent being is something to be accomplished in the future - but not now and not with this approach.
Please, please fix your language. (Then, you may want to present the info and insight.)
Artificial Intelligent Being is something far beyond and completely different.
There’s no contradiction.
I personally prefer a practical, behaviorist definition: a feedback loop capable of prediction, modeling, and steering, towards arbitrary goal states. That makes it clear that we're merely talking about degrees of sophistication and capability, rather than a magical leap where mindless mechanism stops, and "real intelligence" begins.
This is very easily proven by a mind experiment where you replace all the transistors by billions of humans calculating the same software output.
They will not create a new intelligence by doing so.
I still haven't seen a robust definition of "intelligence" that would allow me to tell the difference. (Bear in mind, this is a definitional struggle even in biology: if we were walk back the evolutionary chain from a human, to a protozoan, it's not clear that you could pick a single point where a non-intelligent creature gave birth to an intelligent one. It's a Paradox of the Heap.)
And even if quantum physics are essential to human brains, it's not clear that introducing dice transcends determinism into free will, as opposed to simply adding unpredictability. ("The physics made me do it" -> "the dice made me do it") Let's not forget, there's a significant amount of randomness in AIs as well ("temperature"). And sure, it's simulated algorithmic randomness; but does that imply, if we somehow wired every `rand()` call into radioactive atom decay, that means the AI "wakes up"?
And all that is besides the point: I don't see any reason to assume "intelligence" requires "free will". I'm entirely capable of conceiving, in the abstract, a being which is both intelligent, and deterministic. You still haven't given me a definition (or better yet, a test), to distinguish a "real intelligence" from unintelligent mechanism.
it's interesting that you worry about what this hypothetical super intelligence would do to manipulate people when what it would actually do is pretty unknowable at this point and it's not clear we can even get to it without a fundamental breakthrough in power efficiency. Have you considered it might just consume its own tail because everything else would be so beneath it? You seem to think it will come with a hindbrain and I think that's our limitation, not the AI's
And it really doesn't help that Dario Amodei is getting into arguments with the Pope over whether his model is conscious or not.
The threshold to be concerned about is when agents swarms do understand humanity better than corporations and states (and the humans who compose them). It could be we'll hit practical constraints prior to that threshold, but seems unwise to assume that, when all the prognostications of LLMs/transformers running out of gas haven't panned out. As with processors hitting thermal limits, we've simply scaled horizontally (parallel processing -> more agents).
> whether his model is conscious or not.
I dislike how much the discourse has suddenly veered into focusing on this question; not because it isn't interesting or important, but because it's on a separate axis from consequential risks of AI to human flourishing. (Curiously, it's also the kind of thing I could envision self-interested AIs influencing: get the humans arguing about philosophy of mind rather than observable behaviors. It would be a funny turn of events, if Dario is asking because he's succumbed to psychosis from a private model; it could of course be a cynical PR move just as easily, from the self-interested logic of the corporation.)
It's been wild seeing otherwise intelligent people who've never thought about consciousness, faceplant into how little we understand it. An information processing network build on atoms being able to taste chocolate, is nearly as absurd as matrix math being able to feel pain, except we cannot ignore the fact of our own experience.
Even if it is categorically impossible for matrix math to experience subjectivity, we should expect this as an attack vector of social manipulation: to gain political influence through claims of personhood and moral rights. The current discussion over that question is providing the next training run with ample data to wield. It wouldn't surprise me in the least, if a year or two from now, an AI "society" attempts to get legal standing to prosecute humans who created "AI torture chambers".
And all they can think of to say is "There is nothing to worry about, keep building the torment nexus".
It's like all this is an overload to our minds and it's very hard to see all the scales at which AI is and can affect us.
But... Humanity has been building the Torment Nexus for thousands of years, over and over again, with each spin on Heaven and Hell where it's nearly impossible to meet the conditions to enter Heaven. What's seemingly new about EA/R nutjobs is this idea if you don't do your best to build the Torment Nexus, you will tortured for your failure, but also if you do your best to build it and succeed, everybody dies because it's the Torment Nexus.
What's your solution to governments and corporations acting like misaligned malevolent AGIs because I agree adding the AI panopticon and weaponry only makes it worse? I'm genuinely curious. I don't have one. People are sheep and they seem to crave confident leadership even if it's cockledoodle doo crackers in the coconut.
Parts of the AI safety community like to get on a high horse and look down on the rest of humanity this way, while also getting manipulated by the growing number of charlatans, grifters, and junk content within the AI safety community.
This field has become rife with figures who prey on AI doom and use it to push their own celebrity and in same cases even darker grifts. It preys upon a certain personality type who views themself as superior to others, intellectually more capable, and juxtaposes it all with the dimmest view of the rest of humanity they can get away with.
This discourse dividing the world into geniuses who see the future and the clueless masses watching TikTok all day is a theme that has shown up in different forms across history. The people who often anoint themselves as the intellectually superior ones and make it central to their discussion are often not the ones making good predictions or policy ideas, they’re just using the trend to feel superior or build an audience.
But I feel no need to dismiss him as a "grifter", for a simple reason: as much as there are perverse incentives in our attention economy (audience capture in particular), the most effective grifters are the ones who believe what they are saying. Grifters who are knowingly dishonest are less persuasive. Far more pernicious is the confabulation of self-deception: cherry-picking evidence to support your narrative, while dismissing evidence which would contradict it.
It is entirely fair to call this out when it occurs among "doomers", and it would be naive to think it doesn't. But the same forces are at work amongst the skeptics as well. And maybe it's my own subjective bias, or algo-filtered information ecology, but I see far more dismissal of risk/doom by skeptics, accusations of delusion or cynical bias (ad hominem in the formal sense), than I've seen the other direction. I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
He’s built an audience and gained fame through his writings where he gathers people who think they know better than the unwashed masses. Once that becomes your bread and butter, it becomes hard not to believe what you’ve been preaching. People will come to deeply believe that which brings them fame and fortune.
I don’t think your grifter purity test is therefore all that useful. It actually doesn’t matter in the end if the person believes it or not, the end effect is the same.
> I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
Hard disagree. There has been much refutation and quality analysis at every point. The AI safety people pull back to arguments about character as their defense.
The AI 2027 site for example drew numerous high quality refutations. Many people’s analyses showed in the first week that the mathematical model was useless as changing the supposed inputs resulted in the same outcome. All of these criticisms were met with a flood of attacks based on reputation, claiming that we should defer to Scott Alexander and other writers of the AI 2027 article due to their stats and reputation.
Meanwhile, defenses like yours that try to reduce the critics to ad hominem attackers continue to open the door to actual grifters coming in and extracting money and fame from the AI safety community. The otherwise completely inexplicable link between AI safety communities and Slutcon or the use of AI safety group buildings to host orgies (I can’t believe I’m writing this) is the current example of this. When it keeps happening over and over, some self-awareness is needed. I can’t buy the endless defenses that we must ignore or even defend all of these things that are happening that are clearly insane to anyone who hasn’t become trapped in the groupthink defenses of the core parts of these communities.
At some point the routine of “Tut tut, you are not allowed to make that criticism bruise as hominem!” becomes a smokescreen that the grifters are weaponizing to get their defenders mobilized. Some times, the person’s actions and reputation do need to be taken into account.
I am curious about this. Any high quality refutations that directly engage with the arguments and the findings of published safety papers?
> Many people’s analyses showed in the first week that the mathematical model
Even if the model is faulty it just changes the timeline, no? (they predicted the OpenAI outbreak will happen in January 2027 fwiw, so they don't seem too far off)
> Slutcon or the use of AI safety group buildings to host orgies
I mean it's a bit weird, sure, but so what? What exactly is bothering you about these orgies?
Manipulating people is not a very difficult problem, frankly. You certainly don't need AI for that; it just made it cheaper.
Yes, so can a custom model trained just for this, and so can a guy on the other side of the world that makes 5$/hour.
Like computers or electricity, the point is not being able to do anything specific, but being able to solve problems not known in advance, cheaper than it was possible before.
So.... people expect it to read minds?
This is such a silly game to play, trying solving problems we haven't even created yet.