The value of money, and the ability for average people to have influence on future events, ultimately is tied to the fact that the labor of an average person has inherent value.
Once that is no longer true, all bets are off: All promises of compensation become meaningless, and it's easy for those in power to rationalize to themselves why they no longer are beholden to their earlier promises. They will say to themselves "The situation ended up more nuanced than we expected"
(Which is exactly what Google said in February 2025 when they reneged on their pledge not to use their AI for military applications)
We also have the power of the vote, at least for now. I understand that that is limited, but it isn't nothing.
There is also just the power of having a lot of people. We don't yet have the technology where one person with a lot of machines can dominate a society. It still takes people.
By who? Lula just got reelected and a blue tsunami is coming on the heels of the Mamdani Mandate.
Donald Trump would agree that the government should have a large amount of power.
Hmm. Let's give the clanker a try.
They don't want us to have any power left a few years from now, and they're winning.
Your feelings towards those individuals doesn’t matter in the grand scheme of things, what matters is that everyone’s opinions are shaped/influenced by the few with money.
Although if you have killer robots, then all bets are truly off, and we go the way of the dodo, literally.
(I don’t actually believe in this LessWrong-esque rhetoric, reality is more complicated, but it’s fun to muse about.)
EDIT: If oligarchs with magic AGI really don’t need our labor, nor are they concerned by our violence, then perhaps we become more like ants rather than dodos: we organize to help each other, creating a somewhat traditional economy around the scraps that don’t matter to the oligarchs, relying on obscurity and harmlessness to avoid getting toyed with a magnifying glass. I suppose that’s slightly better.
Even like 1-2% of the adult population going in the streets with makeshift weapons is enough to take over any country on this planet.
Data centers are extremely easy to disable too, take down power, data or both
Although actually: that’s kind of what humans are… Hmm.
I suppose humans can’t make better and better humans, that’s the difference.
Although actually: that’s what technological progress is… Hmm.
We still live in a finite, constrained, physical world that has to obey the laws of physics. We have a looong time before we get fully autonomous self replicating decentralised robot armies.
I think people conflate the real world, sci fi and their wet dreams a bit too much.
We're at the "this thing is kinda useful sometimes but still mostly annoying" part, not at the "skynet in 48hrs" part
We do have them actually. It's called: an engineered virus.
Without some really fundamental social and economic reform, I'm anticipating a squeeze as automation increases, that ends with something like robot armies finally reducing the working class to barbarism, by blocking access to all the valuable resources.
And it will be some oligarch(s) commanding it, not some rogue autonomous AI.
So not "skynet in 48hrs," but capitalism taken to its logical extreme.
I'm not so sure. The front line in Ukraine seems to be pretty good at stopping people with access to military technology. I don't think an untrained and uncoordinated mass of civilians could do a better job, especially against future tech that's more sophisticated and available in larger quantities.
As military tech gets more and more inhuman, the power of masses of bodies is likely to decrease, and there will be even fewer opportunities for moral qualms to give civilians a chance.
https://en.wikipedia.org/wiki/2013_Metcalf_sniper_attack
Some dude in France just carried an attack with a ww2 sten gun, there are hundreds if not thousands of AK47s, grenades, etc. available on the black market. You can buy drone ready fiber optics cans on aliexpress, thermal/night vision cameras, etc.
Military tech is good but you still need people to follow orders, I very much doubt they'd side with the sociopath tech execs, nor their governments.
Data centers might be easier but
a) Will have a lot of security. It's much easier for somebody with money and connections (and now even intelligence+knowledge) to assembly a defense force than it's for a ordinary people to learn how to make proper molotovs, let alone learn how to shoot. And they will know when it's coming because even today's LLMs are good enough to surveil the entire internet.
b) I don't think they burn that well. When Ukraine strikes oil depots or warehouses, you can easily find images of them burning. I haven't looked particularly hard but images of droned data centers burning are quite a bit rarer. (I haven't seen a single one before actively searching, they do burn, but that's with military grade explosives.)
"Hundreds if not thousands"? That may be enough for "rando attention-grabbing but ineffectual terrorist attack," but we're talking about violent resistance to a fully-automated tech bro.
> Military tech is good but you still need people to follow orders, I very much doubt they'd side with the sociopath tech execs, nor their governments.
I already addressed that, and that may not be true in the future:
>> As military tech gets more and more inhuman, the power of masses of bodies is likely to decrease, and there will be even fewer opportunities for moral qualms to give civilians a chance.
Say you have some area you want to exclude a desperate mass of civilians from. Technically you could do that today: just stick a guy in a pill box with a machine gun, and slaughter them like it's World War I. But said guy might have a problem with that, and refuse to pull the trigger (or stop after gunning down a few hundred).
An automated AI machine gun in the same pill box will have no such qualms, and will mercilessly gun down civilians to the last bullet. And that gets more and more true the more and more you automate combat. Maybe a technician wouldn't setup the machine gun in that place or reload it after what happened, but that goes away if setup and resupply is automated, etc.
This is pure sci-fi...
this conversation is making my day!
We just call them "virus".
I feel like I still don't understand your logical endgame.
On the other hand, that way we'll finally get to "reconnect" with nature...forcefully.
In the second world war, Germany had impressive science and technology, but, as Stalin said, quantity is a quality in itself, and The Third Reich found out that it was an impossible task to get the metals and fuels for their advanced jet engines, rockets and U2 boats, when between then and those resources there were masses of tens of millions of under-equipped and malnourished soviet soldiers fighting what was an existencial war for them.
Bombs. You mean bombs. Throwing killer robots at a "problem" is like throwing knives or spears: you just gave the enemy your weapon. Unless they explode. But then you just wasted a lot of money on a really smart grenade.
we already do..
You might lose your value, others might gain.
All real world power, including ownership (both physical and intellectual), is based on the ability to use violence.
As long as the majority - normal working people - have the potential to do more violence than those who want to control us, we have negotiating power. Minimum wage, 8 hour day, etc - those were all negotiated with the alternative being that those who wanted to pay us less and work more got killed.
I know it seems laughable, but we’re in an era right now of extreme competition where you cannot afford to just be nice to people.
But eventually there comes a point where companies have enough power that they don’t need to keep people in line by force: just give people their bread and circus, watch your little AI videos and talk to your AI friends and enjoy your standard housing and provisions. Do some work if you want, no one cares if you do or don’t, it’s largely performative. This basically kills all motivation for resistance or lashing out. Why go out and risk your life fighting for vague reasons when you can be perfectly comfy at home instead and have all your needs met?
If you can control the masses without their complaints, you can solidify power for a long time.
So perhaps they would only respect money and property up to the point that it's necessary.
The more realist answer is that they will keep respecting money and property as capital works to reinforce the status quo and enrich shareholders. You think just because you can do your job faster--or completely--with a machine that you have to work less? Hardly! Now this is where interesting policy questions spring up but I digress.
https://archive.is/20251015175242/https://www.bloomberg.com/...
As long as they're still looking for prompting customers, they're clearly not there yet. There is some solace in that.
People can not be trusted to preserve the agency of others. See history.
I think this is the major hole in the argument, and in the "AI maximalism" position you present. There is no such thing as an omnipotent genie, or a magic eight ball that can answer any question. Any technology has limitations, and if you assume as a premise that the technology does not have meaningful limitations (or that the limitations are irrelevant, which is really what you said), the conclusions you draw will also make no sense. This is how you get nonsensical ideas like the singularity.
I remember the discussions and predictions when e-books or internet newspapers were first introduced. In 2026 they still sell actual newspapers where they print yesterday's news on large sheets of paper. People still pay for that instead of checking X for free. We all buy and read actual books. TV hasn't killed cinema and YouTube hasn't killed TV. Real world doesn't work in extremes.
Real world also doesn't have 'infinity'. You can't just use undergraduate engineering math to create a feedback loop and explode a variable to infinity. Things don't work like that.
What I'm getting at is that even if it isn't literally omnipotent, it's indistinguishable from the impact it's likely to have on society.
because they don't control the physical paper that is money.
I dont understand what you mean by "fiat tokens". I don't imagine most people will accept tokens as a currency because they are as good hold of value and predictable as bitcoin.
In a token economy, certainly what you say is a possibility, but as you can imagine, that economy will be unsustainable so back to "regular" money.
But if I have a near magical genie who can make me anything - software, farming business, physical things - and at a fraction of economic input compared to the legacy monetary policy - then the money is worthless. As the Aztec shells were to the conquistadors.
That they can’t print USD is irrelevant.
> Money is a medium of exchange
That's an oversimplification. It comes with a lot of rules and authority that will be enforced by the issuing authority that a simple "IOU" on a sheet of paper lacks. IN the continental US, you cannot refuse to accept a dollar note. You certainly can refuse to accept the very IOU we agreed on a minute earlier.
Most of the developed world already has magical genies - certainly an overwhelming majority of people on HN earn enough to have LLM subs that build code, 3D Printers that print physical things.
The challenge always has been and will be humans.
You just cannot live in a vacuum. If you could subsist entirely on cheetos and figured out how to make an entire lifetime's supply of cheetos, after N days of watching Netflix, arguing on HN, raising chickens and goats, one day you will want to go and talk to a human.
Soon after, you will need a currency.
This is a false–albeit common–misunderstanding. "Federal statutes do not require a seller to accept cash as a form of legal tender for payment of goods or services that were rendered. Thus, businesses may establish their own policies regarding whether they will accept cash as legal tender" [1].
Yeah you can. Tell that to all the stores in rougher neighborhoods that kept getting held up at gunpoint for the cash in the register, so they stopped taking cash. Bit harder to get money out of a stolen credit card terminal.
And if your "near magical genie" can make an operate military equipment, you don't have to respect existing authorities anymore.
Basically, the "AI maximalism" scenario kicks the legs out from under all these concepts and social institutions you're relying on.
> Most of the developed world already has magical genies - certainly an overwhelming majority of people on HN earn enough to have LLM subs that build code, 3D Printers that print physical things.
Not like in the scenario we're discussing. Whatever "magical genies" we already have are incomplete and need operators or have serious quality problems. Most of us depend on those limitations for our livelihoods, and those limitations also give levers to control them.
> You just cannot live in a vacuum. If you could subsist entirely on cheetos and figured out how to make an entire lifetime's supply of cheetos, after N days of watching Netflix, arguing on HN, raising chickens and goats, one day you will want to go and talk to a human.
An oligarch with the "near magical genie" could easily build a town of 5,000 to satisfy his needs to "talk to a human."
Minor side note: this is true only for repayment of debts. Some local jurisdictions require private parties to accept cash for new purchases, but there's no federal requirement to do so.
I'm also interested what this "entity in charge" does? and why would it do anything at all?
Absolutely untrue. Outside of some local laws governing cash at retail you're under no general obligation to trade with someone if you only want payment in cybertrucks and all they've got is dollars.
He just means it as in “credits”. “Fiat tokens” as in arbitrary units that are said to have value de facto by consensus, but don’t have any inherent value.
Yeah not the best choice of words in this context. And it’s quite a LessWrong-esque leap of reasoning.
But I do get the point: either Anthropic is successful and they transcend money, and thus (somehow) screw over their investors, or they are not successful. Their investors get screwed regardless.
Again, I don’t really agree with it, but it’s an intriguing point.
They won't be able to control saud AGI. Ref Ultron
So why would you allow that kind of power to be concentrated in the hands of a few companies? If the state can use this technology, which keeps improving on its own, to manage an entire nation's resources better than any human or group of humans, why would we still need private companies to do it? Especially when the ones with the most resources/capital would be able to just win with everyone else and accumulate all the existing resources.
But the bigger issue lies elsewhere. When you replace human labor, you also replace soldiers with robots. Until now, a leader with autocratic tendencies couldn't easily order the military to turn against domestic "terrorists" who are actually their political opponents. But once armies are replaced by robots, that could all change, and we could see democracy go into decline.
Doesn't that factor into the future revenue? Feels like people imagine suddenly fully owning the means of production in a vibrant economy but how does that really make sense?
If Microsoft Windows and Office can have a valuation in the trillions then Claude eventually working its way to the same ballpark isn’t a crazy assumption.
Yeah I agree with that. However a "valuation in the trillions" ain't very different from their IPO price.
Seems like lots of risk for proportionally very little return.
A $2T valuation may not be that crazy but thinking it'll make returns like Amazon/Google/Meta/Nvidia isn't reasonable.
" It is a well-documented fact that Bill Gates was initially reluctant to take Microsoft public. His primary concern was that the pressures of quarterly earnings reports and the scrutiny of public shareholders would distract the company from its core mission of writing great software. However, a specific regulatory requirement known as the 500-Shareholder Trigger eventually forced the company's hand.
Under Section 12(g) of the Securities Exchange Act of 1934, any private company with more than 500 shareholders and a certain level of assets was required to register with the SEC and begin filing public financial reports. By 1986, Microsoft had issued stock options to so many employees that it was rapidly approaching this limit. Gates realized that if the company had to disclose its financials anyway, it might as well reap the benefits of being public, such as increased liquidity for employees and a stronger currency for acquisitions. "
https://learn.oreateai.com/learn/when-did-microsoft-go-publi...
You have to wonder how many people in AI leadership positions are already making decisions based on this expected outcome.
"AI" doesn't pump/treat sewage or drinking water, grow crops, mine metals, change wound dressings, deliver babies, stock grocery stores, drill teeth, walk dogs...
It's not the replicator from Star Trek!
AI maximalism is a Silicon Valley religion. You don't need anything close to it for some of these valuations to work. (Anthropic would be worth ~$500bn at Google's IPO multiple on revenue [1]. Anthropic is currently growing 6x faster.)
To be clear, I think these valuations are rich. But the notion that AI maximalism is required to sustain them is just false. $1.5tn sits comfortably on about $60bn of earnings, which with Anthropic's gross margins requires on the order of hundred of billions of dollars of revenue. That's super ambitious. But not Anthropic is the economy ambitious.
Really the only scenario where Anthropic could screw over its investors in the way you're describing is if it could also take over the legal system and modify its own contractual obligations. Which, I mean, I guess anything is possible if we are in AI maximalist-land. But "AI is the law now" would be a much bigger story than "investors frustrated with low dividends."
If the economy is planned based on datasets, which are inherently backward looking then our best option is our ability to surprise. But that’s for the artists and philosophers to figure out.
There will be an oversupply, demand will be there too, but no one can afford to buy since most jobs have been replaced by AI. So the price of goods and services should go down, or some form of UBI will have to be implemented which is just socialism for the masses. Its either that or total anarchy (also socialism).
If humans remain in control of it, then how can it possibly lead to anything other than extreme oligarchy?
Depends on the distribution of such AI. For example, if everyone on Earth has their own, personal superhuman AI, it would not lead to extreme oligarchy. If only a handful of humans control superhuman AI, it would.
1. the goal of ai is to end capitalism by disrupting scarcity in economics
2. if ai can build a closed system that prints ai factories, the 99% of humanity essentially are just genetic diversity live stock.
I don't think the ultra-wealthy care about genetic diversity.
not an original thought, ive seen Her
suppose this is just another way to say, "wtf are we doing?"
The closer you get to the real or plausible "omnipotent genie", why would the person in possession of such a thing respect the rights of the formal owner - and why the engineers and operations managers respect the right of the management - this along with the question why the maybe-conscious super-intelligent machine itself would respect the rest of the hierarchy?
For now, these structures actually need massive resource inputs and that doesn't look to change soon but naturally one should be on look-out for the possibility of the tide shifting.
Lots of other fun risks (highly recommend checking that section) but not that one :(
We don't have democracy because somebody thought it was fair or just for people to rule themselves. We have democracy because in the past, people got angry, picked up rifles and shot enough authoritarians (and their submissives!) for the rest of them to understand the group with more people has more firepower and tends to win.
So now we periodically check which group has more people with ballots instead of bullets.*
With autonomous drones and robots, democracy stops being the local optimum.
And once we slide (are pushed off) into a new more stable state (pun not intended), we won't have the (fire)power to get out of it.
* actual AI, which does exactly what it should correctly, not a statistical model of behavioral patterns
* but indirect democracy is expressing a point in n-dimensional space (your opinion) by selecting one of a predefined set of points (parties/politicians available) - which is stupid - and the cracks are already starting to show as e.g. nobody wants more surveillance, yet we keep getting more of it in both EU and US
---
The "permanent disempowerment" people keep talking about won't be a "rogue" model, it'll be doing exactly what told by the rich and connected.
EA-aligned AI labs share this exact ethos. In their view, fiat money and equity markets are merely temporary scaffolding used to raise capital until AGI is achieved. Once a central superintelligence controls manufacturing, software, and logistics, resource distribution shifts from market supply-and-demand to algorithmic optimization.
[0] https://digital.library.cornell.edu/catalog/ss:34227574
[1] https://en.wikipedia.org/wiki/Joshua_N._Haldeman?hl=en-GB
AI oracles are the central economic planners. But after years of perfection they have made a couple unusual "mistakes" and a human is trying to discover why.