If Anyone Builds It, Everyone Dies https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...
I don't think this matches the guidelines of this website.
Are you saying that superintelligence is impossible?
Are you saying that the alignment problem will certainty be solved before superintelligence emerges?
Are you saying that a superintelligent being connected to the internet would be unable to gain resources such as GPU time, money, and social influence?
Are you saying that a superintelligent being would for some reason be incapable of deception and cunning?
Are you saying that a superintelligent being would necessarily regard human flourishing as a prime objective to be prized above it's own goals and ambitions?
If it's really just doomerism we should be able to point to the flaws in his argument instead of making ad hominem attacks.
Yes, end of the discussion, I don't debate metaphysics with crackheads, sociology with psychopaths or geography with flat earthers, or very bad science fiction with yudkowski. Going on about "exposing the flaws in the argument" of the crackhead just means wasting your time.
-- Lord Kelvin, 1895
AI doomerism is psychologically attractive to "people with autistic cognitive traits, including dichotomous (black-and-white) thinking, intolerance of uncertainty, and a tendency toward catastrophizing". They are pascal's mugging themselves, to ironically use one of their terms. It's fundamentally a cognitive distortion.
"What if AI doom is all fear-mongering, and we create AI less prone to make up dangerous stuff or mistake buggy goals for real ones" (which is what alignment is) "for nothing?"
Even if Yudkowsky is autistic, you're muddling the condition. Autistic people have a *practical* intolerance of uncertainty in the moment (everything unexpected from a noise to a missed turn can be a jump-scare in their day-to-day activities), but often they're absolutely fine with intellectual uncertainty, unconventional ideas, abstract ambiguity, nonconformity, etc. Indeed, one of Yudkowsky's main things is Bayesianism, i.e. being precise about uncertainty.
Yudkowsky's reported P(doom) is somewhere around 90%, which is very much in the realm of "we might eventually be able to figure this out, *but we're not even close to ready so for the love of everything slow down so we can figure this all out*"; the book title comes from a long tradition of authors noticing you need to beat readers over the head with your point for them to notice it.
Anthropic (like at least also OpenAI), appears to think they can solve the problems that Yudkowsky has found. They're a lot more optimistic than him, but they take these problems seriously.
For his work on AI, Hinton got a Nobel prize in Physics, a Turing Award, the inaugural Rumelhart Prize, a Princess of Asturias Award, a VinFuture Prize, and a Queen Elizabeth Prize for Engineering. Calling him a "patron saint" of "doomerism" is like calling Paul Krugman (Nobel laureate in Economics) a patron saint of "Trump Derangement Syndrome" on the basis of what he says in his YouTube channel: a smart person's considered opinions are worth listening to even if you have not got time for the details, because you can be sure someone else has considered the details and will absolutely be responding to even an i missing a dot.
A Pascal's mugging would be more like S-risk (S stands for suffering) than doom risk: https://en.wikipedia.org/wiki/Risk_of_astronomical_suffering
"Oh we'll at least get some side benefit" is not actually what is coming out of the endlessly circular forums talking about the apocalypse.
Different people can care about different things; it's good that each of us gets to focus on what motivates us, rather than all chasing the same thing, because when multiple teams do all chase the same thing typically only the best few of them actually make a difference.
* as it happens, there is some overlap. Knowing more about how a narrow utility function behaves outside distribution is useful for both capabilities and safety. We're not even at the stage of being able to make AI not kill random subsets of the users with bad advice, nor reliably prevent users from falling into delusions of grandeur, let alone giving AI a reliable sense of liberty and the pursuit of happiness to maintain.
The people who've made the biggest contribution to creating a better world over the last 50 years have been the Chinese; powered largely by coal and petroleum. And in one of the most ironic results in the 21st century, they're now the leaders in solar panel production on the back of the largest investment in fossil fuel energy in global history.
The comic ran into the same problem as the climate change movement in general - they proposed ideas that generally made people worse off. And if measured in terms of CO2 emissions achieved nothing except pushing wealth creation to Asia. Which, in fairness, is probably appreciated by the Asians.
BYD had release the first plug in hybrid the year before.
The Beijing Olynpics had made air pollution a hot topic in China in 2007-8.
Wind power had accelerated after their 2005 Renewable Energy Law.
Solar panel production rose around this time, taking over the market from European manufacturers when the Financial Crisis hit and they pulled back investments.
So China at that time, was doing all the things on the cartoon's presentation list, and has benefitted greatly from them.
" “We see addition, not transition,” said Yasheng Huang, a professor of global economics and management at the MIT Sloan School. “China is building alternative sources of energy as well as fossil energy sources, simultaneously. In terms of the global footprint on CO2, China is emitting twice as much as Europe and the United States. I don’t think there’s a transition going on.” "
https://news.harvard.edu/gazette/story/2026/02/yes-china-has...
He's called out in the sub-head as an "expert" but what is he an expert in? Renewables? Energy policy? No, he's an expert in saying that China is too state-led. Why would an expert in that want to downplay their success, apart from all the obvious reasons?
Though Chinese President Xi Jinping promised in 2021 to detail a reduction in coal energy use in the 2026-31 plan, it contains "no clear phase-down plan, no clear fossil fuel cap," said Qin. "The language is much more conservative than many people expected," she told DW. One reason is the continued influence of the powerful coal lobby on Chinese government policy. "
https://www.dw.com/en/china-five-year-plan-energy-transition...
> New Chinese government guidelines on fossil fuels released on April 22 support the view that the country is willing to move away from finite fossil fuels, strengthen energy independence and still achieve its climate targets, says Qin.
> "The new central guideline talks about strictly controlling fossil fuel consumption, reducing coal and controlling oil. It still leaves room for flexibility, but these are concrete policy levers," Qin said of the document, which also indicated a desire to increase clean energy consumption.
Elsewhere Climate Action Tracker on the USA:
> The Trump Administration is pursuing an executive and legislative agenda to systematically repeal targets, policies, and funding for climate change mitigation and science. The administration is actively obstructing the buildout of renewable energy, while encouraging the production and consumption of fossil fuels, completely reversing the Biden Administration’s course on climate action. This is the most aggressive, comprehensive, and consequential climate policy rollback that the Climate Action Tracker has ever analysed.
They have a worse score than China:
https://climateactiontracker.org/countries/china/
https://climateactiontracker.org/countries/usa/
All of which, even the bits quoted to claim "no transition is happening", support my original contention that all the things mentioned in the cartoon were being strongly pushed by China in 2009. They have only gained momentum since and they've profited from doing so.
Lets drill more into details:
" Against this backdrop, coal has re-emerged as a critical stabilizing force in China’s power system. This helps explain the relatively cautious policy signals embedded in China’s 15th Five-Year Plan. While the plan reiterates long-term decarbonization commitments — including reducing carbon intensity by 17 percent and raising the share of non-fossil energy to 25 percent by 2030 — it still stops short of setting explicit timelines for coal or oil consumption to peak. This reflects a deliberate effort to preserve flexibility as Chinese policymakers balance energy transition goals with near-term electric system stability.
In practice, China’s coal production has rebounded significantly. After nearly a decade of supply-side reforms that kept output around 4 billion tonnes annually, coal production rose sharply following the 2022 power shortages and has continued to increase, reaching a record 4.85 billion tonnes in 2025. "
https://thediplomat.com/2026/04/coal-is-rising-in-chinas-cle...
I would like to note "reducing carbon intensity", doesn't mean reducing total carbon emissions, it's reducting carbon emissions per unit GDP.
This is in strong contrast to EU
"Specifically, the EU has a legally binding headline emission reduction target of 90% by 2040 relative to 1990, with a domestic target of 85% and up to 5% of international carbon credits."
https://climate.ec.europa.eu/eu-action/climate-strategies-ta...
https://www.statista.com/statistics/690823/china-annual-pm25...
CO2 emissions are a different kind of "pollution". They are not visible and diffuse quickly over the whole Earth.
We can actually address problems when we want to. It's just pretty slow and requires people to actually give a shit and put in the effort to build support.
The unfortunate comparable here is that all the people who care about making sure their AI is safe, regardless of what they mean by that, are beaten to the market by the people who don't.
Maybe they should pay more attention to real problems like the sycophantic nature of current LLMs causing psychosis in people and worry less about theoretical AGI.
Huuh, a lunatic releases an ultimate fearmongering book that rides on top of fearmongering wave. He's totally trying to help everyone.
Yudkowsky is obviously read a lot by some people working in AI. That doesn't make his ideas prescient.
It wasn't a homage to 70s/80s/90s nerd culture and a hopeful glimpse of what VR tech could be.
It was a warning for people to get off their fucking phones and to work together at improving the real world, versus ignoring it and living out unrealistic fantasies inside a digital ecosystem that makes us all a bit less human.
The whole point of the book is that VR and addictive tech is a red herring. It was deliberately misunderstood by Zuck and his ilk.
Edit: i was mistaken and people clearly do take this seriously now. Oh dear
I guess we’ll find out.
Nice to hear from an optimist sometimes, but it’s hard to be one when meat compute substrate can do all those amazing things in a 4U package on 20W and you extrapolate to silicon
Luckily for planetary overlord hopefuls, you probably don't need the whole package to become overlord. Just machines that can build machines.
I will remark that I don't really understand why any of the current idiot overlord hopefuls even want the job. The entire world is _already_ functionally their slaves. The only thing jeff bezos doesn't have that I can imagine he wants is the world to not think he's an asshole. But short of complete genocide of the human race, I don't think even overlord status will make progress on that. Might even be counterproductive.
[0] https://www.wired.com/story/super-pac-backed-by-openai-and-p...
And when the AGI comes, they won't unleash it to defeat US enemies, they'll first unleash it to make more US workers redundant and boost their stock valuation.
I’d love to hear how you’re going to do the Bolshevik style 1917 to 1918 consolidation, or maybe the 1950 Chinese ROC explusion by the KMT
Where’s this revolutionary group that doesn’t exist that’s going to somehow form to depose … who? Is Travis Kalanick on that list, how about Woz? No we like Woz…so he’s clearly out, despite the fact that he has been visiting the White House for decades, and been leading the promotion of corporate tech since the 80s
Lilliputian dictators like yourself always seem to have a really great idea in their head, but absolutely no experience or competence or capability to know how to actually do a revolution. Always ready to create a list of who’s good and who’s band.
Oh and by the way I’ve been saying on this forum for over a decade tech workers need to unionize
100% of this forum responds in the narcissistic manner; “why would I ever unionize I’m so great I can always be a psychopath like the rest of the psychopaths and make my bank and leave”
The call is coming from inside the house
if you think that there’s not a line of people who are ready to fuck over all of their coworkers on behalf of a bigger bank, so they could be the intermediary between investors and a company they will absolutely jump at it in a heartbeat
* The amount of bad in a revolution is unpredictable and very large. You are fundamentally disrupting the institutions of law and order, which will embolden the worst in society and stroke a fear and self-preservation response in the population.
* Almost every revolution does not result in a high quality government taking power. The most common outcome is that the new government is worse than what it replaced, as the most violent and ambitious tend to thrive in revolutionary conditions.
AGI is nice, yet not necessary. The orbit filled with Starlink descendants and datacenters will be the it. Anybody else wanting to get there would have to get permission. SpaceX/Musk have all the components for it to happen - from Starship to AI (including the army of robots on the ground). The governmental power/sovereignty of US will be used as a stepping stone (that is the strategy described in the Palantir's Karp's book "Technological Republic") for such global techno-feudal regime establishment.
The USA, China, and Russia have all successfully tested anti-satellite weapons. If anything, any company that operates a constellation of space-based data centres would need 'permission' to keep them working.
No state has deployed a kinetic or explosive weapon from orbit to strike a ballistic missile or launch vehicle during ascent.
No operational system exists where satellites are used as strike platforms against Earth-launched rockets in real time.
Russia has done ground-to-orbit anti-satellite missiles though.
Any directed energy system shooting up would be strictly easier than one pointing down, not only because of thermal issues and power supply but also because it's easier to hide ground installations than satellites.
Should we ever get to a point where a country is considering shooting down space datacentres, considerations about the impact on Earth is unlikely to stop them.
if it is designed to breakup. And not if it isn't.
>no nuclear material involved
that is the beauty. No contamination.
>that's not going to have a huge impact.
in my comment i already specified the TNT equivalent of such an impact.
>there's a 0.7 probability
It isn't a matter of probability. You can deorbit with high precision, and pretty much hit any desired target on the ground if you have thousands of objects in space on a bunch of various orbits.
>Should we ever get to a point where a country is considering shooting down space datacentres, considerations about the impact on Earth is unlikely to stop them.
13 ton GBU-57 reaching M 2-3 gets 200 feet deep. De-orbitted 1-2 ton steel rods will have about the same effect - ie. you can hit many strategic objects of your attacker. And having in orbit, just in case, a ball or rod of 30-50 tons will get you a small tactical nuke equivalent.
"In the case of the system mentioned in the 2003 Air Force report above, a 6.1 by 0.3 metres (20 ft × 1 ft) tungsten cylinder impacting at Mach 10 (11,200 ft/s; 3,400 m/s) has kinetic energy equivalent to approximately 11.5 tons of TNT (48 GJ)."
I still believe that it is possible to do noticeably better, yet it would take some technological effort - somewhat similar to how Russian supercavitating torpedo https://en.wikipedia.org/wiki/VA-111_Shkval does or like some hypersonic weapons do by creating a plasma cloud ahead.
Even more - we can create plasma channel by using tandem schema. The first, smaller rod flies down heating and ionizing the air thus "paving" the way for the second, main, much larger rod. The delay between them is such that the main rod passes almost immediately right behind the second in the a-max (maximum deceleration) layer between 3 and 5 km altitude. If they enter 5km at 20 Mach, the first will slow down to almost 10 Mach at 3km, so the main should enter the 5km altitude 0.2 seconds behind the first. The wind speed at those altitudes is 10-20m/s, thus the channel has to be only about 2-4m wide. I actually have been thinking about this from the other side - a large gun for space launch.
The orbital kinetic strike weapons that have been proposed in the past are usually 2 ton titanium rods that would hit at about Mach 10, and even with that level of force they've been dismissed as less useful than ballistic warheads. Things falling from space just aren't as dangerous as people tend to assume.
Why are we suffering fools steering us into the worst of all possible worlds? Are we hoping for some kind of integer overflow?
Just a very rough primitive illustration - a land for a house in SV is like a $1M, and putting a 10 ton house into space at $100/kg - $1M. Existence of supposedly cheap land somewhere (with not much infrastructure usually) doesn't help as you put your computer nodes into a datacenter building with all the required infrastructure which cost more than the SV land on a sq foot basis.
And that is without consideration of how powerful a weapon is the energy generated by a humongous field of solar panels in space. Remember Reagan's Star Wars? Nuclear explosions as a source of power for the direct energy weapons like lasers, etc. Well, you wouldn't need the nukes anymore. Just redirect a bit of power from your compute nodes. And as i already wrote, the large transnational companies will have to take care about their own defense themselves https://news.ycombinator.com/item?id=47981423 - one more "feudal" aspect of the coming techno-feudalism.
Defense is one of the most important sovereign aspects, and upon acquiring it the transnationals will be able to acquire pretty fast the other sovereign aspects. Like enforcement of the Criminal Code of the Mars Colony - again pretty rough primitive illustration of course.
The feudal Europe emerged on the outskirts of the Roman Empire, and in our world the new order will be emerging faster on the outskirts (i.e. where reach and strength of the existing order is weaker), the space being one such "outskirts" dimension and the AI/hypercompute virtual world being the other.
To the commenter below with reddit link : they use human env temp for heat radiation estimate. That lowers the numbers and requires AC equipment. Ie they estimate space station, not datacenter
Better napkin math that is still being unrealistic compared to the true costs of space-based datacenters: https://www.reddit.com/r/theydidthemath/comments/1quvbi4/sel...
Just contemplate what the radiator array and solar array needed a 1GW datacenter and all the cooling equipment and coolant, and imagine the harsh environment in space degrading it constantly.
The only point of the space-based datacenter idea is to pump the Spacex IPO
This is a terrible argument, given that space has zero infrastructure.
Once you can break a data centre into a million sub-units and spread them over a sun-synchronous orbit or ten and cool them radiatively, you can also spread those sub-units on desert land with no water or electricity and cool them radiatively.
The units on the ground would look about 6x larger because ground experiences night and even deserts have clouds, but their PV also lasts 30+ years rather than burning up every 5 years or so, which means the factory making the PV to supply them is the same size.
The main thing you save on is batteries. Tesla already supplies enough batteries that it can manage a "mere" one million 25kW compute modules.
> And that is without consideration of how powerful a weapon is the energy generated by a humongous field of solar panels in space. Remember Reagan's Star Wars? Nuclear explosions as a source of power for the direct energy weapons like lasers, etc. Well, you wouldn't need the nukes anymore. Just redirect a bit of power from your compute nodes. And as i already wrote, the large transnational companies will have to take care about their own defense themselves https://news.ycombinator.com/item?id=47981423 - one more "feudal" aspect of the coming techno-feudalism.
While true, attacking up is easier than attacking down. Anything on the ground has a massive heat-sink all around it, the stuff in space does not. Right now, an attack up is already only limited by the supply of adaptive optics to get through atmospheric distortion.
no, you can't.
>attacking up is easier than attacking down.
no.
Nothing prevents SpaceX or anyone else from buying up the right to put these things on cheap desert land. They don't even need to own the land, just the right to wheel these things out on a trailer or a helicopter and leave them there.
A desert is significantly less harsh than space. If your radiator is sized for space, it's overkill in an atmosphere.
And for your edit: https://www.youtube.com/watch?v=xNmbvaUzC8Q
no. Again totally wrong.
The 20-40C air surrounding the radiator radiates at the radiator too. This is why a human immediately gets stone cold in space while not in the atmosphere - our body radiates away about 900W and receives 800W+ back from the atmosphere - our internal heat 'generation has to cover only the difference - less than 100W usually.
You probably meant forced convection cooling. That requires additional machinery. And that additional machinery is a significant part why ground based datacenters such expensive to build and operate.
To the comment below:
>The planet underneath anything in low orbit also does this, making this argument irrelevant.
no. Again, totally wrong. You've just stated that a human in LEO wouldn't get immediately cold when exposed to space. Just think about it for a second. And after that plug the numbers in thermodynamic calculator. You'll see your error.
>Likewise, the fact that convection exists even without the adjective "forced".
no. Again, wrong. Non-forced convection is pretty small. Use the calculator. And you'll understand why datacenters use forced convection.
As does the fact that humans are only slightly warmer than their surroundings. A human-sized object at the operating temperature of a GPU would have a net radiative loss in Earth's atmosphere of around 0.9-1.3 kW.
Likewise, the fact that convection exists even without the adjective "forced". Again, replace a human with an identically shaped android at maximum GPU operating temperatures of 80-100 °C, normal (non-forced) convection goes from ~117 W (human) to 0.9-1.3 kW (80 °C) to 1.2-2 kW (100 °C).
> no. Again, totally wrong. You've just stated that a human in LEO wouldn't get immediately cold when exposed to space. Just think about it for a second. And after that plug the numbers in thermodynamic calculator. You'll see your error.
I already did before previous comment. I was also considering adding "don't forget evaporative cooling for human bodily fluids" to previous comment, but it seemed an irrelevant tangent to discussing data centres.
Now, if you plug the mass of a human and the specific heat capacity of water into a thermodynamic calculator, tell me how long it would take for a human to cool one degree?
https://www.wolframalpha.com/input?i=%2870+Kg+*+%28specific+...
And that's with the 1 kW radiative losses from being in shadow far enough from Earth to not get meaningful thermal radiation from the planet itself. Even at 500 km, thermal radiation from Earth will still add 200 W/m^2. This is comparable to the thermal paint previously mentioned, whose peak emissivity (and by extension absorption) is chosen to be a different wavelength than the thermal emission of air temperature.
> >Likewise, the fact that convection exists even without the adjective "forced".
> no. Again, wrong. Non-forced convection is pretty small. Use the calculator.
I did, for both humans and GPUs, you saw the results. Humans are the wrong reference class.
In your own words, "Just think about it for a second": a human in humid 40°C air is in immediate danger because then all the sources of cooling have been blocked off. Radiation becomes balanced, I said humid to block off evaporation. Conduction and convection there have the same problem there as radiation. A GPU wouldn't have a problem with 40°C ambient, because it will still be radiating heat, conducting heat, and by conducting heat to the air specifically also convecting it away.
My point, i'll repeat, is that while 80C GPU will still radiate while surrounded by 40C air, it will be receiving back the radiation from the 40C air, whereis in space it will radiate the same while receiving practically nothing back from the environment. Both cases obviously is considered when in shadow.
To the comment below:
>False
you wasted my time as you don't seem to understand the basics of thermodynamics.
>and also irrelevant as if you let the space based ones go into shadow you wasted most of the point of going to space.
again, you wasted my time as you don't understand the datacenter construction discussed in the sibling comments.
so, you also think like ben_w that if we put something into a vacuum bottle here on surface on the Earth it will get cool down like in the vacuum of space.
I do not waste words, perhaps read them and you will find out.
> My point, i'll repeat, is that while 80C GPU will still radiate while surrounded by 40C air, it will be receiving back the radiation from the 40C air, whereis in space it will radiate the same while receiving practically nothing back from the environment. Both cases obviously is considered when in shadow.
False as demonstrated in the words you didn't see the point of, and also irrelevant as if you let the space based ones go into shadow you wasted most of the point of going to space.
Starlink numbers already in thousands (and cost much cheaper than 10M). And that is still using Falcon, not Starship. And a ground launched missile would be easily "cooked", once it exits the atmosphere, by a direct energy weapon - very easy in space.
To the commenter below: yes, exactly, this is where my thinking on that started at the cryptocurrency boom - https://news.ycombinator.com/item?id=26289423 - as you don't need close connection between mining GPUs. For AI you'd need to cluster several together while still overall scheme is the same.
>what the equilibrium temperature of a black planar surface is at a given distance from the sun.
it is 120C at the Earth orbit. So you do need to have some reflection, either back through the solar panels, or the radiators to have a reflective back toward the solar panels in the shadow of which they are to be located.
Perhaps running pumps that move around coolant passing over the cubes of GPUs? ..
That would be extra weight/cost into orbit though...
Also, don't solar panels have reduced efficiency when they're hot? And having anything hot surely increases failure rates.. with metals getting closer to melting points...?
Ideally this is a static structure with an equilibrium temperature acceptable for the silicone to operate. If the required panel area is too hot on its own then a perpendicular cooling fin on the back that falls entirely within the shadow is added.
Solar energy isn't stupendously more available in space than on earth. Even if somehow you get super robots that are able to perform the continuously required maintenance and installation of new equipment, transporting materials into space is very expensive. Venting waste heat in space is incredibly difficult. Dealing with some unexpected situation that requires manual intervention becomes impossible.
(This deity is called the stock market)
If such an AI can be reliably made to never ever come back to Earth, they were never a threat in the first place. Nobody knows how to fully test an AI's utility function yet, only randomly test inputs and hope the random distribution we chose is helpful; but every time a diffusion model's output is body horror, every time an LLM makes buggy code (and even every time it gets the pelican-on-bike wrong), this is an example of the test distribution not being good enough.