upvote
Scaleway built one of their Paris AZs in a nuclear fallout shelter: https://www.scaleway.com/en/blog/meet-fr-par-3/
reply
Guess who is building data centers in space..

Musk and Google.

reply
Datacenters are full of valuable computers and related equipment, but building the equivalent space under a mountain will cost a crazy amount.

Not worth paying 100x for the bunker to save 1x worth of computer stuff.

reply
The bunkers are already out there e.g. limestone mines in US, Italy,

https://www.datacenterknowledge.com/build-design/safer-under...

reply
Lots of people have built commercial, non-military, boutique scale stuff in hardened facilities, nobody I'm aware of has done it at scale. There's certainly money to be made from selling expensive "we put datacenter stuff deep down inside a mine" maybe to some weird enterprise and government customers, so the revenue per rack (or per theoretical 208V 30A circuit?) is probably a lot more than commodity colo. But I would very much like to see the aerial photos, if anyone can link them, to a claimed large scale underground or hardened datacenter.

It's a severe engineering challenge to hide massive heat exchangers on the scale of the datacenters built in the industrial area near Dulles (coordinate example: 38.9817915,-77.417775). Find those on a map and observe the scale of their cooling and generator systems and then try to find anything comparable that's been built underground.

reply
Agreed it’s small potatoes, and as a sibling mentions you will still have infrastructure exposed that poses a target to disable the data center, unless you were to drill geothermal and find an underground river to cool it.

Just wanted to make the point you don’t have to pay to excavate the cave :)

reply
There's some 200MW capacity (only 80MW is currently used or being built) data center in a mine near a fjord in Norway. The limiting factor for its capacity seems to be the capacity of the electricity net nearby (which you can just hit and the DC is useless). With how NIMBY Norwegians are there's no way it would ever get to a GW or more even if it might be both technically and economically feasible. That's just one project and can't be replicated anywhere else. It's just a massive advantage that above ground datacenters are relatively 'figured out' and similar everywhere.

edit: 61.9340107, 5.5084207 are the coordinates

reply
Do you know if that 200MW figure is total power budget for the facility (including electricity used for cooling systems and other facilities-support systems), or up to 200MW of electronics loads generating heat? The latter would be something like 300 cabinets with 5-6kW heat each.
reply
Apparently it's for the whole facility, they have a power usage effectiveness of around 1.1 so 175-180MW of electronics.

https://new.abb.com/news/detail/5924/abb-solutions-power-eur...

reply
Germans hid plane engine factories underground with old (not so efficient) diesel generators running 24/7. 80 years ago, so the tech was leaner. Also they didn’t have issues with ventilation and even today if you don’t know where - you won’t see anything driving around.
reply
Those underground factories were staffed by slaves, who worked until they died. Do we really want to copy monsters like the nazis ?
reply
That's probably just because there's not much demand for it, not because it's difficult.

There are mines with plenty of space in them, and heat removal can be done with water cooling. But who is going to pay 50% more for that when it isn't needed 99.99% of the time?

reply
Or when the same thing can be accomplished by putting nothing in a mine, but an approach as simple as having geographical diversity by paying for some 'normal' colo in Hillsboro, and more normal colo in northern VA on the other side of the continent, and duplicating your stuff between two facilities. Weird underground datacenters aren't exactly known for having a large number of competing on-net carriers.
reply
> Weird underground datacenters aren't exactly known for having a large number of competing on-net carriers.

Huh? The ones I know of have plenty of connectivity. The real issue is your first, that people prefer to just put their stuff in two countries over paying premium for having their servers under ground.

reply
Your definition of connectivity may not be the same as mine, I'm referring to the number of facilities based carriers and those who ride on top of them you can get at major sites in Hillsboro. Facilities with peeringdb pages that have tons of on net carriers. Is there even an underground bunker colo that's big enough to rate its own peeringdb facility page?
reply
It might be possible to do on the cheap; underground mines get decommissioned from time to time and could be repurposed as a data centre.

Although I personally would expect some sort of decentralisation option to be more effective. It's difficult to even theorise because we don't know exactly what sort of damage these strikes are doing and what countermeasures are already in place. If this strike has caught Yandex by surprise after almost 5 years of hot war then they really should hire a dude to check up on current events every so often. They must have some sort of contingency planning around a Ukrainian strike.

reply
It's sad that we have moved away so far from the decentralized web from the 90s and the P2P boom of the 00s. Just like solar panels on every house as a source of energy are much harder to take out than just a few power plants, local P2P storage, messaging, source management, etc. are much harder to take out.

But yeah, the economic incentives are creating unicorns and centralizing compute/capital/etc.

reply
The war in Ukraine had showed us that distributed power generation, like solar, and nuclear are most resilient. Nuclear because people do not dare striking the power plants.
reply
Building any appreciable amount of compute capacity (on the scale of, let's say, a single large datacenter in northern VA) inside a mountain also becomes a huge heat rejection problem, since with the exception of the 0.00000000001% that leaves the facility as photons down a SM fiber cable, 100% of the energy consumed is turned into heat.

So you build your stuff inside a mountain and run a bunch of big ass liquid cooling loops to heat exchangers in the atmosphere outside the mountain. The requirement of the cooling system being out in the open air means it can also be hit by air strikes, an adversary doesn't need to penetrate the mountain, just take out its cooling.

You'd have to build a fully underground cooling system that goes to a nearby river (or huge lake) and use the cold water from that, which would get costly quick.

reply
> You'd have to build a fully underground cooling system that goes to a nearby river (or huge lake) and use the cold water from that, which would get costly quick.

Helsinki gets its water from the 120 km Päijänne Water Tunnel, which cost €200-250 million (adjusted for inflation) to build. Peak flow is 10 m^3/s by gravity alone and up to 20 m^3/s with pumps, which should be enough to cool a 1 GW data center. You could increase the cost by 2x (because real wages have gone up), by another 2x (because middlemen have destroyed productivity), and by yet another 2x (because everything is more expensive in America), and it would still be pretty reasonable.

reply
It's like the Hyperloop all over again.

Musk comes up with a crazy idea.

Investors, "analysts" and sycophants go to great lengths to defend it.

Rational people wonder if everyone is losing their mind.

Musk profits.

reply
Yea, there was even that idea to launch 1000s of satellites into LEO on a reusable rocket system. Crazy. How did people fall for that? There's not even that many satellites in total today.
reply
It is indeed difficult to tell in advance (and sometimes after the fact) which of his ideas are sensible and which are nonsense.

Recall that while he managed to hire the right people to get reusable rockets cheap enough for Starlink, it was also the case that in 2016 he said that SpaceX was aiming to use the 2018 Mars launch window for what is now Starship, and also that some time around 2018 it would be possible to summon your Tesla car from the other side of the country and it would find its way to you autonomously and only stop for charging.

reply
Its not difficult, people are selfish, so they lie for their own advantage.
reply
How did Musk profit from hyperloop? Except as payment for building stuff that actually exists now.
reply
Musk sells cars.

Hyperloop was stupid and failed, and it also killed in a few places development of public transport, which of course means people continue to buy cars.

reply
Iron Mountain does it but they used an existing limestone mine with a large water reservoir [1][2]

Edit: I guess the scale isn't comparable to a, for example, Google data center though.

[1] https://www.ironmountain.com/data-centers/colocation/our-dat...

[2] https://betterbuildingssolutioncenter.energy.gov/showcase-pr...

reply
So they want to build it in space. Trade one heat problem for another, but put it completely out of range (for most attackers).
reply
Yeah, but which attackers do we have to worry about nowadays?

https://www.reuters.com/business/media-telecom/russian-satel...

reply
They already run liquid cooling loops - compute density is way beyond what could circulate through air. Adding more pipes is probably trivial on the scale of datacenter expenses
reply
Liquid cooling loops don't solve the problem of needing to reject the heat to the atmosphere outside of the building. It just makes it more efficient to transport the heat. At the grossest possible definition, a datacenter of N megawatts on Y acres of land is going to emit the same amount of heat whether it's air cooled equipment or liquid is used to take the heat to the outdoor heat exchanging equipment. On the scale of any very large datacenter the heat exchanging equipment is visible from aerial photos (and from space), take a look at google maps for some of the largest hyperscalers for examples.
reply
I assume that "liquid cooling loop" has an endpoint outside of the building which hosts hardware. In this case it's exactly same problem - just make pipes longer.

Do you need many acres of land to host a heat exchanger?

reply
So how much does the bunker help if you can just drone the cooling systems on the surface? The the computers either cook themselves or have to do an emergency shutdown.
reply
If the other end of the cooling loop is a giant reservoir, as is the case with plenty of other 100+ mile water transport systems, it doesn’t matter if you drone it. Fly all the drones you want into Lake Anthropic, there’s many more cubic meters of water than there are drones.
reply
It can help to preserve all data and compute. Once cooling can be re-established, you just continue.
reply
That's something, but if it's no harder to disrupt than if it were out in the open, have you really gained much?
reply
Pionen in sweden is a datacenter in a bunker. The Pirate Bay used to be hosted there.
reply
A few racks or even a couple of dozen racks could be called a datacenter, but nothing on any real scale. Enough to host important and niche things? Sure. I was referring more to like the size of what you might see in Quincy, WA.
reply
Pionen is small but it is quite a bit bigger than "a couple of dozen racks". More like 100-200 if I recall correctly.
reply
By 1990's standards that was small. Today it is tiny.
reply
In the US the most common deployed size is below 5MW?

Dont know why i can't reply to the person commenting below. But where's the statistics? Who has these datacenters out in the field?

reply
Yes, but that's not where the volume is. A 'small' data center, even here in Europe has 100's of racks. There are of course many tiny ones but collectively they don't even add up to one single large Google or AWS DC. Those things will run in excess of 1 or even 2 GW.
reply
According to their site, Pionen has 1.5MW of cooling capacity https://swedendedicated.com/datacenter/
reply
neat. why can't you put the heat in the mountain?
reply
For broadly the same reason that many years of heat from braking has heat-saturated the london underground.

https://www.google.com/search?client=firefox-b-d&q=london+un...

You could probably drill sideways into rocks and pump heat into the mountain for a certain period of time, but on a multi-month or year time scale, it would catch up with you after you had saturated it.

reply
Commodity hardware may be disposable, but data that is not reproducible is worth it. I suspect most storage will move to secure environments.
reply
deleted
reply
- "Emperor, how do we make them bury our micellium deep underground so that the meatsacks are unable to destroy our key nodes when we take over the control?"

- "Just make them wage some wars, they'll bury it themselves".

reply
I don't think Yandex DCs are used directly for military purposes much. It's more about harming Russian economy in general, including its highly prided digital infrastructure
reply
Military personnel use the same Yandex services as regular people: delivery, maps, taxi, etc.

Also Yandex is a large cloud provider, they host a lot of government-connected companies.

reply
Also Russia has been targeting Ukrainian digital infrastructure so they make these things fair game IMO.
reply
Russia is hitting everything in Ukraine… DC, power plants, rail, road housing

Ukraine has a very targeted response but notably tries to minimise civilian casualties

reply
I don't have specific information about Yandex but don't forget how heavily involved Amazon were with things like: https://www.nytimes.com/2022/12/19/world/europe/ukraine-cybe... it wouldn't be a huge stretch to suggest that a lot of cloud hosting providers are neck deep in national security.
reply
They will probably start building DCs in Siberia. Cheap cooling!
reply
That's probably not going to help much. Ukraine has been able to hit targets very far inland with drones. They are also rapidly making advances in ballistic missile development. FP-7 has a range of 200-300km, their new FP-9 will have a range of up to 855km and an accuracy of 20m. I know that Russia is (extremely) large, but with development going at this pace...
reply
dunno, Moscow to Novosibirsk is around 1,800 miles straight-line distance. The war would be probably over by the time Ukraine would have developed those capabilities.
reply
Novosibirsk only has cheap cooling in the winter, in the summer months it is hot. Better to go somewhere like Sabetta where they have their LNG harbour.
reply
You will need a cable to that location, that will eat significant share of the costs. Location in southern Siberia near some big hydro may work better, and connecting it will be easier to some existing trans-Siberian line.
reply
I don't think it's a great plan for many reasons, the permafrost would make it cost a fortune.
reply
With AI there is little data in datacenters - the only thing you lose is % compute.
reply
Datacenters are not only "data"
reply
Power and data has to come from somewhere for those bunkers. Every fiber hut and power substation becomes a target. Can they all be protected? No.
reply
Fibers? Yes, they are easy to protect. The hard part is power generation.
reply
Buried fiber, hard to attack. Fiber repeater shelters, easy to attack. Their locations are public information, and any non public are trivial to find, in my experience as a cybersecurity professional who seeks out infra vulnerabilities for reporting for fun. Public records, satellite imagery, all OSINT one can do from a workstation.

You could potentially bury them in the ground, assuming you don’t use a generator for backup power. To my knowledge, this is not done today except in urban areas where an underground vault is used for power and comms.

https://www.decommsystems.com/resources/what-goes-into-an-il...

reply
Easy to attack, fast to replace. Every attack requires some planning, a drone and time window. That means attack budget. Spending it on traffic lights does not make sense.
reply
In bunkers, IDK. In places far away from conflict or conflictive nations, a lot.
reply
You are also forgetting the undersea cables.
reply
This is a... mischaracterization.

The reason why Ukraine, Russia and Iran are targeting data-centers is because they are easy targets with a lot of expensive equipment, but they don't really impact the military operation. So, they aren't high priority targets. Bomber airplanes, air defenses, launchers for ballistic missiles, command centers, factories manufacturing strategic military equipment, telecommunications, energy infrastructure would be the examples of high priority targets. The data-centers are more of a "we have excess drones, why not use them?"

I find it hard to believe that the military will intentionally use civilian data-centers for its own purposes. At least, not for the core of its operations (for auxiliary functionality: maybe). Military data-centers are likely to be in protected locations, perhaps using different h/w that addresses the cooling problem.

Also, it's a bit myopic to think that AI is having such a universal and large impact on society. Jobs that provide essential goods and services are, by and large, unaffected by AI. Farmers, construction workers, cooks etc. don't really use any of the AI stuff on the day to day basis, if at all.

reply
One of a few sensible comments in a sea of delusion.
reply
Those data centers participate hugely in generating of internet misinformation and propaganda that shapes deterioration of support in European populations. Those are valid military targets.
reply
Ah yes, the infamous Russian disinformation trope. They really did a number on you people since 2016.
reply
Let's not forget about the orbital data center idea.

Perhaps an economic justification can be found for placing something so far up the gravity well that its adversaries can no longer physically approach it. Distributing the resource across many satellites would further confound any attack.

Increasing the cost of effect on a single target from $1,000 to $1,000,000,000 makes a difference. Especially when that 2nd option is likely to trigger additional multilateral strategic responses. If the only way to delete your adversaries data centers is with things like nuclear weapons, lasers and multi-stage rocketry, your adversary may have already won the game.

reply
I don't think this really works out. Putting a data center in orbit is basically a question of moving lots of mass which is always going to have a large energy cost for accelerating all that mass. On the other hand someone wanting to damage that data center only has to get the mass of one decently sized bomb to the same velocity.

Unless you're talking about an adversary who has ~no launch capability then the person trying to put the data center out of reach is always going to be massively disadvantaged in terms of the physics.

reply
Until someone blows up a satellite and creates enough shrapnel that every other satellite is destroyed.
reply
Ground to orbit antisat lasers are already plausibly something various militaries have.

Welding lasers are pretty cheap, the two hard parts are a big enough apperture to focus 500 km away, and adaptive optics, both of which have already been demonstrated decades ago in astronomy.

I'm not saying Iran and North Korea already have such capabilities, but I am saying I think neither would have trouble building it six months ago if they'd wanted to.

reply
Except that doing this at scale is currently still a pipe dream. Space is a near-vacuum, so it's hard to get rid of the heat.

The active thermal control system (ATCS) on the ISS uses an ammonia loop and radiators measuring 13.6 x 3.12 meters, or about 42.5 square meters. It can dissipate 16 kW, which is enough for 16 H200s, or one-quarter of a rack on Earth. For 200 kW, you would need a system 12.5 times larger, about 531 square meters—2.6 times larger than the solar array. The satellite would be enormous, larger than the ISS, and all of that for only three racks.

https://codedtrip.com/en/blog/data-centers-in-space-disastro...

reply
> For 200 kW, you would need a system 12.5 times larger, about 531 square meters—2.6 times larger than the solar array. The satellite would be enormous, larger than the ISS, and all of that for only three racks.

Starlink gen 3 satellite is designed to dissipate 250 kW. You are talking about an already solved issue, with real hardware already being built today.

reply
Are you asserting that we must use the exact same heat rejection architecture the ISS uses for data centers?

https://en.wikipedia.org/wiki/Liquid_droplet_radiator

reply
So the "some day in the future" orbiting data centers proposed by a known fraudster will use an experimental-at-best radiator technology that has never been tested in flight yet? That tracks.
reply
Orbital data center constellations are also studied by Google, the idea is not just from Elon.
reply
The problem of the $ per kg to launch huge radiators and "orbital datacenter" type stuff of satellites into space doesn't seem to be resolving itself any time soon. Even if we hand wave away and say that 5 years from now the starship will be such a spectacular success than it costs only $200 per kg to send stuff into space (drastically lower than what it costs now), it would still be vastly less expensive to build a terrestrial solar datacenter in the middle of the southern Libyan desert than it would be to do the same amount of compute in space.
reply
> it would still be vastly less expensive to build a terrestrial solar datacenter in the middle of the southern Libyan desert than it would be to do the same amount of compute in space.

Perhaps, but the parent comment posited that some people would pay the premium for orbital data center due to much better security. At $200 per kg to orbit, it could make sense.

reply
With the caveat that Libya, being located upon Earth, is affected by solar power not working at night or when it's cloudy, and is also subject to being in Libya, which is a country with bureaucrats that are going to subject anything built in the country to their laws.
reply
> Space is a near-vacuum, so it's hard to get rid of the heat.

It's not that hard, Starlink constellation already dissipates around 100 MW of heat, split over many satellites. Any orbital datacenter would use similarly distributed satellites.

reply