upvote
The bunkers are already out there e.g. limestone mines in US, Italy,

https://www.datacenterknowledge.com/build-design/safer-under...

reply
Lots of people have built commercial, non-military, boutique scale stuff in hardened facilities, nobody I'm aware of has done it at scale. There's certainly money to be made from selling expensive "we put datacenter stuff deep down inside a mine" maybe to some weird enterprise and government customers, so the revenue per rack (or per theoretical 208V 30A circuit?) is probably a lot more than commodity colo. But I would very much like to see the aerial photos, if anyone can link them, to a claimed large scale underground or hardened datacenter.

It's a severe engineering challenge to hide massive heat exchangers on the scale of the datacenters built in the industrial area near Dulles (coordinate example: 38.9817915,-77.417775). Find those on a map and observe the scale of their cooling and generator systems and then try to find anything comparable that's been built underground.

reply
Agreed it’s small potatoes, and as a sibling mentions you will still have infrastructure exposed that poses a target to disable the data center, unless you were to drill geothermal and find an underground river to cool it.

Just wanted to make the point you don’t have to pay to excavate the cave :)

reply
There's some 200MW capacity (only 80MW is currently used or being built) data center in a mine near a fjord in Norway. The limiting factor for its capacity seems to be the capacity of the electricity net nearby (which you can just hit and the DC is useless). With how NIMBY Norwegians are there's no way it would ever get to a GW or more even if it might be both technically and economically feasible. That's just one project and can't be replicated anywhere else. It's just a massive advantage that above ground datacenters are relatively 'figured out' and similar everywhere.

edit: 61.9340107, 5.5084207 are the coordinates

reply
Do you know if that 200MW figure is total power budget for the facility (including electricity used for cooling systems and other facilities-support systems), or up to 200MW of electronics loads generating heat? The latter would be something like 300 cabinets with 5-6kW heat each.
reply
Apparently it's for the whole facility, they have a power usage effectiveness of around 1.1 so 175-180MW of electronics.

https://new.abb.com/news/detail/5924/abb-solutions-power-eur...

reply
Germans hid plane engine factories underground with old (not so efficient) diesel generators running 24/7. 80 years ago, so the tech was leaner. Also they didn’t have issues with ventilation and even today if you don’t know where - you won’t see anything driving around.
reply
Those underground factories were staffed by slaves, who worked until they died. Do we really want to copy monsters like the nazis ?
reply
That's probably just because there's not much demand for it, not because it's difficult.

There are mines with plenty of space in them, and heat removal can be done with water cooling. But who is going to pay 50% more for that when it isn't needed 99.99% of the time?

reply
Or when the same thing can be accomplished by putting nothing in a mine, but an approach as simple as having geographical diversity by paying for some 'normal' colo in Hillsboro, and more normal colo in northern VA on the other side of the continent, and duplicating your stuff between two facilities. Weird underground datacenters aren't exactly known for having a large number of competing on-net carriers.
reply
> Weird underground datacenters aren't exactly known for having a large number of competing on-net carriers.

Huh? The ones I know of have plenty of connectivity. The real issue is your first, that people prefer to just put their stuff in two countries over paying premium for having their servers under ground.

reply
Your definition of connectivity may not be the same as mine, I'm referring to the number of facilities based carriers and those who ride on top of them you can get at major sites in Hillsboro. Facilities with peeringdb pages that have tons of on net carriers. Is there even an underground bunker colo that's big enough to rate its own peeringdb facility page?
reply
It might be possible to do on the cheap; underground mines get decommissioned from time to time and could be repurposed as a data centre.

Although I personally would expect some sort of decentralisation option to be more effective. It's difficult to even theorise because we don't know exactly what sort of damage these strikes are doing and what countermeasures are already in place. If this strike has caught Yandex by surprise after almost 5 years of hot war then they really should hire a dude to check up on current events every so often. They must have some sort of contingency planning around a Ukrainian strike.

reply
It's sad that we have moved away so far from the decentralized web from the 90s and the P2P boom of the 00s. Just like solar panels on every house as a source of energy are much harder to take out than just a few power plants, local P2P storage, messaging, source management, etc. are much harder to take out.

But yeah, the economic incentives are creating unicorns and centralizing compute/capital/etc.

reply
The war in Ukraine had showed us that distributed power generation, like solar, and nuclear are most resilient. Nuclear because people do not dare striking the power plants.
reply
Building any appreciable amount of compute capacity (on the scale of, let's say, a single large datacenter in northern VA) inside a mountain also becomes a huge heat rejection problem, since with the exception of the 0.00000000001% that leaves the facility as photons down a SM fiber cable, 100% of the energy consumed is turned into heat.

So you build your stuff inside a mountain and run a bunch of big ass liquid cooling loops to heat exchangers in the atmosphere outside the mountain. The requirement of the cooling system being out in the open air means it can also be hit by air strikes, an adversary doesn't need to penetrate the mountain, just take out its cooling.

You'd have to build a fully underground cooling system that goes to a nearby river (or huge lake) and use the cold water from that, which would get costly quick.

reply
> You'd have to build a fully underground cooling system that goes to a nearby river (or huge lake) and use the cold water from that, which would get costly quick.

Helsinki gets its water from the 120 km Päijänne Water Tunnel, which cost €200-250 million (adjusted for inflation) to build. Peak flow is 10 m^3/s by gravity alone and up to 20 m^3/s with pumps, which should be enough to cool a 1 GW data center. You could increase the cost by 2x (because real wages have gone up), by another 2x (because middlemen have destroyed productivity), and by yet another 2x (because everything is more expensive in America), and it would still be pretty reasonable.

reply
It's like the Hyperloop all over again.

Musk comes up with a crazy idea.

Investors, "analysts" and sycophants go to great lengths to defend it.

Rational people wonder if everyone is losing their mind.

Musk profits.

reply
Yea, there was even that idea to launch 1000s of satellites into LEO on a reusable rocket system. Crazy. How did people fall for that? There's not even that many satellites in total today.
reply
It is indeed difficult to tell in advance (and sometimes after the fact) which of his ideas are sensible and which are nonsense.

Recall that while he managed to hire the right people to get reusable rockets cheap enough for Starlink, it was also the case that in 2016 he said that SpaceX was aiming to use the 2018 Mars launch window for what is now Starship, and also that some time around 2018 it would be possible to summon your Tesla car from the other side of the country and it would find its way to you autonomously and only stop for charging.

reply
Its not difficult, people are selfish, so they lie for their own advantage.
reply
How did Musk profit from hyperloop? Except as payment for building stuff that actually exists now.
reply
Musk sells cars.

Hyperloop was stupid and failed, and it also killed in a few places development of public transport, which of course means people continue to buy cars.

reply
Iron Mountain does it but they used an existing limestone mine with a large water reservoir [1][2]

Edit: I guess the scale isn't comparable to a, for example, Google data center though.

[1] https://www.ironmountain.com/data-centers/colocation/our-dat...

[2] https://betterbuildingssolutioncenter.energy.gov/showcase-pr...

reply
So they want to build it in space. Trade one heat problem for another, but put it completely out of range (for most attackers).
reply
Yeah, but which attackers do we have to worry about nowadays?

https://www.reuters.com/business/media-telecom/russian-satel...

reply
They already run liquid cooling loops - compute density is way beyond what could circulate through air. Adding more pipes is probably trivial on the scale of datacenter expenses
reply
Liquid cooling loops don't solve the problem of needing to reject the heat to the atmosphere outside of the building. It just makes it more efficient to transport the heat. At the grossest possible definition, a datacenter of N megawatts on Y acres of land is going to emit the same amount of heat whether it's air cooled equipment or liquid is used to take the heat to the outdoor heat exchanging equipment. On the scale of any very large datacenter the heat exchanging equipment is visible from aerial photos (and from space), take a look at google maps for some of the largest hyperscalers for examples.
reply
I assume that "liquid cooling loop" has an endpoint outside of the building which hosts hardware. In this case it's exactly same problem - just make pipes longer.

Do you need many acres of land to host a heat exchanger?

reply
So how much does the bunker help if you can just drone the cooling systems on the surface? The the computers either cook themselves or have to do an emergency shutdown.
reply
If the other end of the cooling loop is a giant reservoir, as is the case with plenty of other 100+ mile water transport systems, it doesn’t matter if you drone it. Fly all the drones you want into Lake Anthropic, there’s many more cubic meters of water than there are drones.
reply
It can help to preserve all data and compute. Once cooling can be re-established, you just continue.
reply
That's something, but if it's no harder to disrupt than if it were out in the open, have you really gained much?
reply
Pionen in sweden is a datacenter in a bunker. The Pirate Bay used to be hosted there.
reply
A few racks or even a couple of dozen racks could be called a datacenter, but nothing on any real scale. Enough to host important and niche things? Sure. I was referring more to like the size of what you might see in Quincy, WA.
reply
Pionen is small but it is quite a bit bigger than "a couple of dozen racks". More like 100-200 if I recall correctly.
reply
By 1990's standards that was small. Today it is tiny.
reply
In the US the most common deployed size is below 5MW?

Dont know why i can't reply to the person commenting below. But where's the statistics? Who has these datacenters out in the field?

reply
Yes, but that's not where the volume is. A 'small' data center, even here in Europe has 100's of racks. There are of course many tiny ones but collectively they don't even add up to one single large Google or AWS DC. Those things will run in excess of 1 or even 2 GW.
reply
According to their site, Pionen has 1.5MW of cooling capacity https://swedendedicated.com/datacenter/
reply
neat. why can't you put the heat in the mountain?
reply
For broadly the same reason that many years of heat from braking has heat-saturated the london underground.

https://www.google.com/search?client=firefox-b-d&q=london+un...

You could probably drill sideways into rocks and pump heat into the mountain for a certain period of time, but on a multi-month or year time scale, it would catch up with you after you had saturated it.

reply
Commodity hardware may be disposable, but data that is not reproducible is worth it. I suspect most storage will move to secure environments.
reply
deleted
reply