That's entirely why they can blow compute on the fun projects like this. If they had to pay extra for the electricity, they wouldn't do it.
I wonder why they don't have some kind of scheduler which makes sure there are never any idle minutes. One would imagine they at least would have autoscaling on their production serving workload and use the freed compute capacity for model training for example.
Not using it would not save them any money, they already paid for it.
The second is approximately no better than astrology.
There is cost, but the cost is mostly the opportunity cost of not being able to do something else.
I don't have any insight on modern GPU datacenters, but in decades past, some owned and operated datacenters didn put effort into making sure power management worked because the cost savings were worth it. I'm pretty sure I saw plans to shed load and power off servers if a utility made a demand response request or in case of loss of cooling. I wouldn't be surprised if some owned and operated data centers do regular full shutdowns at off peak... WOL, IPMI or RTC wakeup can bring them back when needed and if you already have a dynamic service orchestrator and setup times are acceptable, why not shut down if there's no actual priority work and there's also no idle priority opportunistic load either...
> why not shut down if there's no actual priority work and there's also no idle priority opportunistic load either...
Full shutdown and startup often kills capacitors and used to be dangerous for rotational HDD.
Sometimes once you turn things off, they simply don't come back on. It happens.