First, you have every client retrying connections, whether automated or manual retries. This generates extra load over normal.
In addition, whichever systems come online first often get a ton of traffic sent to them, which can immediately bring them back down if you don't have ways to prevent it. You will get a wave of systems coming back online, a ton of traffic hitting them, and then falling over again.
Now, I am sure Microsoft is used to this and has procedures and systems in place to mitigate the issues (I know we did).
However, sometimes if the right combination of services are failing, even the strategies you had to avoid this sort of failure can themselves fail.
For Copilot.
This seems to be the smoking gnu, now what would it be ...
(I've recently been revisiting the X-Files. Can recommend.)
I saw a frog smoking on The Little Rascals once, but never a gnu.
EDIT: Seems like now they confirmed that: "We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service."
@MSFT365Status
16m
We've not identified any recent change correlated with the start of impact; however, our investigation indicates the failed health checks may originate from a specific component that's responsible for processing server health. We've taken steps to prevent additional servers from being impacted while we work to return the affected servers to service. More info under EX1550543 and MO1550821 in the Microsoft 365 admin center, as well as status.cloud.microsoft if the admin center is not available to you.
> Previous status: We're continuing to work through restarting and validating the affected servers to safely return them to service, and we're exploring opportunities to expedite this process. In parallel, we're assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.
If failures started at some point in time then something must have changed just prior to that. If no changes in traffic were detected (no DDOS) then it must have been a change, be it code or config. Duuuh.
Not at all, they could have simply hit an unexpected behavior for their infra they hadn't seen, tested, or expected before. Global scale infra gets very, very complex, and tbh this is Microsoft... they're not very good at it.
More often than not, it's DNS :)
Edit: Oh, there it is.