upvote
Currently (17:22EST):

> Previous status: We're continuing to work through restarting and validating the affected servers to safely return them to service, and we're exploring opportunities to expedite this process. In parallel, we're assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.

If failures started at some point in time then something must have changed just prior to that. If no changes in traffic were detected (no DDOS) then it must have been a change, be it code or config. Duuuh.

reply
> then something must have changed just prior to that

Not at all, they could have simply hit an unexpected behavior for their infra they hadn't seen, tested, or expected before. Global scale infra gets very, very complex, and tbh this is Microsoft... they're not very good at it.

reply
Unexpected behavior doesn't happen out of the blue, it's not LLM, these are deterministic systems. It's usually config change or code change, not necessarily in the component that's misbehaving, but it is somewhere.

More often than not, it's DNS :)

reply
[delayed]
reply
Is this a case of "who shall healthcheck the health checker?"
reply