upvote
To add to the speculation. The error messages we were observing along with the Microsoft status updates read like connection pool exhaustion. The ticket update history started with Microsoft thinking a 'restart of affected servers' fixed it and troubleshooting 'health check failures'.

Anyone who has seen connection leaks/pool exhaustion in practice, it looks like an otherwise healthy low utilization server right up until it stops replying, and a restart appears to fix it until it runs into the limit again. Easy to think that your health checks are the problem. Per MS updates, they rabbit holed on this issue and built 'an alternative health check method' which worked in testing (i.e. low number of test connections); but when they tried to apply this change, servers would go unresponsive anyway.

My guess is a rushed update with a connection leak somewhere in the stack. A call that opens, but does not close. Just the kind of thing that AI can miss because it's only trying to hit the goal and not necessarily clean up after itself.

reply
> But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.

I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?

Exchange and Active Directory are easily my favorite MSFT products. I've always wished they would port them to Linux, even though I know that OS dependencies make that a non-starter (though they surprised me w/ SQL Server for Linux).

I'll concede that the the biggest Exchange environment I ever ran was around 1,000 mailboxes. I've never seen how it fails in really large environments. I was also fortunate that I only ran the earlier versions (pre-2013) in sub-500 mailbox environments. In those settings, though, I found it worked well and wasn't bad to administer.

It was very reasonable to use in small business environments, too (sub-100 mailboxes). I put a lot of little Exchange 2003 boxes out in small offices. It plugged-away very effectively.

Yes-- there were updates here-and-there that had issues, for sure. By-and-large it was a stable and predictable product.

I always tried to be mindful of maintaining free disk space, following MSFT's recommendations for speeds-and-feeds for I/O, and running on solid hardware (ECC memory, redundant power, UPS). I feel like I would have had more problems if I'd scrimped on that stuff, for sure, but I was unwilling to let Customers cheap-out.

With Exchange 2019 and high-availability (database availability groups) I was easily able to do zero-downtime maintenance. That "big for me" Exchange Customer I worked with had more downtime in the last 24 hours than they did the entire time they used Exchange 2019 in production.

I haven't seen the code, so I can't speak to the "nightmare of code". Large parts of it were ported to the .NET Framework in between the 2010 and 2013 releases. The 2013, 2016, 2019, and current Subscription Edition releases appear to be an evolution of the codebase with no signs of major rewrites. I question how much pre-2013 code is even present in the codebase anymore.

I dunno. My experience just hasn't been that bad. I don't want to be an apologist for MSFT. I've never fully gotten over the suspicion I have for their motives going all the way back to the Halloween documents. I don't like how they're forcing everyone to hosted offerings and killing their on-prem products. I've been pleased with the value my Customers realized from Exchange.

reply
> Exchange and Active Directory are easily my favorite MSFT products. I've always wished they would port them to Linux […]

How is Samba 4's AD 'emulation' these days?

reply
You know, I still haven't tried it. That's been on my to-do list for awhile now. I really like Samba (and used it as my homelab CIFS server for years) but I never explored the AD Domain Controller functionality.
reply
It's pretty good now, although, like all open source projects we're being hit with a tsunami of AI reported bugs right now.
reply
> I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?

yup. You're the only one.

When we ran our own mail servers, "some fuckery with client's Exchange server config" was good 90% of mail problems we had.

And my experience with the cloud version management decided we will migrate to has been entirely awful too, simple stuff like "stop spamming the mailing list I get constantly" is entirely impossible to set up, despise both global and per mailbox rules to let it thru, it still lands in spam and the useless support we apparently pay for can't figure out why

reply
> When we ran our own mail servers, "some fuckery with client's Exchange server config" was good 90% of mail problems we had.

As an Exchange admin (and Postfix, Sendmail, qmail, exim) that was my experience, too. I think the popularity of the product, questionable defaults, and the low barrier to entry for admins (read: lacking in experience and knowledge) contributed to that.

reply
Managing it in small and large corporate enviornments for over a decade led me to believe that when it was working, Exchange was very stable and reliable.. But when things would break, they would break in a way that required extensive knowledge of all the areas to fix. Which was a form of job security so Im not complaining.

I guess past reliability might be irrelevent here if an AI slop patch for AI discovered exploits caused the outage. Just speculation though. Since exchange has so many moving parts and so much legacy code, with even more bolted on to make it a cloud service, I could see how an AI exploit fix would not have all the context needed.

There is no information that it was caused by Exchange either, guess we have to wait on those answers. It is just a reasonable assumption.

reply
Open source options tend to require a bunch of knowledge to set up but after that you generally are equipped with that knowledge to solve any configration issue. I wrote "configuration issue" because aside from "ran out of disk space" none of them really "broke", so it was never "fixing the thing", just the usual mail server admin stuff of figuring out why this or that server didn't like us.
reply
A lot of the bad situations I helped bail other contractors out of, when it came to Exchange, related to disk full conditions, a lack of understanding how transaction logging in the database worked, and database inconsistencies brought on by hardware RAID controller issues or hard power-offs. Very nearly all of my ugly ESEUTIL and ISINTEG experience came from people not respecting the product's storage needs.

Sysadmins who were used to running file servers and thought Exchange was the same thing were the problem. A DBA was actually more apt to understand the product's storage needs because it wasn't wholly dissimilar from the RDBMS they supported.

reply
> The postmortem on this will be interesting.

Any bets on whether it's DNS?

reply
Well, my home computer with ddns lost network connection at the exact same time and I had to reboot it for it to work so, maybe?
reply
my bet is on some copilot integration
reply
It's never DNS
reply
That's just what big domain wants you to think. But we know better.
reply
DNS, IPV6 or BGP. It’s always those guys
reply
Certificates expiring
reply
...given it is windows environment... maybe
reply