upvote
Depends on what low/high end are. If low end is 4 core and deployment is 128 cores, this is not acceptable.

But if low is 64 and high is a whole cluster, now that's a different kind of maths.

Edit: The article states the following:

> "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."

So, it's more the former. The testing machine is probably a 8 year old server on an office shelf, and prod machines are 64-128 core cutting edge Xeons/EPYCs.

Lock contention. It was MUTEX spins in Windows 95, it's lock contention in 2026. Why this is still a thing, Microsoft?

(Man, I felt old while writing this.)

reply
I always this was standard to have a test setup in the production environment on the latest hardware (that’s how you qualified that hardware in the first place) running one “unit” of the system (whatever that is) at full scale and load (ideally higher).
reply
We had this conundrum early on when we were building the Neo4j DBaaS - there was zero chance to afford to run test environments as big as production.

We solved that by sharding customers, so about every 1000 database clusters got their own k8s cluster; production then could run hundreds of those “orchestras”, but we could afford to load test one of them and be relatively confident the test was valid for production.

This relied on the various cloud vendors running the front-line load balancers that took traffic and moved it into the shards though. I was about to say that seemed to be the case for Microsoft here as well, but actually it seems they load tested an individual server with different specs than prod.

Tricky problem though - it seems they reasoned if a lower specced server passed the load test, a much beefier prod server would handle the load too. Wrong, but you can see why someone made that assumption.

reply
When i saw the lock contention - I immediately thought distributed lock component
reply
At Microsoft's scale, it's extremely difficult to simulate production scale because the sheer size of the production system is overwhelming, and many issues emerge from the combination of sheer volume and the complexity that reality tends to introduce.

Microsoft tests its stuff on their own employees (old citation, but still true today: https://devblogs.microsoft.com/oldnewthing/20110802-00/?p=10...), and that includes Exchange online. However, even though Microsoft is a pretty big company, it's still teeny compared to its customer base.

reply
Testing at production scale is extremely difficult. It’s also financially expensive. In my experience, you balance the two where big changes are tested with some simulation setup to gain confidence. Though, the ci/cd pipeline never has close to full scale automated testing.
reply
...and many times you test to experience, and experience cannot possibly build interferences that simulate the instances you have not experienced nor what has been imagined based on that experience. some ways of precipitating the demons that are in the ontology of not knowing what you don't know are monte carlo, or perhaps noise injection, but those are time consuming, quite wasteful, and truly hated by people used to working with deterministic machines. that makes finding corner cases very difficult

plus, mistakes are the essence of learning and sometimes mistakes are necessary.

likewise, we are a very efficient world and efficiency hates redundancy, and hates nonconformity...of which those are two of the features that make exploits widely successful and failures massively impactful

reply
I think MS could afford a bit of a test lab?
reply
Yes but something that works fine in a test lab won't necessarily cope when confronted with hundreds of millions of users.
reply
I read that part of their statement as "QA were set a priority to ensure existing hardware wasn't affected, which reduced their (QAs) capacity to test on new hardware which was assumed to be safe"

They find QA dirty in their statement, because they don't state who set that priority, nor why the new hardware was felt to be less likely to be affected (what did their risk matrix look like for their hardware profiles i wonder)

reply
[flagged]
reply