Yeah, this one is bad because it’s off by so much I’m shocked it wasn’t caught by tests, alerts about unusual changes in the billing system, or even accounting. Like surely the P&L reports look all kinds of wrong right now, they have to be showing like 6M% profit margins and revenue measured in quadrillions.
replyI’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
If I were to guess this bug is in the "display" part of the system which is probably distinct from the "actually take money from the customer" part of the system. One can imagine they have gates on the "actually take money" part, especially for a large bill like ours which was ~$300b or about 2.5x AWS' 2025 revenue... In one month. Surely if we had actually accumulated that bill they would be the ones with the problems when we can't pay it.
replydeleted
replyI don’t know how something like this makes it to prod. That’s multiple levels of failure.
replySame here. I got an email with a bill of $233 million and an estimated $433 million until the end of the month. I panicked and nuked my entire setup (which wasn't used that much, anyway, the alert threshold was $1) - I really wonder how many people did the same.
replyIt's been 2 hours and I still haven't fully calmed down.
Same - just had some malicious bots running through my platform last week and really thought they found a security hole after all. Even though the amount sounded ridicoulus, I got quite nervous and a very bad feeling when I logged-in AWS and saw that price.
replyThis is embarrassing for Amazon, but I’d take laughably wrong over subtly wrong any day. If the bug made bills 20% higher I probably wouldn’t have queried it.
replySame. Cold sweat for about 20 minutes. Even though I saw the service health notification, I still spent the last hour trying to find where my storage spiked. In any case, I'll be tearing down plenty of stale infra after this!
replyThe should pass a law saying they should have to pay you the amount over the correct bill as compensation; I bet they'll stop making mistakes like this pretty quickly after that
reply