upvote
No tests? Just mess up some mundane detail [1] and voila! Wake-up calls and heart attacks for 100,000s of administrators?

1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo

reply
There will have been tests, but there will have been missing end-to-end tests. Test 1 will verify that the new system/product emits billing entries in some expected way ("We did 100 bytes of operations and we see we called the billing system for 100 bytes of stuff, yay, test pass"). Test 2 will be in the billing system ("We provide an incoming bill for SKU#12345 for 100 gigabyte-units and we see it costs $17, yay, test passes"). But they won't test the two things together because it will be harder to do and the teams will have different management chains. Seen it happen several times at several companies. Somebody will have said at some point "we should actually have the tests charge money" and somebody else will have said "well we can't have the tests actually charge money, that's a legal/accounting problem, it might even be a crime" and then nobody would have asked what the next best thing was.
reply
But these aren't the right services where the test should be, right?

There's another service that says "ok we take the 100 bytes from A, and we take the $17 SKU from B, and this should equal $X".

It's the third service that multiplies these things that failed. Where are the tests for that?

reply
To me this sounds like a human-or-LLM-driven error. There must be a pretty limited set of factors that determine a pricing unit: I’m not really sure how a deterministic system could do that infrequently enough to not be a bigger story. Maybe a reeeeaaaallly rare race condition or something like that? To me this smells like having enough manual work involved in the process to fuck something up, but not nearly enough eyes on it to notice.

I’m totally guessing though.

reply
Not even tests but just some basic anomaly detection lol.

Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it

reply
You overestimate how much people give shits at big techs like Amazon. When literally everything is driven with sticks instead of carrots, the work culture does not invite employees to proactively care about product quality.

You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).

reply
Speaking from my experience at Amazon this is not the case. Any customer impact like this would necessitate a COE (correction of errors) report, which means a list of required action items to prevent such issues from happening again, which typically suck up at least man-month of labor. Not to mention the report itself, which has to be written by a manager.

In fact, there are regular AWS-wide meetings where L10 technical staff will randomly pick and review reports from across the organization. Getting picked for one of these is not a fun experience.

COEs are such a huge annoyance for teams that they create a strong incentive to be proactive in preventing issues like this from happening. One of the rules when it comes to writing COEs is that they are not the fault of individuals but processes; but in reality, no one wants to be the cause of one.

reply
Amazon is heterogeneous. So much so, that positive anecdotes and negative anecdotes are near worthless without specifying the org.

Depending on if you're a cost cutting team, fixed expense team or organization, if you're a revenue driving team, or if you're a core team, or the very many other splits you can come up about the relationship between the expense/balance sheets and the team itself...there are very very different attitudes towards COEs and leadership principles.

reply
Having been the manager writing those reports, you can only practically find causes that are within a single team’s ability to resolve.

If you find a problem like this thread’s hypothetical, the process stops being an annoyance just to line level managers, and something that directors and vice presidents need to handle by changing strategic priorities within their organizations.

That entails a real loss of face for them, and because they are the ones who actually run the show, it would will only happen if you have one that is naïve or a masochist. In either case that moves them out of management.

reply
From the manager's side, you're absolutely correct. The SEV looks bad on you, and is a headache to document.

From the perspective of people you manage, it's a very different picture.

We (I say "we", because I was an IC) sit under you, and every year at performance review time you're effectively required to put some percentage of us in the "LE" bucket. Never mind that we could theoretically all HV3+ if you measure by "normal" peoples' standards, your manager isn't going to let you mark all of us as HV3 at the performance meetings. I know this, because I've been there as well at those meetings where truly high performing people were downrated to fit a distribution.

So what happens? When I see a peer's critical lurking bug, I have no incentive to fix it for the sake of prevention. If I fix it quietly so that it never surfaces, it looks like I haven't done any work for the week, or have done un-impactful work, and I get the stick from you. Preventing fires doesn't look like work, to non-technical eyes.

The only "safe" way to play this game of "survivor" is to let that bug surface eventually, then when the SEV comes up, I jump in and fix it, earning your approval, skip approval, VP approval, as well as potentially the other person gets the stick from you, because you have to give the stick to someone anyway, you get a reason to stick it on them. At least it's not getting stuck on me.

I'm sorry if this comes off as shocking to you, but it really shouldn't; the incentive structure is NOT set up for teamwork, plain and simple. If "putting customers first" is a value, then it absolutely needs to start from systematic changes of how people are managed.

reply
This ^^^ amplified by indifference and not giving a shit caused by "AI Adoption".

There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job.

And no one is shooting for better than OK.

reply
Are you speaking from experience or simply making things up? I know a fair number of former AWS engineers and managers. None of them think like this.
reply
I am former AWS and this is pretty accurate.

The other factor to add here is that, with some exceptions, the whole company feels like a Rube Goldberg machine and very few people care about what happens outside their cog (because they’re not incentivized to do so).

reply
Rube Goldberg machine attached with used bubble gum and somehow the bubblegum was chewed in all the wrong ways
reply
Maybe they are former AWS employees for a reason and now want things to go better than they were at AWS.
reply
[dead]
reply
"Former" seems to an important detail here.
reply
If I worked at a place like that, I'd sure as hell work my butt of to get a job somewhere else.

Or in my case, actively ignore any and all recruiting from that sesspool.

reply
If someone quits their job, do all their opinions suddenly become suspect? You're kind of damned-if-you-do-damned-if-you-don't. Either you work for the company and you are biased one way, or you quit and now your bias is now suddenly the other way. I've joined and quit many jobs and my opinion may or may not have changed due to my change in status but it is clearly and ad hominem attack.
reply
Not the OP, but:

The point was not that their opinion is suspect, the point was that they are former because people who care about the customer get fired and/or that everyone who cared is former, so nobody who is left cares.

reply
Yes, I am a former AWS employee.

I got put on Focus because my "contributions were not coming through" to leadership.

reply
> take the 3am on-call and be the hero instead

Ah yes, the good old ITism "Everything's good, what are we even paying you for?" followed by "Everything's on fire, what are we even paying you for?"

I moved out of it largely for that reason, am now an infrastructure/IT project manager, quite refreshing actually.

reply
The trick to surviving under such management is to jump in and put out other peoples' fires but not spend time preventing them even if you know how to.
reply
How did you swing that transition? Did you study for PMP before applying around, leverage network to get in the door then backfill skills, or what?
reply
Yeah, I got my PMP before applying around, combined with some luck I suppose. My IT role was basically a solo sysadmin before where I basically was the technical PM + engineer in one, and I did that for about 8 years so I had a ton of experience I could spin on my resume.
reply
Clearly some folks here have never worked at Amazon before. It's genuinely terrifying to see what most of the internet runs on.
reply
While I didn't work on AWS, I did intern on the retail side of Amazon, and there's definitely this sort of monitoring in place. Surely somebody was paged. And even if not, this is "just" the cost explorer estimations, not what is ending up on folk's bills.

I learned about <https://en.wikipedia.org/wiki/2011_T%C5%8Dhoku_earthquake_an...> from alarms like this, as sales in Japan almost entirely stopped.

I've been told a tale of another incident where some customer ran some huge cpu-intensive workload that didn't do any networking. It caused various alarms to fire because it "looked like" a part of the network was idle (potentially indicating some sort of networking failure)

It's generally (in the broad sense) easy to add alarms for things going wrong, but in my experience anomaly detectors are just as likely to fire from other weird things like that happening.

reply
I think billing is the only thing AWS doesn't really care about optimizing or putting enough tests to avoid anomalies lol.
reply
They've already got anomaly detection: their users.
reply
If only there was some way to get anomaly detection services [1] inside of AWS...

1: https://aws.amazon.com/what-is/anomaly-detection/

reply
It’s fractions of a penny Peter.
reply
It's just like in Superman III
reply
I'm sorry but anyone that sees a multi million or billion dollar bill on an account that does nowhere near that should not be scared. It's obviously a mistake. Stories like this have happened with banks in my country. Check your account and you have billions in there. Guess what happened to those that withdrew money? The judge told them any reasonable person would know this is a bug. Had to give it back. Same thing here, any reasonable person doesn't get scared.
reply
Why would you think there are "no tests"?
reply
We have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.
reply
why would a test setting unit to Bytes fail and not MB, KB, or GB, and so on? That's like trying create a unit test for email opt-in, both true and false are valid values. It's up to the user to select the right one.
reply
I'm not quite following your objection.. I'd expect a test that checks the multiplier is correct would detect orders of magnitude discrepancy. So if you're billing $x/byte you'd write a test for the billing thing that checks that, given y bytes, the bill is x*y.

[edit] This may need to be an integration test to be effective, there is a certain peril to mocking that could bite you here. But that's fine, we have the technology.

reply
missing canaries more likely?

insufficient tests that dont assert on the right things?

the existence of a test doesnt mean it catches the right thing

based on the description, id bet the COE action item will be to do a migration that enforces units are passed at the billing service level

theres no good reason for the billing service to make up its own units.

reply
Yeah types and APIs that are difficult to misuse are way better than any regression test, canary, or what have you. Systems that are correct by construction always beat "you're holding it wrong".
reply
Technically, there could be a test. It could just be wrong!
reply
If a tree falls in the forest and nobody hears it...

[edit] Testing your tests, like testing your backups, is a good idea

reply
Yes, test the negative case as well. eg if you get the system setup so you can log in, also make sure you get permission denied for bad login info.
reply
I wonder if AWS billing still uses CSV files for passing data around.

IIRC it was one of my first on-calls at AWS over a decade ago now, and I got a page early evening because some stuff we did with billing records broke because some "smart" engineer thought it'd be a great idea to put an experimental record in with a description something like "I wonder what happens if I put, a comma in this field", into the production record. I watched region after region fail the same way as the record spread. That one engineer made a mess of lots of people's evenings. They could have used the test endpoint, but no. Much better to test in production!

reply
That was just a massive operational failure, not the fault of any single engineer. No change, except hotfixes, should be able to land on prod unless it has at least go through test, staging, and at the scale of Amazon, shadow testing.

Engineers will do what engineers will always want to do, they want to see how things break, and sometimes they manage to fix it.

reply
Kinda want to push back on that. It's not like engineers are separate from operations, not really. A lot of operations is just what engineers have been socialised to do.
reply
CSV files are widely known to be used by the most frugal companies so of course it is.
reply
Have you seen what RDS costs‽ AWS couldn't possibly afford that!
reply
This isn't a flippant comment. Imagine though, being presented with this. Imagine having some underlying health problem (e.g. cardiovascular).

Do not be surprised if real people actually die from this mistake, from the anxiety, the surprise, the helplessness.

reply
But someone that susceptible is likely going to have a bad reaction to many possible unexpected things. How would they react to a minor traffic accident? a family member getting hurt? a letter from tax authorities asking questions?

Having that serious an underlying health problem means everyday life represents risk for you. I don't think that means everyone else has to behave differently wrt (in this case wrt to billing mistakes) to keep you healthy.

reply
its so far out as to be obviously not real

a smaller error by say, just one or two orders of magnitude are much more believable as a reader

reply
I've seen people have panic attacks over much smaller amounts.

Just because you've not seen it or cannot fathom it happening in your world doesn't mean it doesn't happen.

reply
Was this the day that gp3 EBS volumes came out by any chance?
reply
This is why I always fail loud rather than pick a stupid default.
reply
Had it been half a million dollars or something or say like a few hundred dollars?
reply
Imagine a programming language that has physical measurement unit support so this could have never happened.
reply
"I must've put a decimal point in the wrong place or something. I always mess up some mundane detail."
reply
Unit mistakes happen all the time, which is why you should be using your units library religiously and still being vigilant even then.

Worst case I've found was off by 15 orders of magnitude.

reply
Degrees, very funny.
reply
It's not difficult to write regression tests that catch unit mistakes.
reply
One of the Mars landers famously failed due to unit conversion errors from metric to standard.
reply
I didn't know the imperial system was named "standard". Funny, cause its everything but standard both internationally and its definitions (which are not standard as based on SI)
reply
Claiming something isn't standard because it isn't based on SI is entirely circular in the case of weights and measures.

That said I wish the US would bite the bullet and make the switch. Mandating dual labeling on everything would be a great start. Then in 20 years we could narrow it back down to one.

reply
You just need to start teaching kids the metric system. Then when they'll grow up the switch will almost magically happen by itself.
reply
I know you’re being snarky, but the US system is not “imperial” anyway. It’s properly “US Customary” but is often called “US Standard.”
reply
Wasn't it (also?) the Ariane V flight in 1996? Oh, NVM, that was an overflow error.
reply
Multiple rockets exploded and space missions failed because of the imperial vs. metric BS, and those mistakes were made by people all with PhDs or equivalences. This is still pretty mild in comparison haha.

Units and datetime will always be the bane of any professions ...

reply