Edit: and when I say “99% of products”, I mean “99% of products where the team thinks they are building something too complicated for a simple setup”
Every time I've needed to manage something on AWS I've been shocked at just how over wrought the whole system is. There's tons of As-specific terminology for everything, and lots of stuff is tremendously complicated to manage. I can definitely understand why companies need to hire people who are experts in AWS specifically, it's complicated enough to justify that. However, for me personally I'd rather learn more traditional sysadmin systems. The skills are more evergreen, and I'd rather spend my time learning open systems than one tech giant's specific system.
About 6 months ago I needed to migrate some of our systems from DigitalOcean to Hetzner. It was a 2 day process that was very painless. The only complicated bit was managing the DNS switchover with zero downtime. If we were moving those same 3 components from AWS to GCP or Azure, it would have involved needing to rearchitect and rewrite a lot of software.
I remember many years ago we hired a junior developer who just finished his internship at AWS and he showed me the dashboard he shipped all by himself in the summer with no product or designer help. It looked horrible.
Some devs have a good product/UX sense but the vast majority are horrendously bad at UX.
My point is that maybe it was intentional, but just bad UX culture.
Edit: It wasn't intentional
Some background. I work at an Amazon sub. This is a good UI for the way we work. We don't spin up a single machine pretty much ever unless it's a cloud dev machine, at which point the price is listed at startup on a custom internal UI. They should consider putting that UI in the ec2 console.
When I spin up machines I pick an instance class by looking through specs and the price chart and set it via AI into a cdk construct. Usually pick a relatively normal machine type digging through all the ilvarious enterprise discounts (which are not reflectedin the prices in the console). Then as I roll out or when I get resource limit alarms on the fleet I adjust the instance types. Or when accounting asks me about price. In those cases I usually look if it's worth it to optimize.
The enterprise discounts are a big consideration. Every year new hires make bad decisions because they don't know about the discounts. They wildly affect total cost. Some things are more expensive (lambda first few years), and others are very cheap so we dog food. The console price in no way reflects reality.
In 15 years we've had about 1k services stood up, around 700 are active. 2000 or total counting tutorials and tests. That means out of an eng org of 500, we've made those decisions maybe 10k times total.
That's how Amazon thinks about it as well. So yeah I agree that the UI isn't meant to be like one where your spinning up a host. I haven't spun up a single host in like 5 years, but I've made many clusters.
But that doesn't mean it shouldn't be better to work for a wider audience. Customer obsession and all
In the end, our leadership changed what we were building so often that all of the UI work was scrapped long before we shipped. We ended up launching a janky console, quickly assembled by SDEs who were racing against deadlines. We skipped virtually all operational readiness work to meet the launch deadline. After claiming the launch win, the director, two managers, and the pm promptly left for other orgs.
This may be valid, but even if it is someone (or a group of people) at Amazon are violating one of their core leadership principles - Customer Obsession
https://www.amazon.jobs/content/en/our-workplace/leadership-...
A useful (and hopefully delightful) UX is key to showing customer obsession.
That being said, I personally feel the UX at Amazon sucks overall, not just for pricing/packaging but even getting basic shit done. So perhaps Amazon (or at least AWS) doesn't think a good UX is a key ingredient to demonstrating Customer Obsession.
AWS services names are notoriously bad at communicating what they actually do: https://expeditedsecurity.com/aws-in-plain-english/
So no, they care zero about their customers, except maybe for getting as much money as possible out of it.
Ask me how I know
Often I see something that's supposed to be leaner - like Fargate is leaner than renting a whole server to run docker, right?
So it's cheaper as well? - Well, no.
Also if you reach any appreciable level of complexity, you should move to IaC - configuring all that stuff on the UI, and getting it right is torture.
> the UI flow is geared towards the idea that engineers don't really see the costs, they just build stuff and then management pays at the end of the month.
So this is about the engineers consuming AWS, not the ones who designed and implemented AWS
It was absolutely hammered into me in the years of working for startups that followed, though.
I think that applies both to Amazon's dev system and pricing system. From what I hear about the insides, alignment is chaotic neutral inside of Amazon, but that shouldn't affect how we judge the system itself.
I think the problem is that nobody understands the size of the problem.
For most tasks, the accomplishment is getting something to work. That takes 90% of the time. But the UI requires polish, working things out, backing out and trying again, and takes the OTHER 90% of the time.
I remember talking to a friend who worked with apple to port some dvd authoring software. And steve jobs started with the UI, and said "this is what you do". I think it was just a blank screen and you drag your video onto it. the software they were porting was a bunch of windows type confusing nonsense, and they had big changes to make.
That said, AWS might be a dark pattern. Remember the cable companies that didn't WANT to show the hidden fees? because $29.99 a month was really $71.41?
When I started my latest project my first rule was: I never have to login to AWS console. I didn’t achieve ‘never’ but I am pretty close and the experience is a lot better
"Azure’s Security Vulnerabilities Are Out of Control" - https://www.lastweekinaws.com/blog/azures_vulnerabilities_ar...
I just read:
> If I had learned one thing from my past life was that if you see the signs of an abusive relationship, you have the option to walk out, and you don't, all that follows is your own fault.
so... :)
https://eng121.net/online%20textbook/cause-effect/The%20Wall...
If you flee domestic violence you are more likely than not to lose custody of your children to your abuser.
That's because joint custody is the default and you need to have really good evidence when you want to restrict a kids access to their father.
> If you flee domestic violence you are more likely than not to lose custody of your children to your abuser.
"Being forced to allow kids to see their father" is, to you, the same as "losing custody of your children"?
You're talking absolute horse puckey here. I'm also pretty certain you don't believe it.
No matter what you may think of parents, it is absolutely horrific that someone will argue for restricting the rights of children, and do it in a way that he feels is acceptable in society (custody is only in small part about having access to one's children; the actual right is to the child, not the parent - the child has the right to access to their parent).
I wanted to make him understand that trampling over children's rights is not acceptable.
Haven't had anything impacting in GovCloud, but if you're not there yet I'm sure there's shenanigans in the consumer version.
The idea that AWS is abusive seems a bit much to me. There is Amazon Lightsail for people who prefer pay-monthly upfront costs.
> The UI to fire up a new machine did not show me the price. I had to look up the price in another table that did not have the specs.
I don’t want to be the one defending AWS, but I don’t think that this is a valid reason not to like them. I mean, pricing depends on so many factors like reserved/dedicated/spot/on-demand instances have all different prices.
I don’t even think that using the UI to spin up the machine is the right way to do that in an enterprise setting, you should always do that through Infrastructure as Code, to know exactly what you have up and running, just by looking at that as you would with any program. I’d suggest to use the UI for simple testing, for which the costs are often (but not always) negligible.
Jeff Bezos if you see this please send me some cash.
About using IaaC to set-up the infrastructure, sure, but sometimes you just need to browse stuff before actually writing code to get a feel.
Let’s look at Lambda for a second. Deploying a lambda function to AWS costs literally nothing. And yet, depending on how it’s used, it can cost an infinite amount of money. Which price should it show?
There are far more sevices like Lambda than EC2.
"Estimated cost per 1000 invocations"
Or you can have your own negotiated private pricing which is a whole different story in itself.
But that's the problem: The complexity of doing that properly is pretty much the same as just doing your own hardware (which is what I'm working with most of the time - handling stuff on physical servers). And at that point the question should be why you're paying AWS so much money and pay your people to automate AWS workflows when you could just pay them to automate workflows on physical hardware, which would be way cheaper to run than the AWS instances.
If they know how to bill you then they obviously know how to consider and calculate all of these factors, they just choose not to show you up front.
Heck, I even have a hard time telling the price I pay on an account by account basis; because we have savings plans, those get charged against the root account and then I see $0 spent on EC2 in the individual account because it's all covered with a savings plan.
And when I'm putting together that IaC and trying to decide which new instance type to upgrade to, I have to dig through multiple confusing interfaces to figure out that what I want is to upgrade from m8a.4xlarge to c8a.8xlarge and how much that is going to cost me.
I'm tired of people acting like complex infrastructure tooling is adversarial because it's not completely intuitive. Infrastructure is hard. AWS can give you tooling and docs with patterns to follow, but they can't read your mind. Neither can the PaaS providers - they just make choices on your behalf and hope it won't matter to you.
This is still hugely prevalent at some of the largest companies in the world
I get to see how a lot of companies use AWS. The console does make its appearances, but less and less often these days.
i just use vantage (https://instances.vantage.sh/) now. their api is functional and reasonable.
It should really be a read-only layer for metadata and logs.
A) they are receiving massive discounts off of list prices, and
B) they’ve setup everything such that no-one working on the cloud can see the spend.
Companies just really don’t want employees to know what their spend is.
Einstein split the atom. Newton explained gravity. Musk can land rockets backwards on floating platforms in the ocean.
But none of them could answer the ultimate question:
How do I stop AWS from charging me $47k because someone forgot to turn off a Kubernetes cluster?
This is false. The price shows up right away when you select a machine. I dont work for AWS...
It gets far more complicated when you have reserved instances, and combine reserved instances with RAM sharing when working in a larger org.
I'm sorry, what? I just tried the EC2 launch wizard, and the price is listed right in the dropdown with the instance types. Or you can open a table with comparisons and enable the price there, along with ~20 other instance type properties.
Yeah, the AWS UI is not great. But they go out of way to make pricing predictable and public.
Okay but https://ec2instances.info/ is right there. It's valid to point out that you shouldn't have to do that, but sometimes you just have to live with the relationships you have.
If it bothers you that you need to open two tabs for cross-checking the costs, you may want to avoid every cloud provider, not just AWS.
Once you have NAT gateways, CloudFront, S3, auto scaling, loadbalancers, etc, calculating the cost becomes an art rather than an exact science. And if you don't use these, there is no point of using AWS, there are plenty of "cheap" VPS providers.
You might have leftover reserve instance that applies, which make the listed price inaccurate. That reservation might even be in a different AWS account in the same organization that you don't have access to. That reservation might not even be there between the time you quote and the time you actually launch it if someone/something did launch before you.
Your organization might also have discounts. I believe some discount may also be very confidential. For example, my reseller policy is that the customer must not be able to see AWS Billing in the organization root account as supposedly the price in that console are the price AWS charged the reseller, while we pay listed price minus any discount we negotiated ourselves.
Finally, I suppose they don't want to have prices shown in multiple places as they will need to update it when prices changes. Doesn't want to risk forgetting one place and getting sued for it. You can see that AWS documentations often do not want to mention the price at all, even if that price is currently free.
Chinese clouds kinda make this simple by making reservation part of the buying machine itself - you have to mark that particular machine as monthly/yearly committed when you start it (or convert it later). The complicated part is recycling instances - if you delete a server before its reservation ends it ends up in a recycle bin that you need to look before making new reservations.
They cannot predict what my bandwidth consumption will be, or other such variable costs. For those, they tell you rates.
That's just for ec2. Everything is like this. Super awesome when you're being brought onto a new project and trying to estimate costs for your client. And let's not forget the little tiny things that should cost nothing. A NAT gateway with no redundancy is $30/mo. That's a fun surprise.
This is the "Comparison Table" from the EC2 launch wizard: https://imgur.com/a/YjFhkzb
The pricing is right there, along with filtering and sorting.
The main reason is this is only true for ec2 and every other resource has its own slightly different way of getting the cost, making it really easy to miss things like this. But here are the steps we take to get to your image.
- First you click compare instance types, and you're brought to a completely different page with a table.
- By default, there is no column for pricing, but two columns for "storage space" even though most of the instance types have these blank.
- There's nothing that says you can add columns to this page. You eventually figure out it's the gear icon.
- Then you click the gear on the top right to look at column names. You try searching the 44 column names for "price" or "cost" but both of those turn up blank, because there's no fuzzy searching.
- So rather than use the search box, you manually scroll through all 44 column names and find pricing at the bottom of the list.
This is the definition of out of the way. It's hard to imagine why you would default to showing two different storage columns over the pricing column, when half the instances are blank on storage.
Now do FSx, which has no pricing information at all, or any links to pricing information. They have an info tab telling you your backups are incremental, which would make you think they are fairly inexpensive. Not more expensive than the filesystem itself!
So they just added a default table widget, and they didn't even bother with customizing it. You can enable the context menu for the table's rows, which works and is empty.
I worked at AWS around 6 years ago, and we had a great win with just getting access to a service that provided the full list of available instance types and base prices.
This kind of disjointness is both good and bad. It's good in the sense that individual services stay within reasonable complexity, and usually all the functionality is available through the public APIs because the UI console is just another consumer of these APIs. AWS is also very careful with permissions, internal services try to avoid escalating privileges and try to perform everything using the user-visible access policies.
But it's bad because integration just sucks, and the UI layer is the ultimate example of this. AWS console _is_ really messy.
>AWS is not built around hobbyist needs
Yes, as if no startup teams are tasked to remain within hard spending targets when they're trying to build a POC with technologies that they are not initially experts in.
The suggestion to setup some kind of IAM policy to shut things down and stop resource usage is insanely complicated for users who need this kind of feature the most. If I’m learning AWS and just added my CC to it, I am the last person to be qualified to setup this kind of an alert and policy from scratch. This needs to be a single text input in the billing page, like it is for countless spend-as-you-go services. When the limit is hit, the service needs to stop the usage at the customers peril, because that’s what they customer requests.
Hope this helps.
We set this up at my last job like in 10 minutes. Complexity is a matter of perspective, and if your job to do this, you have done this many-many times, and you have ready to use infrastructure as code templates.
Yes, AWS is massive, the documentation is huge and makes things inherently complex, but flexible too. You can define what behavior do you want when you exceed your limits. We can argue whether this is obfuscation or complexity or what, but based on my experience AWS optimizes it's product for enterprise-ish companies, that can afford to have SREs who knows exactly what to do in such cases. That is where they have their own training/certification program. For simple use cases there is AWS Lightsail where pricing is simple and easy to understand.
But even if it would be insanely complicated, that is a reason to downvote? HN used to be better than this kind of "I don't like your comment, let's downvote it".
I think about the diversity in usage patterns: from generating giant video stream broadcast somebody trying to calculate yet another digit of pi. It’s wild.
Is true, probably, that AWS doesn’t know how much anyone’s use case will cost (even when it’s yet another version of something we’ve seen before). Too many variable.
If only there were some kind of software with a text based, natural language interface that we could ask a question like “how much would it cost to do XYNZ on AWS?”
Yes, as long as you do not have seasonal traffic, auto-scaling, spot instances, burstable instances, saving plans, reserved instances, floor/custom pricing, etc. These are tools to optimize your spendings and spend less if you know what you are doing.
> defending deliberately obfuscated pricing
A bit contradictory that price simulators are fine, but then the pricing is deliberately obfuscated. Then which one?
AWS is almost never required and almost never the best option. It's the Cisco of options, it's often the default but for no good reason other than someone on the team probably knows enough about AWS to make it work.
Almost every startup I've worked at has leveraged AWS as their primary but when not they end up using AWS for something. And in every startup there's always contention with AWS spend and all of these startups invest significant time and, funny enough money (via cost savings products or consulting), to reduce their AWS bill. And yet, they never seem to try anything else. Doomed to the cyclical cost savings cycle. Amazon knows this and the UI/UX is designed to keep companies in this money burning loop.
Finally, AWS isn't a silver bullet. For anyone in us-east-1, you know [0].
[0] https://mashable.com/article/amazon-web-services-outage-may-...
I probably should have commented on the original article here, but I pulled all of my company's production infra out of that AZ back in 2019 because AWS dragged its feet for too long deploying 5th gen hardware there.
I assumed the racks were full or something. I still don't know if they ever did get newer hardware in that AZ—I just avoid it like the plague.
I had a light chuckle this week when I discovered the work I did out of sheer frustration saved us from a partial outage seven years later.
On Google cloud compute, the ui shows an updated 'cost' as you start building your machine.
No thank you
I’ve never felt surprised by pricing. Cost has been surprising, but that happens when usage is surprising in my experience.
They absolutely could to you a base price on the ec2 setup page, but they don't. And I have been absolutely surprised by pricing. Services that do almost nothing could cost more than your ec2.
I've been working with AWS for nearly 10 years. Many people I know, both small and large, just don't even use the console. If I need to figure out how much a project costs I use the AWS pricing calculator. Having an ec2 pricing on the pricing page is meaningless once you spend any meaningful amount of time in AWS. Once you add discounts and reserved instances, that number is going to be inaccurate anyways.
If you just need a VPS provider, there are better, less complex options. I find these complaints kind of like stepping into an F1 car and complaining that the F1 car is deceiving you because theres no fuel gauge.
> If you just need a VPS provider, there are better, less complex options. I find these complaints kind of like stepping into an F1 car and complaining that the F1 car is deceiving you because theres no fuel gauge
That's fine if you feel that way. The article and following discussion is clearly about the smaller audience, and I think you're underestimating how far up these little problems stack and scale. If a couple grand is a rounding error to you, that's great. Most businesses fall firmly in the place where that would be a problem.
I think there is a value add for large companies on AWS, but for smaller ones, I don't particularly feel like AWS is an F1 car, more like a self driving Tesla that locks you inside when it's on fire. And I find the cavalier attitude that these companies aren't important enough to add the distinction to be exhausting. AWS is being pushed on everyone.
The complexity of AWS is because a service like AWS is complex. Neither Azure or GCP has any less complexity. DigitalOcean offers way less services and as a result is way less complex.
>And I find the cavalier attitude that these companies aren't important enough to add the distinction to be exhausting
They aren't important in the same way a F1 car doesn't think families are important enough to add a back row seat. No company is going to have fidelity to serve a perfect product to every market. The frustration comes from the misplaced belief that a product should serve every kind of user in the market.
I don't know of anyone saying you should buy an F1 car for your family, do you?
I do see people in this very thread with very different ideas of when AWS makes sense for you.
It's a metaphor. Your clients telling you they need you to deploy on AWS are the kind of people I believe are telling you to buy an F1 car to daily drive to whole foods. You said it yourself: "AWS is being pushed on everyone".
>I do see people in this very thread with very different ideas of when AWS makes sense for you.
Naturally. However, 99% of, what I believe are illegitimate complaints of AWS (AWS has tons of legitimate complaints), are from people who were probably better served by a using a simple VPS provider than a cloud provider. A VPS provider is simpler, easier to understand wrt to pricing, and cheaper. Most of the complexity in AWS comes from the fact that AWS itself is a very complex tool targeted to large organizations and deployments where people aren't using EC2 instances, or are using 100s of them. The complaint that the UI doesn't have enough affordances when trying to create a single EC2 instance is kind of ridiculous when you consider it's a tool designed for people launching 100s of instances. Nobody is reasonably launching 100 instances through the dashboard. Furthermore, if vendor lock-in is a concern you have AWS is the wrong tool.
Likewise for IAM. People complain a lot about IAM. But AWS has thousand different user types, and a 1000 different services. I've written my fair share of permission systems with a fraction of the amount of permutations. They always become complex due to the combinatorial nature. GCP manages to somehow be even worse. But you wouldn't need to deal with something like IAM if you just stuck with a VPS.
For instance, I don't see any pricing information when setting up an FSx filesystem, even for the size you setup. And there's definitely nothing saying backups will cost you more than storage (even though they are incremental?)
I spent 5 years optimizing spendings on AWS at various companies. Yes, it does come with traps and footguns. On the other hand if you know what are you doing, there are plenty of tools to optimize your spendings with RIs, saving plans, auto-scaling, etc, and spend less than the list prices.
Based on my experience AWS for the companies that can afford to pay surprise bills out of pocket if something goes wrong.
I agree with him/her, just shared my more nuanced take, based on my experience coming from my past workplaces.
if you see the signs of an abusive relationship, you have the option to walk out, and you don't, all that follows is your own fault.
Ignores nothing, and blames no victim.
It advises people to avoid becoming one when possible.
Thing about abusive relationships is, though, many (I would go so far as to say "most" but I'm no expert on the numbers) people in one have lots of options to walk out... but they either don't know they can walk out, or they don't feel that they can.
So telling them it's their own fault for leaving, when they didn't really understand that leaving was an option, does blame them unfairly.
Now, when the analogy is employee-employer, the "don't feel that they can" so often doesn't apply: the psychological reason for not leaving ("but I love him!") is almost never something the employee feels. But the "I feel trapped" reason (it's the only job I could find that makes nearly the money I need for my mortgage, if I leave then we might lose the house, etc.) VERY often applies.
EDIT to add this P.S.: I understand the intent of saying that was to advise people "Hey, walk away when you get the chance, otherwise everything that happens to you was 100% avoidable". But saying "it's your fault" is going too far. I've seen people claim that statements purely intended as advice (like "Hey, if you park your car in THAT neighborhood, you might wanna lock your doors and not leave any valuables in sight so nobody smashes your window") are victim-blaming. But it's really, really about the phrasing. The example I gave was definitely NOT victim blaming. Saying "Well, you were asking for it by parking your car there" WOULD be victim-blaming. The way it's phrased is very important. And saying "all that follows is your fault" is most definitely wrongly blaming the victim.
If I know a dog is dangerous, but try to touch it anyway and get bitten - then yes the evil dog bit me, but it was my fault for not reacting to danger. Same way with a abusive company, if you know they are, but still make a contract because it seems convenient, then it is still a abusive company, but your fault for getting into a relationship with them.
At some stage, (regardless of law or what’s right), standing in a pedestrian-crossing on a busy thoroughfare is foolish.
Keep hanging out in the crosswalk hoping Bezos will stop for you,
if you want,
but don’t chastise those warning others to move.
But yes, people in abusive relationships (whether in their personal or professional lives) should be advised to get out of there, and should be helped to do so as best as you can. No qualms with that.