upvote
One of the biggest problems with the self-hosted situations I’ve seen is when the senior engineers who set it up leave and the next generation has to figure out how to run it all.

In theory with perfect documentation they’d have a good head start to learn it, but there is always a lot of unwritten knowledge involved in managing an inherited setup.

With AWS the knowledge is at least transferable and you can find people who have worked with that exact thing before.

Engineers also leave for a lot of reasons. Even highly paid engineers go off and retire, change to a job for more novelty, or decide to try starting their own business.

reply
>With AWS the knowledge is at least transferable

unfortunately it lot of things in AWS that also could be messed up so it might be really hard to research what is going on. For example, you could have hundreds of Lambdas running without any idea where original sources and how they connected to each-other, or complex VPCs network routing where some rules and security groups shared randomly between services so if you do small change it could lead to completely difference service to degrade (like you were hired to help with service X but after you changes some service Y went down and you even not aware that it existed)

reply
Not much different from how it worked in companies I used to work for. Except the situation was even worse as we had no api or UI to probe for information.
reply
There are many great developers who are not also SREs. Building and operating/maintaining have their different mindsets.
reply
The end result of all this is that the percentage of people who know how to implement systems without AWS/Azure will be a single digit. From that point on, this will be the only "economic" way, it doesn't matter what the prices are.
reply
That's not a factual statement over reality, but more of a normative judgement to justify resignation. Yes, professionals that know how to actually do these things are not abundantly available, but available enough to achieve the transition. The talent exists and is absolutely passionate about software freedom and hence highly intrinsically motivated to work on it. The only thing that is lacking so far is the demand and the talent available will skyrocket, when the market starts demanding it.
reply
They actually are abundantly available and many are looking for work. The volume of "enterprise IT" sysadmin labor dwarfs that of the population of "big tech" employees and cloud architects.
reply
I've worked with many "enterprise IT" sysadmins (in healthcare, specifically). Some are very proficient generalists, but most (in my experience) are fluent in only their specific platforms, no different than the typical AWS engineer.
reply
Perhaps we need bootcamps for on prem stacks if we are concerned about a skills gap. This is no different imho from the trades skills shortage many developed countries face. The muscle must be flexed. Otherwise, you will be held captive by a provider "who does it all for you".

"Today, we are going to calculate the power requirements for this rack, rack the equipment, wire power and network up, and learn how to use PXE and iLO to get from zero to operational."

reply
This might be my own ego talking (I see myself as a generalist), but IMHO what we need are people that are comfortable jumping into unfamiliar systems and learning on-the-fly, applying their existing knowledge to new domains (while recognizing the assumptions their existing knowledge is causing them to make). That seems much harder to teach, especially in a boot camp format.
reply
As a very curious autodidact, I strongly agree, but this talent is rare and can punch it's own ticket (broadly speaking). These people innovate and build systems for others to maintain, in my experience. But, to your point, we should figure out the sorting hat for folks who want to radically own these on prem systems [1] if they are needed.

[1] https://xkcd.com/705/

reply
Yeah, anyone who has >10 years experience with servers/backend dev has almost certainly managed dedicated infra.
reply
> and the talent available will skyrocket, when the market starts demanding it.

Part of what clouds are selling is experience. A "cloud admin" bootcamp graduate can be a useful "cloud engineer", but it takes some serious years of experience to become a talented on prem sre. So it becomes an ouroboros: moving towards clouds makes it easier to move to the clouds.

reply
> A "cloud admin" bootcamp graduate can be a useful "cloud engineer"

That is not true. It takes a lot more than a bootcamp to be useful in this space, unless your definition is to copy-paste some CDK without knowing what it does.

reply
> A "cloud admin" bootcamp graduate can be a useful "cloud engineer",

If by useful you mean "useful at generating revenue for AWS or GCP" then sure, I agree.

These certificates and bootcamps are roughly equivalent to the Cisco CCNA certificate and training courses back in the 90's. That certificate existed to sell more Cisco gear - and Cisco outright admitted this at the time.

reply
> The only thing that is lacking so far is the demand and the talent available will skyrocket, when the market starts demanding it.

But will the market demand it? AWS just continues to grow.

reply
Only time will tell. It depends on when someone with a MBA starts asking questions about cloud spending and runs the real numbers. People promoting self hosting often are not counting all the cost of self hosting (AWS has people working 24x7 so that if something fails someone is there to take action)
reply
> AWS has people working 24x7 so that if something fails someone is there to take action..

The number of things that these 24x7 people from AWS will cover for you is small. If your application craps out for any number of reasons that doesn't have anything to do with AWS, that is on you. If your app needs to run 24x7 and it is critical, then you need your own 24x7 person anyway.

reply
All the hardware and network issues are on them. I agree that you still need your own people to support you applications, but that is only part of the problem.
reply
I've got thousands of devices over hundreds of sites in dozens of countries. The number of hardware failures are a tiny number, and certainly don't need 24/7 response

Meanwhile AWS breaks once or twice a year.

reply
From what I've seen, if you're depending on AWS, if something fails you too need someone 24x7 so that you can take action as well. Sometimes magic happens and systems recover after aws restarts their DNS, but usually the combination of event causes the application to get into an unrecoverable state that you need manual action. It doesn't always happen but you need someone to be there if it ever happens. Or bare minimum you need to evaluate if the underlying issue is really caused by AWS or something else has to be done on top of waiting for them to fix.
reply
How many problems is AWS able to handle for you that you are never aware of though?
reply
Distributed systems can partly fail in many subtly different ways, and you almost never notice it because there are people on-call taking care of them.
reply
How many problems do you think there are?

I've only had one outage I could attribute to running on-prem, meanwhile it's a bit of a joke with the non-IT staff in the office that when "The Internet" (i.e. Cloudflare, Amazon) goes down with news reports etc our own services are all running fine.

reply
It’s all anecdotal but in my experiences it’s usually opposite. Bored senior engineer wants to use something new and picks a AWS bespoke service for a new project.

I am sure it happens a multitude of ways but I have never seen the case you are describing.

reply
I’ve seen your case more than the ransom scenario too. But also even more often: early-to-mid-career dev saw a cloud pattern trending online, heard it was a new “best practice,” and so needed to find a way to move their company to using it.
reply
Is that what I should be doing? I'm just encouraging the devs on my team to read designing data intensive apps and setting up time for group discussions. Aside from coding and meetings that is.
reply
> 7) Paired with some "certified AWS partner"

What do you think RedHat support contracts are? This situation exists in every technology stack in existence.

reply
> 3) New engineer comes in and panics

> 4) Ends up using a "managed service" to relieve the panic

It's not as though this is unique to cloud.

I've seen multiple managers come in and introduce some SaaS because it fills a gap in their own understanding and abilities. Then when they leave, everyone stops using it and the account is cancelled.

The difference with cloud is that it tends to be more central to the operation, so can't just be canceled when an advocate leaves.

reply
> One of AWS's favorite situations

I'll give you an alternative scenario, which IME is more realistic.

I'm a software developer, and I've worked at several companies, big and small and in-between, with poor to abysmal IT/operations. I've introduced and/or advocated cloud at all of them.

The idea that it's "more expensive" is nonsense in these situations. Calculate the cost of the IT/operations incompetence, and the cost of the slowness of getting anything done, and cloud is cheap.

Extremely cheap.

Not only that, it can increase shipping velocity, and enable all kinds of important capabilities that the business otherwise just wouldn't have, or would struggle to implement.

Much of the "cloud so expensive" crowd are just engineers too narrowly focused on a small part of the picture, or in denial about their ability to compete with the competence of cloud providers.

reply
> Much of the "cloud so expensive" crowd are just engineers too narrowly focused on a small part of the picture, or in denial about their ability to compete with the competence of cloud providers

This has been my experience as well. There are legitimate points of criticism but every time I’ve seen someone try to make that argument it’s been comparing significantly different levels of service (e.g. a storage comparison equating S3 with tape) or leaving out entire categories of cost like the time someone tried to say their bare metal costs for a two server database cluster was comparable to RDS despite not even having things like power or backups.

reply