You should have a method to deploy hotfixes to code that bypasses your development pipeline though, in case the development pipeline breaks while you need to fix production- because you must be able to fix PRODUCTION even if your dev pipeline is broken ( because again your dev pipeline, while important, is not production)
To the product developers and operators, customer-facing systems are production.
To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and start screaming.
Within the infra-ops team, our config management tests and the deployment pipelines are production. If those don't work, infra operators cannot test or roll out changes to the infrastructure.
This was recently discovered by a dev team providing a cross-cutting service: Their testing environment can halt work for a lot of other teams, so they have to be really careful with their testing environment.
Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?
The teams are running automated end-to-end tests against this environment. On top, pre-sales uses it to build demo and sample environments, sales shows demos on it, solution engineers use it to test/try out the important parts of projects.
That generates a good production-like experience for the team: You have customers using the system, and possibly screaming if you break it. And it has been pretty successful at catching problems missed or integration failures.
You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).
Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a single machine? Everywhere that I've seen it done, it's a horrible kludge of scripts and hacks that's almost as difficult to understand as the code under test.
Way back when, you'd see a lot of people stuff their jenkins in their dev account. There's no way that the system that builds, publishes and deploys your application should be treated as anything but as sensitive as the production system itself.
( To be fair there are mitigations such as reproducible builds, but you're now doing a bunch of engineering to tie in your deploy system. )
Agreed that many parts of the development pipeline can be hit-or-miss. At scale, it's great to have a dedicated "Developer Experience/Tools" org, though I've seen mixed results even with one.
In my experience, we treat "not being able to deploy to prod" as an warning light, because our ability to fix an actual issue or outage in prod is impacted.
i.e. if there is a prod issue, then we can often ether fix it in code, or roll back to an earlier release. But we need the deploy pipeline to be working in order to do that. The longer a pipeline outage persists over time and across multiple services, the more likely it is that there will be some impact.
It's like driving without a seatbelt - you're fine so long as nothing else goes wrong, so it's not recommended as a state to stay in.
Genuine question, does anyone here ITT working in software still have dedicated QA? They laid off all our QA engineers about a year ago, and talking to friends and former colleagues it seems to be the industry wide trend?
FWW I think a good QA person is worth their weight in gold and this has been a horrible mistake. I'm just curious if there are any left.
Those things can't be tested with Playwright et al. So the team had a "every pull request gets an instance like pr143.company.org" pipeline where QA approved. They must still have it.
The other companies tested "on demand."
A few years ago I worked at a company who had a team of QAs dedicated to do manual acceptance tests. Up until the day where a critical regression went unnoticed and it turned out QAs were going through the test suite without actually checking the behavior. Following that the company got rid of all QAs except the software engineer in testing who was responsible for the whole automated test infrastructure, and instead of QAs the team held test days to go through the same manual test suite while we progressively automated them away.
And the whole process improved all across the board.
I also don't find most QA to be worthwhile compared to a good, quality focused dev.
There are several cultural issues with how QA are treated (low pay) and how they think they're expected to behave (e.g. hands off the code, hands off types, hands off unit tests) which make them less useful than a dev who is really good at engineering for reliability.
Where we had good engineering practices and that type of "black box" QA I noticed that product tended to find the majority of bugs, followed by devs, followed by customers, followed by QA.
Only when engineering practices sucked and they were used as a crutch by devs did they really catch a lot of bugs.
Of course, in these days of vibe coded madness they're probably getting more useful, not less.
A buggy release is fine, if it came out faster, and costed less. The bugs in the release can be fixed in the future, and the "heroic" fixes are "creditworthy".
Therefore, over time, the QA team is politically relegated as a "useless" team in the department, and thus given how politics play out, they get sidelined until the entire team is gone.
Deploy was an rsync of a few directories to a server. That server directory also held .env.local, and a content/ dir that a scheduled job writes to directly. Neither is in git. Someone (me) added --delete so the destination would "match source", and rsync faithfully removed everything outside the source set: the env file, the five .bak copies sitting next to it, and a week of generated content.
Two accidents kept it from being worse. The rsync excluded the build output, so the already-running process kept serving; and because the app had read its secrets into memory at boot, checkout kept working off a file that no longer existed on disk. That bought a few hours to restore from git.
What I'd add to the article: treating the pipeline as production should include auditing the pipeline's write scope. Mine had unrestricted write+delete over a directory holding live credentials, and that was invisible until it mattered. Also — backups in the same directory as the original are not backups.
I think for dev, it’s fine not to have blue/green. You want dev deployments to be fast. UAT / pre-production/ or whatever your final env before prod — and particularly if stakeholders are intended to test on that env — is a different matter.
The premise is that Software Development is akin to Production which it should be.
And yes, your dev environments should mirror the production environments bar production data. That’s software development 101.
Within your software development environment (which should be treated as prod to maintain velocity), you have blue/green when doing deployments, established SLAs, backups and DR.
This isn’t only for your (deployed) Production environment.
It’s also not software development 101. This has roots in ITIL.
EDIT. In particular the author hasn't acknowledged the prioritization problem, hasn't explained what factors might go into the necessary cost benefit analysis. The author merely points out "developer productivity is important!" You might as well say "invest in your tools!"
In some places shipping quickly is part of the value. In others, the product works and delivers value and having slow releases (implying eg a manual release process) is a feature.
What this article advocates for is certainly a valid lens, by IMO it’d be a mistake to take it as universal.
Existence of recruiters and recruitment companies as separate job entities is a proof positive that you do not, in fact, need a working recruitment pipeline.
If anything, recruitment in our industry was always broken. And yet here we are.
There is a production eco-system - let’s call it The Hotel. It runs, people work in it and customers pay and it makes money.
But it constantly needs changing. New rooms need to be built, the kitchen rearranged as the chef changes his mind.
So they hire an architect and building firm. Who plan the chnages and then this awesome automated robot comes and builds the extra room.
The architect firm owns and is responsible for the robot and its changes.
Not being able to make a change really really matters, but I don’t call that production because the hotel is still able to operate
Most of the misunderstanding between business and tech is a misunderstanding of are you working on the architect firm or the hotel
Of course I know the importance of preventing problems, but how do you properly measure it? A data leak is immediately visible, but preventing them is less so.
Shameless plug: To fix all those development pipeline issues, I am working on StableBuild, a tool to easily freeze and pin Docker images, operating system packages, Python packages, and arbitrary build dependencies; in 5 lines of code: https://stablebuild.com .
To be honest, they shouldn't do so directly. For the obvious reason you stated, they don't have a SLA to you or your company. But also because it is a shitty thing to do as a company. Companies should have their own proxies and mirrors for any repository they pull from. Both to be in control of the dependencies you as a company rely on and to not unnecessarily put load on free third party repositories.
Had not heard of your solution. More often than not I come across Harbor, Nexus or things like GCPs internal stuff.
Maybe I'm out of touch, but I think control over dependencies is underrated. It's not just about freezing and pinning. There should be better tooling to support the whole dependency evolution pipeline. For example updates should be reviewed/ingested/integrated/validated based on local policy not based on source release schedule.
The problem is, there's a difference between doing that for a handful of C or Go libraries, versus trying to vendor thousands of NPM libraries and all of their interdependencies. So it's very ecosystem dependent.
You should never fix an incident by deploying another fix as it risks making things worse especially while everyone is panicking. Better to roll back - and spread non-backwards compatible work across multiple deploys that each are backwards compatible - always giving yourself a known good state to roll back to.
Rollback should not done be via your CI either! Set up some low dependency thing - a separate tool or script or system.
Once you've rolled back and got the system stable, then produced a fix at your leisure, that's when CI comes back into the picture :)
Acknowledging that developers are in fact customers, is often where it comes undone in my experience.
In a lot of places, only revenue paying people are considered customers, everyone else is a cost centre.
Dev tools are often seen as a tax the business pays to be able to generate revenue. Investing in them, or treating them like the production level systems they are, isn’t intuitive for most, and for some a hard cost to justify.
On the other hand if the email server is down that's just as big a problem for any biz as if customer facing product was not working so some workflow things are already treated as production tier priority
There’s the system you are building…and the system that builds the system or the SDLC. Both are critical.
Solid read.
You too can enjoy being paged at 2am for a flaky disk, a cron job related traffic spike, and AWS outage, or any of the wonderful things pagers get paged on.
And you can re-read your essay after being awakened repeatedly for false alerts, knowing that non-prod alerts are now first class citizens and you are their Shepherd.
Tldr? Go ahead and page yourself for non-prod alerts. No one will stop you.
zZzzZ...
2:35 AM
ALERT #42 - CRITICAL
!!!
ALERT #42 - CRITICAL - Joan forgot feir password to npm
Help feir!
> A team with a broken development pipeline can’t produce software, and must treat this as a production outage.
While I sympathize, I can't go that far. An "outage" means user- or stakeholder-facing.
To the extent that a developer is a stakeholder (surprisingly common) you might be right, and they would prioritize such processes.
Increasingly though founders and executive team are non-technical and simply don't care until it shows up on a graph that piques their interest (the ones with $ somewhere).
It's trendy to act like devs are fungible resources or that AI will replace them, but the truth is that hiring smart people, paying them lots of money, and then forcing them to spend half their time dealing with prod issues that they're not empowered to fix is a staggering waste of resources.
Since deployment or testing issues are mostly invisible to upper management (who never have to spend half of THEIR time fixing them) this doesn't come with any recognition or promotions, so the only thing left to absorb the impact is morale and goodwill towards the company.
Sadly, engineer-driven companies are increasingly rare.