If the dev is pushing code to production they should not simultaneously be pushing environment configs, this is doing two logically distinct things at once: Changing application behavior AND reconfiguring the server environment.

If the dev is adding secrets to their local config and they’re pushing that config to insecure places that means their deployment pipeline is broken and it should be fixed. .env is never committed to source for this reason, for example.

I was a massive fan of dotenv but sOps is so much cleaner, easy enough to use and works well enough in k8s.

It's got a maximum size cap, is trivially introspectable by via any process that can read `/proc`, and sucks at representing hierarchical or structured data beyond k=v.

The proliferation of tools that come up with all sorts of contortions to encode e.g. JSON-ish structures into the environment is evidence that this ain't a great way to go. I hope we're moving towards a container-orchestrator-by-default future; mounting structured data into pseudo-files at runtime is a really nice alternative.

Times have changed since then, and there’s much better tooling available to help with this problem space and surface area these days.

Teach them to use dotenv.

We are moving away from configuration in config files because it is a pain to modify, especially if part of that configuration is secrets. You have to throw everything into your secrets vault of preference, and editing it requires extracting and reuploading the whole thing.

We are currently doing config in env by loading one or multiple secrets per kubernetes pod (mix and match).

What would be your suggestion?

1. Keep secrets in a dedicated secrets store.

2. Read directly from the secrets store in application code. There is no environment, there are no environment variables. Yes, even on local.

Buy-into storing credentials into environment variables.

Then, and this is important - MANAGE YOUR ENVIRONMENTS.

You shouldn't have prod level s3, or aws creds accessible openly in your environment. If someone can steal those values, they can steal the code, and pretty much everything else. This is very bad.

For prod (and possibly staging), use a lib that loads in values securely from an actual secrets service.

For cloud services, it would typically be called a vault. But it could also be a hardware security module (HSM) with bring-your-own-key (BYOK, eg for certificates.)

Ansible also calls it a vault and encrypts it with a password — that file you can check into version control.

The point is that you must keep secrets, and anything environment-specific, out of the code. Follow the spirit of the law, not the letter.

The environment is a standard, locate-able, read-only at runtime k/v store in every process. That makes it an incredibly juicy target for exploits. There are tons of remote exploits well short of RCE which can access all or part of a server process's environment. If that environment contains secrets for everything that process might do, that's asking for trouble.

Consider a user-facing webserver with a rarely-used, admin-only route that talks to AWS APIs. Unless it's deployed on AWS and using IMDS, the 12-factor best practices say there should be AWS credentials in its environment.

Consider a service which, at startup, opens a connection to a telemetry/logging system, then drops privileges and handles requests. 12-factor best practices say there should be a secret for that telemetry system in its environment.

Additional examples abound. Most applications (even ones that aren't internet-facing web servers) use configured secrets infrequently--often only once, to open connections to external services--and not during the vast majority of requests they serve, but we put all secrets in the environment anyway.

Vaults don't automatically solve this problem either; many vaults provide secrets to applications by injecting them into process environment at start.

Good secret management at runtime should ideally be:

1. Mutable or at least delete-able. I really wish there were ways to remove environment variables after they're used (so I could say "once you have an authenticated, open socket or a refreshable auth token to $service, remove the initial login secret from memory entirely"), but absent highly complex multi-process/re-exec dances, that doesn't really exist. If, in Python, you 'del os.environ["foo"]', you haven't modified the environment segment of your program's memory.

2. Not in one common/uniform memory area or key-value API. Hell, it's slightly preferable to have secrets be stored piecemeal in regular variables in memory scattered around your code. Those are going to be slightly harder to find for malware that gets a foothold--security by obscurity, true, but the environment memory block/API is such a tempting and easy target that it buys you a bit more than a false sense of security here.

3. Ideally, stored or encrypted in memory (for secrets that have to stay in memory) such that an exploit which can read process memory doesn't get them for free. Some vaults have a host-local sidecar which provides secrets or a decryption key for them; that way, if an attacker gets memory-read without RCE they can't just exfil a memory image and figure out the decryption key later, but you don't have to be reliant on a remote networked service's uptime for all secret accesses. Even if you don't go that far, securing secrets in-memory at least gives you the option of doing zero-trust stuff based on request payloads, or even just making good-hygiene backend APIs that encode "you can only read the value for secret X if the request is for an admin route and authenticated" (which is a good idea for internet-exposed services with seldom-used risky secrets anyway, but doesn't help with parts 1 and 2 if that API is just wrapping env.get() or whatever).

What makes it bad advice?

> this had the second-order effect of leading devs to believe they could put all their local env secrets in ~/.bashrc files

You need some way to pass secrets to the app; doesn't every other way also suffer the same kind of issue?

Can you explain why do you feel MFA is painful/unsustainable? How would you fix it?

This very web-forum was a very big proponent of those politics a few years ago.

The environment was only ever good for things like GOMAXPROCS where you want a single point of truth for all processes on a machine but in a containerized world even that point is moot. Where regular config in the environment just problematic it is outright dangerous for secrets.

I hired about 50 software interns and junior in the past 5 years. None had ever heard of it before I told them.

For port binding, for example, it used to be that you'd deploy your app to the web application container, rather than bundling them together. e.g., deploying your WAR file to Tomcat, rather than building a self-executing JAR which included Tomcat. The wording is a bit odd, but I think they were trying to make the point very generic, and not specifically about the Enterprise Java world.

For the backing resources, it's a combination of point 3, config often living inside the codebase, applications just shelling out to /usr/sbin/sendmail or what have you, and applications living on the same host as the DB, such that bringing up a new application necessarily required bringing up a new DB as well. Which also made it hard it to scale horizontally.

The whole "12 Factor" thing was partly because Heroku had specific solutions for all of these, so going down this road made it much easier to then sell Heroku, and partly because they really were frustrating. I'd say that the port binding one is more targeted at, say, WebSphere, and all that came along with it, such as sharing a single heap across multiple apps, needing to talk to the WebSphere admins to change configuration, needing to use a "lite" version of WebSphere to test locally, if that was even possible, and so on.

They sound super-obvious these days, but at the time, for a lot of us, they were really nice to see.

The port binding was really a response to tomcat or modphp being modules in the webserver, as opposed to hosting their own web service internally. This was before nginx took off, and proxying to internal application ports was common.

edit:

I was wrong about the backing services, it seems it is really about treating them as configurations and being able to swap them out without making code changes.

https://12factor.net/backing-services

The same reason many older films seem cliche - because they were the first to do it, and it's accepted standard now. Heroku very much shaped how we think of "cloud applications", autoscaling, and containerization.

Drivers protested against seat belts that would save their own lives.

Times change and hindsight is 20/20. Let's just say 10 years ago I worked at a company that broke all 12 factors.

[1]: https://forcesunseen.com/blog/stop-storing-secrets-in-enviro...