Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?
The teams are running automated end-to-end tests against this environment. On top, pre-sales uses it to build demo and sample environments, sales shows demos on it, solution engineers use it to test/try out the important parts of projects.
That generates a good production-like experience for the team: You have customers using the system, and possibly screaming if you break it. And it has been pretty successful at catching problems missed or integration failures.
You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).
Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a single machine? Everywhere that I've seen it done, it's a horrible kludge of scripts and hacks that's almost as difficult to understand as the code under test.
I write a lot of Terraform to manage internal infrastructure with dependencies on other internal and external infrastructure that's outside of my control. I define a module parameter for each dependency (usually with a default value that points to the real service URL) and then create a second module that uses Docker to spin up and configure local instances of each dependency and overrides the corresponding parameter.
This makes local testing very straightforward: `terraform -chdir=tests apply`
Way back when, you'd see a lot of people stuff their jenkins in their dev account. There's no way that the system that builds, publishes and deploys your application should be treated as anything but as sensitive as the production system itself.
( To be fair there are mitigations such as reproducible builds, but you're now doing a bunch of engineering to tie in your deploy system. )