upvote
It's much easier to protect yourself if you have your infra behind a perimeter firewall you control.
reply
Until there's a exploit for Palo Alto, Juniper, Cisco, F5, <pick your defender>.
reply
You can scratch Cisco from the list - they just announced they will re-vibecode all of their codebase! So there will not be any explots for their hardware in the future - one way or the other. ;-)
reply
This solving the problem once and for all.
reply
My hate for hyperscalers is greater than my hate of being SRE.

It's time to leave GCP, AWS, and the rest.

Google and Amazon markup can eat it. Fuck those guys.

Let's not forget that Google and Amazon tried to offshore us, played a role in inventing the labor-displacing models in the first place, and that don't give a shit about any of us.

I hope they ossify and turn into the next IBMs.

They thought AI would lead to growth and a greater concentration of wealth for themselves, but it's actually the seeds of their own destruction and dismantlement.

reply
Only tangentially related, but recently when I was working on a feature using iCloud, I noticed that it's just using Google storage APIs. I read Apple uses AWS S3 too.

Supposedly Apple holds the encryption keys on their own infra, but I still found it interesting how they are just reselling Google Cloud storage here.

reply
Or there's some exploit in an endpoint device, and the problem moves laterally from inside your firewall.
reply
Yep, but the attack surface is much smaller!
reply
yeah remove the door locks, the thief can just break the window
reply
It's also harder to explain to your customers why you're the only product they use that has a security problem when an exploit does eventually happen.

You see this already with uptime. If Github or AWS has a major outage, it's in the news and even people that don't understand software realize it's just "something happening today, everything is down".

And, unfortunately, in practice customers don't care about this inverse, when "everyone else is down but not that one product".

reply
Good like trying to explain to your customers that their data is no more accessible because some api key ended up in a public git repo or since you admin panel is exposed to whole internet someone was able to log in.
reply
With the speed of exploitation hitting right now your analysis will be out of date in a few weeks to months. We are about to have very palpable fear hitting all software vendors. It will be seen as irresponsible to use centralized systems
reply
Especially when the customers' own websites or services are down, they are a lot more understanding.
reply
Until the attacker bypasses your perimeter firewall by convincing an employee to download and install malware.
reply
And so? This is true for any kind of infra!

You can't defend yourself from everything, but you can try to make you life easier by reducing the attack surface and make side movements more difficult.

reply
Having employees (or system like bastion hosts) with unrestricted access is one of the recurring mistakes in such events. I can’t understand it.
reply
Zero Trust is expensive to implement and the business people hear "at the end, the systems will ideally work as good as before".

It will take a few bankruptcies until security is taken seriously.

reply
[dead]
reply
No self-service sign up for your app. No public access to the app itself. Website that let's you submit a sign up request form. Team reaches out and reviews your request, creates an account in the app dash for you, and you set up mTLS to get access to the dash.
reply
In-person requests for access to any app. Request interview includes word association exercise that monitors your emotional reaction to a string of key words.
reply
Security question 2: You reach down and flip the tortoise onto its back…
reply
You joke, but the space I work in actually does do video calls (or in our case in-person meetings if they're local to SF, often) before onboarding new customers.
reply
Having them hold 3 fingers up in front of their face?
reply
LLM discovers exploit -- you don't realize it in time because you don't have a security team as big as the PaaS you used to use -- you get pwned anyway.

This has happened to many high-profile self-hosted pieces of software just this past month.

reply
An exploit in what, the network stack of the Linux distro you're using for the mTLS proxy? Or the crypto module? Sure, could happen. But then, you're unlikely to be the top target of anything.

Or do you mean a customer was running it?

reply
There have been a lot of exploits for a lot of software on-prem tech stacks rely on just over the past month - proxmox, php, nodejs, palo alto, eMudhra, etc come to mind.

> But then, you're unlikely to be the top target of anything.

That would have protected you 3 years ago when a human had to bother to attack you. Today it doesn't protect you because it's trivial to deploy an LLM to act like an APT against any random target that shows up in shodan.

It takes more effort to secure on-prem software today than it did just a couple of years ago. Updates are more frequent, zero days are more frequent, attacks are more common, more persistent, and more automated.

I think most people responsible for securing production systems today will share the experience that their workload has increased.

reply
You're still describing an environment that presents a pre-auth surface area beyond a listening port.

And I am literally a security engineer; the work has absolutely not gotten harder, it's just changing in how you think about security, just like it shifted before from "secure the perimeter" to "defense-in-depth", it's now changing again to focus on automated response and compartmentalization/ segmentation (and I mean real segmentation, not the checklist-style "we have 3 vlans, but they can all talk to each other" 'segmentation' you used to see.

reply
Yeah, most organizations need to expose something to the internet as a matter of normal operations -- at least some sort of a web app if not dozens. It would be great if we could all just have udp 56820 listening silently for a key, but that isn't reasonable for most organizations that have normal users expecting normal applications.

Yes, everyone is rushing to implement zero-trust principles for real now. It's getting harder to trust that your public facing endpoints don't have vulnerabilities, and it's getting harder to trust that your internal software and endpoints haven't been compromised from within your pre-existing trust boundaries. A lot of the increased workload is due to patching up the existing security infrastructure, and on top of that, building out new solutions to better fit today's threat landscape.

reply
Ideally they need to divide those apps entirely; your public website doesn't need to talk to your app.

For stuff like Netflix that is truly consumer-facing, where mTLS or IP whitelisting are non-starters, the reality is you have to rethink your model entirely.

Also, I'm not sure where you got the idea I'm talking about on-prem only; this also applies to using IaaS/ cloud hosted infra.

reply
The reason we were talking about on-prem is because we're talking about the effort of doing these things yourself vs farming it out to a third party.

If you're a security engineer responsible for securing only:

1. a public website

2. stuff that can be ip whitelisted and/or use mTLS

Consider yourself very lucky. Most orgs need to expose a lot more than that.

reply
Almost no companies farm their security out to third parties. The era of MSSPs is long dead, except for small non-IT businesses like mom-and-pop law firms.

> Most orgs need to expose a lot more than that.

No, in fact there are only 2 kinds of connections:

1) public, necessarily allows anonymous/unauthenticated connections

2) non-public, doesn't need to allow unauthenticated connections

Name a service, and I can tell you which one it is.

Or put another way, there are authenticated and unauthenticated services. If you can authenticate someone at the application level, you could do it at the network level.

Most places have combined them at some boundary, for ease of access, but there is no case in which an authenticated application endpoint (even /login) has to allow unauthenticated network access to it.

And in all seriousness, name a service that is not a public website, but cannot use mTLS or IP allowlisting.

SFDC? mTLS. Customer webhooks? IP whitelisting. Data warehouse? PtP VPN.

reply
> Almost no companies farm their security out to third parties.

They do it all the time, piecemeal, that's part of what anything-as-a-service delivers. If you don't want to secure an app, buy a SaaS one. Don't want to secure infrastructure?, buy an IaaS.

> Most places have combined them at some boundary, for ease of access, but there is no case in which an authenticated application endpoint (even /login) has to allow unauthenticated network access to it.

> And in all seriousness, name a service that is not a public website, but cannot use mTLS or IP allowlisting.

Literally any SaaS software where the customer does not or cannot practically install a certificate for you or provide a list of IPs in advance? mTLS is almost never used in a browser unless you are connecting to a government website with a pre-shared smartcard. IP whitelisting is great if the end users only work from behind a VPN or in an office. But there's a whole big world of people who cannot or will not do that.

reply
And that is going to have to change.

There are already plenty of methods that are low-impact to users, like certificate-pinning in apps, and all you have to do to make it per-user is roll in their user-specific key JIT when they download the app.

Users will not be happy (just as no one is happy Reddit is going app-only), but they will do it, especially when it becomes common as companies decide they don't want to be the 5th breach notification on HN in a given day.

Today we have Bun, Cursor, Northflank, and now Linode!

reply
Or the configuration parameter that you got wrong because you rolled your own and didn't know any better.
reply
Why would you roll your own?
reply
You would inherently be rolling your own configuration when deploying something on-prem.
reply
1) where did you get "on-prem" from?

2) that's not what "rolling your own" means. Building your own web proxy is "rolling your own". Standing up e.g. Caddy and configuring it is not.

reply
1. that's what this thread is about?:

> I think people will move back to their own hardware

> self-hosted pieces of software

2. arguably -- but lets set aside what you think the word means and respond to the obvious meaning of the comment you're replying to.

The point is -- doing shit yourself makes it your problem to do correctly and requires sufficient resource allocation

reply
1. That's fair, I didn't realize people were expecting my comment to be perfectly-aligned to every aspect of the thread parent, including the "their own hardware" part, but now I at least see where the confusion came from. For the record, any issues at this level of the stack are also present in IaaS, not just on-prem.

2. I still don't see where you got that they are only talking about configurations versus messing up an implementation (I.e. in a paramater they coded incorrectly)...

But that aside, even on AWS you're still configuring your proxies, gateways, load balancers, etc. AWS gives you just as much rope to hang yourself with as a box in a colo network cage does. Companies are already doing this stuff themselves, on-prem is not more security work. IaaS is about scaling and maintenance load, not security being handled for you.

reply
For one, I have strict iptable rules to drop inbound from outside my specific commercial subnet and my specific residential IP. After that, all the usual best practices.
reply
Breaking a lock still costs approximately the same, whether the lock is protecting a small barn or a huge hangar. A large number of small but reasonably tough (not unbreakable) targets makes the payoff per target smaller. This necessitates target selection, and chances of being deliberately skipped grow for an average target.
reply
Everyone knows Fort Knox has gold, and where it is, that’s railway, heroku. No one knows where my personal safe is or whether it has anything worth taking. That’s a vps in a colo
reply
However, the large hangar is likely to have armed guards inside, or at least able to arrive on the scene within two minutes of the alarm tripping. Your regular household with their regular locks on their regular doors does not.
reply