Why the heck would this be necessary? We have VPNs holy crap, the entire genesis of the technology was to solve this fucking problem. If it's not literally intended for the general random public to be able to randomly visit and use themselves, it shouldn't be exposed on the public internet it should be (heavily restricted) LAN-only with remote access done via VPN. While it's even better if every single last thing is well secured at the client-level too, it goes a long way with ancient (or new, but with ancient-equivalent software design expenditure/decisions) kit to protect it at the network level.
Creating a false binary of "take it offline!" when businesses have very good reasons for networking many classes of industrial hardware seems self-defeating vs "just get the cheap bare minimum basic network protection done". I mean shit, even if somehow they've gotten suckered into freezers which demand access to a non-self-hostable manufacturer server (hey EU! here's something useful to regulate! if nothing else order manufacturers to have WireGuard end points for connection) they could at least have the firewall allow !RFC1918 solely and exclusively to that address on those ports.
For example, where does the VPN connect to? The vendor? Do you trust the vendor more than the internet. Sure. Do you trust the vendor, that does not manage the firmware of their devices and uses inbound connections in the first place, enough to put them on a VPN?
How many devices and vendors do you have? How many VPNs can you establish with the existing networking hardware, and how many can you manage without IP clashes etc. IP clashes work both ways, btw. The vendor has to manage this themselves also if all their clients want VPNs.
Retailers are enterprises with 100s of vendors that run on a shoestring. On top, they use a lot of legacy architecture and hardware. They have teams that have a hard time making an architectural decision in the best of times, but even less so under stress. It is probably the same on the vendors side.
In my mind, the correct solution is indeed to deny any inbound connection to the network, now, to prevent further damage.
Incident response is, in order:
1. preparation (failed)
2. identification (done)
3. containment <--- we are here (take offline, put manual rota in place to manage fridge defrost cycles)
4. eradication (VPN, etc)
5. recovery (take it back online, relieve manual rota)
6. postmortem etcThat's a really strong argument for why systems like this need layered security and should not accept incoming connections from the internet at large.
However, we're talking about companies that put unprotected systems on the public network and did nothing about it even after getting three weeks advance warning.
You can lead a horse to water, but ...
I don't know the companies in question here, I'm not European and my EU family lives in France not Belgium or Sweden, but fwiw here in the states a lot of supermarkets are independent grocers or in a sort of formal or quasi-franchise arrangement where it's part of a "chain" but each store is still somewhat or quite decentralized in management when it comes to the physical stuff at the local store. Which is probably a strength in a lot of cases, makes it easier to adapt to local environment faster, but I suspect there's just plain a knowledge/skill/priority mismatch when it comes to IT. That'd also line up with some of the reports on this I read where there was some official central management system, except certain grocery stores just weren't using it?
The components of solutions for this exist and could even be really cheap, we're not talking much bandwidth here. Not hard to imagine even in a independent grocer association or loose franchise system they could tape together an SBC with preconfigured OPNsense or VyOS or whatever on it and a switch and be able to just tell stores "here, plug your freezers into this, access the portal here" as a default and it'd just route all their traffic to the franchise/IGA's management system. Someone still has to secure that of course, but at least it wouldn't be on the open internet.
Places that wanted to could of course do this other ways, I'm just saying that even with the very tight margins grocery stores tend to run on I think it's very doable to do better going forward, just will take some coordination and knowledge building. "The future is already here, it's just not evenly distributed" and all that.
Each to their own… but why do fridges need to be connected to the internet in the first place?
- Remote alerting for the store manager/head office to quickly respond to a refrigerator failure after hours, providing options for saving valuable stock (e.g. move stock to another refrigerator).
- Maintaining an audit trail for cold chain management (e.g. regulator says you must keep an audit trail proving that a frozen fish has been kept below -30oC since it was processed on a ship all the way up to point of sale in a supermarket).
- Refrigerator manufacturer says a condition of warranty is that the refrigerator must provide remote telemetry proving some warranty condition has been met (e.g. regular cleaning cycle operated).
Note: all these speculated reasons are unidirectional data flows. So even if a valid reason exists for Internet connectivity, why can't it be limited to unidirectional data flows?
It's this, primarily, or most importantly.
I work in the restaurant industry. Health departments require you to assure them that your hot things are kept hot and your cold things are kept cold. The old way was staff walking around the store reading thermometers and writing temperatures down on paper; the newer way was your staff walking around the store inputting temperatures on an iPad connected to your business-management software. The trouble with that is that staff forget to do it (despite threats of discipline, and push notifications to their phones), or even blatantly fake it.
Failing to keep an accurate temperature log means fines, and other consequences, up to and including loss of your health license. I am 100% in favor of this legislation, and solid enforcement of it. The main improvement I would make is to harmonize codes and enforcement across municipalities. I'm in the USA, every county has their own requirements - and their own "portal" for submitting documentation. It's a massive pain in the ass to write Standard Operating Procedures for staff across multiple stores in multiple enforcement districts when there simply aren't any SOPs.
The majority of the fines our (responsible, and genuinely trying-to-do-it-right) company received were things like a staff member moving from a store in County A to a store in County B - just for a few days, to cover someone's PTO, or something - and inadvertently followed County A's logging requirements rather than County B's. Two genuine cases: logging every two hours, instead of every ninety minutes; using whole degrees, instead of tenths. We could appeal, but that takes a lawyer's time, so even if successful it wasn't always worth it.
Real-time data, from connected sensors, is unambiguously better. Staff get loads of time back, to make food, or help guests; all the logging gets done "right", for whatever jurisdiction; data's held by a third-party, so nothing can be faked.
That said, we're not connecting our infrastructure to the internet! We have a separate IoT sensor service that tracks the temps for us. I'm sure their security is also terrible, but that doesn't matter nearly as much. A year of their service is ~80% of the "routine", and totally innocent, fines and other costs that we were previously paying. That's the way all of these stores should go, in my opinion.
The big IT firms syphon off all the talent with compensation these fridge and supermarket companies can't match. You won't find much talent there. Who is brilliant with computers and would want to work there?
But look also at how underfunded the basic blocks of IT are, how open-source is left to fend for itself. It is the whole culture of modern IT to hack things together and not make them well-defined and reliable.
We've been telling people for years that naive threat modeling was going to bite everybody one day. And finally, here we are.
My org has lots of freezers that need to be at precise temperatures. Sometimes they drift so you have to check them a few times per day, log the check, and adjust as necessary. Doing this for a thousand freezers is tedious and error prone. Networking helps. “Cloud” lets it all be done remotely without getting IT very involved for vpns to be set up with specific vendors. It also helps with patching and auditing security so seems odd that these supermarket ones were out of date.
(Assuming they didn't also patch the telemetry software, but that is a much harder attack than just raising the temperature).
Compliance tends to warp the underlying goal quite a bit.
Hackers can still tamper with the telemetry and create false reports of problems, but actually changing the thermostat settings would require local access, so there's opportunity for somebody to verify the problem really exists.
If I put a Zigbee thermometer in the freezer they can hack it all they want, but I am pretty sure they will not be able to set the temperature to 50C.
Those freezers seem to be having lots of fun :-)
I don't care whether it's the Russians, Chinese, Norks or script kiddies. This should never have happened, especially since a similar incident happened in France three weeks ago so they got ample time to prepare themselves. They just don't care.
The issue is this needs to be handled centrally by the parent org also. They have IT ofc but they usually do POS and desktops/servers. The OT/SCADA systems are usually totally separate and not touched by the white collar IT systems.
It is incompetence to not have this capabilty.
The question is whether this is attack has cost them more than that would have.
Came into the thread thinking it would be a new report about the same incident not a new one, yikes.
To be fair, wouldn't the chains probably be reliant on their vendors for an update? and turning off all the fridges for weeks is probably not a sustainable solution... :/
I believe gigantic fines are appropriate. They need to be of a size where the entire management team has significant financial issues for many years to come, even after having sold their houses, big cars, etc.
Nothing will improve until there are personal consequences.