upvote
I would maybe tone that down that sentiment a bit and limit the adjustement.

You actually do want to be able to monitor them remotely, like most industrial/commercial refrigeration, to detect failures faster and more reliably. They should still be at least secured with normal auth/autz in addition to perimeter security, though.

If I can know when a fridge failed within the hour (even at night) then I can also dispatch a technician same/next day more easily.

reply
If what's needed is the ability to monitor them, then why are these things remote-controllable?!

Obviously read-only access to some sensor is not the same attack surface as having the ability to remotely shut-down the refrigeration.

reply
Same reason website's are remote controllable. There's someone who's job it is, is to make sure the fridge is up and running, 24/7 and for the same reason that website operators/SREs don't live at each POP/DC, that fridge operator wants to remotely be able to flip some relays off and on to fix the problem without having to drive onsite, if possible. Additional, a single person/fewer people can then also be responsible for multiple/more sites.
reply
That’s exactly right. There all all sorts of read-only architectures where the device can be monitored without every directly accessing it:

- The device’s control has a radio transmitter that emits its readings, but no receiver. Listen for those messages to monitor it but there’s not a physical path to reply to it.

- The device’s controller writes a database or message queue. Its code never reads from that source. Monitors listen to those messages.

And many, many similar methods with the same core concept: the item transmits its status but doesn’t have a method to receive replies. You just have to commit to using any one of those thousands of mechanisms.

reply
The fridges are cheaper to make because they don't need a bunch of knobs and switches and temperature displays - just an app on employee phones.
reply
How is a button more expensive than a computer with network access?
reply
Read-only access is mostly an illusion that itself can be hacked to gain control.
reply
I have a read only temperature sensor in my old-school non-computerized refrigerator and freezer. Hack until the cows come home but it still can't turn off my freezer.
reply
Off topic but "hack until the cows come home" put a smile on my face.
reply
Air-gapping, network segregation, data diodes, etc. are all real things. It's possible to make a read-only refrigerator that is "unhackable". I have a temperature monitor in my refrigerator right now. Hack that monitor and it doesn't affect my refrigerator at all.
reply
deleted
reply
not in this case where a thermosensor can do nothing but monitor a temperature.
reply
Yeah? We had an old Unix server that printed its logs in realtime to a dot matrix printer. How are you going to gain control of that?

Read-only is a strong control with lots of solid implementations.

reply
Depending on the make and model, send a reverse feed code and then overwrite with block characters. You will know that something bad happened, but you won't know what.
reply
That makes it inconvenient to read, not impossible. E.g. use transmitted instead of reflected light (shine a bright light behind the paper) to see how much ink is in each place.
reply
because you might not want to send a technician to every single one of them once a firmware update is needed
reply
There is value in nuance and debating whether the solution should be scaled down to say a store ( or only available on local vlan or something), but I think the value of discussing that is kinda diminishing, when, apparently, the controls for all stores can be done from the same hit.

But to address your specific point:

being able to monitor does not ( or at least should not ) automatically equal put it on the web

reply
There are most likely IT governance problems: builders bought fridges, IT didn't provision sufficient access, installers put 3G keys instead of spending time getting the correct VPN/networking, and a few months later no one knows how this thing works, and IT is not mandated to find out that shops ever did that.
reply
This still shouldn't be online. They can be networked, if you need remote monitoring, but that's different from being online (at least in my definition).

Obviously it's a question of cost and convenience, and just dumping your appliance on the internet and make it phone home is going to be the cheaper option. If you want security those appliances goes on a separate network, which cannot reach the internet.

reply
> This still shouldn't be online.

This line suggests the units themselves were targeted individually with a hack:

> Danfoss said the affected cold units were older models whose security software could not be updated to ward off a risk they had identified.

But the problem with networked devices spread across tens of thousands of pretty much unguarded locations, surrounded by unvetted employees, but taking orders form a central location is that there is no such thing as "not online". They'll talk to the management server used to set up the nightly defrost schedules across the fleet. Even if they are in an isolated VLAN, it's always one step away from someone connecting a Pi to that network and off they go online. There's only so much zero-trust, NAC/802.1x and segmentation can do when some devices will be too old to support these and their network will be the entry point to the management server, then to the whole fleet.

I'm honestly very surprised that a couple of things things haven't happened yet: the central server for managing these fleets getting hacked and allowing access to the entire fleet, and/or the companies realizing that the only way to limit the reach of an attack is to physically not have the hardware needed to do remote control.

reply
So this is how it was done 25 years ago: Only the used ports (ethernet jacks) on a network is activate, disconnecting a device and the port closes. That strictly speaking doesn't mean that there aren't way to get devices on the network, but it does make it a lot harder. You're absolutely correct about these networks being in hostile environments, but why then do we pretend that they are not? Fridges like these have never not been in a hostile environment, so why aren't basic security measures taken to secure the physical side of things?

In my world the central server can be hack in two ways, that is by a malicious operator or via some scheme like the US/Israeli attack on Irans uranium centrifuges. The central server also have no business being online, that to is on a closed network, running the whitelisted applications from the manufacturer.

reply
This is how it was done in this [0] very nice 60 years old (I'm sad now) movie. [Spoiler alert] An alarm is triggered so many times the guards just disable it. Alert fatigue has worked the same forever. Nobody wants to check the thousands of false alarms that probably happen every day across thousands of locations.

As long a schedule is conceived in one place and applied in another, even with human intervention on both sides, either one of the sides can be swindled. At this point with the bar for a successful hack being lower than ever, completely removing the hardware necessary to make remote changes is probably the only way to limit the blast radius.

Maybe allowing to send the schedule but requiring a local employee to operate a local hardware only control to apply it could work, but who has the money for this kind of complexity. Something like this on every freezer (device, really) in a fleet probably costs multiples of the money they lost with the spoiled food.

[0] https://www.imdb.com/title/tt0060522

reply
this definitely does not require a fridge be "online" and have the ability to be remotely programmed. At the very least have the monitoring and potentially I/O capabilities in a different device that can easily be removed from the system. Integrated smart fridges deserve to Suck it, Jin-yang!
reply
To be clear, they didn't monitor it 24/7, otherwise they would have noticed the fridges were grilling the pizzas.
reply
Like the other poster mentioned, blanket statements like this ignore the incredibly smart people at major international corporations and their strategies.

I have a smart fridge and washer. I won't name names. But for several years they were never connected or online at all (unless they was compromised and I didn't know). I connected them and one of the first updates included in the washer was changing the default wash cycle from something like 110 minutes to 46 or something. I don't remember all the values if I'm honest.

There are executives who have spreadsheets and charts of how much water that saved per household over varying periods of time. How much it would save with greater adoption. And the unsaid message, or maybe they'll be explicit, of how many people are going to face tragic water supply issues in the future.

It's a powerful message, even if we know it's cyncical and calculated to push not just washers, but firmware security systems and institutions to govern those systems.

And yes, compromised systems can also be used to build zombie networks to push horrible content.

I feel horrible that this critically important surface issue of water supply is used to push other agendas. I honestly wish I could do other work not related to it.

reply
<< ignore the incredibly smart people at major international corporations and their strategies.

Ignore may be a tad strong. I am merely asking for hardware that is subject to my control to be within my control. I think somewhere along the way executives got a little to comfortable saying 'they will buy what we give them'.

More to the point, maybe there is exactly too many smart people with agendas that don't seem to align well with my current needs vis-a-vis fridges:

I want something that keeps things cool with minimum of fuss and does not die upon update.

In short, that executive should be fired as they are not doing their job: they are actively undermining their own market.

reply
>Ignore may be a tad strong. I am merely asking for hardware that is subject to my control to be within my control.

Which is how they got into this situation. These are industrial refrigeration systems subject to a huge list of variables (how many units, their sizes, and doors hooked up to a refrigeration circuit, open vs closed cases, ambient conditions, hours of store operation, what type of refrigerant, whether heat is going to a cooling tower or air heat exchanger). They use a ton of energy, and percent optimization leads to millions of dollars over a fleet. They are designed to let the user dial in every operating parameter, like defrost.

Compare this to my home fridge where all these parameters are fixed in the firmware because the hardware is 100% identical.

Industrial systems are necessarily different from mass-produced home appliances.

reply
<<Industrial systems are necessarily different from mass-produced home appliances.

Honestly, reality begs to differ.

reply
Agree. I don't want fridge company or anyone else to turn my fridge into residential proxy with its access sold to the highest bidder to build web scrapers.
reply
Monitoring would be one thing, but there's almost no conceivable reason to allow setting the temperature or triggering a defrost cycle remotely. When was the last time you changed the temperature setting on your fridge or freezer? It's something you set once when you install it and then never touch again.

This kind of inspector gadget "smart tech" lunacy has infected basically all consumer products. It's got to stop.

reply
> This kind of inspector gadget "smart tech" lunacy has infected basically all consumer products. It's got to stop.

Absolutely, and this is why I refuse to connect any home appliances to the Internet. I have a limited number of Zigbee devices, and they are managed through Home Assistant and can't access the Internet.

reply
Yes, I have similar strategy. But in all cases I'd prefer to just buy analog appliances. There's no good reason to put a digital computer in a refrigerator. A bimetallic thermostat does the job just fine.
reply
> When was the last time you changed the temperature setting on your fridge or freezer?

Then again when was the last time you opened your freezer hundreds of times during one day, letting humid air in and building ice at an accelerated rate? These units actively heat up the back of the freezer (evaporator plate) nightly to quickly melt the ice without warming up the food.

There is no reason to directly trigger it remotely but they probably work on a customizable schedule, to account for different store opening times, timezones, or freezer size and model. The hackers could target that schedule to make the defrost last longer.

reply
You're right, a commercial freezer is a different beast than a household one. But what good does it really do anyone to be able to adjust that defrost schedule remotely? Surely the personnel on site are the ones best equipped to schedule the defrost cycles of their equipment?

Within a store certain freezers get more traffic than others, depending on how they're stocked, where they're located, etc, and therefore likely require more frequent defrosting. But that's something the store managers can figure out. It doesn't need to be managed remotely by some third party. So why did someone engineer that capability?

More features, generally, means more failure modes. More ways to exploit the system. Adding fringe features that aren't actually necessary (or desired) has become a strange industrial compulsion. It's very unhealthy and disordered behavior from an engineering standpoint--sacrificing reliability and simplicity for... what exactly? Some centralized control capability? But why?

reply
To be clear I don't think the defrost schedule is the way the hackers enabled this or else every device would be affected. The older devices probably connect to the SaaS server in an insecure way and could be exploited en-masse.

> Surely the personnel on site are the ones best equipped to schedule the defrost cycles of their equipment?

The purpose of the automation is the same as always, to run at peak effectiveness and reduce human error. Most of the workers in this industry much like the fast food one, are low paid, low skill, receiving only basic training, with a high turnover rate, across tens of thousands of locations from major cities to somewhere in the sticks. Even a disgruntled employee can do a lot of damage that's only obvious too late. So this extra work is taken out of their hands. You should really see a lot of the machines the big fast food chains use, you'd be amazed.

This kind of decision was never left to the store manager. In the past they'd get a printout and probably fail to follow the instructions occasionally, which may or may not cause damage or increases operating costs across the entire fleet. Now it's done over the network like everything else, for better or worse.

> Adding fringe features that aren't actually necessary (or desired) has become a strange industrial compulsion.

> More features, generally, means more failure modes.

Here's the kicker, they are necessary and desired by the customer and the failure scenarios were almost definitely factored into their calculations. I know you had a solid 5-10 minutes of though on the topic but for supermarket chains that's how they manage to operate cheaper, that's how they don't increase the cost of the food.

And normally I'd agree (more) with you but the small scale of the hack makes me think they've done a decent job at securing them so far. Spending more on people and security to avoid any hack (right?), or losing efficiency over tens of thousands of locations costs more than the spoiled food.

reply
I'd be fascinated to know how much impact on TCO remote control actually has. There's definitely an economy of scale argument, if you can make tens of thousands of devices a couple percent more efficient that has a real impact, but does the remote control capability actually move the needle that much?

There isn't a complex real time control scenario here.. Obviously defrosting too much wastes energy, so you'll want to get that one right. But does having centralized control over that really make a difference at scale? Does having centralized control over temperature set-point make any difference?

From the basic physics of the process it's hard to imagine real time remote control being a decisive factor in efficiency.

> I know you had a solid 5-10 minutes of though on the topic but for supermarket chains that's how they manage to operate cheaper, that's how they don't increase the cost of the food.

I mean.. maybe? But nothing you wrote actually demonstrates that.

[edit] I guess if it's not about efficiency but instead a trust issue then it makes more sense from a marketing standpoint. But generally remote control isn't a great way to work around untrusted operators with physical access. They could just prop a door open or slice the seal if they wanted to..

reply
> something the store managers can figure out

That's like saying the store manager should manage the store website and networks instead of the chain's IT department.

No grocery store manager is going to be a licensed refrigeration technician. Ask them if they know about basic refrigeration concepts like superheat and subcooling, high side and low side pressures, compressor oils...

Further, a grocery store wants to optimize efficiency down to the percent, because that will save tens of millions of dollars over the fleet, and CO2 emissions. A store manager hardly has the capability.

reply
You're confusing specialized refrigeration maintenance (which, at least in the US requires EPA certification) with general end user operations tasks. It is the store manager's job to know how to operate, optimize, and take basic care of their equipment. Just like it's an excavator operator's job to know how to check the fluids and grease the machine. They don't need to rebuild the engine or do specialized maintenance like lineboring, but they are responsible for basic care of the machinery.
reply
basically SCADA but I bet no one remembers that now.
reply
Monitor yes

Adjust: possibly, withing a range

Shutting down: Big no no

reply
well they weren't shut down. just adjusted. and the adjustment range apparently went up all the way to +50 centigrade...
reply
It's not a maybe. They simply shouldn't. The level of idiocy of not only management but also those who implemented the "security" protecting those online fridges cannot be understated: we're talking about real dumbfucktards.

I see such idiots daily, in all venues of life: the world is simply really full of dumbfucks.

It's simply how things are and we shouldn't be afraid of pointing it out.

reply