upvote
My first week on the job they told me they're about to manufacture 20k units and can you please fix this bug in the firmware by Friday?

I've never shipped anything to real customers in the wild before, so let me tell you how insanely stressed I was to open the firmware and find a 10k lines of C contained entirely within a single switch statement. I think they used some no-code tool to graphically design a state machine then plopped the generated code straight into the device.

reply
Nearly the same experience. Had to fix an issue in a boot loader. It came down to improper setup of the memory controllers ECC engine. It would correct and ignore a single fault. If you managed to get two faults it would raise an exception that was not handled and the boot would fail. For the customer this meant that a reboot might randomly brick the unit until you go in and manually power cycle it.

Just convincing them that their problem boiled down to a single incorrect bit was difficult enough but then having to, in a day, build and successfully operate a test harness to prove the fix worked was the real stress.

I do not miss embedded engineering.

reply
Anything special you noticed about the deployment processes involved with that versus more typical software engineering work?
reply
Software can be updated and patched, even if you have to manually email customers a bespoke exe that pokes bytes into a compiled dll.

Generally firmware can't be updated by the end user because there is physically no way to do so without returning the hardware. (Unless an update mechanism is specifically implemented in hardware, obv)

Pucker factor goes way up because if you ship a bug, there's no way back. If you aren't careful, you can break physical devices which can have consequences anywhere from thousands of RMAs to burning down a user's house depending on the hardware and how bad you fucked up.

The deployment process itself is about the same. Tests and more tests, including testing on prototype and/or pre-production units. Hardware testing can get wild depending on application, but I don't think any SWE would find it too surprising. Then you email a binary to your manufacturer and pray

reply
I can’t quite imagine, even shipping on prem stuff is much harder than the cloud. Especially when people can mess with stuff
reply