upvote
Recently started an embedded hardware/software job. Shipping firmware to the manufacturer feels like that for the device classes that have no internet.
reply
My first week on the job they told me they're about to manufacture 20k units and can you please fix this bug in the firmware by Friday?

I've never shipped anything to real customers in the wild before, so let me tell you how insanely stressed I was to open the firmware and find a 10k lines of C contained entirely within a single switch statement. I think they used some no-code tool to graphically design a state machine then plopped the generated code straight into the device.

reply
Nearly the same experience. Had to fix an issue in a boot loader. It came down to improper setup of the memory controllers ECC engine. It would correct and ignore a single fault. If you managed to get two faults it would raise an exception that was not handled and the boot would fail. For the customer this meant that a reboot might randomly brick the unit until you go in and manually power cycle it.

Just convincing them that their problem boiled down to a single incorrect bit was difficult enough but then having to, in a day, build and successfully operate a test harness to prove the fix worked was the real stress.

I do not miss embedded engineering.

reply
Anything special you noticed about the deployment processes involved with that versus more typical software engineering work?
reply
Software can be updated and patched, even if you have to manually email customers a bespoke exe that pokes bytes into a compiled dll.

Generally firmware can't be updated by the end user because there is physically no way to do so without returning the hardware. (Unless an update mechanism is specifically implemented in hardware, obv)

Pucker factor goes way up because if you ship a bug, there's no way back. If you aren't careful, you can break physical devices which can have consequences anywhere from thousands of RMAs to burning down a user's house depending on the hardware and how bad you fucked up.

The deployment process itself is about the same. Tests and more tests, including testing on prototype and/or pre-production units. Hardware testing can get wild depending on application, but I don't think any SWE would find it too surprising. Then you email a binary to your manufacturer and pray

reply
I can’t quite imagine, even shipping on prem stuff is much harder than the cloud. Especially when people can mess with stuff
reply
The Zvezda module has been in orbit since July 2000.

I don't think any crewed interplanetary mission is going to last that long for the foreseeable future.

reply
Ideally your Mars transit vehicle hasn't been taking 90 minute heating and cooling cycles nonstop for 26 years.
reply
Well one side will be facing the sun and the other will be facing the void, so there might be similar issues.
reply
IIRC during transit you'd want as much mass between you and the Sun (as shielding), and as small of a cross section facing the Sun. Probably also to reduce heat reaching the propellants.

So in a cylindrical ship you'd want to have one end pointing at the Sun most of the trip. This is, of course, very different in effect on the hull compared to the repeated expansion and contraction of heating cycles.

reply
There'a maneuver called a "BBQ roll" where you basically set the craft to doing a barrel roll in order to prevent any one side from overheating. I image that could help some.
reply
That's not necessarily true. Even spaceships in LEO will perform temperature-driven rolls so as to distribute heat and radiation. I have to assume that long-term ships like interplanetary transport will do the same.
reply
Interesting thought. Isn't it possible to design around this?

Surely this was considered when building the first modules.

reply
Of course you can, but "needs to survive 26+ years" was very likely not part of the original design goals. The designers of the time probably wouldn't have expected the dysfunction to be so deep that 26 years later, only the Chinese can seem to stick to a plan.
reply
You can design around a lot of stuff but what you encounter in orbit will ultimately laugh at that bandage and eat it away. AtOx, hard UV, and radiation levels you don't get on Earth just have their way with everything in orbit over time.

You don't get the AtOx going to mars but you have everything else which will utterly take its toll on a traveling craft.

reply
Sort of like what happened on the Apollo 13 mission in 1970. Engineers on the ground were able to devise a makeshift fix to adapt the control module airscrubber filters to fit the lunar module so the astronauts could shelter in the LM for several days before getting back into the CM and coming home.
reply
Yeah I was thinking about that, the big difference being that you are months out instead of hours/days, if a return to Earth is even possible.
reply
I'm not sure distance matters. They're still stranded with virtually no possibility of rescue from the ground. Apollo 13 was extremely lucky that the hull wasn't breached, the spacecraft could still be controlled, that some very smart guys on the ground were able to devise the fix using bits of stuff known to be on board, the filter could actually be made, and, most of all, that it worked.
reply
There is less debris around on the way to mars and this is a known and worsening for the ISS due to its age.
reply
A top (arguably, the top) metallurgist who studied previous failed parts told me it's corrosion of the Russian alloy used.

Corrosion is a hard problem in living quarters (ie moisture and salt) in space (sealed with no gravity)

reply
In microgravity, everything gets everywhere. My mother worked on NASA funded research for diagnostic spit tests to determine chronic versus acute stress, which previously required blood draws, which are a less than optimal choice in space. It's all very stressful.
reply
I was wondering about this as well. In theory, there are also some metals and compounds that react with each other with just simple contact which result in some kind of amalgamation which can result in disastrous structural loss. Veratassium recently did a video on this kind of effect[1]. Could this be happening here?

[1]: https://youtu.be/ksn5yrsC3Wg

reply
Are you referring to galvanic corrosion? That's well understood and I'd hope not an issue in spacecraft manufacturing.
reply
It seems as though the leaks are always in the Russian section? Perhaps this is why. Humans are the weak link. Damn breathers.
reply
Debris from what? Satellite debris get in that orbit?
reply
Most of the things that will be a common danger (that is too small to track) are tiny pieces of stuff. Think paint chips and sand grain sized objects. These can be from things that came off rockets and ships, and things we've left behind like experiments and satellites. When these tiny things intercept you at many kilometers per second it can be dramatic.

Anything larger, say a lost screw driver, would punch thru the ISS like it wasn't even there leading to some ugly consequences.

reply
I did an internship at NASA. What they told me is that anything larger than a golf ball they track while anything smaller than, I think they said a penny, is too small to do damage. The problem is debris that's in between the two. In that case they only get a relatively short warning (it's been a while but I think it was on the order of a couple hours).

The ISS can dodge debris by adjusting the height of its orbit.

reply
Bits of spacecraft falling off (Challenger's windshield was famously cracked by a paint chip), debris from satellite collisions, even anti-satellite weapons tests.
reply
Debris from space. Lots of rocks are constantly falling from space from all over. Sometimes they're big and make pretty lights in the sky as they fall, often they are practically invisible.
reply
Seems like these structural integrity problems are always inside the Russian section. So if you're on a Russian mission to Mars, yes it would be reasonable to be worried. Otherwise this seems like a non-issue.
reply
This is just not true. There have been leaks due to micrometers in just about every section of the ship at one point or another. A quick search pulls up examples of US modules having issues, especially around interfaces and seals. NASA had a whole investigation between 2018 and 2021 about the recurring issue.
reply
This is just wrong. All serious issues that turned out to be safety concerns were in Russian modules. The 2018 leak you refer to here was in a Soyuz capsule and the 2021 leaks were in the Zvezda module (same place they are this time). In between there were also minor leaks in the Zvezda connection tunnel.
reply
If you count the Soyuz leak, then the Boeing counts too! That was far more serious than anything you listed.

Two astronauts stranded for nine months taking the ISIS supplies intended for others. This is after they safely docked, which was considered risky at the time.

reply
You brought it up. I have been talking about structural issues with long term core modules. And that is clearly a Russian issue.
reply
The Boeing mission was scrubbed out of an abundance of caution. IIRC, nothing bad actually happened.
reply
Maybe we can use the goop from those self sealing bike tires to have self sealing space station modules
reply
Unless your spacecraft is built by Boeing.

We had two astronauts stranded in space for the better part of a year just last year!

reply
>Otherwise this seems like a non-issue.

Except you forgot to mention an epic leak in Destiny just three years after it was attached to the ISS: "At its highest rate, the station was leaking about 5 pounds of air per day overboard." [0] Imagine that happening on the 4th year of American Mars mission.

Also, if you on American mission to Mars, it would be reasonable to worry about cooling system dying mid-flight requiring three spacewalks to fix it: "We'd lose cooling capability to half of the electronics on the U.S., European and Japanese part of the space station." [1]

[0] https://www.nbcnews.com/id/wbna3882962

[1] https://spaceflightnow.com/news/n1007/31station/

reply
Ah yes, the well traveled and highly tested human mission to Mars.
reply
The 10 non-Russian modules have been in vacuum for a quarter century and have done just fine despite facing more debris than in interplanetary space. So yes, this aspect is well tested. This stuff is literally part of the reason why the ISS exists in the first place.
reply
The hubris of forgetfulness; to think that until Elon showed up the West couldn't even put a person in space anymore.

The Soyuz, the MIR, the human space records, the Venera program, closed cycle rockets, all have no equivalent in the West. Even their version of the shuttle was superior (it flew 100% autonomously).

I don't like Musk, but he single handedly saved the Western space programs.

reply
I didn't realize Buran flew, and flew autonomously. Impressive for the times.
reply
This sense of national pride based on long past achievements will always be bewildering to me. Do you really think a country that is actively engaged in a full scale open land war and whose economy is in shambles is able to maintain (much less build) a venerable space program? Elon might have saved the American tax payer from the senate launch system jobs program, but the majority of the global space industry is and always has been in the west. Russia has been an afterthought since the collapse of the Soviet Union. And it shows in everything they have done in space since.
reply
Well, they managed to rebuild their launchpad ahead of schedule and launched this big boy not long ago:

https://www.space.com/space-exploration/launches-spacecraft/...

They've also got some new passenger jets certified and about to enter production (MC-21 and SU-100).

reply
A bit of a tangent, but the fictional book "Children of Time" takes this to wild extremes. Really fun read
reply
A little off-topic - the movie Stowaway (on Netflix) is a good movie about journey to Mars.
reply
Then you die and go into the history books.
reply
They're not flying to Mars in a 30 year old Russian rust bucket so
reply