A number of years ago I was working on something professionally and there was a problem. Only about 1 in 5 boards assembled wouldn't crash the CPU. After much debugging it turned out one of the ICs had an open collector output and it wasn't loaded correctly with a pull up resistor. This caused a cascading failure, held the bus up when initialising the hardware which hit the WDT and reset the CPU over and over again.
If you aren't there designing the thing in the first place, you never read the datasheets, never drew the schematic, never placed the components and thus don't know where to look when something goes wrong. And it does go wrong. And then you're in deep shit.
I worry about people who think they can get a product out of the door with this stuff but can't.
Embedded might be resistant to it, because software-hardware interactions are notoriously hard to sim, and AI still struggles with meatspace operations.
Not that it would stop anyone!
You say "people who think they can get a product out of the door with this stuff but can't" and I immediately think: Arduino. That was also seen as a way to introduce people who understand nothing about embedded to embedded. Surely no one would ever go from an Arduino prototype to an actual production run?
Ha ha WRONG. I've seen actual production hardware ship with Arduino firmware, because no one cared enough to fully rewrite that cobbled together Arduino firmware from the first prototype. The FW team just went over it enough to whack-a-mole the most obvious issues, and shipped the result.
So, no. People are absolutely going to ship AI genned embedded hardware, and get away with it too. I bet that by now, someone already did.
That's just basic design sensibilities.
In the given example, the human process obviously failed, right?
If you can’t or don’t entirely go over the output, the failure mode is invisible.