They said this:

> Even so much as debugging it would require shutting down the entire factory and replacing all the crucibles afterwards ($$$).

I'm guessing the fundamental reason debugging would require that isn't something an AI could get around. My guess is the LLM would immediately disregard whatever instructions you gave it not to cause having to replace all the crucibles, then say "Oops! You're right." when you point out that you asked it not to do that.