> Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice.

Yep.

The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.

The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.

It is like passive voice, blaming the algorithm instead of the researchers running it.

The harness and the model are a system with emergent properties.

The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.