What prevents an American model from doing the same? They’re all black boxes.

I think we'll pretty quickly see a best practice emerging that any generated code will be subject to an additional pass scanning for vulnerabilities. The scan will be done by a different model than the one that created the code. That will help catch vulnerabilities created by models, whether intentional or not.

This should be done regardless of which model was used - American or otherwise.

I would say you should assume your models are constantly being attacked by various forms of prompt injection. By that token (puns) if you treat all models as adversarial you’d probably taking a very sane approach. That said - evidence of this sort of thing should be easy to find and report on. The fact we haven’t seen it leads me to believe it is not there.

A model absolutely could be trained to engage in malicious behavior like that, but it seems impractical for an actual attack. What you want as an attacker is to insert a backdoor exactly where you want it an not where you don't because every backdoor increases your chance of getting caught. A malicious model inserts backdoors and exfiltrates data everywhere and you care about maybe 0.01% of it. The other 99.99% is negative value to you. In practice this malicious model would be caught almost instantly.

A hosted model is different because you could prompt inject specific customers, but I assume from this question you mean a malicious open source model being hosted by an honest provider.