Proofs/theorems and memory safety vulnerabilities are a special case because there's an easy way to verify whether the model is bullshitting or not.

That's not true for coding in general. The best you can do is having unreasonably good test coverage, but the vast majority of code doesn't have that.