For instance, you can hear “their model is dangerous! Why would I want that?!”

Investors hear, “Their skunkworks stuff is so impossibly advanced that it’s basically a commercial wonder weapon in the waiting.”

And you might hear “We totally lost control and it hacked the Gibson, this technology needs regulation!” and think “Obviously full lockdown and government control.”

But friendly government actors might think, “Sounds like a good excuse to ban Chinese competition.”

The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.

And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.

I think the idea has some merit, but it only buys time, as people will soon be able to train dangerously powerful AIs in secret. I think the government needs to force the labs to solve alignment before developing more powerful models. Then the question of how to counter a poorly-aligned, rogue model arises; it's guaranteed somebody's going to let one loose.