Should be treated like a digital cousin of gain-of-function research.
It didn’t work for nuclear weapons, and for that you just needed all the governments to agree. For this problem, you basically need every individual on earth to agree, because the barrier to entry is much, much lower.
Regarding the "barrier to entry" for AI, this is not really true. Training a frontier AI model takes an enormous amount of compute using chips that have a supply chain with several chokepoints, like EUV machines and HBM which are only made by a few firms.
[1]: https://cs.stanford.edu/people/eroberts/courses/soco/project...
[2]: https://www.atomicarchive.com/resources/treaties/index.html