This is the sort of pseudo-scientific reasoning by analogy that leads Elizier Yudkowsky to argue we should be terrified that an unfriendly ASI could rapidly develop "diamondide" nanobot viruses, because it will be super-intelligent and super-intelligence can do anything by first-principles:

  > The concrete example I usually use here is nanotech, because there's been pretty detailed analysis of what definitely look like physically attainable lower bounds on what should be possible with nanotech, and those lower bounds are sufficient to carry the point.  
  ...
  > The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer. 

Thought experiments ungrounded by any realistic technological constraints or scientific evidence are pretty much useless for actual forecasting except as an exercise in sci-fi worldbuilding.

https://www.lesswrong.com/posts/bc8Ssx5ys6zqu3eq9/diamondoid...

https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

> superintelligence is, like rocket fuel, an extremely powerful force that has a default tendency to break containment and go boom

what evidence do we have this is the case?

Eh. An intelligence with perfect knowledge of the entire universe, unlimited memory, and infinite processing speed, would have a tendency to go boom. But the real world has limits, and intelligence, even superintelligence, does not equate to godhood. Some tasks are still hard no matter how smart you are.

Plus, we have no real reason to think LLMs are anywhere close to AGI or ASI. So arguments like these are just distracting from the very real, very present danger that LLMs pose: information breakdown, societal collapse and environmental destruction. In other words, this is criti-hype.

Current AIs are closer to bottle rockets than to the Starship on the intelligence scale. Some property damage already happened, but you can't master the art of rocketry without trial and error.