The thing is, I'm not even looking at this from a "how do we keep a superintelligent AI contained" viewpoint. The standard protocol[0] in IT for dealing with a compromise is, more or less, "kill it with fire". Wipe everything and replace with known good backups. And my opinion of basically anything AI writes is "less trustworthy than random apps you get off the Google Play Store". Likewise, "Airgap it behind a serial console" used to be state of the art for computers that handle root CA key material.

I also wouldn't necessarily call myself an AGI/ASI believer - it's my belief these models are actually still subhuman in capability. But they are also superintelligent in one particular direction: speed. Once you have a model that can do something, it can do that something 100x faster than a person on suitably capable hardware; and we've had the hardware to do that for at least a decade. They can also be ran in parallel. Which means you can throw a lot of bullshit at the wall.

So long as there's some kind of process that lets a particular context learn has already been tried and failed (see what I mentioned above about qntm's There is no Antimemetics Division series), eventually one of these models will break containment. The only thing that changes is how many instances you have to run to get a breach. At some point model capability will catch up to hardware limitations and the explosive growth of AI capability will slow.

I'm not sure if I want to call this the "dumb ASI" theory or the "superheated bucket of water" theory yet.

As for automated attacks, that's already the background radiation of the Internet. There's whole frameworks (e.g. Metasploit) for building automated scanners for known CVEs; I would not be surprised if existing (non-Mythos-class) LLMs are already capable of turning a CVE report into a Metasploit module. The usual cybercrime ecosystem is that someone runs an automated scanner on hijacked machines[1], then they compromise the machines that they find and add it to their botnet. This is done by people with almost no actual programming or security skill copypasting commands from PDFs they bought from an exploit dealer. It's all script kiddies.

What changes with a Mythos-class model is that instead of copypasting commands from PDFs, they can ask the model to find an exploit, and possibly get an exploit chain out of it that nobody has seen before. "NOBUS[2]" vulns used to be the exclusive domain of nation-state actors and zero-day brokers spending millions of dollars on exploit kits; but now all of that is potentially under the domain of randos - at least, until the backlog of obvious vulns the CIA had been stockpiling finally gets cleared out, and the Internet returns to merely being as hazardous to your health as the 2b2t spawn.

[0] Okay, the "hardware SATA overlay" idea is, AFAIK, never been tried before.

[1] I have personally been victimized by this

[2] NObody But US