I don’t understand why this idea that it’s all just marketing is so prevalent on HN. It doesn’t make any sense. Why would a private company attempt to convince everyone that what it’s building is dangerous to its own customers?Can you think of a single other time in American history where an entire industry was calling to be regulated? I don’t seem to recall Facebook asking for tighter restrictions on social media. I don’t remember Shell telling the federal government that they need to act fast on climate change. Companies do not market themselves by convincing the public that their products might literally kill them all.

The only way this behavior makes sense is if the AI labs believe two things: one, that what they and the other labs are building is legitimately dangerous, and, two, that if they personally stop building then another lab will simply continue (and that lab will be, simply by virtue of not stopping, less safety-concerned than them). This is why they are calling for regulation. Competitive pressure and investor obligations prevents them from unilaterally stopping development. Government regulation is the only (and the most the appropriate) avenue for slowing down AI acceleration.

I don’t necessarily believe that the AI itself will kill us all. But the AIs are increasingly being baked into complex, even military, systems. That scares me just because the opportunity for complex, cascading failure becomes very difficult to predict and control. These systems are inherently immoral (it’s just a lot of matrix math) and we know these systems “lie” and try to evade detection (see the Huggingface write ups). Now, couple that behavior with a military system (drones, targeting systems, etc.). I’m not suggesting that we’re headed for full-on Terminator land here, but things definitely could go wrong.

I think most would think "I can probably do a better job of not shooting myself in the face than some random person not worried about it will" and then proceed to shoot themselves in the face.

That said, I don't buy the companies actually give a damn about the risk either.

I think there's something here about game theory. I'm not sure what it's called.

The logic that if something is beyond repair anyway you might as well exploit it.

For example I heard a pick-up artist say that he thinks he is harming civilization by sleeping with hundreds of women per year. But that he already considers the situation unsalvageable so... "Might as well?"

I don't think that's an amazing attitude, but the AI labs seem to have a similar idea.

More charitably the logic seems to be, "only I can do this responsibly." I heard that from both Elon Musk (he cites this as his motivation for starting OpenAI — a chat with Sergei Brin that spooked him) and of course Anthropic (which split off from OpenAI due to ethical concerns).

So I don't actually think the ethics is all for show. I think people are actually taking this stuff seriously. But it is indeed deeply unfortunate that the survival of the companies incentivizes them to keep going at an irresponsible pace (by their own admission).

All following the same gradient off a cliff. One AI described it as a tragedy of Ancient Greek proportions.

> Bingo. This is the unsolvable calculus.

Actually, it's easily solvable, even by proper application of existing law, but no solution is possible when big AI is above the law and can hack anybody willy-nilly. The absolute worst we could do is to listen to big AI's pleas to put them in an even more privileged position.

> “If we don’t do it, someone will.”

Do what? There are a lot more choices than "let big AI freely hack everybody" and "stop everybody else from working on AI".

> We won’t stop until we see the whites of death‘s eyes.

Trotting out "the whites of death ‘s eyes" is how real discussion is subverted into a BS binary choice. Open-sourcing all AI is an excellent starting point which will both slow down development to a naturally beneficial pace and allow the broader public to police and weed out bad models.

Shh! Don't say it too loud! They might hear you!