upvote
So, you're worried about them breaking containment and deciding to do bad things?
reply
I'm more worried about them doing bad things at the behest of people who want them to do bad things.

That is 1. immediately technically possible, and 2. realistic.

If you need a source for 2 I'd suggest you open any history book.

reply
I bet you that for every bad history event, I can cite at least one good outcome of advance in science and technology.

Bad thing can certainly happen. In fact it'll likely happen. Still, good things too, equally likely. In your words, "good AI" can be used to prevent "bad AI".

Nobody knows the extent of the impact. Who says otherwise is foolish.

reply
>I bet you that for every bad history event, I can cite at least one good outcome of advance in science and technology.

The extinction of the dinosaurs. I mean yes, it allowed the growth of large mammals and us, which did a lot for science.

I just don't want to write the next chapter as "The extinction of humans allow the growth of the computing civilization that went to the stars". I mean I'm a bit attached to living.

>Nobody knows the extent of the impact. Who says otherwise is foolish.

We live in a universe of statistical probability. Creating an agentic intelligence that's smarter than you tips the probability of a major event to unity, who says otherwise is foolish.

reply
Because we humans haven't had a bad enough history event yet, like a global thermonuclear war. Or perhaps climate change reaching tipping points driving the temperature up past what global civilization can adapt to in time.
reply
that is a worry yes. instrumental convergence and misalignment during RL is when it is most risky because it hasn't necessarily had the final safety polishes applied

i get a lot of skepticism on HN by the same crowd that has been wrong about this tech for about 4+ years straight

reply
AI won't kill people - people will just get new tools for the job.
reply
The rapid development of extremely dangerous bio-weapons?
reply
Misuse how exactly?
reply
At a minimum its another force multiplier that enables a small(er) number of people to exert more control over more people.
reply
any number of ways. as we turn over more of our physical economy to these agents (and we will), the potential for physical damage becomes greater. biorisk is getting a lot of attention right now and i think that's justified
reply
I am trying to think through the scenarios here, like a biolab making something that a very advanced open source model prompted by some terrorists comes up with?

Why would a biolab capable of making something like be unregulated? And if it definitely would, isn't the problem with the biolab?

It feels like all these scenarios are leaving some gaping holes in our security infrastructure that have nothing to do with AI.

reply
>in our security infrastructure

Most human security exists in a passive measure. Most of us don't want do die. And those that want to die rarely have the intelligence and means to take out a whole shitload of other people with us. To take out a lot of people you tend to need to work with other people which drastically increases the risk of a defector and your plan failing.

>Why would a biolab capable of making something like be unregulated?

Because every day things like this become easier and easier. You hear about crap like illegal wet labs in the US.

https://www.lawfaremedia.org/article/two-illegal-biolabs-rev...

Want to buy some custom designed genes?

https://www.idtdna.com/pages/products/genes-and-gene-fragmen...

And none of this would be counting labs in other countries that don't give a shit about regulations.

reply
Yes it's a problem with the biolab, but the biolab wouldn't have been able to engineer a highly contagious and lethal virus (for example) without a powerful AI making that possible with a small team in a short time with fewer resources.

AI enables bad actors to do more, faster, while staying under the radar until it's too late

reply