I'm not an AI doomer but, it does not reduce to that claim. The threat is not just one smart being. The threat is beings that and (1) multiply themsevles instantly, unlike humans that take 15+ years (2) share knowledge instantly "I know kung-fu" matrix style. Humans can share knowledge but they can not absorb it like an AI can/could/will. Human armies have conquered other humans. An army with infinite soldiers will win against one with finite soldiers

Yes, you'll come up with all kinds of objections like AI doesn't have presence in the physical world, etc... That's fine. I'm not arguing my example is perfect, I'm only arguing your characterization of one smart being is not the threat being considered.

Have we quantified the x-risk of Magnus Carlsen?

Not to mention, it relies on a ton of completely undefined concepts.

No one can actually tell you what ASI is or entails because it isn't a legitimate, operational concept. It is a fairy tale.

We can't even define "alignment". As people have finally started pointing out, humanity has never had a collective agreement on what values it should uphold or what ultimate goods are. Your alignment is not my alignment.

AI is not even autonomous. Every system we have today has to be initiated by a human actor. "AI" wouldn't create catastrophic bio weapons, it would help humans create them. The humans are the source of the intent.

Everyone has just completely given in to empty language and marketing nonsense. Honestly it seems like been the people at the labs are drinking their own kool aid and are themselves deeply confused about what they are even building at this point. It is a stateless statistics function running on a bunch of data centers. We aren't even close to an embodied, conscious synthetic being. It doesn't even have state, which is like prerequisite number one, nor is it plastic.