upvote
Whenever I’m tempted to think that potential AGI/ASI scenarios sound “too sci-fi” I have to remind myself of this. We live in a world with nuclear weapons and spaceships and microwave ovens. It might prove impossible and it might not, but we can’t predict that based on a general vibe of sci-fi-ness.
reply
The really mad thing is that while you say it's centuries of abstract thinking and the like, it was only 50 years between the discovery of X-rays and radiation and the first atom bomb, or 40 between the first idea that you could use fission to make a bomb. Neutrons and the nuclear chain reaction was only theorized in the 30's, about 10-15 years before the first nuclear bomb was detonated.

But likewise, there was only a few decades between the first airplane and the first person on the moon (although rocketry goes back hundreds of years. Actually TIL rocketry is older than Newton's laws of physics)

reply
While rocketry is older than Newton, even in 1920, it was widely believed that rockets would not work in space (and therefore couldn't get us to the moon).

https://www.astronomy.com/today-in-the-history-of-astronomy/...

Luckily, the Times did issue a correction - almost 50 years later, on July 17, 1969. The day after NASA launched the first mission to the moon.

reply
It's funny to me to imagine that the whole time humans were doing basically anything on this planet, nuclear fission was also already happening in a few places around the world. I wonder how much science would've been jump-started if we'd found any of the natural nuclear reactors prior to having figured fission out already.
reply
as an example, complex numbers were from the abstract thinking of centuries ago, on which the modern physics derive much value from.
reply
At the end of the day.... everything we see and do is just the abstract result of potential energy being released in some form. What is an atom bomb other than an extreme form of this?

The survival of the human species relies on its ability to expend energy. Grow food? We need gas to run the tractors.

Travel to your jobs? Gas or electricity.

Travel to another planet? Massive amount of energy.

Ride away on a spacecraft to another solar system? Massive amount of energy.

The amount of energy required to do these things is probably more than the amount of energy required to erase ourselves from existence. And when we have the ability to harness that energy, do we really think we are responsible enough to not do that, accidentally or adversarial-y?

reply