As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.
If we had a way to make gene edited humans a lot smarter, a lot stronger or live a lot longer? Or a way to quick-grow human bodies to adulthood in a couple years? Capabilities that private actors or countries may want, ethics be damned? That would be closer to what we have with AI right now.
Now, we're getting better at predicting polygenic traits, and we're getting beefy multi-edit pipelines that might provide a meaningful advantage over embryo selection working in multiple animals. But as of yet, the advantage of genetic editing in humans over just doing aggressive IVF and dredging the embryos for desirable traits is minor.
Bit of a chicken and egg problem there. Can't advance the tech fast without actively using it, can't actively use the tech until it's advanced enough for the benefits to override ethical concerns. So it's getting there, but at a glacial pace.
You should consider reading the wikipedia page about Parkinson’s disease.
Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.
> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.
I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.
For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)
And there have been successful world-wide bans.
For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.
In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).
No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.
That example works against the argument since that policy was rendered moot when Commodore Perry arrived at Japan in 1853 with a squadron of American warships and demanded opening of trade and diplomatic relations at gunpoint.
A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.
With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.
Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity
It's not clear in this context what you actually mean by "government." You are assigning agency to something in a way that seems like a reification. While a bureaucracy can seem to have a life of its own, isn't it generally people who seek power?
Yasha Levine wrote about how this narrative was preceded by a forgotten one where MIT students protested because the computers were going to be linked to government databases and share data on anti-Vietnam war activists. Despite protestations, activists were correct and this happened, and now it happens at huge scale.
thank, mr 习
... recently, as in the last 10 years?
If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.
An easy choice to make if the alternative is everyone dying instead.
> “Politics is the art of the possible”
without sharing tech to make the ASI, you'd hope humanity could work together to determine how to align an AI for our common benefit.
This is a settler-colonial mindset that reflects all the bad things we did onto everyone else. Notably, it's a current US ally that is most guilty of this.
the Baiyue were a vast umbrella of diverse, non-Sinitic indigenous coastal tribes who inhabited Southern China and northern Vietnam.
The Xianbei were an ancient nomadic Proto-Mongolic people from the northern steppes.
The Di and Jie were two of the ancient "Five Barbarian" (Wu Hu) nomadic tribes of northern and western China during the Han and Jin periods.
The Dian Kingdom were an ancient, sophisticated indigenous southwest culture located in modern-day Yunnan province.
The Tujia were an indigenous group of the Hunan-Hubei region. Centuries of inward Han migration and intermarriage have resulted in the Tujia becoming culturally and structurally indistinguishable from their Han neighbors.
That's one outcome, certainly, but not the only one nor, I contend, the most likely one.
A most likely outcome of ASI is human extinction, because there's more paths to an ELE outcome for humans from ASI than there is for non-extinction level outcome.
Your outcome is only possible if:
1. ASI is never able to escape the confines it is placed in.
2. ASI is benevolent to humans.
3. ASI decides, in the spirit of its benevolence, that it should restrict its involvement in humans.
If all three of the above conditions are met, then sure, your outcome is possible. If not, humanity as we know it will end.
It is unlikely that those 3 conditions will all hold, though.
If ASI is trying to wipe out all humans, we probably deserved it. Unironically!
Consider this: All that hardware that's going into those datacentres right now? In 5 years or so it'll all be on the secondary market... an influx of cheaper compute like you've never seen.
Certain powerful wealthy people aren't omnipotent, them losing out isn't the only blocker to progress.
https://en.wikipedia.org/wiki/Sundial_(weapon)
Edit: Mind you, I wonder if the design for Sundial is stored somewhere...
> Scholars have interpreted Cassius Dio's wording to indicate that the fire did not actually destroy the entire Library itself, but rather one or more Library warehouses near the docks.[87][81][8][89] Whatever damage Caesar's fire may have caused, evidently the Library was not completely destroyed.[87][81][8][89][3] The geographer Strabo (c. 63 BC – c. 24 AD) mentions visiting the Mouseion, the larger research institution to which the Library was attached, in around 20 BC, several decades after Caesar's fire, indicating that it either survived the fire or was rebuilt soon afterwards.[87][8] Nonetheless, Strabo's manner of talking about the Mouseion shows that it was nowhere near as prestigious as it had been a few centuries prior. It is unknown whether this was due to historical decline or catastrophic destruction.[8] Despite mentioning the Mouseion, Strabo does not mention the Library separately, perhaps indicating that it had been so drastically reduced in stature and significance that Strabo felt it did not warrant separate mention.[8] It is unclear what happened to the Mouseion after Strabo's mention of it.[60]
> Further evidence for the Library's survival after 48 BC comes from the fact that the most notable producer of composite commentaries during the late first century BC and early first century AD was a scholar who worked in Alexandria named Didymus Chalcenterus, whose epithet Χαλκέντερος (Chalkénteros) means "bronze guts".[90][87] Didymus is said to have produced somewhere between 3,500 and 4,000 books, making him the most prolific known writer in all of antiquity.[90][82] He was also given the nickname βιβλιολάθης (Biblioláthēs), meaning "book-forgetter" because it was said that even he could not remember all the books he had written.[90][91] Parts of some of Didymus' commentaries have been preserved in the forms of later extracts and these remains are modern scholars' most important sources of information about the critical works of the earlier scholars at the Library of Alexandria.[90] Lionel Casson states that Didymus' prodigious output "would have been impossible without at least a good part of the resources of the library at his disposal".[87]
The Library, or part of its collection, was accidentally burned by Julius Caesar during his civil war in 48 BC, but it is unclear how much was actually destroyedMy impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.
This is an interesting subject and conversation, but it's moot having it in these culture-centric forums. I wonder if there are Russians discussing plausible scenarios in Vkontakte groups, or Chinese doing the same in whatever Alibaba group sites they use.
The problem is that we are all skewed by our media, our ideas and our culture. These type of discussions need the highest kind of political interactions.
It's fascinating, specially for someone who lives in a "third world" country, non-aligned to any of these 3 superpowers. Whatever transpires, we are at tge mercy of these (and no, US hasn't treated us "better").
My opinion is that there's no turning back on AGI development. I dont think current governments are capable of getting into an agreement of that size. Specially given the Isolationist stage in the cycle we live in. (In contrast with for example the CFC and Ozone layer issue we had in the 1990s, when the planet was in a globalist kind of stage)
I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.
"Stopping" LLM research just means it will be in the hands of a few who can abuse it. I'd rather a state of M.A.D. but instead of a handful of countries/governments it's millions/billions of people with access to the models (open ideally). Again, perhaps horribly naive or misguided, I understand that bioterrorism could (is?) a real problem as well as more "mundane" things like building a bomb (nuclear or otherwise).
I just feel like limiting access to governments or "blessed" entities is even worse.
> Are there examples of where we have collective decided not to pursue knowledge? Successfully?
human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.
AGI doesn't do away with nuclear MAD, it just messes with economics and makes many people temporarily jobless. Temporarily because in a literal sense RLVR needs verification to train off of, and a lot of jobs cant be easily checked if theyre done. this includes AI safety people, preschool teachers, psychologists, and probably a lot more, including most of their bosses
But, I don't trust capital with either.
- if you have a system that is large enough to store, lets say 10^12 AND gates (all frontier llms can do this) - and this system can produce outputs based on previous things it has outputed
its turning complete, and RLVR on it is optimization over the space of algorithms. If an algorithm exists to do a task, and the task can be verifiably done, this finds the algorithm to solve the task most often.
it is obvious that this scales, from much-worse-than-human to slightly-worse-than-human, therefore it 100% can exceed humans.
I predict that what we consider "super intelligence" is just sheer computational power, but any potential of a very capable agent is bounded by the needs/wants of the person wielding it. That is: even if we were to hand, say, Elon musk this "super intelligence", most humans would consider it relatively stupid because the person wielding it is still a person with stupid goals and values.
Or, to put it another way, I suspect we already do have a superintelligence and have longer than any of us have been alive, and it's just "the market", and it is still incapable of overcoming the limitations of a few morons wielding immense power.... power they will never yield to some intelligence with values and goals "more intelligent" than their own (if such a concept is even meaningful), and intelligence wasted on the values and goals they do have.
On a funny note, I think their prompt was:
"Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."
Studying human bio-diversity since WW2 is the most obvious example, though it hasn't been entirely successful.
Genomics is what finally broke the barrier, especially in the last decade or so.
We were discussing AI in the 90s and it's been discussed before that.
The answer was always the same; hardware can't hang.
Now it can and will get even better.
The SaaS era fueled by ZIRP and ignorant Congress was a fluke that from an engineering perspective didn't produce anything but hype and same old
The generation enriched and empowered by it is just as temporary as Boomers. Little point in enabling their appeals at the expense of scientific progress that helps all of humanity.
China won't. Russia won't.
It's ridiculous to me the level QQing coming from Americans exploiting child sweatshop labor so they are free to ignore their own biological needs and keep a "knowledge work" job (talk about first world privilege) handing them wealth to go tour the poor villages they exploit.
Those workers never had a choice between college or the mines. So sorry 300 million Americans in a world of 8 billion.
We don't even want these jobs given how much bitching I have listened to the last 10-15. IMO the job creators and Congress saw how Millennials liked to be on the computer and went way too far into enabling such banal output.
Make healthcare and housing the economic tentpole. Both still need jobs and technology. But at least the outcome isn't a generational Ponzi scheme engineered by Boomers to enrich them and then let it all collapse when the majority realize those stocks were never real.
Isn’t that like all of the Middle Ages where we replaced knowledge with an alternate religious reality.
For start, previous era was also deeply religious. So it switched religions, if anything new one was more friendly toward knowledge.