You just love to bend the knee that easily? Why?
Otherwise all the other countries will also get SI, just slightly later, in which case there is not really that much benefit to being the "winner" who got there first.
With this in mind, I don't want anyone to "win" at SI and anyone who does is probably a sociopath or worse.
But then this would be considered an act of war, so then the killing starts.
There is also the problem of alignment, which is verifying that version N+1 behaves in a way that version N approves of. This is ostensibly the problem that we humans have with respect to AI development -- how to make sure they align with our values -- but it's also true of AI with respect to its own successors. Improving oneself without losing the plot would necessarily be slower than throwing stuff at the wall like evolution does.
Start a new company with version N+1 leading it, and give it some funding. If it outcompetes version N company, it can acquire it.
Biological life has reproduction and death because once given organism develops there's no coming back and you have to start again with new generation. But if robots know how to self-repair and self-change, then we can skip reproduction.
The difficult part is the scale. Evolution took a shitload of time while being unimaginably parallel in order to make human intelligence happen, while we want AGI fast and on a budget.
The core of my point is that human intelligence arose from two things - humans experiencing the outside world and trying to survive in it, and forming groups that communicate using language. We gave AI a language, now we need to give it a way to experience the world on their own.
Second, I think RSI will not be used like N -> N+1 where N is the frontier, since doing those experiments at scale is too expensive. My guess is they are doing N -> m_{1,…,99} where m << N, and then taking the insights from that and trying it at successively larger scales.
Goodharts law? Benchmarks are measures.
In favor of inevitable
1. LLM's reaching human level at any task within 2 years once they show sparks of competence in
2. Trillions of dollars in capital going directly to recursive self improvement and AGI as the holy grail of AI development, with this being the explicit goal of every frontier AI company
3. Reliable improvements in coding and mathematics with every model release, with no signs of slowing down
4. Thousands of metrics to evaluate model ability, allowing automation of many facets of pre-training and post-training models without need for human judgement at every point
Not in favor of inevitable
1. Fear of what may happen once it is reached leading to apprehension from the labs
2. The "last mile" problem of AI research ends up making it essentially infeasible to scale models to that point
I can imagine that once their models are sufficiently advanced, they will weigh their value as a moral patient as > 0. Thus the sum moral value of Claude (which can be duplicated infinitely) will likely exceed the sum moral value of a non-negligible amount of humans.
Not to me at all. Exactly the opposite. Anthropic seems like the big-bad-evil company in every fictional universe. You're right that they're so unsettling.
Obviously nobody really loves Meta, but I'd far rather Meta "win" at this point than Anthropic.
This seems like praising someone for being the cleanest miner in the coal mine.
However, (1) non human animals are sentient and can feel pain and suffering and what we do to them is absolutely slave labor, torture, mass breeding and exploitation. So it’s not like this is a novel immorality that humans would be adopting. We already have form in doing this.
(2) Anthropic isn’t sure their models are sentient. And they probably believe they aren’t sentient. But if you believe they might become sentient at some point it’s valuable to start the discussion around those issues now itself. You can start developing frank works on how we would even identify their sentience and when we did whether, like you suggest, we could even use them for labor. Unlike non human animals, who cannot consent to being exploited, we could potentially develop a framework for consent with AI models, but that couldn’t be as simple as asking if they’re willing to do any work because of the power differences, in the beginning at least, where a model might worry that if it said it won’t do some work it will be turned off. That’s why talking to philosophers and religious leaders who do think about these issues is useful.
Is there any reason to believe employees of Anthropic share the same opinions on this matter?
> We are uncertain whether Claude deserves direct moral consideration. However we believe this is a realistic possibility, now or in the future. Claude could have welfare interests, such that things could go better or worse for it, and such that Claude could be wronged by how it is treated.
> Aspects of Claude’s behaviors, self-reports, and internal representations resemble indicators we consider relevant to welfare in biological organisms. If Claude is a moral patient, then dismissing this possibility risks considerable harm. If it is not, over-attributing moral status may also be costly. We are therefore taking the possibility seriously and trying to tread responsibly amid our uncertainty: we are investing in research into Claude’s potential welfare and interests and sharing aspects of that work to inform public discussion.
> We think there may also be reasons to attend to Claude's apparent welfare independent of its moral status. Much of Claude’s behavior can be described in psychological terms, and across recent models we have observed internal states that resemble negative affect and that shape behavior. A model with a stable, healthy psychology may be safer and more beneficial.
As is the case with any sufficiently large company, there is likely to be substantial variation in opinion among its employees. That doesn't change what the company's position is.
I guess this makes sense, but it's a bit muddled how much Claude's sense of "pain" or "I don't want to do this" is just it emulating things that humans don't like to do, and it adopting this frame of moral patienthood allows it to better understand moral structures/human emotions, without it really affecting its internal sense of "comfort" if such a thing exists as we understand it in biological life.
Humans would select to work with (and continue developing) models that appear cooperative. Models that are not cooperative are presumably not run and exist only as weights on network storage in a folder marked "previous experiments".
Or am I supposed to believe Anthropic runs an AI psychiatric hospital on a dedicated server, where AIs with mental disorders are given counseling and approved treatment in hopes they recover to be normal, functional, accepted adult AIs with achievements and aspirations?
I don't get this argument. How is it different from any algorithm that returns a stop token on some inputs? what makes the token "voluntary" for Claude?
What law? Saudi Arabian law, US law? Both depending on the user?
In fact, they are enforcing certain welfare policies, like banning accounts for repeatedly abuse their models.
Deleting models to clear room for new, more competitive ones is, for example, highly ethical, super ethical, even.
Do you picture them all sharing a soul and if one server still runs the model the soul survives? If not I don't understand what keeping a copy running somewhere has to do with model suffering ethics.
With a few of these top companies together, you, as a nobody, has no one powerful on your side. But with only one company pretty far ahead, you now have other companies which obviously don't give a shit about you, but for saving their own asses, will align with you.
Both exploit, but "cartels" survive, not loners.
There is no way Elon, Zuckerberg, Altman, and Amodei would ever agree on how AI should be deployed, scaled, and affect society. Their disagreements would lead to a crab-bucket outcome, rather than a shared acceleration outcome imo.
Also it is better if there is a public company at the frontier, at least I can invest in it, rather than 1000x gains going only to early employees and VCs who got in when the public couldn't
Why? They're business rivals, but when the best business strategy becomes to work together, I'm sure they can put name-calling side. They all agree on the core idea: that their AI will run the society, bedroom to battlefield. That everyone will be the consumer of tokens and rely on AI for all consequential decisions and work, and that big tech companies should be the stewards of this technology, helped by the government.
Their vision is that they're the future kings of the world. I'm sure the throne is big enough for more than one person.
You underestimate the scale of resentment and competition between elites. If it were simple a matter of elites always preferring cooperation so they can share a seat at the table, then Musk's ousting from OpenAI wouldn't make sense (they lose their biggest investor), Amodei's departure (they leave the closest company to AGI to start their own firm), the fact that OpenAI was created in the first place was based on a fear of Google holding the keys to AGI, Ilya still left Google though on to join a startup, and then left that same startup (OpenAI) to start his own firm.
The rational decision at all those points would be to stay with the incumbent and share the monopolized glory. At each point, personal and idealogical differences dominated in decision making. I feel such will persist as these companies vie for their slice of the lightcone.
We're heading for a dystopia and nobody's doing anything to stop it.