The thing that I'm less clear on is, if and when they achieve some kind of recursive improvement exponential growth thingy, isn't that going to heavily disrupt the entire economy / stock market? In that environment, would Anthropic's stocks really be a good bet, given they have to operate within the economy?
That's a long-term problem to be dealt with later, greed dictates to pursue the former as long as the music is playing and everyone is dancing. Similar to 2007-2008 when it was clear that things were breaking but the incentives were to play the game for as long as possible before it imploded.
It is a myth. They are chasing science fiction. They either dream of an omniscient overseer who will take control of society and nominate its shareholders-parents into the governing elite, or they envision a chatbot that will solve every problem, again turning shareholders into the power brokers. They forget that nobody will ever be allowed to "win capitalism" and take over the planet. Companies have are torn down for far far less. Every AI model is data, ones and zeros. It therefore can be copied/shared/leaked. Intellectual Property law is not a ladder for assent to lordship. It can and will be rewritten or outright ignored rather than allow capitalism to collapse.
I think that at this point gAI is just a marketing blurb, and the actual race is to dominate current practical applications involving basic RAG and ReAct loops. This can go from anything between software development, business intelligence, and market research. A company whose top brass can use a team of agents to do what nowadays they assign to mid- and top-level management is an executive with tighter control over the whole business process. For this you don't need gAI.