We chose the problems to value. Literally every step of a proof is equally proved as the starting point. The choice of the starting point is what makes the problem, and it's a normative choice, which is to say it is a social choice.
I would like to be optimistic about the whole thing falling apart, but I don't know that it will.
They are saying this thing will soon obsolete human thought; meanwhile I use these tools every day and can't get it to refactor a React app cleanly without making a mess.
It's telling that OpenAI's decided they're going to sell ads in ChatGPT. Instead of, you know, using their AI themselves to produce all these valuable and revolutionary breakthroughs they claim it's capable of making.
If you have a super-intelligent AI that can cure cancer, just sell the cure for cancer already?
Intelligence is very rarely the limiting reagent in progress.
At issue with AI is that it's searching an existing corpus for fragments of solutions generated by humans - that's not necessarily novel, and we've not yet worked out how to make the machines actually invent.