The incentives are massively skewed towards the AI labs investing their massive amounts of compute into being the first to solve an outstanding problem.

It's a marketing game for them, any societal benefits are secondary. Winning a prize is going to get headlines and feed into the "AGI soon, machine replaces another career" narrative they crave unlike coming up with some (possibly) interesting problems.

I think the problem with AI asking questions is that it will ask questions that are interesting to it but not necessarily us. AI, as a model, will never be a perfect copy of a human. It will always be a simulation, and thus to some extent, will ask questions that humans find irrelevant and solve problems that humans find irrelevant.

For anyone facing an existential crisis on AI, your ace in the hole is your humanity. Only you have it, and only you will be the best judge of what is good and interesting (to a human at least).

If AI can generate questions and then answer them, what are the people for?

> ask challenging questions

As far as I can tell, it's still not possible for an agent to reliably determine if a question is a good question. That means the test part of the loop cant be fulfilled.