This is certainly my experience of using Claude every day, in a much less cutting edge domain.
And Claude itself cannot tell the difference between the two. The best it can do is use some sort of external constraint (e.g. verification) to filter them.
And then they are simply boldly wrong at times. About mathematical facts. That’s why we need formalisation when working with them. They are still stupid and can’t be trusted. But theyvare also smart and can come with amazing insight. It’s a tool, but definitely one which cuts both ways.
Do we even know what Alpöge had at his disposal? I think most people underestimate what working with an agent swarm even with the current public technology would be like. We can't assume he didn't have something more advanced either.
I don't feel confident my usage of AI on a personal level reflects any of what has been done here.
A non math person may not be able to comprehend the output, but even professional mathematicians have trouble reading through the huge “alien” AI generated LEAN formalised proofs.