> Why is coming up with a better proof (e.g. shorter, more intuitive, more general, connected to existing frameworks) thankless work?

I think the plausible version of this claim is that it's thankless work compared to solving the problem itself. The Quasi-Riemann Hypothesis and Hilbert's 10th problem over Q have both been described as "instant Fields medal worthy, no question asked". Explaining and cleaning up the proof is obviously important, but also quite a bit easier, and we don't really know what future 'Fields medal worthy' achievements might look like.