Currently, with sex-based categories, a woman can be declared "the best in the world" and most people won't waste much time on the question "yeah, but could she beat the best men?" (granted, some will). They will accept that, e.g. she has the fastest time over 26.2 miles in the world right now, even though a few hundred or a few thousand men worldwide are faster.

If you use performance based metrics to create the categories (the way that road cycling does, for example, though still within gender divisions), that "title" would go away, and likely a woman would only be "the best in the world in division X", other than in (as you noted) some endurance, climbing and gymnastics sports where an elite subset of women could potentially be the best of "top" category.

It isn't completely obvious that this is a negative - how much of a change it would be would depend on a lot of other changes (or lack thereof) in how sport was organized. Certainly if it continued to focus on only the top division, then women would be shut out of most opportunities to be professional. But that's not inherent in the design. I do concede, however, that it is quite a likely outcome of such a category structure.