If you wanted to divide by height or weight in a binary fashion to reduce the number of teams, then obviously you'll just have some sports where everyone in the under-6' team is 5'11.5, which seems not optimal and unfair.
I wish there was a good solution.
I mean, we do have weight categories in combat sports, right? I don't see why we couldn't come up with similarly neutral categories if we think it's good to segment people out by physical advantages. The parent comment is making a good point, though: it feels like some people care a lot about physical advantages that map onto gender stuff they care about, and not a lot about weird genetic anomalies that provide physical advantages that aren't gendered.
Re: anomalies - I think this is just unavoidable in any sort of category system, and I don't have a good solution for it except to consider frequency and severity.