Indeed, but this is only a good argument for barring trans women from competing against females. You see, if trans athletes are so rare, only a very small number of people would be adversely affected by such a restriction, they can live with it.
On the other hand, the ban would calm down a large number of female athletes who are seriously disturbed by the mere possibility of competing against men, especially in contact sports, but not only.
Women are women, not only physically but also emotionally and mentally. Setting out on a crusade to change the thinking of millions of women is seriously dumb when a simple restriction, affecting 3 people total, can avoid it.
Now, think about making such a dumb idea a cornerstone of some party's political messaging... that can happen only if said party wants the other side to win.
What's the point of allowing trans women in women's sports anyway, especially at a top level? To affirm their identity? That throws an entire class of people, women, under the bus. Top performing males have an indisputable competitive advantage against top performing females in athletics.