One easy answer up there with "more study is needed" in terms of being a bit cliche, but cliche because it's often true, is that there may be a richer spectrum or something that isn't even 1-dimensional then we're currently able to grasp, which will become more clear over time and more study. Perhaps it's less rare than we think and the rareness is only those we can talk into agreeing their qualia really is different from others. Or the way we are describing it leads people in the "1-2 out of 5" category to declare themselves aphantastic on HN when they aren't really properly so.
I agree this is an annoying confound, and also that it feels highly intuitive that being able to consciously visualize ought to correlate with better visuospatial (and thus much mathematical) cognitive ability.
I think the solution is recognizing that the reason cognitive deficits are mostly unobserved (visual memory recall studies aside) is recognizing that aphantasia is almost certainly a conscious access problem, i.e. most aphantasics are still using imagery too, just unawares. Last I checked, this is the leading hypothesis / explanation.
Also, plasticity and the ingenuity of the brain is clearly such that an aphantasic can also find other ways (e.g. other sensory modality imagery, pure logical relations, etc) of representing the problems that seem like they should rely on visualization. It seems obvious to me that these other ways can be both advantageous/disadvantageous, depending on the situation / phenomenon.