Actually, I think you're really underestimating how many people struggle to find clothes that fit them well. Most people's bodies are not perfectly average. For clothes not to fit well, you don't need to be a significant outlier in one dimension. Just the accumulation of several smaller deviations from average can be enough to create an awkward fit.

Beyond that, even if we limit it to height alone, there are hundreds of millions of people who are much shorter or much taller than average.

I can assure you having observed the process of clothes shopping for the women in my life, that as far as they are concerned, clothes do not just “fit them”.

Yeah there was no way Amazon could expand from its niche of specialist books to anything else, because that market was already served. Oh, wait...