Every time a wild female mates with a sterile male, that's one less time they've mated with a viable male. It greatly reduces the number of successful females and therefore the size of the following generation.