Technically correct but not really. It's a biased random walk. While outliers are possible betting against the law of large numbers is a losing proposition. More often it's that we as observers lack the ability to see the system as a whole and so fail to properly attribute the net outcome.

It's true that sometimes something can get taken along for the ride by luck of the draw. In which case what's really being selected for is some subgroup of genes as opposed to an individual one. In those cases there's some reason that losing the "detrimental" gene would actually be more detrimental, even if indirectly.