If rich people can't buy luxury housing, they buy non-luxury housing, driving up its price. If you don't want rich people driving up the price of affordable housing, make sure there is lots of luxury housing available so they don't drive up the price of the cheap stuff.

That's not how it works. More competition will bring the price of those down.