You don't get rid of the 'poor'/less wealthy if you allow building higher and thus have a more efficient use of land as land values increase. All owners would share the cost of their land value tax.
The problem is not building higher in areas with very high demand. Of course they become 'rich only neighbourhoods' or whatever.
If a house is worth £1M, a flat in a 5-story building should be less than 200k in the same location. We can't have houses in city centres! We can't have houses 5 min from the city centre! We could definitely have houses 1h from the city centre in a small town. In London or NY we shouldn't, though!
I always tell my internet friends that they should think hard about second order effects. Stamo duty has very negative second order effects. Land value tax has very positive second order effects. Property taxes sit somewhere in the middle.