There may be fewer people in manhattan, but that's mostly because fewer people live in each living unit. The same number of living units is being demanded by the market because of evolving living preferences.
If you allow sufficient living units to be built, it doesn't matter how much landlord try to collude, they won't be able to keep rent high. Someone will break when the vacancy rate reaches 15%.