Rent is always going to go up there even if they build more. Same in other places. As long as rent setting tools exist to collude - we will see the rent not go down. You're not gonna dump $100m in new buildings and not maximize your return.
There may be fewer people in manhattan, but that's mostly because fewer people live in each living unit. The same number of living units is being demanded by the market because of evolving living preferences.
If you allow sufficient living units to be built, it doesn't matter how much landlord try to collude, they won't be able to keep rent high. Someone will break when the vacancy rate reaches 15%.
"The very reason why we object to state ownership, that it puts a stop to individual initiative and to the healthy development of personal responsibility, is the reason why we object to an unsupervised, unchecked monopolistic control in private hands. We urge control and supervision by the nation as an antidote to the movement for state socialism. Those who advocate total lack of regulation, those who advocate lawlessness in the business world, themselves give the strongest impulse to what I believe would be the deadening movement toward unadulterated state socialism."
--Theodore Roosevelt