It seems obvious to me that there is no moral reason that some people should only be allowed to live in certain places.
Having all countries open the borders to anyone (ignoring security risks for the sake of the argument) would mean all poor people would emigrate to rich countries and strain the economy, while their home country would collapse from lack of workforce.