Thats not what the issue is, it's that they aren't free to job-hop in the way that other people are. H1B keeps their wages artificially depressed and they have a sword of damocles over their heads threatening deportation.

Random refugees coming into the country "takin my jerb" isn't happening. Large companies importing skilled labor for specific positions to either drive down sector wages or acquire workers who are more willing to put up with bad working conditions (this is nursing right now) is a different beast.