I know most people (of any nationality, not just Americans) don't understand this, but you and I are not most people. I believe things because it is correct to believe them, not because most people believe them (or not).
edit: to be clear, you can say "immigration control lessens freedom, but I'm willing to give up freedom—both yours and mine—because I value certain things more". That is a reasonable point, though I might as a follow-up probe into the motivation behind that. "I think the government gets to stop you from living with your spouse, but I'm okay with that because XYZ" better have a pretty robust XYZ behind it!