For me it was more about the mismatch between what my eyes saw and what my inner ear felt.

Running around in first person while sitting or standing in a room took some real getting used to. But cockpit games or mostly stationary (in a reasonably small volume) standing-perspective games were fine.