It's pretty widely accepted in the climbing world that the primary effect of altitude in the short-term is a reduction in your cardiovascular fitness.

The better your heart is at getting oxygen into your muscles and organs, the better it can compensate for less oxygen.

Not a bulletproof solution to altitude sickness, but it's definitely one of a lot of variables that matters. It's also just true that some people are way more susceptible regardless, I've got friends who run competitive marathon times who get splitting headaches flying from sea level to denver.

If anything, fitness makes you more susceptible to altitude sickness. It's not an inherent effect, but rather your habits driving you to do things you shouldn't. You are supposed to take things lazy and slow when acclimatizing to high altitude. But if you're fit, you may be used to moving too fast and pushing yourself too hard. You may not recover from exertion as quickly as you expect, and you may end up climbing higher every day than you should.

Altitude sickness typically starts after 12–24 hours. If you climb high and come back down in the same day, there is usually not enough time for the symptoms to start. And 11,500ft is not that high altitude. People routinely fly to Cuzco, La Paz, Lhasa, and Leh from sea level, and most of them suffer no serious ill effects.

>I guess fitness makes a difference

Not really. Altitude sickness seems quite random in who it effects worst. I trekked to the top of Mera Peak (~21,000 ft) many years ago. 3 of the fittest people in our party got altitude sickness and didn't make it to the peak.