I suspect your family was not as middle class as you think it was. You're describing a very similar childhood to what I had in the late 80s, but we were lower class for sure

I'm not saying that middle class families flew all the time in the 80s, but they absolutely could afford to if they wanted to make it a priority

A cursory google search seems to bear this out. Cheap flights in north america started in 1978 with some air travel deregulation.