Yes, TSA is a big part of the problem. It's less "how long it took" and more "how long can it take". I've personally experienced those days where "TSA decided to go slow" and a couple hours disappears. The 5 minute days just make that worse.
Yes, the airport matters. If you're at some small regional it's no big deal. JFK or Atlanta etc is another thing entirely.
Yes, domestic or international matters. Yes, flying business class makes it faster. Yes signing up for "special status" makes things faster.
But airports are typically some drive away from city center (both ends, in traffic). Security and immigration both take time (often significant time.) Door to door time is easily 6 hours more than flight time.
On international returns, both Global Entry or MPC lines are virtually empty when I arrive (SFO)
The worst part is international arrivals in foreign countries, where immigration can soak up a lot of time, and you have no choice but to stand in line. Luckily I don't have to fly internationally too many times a year.
Lucky you :). And I hope you stay lucky! Alas your experience is not universal.
If I catch a train that is 10 minutes before the train departs on a metropolitan train station in Europe.
With planes in my experience arriving two hours before the actual departure is not uncommon at bigger airports, since there are more insecurities involved like how crowded security checks are, where your gate is, etc.
If I come earlier, it's with the expectation that I'll waste some time hanging out in a lounge.
(Train situation is similar in the US, in places where we actually have those. Generally fine to show up right before departure.)