That depends. It can still affect genetic fitness if it affects an individual's ability to confer benefits on their descendants. Of note: most of the most wealthy and influential people in our society are beyond their reproductive years (not technically true for men, but mostly true in practice).
That's a simplification, but you get the idea.
Edit: why am I being downvoted for this?
No, it isn't. It can be socially impolite to yawn unexcused, when someone is talking to you, as it has come to be interpreted as boredom rather than tiredness or similar. But it isn't inherently impolite to, for instance, yawn when walking down the street, or in a setting where someone isn't talking to you.
What you describe is in my opinion true for western cultures. In Brazil they are not so relaxed about it. Asia even less so.
Because you don’t know what detrimental means in this context and clearly don’t understand evolutionary timescales?
Our ancestors used to have tails. We no longer have tails. Plenty of people wear artificial tails today and get laid, it's not a 100% disqualifying trait
When the woods in eastern Africa changed into savannah, we shifted to two legs and adopted a persistence hunting strategy. The tail became useless, even a liability, and mutations that resulted in reduced tails were not selected against anymore.
…Do they? What did I miss?