Gloves require your hands to be perfectly dry to put on effectively.
You can dry your hands on a towel in seconds. I don't know what you mean by "perfectly dry"...? Like, nobody needs to blow-dry their hands before putting gloves on or anything.
If you don't flap your hands around for 30+ seconds, any remaining moisture from handwashing (or sweat) makes them stick to your skin and you wind up fighting them (and about half the time, ripping a hole). A towel is not enough.
I'm not doubting your personal experience. I'm just saying it's in no way a universal rule. I'm sure experiences will be different depending on glove material, glove size, and just the different shapes of different people's hands.
But for me and for plenty of people I've worked with earlier in my life, swapping gloves was way faster and easier than washing hands again. Plus, washing your hands like 40 times in a shift is going to dry them out. It's not great.