That's more work than even following someone, because it asks for confirmation or pops up a separate modal to unfollow, which it doesn't do for following someone. And so I don't even bother.
This leads to stale social networks and algorithmic timelines.