No, I definitely wouldn't waste the 5 seconds it takes to unfollow someone. I'd rather spend that 5 seconds on the next interesting item instead.