Most people are "oh that's fun to watch ok" and then when they don't like it anymore, they get bored and forget about it entirely.
The anti-fans continue to follow it, but rabidly hate it.
Think Syndrome from Incredibles. He's always been the biggest fan.