I think it's just people being snooty. Bestsellers are trash, and by definition a plurality of readers are reading them. I don't think someone who reads Gladwell has any greater cognitive power than somebody who reads twitter.
Outside of that, I don't think we should gatekeep it too much though - the biggest benefits come from reading anything at all.