That's what Virgina Woolf said.

To be highbrow, you can like whatever you like as long as other people don't like it too.

(She didn't say that last part out loud, but it's strongly implied.)