One can publicly declare a speaker to be a wrongheaded pig that all decent people should ignore, but that's a wholly different thing from ensuring that said pig is unable to utter its wrongheaded ideas in public.
Free speech advocates support the right of everyone to set up a stall in the marketplace of ideas, but don't have to support every idea that sets up a stall.
A publisher of gay erotica might support free speech, and find themselves alongside revolutionary communists, hitler-heiling nazis and radical islamists distributing bomb-making manuals - but the groups aren't allies or friends outside of that.