It just isn’t what free speech is.
It’s a separate thing.
Speaking more generally, don't let your ideals bias your judgment. Just because you support policy X and also view ideal Y as good that doesn't mean that X isn't detrimental to Y. It's important to be objective about these things.