It doesn't seem like most people are taking the agnostic "wait and see" approach when it comes to judging effective altruists (or vegans).
To accept that someone else might be right on a moral issue also means accepting that you might be morally wrong.
I don't send 10% of my income to charity or eat vegan, but it does seem to me that people feel threatened when they see others doing this