I feel like there is this weird thing where people will criticize people who do charity work in not quite the right way (from the criticizers perspective), but are totally fine with people who just use their money to buy a yacht.
I'd much rather have people spend their money on feeding the hungry, even imperfectly then spending it on other pursuits.
I sometimes wonder what the critics of EA are doing to eradicate the ills of the world or even their local community. Is it more than what EA is doing? If not, i'd say those in glass houses should be wary of throwing stones.
This is a very uncommon idea within EA. There are some people who support it ("patient philanthropy") but almost everyone within EA either thinks you should be working / donating to help people/animals (most "neartermists") or you should be working / donating to help reduce extinction risk (most "longtermists").
Are they? To what degree are they? And to what degree are they not helping the hungry today, in order to amass more money so that they can in theory help the hungry more tomorrow?
I can respect someone like Bill Gates. He hasn't been perfect, not by a long shot, but he's put real money into really trying to help real problems. What's the track record of the effective altruists? (Genuine question - I haven't researched them enough to know.)
Is that the standard you hold everything to?
According to AI, the biggest EA charity is GiveWell, which spends about 600 million per year. 95% of the money on low-cost high-impact global health programs like mosquito Nets and vitamins for children.
There is also open philanthropy, which gives about 500 million per year to Global Health, Existential Risks, and then Animal Welfare in that order.
That said, I'm not really trying to argue the case. If the simple reason is that people have characterized EA based on SBF, and that's how they think about these things, then that is a complete answer.