I think much of the opposition to data centers is because companies are rushing to build them as quickly as possible. When you are building a traditional factory, you probably expect it to remain operational for 30+ years, so it makes sense to get the local community on board first. But a data center building could be obsolete in 10–20 years, if the requirements change enough that it's not feasible to retrofit it for new hardware.

Yes, if the company that wants to build the factories became known for operating it on the most polluting way possible¹, you can bet nobody would let it open either.

1 - Way more polluting than anybody even considered possible before.

It’s so crazy that nobody wants a gigantic, loud, ugly building that raises their power bill and puts them out of a job in their neighborhood

Not in all places. However you will see the places where they are stopped in the news.