A flaw in the mirrors wouldn't leave the anomaly in a consistent place, it would keep causing problems no matter where you look.

But I'm pretty sure they thought of all of this and many more objections already. It's not like this is a super advanced thread of skepticism that physicists would have overlooked

Afaik they did that during calibration. Take known close by objects, compare results, make sure they are the same (up to the capabilities of your ground truth).