> the problem that he had an affair, or that he was targeted by a spy?

The problem is he was close to a Chinese spy while advocating for increasing exports to China. Then, when public reports emerged about Alice Huang’s true identity, he tried to ignore it. As did the White House. All the way until electoral reality made it clear the cover-up was over.