Can be mitigated, as the sibling comment points out, but even in the situation you described, the blast radius is reduced, especially for frontend libs.

As long as you embed it with an SRI integrity hash, you're safe, even if the remote server is compromised.