What's more likely right now:
- Your cryptosystem is compromised at some point in the future if/when quantum computers exist and can effectively attack EdDSA/X25519. Something no one has yet demonstrated or come close to demonstrating.
- Someone implementing PQC in a library/software/hardware follows the standard which does not care at all about side channel resistant implementation of critical algorithms, resulting in your private keys being leaked. Demonstrated repeatedly over 20+ years.
[1] https://cr.yp.to/papers/safecurves-20240809.pdf#chronology
this isn't to say everything will go perfect, but the people doing implementations are more experienced now, and the problem is an easier one to do (though in a certain sense, optimizing compilers are making any side-channel resistance a harder goal to achieve. an implementation that is resistant under one compiler version may not be resistant in the future).