Even if you are not hacked, or otherwise lose access to the funds, they are not unlikely to become worthless simply due to the system with the stable coins collapsing.
The liquidity can easily disappear, much more so than with Fiat currency or other assets linked to real economies.
You cannot protect yourself from being scammed in that way, even if you do hold on to your keys.