If you have just one Yubikey and you lose it, then you lose your SSH access.
But that's true as well if you have just one SSH key on your computer and you lose your computer, right?
I have offline key on two different devices, haven't lost one yet over the last decade!