Yubikeys (or something like keys in the Secure Enclave of a Mac or iPhone) are themselves not accessible by software, you can only ask the hardware to perform operations with them (e.g. decrypt or sign something).

This means someone with access to your filesystem can’t just copy the key off.

Of course a passphrase is another protective layer. But passphrases can be keylogged, or the decrypted key (or passphrase, depending) in the memory of an ssh agent can be can potentially be read out by an attacker.

With separate hardware, the assumption is that the hardware is much much harder to attack, so the keys stay on there and only on there.

If someone compromises your computer, they can copy your SSH keys and their passphrase and then, well, use them. And you may never know it.

If someone steals your Yubikey without compromising your computer, they cannot use it without the passphrase.

If someone compromises your computer to copy your passphrase, AND steals your Yubikey, then they can use it. But you will know that you are not in possession of your Yubikey anymore, and you can do something about it.

The solution to the second is to have 2-5 devices, each with their own key. I have a nano in each laptop, one backup one at each house, and a floating one on my keyring.

You can require physical user interaction to use the key. If the mossad is after you, this doesn't really help you because they'll just replace the ssh binary and intercept one of your requests, but against a less motivated attacker, it makes it harder to pivot from one machine into others.

My employer gives us yubikeys. The first one I had randomly stopped working after a year. I wouldn't use it for anything where pairing a new key is difficult if you cannot use the old one anymore.