Yeah, I agree. My policy is much the same as yours: set a passkey only when a site makes logging in with a password such a PITA that the PITA of a passkey becomes the easier option. Google comes to mind. They seem to actively not want anyone to log in. At some point sites will start to 2FA you even with a passkey, but we don't seem to be there yet.