It’s been a while, but I’ve used file-backed client certs issued by a private CA in an ordinary browser without installing anything into the trust store, and it worked fine. I don’t see why a client cert using PKCS11 or any other store would work any differently. Why would the browser want to verify a certificate chain at all?
I guess you could flag the leaf (user) cert as ultimately trusted and that should be fine, but if the browser doesn't see that trust notation, and does see an intermediate CA, it's going to try to pull that back to a trusted root.
One way or the other the user will have to fiddle with browser settings to make a CAC work, either to tell the browser to trust their cert explicitly, or to have the browser trust DoD certs.