The more I think about it, the more I think it should be behind CAP_SYS_ADMIN, or a new CAP_KCRYPT (better name TBD. CAP_CRYPT_OFFLOAD?)
Still a risk that some admin-enabled method (like enabling an IPsec VPN) provides a path to it, but would reduce the potential for crafting weird inputs.