Secretive (and the similar built-in functionality [1]) both allow you to require TouchID too. I found an okay balance using two SSH keys: one for commit signing (w/o TouchID) and one for everything else (requires TouchID, or PIN on Linux)

So, the actions that I really don't want the agent to take (establishing an SSH connection, pushing to a git remote) always require my manual intervention.

[1] https://gist.github.com/arianvp/5f59f1783e3eaf1a2d4cd8e952bb...