I used to install ssh_locker [1] (I don’t know if it still works), and have the passphrase in a keychain separate from the default `login` that I then set in Keychain Access to autolock after a set period of time. Handy for keeping the private key for a while in RAM and then autoremoved, instead of having it there for the whole session. It worked transparently and is a feature that should come with the OS.

[1]: https://github.com/ddribin/ssh_locker