Huh? I run ssh (OpenSSH) on port 22 with root login enabled and no fail2ban on a single-core laptop cpu from fifteen years ago. The load average is 0.00 even with the small mail and web server that runs on it
What cpu and especially what disk load are you talking about? Also not running out of sockets, not sure if that's RAM-dependent (the box has 2GB)