Yes, you want to guard the machine that hosts your passwords. You can even physically keep it at home, and only proxy its port 443 wherever you have a presence in the public Internet.
That’s not to say anything is bulletproof… nothing useful is… just that I don’t entirely trust myself to be 100% on top of something like that as a hobby hosting endeavor.