Then firewall traffic that doesn't come from your local ISP blocks or authenticated users.

You could set up two way TLS with client certificates