When doing stuff on the internet, I've just decided to stop worrying and treat these scans like that above mentioned QRM. You can filter it a bit if you like [1], but really, a sensibly configured and maintained SSH server is as secure as it gets as far as I can see.
[1] https://alastairbarber.com/Building-Anycast-Network/#securit...
This has been the case for years. I can remember this from logs for port 22, more than 20 yeas ago, I saw this.
If you don't have a wp-admin.php who cares if someone is trying to access it? If you have one but it correctly validates your admin credentials, again who cares?
You can turn it into a fun project of making a honeypot.
It's plenty possible to run an independent site with no issues if you keep things up to date and change a few things to thwart the most common attack attempts.