There was a comment in a different thread that suggested they may respect the robots.txt for the most part, but may ignore wildcards: https://news.ycombinator.com/item?id=46975726

Maybe this is worth trying out first, if you are currently having issues.

If possible block I would block by country first. Even on public websites I block Russia/China by default and that reduced port scans etc.

On "private" services where I or my friends are the only users, I block everything except my country.