Semi-off-topic but...
replyOn my sites, I see ClaudeBot consistently (and has been like this for over a year) ask for "${SITE}.com/base_dir1" and then get the redirect (Caddy does this automatically) to get "${SITE}.com/base1/base_dir1/" (trailing slash).
The hrefs on my sites include the trailing slashes for directories, so looks like ClaudeBot's internal code is stripping them off before requesting them, and therefore essentially makes almost 2x the requests to my sites for directories, half of them ending up being redirects back to the same url but with the trailing slash.
At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.
reply> I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data
replyProbably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
If you block it, it comes back with a residential proxy network and headless Chrome. Better not to block based on the obvious signs.
replyBlocking is too obvious. I would prefer to feed back false information but only to LLM crawlers.
reply