> trawlerbots use smart TV residential proxies and will copy someone else’s UA.
replyWhat bugs me the most are the ones that cycle through a list of LLM user agents.. What are you actually? You can't be Baidu and Bing at the same time.
> My experience with a crawler is that people don’t actually provide a robots.txt far more often.
replyI think many don't see the point of robots.txt as it will just be ignored by most bots anyway, so instead concentrate on more complicated measures.
[dead]
replyThose would fall under the "stealth" bots subject to a fine.
replyThe whole point is that if you can prove a company is doing this kind of shit, they should be on the hook for it. Of course, I'm not expecting any company in China to actually pay the fines, but this opens up some options such as seizing foreign assets.