upvote
> it's not actually obvious who's running these bots

This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!

Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option

reply
ChatGPT uses a distinct bot type for ChatGPT-initiated web search versus model training:

GPTBot → model improvement / training crawl

OAI-SearchBot → search index / ChatGPT Search

ChatGPT-User → fetch initiated by a user's ChatGPT interaction

OAI-AdsBot → advertising landing-page crawl

reply
Especially since if any crawler IP is in your country, you can get a subpoena to find out who it is.
reply
Meta is an exception. They intentionally ignore robots.txt rules.

(No second chances, I blocked their whole IP range.)

reply
Everyone ignores robots.txt because every robots.txt blocks everyone.
reply
I allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.
reply
Think about incentives, what benefit does Anthropic obtain by going to hell?
reply
Why do we think that only “good guys” are training LLMs? I imagine organized crime is getting in on the game too.
reply