upvote
I think the implicit assumption is that if you explained to the owner of the website what the scraper was doing, they would be more than happy to allow it.

The bot was also being generally courteous, only scraping the 2-3 pages needed to get the information for the human who spun up the LLM.

I don't think GP was at all suggesting he should be able to act however he wants, just because he's a paying customer.

I run an ecommerce site and I'm actually more worried about bots being rejected than accepted because of stuff like GP. We get a decent amount of business through ChatGPT.

reply
I think he's saying that his use of the site, via the LLM, is consistent with what the site owner-operator likely desires.

To give a concrete example: I am a physician and have a site for my private clinic. I am not only unopposed to being scraped by bots/LLMs, I prefer it. The only value I gain from users being on my site is referral to my practice, and that same information being surfaced by search engines and LLMs just means I'm getting in front of more potential patients.

reply