This is giving "why is this plane full? What in doing is important but what is everyone else doing here?"
To be fair, I do not generally support wholesale banning of scrapers and non-browser clients, since this tends to have lots of ugly side effects.
But regardless, I'm interpreting your last comment as saying that you should be entitled to behave as you please because you are a paying customer. If that's the case, then you are in for a rude awakening one day
The bot was also being generally courteous, only scraping the 2-3 pages needed to get the information for the human who spun up the LLM.
I don't think GP was at all suggesting he should be able to act however he wants, just because he's a paying customer.
I run an ecommerce site and I'm actually more worried about bots being rejected than accepted because of stuff like GP. We get a decent amount of business through ChatGPT.
To give a concrete example: I am a physician and have a site for my private clinic. I am not only unopposed to being scraped by bots/LLMs, I prefer it. The only value I gain from users being on my site is referral to my practice, and that same information being surfaced by search engines and LLMs just means I'm getting in front of more potential patients.
The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money.
I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am considering as well as staples/groceries and everything in between. I have this little scrape/ingest pipeline because there's no other way to obtain this data... not even an API that I can pay for access to.
One of the large appliance sellers that I have in the scrape queue has gotten _hyper_ aggressive with bot detection to the point where even my personal head-full chrome instance doesn't always get to load the page. Guess who I will never buy that ~$2000 appliance from.
Significant sales (beyond for excess stock clearance/discontinued items) exist largely to generate some excitement from potential customers and to lure them in to taking a look at the site/store and to hopefully get them to buy some extra stuff that's less discounted while they're there.
The actual on-sale item is potentially losing money for the business or at break-even.
------
A person who's running a series of bots like this to track the exact lowest price on the internet for items over time and to always only buy at the lowest price it's ever been sold for....is probably the least likely kind of customer to buy anything extra that's actually profitable for the business.
Almost certainly! No eComm site wants the cautious shopper that's done their comparisons and knows price trends; they want the impulsive people with a line of credit. I doubt that my local grocery store is happy that I index all of my receipts so I know what I buy, how often and when the prices change so I know when a discount on something I buy every once in a while is actually a discount.
When I built this a few years ago, I had to write a lot of code to make all this work.
Now that's a simple "hey $agent, check the price of https://some-site.com/appliances/that-new-fridge.html and https://other-site.com/shop/appliances/that-new-fridge.aspx daily and notify me when it's on sale or has a promotion"
I would expect that more and more people will discover that their LLM of choice can do this, that their LLM of choice can help them comparison shop and do analysis of when - historically - is the best time to buy that new TV or whatever (yes, pricing on this stuff is seasonal!).
Maybe _most_ people will never do this, maybe I'm wrong. But if i'm not, blocking all bots "because bots are bad, mmmkay" is how you loose a sale.
Ignoring the specific price data example, more and more and more people _are_ browsing the web through an LLM. Assuming that continues, blocking bots is going to have the same effect on your traffic as doing everything in the "what not to do" section of the google SEO guide book.
I have a business where I clearly don’t want any bots. A retailer, maybe they do.
Not a rhetorical question, because most companies would welcome that interest from customers.
That's the thing; they can't, really.
Best we can do is compromise; allow bots and rate limit them. Maybe someday there will be a relatively universal standard for fetching pages akin to the `index.html/index.md` pattern that is emerging now. Put the bot filters on the .html path and some sort of micro transactions system on the .md endpoint.
In basically every cat/mouse game defense is forever one step behind...