upvote

points

by TeMPOraL14 hours ago |
comments
upvoteby bradly13 hours ago|
[-]
Do you think there should be a way for a site to tell an agent it isn't allowed access? I'm not sure where I land on this exactly tbh, so no judgment cast.
reply
upvoteby aaron_m0413 hours ago|
parent|
next
[-]
robots.txt?
reply
upvoteby bradly13 hours ago|
parent|
[-]
Has it been settled whether robots.txt applies to user-driven chat sessions and if things like the crawl delay should be applied to say an end-user, an ip address, a harness provider, etc? My understanding is robots.txt is more for training exclusions, but less so for agent work.
reply
upvoteby dhx2 hours ago|
parent|
next
[-]
robots.txt was only intended to help search index crawlers not get stuck in endless crawl loops for badly designed websites.

What you suggest is explicitly not a purpose of robots.txt per RFC9309[1]:

"These rules are not a form of access authorization."

HTTP 429 and HTTP 403 are what servers are meant to return to clients to slow them down or tell them to stop doing something without having first gained authorisation.

[1] https://datatracker.ietf.org/doc/html/rfc9309#section-1

reply
upvoteby bityard11 hours ago|
parent|
prev|
next
[-]
robots.txt applies (or should, in my opinion) to anything that automatically follows a link. Basically any software that is not a human-controlled web browser or single-shot curl command. Everything else: robot.
reply
upvoteby xena12 hours ago|
parent|
prev|
[-]
AI bros think they should be exempt from robots.txt. Administrators of big services beg to differ. No solid consensus has arisen. I bet it's gonna take a lawsuit or two to see how it shakes out.
reply
upvoteby ghaff11 hours ago|
parent|
next
[-]
From the start, robots.txt has always been an indicator of a site's preference with no actual legal significance.
reply
upvoteby Dylan1680710 hours ago|
parent|
prev|
next
[-]
wget ignores robots.txt outside of recursive mode. I think it's correct to do so, and I think an AI loading a handful of pages in response to a command should be about the same.
reply
upvoteby recursive11 hours ago|
parent|
prev|
[-]
If a new directive was introduced that allows for an explicit setting in robots.txt, do you think the bros would follow it anyway? Something like `ALLOW AGENTS` or `DISALLOW AGENTS`
reply
upvoteby Analemma_12 hours ago|
parent|
prev|
[-]
I want agents to be able to act on my behalf, that’s the entire point. An agent should be able to do anything I can do sitting at my browser.
reply
upvoteby compiler-guy12 hours ago|
parent|
next
[-]
I suspect most people would be ok with this if they could only do it at the rate and frequency you yourself can do it. The problem is largely one of scale.
reply
upvoteby daveoc6411 hours ago|
parent|
next
[-]
Is scale what we're discussing though?

e.g. a prompt of "fetch <article URL> and summarise it for me" is very close to what a human would be doing with a web browser, and doesn't seem to involve any kind of scaling issue.

reply
upvoteby kelnos3 hours ago|
parent|
next
[-]
Sure, but all the time I'll ask Claude a question, and then I'll see it fetch 5-10 different URLs to come up with answer. I certainly would not be fetching those URLs at that rate if I were doing it myself. I would probably be visiting those pages, one by one, over the span of 10-20 minutes.

That's the scale argument.

reply
upvoteby TeMPOraL12 minutes ago|
parent|
[-]
As would I when researching anything myself. I'll do a web search, and if I see some highly relevant results, I'll middle-click them so they open in a new tab, and I'll easily do 5+ at a time, before then going to read the first one.

Same with browsing HN, btw. I have a row of 9 HN tabs open, all of them opened at the same time, as I scrolled the front page and middle-clicked on thread link to anything interesting.

reply
upvoteby compiler-guy9 hours ago|
parent|
prev|
[-]
It’s easy to write instructions that have the agent check once every fifteen minutes, or even once an hour, in perpetuity, which never sleeps. And people do write such instructions. A human can’t do that by hand for very long.

The problem is that it is hard to distinguish your one off (which seems perfectly fine) from the tidal wave of bad actors.

reply
upvoteby cruffle_duffle12 hours ago|
parent|
prev|
[-]
Then make agent friendly content. Take the text and make a markdown version.
reply
upvoteby fineIllregister11 hours ago|
parent|
[-]
People doing this say it makes things worse because then the bots download both.
reply
upvoteby TeMPOraL5 minutes ago|
parent|
next
[-]
Because not enough people do this earnestly, and many more do it maliciously (bot endpoints that lie, or provide significantly less information than people endpoints) or put it behind a business contract (yes, APIs), so the bots or agents can't trust it in general.

Also let's not forget that innocent sites suffering from floods of scrapers are actually the minority here - this is just a special case; the main reason for the tension is simply that most websites and businesses on-line rely on users wasting their time, and cannot abide any form of end-user automation. Their business plans hinge on their ability to force themselves on you.

reply
upvoteby compiler-guy11 hours ago|
parent|
prev|
[-]
Not to mention that it solves none of the rate issues. If the scrapers are hitting your site 10,000 times a day, adding markdown isn’t going to change that at all.
reply
upvoteby ryandrake11 hours ago|
parent|
prev|
next
[-]
Technically, even your browser is an agent. It says it in the HTTP: User-Agent. So is cURL. Every application the user runs is acting on the user's behalf.
reply
upvoteby cruffle_duffle12 hours ago|
parent|
prev|
[-]
Dunno why the downvotes. I feel that is reasonable as well. Owners that block that stuff are doing so only to their detriment.
reply