undefined

points

[-]

Is scale what we're discussing though?

e.g. a prompt of "fetch <article URL> and summarise it for me" is very close to what a human would be doing with a web browser, and doesn't seem to involve any kind of scaling issue.

by kelnos3 hours ago|

parent|

[-]

Sure, but all the time I'll ask Claude a question, and then I'll see it fetch 5-10 different URLs to come up with answer. I certainly would not be fetching those URLs at that rate if I were doing it myself. I would probably be visiting those pages, one by one, over the span of 10-20 minutes.

That's the scale argument.

by TeMPOraL13 minutes ago|

parent|

[-]

As would I when researching anything myself. I'll do a web search, and if I see some highly relevant results, I'll middle-click them so they open in a new tab, and I'll easily do 5+ at a time, before then going to read the first one.

Same with browsing HN, btw. I have a row of 9 HN tabs open, all of them opened at the same time, as I scrolled the front page and middle-clicked on thread link to anything interesting.

by compiler-guy9 hours ago|

parent|

prev|

[-]

It’s easy to write instructions that have the agent check once every fifteen minutes, or even once an hour, in perpetuity, which never sleeps. And people do write such instructions. A human can’t do that by hand for very long.

The problem is that it is hard to distinguish your one off (which seems perfectly fine) from the tidal wave of bad actors.

by cruffle_duffle12 hours ago|

prev|

[-]

Then make agent friendly content. Take the text and make a markdown version.

by fineIllregister11 hours ago|

parent|

[-]

People doing this say it makes things worse because then the bots download both.

by TeMPOraL6 minutes ago|

parent|

[-]

Because not enough people do this earnestly, and many more do it maliciously (bot endpoints that lie, or provide significantly less information than people endpoints) or put it behind a business contract (yes, APIs), so the bots or agents can't trust it in general.

Also let's not forget that innocent sites suffering from floods of scrapers are actually the minority here - this is just a special case; the main reason for the tension is simply that most websites and businesses on-line rely on users wasting their time, and cannot abide any form of end-user automation. Their business plans hinge on their ability to force themselves on you.

by compiler-guy11 hours ago|

parent|

prev|

[-]

Not to mention that it solves none of the rate issues. If the scrapers are hitting your site 10,000 times a day, adding markdown isn’t going to change that at all.