I'd hope so, because that's what we're doing right now. Your browser is automatically speaking HTTP for you so that you don't have to.
Am I having a bit of a laugh? Maybe. But really, services should be user-agent agnostic. That's the whole "agent" part of User Agent and the founders of the Internet had incredible foresight to name it this way.
> Is it allowed to scrape data?
You mean, request data and receive what the other server voluntarily transmits?
> Are there any laws for this?
There was a court case that said the above is fine, thankfully, since that's how the internet works. There's probably other cases going on and I'm sure at least one of them will have some unfortunate tech-illiterate result that makes things worse for anyone who understands this stuff.
Now it's the Agentic User Agent.
Taking your position to its logical conclusion implies that we shouldn't try to mitigate DDoSes either.
In many cases, what the other server voluntarily transmits has so far been based on the tacit assumption that a person, with person-level time and computational power, is doing the receiving. While in principle a machine could be doing it even in pre-LLM times, in practice many websites, including all the biggest ones, have implemented a wide range of approaches to try to curb machine access, starting with user agent checks and rate limits but by no means ending there.
The question is: Given the new landscape, where this assumption increasingly does not hold (because AI agents are increasingly able to simulate anything a person could do online), would those servers voluntarily transmit that data? In many cases, the answer is no.
Not really. At any time you can, and should, choose not to reply to traffic that is wasting your bandwidth - ban IPs, use DDOS mitigation services, etc. My position is simply that regulation doesn't belong in this space, and it's ok for the 'net to be a dog eat dog world. Kind of what keeps technology advancing and exciting.
Is it me (the site owner in this example)? If so: Since it's my subjective decision to make, couldn't I equally legitimately decide that traffic I serve to non-human entities is "wasting my bandwidth"?
To be clear, I'm not trying to make the case that there should be some law in place that prevents scraping or machine access across the board -- only that it would make sense for website operators to be able, optionally, to include that kind of usage restriction in an ordinary contract and legally enforce it by the usual means (lawsuits), in addition to any kind of technical restrictions they are able to put in place.
The thing about most sites is they're public and you don't need to sign a real contract to use them. Can't have it both ways.
Yes, but it's not filling in the forms or clicking the buttons for me. HTTP is just infrastructure. Are LLMs infrastructure? Are we too maybe infrastructure? Where do we draw the line?
> You mean, request data and receive what the other server voluntarily transmits?
I mean to go over a large collection of publicly or privately (to you) available pages and parse and collect the data, with idea of using it in other purposes.
Regarding scraping, considering that this whole AI phase was built on illegal scraping, I don't think they can say anything now...
I've always called that "learning" but I guess it's called something else when a robot does it :)