How is geofencing supposed to help? The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country and you ultimately end up blocking primarily real humans who don't all know how to change their apparent geolocation.
> A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question.
There is a major way to distinguish them. If you pass a law requiring them to do something, journalists from major publications have legal departments who then order them to follow the law (and then can't do their journalism), whereas the bad actor is either not interested in following laws or is not in your jurisdiction to be subject to them to begin with.
That costs the crawler money, not many do it and is something that can be fixed over time
banning country and AS numbers for cloud companies is pretty effective in my experience
They're for-profit operations. They have money, and it doesn't cost that much. Proof of work is significantly more expensive (but then you put delays on even the users in your own region).
> banning country and AS numbers for cloud companies is pretty effective in my experience
It's the sort of thing that works when the first person initially rolls it out because the crawlers only bother to do something about it once a threshold percentage of people are doing it, at which point they use the workaround.
And then you have people insisting that it works even though it only used to work, and meanwhile you're still blocking actual humans.
From hundreds a day, to zero.
As for journalists, nothing is stopping their legal department from petitioning a judge to issue a temporarily exemption from the law for a specific period of time under a specific written purpose.
and this is the point.
so people without a 'legal department' are shut off.
Eh?
a) Window shopping for the next time the out-of-country visitor is in the country.
b) An out-of-country visitor can contract someone who's in the country to purchase things from the shop and ship those things to them.
c) Queries and requests from out-of-country visitors can cause the shop to notice that setting things up so that they can ship internationally actually would be a money-making thing for them to do.
I do agree that the obscenely antisocial bots run for the benefit of the LLM Manufacturing Industry make it very hard for small-time operators to keep their websites up and running at any reasonable cost. Hopefully the major LLM Manufacturers will fail to secure enough funding over the next few months, their VC-backed primary customers will then have their money faucets shut off, and the attack bots will reduce down to a dull roar because folks have stopped paying "infinity" dollars for them.
They're also stealing your curated dataset: if you take the time to correctly label your products, add specifications and manuals because you want this to be a competitive advantage, your work was just re-appropriated for the benefit of humanity. But primarily for your larger competitors willing to pay for it.