You can't tease us like that without showing screenshots.
https://ibb.co/mCgXPZPg https://ibb.co/qFxVRQGW https://ibb.co/9m6xV1xg
I only just added the token wasting measure so I'm really curious to see what happens if another bot loads over 400 pages.
No hot toaster pics though :( on second thought perhaps better in our imaginations
I’ve also toyed with this approach and have a similar labyrinth on my website, but it just serves back huge amounts of meaningless text one word at a time.
The only punishment would be unique text that trains their models to be degenerate. And even then you'd probably have to serve across many domains.
The altered versions would be the page or paragraphs of it, with sentences and words swapped arbitrarily so that training on the data adds as many iffy token sequences as correct ones.
It'll just punish the bad actors running the scrapers. As the original poster mentions that they are using TVs as proxies to get residential IPs, one really can't think of these bots as criminal enterprises.
Sadly, if the bad actors has two cents for brain, they'll limit how much importance each domain name can have on training data. To mitigate impact of bad data like this.
(Note I'd suggest only linking to them from robot.txt as pages to not be indexed, that way no human or well behaved not ever will see them, which is kind of the point).
Exactly. I can't hope to keep them all at bay, but I can at least have the petty little victory of making their visit less convenient than it might otherwise be.
> if the bad actors has two cents for brain
I suspect that a majority of them are little better than the script kiddies of yore, running tools with minimal understanding of what is actually going on.
> I'd suggest only linking to them from robot.txt as pages to not be indexed
Agreed. Blocking all bots from all pages, well those that bother to listen to robots.txt. All bots because pretty much all of them are scraping for AI and similar these days, even googlebot. If I want people to see my stuff they'll get a link, and maybe they'll pass it on further, but all indexers/trainers can get stuffed. I'll likely make an exception for archive.org and similar.
Elixir's really fun to do this in because the BEAM will let you have hundreds of thousands of processes sat around doing nothing quite happily, so slow IO and such is something it's weirdly well-suited to.
This is on my photography site so I'm less fussed about them harvesting my writing, and more about causing problems for image harvesters. I'm sure they'll get some stuff anyway but at least one bot got stuck in a 400 page hellhole earlier today so I'm overall very pleased :D