But there are bad actors - bots that try every possible urls and traversals and cross scripting and so on… I see it in our access logs. This cannot be solved otherwise than some greylisting/blacklisting on ISP level. E.g. this is so huge that we have started to block all Chinese inbound traffic. I am sorry for that because Deepseek and other good actors are probably blocked as well but IP rotation made it impossible to filter bad ones only… (and I suppose that DS uses EU or US addresses as well so they shouldn’t be blocked in full).
Indeed, they use an edge SSL Bump proxy that is already bandwidth limited and has to speak real HTTPS.
I do not focus too much on IP addresses but rather the behavior that they can not hide. LLM's can help you spot this behavior. Good LLM vs. Bad LLM. Gemini knows a few tricks but you have to coax it out of them. It seems to not want to overload people with information. That said there are some good IP lists that can be used in blackhole routing tables, nginx geo tables, etc...
I focus on content on our site so blocking part of internet with wordfence was brute but as life has only few thousands of weeks, it was the least I wanted to think about :).
Would you mind to share a source of good IP lists pls?
LLM's can help you implement iprange into IP list building scripts to keep the lists smaller and more efficient.
[1] - https://github.com/firehol/blocklist-ipsets/
Before pushing such things to production consider first testing in dev/qa then staging or some canary environment. I use those IP's to determine who gets into a moderator review queue so worst case it just makes more work for me. Consider using those in conjunction with whatever rate limiting capabilities your web server provides first so it is not a hard block in the unlikely event a real customer shows up on one of them.
I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.
The bots may have layer 7 signatures that make them "easy" to block, but I'm dealing with COTS software with no ability to make changes to the stack. Even if I could by the time they're blocked the work is done and they're never coming back anyway.
I don't think I'm seeing the LLM bots in the scraping on this particular site. This site's data is highly sought-after by data brokers, legal services advertising, debt collectors, etc. I tend to think I'm seeing homespun scrapers. This site has been up for 20+ years and the scrapers seem to be well-adapted to using the HTML as an ad-hoc API for the data. The difference in the last few years has been the availability of vast numbers of residential proxies to mask the traffic.
Some of the things that seemed like they'd cut out lazier bots/scripters involved browser fingerprinting that I didn't want to yet have to deal with disclosures or approval for. But also even some of these browser fingerprinting things were disable-able via flags when you start up Chrome or whatever.
I think what would work overall is to track all requests for a pattern of page access and start blocking other systems that are part of that pattern. But that also wouldn't account for like... landing on the front page of HN, which happens not infrequently.
All of this is aside from per-IP or per-user proof of work. Whenever someone lands on a more expensive page I make the browser do proof of work and store it in a cookie, unique to the IP or user. (Signed in users don't have to do proof of work). This is a perfect solution except for the IP rotation I'm talking about above.
edit: ah also, I don't do SSL termination so can't spot the SSL configuration. But Cloudflare does. And all of this is even after Cloudflare paid bot traffic systems are in place.
edit 2:
> tiny percentage of bots will still get through but the load becomes much more manageable.
The load is not the problem. It's only at most a few bot hits per second. I'm just having trouble removing bot attribution from view counts.
Youtube wont count a view until you watch for some percentage of the video. Maybe consider something similar.
Just my two cents, you'll figure out a good solution.
I love sharing methods with the botters so they step up their game, thus forcing me to step up my game. Everything I use does not require javascript, cookies or authentication. I save authentication as a last resort which means I lost that round.
Botters are welcome and encouraged to test their setup on my silly hobby sites. Here's one of them: [1]
[1] - https://altchan.org/sh/33
P.S. These are the simplest crude methods for the obvious bots, let your LLM to help spot the fakers. pre-2026 Safari will not work. Do not visit image boards with unpatched browsers.
i heard hcaptcha is good
I was told that I could only lead a horse to water but I could not make it drink. Turns out that was not at all true. I had to find which horse was being the ass and re-educate it so the yearlings could get to the water.
That said I think most people here can at very least have discovery conversations with LLM's to explore their options. They can share their architecture, infrastructure design and have LLM's walk them through gathering more information. It may be bumpy at first but I suspect some of them will be able to give the bots a few flat tires. I have confidence that necessity will drive innovation and they most likely will not even have to spend a dime.