upvote

points

by avastel4 hours ago |
comments
upvoteby Havoc4 hours ago|
[-]
Wouldn’t this end up flagging a lot of residential IPs due to residential proxies?
reply
upvoteby avastel4 hours ago|
parent|
[-]
The DB contains different types of proxies: - Residential - ISP - Data center

I don't include mobile proxies since they're heavily shared, so knowing that an IP address was used as a proxy at some point is basically useless.

Regarding your remark, indeed, there are several shared residential IPs, including IPs of legitimate users who may have a shady app that routes traffic through their device. That's why I don't recommend blocking using IP addresses as is. It's supposed to be more of a datapoint/signal to enrich your anti-fraud/anti-bot system. However, regarding the block list, I analyze the IPs on bigger time frames, the percentage of IPs in the range that were used as proxies, and generate a confidence score to indicate whether or not it is safe to block.

reply
upvoteby Havoc3 hours ago|
parent|
[-]
Sounds like pretty sophisticated filtering!

I’m working on a scraping project at the moment so looking at this too but from the other end. Super low volume though so pretty tame - emphasis on success rate more than throughput

I bought a 4G dongle for use as last resort if nothing else gets through. And also investigating ipv6

reply
upvoteby avastel2 hours ago|
parent|
[-]
Using a 4G dongle makes it easier to hide in the crowd indeed. Since your traffic will go through heavily shared mobile IPs, probably with thousands of users behind them, anti-bot vendors won't/shouldn't block per IP, but per fingerprint/session cookie instead.
reply
upvoteby Havoc1 hours ago|
parent|
[-]
Ah hadn’t realised it’s the NAT. I thought it’s because the IPs are dynamic and rotate too much. Interesting.

Currently planning on doing a layered approach. Cloud IPs first etc.

Interesting challenge but also trying to be somewhat respectful about it since nobody likes aggressive bots

reply