Sounds like pretty sophisticated filtering!
replyI’m working on a scraping project at the moment so looking at this too but from the other end. Super low volume though so pretty tame - emphasis on success rate more than throughput
I bought a 4G dongle for use as last resort if nothing else gets through. And also investigating ipv6
Using a 4G dongle makes it easier to hide in the crowd indeed. Since your traffic will go through heavily shared mobile IPs, probably with thousands of users behind them, anti-bot vendors won't/shouldn't block per IP, but per fingerprint/session cookie instead.
replyAh hadn’t realised it’s the NAT. I thought it’s because the IPs are dynamic and rotate too much. Interesting.
replyCurrently planning on doing a layered approach. Cloud IPs first etc.
Interesting challenge but also trying to be somewhat respectful about it since nobody likes aggressive bots