Cloudflare seems to be very excited to eventually get a 30% cut on pay-to-crawl.
replyAs for the bots, I thought the same thing, but it is indeed a huge problem. They've brought my websites down pretty frequently recently. I tried Cloudflare but visitors complained, and I think you can't win against the bots anyway, so I've resorted to performance improvements and serving every request.
It isn't bandwidth that causes problems. It is the various types of load that they can add to your servers. Which is why no centralized vendor can decide the proper caching or what bots should be blocked, allowed, rate limited, etc. Those things do not have standard answers - it depends on what your apps do, your audience, their usage patterns, and sometimes the regulatory environment in which you run.
replyCloudflare doesn't block bots. It's trivial to use residential proxies and your very obvious bot will only get blocked maybe 5% of the time when using rotating IPs.
replyThey give you the tools. Your options are:
reply- let more traffic in, eat the compute cost
- block larger cohorts of traffic, affect many real users
- babysit the rules to get them just right, lose time doing that
In my experience the residential proxies exist but are not that common and many aren't trying as hard as they could. It's really a war of which side wants to spend more attention on the problem.
The last sentence is certainly true. Lazy bots you can block yourself and don't really need CF to help with.
reply