A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
replyCloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
replycloudflare routinely denies my human-piloted browser now, on many sites.
replyyou're not a bot thats irrelevant
reply[dead]
reply> I think that nobody would care if I use wget or curl for few pages
replyIf only you were the only one doing it...