It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.

If you scrape 6TB from across the web or grab it from one place, it's still 6TB