upvote
Sorry, I wasn't clear. In order to know you have an up to date copy of the page you need two pieces of information:

1. When the page itself was last updated

2. When the crawled copy was last updated

In order to get an accurate date for 1, you have to crawl it, and if you are crawling it you might as well use that copy you just crawled.

The missing here is that for pretraining AI models should accept a cut off date and not worry about being perfectly up to date. Keeping things up to date is more useful developing internet search engines for grounding.

reply