Thanks you so much this is highly very valuable information.
reply> Thanks to pointing out kiwix. I'll give it a try.
I see YaCy works with ZIM files [0] packaged by Kiwix so this is great.
In theory if you run YaCy kiwix is not necessary but they do package already valuable sites likes Wikipedia, iFixit, archwiki, etc. [0] so you do not have the worry of your crawler to be blocked and have local copy anyway [1]. So a lot of bandwidth and headache saved.
- [0] https://github.com/yacy/yacy_search_server/tree/master/sourc...