If CBP's systems go down, they will not process (foreign, they'll process US citizens still) arrivals [1], even with physical passports in front of them. I assume the EU ESS works the same.
"If the internet goes down, your border checkpoint is down" is not some terrifying future we need to protect against, it's the reality of the world as you live in right now.
[1]: I've had to wait for an hour, at SFO of all places, because of exactly that happening.
Why would you need internet? Document holder smartphone can cache the document for years and present it over NFC (including photo, signature, etc). Just like existing biometric passports work, but replace the physical passport with smartphone app.
The internet requirement is not there for the person presenting the document, it's for the person/system checking it.
History of entry and visas/etc could be stored on device as well
But in the real world, the systems that deal with processing people's entries already cross-reference multiple other existing databases, require internet connectivity to do so, and I think you'll have hard time convincing anyone to stop doing that.