Building a robust annotation system for both PDF and EPUB is a tough technical challenge, but definitely needed. I've been working in a similar space and recently open-sourced it. It's a local-first 3D library for organizing and reading EPUBs and PDFs directly in the browser. It's called KoreShelf on Github