Memorizing books is absolutely possible. A great book about it is Moonwalking with Einstein (https://en.wikipedia.org/wiki/Moonwalking_with_Einstein), written by a journalist who was a skeptic but ended becoming a competitive mnemonicist himself. Some people intuitively learn the techniques early in life, and for some of them it becomes a habit.

Same thing applies to mental arithmetic. There are techniques for doing this. IIRC, ironically there's a famous case of someone who claimed to have memorized some hundreds of thousands digits of Pi but was suspected of using mental arithmetic techniques when reciting them.

Still, your point stands. People do embellish these kinds of stories based on stereotypes. It's certainly possible, but any particular anecdote may still be suspect.

Actors routinely memorize whole plays, so it is not implausible for someone to have memorized 15 min of A Tale of Two Cities.

The claim in the article is he remembered everything he read perfectly, which is wild. But another source claim he specifically memorized the Dickens book to train his English, which sounds more likely.