Reading a book, require attention but of lower intensity. While watching an hour long fast paced video, require a high intensity attention.
In the case of watching an hour of video, you're just there consuming what's going on. Reading a novel requires you to world build internally. I'd say that sort of attention is a much higher intensity version. Or at least it takes a lot more active involvement.
If you've ever sat for meditation you'll know that low-input stimulation can be much harder to keep your attention on, but being lost in daydreaming and 'monkey-mind' chatter is pretty effortless. Once you train in it it becomes no big deal, though. Same is true for reading novels.
My attention span went (back) up after I forced myself to read some books start to finish. It’s something you can lose, but fortunately it’s also something you can regain.
I like to think that books (novels and fantasy) are low-resolution prose, so the crux of the matter is distributed — mostly useless info — across several pages. While forums — like HN or Reddit — are high-resolution prose. I don't know if I make sense.
Well, novels are just more subtle. A good novel will get you deep into the emotional landscape of it's subject, or give you a vivid portrait of a scene that is happening, or transport you to a historical or future time. You get to embody a particular character or world, which builds your own personal knowledge and empathy. We're not just reading a collection of facts or statements. We can get lost in the beauty of a landscape we've never visited before in a novel, which is the crux of the matter, even if it doesn't seem like it.
Those sorts of novels tend to be challenging to read, but most things worth doing challenge us. If you've never read something like Cormac McCarthy's Blood Meridian, or Toni Morrison's Beloved, or Gabo's 100 Years of Solitude, or many of the other great artistic achievements in literature, you really should challenge yourself to do so. They make us better.