For the ultimate in non-basic vocab, try Gene Wolfe's Book of the New Sun.
The English translation of Metro 2033 is amazing. You can feel the stylistic differences originating in the Russian language, and it's a very evocative horror story with slightly longer sentences and richer language than ordinary books.
Both the Babel books and Metro 2033 (especially) were inspirations for me, writing my novel. And Wolfe is one of my literary heroes, one of the most amazing writers I have ever read. Every one of his novels is rewarding, often through not just language but complexity of plot and narrator. He uses unreliable narrators a lot; beautifully, in one of his series the narrator has amnesia and cannot remember what occurred the previous day, so writes notes to himself. (Soldier of the Mist. Also rewarding once you figure out the name and location references.)
An older classic: Mervyn Peake's Titus Groan is extraordinary in language, but also in the rhythm of sentences. Definitely one written to be read with a voice, not just internally.
Perdido Street Station by China Mieville has dense prose. Wonderful book, very rewarding to read.
But simpler prose but it can be just as rewarding when it is written for a purpose, and I'd love to recommend to you Lavie Tidhar's Osama. It is written in noir style, and so the sentences are far simpler, sparser. Yet there are multiple styles in this book and the kind of prose tells you a lot about story or the place in the world. I am writing vaguely because I don't want to give spoilers. He is an excellent writer and he chose a specific style in this book that he executed very well.
Moby Dick is unexpectedly stylish. You'd think a story about a whale would be more genre-ish, but the writing is very fun.
Then there are heavy hitters are writers like Gabriel García Márquez, Tom Wolfe, and Virginia Woolf.
James Joyce is the end level boss. Ulysses is readable, but by the time you get to Finnegan's Wake it's hard to imagine anyone getting more abstract.
But is heaviness truly deplorable and lightness splendid?
The heaviest of burdens crushes us, we sink beneath it, it pins us to the ground. But in the love poetry of every age, the woman longs to be weighed down by the man's body. The heaviest of burdens is therefore simultaneously an image of life's most intense fulfillment. The heavier the burden, the closer our lives come to the earth, the more real and truthful they become.
Conversely, the absolute absence of a burden causes man to be lighter than air, to soar into the heights, take leave of the earth and his earthly being, and become only half real, his movements as free as they are insignificant.
What then shall we choose? Weight or lightness?
Parmenides posed this very question in the sixth century before Christ. He saw the world divided into pairs of opposites: light/darkness, fineness/coarseness, warmth/cold, being/non-being. One half of the opposition he called positive (light, fineness, warmth, being), the other negative. We might find this division into positive and negative poles childishly simple except for one difficulty: which one is positive, weight or lightness?
I cannot fathom how people enjoy that book. I appreciate the writing (and love other CM books) but Blood Meridian felt like a gratuitous ordeal to me. I tried watching lit professors lecture about it too but I still just don’t get it.
Umberto Eco and Ursula Le Guin I agree on.
Another author that comes to mind is Zizek. His books could easily be reduced to a boring 2 page essay, but instead they are taken on a wild ride through cultural and theory and philosophical history and virtually encoded in philosophy jargon in order to absolutely reek of intellectualism. I'm 0% ashamed to admit openly that keeping up with him as I read makes me proud of my brain and attention capacity way more than I actually agree with his specific political philosophy in a work.
I will give you approximately twenty delightful novels where the prose and the stories are both excellent: Patrick O’Brian’s Aubrey–Maturin series.
I avoided these books for the longest time because I didn't believe I was the kind of person who would read “books about tall ships.” But so many people recommend these particular books because of the masterly writing that I tried the first, Master and Commander. Immediately, I was taken by O’Brian’s writing: vivid, lively, and engaging, while also utterly precise.
Give that first novel a try. It’s not easy, but it is oh so rewarding.
Edited to add some examples from the first novel:
“I have never yet known a man admit that he was either rich or asleep: perhaps the poor man and the wakeful man have some great moral advantage.”
“Where there was no equality there was no companionship: when a man was obliged to say ‘Yes, sir,’ his agreement was of no worth even if it happened to be true.”
“He held up two fingers, in case a landman might not fully comprehend so great a number.”
“But you know as well as I, patriotism is a word; and one that generally comes to mean either my country, right or wrong, which is infamous, or my country is always right, which is imbecile.”
Nabokov? His protagonists have very special styles of speaking. I'd recommend "Pale Fire" (which is also very funny and has an interesting hypertext-like structure) as a starter.
Thankfully, the story is as strong as the prose. Excellent book all around.
E.L. Doctorow is another such example. Try The Book of Daniel, Ragtime, Billy Bathgate, Loon Lake.
In the scifi genre (aside from the Shikasta series already mentioned), M. John Harrison is worth a look. He’s got a long bibliography but Light might be a good starting point.
Umberto Eco: The Name of the Rose, Foucault’s Pendulum.
Some of Ursula Le Guin’s more serious books might fit the bill, like The Dispossessed or The Left Hand of Darkness.
Someone already mentioned Mervyn Peake, he definitely fits this category.
No need to always flex with some highbrow stuff to find a masterful style.
He writes sentences around how they're, eh, delivered? Can't think of a better word. Lots of footnotes and clauses and parentheticals. And the timing of his words make the jokes.
I would absolutely call him a landmark writer, but I wouldn't call him lit fic in the publishing sense.
I'll be honest: my (for now quite small) experience with the /lit/ canon has been very hit-or-miss for now. It feels like Literature with a capital L is something only ivory tower academics and New-Yorker or Parisian critics have the right to stick to works and they decided that comedy cannot be part of it. They're also very quick to ignore small and larger crossings into purple prose territory when it pleases them.
Similar in a sense to the music world, where the crowd that decides what is Serious Work^tm is enshrining contemporary classical, post-modernism and performance art instead of sifting through the (admittedly lackluster) popular music world that has produced genuine pearls often rivaling, sometimes eclipsing their dominion.
------
End of rant, sincerly sorry.
tl;dr: yes, Pratchett isn't literature in any sense of the word (nobody claimed so, anyway) but your scale may be a bit too logarithmic, putting him and much worse stuff in the same basket.
> He writes sentences around how they're, eh, delivered? Can't think of a better word. And the timing of his words make the jokes.
You put the words in my mouth, very accurate way of describing his style!