You've read many stories set in all the settings you mentioned. You have never read a story in which the fundamental shape of space-time is two time dimensions and two space dimensions before, unless you have also read Dichronauts. This is the supplementary material to the novel, which is mostly not in the novel and is not the story itself, just the background: https://gregegan.net/DICHRONAUTS/01/World.html You don't need that provided for something set in the Jazz Age, or a fantasy story explicitly based on myths that had been floating around for centuries, or a historical fantasy. Someone could write some equivalent, but you don't need it; it's already loaded into your head. That's the point.
>With this sort of story, as a reader you are putting all your chips on the setting. Going in to this sort of work you should expect the conventional measures of a story to at least take a hit, and in many cases a fairly large one. And of course, if that's not what you want, then you're not going to enjoy it and I have no problem with that.
Where I would challenge that as it relates to TINAMD is I am not sure it fully succeeds even against this basic bargain. By contrast I would note Annihilation which is exactly as you describe, light on characters and plot and entirely about setting, and I think it sticks the landing on those terms in ways this book could have. But still, love its premise, love the traction its getting and I think the healthier way to engage with it is to cheer it on for its successes, which are significant.
His characters do tend to be a little flat, but I think I almost always found them compelling. His books tend to be a physics or mathematics primer, wrapped in a pretty thin plot, but as soon as you poke at that plot-wrapper, most of the time some pretty good social commentary comes steaming out.