Thing is, stories are not gameplay. You praise "like a novel" but oppose "like a movie". I don't see much difference, and I don't want my game to be either of those things. I want it to have mechanisms, which I interact with tactically to bring out their emergent properties and exploit them. I want it to be a functioning, complex, living thing. So the mention of NPC behavior in Ultima is the right idea: you can understand how the NPCs work, and you can make use of your understanding for fun and profit.

Those mechanisms ideally also represent some fictional world, so there's something approximately similar to a story to be found there, but that's plenty enough story already. The rest is the player actually having adventures, spontaneously. There's some use for a story in providing a goal and an ending, but beyond that it's likely to put the game on rails, like a movie or a novel.

I mean, a linear game has a charm of its own. I also enjoy point-and-clicks. There are different genres of game. But I think an interactive organic open world is the true one.

1. there's a trade-off between graphics/sound quality and story complexity. The better quality your voiced dialogue is - the more you have to pay for every additional line - so you tend to shorten it. Same with graphics - it's one thing to paint 8 frames of 32x32 sprites. It's another to motion-capture, model, texture, and process 100s of different versions of each character animations.

2. you're comparing unfairly (looking at the most complex examples from the past and comparing them to modern average). There were LOTS of very simplistic games in the past. You just don't think about most of them. Some genres went extinct because of how simplistic they were (see the dungeon crawlers where there was no dialogue or story whatsoever - just moving at 90 degree and hitting monsters) - it's the "old music was better" fallacy - you don't remember the old music that sucked.

If you compare most complex modern games they blow out of the water anything from the past. Let's say Baldur's Gate 3 or Dwarf Fortress or Minecraft compared to let's say Elite or Betrayal at Krondor.