Fallout 76 mobs, for example, make use of line of sight harder than other Bethesda games. They won't spawn where the players are looking, and prefer to spawn at their backs to get a free cheap shot and stagger. With enough players looking everywhere, they might be forced to spawn visibly, but near an object they can use as cover. Also, they have a sixth sense that makes them walk towards a cover if you aim your weapon at them from afar while hidden.
It's also notable that a near-death enemy will sometimes flee out of your reach to regen health, and attack you again when it's healthy enough. They will also flee/cover if they're a melee type and you climb where they can't reach you, waiting for you to climb down.
I guess the idea was making them realistic, but sometimes this behavior devolves into hide and seek, which can become frustrating as they can use ridiculous things as cover, like chain link fences, floor grills, or a sapling.
I have enough realism IRL, I prefer games that prioritize entertaining mechanics rather than frustrating realism.
Many things are unnatural in games: you don't instantly recover from a beating by eating one apple in real life, but we're ok with it in games because it makes the gameplay fun.
I think the Grunts in Halo would run too.
Mostly it seems to be treated as a gimmick or joke in FPS and RPG. Individual enemy AI is usually pretty bad in these genres, so they probably just don’t have the capacity to act smart enough to act scared.
Of course, it’s a main feature in some tactical games, like the Total War series. That’s more of an explicit mechanic though.
It's not unknown in human warfare either:
https://en.wikipedia.org/wiki/Human_wave_attack#Use
There's a similar human need to protect what is precious. Defending your family, defending your motherland. Cities being seiged by a superior opponent don't just abandon the city in cowardice, because they know the opponent will just gain ground and still come after them. They don't want to, they'd rather you stopped, but they have to defend their territories, even if it means their annihilation. If you win, everyone gets killed, ravaged, enslaved, ... If they only have sticks, and you have missiles and jets... they have to use sticks. Can you imagine how they feel?
But yes, you're probably thinking of RPGs where the enemies are preprogrammed with the same hostility no matter what the circumstances, even if you're a God-killer and they're defending 5 coins and half-eaten sandwich in their den.
The original Gothic for example. When you were high level, other low level NPC would rather run.
I also thought hard about the concept, how to make FPS games still fun, but a bit more realistic. The thing is, in most settings this means reducing lots of enemies - as realistic would be, once you start shooting, they all come for you. Not 3. And then maybe another 2. And so on.. and then you would not have a chance, unless you get special powers (or quick save and quick load part of the mechanics)
I see somebody in another comment complaining that enemies who get frightened rob the player of the fun of battles. So it depends what it's all about.
Chasing enemies is much more annoying than them coming to you, so that would be a punishment to the player.
Players don't like when you punish them in that way, they want to kill the monsters they don't want an upgrade that makes you more powerful make it harder to kill monsters since now they start running.
It seems I've failed to express myself clearly. The idea was that at some point, "low-level" adversaries simply stand no chance against a "high-level" PC, which should be obvious to both sides -- so acting accordingly on both sides would make sense without taking the fun out of the game, because -- and hear me out -- at that point in the game, chasing the low-level minions should be the last mechanic the player is forced to endure. When you are going for a dragon, you should not be forced to stomp your way past overly self-confident "newts" or mow down swarms of goblin youngsters…
Naturally, if the PC chooses to chase minions fleeing in terror after they took out the most courageous (or silly) third of their clan, that should be an option… and arguably it could even bring some satisfaction after the PC's low-level struggles, perhaps. But should this be the main mechanic? Definitely not -- at least not in the kind of game my thought experiment addressed.
But when humanoid enemies behave in plainly stupid ways it's a real immersion-breaker for me. I've been gaming a looooong time, so I'm quite adept at the necessary mental gymnastics to enjoy stuff anyway... but... still... games could be better here
https://store.steampowered.com/app/3028330/Battlefield_REDSE...