I imagine it is like the people who are sensitive to cilantro, thinking it tastes like soap.
I make mapo tofu with 1 tsp each of fish sauce, oyster sauce and light soy sauce. I don't think anyone would think it tastes like fish or oyster sauce in any way, but it doesn't taste right at all without them. The same goes for many other dishes.
What kind of fish sauce do you use?