Polygamy is not concubinage is not having extramarital affairs
Making a promise to someone of exclusive sexual intimacy and then breaking that promise is recognized near universally as a moral failing. He knew such an affair would hurt not just shareholders financially but people emotionally.
Maybe the guy isn't totally morally fiberless, but I think most people would agree that such behavior is a (non-trivial) moral failing.
Particular notions or feelings of right and wrong that you may have because of prevailing norms in the particular era and location which you live or grew up in don’t necessarily have any bearing on what is moral or not.