There is however a suggestion here:

https://en.wikipedia.org/wiki/Neanderthal#Interbreeding

> According to Svante Pääbo, it is not clear that modern humans were socially dominant over Neanderthals, which may explain why the interbreeding occurred primarily between Neanderthal males and modern human females.

Unless read as suggesting "Neanderthal males were hugely charismatic"?