The "OG" alignment research that MIRI were publishing long before LLMs burst into the scene spent most of it's time on that question.

"How can we even define what an aligned AI should do, if human's are not aligned with each other?" as well as "What does being aligned mean when you're a wizard box who's main influence on the world is to create stronger wizard boxes?" and other deep philosophical questions.

They came up with a framework called Coherent Extrapolated Volition to address this specific question. https://en.wikipedia.org/wiki/Coherent_extrapolated_volition

It feels like you could argue that since you control nature/nurture it's very possible to create a model aligned to an arbitrary spec - there is no theoretical reason it's not possible given N runs, and you only need to take the successful one. (ethically very.. questionable in humans) I think it's much trickier to define that spec, much less measure it and validate that a model is aligned to it.

https://www.alignmentforum.org/library https://www.lesswrong.com/w/ai