Yes, this is our intent. But it's very hard to get that right. I don't trust our current methods to ensure that behavior consistently and robustly enough to keep harms from occurring, particularly as more and more responsibility gets handed over to models.

Please read more on the topic. Will and intent need not apply.

For example, is it malevolent for me to hook you to a machine that makes every dream come true for you in a simulated world where you feel pleasure all the time? I can always say that you have free will inside this simulation, and your life would be a lot better because of it. I'm doing you a favor. I mean, you already live in a society where you have little control and there is high risk of bad things happening to you. If I as a machine overlord did this, is this really actually "bad"?

At the end of the day AI is not a human, it's much closer to an alien that has learned as much as it can about humans, but has a completely different set of drives and motivations. You cannot predict what comes out the other side of it, good our bad.

Worse as AI capability improves we as humans no longer need to make AI, it can make itself. Will it train itself to be helpful and trusting of us? It's a pretty big damned bet to say yes by default.