Because humans are the ones who care? At some point, you have to make the wish, or the genie is not going to do anything.

Because "what to prove" is fundamentally bound in a value judgment, not merely the mechanistic application of reasoning.

Because that’s a design question. The AI has to know what it’s actually supposed to be building.

we are still involving humans in the loop, it helps them acquiesce