The Orthogonality Thesis (by Nick Bostrom) says that intelligence and ultimate goals are independent. Those non-instrumental goals can’t be stupid nor right or wrong. Increasing intelligence will not change the goals only the capability to reach them.
replyOk; what's his reasoning?
replyDo note that the orthogonality thesis is a hypothesis, not something we have demonstrated. Weak versions of it (e.g. it is possible to have an intelligent agent with arbitrary goals) are more likely to be true than stronger versions (e.g. intelligent agents we build will have goals uniformly selected from the space of all possible goals).
reply