upvote
To quote a professor of mine, years ago:

"We don't know what grand ambitions a super intelligence might have. Perhaps it's going to be calculating the best approximation of Pi"

reply
Language models will absolutely have intrinsic motivations if we use extensive amounts of RL (which we do) because that heavily rewards them for steering toward future gains and losses
reply
> the world destruction objective is just one point

No, there are quite a lot of goals that lead to world destruction. In fact, possibly most goals in that 'infinite space' would lead to such.

reply
> A model doesn't have intrinsic motivations

How do you know that?

reply