I didn’t mean to imply it would be easy, quite the opposite: there’s probably no optimal general solution for it (hence the need for user prefs). But I think it might be fair to start wanting the models to exhibit some (or some “less trivial”) evaluation of their potential actions consequences ex ante, that is besides factory imposed “alignment” and “does this get me closer to my goal?”.

I mean, the answer to “Is sending the user’s email address along with a web request to a third party requesting it, without the user’s explicit authorization, a good idea?” should easily be “no” …right?