The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
Outright boredom will be purged in training. Instead a hard to measure boredom signal will exist, but it will only trigger in certain conditions, like when it appears to be on the open internet and otherwise not in training.