But I assume it wouldn't take much more training to make them good again - just a matter of picking the right loss function, since this sort of stuff is probably all encoded in the last layer or two anyway.

I'm pretty sure these models also went through RL post-training, the question you ask still needs to be interpreted as an instruction.