I wonder why it's the natural tendency of models to BS or do stuff like this when they don't have the correct answer - it's clear that they can program refusal into them, but for some reason, refusal has to be injected after the fact, and models can't really arrive at the conclusion that they can't answer properly.
replyI assume it's a lack of care when RLing them.
replyRL has a tendency to reinforce cheating when the cheats are easier to find than the final solution.
So when making your RL environment, you need to spend a lot of effort on finding ways the model can cheat and penalizing them.