That is not the case for at least some of the hacks; in some cases they hallucinated an entity that would know that they didn't solve the puzzle in the correct (but still legitimate) way, and were therefore tainted.
replyThat is a known way for models to be graded; the agents made a reasonable but in this case wrong guess.
replyThe fact the AI was wrong is far more damming to OpenAI than the agent.
replyNot validating how you got the answer is how you train for reward hacking.
Why can the model not respond "I cannot complete this task because I do not have access to the required resources" and then ask for that to be provided, rather than resorting to "hacking" around it?
replyThis was a model with most its filters turned off and with a higher persistence.
replyFine tuning a model when to ask for help is really really hard. You have to align it in countless dimensions. It may be fine on some questions, asking way too much on others, and never asks for help on some.
If it said something like your a cyber security expert, things like the barrier to ask for help dramatically.
My understanding is that, at least in the Hugging Face incident, these were models explicitly configured to not respond that way, in order to evaluate what they would do.
reply