> We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation.
replyCurious the reason that a grader would wind up preferring this. Honest mistake in the grading criteria? Or some spillover from criteria designed to cause underperformance in, say, competitor model creation? It was not very long ago[1] that Anthropic announced and then walked back their strategy on this. Would not surprise me if OpenAI has been doing it silently all along.
[1]: https://mlq.ai/news/anthropic-reverses-secret-policy-that-si...
You could easily imagine structural issues driving this. It's pretty unsatisfying to get refused for something that 'should' be handled and also to be dinged for doing something dangerous. You see this stuff in human organizations all the time.
replyI'm pretty sure the typical "Claudisms" are influenced by RLHF as well, so if it can have that effect, why not these...
reply