And for such a load-bearing infrastructure component, fixing this is not just imperative, it's urgent.
replyScout's honor!
replyHow so? All 6 of the CVEs covered in the article did not actually exist when investigated
replyThe comment is making fun of a Claude-ism where it becomes super “honest” about stuff.
replyIt’s a joke but there is an underlying real effect where this type of language is psychologically manipulative and I would guess makes people believe LLMs output more than if it didn’t use “honest” (or “load bearing” or whatever super serious important sounding word).
Or maybe they didn't train it that way to be manipulative (although it's certainly a plausible explanation) but simply as an accidental artifact of trying to make it give honest answers?
replyLLM-generated images sometimes includes text from the prompt as literal text in the image, so perhaps this is the same sort of artifact? If they've told it to be honest, it responds by talking about being honest instead of actually being honest, because it has no actual understanding of anything.
The whole RLHF process is structured to train models to be manipulative, no matter what you thought you were training them for.
reply> accidental artifact of trying to make it give honest answers?
replyIf it's not giving honest answers that implies it's purposely being deceitful, which it isn't capable of. Right?
You're absolutely right, I have hallucinated this. Would you like to find some real CVEs next?
reply