The probability values don’t really represent confidence in modern LLMs though, especially after RLHF and RLVR.
replySystem One says they use RLCD, Reinforcement Learning for Calibrated Decisions, which presumably has accurate probabilities as an explicit optimisation goal.
How is that different from RLVR?
replyRLVR generally upweights tokens along the whole thinking trace that led to a correct answer, whether each token was "correct" or not. RLVR doesn't train a model to output an 80% likelihood, it just trains it to produce correct answers, and not to produce incorrect ones.
replySystem One hasn't said how RLCD works, but they do say it is explicitly training models to output "calibrated" probabilities, which makes it distinct from RLVR. This is how they describe it:
> System One models are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty.
… which they could provide in their APIs but are vehemently opposed to because it makes distillation much easier, and faster.
replyYeah. Yet another reason why open-weight models are better. If I want to use the logits, I can.
reply