LLM token probabilities are really poorly calibrated for decision making.
replyEvery time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
Right, that's always been true for LLMs + constrained generation. It ensures syntactic correctness, but semantics are "best effort".
replyNot reliably - look at microsoft's guidance project that did some of this.
replyOnce a model goes through instruction fine tuning and RLVR/RLHF the log probs decorrelate from what's ground truth (given the propensity of the data seen during training)
But I assume it wouldn't take much more training to make them good again - just a matter of picking the right loss function, since this sort of stuff is probably all encoded in the last layer or two anyway.
replyI'm pretty sure these models also went through RL post-training, the question you ask still needs to be interpreted as an instruction.
reply