undefined

points

by computably15 hours ago |

comments

by 3 hours ago|

[-]

deleted

by orbis-tertius4 hours ago|

prev|

[-]

Wouldn't learning a calibrated probability distribution just be a consequence of the CE loss being a proper scoring rule? So, by construction, they are approximating the distribution of the labels. Of course in practice there might be issues with overfitting the log-likelihood and the with the labels being a finite sample of the population distribution

by reexpressionist2 hours ago|

parent|

[-]

Yes, in a sense, with some important caveats. Cross-entropy will tend to yield reasonable marginal (i.e., not class-wise, not approximately conditional) calibration for in-distribution (i.i.d. data). The challenge with LMs, and neural networks more generally, is that the resulting estimators that are based on the output logits are very brittle to covariate shifts and out-of-distribution data, and that for real applications, we often need robust estimates in high-probability regions (e.g., of class- and prediction-conditional accuracy at least some value near 1) for conditional-branching decisions (by a human or a model). (ECE is the wrong metric for that quantity.)

The various decision APIs don't give you that, and would themselves need to be recalibrated with constraints in feature-representation space over the applicable reference class, as with https://pypi.org/project/reexpress-sdm/.

by JacobAsmuth15 hours ago|

prev|

[-]

For a single prediction, sure. But the model doesn't make a single prediction and then stop existing. It makes an unbounded number of future predictions and under that assumption it is reasonable to expect the model to behave optimally under Bayes rule - which has a unique global minimizer at E[(Y-F(X))^2].

So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.

by sdl5 hours ago|

parent|

[-]

The thing I had to wrap my head around is: For a "classic" classifier, the classes are fixed and the data distribution is p(x,y). The calibration is checked over this single distribution. Jev makes the tasks part of the input, so the data distribution is p(task,x,y) and calibration is averaged over that joint distribution. What follows is that while for classic classifiers, calibration brakes down if your x was out-of-distribution, it can now also break if your task is out of distribution. The big question now is: Can subtle differences in phrasing your task already lead to this? And the only way to verify calibration is having ground truth data to validate on. Also: Calibration over p(task, x, y) means calibrated on average across tasks. Even an in-distribution task can be over- or underconfident while the average looks fine.

by IanCal3 hours ago|

parent|

[-]

> And the only way to verify calibration is having ground truth data to validate on

I’m not sure that’s true in real cases.

I would expect that in distribution tasks have quite consistent outcomes with variations on phrasing. I would expect that to be much less true when tasks are out of distribution. So could we randomly adjust phrasing and see how much the results change - looking less for truth but consistency?

by coderenegade3 hours ago|

parent|

[-]

A model can be consistently over- or under-confident for similar inputs. The only way to check if it's actually calibrated is to compare against some known ground truth. I don't see any way around that.

by coderenegade3 hours ago|

parent|

prev|

[-]

In my experience, calibration is only ever in expectation, even for a classic classifier. Any given prediction is often still over- or under-confident.

by computably6 hours ago|

parent|

prev|

[-]

> It makes an unbounded number of future predictions

nit: it's trained on finitely many samples.

> So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.

Huh? That's not counter to anything I said. All I said was it's not trivially implied.