The thing I had to wrap my head around is: For a "classic" classifier, the classes are fixed and the data distribution is p(x,y). The calibration is checked over this single distribution. Jev makes the tasks part of the input, so the data distribution is p(task,x,y) and calibration is averaged over that joint distribution. What follows is that while for classic classifiers, calibration brakes down if your x was out-of-distribution, it can now also break if your task is out of distribution. The big question now is: Can subtle differences in phrasing your task already lead to this? And the only way to verify calibration is having ground truth data to validate on.
Also: Calibration over p(task, x, y) means calibrated on average across tasks. Even an in-distribution task can be over- or underconfident while the average looks fine.
reply> And the only way to verify calibration is having ground truth data to validate on
replyI’m not sure that’s true in real cases.
I would expect that in distribution tasks have quite consistent outcomes with variations on phrasing. I would expect that to be much less true when tasks are out of distribution. So could we randomly adjust phrasing and see how much the results change - looking less for truth but consistency?
A model can be consistently over- or under-confident for similar inputs. The only way to check if it's actually calibrated is to compare against some known ground truth. I don't see any way around that.
replyIn my experience, calibration is only ever in expectation, even for a classic classifier. Any given prediction is often still over- or under-confident.
reply> It makes an unbounded number of future predictions
replynit: it's trained on finitely many samples.
> So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
Huh? That's not counter to anything I said. All I said was it's not trivially implied.