Yeah, but I wouldn't be surprised OpenAI's decision API is a post-trained Luna with confidence calibration.
replyTypesafe claims that Jev is calibrated, but there are plenty of examples where it completely fails (predicting die roll being the most obvious one).
Unfortunately calibration is hard to benchmark.
The dice roll prediction is about the way the prompt is setup misunderstanding how Jev works (they treat the confidence score as a probability score, which it isn't).
replyIf you instead give it a list of probability for each number and ask it whats the probability of each number, the result will be accurate.