It is a different problem if we pick two sets from the same data distribution, A and B, and first we have a score on A, then on B. Here we re-run on precisely the same set of Terminal Bench 2.1 problems. It may be that results are so random between runs that each single task has the same probability in a Bernoulli distribution. But more likely, many problems are easy (i.e. each run will solve them consistently), many are too hard (i.e. no run is going to solve them) and only a fraction is somehow in between.
Maybe there is some good trick to find a proper distribution, but to my knowledge, we would need to run it at least two times on TB2.1 to get any more educated estimates. That said, I am open to new ideas.
That said, I consider frequentist probability a dirty trick, and that Bayesian is the proper way of doing things (vide David J.C. MacKay" Information Theory, Inference, and Learning Algorithms" and Cam Davidson-Pilon "Probabilistic Programming & Bayesian Methods for Hackers" https://www.inference.org.uk/itprnn/book.pdf, https://dataorigami.net/Probabilistic-Programming-and-Bayesi...).
I'm not that well versed in statistics, but a standard box plot is probably the best alternative
They're really reaching here. There's a 95% chance it's 1 and 5% it's 0, and we don't know which, so it's 0.95. Like saying the probability your uncle died of a shark attack is either 0 or 1. But in reality I don't know anything about your uncle so it's the average probability that someone picked randomly died of a shark attack, until I know it about your uncle.
> Like saying the probability your uncle died of a shark attack is either 0 or 1.
For a frequentist, it is. If you want the Bayesian interpretation of probability, you must calculate the probability that the interval you ended up getting contains the true value, and you can easily end up with a value different from 95%. You can’t just stop at “it was drawn from a pool of intervals of which 95% contain the true value” since you have more information than that (the numerical bounds).
Saying there's a confidence interval for run-to-run makes no sense, from the way I understand it, because each run could have a result that's all over the place.
Let’s say that the “true” distribution of the data has mean μ=100 and standard deviation σ=15, but we don’t know that.
95% confidence interval for μ = “if we repeatedly draw N samples from the true distribution and compute a confidence interval every time, 95% of those intervals will contain μ.” That’s all that the definition of a confidence interval guarantees. It does not follow that if we take one of those intervals, it, specifically, has a 95% chance of containing μ. For a frequentist, that’s a meaningless statement (both the interval and μ are fixed so there’s no frequentist probability about it); for a Bayesian, there is no guarantee that that probability is 95%. 95% is instead the probability of “sampling data that will happen to generate an interval that contains μ”.
95% Bayesian credible interval for μ = interval that can be interpreted as having a 95% probability of containing μ, generally obtained by computing the posterior probability density distribution for μ and finding an interval that encompasses 95% of the probability mass. Conventions include highest-density intervals (HDIs), which are obtained by making sure that the PDF is equal at both bounds, and equal-tailed intervals (equal probability mass before and after the interval). With enough samples, it may become arbitrarily narrow (“we are very sure of the mean”), despite the standard deviation of 15 that is built into the “true” distribution that we are estimating, and a Jeffreys prior will happen to make it satisfy the definition of a confidence interval as well (https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... ).
Posterior predictive distribution = taking into account the uncertainty on both μ and σ, distribution of samples that would be obtained by sampling from N(μ, σ) (which, because of that uncertainty, is a https://en.wikipedia.org/wiki/Compound_probability_distribut... but may have a convenient closed form https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conju... ), from which we can likewise extract a 95% interval.
https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... (+ Cauchy example below)
It’s like with test accuracy. Test accuracy is the pre-test probability that the test will give a correct result. But once you have a positive or negative result, which way it turned out plays a part in computing the predictive value. Likewise, once you have computed the interval, the specific bounds you ended up getting can affect the plausibility that they contain the true value.
The conclusion they come to is "it is possible to do better in the individual case by taking into account evidence from the sample that the confidence interval method throws away". That means the confidence interval is inefficient, not incorrect.
But once you have picked one, and you know its bounds (say, [12.1471, 13.8264]), then it’s fallacious to make the post-data reasoning that “because it was picked at random from the set of 90% confidence intervals, it has a 90% chance of containing the true parameter”.
Again, it’s like with medical tests. If a test has 90% sensitivity and 90% specificity, it has 90% accuracy (it will, in 90% of cases, produce a result that matches disease status) – a pre-data statement on the test result (/ on the confidence interval that we will compute). But it does not follow that, if you screen an asymptomatic patient with low prior probability of disease and get a positive result, they have a 90% chance of having the disease – a post-data statement on disease status, given the test result (/ on where the parameter lies, given the interval).
> The conclusion they come to is "it is possible to do better in the individual case by taking into account evidence from the sample that the confidence interval method throws away". That means the confidence interval is inefficient, not incorrect.
It means you know in the individual case that the specific confidence interval does not in fact have a 90% chance of containing the parameter.
Correct, which is why it's crazy to consider a Jeffrey's credible interval as valid, but not a confidence interval.
Perhaps the confidence interval might not be valid under a frequentist interpretation, but the same math can be valid under a Bayesian interpretation even if the person publishing didn't mean it that way.
Does that mean it's valid? Well, the confidence interval is as equally valid as a lot of credible intervals, which is why the credible interval is not some magical better alternative - it's mostly just a choice of how you interpret it, it is not a different procedure that produces different numbers.
The confidence interval is “valid” for what its definition is. But:
- saying “there is a 95% chance that this interval contains the true value because it’s a 95% credible interval” is valid;
- saying “there is a 95% chance that this interval contains the true value because it’s a 95% confidence interval” is not.
It’s very much possible to reach a correct conclusion with invalid reasoning. As Jaynes put it:
“As we stress repeatedly, from a false proposition – or from a fallacious argument that leads to a false proposition – all propositions, true and false, may be deduced. But this is just the danger; if fallacious reasoning always led to absurd conclusions, it would be found out at once and corrected. But once an easy, shortcut mode of reasoning has led to a few correct results, almost everybody accepts it; those who try to warn against it are not listened to.”
> Perhaps the confidence interval might not be valid under a frequentist interpretation, but the same math can be valid under a Bayesian interpretation even if the person publishing didn't mean it that way.
Right, but if you get that by happenstance and not from the fact that it’s a confidence interval, the latter isn’t really getting you anything.
Per my first link: “One of the misconceptions regarding the relationship between Bayesian inference and frequentist inference is that they will lead to the same inferences, and hence all confidence intervals can simply be interpreted in a Bayesian way. In the case where data are normally distributed, for instance, there is a particular prior that will lead to a confidence interval that is numerically identical to Bayesian credible intervals computed using the Bayesian posterior (Jeffreys 1961; Lindley 1965). This might lead one to suspect that it does not matter whether one uses confidence procedures or Bayesian procedures. We showed, however, that confidence intervals and credible intervals can disagree markedly. The only way to know that a confidence interval is numerically identical to some credible interval is to prove it. The correspondence cannot — and should not — be assumed. More broadly, the defense of confidence procedures by noting that, in some restricted cases, they numerically correspond to Bayesian procedures is actually no defense at all. One must first choose which confidence procedure, of many, to use; if one is committed to the procedure that allows a Bayesian interpretation, then one’s time is much better spent simply applying Bayesian theory. If the benefits of Bayesian theory are desired — and they clearly are, by proponents of confidence intervals — then there is no reason why Bayesian inference should not be applied in its full generality, rather than using the occasional correspondence with credible intervals as a hand-waving defense of confidence intervals.”
> which is why the credible interval is not some magical better alternative - it's mostly just a choice of how you interpret it
You don’t get to choose to interpret a confidence interval as if it had the meaning of a Bayesian one. If you want to be able to interpret a computed interval as “there is a 95% chance that this interval contains the true parameter value”, only the Bayesian procedure guarantees it, and it can very well produce different numbers. So it is a better alternative in that sense.