Let us know when you have Qwen vs Qwen comparison stats. As long as there's not a regression, that'd be awesome.
reply4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.
replyDepends on where you sit on the binomial curve. At p=0.04 for n=250 4% points would not be within margin of error.
reply[dead]
reply[dead]
reply>> We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios
replyOkay but the parent said real-world usage, presumably meaning coding tasks.
We have a whole bunch of complex evals that Haiku 4.5 passes. That doesn't mean it is a good model for coding.
They literally stated in their first sentence that it was coding tasks.
replyYes, these are coding tasks in the embedded systems domain (I mentioned Rust and C).
replyHow much does it score though? 0% would be 4% less if Opus was at 4%. Unless you mean relative fraction not percentage points - but people usually mean percentage points in such situations.
reply0% is not 4% less than 4%, that would be 3.84%.
reply0% is 4 percentage points (pp) less than 4%.
[dead]
reply