Are these novel coding tasks, or could they have ended up in the training set?
replyDoes the number/frequency of safeguard stops affect the measured speed?
Private benchmark from private repos so yeah, novel.
replyI think I included the Opus 4.8-backoff runs in the speed number and they were on average faster than the actual Fable 5.1 runs, so the speed result might be even bigger if I excluded the Opus 4.8 runs.