We don't even need a standard for reasoning levels.
replyJust accept whatever the provider names as "medium" and compare the accuracy between models vs the time it takes to complete a task.
You don't need to directly compare the same reasoning levels. Today you can compare Sol on Low vs Haiku on Max, for example, but you can't look at GLM 5.3 High, only Max.
replyFor some tasks I genuinely feel Opus 5.5 does better on low than medium or high. The trouble, as always, is figuring out which.
replyEdit: I realize after commenting that you're not suggesting medium is useless, but just arbitrary and differs between models. Personally I don't want to directly compare mediums between models, I just want to be able to compare across effort levels.
i run opus on medium, sonet on high, and haiku on high for implementation based how complicated is
for planing and specing and review opus is on high usually, except if if the task is contained.
replydid not try to run it on low, will have to try it.
Opus 5.5 on low is good if you are relatively sure a given task is straightforward and time spent gathering background information is purely wasted or possibly even detrimental. Opus 5.5 low seems great at doing the most obvious thing based on the prompt without thinking much about it.
replyI agree - Opus 5.5 on low is a really great model if it has some direction and the tasks aren't complex.
replyIf the tasks are complex cranking Opus up to high or xhigh and getting it to delegate to Opus low subagents is pretty effective and much faster than running everything at high.
A graph of time (or cost) per task against success rate does it. Just include all the different levels.
replyThis exactly. The three interesting parameters are total time to answer, total cost, and output quality (for any given purpose).
reply[flagged]
replyThat's the beauty of the line charts; the labels are arbitrary points on the line. You can easily compare the series visually.
reply