upvote
The real issue with coding harnesses is that you get a cache miss any time you switch models. I might have some easy task (e.g. "fix lints") that I could delegate to Terra, because my session has been using Sol up to this point, I'd pay a full uncached prefill if I routed it to Terra. It ends up being cheaper in practice to just stick with Sol.

Spinning up a fresh session would be a reasonable choice in the above case, but sometimes you still need the retained context.

reply
Yes but you are missing the point. It's the case where Luna is not good enough, and Sol is too much. The problem is that on benchmarks Luna xhigh is typically comparable to Sol low in terms of capability. So where does Terra fit in?
reply
World knowledge, factual recall without grounding, more complex instruction following, etc.

Benchmarks don’t really tell you much. For API best will always be a model+effort sweep on your internal evals (and if you don’t have internal evals, do that first).

For individual coding harnesses, honestly I just go with vibes. The best workflow is the one that works for you.

reply