Yeah, I'd expect model performance to be super spiky on SWE work, at least they admit it with the name of the model. It's distilled from an already-distilled model.
replyMaybe still worth it if their "64% cheaper" figure holds.
I don't think you know what distill means
replyI guess I don't. Does post-training from another (larger) model not fall under the umbrella of distillation? I'd imagine it leads to the same spiky-ness issues...?
replyDistilling you don't have the actual model weights of the teacher. All you have are the teachers answers to a lot of questions. You then teach your own smaller model to answer more similarly to the big teacher model.
replyFine tuning you have the actual model weights of the original model, you then train that model to answer in a different (or better) way.
Distillation requires you to have the actual logits of each token from the teacher model, which in practice means having the model itself.
replyWhat you're describing is just synthetic data.
Note Anthropic misused the term in their post about Chinese model distillation, deliberately I assume.
There are multiple kinds of distillation
replyWith the Devin subscription even at the 20$ plan, they offered unlimited SWE 1.7 usage. Wondering if they do the same for SWE 2.
replySWE-2 is free for all subscribers on the CLI to try out for the next month :)
replyWhat about the gui/windsurf app? Same as cli?
replyI presume post training is significantly easier than the distillation/training the top Chinese labs are doing.
replyI wonder if, similar to the American labs, they'll become stingy with their weights once they start getting immediately undercut by a wave of slightly better derived models.