These models exist because better models exist, these models can't exist without better models existing, you will never beat the frontier with smaller model sizes. Because all these models are trained again larger models, generally a teacher model of sorts.
replyLots of synthetic data, distilled data etc.
The problem is if we can pack a Fable in Qwen 4 27B who will pay subscription prices?
Synthetic data, distilled data, etc., does not necessarily mean data coming from outside of an organization. When we see performance increases with models like Haiku, it is because they are able to leverage the knowledge of larger architectures.
replyI fully expect that we will see increased performance from smaller architectures (to a point) simply because they are refined using larger ones.
> The problem is if we can pack a Fable in Qwen 4 27B who will pay subscription prices?
This is available to everyone developing these systems. Prices per relative unit of intelligence will continue to decrease. And market economics will continue to drive the cost down.
There is nothing inherent in the minimization process that says you must distill from a larger model. It is just the thing we know how to do right now.
reply