undefined

points

by minraws5 hours ago |

comments

by bitexploder4 hours ago|

[-]

Models like Qwen 27B 3.8 and Flash Next 3.8 show there is a lot of room to pack more into less parameters.

by minraws4 hours ago|

parent|

[-]

These models exist because better models exist, these models can't exist without better models existing, you will never beat the frontier with smaller model sizes. Because all these models are trained again larger models, generally a teacher model of sorts.

Lots of synthetic data, distilled data etc.

The problem is if we can pack a Fable in Qwen 4 27B who will pay subscription prices?

by tcdent3 hours ago|

parent|

[-]

Synthetic data, distilled data, etc., does not necessarily mean data coming from outside of an organization. When we see performance increases with models like Haiku, it is because they are able to leverage the knowledge of larger architectures.

I fully expect that we will see increased performance from smaller architectures (to a point) simply because they are refined using larger ones.

> The problem is if we can pack a Fable in Qwen 4 27B who will pay subscription prices?

This is available to everyone developing these systems. Prices per relative unit of intelligence will continue to decrease. And market economics will continue to drive the cost down.

by bitexploder1 hours ago|

parent|

prev|

[-]

There is nothing inherent in the minimization process that says you must distill from a larger model. It is just the thing we know how to do right now.

by khimaros4 hours ago|

prev|

[-]

that was always the promise wasn't it? linear increase in intelligence in exchange for exponential increase in compute?

by minraws3 hours ago|

parent|

[-]

It's not really if you mean to point to the bitter lesson, it was models/intelligence scales better with general/simple methods with more compute better than it scales with any hand-crafted human program/method.

Ala intelligence is a function of compute, there was no guarantee that you need to 10x compute for 10% better performance or something like that.

If you ask the labs they will say the scaling is fairly linear or atleast sub-quadratic, they definitely won't say it's exponential.

No one was thinking we need to 10x compute from here again for another 10-20% increase in performance.

It's only been since Opus 4.6 that the compute growth vs improvements are starting to feel this way.

For instance GPT-3 vs GPT-4 was practically a 10-100x jump for a 10-100x increase in compute, we went from broken sentences to fully formed essays writing, speech and so on.

GPT-4 vs GPT-5 was just as massive we went from no agentic capabilities to fully agentic, 10-100x for a 10-100x increase in compute usage.

GPT-5 vs GPT-6 was atleast a 10x (upto 100x) increase in compute, but the difference is not that clear sure people say that one is significantly better than the other, but GPT-5.5 was good enough compared to where we are today.

Even on benchmarks compared to GPT-5.2 Codex we have almost 2x'ed the performance on say artificial analysis (53 vs 29). This is absurd. 10-100x increase in compute for 2-5x performance improvement is a massive gap.

[1]: https://bitterlesson.ai/ [2]: https://artificialanalysis.ai/?models=gpt-6-astra%2Cgpt-6-lu...