upvote

points

by rfv672315 hours ago |
comments
upvoteby peepeepoopoo13714 hours ago|
next
[-]
"""The bitter lesson""" is how you get the current swath of massively unprofitable AI companies that are competing with each other over who can lose money faster.
reply
upvoteby furyofantares13 hours ago|
parent|
[-]
I can't tell if you're perpetuating the myth that these companies are losing money on their paid offerings, or just overestimating how much money they lose on their free offerings.
reply
upvoteby janalsncm10 hours ago|
parent|
[-]
If it costs you a billion dollars to train a GPT5 and I can distill your model for a million dollars and get 90% of the performance, that’s a terrible deal for you. Or more realistically, whoever you borrowed from.
reply
upvoteby yorwba10 hours ago|
prev|
next
[-]
They took a simple technique (normalizing flows), instantiated its basic building blocks with the most general neural network architecture known to work well (transformer blocks), and trained models of different sizes on various datasets to see whether it scales. Looks very bitter-lesson-pilled to me.

That they didn't scale beyond AFHQ (high-quality animal faces: cats, dogs and big cats) at 256×256 is probably not due to an explicit preference for small models at the expense of output resolution, but because this is basic research to test the viability of the approach. If this ever makes it into a product, it'll be a much bigger model trained on more data.

EDIT: I missed the second paper https://arxiv.org/abs/2506.06276 where they scale up to 1024×1024 with a 3.8-billion-parameter model. It seems to do about as well as diffusion models of similar size.

reply
upvoteby janalsncm10 hours ago|
prev|
next
[-]
The bitter-er lesson is that distillation from bigger models works pretty damn well. It’s great news for the GPU poor, not great for the guys training the models we distill from.
reply
upvoteby sipjca14 hours ago|
prev|
next
[-]
somewhat hard to say how the cards fall when the cost of 'intelligence' is coming down 1000x year over year while at the same time compute continues to scale. the bet should be made on both sides probably
reply
upvoteby furyofantares13 hours ago|
parent|
[-]
10x year over year, not 1000x, right? The 1000x is from this 10x observation having held for 3 years.
reply
upvoteby echelon14 hours ago|
prev|
[-]
Edge compute would be clutch, but Apple feels a decade too early.
reply
upvoteby 7speter11 hours ago|
parent|
[-]
Maybe for a big llm, but if they add some gpu cores and added a magnitude or 2 more unified memory to their i devices, or shoehorned m socs into high tier iDevices (especially as their lithography process advances), image generation becomes more viable, no? Also, I thought I read somewhere that apple wanted to infer simpler queries locally and switch to datacenter inference when the request was more complicated.

If they approach things this way, and transistor progress continues linearly (relative to the last few years) maybe they can make their first devices that can meet these goals in… 2-3 years?

reply