And in terms of interesting use cases: recently pointed an agent at Blender and gave it vision. That setup can essentially iterate on a scene forever.
Because I don’t think “the subsidization train is going to end” necessarily means current pricing becomes impossible.
If capital keeps pouring into frontier AI, companies still have an incentive to subsidize access while competing for users and market share. And if that subsidization starts drying up, there’s even more incentive to bring inference costs down by making smaller and cheaper models catch up to today’s frontier capabilities.
So either way, I’m not sure you can extrapolate from the cost of serving current frontier models to what equivalent capability will cost two years from now.
A single six year old RTX 3090 works great: https://www.reddit.com/r/LocalLLaMA/comments/1vkm42m/muse_gl...
I fully expect Meta will release other, smaller Muse models in the near future too.
The 5090 is also supposed to be a $2000 GPU, not a $5000 one. The entire market is utterly distorted right now, which will impact cloud inference more and more over time too. They are not immune to the absurdly high RAM prices, so their prices will have to go up over time too until the RAM supply chain goes back to normal.
Prefill: ~1000 tok/s
Decode: 75-100 tok/s
It'll be far faster on a 5090, but I find the above performance to be acceptable. I've seen some claims that it even works OK on an AMD RX 7900XT (~$500USD)