Maybe with a decent consumer GPU like a 4090, you could do experiments like distilling and fine tuning a small image model for edge deployment for specific tasks.
Even there, many use cases might require renting compute for $10/hour and investing a few hundred.
A LLM from scratch? Forget it. You can do theoretical experiments, but not build anything remotely useful with that kind of budget.
If you're talented enough to come up with revolutionary methods, maybe an university or AI lab would be the place to be.
10000%.
Even more so when things are not just expensive by nature, but truly overpriced beyond that point.
>10000%
Once in a while you do get somebody who only spends a dollar and gets more out of it than a seasoned high-roller spending $10000. Most of the time the waste is borne by those who can afford to throw away $10000 more easily than an economizer can afford to lose one dollar, so nobody is crying about it.
With how ridiculously large the language models have gotten though, a 10000x improvement in actual intelligence does seem like it could be lurking unrecognized at a different point on the compass.