Woah, is this part of the future of models?
Basically little models you can use as tools.
replyIt's looking like running your own mini ecosystem is the way of the future to me. No data centers, just a decent GPU 16-24gb of VRAM, CPU, and 32gb of RAM.
replyThis is Apple's bet, among others.
replyTraining purpose-specific miniature models lets you have a lot of tasks you can run with high confidence on consumer hardware.
Or on a commodity EC2 instance with a relatively cheap inference sidecar.
replyEventually we'll have models small enough to do a single thing really well and we'll call them functions.
replyI'm pretty sure there's someone somewhere who'll create a proper harness that's equivalent to one giant model. The difficulty is mostly local hardware has lot of memory constraints. Targeting 128GB would seem to be the current sweet spot. If we could get out of the corporate market movers of buying up all the memory, we could maybe have more.
replyRegardless, the people in the 80s capable of pruning programs to fit on small devices is likely happening now. I'd bet most of the Chinese firms are doing it because of the US's silly GPU games among other constraints.