upvote

points

by ux26647813 hours ago |
comments
upvoteby rurban8 hours ago|
next
[-]
You got it wrong. Inference can use crap GPU's. Training needs the 100x more expensive big guns. Our training machine is 100x more expensive than our inference machine.
reply
upvoteby bombcar2 hours ago|
parent|
[-]
How is the result of training stored? How big is that? It seems reasonable to assume we’ll eventually plateau and all we’ll need is relatively infrequent training.
reply
upvoteby rurban38 minutes ago|
parent|
next
[-]
Not so often. The GPU's are running 100% for 3 weeks for a training run. We do images only, but it's the same process. And then we can use the costly GPU's for inference, local model coding agents. Training is about 4x a year. But it depends what ideas the PM or the costumers have. If they has more, more training tasks. Eg. more viruses to detect.
reply
upvoteby brandensilva1 hours ago|
parent|
prev|
[-]
I agree, leave the training to open source federations that roll out like operating systems. Minimal training over time.

Then have inference go down to the next layer to use those models as a P2P decentralized network.

Maybe like open router could tap federation networks.

reply
upvoteby sho12 hours ago|
prev|
next
[-]
> AI hardware is for inference, not training

Not sure what you are referring to, unless you don't think h100/h200/b200 are "AI hardware"

> Superpods aren't really power efficient

Maybe not compared to a specialized rig with multiple 4090s, but that is the best case for consumer hardware - the vast majority will be dramatically less efficient than that

Anyway, I agree the interconnect is by far the biggest obstacle and seems insurmountable, I should probably have led with that.

reply
upvoteby pksebben13 hours ago|
prev|
next
[-]
Bit of a doozie though, that one.

I recall getting really excited over hinton's FF foray, right before he bailed on AI as a societal direction (which, if anyone ever had the right, I suppose he does). If one squints, one can see a backprop-free base being much easier to train on geographically distributed and heterogenous hardware.

reply
upvoteby Davidzheng9 hours ago|
prev|
next
[-]
Are you sure most of frontier cost isn't inference in RL environments?
reply
upvoteby dyauspitr12 hours ago|
prev|
[-]
That makes no sense. It’s basically the same calculations for training as well.
reply