A 1 gbe local network should have < 1 msec latency per hop so theoretical upper bound is substantially higher than 30 tps (again assuming instantaneous compute) => thus network latency should not be the limiting factor in reality, no?
replyOn a fast network, yes. If you want to create a distributed network over the internet, then it will.
replyAh, that's interesting. I though there was more data crossing the network. So, why does a DGX Spark come with super fast network if 10Gbps ethernet would be sufficient for splitting a model? I never bought a second Strix Halo on the assumption that the pipe between them would be a limiting factor to using larger models, so obviously there's something I don't understand.
replyThe amount of data is only low for inference, not for training, and AFAIK DGX spark is supposed to be a researcher's machine that can do small-scale training.
replyI’m staring at this comment for a while now: With 3ms latency combined per token, wouldn’t that mean (1 / latency) = 333 token/s for the theoretical upper bound? I’m not trying to nitpick, just curious if I misunderstand something.
replyIndeed, I completely screwed my math up. Looks like 10am is too early in the morning for a Sunday.
reply33 tps max token generation speed would be for 10ms of network latency in the above example.
deleted
reply