My understanding is that in addition to your comment and the development of a method to separate the training data for distributed learning, the latency/bandwidth of systems connected on the internet is a challenge, too. Information has to be sent around before and after any hypothetical number crunching.
replyYou would probably not be able to go down to the scale of a single PC, but it should be possible to train models focusing on different specialties on different nodes and then have them periodically "mix" together.
replyWith current paradigms, yes. I'm hoping to see more focus on architectures that are more amenable to distributed training in the near future.
reply