See the recent development of DiLoCo at Nous Research and Prime Intellect.