See the recent advance of DiLoCo at Nous Research and Prime Intellect.