Then the local cache system needs to provide or cache those git packs

Exactly. I would love to see that design work, but it doesn't for any realistic workload.