undefined

points

[-]

This would be us doing a shallow checkout of individual commits (not cloning the full history) if we downloaded the 25GB compressed data for whatever commit we needed multiplied by the number of jobs that check out code.

We don't do it this way, for obvious reasons, but it means that there are different levels of caching and infrastructure required to make git work at repo sizes far smaller than what we have.

by nh25 hours ago|

prev|

[-]

No, of course not.

As the article explains, "over 600 PB" = 25M code checkouts * 25 GB "compressed working copy".

So "fetched naively" here means doing a full `git clone` on every need for `git checkout`, which of course nobody should do, nor does that refer to "code that needs changing daily".

by steveklabnik5 hours ago|

parent|

[-]

> "fetched naively" here means doing a full `git clone` on every need for `git checkout`, which of course nobody should do,

It can be cheaper to do a full clone than a shallow one, because with the shallow one, you can't re-use packfiles. GitHub has fairly regularly asked people to stop doing shallow clones because it's harder on their servers than full ones.

by nh22 hours ago|

parent|

[-]

Makes sense. I wasn't suggesting that one should do a shallow clone though. If your company repo is 100 GB, it seems sensible to keep it cached locally on each machine once (100 GB isn't large for current NVMe disks), and `git fetch` the updates the normal way. Then checkouts don't fetch anything large.

by steveklabnik2 hours ago|

parent|

[-]

Ah, yes, you can do stuff like that too, for sure.

At some point you still end up running into the issues the article talks about, all of these approaches only get you so far.

by falcor845 hours ago|

prev|

[-]

I don't think anyone "needs" more than perhaps 640KB of git traffic per day, but we want more.

by bjornsing5 hours ago|

prev|

[-]