upvote
Git at any scale

(cursor.com)

It’s hard to overstate the reputation of the author of this post. Everything good about GitHub’s internal systems seemed to have his name all over it (I realize that today this statement hits different than a few years ago). Our times at GH didn’t overlap much but hearing the fact that he’s working at cursor increases my estimation of their engineering org by leaps and bounds.
reply
Came here to share a related fact :)

This solution with WAL and compact resembled the database internals as taught by Martin Kleppmann.

I asked Claude about this observation, and learned that the author (Vicent Martí) wasn't only core Github developer but also worked on Vitess from PlanetScale.

So no surprise these two contexts resulted in the design of Continuity as it is.

https://vitess.io/ - Vitess is a database clustering system for horizontal scaling of MySQL.

reply
Is this enough to counteract the reputation of his new boss, Elon Musk?

<edit>: there is lots of demand for a GitHub alternative due to poor uptime, poorly implemented components (CI), and Microsoft stewardship. Switching, especially at this early stage requires a huge component of trust.

reply
Cursor isn't covering itself in glory regardless. The flagship app is getting enshittified at a surprising clip. It constantly pops up and interrupts your work pushing new features, changes your model to whatever the latest Grok is without prompting, has this mystery meat UI that is constantly changing, pushes cloud agents in ways that are definitely designed to trick you. We're actively looking at alternatives, I wouldn't touch anything this company produces from here on out.
reply
[flagged]
reply
Did you really create an account for this?
reply
its literally elon getting mad and making an account calling people cock suckers
reply
* spent tokens on this
reply
But OP is the loser for leaving a comment.
reply
tired of this
reply
There's a reason you say that with a throwaway account. Even you know it's shameful.
reply
Just like with climate change, the problems Elon creates don’t go away just because people are reminded of them.

He still pushes his agenda, he still has lots of followers who believe his nonsense

reply
Well, there's going to be a lot more of it. You can't be an outright ultra-right wing racist and expect that it doesn't cause brand damage. It never made business sense for Musk to act the way he does, but he couldn't help himself. Do not expect people to just be ok with it.
reply
tired of you
reply
Fantastic write up, but I still have doubts. As the author states: Serving git at scale is hard. I can only concur.

I feel like the article is a bit light on the design of the WAL and maintenance operations, but maybe it's me.

Anyway, I'd love to have a friendly chat with the author.

reply
> What about consensus? Elections? Which server is the primary for a given repository? It also doesn't matter! There's no state and no consensus here. Any server can be the primary. All updates to the write-ahead log are synchronized with an atomic compare-and-swap (CAS) operation on S3, so it's always safe for any instance of a repository to receive a push.

Again reminded of what an amaizing piece of engineering S3 is (99.999999999% - 11 nines of durability) [1]

1: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...

reply
Not just durability. But read after write guarantee on a distributed object store.

How many startups and corporations built on top of S3 semantics?

Snowflake and the big lakehouses are one of them. They use S3 as giant disk in the sky.

reply
https://cursor.com/en-US/blog/git-at-any-scale to avoid automatic translations (couldn't find a button to switch)
reply
Thank you so much, what an obnoxious behavior!
reply
The language selection is in the footer. Interestingly, the URL for English is the original post without en-US.
reply
It's time to look back on S3's 20-year history

https://blog.fnil.net/s3-at-twenty/

reply
Took me some thinking to figure out where this design is not optimal.

The first step is to separate the problems. The content addressable object storage and storing the refs. libgit2 has two backends for those already.

Then use appropriate distributed dbs for the two backends. With large scale something like Cassandra for the objects (AP) and FoundationDB (CP) for the refs (or the equivalent Amazon services).

For performance the objects need to be cached locally and requests routed to nodes where the objects are cached locally.

reply
Part of the problem is that the API you're supposed to provide externally depends on git packs which you might have to reconstruct on the fly from such a system.
reply
Exactly. I would love to see that design work, but it doesn't for any realistic workload.
reply
> The "fan-out" is synchronized with a classic consensus algorithm called 3PC (three-phase commit) so that a push is only accepted if a majority of the nodes acknowledge it.

Doesn't 3PC require all nodes to agree, not just a majority?

reply
I would think so - outside this sentence the author indicates they're talking about consensus across all members.

IE - the interactive diagram and note about latency being bounded by the slowest replica

reply
There’s a trend of doing impressive things by pushing many of the hard problems into S3 and assuming S3 “just works”, then not bothering to explain how S3 works. I guess we do the same thing all the time with other miraculous solutions to hard problems, like assuming that superscalar out-of-order processors “just work”, but in cases like this it makes for an unsatisfying explanation. Especially because S3 is a proprietary product, not an algorithm.
reply
at this point, s3 just means object storage and doesnt mean it actually has to be hosted on aws. theres plenty of other companies that provide s3 compatible storage apis.
reply
APIs, sure, but do they provide the semi-magical consistency, atomicity, durability, and latency guarantees that this depends on?
reply
> The productivity cost of five minutes of downtime in your CI system is hard to quantify in dollars, but it is, by any measure, a humongous amount.

Not sure the last month GH had fewer than five minutes of downtime - but I think it's been a while

reply
The key part: they decided to use the unchanged git codebase as their building block. Likely assuming that the people who wrote the Linux kernel probably knew what they were doing. That explains some features that look bizarre from the generic distributed-database perspective. I think the author has had this argument many times, because he reiterates it several times in the post. The rest follows from it.
reply
That was a very well written and interesting article. I really enjoyed it. Some good background for people telling GitHub to "just scale".
reply
GitHub issues are mostly with PRs, CI, etc, not with git
reply
Reading this, I can't come to a conclusion other than "git rapidly starts sucking balls for use cases outside of Linux development". Github started with ze_current_fastness but then ended up recurrently smashing into walls because of it.

They should have just failed to scale git and leave place for the less idiosyncratic bazaar or mercurial to take the lead. These were slower, but would probably have scaled much better in the long term.

Github's core innovation was re-centralizing a decentralized SCM (as per article) by giving it nice web UI and features such as PRs. Oh and also free code hosting space. None of which relied on git's peculiarities.

reply
I don’t see a button to switch to English and the German text is a pain to read, comparable to Reddit auto translate..
reply
The button is apparently in the footer: Another comment earlier has the english-only link: https://news.ycombinator.com/item?id=49376960
reply
Very cool write up. But I do have to wonder, is Git the right solution for version control given where we are heading? The commands are all muscle memory for me now at this point, but all the projects I've been working on lately have just all moved to trunk based development, rebasing, and squashing merging feature branches onto main.

I've been working on another project where I have binary files checked in, assets that move or are moved and then changed that git "loses track of" due to the significant change. This just may be me doing silly things like moving a file and then editing it and then committing it and git thinks I deleted it and created a new file.

reply
WAL on S3 is the new normal.
reply
Interesting, wonder how Cursor will scale with Git with this approach.
reply
Related:

Cursor Origin Code Hosting

https://news.ycombinator.com/item?id=49334209

reply
Remember, don't put large objects in cnt, it makes it a real pain later.
reply
cnt?
reply
Continuity system of Cursor. Brilliant name, just like the octo"cat"! Elon will approve.
reply
how does git lock the repo on extreme concurrency? let's say 1000 devs writing to the same repository? eventually you'll have saturation of the resource locking everyone out.

When I've worked at big companies they had bespoke SCM that resolved this. But i'm curious about mid size teams that are big enough to overwhelm git but not so that they can rewrite it.

reply
If those 1000 devs write to different branches then you only take a light lock on the branch, usually that is not the problem.

The problems start when there is a lot of changes on one branch and usually the first band-aid is to put a commit queue in place. The few very big companies that need even more than that probably aren't going to put their code on a new git hosting provider tomorrow.

reply
that sounds like a practical solution once you reach that size. yeah you're right i think only the refs are locked, the rest of the objects can all be submitted concurrently, so it would be high contention refs/branches that would be a pain point.
reply
It doesn't matter how many technical blog posts they put out, I will still never host my code with Elon Musk.
reply
Same. Sometimes you have to live with inconvenience to keep a moral stand. Those down-voting don't understand why any sacrifice of convenience or tech is worth it.
reply
I agree, although I don't think their offering is even convenient. I also don't use Cursor (I think most people have moved on to Claude Code/Codex), so also not helpful from that point of view. Regardless, not only do I find it morally wrong to give Musk money, I think it's a bad business idea. He can't be trusted and your code is at risk on his platform.

> Those down-voting don't understand why any sacrifice of convenience or tech is worth it.

FWIW my comment was actually highly upvoted but it looks like X employees and/or Musk fans showed up and downvoted all criticism in this thread.

reply