upvote
Improving on "git not handling non-text files" is a semantic understanding aka parse step in between the file write.

Take a docx, write the file, parse it into entities e.g. paragraph, table, etc. and track changes on those entities instead of the binary blob. You can apply the same logic to files used in game development.

The hard part is making this fast enough. But I am working on this with lix [0].

[0] https://github.com/opral/lix

reply
We're working on this project to help with the non-text file and large file problem: https://github.com/Oxen-AI/Oxen

Started with the machine learning use case for datasets and model weights but seeing a lot of traction in gaming as well.

Always open for feedback and ideas to improve if you want to take it for a spin!

reply
Totally agree. After trying to flesh out Unreal's git plugin, it really shows how far from ideal git really is.

Partial checkouts are awkward at best, LFS locks are somehow still buggy and the CLI doesn't support batched updates. Checking the status of a remote branch vs your local (to prevent conflicts) is at best a naive polling.

Better rebase would be a nice to have but there's still so much left to improve for trunk based dev.

reply
Has there ever been a consideration for the git file format to allow storage of binary blobs uncompressed?

When I was screwing around with the Git file format, tricks I would use to save space like hard-linking or memory-mapping couldn't work, because data is always stored compressed after a header.

A general copy-on-write approach to save checkout space is presumably impossible, but I wonder what other people have traveled down similar paths have concluded.

reply
What strategies would you like to use to diff the binaries? Or else how are you going to avoid bloat?

Is it actually okay to try to merge changes to binaries? If two people modify, say, different regions of an image file (even in PNG or another lossless compression format), the sum of the visual changes isn't necessarily equal to the sum of the byte-level changes.

reply
I really think something like Xet is a better idea to augment Git than LFS, though it seems to pretty much only be used by HuggingFace for ML model storage, and I think their git plugin was deprecated? Too bad if it ends up only serving the HuggingFace niche.
reply