I have a project[0] that does the large file thing well, but is missing most of the version control porcelain. I've been looking for the path of least resistance to integrate it into something with a larger user base.
I suspect if you came by the jj discord, folks could help you with more detail than that.
It looks like this treats files as blobs just like Git, and trees as single objects which fit in memory. Assuming that is a correct understanding, this core abstraction would need to change to handle large files and directories well.
All the well known version control systems do this though, and it simplifies the system significantly. It's the right model for source code, but it doesn't translate well to arbitrary data.
(LFS support is in progress though)