NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed small tools. I wonder if anyone bothered to look into this problem.
replyIt was the second most reported problem on the 2025 Rust survey, after compilation times.
replyI think the parent was referring to hosting the entire NPM registry, not having a project that uses NPM. In that case there's substantial duplication from version to version.
replyI've done similar things for large container images. My format allows for using FastCDC to chunk files, but there's a tradeoff between number of shared chunks and between number of HTTP requests. I keep it turned off by default.
replyI'd be curious whether block level de-duping would add value too in their case. You effectively achieved that to some degree with storing the deltas.
replyYou may or may not be familiar with the content defined chunking family of algorithms like FastCDC.
replyhttps://joshleeb.com/posts/chunking.html
https://www.usenix.org/conference/atc16/technical-sessions/p...
On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.