I wrote https://github.com/anacrolix/possum for this exact use case. It has a few rough edges but it absolutely hauls ass for caching on disk. Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck.
sqlite transaction commit overhead is too high and uncompetitive without batching but it's a design flaw specific to being multiprocess.
Or is there a better way?
So, on a filesystem?
Why sqlite instead of RocksDB ?
What you need at the end of the day for a blob storage is a transactional key value store.
Which RocksDB will give you with much better throughput (and latency) and solves your transaction problem.
This is true of every RDBMS I am aware of. It would be horribly inefficient to constantly run page merges in order to be able to free some of them.
Whenever I read things on posts that are a. discussed in documentation b. should be apparent from first principles, I immediately lose faith in the product.
I mean, today, why would you? It used to be that starting a new CRUD project meant at least an hour of setting up web servers, database servers, proxies, etc. Today, that's all abstracted away for pennies an hour.
It was hard to take him seriously after that
I mean you're one-off vibe-coded distributed database isn't a database until aphyer puts it through it's paces, and until it encounters most CAP its edge cases over years of use.
Otherwise, this isn't a database. It's a black hole of uncertainty that will almost certainly drop writes and commits silently
> While raw files start to beat SQLite above ~100 KiB, we compress blobs before writing them. And thankfully, 99% of our compressed values sit well below 100 KiB.
My question is, were the other databases not tried with compression on the application side?
An alternative to faster blob storage is to represent the blob's schema in SQL directly.
If most blob updates are incremental you can eliminate a significant amount of storage IO and garbage collection this way.
One SQLite database per blob type with a metadata SQLite db that tracks each. We could leverage powerful AI reasoning models to produce SQLite schemas and virtual table adapters for each encountered JSON schema type.
sqlite offers this and this is being redone or what is being reinvented here
Problem is that it is not really distributed (static sharding, no fault tolerance, no consistency model).. on top of that poor operational support - backup/restore, recovery. Stateful infra pieces need lot more rigour, you can't iterate your way through such fundamental capabilities once it is in production use.
These are user preferences for UI setup, and they have a lot of write traffic. But probably each user changes their preferences only by a little bit each time. Their 1MB writes would go down to just the small diffs.
They could store the diffs and transparently apply them before sending to the client. After some timeout or maybe after a certain number of diffs have accumulated, squash them all and write back.
They’d still need a server and custom code to handle this but it would probably reduce the database load and SSD writes by a lot. And let them just use something off-the-shelf for storage.
The network interface is the Redis protocol.
This doesn't describe rqlite, at least.
Correct, I read the post. Agreed, rqlite is not a great fit for high-throughput durable writes since all writes go through the Raft Leader. This is not specific to rqlite -- all systems which put Raft in the data plane[1] have the same challenge.
rqlite does offer a Queued Writes mode[2], which allows you to trade durability for much greater write throughput. The post mentions something similar -- async_write -- but their post claims they also make the data queryable before it is durable, unlike rqlite.
https://github.com/Safebots/Safecloud
It can also stream videos to any domain, like a decentralized YouTube. Here is a demo:
https://youtu.be/glWQM0BIcDQ?si=VV7YFiElGKBC82Bv
This is what we encourage to store in it:
https://safebots.ai/datasets.html
Oh, and it is also compatible with S3 bucket protocol, IPFS, and even MySQL protocol so it can replace your S3 bucket or database with a distributed database — just like that!
PS: if you’re curious, it uses a columnar store and SQLite’s query planner, to be as efficient as possible.
They also mention considering this in the post.
Bad fault tolerance, no words on replication or backup. Offline rebalance... meh. I am not sure how I feel about their choice of Redis protocol.