Several megabytes/gigabytes assets and you want to extract metadata, a preview, running as a batch some filter/compression/, conversion to CMYK, text injection ... fast partial read/write access would be nice. Right now, most reads are performed through indexes because those files are slow to read.
If we take 10k sqlite files and want to retrieve a row, we would be around 3s on SSD, maintaining preemptive indexes become less important for a lot of use cases.
Change management and versioning also becomes quite efficient - sqlite can be configured to not offset bytes, so CVS like Epic Lore can efficiently delta the files and store minimal delta, or the file format itself can keep its edit history. Oh and it's 3x-10x less bytes without compressing the whole thing, so pages are stable through time.
About needing SQLite client, it's real but it's roughly the same size as an XML parser.
what I like about it being XML is I can just open it with any text tool and inspect it. It's human readable so I can edit and debug it manually, no need to have a parser or an extra application just to see what's in my file
Nothing wrong with XML, it is well understood, and the tooling is pretty good. But partial loads/edits is one thing it can not do.
I agree with you that SQlite is overkill but honestly curious to know why you think xml+zip is strange? what would you use instead?
Sure you will. Plenty of features that don't exist, or are implemented badly, because you can't easily do it.
Quick mental translation table: if you think "iterate over every ..." or a `for` loop, that's your SELECT query. If you think about `if` conditions, that's the parts that go after FROM clause.
All that tends to fit sensible schemas and managing it is what SQLite shines at.
- Continuously parsing and writing and reparsing text, 90% of which is useless (that's the JSON/S-expressions vs XML argument)
- Forcing a diverse relational structure to fit a tree hierarchy, hand-writing all the logic that manages representation change - either explicitly, at serialization boundary, or implicitly, in every single access operation you're doing to refer to some data;
- Or worse, using an off-the-shelf, generic object/XML mapper, in which case you just compound the bloat even more.
SQLite is one of the single most battle-tested and ubiquitous piece of software in the history of mankind. Anything "simpler" you're going to pick up is much more likely to be buggy and broken, and will definitely be orders of magnitude slower.
With a zip file at least you know your file was corrupted.
It's all the metadata around the image that's interesting. Images have layers, dozens or hundreds of them (this literally scales with how good your software is at handling those - the faster, and more powerful layer UX is, the more they get used). Some are pixel layers, other are effect layers, text layers, vector layers. Layers have metadata - names, sizes, colors, tags, types, special effects, and a bunch of other stuff I don't know because I don't use that 80% of features of GIMP/Photoshop/Affinity.
Then you have document level metadata, UI-specific metadata, etc. Also undo history. A lot of that is relevant to the work on images themselves, and changes in realtime, and can get even more useful if querying it wasn't such a PITA.
That - not the binary pixel blobs - is the selling case of using SQLite as application data format.
That being said, SQLite would still be a good choice, especially as it's a format that's really intended to be modified in-place, and has good data integrity features (eg: keep WAL enabled so that mid-save crashes/shutdowns don't corrupt your file), neither of which are provided by Zip. You could even just run zlib on data (be it XML or what have you) if optimizing the on-disk size of the file is desirable.
Not really a great option for an image format, where we therefore can't have multiple implementations. Unless I misremembered.