upvote
One of the cool things about Vortex (which is a file format, but in this context can be better thought of as the storage layer of a DBMS) is that we have implemented compression encodings that _do_ support fast random access. For example, FastLanes allows us to bit-pack values extremely effectively while retaining random access, and this can be combined with other encodings such as ALP and FSST.

I believe Vortex is unique in its ability to retain random access while recursively compressing data. And turns out that this is helpful for both in-memory _and_ on-disk (we might write about that in the future)!

I unfortunately didn't have much space to write about that in this blog, but in the compression section I did link to other blogs that we have written that go into much more detail, if you are interested :D

reply
Fun fact, the original Pokemon used both DE and RLE to compress the sprites. It's one of the things that makes missingno screw up sprites (it tries to decode game data as if it's a sprite encoded this way and sometimes causes buffer overruns)
reply
Appreciate you going into more detail about how run-length encoding works in any case. I had an abstract idea it was something like that, but didn't realize it could actually be implemented so 'simply'.
reply
On the one hand, you missed me mentioning RLE. Oh no how dare you. On the other hand, you added a bunch of extra detail, including stuff I didn't know about.

I call that a pretty positive tradeoff :)

reply