upvote
It's the same basic concept, yes. The details matter, though, and the fact that AoS/SoA is mostly about in-memory layout, whereas row/column store is about on-disk layout means that the performance compromises are very different. E.g. column store databases frequently use RLE for compression, but that's probably _way_ too expensive for SoA.
reply
Thank you!
reply
One thing to add is that databases tend to take it further using compression. For instance, if you have a column that is all either A or B, you can use run-length encoding ("the next thousand entries are B"). Or you can do delta encoding: if you have some very large number that takes a lot of bits to store, but the deltas between them are quite small, you can save a lot of disk space and I/O by just storing the deltas.

You can also combine these two: if you have a numeric id that increases by 1 every row, you can delta-encode that to just get a massive list of 1's, which you can then run-length encode to just be "the delta for the next million rows is 1", saving huge amounts of space.

You generally don't do this with AOS/SOA, because you do want to be able to random-access your entries ("index" them, in database terms) and I/O performance is much less important. Databases have this issue too, which is why indexes tend to either not be available or much slower in column-oriented DBs.

EDIT: ah, read the original commenter a bit too quickly, missed that they mentioned run-length encoding. Apologies.

reply
One of the cool things about Vortex (which is a file format, but in this context can be better thought of as the storage layer of a DBMS) is that we have implemented compression encodings that _do_ support fast random access. For example, FastLanes allows us to bit-pack values extremely effectively while retaining random access, and this can be combined with other encodings such as ALP and FSST.

I believe Vortex is unique in its ability to retain random access while recursively compressing data. And turns out that this is helpful for both in-memory _and_ on-disk (we might write about that in the future)!

I unfortunately didn't have much space to write about that in this blog, but in the compression section I did link to other blogs that we have written that go into much more detail, if you are interested :D

reply
Fun fact, the original Pokemon used both DE and RLE to compress the sprites. It's one of the things that makes missingno screw up sprites (it tries to decode game data as if it's a sprite encoded this way and sometimes causes buffer overruns)
reply
Appreciate you going into more detail about how run-length encoding works in any case. I had an abstract idea it was something like that, but didn't realize it could actually be implemented so 'simply'.
reply
On the one hand, you missed me mentioning RLE. Oh no how dare you. On the other hand, you added a bunch of extra detail, including stuff I didn't know about.

I call that a pretty positive tradeoff :)

reply
Please introduce your acronyms!

Array of Structures vs Structure of Arrays.

reply
It's a technical forum. That's like asking to "please introduce the definition of a linker" the first time it crops up. If somebody really doesn't know, searching "aos vs soa" in Google yields a largely correct AI summary followed by a Wikipedia article on the topic followed by a StackOverflow discussion on pros/cons followed by a similar Reddit discussion. Aside from maybe the image/video results, literally everything on the first page looks like it would explain the concept well.
reply
Expanding acronyms helps bring people into your field of interest.

People who might be technical and might even intuitively understand what is meant by "Array of Structures vs Structures of Arrays" might not be familiar with the acronym AOS vs SOA, especially given SOA is already a technical acronym their brain might shortcut to, "Service Oriented Architecture", even if instantly dismissed as clearly incorrect.

By expanding the algorithm you also help let people better decide if they want to spend the time reading up more about it.

reply
It's the same problem you see with variable names though -- longer names add local clarity but give disproportionate weight to those local concepts rather than the global structure. There's value in selectively using less precise identifiers.

Maybe you agree with that and just disagree with where the line should be drawn?

reply
My line on acronyms is fairly clear, I think that introducing acronyms by expanding them on first-use is a useful guideline and principle to try to stick by.

I'd also err on the side of generally assuming that most people, even in a technical forum, don't know all acronyms unless they're very widely known outside of the topic at hand. I don't think AOS vs SOA comes anywhere close to that line.

Some acronyms are of course better understood by their acronym, I wouldn't introduce a Local Area Network (LAN), or Domain Name Resolution (DNS), so like all guidelines there's judgement to be made.

Even then, if I were writing an intro to DNS, then I would call it an Introduction to Domain Name Resolution (DNS), because it is specifically targeted as an introduction. I wouldn't do the same for SQL, because no-one calls it Structured Query Language. Fairly often it would be more confusing if they did expand it, because quite often when people talk about SQL they're not really talking about the language at all, but using it as a short-hand for talking about SQL-compatible databases, or more specifically talking about relational database management systems (RDBMS), but "SQL" is a useful shorthand to contrast to "No-SQL" databases that were popularised a decade or two ago.

reply