upvote
I noticed some prior art when I wrote a similar library in C++ earlier this year, but unfortunately it's pretty much all deeply theoretical. The magic search term is "monadic unions of conjunctive queries", which is the class of queries this kind of system can express.

I'm still surprised there aren't more implementations out there, because it's a very useful class in application programming. For example, gameplay tags in unreal engine are queries in this class [0], and Intel ships a library using them for binary message parsing. It can also be done almost completely in SIMD, which is the route my shitty library takes. If you add recursion on top, you get queries that can express regular languages / monadic datalog.

My library quickly bottlenecked on memory bandwidth though. It doesn't seem like you're ordering joins by selectivity or a bitmap index to avoid reading unnecessary memory, so is this a direction you plan to explore more in the future?

[0] https://dev.epicgames.com/documentation/unreal-engine/using-...

reply
> It is mostly a bitmap index, generated from your struct so you don't have to write and maintain it yourself.

That explanation makes complete sense, because with this in mind, I can compare it to Apache Druid which does the same thing for data spilled onto S3.

Since this is immutable, you only had to solve "naming things", since the caches didn't need to be invalidated.

Druid is technically mutable, since it lets you mutate whole segments & then cache these bitmasks without having the original data in memory.

But if your dataset is already in memory, this is a neat implementation of an index and it should be faster than Roaring since you can skip the extra translation to the row-id that requires.

Druid uses Roaring - https://dl.acm.org/doi/10.1145/2938503.2938515

reply