upvote
I currently manage 50m active series and it’s brain dead easy using Mimir (which can also easily do 100-200m).

That being said, perhaps this Parseable project is much more efficient (RAM) in storing that 1b samples compared to Mimir (RAM). Once we add in object stores which cheapen the storage cost by orders of magnitude (by going from memory to disk), the comparison is even weaker. Mimir and Cortex/Thanos are quite happy to pull cold data from S3.

So I too expected to see “billions” as well.. hmm.

reply
We haven't yet tried pushing it to the scale of billions yet. The max that we've gone to is 150-180 million.
reply
Fair question. 100M isn't a ceiling, it's what we have seen in that deployment. We have not run a billion series test yet.

The reason we think it scales differently - labels are just columns in Parquet, so there is no per series index that grows with cardinality. In that deployment one label alone has ~2.5M distinct values among 500+ labels, which would be painful for an index based TSDB but here is just a high cardinality column. What drives cost for us is ingestion rate (data points/s) and how much data a query has to scan for a particular time range not series count. Ingest scales horizontally by adding ingestors, and queries prune by time partition and column stats.

A billion series benchmark is on our list, and we'll publish the numbers when we run it.

reply
Definitely publish it! I’ll be following closely! 100M seems a bit too low to turn heads, but cool project nevertheless. Always exciting to see open source observability tools pop up!
reply