Why didn't you just use Iceberg and Trino from the get go (or Paimon if you are fancy)? This has been the accepted answer for quite some time now. The fast OLAP stores just don't scale as well as going object store to a proper query engine like Trino.
replyWe tried Iceberg and it was too slow writing data is the short story.
reply>The fast OLAP stores just don't scale as well as going object store to a proper query engine like Trino
this exactly why we started with this project.
> Happy to answer more questions.
replyAh yay, I appreciate it! I feel like its rare to get a look at how these decisions cone about.
What did you measure? How'd you realise there was a problem? And did you measure anything to give you confidence that Trino won't hit the same issues?
With DuckDB and the queries we got, we had to provision such huge pods/nodes that we made it work but could never get to an economical viability. In the end, most of our customers had pods running 46 vCPUs / 360GB and asked for more. DuckDB is amazing, it really does everything right on a single node to make your query run successfully to the end almost always but it just doesn't work in our cloud env. Imagine when a query is running for 12h with this since and then another 10 from the same customer come in :)
replyTrino on the other hand has a much more convenient distributed query concept, so you can much more dynamically assign resources.