They got to put ClickHouse, DuckDB and Trino on their resume & LinkedIn. Problem solved!

Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?

The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.

If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.

This reads to me like a recruiting post; thats why they are open if it solved their problems. I'm not a fan of the way they framed it, it seems they have made their architecture decision and now need help implementing this.

I'd imagine if you are the kind of person that wants to work on large scale data query problems this is an invite for you to apply. That said, if you are that person it might not be ideal to join after all the fun decisions have been made.

I know snowflake does this, they do the query optimization before sending it for execution. So it knows about how many resources will be needed (it can obviously be wrong).

We engineered, then we measured and then we tried to make it work, probably for a too long of a time before we decided to completely scrap our current architecture and go back to the drawing board. Happy to answer more questions.

They are big on shipping slop at lightnigg speed. So this is not surprising. Checkout posts by their ceo on cringedin.

It is not to serve high quality and timely data to make good decisions, is to avoid hiring a specific role, lol. So bad.

As much as I'd love to claim victory, we are not there yet :)

The first big system I worked on is the one where I learned not to generate expensive responses from a read mostly system, instead you should make writes expensive so reads are cheap. That project had an almost equally convoluted responses, and yet we were doing 30 req/s per instance on, IIRC, 4 core boxes. Adjusted for hot spare capacity, that ancient system was servicing 4x as many requests per core, 8x as configured.

A big part of the problem, besides selling a “premium” service to customers, was drinking the flavor of Knuth’s “premature optimization” misquoted koolaid that convinces people that all substantial speed improvements come from major architectural shifts. Which is how in the time I was on that team of 6-9 people, I delivered over half of the response time and cluster size improvements. If you count the work done to support upgrading the runtime, which brought perf improvements of its own, then more like 2/3.

Sounds a bit like this team might have the same disease.

The problem with internal service teams, of which this seems to be one, is that dealing with noisy neighbors is always a bigger problem than anyone is willing to tackle because it involves confrontation. Team X gets punished every time Team Y introduces a new feature that leans on the shared resource harder. It’s a multi tenant system without the fairness guarantees that we usually build into proper multi tenant systems.

Sometimes the easiest way to solve this is sharding. You end up deploying the same service many times and routing different classes of requests to different machines so if Team Y keeps trying to wreck the Commons they slow themselves down faster than they slow everyone else down. Those sped up consequences create a feedback loop that almost works instead of an amorphous one with consequences so far removed from actions as to become academic.

On that second team I mentioned we had an unfortunate situation of introducing things that took 100 ms off of response time and then three months later we were right back where we had been due to new features and creeping complexity of user content. It was demotivating for at least half of the team. My biggest contributions lowered the slope of that line, which allowed us to finally carry improvements over from year to year instead of being on the Red Queen problem.

Stateless CH on Iceberg sounds interesting, we looked into it but Delta/Iceberg had other downsides for us.

But it seems to have gotten a re-skin recently, and I preferred the previous more pixelated version. Also, there used to be a way to render a "normal" version of the website, but I can't find it anymore.

We built up all these UI conventions from decades of desktop development, then abandoned them for the web (and worse, electron). It's annoying learning a new convention for every website. We could do a lot worse than bringing back menus.