They got to put ClickHouse, DuckDB and Trino on their resume & LinkedIn. Problem solved!
Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?
The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.
If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.
Counter-point - the longer you delay a crucial decision the more information you have to make it a correct one, so you might avoid migrating DB engine twice a year
This reads to me like a recruiting post; thats why they are open if it solved their problems. I'm not a fan of the way they framed it, it seems they have made their architecture decision and now need help implementing this.
I'd imagine if you are the kind of person that wants to work on large scale data query problems this is an invite for you to apply. That said, if you are that person it might not be ideal to join after all the fun decisions have been made.
As it was written, it looks like the engineering team is migrating the core infrastructure of the company around.
I'm not really surprised. Data storage and data access technology is a pretty large subject all by itself, and a "jack of all, master of none" developer might never develop the expertise needed to properly optimize a database.
Duckdb is an interesting approach to resource management (all ephemeral), but TB's are TB's. I doubt it was finding data where it needed it.
Clickhouse seems slow at getting to the top end of scaling; it's stuck with the 99% of its userbase doing 1 GB queries. I'm hoping they put some of their money into it.
https://clickhouse.com/blog/we-stand-with-ukraine
> We have no infrastructure, servers or people in Russia. The original engineering team successfully moved out of Russia and are now working in our Amsterdam office.
I kid but that's probably what people have a problem with.