upvote
> What are you guys using it for?

Transparent queries on server/browser in identical sql.

If you play all the tricks Google's Capacitor does to the data layout, you can shove ~2-3 million rows to a browser based query engine in 16-20Mb with duckdb-wasm. We can scale it up to ~1Gb with HTTP range queries, with cache-control.

Then we use Jev to write SQL queries over it in 1-2s, whenever someone has a followup question. Getting Jev to write SQL takes a bit of prep, but my co-founder wrote a paper about clairvoyant query execution a few years ago [1].

I could have 3000 users logged into the same application, but all of them would be bringing their own CPU to run duckdb. My compute scaling problems for my data cube is a solved problem with this approach.

[1] - https://dl.acm.org/doi/10.14778/2733004.2733064

reply
note that if you want range queries in duckdb-wasm to work, you currently need to make sure that you disable `builtin_httpfs`, otherwise you can end up pulling a ton of data: https://github.com/duckdb/duckdb-wasm/issues/2228#issuecomme...
reply
This week I've been debugging a performance issue in a Ducklake and knocking out some good improvements, but this precise symptom—read_parquet() making tons of tiny range requests to load a parquet's footer from S3—was still baffling me. How lucky am I that I stumbled on your comment! Amazing, thank you for the pointer.
reply
> read_parquet() making tons of tiny range requests to load a parquet's footer from S3

This is somewhat unavoidable if you have a number of files and a tight filter which rejects most of them. The footer choice was a bad one in hindsight (not for HDFS), but for anything which uses XOR or other kind of erasure coding. The other other file format I worked with used headers only instead of footers, to save a few ms in latency for the initial GET

I don't think Ducklake can maintain a zone-map or ART index over parquet files, but there is no equivalent of iceberg metadata/something which has min/max/HLL compacted across multiple files yet.

The secondary indexes with file identifiers stored in a sequential file is most likely the next useful thing duckdb can make.

reply
Amazing we did the exact same thing at work. Been working great. A lot of BI datasets can fit within sub 20MiB of ZSTD compressed parquet.
reply
do you have not-big big data* that individuals or sub-teams/groups use? Lots of companies have centralized data <things> (bases, warehouses, lakes, swamps, etc) and runtime security is a challenging issue (i.e row-based security). If your audience measures their data in GB or MB, DuckDB can be an excellent way to punt on the security challenges and just provide them with ALL their data - maybe even locally - while they still get first-class DB tooling and performance (or even better than classic client-server). Example - old way: build a bunch of dimensions and facts for marketing, manage their access and permissions, produce reports, make sure you don't leak data they shouldn't see. new way: write (explore using AI) some extract/transforms from your SoT & create parquet files, use duckdb to serve that up to an individual or group's reporting solution or adhoc usage. IME that one-time investment is not that big and replaces a lifetime of scaling & access issues, or worrying about someone writing a query that doesn't join with the user table correctly!

* IME most companies don't actually have BIG data, but are talking about (at most) a few TB, which DuckDB is fine with.

reply
I recently used it for a Rust profiler which writes Parquet files, so DuckDB then gives an easy SQL interface for things like summarization, analysis, and joins across runs.
reply
You can use it to connect every single data source that's around your company and then use it as a federation engine (e.g. spreadsheets!)
reply
Use it for viewing csv and json files. Also, converting between formats can be a one liner.
reply
Write a log processing/aggregation tool!
reply