upvote

points

by 0xferruccio14 hours ago |
comments
upvoteby skeeter20203 hours ago|
next
[-]
>> DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop

I agree, and the dirty (not so) secret big data providers like Snowflake try to hide: the majority of your work is not big data and WILL fit on your local machine. My last company was spending $2M/yr on contract with Snowflake, and another million between Fivetran and Matillion. Of the 1200 clients using analytics maybe 2 had enough data to warrant "infinite scalability" and a dozen wanted Snowflake because they already had corporate warehouses in Snowflake (they probably didn't need it either). Turns out the Extract and Load could be handled by bog-standard C# code and a bunch of SQL, while almost everyone was better off with a DuckDB database running locally, often in the browser. You've probably heard YAGNI before (You Ain't Gonna Need It) but it's even more likely with "Big Data". #SmallDataConvert

reply
upvoteby tomjakubowski3 hours ago|
parent|
[-]
Folks have been beating this drum for as long as I've worked in software, dating to the Hadoop era, and it remains true today. So much of "big data" only appears big because it's poorly stored, or is represented wastefully (in persistent storage or in memory).
reply
upvoteby zurfer12 hours ago|
prev|
next
[-]
Like sqlite, duckdb is underappreciated as a production database. You can totally run it on servers or even "serverless" and do some heavy data transformations or with the right server size work with large scale datasets (up to a TB compressed seems fine).
reply
upvoteby ndr10 hours ago|
parent|
next
[-]
This. I've recently used both duckdb and sqlite to power a dashboard for a small restaurant of a family member. It converts all their sales to a very tiny parquet files, daily.

The file fits in memory and can do all sort of computation in the browser itself. The backend is extremely simple, it just loads the JS and serves the parquet files.

It was also trivial to let the owner do their own queries, just give the schema to an LLM and let it use the charting library, no data hallucinations. If they need it in the dashboard they can either use that one or ask me to review that query.

To be honest, given how simple some things became, it's been really fun to work on.

reply
upvoteby skeeter20203 hours ago|
parent|
next
[-]
Similar experience here. The best thing I've built in a long time is replacing a complex (and scary) permissions system built on top of Snowflake with single role duckdb databases that - aside from no longer worrying about bugs leaking data across roles - are more performant, timely and flexible. Combined with the use of AI this is the way forward IMO.

At the other end of the spectrum, working with random data on "what if?" and exploration tasks with DuckDB is fun again. it's so straightforward and fast, with tools and functions for pretty much everything.

reply
upvoteby kristjansson2 hours ago|
parent|
prev|
next
[-]
> no data hallucinations

Dangerous thing to assert. It’ll happily run SQL that works, but doesn’t necessarily correspond to intentions or unstated assumptions about the data.

reply
upvoteby ndr2 hours ago|
parent|
[-]
Of course I meant that it won't make data up.

It can only emit SQL and the json spec of the chart.

Since shipping I've reviewed dozens of queries and charts it produces answering the user. I'm yet to catch sonnet off guard.

reply
upvoteby noworriesnate3 hours ago|
parent|
prev|
next
[-]
I have a a theory that LLMs are going to be the death knell of big SaaS. It's so much harder to build and maintain an massive SaaS that does 80% of what 80% of your customers want, than it is to build something small and simple that does 100% of what one customer wants.
reply
upvoteby kristjansson2 hours ago|
parent|
[-]
Maybe once the model can administer and operate the service too.

For now building the 10% of the SaaS that you need still leaves you operating 100% of a new service/process

reply
upvoteby tomnipotent12 hours ago|
parent|
prev|
[-]
Not to mention it can query across heterogeneous sources, so the same query can use a duckdb table, sqlite, csv, and parquet (including predicate pushdown).
reply
upvoteby tosh6 hours ago|
prev|
next
[-]
Agree, in addition to that DuckDB also works quite well for data that is too big to fit in memory or on the machine DuckDB is on (predicate push down, out of core processing, …).
reply
upvoteby cyanide91112 hours ago|
prev|
next
[-]
>Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them

Nice! How do you set things up so that your engineers's claude code sessions upload to S3? Thanks for the help in advance

reply
upvoteby _boffin_6 hours ago|
parent|
next
[-]
Probably on a business / Enterprise plan, which has managed settings and also telemetry export. Give it a collector endpoint to export to and then have collector send to s3.
reply
upvoteby pimeys8 hours ago|
parent|
prev|
[-]
If you use OpenCode, the sessions are all in a local sqlite database. After lunch I'm pushing one of my agents to crunch some data from that using duckdb...
reply
upvoteby ashu14618 hours ago|
prev|
next
[-]
Can you please expand more on the claude analysis part. What exactly you analysed and what outcome it helped with ?
reply
upvoteby ryanchants6 hours ago|
parent|
[-]
Not who you responded to, but I've been working on cctx. It's an open source tool for analyzing claude code sessions to see where things went wrong(tool failure loops, bloated context, and the like).

https://github.com/jacquardlabs/cctx

reply
upvoteby creatorpilot10 hours ago|
prev|
[-]
[flagged]
reply