Datafusion has all of the individual pieces, especially if you just want to data analysis, but is far from an embedded database.
replyIndexes, transactions, a first-class storage format are all things that come included with DuckDB, that you won't have with Datafusion.
indexes are very weak in duckdb, need to fit memory.
replyThis fall, DuckDB will support larger than memory indexes! I'm really excited about it.
https://duckdb.org/2026/08/17/duckdb-20-highlights#7-storage...
replythey don't say "this fall", they say later this year, I think it is some parallel effort, and it is actually going for few years already (rewriting index logic to have them in mmapped buffers and not in memory), finger crossed they will finish it finally.
replyIt's juts not as user friendly as DuckDB.
replyYou can even use DuckLake with Datafusion: https://github.com/tobilg/datafusion-ducklake-provider
replyhttps://github.com/datafusion-contrib/datafusion-ducklake is more established, and linked to directly from the ducklake docs
replyIs Datafusion related to Polars at all? I ask because you mentioned Rust integration.
replyNo, it's a competing ecosystem, even though both are built on Apache Arrow, and their dataframe APIs might look similar on the surface.
replyDataFusion is being used as a building block for a growing number of databases and data processing engines in Rust, as it offers the necessary primitives.
Awesome, thank you for the detail.
replyCame here to post the same thing. DataFusion is fantastic.
reply