Just to add, people in my network have been talking about polars (as an alternative to pandas) and other dataframe libraries. They're much easier to use now thanks to the Narwhals compatibility layer (for example, Narwhals was recently added as a dependency to scikit-learn).
replyWhat's different between Narwhals and Ibis? Why does the former exist when the latter has already (struggled along) existing for a while? Narrower scope / benefit of hindsight? No support for "cloud" data frames like Ray and Spark?
replyAlso Pandas is very much still a great tool and it's only getting better. It has some fundamental limitations that are relevant for processing bigger datasets or running things with higher performance. But it's still my preferred data frame for interactive day-to-day work. I only switch to Polars (or DuckDB) when I want to maximize performance.
I love what I am hearing.
I still see a lot of engineers using pandas, but it is such a horrible tools. You usually find an abandoned notebook with 100s of ”df_final_2” with sequential wrangling, making it impossible to understand what’s happening. Notebooks are also horrible for the same reason IMO.
replyWith chat-your-data you have Hex, Claude + MCP, snowflake, Databricks etc… everyone’s in on it.