> 1 Billion Row Challenge benchmark: Pandas took 4m28s vs. Polars 5.04s and DuckDB 5.19s — DuckDB also used 19x less memory
Python Vs Rust : In terms for speed - No comparison
(The above episode transcript has a link to blog post titled "Pandas should go extinct" )
Polars is great, but I'm just too used to the Pandas API to use it as a replacement for the cases where DuckDB is overkill.
Part of the problem, as I said, is that I'm just spoiled by DuckDB when performance matters.
from https://github.com/pola-rs/geopolars/tree/main
Comparison with GeoPandas
Imitation is the sincerest form of flattery! GeoPandas — and its underlying libraries of shapely and GEOS — is an incredible production-ready tool.
GeoPolars is nowhere near the functionality or stability of GeoPandas, but competition is good and, due to its pure-Rust core, GeoPolars will be much easier to use in WebAssembly.
This is awesome!! I'd looked at the project only a month or so and it appeared abandoned, but I must have missed the off-main-branch development going on!
It's been impressive!
Vast majority of skilled developers are now using Polars, unless they are constrained by lack of Narwhals support in their third-party library of choice (e.g. Great Expectations, SHAP). That's the more important trend to follow.
In that regard, I’m still waiting for a credible jq replacement…
Also, try fx.wtf as a replacement for jq. it comes with a in-built tui viewer that supports vi-keybindings. Ecmascript is built into fx.wtf so you can query the JSON with JS notation (where JSON was born). You can use any JS functions including map/reduce/filter or perform any kind of transformation instead of learning jq dsl that you will forget tomorrow.
tl;dr yes
DnD does the same thing: it's the most popular but its rules are this awkward hybrid of legacy cruft and some modern ideas, so learning it a huge effort, which means most people who play it aren't willing to try any other RPG systems even though most of them are dramatically easier to learn because they were built with a clean design from the ground-up.
It's the sunk-cost fallacy as applied to learning something complex, combined with something like the horn effect (inverse of the halo effect) making any competitors look equally complex even if they're not, causing long-time Pandas users/DnD players to strongly resist even looking at other options. Basically the frustration of learning these older systems seemingly traumatizes some people into never straying.
https://docs.pola.rs/api/python/stable/reference/expressions...
Pandas has a really simple ability to just define a new column with
`df['col_a'] + "text" + df[col_b']` where "text" can be any string text inbetween your column values from col_a and col_b
If i remember correctly while you can do pl.col("col_a") + pl.("col_b") for plain concatenation, you can't mix in static text strings like you can with pandas and I haven't found really elegant ways to do that personally. Whereas I've found polars doesn't have as simple of a way to do that. You can choose to add one separator and make that anything you want, but only one separator and only inbetween the two values (so no suffixes or prefixes for example).
That being said, I hate everything to do with the pandas API (especially with its indexing system) and really prefer the more polars API for anyone coming from a SQL or database background. Pandas really shows its sort of academia background rather than a data engineering origin.
>>> df = pl.DataFrame({"x": ["a", "b", "c"], "y": ["d", "e", "f"]})
>>> df.with_columns(new=pl.col.x + " text " + pl.col.y)
shape: (3, 3)
┌─────┬─────┬──────────┐
│ x ┆ y ┆ new │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str │
╞═════╪═════╪══════════╡
│ a ┆ d ┆ a text d │
│ b ┆ e ┆ b text e │
│ c ┆ f ┆ c text f │
└─────┴─────┴──────────┘