upvote

points

by mjhay7 hours ago |
comments
upvoteby pks0164 hours ago|
next
[-]
I would be in minority. But, I don't like tidyverse ecosystem. I prefer data.table for most of my uses.
reply
upvoteby donkeyboy1 hours ago|
parent|
[-]
Data.table is just so much faster, and the sql-like stntax is easier tonunderstand
reply
upvoteby chocochunks8 minutes ago|
parent|
[-]
I can never remember the data.table syntax, every time I use it I have to re-learn it. It doesn't feel very SQL to me either. There is an interface to use tidy syntax on data.tables and get's you 90% of the speed.
reply
upvoteby Mairoce1 hours ago|
prev|
next
[-]
The core of the problem is that the tidyverse is trying to turn R into a user-friendly real-time calculator, rather than a tool for stable, deterministic, and literate data analysis.
reply
upvoteby levocardia36 minutes ago|
parent|
[-]
"The real problem with this package is that it makes things easy, instead of hard"
reply
upvoteby Mairoce22 minutes ago|
parent|
[-]
That’s a rather glib interpretation of what I said.

I’m being rather charitable when I call the tidyverse “user-friendly”. That might be the goal of tidyverse devs, but it hasn’t been the reality since inception.

The point is even if we assume it is or will eventually become user-friendly, are we willing to accept the trade-offs that come with that?

reply
upvoteby dash29 minutes ago|
parent|
[-]
I think the users have clearly delivered the verdict that it is friendlier than base R. Admittedly a low bar. Non R users, which do you prefer:

    foo[foo$bar == “baz”,]

    foo |> filter(bar == “baz”)

?
reply
upvoteby 331c8c717 hours ago|
prev|
[-]
It's certainly quite pleasant to work with...but I would rather use sql for etl, the backend be whatever it needs to be...

The real world data transformations can get gnarly very quickly and sql is the perfect common debiminator compared to dplyr which is still niche...

How do you feel about polars?

reply
upvoteby mjhay7 hours ago|
parent|
[-]
I’m a big fan of Polars. It’s really fast and memory efficient. With the lazy streaming functionality, I’ve been able to easily process 1 Tb+ data on a single machine (you do have to be careful to not do any operation that would cause the whole DF to materialize in that case).

It’s certainly miles better than Pandas, which has a terrible API in addition to being comically inefficient. In my group, we generally use it for any new work, and have also swapped out pandas for polars in critical spots of our existing code - the latter giving a huge benefit relative to the amount of work it took.

I largely agree with you on SQL being the common denominator, but there are some things that are just awkward in SQL, and much easier to do in Python or other general purpose language.

reply