My favourite trick for working with huge data sets in R. If your dataset is larger than memory and the query result is also larger than memory, you can still use dplyr/arrow pipelines. Example:
library(arrow)
library(dplyr)
nyc_taxi <- open_dataset("nyc-taxi/")
nyc_taxi |>
filter(payment_type == "Credit card") |>
group_by(year, month) |>
write_dataset("nyc-taxi-credit")
Input is 1.7 billion rows (70GB), output is 500 million (15GB). Takes 3-4 mins on my laptop 🙂
@ryderdavid One really nice thing, if you decide to learn arrow, is that it's very interoperable with Python because the in-memory data structure is the same regardless of which language you're using. So it's pretty easy to pivot between R and Python within a workflow if you find yourself needing that.
(I wrote up some notes on it a couple months ago, actually: https://blog.djnavarro.net/posts/2022-09-09_reticulated-arrow/)
In a multi-language ‘polyglot’ data science world, it becomes important that we are able to pass large data sets efficiently from one language to another without making unnecessary copies of the data. This post is the story of how I learned to love using reticulate to pass data between R and Python, especially when used together with Apache Arrow which prevents wasteful copying when the handover takes place
Danielle Navarro
Ryder C 
🔭 🚲