Christos Argyropoulos MD, PhD
It is all #rstats fun and giggles until the dataset is too big to fit in memory. Then, it is time for the database servers and #SQL to save the day.
Apr 14, 2025 at 5:20pmWeb
Show threadFabianApr 14, 2025
@ChristosArgyrop While this has truth, there is still #DuckDB and #ApacheArrow that can help tons with large datasets before we need actual SQL! Don't know if that's counting as giggles though 😅
Show threadnecrophcodrApr 14, 2025
@radlfabs @ChristosArgyrop DuckDB is still SQL too. Both that and SQLite will handle files much larger than what fits in memory. Neat tricks in both of them!
Show threadFabianApr 14, 2025
@necrophcodr Fair enough! @ChristosArgyrop
Show threadChristos Argyropoulos MD, PhDApr 14, 2025
@necrophcodr @radlfabs For in memory, I prefer data.table until it craps out. The source data are in SQLite, but we got lazy doing things in #rstats until our 0.5TB RAM failed us. Sql is a nice trick to have
Show threadSTOP THE WAVE 🇨🇦Apr 14, 2025
@ChristosArgyrop NoSQL?