Stuff I would like to read when I have time: [1] Notebook examples about the query optimization [2] the slide deck going into more technical details, [3] other links from the homepage
[1] https://fireducks-dev.github.io/posts/sourav_cse_demo_20240701/
[2] https://fireducks-dev.github.io/files/20241003_PyConZA.pdf
[3] https://fireducks-dev.github.io/
In general, a Data Scientist spends significant efforts in transforming the raw data into a more digestible format before training an AI model or creating visualizations. Traditional tools such as pandas have long been the linchpin in this process, offering powerful capabilities but not without limitations. With the pitfall of its single-core implementation and inefficient data structures, often we face performance issues when dealing with pandas for relatively larger data, but its performance is also highly impacted by the choice of APIs, their parameters and execution orders.
Just learned there's a drop-in replacement for pandas that can be 16x faster, it's called fireducks. You call it with `import fireducks.pandas as pd` & it's built by some supercomputing folks (NEC) specifically with parallelization and automatic query optimization in mind. https://hwisnu.bearblog.dev/fireducks-pandas-but-100x-faster/
Disclaimer: I haven't tried it yet. Also: tragically I really dislike the name ;__;
