Mastodawn

I like to do math, small prototypes and other experimenting with Python in Jupyter notebooks. NumPy, SciPy, SymPy and Matplotlib have been very handy.

But recently Python was too slow for what I was doing and I had trouble juggling Numpy arrays like I often have (it is required for speed). I know some #Rust so I decided to give #evcxr a go because it has a Jupyter kernel. And omg, I wasn't disappointed! It was mindbogglingly fast compared to what I'm used to and it also made the hard part of my code way easier to get right.

However, I only needed math functions provided by std this time. Now I'm trying to figure out if there are science Rust libs that are convenient enough to replace most Python in my notebooks. I know it's a fragmented space, but maybe one could already gather a comprehensive collection of crates manually.

Any recommendations?

#RustLang #SciPy #NumPy #SymPy #SciComp #Math

nen Dec 25, 2024

Some promising crates I have found thus far, but haven't tried yet:

- ndarray: convenient NumPy-like array data structure
- nalgebra: linear algebra (more than in ndarray?)
- faer: another linear algebra lib with focus on medium and large matrixes
- num: numeric types (bigint, complex, rational, some functions like gcd)

- peroxide: some scientific computation, quite many things
- mathru: like peroxide (but less messy? idk)
- polars: Pandas replacement
- argmin: optimization algorithms
- petgraph: graph algorithms and types

- rustfft & realfft: FFT for arbitrary sizes, but no fftfreq, fftshift
- ndrustfft: rustfft/realfft (+DCT) for ndarray
- hifitime: time and date for scientific computation

- plotpy: Matplotlib wrapper (supports evcxr/Jupyter and I expect great visual quality)

Some relevant crates I have used at least once:

- rand: random number generation, some distributions
- rayon: easy parallelism, iterators etc
- plotters: plotting (supports evcxr/Jupyter, but seems quite rough)

miaouPlop Dec 25, 2024

@nen I’ve never used it myself but faer (https://docs.rs/faer/latest/faer/) from @sarah_vegan_btw seems very promising

faer - Rust

`faer` is a general-purpose linear algebra library for rust, with a focus on high performance for algebraic operations on medium/large matrices, as well as matrix decompositions

nen Dec 25, 2024

@miaouPlop @sarah_vegan_btw Thanks, I'll add it to the list!

Alavi | علوی Dec 25, 2024

@nen
Doesn't Julia also have C-like performance?
I think that would be a better choice.

nen Dec 25, 2024

@alavi Oh, somehow I hadn't considered Julia from performance standpoint at all. I have pretty limited mental capacity for learning new languages at the moment, but maybe I should seriously think about it.

Pluto seems awesome, btw: https://plutojl.org/

Pluto.jl — interactive Julia programming environment

A free, open source Julia programming environment. Designed to make learning and teaching scientific programming simple!

Alavi | علوی Dec 25, 2024

@nen
I don't know Julia but I think languages like matlab, julia etc. Can be learnt in 3-4 days and you just need a cheatsheet and internet by your side, and you will be good to go.

nen Dec 25, 2024

@alavi Yeah, you are probably right. I'll take a small bite first and try to port my last experiment to Julia and see how easy it is to get to work and how fast I can get it.

I browsed the documentation and had some doubts about parallelism, but this answered them pretty well:

https://stackoverflow.com/questions/73375161/multithreading-vs-multiprocessing-in-julia/73375947#73375947

Multithreading vs Multiprocessing in julia

I am new to Julia and have few confusions regarding Multiprocessing and multithreading. Previously many people have asked similar questions in stackoverflow, but I still have confusions. They are as

Stack Overflow

@nen @alavi Although I am a huge Rust fan, Julia is a better fit for interactive notebook workflows.

See my related blog post comparing Rust and Julia in scientific computing:
https://mo8it.com/blog/rust-vs-julia

Rust vs Julia in scientific computing

Does Julia solve the two-language problem and when should you use Rust instead?

nen Dec 26, 2024

@mo8it @alavi Good summary of why I love Rust. 😄 I really would like to know what it would be like if I had Rust's strong guarantees also when I'm just quickly fooling around with data/maths/stuff. Writing Python I tend to make small mistakes that are very frustrating to debug. All the time.

My initial impression of Julia+Pluto is good! I got distracted a bit, though.

@nen @alavi Pluto notebooks are awesome for teaching or presenting results. But you can also use Jupyter notebooks with Julia :)

nen Dec 26, 2024

@mo8it @alavi A very common case is that I'm just loading updated data or tweaking parameters and then re-plotting, so Pluto's input widgets and automatic execution of dependent cells are super convenient.

guenther Dec 25, 2024

@nen seconding what @alavi said. I usually advocate for Rust any time I have the chance, but if your use case is more like a scipy/matplotlib data analysis thing, I don't think it's worth it.

Aside from Julia, I've heard good things about numba (opt-in JIT for a subset of Python).

https://numba.pydata.org/

Numba: A High Performance Python Compiler

Numba is an open source JIT compiler that translates a subset of Python and NumPy code into fast machine code.

guenther Dec 25, 2024

That said, if you have like a specific, reusable component that's slow in python, maybe you can make use of pyo3.

https://pyo3.rs/

It's basically like pybind for c++, so you can write Rust modules and use them from python (or vice versa, but there are probably fewer use cases for that). It also relatively seamlessly bridges numpy with rust's ndarray.

Introduction - PyO3 user guide

PyO3 user guide

nen Dec 26, 2024

@guenther @alavi I have used Numba occasionally. It's nice when it works. I also tried it in this specific case, but got weird crashes. Then I gave up with Python altogether. It was possibly somehow related to mistakes I had introduced when I translated math from my paper notes to Python. The bugs were finally identified when I ported the code to Rust (if they were the same bugs). Rust complained about out of bounds indices and it was quickly fixed.

The Python code without Numba ran (slowly), but produced unexpected garbage.

CC: @mo8it (why I wanted to try Rust in notebooks)

Alexandre B A Villares 🐍Dec 31, 2024

@nen while still on Python, have you tried scientific computation speed up things like #Numba ? Also, I once saw a #Cython talk and I was almost convinced to give it a go (but I'm too lazy) :D

nen Dec 31, 2024

@villares Yes, I have sometimes. But another problem Python has is that it is difficult to write correct programs with if one is as scatterbrained as I am. ;)

https://mementomori.social/@nen/113720349987988317

nen (@[email protected])

@[email protected] @[email protected] I have used Numba occasionally. It's nice when it works. I also tried it in this specific case, but got weird crashes. Then I gave up with Python altogether. It was possibly somehow related to mistakes I had introduced when I translated math from my paper notes to Python. The bugs were finally identified when I ported the code to Rust (if they were the same bugs). Rust complained about out of bounds indices and it was quickly fixed. The Python code without Numba ran (slowly), but produced unexpected garbage. CC: @[email protected] (why I wanted to try Rust in notebooks)

Memento mori