royalMay 2, 2024

I think in PowerShell and can manage in Python. I want to learn Rust to the degree I can write in it directly, rather than prototyping in PowerShell and then converting.

A lot of what I do is data manipulation and analysis. (Take several CSV files as input, and output new CSV files that answer business questions based on the inputs.) I'm seriously impressed with Rust's performance here.

If you've made this transition, advice on where to begin?

#python #rust #powerShell #programming #dataScience

Show threadJoeMay 2, 2024

@royal Hi, royal. I’m a little curious about your reasoning leading you from A to B.

  • Python has extensive support in modules for data analysis & manipulation, including CSV in & out – pandas comes right to mind. Most would probably assert that Python and R are the big two for this kind of work. Why would you move away from the “center of gravity” for analysis toward more of a systems-level language?

  • How do you use PowerShell? I’d guess gathering the data you intend to analyze from windows systems – generating your CSVs for input? What do translate that prototype into?

  • I hear performance. How much are you finding that to be an issue in your work?

It’s not criticism – just my curiosity :-)

Show threadroyalMay 2, 2024
@MindOfJoe Performance. My last job yielded a 3 million line output and took multiple hours with PS. I know python is faster than PS, but Rust dispatched the same job in a few seconds.
Show threadJoeMay 2, 2024
@royal Performance in performing what tasks? I mean, you might have "gather the data from windows systems" where that's particularly well suited to PowerShell, "process the data" where that's particularly well suited to python or R, and "show the results" which might be something else like excel, powerbi, or whatever. Are you doing everything in powershell and transitioning to everything rust?
Show threadroyal
@MindOfJoe These are small tasks I write myself, not collaborative projects. Windows is not involved. I'm going for fast execution time, which compiled languages are typically good for. I very much like the interactive nature of PowerShell and Python, but am willing to give it up in this case.
May 3, 2024 at 7:50pmWeb
Show threadJoeMay 3, 2024
@royal 20+ years or so ago I was a C & assembly guy handling massive telcoms data sets and line speeds, trying on average to keep up with a bunch of special processing. At the time, C++ was just some add-on libraries for C that made prototyping awesome, but slowed down things too much to deploy. #thegoodoldays :-)

Rust is definitely the sexy thing -- haven't tried it myself as I haven't been in the *must go faster* game for a while, more needing the higher order functions and proven libraries, etc. Definitely curious how it plays out for you on the data analysis front!

Pleased to meet you, BTW :-) // Happy Friday!
Akkoma

Show threadroyalMay 3, 2024

@MindOfJoe Well, when I'm asking questions of my data and won't have an answer until at least the next day, go-fast becomes pretty attractive.

Which assembly were you working with?

Show threadJoeMay 3, 2024
@royal For assembly, I started in the 80's on an IBM PC 8088 and did a little more later on the Intel 386x -- mostly things for fun when the architectures were more open. When I was in the business of making things go really fast, that was the occasional examination and tweaking of inner loops (generated by cc or gcc compilers) on Sparc / Solaris, DEC Alpha, and very briefly on a Cray. Later is was mostly reading 386-family stuff for reverse-engineering and integration tasks. I'm happy enough not to do any of that for a living, but happy to know I could again :-)