Jesper Dr.amsch – Realworld MLOct 30, 2022

Are you a scientist applying Machine Learning?

I wrote a tutorial with ready-to-use notebooks to make your life easier!

Let's focus on 3 aspects:
• More Citations
• Easier Review
• Better Collaboration

☝ First things first!

This was a #EuroScipy2022 tutorial.

In the future, there will be a talk recording. Until then, the gist:

1. Model Evaluation
2. Benchmarking
3. Model Sharing
4. Testing
5. Interpretability
6. Ablation

https://github.com/JesperDramsch/euroscipy-2022-ml-for-science-reproducibility-tutorial

🧵👇

#machinelearning #science #ml4science

GitHub - JesperDramsch/euroscipy-2022-ml-for-science-reproducibility-tutorial: Increase citations, ease review & collaboration A collection of "easy wins" to make machine learning in research reproducible. This tutorial focuses on basics that work. Getting you 90% of the way to top-tier reproducibility.

Increase citations, ease review & collaboration A collection of "easy wins" to make machine learning in research reproducible. This tutorial focuses on basics that work. Getting yo...

GitHub
Show threadJesper Dr.amsch – Realworld MLOct 30, 2022

📐 Model Evaluation

In science, we want to describe the world.

Overfitting gets in the way of this.

With real-world data, there are many ways to overfit, even if we use a random split and have a validation and test set!

Save yourself the pain!

https://dramsch.net/articles/euroscipy-2022/euroscipy-tutorial-1-model-evaluation/

A machine learning model that isn't evaluated correctly is not a scientific result.

This leads to desk rejections, tons of extra work, or in the worst case redactions and being the "bad example".

See:
https://dramsch.net/books/

Show threadJesper Dr.amsch – Realworld ML

🔬 Benchmarking

Compare your models using the right metrics and benchmarks.

Here are great examples:
• DummyClassifiers
• Benchmark Datasets
• Domain Methods
• Linear Models
• Random Forests

Always ground your model in the reality of science!

https://dramsch.net/articles/euroscipy-2022/euroscipy-tutorial-2-benchmarking/

Proper benchmarks make stronger papers!
Metrics on their own don't paint a full picture.

Use benchmarks to tell a story of "how well your model should be doing" and disarm many comments by Reviewer 2 before they're written down.

Oct 30, 2022 at 8:13pmWeb
Show threadJesper Dr.amsch – Realworld MLOct 30, 2022

🤝 Model Sharing

Sharing models is great for reproducibility and collaboration.

Export your models and fix the random seed for paper submissions.

Share your dependencies in a requirements.txt or env.yml so other researchers can use & cite your work!

https://dramsch.net/articles/euroscipy-2022/euroscipy-tutorial-3-model-sharing/

Good code is easy to use and cite!

Use these libraries:
• flake8 for linting
• black for formatting

Write docstrings for docs!
(VS Code has a fantastic extension called autoDocstring)

Your peers will thank you.

Show threadJesper Dr.amsch – Realworld MLOct 30, 2022

⚗️ Testing

I know code testing in science is hard.

Here are ways that make it incredibly easy:
• Doctests for small examples
• Data Tests for important samples
• Deterministic tests for methods

https://dramsch.net/articles/euroscipy-2022/euroscipy-tutorial-4-testing/

You can make your own life and that of collaborators 1000 times easier!

Use Input Validation.

Pandera is a nice little tool that lets you define how your input data should look like. Think:
• Data Ranges
• Data Types
• Category Names

It's honestly a game changer and easy!

Show threadJesper Dr.amsch – Realworld MLOct 30, 2022

🧠 Interpretability

This is a great communication tool for papers and meetings with domain scientists!

No one cares about your mean squared error!

How does the prediction depend on changing your input values?!

What features are important?!

https://dramsch.net/articles/euroscipy-2022/euroscipy-tutorial-5-interpretability/

Show threadJesper Dr.amsch – Realworld MLOct 30, 2022

✂️ Ablation Studies

You know it. I know it.

Data science is trying a lot and finding what works.
It's iterative!

Use ablation studies to switch off components in your solution to evaluate the effect on the final score!

This care is great in a paper!

https://dramsch.net/articles/euroscipy-2022/euroscipy-tutorial-6-ablation-study/

Show threadJesper Dr.amsch – Realworld MLOct 30, 2022

TL;DR

Overall, this tutorial is aimed at applied scientists, exploring ML for their problems.

We looked at 6 ready-to-use notebooks to make your life easier.

This resource is for you to steal and make better science.

Each tool makes it more likely for
• Your results to go through review
• Others to use and cite your stuff
• The code fairy to smile upon you

We focused on “easy wins” scientists can implement in research to avoid catastrophic failures and increase reproducibility.