RE: https://fediscience.org/@snakemake/116571962095785816

This little bit "performance improvements" lowered the number of file system access events for considerably! #Snakemake triggers many such events for keeping track of metadata. Which is important, but may cause some delays due to file system overhead - particularly on parallel and/or network file systems. The feature to outsource parts of this to sqlite was implemented during the #SnakemakeHackathon2026 . I hope, I can test the improvements next Monday!

#HPC #ReproducibleComputing

RE: https://fediscience.org/@biohackrxiv/116477505622903440

Today, we published the summary (as a #BioHackrXiv preprint) of the #SnakemakeHackathon2026 . All the accomplishments which contribute once more to improve the #Snakemake "ecosystem" for #reproducibleResearch & #Dataanalysis .

Many thanks to @egonw from BioHackrXiv for helping us!

"Snakemake Hackathon 2026" https://doi.org/10.37044/osf.io/h6zqj_v1

"Nonetheless, the platform’s continued evolution faces several open challenges: improving core performance on heterogeneous high-performance-computing (HPC) resources, extending the plugin architecture for domain-specific extensions, and lowering the entry barrier for novice users while preserving full reproducibility. Here we report on the Snakemake Hackathon 2026, convened in Munich, Germany (9–13 March 2026) with more than 40 participants representing academia, industry, and national-level research infrastructure." https://index.biohackrxiv.org/2026/04/27/h6zqj.html

#SnakemakeHackathon2026 #biohackathon #snakemake

The #SnakemakeHackathon2026 has ended, we are still preparing our preprint release. But, our host has prepared a note on their homepage: https://go.tum.de/946236 🥳

#Snakemake #ReproducibleComputing

RE: https://fediscience.org/@snakemake/116295568336688286

This is a big step forward: The SLURM plugin for Snakemake now supports so-called job arrays. These are cluster jobs, with ~ equal resource requirements in terms of memory and compute resources.

The change in itself was big: The purpose of a workflow system is to make use of the vast resources of an HPC cluster. Hence, jobs are submitted to run concurrently. However, for a job array, we have to "wait" for all eligible jobs to be ready. And then we submit.

To preserve concurrent execution of other jobs which are ready to be executed, a thread pool has been introduced. In itself, I do not see job arrays as such a big feature: The LSF system profited much more from arrays than the rather lean SLURM implementation does.

BUT: the new code base will ease further development to pooling many shared memory tasks (applications which support no parallel execution or are confined to one computer by "only" supporting threading). Until then, there is more work to do.

#HPC #SLURM #Snakemake #SnakemakeHackathon2026 #ReproducibleComputing #OpenScience

Did you know? We started to gather cluster profiles for Snakemake users in a little repo: https://github.com/snakemake/snakemake-cluster-profiles

They should serve as a template for others.

#HPC #Snakemake #ReproducibleComputing #SnakemakeHackathon2026

2/3 accomplished ✅

Now, for a release a dependency has to be released with my two fixes, which have already been merged. Then, the world will see a new #nanopub application.

A #SnakemakeHackathon2026 result.

RE: https://fediscience.org/@snakemake/116245074392817916

Hm, I thought(!) that I fixed some issues with this bot. Appears not to be the case. Moreover, the `**docs** should be bold, not ignored. 🤦

Any way, this release is yet another leftover from the #SnakemakeHackathon2026 !

Personally, the week in Munich at the #SnakemakeHackathon2026 was really neat. I met friends and acquaintances, took the time to meet an old friend of mine not working in academia any more and the wonderful @FrankSonntag from our #FediScience association.

And now, tired, on my way back. Thanks to the railway service in Germany, I enjoy some boredom and the opportunity to do something else. Even reading a disc world novel (which is not "novel" any more) does not help.

#lifeis2short