DIe Inititative der #IUCr, Rohdaten #FAIR verfügbar zu machen, erfordert ein kompatibles Datenformat. Dieses wird in den Communitys als CIF/CBF bzw. NeXuS in #HDF5-Containern definiert.

Dies führt unter anderem dazu, dass Industrie entsprechende Konverter aus ihren proprietären Dateiformaten bereitstellt 🥳

📢Blosc2 3.10.2 Released! 📢

Building on the functions introduced in version 3.9, we have extended lazy evaluation to general expressions with ALL blosc2 functions in version 3.10.

Blosc2's compute engine ingests many array formats, as well as blosc2 arrays, with impressive results - see the graphic below!
Details on the comparisons in our blog: https://www.blosc.org/posts/tensordot-pure-persistent/

Blosc2's compute engine also powers Cat2Cloud (visit the demo site here https://cat2.cloud/demo/)!

#Zarr
#HDF5
#Python

🚀 Caterva2 2025.8.7 is out!

This release features a major refactoring for a simpler, more robust system. Best of all, client APIs are unaffected. ✅

🌟 Highlights:

The new cat2agent 🤖: A CLI client to watch a directory and auto-sync it with a Caterva2 server.
New stack and concat commands in the web UI for easier data manipulation.
Full release notes: https://github.com/ironArray/Caterva2/releases

Learn more about Caterva2: https://ironarray.io/caterva2

#Caterva2 #Blosc2 #HDF5 #DataStorage #OpenSource #DataScience

"I’m experiencing significant slow-downs when converting data from #LMDB to HDF5 format."

Solution: don't do that. #HDF5 is a pig.

https://stackoverflow.com/questions/79706522/hdf5-write-performance-degrades-over-time-when-converting-from-lmdb-3-7m-entri

I've surprised myself how fun it is to write this pure #haskell #hdf5 parser. It's, in a way, gruesome, fiddling with raw bytes, guessing some details, getting new files with features still unsupported, and I have no clue how exciting it will be to implement all the common filters. But the language is delightful and I'm making constant glacial progress. That accounts for a lot I suppose...

🚀 Excited to share more about Caterva2, your ultimate gateway to Blosc2/HDF5 repositories! 🚀

Caterva2 is designed to redefine how you interact with large datasets.

Want to see it in action? 🤔 We've just released a new introductory video showcasing Caterva2's main functionalities! 🎬

👉 https://ironarray.io/caterva2

#Caterva2 #Blosc2 #HDF5 #BigData #DataManagement #FreeSoftware #Python #DataScience #Tech

✨ Ever wondered how to effortlessly apply lazy expressions to massive tomographic HDF5 datasets and visualize them instantly in Cat2Cloud? 🤯

🎬 Check out this fantastic new video from our very own Luke Shaw! Discover how a bit of Python 🐍 magic and our user-friendly web interface simplify the process.

👇 Watch the full video to see the magic unfold: https://ironarray.io/cat2cloud

#HDF5 #Tomography #LazyEvaluation #DataVisualization #Cat2Cloud

Now we have THomas from #EuXFEL with “down to the bytes: can we simplify alternative access to #HDF5?“

– h5glance

#HDF5 User Group meeting, preparing for day two …

Zdenek from MAX IV #Synchrotron in Lund shows their “innovative data acquisition solutions with HSDS“.

Now it's Gerd from the #HDF5 group showing the history and future of the “most versatile container for sharing scientific and engineering data”.