#AcademicChatter Hello, fellow academic chatters. I'm an experimental scientist. And I regularly have the problem of how to organize acquired data by different participants to a same experiment. I'd like to have some feedback of how other people do this. Not necessarily specific computer programmes (such as digital labbooks), but rather how-tos based on robust, basic things, such as text and csv files, folder structures etc. I'm often battling with the workflow which is including handwritten measurements, their digital counterparts, metadata for the measurements, codes for samples, all of these done by different people, but all will have to end in one final dataset, where I would like to be able to trace data back to the paper files, and their metadata. And all of this with not too much additional effort by everybody ;-) Have done this usually with some shared server space, but usually it ends up as something a bit difficult to get through. I'm now rather thinking about a system involving perhaps something based on #MarkDown or even #RMarkdown or #knitr etc. Any feedback much welcome 😁
#ReproducibleResearch
@olibrendel maybe you could have a look into mqtt (which is useful to collect and distribute data from different places/sources), jsonl/json, ... e.g. with an App like lablog ( https://f-droid.org/packages/si.uni_lj.fe.lablog ) .
I find it convenient to handle/extract from/browse/search through those json files (if the data structure is well designed).
LabLog | F-Droid - Free and Open Source Android App Repository

A digital lab notebook with MQTT

@olibrendel You might try reading what @briney has published and see if any of it resonates! There was even a recent book: https://pelagicpublishing.com/products/the-data-management-workbook
The Data Management Workbook

@QuantumDot2 @olibrendel Thanks for the mention! It sounds like some good data management is needed. I would actually check your library because you might have someone at your institution who can do a consult on this topic. But also, yes, I have a bunch of materials available (book and blog) to help in this area.
@QuantumDot2 Thanks Andy,
@briney Hello Kristin, very interesting blog, lots of usefull informations. I will have a closer look and also at your latest book. I already have "Reproducible Research with R and RStudio" by Christopher Gandrud, which also contains a chapter on data management, using markdown etc. but going further into knitr.
https://www.taylorfrancis.com/books/mono/10.1201/9780429031854/reproducible-research-rstudio-christopher-gandrud
Reproducible Research with R and RStudio | Christopher Gandrud | Taylo

Praise for previous editions:"Gandrud has written a great outline of how a fully reproducible research project should look from start to finish, with brief

Taylor & Francis
@olibrendel @QuantumDot2 It sounds like you'll want a balance of data management principles applied using software tools. E.g. determining a file naming convention (see this worksheet: https://doi.org/10.7907/894q-zr22) and then integrating that into your code/Markdown. If you understand some of the data management principles, it will be easier to streamline your analysis.
@briney @QuantumDot2 Yes, naming convention is the foundation. I guess for what I was looking for returns of experience is how to tie the different files together, from raw data coming from different manual measurements or equipments, to consolidated data which would lead to some type of analysis to finalised data to be submitted to some repository. Everything could be traced by a readme.txt (or readme.md ;-) per folder, but when the data-collection is big enough, these readme's become rapidly un-readable. I was thinking about MD because some MD systems allow for automatic summaries, and MD allows linking among files, and it is pretty versatile and independent from specific software, as still human readable. I guess I'm looking for some experiences with these kind of set-ups 🙂
#AcademicChatter