@jonny @cogsci @neuralreckoning
Here is the query I raised in a couple of off-fediverse forums:

I would greatly appreciate pointers to any introduction/guide/tutorial on appropriate statistical methods for evaluating computational research studies.

Context: I am starting to do some work in a research field where the researchers are mostly computer scientists and most of the studies are computational experiments. The statistical analysis of the results generated by those studies is nonexistent to naive - often results consist only of a table of means, and you might get standard deviations (not standard errors). I would like to do better from a statistical point of view, but (a) it's a very long time since I had to think about analogous issues in non-computational disciplines, and (b) some statistical issues may be specific to computational experiments, e.g. re-using the same random number stream as input, and non-independence between the folds of a multi-fold validation. Also, I am concerned that trying to introduce some statistical sophistication will attract negative comments from reviewers - so I want to be able to cite something that points out what the statistical problems are and how to deal with them.

Example: Graph Neural Networks are machine learning models that operate on graphs as input. A typical task is to learn to label graphs (classification of graphs), for example, represent chemical molecular structures as graphs and classify them as mutagenic or not. Researchers develop new GNN algorithms and want to compare their performance to other GNN algorithms. There is an archive of graph datasets, which is typically used for this comparison. There are many datasets in the archive, but of course this archive is just a convenience sample of all possible graph datasets (if that even makes sense). Each dataset contains some number (not always large) of labelled graphs (cases). The cases are randomly partitioned into train/test sets, the GNN trained on the train set, then the trained GNN evaluated on the test set. The evaluation metric is usually a single number summary - accuracy (they really love accuracy as the metric). The random train/test partitioning is repeated some small number of times (k-fold validation) to get a distribution of evaluation metrics, and the mean value is reported. This is done independently for each of the GNNs to be compared.

I have not, so far, seen any discussion of: the convenience sample nature of the dataset archive, possible advantages of comparing GNNs on the same train/test partition, issues around accuracy as a metric, or statistical dependence between folds in k-fold validation because of sharing cases. So I am trying find resources identifying statistical issues in that kind of research and potential statistical approaches for analysing the results of the experiments.