Permutation Tests for Regression, (Repeated Measures) ANOVA/ANCOVA and Comparison of Signals

Functions to compute p-values based on permutation tests. Regression, ANOVA and ANCOVA, omnibus F-tests, marginal unilateral and bilateral t-tests are available. Several methods to handle nuisance variables are implemented (Kherad-Pajouh, S., & Renaud, O. (2010) <doi:10.1016/j.csda.2010.02.015> ; Kherad-Pajouh, S., & Renaud, O. (2014) <doi:10.1007/s00362-014-0617-3> ; Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Nichols, T. E. (2014) <doi:10.1016/j.neuroimage.2014.01.060>). An extension for the comparison of signals issued from experimental conditions (e.g. EEG/ERP signals) is provided. Several corrections for multiple testing are possible, including the cluster-mass statistic (Maris, E., & Oostenveld, R. (2007) <doi:10.1016/j.jneumeth.2007.03.024>) and the threshold-free cluster enhancement (Smith, S. M., & Nichols, T. E. (2009) <doi:10.1016/j.neuroimage.2008.03.061>).

Significance tests, p-values, and falsificationism

This thread takes its inspiration from the recent discussions in social science and statistics about significance tests, what they’re good for, whether p-values should be banned, and what all of that has to do with general scientific methodology, particularly the Popperian one called falsificationism. See this Twitter thread for a random place to jump into the discussion. That place, however, was where @f2harrell suggested we create a thread over here. I’ll start this thread off, if I may, with...

A Pragmatic Approach to Statistical Testing and Estimation (PASTE)

The p-value has dominated research in education and related fields and a statistically non-significant p-value is quite commonly interpreted as ‘confirming’ the null hypothesis (H0) of ‘equivalence’. This is unfortunate, because p-values are not fit for that purpose. This paper discusses three alternatives to the traditional p-value that unfortunately have remained underused but can provide evidence in favor of ‘equivalence’ relative to ‘non-equivalence’: two one-sided tests (TOST) equivalence testing, Bayesian hypothesis testing, and information criteria. TOST equivalence testing and p-values both rely on concepts of statistical significance testing and can both be done with confidence intervals, but treat H0 and the alternative hypothesis (H1) differently. Bayesian hypothesis testing and the Bayesian credible interval aka posterior interval provide Bayesian alternatives to traditional p-values, TOST equivalence testing, and confidence intervals. However, under conditions outlined in this paper, confidence intervals and posterior intervals may yield very similar interval estimates. Moreover, Bayesian hypothesis testing and information criteria provide fairly easy to use alternatives to statistical significance testing when multiple competing models can be compared. Based on these considerations, this paper outlines a pragmatic approach to statistical testing and estimation (PASTE) for research in education and related fields. In a nutshell, PASTE states that all of the alternatives to p-values discussed in this paper are better than p-values, that confidence intervals and posterior intervals may both provide useful interval estimates, and that Bayesian hypothesis testing and information criteria should be used when the comparison of multiple models is concerned.

how to cheat at settlers by loading the dice