Statistics#

Group-level tests on a feature table (features), with permutation inference and correction for testing many features at once.

Designs#

Function

Tests

Model

test_groups(table, column, (a, b), covariates=[...])

group a minus group b

GLM: intercept, a 0/1 dummy for a, centred covariates

test_groups(..., welch=True)

the same, without covariates

Welch’s t

test_paired(table, (a, b), covariates=[...])

condition a minus b within each recording

one-sample GLM on the differences, by sign flipping

test_correlation(table, score, covariates=[...])

the slope of a numeric column

GLM: intercept, the score, centred covariates

  • Covariates: numeric columns are centred. Text columns (such as sex) are dummy-coded against their first level, then centred.

  • Missing values: rows missing any value the design needs are dropped, and report.dropped counts them.

  • One row per subject: every design needs exactly one row per subject, because rows must be exchangeable under the null. Average runs, or filter, first. A table built from epochs has one row per condition, so test_groups and test_correlation need condition=.

Inference#

  • Test statistic. Each design is a general linear model, and the statistic is the t of the coefficient of interest. Nuisance covariates are handled by the Freedman–Lane procedure: the residuals of the covariates-only model are permuted (or sign-flipped) and added back to its fit (Freedman & Lane, 1983; Anderson & Robinson, 2001; Winkler et al., 2014). With welch=True and no covariates, two groups are compared with Welch’s t, which does not assume equal variances (Delacre, Lakens & Leys, 2017).

  • Paired designs flip the signs of the within-subject differences, assuming they are symmetric around zero under the null (Nichols & Holmes, 2002).

  • p-values are (b + 1) / (m + 1), where b of m rearrangements reach the observed statistic, so they are never zero (Phipson & Smyth, 2010). When every sign flip fits in n_permutations (2ⁿ of them for n subjects), all are enumerated and the p-value is exact.

  • Family-wise error uses the null distribution of the largest statistic over all features (Nichols & Holmes, 2002; Westfall & Young, 1993).

  • False discovery rate: Benjamini–Hochberg by default, for independent or positively dependent tests, or Benjamini–Yekutieli (fdr="by") for any dependence (Benjamini & Hochberg, 1995; Benjamini & Yekutieli, 2001).

  • Network-based statistic: edges above nbs_threshold form connected components, each sized by its number of edges, and each component’s p comes from the null distribution of the largest component (Zalesky, Fornito & Bullmore, 2010).

Tails. tail="both" uses |t|; "greater" and "less" test one direction. For NBS, test one direction at a time (Zalesky, Fornito & Bullmore, 2010): pass tail="greater" or "less" together with nbs_threshold.

NBS across modes. Each mode’s edges form a separate network (m{k}). Components never cross networks. The maximum is taken over all networks, so FWER covers every mode tested.

Ties. A rearranged statistic within a relative 10⁻¹² of the observed one counts as at least as large. Rearrangements that are equal in exact arithmetic (flipping every sign in a two-sided test gives exactly −t) must not be split by rounding, which differs between builds.

Reproducibility. The same seed gives the same p-values on any machine and any number of threads, because each rearrangement draws from its own random stream (ChaCha8).

Equal variances. The GLM assumes equal residual variance across groups. With unequal group sizes and variances, permutation p-values for group differences can be inexact. Use welch=True when there are no covariates; Winkler’s variance-group v statistic, the general fix, is planned.

Planned#

  • General contrasts and F tests.

  • Winkler’s v statistic for unequal variances with covariates.

  • Restricted exchangeability (blocks) for several rows per subject.

References#

  • Anderson, M. J., & Robinson, J. (2001). Permutation tests for linear models. Australian & New Zealand Journal of Statistics, 43(1), 75–88. https://doi.org/10.1111/1467-842X.00156

  • Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B, 57(1), 289–300.

  • Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29(4), 1165–1188. https://doi.org/10.1214/aos/1013699998

  • Delacre, M., Lakens, D., & Leys, C. (2017). Why psychologists should by default use Welch’s t-test instead of Student’s t-test. International Review of Social Psychology, 30(1), 92–101. https://doi.org/10.5334/irsp.82

  • Freedman, D., & Lane, D. (1983). A nonstochastic interpretation of reported significance levels. Journal of Business & Economic Statistics, 1(4), 292–298.

  • Nichols, T. E., & Holmes, A. P. (2002). Nonparametric permutation tests for functional neuroimaging: A primer with examples. Human Brain Mapping, 15(1), 1–25. https://doi.org/10.1002/hbm.1058

  • Phipson, B., & Smyth, G. K. (2010). Permutation p-values should never be zero. Statistical Applications in Genetics and Molecular Biology, 9(1), Article 39. https://doi.org/10.2202/1544-6115.1585

  • Westfall, P. H., & Young, S. S. (1993). Resampling-Based Multiple Testing. Wiley.

  • Winkler, A. M., Ridgway, G. R., Webster, M. A., Smith, S. M., & Nichols, T. E. (2014). Permutation inference for the general linear model. NeuroImage, 92, 381–397. https://doi.org/10.1016/j.neuroimage.2014.01.060

  • Zalesky, A., Fornito, A., & Bullmore, E. T. (2010). Network-based statistic: Identifying differences in brain networks. NeuroImage, 53(4), 1197–1207. https://doi.org/10.1016/j.neuroimage.2010.06.041