Do the limits of methods reach those who use them? What citations in the imaging-neuroscience literature suggest

Progress note from the assessment of the foundations of imaging neuroscience, week to 11 Oct 2026

Every measurement method has known limits. In imaging neuroscience they were usually described by the method’s own authors or their closest colleagues, carefully and with numbers: under what conditions, and by how much, the method gives a distorted result. Such a paper is a warning — against misreading a result when the method is used outside the conditions in which it holds. It is addressed to researchers who use the method in their own studies. We asked: does it reach them?

A warning can travel to the user through many channels — the research literature, software documentation, teaching, peer review, laboratory habits. We follow a single channel and a single trace in it: whether a study that uses the method also cites the paper on its limit. Citing is not reading, understanding or complying; it is only the visible trace that the warning was present in that text.

The picture this trace shows is more interesting than we expected. Limits appear in the citations of application studies neither „all“ nor „none“, but selectively — some kinds more often than others.

What we counted and how we read it

We distinguish four things. Measurement is a number with a well-defined quantity, scale, limits and conditions of validity. Testing is measurement with a criterion fixed in advance. Qualified assessment is an expert classification or selection that is not a measurement. Qualified estimate is a judgement about what was not measured, resting on what was. Our work lies mostly between measurement and assessment — which is why we say so up front.

Two methods used by thousands of studies: the canonical haemodynamic response in fMRI analysis (Friston et al. 1998) and FreeSurfer, software for measuring brain structure (Fischl et al. 2002). For each, the papers that state its limits: Boynton et al. (1996) — linearity holds within a range; Glover (1999) — separate events by at least 4 s; Vazquez & Noll (1998) — below 4 s the response is not linear; Birn et al. (2001) — for short stimuli the true response can be up to ten times off the linear prediction, differently in different brain regions. For FreeSurfer: Jovicich et al. (2009) — scanner and site effects; Gronenschild et al. (2012) — software version and operating system change the results, between FreeSurfer versions structure volumes differ by 8.8 % on average (up to 64 % in the extreme) and cortical thickness by 2.8 %, workstation type and OS version by about half that; Glatard et al. (2015) — operating system and numerical precision.

We counted, in the OpenAlex citation database (snapshot of 11 Oct 2026), how many studies that cite the method also cite the paper on its limit, compared each limit with a „sibling“ from the same method family (Friston et al. 1994; Dale et al. 1999), split citers into methods and application literature, and counted only studies published after the limit paper. The count itself is exact. But it is a proxy: whether a warning reaches the method’s users is something we infer from it, not something we measure.

We tested one thing only: for Birn’s paper we recorded in advance that it would be cited markedly less than its sibling in applications. The other results are counts, not a test.

We assessed — and here the grey zone begins for us too. The choice of the seven limit papers, the choice of siblings and the split into limits that say where a method holds and those that say by how much it can fail are our classifications. The split into methods and application literature we took from OpenAlex’s automatic classification and accepted as good enough without measuring its accuracy. The same holds for the completeness of citation links. These are exactly the attributes of a measurement that in practice are quietly taken for granted.

What the citation counts showed

For each limit we compare two shares: how many of the studies citing the method cite the limit paper, and how many cite the sibling. The ratio r is their quotient (1 = the limit is cited as often as the sibling).

Glatard’s paper is cited so rarely that there are not enough data for a statistical test; listed for completeness.

Of 1,448 application studies citing the canonical response, four cite Birn (0.28 %, 95 % interval 0.11–0.71 %). We wrote down the direction for Birn before running the computation. How strong is the evidence? Recomputed on frozen data with every study counted exactly once, and corrected for seven statistical tests: the direction is the same in every variant for all three „alarming“ limits, but it is statistically secure only for Jovicich (p = 0.002); Birn is borderline (0.10; 0.024 in one variant) and so is Gronenschild (0.10). Our first analysis showed stronger results — it counted studies citing both papers twice. We withdrew it. In the open full texts of our corpus, 10 of the 24 studies that name their analysis software give its version, and 4 of 24 name the operating system.

What follows in practice? Three conclusions, each modest and well supported.

1. Limits that say where a method works do appear in the citations of application studies. „Linearity holds“ and „use 4 s spacing“ are cited in applications as often as, or more often than, the comparison („sibling“) paper of the same method. The split into „where“ and „by how much“ is our working classification and does not fit perfectly: Vazquez & Noll says where linearity fails, yet is cited little in applications (r = 0.64).
2. Limits that say by how much it can fail appear there less — so the data suggest: the direction is consistent, statistically secure so far for one of three such papers. The tenfold deviation for short stimuli stays in the methods literature; so does the finding that a structural result depends on the software version.
3. Not citing is not violating. A study can follow Glover’s rule without citing him, and its stimuli may be long enough for Birn not to apply. We followed the trace of warnings in one channel, not whether studies are wrong. A warning missing from citations may reach authors by another route; and a citation, conversely, does not guarantee it was taken in.

Questions we ask openly

In the academic paper we will phrase these more cautiously. Here we ask them directly.

• Is citing a limit a warning or a licence? Glover is cited to justify a design: „4 s spacing, following Glover“. The citation does not warn; it permits. Birn permits nothing — it only says the result may lie elsewhere. Do mainly the limits that can serve as a stamp of approval make it into citations?
• Who should cite Birn? Precisely the studies with short stimuli in regions of strong non-linearity — the studies for which the citation would raise an awkward question. Are warnings selected by whom they would inconvenience?
• Why do even cheap precautions not take hold? „Freeze the version and report it“ costs nothing, yet in the openly available full texts fewer than half of the studies that name their software report the version (14 of 24 do not — a small, selective sample). Is a limit that questions the instrument less comfortable than one that questions the experiment?
• Which attributes of a measurement get „trimmed“, when and by whom? On the way from method to application, a well-defined measurement turns into one where some conditions of validity are accepted as self-evident or irrelevant without anyone showing it. Which ones, at which step are they lost, and what is the cause — and what the motive, since the two need not coincide? And how large a role do the culture of the field, institutions and still softer factors play: habit, convenience, publication pressure? The licence and the discomfort in the previous questions are two cases of the same thing.
• Should every method carry a package insert? A medicine’s side effects travel with every box. A method travels with its name only. Who in a field decides what is packed with a method?

Citations cannot answer these questions. They do show which questions a field does not ask itself.

References

• Boynton, G. M., Engel, S. A., Glover, G. H. & Heeger, D. J. (1996). Linear systems analysis of functional magnetic resonance imaging in human V1. Journal of Neuroscience, 16(13), 4207–4221. https://doi.org/10.1523/jneurosci.16-13-04207.1996
• Vazquez, A. L. & Noll, D. C. (1998). Nonlinear aspects of the BOLD response in functional MRI. NeuroImage, 7(2), 108–118. https://doi.org/10.1006/nimg.1997.0316
• Friston, K. J. et al. (1998). Event-related fMRI: characterizing differential responses. NeuroImage, 7(1), 30–40. https://doi.org/10.1006/nimg.1997.0306
• Friston, K. J. et al. (1994). Analysis of functional MRI time-series. Human Brain Mapping, 1(2), 153–171. https://doi.org/10.1002/hbm.460010207
• Glover, G. H. (1999). Deconvolution of impulse response in event-related BOLD fMRI. NeuroImage, 9(4), 416–429. https://doi.org/10.1006/nimg.1998.0419
• Dale, A. M., Fischl, B. & Sereno, M. I. (1999). Cortical surface-based analysis. I. Segmentation and surface reconstruction. NeuroImage, 9(2), 179–194. https://doi.org/10.1006/nimg.1998.0395
• Birn, R. M., Saad, Z. S. & Bandettini, P. A. (2001). Spatial heterogeneity of the nonlinear dynamics in the FMRI BOLD response. NeuroImage, 14(4), 817–826. https://doi.org/10.1006/nimg.2001.0873
• Fischl, B. et al. (2002). Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron, 33(3), 341–355. https://doi.org/10.1016/s0896-6273(02)00569-x
• Jovicich, J. et al. (2009). MRI-derived measurements of human subcortical, ventricular and intracranial brain volumes: reliability effects of scan sessions, acquisition sequences, data analyses, scanner upgrade, scanner vendors and field strengths. NeuroImage, 46(1), 177–192. https://doi.org/10.1016/j.neuroimage.2009.02.010
• Gronenschild, E. H. B. M. et al. (2012). The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements. PLoS ONE, 7(6), e38234. https://doi.org/10.1371/journal.pone.0038234
• Glatard, T. et al. (2015). Reproducibility of neuroimaging analyses across operating systems. Frontiers in Neuroinformatics, 9, 12. https://doi.org/10.3389/fninf.2015.00012
• OpenAlex (Priem, Piwowar & Orr 2022), queries 11 Oct 2026. https://openalex.org · https://arxiv.org/abs/2205.01833

Our reference documentation (working .md; TeX typesetting in preparation): neuro-assessment/PART-2-PHYSICS-v0.1-EN.md §3h (A7, A7b, A7c, A7d and the A8 recomputation on a frozen snapshot, including withdrawn and corrected measures); scripts scripts/a7.py, a8_snapshot.py, a8_analyse.py; data with checksums in data/A7 and data/A8-. Currently in a private working repository. References verified 12 Oct 2026: Friston 1998, Dale 1999, Fischl 2002 and Jovicich 2009 via PubMed (PMID 9500830, 9931268, 11832223, 19233293), Friston 1994 via Crossref.*

Forthcoming. This result will be part of an academic paper we are preparing for arXiv, with full method, data and code. The text will pass a second review and be announced here.

Method and limits. Citations from OpenAlex, whose citation links are incomplete. The methods/application split follows OpenAlex topic classification (T10378) and is approximate. The statistical test is computed on a frozen snapshot of 23,390 citing works (11 Oct 2026, two independent snapshots identical) with disjoint groups — every study exactly once — and Holm correction for seven tests; the first version, which counted studies citing both the limit and the sibling twice, is withdrawn. Not citing does not mean not complying. A measure we withdrew along the way (ratio to the field average) is documented: it gave the control pair the same value and therefore said nothing. Three times: generator of text and analysis scripts — Claude language model (Cowork), 2026-10-11 to 12; oracle — our own expectations recorded in advance (commit 3173329) and the disjoint Fisher test; domain — the OpenAlex citation graph as of snapshot 2026-10-11T11:52Z.