Evidence map›Paper›PMID 34100223›Full record

ReviewPsychonomic bulletin & review2022

Methods to split cognitive task data for estimating split-half reliability: A comprehensive review and systematic assessment.

Thomas Pronk, Dylan Molenaar, Reinout W Wiers, Jaap Murre

Abstract readReview
PubMed Full text at PMC Publisher
In one paragraph

Review in Psychonomic bulletin & review, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 45 papers.

0numbers the graph read from it
0cells of the map it votes in
45citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

45 citing papers in PubMed.

  1. Trial
  2. The impact of alcohol consumption on implicit racial bias.Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors · 2024
    Trial
  3. Trial
  4. Article
  5. Article
  6. Article
  7. Cross-frequency interactions in band importance functions.The Journal of the Acoustical Society of America · 2026
    Article
  8. Article
  9. Article
  10. Article
  11. Article
  12. Article
  13. Article
  14. Article
  15. Review
  16. No Evidence of Reliability Across 36 Variations of the Emotional Dot-Probe Task in 9,600 Participants.Clinical psychological science : a journal of the Association for Psychological Science · 2025
    Article
  17. Article
  18. Article
  19. Article
  20. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

4 authors.

Thomas PronkFaculty of Social and Behavioural Sciences, Department of Psychology, University of Amsterdam, P.O. Box: 15969, 1001 NL, Amsterdam, The Netherlands. t.pronk@uva.nl.ORCID http://orcid.org/0000-0001-9334-7190
Dylan MolenaarFaculty of Social and Behavioural Sciences, Department of Psychology, University of Amsterdam, P.O. Box: 15969, 1001 NL, Amsterdam, The Netherlands.
Reinout W WiersFaculty of Social and Behavioural Sciences, Department of Psychology, Addiction Development and Psychopathology (ADAPT) lab, University of Amsterdam, Amsterdam, The Netherlands.
Jaap MurreFaculty of Social and Behavioural Sciences, Department of Psychology, University of Amsterdam, P.O. Box: 15969, 1001 NL, Amsterdam, The Netherlands.

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

Estimating the reliability of cognitive task datasets is commonly done via split-half methods. We review four methods that differ in how the trials are split into parts: a first-second half split, an odd-even trial split, a permutated split, and a Monte Carlo-based split. Additionally, each splitting method could be combined with stratification by task design. These methods are reviewed in terms of the degree to which they are confounded with four effects that may occur in cognitive tasks: effects of time, task design, trial sampling, and non-linear scoring. Based on the theoretical review, we recommend Monte Carlo splitting (possibly in combination with stratification by task design) as being the most robust method with respect to the four confounds considered. Next, we estimated the reliabilities of the main outcome variables from four cognitive task datasets, each (typically) scored with a different non-linear algorithm, by systematically applying each splitting method. Differences between methods were interpreted in terms of confounding effects inflating or attenuating reliability estimates. For three task datasets, our findings were consistent with our model of confounding effects. Evidence for confounding effects was strong for time and task design and weak for non-linear scoring. When confounding effects occurred, they attenuated reliability estimates. For one task dataset, findings were inconsistent with our model but they may offer indicators for assessing whether a split-half reliability estimate is appropriate. Additionally, we make suggestions on further research of reliability estimation, supported by a compendium R package that implements each of the splitting methods reviewed here.

Indexed as

AlgorithmsCognitionHumansMonte Carlo MethodReproducibility of ResultsCognitive tasksConfounding effectsNon-linear scoring algorithmsReliability

Identifiers

PMID34100223
PMCPMC8858277

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.