Evidence map›Paper›PMID 39050728›Full record

ArticleRoyal Society open science2024

Subjective evidence evaluation survey for many-analysts studies.

Alexandra Sarafoglou, Suzanne Hoogeveen, Don van den Bergh, Balazs Aczel, Casper J Albers, Tim Althoff, Rotem Botvinik-Nezer, Niko A Busch, Andrea M Cataldo, Berna Devezer and 30 more

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Royal Society open science, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 1 paper.

0numbers the graph read from it
0cells of the map it votes in
1citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

1 citing paper in PubMed.

  1. Heterogeneity in effect size estimates.Proceedings of the National Academy of Sciences of the United States of America · 2024
    Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

40 authors.

Alexandra SarafoglouDepartment of Psychology, University of Amsterdam, Amsterdam, The Netherlands.ORCID 0000-0003-0031-685X
Suzanne HoogeveenUtrecht University, Utrecht, The Netherlands.ORCID 0000-0002-1304-8615
Don van den BerghDepartment of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
Balazs AczelInstitute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary.ORCID 0000-0001-9364-4988
Casper J AlbersHeymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands.
Tim AlthoffAllen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA.
Rotem Botvinik-NezerHebrew University of Jerusalem, Jerusalem, Israel.
Niko A BuschInstitute for Psychology, University of Münster, Münster, Germany.
Andrea M CataldoCenter for Depression, Anxiety and Stress Research, McLean Hospital, Belmont, MA, USA.
Berna DevezerDepartment of Business, University of Idaho, Moscow, ID, USA.ORCID 0000-0002-5979-2781
Noah N N van DongenDepartment of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
Anna DreberStockholm School of Economics, Stockholm, Sweden.ORCID 0000-0003-3989-9941
Eiko I FriedDepartment of Psychology, Leiden University, Leiden, The Netherlands.
Rink HoekstraNieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands.ORCID 0000-0002-1588-7527
Sabine HoffmanDepartment of Statistics, Ludwig-Maximilians-Universität München, Munchen, Bayern, Germany.ORCID 0000-0002-1304-8615
Felix HolzmeisterUniversity of Innsbruck, Innsbruck, Tirol, Austria.
Jürgen HuberUniversity of Innsbruck, Innsbruck, Tirol, Austria.
Nick Huntington-KleinSeattle University, Seattle, WA, USA.
John IoannidisMeta-Research Innovation Center at Stanford (METRICS) and Departments of Medicine, of Epidemiology and of Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Stanford, CA, USA.ORCID 0000-0003-3118-6859
Magnus JohannessonStockholm School of Economics, Stockholm, Sweden.
Michael KirchlerUniversity of Innsbruck, Innsbruck, Tirol, Austria.
Eric LokenUniversity of Conneticut, Storrs, CT, USA.
Jan-Francois ManginUniversity Paris-Saclay, Gif-sur-Yvette, France.
Dora MatzkeDepartment of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
Albert J MenkveldVrije Universiteit Amsterdam, Amsterdam, Noord-Holland, The Netherlands.
Gustav NilsonneKarolinska Institutet, Solna, Sweden.ORCID 0000-0001-5273-0150
Don van RavenzwaaijHeymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands.ORCID 0000-0002-5030-4091
Martin SchweinsbergESMT Berlin, Berlin, Germany.
Hannah Schulz-KuempelDepartment of Statistics and The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich, Munchen, Bayern, Germany.
David R ShanksDivision of Psychology and Language Sciences, University College London, 26 Bedford Way, London WC1H 0AP, UK.ORCID 0000-0002-4600-6323
Daniel J SimonsUniversity of Illinois-Urbana-Champaign, Urbana, IL, USA.
Barbara A SpellmanSchool of Law, University of Virginia, 580 Massie Road, Charlottesville, VA, USA.
Andrea H StoevenbeltNieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands.
Barnabas SzasziInstitute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary.
Darinka TrübutschekMax Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany.
Francis TuerlinckxUniversity of Leuven, Leuven, Belgium.
Eric L UhlmannINSEAD, Fontainebleau, Île-de-France, France.
Wolf VanpaemelUniversity of Leuven, Leuven, Belgium.ORCID 0000-0002-5855-3885
Jelte WichertsDepartment of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands.
Eric-Jan WagenmakersDepartment of Psychology, University of Amsterdam, Amsterdam, The Netherlands.

Funding

Sequence-structure-function relationships in human visual photopigmentsP20GM104420 · NIGMS · UNIVERSITY OF IDAHO · PI WICHMAN, HOLLY A · 2015 to 2024
$22.9M
NIGMS NIH HHS P20 GM104420
6 · The paper itself

Abstract

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

Indexed as

crowdsourcing analysismetascienceopen sciencescientific transparencyteam science

Identifiers

PMID39050728
PMCPMC11265885

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.