Evidence map›Paper›PMID 31744489›Full record

ArticleBMC medicine2019

Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study.

Anthony Chauvin, Philippe Ravaud, David Moher, David Schriger, Sally Hopewell, Daniel Shanahan, Sabina Alam, Gabriel Baron, Jean-Philippe Regnaux, Perrine Crequit and 6 more

Registry-linked trialAbstract readEvaluation Study
PubMed Full text at PMC Publisher
In one paragraph

Article in BMC medicine, 2019. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It is linked to trial NCT03119376 (Evaluation of an Online Tool), which is not on this map. Cited by 23 papers, 4 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
23citing papers in PubMed, 4 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

NCT03119376 nacompletednot on this map

Evaluation of an Online Tool (COBPeer) Dedicated to Junior Peer Reviewers for Assessing the Reporting of Randomized Controlled Trial Reports

TypeinterventionalSponsorAssistance Publique - Hôpitaux de ParisRan2017 to 2018Enrolled120ConditionsPeer ReviewArmsCOBPEER
3 · Its place in the literature

Who cites it

23 citing papers in PubMed, 4 syntheses or guidelines pooled it.

  1. Guideline
  2. Guideline
  3. Guideline
  4. Guideline
  5. Article
  6. Article
  7. Article
  8. Article
  9. Review
  10. Review
  11. Article
  12. Article
  13. Article
  14. Article
  15. Article
  16. Article
  17. Article
  18. Article
  19. Article
  20. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

16 authors.

Anthony ChauvinUniversité de Paris, CRESS, Inserm, INRA, F75004, Paris, France. anthony.chauvin@aphp.fr.ORCID 0000-0001-8819-0061
Philippe RavaudUniversité de Paris, CRESS, Inserm, INRA, F75004, Paris, France.
David MoherCentre for Journalology, Ottawa Hospital Research Institute, School of Epidemiology and Public Health, University of Ottawa, Ottawa, Canada.
David SchrigerDepartment of Emergency Medicine, School of Medicine, University of California, Los Angeles, CA, USA.
Sally HopewellCentre for Statistics in Medicine, University of Oxford, Oxford, UK.
Daniel ShanahanCochrane Central Executive, London, UK.
Sabina AlamTaylor and Francis group, Abingdon, UK.
Gabriel BaronUniversité de Paris, CRESS, Inserm, INRA, F75004, Paris, France.
Jean-Philippe RegnauxUniversité de Paris, CRESS, Inserm, INRA, F75004, Paris, France.
Perrine CrequitUniversité de Paris, CRESS, Inserm, INRA, F75004, Paris, France.
Valeria MartinezDepartment of Anesthesiology, Assistance Publique Hôpitaux de Paris, Hôpital Raymond Poincaré, Garches, France.
Carolina RiverosUniversité de Paris, CRESS, Inserm, INRA, F75004, Paris, France.
Laurence Le CleachService de Dermatologie, Hopital Mondor, Assistance Publique des Hôpitaux de Paris, Université Paris Est Creteil, EpidermE, 94000, Créteil, France.
Alessandro RecchioniBMC Medicine, London, UK.
Douglas G AltmanCentre for Statistics in Medicine, University of Oxford, Oxford, UK.
Isabelle BoutronUniversité de Paris, CRESS, Inserm, INRA, F75004, Paris, France.

Funding

Cancer Research UK 16895
6 · The paper itself

Abstract

backgroundThe peer review process has been questioned as it may fail to allow the publication of high-quality articles. This study aimed to evaluate the accuracy in identifying inadequate reporting in RCT reports by early career researchers (ECRs) using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process.

methodsWe performed a cross-sectional diagnostic study of 119 manuscripts, from BMC series medical journals, BMJ, BMJ Open, and Annals of Emergency Medicine reporting the results of two-arm parallel-group RCTs. One hundred and nineteen ECRs who had never reviewed an RCT manuscript were recruited from December 2017 to January 2018. Each ECR assessed one manuscript. To assess accuracy in identifying inadequate reporting, we used two tests: (1) ECRs assessing a manuscript using the COBPeer tool (after completing an online training module) and (2) the usual peer-review process. The reference standard was the assessment of the manuscript by two systematic reviewers. Inadequate reporting was defined as incomplete reporting or a switch in primary outcome and considered nine domains: the eight most important CONSORT domains and a switch in primary outcome(s). The primary outcome was the mean number of domains accurately classified (scale from 0 to 9).

resultsThe mean (SD) number of domains (0 to 9) accurately classified per manuscript was 6.39 (1.49) for ECRs using COBPeer versus 5.03 (1.84) for the journal's usual peer-review process, with a mean difference [95% CI] of 1.36 [0.88-1.84] (p < 0.001). Concerning secondary outcomes, the sensitivity of ECRs using COBPeer versus the usual peer-review process in detecting incompletely reported CONSORT items was 86% [95% CI 82-89] versus 20% [16-24] and in identifying a switch in primary outcome 61% [44-77] versus 11% [3-26]. The specificity of ECRs using COBPeer versus the usual process to detect incompletely reported CONSORT domains was 61% [57-65] versus 77% [74-81] and to identify a switch in primary outcome 77% [67-86] versus 98% [92-100].

conclusionsTrained ECRs using the COBPeer tool were more likely to detect inadequate reporting in RCTs than the usual peer review processes used by journals. Implementing a two-step peer-review process could help improve the quality of reporting.

trial registrationClinical.Trials.gov NCT03119376 (Registered April, 18, 2017).

Indexed as

Cross-Sectional StudiesHumansPeer ReviewPeriodicals as TopicPublishingResearch ReportCONSORT statementPeer reviewersRandomized controlled trialsReporting

Identifiers

PMID31744489
PMCPMC6864983

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.