Evidence map›Paper›PMID 34809639›Full record

ArticleBMC medicine2021

ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis.

Virginia Chiocchia, Adriani Nikolakopoulou, Julian P T Higgins, Matthew J Page, Theodoros Papakonstantinou, Andrea Cipriani, Toshi A Furukawa, George C M Siontis, Matthias Egger, Georgia Salanti

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in BMC medicine, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 53 papers, 21 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
53citing papers in PubMed, 21 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

53 citing papers in PubMed, 21 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Pooled it
  8. Pooled it
  9. Pooled it
  10. Interventions for central serous chorioretinopathy: a network meta-analysis.The Cochrane database of systematic reviews · 2025
    Pooled it
  11. Pooled it
  12. A meta-analysis of the effect of visiting zoos and aquariums on visitors' conservation knowledge, beliefs, and behavior.Conservation biology : the journal of the Society for Conservation Biology · 2025
    Pooled it
  13. Multimodal treatment improves survival in patients with lung metastases from colorectal cancer: A network meta-analysis.Colorectal disease : the official journal of the Association of Coloproctology of Great Britain and Ireland · 2025
    Pooled it
  14. Pooled it
  15. Pooled it
  16. Pooled it
  17. Pooled it
  18. Pooled it
  19. Pooled it
  20. Pooled it
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

10 authors.

Virginia ChiocchiaInstitute of Social and Preventive Medicine, University of Bern, Bern, Switzerland. virginia.chiocchia@ispm.unibe.ch.ORCID 0000-0002-6196-3308
Adriani NikolakopoulouInstitute of Social and Preventive Medicine, University of Bern, Bern, Switzerland.
Julian P T HigginsPopulation Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK.
Matthew J PageSchool of Public Health and Preventive Medicine, Monash University, Melbourne, Australia.
Theodoros PapakonstantinouInstitute of Social and Preventive Medicine, University of Bern, Bern, Switzerland.
Andrea CiprianiDepartment of Psychiatry, University of Oxford, Oxford, UK.
Toshi A FurukawaDepartment of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine and School of Public Health, Kyoto, Japan.
George C M SiontisDepartment of Cardiology, Bern University Hospital, Inselspital, Bern, Switzerland.
Matthias EggerInstitute of Social and Preventive Medicine, University of Bern, Bern, Switzerland.
Georgia SalantiInstitute of Social and Preventive Medicine, University of Bern, Bern, Switzerland.

Funding

Medical Research Council MR/K025643/1
6 · The paper itself

Abstract

backgroundSelective outcome reporting and publication bias threaten the validity of systematic reviews and meta-analyses and can affect clinical decision-making. A rigorous method to evaluate the impact of this bias on the results of network meta-analyses of interventions is lacking. We present a tool to assess the Risk Of Bias due to Missing Evidence in Network meta-analysis (ROB-MEN).

methodsROB-MEN first evaluates the risk of bias due to missing evidence for each of the possible pairwise comparison that can be made between the interventions in the network. This step considers possible bias due to the presence of studies with unavailable results (within-study assessment of bias) and the potential for unpublished studies (across-study assessment of bias). The second step combines the judgements about the risk of bias due to missing evidence in pairwise comparisons with (i) the contribution of direct comparisons to the network meta-analysis estimates, (ii) possible small-study effects evaluated by network meta-regression, and (iii) any bias from unobserved comparisons. Then, a level of "low risk", "some concerns", or "high risk" for the bias due to missing evidence is assigned to each estimate, which is our tool's final output.

resultsWe describe the methodology of ROB-MEN step-by-step using an illustrative example from a published NMA of non-diagnostic modalities for the detection of coronary artery disease in patients with low risk acute coronary syndrome. We also report a full application of the tool on a larger and more complex published network of 18 drugs from head-to-head studies for the acute treatment of adults with major depressive disorder.

conclusionsROB-MEN is the first tool for evaluating the risk of bias due to missing evidence in network meta-analysis and applies to networks of all sizes and geometry. The use of ROB-MEN is facilitated by an R Shiny web application that produces the Pairwise Comparisons and ROB-MEN Table and is incorporated in the reporting bias domain of the CINeMA framework and software.

Indexed as

Publication BiasAdultHumansMajor Depressive DisorderNetwork Meta-Analysis as TopicRisk AssessmentEvidence synthesisMissing evidenceNetwork meta-analysisPublication biasReporting biasRisk of biasSelective outcome reporting

Identifiers

PMID34809639
PMCPMC8609747

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.