Evidence map›Paper›PMID 34696810›Full record

ReviewSystematic reviews2021

A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2.

R Perry, A Whitmarsh, V Leach, P Davies

Abstract readReview
PubMed Full text at PMC Publisher
In one paragraph

Review in Systematic reviews, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 61 papers, 18 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
61citing papers in PubMed, 18 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

61 citing papers in PubMed, 18 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Pooled it
  8. Pooled it
  9. Pooled it
  10. Pooled it
  11. Pooled it
  12. Pooled it
  13. Pooled it
  14. Pooled it
  15. Health Effects of Electronic Cigarettes: An Umbrella Review and Methodological Considerations.International journal of environmental research and public health · 2022
    Pooled it
  16. Pooled it
  17. Pooled it
  18. Association Between Periodontitis and Mild Cognitive Impairment, Alzheimer's Disease, and Dementia: An Umbrella Review With Evidence Stratification.Special care in dentistry : official publication of the American Association of Hospital Dentists, the Academy of Dentistry for the Handicapped, and the American Society for Geriatric Dentistry
    Pooled it
  19. Review
  20. Review

1 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

4 authors.

R PerryNational Institute for Health Research Bristol Biomedical Research Centre, University Hospitals Bristol and Weston NHS Foundation Trust and University of Bristol, Bristol, UK. Rachel.Perry@bristol.ac.uk.ORCID 0000-0001-5874-3016
A WhitmarshNational Institute for Health Research Bristol Biomedical Research Centre, University Hospitals Bristol and Weston NHS Foundation Trust and University of Bristol, Bristol, UK.
V LeachThe National Institute for Health Research Applied Research Collaboration West (NIHR ARC West) at University Hospitals Bristol NHS Foundation Trust, Bristol, UK.
P DaviesThe National Institute for Health Research Applied Research Collaboration West (NIHR ARC West) at University Hospitals Bristol NHS Foundation Trust, Bristol, UK.

Funding

Department of Health
6 · The paper itself

Abstract

backgroundAMSTAR-2 is a 16-item assessment tool to check the quality of a systematic review and establish whether the most important elements are reported. ROBIS is another assessment tool which was designed to evaluate the level of bias present within a systematic review. Our objective was to compare, contrast and establish both inter-rater reliability and usability of both tools as part of two overviews of systematic reviews. Strictly speaking, one tool assesses methodological quality (AMSTAR-2) and the other assesses risk of bias (ROBIS), but there is considerable overlap between the tools in terms of the signalling questions.

methodsThree reviewers independently assessed 31 systematic reviews using both tools. The inter-rater reliability of all sub-sections using each instrument (AMSTAR-2 and ROBIS) was calculated using Gwet's agreement coefficient (AC

resultsThirty-one systematic reviews were included. For AMSTAR-2, the median agreement for all questions was 0.61. Eight of the 16 AMSTAR-2 questions had substantial agreement or higher (> 0.61). For ROBIS, the median agreement for all questions was also 0.61. Eleven of the 24 ROBIS questions had substantial agreement or higher.

conclusionROBIS is an effective tool for assessing risk of bias in systematic reviews and AMSTAR-2 is an effective tool at assessing quality. The median agreement between raters for both tools was identical (0.61). Reviews that included a meta-analysis were easier to rate with ROBIS; however, further developmental work could improve its use in reviews without a formal synthesis. AMSTAR-2 was more straightforward to use; however, more response options would be beneficial.

Indexed as

Reproducibility of ResultsBiasHumansSystematic Reviews as TopicAMSTAR-2Methodological qualityRisk of biasROBISSystematic reviews

Identifiers

PMID34696810
PMCPMC8543959

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.