Evidence map›Paper›PMID 26092286›Full record

ArticleJournal of clinical epidemiology2016

ROBIS: A new tool to assess risk of bias in systematic reviews was developed.

Penny Whiting, Jelena Savović, Julian P T Higgins, Deborah M Caldwell, Barnaby C Reeves, Beverley Shea, Philippa Davies, Jos Kleijnen, Rachel Churchill, ROBIS group

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Journal of clinical epidemiology, 2016. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 1,035 papers, 90 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
1,035citing papers in PubMed, 90 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

1,035 citing papers in PubMed, 90 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Dental caries and periodontal status in children and adolescents with type 1 diabetes mellitus: an umbrella review.European archives of paediatric dentistry : official journal of the European Academy of Paediatric Dentistry · 2026
    Pooled it
  8. Pooled it
  9. Pooled it
  10. Pooled it
  11. Exploring the influence of long COVID upon self-identity: a systematic literature review.Archives of public health = Archives belges de sante publique · 2026
    Pooled it
  12. Pooled it
  13. Pooled it
  14. Pooled it
  15. Pooled it
  16. Pooled it
  17. Pooled it
  18. Pooled it
  19. Pooled it
  20. Pooled it

975 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

5 · Who and what money

Authors and funding

10 authors.

Penny WhitingSchool of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK; The National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West at University Hospitals Bristol NHS Foundation Trust, 9th Floor, Whitefriars, Lewins Mead, Bristol BS1 2NT; Kleijnen Systematic Reviews Ltd, Unit 6, Escrick Business Park, Riccall Road, Escrick, York YO19 6FD, UK. Electronic address: penny.whiting@bristol.ac.uk.
Jelena SavovićSchool of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK; The National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West at University Hospitals Bristol NHS Foundation Trust, 9th Floor, Whitefriars, Lewins Mead, Bristol BS1 2NT.
Julian P T HigginsSchool of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK; Centre for Reviews and Dissemination, University of York, York YO10 5DD, UK.
Deborah M CaldwellSchool of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK.
Barnaby C ReevesSchool of Clinical Sciences, University of Bristol, Bristol Royal Infirmary, Level Queen's Building, 69 St Michael's Hill, Bristol BS2 8DZ, UK.
Beverley SheaCommunity Information and Epidemiological Technologies Institute of Population Health, 1 Stewart Street, Room 319, Ottawa, Ontario, K1N 6N5, Canada.
Philippa DaviesSchool of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK; The National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care West at University Hospitals Bristol NHS Foundation Trust, 9th Floor, Whitefriars, Lewins Mead, Bristol BS1 2NT.
Jos KleijnenKleijnen Systematic Reviews Ltd, Unit 6, Escrick Business Park, Riccall Road, Escrick, York YO19 6FD, UK; School for Public Health and Primary Care (CAPHRI), Maastricht University, PO Box 616, 6200 MD, Maastricht, The Netherlands.
Rachel ChurchillSchool of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK.
ROBIS group

Funding

Department of HealthMedical Research Council G0902118Medical Research Council MC_UU_12013/9Medical Research Council MR/K01465X/1
6 · The paper itself

Abstract

objectiveTo develop ROBIS, a new tool for assessing the risk of bias in systematic reviews (rather than in primary studies). STUDY DESIGN AND

settingWe used four-stage approach to develop ROBIS: define the scope, review the evidence base, hold a face-to-face meeting, and refine the tool through piloting.

resultsROBIS is currently aimed at four broad categories of reviews mainly within health care settings: interventions, diagnosis, prognosis, and etiology. The target audience of ROBIS is primarily guideline developers, authors of overviews of systematic reviews ("reviews of reviews"), and review authors who might want to assess or avoid risk of bias in their reviews. The tool is completed in three phases: (1) assess relevance (optional), (2) identify concerns with the review process, and (3) judge risk of bias. Phase 2 covers four domains through which bias may be introduced into a systematic review: study eligibility criteria; identification and selection of studies; data collection and study appraisal; and synthesis and findings. Phase 3 assesses the overall risk of bias in the interpretation of review findings and whether this considered limitations identified in any of the phase 2 domains. Signaling questions are included to help judge concerns with the review process (phase 2) and the overall risk of bias in the review (phase 3); these questions flag aspects of review design related to the potential for bias and aim to help assessors judge risk of bias in the review process, results, and conclusions.

conclusionsROBIS is the first rigorously developed tool designed specifically to assess the risk of bias in systematic reviews.

Indexed as

BiasReview Literature as TopicRisk AssessmentEvidenceMeta-analysisQualityRisk of biasSystematic reviewTool

Identifiers

PMID26092286
PMCPMC4687950

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.