Evidence map›Paper›PMID 30577874›Full record

SynthesisSystematic reviews2018

The risk of bias in observational studies of exposures (ROBINS-E) tool: concerns arising from application to observational studies of exposures.

Lisa Bero, Nicholas Chartres, Joanna Diong, Alice Fabbri, Davina Ghersi, Juleen Lam, Agnes Lau, Sally McDonald, Barbara Mintzes, Patrice Sutton and 2 more

Abstract readSystematic Review
PubMed Full text at PMC Publisher
In one paragraph

Synthesis in Systematic reviews, 2018. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 140 papers, 67 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
140citing papers in PubMed, 67 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

140 citing papers in PubMed, 67 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Pooled it
  8. Fatty acid dietary intakes and blood concentrations in relation to hip fracture risk in adults: a systematic review and meta-analysis of prospective cohort studies.Osteoporosis international : a journal established as result of cooperation between the European Foundation for Osteoporosis and the National Osteoporosis Foundation of the USA · 2025
    Pooled it
  9. Pooled it
  10. Pooled it
  11. Effects of the inhaled dose of air pollution on health: a systematic review.European respiratory review : an official journal of the European Respiratory Society · 2025
    Pooled it
  12. Pooled it
  13. Pooled it
  14. Pooled it
  15. Pooled it
  16. Pooled it
  17. Pooled it
  18. Pooled it
  19. Pooled it
  20. Pooled it

80 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

12 authors.

Lisa BeroCharles Perkins Centre and School of Pharmacy, Faculty of Medicine and Health, The University of Sydney, D17, The Hub, 6th floor, Sydney, New South Wales, 2006, Australia. Lisa.bero@sydney.edu.au.ORCID 0000-0003-1893-6651
Nicholas ChartresCharles Perkins Centre and School of Pharmacy, Faculty of Medicine and Health, The University of Sydney, D17, The Hub, 6th floor, Sydney, New South Wales, 2006, Australia.
Joanna DiongSchool of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia.
Alice FabbriCharles Perkins Centre and School of Pharmacy, Faculty of Medicine and Health, The University of Sydney, D17, The Hub, 6th floor, Sydney, New South Wales, 2006, Australia.
Davina GhersiNational Health and Medical Research Council, Canberra, Australia.
Juleen LamDepartment of Ob/Gyn & the Institute for Health Policy Studies, University of California, San Francisco, USA.
Agnes LauSchool of Pharmacy, University of California, San Francisco, USA.
Sally McDonaldCharles Perkins Centre, The University of Sydney, Sydney, Australia.
Barbara MintzesSchool of Pharmacy, Faculty of Medicine and Health and Charles Perkins Centre, The University of Sydney, Sydney, Australia.
Patrice SuttonDepartment of Ob/Gyn & the Institute for Health Policy Studies, University of California, San Francisco, USA.
Jessica Louise TurtonCharles Perkins Centre, The University of Sydney, Sydney, Australia.
Tracey J WoodruffDepartment of Ob/Gyn & the Institute for Health Policy Studies, University of California, San Francisco, USA.

Funding

The UCSF Pregnancy Exposures to Environmental Chemicals (PEEC) Children's CenterP01ES022841 · NIEHS · UNIVERSITY OF CALIFORNIA, SAN FRANCISCO · PI WOODRUFF, TRACEY J. · 2013 to 2018
$3.9M
national health and medical research council APP1139997NIEHS NIH HHS P01 ES022841
6 · The paper itself

Abstract

backgroundSystematic reviews, which assess the risk of bias in included studies, are increasingly used to develop environmental hazard assessments and public health guidelines. These research areas typically rely on evidence from human observational studies of exposures, yet there are currently no universally accepted standards for assessing risk of bias in such studies. The risk of bias in non-randomised studies of exposures (ROBINS-E) tool has been developed by building upon tools for risk of bias assessment of randomised trials, diagnostic test accuracy studies and observational studies of interventions. This paper reports our experience with the application of the ROBINS-E tool.

methodsWe applied ROBINS-E to 74 exposure studies (60 cohort studies, 14 case-control studies) in 3 areas: environmental risk, dietary exposure and drug harm. All investigators provided written feedback, and we documented verbal discussion of the tool. We inductively and iteratively classified the feedback into 7 themes based on commonalities and differences until all the feedback was accounted for in the themes. We present a description of each theme.

resultsWe identified practical concerns with the premise that ROBINS-E is a structured comparison of the observational study being rated to the 'ideal' randomised controlled trial. ROBINS-E assesses 7 domains of bias, but relevant questions related to some critical sources of bias, such as exposure and funding source, are not assessed. ROBINS-E fails to discriminate between studies with a single risk of bias or multiple risks of bias. ROBINS-E is severely limited at determining whether confounders will bias study outcomes. The construct of co-exposures was difficult to distinguish from confounders. Applying ROBINS-E was time-consuming and confusing.

conclusionsOur experience suggests that the ROBINS-E tool does not meet the need for an international standard for evaluating human observational studies for questions of harm relevant to public and environmental health. We propose that a simpler tool, based on empirical evidence of bias, would provide accurate measures of risk of bias and is more likely to meet the needs of the environmental and public health community.

Indexed as

BiasEnvironmental ExposureObservational Studies as TopicPublic HealthEnvironmental HealthNutritional StatusCochraneEnvironmentGRADEGuidelinesNutritionObservational studyPublic health guidelinesQuality assessmentRisk of biasSystematic review

Identifiers

PMID30577874
PMCPMC6302384

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.