Evidence map›Paper›PMID 30606262›Full record

ArticleSystematic reviews2019

The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies.

Caitlin E Kennedy, Virginia A Fonner, Kevin A Armstrong, Julie A Denison, Ping Teresa Yeh, Kevin R O'Reilly, Michael D Sweat

Open access · goldAbstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Systematic reviews, 2019. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 103 papers, 61 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
103citing papers in PubMed, 61 pooled it
16.6field-weighted citation impact, top 1% of its field
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

103 citing papers in PubMed, 61 syntheses or guidelines pooled it, 207 citations in OpenAlex.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Pooled it
  8. Pooled it
  9. Pooled it
  10. Pooled it
  11. Pooled it
  12. Pooled it
  13. Pooled it
  14. Pooled it
  15. Effects of exercise on life satisfaction of people diagnosed with cancer: a systematic review and meta-analysis.Supportive care in cancer : official journal of the Multinational Association of Supportive Care in Cancer · 2024
    Pooled it
  16. Pooled it
  17. Pooled it
  18. Pooled it
  19. Pooled it
  20. Self-assembling peptides for managing white spot lesions: a systematic review and meta-analysis.European archives of paediatric dentistry : official journal of the European Academy of Paediatric Dentistry · 2023
    Pooled it

43 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

7 authors at 2 institutions in 1 country.

Caitlin E KennedySocial and Behavioral Interventions Program, Department of International Health, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, Room E5547, Baltimore, MD, 21205, USA. caitlinkennedy@jhu.edu.
Virginia A FonnerDivision of Global and Community Health, Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina, 176 Croghan Spur Road, Suite 104, Charleston, SC, 29407, USA.
Kevin A ArmstrongDivision of Global and Community Health, Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina, 176 Croghan Spur Road, Suite 104, Charleston, SC, 29407, USA.
Julie A DenisonSocial and Behavioral Interventions Program, Department of International Health, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, Room E5547, Baltimore, MD, 21205, USA.
Ping Teresa YehSocial and Behavioral Interventions Program, Department of International Health, Johns Hopkins Bloomberg School of Public Health, 615 North Wolfe Street, Room E5547, Baltimore, MD, 21205, USA.ORCID 0000-0002-7425-0382
Kevin R O'ReillyDivision of Global and Community Health, Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina, 176 Croghan Spur Road, Suite 104, Charleston, SC, 29407, USA.
Michael D SweatDivision of Global and Community Health, Department of Psychiatry and Behavioral Sciences, Medical University of South Carolina, 176 Croghan Spur Road, Suite 104, Charleston, SC, 29407, USA.
Medical University of South Carolina · USJohns Hopkins University · US

Funding

Synthesizing HIV Behavioral Intervention Effectiveness in Developing CountriesR01MH090173 · NIMH · MEDICAL UNIVERSITY OF SOUTH CAROLINA · PI SWEAT, MICHAEL D · 2010 to 2019
$4.2M
HIV Intervention Effectiveness in Developing CountriesR01MH071204 · NIMH · MEDICAL UNIVERSITY OF SOUTH CAROLINA · PI SWEAT, MICHAEL D · 2004 to 2007
$1.2M
NIMH NIH HHS R01 MH071204NIMH NIH HHS R01 MH090173
6 · The paper itself

Abstract

backgroundDifferent tools exist for assessing risk of bias of intervention studies for systematic reviews. We present a tool for assessing risk of bias across both randomized and non-randomized study designs. The tool was developed by the Evidence Project, which conducts systematic reviews and meta-analyses of behavioral interventions for HIV in low- and middle-income countries.

methodsWe present the eight items of the tool and describe considerations for each and for the tool as a whole. We then evaluate reliability of the tool by presenting inter-rater reliability for 125 selected studies from seven published reviews, calculating a kappa for each individual item and a weighted kappa for the total count of items.

resultsThe tool includes eight items, each of which is rated as being present (yes) or not present (no) and, for some items, not applicable or not reported. The items include (1) cohort, (2) control or comparison group, (3) pre-post intervention data, (4) random assignment of participants to the intervention, (5) random selection of participants for assessment, (6) follow-up rate of 80% or more, (7) comparison groups equivalent on sociodemographics, and (8) comparison groups equivalent at baseline on outcome measures. Together, items (1)-(3) summarize the study design, while the remaining items consider other common elements of study rigor. Inter-rater reliability was moderate to substantial for all items, ranging from 0.41 to 0.80 (median κ = 0.66). Agreement between raters on the total count of items endorsed was also substantial (κ

conclusionsStrengths of the tool include its applicability to a range of study designs, from randomized trials to various types of observational and quasi-experimental studies. It is relatively easy to use and interpret and can be applied to a range of review topics without adaptation, facilitating comparability across reviews. Limitations include the lack of potentially relevant items measured in other tools and potential threats to validity of some items. To date, the tool has been applied in over 30 reviews. We believe it is a practical option for assessing risk of bias in systematic reviews of interventions that include a range of study designs.

Indexed as

BiasSystematic Reviews as TopicHumansNon-Randomized Controlled Trials as TopicObservational Studies as TopicObserver VariationRandomized Controlled Trials as TopicReproducibility of ResultsRisk AssessmentCritical appraisalQuality assessmentRigor assessmentRigor scoreRisk of biasStudy qualityStudy rigor

Identifiers

PMID30606262
PMCPMC6317181
OpenAlexW2939489030

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.