Evidence map›Paper›PMID 26311020›Full record

ArticleBMJ quality & safety2015

Development of the Quality Improvement Minimum Quality Criteria Set (QI-MQCS): a tool for critical appraisal of quality improvement intervention publications.

Susanne Hempel, Paul G Shekelle, Jodi L Liu, Margie Sherwood Danz, Robbie Foy, Yee-Wei Lim, Aneesa Motala, Lisa V Rubenstein

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in BMJ quality & safety, 2015. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 82 papers, 36 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
82citing papers in PubMed, 36 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

82 citing papers in PubMed, 36 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Increasing the environmental sustainability of operating rooms in Canada: an evidence-informed guideline for policy.CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne · 2026
    Guideline
  5. Pooled it
  6. Pooled it
  7. Pooled it
  8. Pooled it
  9. Pooled it
  10. Pooled it
  11. Pooled it
  12. Pooled it
  13. Pooled it
  14. Pooled it
  15. Pooled it
  16. Pooled it
  17. Pooled it
  18. Pooled it
  19. Pooled it
  20. Pooled it

22 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

8 authors.

Susanne HempelRAND Corporation, Santa Monica, California, USA.
Paul G ShekelleRAND Corporation, Santa Monica, California, USA Veterans Affairs West Los Angeles Medical Center, Los Angeles, California, USA.
Jodi L LiuRAND Corporation, Santa Monica, California, USA.
Margie Sherwood DanzRAND Corporation, Santa Monica, California, USA Veterans Affairs Greater Los Angeles, North Hills, California, USA.
Robbie FoyUniversity of Leeds, Leeds Institute of Health Sciences, Leeds, UK.
Yee-Wei LimNational University of Singapore, Saw Swee Hock School of Public Health, Singapore.
Aneesa MotalaRAND Corporation, Santa Monica, California, USA.
Lisa V RubensteinRAND Corporation, Santa Monica, California, USA Veterans Affairs Greater Los Angeles, North Hills, California, USA University of California, Department of Medicine, Los Angeles, California, USA.

Funding

EXPERT PANEL MEETING ADVANCING THE SCIENCE OF CONTINUOUS QUALITY IMPROVEMENTR13HS018139 · AHRQ · RAND CORPORATION · PI RUBENSTEIN, LISA V · 2009 to 2009
$15k
AHRQ HHS 1R13HS018139-01AHRQ HHS R13 HS018139
6 · The paper itself

Abstract

objectiveValid, reliable critical appraisal tools advance quality improvement (QI) intervention impacts by helping stakeholders identify higher quality studies. QI approaches are diverse and differ from clinical interventions. Widely used critical appraisal instruments do not take unique QI features into account and existing QI tools (eg, Standards for QI Reporting Excellence) are intended for publication guidance rather than critical appraisal. This study developed and psychometrically tested a critical appraisal instrument, the QI Minimum Quality Criteria Set (QI-MQCS) for assessing QI-specific features of QI publications.

methodsApproaches to developing the tool and ensuring validity included a literature review, in-person and online survey expert panel input, and application to empirical examples. We investigated psychometric properties in a set of diverse QI publications (N=54) by analysing reliability measures and item endorsement rates and explored sources of disagreement between reviewers.

resultsThe QI-MQCS includes 16 content domains to evaluate QI intervention publications: Organisational Motivation, Intervention Rationale, Intervention Description, Organisational Characteristics, Implementation, Study Design, Comparator Description, Data Sources, Timing, Adherence/Fidelity, Health Outcomes, Organisational Readiness, Penetration/Reach, Sustainability, Spread and Limitations. Median inter-rater agreement for QI-MQCS items was κ 0.57 (83% agreement). Item statistics indicated sufficient ability to differentiate between publications (median quality criteria met 67%). Internal consistency measures indicated coherence without excessive conceptual overlap (absolute mean interitem correlation=0.19). The critical appraisal instrument is accompanied by a user manual detailing What to consider, Where to look and How to rate.

conclusionsWe developed a ready-to-use, valid and reliable critical appraisal instrument applicable to healthcare QI intervention publications, but recognise scope for continuing refinement.

Indexed as

Health Services ResearchHumansPeriodicals as TopicPsychometricsPublishingQuality ImprovementReproducibility of ResultsEvaluation methodologyEvidence-based medicineHealthcare quality improvementQuality improvementQuality improvement methodologies

Identifiers

PMID26311020
PMCPMC4680162

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.