ArticleBMC medical research methodology2011
Conducting online expert panels: a feasibility and experimental replicability study.
Article in BMC medical research methodology, 2011. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 63 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
63 citing papers in PubMed, 2 syntheses or guidelines pooled it, 112 citations in OpenAlex.
- Assessing the process and outcome of the development of practice guidelines and recommendations: PANELVIEW instrument development.CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne · 2020Pooled it
- Eight Priorities for Improving Primary Care Access Management in Healthcare Organizations: Results of a Modified Delphi Stakeholder Panel.Journal of general internal medicine · 2020Pooled it
- Curating lesson level learning objectives for non-majors introductory biology.Journal of microbiology & biology education · 2026Article
- Home- and Community-Based Service Use and Preferences Among Post-9/11 Veterans With or at High Risk of Alzheimer Disease and Related Dementia and Their Caregivers: Protocol for a Mixed Methods Observational Study.JMIR research protocols · 2026Observational
- Integrating real-time polling into modified Delphi methodology: lessons from a consensus study on cardiac implantable electronic device remote monitoring.BMC medical research methodology · 2026Article
- Content validity of the Scale for Assessing the Risk of Phlebitis in Hospitalized Adults.Revista brasileira de enfermagem · 2026Article
- Surgeon-Informed Clinical Decision Support Software for Surgical Risk Prediction and Outcomes Tracking.The Journal of surgical research · 2025Article
- AI-Augmented Content Validation in Behavioral Research: Development and Evaluation of the RATER System.MIS quarterly : management information systems · 2025Article
- AIFM-ed Curriculum Framework for Postgraduate Family Medicine Education on Artificial Intelligence: Mixed Methods Study.JMIR medical education · 2025Article
- Inventory of ethical problems in mobile pre-hospital care.Revista brasileira de enfermagem · 2025Article
- Reporting Guideline for Therapeutic Immersive Virtual Reality Interventions: International eDelphi Study (INVIRTUE).Journal of medical extended reality · 2025Article
- Assessing the feasibility and likelihood of policy options to lower specialty drug costs.Health affairs scholar · 2024Article
- Quality Indicators for Transition from Pediatric to Adult Care for Youth With Chronic Conditions: Proposal for an Online Modified Delphi Study.JMIR research protocols · 2024Article
- Hospital Quality Indicators for Opioid-Exposed Infants: Results From an Expert Consensus Panel.Pediatrics · 2024Article
- ROMPER: The RAND/USC OPTIC Method for Policy Expert Ratings.MethodsX · 2024Article
- Article
- Expert Panel Consensus on State-Level Policies to Improve Engagement and Retention in Treatment for Opioid Use Disorder.JAMA health forum · 2022Article
- Planning and Reporting Effective Web-Based RAND/UCLA Appropriateness Method Panels: Literature Review and Preliminary Recommendations.Journal of medical Internet research · 2022Review
- Consensus-Based Guidance on Opioid Management in Individuals With Advanced Cancer-Related Pain and Opioid Misuse or Use Disorder.JAMA oncology · 2022Article
- Expert views on state-level naloxone access laws: a qualitative analysis of an online modified-Delphi process.Harm reduction journal · 2022Article
3 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors at 4 institutions in 2 countries.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundThis paper has two goals. First, we explore the feasibility of conducting online expert panels to facilitate consensus finding among a large number of geographically distributed stakeholders. Second, we test the replicability of panel findings across four panels of different size.
methodWe engaged 119 panelists in an iterative process to identify definitional features of Continuous Quality Improvement (CQI). We conducted four parallel online panels of different size through three one-week phases by using the RAND's ExpertLens process. In Phase I, participants rated potentially definitional CQI features. In Phase II, they discussed rating results online, using asynchronous, anonymous discussion boards. In Phase III, panelists re-rated Phase I features and reported on their experiences as participants.
results66% of invited experts participated in all three phases. 62% of Phase I participants contributed to Phase II discussions and 87% of them completed Phase III. Panel disagreement, measured by the mean absolute deviation from the median (MAD-M), decreased after group feedback and discussion in 36 out of 43 judgments about CQI features. Agreement between the four panels after Phase III was fair (four-way kappa=0.36); they agreed on the status of five out of eleven CQI features. Results of the post-completion survey suggest that participants were generally satisfied with the online process. Compared to participants in smaller panels, those in larger panels were more likely to agree that they had debated each others' view points.
conclusionIt is feasible to conduct online expert panels intended to facilitate consensus finding among geographically distributed participants. The online approach may be practical for engaging large and diverse groups of stakeholders around a range of health services research topics and can help conduct multiple parallel panels to test for the reproducibility of panel conclusions.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.