ArticleBMC medical education2021
Reliability of simulation-based assessment for practicing physicians: performance is context-specific.
Article in BMC medical education, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 12 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
12 citing papers in PubMed.
- Assessment of prehospital tracheal intubation technique using initial direct laryngoscopy during videolaryngoscopy: randomized controlled simulated trial.BMC emergency medicine · 2025Trial
- Proximity to primary board certification and prior crisis management experience are associated with anesthesiologists' performance in high-fidelity acute care scenarios.medRxiv : the preprint server for health sciences · 2026Article
- Maternal Health Nurse Competency in Addressing Emergency Obstetric Care: An Assessment in Southern Mozambique.Nursing reports (Pavia, Italy) · 2026Article
- How might we interpret quantitative findings in simulation research more meaningfully?Advances in simulation (London, England) · 2026Article
- Reframing Validity: How Values Shape Assessment in Postgraduate Medical Education.Perspectives on medical education · 2026Article
- Simulation-based training for fascial plane blocks: A scoping review mapped to the Kirkpatrick evaluation framework.Indian journal of anaesthesia · 2026Review
- Reliability and validity of simulation-based Electrocardiogram assessment rubrics for cardiac life support skills among medical students using generalizability theory.Medical education online · 2025Article
- Simulation-based training program effect on pediatric nurses' knowledge and performance regarding heel-prick during newborn blood screening test.BMC nursing · 2025Article
- Real-time assessment of triage nurse situational awareness (SA) using the situation awareness global assessment technique (SAGAT).PloS one · 2025Observational
- Community Pharmacists' Knowledge in Managing Minor Ailments: A Focus on Childhood Gastroenteritis in Saudi Arabia Using a Simulated Patient Approach.Healthcare (Basel, Switzerland) · 2024Article
- Development of peer assessment rubrics in simulation-based learning for advanced cardiac life support skills among medical students.Advances in simulation (London, England) · 2024Article
- Simulation-based summative assessment in healthcare: an overview of key principles for practice.Advances in simulation (London, England) · 2022Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
15 authors.
Funding
Abstract
introductionEven physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.
methodsStandardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.
resultsThe behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.
conclusionsThe performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.