ArticleBMJ (Clinical research ed.)2022
Can artificial intelligence pass the Fellowship of the Royal College of Radiologists examination? Multi-reader diagnostic accuracy study.
Article in BMJ (Clinical research ed.), 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 14 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
14 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Artificial intelligence in commercial fracture detection products: a systematic review and meta-analysis of diagnostic test accuracy.Scientific reports · 2024Pooled it
- External validation of an artificial intelligence tool for fracture detection in children with osteogenesis imperfecta: a multireader study.European radiology · 2026Article
- AI in peer review: can artificial intelligence be an ally in reducing gender and geographical gaps in peer review? A randomized trial.Research integrity and peer review · 2025Article
- Utilization of Generative AI-drafted Responses for Managing Patient-Provider Communication.NPJ digital medicine · 2025Article
- Bias Sensitivity in Diagnostic Decision-Making: Comparing ChatGPT with Residents.Journal of general internal medicine · 2025Article
- Enhancing Diagnostic Accuracy of Lung Nodules in Chest Computed Tomography Using Artificial Intelligence: Retrospective Analysis.Journal of medical Internet research · 2025Article
- Age against the machine-susceptibility of large language models to cognitive impairment: cross sectional analysis.BMJ (Clinical research ed.) · 2024Article
- Analysis of large-language model versus human performance for genetics questions.European journal of human genetics : EJHG · 2024Article
- Generative pretrained transformer-4, an artificial intelligence text predictive model, has a high capability for passing novel written radiology exam questions.International journal of computer assisted radiology and surgery · 2024Article
- Commercially available artificial intelligence tools for fracture detection: the evidence.BJR open · 2024Review
- Improved Performance of ChatGPT-4 on the OKAP Examination: A Comparative Study with ChatGPT-3.5.Journal of academic ophthalmology (2017) · 2023Article
- Large language models (LLM) and ChatGPT: what will the impact on nuclear medicine be?European journal of nuclear medicine and molecular imaging · 2023Article
- Efficacy of Artificial Intelligence in the Categorisation of Paediatric Pneumonia on Chest Radiographs: A Systematic Review.Children (Basel, Switzerland) · 2023Review
- Assessment of a combined musculoskeletal and chest deep learning-based detection solution in an emergency setting.European journal of radiology open · 2023Article
Corrections and comments
- Commented on byBeyond the2023
Authors and funding
6 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectiveTo determine whether an artificial intelligence candidate could pass the rapid (radiographic) reporting component of the Fellowship of the Royal College of Radiologists (FRCR) examination.
designProspective multi-reader diagnostic accuracy study.
settingUnited Kingdom.
participantsOne artificial intelligence candidate (Smarturgences, Milvue) and 26 radiologists who had passed the FRCR examination in the preceding 12 months.
main outcome measuresAccuracy and pass rate of the artificial intelligence compared with radiologists across 10 mock FRCR rapid reporting examinations (each examination containing 30 radiographs, requiring 90% accuracy rate to pass).
resultsWhen non-interpretable images were excluded from the analysis, the artificial intelligence candidate achieved an average overall accuracy of 79.5% (95% confidence interval 74.1% to 84.3%) and passed two of 10 mock FRCR examinations. The average radiologist achieved an average accuracy of 84.8% (76.1-91.9%) and passed four of 10 mock examinations. The sensitivity for the artificial intelligence was 83.6% (95% confidence interval 76.2% to 89.4%) and the specificity was 75.2% (66.7% to 82.5%), compared with summary estimates across all radiologists of 84.1% (81.0% to 87.0%) and 87.3% (85.0% to 89.3%). Across 148/300 radiographs that were correctly interpreted by >90% of radiologists, the artificial intelligence candidate was incorrect in 14/148 (9%). In 20/300 radiographs that most (>50%) radiologists interpreted incorrectly, the artificial intelligence candidate was correct in 10/20 (50%). Most imaging pitfalls related to interpretation of musculoskeletal rather than chest radiographs.
conclusionsWhen special dispensation for the artificial intelligence candidate was provided (that is, exclusion of non-interpretable images), the artificial intelligence candidate was able to pass two of 10 mock examinations. Potential exists for the artificial intelligence candidate to improve its radiographic interpretation skills by focusing on musculoskeletal cases and learning to interpret radiographs of the axial skeleton and abdomen that are currently considered "non-interpretable."
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.