Quantifying human variability in pediatric bone age assessment: a multi-institutional multi-reader study and benchmark for AI evaluation.

backgroundBone age assessment using the Greulich and Pyle atlas is widely used in pediatrics but is subject to substantial reader variability. As artificial intelligence (AI) systems increasingly support clinical workflows, understanding the magnitude and structure of human variability is essential for contextualizing AI performance.

objectiveTo quantify human variability in pediatric bone age assessment across multiple institutions and to compare human variability with two automated bone age estimation methods. MATERIALS AND

methodsIn this multi-reader, multi-case study, 1,285 left-hand and wrist radiographs from five US academic centers were independently interpreted by four radiologists per case from a centrally administered pool of 22 radiologists recruited across nine institutions. Bone age estimates were obtained using the Greulich and Pyle atlas. Variability was assessed at the image, rater, and institution levels using mixed-effects modeling. Inter-reader variability was estimated while accounting for patient age and sex. Performance of two automated methods (BoneXpert and a deep-learning algorithm) was evaluated in an interchangeability analysis using a three-rater consensus reference. Accuracy metrics included mean absolute error (MAE), root mean square error (RMSE), and rates of substantial deviation (>1.8 years).

resultsWithin-image variability among radiologists was higher in younger patients (<12 years) than in older patients (standard deviation (SD) 8.7 months vs. 4.2 months, P<0.001). Between-institution variability decreased substantially after adjustment for patient age (8.9 months to 0.3 months). Inter-rater variability remained stable after adjustment (1.2 months). Variability was greater in male patients and younger age groups. Several radiologists demonstrated systematic biases related to age or sex. In comparison with individual human readers, BoneXpert showed lower error (MAE 4.8 months vs. 6.5 months) and variability, while the deep-learning algorithm performed similarly to human readers (MAE 6.3 months). Substantial deviations occurred in 0.7% of BoneXpert estimates, 2.4% of deep-learning estimates, and 3.4% of single human readings.

conclusionPediatric bone age assessment using the Greulich and Pyle method demonstrates measurable human reader variability, particularly in younger children and male patients. Automated methods performed within or beyond the observed range of human variability. These findings provide a quantitative benchmark for interpreting AI performance in skeletal maturity assessment.