Internal validation methods to detect patient characteristics associated with differential item functioning in patient-reported outcome measures.

purposeDifferential item functioning (DIF) detection methods are one tool to assess validity of patient-reported outcome measures (PROMs). Given the growing interest in machine-learning models for the analysis of PROMs data, including for DIF detection, internal cross-validation (CV) is recommended to assess model performance. We compared two CV methods for DIF models in simulated and real-world data.

methodsTwo common CV methods, k-fold and holdout CV, were compared for a machine-learning recursive partitioning model to detect DIF. Simulation parameters included sample size, number of items and covariates associated with DIF, and DIF effect size. Simulation performance was assessed using true positive rate (TPR; power) and false positive rate (FPR; Type I error) for items, covariates, and all item-covariate combinations. DIF performance was also assessed in real-world Hospital Anxiety and Depression Scale (HADS) item responses using area under the receiver operating characteristic (AUROC) curve.

resultsIn simulations, TPRs for consistently identifying items and covariates associated with DIF increased as DIF effect size, sample size, and number of items increased for both CV methods. The 10-fold CV method had the highest TPR in many conditions reaching nearly 100% at the item-covariate level while the 50% holdout CV method had the lowest FPR. For HADS items, DIF detection varied between CV methods, with AUROC ranging from 0.74 to 0.80.

conclusionThe choice of a CV method for a machine-learning model to detect DIF affected consistency of DIF item detection. The 10-fold CV method was advantageous for balancing systematic error and data stability.