Inter-rater reliability and validity of supervision performance assessment and recognition (SPARS) indicators to assess medicines management in public health facilities in Nepal.

Background: Nepal's Ministry of Health and Population piloted the Supervision, Performance Assessment, and Recognition Strategy (SPARS), a multipronged strategy to improve medicines management practices in health facilities. Medicines management supervisors (MMS) periodically assess facility performance using 25 indicators grouped into 5 domains - prescribing, dispensing, ordering and reporting quality and stock and storage management - and provide targeted on-the-job coaching. This study aimed to determine inter-rater reliability (IRR) and validity for SPARS indicators in Nepali context. Methods: We assessed IRR and validity scores with three rater teams assessing medicines management in six public health posts. Each team of three MMS and one technical advisor evaluated two facilities, resulting in 24 total assessments. We calculated the mean and median validity and IRR scores for SPARS overall, and by domain and indicator. Acceptable scores were mean validity and IRR scores ≥75%, moderately acceptable were 50-75%, and unacceptable were <50%. Results: The mean overall IRR (60%) and validity (74%) of all 25 SPARS indicators were significantly different ( Conclusions: This study demonstrated that the SPARS indicators produce moderately accurate findings. Validity was significantly higher than IRR and both need improvement, especially for complex indicators. As such, future SPARS implementation should endeavour to include validity and IRR assessments to help improve the indicators.