Return on investment from quality improvement programmes in mental healthcare; a Delphi study on conceptual uncertainty and ambiguity.

introductionReturn on Investment (ROI) is a recommended and thus legitimatised approach to assessing the value of healthcare programmes, including in Quality Improvement (QI). In its accounting origins, ROI estimates monetary benefits against investments. In health economics, ROI estimates a monetary value of healthcare related benefits against costs. In a recent single site study, we explored the metaphorical use of ROI as a concept of benefits from QI in-order to develop a QI-ROI conceptual framework. We found that in QI, both monetary and non-monetary benefits are deemed as legitimate ROI by mental healthcare leaders. However, various ambiguities and uncertainties associated with QI-ROI conceptualisation were also apparent. As such, in the current study, we explored consensus on the QI-ROI concept with a wider group of mental healthcare leaders across multiple sites. We then assessed the potential impact of the consensus levels on the stability of our QI-ROI conceptual framework.

methodsWe ran two rounds of Delphi through Qualtrics online platform. Public sector mental health leaders were approached UK-wide (N = > 100). Only leaders from England participated (N = 23). This included board members (n = 15), directors (n = 2), and QI leaders (n = 6). Sixty-seven items were rated, including patient outcomes, development (e.g., culture), external benefits (e.g., socio-economic), incentives (e.g., reputation), implementation outcomes (e.g., sustainability), and monetised benefits. Consensus was measured using interquartile range and median. We also collected qualitative data to help explain participant responses.

resultsThere was consensus on 45 of 67 (67%) of the items in our existing QI-ROI conceptual framework. There was dissensus on 22 of 67 (33%) items. Positive consensus (item acceptance) was on 33 of 45 (73.3%), negative consensus (item rejection) was on 5 of 45 (11.1%) items, and indecision was on 7 of 45 (15.6%). Patient outcomes were rated as most relevant, followed by developmental benefits, and benefits for external stakeholders. External incentives and monetised outcomes were rated as least relevant and legitimate as QI-ROI. Areas of indecision and dissensus were largely on implementation outcomes, measurement, and monetisation of QI benefits. Qualitative data indicated that indecision and dissensus was likely related to organisational needs, or the novelty of some related benefits.

conclusionMental healthcare leaders value various QI benefits depending on organisational needs. Thus, although there is consensus on most aspects, some ambiguity and uncertainty may remain. Regardless, QI-ROI is viewed as both monetary and non-monetary benefits. This has implications for similar organisations and policies relating to the assessment of the value of QI.