ReviewTrials2023
How to reach agreement: the impact of different analytical approaches to Delphi process results in core outcomes set development.
Review in Trials, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 4 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
4 citing papers in PubMed.
- Defining delayed and missed diagnosis of acute and chronic coronary syndromes: a modified Delphi consensus method.Open heart · 2026Article
- Article
- Plasma IgG and IgM autoantibodies to COPT1 as potential biomarkers for detection of non-small cell lung cancer.Frontiers in immunology · 2025Article
- Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors.
Funding
Abstract
backgroundCore outcomes sets are increasingly used to define research outcomes that are most important for a condition. Different consensus methods are used in the development of core outcomes sets; the most common is the Delphi process. Delphi methodology is increasingly standardised for core outcomes set development, but uncertainties remain. We aimed to empirically test how the use of different summary statistics and consensus criteria impact Delphi process results.
methodsResults from two unrelated child health Delphi processes were analysed. Outcomes were ranked by mean, median, or rate of exceedance, and then pairwise comparisons were undertaken to analyse whether the rankings were similar. The correlation coefficient for each comparison was calculated, and Bland-Altman plots produced. Youden's index was used to assess how well the outcomes ranked highest by each summary statistic matched the final core outcomes sets. Consensus criteria identified in a review of published Delphi processes were applied to the results of the two child-health Delphi processes. The size of the consensus sets produced by different criteria was compared, and Youden's index was used to assess how well the outcomes that met different criteria matched the final core outcomes sets.
resultsPairwise comparisons of different summary statistics produced similar correlation coefficients. Bland-Altman plots showed that comparisons involving ranked medians had wider variation in the ranking. No difference in Youden's index for the summary statistics was found. Different consensus criteria produced widely different sets of consensus outcomes (range: 5-44 included outcomes). They also showed differing abilities to identify core outcomes (Youden's index range: 0.32-0.92). The choice of consensus criteria had a large impact on Delphi results. DISCUSSION: The use of different summary statistics is unlikely to affect how outcomes are ranked during a Delphi process: mean, median, and rates of exceedance produce similar results. Different consensus criteria have a large impact on resultant consensus outcomes and potentially on subsequent core outcomes sets: our results confirm the importance of adhering to pre-specified consensus criteria.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.