Evidence map›Paper›PMID 37626309›Full record

ArticleBMC cancer2023

Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters.

Ming Li, Qian Gao, Tianfei Yu

Abstract readLetter
PubMed Full text at PMC Publisher
In one paragraph

Article in BMC cancer, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 83 papers, 10 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
83citing papers in PubMed, 10 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

83 citing papers in PubMed, 10 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Pooled it
  8. Pooled it
  9. Pooled it
  10. Pooled it
  11. Article
  12. Article
  13. Article
  14. Article
  15. Article
  16. Review
  17. Article
  18. Article
  19. Article
  20. Article

23 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

3 authors.

Ming LiDepartment of Computer Science and Technology, College of Computer and Control Engineering, Qiqihar University, Qiqihar, 161006, China.
Qian GaoDepartment of Computer Science and Technology, College of Computer and Control Engineering, Qiqihar University, Qiqihar, 161006, China.
Tianfei YuDepartment of Biotechnology, College of Life Science and Agriculture Forestry, Qiqihar University, Qiqihar, 161006, China. yutianfei2001@163.com.

Funding

Chinese Ministry of Education "Chunhui Plan" International Scientific Research Cooperation Project HLJ2019017Fundamental Research Funds in Heilongjiang Provincial Universities 145209121Heilongjiang Province Leading Talent Echelon Reserve Leader Funding Project Not applicable.
6 · The paper itself

Abstract

backgroundIn research designs that rely on observational ratings provided by two raters, assessing inter-rater reliability (IRR) is a frequently required task. However, some studies fall short in properly utilizing statistical procedures, omitting essential information necessary for interpreting their findings, or inadequately addressing the impact of IRR on subsequent analyses' statistical power for hypothesis testing.

methodsThis article delves into the recent publication by Liu et al. in BMC Cancer, analyzing the controversy surrounding the Kappa statistic and methodological issues concerning the assessment of IRR. The primary focus is on the appropriate selection of Kappa statistics, as well as the computation, interpretation, and reporting of two frequently used IRR statistics when there are two raters involved.

resultsThe Cohen's Kappa statistic is typically utilized to assess the level of agreement between two raters when there are two categories or for unordered categorical variables with three or more categories. On the other hand, when it comes to evaluating the degree of agreement between two raters for ordered categorical variables comprising three or more categories, the weighted Kappa is a widely used measure.

conclusionDespite not substantially affecting the findings of Liu et al.?s study, the statistical dispute underscores the significance of employing suitable statistical methods. Rigorous and accurate statistical results are crucial for producing trustworthy research.

Indexed as

Research DesignUpper ExtremityHumansReproducibility of ResultsCohen’s KappaDWIIntra-rater reliabilityKappa statisticLiver metastasesRECIST 1.1 criteriaWeighted Kappa

Identifiers

PMID37626309
PMCPMC10464133

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.