Generalizations of the Jaccard index and Sørensen index for assessing agreement across multiple readers in object detection and instance segmentation in biomedical imaging.

Significance: Manual annotations are necessary for training supervised learning algorithms for object detection and instance segmentation. These manual annotations are difficult to acquire, noisy, and inconsistent across readers. Aim: The goal of this work is to describe and demonstrate multireader generalizations of the Jaccard and Sørensen indices for object detection and instance segmentation. Approach: The multireader Jaccard and Sørensen indices are described in terms of "calls," "objects," and number of readers. These generalizations reduce to the equations defined by confusion matrix variables in the two-reader case. In a test set of 50 cell microscopy images, we use these generalizations to assess reader variability and compare the performance of an object detection network (Yolov5) and an instance segmentation algorithm (Cellpose2.0) with a group of five human readers using the Mann-Whitney U-test with Bonferroni correction for multiplicity. Results: The multireader generalizations were statistically different from the mean of pairwise comparisons of readers ( Conclusions: Multireader generalizations of the Jaccard and Sørensen indices provide metrics for characterizing the agreement of an arbitrary number of readers on object detection and instance segmentation tasks.