AI performance in oral cytology for differentiating poorly defined tumor cells from reactive atypia.

Background: Despite its investigative potential, few studies have reported the use of artificial intelligence (AI) in oral cytology. Oral mucosal cells display significant cellular atypia due to inflammatory stimulation or denaturation, whereas well-differentiated oral squamous cell carcinomas do not always show remarkable cellular atypia. The presence of noncancerous atypical cells alongside ill-defined tumor cells poses significant challenges to the development of effective AI tools. Thus, we aimed to investigate the effect of these atypical cells on AI performance. Materials and methods: We used 29 cases of non-neoplastic lesions, including gingivitis, stomatitis, and 17 squamous cell carcinomas for supervised learning and validation. The cells were classified into four categories: normal, cancer, orange-suspicious, and green-suspicious. Orange and green suspicious cells indicated tumor cells lacking definitive morphological features. Annotation was performed using VoTT v2.2.0, and YOLOv7 as the object detection model, with model training being performed in six ways. Results: The model that learned orange- and green-suspicious cells as cancer exhibited the highest detection capabilities, but also yielded a high number of false positives. In contrast, the model that excluded information about suspicious cells could rightfully identify some suspicious cells as cancer with fewer false positives. Conclusions: Discriminating ill-defined tumor cells from atypical non-neoplastic cells based solely on morphology is challenging. Classifying suspicious cells as cancer often results in numerous false positives. Conversely, AI trained on normal and cancer can reveal previously unnoticed cancerous features in suspicious cells.