Hybrid foundation models: An investigation of foundation anatomy model and foundation segmentation model (FAM-FSM) on thoracic CT images for radiation therapy planning.

In this paper, we investigated a novel hybrid foundation model (HFM) by combining a foundation anatomy model and a foundation segmentation model (FAM-FSM) for automatic object segmentation on thoracic CT images for radiation therapy planning purposes. The HFM framework first performs object recognition and then object delineation for segmentation. The previous hybrid intelligence (HI) segmentation system using natural intelligence through anatomy models to provide recognition results as prompts to FSM enables the combination of existing recognition with the delineation from FSM. SAM-Med2D is adopted as the FSM for segmentation in this study. 249 thoracic CT images are utilized, and 7 objects of interest are assessed in the experiments. 125 data sets from one site are used for fine-tuning FSM, and 104 testing data sets from another 4 different clinical centers are used for testing the performance of segmentation. Different from the typical training strategy used in multi-center deep learning approaches and federated learning, we did not mix data sets from different centers for training purposes in our study. Instead, consistent and strict body region and object definitions as foundational knowledge and guidance were shared and followed by each clinical center in training and testing. HFM achieved promising results from 4 different centers compared to results from FSM using manual ROI prompts. We also show that the existing FSM might not perform well on challenging objects compared to small but specific trained models. The proposed HFM framework can easily integrate any other prompt-based FSMs.