Machine learning prediction of bile salt export pump inhibition with Boruta feature selection in imbalanced data.

Introduction: Bile salt export pump (BSEP) inhibition is an important mechanistic risk factor for cholestatic drug-induced liver injury. Quantitative structure-activity relationship (QSAR) models may support early safety screening, but practical BSEP datasets are often limited in size and imbalanced. Methods: We evaluated a machine-learning workflow combining molecular descriptors and fingerprints, Boruta feature selection, and class-imbalance training. Multiple algorithms were assessed using cross-validation, a held-out test set, and an independent external validation set composed of compounds synthesized in an industrial research setting. Results: Tree-based models generally showed the strongest external performance. In particular, models using RDKit2D descriptors achieved Matthews correlation coefficients (MCC) above 0.65 and a ROC-AUC of 0.87, while several other tree-based model representation combinations achieved MCC values around 0.45-0.50. Boruta markedly reduced feature dimensionality, but its effect on predictive performance depended on both the molecular representation and learning algorithm; it was most beneficial for selected high-dimensional fingerprints and tree-based models, while offering limited or unfavorable effects in some support vector machine settings. Applicability-domain and SHAP analyses further revealed representation-dependent generalization patterns and highlighted contributions from lipophilicity, polarity, solubility, and molecular complexity. Discussion: Overall, the workflow maintained useful predictive performance under small and imbalanced conditions and provides a practical framework for early-stage BSEP risk screening.