Potential and limitations of a large language model in the statistical review of comparative categorical data: an exploratory study of structured prompt-guided approach.

backgroundStatistical review is essential for research quality and integrity, yet traditional manual review is inefficient. Large language models (LLMs) offer potential support but are unreliable when used without guidance for precise calculations and raise concerns about accountability. This study evaluated whether a structured, rule-based prompt can reliably constrain an LLM to perform statistical review of comparative categorical data, and characterized both its feasibility and its inherent risks from an accountability perspective.

methodsThis study employed a two-stage design based on the DeepSeekV3.2. In the first stage, a structured prompt was developed through dozens of "test-fail-iterate" cycles using 20 published medical articles. The prompt assigned the LLM the role of a "statistics expert" and provided a closed set of computational rules and a "recognize data-select calculation formula-calculate" workflow for analyzing categorical data, including Pearson's Chi-square test, continuity correction, and McNemar's tests. In the second stage, the performance of the final prompt was evaluated on a test set of 20 independent manuscripts. The model's output was compared against the results calculated by a senior statistician (the gold standard). The primary outcome measures were the performance in statistical method selection and numerical computation, including accuracy, sensitivity (recall), specificity, positive predictive value, negative predictive value, F1 score, and Cohen's Kappa. Secondary measures included reproducibility and efficiency.

resultsThe test set consisted of 15 manuscripts with independent samples and 5 with paired samples. In the assessment of the appropriateness of statistical method selection for 148 analysis items, the model achieved an accuracy of 99.3% (147/148), a sensitivity of 96.2% (25/26) (F1=98.0%, κ=0.976). For the test of computational consistency in 97 independent sample tests, the accuracy for χ

conclusionsA structured, rule-based prompt can guide the DeepSeek to achieve high accuracy in standardized statistical review tasks, but its reliability is contingent on operational stability. Inherent failure modes, including performance instability and a strong anchoring effect on author-provided data, persist and can lead to significant errors, particularly when source data are flawed. These findings suggest that the the DeepSeek is not suitable for autonomous auditing. Their most appropriate application is as assistive tools within a human-in-the-loop framework, where rigorous human supervision is essential for risk mitigation and to maintain ultimate accountability.