Automating data extraction in meta-research: A multi-model benchmark in network psychometrics papers.

Manual data extraction in meta-research is often tedious, time-consuming, and error-prone. In this paper, we investigate whether the current generation of large language models (LLMs) can be used to extract accurate information from scientific papers. Across the meta-research literature, these tasks usually range from extracting verbatim information (e.g., the number of participants in a study, effect sizes, or whether a study is preregistered) to making subjective inferences. Using a publicly available dataset containing a wide range of metascientific variables from 43 network psychometrics papers, we tested five LLMs (Claude 4.6 Opus, Claude 4.5 Sonnet, Claude 4.5 Haiku, GPT-5.2, and GPT-5 mini). We used an automated API-based pipeline to extract variables from the documents. This approach allows batch processing of research papers. As such, it represents a more efficient and scalable way to extract metascientific data than the default chat interface. The extraction accuracy ranged from 79.6% to 91.3% across the models. The extraction performance was generally higher for more explicit, verbatim information and worse for variables that required more complicated inference. Furthermore, most models were able to convey uncertainty in more contentious cases. We provide a comparison of the accuracy and cost-effectiveness of the individual models and discuss the characteristics of variables that are and are not suitable for automatic coding. Furthermore, we describe some of the common pitfalls and best practices of automated LLM data extraction. The proposed procedure can substantially reduce the time and costs associated with conducting meta-research.