ArticleProceedings of the National Academy of Sciences of the United States of America2024
ChatGPT as Research Scientist: Probing GPT's capabilities as a Research Librarian, Research Ethicist, Data Generator, and Data Predictor.
Article in Proceedings of the National Academy of Sciences of the United States of America, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 21 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
21 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Large Language Models in Lung Cancer: Systematic Review.Journal of medical Internet research · 2025Pooled it
- Article
- Like humans, language models demonstrate face-to-character biases.PNAS nexus · 2026Article
- Using LLM-generated tools to extract information about reporting statistical software in biomedical and health science research articles.BMC research notes · 2026Article
- LLMs as emerging tools for understanding and managing bone metastasis and cancer-induced bone pain.iScience · 2026Article
- A concordance between patent and trademark classes to link technologies to markets.Scientific data · 2026Article
- Deciphering AI-chatbots utilization in clinical practice, research and education among Chinese medical students in generative AI era: a cross-sectional study.BMC medical education · 2026Article
- Moral stereotyping in large language models.Proceedings of the National Academy of Sciences of the United States of America · 2026Article
- iDesignGPT enhances conceptual design via large language model agentic workflows.Nature communications · 2026Article
- Opinion paper: generative AI and the future of scientometrics.Scientometrics · 2026Article
- Toward an integrated animal-human nutrition intelligence via agentic retrieval-augmented language models.Frontiers in nutrition · 2026Article
- ChatGPT does not replicate human moral judgments: the importance of examining metrics beyond correlation to assess agreement.Scientific reports · 2025Article
- Influence of Topic Familiarity and Prompt Specificity on Citation Fabrication in Mental Health Research Using Large Language Models: Experimental Study.JMIR mental health · 2025Article
- Risks of AI scientists: prioritizing safeguarding over autonomy.Nature communications · 2025Review
- Artificial Intelligence in Action: Racial and Gender Disparities in Academic Radiology.Cureus · 2025Article
- Kernels of selfhood: GPT-4o shows humanlike patterns of cognitive dissonance moderated by free choice.Proceedings of the National Academy of Sciences of the United States of America · 2025Article
- Generalization bias in large language model summarization of scientific research.Royal Society open science · 2025Article
- AnnCovDB: a manually curated annotation database for mutations in SARS-CoV-2 spike protein.Database : the journal of biological databases and curation · 2025Article
- Multiple Confabulations Found in Bioinformatics Tasks Carried Out by Several Free Large Language Models.Current genomics · 2025Article
- Should Artificial Intelligence Play a Durable Role in Biomedical Research and Practice?International journal of molecular sciences · 2024Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
Abstract
How good a research scientist is ChatGPT? We systematically probed the capabilities of GPT-3.5 and GPT-4 across four central components of the scientific process: as a Research Librarian, Research Ethicist, Data Generator, and Novel Data Predictor, using psychological science as a testing field. In Study 1 (Research Librarian), unlike human researchers, GPT-3.5 and GPT-4 hallucinated, authoritatively generating fictional references 36.0% and 5.4% of the time, respectively, although GPT-4 exhibited an evolving capacity to acknowledge its fictions. In Study 2 (Research Ethicist), GPT-4 (though not GPT-3.5) proved capable of detecting violations like p-hacking in fictional research protocols, correcting 88.6% of blatantly presented issues, and 72.6% of subtly presented issues. In Study 3 (Data Generator), both models consistently replicated patterns of cultural bias previously discovered in large language corpora, indicating that ChatGPT can simulate known results, an antecedent to usefulness for both data generation and skills like hypothesis generation. Contrastingly, in Study 4 (Novel Data Predictor), neither model was successful at predicting new results absent in their training data, and neither appeared to leverage substantially new information when predicting more vs. less novel outcomes. Together, these results suggest that GPT is a flawed but rapidly improving librarian, a decent research ethicist already, capable of data generation in simple domains with known characteristics but poor at predicting novel patterns of empirical data to aid future experimentation.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.