ArticleScientific reports2024
Empirical assessment of ChatGPT's answering capabilities in natural science and engineering.
Article in Scientific reports, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 8 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
8 citing papers in PubMed.
- Comment to "Artificial Intelligence (AI)-Assisted Patient Education and Concerns Following Facelift Surgery: A Study on ChatGPT-4 and Gemini".Aesthetic plastic surgery · 2026Article
- ChatGPT and deepseek in head and neck cancers: objective scales could strengthen AI research in medicine.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- CarD-T: an automated pipeline for the nomination and analysis of potential human carcinogens.Carcinogenesis · 2025Article
- Automatic- and Transformer-Based Automatic Item Generation: A Critical Review.Journal of Intelligence · 2025Review
- Open- and closed-source LLMs in medical and engineering education.Frontiers in medicine · 2025Article
- Leveraging enhanced egret swarm optimization algorithm and artificial intelligence-driven prompt strategies for portfolio selection.Scientific reports · 2024Article
- CarD-T: Interpreting Carcinomic Lexicon via Transformers.medRxiv : the preprint server for health sciences · 2024Article
- Performance of chatbots in queries concerning fundamental concepts in photochemistry.Photochemistry and photobiologyArticle
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
ChatGPT is a powerful language model from OpenAI that is arguably able to comprehend and generate text. ChatGPT is expected to greatly impact society, research, and education. An essential step to understand ChatGPT's expected impact is to study its domain-specific answering capabilities. Here, we perform a systematic empirical assessment of its abilities to answer questions across the natural science and engineering domains. We collected 594 questions on natural science and engineering topics from 198 faculty members across five faculties at Delft University of Technology. After collecting the answers from ChatGPT, the participants assessed the quality of the answers using a systematic scheme. Our results show that the answers from ChatGPT are, on average, perceived as "mostly correct". Two major trends are that the rating of the ChatGPT answers significantly decreases (i) as the educational level of the question increases and (ii) as we evaluate skills beyond scientific knowledge, e.g., critical attitude.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.