ArticleDigital discovery2023
Assessment of chemistry knowledge in large language models that generate code.
Article in Digital discovery, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 25 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
25 citing papers in PubMed.
- Transforming Molecular Synthesis With Large Language Models.Chemistry (Weinheim an der Bergstrasse, Germany) · 2026Review
- Censoring chemical data to mitigate dual use risk.Digital discovery · 2026Article
- Bayesian Optimization of Catalysis with In-Context Learning.ACS central science · 2026Article
- Looking back and to the future after four-plus years of language in chemistry.Digital discovery · 2026Review
- Synergizing Chemical and AI Communities for Advancing Laboratories of the Future.ACS central science · 2026Review
- Word embeddings as autonomous predictors in materials design-the effect of inherent variability on information transfer.Journal of cheminformatics · 2026Article
- Assessing the Chemical Intelligence of Large Language Models.Journal of chemical information and modeling · 2026Article
- A benchmark for assessing large language models on molecular-to-food and food-to-molecular prediction tasks.Frontiers in artificial intelligence · 2026Article
- Large language model guided automated reaction pathway exploration.Communications chemistry · 2025Article
- Artificial Intelligence Paradigms for Next-Generation Metal-Organic Framework Research.Journal of the American Chemical Society · 2025Review
- Machine Learning for Toxicity Prediction Using Chemical Structures: Pillars for Success in the Real World.Chemical research in toxicology · 2025Review
- Augmented and Programmatically Optimized LLM Prompts Reduce Chemical Hallucinations.Journal of chemical information and modeling · 2025Article
- A review of large language models and autonomous agents in chemistry.Chemical science · 2025Review
- Integrating Machine Learning and Large Language Models to Advance Exploration of Electrochemical Reactions.Angewandte Chemie (International ed. in English) · 2025Article
- Comparing SMILES and SELFIES tokenization for enhanced chemical language modeling.Scientific reports · 2024Article
- LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset.... Conference on Language Modeling · 2024Article
- Deductive machine learning models for product identification.Chemical science · 2024Article
- Can large language models predict antimicrobial peptide activity and toxicity?RSC medicinal chemistry · 2024Article
- Application of Transformers in Cheminformatics.Journal of chemical information and modeling · 2024Review
- ChatMOF: an artificial intelligence system for predicting and generating metal-organic frameworks using large language models.Nature communications · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
Abstract
In this work, we investigate the question: do code-generating large language models know chemistry? Our results indicate, mostly yes. To evaluate this, we introduce an expandable framework for evaluating chemistry knowledge in these models, through prompting models to solve chemistry problems posed as coding tasks. To do so, we produce a benchmark set of problems, and evaluate these models based on correctness of code by automated testing and evaluation by experts. We find that recent LLMs are able to write correct code across a variety of topics in chemistry and their accuracy can be increased by 30 percentage points
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.