ArticleProceedings of the National Academy of Sciences of the United States of America2024
Embers of autoregression show how large language models are shaped by the problem they are trained to solve.
Article in Proceedings of the National Academy of Sciences of the United States of America, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 29 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
29 citing papers in PubMed.
- Virtual Reality Therapy Assistants for Adjunctive Cue Exposure Training in the Treatment of Alcohol Use Disorder (VITAACue-AUD): Protocol for a Technical Feasibility Study.JMIR research protocols · 2026Article
- A medically grounded LLM agent-based tool to detect patient safety events in medical records.PLOS digital health · 2026Article
- A Generative Neuro-Symbolic AI for Protein Sequence Design.Advanced science (Weinheim, Baden-Wurttemberg, Germany) · 2026Article
- Predicting the evolutionary and functional landscapes of viruses with a unified nucleotide-protein language model: LucaVirus.National science review · 2026Article
- TechnologicalNature. Mental health · 2026Article
- Review
- Artificial Authority: The Promise and Perils of LLM Judges in Healthcare.Bioengineering (Basel, Switzerland) · 2026Review
- Article
- Vector symbolic algebras for the Abstraction and Reasoning Corpus.Npj unconventional computing · 2026Article
- Advances in LLM Reasoning Enable Flexibility in Clinical Problem-Solving.AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science · 2026Article
- Assessing the Relational Abilities of Large Language Models and Large Reasoning Models.Behavioral sciences (Basel, Switzerland) · 2025Article
- An External Validation Study on Two Pre-Trained Large Language Models for Multimodal Prognostication in Laryngeal and Hypopharyngeal Cancer: Integrating Clinical, Treatment, and Radiomic Data to Predict Survival Outcomes with Interpretable Reasoning.Bioengineering (Basel, Switzerland) · 2025Article
- Mining the gaps: Deciphering Alzheimer's biology through AI-driven reconciliation.The journal of prevention of Alzheimer's disease · 2025Article
- Limitations of large language models in clinical problem-solving arising from inflexible reasoning.Scientific reports · 2025Article
- Revealing emergent human-like conceptual representations from language prediction.Proceedings of the National Academy of Sciences of the United States of America · 2025Article
- Performance of the Large Language Models on the Chinese National Nurse Licensure Examination: Cross-Sectional Evaluation Study.JMIR medical informatics · 2025Article
- Toward large reasoning models: A survey of reinforced reasoning with large language models.Patterns (New York, N.Y.) · 2025Review
- Neural Generative Models and the Parallel Architecture of Language: A Critical Review and Outlook.Topics in cognitive science · 2025Review
- Probing the limitations of multimodal language models for chemistry and materials research.Nature computational science · 2025Article
- Re-evaluating Theory of Mind evaluation in large language models.Philosophical transactions of the Royal Society of London. Series B, Biological sciences · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors.
Funding
Abstract
The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that to develop a holistic understanding of these systems, we must consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts, we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. Using this approach-which we call the teleological approach-we identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. To test our predictions, we evaluate five LLMs (GPT-3.5, GPT-4, Claude 3, Llama 3, and Gemini 1.0) on 11 tasks, and we find robust evidence that LLMs are influenced by probability in the hypothesized ways. Many of the experiments reveal surprising failure modes. For instance, GPT-4's accuracy at decoding a simple cipher is 51% when the output is a high-probability sentence but only 13% when it is low-probability, even though this task is a deterministic one for which probability should not matter. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system-one that has been shaped by its own particular set of pressures.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.