ArticleBehavior research methods2024
Can large language models help augment English psycholinguistic datasets?
Article in Behavior research methods, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 22 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
22 citing papers in PubMed.
- Contextualized sensorimotor norms: Multi-dimensional measures of sensorimotor strength for ambiguous English words, in context.Behavior research methods · 2026Article
- Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings.Behavior research methods · 2026Article
- Relationships Between Subjective Iconicity Ratings and Phonological Variables in English and Spanish.Quarterly journal of experimental psychology (2006) · 2026Article
- Automatic quantification of lexical ambiguity using large-scale word association data.Behavior research methods · 2026Article
- Extrapolated Persian Lexical Affect Norms (E-PLAN) from best-worst judgments of valence, arousal, dominance, and concreteness.Behavior research methods · 2026Article
- AI-Enhanced Semantic Feature Norms for 786 Concepts.Topics in cognitive science · 2026Article
- The Multidimensional Nature of Semantic Transparency in a Cross-Linguistic Perspective: Evidence From Human Intuitions, Computational Estimates, and Processing Data for Chinese Compounds.Cognitive science · 2026Article
- How well do large language models mirror human cognition of word concepts?: A comparison of psychological ratings for early-acquired English words.Behavior research methods · 2026Article
- Incipient Constituents: Phonesthemes Facilitate Word Processing in English.Open mind : discoveries in cognitive science · 2026Article
- Updating the German Psycholinguistic Word Toolbox with AI-Generated Estimates of Concreteness, Valence, Arousal, Age of Acquisition, and Familiarity.Journal of cognition · 2026Article
- Large Language Models as Distributional Baselines for Language Tasks.Open mind : discoveries in cognitive science · 2026Article
- Contextual assembly of lexical functions in large language models.Behavior research methods · 2025Article
- Crowdsourced and AI-generated age-of-acquisition (AoA) norms for vocabulary in print: Extending the Kuperman et al. (2012) norms.Behavior research methods · 2025Article
- Simulating lexical decision times with large language models to supplement megastudies and crowdsourcing.Behavior research methods · 2025Article
- Turing Jest: Distributional Semantics and One-Line Jokes.Cognitive science · 2025Article
- Article
- Can Large Language Models Generate Useful Linguistic Corpora?: A Case Study of the Word Frequency Effect in Young German Readers.Open mind : discoveries in cognitive science · 2025Article
- Article
- Investigating the Effects of Valence, Arousal, Concreteness, and Humor on Words Unique to Singapore English.Journal of cognition · 2025Article
- Moving beyond word frequency based on tally counting: AI-generated familiarity estimates of words and phrases are an interesting additional index of language knowledge.Behavior research methods · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
1 author.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Research on language and cognition relies extensively on psycholinguistic datasets or "norms". These datasets contain judgments of lexical properties like concreteness and age of acquisition, and can be used to norm experimental stimuli, discover empirical relationships in the lexicon, and stress-test computational models. However, collecting human judgments at scale is both time-consuming and expensive. This issue of scale is compounded for multi-dimensional norms and those incorporating context. The current work asks whether large language models (LLMs) can be leveraged to augment the creation of large, psycholinguistic datasets in English. I use GPT-4 to collect multiple kinds of semantic judgments (e.g., word similarity, contextualized sensorimotor associations, iconicity) for English words and compare these judgments against the human "gold standard". For each dataset, I find that GPT-4's judgments are positively correlated with human judgments, in some cases rivaling or even exceeding the average inter-annotator agreement displayed by humans. I then identify several ways in which LLM-generated norms differ from human-generated norms systematically. I also perform several "substitution analyses", which demonstrate that replacing human-generated norms with LLM-generated norms in a statistical model does not change the sign of parameter estimates (though in select cases, there are significant changes to their magnitude). I conclude by discussing the considerations and limitations associated with LLM-generated norms in general, including concerns of data contamination, the choice of LLM, external validity, construct validity, and data quality. Additionally, all of GPT-4's judgments (over 30,000 in total) are made available online for further analysis.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.