Turing Jest: Distributional Semantics and One-Line Jokes.

Humor is an essential aspect of human experience, yet surprisingly, little is known about how we recognize and understand humorous utterances. Most theories of humor emphasize the role of incongruity detection and resolution (e.g., frame-shifting), as well as cognitive capacities like Theory of Mind and pragmatic reasoning. In multiple preregistered experiments, we ask whether and to what extent exposure to purely linguistic input can account for the human ability to recognize one-line jokes and identify their entailments. We find that GPT-3, a large language model (LLM) trained on only language data, exhibits above-chance performance in tasks designed to test its ability to detect, appreciate, and comprehend jokes. In exploratory work, we also find above-chance performance in humor detection and comprehension in several open-source LLMs, such as Llama-3 and Mixtral. Although all LLMs tested fall short of human performance, both humans and LLMs show a tendency to misclassify nonjokes with surprising endings as jokes. Results suggest that LLMs are remarkably adept at some tasks involving one-line jokes, but reveal key limitations of distributional approaches to meaning.