ArticleAddiction (Abingdon, England)2020
Mechanical Turk data collection in addiction research: utility, concerns and best practices.
Article in Addiction (Abingdon, England), 2020. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 73 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
73 citing papers in PubMed.
- Patient-Representing Population's Perceptions of GPT-Generated Versus Standard Emergency Department Discharge Instructions: Randomized Blind Survey Assessment.Journal of medical Internet research · 2024Trial
- Illegal product purchasing in the experimental tobacco marketplace: Effects of menthol cigarette and cigarette ventilation ban.Drug and alcohol dependence · 2023Trial
- Narrative theory v: Narratives about penalties resulting from illegal tobacco purchases increase psychological distress.Experimental and clinical psychopharmacology · 2023Trial
- Incorporating Social Networks and Event-Specific Information in a Personalized Feedback Intervention to Reduce Drinking Among Young Adults.Alcohol and alcoholism (Oxford, Oxfordshire) · 2022Trial
- Quality control questions on Amazon's Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7.Behavior research methods · 2022Trial
- Evaluating the Effectiveness of NoteAid in a Community Hospital Setting: Randomized Trial of Electronic Health Record Note Comprehension Interventions With Patients.Journal of medical Internet research · 2021Trial
- Perception of Resource Allocations to Address the Opioid Epidemic.Journal of addiction medicineTrial
- Applying the reinforcer pathology model to alcohol use and suicide risk: An initial test.Journal of the experimental analysis of behavior · 2026Article
- Perceived norms and e-cigarette use among young adults in US college: The mediation role of outcome expectancies.Journal of American college health : J of ACH · 2026Article
- Planning for New Threats to Online Research Data Validity: The Issue of Computer-Using Agents.Evaluation & the health professions · 2026Article
- Narrative simulation of financial scarcity: Effects on delay discounting, cigarette craving, and cigarette demand in adults who smoke cigarettes.Experimental and clinical psychopharmacology · 2025Article
- Opioid Use Among Individuals in Intimate Relationships.Couple & family psychology · 2025Article
- Relation of Cannabis Use Frequency and Gambling Behavior in Individuals Who Gamble Under the Influence of Cannabis.Journal of gambling studies · 2025Article
- Health Literacy-Based Heart Failure Self-care (H2Lit) Application: Development and Usability Testing.Computers, informatics, nursing : CIN · 2025Article
- Enhancing patient engagement and understanding: is providing direct access to laboratory results through patient portals adequate?JAMIA open · 2025Article
- High in the Cloud: Alcohol-, Cannabis-, and Co-Use Before and During Remote Research Participation.Substance use & misuse · 2025Article
- Understanding the Psychosis Spectrum Using a Hierarchical Model of Social Cognition.Schizophrenia bulletin · 2024Article
- The subjective value of social context in people who use cannabis.Experimental and clinical psychopharmacology · 2024Article
- Clustering of behavioral economic biases in decision-making and risk for cigarette smoking and other substance use in women and men.Preventive medicine · 2024Article
- The Cyborg Method: A Method to Identify Fraudulent Responses from Crowdsourced Data.Computers in human behavior · 2024Article
13 more citing papers are in PubMed but not listed here.
Corrections and comments
- Commented on by
- Commented on by
Authors and funding
2 authors.
Funding
Abstract
aimsAmazon Mechanical Turk (MTurk) provides a crowdsourcing platform for the engagement of potential research participants with data collection instruments. This review (1) provides an introduction to the mechanics and validity of MTurk research; (2) gives examples of MTurk research; and (3) discusses current limitations and best practices in MTurk research.
methodsWe review four use cases of MTurk for research relevant to addictions: (1) the development of novel measures, (2) testing interventions, (3) the collection of longitudinal use data to determine the feasibility of longer-term studies of substance use and (4) the completion of large batteries of assessments to characterize the relationships between measured constructs. We review concerns with the platform, ways of mitigating these and important information to include when presenting findings.
resultsMTurk has proved to be a useful source of data for behavioral science more broadly, with specific applications to addiction science. However, it is still not appropriate for all use cases, such as population-level inference. To live up to the potential of highly transparent, reproducible science from MTurk, researchers should clearly report inclusion/exclusion criteria, data quality checks and reasons for excluding collected data, how and when data were collected and both targeted and actual participant compensation.
conclusionsAlthough on-line survey research is not a substitute for random sampling or clinical recruitment, the Mechanical Turk community of both participants and researchers has developed multiple tools to promote data quality, fairness and rigor. Overall, Mechanical Turk has provided a useful source of convenience samples despite its limitations and has demonstrated utility in the engagement of relevant groups for addiction science.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.