ArticlePloS one2023
Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA.
Article in PloS one, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 283 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
283 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Parenthood and care in academia: Publication counts, self-assessed productivity, and gendered differences.PloS one · 2026Pooled it
- A crowdsourced megastudy of 12 digital single-session interventions for depression in US adults.Nature human behaviour · 2026Trial
- Trial
- Measuring the effect of short, animated storytelling videos to boost psychological capital in US adults: a randomized controlled trial.Scientific reports · 2025Trial
- Warning label messages about the cancer risk associated with alcohol: Effects of causal language.Addictive behaviors · 2025Trial
- On the conversational persuasiveness of GPT-4.Nature human behaviour · 2025Trial
- Impact of AI-Assisted Diagnosis on American Patients' Trust in and Intention to Seek Help From Health Care Professionals: Randomized, Web-Based Survey Experiment.Journal of medical Internet research · 2025Trial
- Alternative Presentations of Overall and Statistical Uncertainty for Adults' Understanding of the Results of a Randomized Trial of a Public Health Intervention: Parallel Web-Based Randomized Trials.JMIR public health and surveillance · 2025Trial
- Comparing the effectiveness of animated videos and talking-head videos in science communication.British journal of health psychology · 2025Trial
- Effect of a Short, Animated Storytelling Video on Transphobia Among US Parents: Randomized Controlled Trial.JMIR public health and surveillance · 2025Trial
- Well-being comparisons and mental health outcomes following loss: a two-wave longitudinal study.European journal of psychotraumatology · 2026Article
- Factors associated with patient preferences for statin therapy in primary prevention.American journal of preventive cardiology · 2026Article
- Task-specific associations between holistic processing and individual differences in face memory.British journal of psychology (London, England : 1953) · 2026Article
- Understanding when mixture models effectively identify careless responding: Simulation and validity evidence.Behavior research methods · 2026Article
- Article
- Further adaptation of cross-commodity purchase task methodology: The impact of preference rank on demand.Experimental and clinical psychopharmacology · 2026Article
- Improving Medication Classification in GLP-1RA Research: Feasibility of a Photo-Review Protocol.Obesity science & practice · 2026Article
- Comparing current veterans affairs contingency management incentive structures against common alternatives on intentions to abstain from alcohol among veterans.Experimental and clinical psychopharmacology · 2026Article
- Digital Technologies for Community Consultation and Public Disclosure in Exception From Informed Consent Trials.Journal of the American College of Emergency Physicians open · 2026Article
- Differentiating e-cigarette use and dual use: the role of impulsivity.Psychopharmacology · 2026Article
223 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
With the proliferation of online data collection in human-subjects research, concerns have been raised over the presence of inattentive survey participants and non-human respondents (bots). We compared the quality of the data collected through five commonly used platforms. Data quality was indicated by the percentage of participants who meaningfully respond to the researcher's question (high quality) versus those who only contribute noise (low quality). We found that compared to MTurk, Qualtrics, or an undergraduate student sample (i.e., SONA), participants on Prolific and CloudResearch were more likely to pass various attention checks, provide meaningful answers, follow instructions, remember previously presented information, have a unique IP address and geolocation, and work slowly enough to be able to read all the items. We divided the samples into high- and low-quality respondents and computed the cost we paid per high-quality respondent. Prolific ($1.90) and CloudResearch ($2.00) were cheaper than MTurk ($4.36) and Qualtrics ($8.17). SONA cost $0.00, yet took the longest to collect the data.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.