ReviewThe International journal of eating disorders2022
Concerns and recommendations for using Amazon MTurk for eating disorder research.
Review in The International journal of eating disorders, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 44 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
44 citing papers in PubMed, 1 synthesis or guideline pooled it, 106 citations in OpenAlex.
- Psychological mediators of the associations between adverse and benevolent childhood experiences and posttraumatic stress disorder (PTSD) and complex PTSD: A systematic review and meta-analysis.Journal of traumatic stress · 2026Pooled it
- Associations between intuitive eating and health behaviors and physical health in U.S. adults.Journal of behavioral medicine · 2026Article
- Optimizing participation in digital health studies: understanding appointment attendance.Journal of the American Medical Informatics Association : JAMIA · 2026Article
- Safeguarding Online Research in Eating Disorders Against Fraud: Increasing Risks and Practical Recommendations.The International journal of eating disorders · 2026Article
- Intraminority Body Stigma and Body Dissatisfaction Clarify the Prospective Link Between LGBTQ+ Community Involvement and Thinness-Oriented Disordered Eating in Sexual Minoritized Men.The International journal of eating disorders · 2026Article
- Predicting Occupational Outcomes for Individuals with ADHD: The Role of Hyperactivity/Impulsivity and Executive Functioning.Journal of occupational rehabilitation · 2026Article
- Article
- Human approach-avoidance conflict behaviour relates to transdiagnostic psychiatric symptom dimensions.Translational psychiatry · 2026Article
- Imposters, Bots, and Other Threats to Data Integrity in Online Research: Scoping Review of the Literature and Recommendations for Best Practices.Online journal of public health informatics · 2025Review
- Improving Data Integrity in Samples Obtained From Web-Based Recruitment: Protocol for the Development of a Novel System for Assessing Participant Authenticity in a Remote Longitudinal Cohort Study of Polysubstance Use.JMIR research protocols · 2025Article
- Assessing the quality and reliability of the Amazon Mechanical Turk (MTurk) data in 2024.Royal Society open science · 2025Article
- Preventing fraudulent research participation: Methodological strategies and ethical impacts.Journal of genetic counseling · 2025Article
- Generative AI in healthcare: challenges to patient agency and ethical implications.Frontiers in digital health · 2025Article
- Pragmatic recruitment strategies for a text-based smoking cessation intervention for sexual and gender minority groups.Discover public health · 2025Article
- Addressing Fraudulent Responses in Quantitative and Qualitative Internet Research: Case Studies from Body Image and Appearance Research.Ethics & behavior · 2024Article
- A quasi-experimental study examining the efficacy of multimodal bot screening tools and recommendations to preserve data integrity in online psychological research.The American psychologist · 2024Article
- Weight stigma and binge eating related to poorer perceptions of healthcare provider interaction quality in a community-based sample.Journal of eating disorders · 2024Article
- Reliable, rapid, and remote measurement of metacognitive bias.Scientific reports · 2024Article
- Engagement in binge eating and fasting associated with poorer sleep quality in an online sample of adults.Journal of eating disorders · 2024Article
- Evaluating the Problem of Fraudulent Participants in Health Care Research: Multimethod Pilot Study.JMIR formative research · 2024Article
Corrections and comments
- Commented on by
- Commented on by
- Commented on by
- Commented on by
- Commented on by
- Commented on by
Authors and funding
8 authors at 3 institutions in 1 country.
Funding
Abstract
objectiveOur original aim was to validate and norm common eating disorder (ED) symptom measures in a large, representative community sample of transgender adults in the United States. We recruited via Amazon Mechanical Turk (MTurk), a popular online recruitment and data collection platform both within and outside of the ED field. We present an overview of our experience using MTurk.
methodRecruitment began in Spring 2020; our original target N was 2,250 transgender adults stratified evenly across the United States. Measures included a demographics questionnaire, the Eating Disorder Examination-Questionnaire, and the Eating Attitudes Test-26. Consistent with current literature recommendations, we implemented a comprehensive set of attention and validity measures to reduce and identify bot responding, data farming, and participant misrepresentation.
resultsRecommended validity and attention checks failed to identify the majority of likely invalid responses. Our collection of two similar ED measures, thorough weight history assessment, and gender identity experiences allowed us to examine response concordance and identify impossible and improbable responses, which revealed glaring discrepancies and invalid data. Furthermore, qualitative data (e.g., emails received from MTurk workers) raised concerns about economic conditions facing MTurk workers that could compel misrepresentation. DISCUSSION: Our results strongly suggest most of our data were invalid, and call into question results of recently published MTurk studies. We assert that caution and rigor must be applied when using MTurk as a recruitment tool for ED research, and offer several suggestions for ED researchers to mitigate and identify invalid data.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.