Evaluating Crowdsourced Data Collection for Carceral Death Surveillance: Pilot Study Using Amazon Mechanical Turk.

backgroundPeople who are incarcerated face significantly higher health risks than the general population, yet deaths in custody remain underreported and poorly monitored by public health systems. Although the federal Death in Custody Reporting Act requires reporting of all deaths in correctional facilities to the US Department of Justice, reporting has been inconsistent, delayed, and often publicly inaccessible. Consequently, researchers have turned to press releases issued by correctional agencies as one of the few timely sources of information on deaths in custody. However, these press releases vary widely in content and structure, making standardized data extraction difficult. Crowdsourcing platforms such as Amazon Mechanical Turk (MTurk) may offer a faster, low-cost method for gathering data, but their utility in this setting remains untested.

objectiveThis pilot study evaluated whether MTurk could be used to extract structured information from press releases about deaths in custody.

methodsWe selected 144 press releases describing deaths between 2000 and 2023 from state prison systems and Immigration and Customs Enforcement. Each press release was assigned to 3 MTurk crowd workers (who were required to be English speaking and located in the United States), resulting in 432 individual responses. Workers were informed in advance that the task involved reviewing sensitive content related to deaths in custody. Crowd workers completed a 16-question form aligned with Death in Custody Reporting Act variables, including age, race and ethnicity, date of death, and facility location. Data quality was assessed using strict concordance (all 3 responses matched), 2-way concordance (2 of 3 responses matched), and qualitative review of common errors. Task completion time was also recorded. Sampling included complete subsets of selected press releases and a stratified subset from systems with more complex reporting formats.

resultsAll 144 entries were completed within 48 hours. However, agreement across crowd workers was low: strict concordance was 14.2% (20/144) for age, 12.3% (18/144) for race or ethnicity, and 11.4% (16/144) for date of birth. Qualitative review identified frequent errors, missing data, and inattentive or automated responses. Crowd workers often misinterpreted system-specific terminology or copied placeholders instead of extracting information from the source. The low agreement indicated that this baseline MTurk configuration produced insufficient data quality for more resource-intensive use.

conclusionsMTurk enabled rapid task completion but produced low-quality results when applied to extracting structured data from carceral press releases. These findings suggest that general crowdsourcing platforms are poorly suited to complex data abstraction tasks without additional training or oversight. With improved task design or support from artificial intelligence tools, crowdsourcing may help address gaps in the surveillance of deaths in custody. Long-term improvements will require consistent, transparent, and standardized reporting practices across correctional institutions.