ArticleDatabase : the journal of biological databases and curation2015
Scaling drug indication curation through crowdsourcing.
Article in Database : the journal of biological databases and curation, 2015. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 23 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
23 citing papers in PubMed, 1 synthesis or guideline pooled it, 42 citations in OpenAlex.
- Mapping of Crowdsourcing in Health: Systematic Review.Journal of medical Internet research · 2018Pooled it
- Evaluating NLP Approaches to Extract Drug Indications.AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science · 2026Article
- Facial Anthropomorphic Trustworthiness Scale for Social Robots: A Hybrid Approach.Biomimetics (Basel, Switzerland) · 2023Article
- Developing Crowdsourced Training Data Sets for Pharmacovigilance Intelligent Automation.Drug safety · 2021Article
- InContext: curation of medical context for drug indications.Journal of biomedical semantics · 2021Article
- DICE: A Drug Indication Classification and Encyclopedia for AI-Based Indication Extraction.Frontiers in artificial intelligence · 2021Article
- A hybrid approach toward biomedical relation extraction training corpora: combining distant supervision with crowdsourcing.Database : the journal of biological databases and curation · 2020Article
- Applying citizen science to gene, drug and disease relationship extraction from biomedical abstracts.Bioinformatics (Oxford, England) · 2020Article
- Applications of crowdsourcing in health: an overview.Journal of global health · 2018Review
- Comparing Amazon's Mechanical Turk Platform to Conventional Data Collection Methods in the Health and Medical Research Literature.Journal of general internal medicine · 2018Review
- A dataset of 200 structured product labels annotated for adverse drug reactions.Scientific data · 2018Article
- Article
- Developing a framework for digital objects in the Big Data to Knowledge (BD2K) commons: Report from the Commons Framework Pilots workshop.Journal of biomedical informatics · 2017Article
- Extraction and analysis of signatures from the Gene Expression Omnibus by the crowd.Nature communications · 2016Article
- Is the Juice Worth the Squeeze? Costs and Benefits of Multiple Human Annotators for Clinical Text De-identification.Methods of information in medicine · 2016Article
- Using Nonexperts for Annotating Pharmacokinetic Drug-Drug Interaction Mentions in Product Labeling: A Feasibility Study.JMIR research protocols · 2016Article
- A survey of current trends in computational drug repositioning.Briefings in bioinformatics · 2016Review
- A crowdsourcing workflow for extracting chemical-induced disease relations from free text.Database : the journal of biological databases and curation · 2016Article
- Overview of the interactive task in BioCreative V.Database : the journal of biological databases and curation · 2016Article
- Crowdsourcing and curation: perspectives from biology and natural language processing.Database : the journal of biological databases and curation · 2016Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors at 2 institutions in 1 country.
Funding
Abstract
Motivated by the high cost of human curation of biological databases, there is an increasing interest in using computational approaches to assist human curators and accelerate the manual curation process. Towards the goal of cataloging drug indications from FDA drug labels, we recently developed LabeledIn, a human-curated drug indication resource for 250 clinical drugs. Its development required over 40 h of human effort across 20 weeks, despite using well-defined annotation guidelines. In this study, we aim to investigate the feasibility of scaling drug indication annotation through a crowdsourcing technique where an unknown network of workers can be recruited through the technical environment of Amazon Mechanical Turk (MTurk). To translate the expert-curation task of cataloging indications into human intelligence tasks (HITs) suitable for the average workers on MTurk, we first simplify the complex task such that each HIT only involves a worker making a binary judgment of whether a highlighted disease, in context of a given drug label, is an indication. In addition, this study is novel in the crowdsourcing interface design where the annotation guidelines are encoded into user options. For evaluation, we assess the ability of our proposed method to achieve high-quality annotations in a time-efficient and cost-effective manner. We posted over 3000 HITs drawn from 706 drug labels on MTurk. Within 8 h of posting, we collected 18 775 judgments from 74 workers, and achieved an aggregated accuracy of 96% on 450 control HITs (where gold-standard answers are known), at a cost of $1.75 per drug label. On the basis of these results, we conclude that our crowdsourcing approach not only results in significant cost and time saving, but also leads to accuracy comparable to that of domain experts.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.