ArticleScientific data2022
A large-scale study on research code quality and execution.
Article in Scientific data, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 35 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
35 citing papers in PubMed.
- hpci improves reproducibility of computational biology research by integrating Continuous Integration with High Performance Computing clusters.Bioinformatics (Oxford, England) · 2026Article
- Article
- Do papers tell the whole story? A benchmark and framework for uncovering hidden implementation gaps in bioinformatics.Briefings in bioinformatics · 2026Article
- Coala: a standard-based framework for converting CWL-described command-line tools into agentic toolsets.Bioinformatics (Oxford, England) · 2026Article
- Article
- Unlocking the benefits of transparent and reusable science for climate risk management.Proceedings of the National Academy of Sciences of the United States of America · 2026Article
- Best Practices for Machine Learning-Assisted Protein Engineering.Journal of chemical information and modeling · 2025Review
- Article
- Building a continuous benchmarking ecosystem in bioinformatics.PLoS computational biology · 2025Article
- Improving reproducibility of data analysis and code in medical research: 5 recommendations to get started.BMJ open · 2025Review
- Agentic Superoptimization of Bioimaging Analysis Workflows.... Conference on Language Modeling · 2025Article
- Open science interventions to improve reproducibility and replicability of research: a scoping review.Royal Society open science · 2025Article
- The reliability of replications: a study in computational reproductions.Royal Society open science · 2025Article
- How will we prepare for an uncertain future? The value of open data and code for unborn generations facing climate change.Proceedings. Biological sciences · 2025Article
- Building Portable and Reproducible Cancer Informatics Workflows for Scalable Data Analysis: An RNA Sequencing Tutorial.Methods in molecular biology (Clifton, N.J.) · 2025Article
- Article
- Open Science at the generative AI turn: An exploratory analysis of challenges and opportunities.Quantitative science studies · 2025Article
- Exploratory Insights into Barriers and Open Practices for Computational Reproducibility in Scientific Research.F1000Research · 2025Article
- Assessing computational reproducibility in Behavior Research Methods.Behavior research methods · 2024Article
- Ten simple rules for training scientists to make better software.PLoS computational biology · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
Abstract
This article presents a study on the quality and execution of research code from publicly-available replication datasets at the Harvard Dataverse repository. Research code is typically created by a group of scientists and published together with academic papers to facilitate research transparency and reproducibility. For this study, we define ten questions to address aspects impacting research reproducibility and reuse. First, we retrieve and analyze more than 2000 replication datasets with over 9000 unique R files published from 2010 to 2020. Second, we execute the code in a clean runtime environment to assess its ease of reuse. Common coding errors were identified, and some of them were solved with automatic code cleaning to aid code execution. We find that 74% of R files failed to complete without error in the initial execution, while 56% failed when code cleaning was applied, showing that many errors can be prevented with good coding practices. We also analyze the replication datasets from journals' collections and discuss the impact of the journal policy strictness on the code re-execution rate. Finally, based on our results, we propose a set of recommendations for code dissemination aimed at researchers, journals, and repositories.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.