ArticleJournal of general internal medicine1993
How reliable is peer review of scientific abstracts? Looking back at the 1991 Annual Meeting of the Society of General Internal Medicine.
Article in Journal of general internal medicine, 1993. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 14 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
14 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants.PloS one · 2010Pooled it
- More insight into the fate of biomedical meeting abstracts: a systematic review.BMC medical research methodology · 2003Pooled it
- Fostering the Next Generation of Researchers: a Sustainable Mentoring Program for Early Career Toxicologists in Scientific Abstract Review.Journal of medical toxicology : official journal of the American College of Medical Toxicology · 2023Article
- Peer Review of Abstracts Submitted to An Internal Medicine National Meeting: Is It a Predictor of Future Publication?Journal of general internal medicine · 2018Article
- How do Medical Societies Select Science for Conference Presentation? How Should They?The western journal of emergency medicine · 2015Article
- Improving the quality of abstract reporting for economic analyses in oncology.Current oncology (Toronto, Ont.) · 2012Article
- Selecting the best clinical vignettes for academic meetings: should the scoring tool criteria be modified?Journal of general internal medicine · 2012Article
- Editorial peer reviewers' recommendations at a general medical journal: are they reliable and do editors care?PloS one · 2010Article
- Poster exhibitions at national conferences: education or farce?Deutsches Arzteblatt international · 2008Article
- Reviewer agreement trends from four years of electronic submissions of conference abstract.BMC medical research methodology · 2006Article
- Development and evaluation of a quality score for abstracts.BMC medical research methodology · 2003Article
- Scientific meeting abstracts: significance, access, and trends.Bulletin of the Medical Library Association · 1998Article
- An observational study of the proceedings of the All India Ophthalmological Conference, 2000 and subsequent publication in indexed journals.Indian journal of ophthalmologyArticle
- The fate of neuroradiologic abstracts presented at national meetings in 1993: rate of subsequent publication in peer-reviewed, indexed journals.AJNR. American journal of neuroradiologyArticle
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectiveTo evaluate the interrater reproducibility of scientific abstract review.
designRetrospective analysis.
settingReview for the 1991 Society of General Internal Medicine (SGIM) annual meeting. SUBJECTS: 426 abstracts in seven topic categories evaluated by 55 reviewers. MEASUREMENTS: Reviewers rated abstracts from 1 (poor) to 5 (excellent), globally and on three specific dimensions: interest to the SGIM audience, quality of methods, and quality of presentation. Each abstract was reviewed by five to seven reviewers. Each reviewer's ratings of the three dimensions were added to compute that reviewer's summary score for a given abstract. The mean of all reviewers' summary scores for an abstract, the final score, was used by SGIM to select abstracts for the meeting.
resultsFinal scores ranged from 4.6 to 13.6 (mean = 9.9). Although 222 abstracts (52%) were accepted for publication, the 95% confidence interval around the final score of 300 (70.4%) of the 426 abstracts overlapped with the threshold for acceptance of an abstract. Thus, these abstracts were potentially misclassified. Only 36% of the variance in summary scores was associated with an abstract's identity, 12% with the reviewer's identity, and the remainder with idiosyncratic reviews of abstracts. Global ratings were more reproducible than summary scores.
conclusionReviewers disagreed substantially when evaluating the same abstracts. Future meeting organizers may wish to rank abstracts using global ratings, and to experiment with structured review criteria and other ways to improve raters' agreement.
Indexed as
Identifiers
8505684What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.