Evidence map›Paper›PMID 30928941›Full record

SynthesisBMJ open2019

Comprehensive Researcher Achievement Model (CRAM): a framework for measuring researcher achievement, impact and influence derived from a systematic literature review of metrics and models.

Jeffrey Braithwaite, Jessica Herkes, Kate Churruca, Janet C Long, Chiara Pomare, Claire Boyling, Mia Bierbaum, Robyn Clay-Williams, Frances Rapport, Patti Shih and 8 more

Open access · goldAbstract readSystematic Review
PubMed Full text at PMC Publisher
In one paragraph

Synthesis in BMJ open, 2019. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 10 papers, 2 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
10citing papers in PubMed, 2 pooled it
4.2field-weighted citation impact, top 4% of its field
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

10 citing papers in PubMed, 2 syntheses or guidelines pooled it, 36 citations in OpenAlex.

  1. Pooled it
  2. Pooled it
  3. Maximizing the Impact of Your Academic Scholarship: Strategies for Early Career Professionals.Academic psychiatry : the journal of the American Association of Directors of Psychiatric Residency Training and the Association for Academic Psychiatry · 2026
    Review
  4. Article
  5. Article
  6. Article
  7. Article
  8. Review
  9. Article
  10. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

18 authors at 2 institutions in 1 country.

Jeffrey BraithwaiteAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.ORCID 0000-0003-0296-4957
Jessica HerkesAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Kate ChurrucaAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Janet C LongAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Chiara PomareAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.ORCID 0000-0002-9923-3116
Claire BoylingAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Mia BierbaumAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Robyn Clay-WilliamsAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.ORCID 0000-0002-6107-7445
Frances RapportAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Patti ShihAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.ORCID 0000-0002-9628-7987
Anne HogdenAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.ORCID 0000-0002-4317-7960
Louise A EllisAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.ORCID 0000-0002-2012-1980
Kristiana LudlowAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.ORCID 0000-0001-7284-5625
Elizabeth AustinAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Rebecca SeahAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Elise McPhersonAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Peter D HibbertAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Johanna WestbrookAustralian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia.
Macquarie University · AUUniversity of South Australia · AU

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

objectivesEffective researcher assessment is key to decisions about funding allocations, promotion and tenure. We aimed to identify what is known about methods for assessing researcher achievements, leading to a new composite assessment model.

designWe systematically reviewed the literature via the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols framework. DATA SOURCES: All Web of Science databases (including Core Collection, MEDLINE and BIOSIS Citation Index) to the end of 2017. ELIGIBILITY CRITERIA: (1) English language, (2) published in the last 10 years (2007-2017), (3) full text was available and (4) the article discussed an approach to the assessment of an individual researcher's achievements. DATA EXTRACTION AND SYNTHESIS: Articles were allocated among four pairs of reviewers for screening, with each pair randomly assigned 5% of their allocation to review concurrently against inclusion criteria. Inter-rater reliability was assessed using Cohen's Kappa (ĸ). The ĸ statistic showed agreement ranging from moderate to almost perfect (0.4848-0.9039). Following screening, selected articles underwent full-text review and bias was assessed.

resultsFour hundred and seventy-eight articles were included in the final review. Established approaches developed prior to our inclusion period (eg, citations and outputs, h-index and journal impact factor) remained dominant in the literature and in practice. New bibliometric methods and models emerged in the last 10 years including: measures based on PageRank algorithms or 'altmetric' data, methods to apply peer judgement and techniques to assign values to publication quantity and quality. Each assessment method tended to prioritise certain aspects of achievement over others.

conclusionsAll metrics and models focus on an element or elements at the expense of others. A new composite design, the Comprehensive Researcher Achievement Model (CRAM), is presented, which supersedes past anachronistic models. The CRAM is modifiable to a range of applications.

Indexed as

BibliometricsPeriodicals as TopicResearch PersonnelAlgorithmsHumansJournal Impact FactorModels, StatisticalProgram EvaluationcitationsComprehensive Researcher Achievement Model (CRAM)h-indexjournal impact factoroutputsresearcher assessmentresearch metrics

Identifiers

PMID30928941
PMCPMC6475357
OpenAlexW2933830696

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.