Evidence map›Paper›PMID 39612724›Full record

Observational studyBehaviour research and therapy2025

A comparison of scalable routine clinical materials and observer ratings to assess CBT fidelity.

Amber Calloway, Torrey A Creed, Niki B Gumport, Cassidy Gutner, Luana Marques, Samantha Hernandez, Jiyoung Song, Clara Johnson, Soo Jeong Youn, Sohayla Elhusseini and 14 more

Registry-linked trialAbstract readComparative StudyObservational Study
PubMed Full text at PMC Publisher
In one paragraph

Observational study in Behaviour research and therapy, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It is linked to trial NCT03479398 (Leveraging Routine Clinical Materials and Mobile Technology to Assess CBT Quality), which is not on this map. Not yet cited in PubMed.

0numbers the graph read from it
0cells of the map it votes in
0citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

NCT03479398 unknown statusnot on this map

Leveraging Routine Clinical Materials and Mobile Technology to Assess CBT Quality

TypeobservationalSponsorPalo Alto Veterans Institute for ResearchRan2017 to 2023Enrolled439ConditionsMobile Application-Based CBT Worksheet Collection, Paper-Based Worksheet CollectionArmsCognitive Behavioral Therapy
3 · Its place in the literature

Who cites it

0 citing papers in PubMed.

No citing paper in PubMed yet.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

24 authors.

Amber CallowayPenn Collaborative for CBT and Implementation Science, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA. Electronic address: Amber.Calloway@pennmedicine.upenn.edu.
Torrey A CreedPenn Collaborative for CBT and Implementation Science, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
Niki B GumportDepartment of Psychiatry and Behavioral Sciences, Stanford University, 401 Quarry Road, Stanford, CA, 94305, USA.
Cassidy GutnerDepartment of Psychiatry, Boston University School of Medicine, Boston, MA, 02118, USA.
Luana MarquesDepartment of Psychiatry, Massachusetts General Hospital, One Bowdoin Square, Boston, MA, 02114, USA.
Samantha HernandezRutgers, The State University of New Jersey, 152 Frelinghuysen Road, Piscataway, NJ, 08854, USA.
Jiyoung SongDepartment of Psychology, University of California, Berkeley, 2121 Berkeley Way, Berkeley, CA, 94704, USA.
Clara JohnsonDepartment of Psychology, University of Washington, Guthrie Hall 119A, Seattle, WA, 98195, USA.
Soo Jeong YounReliant Medical Group, OptumCare, Harvard Medical School, 5 Neponset St, Worcester, MA, 01606, USA.
Sohayla ElhusseiniDepartment of Psychiatry and Behavioral Sciences, Stanford University, 401 Quarry Road, Stanford, CA, 94305, USA; National Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA.
Regine M Deguzman-LuceroNational Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA.
Taylor LaskotNational Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA.
Heidi La BashDepartment of Psychiatry and Behavioral Sciences, Stanford University, 401 Quarry Road, Stanford, CA, 94305, USA; National Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA.
Yesenia Aguilar SilvanDepartment of Psychology, University of California, Los Angeles (UCLA), 5505 Franz Hall, Los Angeles, CA, 90095, USA.
Caroline CassotteNational Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA; VA Boston Healthcare System, 150 S Huntington Ave, Boston, MA, 02130, USA.
Alayna L ParkUniversity of Oregon, 1227 University St, Eugene, OR, 97403, USA.
Kimberlye DeanDepartment of Psychiatry, Massachusetts General Hospital, One Bowdoin Square, Boston, MA, 02114, USA.
Anna D BartuskaUniversity of Oregon, 1227 University St, Eugene, OR, 97403, USA.
Booil JoDepartment of Psychiatry and Behavioral Sciences, Stanford University, 401 Quarry Road, Stanford, CA, 94305, USA.
Paul BarnettPalo Alto Veterans Institute for Research, 3801 Miranda Avenue Palo Alto, CA, 94304, USA.
Eric KuhnDepartment of Psychiatry and Behavioral Sciences, Stanford University, 401 Quarry Road, Stanford, CA, 94305, USA; National Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA.
Robert DeRubeisDepartment of Psychology, University of Pennsylvania, Philadelphia, PA, USA.
Dawne VogtDepartment of Psychiatry, Boston University School of Medicine, Boston, MA, 02118, USA; National Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA; Department of Psychology, University of California, Los Angeles (UCLA), 5505 Franz Hall, Los Angeles, CA, 90095, USA.
Shannon Wiltsey StirmanDepartment of Psychiatry and Behavioral Sciences, Stanford University, 401 Quarry Road, Stanford, CA, 94305, USA; National Center for PTSD, VA Palo Alto Healthcare System, 795 Willow Road (NC-PTSD), Menlo Park, CA, 94025, USA.

Funding

Leveraging Routine Clinical Materials and Mobile Technology to Assess CBT QualityR01MH112628 · NIMH · PALO ALTO VETERANS INSTIT FOR RESEARCH · PI WILTSEY STIRMAN, SHANNON · 2017 to 2020
$2.6M
NIMH NIH HHS R01 MH112628
6 · The paper itself

Abstract

Decades of research have demonstrated the efficacy of cognitive behavioral therapies (CBTs) for a wide variety of psychiatric diagnoses, resulting in the inclusion of CBT as a first-line evidence-based practice (EBP) in treatment guidelines for mood and anxiety disorders. However, some research suggests that many providers do not implement EBPs as intended. Ongoing quality monitoring is needed to support EBP implementation and sustainability, but "gold standard" fidelity monitoring (e.g. observer ratings) is time-consuming, requires extensive training, and may feel intrusive to providers and clients. In the current study, we aimed to develop a scalable method of assessing CBT fidelity that leverages information generated in routine clinical care (e.g. session worksheets and clinician checklists). Ratings of adherence based on worksheets were not correlated with ratings of adherence based on observer ratings. However, ratings of competence based on worksheets were significantly correlated with observer ratings of competence. Ratings of adherence based on clinician checklist ratings were also significantly correlated with observer-rated adherence. Results did not indicate a strong relationship between adherence or competence measured by worksheet ratings or observer and symptom change. However, adherence as measured by clinician checklists were associated with subsequent depression symptom change. Findings have a strong potential to impact fidelity monitoring strategies for a variety of CBTs. Given the limited resources and time to do full audio review in routine care settings, findings suggest that using routine materials generated in session to assess therapist competence may be a feasible alternative to the "gold standard" audio review. The trial is registered at ClinicalTrials.gov, number NCT03479398.

Indexed as

Cognitive Behavioral TherapyAdultAnxiety DisordersChecklistClinical CompetenceEvidence-Based PracticeFemaleGuideline AdherenceHumansMaleMiddle AgedAnxietyCognitive behavior therapyDepressionEvidence-basedFidelityImplementationPractices

Identifiers

PMID39612724
PMCPMC11717604

What OpenQuestion holds

Textmetadata
LicenceTDM
Read underepoch 390

Registered trials

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.