Evidence map›Paper›PMID 35292073›Full record

ArticleHuman resources for health2022

Analysing 3429 digital supervisory interactions between Community Health Workers in Uganda and Kenya: the development, testing and validation of an open access predictive machine learning web app.

James O'Donovan, Ken Kahn, MacKenzie MacRae, Allan Saul Namanda, Rebecca Hamala, Ken Kabali, Anne Geniets, Alice Lakati, Simon M Mbae, Niall Winters

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Human resources for health, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 3 papers.

0numbers the graph read from it
0cells of the map it votes in
3citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

3 citing papers in PubMed.

  1. Trial
  2. Article
  3. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

10 authors.

James O'DonovanDivision of Research and Health Equity, Omni Med Uganda, Mukono District, Makata, Uganda. jamesodonovan@post.harvard.edu.ORCID 0000-0002-7248-5436
Ken KahnLearning and New Technologies Research Group, Department of Education, University of Oxford, Oxford, United Kingdom.
MacKenzie MacRaeDivision of Research and Health Equity, Omni Med Uganda, Mukono District, Makata, Uganda.
Allan Saul NamandaDivision of Research and Health Equity, Omni Med Uganda, Mukono District, Makata, Uganda.
Rebecca HamalaDivision of Research and Health Equity, Omni Med Uganda, Mukono District, Makata, Uganda.
Ken KabaliDivision of Research and Health Equity, Omni Med Uganda, Mukono District, Makata, Uganda.
Anne GenietsLearning and New Technologies Research Group, Department of Education, University of Oxford, Oxford, United Kingdom.
Alice LakatiDivision of Health and Social Sciences, Amref International University, Nairobi, Kenya.
Simon M MbaeDivision of Research, Medic Mobile, Nairobi, Kenya.
Niall WintersLearning and New Technologies Research Group, Department of Education, University of Oxford, Oxford, United Kingdom.

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

backgroundDespite the growth in mobile technologies (mHealth) to support Community Health Worker (CHW) supervision, the nature of mHealth-facilitated supervision remains underexplored. One strategy to support supervision at scale could be artificial intelligence (AI) modalities, including machine learning. We developed an open access, machine learning web application (CHWsupervisor) to predictively code instant messages exchanged between CHWs based on supervisory interaction codes. We document the development and validation of the web app and report its predictive accuracy.

methodsCHWsupervisor was developed using 2187 instant messages exchanged between CHWs and their supervisors in Uganda. The app was then validated on 1242 instant messages from a separate digital CHW supervisory network in Kenya. All messages from the training and validation data sets were manually coded by two independent human coders. The predictive performance of CHWsupervisor was determined by comparing the primary supervisory codes assigned by the web app, against those assigned by the human coders and calculating observed percentage agreement and Cohen's kappa coefficients.

resultsHuman inter-coder reliability for the primary supervisory category of messages across the training and validation datasets was 'substantial' to 'almost perfect', as suggested by observed percentage agreements of 88-95% and Cohen's kappa values of 0.7-0.91. In comparison to the human coders, the predictive accuracy of the CHWsupervisor web app was 'moderate', suggested by observed percentage agreements of 73-78% and Cohen's kappa values of 0.51-0.56.

conclusionsAugmenting human coding is challenging because of the complexity of supervisory exchanges, which often require nuanced interpretation. A realistic understanding of the potential of machine learning approaches should be kept in mind by practitioners, as although they hold promise, supportive supervision still requires a level of human expertise. Scaling-up digital CHW supervision may therefore prove challenging.

trial registrationThis was not a clinical trial and was therefore not registered as such.

Indexed as

Community Health WorkersMobile ApplicationsAccess to InformationArtificial IntelligenceHumansKenyaMachine LearningReproducibility of ResultsUgandaArtificial intelligenceCommunity Health WorkerDigital HealthMachine learningSupervisionTraining

Identifiers

PMID35292073
PMCPMC8925202

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.