Analysing 3429 digital supervisory interactions between Community Health Workers in Uganda and Kenya: the development, testing and validation of an open access predictive machine learning web app

O’Donovan et al. Human Resources for Health https://doi.org/10.1186/s12960-021-00699-5 (2022) 20:6 Open Access RESEARCH Analysing 3429 digital supervisory interactions between Community Health Workers in Uganda and Kenya: the development, testing and validation of an open access predictive machine learning web app James O’Donovan1,2* , Ken Kahn2, MacKenzie MacRae1,3, Allan Saul Namanda1, Rebecca Hamala1, Ken Kabali1, Anne Geniets2, Alice Lakati4, Simon M. Mbae5 and Niall Winters2 Abstract Background: Despite the growth in mobile technologies (mHealth) to support Community Health Worker (CHW) supervision, the nature of mHealth-facilitated supervision remains underexplored. One strategy to support supervision at scale could be artificial intelligence (AI) modalities, including machine learning. We developed an open access, machine learning web application (CHWsupervisor) to predictively code instant messages exchanged between CHWs based on supervisory interaction codes. We document the development and validation of the web app and report its predictive accuracy. Methods: CHWsupervisor was developed using 2187 instant messages exchanged between CHWs and their supervisors in Uganda. The app was then validated on 1242 instant messages from a separate digital CHW supervisory network in Kenya. All messages from the training and validation data sets were manually coded by two independent human coders. The predictive performance of CHWsupervisor was determined by comparing the primary supervisory codes assigned by the web app, against those assigned by the human coders and calculating observed percentage agreement and Cohen’s kappa coefficients. Results: Human inter-coder reliability for the primary supervisory category of messages across the training and validation datasets was ‘substantial’ to ‘almost perfect’, as suggested by observed percentage agreements of 88–95% and Cohen’s kappa values of 0.7–0.91. In comparison to the human coders, the predictive accuracy of the CHWsupervisor web app was ‘moderate’, suggested by observed percentage agreements of 73–78% and Cohen’s kappa values of 0.51–0.56. Conclusions: Augmenting human coding is challenging because of the complexity of supervisory exchanges, which often require nuanced interpretation. A realistic understanding of the potential of machine learning approaches *Correspondence: 1 Division of Research and Health Equity, Omni Med Uganda, Mukono District, Makata, Uganda Full list of author information is available at the end of the article © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativeco mmons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. O’Donovan et al. Human Resources for Health (2022) 20:6 Page 2 of 8 should be kept in mind by practitioners, as although they hold promise, supportive supervision still requires a level of human expertise. Scaling-up digital CHW supervision may therefore prove challenging. Trial registration: This was not a clinical trial and was therefore not registered as such. Keywords: Machine learning, Artificial intelligence, Supervision, Community Health Worker, Digital Health, Training Background By 2030 The World Health Organization (WHO) estimates there will be a global shortage of 18 million health workers, which will be most pronounced in countries defined as low- or middle-income (LMIC) [1]. To address this gap in human resources for health, Community Health Workers (CHWs) have been trained to deliver primary healthcare services [2], especially in remote or rural communities. Although there is no fixed definition for a CHW, the term is generally used as an umbrella description for groups of “…paraprofessionals or lay individuals with an in-depth understanding of the community, culture and language, who have received standardised job-related training of a shorter duration than health professionals and whose primary goal is to provide culturally appropriate health services to the community” [3]. In 2010, a report from the WHO stated that supervision is one of the “weakest links in CHW program(me) s” [4], for reasons including a shortage of supervisors and poorly designed programmes. As a result, the use of mobile technologies (mHealth) has been proposed as one way to address these challenges [5, 6] and in 2018 a $100 million fund was announced at The World Economic Forum to support mHealth-facilitated training and supervision of 50,000 CHWs across sub-Saharan Africa [7]. Yet, from a pedagogical perspective, the evidence regarding the use of mHealth to support CHW supervision is weak. A systematic scoping review by Winters et al. found that of 24 studies which described the use of mHealth to support CHW training and learning, only four drew upon established theories of learning [8]. The authors of this study suggest that “mHealth suffers from a reductionist view of learning that underestimates the complexities of the relationship between pedagogy and technology” [8]. It is therefore vitally important that we understand the nature of mHealth-facilitated supervision occurring between CHWs in order to ensure it facilitates CHW learning and professional development in a rigorous manner. To try and capitalise on the promise of mHealth to support CHW supervision, interactive forms of learning— which are supported by the technological capabilities of mobile technologies—are beginning to be explored [9]. Examples include the use of instant messaging applications (apps) to encourage interactive and peer-to-peer forms of learning [10]. Such approaches could help to facilitate a move away from less pedagogically sophisticated means of supervision, which have traditionally focussed on simplistic, information dissemination style interventions (e.g. one way messaging) [8, 11, 12]. These have been critiqued in the wider literature for simplified approaches to supervision which fail to promote CHW collaboration, agency and professional growth [8, 12]. Yet, despite emerging attempts to understand how the use of instant messaging apps (...truncated)