An Assessment of Inter-Observer Agreement in Water Source Classification and Sanitary Risk Observations

Exposure and Health https://doi.org/10.1007/s12403-019-00339-3 ORIGINAL PAPER An Assessment of Inter‑Observer Agreement in Water Source Classification and Sanitary Risk Observations Joseph Okotto‑Okotto1 · Peggy Wanza2 · Emmah Kwoba2 · Weiyu Yu3 · Mawuli Dzodzomenyo4 · S. M. Thumbi2,5 · Diogo Gomes da Silva6 · Jim A. Wright3 Received: 13 September 2019 / Revised: 22 November 2019 / Accepted: 10 December 2019 © The Author(s) 2019 Abstract Sanitary risk inspection, an observation protocol for identifying contamination hazards around water sources, is promoted for managing rural water supply safety. However, it is unclear how far different observers consistently identify contamination hazards and consistently classify water source types using standard typologies. This study aimed to quantify inter-observer agreement in hazard identification and classification of rural water sources. Six observers separately visited 146 domestic water sources in Siaya County, Kenya, in wet and dry seasons. Each observer independently classified the source type and conducted a sanitary risk inspection using a standard protocol. Water source types assigned by an experienced observer were cross-tabulated against those of his colleagues, as were contamination hazards identified, and inter-observer agreement measures calculated. Agreement between hazards observed by the most experienced observer versus his colleagues was significant but low (intra-class correlation = 0.49), with inexperienced observers detecting fewer hazards. Inter-observer agreement in classifying water sources was strong (Cohen’s kappa = 0.84). However, some source types were frequently misclassified, such as sources adapted to cope with water insecurity (e.g. tanks drawing on both piped and rainwater). Observers with limited training and experience thus struggle to consistently identify hazards using existing protocols, suggesting observation protocols require revision and their implementation should be supported by comprehensive training. Findings also indicate that field survey teams struggle to differentiate some water source types based on a standard water source classification, particularly sources adapted to cope with water insecurity. These findings demonstrate uncertainties underpinning international monitoring and analyses of safe water access via household surveys. Keywords Water safety · Rainwater · Inter-observer agreement · Sanitary inspection · Sustainable development goals Electronic supplementary material The online version of this article (https://doi.org/10.1007/s12403-019-00339-3) contains supplementary material, which is available to authorized users. Abbreviations CHIRPS Climate hazards group infra-red precipitation with station data CI Confidence interval DHS Demographic and health survey * Joseph Okotto‑Okotto 4 Ghana School of Public Health, University of Ghana, P.O. Box LG 13, Legon, Accra, Ghana * Jim A. Wright 5 Paul G Allen School for Global Animal Health, Washington State University, Pullman, WA 99164‑ 7090, USA 6 School of Environment and Technology, University of Brighton, Cockcroft Building, Lewes Road, Brighton BN2 4GJ, UK 1 Victoria Institute for Research on Environment and Development (VIRED) International, off Nairobi Road, Rabour, P.O. Box 6423‑40103, Kisumu, Kenya 2 Centre for Global Health Research, Kenya Medical Research Institute, P.O. Box 1578‑40100, Kisumu, Kenya 3 School of Geography and Environmental Science, University of Southampton, Building 44, Highfield, Southampton SO17 1BJ, UK 13 Vol.:(0123456789) J. Okotto‑Okotto et al. JMP Joint monitoring programme RADWQ Rapid assessment of drinking-water quality SDG Sustainable development goal WHO World Health Organization Background Target 6.1 of the Sustainable Development Goal (SDG) aims to ‘By 2030, achieve universal and equitable access to safe and affordable drinking water for all’ (United Nations 2019). To deliver this target, the World Health Organization has promoted water safety plans as a tool for rural water supply managers to ensure the safety of such supplies (Rickert et al. 2014). In remote and resource-poor settings, however, microbiological testing is often unavailable given its cost, lack of consumables or distance to laboratory infra-structure and skilled staff (Wright et al. 2014), with far less testing being completed on non-piped than piped supplies in subSaharan Africa (Kumpel et al. 2016). Where microbiological testing does take place, supply managers require methods for identifying the hazards responsible for the microbiological contamination identified through water testing so that these can be remediated. For this reason, the World Health Organization (WHO) has promoted the use of structured observation protocols for identifying faecal contamination hazards at and surrounding rural water sources (World Health Organization 1997). These protocols, often referred to as sanitary risk inspections, identify hazards such as problems with the structural integrity of source protection measures (e.g. blocked drainage channels or broken fencing around protected wells) and contamination sources in the surrounding environment (e.g. pit latrines or livestock immediately upstream of a spring). As well as being promoted as a tool for water supply managers, sanitary risk inspection has also been used in national water source surveys, such as the Rapid Assessment of Drinking-Water Quality (RADWQ) survey series (World Health Organization and UNICEF 2012). Although sanitary risk inspection has been promoted for over two decades and has been widely used in many settings (Ercumen et al. 2017; Howard et al. 2003; Luby et al. 2008), it is unclear how reliably different surveyors can identify a given set of hazards at or surrounding a given set of water sources using these protocols. Reliability refers to the repeatability or consistency of measurements (Heale and Twycross 2015) and may vary both over time and between observers. Consistency in repeated measurements based on the same protocol is often referred to as stability, whilst consistency in measurements made by different observers using the same protocol is referred to as equivalence (Heale and Twycross 2015). In public health, inter-observer agreement studies are commonly used to 13 assess whether observations or measurements can reliably be made by community-based healthcare professionals (Laar et al. 2018; Triasih et al. 2015) rather than specialists. However, studies of inter-observer agreement are less common in low and middle income countries (Bolarinwa 2015) and environmental management. If sanitary risk inspection protocols are to form a robust basis for water source remediation or comparing the relative safety of sources in different areas via water source surveys, then observations need to be consistent across observers. We recently conducted a small-scale study of inter-observer agreement of sanitary risk observations at groundw (...truncated)