Multiple Indicator Cluster Survey 2003 in Afghanistan: Outdated Sampling Frame and the Effect of Sampling Weights on Estimates of Maternal and Child Health Coverage

J HEALTH POPUL NUTR 2011 Aug;29(4):388-399 ISSN 1606-0997 | $ 5.00+0.20 ©INTERNATIONAL CENTRE FOR DIARRHOEAL DISEASE RESEARCH, BANGLADESH Multiple Indicator Cluster Survey 2003 in Afghanistan: Outdated Sampling Frame and the Effect of Sampling Weights on Estimates of Maternal and Child Health Coverage Shivam Gupta1, Muhammad Shuaib2, Stan Becker1, Md. Mokhlesur Rahman3, and David H. Peters1 1 Department of International Health, Johns Hopkins Bloomberg School of Public Health, 615 N Wolfe Street, Baltimore, MD 21205, USA, 2Institute of Statistical Research and Training, University of Dhaka, Dhaka, Bangladesh, and 3 Survey Research, Dhaka, Bangladesh ABSTRACT Due to an urgent need for information on the coverage of health service for women and children after the fall of Taliban regime in Afghanistan, a multiple indicator cluster survey (MICS) was conducted in 2003 using the outdated 1979 census as the sampling frame. When 2004 pre-census data became available, population-sampling weights were generated based on the survey-sampling scheme. Using these weights, the population estimates for seven maternal and child healthcare-coverage indicators were generated and compared with the unweighted MICS 2003 estimates. The use of sample weights provided unbiased estimates of population parameters. Results of the comparison of weighted and unweighted estimates showed some wide differences for individual provincial estimates and confidence intervals. However, the mean, median and absolute mean of the differences between weighted and unweighted estimates and their confidence intervals were close to zero for all indicators at the national level. Ranking of the five highest and the five lowest provinces on weighted and unweighted estimates also yielded similar results. The general consistency of results suggests that outdated sampling frames can be appropriate for use in similar situations to obtain initial estimates from household surveys to guide policy and programming directions. However, the power to detect change from these estimates is lower than originally planned, requiring a greater tolerance for error when the data are used as a baseline for evaluation. The generalizability of using outdated sampling frames in similar settings is qualified by the specific characteristics of the MICS 2003—low replacement rate of clusters and zero probability of inclusion of clusters created after the 1979 census. Key words: Maternal health; Child health; Cluster survey; Sampling frame; Sampling weight; Afghanistan INTRODUCTION The Afghanistan Ministry of Public Health (MoPH) initiated a strategy to reconstruct the health system in 2002 with a focus on laying “the foundations for equitable, quality health care for the people of Afghanistan” (1). The MoPH and other stakeholders required baseline population-level health data for planning and evaluation of this health strategy. Correspondence and reprint requests should be addressed to: Dr. Shivam Gupta Department of International Health Johns Hopkins Bloomberg School of Public Health 615 N Wolfe Street Baltimore MD 21205 USA Information was particularly needed on the coverage of health services to identify provinces with the greatest problems and to provide a reasonable starting point to gauge future change in the health sector. In the post-Taliban period, the first population-based health survey of national scope was conducted by the United Nations Children’s Fund (UNICEF) and the Central Statistics Office (CSO) for the MoPH in 2003. This Multiple Indicator Cluster Survey (MICS) used data of the outdated population census from 1979 for sampling of households. This pragmatic decision was guided by the lack of a national census since 1979 and the urgent need to collect information on the coverage of health services across the country (2). However, questions persisted about the accuracy of the 2003 MICS estimates, given the substantial changes that occurred Outdated sampling frames and population surveys in the population since the sampling frame was constructed in 1979. An opportunity presented itself to re-assess the 2003 estimates when the CSO conducted a pre-census enumeration in 2004 and, in 2006, published the national and provincial census figures (3). Population surveys, such as MICS, are important tools for planning, monitoring, and evaluation of health programmes in developing countries. The results of these surveys are used for summative evaluations and for influencing significant policy decisions on allocation of resources, continuation, and restructuring of programmes (4). In recent times, the ‘instrumental’ use of such results has increased as a greater proportion of decisions on programme oversight is directly based on these results (5). The estimates from the MICS 2003 have been put to ‘instrumental’ use as official health indicators for Afghanistan and have been used as benchmarks for health policy (6). Although the MICS 2003 was the first quantitative assessment of coverage of services targeted to women and children in the post-Taliban period, a further study was needed to assess whether these estimates would be adequate for providing baseline estimates for future evaluation of healthcare coverage in Afghanistan (7). The basic approach in population-based surveys is to collect information from a random sample of people that is representative of the population (8). The sampling and data-collection are usually conducted in multiple stages to overcome the constraints of time, money, and logistics. In order for the results to reflect the situation in the population from which the data are collected, the sampling scheme must be incorporated in the analysis. This usually requires the use of sampling weights and statistical techniques to accommodate for the multi-stage sampling design. The purpose of weighting sample data is to assure the representativeness of the sample vis-a-vis the study population. The inverse of the selection probability of a sampled unit is used as the sampling weight for that unit. The population estimates generated without sampling weights could be biased (8,9). Evaluations of programmes based on the ‘instrumental’ use of these survey results can be adversely affected by this potential bias and lead to incorrect conclusions. The field of summative evaluation of health programmes can benefit from applied research on this aspect of survey methods. This is especially true in post-conflict settings where the lack of good, routine health information systems, vital registration systems, and census data make household surveys indispensable Volume 29 | Number 4 | August 2011 Gupta S et al. for information on the health of the population (10). The scarcity of reliable, comprehensive data is considered one of the greatest challenges in planning and evaluating post-conflict reconstruction of the health systems (11). The clusters for the MICS 2003 were systematically sampled according to the 1979 census using t (...truncated)