Evaluating eHealth: How to Make Evaluation More Methodologically Robust

Pringle M (2009) Evaluating eHealth: How to Make Evaluation More Methodologically Robust. PLoS Med 6(11): e1000186. doi:10.1371/journal.pmed.1000186 Evaluating eHealth: How to Make Evaluation More Methodologically Robust Richard James Lilford 0 Jo Foster 0 Mike Pringle 0 Aziz Sheikh, Edinburgh University, United Kingdom 0 1 Division of Primary Care, School of Health and Population Sciences, University of Birmingham , Birmingham , United Kingdom , 2 School of Community Health Sciences, University of Nottingham , Nottingham , United Kingdom This is the third in a series of three articles on evaluation of eHealth. eHealth-the organisation and delivery of health services and information using information technology (IT) systems-is playing an increasingly important role in shaping health care systems. However, as Catwell and Sheikh described in the first article in this series [1], IT systems can introduce harms as well as benefits. Catwell and Sheikh argued for a general scheme of evaluation starting with careful specification of need and pre-implementation testing in their article. This philosophy of pre-implementation testing resonates strongly with the UK Medical Research Council (MRC) framework for evaluation of complex interventions [2], the tenets of safety science (which endorses the use of analytic procedures to predict the failure rate of a system still in the design phase), and established principles in the IT field where ''alpha testing'' is routine. But how should IT systems be evaluated as they are rolled out following pre-implementation testing? This is the aspect of eHealth we will consider in this essay. - eHealth in the UK Our approach to the evaluation of eHealth has been strongly influenced by the roles we have played in commissioning evaluation research on behalf of the National Programme for Information Technology (NPFIT), an initiative by the Department of Health in England to move the English National Health Service (NHS) towards a single, centrally mandated electronic care record for patients and to connect general practitioners to hospitals. The NHS is investing several billion pounds sterling each year in IT. Investment in a programme of evaluation alongside the NPFIT programmes has The Essay section contains opinion pieces on topics of broad interest to a general medical audience. Summary Points N Evaluation of information technology (IT) systems often requires a mixed methods approach. N External evaluations have many advantages, especially in terms of standardisation, independence, and the possibility of using controlled before and after designs. N Difficulties arise when commissioners ask external evaluations to also provide formative assessments designed to assist in the implementation itself. Under these circumstances the summative results, which encapsulate the overall benefits and harms of a system, may be rendered less generalisable. N We think researchers and commissioners should resist the current fashion of asking external academic teams to combine formative with summative assessments. provided an excellent opportunity to identify newly installed IT systems and to commission prospective studies to assess these programs as they are rolled out. Such evaluations fulfil a real need, since a recent systematic review showed that most of the high quality literature regarding multifunctional health information technology systems comes from 4 benchmark research institutions and that little evidence is available on the effect of multifunctional commercially developed systems [3] such as those that are increasingly implemented in the NHS. Because these evaluations would inevitably be commissioned under intense political and media attention, NPFIT took the principled decision to contract the University of Birmingham to commission the research independently under Department of Health procurement rules. NPFIT could thus influence what was commissioned (i.e., research topics) but not the results obtained. All research commissioners have to take some responsibility for determining the form that research takes. This is particularly so when, as in the case of the evaluation of NPFIT, it is the research commissioner, rather than the researcher, who puts the ball in play. It is the research commissioner who specifies what is to be researched, over what time scale, and with what level of resource. During the course of commissioning evaluation studies for NPFIT, we identified four tricky issues that we think both commissioners of eHealth research and eHealth researchers need to consider, namely: (1) which research methods are suitable for the evaluation of highly complex interventions with diffuse effects, such as IT systems; (2) whether it is necessary to make observations at both the patient and the system level; (3) whether to conduct research that strengthens or improves the intervention being evaluated (formative research) and/ or research that examines the benefits or outcomes of that intervention (summative research); and (4) whether to evaluate research both externally and internally. There is a consensus about the evaluation of clinical treatments, such as drugs in which randomised control trials are state of the art. No such consensus exists yet for the evaluation of highly complex service interventions such as computer systems. However, we believe that the best way to evaluate eHealth is through methodological pluralism [48]. That is, research commissioners and research teams need to recognise the importance of undertaking combined quantitative and qualitative work when evaluating IT systems. Quantitative research can provide important numerical information about how IT systems are performing and is important in theory building, which is necessary to understand how interventions work (not just whether they worked in a particular set of instances) and hence to inform judgements about the generalisability of results from one context to another. Qualitative research can provide information on topics such as ease of use, which will ultimately affect whether the IT system is successful. So, for example, quantitative data may show that computer decision support has little impact on clinical error, while qualitative work explains whyfor instance, clinicians may experience alert fatigue. More controversially qualitative research can also contribute to parameter estimation, particularly under a Bayesian framework [9]. More detailed accounts of methodological pluralism can be found elsewhere [58,1013]. Observations at Patient and System Level Although IT systems can sometimes be studied at the level of individual patients (e.g., computerised decision support) [1416], they often need to be studied at the organisational level [17]. In some cases, this is because IT systems simply cannot be restricted to certain individuals in a group (for example, a computerised theatre booking system). In other cases, contamination (where an intervention le (...truncated)