Estimating recruitment rates for routine use of patient reported outcome measures and the impact on provider comparisons

Andrew Hutchings 0 Jenny Neuburger 0 Jan van der Meulen 0 Nick Black 0 0 Department of Health Services Research & Policy, London School of Hygiene & Tropical Medicine , 15-17 Tavistock Place, London WC1H 9SH , UK Background: The routine use of patient reported outcome measures (PROMs) aims to compare providers as regards the clinical need of their patients and their outcome. Simple methods of estimating recruitment rates based on aggregated data may be inaccurate. Our objectives were to: use patient-level linked data to evaluate these estimates; produce revised estimates of national and providers' recruitment rates; and explore whether or not recruitment bias exists. Methods: Case study based on patients who were eligible to participate in the English National PROMs Programme for elective surgery (hip and knee replacement, groin hernia repair, varicose vein surgery) using data from pre-operative questionnaires and Hospital Episode Statistics. Data were linked to determine: the eligibility for including operations; eligibility of date of surgery; duplicate questionnaires; cancelled operations; correct assignment to provider. Influence of patient characteristics on recruitment rates were investigated. Results: National recruitment rates based on aggregated data over-estimated the true rate because of the inclusion of ineligible operations (from 1.9% - 7.0% depending on operation) and operations being cancelled (1.9% - 3.6%). Estimates of national recruitment rates using inclusion criteria based on patient-level linked data were lower than those based on simple methods (eg hip replacement was 73% rather than 78%). Estimates of provider's recruitment rates based on aggregated data were also adversely affected by attributing patients to the wrong provider (2.4% - 4.9%). Use of linked data eliminated all estimates of over 100% recruitment, though providers still showed a wide range of rates. While the principal determinant of recruitment rates was the provider, some patients' socio-demographic characteristics had an influence on non-recruitment: non-white (Adjusted Odds Ratio 1.25-1.67, depending on operation); most deprived socio-economic group (OR 1.11-1.23); aged over 75 years (OR 1.28-1.79). However, there was no statistically significant association between providers' recruitment rates and patients' pre-operative clinical need. Conclusions: Accurate recruitment rates require the use of linked data to establish consistent inclusion criteria for numerators and denominators. Non-recruitment will bias comparisons of providers' pre-operative case-mix and may bias comparisons of outcomes if unmeasured confounders are not evenly distributed between providers. It is important, therefore, to strive for high recruitment rates. - Background There is increasing interest in using patient reported outcome measures (PROMs) to assess the quality of providers of surgery, most notably in Sweden [1] and, more recently, in England [2]. The National PROMs Programme that started in England in April 2009 covered four common operations and aimed to allow comparisons of providers in two principal ways: the appropriateness of patients undergoing surgery according to their preoperative health status and quality of life; and the outcome of care as judged by the change in health status and quality of life [3]. One particular concern is the extent to which incomplete recruitment of patients might bias comparisons of providers. This raises two methodological questions: are the estimates of recruitment rates accurate; and to what extent do those recruited differ from those not recruited? As regards the first question, simple recruitment rates for each provider are routinely reported by the Health and Social Care Information Centre (HSCIC) [4] based on the number of completed pre-operative PROM questionnaires and the number of procedures undertaken according to a routine administrative data source, the Hospital Episode Statistics (HES). The shortcomings of this approach are evident in the number of providers who are reported as having more than 100% recruitment. Potential causes of inaccuracies in the estimates of the numerator include: patients being included who underwent an ineligible procedure; despite having completed a pre-operative questionnaire, the procedure did not take place; the procedure was delayed such that it took place outside the time period being assessed; and patients being inadvertently asked to complete more than one pre-operative questionnaire, creating duplicate entries. Errors in the denominator can arise because patients are attributed to the wrong provider, a particular risk when operations are sub-contracted to another provider. To investigate the extent to which these problems occur, data on individual patients completing a pre-operative questionnaire need to be linked to their data in the routine administrative data. In this way it is possible to ensure the individual patients included in the numerator a (...truncated)