Development and validation of a patient no-show predictive model at a primary care setting in Southern Brazil

PLOS ONE, Apr 2019

Patient no-show is a prevalent problem in health care services leading to inefficient resources allocation and limited access to care. This study aims to develop and validate a patient no-show predictive model based on empirical data. A retrospective study was performed using scheduled appointments between 2011 and 2014 from a Brazilian public primary care setting. Fifty percent of the dataset was randomly assigned to model development, and 50% was assigned to validation. Predictive models were developed using stepwise naïve and mixed-effect logistic regression along with the Akaike Information Criteria to select the best model. The area under the ROC curve (AUC) was used to assess the best model performance. Of the 57,586 scheduled appointments in the period, 70.7% (n = 40,740) were evaluated including 5,637 patients. The prevalence of no-show was 13.0% (n = 5,282). The best model presented an AUC of 80.9% (95% CI 80.1–81.7). The most important predictors were previous attendance and same-day appointments. The best model developed from data already available in the scheduling system, had a good performance to predict patient no-show. It is expected the model to be helpful to overbooking decision in the scheduling system. Further investigation is needed to explore the effectiveness of using this model in terms of improving service performance and its impact on quality of care compared to the usual practice.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0214869&type=printable

Development and validation of a patient no-show predictive model at a primary care setting in Southern Brazil

April Development and validation of a patient no- show predictive model at a primary care setting in Southern Brazil Henry Lenzi 0 1 A? ngela Jornada BenID 1 Airton Tetelbom Stein 0 1 0 Servic ?o de Sau ?de Comunita ?ria-Grupo Hospitalar Conceic ?a?o, Porto Alegre, Brazil, 2 Department of Health Sciences, Vrije Universiteit Amsterdam , Amsterdam , The Netherlands , 3 Departamento de Sau ?de Coletiva, Universidade Federal de Cie?ncias da Sau ?de de Porto Alegre , Porto Alegre , Brazil 1 Editor: Antonio Palazo ?n-Bru, Universidad Miguel Hernandez de Elche , SPAIN Patient no-show is a prevalent problem in health care services leading to inefficient resources allocation and limited access to care. This study aims to develop and validate a patient noshow predictive model based on empirical data. A retrospective study was performed using scheduled appointments between 2011 and 2014 from a Brazilian public primary care setting. Fifty percent of the dataset was randomly assigned to model development, and 50% was assigned to validation. Predictive models were developed using stepwise na?ve and mixed-effect logistic regression along with the Akaike Information Criteria to select the best model. The area under the ROC curve (AUC) was used to assess the best model performance. Of the 57,586 scheduled appointments in the period, 70.7% (n = 40,740) were evaluated including 5,637 patients. The prevalence of no-show was 13.0% (n = 5,282). The best model presented an AUC of 80.9% (95% CI 80.1-81.7). The most important predictors were previous attendance and same-day appointments. The best model developed from data already available in the scheduling system, had a good performance to predict patient noshow. It is expected the model to be helpful to overbooking decision in the scheduling system. Further investigation is needed to explore the effectiveness of using this model in terms of improving service performance and its impact on quality of care compared to the usual practice. - Data Availability Statement: All relevant data are within the manuscript and its Supporting Information files. Funding: The author(s) received no specific funding for this work. Competing interests: The authors have declared that no competing interests exist. Introduction Patient no-show is defined as a scheduled appointment that the patient neither attended or canceled on time to be reassigned to another patient [ 1,2 ]. It implies ineffective use of human and logistic resources in a scenario where the demand for health care is greater than the supply. Beyond that, the patient non-attendance could compromise the core principles of primary care: the accessibility and the continuity of care [3]. Whenever a patient misses an appointment, two patients fail to access health care: the no-show patient and the patient who could not book an appointment. Also, patient non-attendance leads to a discontinuity of care, which is associated with worsening of health outcomes such as increasing of hospitalization rates due to exacerbations of chronic conditions [ 4?6 ]. There are also additional costs, e.g., time spent on mitigation strategies and health care staff idle time [ 7 ]. The prevalence of no-show varies worldwide. It has been shown to be higher in low income and developing countries [ 1 ]. Dantas et al., in a literature review, found the second highest noshow prevalence in South America (27.8%) after the African continent (43.0%) [ 1 ]. In Brazil, despite the shortage of data on this issue, studies have reported no-show rates of 48.9% at primary care [ 8 ] and 34.4% at specialized point-of-care service [ 9 ]. It has been described that decreasing no-show rates could have resulted in substantial savings especially in universal health care systems[ 10 ]. For instance, in the National Health Service of the United Kingdom, a reduction in no-show prevalence from 12% to 10.8%, would decrease the annual public expenses by 10% [ 10 ]. Given the aforementioned, factors associated with patient no-show have been investigated to provide insights about target interventions. Young age and previous patient non-attendance have been consistently reported [ 11?16 ]. An association has also been found between longer lead time (the time between the scheduling and the appointment) and higher no-show rates [1]. Other factors are related to the type and severity of the problem; sociodemographic conditions; appointment period of the year and distance to service [ 17 ]. Since these factors may vary across populations and health care services, a common set of universal determinants is unlikely to be found. Hence, this implies that it behooves each service to investigate local predictors, to tailor actions to address the issue. Based on that, no-show predictive models have been developed to optimize the scheduling process and service performance, but mainly in developed countries setting [ 17?21 ]. To the best of our knowledge, there have been no published studies about developing noshow predictive models based on data from a Brazilian public health care scenario. Therefore, the present study aims to explore the factors associated with a no-show at a Brazilian primary care setting and to develop and validate a patient no-show predictive model based on empirical data. Materials and methods Study design A retrospective study was performed based on the scheduled appointments registered in the scheduling system of a public primary care service of the Grupo Hospitalar Conceic??o between November 1, 2011, and March 31, 2014. Patient record numbers were irreversibly replaced by a sequence of random characters (fully anonymized) before the analysis. The study was approved by the Ethics Committee of the Grupo Hospitalar Conceic??o (number 2.349.672). Ethics committee waived the requirement for informed consent. Patient no-show predictors The no-show predictors were chosen based on literature, on the experience of the primary care service team and considering the availability of data in the scheduling system. The unit of analysis was the scheduled appointment. No-show was defined as non-attendance at the appointment day until the closing time of the service at 6 pm. For each unit of analysis, the following variables were available in the scheduling system: patient record number; age (years); gender (male/female); self-reported race/ethnicity registered in the scheduling system according to the Ethno-Racial Characteristics of the Brazilian Population [ 22 ] and dichotomized as white and non-white; appointment day; date and time of the scheduling; date and time of the appointment; appointment shift (morning or afternoon); appointment weekday; appointment 2 / 14 month; appointment attendance (attendance/no-show); health professional categories (nursing, dentist, general practitioner, pharmacist, nutritionist, psychologist, social worker and oral health technician) and types of appointment?S1 Table. For each appointment scheduled, the following metrics were calculated: ?lead time? (the time between the scheduling and the appointment in days); ?waiting time? (difference between the appointment time and the time it was held in minutes); ?patient previous attendance? (number of times the patient has attended the previous appointments) and ?patient previous same-day appointment? (number of times the patient had previous same-day appointments). A dichotomous variable ?same-day appointment calculated? was generated for those appointments scheduled and held on the same day (1) or not (0) to verify consistency with the category ?same-day appointment? of the variable ?types of appointment??S1 Table. Observations were included if they did not fulfill the exclusion criteria. Observations were excluded and deleted from the analysis if: 1) the information on the outcome was not registered, and 2) there was no possibility to derive metrics from the available data. Statistical analysis Descriptive analysis of the dataset. First, a descriptive analysis of variables was performed by the categorical outcome: attendance (0) and no-show (1). Means and standard deviations were calculated for normally distributed variables and medians and quartiles for nonparametric variables. Frequencies and percentages were calculated to describe categorical variables. The normality of continuous variables was evaluated by the D?Agostino skewness test [ 23 ]. All variables were included as predictors in the process of model selection. Analyses were performed in R software, version 3.5.2. Model development and selection. Afterwards, the dataset was randomly divided into two subsets by using the Caret R package [ 24 ]: 1) 50% of the dataset was assigned to develop the logistic regression model (training subset) and 2) the remaining 50% was assigned to validate the model (validation subset). Subsequently, a na?ve logistic regression was performed along with the Akaike Information Criteria (AIC) to select the most parsimonious model (best model) in a stepwise backward algorithm by using the stepAIC function [ 25 ]. In the stepwise backward AIC, the selection process starts with a model including all variables of interest. At each step, a variable is excluded from the model if its elimination results in a higher AIC value than the previous model [ 25 ]. The best model is defined as the one with the lowest AIC value compared to the other possible explanatory models. Thirteen variables of interest were considered with their respective dummy variables resulting in 8,191 possible models. Additionally, a mixed-effect model was developed considering patients and health professionals as random intercepts and hence, accounting for the variance between- and within-patients and professionals on the outcome [ 26 ]. The glmer function was used to perform the mixed-effect model analyses [ 27 ]. To select the best mixed-effect model, a forward and backward stepwise algorithm was performed based on the AIC criteria. The intra-class correlation coefficient (ICC) [ 26 ] within-clusters was calculated for the best mixed-effect model to verify within-clusters dependency using the sjstats R package [ 28 ]. The best na?ve model and the best mixed-effect model were compared, and the model with the smallest AIC value was considered the final best model. The variable importance in the best final model was estimated using the permute. varimp R function [ 29 ]. In this function, the values of each predictor are randomly permuted to break their association with the response, and the model is re-fit to a new dataset containing the permuted values [ 29 ]. The fit of the new model is compared to that of the original model [ 29 ]. The variables presenting the higher AICc difference between the original model and the 3 / 14 model with the permuted predictor were considered as having higher importance in the model. Akaike inflection criteria. The Akaike Information Criteria (AIC) was developed by Hirotugu Akaike to identify which combination of variables would best explain an outcome given a universe of potentially explanatory models [ 30 ]. The AIC is defined by the expression: AIC = - 2log (L (? | y)) + 2K. Where, log (L (? | y)) represents the maximum likelihood ratio (model quality), and K represents the number of variables included in the model (complexity) [ 30 ]. The best fit model is the one with the smallest possible number of parameters (parsimony) with a higher probability of explaining the outcome. This method is indicated to select models, which are developed based on observational data [ 30,31 ]. Model performance and validation. The area under the ROC curve (AUC) was used to assess the performance of the best model by using the pROC R package [ 32 ]. The AUC was calculated based on the training and the validation subsets and compared by using the roc.test function [ 32 ]. Additionally, the threshold that maximizes the sensitivity and specificity in classifying patients as no-show was identified. Sensitivity analysis. A sensitivity analysis was conducted based on a model developed on 80% of the dataset (p80) and validated on the remaining 20% of the dataset to explore if it would improve the final best model performance. The model selection of the final best p80 model was carried out following the methodology applied to select the final best p50 model. The AUC?s of both best models were compared by using the roc.test function [ 32 ]. Results Descriptive analysis of the dataset Of the 57,586 scheduled appointments in the period, 70.7% (n = 40,740) fulfilled the inclusion criteria including 5,637 patients. The prevalence of no-show was 13.0% (n = 5,282). The mean age of the sample was 41 years (SD 23.2). Thirty percent of the appointments were scheduled by male (n = 12,219) and 82.1% (n = 33,442) by patients self-reported as white. Thirty-six percent of the sample was delivered as same-day appointments (n = 14,653). The mean age of the attendance group was 41.2 years (SD 23.3), and the mean age of the no-show group was 39.8 years (SD 22.0). The median of patient previous attendance was smaller in the no-show (4.0, IQR 7.0) group compared to the attendance group (5.0, IQR 8.0), whereas the median lead time was higher (14.0 days, IQR 17.0) in the no-show group compared to the attendance group (2.2 days, IQR 14.1)?Table 1. Model development and selection The backward stepwise algorithm compared all possible models based on 20,368 scheduled appointments (50% of the dataset). Two observations were excluded due to missingness in the waiting time variable. The stepwise procedure fitted forty models. The best na?ve model presented an AIC value of 12,974. The best mixed-effect model presented an AIC value of 12,763 which was smaller than the na?ve model and hence it was considered the best final model (p50). Table 2 presented the final best model results including the combination of variables that would best estimate the probability of no show, given a universe of potentially explanatory models. The most important variables in the p50 model were the type of appointment (difference in the AICc = 804), previous attendance (AICc difference = 281), previous same-day appointment (AICc difference = 114) and same-day appointment (AICc difference = 110). The variance within patients was 0.185 (SD 0.430) and within health professionals was 0.020 (SD 0.143). The intra-class correlation coefficient between patients was 0.05 and between health professionals was 0.003. 4 / 14 Total N = 40,740 41.0 (23.2) 12,219 (30.0) 33,442 (82.1) 5.0 (8.0) 2.0 (3.0) Mental health group: n (%) Quality of life group: n (%) Model performance and validation Practical application of the predictive patient no-show model A patient of 20 years-old, non-white, 0 previous attendance and 1 previous same-day appointment scheduled an appointment with the psychologist within 14 days (appointment weekday 6 / 14 User embracement Same-day appointment Extra-same-day appointment Extra-scheduled appointment Dental urgency/emergency Extra-scheduled dental appointment Rapid HIV test First dental appointment HT/DM dental appointment Dental appointment Pharmacist appointment Nutritionist appointment Psychologist appointment Social worker appointment Oral health technician appointment Prenatal health care program Child health care program Pap smear screening program HT/DM health care program Tuberculosis control program Adult health care program Return Individual appointment Elderly group Asthma control group Tabaco control group Mental health group Quality of life group Afternoon Monday Tuesday Wednesday Thursday Appointment shift Appointment weekday Regression coefficients -1.193 -0.007 -0.018 -0.119 and month: Monday and March, respectively). The probability of patient-no-show, according to the p50 model was 0.591. Based on the threshold of 0.140, the patient would be classified as a no-show. If an appointment is overbooked in this slot, the model would have a probability of 81% to correctly identify the true positives and negatives of no-show?S1 File. Discussion This study explored the factors associated with no-show at a primary care setting in Southern Brazil and developed and validated a patient no-show predictive model based on empirical data. It revealed that previous patient attendance and same-day appointments were the most important predictors of a no-show in the service investigated. More importantly, the results showed that the best model, developed from data already available in the scheduling system, had a good performance with a probability of 81% to correctly identify the true positives and negatives of a patient no-show. Alike previously published models, our results revealed previous patient attendance as one of the most important predictors of no-show [ 18?21,33 ]. Aware of this, Harris et al. developed a predictive no-show model including solely the patient?s past attendance history, observing an accuracy around 0.70 [ 21 ], which is slightly lower than our best model and other models with additional factors (i.e., sociodemographic and medical background). Nevertheless, it is difficult to compare the performance of all these models because they (1) were based on different population, (2) included different predictors (i.e., marital status, religion, socioeconomic status, insurance coverage, and comorbidities), (3) used different modelling techniques and (3) considered different methods to estimate patient past attendance history. However, their performances were very similar to what we found, ranging from 0.69 to 0.82 [ 18?21,33 ]. Our study differs from the previous models because we used a mixed-effect modelling approach to account for the variance across patients and health professional and developed relatively simple models and compared them using a multimodel inference method. The AIC 8 / 14 allows selecting models considering the strength of evidence and uncertainty in the selection process [ 34 ]. This information-theoretic approach has been deemed more appropriate to deal with the complexity of the real world problems and has been mainly used by biologists [ 34 ]. In this method, the goal is to identify the best fit model given the data available, which is quite different than finding full truth [ 34,35 ]. Given the complexity of the no-show issue, which encompasses several factors, the final best model probably did not include all universe of variables to explain the outcome. Aware of this, our study had conducted further testing and, hence, observing a good performance of the final best model to predict patient no-show when validated on empirical data. Based on the results of this study, one could explore the potential of incorporating the patient no-show predictive model into the scheduling system of the service, which might aid overbooking approaches in programs associated with high non-attendance rates (i.e., Pap smear screening or Hypertension/Diabetes health care program). Additionally, adopting overbooking based on patient no-show predictive models, instead of using only a flat non-attendance prevalence, would avoid false positives and hence avoid excessive work load for the healthcare team [ 17 ]. As expected, we found that same-day appointment is less likely of nonshow than scheduled appointment ahead of time. In a systematic review, Ansell et al. [ 36 ] found that no-show rates decreased after the implementation of same-day appointments, further referred to as ?open access scheduling.? The rationale of the open access approach is that patients would have access to care in the time when they need most. On the other hand, open access scheduling may overload the health care staff if the demand exceeds the supply, which could compromise the quality of care [ 37 ]. Hitherto, its implementations would require at 10 / 14 Fig 1. Performance of the patient no-show predictive model developed on 50% (p50) of the dataset. (A) shows the AUC (95% confidence interval) of the p50 model tested on the same subset from which it was developed (training subset). (B) shows the AUC (95% confidence interval) of the p50 model tested on the remaining 50% of the dataset (validation subset). The point in the curve is the threshold that maximizes the sensitivity and specificity of the model. The sensitivity and specificity are in the parenthesis. AIC: Akaike Information Criteria. AUC: area under the Receiver Operating Characteristic curve. least a resizing of the service?s patient load [ 37 ] and an evaluation of its impact on quality of care [ 36 ], which are beyond the scope of this study. However, our results may provide some insight about the daily agenda optimization, aiming at reaching a balance between same dayappointments and advanced-scheduled ones. This study has some limitations. Despite the advantage of the stepwise algorithm of comparing predictors automatically, it may lead to spurious associations [ 25,38 ]. Taking this into account, we focused on select no-show predictors reported on literature and on the experience of the primary health care team instead of just relying on the algorithm?s choice. Another issue regards to missing data excluded from the analysis. However, we assume the missingness was related to data registration issues and not associated with variables of interest. Hence, the exclusion of missing cases is known to produce unbiased estimates and conservative results [39]. Also, we did not include sociodemographic statuses such as educational level and income due to missingness in the database. Further investigation should explore if these inputs improve the performance of predictive no-show models. Conclusions This study developed and validated a patient no-show predictive model based on data from a public primary care setting in Southern Brazil. It mainly revealed that using the information already available in the scheduling system, the best fit model presented a good performance to predict noshow when empirically validated. Additionally, the methodology applied in this study may be useful to other health care services to develop predictive no-show models based on their specific population. It is expected this approach to be helpful to overbooking decision in scheduling systems. Further investigation is needed to explore the effectiveness of using this model in terms of improving service performance and its impact on quality of care compared to the usual practice. Supporting information S1 Table. Types of appointment: Definition based on the primary care service database. (DOCX) 11 / 14 S1 File. No-show predictive model calculator. Practical application of the predictive patient no-show model. (XLSX) Acknowledgments The primary care team of the Grupo Hospitalar Conceic??o for contributing with their experience. In memoriam of our extraordinary colleague Jose? Mauro Ceratti Lopes who introduced the authors. Author Contributions Conceptualization: Henry Lenzi, A?ngela Jornada Ben, Airton Tetelbom Stein. Data curation: Henry Lenzi, A? ngela Jornada Ben. Formal analysis: Henry Lenzi, A?ngela Jornada Ben. Investigation: Henry Lenzi, A?ngela Jornada Ben. Methodology: Henry Lenzi, A?ngela Jornada Ben, Airton Tetelbom Stein. Project administration: Henry Lenzi. Resources: Henry Lenzi. Software: Henry Lenzi, A?ngela Jornada Ben. Supervision: A?ngela Jornada Ben, Airton Tetelbom Stein. Validation: A?ngela Jornada Ben. Writing ? original draft: Henry Lenzi, A? ngela Jornada Ben, Airton Tetelbom Stein. Writing ? review & editing: Henry Lenzi, A? ngela Jornada Ben, Airton Tetelbom Stein. 12 / 14 13 / 14 1. Dantas LF , Fleck JL , Oliveira FLC , Hamacher S. No-shows in appointment scheduling-a systematic literature review . Health Policy . 2018 ; 122 : 412 - 421 . https://doi.org/10.1016/j.healthpol. 2018 . 02 .002 PMID: 29482948 2. Tuso PJ , Murtishaw K , Tadros W. The Easy Access Program: A Way to Reduce Patient No-Show Rate, Decrease Add-Ons to Primary Care Schedules, and Improve Patient Satisfaction . The Permanente Journal . 1999 ; 3 : 68 - 71 . 3. George A , Rubin G . Non-attendance in general practice: a systematic review and its implications for access to primary health care . Fam Pract . 2003 ; 20 : 178 - 184 . https://doi.org/10.1093/fampra/20.2.178 PMID: 12651793 4. Nguyen DL , DeJesus RS , Wieland ML . Missed Appointments in Resident Continuity Clinic: Patient Characteristics and Health Care Outcomes . J Grad Med Educ . 2011 ; 3 : 350 - 355 . https://doi.org/10. 4300/ JGME-D- 10- 00199 .1 PMID: 22942961 5. Nuti LA , Lawley M , Turkcan A , Tian Z , Zhang L , Chang K , et al. No-shows to primary care appointments: subsequent acute care utilization among diabetic patients . BMC Health Services Research . 2012 ; 12 : 304 . https://doi.org/10.1186/ 1472 -6963-12-304 PMID: 22953791 6. Hwang AS , Atlas SJ , Cronin P , Ashburner JM , Shah SJ , He W , et al. Appointment ?no-shows? are an independent predictor of subsequent quality of care and resource utilization outcomes . J Gen Intern Med . 2015 ; 30 : 1426 - 1433 . https://doi.org/10.1007/s11606-015 -3252-3 PMID: 25776581 7. Berg B , Murr M , Chermak D , Woodall J , Pignone M , Sandler RS , et al. Estimating the Cost of No-shows and Evaluating the Effects of Mitigation Strategies . Med Decis Making . 2013 ; 33 : 976 - 985 . https://doi. org/10.1177/0272989X13478194 PMID: 23515215 8. Izecksohn MMV , Ferreira JT . Falta ?s consultas me?dicas agendadas: percepc??es dos usua?rios acompanhados pela Estrate? gia de Sau?de da Fam??lia, Manguinhos, Rio de Janeiro. Rev Bras Med Fam Comunidade . 2014 ; 9 : 235 - 241 . https://doi.org/10.5712/rbmfc9( 32 ) 960 9. Bender A da S , Molina LR , Mello ALSF de. Absente??smo na atenc??o secunda?ria e suas implicac??es na atenc??o ba?sica . Espac?o para a Sau?de-Revista de Sau?de Pu?blica do Parana?. 2010 ; 11 : 56 - 65 . 10. Ellis DA , Jenkins R . Weekday Affects Attendance Rate for Medical Appointments: Large-Scale Data Analysis and Implications . PLoS One . 2012 ; 7 . https://doi.org/10.1371/journal.pone. 0051365 PMID: 23272102 11. Bean AG , Talaga J . Appointment Breaking: Causes and Solutions . Journal of Health Care Marketing . 1992 ; 12 : 14 - 21 . PMID: 10123581 12. Lacy NL , Paulman A , Reuter MD , Lovejoy B. Why We Don't Come : Patient Perceptions on No-Shows . Ann Fam Med . 2004 ; 2 : 541 - 545 . https://doi.org/10.1370/afm.123 PMID: 15576538 13. Kaplan-Lewis E , Percac-Lima S . No-Show to Primary Care Appointments: Why Patients Do Not Come . J Prim Care Community Health . 2013 ; 4 : 251 - 255 . https://doi.org/10.1177/2150131913498513 PMID: 24327664 14. Nancarrow S , Bradbury J , Avila C . Factors associated with non-attendance in a general practice super clinic population in regional Australia: A retrospective cohort study . Australas Med J. 2014 ; 7 : 323 - 333 . https://doi.org/10.4066/AMJ. 2014 . 2098 PMID: 25279008 15. Norris JB , Kumar C , Chand S , Moskowitz H , Shade SA , Willis DR . An empirical investigation into factors affecting patient cancellations and no-shows at outpatient clinics . Decision Support Systems . 2014 ; 57 : 428 - 443 . https://doi.org/10.1016/j.dss. 2012 . 10 .048 16. Torres O , Rothberg MB , Garb J , Ogunneye O , Onyema J , Higgins T . Risk factor model to predict a missed clinic appointment in an urban, academic, and underserved setting . Popul Health Manag . 2015 ; 18 : 131 - 136 . https://doi.org/10.1089/pop. 2014 .0047 PMID: 25299396 17. Huang Y , Hanauer DA . Patient No-Show Predictive Model Development using Multiple Data Sources for an Effective Overbooking Approach . Appl Clin Inform . 2014 ; 5 : 836 - 860 . https://doi.org/10.4338/ ACI-2014 - 04-RA-0026 PMID: 25298821 18. Daggy J , Lawley M , Willis D , Thayer D , Suelzer C , Delaurentis PC , et al. Using no-show modeling to improve clinic performance . Health informatics journal . 2010 ; 16 : 246 - 259 . https://doi.org/10.1177/ 1460458210380521 PMID: 21216805 19. Mark Reid P , Samuel Cohen MD , Hank Wang MD , Aung Kaung MD , Anish Patel MD , Vartan Tashjian BS , et al. Preventing Patient Absenteeism: Validation of a Predictive Overbooking Model . American Journal of Managed Care . 2015 ; 21. Available: http://www.ajmc.com/journals/issue/2015/2015-vol21 - n12/preventing-patient-absenteeism-validation-of-a-predictive-overbooking-model/ 20. Goffman RM , Harris SL , May JH , Milicevic AS , Monte RJ , Myaskovsky L , et al. Modeling Patient NoShow History and Predicting Future Outpatient Appointment Behavior in the Veterans Health Administration . Military Medicine . 2017 ; 182 : e1708 - e1714 . https://doi.org/10.7205/ MILMED-D- 16-00345 PMID: 29087915 21. Harris SL , May JH , Vargas LG . Predictive analytics model for healthcare planning and scheduling . European Journal of Operational Research . 2016 ; 1 : 121 - 131 . https://doi.org/10.1016/j.ejor. 2016 . 02 . 017 22. IBGE. Ethno-Racial Characteristics of the Population | Statistics | Instituto Brasileiro de Geografia e Estat??stica [Internet]. [cited 16 Feb 2019 ]. Available: https://www.ibge.gov.br/en/np-statistics/social/ population/17590-ethno -racial-characteristics-of-the-population .html?=& t= o -que-e 23. Zhang Z. Univariate description and bivariate statistical inference: the first step delving into data . Ann Transl Med . 2016 ; 4 . https://doi.org/10.21037/atm. 2016 . 02 .11 PMID: 27047950 24. Kuhn M. Building Predictive Models in R Using the caret Package | Kuhn | Journal of Statistical Software . https://doi.org/10.18637/jss.v028.i05 25. Zhang Z. Variable selection with stepwise and best subset approaches . Ann Transl Med . 2016 ; 4 . https://doi.org/10.21037/atm. 2016 . 03 .35 PMID: 27162786 26. Twisk JWR . Applied Longitudinal Data Analysis for Epidemiology: A Practical Guide . Cambridge University Press; 2013 . 27. Bolker B. lme4 -package: Linear, generalized linear, and nonlinear mixed models in lme4: Linear MixedEffects Models using ?Eigen? and S4-Version 1 . 1 - 20 [Internet]. [cited 16 Feb 2019 ]. Available: https:// github.com/lme4/lme4/ 28. Lu ?decke D. sjstats: Collection of Convenient Functions for Common Statistical Computations [Internet]. 2019 . Available: https://CRAN.R-project.org/package=sjstats 29. Grafmiller J. permute.varimp: Permutation variable importance for regression in JGmermod: Custom Functions For Mixed-Effects Regression Models [Internet]. 2017 . Available: https://rdrr.io/github/ jasongraf1/JGmermod/man/permute.varimp.html 30. Akaike H. A new look at the statistical model identification . IEEE Transactions on Automatic Control . 1974 ; 19 : 716 - 723 . https://doi.org/10.1109/TAC. 1974 .1100705 31. Herland M , Khoshgoftaar TM , Wald R. A review of data mining using big data in health informatics . Journal Of Big Data . 2014 ; 1: 2 . https://doi.org/10.1186/ 2196 -1115-1-2 32. Robin X , Turck N , Hainard A , Tiberti N , Lisacek F , Sanchez J-C, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves . BMC Bioinformatics . 2011 ; 12 : 77 . https://doi.org/ 10.1186/ 1471 -2105-12-77 PMID: 21414208 33. Huang Y-L , Hanauer DA . Time dependent patient no-show predictive modelling development . Int J Health Care Qual Assur . 2016 ; 29 : 475 - 488 . https://doi.org/10.1108/IJHCQA-06 -2015-0077 PMID: 27142954 34. Burnham KP , Anderson DR. Model Selection and Multimodel Inference-A Practical Information-Theoretic Approach [Internet]. Springer; 2002 . Available: http://www.springer.com/us/book/9780387953649 35. Grueber CE , Nakagawa S , Laws RJ , Jamieson IG . Multimodel inference in ecology and evolution: challenges and solutions . Journal of Evolutionary Biology . 24 : 699 - 711 . https://doi.org/10.1111/j.1420- 9101 . 2010 . 02210 . x PMID : 21272107 36. Ansell D , Crispo JAG , Simard B , Bjerre LM . Interventions to reduce wait times for primary care appointments: a systematic review . BMC Health Serv Res . 2017 ; 17 . https://doi.org/10.1186/s12913-017- 2219-y PMID: 28427444 37. Kiran T , O'Brien P. Challenge of same-day access in primary care . Can Fam Physician . 2015 ; 61 : 399 - 400 . PMID: 25971751 38. Anderson DR , Burnham KP. Avoiding Pitfalls When Using Information-Theoretic Methods . The Journal of Wildlife Management . 2002 ; 66 : 912 . https://doi.org/10.2307/3803155 39. Kang H. The prevention and handling of the missing data . Korean J Anesthesiol . 2013 ; 64 : 402 - 406 . https://doi.org/10.4097/kjae. 2013 . 64 .5.402 PMID: 23741561


This is a preview of a remote PDF: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0214869&type=printable

Henry Lenzi, Ângela Jornada Ben, Airton Tetelbom Stein. Development and validation of a patient no-show predictive model at a primary care setting in Southern Brazil, PLOS ONE, 2019, DOI: 10.1371/journal.pone.0214869