Development of Decision Support Formulas for the Prediction of Bladder Outlet Obstruction and Prostatic Surgery in Patients With Lower Urinary Tract Symptom/Benign Prostatic Hyperplasia: Part II, External Validation and Usability Testing of a Smartphone App

INJ INTERNATIONAL NEUROUROLOGY JOURNAL pISSN 2093-4777 eISSN 2093-6931 Original Article Volume 19 | Number 2 | June 2015 pages 131-210 INTERNATIONAL NEUROUROLOGY JOURNAL Int Neurourol J 2017;21 Suppl 1:S66-75 https://doi.org/10.5213/inj.1734854.427 pISSN 2093-4777 · eISSN 2093-6931 Official Journal of Korean Continence Society / Korean Society of Urological Research / The Korean Children’s Continence and Enuresis Society / The Korean Association of Urogenital Tract Infection and Inflammation einj.org Mobile Web Development of Decision Support Formulas for the Prediction of Bladder Outlet Obstruction and Prostatic Surgery in Patients With Lower Urinary Tract Symptom/Benign Prostatic Hyperplasia: Part II, External Validation and Usability Testing of a Smartphone App Min Soo Choo1, Seong Jin Jeong2, Sung Yong Cho3, Changwon Yoo4, Chang Wook Jeong5, Ja Hyeon Ku5, Seung-June Oh5 Department of Urology, Hallym University Dongtan Sacred Heart Hospital, Hwaseong, Korea Department of Urology, Seoul National University Bundang Hospital, Seongnam, Korea 3 Department of Urology, Seoul Metropolitan Government-Seoul National University Boramae Medical Center, Seoul, Korea 4 Department of Biostatistics, Robert-Stempel College of Public Health & Social Work, Florida International University, Miami, FL, USA 5 Department of Urology, Seoul National University Hospital, Seoul, Korea 1 2 CROSSMARK_logo_3_Test 1/1 https://crossmark-cdn.crossref.org/widget/v2.0/logos/CROSSMARK_Color_square.svg 2017-03-16 Purpose: We aimed to externally validate the prediction model we developed for having bladder outlet obstruction (BOO) and requiring prostatic surgery using 2 independent data sets from tertiary referral centers, and also aimed to validate a mobile app for using this model through usability testing. Methods: Formulas and nomograms predicting whether a subject has BOO and needs prostatic surgery were validated with an external validation cohort from Seoul National University Bundang Hospital and Seoul Metropolitan Government-Seoul National University Boramae Medical Center between January 2004 and April 2015. A smartphone-based app was developed, and 8 young urologists were enrolled for usability testing to identify any human factor issues of the app. Results: A total of 642 patients were included in the external validation cohort. No significant differences were found in the baseline characteristics of major parameters between the original (n=1,179) and the external validation cohort, except for the maximal flow rate. Predictions of requiring prostatic surgery in the validation cohort showed a sensitivity of 80.6%, a specificity of 73.2%, a positive predictive value of 49.7%, and a negative predictive value of 92.0%, and area under receiver operating curve of 0.84. The calibration plot indicated that the predictions have good correspondence. The decision curve showed also a high net benefit. Similar evaluation results using the external validation cohort were seen in the predictions of having BOO. Overall results of the usability test demonstrated that the app was user-friendly with no major human factor issues. Conclusions: External validation of these newly developed a prediction model demonstrated a moderate level of discrimination, adequate calibration, and high net benefit gains for predicting both having BOO and requiring prostatic surgery. Also a smartphone app implementing the prediction model was user-friendly with no major human factor issue. Keywords: Decision Support Systems, Clinical; Nomograms; Validation Studies; Predictive Values of Tests; Mobile Apps • Fund/Grant Support: This study was supported by grant from the Seoul National University Hospital Research Fund (34-2014-0100). • Research Ethics: The study design and the use of patients’ information stored in the hospital database were approved by the Institutional Review Board of Seoul National University Bundang Hospital (B-1410-272-404) and Seoul Metropolitan Government-Seoul National University Boramae Medical Center (26-2014-99). This study was also approved by the Seoul National University Hospital (H-1406-119-591). • Conflict of Interest: MSC, a member of the Editorial Board of INJ, is the first author of this article. However, he played no role whatsoever in the editorial evaluation of this article or the decision to publish it. No potential conflict of interest relevant to this article was reported. Corresponding author: Seung-June Oh http://orcid.org/0000-0002-0322-3539 Department of Urology, Seoul National University Hospital, 101 Daehak-ro, Jongno-gu, Seoul 03080, Korea E-mail: / Tel +82-2-2072-2406 / Fax: +82-2-742-4665 Submitted: March 22, 2017 / Accepted after revision: April 6, 2017 This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited. Copyright © 2017 Korean Continence Society www.einj.org  Choo, et al. • External Validation of Prediction Nomograms for BOO and Prostatic Surgery and Usability Test of Its App INTRODUCTION Lower urinary tract symptoms (LUTS)/benign prostatic hyperplasia (BPH) is not a life-threatening malignant disease. Treatment practices are generally not standardized within a center because decision-making may vary according to patients’ preferences and the subjective judgment of the surgeon [1,2]. Bladder outlet obstruction (BOO) is one of the most important components to assess in patients with LUTS/BPH. However, it is difficult to implement urodynamic studies to diagnose BOO routinely in all patients [3]. In our previous study, we developed a nomogram to predict having BOO and requiring prostatic surgery based on urodynamically determined BOO using our urodynamics database that we have developed over the last 13 years (since 2004) [4]. These prediction models are expected to help support physicians in decision-making and in informing patients of their risk of requiring prostatic surgery. As a next step, it is necessary to confirm the transportability of the model into different but related populations. The successful implementation of prediction models in clinical practice generally requires validation of their performance [5]. The successful estimation of model performance is needed to make judgments regarding model reproducibility and model transportability [6]. Clinical prediction models are commonly developed to facilitate diagnostic or prognostic probability estimations in daily medical practice [6]. A user-friendly modality is necessary for the use of such formulas and nomograms to become a widespread part of routine practice. However, any such app or software should be validated through usability testing among potential end-users. In this study, we aimed to externally validate the prediction model for both hav (...truncated)