Developing quality indicators and auditing protocols from formal guideline models: knowledge representation and transformations.

Developing Quality Indicators and Auditing Protocols from Formal Guideline Models: Knowledge Representation and Transformations Aneel Advani, MD,a,b Mary Goldstein, MD,a,b Yuval Shahar, MD, PhD,c Mark A. Musen, MD, PhDa a Stanford Medical Informatics, Stanford University School of Medicine, Stanford, California b Geriatrics Res. Education and Clinical Ctr., VA Palo Alto Health Care System, Palo Alto, California c Department of Information Engineering, Ben-Gurion University of the Negev, Beer-Sheva, Israel Automated quality assessment of clinician actions and patient outcomes is a central problem in guideline- or standards-based medical care. In this paper we describe a model representation and algorithm for deriving structured quality indicators and auditing protocols from formalized specifications of guidelines used in decision support systems. We apply the model and algorithm to the assessment of physician concordance with a guideline knowledge model for hypertension used in a decision-support system. The properties of our solution include the ability to derive automatically (1) context-specific and (2) case-mix-adjusted quality indicators that (3) can model global or local levels of detail about the guideline (4) parameterized by defining the reliability of each indicator or element of the guideline. Introduction Clinical guidelines are increasingly being used as tools to improve the quality of medical care.1 An important task in quality improvement using computerized guidelines is that of developing quality assessments to measure concordance of physician actions and patient outcomes in relation the guideline. One proposed approach to guideline-based quality assessment, suggested by the Agency for Healthcare Research and Quality (AHRQ) is to (manually) derive quality indicators (both outcome and performance measures) from the specification of the guideline itself.2 However, currently used methods for quality indicators, such as the National Quality Measures Clearinghouse (NQMC) of the AHRQ,3 are simply long lists of ratio-based measures that are generated and applied outside of any context of a full clinical guideline. For instance, the predecessor to the NQMC, the CONQUEST System, had about 1,200 measures spread over 50 conditions, giving an average of 24 different quality measures for a condition without any of the 24 being reconciled to the appropriate clinical guideline or guideline elements.4 Clearly, giving condition-specific assessments of quality with 24 different answers begs the questions of creating (a) a coherent modeling and reporting structure among these quality indicators preferably in relation to a clinical guideline for the condition, (b) a method of producing a “global assessment” or overall summary measure of quality, and (c) a method for designing an auditing protocol to know which of the measures to sample from the potential set in relation to the full clinical guideline. To answer these three questions, we extend our previous work on the development of guideline-based quality indicators. Our previous work has addressed questions (a) and especially (b) by showing that we can structure discrete ratio-based quality indicators from full clinical guidelines by modeling higher-level intentions of the guideline. 5,6 The intentions of the guideline allow us to model and evaluate higher-level, more global constraints that encapsulate properties relating to many sub-steps of the guideline processes and the relations between processes and outcomes stipulated by the guideline authors. 7,8 In this paper, we address the problem of how to derive a global yet structured set of quality indicators when the guideline is represented as a formal specification used to drive an automated decision support system. We contrast this problem to that of developing quality indicators from guideline texts, where an automated approach to deriving the quality indicators would currently be unfeasible. Our work is based on the QUIL (Quality Indicator Language) system5 for modeling and executing queries for guideline-based quality indicators. Below, we describe how the QUIL system can be applied to the problem of representing and deriving quality indicators from formalized guidelines. We discuss the implementation of the method in the QUIL Modeler component of the QUIL system. We then show how the QUIL representation is the basis for an automated method to derive quality indicators and to design auditing protocols that are (1) context-specific and (2) case-mixadjusted and that (3) can represent global or local levels of detail about the guideline (4) parameterized by defining the reliability of each indicator or element of the guideline. Methods The QUIL system for automated quality assessment scores adherence to hierarchical sets of quality-indicators derived from guideline plan elements or higher-level intentions. Our method is designed to take guidelines expressed in guideline specification languages such as EON,9 ASBRU,10 or GLIF3,11 and produce a set of related quality indicators as individual nodes in a hierarchical guideline-based QUIL Structure. In Figure 1 we present an example based on our current implementation of the QUIL system that starts with a AMIA 2003 Symposium Proceedings − Page 11 model of an EON guideline for hypertension used in the ATHENA clinical decision support system.12 Quality Indicator Language. The QUIL Modeler component takes guideline elements expressed as frames in the Protégé knowledge-modeling tool for the guideline and produces another Protégé knowledge model expressing the QUIL structure of quality indicators (shown with diagram widget in the Figure). To outline our method, we start first with the target language, QUIL, expanding on our previous exposition.5 In this paper, we focus our discussion on the semantic features of the QUIL graph structure. The QUIL structure is a directed acyclic graph (DAG) of nodes representing quality indicators, and (directed) arcs or edges representing elaborations of the quality indicators into more context-restricted or consequential nodes. In the QUIL structure, the higher-level indicators in the hierarchy can be considered higher-level intentions of the guideline while lower-level indicators may be more specific processes or adjusted outcomes. Individual performance criteria or outcome measures can be embedded as nodes in the guideline-based QUIL structure. QUIL nodes define dyadic (or ratio-based) queries consisting of goal (numerator) and enabling (denominator) constraints, preserving the form of ratio-based population quality indicators. Satisfaction of the goal constraint defines a clinical execution of the medical guideline that satisfies the quality indicator, given the appropriate evaluation context defined by the enabling constraint. The evaluation context can be used to model the case-mix associated with a quality indicator. Furthermore, the expansion of the quality indicators into lower-le (...truncated)