Evaluating a handheld decision support device in pediatric intensive care settings

JAMIA Open, Mar 2019

To evaluate end-user acceptance and the effect of a commercial handheld decision support device in pediatric intensive care settings. The technology, pac2, was designed to assist nurses in calculating medication dose volumes and infusion rates at the bedside.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

https://academic.oup.com/jamiaopen/article-pdf/2/1/49/28224192/ooy055.pdf

Evaluating a handheld decision support device in pediatric intensive care settings

Abstract Objective To evaluate end-user acceptance and the effect of a commercial handheld decision support device in pediatric intensive care settings. The technology, pac2, was designed to assist nurses in calculating medication dose volumes and infusion rates at the bedside. Materials and Methods The devices, manufactured by InformMed Inc., were deployed in the pediatric and neonatal intensive care units in 2 health systems. This mixed methods study assessed end-user acceptance, as well as pac2’s effect on the cognitive load associated with bedside dose calculations and the rate of administration errors. Towards this end, data were collected in both pre- and postimplementation phases, including through ethnographic observations, semistructured interviews, and surveys. Results Although participants desired a handheld decision support tool such as pac2, their use of pac2 was limited. The nature of the critical care environment, nurses’ risk perceptions, and the usability of the technology emerged as major barriers to use. Data did not reveal significant differences in cognitive load or administration errors after pac2 was deployed. Discussion and Conclusion Despite its potential for reducing adverse medication events, the commercial standalone device evaluated in the study was not used by the nursing participants and thus had very limited effect. Our results have implications for the development and deployment of similar mobile decision support technologies. For example, they suggest that integrating the technology into hospitals’ existing IT infrastructure and employing targeted implementation strategies may facilitate nurse acceptance. Ultimately, the usability of the design will be essential to reaping any potential benefits. INTRODUCTION Medical errors are the third leading cause of death in the United States.1 Adverse medication events (AMEs) due to medical errors are a major threat to patient safety during inpatient care.2–4 The problem is particularly pronounced in critical care settings and pediatric populations. Critical care settings such as intensive care units are complex, fast-paced environments in which the severity of patients’ conditions, as well as the number and types of medications required for treatment, can all contribute to an increased risk of errors and subsequent AMEs.5,6 AME rates can be even higher among pediatric patients, especially those in critical care settings,7,8 due to weight-based dosing, rapid changes in weight with growth, and the lack of standard pediatric formulations.9–13 Incorrect dosing of intravenous medications is one of the most frequently reported types of medication errors found in pediatric settings, often attributable to human mistakes in computing dosage and dosing intervals.10,14–19 Research suggests that intravenous medications that are administered without first being prepared in the pharmacy are particularly prone to such errors.18,20,21 In this relatively common situation,20,22,23 nurses typically perform manual calculations to translate physician orders into an intravenous flow rate (volume/time) or volume in a syringe, considering the patient’s weight, ordered dose, and concentration of the available medication solutions. Previous studies have shown that human errors are common when performing such complex calculations.24–26 There has been increasing recognition of the value of using decision support technologies (DSTs) to improve clinical practice and reduce medication errors, especially technologies designed to assist medical professionals at the point, that is time and location, the decision is being made.5,27,28 Such just-in-time decision support is usually provided by interactive software and may be delivered through various types of devices (e.g., desktop computer). The algorithms underlying this software are typically based on well-validated, research-driven evidence or on widely accepted standards and guidelines recommended by authoritative organizations. Appropriate use of DSTs should, therefore, lead to significant improvements in quality of care and patient safety. Since pediatric critical care nurses prepare and administer intravenous medications in multiple locations (e.g., bedside, medication room), handheld devices are needed to deliver just-in-time decision support. Unfortunately, little is known about the effectiveness of portable DSTs.28,29 In addition, while more healthcare organizations are implementing commercial products rather than developing their own “homegrown” systems,29,30 even less is known about commercial tools.28,29 Among the few existing studies evaluating mobile DSTs, there is evidence that they may be effective; however, most have been conducted in laboratory settings31,32 and few have included nursing participants.30 Field evaluations of other types of DSTs have shown mixed evidence of success, despite the great promise that they had demonstrated in laboratory settings.28,33 This may be explained, at least in part, by low user acceptance. For example, a systematic review indicated that computer-generated medication safety alerts were overridden by clinicians in 49–96% of cases.34 Low user acceptance, in turn, may be due to technological, behavioral, social, and organizational barriers prohibiting effective technology use.35–37 When such barriers are not identified and addressed, implementing technologies can also result in unintended adverse consequences (e.g., technology-introduced errors).38–41 Thus, there is a critical need to understand the barriers to, and facilitators of, end-users’ acceptance of mobile DSTs, particularly commercial tools, as well as to determine their effectiveness in the field. Towards this end, the study described in this paper, funded by the Telemedicine and Advanced Technology Research Center, Department of Defense (Contract # W81XWH1010606), aimed to deploy and rigorously evaluate a laboratory-tested, commercial handheld decision support tool for reducing dosing errors in administering intravenous and other liquid medications. This multisite, mixed methods study was designed to evaluate (1) nurse acceptance and satisfaction and (2) the tool’s effect on the cognitive load associated with dose calculations and the rate of medication errors. METHODS Handheld decision support tool The commercial DST evaluated in this study (shown in Figure 1) was the pharmaceutical algorithm computerized calculator (“pac2”) manufactured by InformMed Inc. (Peoria, Illinois, USA). It is a standalone handheld device designed to assist nurses in calculating medication dose volumes and infusion rates, to provide easy access to essential drug administration information at the point of care, and to detect unsafe doses. Using pac2 has the potential to reduce medication dosing errors by supporting nurses in preparing, dispensing, administering, and monitoring medication dosing for hospitalized patients.31Figure 2 illustrates typical pac2 workflows. Figure 1. View largeDownload slide Rendering of the pac2TM device. Figure 1. View largeDownload slide Rendering of the pac2TM device. Figure 2. View largeDownload slide Typical workflows of using the pac2 device. Figure 2. View largeDownload slide Typical workflows of using the pac2 device. The decision support algorithms embedded in pac2 reference a built-in pharmaceutical knowledge base to provide immediate warning when doses are outside the recommended safe therapeutic range. The device issues 2 types of warnings, shown in Figure 3: (1) passive, noninterruptive reminders for potential dosing issues with a high likelihood of injury and (2) interruptive alerts when severe dosing errors are detected. While no action is required in response to the first type, users must either override the second type or change the medication dosage. Figure 3. View largeDownload slide The two types of warnings provided by the pac2 device—noninterruptive reminders (left) and Interruptive alerts (right). Figure 3. View largeDownload slide The two types of warnings provided by the pac2 device—noninterruptive reminders (left) and Interruptive alerts (right). In addition to safety warnings, the device will also display essential medication properties such as safe intravenous push rate, route, reversal, and monitoring information. As such, pac2 may be used as an “incidental learning” tool for skill training and retraining, and to familiarize nurses with new medications and recommended safe administration practices. In a laboratory-based evaluation study, pac2 use was associated with significantly improved drug calculation performance by pediatric and critical care nurses in simulated scenarios.31 Empirical study sites and research protocol Figure 4 presents an overview of the study sites and research activities. Briefly, in order to evaluate end-user perceptions and pac2’s effect in the field, we deployed the devices in the Neonatal and Pediatric Intensive Care Units (NICU and PICU) of 2 US health systems (referred to as “HS-A” and “HS-B”). Registered nurses working in one of these care areas during the study period, and involved in direct patient care, were eligible to participate. The devices deployed in the 4 care areas were preloaded with point-of-use drugs and ranges and critical administration information consistent with each hospital’s formulary and protocols. Figure 4. View largeDownload slide Overview of sites and activities (*Lighter rectangles denote qualitative data). Figure 4. View largeDownload slide Overview of sites and activities (*Lighter rectangles denote qualitative data). The empirical study was conducted over a 12-month period across 3 stages: (1) a 3-month “pre-implementation” technology preparation and staff training period prior to deploying 72 pac2 devices in March 2012; (2) a 3-month “disruption” period immediately following deployment, during which participants learned and adapted to the technology; and (3) a 6-month “postimplementation” period after stable usage was achieved. This study’s research protocol was approved by the participating organizations’ Institutional Review Boards. Data sources and measures The goal of our mixed methods approach was to collect qualitative data that would help us to both interpret our quantitative findings and discover unique insights that may not be captured by our quantitative methods. User acceptance and satisfaction We used the unified theory of acceptance and use of technology (UTAUT) to guide data collection for assessing nurses’ technology acceptance.42–44 UTAUT proposes that 4 constructs, performance expectancy, effort expectancy, social influence, and facilitating conditions, are the most influential factors in determining technology acceptance. The use of this well-established theory, in combination with our mixed methods approach, provides a rich, user-centered assessment of the technology’s utility and usability in the field. Specifically, we measured potential moderating variables through a pre-implementation demographics and computer literacy survey. The main UTAUT constructs were assessed pre- and postimplementation through 2 complementary, and commonly used, qualitative data collection methods—ethnographic observations and semistructured interviews. In addition, given that numerous studies have found effort expectancy, also referred to as perceived ease of use or usability, to be particularly important in healthcare settings,41,45 we also quantitatively measured participants’ usability satisfaction through a postimplementation survey. Finally, pac2 use was assessed through automatically logged user interaction data (ie device logs). Demographics and computer literacy We administered a pre-implementation survey to gather participant demographics, such as age and gender, and computer literacy. The latter part of this survey was based on Cork’s instrument for measuring medical professionals’ use of, knowledge about, and attitudes toward computers.46 Cork’s instrument has been validated and used widely in technology evaluation studies. Semistructured interviews The goal of the semistructured interviews was to solicit in-depth elaboration of work processes, as well as to measure UTAUT’s main dimensions. Towards this end, research assistants (RAs) conducted a total of 40 in-person interviews with participants—20 pre-implementation and 20 postimplementation. The pre- and postinterviews were conducted in a similar manner, with the exception that, in the former, the interviewer briefly demonstrated pac2 and answered any interviewee questions. The interview protocol was developed based on UTAUT.42–44 Each interview lasted approximately 15 min. All interviews were recorded and transcribed for subsequent qualitative analysis. Ethnographic observations Two RAs conducted a total of 200 h of ethnographic observations in the 4 care areas.47 The observation hours were evenly split between pre- and postimplementation phases, and occurred during both day (7 am–7 pm) and night (7 pm–7 am) shifts. Data were collected in the form of field notes. During all observations, participants’ routine work activities were noted, with a particular focus on the processes of preparing, calculating, and administering intravenous medications. In the postimplementation period, we were especially interested in: (1) how pac2 was used; (2) reactions to safety alerts; and (3) any unanticipated technology use behaviors (e.g., unintended uses of pac2). Usability satisfaction We conducted a postimplementation survey to assess participant satisfaction with pac2’s usability. The survey was based on a commonly used instrument—the IBM Computer Usability Satisfaction Questionnaire (CUSQ).48 This instrument evaluates overall end-user satisfaction with pac2’s usability, as well as along the dimensions of system usefulness, information quality, and interface quality, on a 7-point Likert-scale where 7 represented most satisfied and 1 least.48 Effect of device The device’s effect was primarily evaluated quantitatively by assessing pre- and postimplementation (i) workload associated with dose calculation and verification tasks and (ii) potential and actual medication errors based on pac2 device logs and voluntary reports. In addition, our qualitative data sources (described above) also provided important insights. Task workload We used the NASA-TLX instrument49,50 to measure participants’ perceived workload associated with dose calculation and verification tasks (referred to as dose calculation below) both pre- and postimplementation. Although it had been used widely in domains such as aviation, and is validated in healthcare settings,51 prior to this study it had rarely been employed to assess the effect of implementing a new healthcare technology in the field. The instrument takes only minutes to complete. In the first of 2 sections, participants individually rated tasks associated with performing dose calculations along 6 dimensions (mental demand, physical demand, temporal demand, performance, effort, and frustration level). In the second, participants selected which of the demands was more important to the experience of workload by completing 15-pairwise comparisons. In other words, overall workload and each dimension’s relative contribution were both measured. Medication errors Finally, we used 2 different data sources to assess potential and actual medication errors. First, we analyzed the device logs; specifically, whether calculations performed using the device were above or below the safe therapeutic range—an indicator of potential pac2-identified dosing errors (prescription or administration). Second, we evaluated the device’s effectiveness by analyzing the voluntary reports of actual medication errors that are collected on an ongoing basis as part of the hospitals’ routine quality improvement practices in June–November 2011 (pre) and July–December 2012 (post). These reports were reviewed, and administration errors were identified. All errors were categorized according to the index developed by the National Coordinating Council for Medication Error Reporting and Prevention.52 Analyses Qualitative Qualitative data collected through the study, including field notes and interview transcripts, were codified, categorized, and analyzed using the constant comparison method.53,54 We employed the framework analysis approach that allows for the inclusion of both a priori and emergent concepts.55,56 We identified emergent themes corresponding to the core constructs of UTAUT to understand the perceived barriers to and facilitators of pac2 use. Quantitative Survey and medication error data were quantitatively analyzed using descriptive statistics. Separate analyses were conducted for the 2 participating hospitals and then combined across sites. When appropriate and possible, we tested for statistical differences between the 2 hospitals or between pre- and post-stages. Specifically, we assessed whether participants from the 2 hospitals differed in demographics, computer literacy, or usability satisfaction using 2 sample Mann–Whitney U tests. In addition, paired t-tests for the difference between the pre-/post-test measures were conducted for each of the 6 dimensions of the NASA-TLX instrument and the global workload score. We applied the Bonferroni correction to account for the number of t-tests. We also analyzed the weighted ratings with a 2 (pre-test, post-test) by 2 (6 dimensions of TLX) repeated-measures ANOVA (as described by Hart and Staveland49). Two participants had missing weights due to incomplete paired comparisons (Section 2 of the NASA-TLX instrument); we addressed this using a common imputation method—replacing the missing weights with the ones most often selected by the other participants. Finally, we used Tukey’s post hoc comparisons of means to identify the differences. RESULTS Participant demographics and computer literacy Table 1 shows the demographics and computer literacy of the 64 nurses who volunteered to participate and who provided valid responses to the pre-implementation survey. The demographics of participants from HS-A and HS-B were comparable. However, on average, nurses at HS-B had more computer experience (3.44 ± 0.65, N = 28 vs 2.85 ± 0.72, N = 36; U = 291, P < .001). Table 1. Demographics and computer literacy assessments Assessment HS-A HS-B Combined Number of participants 36 28 64 Female 36 (100%) 27 (96.4%) 63 (98.4%) Age 32.6 ± 8.9 34.8 ± 8.5 33.6 ± 8.7 Computer experience (1–5)a 2.85 ± 0.72 3.44 ± 0.65 3.11 ± 0.74 Computer knowledge (1–3) 1.73 ± 0.48 1.76 ± 0.40 1.74 ± 0.44 Computer attitude (1–5) 3.89 ± 0.59 3.94 ± 0.46 3.92 ± 0.53  Assessment HS-A HS-B Combined Number of participants 36 28 64 Female 36 (100%) 27 (96.4%) 63 (98.4%) Age 32.6 ± 8.9 34.8 ± 8.5 33.6 ± 8.7 Computer experience (1–5)a 2.85 ± 0.72 3.44 ± 0.65 3.11 ± 0.74 Computer knowledge (1–3) 1.73 ± 0.48 1.76 ± 0.40 1.74 ± 0.44 Computer attitude (1–5) 3.89 ± 0.59 3.94 ± 0.46 3.92 ± 0.53  a Difference between HS-A and HS-B, P < .01. Table 1. Demographics and computer literacy assessments Assessment HS-A HS-B Combined Number of participants 36 28 64 Female 36 (100%) 27 (96.4%) 63 (98.4%) Age 32.6 ± 8.9 34.8 ± 8.5 33.6 ± 8.7 Computer experience (1–5)a 2.85 ± 0.72 3.44 ± 0.65 3.11 ± 0.74 Computer knowledge (1–3) 1.73 ± 0.48 1.76 ± 0.40 1.74 ± 0.44 Computer attitude (1–5) 3.89 ± 0.59 3.94 ± 0.46 3.92 ± 0.53  Assessment HS-A HS-B Combined Number of participants 36 28 64 Female 36 (100%) 27 (96.4%) 63 (98.4%) Age 32.6 ± 8.9 34.8 ± 8.5 33.6 ± 8.7 Computer experience (1–5)a 2.85 ± 0.72 3.44 ± 0.65 3.11 ± 0.74 Computer knowledge (1–3) 1.73 ± 0.48 1.76 ± 0.40 1.74 ± 0.44 Computer attitude (1–5) 3.89 ± 0.59 3.94 ± 0.46 3.92 ± 0.53  a Difference between HS-A and HS-B, P < .01. Not all 64 nurses completed the subsequent study components, and some of them left the study areas before all components were completed. pre-implementation environments and perceptions Overall, pre-implementation observation and interview data indicated that nurses’ use would be contingent upon the device’s convenience, time efficiency, and usability. It also showed that most participants believed that a handheld tool such as pac2 would be useful, but not in every situation. Table 2 summarizes the recurring themes prior to deploying pac2, before nurses had used it in their day-to-day work. Table 2. Recurring themes from pre-implementation ethnographic observations and semistructured interviews Dimension Theme Source Example(s) Effort expectancy Time efficiency Interviews  “The only problem I might foresee is in an emergency situation, it might take too long to use it.” “I could see, just depending on how the device works, I could see it being maybe a cumbersome aspect. Sometimes you just need to get a medication in and stuff to take a minute, 3–5 min depending on how long it takes to program it in, you might just need. That baby might not be able to wait for that.”  Usability Interviews “I think it can [be useful] … depending on how user-friendly the device is because if we’re in a rush, you have a child coding and it’s critical in the situation, depending on how hard the device is to use, it could be more cumbersome and more slowing you down than actually helping.” Learning curve Interviews “Once we get used to them, it’ll be pretty helpful … but at first, just the learning curve that’s going to be a bit of a pain.” Performance expectancy Perceived usefulness Observations “Even though vitamin K is premeasured, the nurse needs to adjust syringe so that the correct amount is administered (ie 0.5 mls).” Interviews  “Yes, I do. I think it will be very useful. Especially if I’m able to have it readily available to calculate and then to reference for any information that I might need on the medication. I think that will be very valuable.” “I think that it’s important and it would be a good tool especially for people that are mathematically challenged or something…”  Conditions of use Observations “Pharmacy precalculates the dosage for each of a patient’s meds, repairs several syringes with the proper dose of these meds, and then puts the syringe is in the med room. The nurse only needs to verify the calculation is correct.” Interviews  “[The device is useful] If it is a very common medication and it is not one that’s prepared by the pharmacy. So for instance, the pharmacy usually prepares antibiotics because they prepare those, like IV drugs, they prepare those under the hood so it’s sterile. If it’s just an intermittently injectable drug like morphine or Ativan or something along those lines or any of the PO drugs that we give, those ones are just in our Pyxis.” “Usually the first time when we give that medication, if it’s unfamiliar, then we would sit down and calculate it out with a calculator. But we usually give the same routine medications.”  Existing alternative tools Observations  “The nurse uses the Pyxis to calculate the dose for 1 of the medications (ie Carofate). The Pyxis gives a printout of the correct dose.” “The nurse calculates what the total parenteral nutrition (TPN) flow rate should be. She makes these calculations using her cell phone calculator.”  Interviews  “[We do the calculation] by hand or on the Pyxis there is a calculator. All of us have our phones. There’s calculators in the unit to check.” “The NeoFax is the one we use up here all the time. And like I said, if we have any questions about a medication, we go to the NeoFax and it tells you everything from, you can run that with your TPN or it needs to run by itself or it tells you compatibilities and it tells you what to watch for.”  Existing safety assurance procedures Observations “The nurse reprograms the pump for the patient’s morphine. He requests that the charge nurse watch him while he reprograms pump to make sure that he doesn’t make a mistake. The nurse mentions that this is part of normal protocol.” Interviews “I think that dosage calculations are always vulnerable to errors, but we do double-check all the meds that we administer in this institution with another nurse and they do the same form of calculation on their own. We verify that we’re getting the same answer.” Facilitating conditions Accessibility Interviews “The only thing I see is like availability because like if we each had one and I have it in my pocket then probably it would be just as easy to pull out as my phone but if I’m having to go get it because somebody else is using it probably in a lot of situations we don’t have the time to do that.” Battery life Interviews “So we did the class the other day, some of the batteries were dead.”  Dimension Theme Source Example(s) Effort expectancy Time efficiency Interviews  “The only problem I might foresee is in an emergency situation, it might take too long to use it.” “I could see, just depending on how the device works, I could see it being maybe a cumbersome aspect. Sometimes you just need to get a medication in and stuff to take a minute, 3–5 min depending on how long it takes to program it in, you might just need. That baby might not be able to wait for that.”  Usability Interviews “I think it can [be useful] … depending on how user-friendly the device is because if we’re in a rush, you have a child coding and it’s critical in the situation, depending on how hard the device is to use, it could be more cumbersome and more slowing you down than actually helping.” Learning curve Interviews “Once we get used to them, it’ll be pretty helpful … but at first, just the learning curve that’s going to be a bit of a pain.” Performance expectancy Perceived usefulness Observations “Even though vitamin K is premeasured, the nurse needs to adjust syringe so that the correct amount is administered (ie 0.5 mls).” Interviews  “Yes, I do. I think it will be very useful. Especially if I’m able to have it readily available to calculate and then to reference for any information that I might need on the medication. I think that will be very valuable.” “I think that it’s important and it would be a good tool especially for people that are mathematically challenged or something…”  Conditions of use Observations “Pharmacy precalculates the dosage for each of a patient’s meds, repairs several syringes with the proper dose of these meds, and then puts the syringe is in the med room. The nurse only needs to verify the calculation is correct.” Interviews  “[The device is useful] If it is a very common medication and it is not one that’s prepared by the pharmacy. So for instance, the pharmacy usually prepares antibiotics because they prepare those, like IV drugs, they prepare those under the hood so it’s sterile. If it’s just an intermittently injectable drug like morphine or Ativan or something along those lines or any of the PO drugs that we give, those ones are just in our Pyxis.” “Usually the first time when we give that medication, if it’s unfamiliar, then we would sit down and calculate it out with a calculator. But we usually give the same routine medications.”  Existing alternative tools Observations  “The nurse uses the Pyxis to calculate the dose for 1 of the medications (ie Carofate). The Pyxis gives a printout of the correct dose.” “The nurse calculates what the total parenteral nutrition (TPN) flow rate should be. She makes these calculations using her cell phone calculator.”  Interviews  “[We do the calculation] by hand or on the Pyxis there is a calculator. All of us have our phones. There’s calculators in the unit to check.” “The NeoFax is the one we use up here all the time. And like I said, if we have any questions about a medication, we go to the NeoFax and it tells you everything from, you can run that with your TPN or it needs to run by itself or it tells you compatibilities and it tells you what to watch for.”  Existing safety assurance procedures Observations “The nurse reprograms the pump for the patient’s morphine. He requests that the charge nurse watch him while he reprograms pump to make sure that he doesn’t make a mistake. The nurse mentions that this is part of normal protocol.” Interviews “I think that dosage calculations are always vulnerable to errors, but we do double-check all the meds that we administer in this institution with another nurse and they do the same form of calculation on their own. We verify that we’re getting the same answer.” Facilitating conditions Accessibility Interviews “The only thing I see is like availability because like if we each had one and I have it in my pocket then probably it would be just as easy to pull out as my phone but if I’m having to go get it because somebody else is using it probably in a lot of situations we don’t have the time to do that.” Battery life Interviews “So we did the class the other day, some of the batteries were dead.”  Table 2. Recurring themes from pre-implementation ethnographic observations and semistructured interviews Dimension Theme Source Example(s) Effort expectancy Time efficiency Interviews  “The only problem I might foresee is in an emergency situation, it might take too long to use it.” “I could see, just depending on how the device works, I could see it being maybe a cumbersome aspect. Sometimes you just need to get a medication in and stuff to take a minute, 3–5 min depending on how long it takes to program it in, you might just need. That baby might not be able to wait for that.”  Usability Interviews “I think it can [be useful] … depending on how user-friendly the device is because if we’re in a rush, you have a child coding and it’s critical in the situation, depending on how hard the device is to use, it could be more cumbersome and more slowing you down than actually helping.” Learning curve Interviews “Once we get used to them, it’ll be pretty helpful … but at first, just the learning curve that’s going to be a bit of a pain.” Performance expectancy Perceived usefulness Observations “Even though vitamin K is premeasured, the nurse needs to adjust syringe so that the correct amount is administered (ie 0.5 mls).” Interviews  “Yes, I do. I think it will be very useful. Especially if I’m able to have it readily available to calculate and then to reference for any information that I might need on the medication. I think that will be very valuable.” “I think that it’s important and it would be a good tool especially for people that are mathematically challenged or something…”  Conditions of use Observations “Pharmacy precalculates the dosage for each of a patient’s meds, repairs several syringes with the proper dose of these meds, and then puts the syringe is in the med room. The nurse only needs to verify the calculation is correct.” Interviews  “[The device is useful] If it is a very common medication and it is not one that’s prepared by the pharmacy. So for instance, the pharmacy usually prepares antibiotics because they prepare those, like IV drugs, they prepare those under the hood so it’s sterile. If it’s just an intermittently injectable drug like morphine or Ativan or something along those lines or any of the PO drugs that we give, those ones are just in our Pyxis.” “Usually the first time when we give that medication, if it’s unfamiliar, then we would sit down and calculate it out with a calculator. But we usually give the same routine medications.”  Existing alternative tools Observations  “The nurse uses the Pyxis to calculate the dose for 1 of the medications (ie Carofate). The Pyxis gives a printout of the correct dose.” “The nurse calculates what the total parenteral nutrition (TPN) flow rate should be. She makes these calculations using her cell phone calculator.”  Interviews  “[We do the calculation] by hand or on the Pyxis there is a calculator. All of us have our phones. There’s calculators in the unit to check.” “The NeoFax is the one we use up here all the time. And like I said, if we have any questions about a medication, we go to the NeoFax and it tells you everything from, you can run that with your TPN or it needs to run by itself or it tells you compatibilities and it tells you what to watch for.”  Existing safety assurance procedures Observations “The nurse reprograms the pump for the patient’s morphine. He requests that the charge nurse watch him while he reprograms pump to make sure that he doesn’t make a mistake. The nurse mentions that this is part of normal protocol.” Interviews “I think that dosage calculations are always vulnerable to errors, but we do double-check all the meds that we administer in this institution with another nurse and they do the same form of calculation on their own. We verify that we’re getting the same answer.” Facilitating conditions Accessibility Interviews “The only thing I see is like availability because like if we each had one and I have it in my pocket then probably it would be just as easy to pull out as my phone but if I’m having to go get it because somebody else is using it probably in a lot of situations we don’t have the time to do that.” Battery life Interviews “So we did the class the other day, some of the batteries were dead.”  Dimension Theme Source Example(s) Effort expectancy Time efficiency Interviews  “The only problem I might foresee is in an emergency situation, it might take too long to use it.” “I could see, just depending on how the device works, I could see it being maybe a cumbersome aspect. Sometimes you just need to get a medication in and stuff to take a minute, 3–5 min depending on how long it takes to program it in, you might just need. That baby might not be able to wait for that.”  Usability Interviews “I think it can [be useful] … depending on how user-friendly the device is because if we’re in a rush, you have a child coding and it’s critical in the situation, depending on how hard the device is to use, it could be more cumbersome and more slowing you down than actually helping.” Learning curve Interviews “Once we get used to them, it’ll be pretty helpful … but at first, just the learning curve that’s going to be a bit of a pain.” Performance expectancy Perceived usefulness Observations “Even though vitamin K is premeasured, the nurse needs to adjust syringe so that the correct amount is administered (ie 0.5 mls).” Interviews  “Yes, I do. I think it will be very useful. Especially if I’m able to have it readily available to calculate and then to reference for any information that I might need on the medication. I think that will be very valuable.” “I think that it’s important and it would be a good tool especially for people that are mathematically challenged or something…”  Conditions of use Observations “Pharmacy precalculates the dosage for each of a patient’s meds, repairs several syringes with the proper dose of these meds, and then puts the syringe is in the med room. The nurse only needs to verify the calculation is correct.” Interviews  “[The device is useful] If it is a very common medication and it is not one that’s prepared by the pharmacy. So for instance, the pharmacy usually prepares antibiotics because they prepare those, like IV drugs, they prepare those under the hood so it’s sterile. If it’s just an intermittently injectable drug like morphine or Ativan or something along those lines or any of the PO drugs that we give, those ones are just in our Pyxis.” “Usually the first time when we give that medication, if it’s unfamiliar, then we would sit down and calculate it out with a calculator. But we usually give the same routine medications.”  Existing alternative tools Observations  “The nurse uses the Pyxis to calculate the dose for 1 of the medications (ie Carofate). The Pyxis gives a printout of the correct dose.” “The nurse calculates what the total parenteral nutrition (TPN) flow rate should be. She makes these calculations using her cell phone calculator.”  Interviews  “[We do the calculation] by hand or on the Pyxis there is a calculator. All of us have our phones. There’s calculators in the unit to check.” “The NeoFax is the one we use up here all the time. And like I said, if we have any questions about a medication, we go to the NeoFax and it tells you everything from, you can run that with your TPN or it needs to run by itself or it tells you compatibilities and it tells you what to watch for.”  Existing safety assurance procedures Observations “The nurse reprograms the pump for the patient’s morphine. He requests that the charge nurse watch him while he reprograms pump to make sure that he doesn’t make a mistake. The nurse mentions that this is part of normal protocol.” Interviews “I think that dosage calculations are always vulnerable to errors, but we do double-check all the meds that we administer in this institution with another nurse and they do the same form of calculation on their own. We verify that we’re getting the same answer.” Facilitating conditions Accessibility Interviews “The only thing I see is like availability because like if we each had one and I have it in my pocket then probably it would be just as easy to pull out as my phone but if I’m having to go get it because somebody else is using it probably in a lot of situations we don’t have the time to do that.” Battery life Interviews “So we did the class the other day, some of the batteries were dead.”  Effort expectancy Three themes emerged in UTAUT’s effort expectancy dimension (1) time efficiency, (2) usability, and (3) learning curve. Many nurses raised concerns regarding the device’s time efficiency. They emphasized that in many situations they need to administer medications as quickly as possible and might not have time to use the device. One nurse explained: “When there is a physician at the bedside yelling orders at me with no patience, I don’t know that there will be tolerance for, ‘Let me check this machine and see if you’re telling me the right dose for fentanyl.’ I don’t know that there because it’ an emergent situation, we have to get it done and it needs to be fast. So if I say to Dr. XXX, ‘Let me look that up first.’ I’m pretty sure the answer is going to be, ‘Just get it!’” Although it did not emerge as a major theme, this quote also highlights the fact that nurses are not working in a vacuum—organizational culture and social dynamics may also be a factor in their technology acceptance. Related to the feedback on time efficiency, participants placed great emphasis on the device’s usability, that is, its ease of use. They stated that they would only use the device if “it is user-friendly” and would not “slow things down.” They also acknowledged the learning curve, especially for those that were not technologically savvy. For example, a participant explained, “Once we get used to them, it’ll be pretty helpful … but at first, just the learning curve that’s going to be a bit of a pain.” Performance expectancy The following themes emerged in UTAUT’s performance expectancy dimension (1) usefulness, (2) conditions of use, (3) existing tools, and (4) existing approaches. During the pre-implementation observations, there were numerous instances when nurses needed to perform bedside dose calculations, which were often completed in their head or with conventional calculators. Indeed, a majority of the nurses interviewed believed that a handheld DST would be very valuable. However, the RAs also observed several common situations where nurses either (i) did not need to perform calculations (e.g., pharmacy prepared the medication) or (ii) did not appear to need a device to perform calculations (e.g., nurse had the dose memorized). Similarly, interview participants named conditions when the device might be helpful, for instance, “[it would be useful] if it is a very uncommon medication and it is not one that’s prepared by the pharmacy.” In addition, through both observations and interviews, we identified several commonly used tools that already existed in the study environments. For instance, some medications were obtained from the Pyxis MedStation Unit, which was equipped with a built-in medication dose calculator. In an interview, one nurse explained, “our Pyxis also has a calculation device on it … down in the bottom it has a button that says, ‘Dose calculation’ or something, and you can hit it and you can put in your dose and it will tell you how much to give.” However, the RAs observed that using this tool sometimes required leaving the patient and going to a medication room, and that the built-in calculator was not always employed. During observations, it was also noted that participants utilized other reference tools such as Micromedex NeoFax (a book) and various online drug databases. Further, most nurses had a smartphone and used either the built-in calculator or professional medication applications for dose calculations. One nurse specifically commented that “there are apps on the iPhone that do similar things in a much easier manner (e.g., Epocrates or PediStat).” That said, the RAs observed that none of these smartphone apps provided warning of doses that were outside the recommended safe therapeutic range. Finally, all 4 areas had double-check safety assurance procedures in place—dose calculations were often witnessed, verified, and signed off by another nurse. Although all nurses recognized that medication administration processes were subject to errors, they were confident in their existing procedures: “I think they [dose calculation errors] can happen, obviously, but I think that we got a pretty good system as far as checking and pull proofing each other.” Facilitating conditions Prior to implementation, participants identified 2 main logistical concerns (1) availability and (2) battery life. One nurse, who was concerned about the first, explained, “… if we each had one [pac2] and I have it in my pocket then probably it would be just as easy to pull out as my phone but if I’m having to go get it because somebody else is using it probably in a lot of situations we don’t have the time to do that.” The second concern arose because pac2’s batteries were dead during a training session. Postimplementation acceptance and satisfaction While many nurses were positive about pac2’s usefulness, our results revealed that they did not actually use the device or did not use it routinely. In the 6-month postimplementation period, only 1202 calculations were performed on the device, with 69% of these completed by HS-A nurses. One participant stated, “I try to use the little green machines but I must admit that I don’t use them very often.” Instead, many continued to calculate doses in their head or use conventional calculators or smartphones. Overall, our data showed that participants were only modestly satisfied with pac2’s usability, and that the device did not seem to meet the requirements identified pre-implementation (e.g., convenient). Additionally, although participants found pac2 useful for complex calculations, they did not find it useful when administering routine medications or when their time was limited (eg emergencies). Usability satisfaction The IBM CUSQ results are reported in Table 3. Participants were modestly satisfied with the device’s overall usability (average: 5.09 ± 1.36 on the 7-point scale, where higher scores indicate higher satisfaction). In terms of the 3 usability dimensions assessed, on average, interface quality was rated lowest (4.91 ± 1.56), followed by system usefulness (4.99 ± 1.35) and information quality (5.34 ± 1.36). As one might expect based on usage, HS-A participants provided higher mean ratings of system usefulness compared to HS-B nurses (5.43 ± 1.01, N = 28 vs 4.27 ± 1.54, N = 17; U = 349.5, P < .01). Table 3. Results of the IBM system usability satisfaction questionnaire Construct HS-A (N = 28) HS-B (N = 17) Combined (N = 45) System usefulness (1–7)a 5.43 ± 1.01 4.27 ± 1.54 4.99 ± 1.35 Information quality (1–7) 5.60 ± 1.09 4.92 ± 1.67 5.34 ± 1.36 Interface quality (1–7) 5.18 ± 1.27 4.44 ± 1.93 4.91 ± 1.56 Overall satisfaction (1–7) 5.45 ± 1.04 4.49 ± 1.62 5.09 ± 1.36  Construct HS-A (N = 28) HS-B (N = 17) Combined (N = 45) System usefulness (1–7)a 5.43 ± 1.01 4.27 ± 1.54 4.99 ± 1.35 Information quality (1–7) 5.60 ± 1.09 4.92 ± 1.67 5.34 ± 1.36 Interface quality (1–7) 5.18 ± 1.27 4.44 ± 1.93 4.91 ± 1.56 Overall satisfaction (1–7) 5.45 ± 1.04 4.49 ± 1.62 5.09 ± 1.36  a Difference between HS-A and HS-B, P < .01. Table 3. Results of the IBM system usability satisfaction questionnaire Construct HS-A (N = 28) HS-B (N = 17) Combined (N = 45) System usefulness (1–7)a 5.43 ± 1.01 4.27 ± 1.54 4.99 ± 1.35 Information quality (1–7) 5.60 ± 1.09 4.92 ± 1.67 5.34 ± 1.36 Interface quality (1–7) 5.18 ± 1.27 4.44 ± 1.93 4.91 ± 1.56 Overall satisfaction (1–7) 5.45 ± 1.04 4.49 ± 1.62 5.09 ± 1.36  Construct HS-A (N = 28) HS-B (N = 17) Combined (N = 45) System usefulness (1–7)a 5.43 ± 1.01 4.27 ± 1.54 4.99 ± 1.35 Information quality (1–7) 5.60 ± 1.09 4.92 ± 1.67 5.34 ± 1.36 Interface quality (1–7) 5.18 ± 1.27 4.44 ± 1.93 4.91 ± 1.56 Overall satisfaction (1–7) 5.45 ± 1.04 4.49 ± 1.62 5.09 ± 1.36  a Difference between HS-A and HS-B, P < .01. Effort expectancy The qualitative results offer insights into issues that may have contributed to the moderate usability scores. These are summarized in Table 4, along with other key findings from the postimplementation observations and interviews. Some nurses perceived pac2’s interface to be too complex and outdated, especially compared to other touchscreen devices. One nurse stated that the device “has way too many buttons,” and that “it is like a dinosaur … it feels like it might have come out in 1987… there are much easier ways to do these things with apps such as the iPhone apps.” Furthermore, many participants commented that pac2 was difficult to learn and use. Finally, although the mean rating for information quality was the highest of the 3 dimensions, several nurses mentioned that they failed to find frequently used medications or doses in pac2’s drug databases. This led to the belief that the databases shipped with the device were not comprehensive enough. One participant commented: “I don’t know if it’s more adult-based or what the difference was, but it just didn’t have pediatric, our stuff I guess.” Table 4. Recurring themes from postimplementation observations and semistructured interviews Dimension Theme Source Example(s) Effort expectancy Usability Observations “… [Two nurses] say they do not use the ‘green machine’ because there are ‘way too many buttons’ and ‘it takes way longer than a calculator’.” Interviews “I guess I wasn’t as comfortable using the keypad and so I would use the stylus and even the stylus wasn’t as quick I guess.” Issues related to medication databases used by pac2 Observations  “Nurse 1 tries to double-check the dose using the pac2… However, digoxin per oral (PO) did not come up so she does it in her head.” “[The nurse] Asks another nurse where to find something on PAC2. The nurse is unable to find a med in the PAC2.”  Time efficiency Interviews  “So while it does give you a lot of redundancy and makes sure that you know what is going on with that baby and that medication, it’s almost overkill in that it’s taking so much time…” “Now, there were a couple of times where I was trying real hard to use it and I would check every med on my MAR with your device and tonight I would not have had time to do that.”  Performance expectancy Perceived usefulness Observations “Nurses (at least 2) report machine is most helpful w/ drips. One nurse used PAC2 in addition to hand calculating doses. Another nurse reports doing the same initially but using the PAC2 in place of hand calculations after having gotten used to using the PAC2.” Interviews  “Instead of me handwriting a system out while I can enter my dose, concentration, whatnot, into there, it will print all that out on the sticker, and I can attach to my drugs at that point…” “I liked it for drips. I don’t know that it really helps that much with our basic medications that we give. I don’t think it makes it them any quicker or decreases the errors, but I do like it for drips.”  Interviews  “It’s always in default to the one that you don’t want 99% of the time on that milligrams per kilo per unit. It’d be nice if it didn’t default to that one since that’s a minority of the ones that are chosen…” “We basically are weaning the patient because the baby is growing and getting bigger, and the dose is staying the same… So, it will go, ‘alarm, alarm, alarm!’ You know, ‘dose is too low.’ And I'm like ‘So what?’”  Conditions of use Observations “The nurse double-checks a medication for another nurse. She has the dose already memorized so she tells me that she doesn’t need to calculate it.” Observations “PAC2 was never used. A calculator was used. The situation was rushed and the nurse had to ask for the pharmacist’s help calculating the second med.” Interviews  “So in the future, if there are times where I feel like there’s no reason I would need to use it, again, referring back to say, Tylenol, as an easy dose, that’s something very easily calculated. So it’s quick to do in your head, or just go on a calculator, and I don’t need to use a complicated system for it.” “[The device is most useful] for drugs that are uncommon to us, that we don’t use that often, is what I found the most beneficial. Otherwise, drugs that we use, sometimes every day, again, they’re just second nature to us and we already know what we’re doing with those.”  Existing alternative tools Observations “Med calculations done w/ calculator.” Interviews  “No, I use other … Like I said, I have other programs that I have on my phone that are pretty much similar … It’s all on my phone.” “It depends on what it is and where we get. Like if it’s a narcotic, then we can use the calculator off the Pyxis system…”  Existing safety assurance procedures Observations “The nurse completes the verbal double check procedure [with another nurse].” Interviews “I feel like it is vulnerable to errors. I feel like we do have a redundancy in checking it with another nurse up here and that’s, there’s two people looking at the same thing and one normally catches it if the other one’s made a mistake.” Facilitating conditions Logistics issues Observations  “They do the calculations in their head. The nurse tries to use the device but it is not charged…” “Finishes calculation with PAC2. The PAC2 doesn’t print. The nurse reports this to be a common problem.”   Dimension Theme Source Example(s) Effort expectancy Usability Observations “… [Two nurses] say they do not use the ‘green machine’ because there are ‘way too many buttons’ and ‘it takes way longer than a calculator’.” Interviews “I guess I wasn’t as comfortable using the keypad and so I would use the stylus and even the stylus wasn’t as quick I guess.” Issues related to medication databases used by pac2 Observations  “Nurse 1 tries to double-check the dose using the pac2… However, digoxin per oral (PO) did not come up so she does it in her head.” “[The nurse] Asks another nurse where to find something on PAC2. The nurse is unable to find a med in the PAC2.”  Time efficiency Interviews  “So while it does give you a lot of redundancy and makes sure that you know what is going on with that baby and that medication, it’s almost overkill in that it’s taking so much time…” “Now, there were a couple of times where I was trying real hard to use it and I would check every med on my MAR with your device and tonight I would not have had time to do that.”  Performance expectancy Perceived usefulness Observations “Nurses (at least 2) report machine is most helpful w/ drips. One nurse used PAC2 in addition to hand calculating doses. Another nurse reports doing the same initially but using the PAC2 in place of hand calculations after having gotten used to using the PAC2.” Interviews  “Instead of me handwriting a system out while I can enter my dose, concentration, whatnot, into there, it will print all that out on the sticker, and I can attach to my drugs at that point…” “I liked it for drips. I don’t know that it really helps that much with our basic medications that we give. I don’t think it makes it them any quicker or decreases the errors, but I do like it for drips.”  Interviews  “It’s always in default to the one that you don’t want 99% of the time on that milligrams per kilo per unit. It’d be nice if it didn’t default to that one since that’s a minority of the ones that are chosen…” “We basically are weaning the patient because the baby is growing and getting bigger, and the dose is staying the same… So, it will go, ‘alarm, alarm, alarm!’ You know, ‘dose is too low.’ And I'm like ‘So what?’”  Conditions of use Observations “The nurse double-checks a medication for another nurse. She has the dose already memorized so she tells me that she doesn’t need to calculate it.” Observations “PAC2 was never used. A calculator was used. The situation was rushed and the nurse had to ask for the pharmacist’s help calculating the second med.” Interviews  “So in the future, if there are times where I feel like there’s no reason I would need to use it, again, referring back to say, Tylenol, as an easy dose, that’s something very easily calculated. So it’s quick to do in your head, or just go on a calculator, and I don’t need to use a complicated system for it.” “[The device is most useful] for drugs that are uncommon to us, that we don’t use that often, is what I found the most beneficial. Otherwise, drugs that we use, sometimes every day, again, they’re just second nature to us and we already know what we’re doing with those.”  Existing alternative tools Observations “Med calculations done w/ calculator.” Interviews  “No, I use other … Like I said, I have other programs that I have on my phone that are pretty much similar … It’s all on my phone.” “It depends on what it is and where we get. Like if it’s a narcotic, then we can use the calculator off the Pyxis system…”  Existing safety assurance procedures Observations “The nurse completes the verbal double check procedure [with another nurse].” Interviews “I feel like it is vulnerable to errors. I feel like we do have a redundancy in checking it with another nurse up here and that’s, there’s two people looking at the same thing and one normally catches it if the other one’s made a mistake.” Facilitating conditions Logistics issues Observations  “They do the calculations in their head. The nurse tries to use the device but it is not charged…” “Finishes calculation with PAC2. The PAC2 doesn’t print. The nurse reports this to be a common problem.”   Table 4. Recurring themes from postimplementation observations and semistructured interviews Dimension Theme Source Example(s) Effort expectancy Usability Observations “… [Two nurses] say they do not use the ‘green machine’ because there are ‘way too many buttons’ and ‘it takes way longer than a calculator’.” Interviews “I guess I wasn’t as comfortable using the keypad and so I would use the stylus and even the stylus wasn’t as quick I guess.” Issues related to medication databases used by pac2 Observations  “Nurse 1 tries to double-check the dose using the pac2… However, digoxin per oral (PO) did not come up so she does it in her head.” “[The nurse] Asks another nurse where to find something on PAC2. The nurse is unable to find a med in the PAC2.”  Time efficiency Interviews  “So while it does give you a lot of redundancy and makes sure that you know what is going on with that baby and that medication, it’s almost overkill in that it’s taking so much time…” “Now, there were a couple of times where I was trying real hard to use it and I would check every med on my MAR with your device and tonight I would not have had time to do that.”  Performance expectancy Perceived usefulness Observations “Nurses (at least 2) report machine is most helpful w/ drips. One nurse used PAC2 in addition to hand calculating doses. Another nurse reports doing the same initially but using the PAC2 in place of hand calculations after having gotten used to using the PAC2.” Interviews  “Instead of me handwriting a system out while I can enter my dose, concentration, whatnot, into there, it will print all that out on the sticker, and I can attach to my drugs at that point…” “I liked it for drips. I don’t know that it really helps that much with our basic medications that we give. I don’t think it makes it them any quicker or decreases the errors, but I do like it for drips.”  Interviews  “It’s always in default to the one that you don’t want 99% of the time on that milligrams per kilo per unit. It’d be nice if it didn’t default to that one since that’s a minority of the ones that are chosen…” “We basically are weaning the patient because the baby is growing and getting bigger, and the dose is staying the same… So, it will go, ‘alarm, alarm, alarm!’ You know, ‘dose is too low.’ And I'm like ‘So what?’”  Conditions of use Observations “The nurse double-checks a medication for another nurse. She has the dose already memorized so she tells me that she doesn’t need to calculate it.” Observations “PAC2 was never used. A calculator was used. The situation was rushed and the nurse had to ask for the pharmacist’s help calculating the second med.” Interviews  “So in the future, if there are times where I feel like there’s no reason I would need to use it, again, referring back to say, Tylenol, as an easy dose, that’s something very easily calculated. So it’s quick to do in your head, or just go on a calculator, and I don’t need to use a complicated system for it.” “[The device is most useful] for drugs that are uncommon to us, that we don’t use that often, is what I found the most beneficial. Otherwise, drugs that we use, sometimes every day, again, they’re just second nature to us and we already know what we’re doing with those.”  Existing alternative tools Observations “Med calculations done w/ calculator.” Interviews  “No, I use other … Like I said, I have other programs that I have on my phone that are pretty much similar … It’s all on my phone.” “It depends on what it is and where we get. Like if it’s a narcotic, then we can use the calculator off the Pyxis system…”  Existing safety assurance procedures Observations “The nurse completes the verbal double check procedure [with another nurse].” Interviews “I feel like it is vulnerable to errors. I feel like we do have a redundancy in checking it with another nurse up here and that’s, there’s two people looking at the same thing and one normally catches it if the other one’s made a mistake.” Facilitating conditions Logistics issues Observations  “They do the calculations in their head. The nurse tries to use the device but it is not charged…” “Finishes calculation with PAC2. The PAC2 doesn’t print. The nurse reports this to be a common problem.”   Dimension Theme Source Example(s) Effort expectancy Usability Observations “… [Two nurses] say they do not use the ‘green machine’ because there are ‘way too many buttons’ and ‘it takes way longer than a calculator’.” Interviews “I guess I wasn’t as comfortable using the keypad and so I would use the stylus and even the stylus wasn’t as quick I guess.” Issues related to medication databases used by pac2 Observations  “Nurse 1 tries to double-check the dose using the pac2… However, digoxin per oral (PO) did not come up so she does it in her head.” “[The nurse] Asks another nurse where to find something on PAC2. The nurse is unable to find a med in the PAC2.”  Time efficiency Interviews  “So while it does give you a lot of redundancy and makes sure that you know what is going on with that baby and that medication, it’s almost overkill in that it’s taking so much time…” “Now, there were a couple of times where I was trying real hard to use it and I would check every med on my MAR with your device and tonight I would not have had time to do that.”  Performance expectancy Perceived usefulness Observations “Nurses (at least 2) report machine is most helpful w/ drips. One nurse used PAC2 in addition to hand calculating doses. Another nurse reports doing the same initially but using the PAC2 in place of hand calculations after having gotten used to using the PAC2.” Interviews  “Instead of me handwriting a system out while I can enter my dose, concentration, whatnot, into there, it will print all that out on the sticker, and I can attach to my drugs at that point…” “I liked it for drips. I don’t know that it really helps that much with our basic medications that we give. I don’t think it makes it them any quicker or decreases the errors, but I do like it for drips.”  Interviews  “It’s always in default to the one that you don’t want 99% of the time on that milligrams per kilo per unit. It’d be nice if it didn’t default to that one since that’s a minority of the ones that are chosen…” “We basically are weaning the patient because the baby is growing and getting bigger, and the dose is staying the same… So, it will go, ‘alarm, alarm, alarm!’ You know, ‘dose is too low.’ And I'm like ‘So what?’”  Conditions of use Observations “The nurse double-checks a medication for another nurse. She has the dose already memorized so she tells me that she doesn’t need to calculate it.” Observations “PAC2 was never used. A calculator was used. The situation was rushed and the nurse had to ask for the pharmacist’s help calculating the second med.” Interviews  “So in the future, if there are times where I feel like there’s no reason I would need to use it, again, referring back to say, Tylenol, as an easy dose, that’s something very easily calculated. So it’s quick to do in your head, or just go on a calculator, and I don’t need to use a complicated system for it.” “[The device is most useful] for drugs that are uncommon to us, that we don’t use that often, is what I found the most beneficial. Otherwise, drugs that we use, sometimes every day, again, they’re just second nature to us and we already know what we’re doing with those.”  Existing alternative tools Observations “Med calculations done w/ calculator.” Interviews  “No, I use other … Like I said, I have other programs that I have on my phone that are pretty much similar … It’s all on my phone.” “It depends on what it is and where we get. Like if it’s a narcotic, then we can use the calculator off the Pyxis system…”  Existing safety assurance procedures Observations “The nurse completes the verbal double check procedure [with another nurse].” Interviews “I feel like it is vulnerable to errors. I feel like we do have a redundancy in checking it with another nurse up here and that’s, there’s two people looking at the same thing and one normally catches it if the other one’s made a mistake.” Facilitating conditions Logistics issues Observations  “They do the calculations in their head. The nurse tries to use the device but it is not charged…” “Finishes calculation with PAC2. The PAC2 doesn’t print. The nurse reports this to be a common problem.”   Time efficiency continued to be mentioned as the most significant factor in participants’ use decisions: “I understand that it has also other checks besides just the calculation but it’s a lot quicker for me just to calculate it real quick and go ahead and give it if it’s something that I give all the time… It slows me down on meds that I give all the time.” The complexity of the device, especially compared to other tools, seemed to be a primary reason for this. One nurse explained: “It was actually a little bit less useful than I thought that it would be, just in that the amount of information that it asks for. Well I feel that redundancy is good, you’re having to look-up so much more information that actually impedes the process of giving the medication … It can be up to 10 to 12 minutes … I think it slows me down… I can use my phone and get the information this quick or just even call the pharmacy and get the information faster.” This suggests that one of the reasons that nurses may not have perceived pac2 to be time-efficient was because it did indeed require additional work and, thus, time—both to find the necessary patient information and to input that information into pac2. Finally, while participants still saw the potential utility of a portable DST like pac2, they envisioned a system that addresses some of the aforementioned usability issues: “… [a decision support tool] that could be easily integrated into the programs itself installed into, say, a tablet… or again our personal devices….” In other words, nurses wanted a tool that integrates with the other systems that they use and on a device with which they are already familiar. Performance expectancy The recurring themes that emerged along this dimension in the postimplementation phase are very similar to those discovered pre-implementation (1) usefulness, (2) conditions of use, (3) existing tools, and (4) existing approaches. First, nurses found pac2 useful for certain types of calculations, and also appreciated specific functions. For instance, some spoke about how the tool helped them with drips, which tend to involve more complicated calculations (ie volume/time rather than just volume). Additionally, many found pac2’s label printing functions useful: “One thing I found that I’ve enjoyed the most about it, is the printing system that you all tied to it as well.” One participant went on to explain that this feature saved her the step of handwriting the label (ie a process efficiency). On the other hand, some nurses also identified functionality issues that affected their perceptions of usefulness, such as the device failing to account for common situations in which a nonstandard dosage may be desired (eg weaning patients off medications). Second, during both observations and interviews, it emerged that many nurses did not find pac2 useful when they were under time pressure or when they were giving “routine” medications with which they were very familiar. For instance, during an emergency, one RA observed, “PAC2 was never used. A calculator was used. The situation was rushed and the nurse had to ask for the pharmacist’s help calculating the second med.” In another instance where pac2 was not used, it was noted: “[the nurse] mentions that the pac2 device is good for infrequently administered drugs (eg dopamine and dobutamine) but not really helpful for ‘everyday’ medications.” Similarly, an interview participant stated, “If it’s something we don’t give very often, I think it’s helpful. If it’s something that we give all the time, I don’t know that it’s that much error proof with it.” This is likely a major reason for the relatively low usage—since most of the medications administered were routine, and nurses were confident in these calculations, pac2 was considered unnecessary. Third, since pac2’s function overlapped with existing tools, participants tended to use the most convenient tool in a given context. For example, if the nurse needed to get a medication from the Pyxis MedStation Unit, they may have chosen to use its built-in calculator. As one nurse explained, “It depends on what it is and where we get [it].” In addition, smartphone applications continued to be widely used postimplementation. Finally, many participants also continued to be confident in their existing safety procedures, which could have also contributed to the perception that pac2 was unnecessary. Facilitating conditions The RAs noticed several instances when the nurses attempted to use pac2, but were unable—the device was inaccessible in the ward or was out of battery charge. There were also times when the device failed to print. Effect of device Workload Table 5 reports the average patient load pre- and postimplementation, the mean weighted ratings for each of the 6 NASA-TLX dimensions, and the global workload score. HS-B nurses reported a lower patient load postimplementation, t (7) = 2.67, P < .05. However, none of the other pre-/post-test differences were statistically significant after applying the Bonferroni correction for the number of t-tests (N = 14, corrected P < .0035). Furthermore, differences between the pre-/post-test scores were not significant based on the repeated-measures ANOVA test; nor was there an interaction between testing period and NASA-TLX dimension. In other words, the cognitive load for the nurses to perform bedside dose calculation tasks, with or without pac2’s assistance, did not differ significantly. Table 5. Mean weighted ratings and global workload from the NASA-TLX and post/predifferences Construct HS-A (N = 16) HS-B (N = 8) Combined (N = 24) Pre Post Diff. Pre Post Diff. Pre Post Diff. Patient load 2 1.93 −0.067 2.5 1.63 −0.88* 2.17 1.83 −0.35 Mental demand 176.09 152.81 −23.28 194.38 188.13 −6.25 182.19 164.58 −17.60 Physical demand 14.84 12.5 −2.34 3.44 10.94 7.5 11.04 11.98 0.94 Temporal demand 175 180.16 5.16 124.38 98.13 −26.25 158.13 152.81 −5.31 Performance 50 66.41 16.41 35.94 19.06 −16.88 45.31 50.63 5.31 Effort 59.06 111.25 52.19 182.50 68.75 −113.75 100.21 97.08 −3.13 Frustration 81.09 128.13 47.03 62.5 56.56 −5.94 74.90 104.27 29.38 Global score 37.07 44.00 6.92 40.21 29.44 −10.77 38.12 39.14 1.02  Construct HS-A (N = 16) HS-B (N = 8) Combined (N = 24) Pre Post Diff. Pre Post Diff. Pre Post Diff. Patient load 2 1.93 −0.067 2.5 1.63 −0.88* 2.17 1.83 −0.35 Mental demand 176.09 152.81 −23.28 194.38 188.13 −6.25 182.19 164.58 −17.60 Physical demand 14.84 12.5 −2.34 3.44 10.94 7.5 11.04 11.98 0.94 Temporal demand 175 180.16 5.16 124.38 98.13 −26.25 158.13 152.81 −5.31 Performance 50 66.41 16.41 35.94 19.06 −16.88 45.31 50.63 5.31 Effort 59.06 111.25 52.19 182.50 68.75 −113.75 100.21 97.08 −3.13 Frustration 81.09 128.13 47.03 62.5 56.56 −5.94 74.90 104.27 29.38 Global score 37.07 44.00 6.92 40.21 29.44 −10.77 38.12 39.14 1.02  * P < .05. Table 5. Mean weighted ratings and global workload from the NASA-TLX and post/predifferences Construct HS-A (N = 16) HS-B (N = 8) Combined (N = 24) Pre Post Diff. Pre Post Diff. Pre Post Diff. Patient load 2 1.93 −0.067 2.5 1.63 −0.88* 2.17 1.83 −0.35 Mental demand 176.09 152.81 −23.28 194.38 188.13 −6.25 182.19 164.58 −17.60 Physical demand 14.84 12.5 −2.34 3.44 10.94 7.5 11.04 11.98 0.94 Temporal demand 175 180.16 5.16 124.38 98.13 −26.25 158.13 152.81 −5.31 Performance 50 66.41 16.41 35.94 19.06 −16.88 45.31 50.63 5.31 Effort 59.06 111.25 52.19 182.50 68.75 −113.75 100.21 97.08 −3.13 Frustration 81.09 128.13 47.03 62.5 56.56 −5.94 74.90 104.27 29.38 Global score 37.07 44.00 6.92 40.21 29.44 −10.77 38.12 39.14 1.02  Construct HS-A (N = 16) HS-B (N = 8) Combined (N = 24) Pre Post Diff. Pre Post Diff. Pre Post Diff. Patient load 2 1.93 −0.067 2.5 1.63 −0.88* 2.17 1.83 −0.35 Mental demand 176.09 152.81 −23.28 194.38 188.13 −6.25 182.19 164.58 −17.60 Physical demand 14.84 12.5 −2.34 3.44 10.94 7.5 11.04 11.98 0.94 Temporal demand 175 180.16 5.16 124.38 98.13 −26.25 158.13 152.81 −5.31 Performance 50 66.41 16.41 35.94 19.06 −16.88 45.31 50.63 5.31 Effort 59.06 111.25 52.19 182.50 68.75 −113.75 100.21 97.08 −3.13 Frustration 81.09 128.13 47.03 62.5 56.56 −5.94 74.90 104.27 29.38 Global score 37.07 44.00 6.92 40.21 29.44 −10.77 38.12 39.14 1.02  * P < .05. Finally, the main effect of the NASA-TLX dimensions, F(5, 115) = 15.14, P < .0001, followed by Tukey’s post hoc comparisons of means indicated that scores on the rating scales for mental and temporal demands were significantly higher than for all other scales except task effort (P < .05). This likely reflects the cognitive effort and fast pace, respectively, that the dose calculation task requires. Medication errors The device log analysis indicated that, of the dosage calculations performed using pac2 (N = 1202), about 20% were flagged as high and about 12% as low (ie above and below the safe therapeutic range, respectively). Our qualitative data suggest that only some of these flagged calculations were actual dosing errors. For instance, one nurse expressed irritation when pac2 alerted her to low dosages that were actually appropriate for their treatment goal (ie weaning patients off medications), “We basically are weaning the patient because the baby is growing and getting bigger, and the dose is staying the same… So, it will go, ‘alarm, alarm, alarm!’ You know, ‘dose is too low.’ And I’m like ‘So what?’” On the other hand, though, during an observation session, a participant described how pac2 supported a prescriber’s decision-making; the RA noted, “She said at another time the device flagged a ‘low’ dose and this helped the MD decide to ‘DC’ (discontinue) the medication.” Another nurse commented during an interview that: “… for our neo-nurses it’s been beneficial. I know they've caught a couple of errors or even just maybe a dose that's been out of range.” Overall, the number of voluntarily reported medication errors attributed to the administration stage was small. In the NICU, there were 5 administration errors reported pre-implementation and 4 reported postimplementation. In the PICU, only 2 administrative errors were reported pre-implementation, and none were reported postimplementation. Among all the reviewed reports, there was only one that “required intervention to save life,” and it occurred pre-implementation. DISCUSSION Successful laboratory tests cannot guarantee that a technology will have desirable outcomes in the field35; thus, we deployed and evaluated a laboratory-tested handheld DST in pediatric critical care settings. The commercial device, pac2, was designed to assist nurses in translating medication orders to correct volumes and rates of administration for intravenous and other liquid medications. It was intended to reduce the incidence of dosing errors during medication administration. Although our qualitative data indicate that pac2 may have had some effect on medication prescription and administration, overall, its effect was limited. This was likely due to low participant use. While many reported that pac2 was useful in certain situations (ie complex calculations) and that some of its features were particularly helpful (ie printing function), major barriers related to nurses’ risk perceptions, the critical care environment, and pac2’s usability emerged. If routine use of such mobile DSTs is necessary to reduce the occurrence of medication dosing errors, these barriers must be addressed in both their implementation and design. Addressing points of vulnerability Our findings offer insights into potential strategies that those responsible for implementing mobile tools like pac2 may need to employ in order to mitigate underlying barriers to use in 2 key situations: when (1) administering “routine” medications and (2) facing time constraints. Routine medications The perception that pac2 was unnecessary when administering common, or routine, medications with which participants were very familiar raises several questions: What level of use is required to reduce dosing errors? Is routine use necessary? Can this outcome be achieved if use is limited to specific situations (eg new medications)? Are errors indeed less likely for routine medications? While knowledge deficits may be an issue for infrequently administered medications,57,58 the lack of familiarity may also lead to greater attention and care in administration tasks.58 In contrast, the familiarity and comfort with routine medications may contribute to the perception that these medications are lower risk,58 leading to decreased vigilance. The research on adherence to safety procedures such as nurse double checks seems to support this—adherence tends to be lower when administering routine medications.57,58 The literature does not provide a definitive answer, but there is some evidence that errors are more likely for common medications.59–61 If this is the case, it indicates a need for educational interventions to address such misperceptions prior to the implementation of mobile DSTs similar to pac2. On the other hand, if errors are less likely, it may not be reasonable to expect routine use of such tools. Instead, nurses should be encouraged to use them to complete complex calculations and as an “incidental learning” tool. In the latter case, those responsible for implementing the technology should expect use to be relatively low and evaluations may need to focus on longer-term effectiveness, as the benefits may not be immediately visible. Further research is needed to better understand the nature and root cause of dosing errors in various settings in order to facilitate effective implementation of similar technological interventions. Time constraints Our data also show that time constraints, whether due to emergencies or high patient load, were barriers to pac2 use. Unfortunately, these situations tend to be particularly error-prone,15,62,63 and are where DSTs may have the greatest potential for impact. Although there are strategies that may be effective in the high patient load context—both organizational (eg increase staff) and individual (eg share suggestions from “super users”)—they do not address emergency situations. Technology gap In both of the above situations, if a tool was needed, nurses tended to use the simplest, most convenient tool possible. Thus, even though existing tools (eg conventional calculators) may be insufficient and nurses seem to desire a more sophisticated solution, pac2 was too complex and resource intensive to be practical in these contexts. Interestingly, these situations have also been associated with non-adherence to other safety protocols.57,58,64 Taken together, it suggests that these may be persistent points of vulnerability in hospitals’ defenses against errors, and that they may continue to be unless the design of mobile DSTs like pac2 is improved. Improving mobile decision support technologies The results of this study provide some insights into how developers may improve nurse acceptance and address this technology gap. Along with existing features that provide process efficiencies (eg printing function), there are opportunities to improve nurse acceptance by customizing the tool for the setting (eg default to the units needed most often) and accounting for exceptions (eg where a nonstandard dose may be required). More importantly, though, convenience and time efficiency may be improved with a system that is integrated into the hospitals’ existing IT infrastructure—leveraging the tools, such as smartphones, and systems, such as electronic health record, that already exist in the environment. The optimal solution may be one that is flexible enough to address context-specific needs and that nurses may customize according to their individual preferences. As many others have discussed, the extent to which commercial technologies, including standalone DSTs, may be customized is often limited.65,66 While there are cases where custom DSTs have been locally developed and integrated into commercial systems,67 this is relatively rare and, obviously, requires sufficient local resources. More commonly, though, vendors want to maintain complete control over their proprietary systems, including the available DSTs and the platforms through which they are accessed.66 Thus, it remains unclear whether a commercial mobile DST like the one our participants envisioned will be developed at all and, if it is, whether it will be flexible enough to meet pediatric critical care nurses’ needs and expectations. Finally, our findings also underscore the importance of extensive user-centered usability testing, especially formative studies, during the development of mobile DSTs so that issues are identified and addressed as early as possible—ideally before the tool becomes widely available. Limitations This study had 2 major limitations. First, participation was voluntary and not all nurses participated in all study activities. Because participants likely represent a particularly motivated and tech-savvy group, it is possible that non-participants would be even less likely to use pac2. This suggests an even greater need to address the barriers identified in this study; however, more research is also needed to determine whether there may be additional barriers when such technology is implemented more broadly. Second, we utilized medication error data from voluntary reporting systems, which tend to underestimate errors.68 For this study, though, the bigger concern was observation bias—that participation in the study would change reporting behaviors. We minimized this risk by leveraging hospitals’ existing quality assurance mechanisms rather than collecting this information directly from participants. Despite these limitations, this was a comprehensive, multi-site evaluation that used a mixed methods approach to generate key insights into nurses’ acceptance of commercial handheld DSTs to reduce medication dosing errors in pediatric critical care settings. End-user acceptance is essential to realizing the value of such technologies; our data suggest ways to improve the design and implementation of mobile DSTs in order to facilitate acceptance. These findings add to the growing evidence base on end-user acceptance and effectiveness of DSTs in the field. It also addresses a gap in this literature, as few studies have evaluated mobile DSTs, particularly commercial tools, in healthcare settings. CONCLUSION The results of this study show that regardless of its potential for reducing AMEs, the commercial, standalone handheld decision support device evaluated was not routinely used by participants and thus achieved very limited impacts on error reduction and workload. We conclude that, although a mobile tool may be necessary to provide just-in-time medication administration decision support to pediatric critical care nurses, portability does not guarantee routine use—implementation efforts, as well as the tool’s usability and time efficiency, are critical for adoption. Nurses may be more likely to accept a mobile DST that is integrated into the existing IT infrastructure, provides clear process efficiencies, is flexible enough to address context-specific needs, and is customizable according to individual preferences. This research should inform efforts to develop and deploy similar patient safety technologies. CONTRIBUTORS TG, PRD, KAE, JGS, and KZ contributed to the conception and design of the study, as well as to obtaining funding and to the manuscript. KZ managed the data, led the qualitative and qualitative data analyses, and significantly contributed to drafting the manuscript. JGS worked with KZ to oversee all project-related activities, and also reviewed and categorized medication errors. PRD coordinated study activities at one site and assisted with data analysis. KAE coordinated study activities at the other site, assisted with data analysis, and developed the training plan for the intervention. NJW and JAR assisted with data collection and revising the manuscript. TLR contributed significantly to drafting and revising the manuscript. ACKNOWLEDGMENTS This research and development project was conducted by Texas Tech University. We are grateful for the contributions of Scott Ciarkowski and Marita G. Titler during early stages of the project. FUNDING This research was supported by the U.S. Army Medical Research & Materiel Command (USAMRMC) and the Telemedicine & Advanced Technology Research Center (TATRC), at Fort Detrick, MD under Contract Number: W81XWH1010606. Conflict of interest statement. During this study, Todd Gage was employed by InformMed, the producer of the device, and served as the administrative project director. None of the other authors have any interests to disclose. NON-ENDORSEMENT DISCLAIMER The views, opinions and/or findings contained in this publication are those of the authors and do not necessarily reflect the views of the Department of Defense and should not be construed as an official DoD/Army position, policy or decision unless so designated by other documentation. No official endorsement should be made. REFERENCES 1 Makary MA , Daniel M. Medical error-the third leading cause of death in the US . BMJ   2016 ; 353 : i2139. Google Scholar Crossref Search ADS PubMed   2 Kohn LT , Corrigan JM , Donaldson MS , eds. To Err Is Human: Building a Safer Health System.   Washington, D.C .: National Academies Press ; 2000 . http://www.nap.edu/catalog/9728. Accessed April 22, 2016. 3 Landrigan CP , Parry GJ , Bones CB , Hackbarth AD , Goldmann DA , Sharek PJ. Temporal trends in rates of patient harm resulting from medical care . N Engl J Med   2010 ; 363 22 : 2124 – 34 . Google Scholar Crossref Search ADS PubMed   4 Classen DC , Resar R , Griffin F , et al. . Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured . Health Aff (Millwood)   2011 ; 30 4 : 581 – 9 . Google Scholar Crossref Search ADS PubMed   5 Prgomet M , Li L , Niazkhani Z , Georgiou A , Westbrook JI. Impact of commercial computerized provider order entry (CPOE) and clinical decision support systems (CDSSs) on medication errors, length of stay, and mortality in intensive care units: a systematic review and meta-analysis . J Am Med Inform Assoc   2016 ; 24(2): 413–22. 6 Moyen E , Camiré E , Stelfox HT. Clinical review: medication errors in critical care . Crit Care   2008 ; 12 2 : 208. Google Scholar Crossref Search ADS PubMed   7 Raju TNK , Thornton JP , Kecskes S , Perry M , Feldman S. Medication errors in neonatal and paediatric intensive-care units . Lancet   1989 ; 334 8659 : 374 – 6 . Google Scholar Crossref Search ADS   8 Agarwal S , Classen D , Larsen G , et al. . Prevalence of adverse events in pediatric intensive care units in the United States* . Pediatr Crit Care Med   2010 ; 11 5 : 568. Google Scholar Crossref Search ADS PubMed   9 Takata GS , Mason W , Taketomo C , Logsdon T , Sharek PJ. Development, testing, and findings of a pediatric-focused trigger tool to identify medication-related harm in US children’s hospitals . Pediatrics   2008 ; 121 4 : e927 – 35 . Google Scholar Crossref Search ADS PubMed   10 Kaushal R. Medication errors and adverse drug events in pediatric inpatients . JAMA   2001 ; 285 16 : 2114. Google Scholar Crossref Search ADS PubMed   11 Kirkendall ES , Kloppenborg E , Papp J , et al. . Measuring adverse events and levels of harm in pediatric inpatients with the global trigger tool . Pediatrics   2012 ; 130 5 : e1206 – 14 . Google Scholar Crossref Search ADS PubMed   12 Rocchi F , Tomasi P. The development of medicines for children. Part of a series on Pediatric Pharmacology, guest edited by Gianvincenzo Zuccotti, Emilio Clementi, and Massimo Molteni . Pharmacol Res   2011 ; 64 3 : 169 – 75 . Google Scholar Crossref Search ADS PubMed   13 Nunn A , Richey R , Shah U , et al. . Estimating the requirement for manipulation of medicines to provide accurate doses for children . Eur J Hosp Pharm Sci Pharm   2012 ; 20 (1):3–7. 14 Folli HL , Poole RL , Benitz WE , Russo JC. Medication error prevention by clinical pharmacists in two children’s hospitals . Pediatrics   1987 ; 79 5 : 718 – 22 . Google Scholar PubMed   15 Cowley E , Williams R , Cousins D. Medication errors in children: a descriptive summary of medication error reports submitted to the United States pharmacopeia . Curr Ther Res   2001 ; 62 9 : 627 – 40 . Google Scholar Crossref Search ADS   16 Keers RN , Williams SD , Cooke J , Ashcroft DM. Prevalence and nature of medication administration errors in health care settings: a systematic review of direct observational evidence . Ann Pharmacother   2013 ; 47 2 : 237 – 56 . Google Scholar Crossref Search ADS PubMed   17 Shaw KN , Lillis KA , Ruddy RM , et al. . Reported medication events in a paediatric emergency research network: sharing to improve patient safety . Emerg Med J   2013 ; 30 10 : 815 – 9 . Google Scholar Crossref Search ADS PubMed   18 Ghaleb MA , Barber N , Franklin BD , Wong ICK. The incidence and nature of prescribing and medication administration errors in paediatric inpatients . Arch Dis Child   2010 ; 95 2 : 113 – 8 . Google Scholar Crossref Search ADS PubMed   19 McLeod MC , Barber N , Franklin BD. Methodological variations and their effects on reported medication administration error rates . BMJ Qual Saf   2013 ; 22 4 : 278 – 89 . Google Scholar Crossref Search ADS PubMed   20 Phillips J , Beam S , Brinker A , et al. . Retrospective analysis of mortalities associated with medication errors . Am J Health-Syst Pharm   2001 ; 58 19 : 1835 – 41 . Google Scholar Crossref Search ADS PubMed   21 Krähenbühl-Melcher A , Schlienger R , Lampert M , Haschke M , Drewe J , Krähenbühl S. Drug-related problems in hospitals. A review of the recent literature . Drug Saf   2007 ; 30 5 : 379 – 407 . Google Scholar Crossref Search ADS PubMed   22 Pedersen CA , Schneider PJ , Scheckelhoff DJ. ASHP national survey of pharmacy practice in hospital settings: dispensing and administration-2002 . Am J Health-Syst Pharm   2003 ; 60 1 : 52 – 68 . Google Scholar Crossref Search ADS PubMed   23 Pedersen CA , Schneider PJ , Scheckelhoff DJ. ASHP national survey of pharmacy practice in hospital settings: prescribing and transcribing-2016 . Am J Health Syst Pharm   2017 ; 74 17 : 1336 – 52 . Google Scholar Crossref Search ADS PubMed   24 Conti A , Beare PG. Performance on a mathematics/drug calculation test: relationship to subsequent reported medication errors . J Nurs Staff Dev   1988 ; 4 2 : 54 – 8 . Google Scholar PubMed   25 Bindler R , Bayne T. Medication calculation ability of registered nurses . Image J Nurs Sch   1991 ; 23 4 : 221 – 4 . Google Scholar Crossref Search ADS PubMed   26 Bliss-Holtz J. Discriminating types of medication calculation errors in nursing practice . Nurs Res   1994 ; 43 6 : 373 – 5 . Google Scholar Crossref Search ADS PubMed   27 Garg AX , Adhikari NKJ , McDonald H , et al. . Effects of computerized clinical decision support systems on practitioner performance and patient outcomes. A systematic review . JAMA   2005 ; 293 10 : 1223. Google Scholar Crossref Search ADS PubMed   28 Kawamoto K , Houlihan CA , Balas EA , Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success . BMJ   2005 ; 330 7494 : 765 – 0 . Google Scholar Crossref Search ADS PubMed   29 Martínez-Pérez B , de la Torre-Díez I , López-Coronado M , Sainz-de-Abajo B , Robles M , García-Gómez JM. Mobile clinical decision support systems and applications: a literature and commercial review . J Med Syst   2014 ; 38 1 : 4 . Google Scholar Crossref Search ADS PubMed   30 Dwivedi R , Ghahramani F , Mahapatra R. Mobile clinical decision support systems—a systematic review. In: AMCIS 2017 Proceedings; 2017 ; https://aisel.aisnet.org/amcis2017/Healthcare/Presentations/5. Accessed August 8, 2018. 31 Deshpande GG , Torres A , Buchanan DL. Computerized pharmaceutical algorithm reduces medication administration errors during simulated resuscitations . J Pediatr Pharmacol Ther   2010 ; 15 4 : 274 – 81 . Google Scholar PubMed   32 Yuan MJ , Finley GM , Long J , Mills C , Johnson RK. Evaluation of user interface and workflow design of a bedside nursing clinical decision support system . Interact J Med Res   2013 ; 2 1 : e4. Google Scholar Crossref Search ADS PubMed   33 Roshanov PS , Fernandes N , Wilczynski JM , et al. . Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials . BMJ   2013 ; 346 1 : f657. Google Scholar Crossref Search ADS PubMed   34 van der Sijs H , Aarts J , Vulto A , Berg M. Overriding of drug safety alerts in computerized physician order entry . J Am Med Inform Assoc   2006 ; 13 2 : 138 – 147 . Google Scholar Crossref Search ADS PubMed   35 Anderson JG , Aydin CE , Jay SJ , eds. Evaluating Health Care Information Systems: Methods and Applications.   Thousand Oaks, CA : Sage Publications ; 1994 . 36 Zheng K. Clinical decision-support systems In: Bates M , Maack M , eds. Encyclopedia of Library and Information Sciences  . New York City, NY : Taylor & Francis Group ; 2009 . http://www.tandfonline.com/doi/book/10.1081/E-ELIS3. Accessed March 30, 2018. 37 Sittig DF. Defining health information technology–related errors: new developments since to err is human . Arch Intern Med   2011 ; 171 14 : 1281. Google Scholar Crossref Search ADS PubMed   38 Koppel R , Metlay JP , Cohen A , et al. . Role of computerized physician order entry systems in facilitating medication errors . JAMA   2005 ; 293 10 : 1197 – 1203 . Google Scholar Crossref Search ADS PubMed   39 Ash JS. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors . J Am Med Inform Assoc   2003 ; 11 2 : 104 – 112 . Google Scholar Crossref Search ADS PubMed   40 Landman AB , Takhar SS , Wang SL , et al. . The hazard of software updates to clinical workstations: a natural experiment . J Am Med Inform Assoc   2013 ; 20 ( e1 ): e187 – e190 . Google Scholar Crossref Search ADS PubMed   41 Gephart S , Carrington JM , Finley B. A systematic review of nurses’ experiences with unintended consequences when using the electronic health record . Nurs Adm Q   2015 ; 39 4 : 345. Google Scholar Crossref Search ADS PubMed   42 Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology . Mis Q   1989 ; 13 3 : 319. Google Scholar Crossref Search ADS   43 Venkatesh V , Davis FD. A theoretical extension of the technology acceptance model: four longitudinal field studies . Manag Sci   2000 ; 46 2 : 186 – 204 . Google Scholar Crossref Search ADS   44 Venkatesh V , Morris MG , Davis GB , Davis FD. User acceptance of information technology: toward a unified view . Mis Q   2003 ; 27 (3): 425 – 478 . 45 Bates DW , Kuperman GJ , Wang S , et al. . Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality . J Am Med Inform Assoc   2003 ; 10 6 : 523 – 530 . Google Scholar Crossref Search ADS PubMed   46 Cork RD , Detmer WM , Friedman CP. Development and initial validation of an instrument to measure physicians’ use of, knowledge about, and attitudes toward computers . J Am Med Inform Assoc   1998 ; 5 2 : 164 – 176 . Google Scholar Crossref Search ADS PubMed   47 Wheeler NJ , DeLucia PR , Esquibel KA , et al. . Focused ethnographic observations: cues to medication administration . Proc Hum Factors Ergon Soc Annu Meet   2014 ; 58 1 : 1257 – 1261 . Google Scholar Crossref Search ADS   48 Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use . Int J Hum-Comput Interact   1995 ; 7 1 : 57 – 78 . Google Scholar Crossref Search ADS   49 Hart SG , Staveland LE. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research In: Advances in Psychology  . Vol 52 . Elsevier . Amsterdam: North-Holland; 1988 : 139 – 183 . 50 Rubio S , Diaz E , Martin J , Puente JM. Evaluation of subjective mental workload: a comparison of SWAT, NASA-TLX, and workload profile methods . Appl Psychol   2004 ; 53 1 : 61 – 86 . Google Scholar Crossref Search ADS   51 Hoonakker P , Carayon P , Gurses AP , et al. . Measuring workload of ICU nurses with a questionnaire survey: the NASA Task Load Index (TLX) . IIE Trans Healthc Syst Eng   2011 ; 1 2 : 131 – 143 . Google Scholar Crossref Search ADS PubMed   52 National Coordinating Council for Medication Error Reporting and Prevention . NCC MERP Index for Categorizing Medication Errors. NCC. http://www.nccmerp.org/sites/default/files/indexBW2001-06-12.pdf. last accessed May 01, 2013 53 Charmaz K. Constructing Grounded Theory  . 2nd ed . London : Sage ; 2014 . 54 Glaser BG , Strauss AL. The constant comparative method of qualitative analysis In: The Discovery of Grounded Theory  . Chicago, IL : Aldine Transaction ; 1967 . 55 Hsieh H-F , Shannon SE. Three approaches to qualitative content analysis . Qual Health Res   2005 ; 15 9 : 1277 – 1288 . Google Scholar Crossref Search ADS PubMed   56 Ritchie J , Spencer L. Qualitative data analysis for applied policy research. In: Bryman A , Burgess B , eds. Analyzing Qualitative Data  . London : Routledge ; 1994 : 173 – 194 . 57 Keers RN , Williams SD , Cooke J , Ashcroft DM. Understanding the causes of intravenous medication administration errors in hospitals: a qualitative critical incident study . BMJ Open   2015 ; 5 3 : e005948 – e005948 . Google Scholar Crossref Search ADS PubMed   58 Gill F , Corkish V , Robertson J , Samson J , Simmons B , Stewart D. An exploration of pediatric nurses’ compliance with a medication checking and administration protocol J Spec Pediatr Nurs   2012 ; 17 2 : 136 – 146 . Google Scholar Crossref Search ADS PubMed   59 Ghaleb MA , Barber N , Franklin BD , Yeung VW , Khaki ZF , Wong IC. Systematic review of medication errors in pediatric patients . Ann Pharmacother   2006 ; 40 10 : 1766 – 1776 . Google Scholar Crossref Search ADS PubMed   60 Anderson DJ , Webster CS. A systems approach to the reduction of medication error on the hospital ward . J Adv Nurs   2001 ; 35 1 : 34 – 41 . Google Scholar Crossref Search ADS PubMed   61 American Academy of Pediatrics . Acetaminophen toxicity in children . Pediatrics   2001 ; 108 4 : 1020 – 1024 . Crossref Search ADS PubMed   62 Montgomery VL. Effect of fatigue, workload, and environment on patient safety in the pediatric intensive care unit . Pediatr Crit Care Med   2007 ; 8(Suppl) : S11 – S16 . Google Scholar Crossref Search ADS   63 Valentin A , Capuzzo M , Guidet B , et al. . Errors in administration of parenteral drugs in intensive care units: multinational prospective study . BMJ   2009 ; 338 : b814. Google Scholar Crossref Search ADS PubMed   64 Davis L , Keogh S , Watson K , McCann D. Dishing the drugs: a qualitative study to explore paediatric nurses’ attitudes and practice related to medication administration . Coll R Coll Nurs Aust   2005 ; 12 4 : 15 – 20 . 65 Cresswell KM , Bates DW , Williams R , et al. . Evaluation of medium-term consequences of implementing commercial computerized physician order entry and clinical decision support prescribing systems in two “early adopter” hospitals . J Am Med Inform Assoc   2014 ; 21 ( e2 ): e194 – e202 . doi: 10.1136/amiajnl-2013-002252 Google Scholar Crossref Search ADS PubMed   66 Payne TH , Hines LE , Chan RC , et al. . Recommendations to improve the usability of drug-drug interaction clinical decision support alerts . J Am Med Inform Assoc   2015 ; 22 6 : 1243 – 1250 . Google Scholar Crossref Search ADS PubMed   67 Hum RS , Cato K , Sheehan B , et al. . Developing clinical decision support within a commercial electronic health record system to improve antimicrobial prescribing in the neonatal ICU . Appl Clin Inform   2014 ; 05 02 : 368 – 387 . Google Scholar Crossref Search ADS   68 Flynn EA , Barker KN , Pepper GA , Bates DW , Mikeal RL. Comparison of methods for detecting medication errors in 36 hospitals and skilled-nursing facilities . Am J Health-Syst Pharm   2002 ; 59 5 : 436 – 446 . Google Scholar Crossref Search ADS PubMed   Published by Oxford University Press on behalf of the American Medical Informatics Association 2019. This work is written by US Government employees and is in the public domain in the United States.


This is a preview of a remote PDF: https://academic.oup.com/jamiaopen/article-pdf/2/1/49/28224192/ooy055.pdf

Reynolds, Tera L, DeLucia, Patricia R, Esquibel, Karen A, Gage, Todd, Wheeler, Noah J, Randell, J Adam, Stevenson, James G, Zheng, Kai. Evaluating a handheld decision support device in pediatric intensive care settings, JAMIA Open, 2019, 49-61, DOI: 10.1093/jamiaopen/ooy055