Inter-rater reliability of the QuIS as an assessment of the quality of staff-inpatient interactions

BMC Medical Research Methodology, Dec 2016

Background Recent studies of the quality of in-hospital care have used the Quality of Interaction Schedule (QuIS) to rate interactions observed between staff and inpatients in a variety of ward conditions. The QuIS was developed and evaluated in nursing and residential care. We set out to develop methodology for summarising information from inter-rater reliability studies of the QuIS in the acute hospital setting. Methods Staff-inpatient interactions were rated by trained staff observing care delivered during two-hour observation periods. Anticipating the possibility of the quality of care varying depending on ward conditions, we selected wards and times of day to reflect the variety of daytime care delivered to patients. We estimated inter-rater reliability using weighted kappa, κ w , combined over observation periods to produce an overall, summary estimate, κ ^ w . Weighting schemes putting different emphasis on the severity of misclassification between QuIS categories were compared, as were different methods of combining observation period specific estimates. Results Estimated κ ^ w did not vary greatly depending on the weighting scheme employed, but we found simple averaging of estimates across observation periods to produce a higher value of inter-rater reliability due to over-weighting observation periods with fewest interactions. Conclusions We recommend that researchers evaluating the inter-rater reliability of the QuIS by observing staff-inpatient interactions during observation periods representing the variety of ward conditions in which care takes place, should summarise inter-rater reliability by κ w , weighted according to our scheme A4. Observation period specific estimates should be combined into an overall, single summary statistic κ ^ w random , using a random effects approach, with κ ^ w random , to be interpreted as the mean of the distribution of κ w across the variety of ward conditions. We draw attention to issues in the analysis and interpretation of inter-rater reliability studies incorporating distinct phases of data collection that may generalise more widely.

A PDF file should load here. If you do not see its contents the file may be temporarily unavailable at the journal website or you do not have a PDF plug-in installed and enabled in your browser.

Alternatively, you can download the file locally and open with any standalone PDF reader:

http://www.biomedcentral.com/content/pdf/s12874-016-0266-4.pdf

Inter-rater reliability of the QuIS as an assessment of the quality of staff-inpatient interactions

Mesa-Eguiagaray et al. BMC Medical Research Methodology Inter-rater reliability of the QuIS as an assessment of the quality of staff-inpatient interactions Ines Mesa-Eguiagaray 0 Dankmar Böhning 2 Chris McLean 1 Peter Griffiths 1 Jackie Bridges 1 Ruth M Pickering 0 0 Medical Statistics Group, Faculty of Medicine, Southampton General Hospital , Mailpoint 805Level B, South Academic Block, Southampton SO16 6YD , UK 1 Faculty of Health Sciences, University of Southampton , Southampton , UK 2 Southampton Statistical Sciences Research Institute & Mathematical Sciences, University of Southampton , Southampton , UK Background: Recent studies of the quality of in-hospital care have used the Quality of Interaction Schedule (QuIS) to rate interactions observed between staff and inpatients in a variety of ward conditions. The QuIS was developed and evaluated in nursing and residential care. We set out to develop methodology for summarising information from inter-rater reliability studies of the QuIS in the acute hospital setting. Methods: Staff-inpatient interactions were rated by trained staff observing care delivered during two-hour observation periods. Anticipating the possibility of the quality of care varying depending on ward conditions, we selected wards and times of day to reflect the variety of daytime care delivered to patients. We estimated inter-rater reliability using weighted kappa, κw, combined over observation periods to produce an overall, summary estimate, κ^w. Weighting schemes putting different emphasis on the severity of misclassification between QuIS categories were compared, as were different methods of combining observation period specific estimates. Results: Estimated κ^w did not vary greatly depending on the weighting scheme employed, but we found simple averaging of estimates across observation periods to produce a higher value of inter-rater reliability due to overweighting observation periods with fewest interactions. Conclusions: We recommend that researchers evaluating the inter-rater reliability of the QuIS by observing staffinpatient interactions during observation periods representing the variety of ward conditions in which care takes place, should summarise inter-rater reliability by κw, weighted according to our scheme A4. Observation period specific estimates should be combined into an overall, single summary statistic κ^w random, using a random effects approach, with κ^w random, to be interpreted as the mean of the distribution of κw across the variety of ward conditions. We draw attention to issues in the analysis and interpretation of inter-rater reliability studies incorporating distinct phases of data collection that may generalise more widely. Weighted kappa; Random effects meta-analysis; QuIS; Collapsing; Averaging - Background The Quality of Interactions Schedule (QuIS) has its origin in observational research undertaken in 1989 by Clark & Bowling [1] in which the social content of interactions between patients and staff in nursing homes and long term stay wards for older people was rated to be positive, negative or neutral. The rating specifically relates to the social or conversational aspects of an interaction, such as the degree to which staff acknowledge the patient as a person, not to the adequacy of any care delivered during the interaction. Dean et al. [2] extended the rating by introducing distinctions within the positive and negative ratings, creating a five category scale as set out in Table 1. QuIS is now generally regarded as an ordinal scale ranging from the highest ranking, positive social interactions to the lowest ranking, negative restrictive interactions [3]. Barker et al. [4] in a feasibility study of an intervention designed to improve the compassionate/social aspects of care experienced by older people in acute hospital © The Author(s). 2016 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Table 1 Definitions of QuIS categories [2] Positive Care (+c) Negative protective (−p) Explanation Interaction principally involving ‘good, constructive, beneficial’ conversation and companionship. Interactions during the appropriate delivery of physical care. Brief, indifferent interactions not meeting the definitions of the other categories. Providing care, keeping safe or removing from danger, but in a restrictive manner, without explanation or reassurance: in a way which disregards dignity or fails to demonstrate respect for th (...truncated)


This is a preview of a remote PDF: http://www.biomedcentral.com/content/pdf/s12874-016-0266-4.pdf

Ines Mesa-Eguiagaray, Dankmar Böhning, Chris McLean, Peter Griffiths, Jackie Bridges, Ruth M Pickering. Inter-rater reliability of the QuIS as an assessment of the quality of staff-inpatient interactions, BMC Medical Research Methodology, 2016, pp. 171, 16, DOI: 10.1186/s12874-016-0266-4