Measuring Client Experiences in Maternity Care under Change: Development of a Questionnaire Based on the WHO Responsiveness Model
Measuring Client Experiences in Maternity Care under Change: Development of a Questionnaire Based on the WHO Responsiveness Model
Marisja Scheerhagen 0 1 2
Henk F. van Stel 0 1 2
Erwin Birnie 0 1 2
Arie Franx 0 1 2
Gouke J. Bonsel 0 1 2
0 1 Erasmus Medical Centre, Department of Obstetrics and Gynaecology, Division of Obstetrics and Prenatal Medicine , Rotterdam , The Netherlands , 2 University Medical Center Utrecht, Julius Center for Health Sciences and Primary Care, Department of Health Technology Assessment , Utrecht , The Netherlands , 3 Institute of Health Policy and Management, Erasmus University Rotterdam , Rotterdam , The Netherlands , 4 University Medical Center Utrecht, Division Women and Baby, Department of Obstetrics , Utrecht , The Netherlands
1 Data Availability Statement: All relevant data are available from the Julius Center for Health Sciences and Primary Care Dataverse: https://dataverse.nl/ dvn/dv/Julius_Center/faces/study/StudyPage.xhtml? globalId = hdl:10411/20363
2 Academic Editor: Sari Helena Raisanen, Kuopio University Hospital , FINLAND
Maternity care is an integrated care process, which consists of different services, involves
different professionals and covers different time windows. To measure performance of
maternity care based on clients' experiences, we developed and validated a questionnaire.
Methods and Findings
We used the 8-domain WHO Responsiveness model, and previous materials to develop a
self-report questionnaire. A dual study design was used for development and validation.
Content validity of the ReproQ-version-0 was determined through structured interviews with
11 pregnant women ( 28 weeks), 10 women who recently had given birth ( 12 weeks),
and 19 maternity care professionals. Structured interviews established the domain
relevance to the women; all items were separately commented on. All Responsiveness
domains were judged relevant, with Dignity and Communication ranking highest. Main
missing topic was the assigned expertise of the health professional. After first adaptation,
construct validity of the ReproQ-version-1 was determined through a web-based survey.
Respondents were approached by maternity care organizations with different levels of
integration of services of midwives and obstetricians. We sent questionnaires to 605 third
trimester pregnant women (response 65%), and 810 women 6 weeks after delivery (response
55%). Construct validity was based on: response patterns; exploratory factor analysis;
association of the overall score with a Visual Analogue Scale (VAS), known group
comparisons. Median overall ReproQ score was 3.70 (range 14) showing good responsiveness.
The exploratory factor analysis supported the assumed domain structure and suggested
several adaptations. Correlation of the VAS rating and overall ReproQ score (antepartum,
Competing Interests: The authors have declared
that no competing interests exist.
postpartum) supported validity (r = 0.56; 0.59, p<0.001 Spearman's correlation coefficient).
Pre-stated group comparisons confirmed the expected difference following a good vs.
adverse birth outcome. Fully integrated organizations performed slightly better (median = 3.78)
than less integrated organizations (median = 3.63; p<0.001). Participation rate of women with
a low educational level and/or a non-western origin was low.
The ReproQ appears suitable for assessing quality of maternity care from the clients'
perspective. Recruitment of disadvantaged groups requires additional non-digital approaches.
Performance of maternity care is primarily determined by its health outcomes, in particular
mortality and morbidity of mother and child over the short and long term. Such outcomes
differ globally, countrywise, and also within countries where health care quality differences may
be in part responsible .
Another dimension of maternity care performance, is the way that clients (here primarily
the women involved) experience the care provided. This includes whether they feel secure, feel
treated with respect, feel adequately informed; are facilities in a broad sense accessible and
client-friendly. These client experiences with health care provision are supposed to be important
for two reasons: 1) client experiences represent an independent outcome of performance,
which may guide choices of health care provider if outcomes are similar ; 2) client
experiences may affect clinical outcomes through several ways, hence may act as determinant of the
aforementioned outcomes in mother and child . According to the World Health
Organization (WHO), which developed an influential concept to measure client experiences, adequate
client orientation ultimately relates to respecting human rights, specified for the context of
health care provision [6,11,12].
To achieve uniform measurement of client experiences as a performance indicator, the
WHO elaborated the so-called Responsiveness model, after comprehensive preparatory studies
and consultation. Following this model, responsiveness is defined as the way a client is treated
by the professional and the environment in which the client is treated, where eight different
domains are suggested to cover the concept. This model deliberately focusses on individual
experiences rather than characteristics of processes or structures, acknowledging that between and
even within countries the same client experiences may be arrived at by various means. The
model has been shown to enable comparison of experienced performance within and between
countries on a general level [6,13].
So far, the responsiveness questionnaires were never specified to a health care subsystem,
such as maternity care. We selected the WHO responsiveness model to measure client
experiences in maternity care in the Netherlands, for reasons explained below.
Measurement of maternity care performance in general is a challenge, because maternity
care consists of different services (e.g. antenatal check-ups, care during the delivery); different
time windows (antepartum phase, childbirth, postpartum phase) and involves several
professions; and professionals (e.g., obstetricians, midwives, and maternity nurses) where many tasks
are executed interchangeably.
Seen from the client's perspective, the health system in many countries shows considerable
variety in health care arrangements, the location of organizations (e.g. urban vs. rural), and
This is particularly true in the Netherlands where currently the maternity care is changing
from a two-tier system to an integrated care system . The current dominant two-tier
system is based on strict division of tasks, with primary care though midwives and general
practitioners for assumed low-risk pregnant women, and secondary/tertiary care for assumed
high-risk women in hospitals and perinatal centers. Primary care and secondary care
professionals each have their own professional autonomy, responsibilities, and financial
arrangements, and integration of processes and risk standards is limited. In view of the unsatisfactory
performance of the Dutch maternity care system (perinatal outcome, maternal outcome,
system weaknesses e.g. in risk management and 24/7 hospital quality), maternity care shifts
towards integrated care, following the 2010 advice of a National Committee on Perinatal Care
established by the Ministry of Health [1,2,1820]. Integrated care combines the delivery and
organization of health services; it assumes one clinical perspective, one risk management
approach and one client orientation .
Existing indicators and questionnaires all appeared limited for our purposes. They either
focus on specific processes, monodisciplinary perspectives or assume a specific maternity care
organization; they usually contain additional modules on outcomes and procedural facts, and
lack a formal aggregate scoring system for the client's experience allowing a graded quality
judgment. For example, the questionnaires of the British National Health Service
(Women's Experience of Maternity Care)  and the National Perinatal Epidemiology Unit
 include only part of the responsiveness domains, focusing on the personal quality of
services. The Dutch Consumer Quality Index for primary maternity care  and a similar survey
for postnatal care  focus on the care delivered by one professional group (community
midwife, maternity nurse) for specific phases (antenatal, delivery, first postnatal week) assuming
monodisciplinary care as standard, i.e. without any involvement of hospital, gynaecologist or
paediatrician. Two other comprehensive interviewer-based instruments are obviously not
suited for self-report. The Maternity Experiences Survey from Canada assumes additional
explanatory support of an interviewer, and its length precludes routine application . Prior to the
ReproQ, we developed a structured face-to-face interview based on the WHO responsiveness
concept to evaluate care in an integrated birth centre, which includes clinical postdelivery
services . Like the Maternity Experiences Survey this interview was too long for routine
application, and results suggested that after a complicated delivery, bias could occur in the
report of client experiences antenatally (carry back effect ). Other surveys, not listed here,
primarily ask for the presence of structural features or care processes rather than for the
performance-as-experienced. International comparisons of health services  have made clear that
one cannot easily rely on the structural features, as a proxy for the actual client centeredness of
services, in particular in case of disadvantaged groups. The WHO model seemed appropriate
and suitable in this case as starting point for a uniformly applicable questionnaire on client
experiences, as it allows for measurement regardless of the particular organizational and
professional characteristics. We expect that this questionnaire is sensitive for performance
characteristics that benefit from integration, such asin terms of the WHO
domainscommunication, prompt attention, information continuity, etc. The questionnaire may also be
sensitive for potential negative aspects of integration such as decreased autonomy if care becomes
more rule-based. Existing indicators and questionnaires either focus on processes and
structural features (from a professional point of view) of maternity care, or are to some extent restricted
to one organizational structure , justifying our comprehensive approach on the base of
a proven concept.
The study presented here describes the development of a client experiences questionnaire
on the basis of the WHO responsiveness model, and presents basic psychometric evidence.
Methods and Materials
The development of the questionnaire, called the ReproQ, covered three phases: 1. overall
design and specific item generation for the client experiences following the WHO concept; 2.
interview study involving relevant stakeholders to determine the content validity of the null
version of the ReproQ; 3. survey study in 4 different regions to enable constructive
psychometric analysis. Prior to the description of the methods used in these phases, we describe the seven
theoretical considerations on which the ReproQ is based. The phasing is shown in Fig. 1.
Content: 1) The WHO responsiveness model was the conceptual basis. This model consists of
four domains concerning the interaction between the client and health professional (dignity,
autonomy, confidentiality, and communication), and four domains concerning the
organizational structure (prompt attention, access to family and community support, quality of basic
amenities, and choice and continuity of care) [6,13].
2) In agreement with the WHO model, the operationalization of the concept into experience
items avoided any implicit preference toward provider or organization structures, leaving
room to different organization structures and different levels of integrated care (high/low). We
did not measure integral working as such; moreover, we assumed performance in terms of the
WHO responsiveness concept would benefit from more integration, if performed well.
3) The questionnaire focussed on performance-as-experienced by the client, rather than on structural features or processes.
Coverage: 4) The mother is the principle bearer of experiences, because choices and
decision-making in maternity care delivery generally rest with the mother or mother-to-be. In
addition, the child's father may not invariably be a desirable or available co-respondent.
Obviously, responsiveness cannot be reported by the neonates themselves.
5) From a system's point of view, maternity care actually consists of service delivery that is
different during pregnancy, during childbirth and postpartum care. The antepartum phase can
be defined as monitoring intermittent preventive care, mostly in an ambulatory facility.
Screening is a particular feature at onset of antenatal care. The delivery is a single, high impact
process, which shows many features of acute curative care. Postpartum care aims at monitoring
the health of both mother and child, and at empowering the parents for the future. In these
three phases, the interaction with health care professionals, facilities, and the time axis of
experiences are quite different. We developed two mirror versions of the questionnaire; one to
measure experiences during pregnancy (antepartum) and one to measure experiences during
delivery and thereafter (postpartum).
Both versions are symmetrical, in that the same type of experiences are asked for and the
way these are asked for is also identical, yet each item is adapted to the context (antepartum vs.
postpartum). In each version we asked the client to judge each item during two reference
periods: in the antepartum questionnaire the first and second half of pregnancy, in the postpartum
questionnaire the event of labour and birth, and the subsequent postpartum week.
Consequently, responses on all responsiveness items existed for 4 different reference periods.
Feasibility: 6) The questionnaire was intended for self-report of clients, without support,
and was primarily developed as online survey. A paper version should be also available,
limiting the possible complexity of the digital version.
7) The questionnaire was suitable for clients with low educational level (defined as duration
6 years for migrant women and 8 years for women of Dutch origin) and migrants and
clients of non-western origin. This was achieved by the following: a) the response mode
uniformly used 4 simple categories: never, sometimes, often, and always, with a
numerical range of 1 (worst) to 4 (best); b) items consist of short sentences; c) common
language was used (reading level B1, checked by word frequency lists ); testing by members of
the target group. We are aware that illiterate clients need another approach, most likely a
Survey structure and item generation
The questionnaire consists of five sections, i.e.: 1) information about the current care process,
the location of care (e.g. home or hospital) and the dominant health professional delivering
care (e.g. midwife or obstetrician); 2) the clinical outcome of both mother and child, as
perceived by the mother in non-medical terms; 3) the client experiences in terms of the eight key
domains of the responsiveness model; 4) information about previous pregnancies; 5)
socio-demographic characteristics of the client.
Section 3 is the key section of the ReproQ. For the generation of the items of this section we
used four sources. First we looked at the responsiveness items of the World Health Survey and
Multicountry Survey model , adapting items with contextual information of maternity care.
Second, we used items generated for a previously developed face-to-face interview . Third,
we explored published questionnaires on the same or related concepts concerning maternity
care . Finally, we used the manual of the Dutch Consumer Quality Index method to
measure client experiences .
The other sections were developed to enable interpretation of the experiences, and
supplementary discriminative content validation, as reported in this paper. The elaboration of these
sections was based on existing formats and will not be discussed further.
Content validity: interviews
Content validity of the pilot version of the ReproQ (version 0) was determined through
structured interviews, supported by questionnaires, with 11 pregnant women, 10 women who
recently had given birth ( 12 weeks postpartum) and 19 maternity care professionals (7
midwifes, 4 obstetricians, 2 maternity nurses, 4 executives and 2 perinatal health officers). In
Spring 2012, the participating clients were approached in three different maternity care
organizations in The Netherlands with different levels of integration: 1) a fully integrated midwifery
practice and a peripheral hospital (Roosendaal); 2) a fully integrated midwifery practice and a
university hospital (Utrecht); 3) a clinic from the university hospital in Rotterdam, with an
adjacent birth centre (Rotterdam). The hospitals involved, and the birth centre provided care to
clients of several associated primary care midwifery practices and clients, which were already
under care from the hospital. Clients were approached either by their professional or a member
of the research team. The maternity care professionals were recruited from the same facilities
through their team manager.
We intended to perform a group interview with each group of relevant stakeholders in each
center, resulting in altogether nine group interviews. We intended to include a minimum of six
participants per interview. All interviews were chaired and performed by the research team.
The number of participants for each organization is shown in Table A1 in the S1 Table.
The group interviews of about 2 hours followed a common structure: 1) prioritisation of the
responsiveness domains; 2) two comments on each item (a. contents and b.
grammar/readability); these were first written down for each item separately, and subsequently discussed in
plenum; 3) systematic check for missing topics or perspectives of the questionnaire. Health
professionals were additionally asked to rate the suitability of the experience items of the
questionnaire (ReproQ core, section 3) from the perspective for women with a low educational level
and non-western women. Because they regularly encounter many of these women during their
consultation hours, we assumed that they could give a reasonable judgment of the suitability.
They separately rated the suitability for women with a low educational level and for women
with different ethnicities on a five point scale [strongly agree-strongly disagree].
More in detail, the client interview first invited the participants to individually describe
their wishes and possible improvements concerning the maternity care they had received.
Discussion could follow. Second, clients were asked which two of the eight
Responsiveness-domains were most important to them. Finally, the clients were asked to fill out the null version
of the questionnaire; comments were noted and discussed plenary. Each group client interview
lasted about 2 hours. We performed some individual interviews, when the number of
participating clients was less than the required six participants per group interview. Each participant
received a compensation of 20 ($27, 16).
The group interviews with maternity care professionals lasted on average 1.5 hours and
were unrewarded. In the group interviews with clients, 7 pregnant women and 9 women who
recently had given birth participated. In addition, we interviewed 4 pregnant women and 1
woman who recently had given birth individually. In the group interviews with health
professionals, 7 midwifes, 4 obstetricians, 2 maternity nurses, 4 executives and 2 perinatal health
The null version of the ReproQ was adjusted based on the joint comments, where comments
of clients and health professionals were regarded as equally relevant. We assumed that the item
content to be valid if the comments involved no or minor changes in item wording or
Survey study to obtain psychometric characteristics
We obtained psychometric characteristics of the adjusted questionnaire in a subsequent survey
study. Pregnant women and women who recently had given birth were asked for participation
when they visited their care provider. After written informed consent, they received an
invitation by email to fill out the web-based questionnaire. Patients were locally recruited with the
support of the organisation.
To qualify for the antepartum questionnaire, women should have a gestational age less than
34 weeks; to qualify for the questionnaire concerning the delivery and postpartum care,
women should have given birth less than 6 weeks earlier. The antepartum questionnaire was
sent in the 34th week of their pregnancy, the postpartum questionnaire was sent 6 weeks after
the expected date of delivery. Non-responding women received an e-mail reminder 2 weeks
after they received the initial questionnaire.
Four maternity care organizations participated for client recruitment. Three of these also
participated in the interview study. The additional organization included four hospitals and
four midwifery practices.
Altogether a wide range of organisational structures and client populations was covered. To
determine the psychometric characteristics of the questionnaire, we aimed at a minimum of
300 completed antepartum and 300 completed postpartum questionnaires. Because the
questionnaire exists of two versions, that are not identical, we aimed at a sample size of 300
respondents for both versions of the questionnaire. The sample size was based on the Dutch
manual to develop Consumer Quality questionnaires .
Interviews relevant stakeholders to determine the content validity. The prioritised domains
will be reported in percentage of domains ranked first or second.
The items were primarily adapted based on the detailed individual written comments.
Combining the comments per item resulted in 1) items needing no change; 2) items to be simplified
or changed to avoid textual ambiguities; 3) adaptation of the response mode in specific cases, e.
g. through addition of the option not applicable, or changes in the labels of the response
levels; 4) items to be removed, if the item did not sufficiently fit to the concept or if the item
showed too much overlap with other items questions
The comments on missing domains or items are reported if multiple comments indicated
The response mode of the five point suitability-questions for women with a low educational
level, and of non-Dutch origin were later reduced to three categories: agree-neutral-disagree, as
extreme categories were rarely used.
Survey study followed by psychometric analyses. We invited 605 pregnant women, of
whom 396 responded (65%), and invited 810 women who recently had given birth, of whom
483 responded (55%). We excluded 45 pregnant women and 50 women who recently had
given birth, because 50% of their answers were missing in 2 of more domains. The first step in
the analysis was the checking for response patterns, such as a floor-ceiling-effect, the
computation of the percentage missing-values per item, and the computation of the digitally measured
response time. The second step involved analysis of the construct validity using Exploratory
Factor Analysis (EFA) . The main goal was to identify items that required replacement to
another domain, rewording, or removal. Because we use a so-called formative measurement
model (pre-stated domain structure) and not a reflective model, the decisions on which item
belongs to which domain finally are based on content and the EFA combined, rather than
The analyses were intended to be performed separately for the four phases of maternity
care, namely first half pregnancy, second half pregnancy, birth, and postnatal care. However, as
answer patterns for the first and second half of the pregnancy were close to identical, we only
present data of the second half of pregnancy, and data of birth and postnatal care (3 reference
In the EFA for labour and birth, and postnatal care, the three questions of the domain Basic
Amenities were not included, because the number of respondents was too small due to routing
in the questionnaire. The EFA was conducted as a principal components analysis followed by
orthogonal rotation (Varimax) . The factors were determined by the Kaiser criterion (i.e.
an Eigenvalue > 1). In addition, we computed Cronbach's alpha to determine the internal
consistency of each factor. Note that internal consistency of items may be empirically low despite a
close relation in terms of contents: e.g. items on the accessibility all refer to one basic concept,
yet the travelling distance to the facility is not empirically associated to the accessibility
Third, convergent validity was tested by the association between an overall 10-point VAS
rating with the overall client experience of women, combining all domain responses. This
10-point VAS rating was based on the recent recommendations of the National Patient Survey
Coordination Centre . The overall client experiences score was obtained by first computing
an average score per domain (where the 1, 2,3 or 4 response was treated numerically), and then
computing an unweighted average across the 8 domain scores, resulting in an overall
experience score with range 14. The association of women's global rating with their experience as a
client was expressed by Spearman's correlation coefficient (rho).
The last step was a preliminary assessment of the discriminative validity of the ReproQ by
three so-called known group comparisons. The client experience was compared applying the
following groupings: 1) pregnant women versus women who recently had given birth; 2)
women with better vs. worse clinical outcome of their baby depending on perceived health
problems by the mother and hospitalization of the baby (altogether 4 groups); and 3) women
who received care in fully integrated facilities versus women who received care in less
We calculated domain scores (giving a profile) and an overall ReproQ score. Domain scores
were declared missing when less than half of the items of that domain were filled out. We
refrained from imputation of missing data. If more than half of the domain scores were missing,
no overall score was computed. Because the experience data did not show a normal
distribution, we report the overall median (MD) and the interquartile range (IQR) of all
Responsiveness domains. To explore if differences in performance were significant between groups, we
performed a Mann-Whitney test or Kruskal-Wallis test depending on the number of
determinant categories (2 or 4, respectively). Significance level was p<0.05, without adjustment for
multiple testing, as this was an explorative study, without prior sample size calculation. For the
statistical analyses we used SPSS 21.0.
The development process was supervised by a steering committee. This group consisted of
representatives from health professionals, health insurance companies, a client-patient association,
and members of the research team. Besides the steering committee, we were advised by a senior
officer of the WHO engaged in the development of responsiveness measurement, with
sufficient knowledge of the Dutch language.
The Medical Ethical Review Board of the University Medical Centre Utrecht approved the
study protocol (study number MEC-2012435).
The test version of our antepartum questionnaire contained 30 experience items. The
postpartum questionnaire contained 36 experience items. The difference is explained by items in the
domains Prompt Attention and Basic Amenities concerning specific elements of the delivery
and postnatal care, such as the facilities during hospitalization after the delivery or the presence
of a maternity nurse.
Interviewing stakeholders to determine the content validity
The mean age of the participating women was 32.3 years (SD = 5.5). Of the 21 women, 6
reported to be of non-Dutch origin (29%). Most women had a high education; 8 women had a
low/middle education (38%). All women were married or living together. Half of the women
gave birth for the first time (52%). 13 of the 21 women received care in an integrated facility
(62%). The characteristics are described in Table 1. All responsiveness domains were
confirmed as being relevant in general. The domains Dignity and Communication were selected as
most important by clients, by health professionals from their own perspective, and from the
proxy-perspective of clients with low educational level or migrant status as expressed by these
% (N = 21)
Antepartum % (N = 351)
Postpartum % (N = 433)
professionals. Clients and health professionals gave altogether 266 comments about the items
in the Responsiveness domains (roughly 1 out of 5 items received a comment). 93 (35%) of
these comments were related to the clarity of the wording of items. The participants stated
problems with specific terms e.g. personal attention, home situation and the meaning of
several options in the item Could you choose from several options for postnatal care?.
Of the 266 comments, 119 comments (45%) concerned the relevance of items. Women
noted difficulty in giving response if they had not been in a situation as described. Health
professionals doubted whether some items could be judged by clients in case of high urgency of
the care provided. They suggested adaptations of question or response (adding not
applicable) in some instances.
54 comments (20%) suggested literal improvements in text of items or the response.
The topics claimed more than once to be missing were the client's judgment of the health
professional's expertise and specific items on cultural customs and traditions of migrant
women. As the ReproQ is to be used in connection to medical outcome measures, we refrained
from adding an item on assigned expertise.
The suitability for women with low educational level was judged as sufficient by 10 of the 18
health professionals, while one health professional thought the questionnaire was unsuitable
for women with low education. All professionals emphasized to be cautious with the
application of standard survey data collection techniques in respondents with a low educational level.
Based on all comments, we left 11 items unchanged; 7 items were slightly rephrased; 5 items
were deleted; 2 items were added; and the response mode of 10 items was rephrased (adding
Survey study to determine psychometric characteristics
The characteristics of pregnant women and women who recently had given birth are presented
in Table 1. As differences were minimal, characteristics are described combined. The
participating women had a mean age of 33.1 years (SD = 4.4). Of the 784 women who responded, 72
(9%) reported to be of non-Dutch origin. 71 women were not living together with the father of
the child, or did not have a relationship with the father at all (9%).
The response pattern of the women generally showed high responsiveness to the client. The
response modus never (representing an adverse experience) was not used in several items.
Never was most often used in the item concerning choice of health care professional
(19.7%) in the antepartum and postpartum questionnaire (18.2%). The response modus
always was least often used in the item concerning waiting in the antepartum questionnaire
(20.3%), and most by the item concerning privacy (94.3%). In the postpartum questionnaire,
the response modus always was least often used in the item concerning the furnishing of the
maternity care organizations (36.1%), and most often in the item treated with respect
(88.6%). The per item missing rates were all below 5%. Filling out the antepartum
questionnaire lasted on average 16 minutes (95% confidence interval (CI): 1121min). The postpartum
questionnaire took on average 14 minutes (95% CI: 1117min).
The EFA revealed 9 factors in the antepartum questionnaire; 7 factors in experience with
delivery, and 5 factors in postnatal care. Table 2 shows the factor loadings of each item (after
rotation) for pregnancy, labour and birth, and postnatal care phase separately. Factor loadings of
items that deviate from the dominant factor (i.e. the domain on which most of the items of
the domain loaded) are shown in italics. The factors that included two items or more had a
Cronbach's alpha varying between 0.68 and 0.89. From the EFA it appears that the factor
solution shows considerable commonality across the three phases.
The median score was 3.69 for the antepartum version (IQR 3.393.87). The median score
of the postpartum version was 3.74 (IQR 3.453.88). In Fig. 2 the global 10-point VAS rating
was related to the overall ReproQ client experience score, to determine the convergent validity.
A low VAS rating was associated with a lower ReproQ score in both the antepartum (r = 0.59;
p<0.001) and the postpartum questionnaire (r = 0.56; p<0.001).
The results on discriminative validity showed that overall, pregnant women and women
who recently had given birth had a similar overall ReproQ score (MD = 3.68, IQR = 3.403.87
vs. MD = 3.73, IQR = 3.443.88; p = 0.23), see Fig. 3. Domain-wise, Autonomy in pregnant
women was experienced better compared to Autonomy in women who recently had given
birth. Women who recently had given birth had better experiences with Prompt Attention.
The average score combining all domains per individual has a median of 3.68 (IQR = 3.403.87)
antepartum, and a median of 3.73 (IQR = 3.443.88) postpartum (p = 0.23). Domain-wise,
Autonomy, Respect, Confidentiality were experienced better in pregnant women compared women
who recently had given birth (p between 0.021 and <0.0001). Women who recently had given
birth had better experiences with Prompt Attention, Social Consideration and Choice and
Continuity (p between 0.033 and <0.0001).
Consistency of advice across professionals
Comprehensibility of explanation
Provision of information while treated
Access for appointment/contact without urgency
Responsive to client questions
Access for appointment/contact in urgent situations
Factor number and factor loading*
Adaptation: NC = no change; AI = adjusted item; AR = adjusted response mode; DD = assigned to different domain based on EFA; DEL = removed
* The last three columns represent 3 separate factor analyses. The number of the factor is listed in the order of the output (F1 = first factor, F2 = second
factor, etc). Only results with a factor loading > 0.3 are shown. If an item corresponds to a factor numbers in italics, then the factor analysis apparently
placed the item into another domain, as we assumed constructing the item cf. the WHO domain structure.
** For reasons of brevity, we indicate the contents of each item as a professional or maternity care organization characteristic. The question to the
respondent refers to the actual experience. E.g. the first item indicated with respecting privacy, in the ReproQ reads as Did the caregiver consider your
privacy during care provision?
Factor number and factor loading*
Women who perceived no health problems in their baby reported best overall ReproQ score
(see Fig. 4), independent whether their baby was hospitalized (MD = 3.78, IQR = 3.523.87) or
not (M = 3.72, IQR = 3.453.85). Within these groups, women with their baby hospitalized
showed more negative experiences concerning Dignity and Social Considerations than women
whose baby was not hospitalized. Women who did perceive health problems in their baby, but
whose baby was not hospitalized showed a lower median overall score (MD = 3.47, IQR =
3.163.80). Women whose baby had been hospitalized with (perceived) health problems
showed the lowest overall scores (MD = 3.42, IQR = 3.103.72). All domains and the overall
score differed significantly between the four subgroups (Kruskal-Wallis, all P<0.001).
During pregnancy, the overall ReproQ score of women who received care in a full integrated
facility (MD = 3.65, IQR = 3.373.86) showed no significant difference with women who
received care in a less integrated facility (MD = 3.74, IQR = 3.423.88; p = 0.14) (see Fig. 5). In
the delivery and postpartum phase women who received care in integrated facilities had a
Figure 2. Convergent validity: association between overall rating of maternity care, and ReproQ score (all domains combined). Fig. 2A shows the
results during the antenatal phase; Fig. 2B shows the results during labour and postnatal care. The overall rating (10-point VAS scale) was significantly
associated with the overall ReproQ score (i.e. the unweighted summation [range 14] of the individual eight domains), in both the antepartum (p<0.001) and
the postpartum phase (p<0.001).
slightly higher score compared to less integrated facilities (Md = 3.78, IQR = 3.533.90 vs. Md
= 3.63, IQR = 3.343.84; p<0.001). All domains except Choice and continuity (p = 0.062)
Based on the results of the statistical analyses we left 19 items unchanged; 10 items were
slightly rephrased; 3 items were deleted; 4 new items from the original item pool were added; 3
items were left unchanged but formally assigned to another domain; and the response mode of
4 items was slightly rephrased (see Table 2). No changes were needed after the second round of
stakeholders experts for final verification that wording of item and response were strictly
neutral and unequivocal for types of organizations. The questionnaires can be accessed online:
antepartum version, postpartum version.
We developed a self-report questionnaire, the ReproQ, to measure the performance of
maternity care from the perspective of clients. We used the WHO responsiveness model, which
evaluates experienced client interactions with professionals and the care providing organisation.
Content validity of the instrument was judged appropriate with balanced contribution of the
WHO domains. According to participants, more attention could be given to sensitivity for the
cultural background and traditions of the client, and the experienced professional expertise.
The null version of the questionnaire was adjusted based on all comments, but we did not
include a domain on the experienced expertise as in our opinion this should be primarily
reflected in the clinical outcome, which is difficult to assess by the client.
The construct validity of the improved ReproQ version 1 was established in a survey study
involving pregnant and recently delivering women. The response pattern showed overall good
to excellent overall ReproQ scores, averaging over domains. The effect that participants rarely
use the most negative response modus, is known from other self-report instruments in
maternity care and may be partially caused by the fact that pregnancy and childbirth are not a disease
and generally have good outcome [23,3436]. The most positive response category (always)
was used the most, demonstrating a ceiling effect also shown in the maternity experience
survey of the National Perinatal Epidemiology Unit .
The exploratory factor analyses largely confirmed the pre-stated domain structure.
However, the EFA strongly suggested to rearrange and reword items from the Confidentiality domain,
because these items loaded on different factors for the different stages of maternity care.
Testing convergent validity, we established a clear association between the overall VAS rating and
the overall ReproQ score.
The known group comparisons revealed literature-expected differences between women
perceiving good vs. bad outcomes in their baby, being aware that this may be a cross-over
effect rather than actually reflect poor responsiveness. In clients who received care in fully
integrated facilities vs. less integrated facilities, we observed differences during birth and postnatal
care, but not during pregnancy as might be expected as integration effects from the perspective
of the client is most clearly experienced at that stage.
The ReproQ focuses on the actual experiences of women with maternity care while existing
questionnaires mainly focus on procedural aspects . While following the adequate
procedure can contribute to responsiveness, it does not replace or predict care provision which is
client-centered. For example, the provision of written information can be a valuable standard
procedure, but it requires verification of utilization and understanding of the information.
The ReproQ is unique in the coverage of the eight responsiveness domains, which were all
considered valuable. The questionnaire of the National Health Service in the United Kingdom
included only 6 of the 8 Responsiveness domains, often using one specific item within a
domain. Prompt Attention was e.g. indicated by the item were you given the help you
needed?. This item was similar in the questionnaire of the National Perinatal Epidemiology
Unit concerning women's experience with maternity care . As it combines promptness and
perceived adequacy, response is difficult to interpret.
To prevent cross-over effects from labour and birth to the antenatal experiences , we
created two separate questionnaires to measure the experiences during pregnancy and the
experiences during delivery and postpartum care. This facilitates quality improvement as the
services involved usually are different.
This study had several limitations.
First, fewer clients participated in the group interviews than anticipated. In order to cover
all relevant perspectives and to maximize input on the issue of comprehensibility for the
deprived, we conducted additional individual interviews. In both forms all participants first wrote
down their individual comments (positive/negative/change) on contents and readability for
each questionnaire item of the ReproQ core separately. No discussion or exchange was allowed
in the group session at this stage. In the group sessions, these items were then presented one by
one, and discussed if asked for. The items were primarily adapted based on the detailed
individual written comments, which frequently converged; occasionally the plenary discussion was
used to solve an arbitrary wording choice. We assume the combination of group and individual
sessions did not compromise the results.
Second, there is no reference standard available to measure performance from the
perspective of clients, which makes it hard to establish the quality of the measured concept. We believe
however that the responsiveness model provides a solid conceptual base, confirmed by
extensive testing during its development and thereafter [6,13]. The comprehensiveness and
cross-cultural suitability has been confirmed in our study.
Third, women with a low educational level were underrepresented in our studies despite
repeated and considerable efforts to engage them. An explanation may be lack of perceived
control of these women, which is reflected in reluctance to participate: they do not believe that
participation or responding matters .
Fourth, a minority of the non-western women participated. This percentage (9%) is lower
than the percentage of non-Dutch pregnant women in the Netherlands (non-Dutch: 16%)
. Possible explanations include a language barrier [39,40] and our reliance on an
anonymous digital procedure. Perhaps the frequent coexistence of low education and non-Western
ethnicity plays a role . To increase their participation, the questionnaire could be adapted
by adding specific questions or by translating the questionnaire into other languages. For both
non-response prone groups the questionnaire could also be presented differently. For example,
with assistance of an independent third person, or by using a face-to-face interview. Another
option would be to ask key figures of their local society to promote participation.
The resulting questionnaire may be used in various types of evaluation studies, dedicated to
compare specific interventions or specific organization structures, or health care providers.
From its conceptual basea complement to medical outcomeit follows that outcomes, like
mortality of both mother and child, or compound measures like the Perinatal Adverse
Outcome Index  are unconditionally required for overall judgement. Interpretation of the
relevance of average ReproQ differences requires further study.
Further research Is needed on the discriminative capacity of the ReproQ to show differences
between care providers, and on the interpretation and relevance of observed differences. Also,
testing the proposed domain structure in a new sample using confirmatory factor analysis
We developed a client experience questionnaire (ReproQ) to measure maternity care
performance based on the WHO responsiveness model. After content analysis the improved ReproQ
questionnaire showed acceptable convergent and satisfactory discriminative validity.
Participation of disadvantaged groups in measurement of client experiences may require
We are grateful for all maternity care organizations that took part in our study, with special
thanks to Richard Pal, Jan Oostenbrink, Marlies Bartels, Hanneke de Graaf, Jannie de Vos,
Valentine Geluk, Anky van Hecke and Karin Scheele. They inspired the maternity care
organizations to participate and contributed to the translation of the WHO concept into the survey
presented. We thank Jacoba van der Kooy and Nicole Valentine (WHO) for their extensive
support designing the questionnaire. We also thank Marit Hitzert, who helped performing the
interviews and the survey study. In addition we are grateful for the support of Augustinus JP
Schrijvers and Eric AP Steegers for valuable organisational and professional support.
Conceived and designed the experiments: MS HFVS EB AF GJB. Performed the experiments:
MS HFVS EB GJB. Analyzed the data: MS HFVS EB GJB. Wrote the paper: MS HFVS EB AF
41. Egelhard D (2007) Met kennis van feiten. vluchtelingen, nieuwkomers en gezondheid in cijfers with
knowledge of facts: Figures about refugees, migrants and health]. Available: http://www.pharos.nl/
documents/doc/webshop/met_kennis_van_feiten.pdf via the Internet.
1. Zeitlin J , Mohangoo AD , Delnord M , Cuttini M ( 2013 ) The second european perinatal health report: Documenting changes over 6 years in the health of mothers and babies in europe . J Epidemiol Community Health 67 : 983 - 985 . doi: 10.1136/jech- 2013 -203291 PMID: 24052513
2. Mohangoo AD , Buitendijk SE , Szamotulska K , Chalmers J , Irgens LM , et al. ( 2011 ) Gestational age patterns of fetal and neonatal mortality in europe: Results from the euro-peristat project . PLoS.One . 6 : e24727. doi: 10.1371/journal. pone.0024727 PMID: 22110575
3. Zeitlin J , Szamotulska K , Drewniak N , Mohangoo AD , Chalmers J , et al. ( 2013 ) Preterm birth time trends in europe: A study of 19 countries . BJOG 120 : 1356 - 1365 . doi: 10.1111/ 1471 - 0528 . 12281 PMID: 23700966
4. de Graaf JP , Ravelli AC , de Haan MA , Steegers EA , Bonsel GJ ( 2013 ) Living in deprived urban districts increases perinatal health inequalities . J.Matern.Fetal Neonatal Med . 26 : 473 - 481 . doi: 10.3109/ 14767058.2012.735722 PMID: 23039164
5. de Graaf JP , Schutte JM , Poeran JJ , van Roosmalen J , Bonsel GJ , et al. ( 2012 ) Regional differences in dutch maternal mortality . BJOG 119 : 582 - 588 . doi: 10.1111/j.1471- 0528 . 2012 .03283.x PMID: 22329532
6. Valentine NB , de Silva A , Kawabata K , Darby C , Murray CJ , et al. ( 2003 ) Health system responsiveness: Concepts, domains and measurement . In: Murray CJ, Evans BE, editors. Health systems performance assessment: debates, methods and empiricism . Geneva: World Health Organization . pp. 573 - 596 .
7. Campbell SM , Roland MO , Buetow SA ( 2000 ) Defining quality of care . Soc Sci Med 51 : 1611 - 1625 . PMID: 11072882
8. Sitzia J , Wood N ( 1997 ) Patient satisfaction: A review of issues and concepts . Soc Sci Med 45 : 1829 - 1843 . PMID: 9447632
9. Wensing M , Jung HP , Mainz J , Olesen F , Grol R ( 1998 ) A systematic review of the literature on patient priorities for general practice care. part 1: Description of the research domain . Soc Sci Med 47 : 1573 - 1588 . PMID: 9823053
10. Williams B ( 1994 ) Patient satisfaction: A valid concept ? Soc Sci Med 38 : 509 - 516 . PMID: 8184314
11. de Silva A ( 2000 ) A framework for measuring responsiveness. global programme on evidence for health policy discussion paper 32 . Available: www.who. int/healthinfo/paper32.pdf via the Internet . 2014 .
12. Gostin L ( 2002 ) The domains of health responsiveness: A human rights assessment . Available: http://www.who. int/entity/hhr/Series_2%20Responsiveness.pdf?ua=1 via the Internet . 2014 .
13. Valentine NB , Bonsel GJ , Murray CJ ( 2007 ) Measuring quality of health care from the user's perspective in 41 countries: Psychometric properties of WHO's questions on health systems responsiveness . Qual Life Res 16 : 1107 - 1125 . PMID: 17624815
14. Offerhaus PM , Hukkelhoven CW , de Jonge A , van der Pal-de Bruin KM , Scheepers PL , et al. ( 2013 ) Persisting rise in referrals during labor in primary midwife-led care in the netherlands . Birth 40 : 192 - 201 . doi: 10.1111/birt.12055 PMID: 24635504
15. Christiaens W , Gouwy A , Bracke P ( 2007 ) Does a referral from home to hospital affect satisfaction with childbirth? A cross-national comparison . BMC Health Serv Res 7: 109. PMID: 17626631
16. Wiegers TA ( 2009 ) The quality of maternity care services as experienced by women in the netherlands . BMC Pregnancy Childbirth 9: 18. doi: 10.1186/1471-2393-9-18 PMID: 19426525
17. Wiegers TA , van der ZJ , Keirse MJ ( 1998 ) Transfer from home to hospital: What is its effect on the experience of childbirth? Birth 25 : 19 - 24 . PMID: 9534501
18. van der Velden K ( 2009 ) A good start. safe care during pregnancy and birth [een goed begin. veilige zorg rond zwangerschap en geboorte] . Available: http://www.rijksoverheid. nl/documenten-en-
19. Ravelli AC , Tromp M , van Huis M , Steegers EA , Tamminga P , et al. ( 2009 ) Decreasing perinatal mortality in the netherlands , 2000 - 2006 : A record linkage study . J Epidemiol Community Health 63 : 761 - 765 . doi: 10.1136/jech.2008.080440 PMID: 19416928
20. Bonsel GJ , Birnie E , Denktas S , Poeran J , Steegers EA ( 2010 ) Trends in the perinatal mortality. description study pregnancy and birth 2010 . PMID: 25506974
21. Grne O , Garcia-Barbero M ( 2002 ) Trends in integrated care-reflections on conceptual issues .
22. Hay H ( 2010 ) A report on the development of the questionnaire for the 2010 maternity survey <br /> . Available: http://www.nhssurveys.org/surveys/483 via the Internet. 2014 .
23. Redshaw M , Heikkila K ( 2010 ) Delivered with care: A national survey of women's experience of maternity care . Available: https://www.npeu.ox.ac. uk/delivered-with-care via the Internet .
24. Wiegers TA , Keirse MJ , Berghs GA , van der ZJ ( 1996 ) An approach to measuring quality of midwifery care . J Clin Epidemiol 49 : 319 - 325 . PMID: 8676180
25. van Wagtendonk I , van der Hoek, Wiegers TA ( 2010 ) Development of a consumer quality index of postnatal care . Available: http://www.zorginstituutnederland.nl/kwaliteit/toetsingskader+en+register/cqindex/cqi-vragenlijsten# CQIKraamzorg via the Internet .
26. Dzakpasu S , Kaczorowski J , Chalmers B , Heaman M , Duggan J , et al. ( 2008 ) The canadian maternity experiences survey: Design and methods . J.Obstet.Gynaecol.Can . 30 : 207 - 216 . PMID: 18364098
27. van der Kooy J , Valentine NB , Birnie E , Vujkovic M , de Graaf JP , et al. ( 2013 ) Validity of the ReproQ; a questionnaire measuring the world health organization concept of responsiveness in perinatal care . In: van der Kooy J , editor. pp. 151 - 174 . doi: 10.1007/s12288- 013 - 0266 -z PMID : 25566426
28. Christensen-Szalanski JJ ( 1984 ) Discount functions and the measurement of patients' values. women's decisions during childbirth . Med Decis Making 4 : 47 - 58 . PMID: 6727587
29. Murray CJ , Evans BE ( 2003 ) Health systems performance assessment: Debates, methods and empiricism . Geneva: World Health Organization.
30. BureauTaal . Dictionary language level B1 and A2 . Available: www.zoekeenvoudigewoorden.nl via the Internet.
31. Koopman L , Sixma H , Hendriks M , de Boer D , Delnoij D ( 2011 ) Manual CQI development: Guidelines and requirements for the development of a CQI measurement . Available: http://www. zorginstituutnederland.nl/kwaliteit/toetsingskader+en+register/cq-index/handboek+ontwikkelen+cqindex via the Internet.
32. Stevens JP ( 2009 ) Applied multivariate statistics for the social sciences . New York : Routledge.
33. Graham C , Maccormick S ( 2012 ) Overarching questions for patient surveys: Development report for the care quality commission (CQC).
34. Smith LF ( 2001 ) Development of a multidimensional labour satisfaction questionnaire: Dimensions, validity, and internal reliability . Quality in Health Care 10 : 17 - 22 . PMID: 11239139
35. Smith LF ( 2011 ) Postnatal care: Development of a psychometric multidimensional satisfaction questionnaire (the WOMBPNSQ) to assess women's views . Br J Gen Pract 61 : e628 - e637 . doi: 10.3399/ bjgp11X601334 PMID: 22152835
36. Peterson WE , Charles C , DiCenso A , Sword W ( 2005 ) The newcastle satisfaction with nursing scales: A valid measure of maternal satisfaction with inpatient postpartum nursing care . J Adv Nurs 52 : 672 - 681 . PMID: 16313380
37. Bobak M , Pikhart H , Hertzman C , Rose R , Marmot M ( 1998 ) Socioeconomic factors, perceived control and self-reported health in russia. A cross-sectional survey . Soc Sci Med 47 : 269 - 279 . PMID: 9720645
38. Ravelli AC , Eskes M , Tromp M , van Huis AM , Steegers EA , et al. ( 2008 ) Perinatal mortality in the netherlands 2000-2006; risk factors and risk selection] . Ned Tijdschr Geneeskd 152 : 2728 - 2733 . PMID: 19192586
39. Fransen M , Harris VC , Essink-Bot ML ( 2013 ) Low health literacy in ethnic minority patients: Understandable language is the beginning of good healthcare] . Ned Tijdschr Geneeskd 157 : A5581. PMID: 23548185
40. Agyemang C , Denktas S , Bruijnzeels M , Foets M ( 2006 ) Validity of the single-item question on selfrated health status in first generation turkish and moroccans versus native dutch in the netherlands . Public Health 120 : 543 - 550 . PMID: 16684550
42. Mann S , Pratt S , Gluck P , Nielsen P , Risser D , et al. ( 2006 ) Assessing quality obstetrical care: Development of standardized measures . Jt.Comm J.Qual.Patient Saf 32 : 497 - 505 . PMID: 17987873