Essential metrics for assessing sex & gender integration in health research proposals involving human participants
Essential metrics for assessing sex & gender integration in health research proposals involving human participants
Suzanne Day 0 1 2
Robin Mason 0 1 2
Cara Tannenbaum 0 2
Paula A. Rochon 0 1 2
0 Funding: PR, RM, and SD are members of Women's Xchange, which is funded by the Ontario Ministry of Health and Long-Term Care (grant
1 Women's Xchange, Women's College Hospital , Toronto, Ontario , Canada , 2 Women's College Research Institute, Women's College Hospital , Toronto, Ontario , Canada , 3 Dalla Lana School of Public Health and Department of Psychiatry, University of Toronto , Toronto, Ontario , Canada , 4 Institute of Gender and Health, Canadian Institutes of Health Research , Montreal, Quebec , Canada , 5 Department of Medicine, University of Toronto , Toronto, Ontario , Canada
2 Editor: Pavel V. Ovseiko, University of Oxford , UNITED KINGDOM
Integrating sex and gender in health research is essential to produce the best possible
evidence to inform health care. Comprehensive integration of sex and gender requires
considering these variables from the very beginning of the research process, starting at the
proposal stage. To promote excellence in sex and gender integration, we have developed a
set of metrics to assess the quality of sex and gender integration in research proposals.
These metrics are designed to assist both researchers in developing proposals and
reviewers in making funding decisions. We developed this tool through an iterative three-stage
method involving 1) review of existing sex and gender integration resources and initial
metrics design, 2) expert review and feedback via anonymous online survey (Likert scale and
open-ended questions), and 3) analysis of feedback data and collective revision of the
metrics. We received feedback on the initial metrics draft from 20 reviewers with expertise in
conducting sex- and/or gender-based health research. The majority of reviewers responded
positively to questions regarding the utility, clarity and completeness of the metrics, and all
reviewers provided responses to open-ended questions about suggestions for
improvements. Coding and analysis of responses identified three domains for improvement:
clarifying terminology, refining content, and broadening applicability. Based on this analysis we
revised the metrics into the Essential Metrics for Assessing Sex and Gender Integration in
Health Research Proposals Involving Human Participants, which outlines criteria for
excellence within each proposal component and provides illustrative examples to support
implementation. By enhancing the quality of sex and gender integration in proposals, the metrics
will help to foster comprehensive, meaningful integration of sex and gender throughout each
stage of the research process, resulting in better quality evidence to inform health care for
#06667, URL: http://www.health.gov.on.ca/en/)
and by the Ontario Strategy for Patient Oriented
Research (SPOR) Support for People and
PatientOriented Research and Trials (SUPPORT) Unit
(grant #1014384, URL: http://ossu.ca/). The
funders had no role in study design, data collection
and analysis, decision to publish, or preparation of
Competing interests: PR, RM, and SD are
members of Women's Xchange, a women's health
research knowledge translation and exchange
centre based at Women's College Hospital in
Toronto, Canada. A mandate of Women's Xchange
is to enhance the integration of sex and gender in
health research. This does not alter our adherence
to PLOS ONE policies on sharing data and
Both sex and gender are fundamental to consider in the design of health research in order to
produce the most complete and accurate evidence to inform health care and improve patient
outcomes [1±3]. The terms `sex' and `gender' refer to two distinct but interrelated factors that
shape health: sex encompasses a set of biological attributes such as chromosomes, gene
expression, and anatomy, while gender refers to socioculturally-constructed roles, behaviours,
norms, identities, and power relations [
]. Despite their importance, the integration of sex and
gender in health research has yet to be consistently adopted as a standard research and
reporting practice [5±9]. This indicates a need for tools and strategies to help researchers enhance
the integration of sex and gender in their work.
Some recently developed tools can help guide sex and gender inclusion in the reporting of
research findings in scientific publications, such as the Sex and Gender Equity in Research
(SAGER) guidelines [
], and criteria for inclusion of sex and gender in Cochrane systematic
]. To conduct robust analyses, sex and gender need to be considered from the
outset and integrated throughout the entire study [
], from research questions through methods,
analysis plan, and dissemination strategy. Amidst growing interest on the part of federal health
research funding agencies to include both sex and gender as a condition of funding [
researchers and reviewers would benefit from a way to assess the strength of that integration in
Some tools have been developed for this purpose. For example, the National Institutes of
Health Research provides a decision tree for reviewers assessing the integration of sex as a
biological variable in proposals [
]. This decision tree is limited to assessing only the integration
of sex and not gender in the proposed research design and planned analyses. GENDER-NET, a
European Research Area Network focusing on the integration of sex and gender analysis, has
developed a set of guidelines and checklists for Integrating Gender Analysis in Research
(IGAR) for both research applicants [
] and application reviewers [
]. These checklists do
not include an assessment scale, and thus can only record the presence or absence of
sex/gender considerations. Similarly, TomaÂs et al.'s 10-item questionnaire, Gender Perspectives in
Health Research [
], permits only `yes' or `no' responses to a series of questions on sex and
gender integration and thus cannot assess integration quality. Additionally, as the authors
themselves note [
], reviewers require training in a gender perspective in order to
appropriately use the questionnaire, posing a barrier to its widespread uptake.
With the objective of addressing the need for a comprehensive tool to assess the quality of
sex and gender integration in health research proposals, we developed the Essential Metrics for
Assessing Sex & Gender Integration in Health Research Proposals Involving Human Participants.
This tool will be useful to both researchers preparing proposals and reviewers assessing
proposals for funding opportunities. In this paper we detail the methods of our metrics
development study and present the resulting tool.
An iterative 3-stage method was used to develop the metrics. The Research Ethics Board at
Women's College Hospital reviewed and provided ethics exemption for this study.
Review of existing tools
We began by compiling existing guidelines and tools for sex and gender integration. A selected
bibliography was developed of 19 sex and gender integration resources that our group had
previously used in our capacity as a women's health research knowledge translation and exchange
centre [1, 18±35]. This initial bibliography was then presented during a 2-day meeting of our
2 / 13
research team. At this meeting, discussions focused on definitions of sex and gender, contexts
for promoting sex and gender integration, and assessing the existing resources for sex- and
gender-based analysis, adding to the initial bibliography based on the collective knowledge of
the research team. The research process was mapped from start to finish to identify points
where sex and gender should be integrated. Drawing upon five key publications unanimously
agreed upon at our meeting to be the most comprehensive instructional tools available at the
14, 23, 29, 36, 37
], we developed an initial set of metrics designed to consider sex and
gender at six pivotal points in the research process: literature review and research question
development, research design and methods, data analysis, data reporting, knowledge
translation planning, and patient engagement. In line with recent recommendations for translating
sex- and gender-inclusive policy into practice [
], a descriptive assessment rather than a
numeric scoring system was devised. Five assessment categories were provided for each
proposal component, scaled from ªpoorº to ªoutstandingº.
Expert review and feedback
Following methods used to develop sex and gender reporting guidelines [
], we used
purposive sampling to recruit Canadian and international health researchers with expertise in
conducting sex- and/or gender-based analysis to review and provide feedback on the draft
metrics. Potential reviewers were identified through a variety of means, including: 1)
membership with the International Society for Gender Medicine and its affiliated national societies in
Austria, the USA, Germany, Israel, Italy, Japan, and Sweden ; 2) involvement in
developing existing tools for sex and gender integration in health research, such as those used to
develop the metrics [
14, 23, 29, 36, 37
]; and 3) known associations among our team members'
research networks. Eligibility criteria included possession of an advanced degree (PhD or MD)
and authorship of instructional resources and/or peer-reviewed journal articles on the
integration of sex and gender considerations in health research.
A total of 55 individuals with expertise in sex- and gender-based health research were
invited to review and provide feedback on the metrics, including 36 Canadian researchers and
19 international researchers. To encourage participation in the review process, invitees
received a personalized email [
] with a link to a questionnaire hosted on a free online survey
Survey questions were developed based on a consensus discussion among our team as to
what kinds of feedback would be most helpful for comprehensively revising the metrics. From
this discussion we devised a survey composed of a mix of Likert scale responses and
openended questions. The Likert scale questions solicited reviewers' opinions on whether the
metrics were clear, useful and complete. The open-ended questions allowed participants an
opportunity to offer specific recommendations for improvement by asking ªWhat additional
questions should we include?º, ªHow could the metrics be made clearer and easier to use?º
and ªWhat additional comments/feedback do you have about the metrics?º. We also collected
information about reviewers' disciplinary backgrounds to better contextualize the
recommendations and to verify that feedback was gathered from differing perspectives and a range of
expertise. All responses were collected anonymously in order to reduce the risk of social
desirability bias influencing reviewers' reactions to the metrics.
Qualitative analysis of feedback and revision
Expert reviewers' responses were analyzed and used to inform subsequent revisions of the
metrics. All survey responses were read and independently coded by two of the authors (SD and
RM) using open codes to identify emergent themes. Open codes were compared and through
3 / 13
discussion, consensus was reached on their synthesis into broader categories [
categories reflect the expert reviewers' primary recommendations, indicating the main changes that
needed to be made to the tool. Over the course of several meetings, SD, RM, and PR discussed
the categories and primary recommendations, and drafted and reviewed subsequent iterations
to finalize the metrics.
No patients were involved in the design of this study, and there are no plans to disseminate the
findings to patient groups.
Expert review and feedback
We received a total of 20 completed online survey responses from 10 Canadian and 10
international (non-Canadian) experts in sex- and gender-based health research. Reviewers
represented diverse disciplinary backgrounds, including epidemiology, physiology, pharmacology,
neuroscience, ergonomics, social sciences, the humanities, medicine, and nursing. In their
own research, 11 respondents indicated that they use both qualitative and quantitative
methods, while 8 reported using quantitative methods exclusively and 1 reported using qualitative
methods exclusively. In Likert scale responses, most reviewers indicated that they agreed or
strongly agreed with questions on the clarity, completeness and utility of the metrics (Fig 1).
Analysis of feedback and revision
All reviewers responded to the three open-ended survey questions. Open coding of this data
yielded three broad categories of recommendations, indicating a need for: 1) clarifying
Fig 1. Reviewer responses to Likert scale questions on the metrics. Fig 1 shows the number of respondents that indicated strongly disagree,
disagree, neither agree nor disagree, agree, and strongly disagree in response to each of the 5 survey questions.
4 / 13
terminology, 2) refining content, and 3) broadening applicability. Regarding terminology,
reviewers noted some words were too vague, confusing or required further explanation. For
example, use of the phrase ªdiverse populationsº was confusing as there are many ways to
address diversity (e.g. racial diversity, diverse socio-economic backgrounds, etc.). A fifth
assessment level, ªoutstandingº, created challenges due to difficulties distinguishing between
ªoutstandingº and ªexcellentº. In terms of content, additional points where sex and gender
integration should be evaluated (such as in planning for retention of participants) were
proposed. Finally, recommendations for broadening applicability of the metrics to a wider range
of study types and research methodologies, such as single-sex studies and qualitative designs,
were supplied by several reviewers.
Based on this analysis of reviewers' recommendations, we revised the metrics in a series of
iterations. Terms flagged by reviewers as being unclear were removed or replaced with clearer
language. The assessment scale was reduced to 4 levels, and examples were embedded within
the tool to help assess each specific measure as well as to illustrate how the metrics can be
applied to different kinds of studies. Our analysis of reviewers' feedback also indicated the
need for a contextualizing set of instructions in order to interpret and effectively apply the
Instructions for metrics use
The metrics are to be used by health researchers as they prepare their funding proposals, as
well as by health research funding bodies as they review proposals submitted for funding. In
applying the metrics, researchers and reviewers should keep four key considerations in mind:
· First, researchers and reviewers should be aware of the difference between the concepts of
sex and gender, with sex referring to biological factors and gender referring to sociocultural
· Second, investigating both sex and gender may not be appropriate for all study analyses. The
relevance of sex and gender will depend upon whether the study proposes to investigate
biological or sociocultural factors [
]. For example, qualitative studies of persons' experiences
with health and illness are more amenable to studying the impact of gender, while clinical
trials of drug therapies are more amenable to study the impact of sex. To accommodate these
possibilities, we used the term ªsex/genderº±not to imply that the two concepts are
synonymous, but to serve as shorthand for ªand/orº throughout the metrics.
· Third, not all elements of the metrics will apply to all types of research studies, nor all study
designs. For example, some of the assessment categories will not be applicable to qualitative
studies (such as ensuring sufficient sample size for powering statistical analyses).
· Finally, the metrics are not only applicable to studies that investigate sex/gender differences,
but are equally useful in study proposals focused on one sex or gender. Understanding the
role of these factors in shaping health experiences and outcomes is also important for
revealing within-group differences in sex- or gender-specific studies [
Essential metrics for assessing sex & gender integration in health research proposals involving human participants
The final version of the metrics is presented in Table 1.
As indicated in the first column of Table 1, the metrics are divided into 3 primary sections
reflecting the stages of research development: 1) Literature Review & Research Objectives, 2)
5 / 13
2. Research Design, a) Population
Methods, & · Has sex/gender been considered in the
Analysis Plan inclusion and exclusion criteria?
· Has sex/gender been considered in the
calculation of sample size?
· Are some populations inappropriately excluded on the
basis of sex/gender by the inclusion/exclusion criteria?
· To what extent has justification been provided for the
inclusion/exclusion of populations based on sex/
· Has the sample size been sufficiently powered to
identify potentially relevant sex/gender findings?
· Excellent: substantial consideration of sex/gender
throughout the literature review; explicit and thorough
exploration of sex/gender identified in research
· Good: some consideration of sex/gender in
literature review and research objectives, but with
some potential to be expanded.
· Fair: minimal consideration of sex/gender in
literature review and research objectives, numerous
critical gaps remain.
· Poor: sex/gender not considered at all in the
literature review and research objectives.
· Excellent: substantial justification provided for sex/
gender related inclusion/exclusion criteria.
· Good: clear justification provided for sex/gender
related inclusion/exclusion criteria, but with some
potential to be expanded.
· Fair: minimal justification provided for sex/gender
related inclusion/exclusion criteria, numerous critical
· Poor: no justification for sex/gender related
· Sufficient sample size to allow for potential sex/
b) Participant Recruitment &
· Has sex/gender been considered in the
recruitment and retention strategies to
ensure as broad as possible study
c) Data Collection Tools:
· Do the data collection tools capture
information relevant to sex/gender?
· Are there potential sex/gender related barriers to · Excellent: consideration of sex/gender in
participation for some populations who should be recruitment and retention strategies will ensure the
included? Consider the accompanying recruitment broadest possible participation by study population(s).
materials; for example, where will recruitment posters be · Good: some consideration given to sex/gender in
located, and who is represented on the posters? the recruitment and retention strategies, but there
· Are there potential sex/gender related barriers to remains some potential to further broaden
retention? For example, will child care be available or participation.
travel costs offset for participants? Is consideration · Fair: minimal inclusion of sex/gender in the
given to sex-specific dosing strategies to prevent recruitment and retention strategies may result in
adverse events that may be common in males/females? numerous populations being inappropriately
· Poor: sex/gender not considered in recruitment and
· Not Applicable: No recruitment needed.
· Do the participant intake forms and other tools (e.g. · Excellent: all tools reflect the widest possible range
questionnaires, interview guides) capture sex (e.g. male, of sex/gender identities; tools will collect extensive
female) and/or gender identities (e.g. man, woman, data relevant to conducting sex/gender analyses.
transgender, two-spirit, etc.)? · Good: tools reflect a range of sex/gender identities
· Will the tools used to collect data include variables to and will collect some data relevant to sex/gender, but
conduct analyses of the influence of sex/gender? with potential to be expanded.
· Fair: tools reflect minimal data on sex/gender
identities and numerous critical gaps remain in the
collection of sex/gender relevant data.
· Poor: no sex/gender data will be collected.
d) Data Analysis Plan:
· Does the proposal include a plan to
analyze the impact of sex/gender on
· Will key variables be analyzed and reported
disaggregated by sex/gender?
· In single-sex/gender studies: is there a plan to
investigate differences within this population?
· Has sex/gender been considered in the
knowledge translation plan?
· Are knowledge translation strategies customized for
relevance to a range of populations? For example, is
there a plan to present findings relevant to specific
participant populations based on sex/gender?
· Excellent: all data will be analyzed in relation to sex/
· Good: most data will be analyzed in relation to sex/
gender, but there is some potential to further expand
the sex/gender analyses.
· Fair: minimal data will be analyzed in relation to sex/
gender, numerous critical gaps remain.
· Poor: no inclusion of sex/gender in the analysis
· Excellent: the knowledge translation plan explicitly
notes sex/gender considerations and is tailored to the
widest possible range of populations.
· Good: the knowledge translation plan shows some
consideration of sex/gender, but there is some
potential to expand the possibilities for tailoring to a
range of populations.
· Fair: numerous critical gaps remain in the
knowledge translation plan's consideration of sex/
gender and tailoring to a range of populations.
· Poor: no consideration of sex/gender in the
knowledge translation plan.
6 / 13
Research Design, Methods, & Analysis Plan, and 3) Knowledge Translation Plan. For each of
these three sections there is a question or series of subsection questions on sex/gender
integration, examples to illustrate each question, and a descriptive assessment scale to evaluate the
extent to which the proposal addresses the questions. Each of the three sections and the
rationale for their inclusion are explored in detail below.
Section 1: literature review & research objectives. Questions asked: Does the literature
review include consideration of sex/gender? Do the research objectives include exploration of
Examples: Are current knowledge, gaps or new questions about sex/gender raised in the
literature review? Are gaps or new questions about sex/gender addressed in the research
Rationale: Inclusion of sex/gender in the literature review is essential to determining what
is already known about the impact of these factors on the proposed topic of study [
Excellent inclusion of sex/gender in the review should not only outline what is presently known
about the incidence, prevalence, risk factors, pathophysiology, treatment and experience of
health and illness, but also gaps in our understanding and questions that might arise from
these gaps [
]. The literature review should be used to inform the basis of an explicit
exploration of sex/gender in the proposed research objectives.
Section 2: research design, methods, & analysis plan. a) Population
Questions asked: Has sex/gender been considered in the inclusion and exclusion criteria?
Has sex/gender been considered in the calculation of sample size?
Examples: Are some populations inappropriately excluded on the basis of sex/gender by the
inclusion/exclusion criteria? To what extent has justification been provided for the inclusion/
exclusion of populations based on sex/gender? Has the sample size been sufficiently powered
to identify potentially relevant sex/gender findings?
Rationale: The proposal should identify and thoroughly justify any inclusion and/or
exclusion criteria based on sex/gender; consideration should be given to whether some populations
have been inappropriately excluded on the basis of sex/gender considerations [
Additionally, other factors that intersect with sex and/or gender that are of relevance to the research
question should be noted. For example, if the inclusion criteria exclude populations based on
age cut-offs, consider potential sex-based differences in the age of onset for the condition
under study [
]. The proposed sample size should be sufficient to permit analyses
illuminating sex/gender findings; in the case of quantitative research, this will require consideration of
whether the sample size is sufficiently powered, while in qualitative research the sample size
needs to be large enough to capture a range of experiences among participants. Note that the
question of sufficient sample size is the only metric for which it is not possible to assess relative
quality; it is either sufficient or not sufficient.
b) Participant Recruitment & Retention (if applicable)
Questions asked: Has sex/gender been considered in the recruitment and retention
strategies to ensure as broad as possible study participation?
Examples: Are there potential sex/gender related barriers to participation for some
populations who should be included? Consider the accompanying recruitment materials; for
example, where will recruitment posters be located, and who is represented on the posters? In terms
of retention, will child care be available or travel costs offset for participants? Is consideration
given to sex-specific dosing strategies to prevent adverse events that may be common in males/
Rationale: The proposed study recruitment and retention strategies should outline possible
sex-/gender-based barriers and provide a clearly articulated plan to ensure the broadest
possible range of participation. For example, gendered differences in help-seeking and health care
7 / 13
] may impact opportunities for recruiting men into the study sample. Similarly,
the gendered division of labour in the home [
] may warrant offering childcare or flexible
participation hours to recruit and retain women participants. This section will not be relevant
for studies that do not involve recruitment of human subjects, which should be evaluated as
ªnot applicableº in the accompanying assessment scale.
c) Data Collection Tools
Questions asked: Do the data collection tools capture information relevant to sex/gender?
Examples: Do the participant intake forms and other tools (e.g. questionnaires, interview
guides) capture sex (e.g. male, female) and/or gender identities (e.g. man, woman, transgender,
two-spirit, etc.)? Will the tools used to collect data include variables to conduct analyses of the
influence of sex/gender?
Rationale: Data collection tools such as surveys, focus group questionnaires, patient intake
and demographic forms should be designed to capture an array of relevant sex/gender
information, allowing for robust analyses of the influence of sex/gender on study outcomes [
Additionally, researchers are urged to move beyond the sex and gender binary of male/female
and man/woman in designing data collection tools. These categories do not the capture the
full range of possibilities in human expressions of sex and gender, which includes intersex and
transgender individuals [
d) Data Analysis Plan
Questions asked: Does the proposal include a plan to analyze the impact of sex/gender on
Examples: Will key variables be analyzed and reported disaggregated by sex/gender? In
single-sex/gender studies: is there a plan to investigate differences within this population?
Rationale: The proposal should include an explicit plan to conduct sex/gender analyses,
ideally by disaggregating and analyzing all research results by sex/gender [
]. Controlling for
sex/gender means that valuable information may be lost regarding whether and how the study
outcomes differ by sex/gender [
]. The proposal should thus clearly articulate how the study's
analyses will investigate the potential impact of sex/gender on study findings [
]. In studies
specific to only one sex or gender, there should be a clearly articulated plan to analyze and
report differences among the study population [
Section 3: knowledge translation plan. Questions asked: Has sex/gender been considered
in the knowledge translation plan?
Examples: Are knowledge translation strategies customized for relevance to a range of
populations? For example, is there a plan to present findings relevant to specific participant
populations based on sex/gender?
Rationale: The knowledge translation plan should reflect sex/gender considerations in
strategies for disseminating the research findings [
]. For example, a proposal with an excellent
knowledge translation plan would articulate a clear plan for selecting and tailoring
interventions based on sex/gender, accounting for potential sex/gender barriers and facilitators to the
knowledge translation process [
The Essential Metrics for Assessing Sex & Gender Integration in Health Research Proposals Involving
Human Participants are among the first to provide a comprehensive guide for assessing the quality
of sex and gender integration in health research proposals. The metrics are designed for use by
researchers preparing proposals and by reviewers evaluating proposals for funding opportunities.
Funding agencies require strategies for encouraging meaningful and thorough consideration of
sex and gender in grant applications [
]. Our metrics are one such mechanism, providing criteria
8 / 13
for proposal reviewers on what constitutes high-quality integration. Further, as evidenced by the
inconsistent and uneven integration of sex and gender across the fields of health research [5±9],
researchers developing a proposal require criteria for `best practices' in sex and gender integration.
While we recognize that these metrics do not necessarily resolve all of the challenges of sex and
gender inclusion [
], they do provide a practical strategy for helping researchers and
reviewers think beyond a tokenistic or `checkbox' approach. The metrics also complement funding
agencies' efforts to ensure that reviewers have sufficient instruction in and understanding of sex and
gender integration when evaluating applications .
Health researchers have been challenged to find ways to reduce research ªwasteº through
better consideration of what evidence is already known, increasing the yield of research
evidence, and enhancing the applicability of their research[
]. Identifying opportunities to
include sex and gender at the proposal stage can help reduce potentially avoidable research
waste by both building our knowledge of how sex and gender impact health, as well as
enhancing evidence to inform health practice and policy. There is substantial evidence to show that
prioritizing the integration of sex and gender±and considering these factors throughout the
research process±leads to innovative evidence to improve the health of all [
]. For example,
studying the influence of sex and gender on pharmacokinetics has advanced our
understanding of differences in drug tolerability and efficacy, helping to inform safer and more effective
]. Sex- and gender-based disparities have been noted in the manifestation and
management of cardiovascular disease, revealing crucial opportunities to enhance treatment
]. Investigating how sex and gender inform risk and protective factors in dementia and
Alzheimer's disease has also helped to establish recommendations for innovative future research
]. Use of the metrics may help ensure similarly promising lines of inquiry are pursued
in all fields of health research.
The metrics address the limitations we have identified in the few existing decision-making
aides for reviewing proposals [15±18], providing benchmarks for excellence in sex and gender
integration as well as an assessment scale for evaluating quality. To the best of our knowledge
the closest comparable evaluation strategy is offered by TomaÂs et al.'s questionnaire, which
allow reviewers to order proposals into three ªlevelsº of integration: ªdifference by sexº (basic
research that disaggregates findings by sex), ªgender sensitiveº (research that looks at the
impact of gender differences on health), and ªfeminist researchº (research that addresses
health inequalities). These are more accurately described as categories or types of research
rather than levels of integration, given that whether a proposed study will examine sex, gender,
or both depends on the purpose of the study and whether biological or
psychosocial/sociocultural factors or both are relevant to that purpose [
]. In contrast, our metrics are broadly
applicable to evaluating integration quality within each phase of the research process and across a
broad range of study designs, regardless of study purpose.
A strength of the metrics is that they have been developed on the basis of published
resources for conducting sex and gender integration, as well as with input from international
experts in sex- and gender-based health research. A potential limitation is that, as our
reviewers of the metrics were sex and gender health research experts, their responses to our review
and feedback survey may have been biased towards reporting the metrics as useful, clear and
complete. To address this potential limitation we have taken steps to assist non-expert users,
including providing contextualizing instructions for appropriate use of the metrics as well as
examples to illustrate the kinds of factors that should be considered in evaluating each proposal
component. Additionally, limiting the assessment scale to four clearly differentiated and
descriptive levels of quality is intended to help all reviewers develop consistency in their
evaluation of proposals. We will continue to update the metrics as new information becomes
available and the metrics are applied by different groups in practice.
9 / 13
The Essential Metrics for Assessing Sex & Gender Integration in Health Research Proposals
Involving Human Participants is a scaled assessment tool that can be used by both reviewers
and researchers to evaluate the quality of sex and gender integration in any health research
proposal. The metrics support growing efforts to promote incorporation of sex and gender as
a means of enhancing the value, applicability and quality of health research. By enhancing the
integration of sex and gender in proposals, use of the metrics will help to foster high-quality,
meaningful integration of sex and gender throughout the research process, resulting in better
quality evidence to inform health care for all.
The authors gratefully acknowledge the valuable feedback provided by all reviewers of the
metrics, including Dr. Cornelia M. Borkhoff, Dr. Marion Doull, Dr. Angel Foster, Dr. Lorraine
Greaves, and Dr. Nazilla Khanlou. The authors also thank Stephanie Lagosky and Marni Wolf
for their contributions in compiling the initial bibliography of sex and gender integration
resources and the first metrics design stage, and Dr. An-Wen Chan for his valuable feedback
in drafting the final version of the metrics. Authors who are members of Women's Xchange
(SD, RM, and PR) acknowledge the support of the Ontario Ministry of Health and Long-Term
Care and the Ontario Strategy for Patient Oriented Research (SPOR) Support for People and
Patient-Oriented Research and Trials (SUPPORT) Unit. PR holds the Retired Teachers of
Ontario (RTO/ERO) Chair in Geriatric Medicine at the University of Toronto.
Conceptualization: PR RM CT.
Formal analysis: SD RM PR.
Investigation: SD RM PR.
Methodology: SD RM PR.
Supervision: RM PR.
Writing ± original draft: SD.
Writing ± review & editing: SD RM PR CT.
10 / 13
11 / 13
WHO Department of Gender WaH. Gender analysis in health: A review of selected tools. Geneva:
World Health Organization; 2002.
12 / 13
1. Coen S , Banister E. What a difference sex and gender make: a gender, sex and health research casebook . Ottawa: Canadian Institutes of Health Research; 2012 .
2. Greaves L. Why put gender and sex into health research ? In: Oliffe JL , Greaves L , editors. Designing and conducting gender, sex and health research. Thousand Oaks: Sage Publications Inc.; 2011 . p. 3 ± 14 .
3. Legato MJ , Johnson PA , Manson JE . Consideration of sex differences in medicine to improve health care and patient outcomes . JAMA . 2016 . https://doi.org/10.1001/jama. 2016 .13995 PMID: 27802499
4. Clayton J , Tannenbaum C . Reporting sex, gender, or both in clinical research ? JAMA. 2016 . https://doi. org/10.1001/jama. 2016 .16405 PMID: 27802482
5. Johnson J , Sharman Z , VissandjeeÂ B , Stewart DE . Does a change in health research funding policy related to the integration of sex and gender have an impact? PLOS ONE . 2014 ; 9 ( 6 ):e99900. https:// doi.org/10.1371/journal.pone. 0099900 PMID: 24964040
6. Oertelt-Prigione S , Parol R , Krohn S , Preiûner R , Regitz-Zagrosek V . Analysis of sex and gender-specific research reveals a common increase in publications and marked differences between disciplines . BMC Med . 2010 ; 8 : 70 . https://doi.org/10.1186/ 1741 -7015-8-70 PMID: 21067576
7. Philips SP , Hamberg K. Doubly blind: a systematic review of gender in randomised controlled trials . Glob Health Action . 2016 ; 9 : 29597 .
8. Foulkes M. After inclusion, information and inference: reporting on clinical trial results after 15 years of monitoring inclusion of women . J Womens Health . 2011 ; 20 ( 6 ): 829 ± 36 .
9. Greyson DL , Becu AR , Morgan SG . Sex, drugs and gender roles: mapping the use of sex and gender based analysis in pharmaceutical policy research . Int J Equity Health . 2010 ; 9 ( 1 ): 1 .
10. Heidari S , Babor TF , De Castro P , Tort S , Curno M. Sex and gender equity in research: rationale for the SAGER guidelines and recommended use . Res Integr Peer Rev . 2016 ; 1 ( 1 ): 1 .
11. Doull M , Runnels VE , Tudiver S , Boscoe M. Appraising the evidence: applying sex- and gender-based analysis (SGBA) to Cochrane Systematic Reviews on cardiovascular diseases . J Womens Health . 2010 ; 19 ( 5 ): 997 ± 1003 .
12. Schiebinger L , Klinge I. Gendered innovation in health and medicine . Gender: Zeitschrift fuÈr Geschlecht, Kultur, und Gesellschaft 2015 ; 2 : 29 ± 50 .
13. National Institutes of Health. Consideration of sex as a biological variable in NIH funded research 2015 [cited 2016 October 24 ]. Available from: http://grants.nih.gov/grants/guide/notice-files/NOT-OD- 15 - 102.html.
14. Canadian Institutes of Health Research. Gender, sex and health research guide: a tool for CIHR applicants 2010 [ cited 2016 August 5] . Available from: http://www.cihr-irsc.gc.ca/e/32019.html.
15. National Institutes of Health. Reviewer guidance to evaluate sex as a biological variable (SABV) 2016 [ cited 2016 November 17] . Available from: https://grants.nih.gov/grants/peer/guidelines_general/ SABV_ Decision_Tree_for_Reviewers .pdf.
16. GENDER-NET . Recommendations for integrating gender analysis into research: grant applicants 2016 [cited 2016 December 22] . Available from: http://igar-tool.gender-net.eu/en/target/grant-applicants.
17. GENDER-NET . Recommendations for integrating gender analysis into research: peer reviewers/evaluators 2016 . Available from: http://igar-tool.gender-net.eu/en/target/peer-reviewers-evaluators.
18. TomaÂs C , Yago T , Eguiluz M , Samitier ML , Oliveros T , Palacios G. A tool to assess sex-gender when selecting health research projects . Aten Primaria . 2015 ; 47 ( 4 ): 220 ±7. https://doi.org/10.1016/j.aprim. 2014 . 05 .010 PMID: 25500216
19. National Collaborating Centre for Methods and Tools. PROGRESS framework: Applying an equity lens to interventions 2015 . Available from: http://www.nccmt.ca/resources/search/234.
20. Doull M , Shea B , Puil L , Runnels V , Welch V , Tudiver S , et al. Addressing Sex/Gender in Systematic Reviews: Cochrane HIV/AIDS Group Briefing Note 2014. Available from: http://equity.cochrane.org/ sex-and -gender-analysis.
21. Bottorff JL , Oliffe JL , Kelly MT , Chambers NA . Approaches to examining gender relations in health research . In: Oliffe JL , Greaves L , editors. Designing and Conducting Gender , Sex, and Health Research. London: Sage; 2012 . p. 175 ± 88 .
22. Ueffing E , Tugwell P , Welch V , Petticrew M , Kristjansson E. Equity checklist for systematic review authors: Campbell & Cochrane Equity Methods Group,; 2012 . Available from: http://methods. cochrane.org/sites/methods.cochrane.org.equity/files/public/uploads/EquityChecklist2012.pdf.
23. Doull M , Runnels VE , Tudiver S , Boscoe M. Sex and gender in systematic reviews: planning tool: The Campbell and Cochrane Equity Methods Group 2011 [ cited 2016 August 17] . Available from: http:// methods.cochrane.org/equity/sites/methods.cochrane.org.equity/files/uploads/SRTool_ PlanningVersionSHORTFINAL.pdf.
24. Office of Research on Women's Health. Moving into the future with new dimensions and strategies: A vision for 2020 for women's health research . National Institutes of Health , 2010 .
25. Clow B , Pederson A , Haworth-Brockman M , Bernier JI . Rising to the challenge: Sex-and gender-based analysis for health planning, policy and research in Canada. Halifax, NS: Atlantic Centre of Excellence for Women' s Health; 2009 .
26. Hankivsky O , Cormier R , de Merich D. Intersectionality: Moving women's health research and policy forward Vancouver , BC: Women's Health Research Network; 2009 .
27. Almey M. Finding data on women: A guide to major sources at Statistics Canada2007 . Available from: http://publications.gc.ca/collections/Collection/SW21-22-2007E.pdf.
28. Johnson JL , Greaves L , Repta R. Better Science with Sex and Gender±A Primer for Health Research . Vancouver, BC: Women's Health Research Network; 2007 .
29. Nieuwenhoven L , Bertens M , Klinge I. Gender awakening tool, bibliography: sex and gender in research: Gender Basic; 2007 [cited 2016 August 17 ]. Available from: http://www.genderbasic.nl/ downloads/pdf/WISER%20fest %20booklet%20extra%20pages.pdf.
30. HammarstroÈm A. A tool for developing gender research in medicine: examples from the medical literature on work life . Gender medicine . 2007 ; 4:S123±S32 . PMID: 18156098
31. Burke MA , Eichler M. The BIAS FREE Framework. A practical tool for identifying and elininating social biases in health research . Geneva: Global Forum for Health Research ; 2006 .
32. Donner L. Including gender in health planning: A guide for regional health authorities2003 . Available from: http://www.pwhce.ca/gba.htm.
34. Baldaszti E , Bloomfield K , den Broeder L , Burke MA , Diggins-RoÈsner A , Doyal L , et al. Gender Based Analysis (GBA) in Public Health Research, Policy and Practice: Documentation of the International Workshop in Berlin2001. Available from: http://www.gesundheit-nds.de/ewhnet/Documentations/GBA_ Dokumentation.PDF.
35. Canadian Institutes of Health Research. Shaping science for a healthier world 2014 . Available from: http://www.cihr-irsc.gc.ca/e/48633.html.
36. Gendered Innovations . Health and medicine checklist 2015 [cited 2016 August 17 ]. Available from: https://genderedinnovations.stanford.edu/methods/health_med_checklist.html.
37. European Commission . Toolkit for gender in research: checklist for gender in research: European Commission; 2011 [cited 2016 October 25 ]. Available from: http://bookshop.europa.eu/en/toolkitgender-in -eu-funded-research-pbKINA24840/downloads/KI- NA- 24840 - EN-C/ KINA24840ENC _002. pdf?FileName=KINA24840ENC_ 002 .pdf&SKU=KINA24840ENC_PDF& CatalogueNumber=KI-NA24840-EN-C.
38. Tannenbaum C , Schwarz JM , Clayton JA , de Vries GJ , Sullivan C . Evaluating sex as a biological variable in preclinical research: the devil in the details . Biol Sex Differ . 2016 ; 7 ( 1 ): 1 .
39. International Society for Gender Medicine. The National SocietiesÐInternational Society for Gender Medicine 2017 . Available from: http://www.isogem.com/?CategoryID= 168 .
40. Sauermann H , Roach M. Increasing web survey response rates in innovation research: An experimental study of static and dynamic contact design features . Res Policy . 2013 ; 42 ( 1 ): 273 ± 86 .
41. Saldaña J. The coding manual for qualitative research . London: Sage; 2015 .
42. Gendered Innovations . Formulating research questions 2015 [cited 2016 Ocotber 24 ]. Available from: http://genderedinnovations.stanford.edu/methods/questions.html.
43. Gendered Innovations . Designing health and biomedical research 2015 [cited 2016 October 24 ]. Available from: http://genderedinnovations.stanford.edu/methods/health.html.
44. Smith JA , Braunack-Mayer A , Wittert G . What do we know about men's help-seeking and health service use? Med J Aust. 2006 ; 184 ( 2 ): 81 ± 3 . PMID: 16411874
45. Armstrong P , Kits O . One Hundred Years of Caregiving . In: Grant K , Amaratunga C , Armstrong P , Boscoe M , Pederson A , Willson K , editors. Caring for/caring about: women, home care, and unpaid caregiving . Aurora, Ontario: Garamond Press; 2004 . p. 45 ± 74 .
46. Bauer G . Making sure everyone counts: considerations for inclusion, identification and analysis of transgender and transsexual individuals in health surveys . In: Coen S, Bannister E , editors. What a difference sex and gender make: a gender, sex and health research casebook Ottawa: Canadian Institutes of Health Research; 2012 . p. 59 ± 67 .
47. Carabez R , Pellegrini M , Mankovitz A , Eliason M , Scott M. Does your organization use gender inclusive forms? Nurses' confusion about trans* terminology . J Clin Nurs . 2015 ; 24 ( 21 ±22): 3306 ± 17 . https://doi. org/10.1111/jocn.12942 PMID: 26263919
48. Canadian Institutes of Health Research. Sex, gender and knowledge translation 2016 [cited 2016 October 26 ]. Available from: http://www.cihr-irsc.gc.ca/e/49933.html.
49. Miller VM . Why are sex and gender important to basic physiology and translational and individualized medicine? Am J Physiol Heart Circ Physiol . 2014 ; 306 ( 6):H781±8 . https://doi.org/10.1152/ajpheart. 00994. 2013 PMID: 24414073
50. Runnels V , Tudiver S , Doull M , Boscoe M. The challenges of including sex/gender analysis in systematic reviews: a qualitative survey . Systematic Reviews . 2014 ; 3 ( 1 ): 33 . https://doi.org/10.1186/2046- 4053-3-33 PMID: 24720875
51. Day S , Mason R , Lagosky S , Rochon PA . Integrating and evaluating sex and gender in health research . Health Res Policy Syst . 2016 ; 14 : 75 . https://doi.org/10.1186/s12961-016 -0147-7 PMID: 27724961
52. Chalmers I , Bracken M , Djulbegovic B , Garattini S , Grant J , GuÈlmezoglu A , et al. How to increase value and reduce waste when research priorities are set . Lancet . 2014 ; 383 ( 9912 ): 156 ± 65 . https://doi.org/10. 1016/S0140- 6736 ( 13 ) 62229 - 1 PMID: 24411644
53. Franconi F , Campesi I . Sex and gender influences on pharmacological response: an overview . Expert Rev Clin Pharmacol . 2014 ; 7 ( 4 ): 469 ± 85 . https://doi.org/10.1586/17512433. 2014 .922866 PMID: 24857340
54. Regitz-Zagrosek V . Sex and gender differences in cardiovascular disease . In: Oertelt-Prigione S , Regitz-Zagrosek V , editors. Sex and Gender Aspects in Clinical Medicine. London: Springer-Verlag London Limited; 2012 . p. 17 ± 44 .
55. Carter C , Resnick E , Mallampalli M , Kalbarczyk A. Sex and gender differences in Alzheimer's disease: recommendations for future research . J Womens Health . 2012 ; 21 ( 10 ): 1018 ± 23 .
56. Rocca WA , Mielke MM , Vemuri P , Miller VM . Sex and gender differences in the causes of dementia: a narrative review . Maturitas . 2014 ; 79 ( 2 ): 196 ± 201 . https://doi.org/10.1016/j.maturitas. 2014 . 05 .008 PMID: 24954700