Observational Review and Analysis of Concussion: a Method for Conducting a Standardized Video Analysis of Concussion in Rugby League
Gardner et al. Sports Medicine - Open
Observational Review and Analysis of Concussion: a Method for Conducting a Standardized Video Analysis of Concussion in Rugby League
Andrew J. Gardner 0 1
Christopher R. Levi 0 1
Grant L. Iverson 2 3 4
0 Hunter New England Local Health District Sports Concussion Program, John Hunter Hospital , Newcastle, New South Wales , Australia
1 Centre for Stroke and Brain Injury, School of Medicine and Public Health, University of Newcastle , Callaghan, New South Wales , Australia
2 Department of Physical Medicine and Rehabilitation, Harvard Medical School, Spaulding Rehabilitation Hospital , Boston, MA , USA
3 Home Base, A Red Sox Foundation and Massachusetts General Hospital Program , Boston, MA , USA
4 MassGeneral Hospital for ChildrenTM Sport Concussion Program , Boston, MA , USA
Background: Several professional contact and collision sports have recently introduced the use of sideline video review for club medical staff to help identify and manage concussions. As such, reviewing video footage on the sideline has become increasingly relied upon to assist with improving the identification of possible injury. However, as yet, a standardized method for reviewing such video footage in rugby league has not been published. The aim of this study is to evaluate whether independent raters reliably agreed on the injury characterization when using a standardized observational instrument to record video footage of National Rugby League (NRL) concussions. Methods: Video footage of 25 concussions were randomly selected from a pool of 80 medically diagnosed concussions from the 2013-2014 NRL seasons. Four raters (two naïve and two expert) independently viewed video footage of 25 NRL concussions and completed the Observational Review and Analysis of Concussion form for the purpose of this inter-rater reliability study. The inter-rater reliability was calculated using Cohen's kappa (κ) and intra-class correlation (ICC) statistics. The two naïve raters and the two expert raters were compared with one another separately. Results: A considerable number of components for the naïve and expert raters had almost perfect agreement (κ or ICC value ≥ 0.9), 9 of 22 (41%) components for naïve raters and 21 of 22 (95%) components for expert raters. For the concussion signs, however, the majority of the rating agreement was moderate (κ value 0.6-0.79); both the naïve and expert raters had 4 of 6 (67%) concussion signs with moderate agreement. The most difficult concussion sign to achieve agreement on was blank or vacant stare, which had weak (κ value 0.4-0.59) agreement for both naïve and expert raters. Conclusions: There appears to be value in expert raters, but less value for naive raters, in using the new Observational Review and Analysis of Concussion (ORAC) Form. The ORAC Form has high inter-rater agreement for most data elements, and it can be used by expert raters evaluating video footage of possible concussion in the NRL.
Concussion; Video analysis; Injury management
Identifying concussion from the sideline during a match is challenging, but with the use of video out-of-view or fleeting signs may be captured and a player can be removed from play.
Having a reliable form for coding and analysing concussion can be a useful adjunct to the sideline clinical management strategy of the athletic trainer and team physician.
We present the first objective and reliable coding form
for rugby league to capture the game situation, the
mechanism of injury, and possible signs of concussion.
Rugby league is a high-intensity collision sport [
game is played continuously in two 40-min halves, and
game-play involves two teams of 13 on-field players and
four interchange players who may be switched in and
out of the game. The published incidence rates of
concussion in rugby league vary [
]; at the National Rugby
League (NRL) level, medically diagnosed concussions in
three clubs from the 2013 season revealed an incidence
rate of 14.8 concussions per 1000 player match hours
], while a rate of 28.3 concussion per 1000 player
match hours were reported from one NRL club over a
15year (1998–2012) period [
]. The incidence of use of the
concussion interchange rule (CIR) was 24.0 (95% CI
20.7–27.9) uses of the CIR per 1000 NRL player
match hours [
] and 44.9 (95% CI 38.5–52.3) uses of
the CIR per 1000 National Youth Competition player
match hours [
One method that has becoming increasingly relied upon
to assist with improving the identification of possible
concussion has been the review of video footage on the
sideline. The use of video for reviewing a concussion may
identify signs of injury that may have been blocked from
view or otherwise missed by medical staff. A number of
professional contact and collision sports have recently
introduced the use of sideline video review for club medical
staff to help identify and manage concussions [
number of studies of video footage have been conducted
in a variety of sports, for example, rugby league [
rugby union [
], and Australian Rules Football [
]. Other sports, such as boxing [
], soccer [
], ice hockey [
6, 10, 18, 19
], and lacrosse
, have also reported on the use of video footage for
understanding the circumstances and mechanisms of injury
unique to their sports. A risk prediction model among
National Hockey League (NHL) players reported that both
visual signs of concussion and information pertaining to
mechanisms of injury improved a clinician’s ability to
identify athletes who should be removed from play and
]. Specifically, the study indicated that
suspected loss of consciousness, motor incoordination or
balance problems, being in a fight, having an initial hit
from another player’s shoulder, and having a secondary hit
on the ice were all associated with increased risk of
subsequent concussion diagnosis.
Sport-specific coding criteria of concussion for game
situational factors and injury mechanisms have been
developed for hockey (e.g., the ‘Heads-Up Checklist’ [
but these criteria do not generalize to other sports like
rugby union or rugby league. Video criteria and coding
forms require validation in each individual sport [
In a more recent NHL study examining the predictive
ability of visual signs of concussion, loss of consciousness,
motor incoordination, and blank/vacant look had a
positive association with concussion diagnosis, whereas
slow to get up and clutching of the head, despite occurring
frequently, had low positive predictive values .
Several video studies have examined signs of
concussion, together with player characteristics, injury
characteristics, and match situational factors, in
professional rugby league [
]. In 2014, video reviews
of injury have been implemented in the NRL to help
medical staff and promote player health and safety.
The aim of this study was to present a standardized
observational recording form and to determine
whether independent raters agreed on the antecedent
events, mechanisms of injury, and concussion signs
when using the form to code digital video records of
concussions in the NRL.
This study was conducted in the national professional
rugby league competition in Australia during the
2013 and 2014 seasons. All medically diagnosed
concussion events during 2013 and 2014 NRL seasons
were available to be included in this study.
For this study, 25 medically diagnosed concussions
were randomly selected from the 2013 to 2014 NRL
seasons’ video library (n = 80). The video library
included only excerpts of the incidents for each case,
not the entire game. The duration of the game
footage recorded for each of the 25 cases selected from
the library and used in this study ranged from 138 to
473 s. Four raters (two ‘expert’ and two ‘naïve’)
independently reviewed the video footage of the 25 NRL
concussion events. The naïve raters were novices of
the sport. They had limited to no knowledge or
experience with rugby league match play and no
experience identifying and managing concussion. The
expert raters were defined as individuals with experience
in rugby league match play and expertise in concussion
management at the professional (NRL) level. Both expert
raters had at least one NRL season of experience working
on the sideline for an NRL club with the responsibility of
identifying and assessing athletes suspected of having
sustained a concussion.
The medically diagnosed NRL concussion library
was gathered from three teams during the 2013
season (n = 20 concussions) [
], and all teams during
the 2014 NRL season (n = 60 concussions). The
Concussion in Sport Group consensus definition of
concussion was used by all clubs involved in this study
]. Raters viewed the match digital records of 25
concussions using the Quicktime Multimedia Player V.7.7.5.
Each rater completed all components of the form for each
of the 25 concussion events. The raters were permitted to
view the incident as many times as required and in any
playback speed as deemed necessary to complete all
categories of the form. All participants provided informed
consent. This study was conducted in accordance with the
standards of the ethics outlined in the Declaration of
Helsinki. Approval of this study was provided by the
University of Newcastle Human Ethics Committee.
A rating form was created to provide a simple but
standardized framework for coding and analysing video
footage of the situations and consequences of concussion
events in rugby league. The form was developed by a
neuropsychologist with extensive experience in the sport
and concussion management and was based on work
conducted previously in ice hockey in North America, and a
similar, but not identical, approach to validating the form
was used for the validation of the ‘Heads-Up Checklist’
]. The form was adapted to include specific
information to rugby league. Concussion signs that have been
previously examined in video review studies in rugby league
were included [
]. The form consists of various
sections related to the player and game characteristics (e.g.,
ball carrier versus tackler, tackle height, type of play, etc.),
the anatomical region of contact, the injury location on
the field of play, the injured player’s on-field management,
and six possible concussion signs (see Fig. 1).
The results from the two naïve raters and the expert raters
were considered separately. The intra-class correlation
(ICC) was used to determine the level of agreement
between the two naïve raters and the two expert raters for
interval and ratio variables (i.e. ‘number of players in
tackle’ and ‘time taken to leave’). Inter-rater reliability
analyses using Cohen’s kappa (κ) statistics [
] were used to
determine the level of agreement between the two naïve
raters and the two expert raters for all other (nominal)
variables. Unlike the total percent agreement, Cohen’s
kappa considers the proportional agreement that could
occur simply by chance. The κ coefficients are calculated
by considering the proportion of rater agreement and the
expected proportion [
]. Using the interpretations of κ
described by McHugh [
], κ agreement was categorized
as almost perfect (>.90), strong (.80–.90), moderate
(.60–.79), weak (.40–.59), minimal (.21–.39) and none
(0–.20). All analyses were performed using IBM SPSS
Statistics V.23.0 [
] and used two-sided tests for
significance at the 0.05 level, with 95% confidence intervals (CIs).
The inter-rater reliabilities for the various components of
the rating form for both the naïve and expert raters are
presented in Table 1. For the naïve raters, 6 of 20 (30%)
components of the form, and 5 of 6 (83%) concussions
signs, had κ values between .60 and .79 (‘moderate’
agreement), while 2 of 2 (100%) interval/ratio variables of the
form had very good ICC. According to the interpretations
of κ described by McHugh [
], 8 of 22 components were
categorized as ‘almost perfect’; 3 components were
classified as ‘strong’; 2 components were classified as ‘weak’; 2
components were classified as ‘minimal’; and one of the
components (playing position) was not classified. For the
concussion signs, the naïve raters had no signs that had
‘almost perfect’ or ‘strong’ agreement; 5 (83%) signs were
classified as ‘moderate’ and 1 (17%) sign was classified as
For the expert raters, 19 of 20 (95%) components of the
form had κ values of between .90–1.00 (‘almost perfect’
agreement) and one (5%) had moderate agreement. The
expert raters had perfect ICC for 2 of 2 (100%) interval/ratio
variables. For the concussion signs, the expert raters had 1
of 6 (17%) of concussions signs with κ values between
.80–.90 (‘strong’ agreement); 4 (67%) of were classified as
‘moderate’; and 1 (17%) was classified as ‘weak’ agreement.
No signs classified by either the naïve or expert raters had a
‘minimal’ or ‘none’ level of agreement.
There were nine components that were all rated with
‘almost perfect’ agreement by both the naïve and the
expert raters (game time, score, whether the concussed
player was a ball carrier or a tackler, the number of
players involved in the tackle, whether the offending
player was placed on report by a match official, the
initial contact, the region of contact, whether the player
was removed from play, and how the player left the
field). The level of agreement between the expert raters
and between the naïve raters was also very consistent for
the tackle height and whether the injury occurred as a
result of foul play (i.e., the offending player was
penalized). The naïve raters had a ‘strong’ level of agreement
for these components. The naïve raters had a moderate
level of agreement on whether the game was played
during the night or day, the tackle number in the set, the
anatomical location of the impact, and the location of
the field where the concussion took place, whereas all of
these components had an ‘almost perfect’ level of
agreement between the two expert raters. The expert raters
also achieved an ‘almost perfect’ level of agreement on
the secondary contact, whether the concussed player
had anticipated the impact that caused the injury, and
the time taken to leave the field of play. However, the
naïve raters only had a ‘minimal’ level of agreement on
these components. For type of play, and whether or not
the player returned to play, the naïve raters had a ‘weak’
agreement on these components compared to the
‘almost perfect’ agreement by the expert raters. Whether
or not there was secondary contact was the most
difficult component to agree upon; the expert raters’
level of agreement was ‘moderate’ for this component,
and the naive raters’ level of agreement was ‘minimal’
(see Table 1).
Regarding concussion signs, slow to get up had the
best level of agreement between expert and naïve raters
of all possible concussion signs (strong and moderate
agreement, respectively), whereas a blank or vacant stare
had the worst agreement (both rater groups had a ‘weak’
level of agreement). Clutch or shake head, gait ataxia (or
having wobbly legs), unresponsiveness, and post-impact
seizure-like features had moderate agreement for both
expert and naïve raters.
Rugby League is a full contact collision sport that has
high concussion incidence rates [
]. The in-game
management and decision-making process surrounding
concussion is a challenge. Video review is increasingly
being used as one method for improving this in-game
decision-making process for medical staff, although a
standardized approach to the use of such information
had not been published. Although there is a large
body of research examining on-field markers of
concussion and their association with outcome [
2–5, 7, 8,
11, 16, 22, 26, 27, 31, 33, 35, 36, 38–40, 42–46
few of these studies have been focused on possible signs of
concussion at the time of injury (versus collected later as
part of a questionnaire or interview with the athlete). This
study presents a standardized observational form and
examines intra-rater and inter-rater agreement on the
antecedent events, mechanisms of injury, and concussion
signs. Overall, the results of this study suggest that a
certain level of knowledge about the game is required to
complete the form components accurately. Expert rates
achieved an ‘almost perfect’ level of agreement on 21/22
(95%) of components compared to only 9/22 (41%)
components for the naïve raters.
In a similar study conducted with the ‘Heads-Up
Checklist’ for National Hockey League (NHL)
concussions, the naïve raters also had worse agreement
across components pertaining to the antecedent
events and mechanism of injury compared to the
expert raters. Of the 15 components in version 1 of the
Heads-Up Checklist, naïve raters 7 (47%) had weak or
minimal agreement, compared to only 1 of the 15
(7%) components for the expert raters [
]. For the
Heads-Up Checklist, the acceleration of the head
(which was not considered a component or review
item in our form) was the single component with the
worst agreement across naïve and expert raters.
Rating secondary contact was also challenging in the
hockey study as it was in the current study. The
location of the playing surface where the concussion
Clutch or shake head 0.73 (0.46–0.93) Moderate 0.63 (0.36–0.87)
Slow to get up 0.83 (0.15–1.00) Moderate 0.52 (0.09–1.00)
Gait ataxia 0.73 (0.47–0.94) Moderate 0.60 (0.58–0.61)
Blank/vacant stare 0.50 (0.23–0.76) Weak 0.44 (0.15–0.71)
Unresponsiveness 0.78 (0.56–1.00) Moderate 0.70 (0.45–0.93)
Post-impact seizure 0.65 (N/A) Moderate 0.53 (0.04–0.90)
McHugh κ Agreement Classification: almost perfect (>.90), strong (.80–.90), moderate (.60–.79), weak (.40–.59), minimal (.21–.39), and none (0–.20)
CIs confidence intervals, ICC intra-class correlation, κ kappa, N/A not applicable, NR not recorded
occurred and the time in the game when the
concussion occurred were the two components with the
strongest agreement by naïve and expert raters for
the hockey study [
]. For the current study, the time
in the game was rated well. However, the location on
the field did not have a high agreement for the naïve
raters. The discrepancy between naïve raters for the
hockey study compared to this rugby league study
may have occurred for at least three reasons. First,
we divided the playing surface in our study into 12
different components and the hockey study used
fewer zones. Second, the hockey study designated
offensive ends and defensive ends, whereas the rugby
league study required the raters to record the
direction of the play, and some of the disagreement
between the naïve raters for the location on the field
was due to the indication of the direction of the play.
Finally, the hockey study used naïve raters who where
more familiar with their sport (i.e., ‘individuals with
limited experience who might have played or coached
[ice] hockey at a competitive level’), whereas our
naïve raters were complete novices, who had limited
to no experience even watching the sport as fans and
certainly no experience identifying concussions.
In the current study, there were a number of
variables that appear to rely on knowledge,
understanding, and experience with rugby league match play
(i.e., the expert raters outperformed the naïve raters).
For example, there were large differences between the
coding by expert and naïve raters of variables such as
secondary contact and anticipation of impact. There
was also a large difference between the coding by
expert and naïve raters on whether the player
returned to play. This variable required the raters to
watch the remainder of a game (following the injury)
to determine if the injured athlete subsequently
returned to the field of play. Interchanges can occur
during play or during a stoppage in play, and they
are not always announced on the broadcaster footage.
It appears that the naïve raters were not as savvy in
identifying the return to play of an interchanged
athlete and/or did not identify the athlete as being
re-involved in match play following their return to
the field of play.
As with our previous video reviews of concussion signs
], we once again found that determining whether
a concussed player had a blank or vacant stare was
difficult to agree upon. We had weak agreement between
naïve (0.44, 95% CI = 0.15–0.71) and expert (0.50, 95% CI
= 0.23–0.76) raters in this study, and our previous work
has also revealed difficulty with agreement between raters
(i.e., 0.36 (95% CI = 0.29 to 0.43)  and 0.62 (95% CI =
0.37 to 0.88) [
]). In a recent Australian Football League
(AFL) video review, inter-rater reliability for the
blank/vacant stare on first review was reported to be 0.24 (95% CI
= 0.04 to 0.41) and minimal improvements were observed
on second review [0.26 (95% CI = 0.07 to 0.43)]. The
intra-rater reliability in the AFL study was somewhat
better for the two raters over the two rating sessions [i.e. 0.63
(95% CI = 0.50 to 0.74) and 0.36 (95% CI = 0.18 to 0.51)].
The concussion sign ‘blank/vacant stare’ was reported to
have 9% sensitivity, 100% specificity, 100% positive
predictive value and 58% negative predictive value in the
sample of AFL concussions [
]. When the quality of the
video (including the zoom capacity to see the players face)
is limited, attempting to code the presence or absence of a
blank or vacant stare from video is challenging [
supports the notion that good-quality video from multiple
camera angles are crucial for effective video surveillance
of injuries [
]. In the current study, however, this was not
a limitation, suggesting that it is also important to have
clear definitions, including the inclusion and exclusion
criteria for coding concussion signs [
]. In a recent series of
video reviews of concussions from the AFL [
9, 29, 30
Makdissi and Davis indicated that video review may be an
avenue that facilitates the assessment of the mechanism
and impact of injury and allows for the identification of
brief early signs of concussion [
]. The authors suggest
that video analysis may be a useful adjunct to the sideline
assessment of possible concussion [
] and that the
implementation of a flowchart may improve the timely
assessment of concussion [
We recently completed a study on the frequency
(or base rates) of concussion signs in NRL match play
(Gardner et al., under review). That study reviewed
every game (n = 201) from the 2014 NRL season,
which included 127,062 tackles, and found
unresponsiveness occurred 52 times [24 (46%) were diagnosed
with a concussion], slow to get up occurred 2240
times [60 (3%) were diagnosed with a concussion],
clutching or shaking the head occurred 361 times [38
(11%) were diagnosed with a concussion], gait ataxia
occurred 102 times [35 (34%) were diagnosed with a
concussion], blank or vacant stare occurred 98 times
[45 (46%) were diagnosed with a concussion], and a
post-impact posturing or seizure occurred 4 times [3
(75%) were diagnosed with a concussion]. The
unresponsiveness sign had 40% sensitivity, 91% specificity,
46% positive predictive value, and 89% negative
predictive value. The slow to get up sign had 100%
sensitivity, 50% specificity, 27% positive predictive
value, and 100% negative predictive value. Clutching
or shaking the head had 63% sensitivity, 46%
specificity, 18% positive predictive value, and 87% negative
predictive value. Gait ataxia had 58% sensitivity, 79%
specificity, 34% positive predictive value, and 91%
negative predictive value. Blank or vacant stare had
75% sensitivity, 84% specificity, 46% positive
predictive value, and 95% negative predictive value.
Post-impact seizure had 5% sensitivity, 100% specificity,
75% positive predictive value, and 85% negative predictive
value in the 2014 NRL season (Gardner et al., under
One of the unusual and unexpected findings of this
study was the discrepancy observed between the naïve
raters in coding variables that were conceivably
thought to be obvious (e.g., game time, score, day/
night game). The naïve raters did not always have
100% agreement. Because rugby league is a
continuous sport, it is common for the game to continue
despite an injury, and therefore, the game clock also
does not stop. As such, an injury can occur well
before the game and the game clock is stopped. The
discrepancies in the ‘time in game’ variable are
explained by this issue; one of the naïve reviewers
recorded the time correctly (i.e., when the injury
occurred), whereas the other naïve rater often
recorded the time when the game clock was stopped.
In terms of the ‘game score’ variable, it is possible
that the naïve raters were unfamiliar with teams, and
therefore, errors were made in coding the score of
each team. For the ‘day/night game’ variable, there
were a number of games that were played during
twilight, as well as the footage of some of those cases
being zoomed in, and the wide view did not make
the day/night difference obvious to the naïve raters
who do not watch NRL games.
Video review appears to be a useful adjunct to
traditional methods for making in-game decisions pertaining
to the identification of potential concussion (and an
athlete subsequently being removed from play). However, to
better understand and quantify the value of this process,
future research should be conducted under time limits
and/or during a game to replicate the
real-world/practical pressure, neither of which was replicated in this
study. Future studies might focus on whether agreement
between experts improves under ‘ideal circumstances’ (i.e.
as many reviews as required without time limitations)
versus ‘real-world circumstances’ (i.e. a quick decision
required to identify a possible injury and immediately
remove the athlete from play).
The current study has several limitations. Firstly, clubs
used their own personnel and methods for identifying
possible injuries on the field and diagnosing concussions
on the sideline, which presumably makes the final
specific criteria for a ‘medically diagnosed concussion’
variable across clubs. The current study does not
generalize to the real-world use of in-game video
analysis because the study was not conducted under the
time pressure associated with in-game decision-making.
Further, the sample size is small, and only two naïve and
two expert reviewers were used. Whether the current
results hold true for more cases and a greater number of
raters is unknown.
The present study suggests that determining the
presence or absence of a blank or vacant stare is challenging
for both naïve and expert raters to rate reliably, but that
showing unresponsiveness (i.e. possible LOC), clutching
or shaking of the head, a post-impact seizure, or being
slow to get up are more reliably rated signs. However, in
this study, there was no variability in the clinical
outcome measure, as our sample came from a pool of
individuals who were all medically diagnosed with a
concussion. Therefore, the predictive value of any one
component or concussion sign, or a combination of these
items, is unknown and may be the focus of future
research. Given the variability of in-game decision-making
in professional rugby league [
], we sought to
provide validation of a standardized approach for collecting
information surrounding possible concussions to help
inform the in-game decision-making process. Although
the form was created for all levels of competition, it only
had a good level of agreement among experienced raters.
Therefore, it might only be useful for those teams or
clubs that have experts available to them (i.e., the
professional level). For lower levels of competition, the form
may have less of a benefit, because the naïve raters had a
low level of agreement on many components of the
form. It is important to note, however, that the
management of suspected concussion at these lower levels
should always be conservative. If a concussion is
suspected, then the athlete should be removed from play
and not returned to play the same day [
]. At the
professional level, data collected from this form may allow
for a thorough understanding of the situational and
contextual factors related to concussion, which may be used
to strategize future interventions to reduce the risk of
concussion at this level.
The authors would like to thank Mrs. Kathryn Gurr and Ms. Vanessa Case
(University of Newcastle, School of Medicine and Public Health) for
completing the video review as the nominated naïve raters and Dr. Jinho
Lee (School of Human Movement and Nutrition Sciences, University of
Queensland) for completing the review as an expert rater.
The funding was provided by Hunter Medical Research Institute (HMRI)
supported by Anne Greaves.
AG conceived the design of the study, designed the data collection tool,
collected the video footage, liaised with the other raters to complete the
data collection and monitored the data collection, managed the study
database, conducted the statistical analysis, and drafted, revised, and
finalized the manuscript. He had the final veto on the submission. CL
assisted with the design of the study and provided the editorial comment to
the drafts of the manuscript prepared for the submission. GI assisted with
the design of the study, assisted with the statistical analysis of the data,
provided expert editorial comment for all the drafts of the manuscript, and
had the final veto on the submission. All authors read and approved the
Andrew Gardner is an early career fellow with the National Health and Medical
Research Council (NHMRC) and is supported by the School of Medicine and
Public Health, University of Newcastle, and the Priority Research Centre for
Stroke and Brain Injury, School of Medicine and Public Health, University of
Newcastle. He has a clinical practice in neuropsychology involving individuals
who have sustained sport-related concussion (including current and former
athletes). He has operated as a contracted concussion consultant to the
Australian Rugby Union (ARU) from July 2016. He has received travel funding
from the Australian Football League (AFL) to present at the Concussion in
Football Conference in 2013 and 2017. The previous grant funding includes the
NSW Sporting Injuries Committee, the Brain Foundation (Australia), and the
Hunter Medical Research Institute (HMRI), supported by Jennie Thomas. He is
currently funded through the HMRI, supported by Anne Greaves, and the
University of Newcastle’s Priority Research Centre for Stroke and Brain Injury.
Christopher Levi declares that he has no conflict of interest. Grant Iverson has
been reimbursed by the government, professional scientific bodies, and
commercial organizations for discussing or presenting research relating to mild
TBI and sport-related concussion at meetings, scientific conferences, and
symposiums. He has a clinical and consulting practice in forensic
neuropsychology involving individuals who have sustained mild TBIs (including
athletes). He has received research funding from several test publishing
companies, including ImPACT Applications, Inc., CNS Vital Signs, and
Psychological Assessment Resources (PAR, Inc.). He received past salary support
from the Harvard Integrated Program to Protect and Improve the Health of
National Football League Players Association Members. He acknowledges
unrestricted philanthropic support from the Mooney-Reed Charitable
Foundation and ImPACT Applications, Inc.
Springer Nature remains neutral with regard to jurisdictional claims in published
maps and institutional affiliations.
1. Andersen TE , Larsen Ø , Tenga A , Engebretsen L , Bahr R . Football incident analysis: a new video based method to describe injury mechanisms in professional football . Br J Sports Med . 2003 ; 37 : 226 - 32 .
2. Asplund CA , McKeag DB , Olsen CH . Sport-related concussion: factors associated with prolonged return to play . Clin J Sport Med . 2004 ; 14 : 339 - 43 .
3. Baker JG , Leddy JJ , Darling SR , Rieger BP , Mashtare TL , Sharma T , Willer BS . Factors associated with problems for adolescents returning to the classroom after sport-related concussion . Clin Pediatr (Phila) . 2015 ; 54 : 961 - 8 .
4. Barlow M , Schlabach D , Peiffer J , Cook C. Differences in change scores and the predictive validity of three commonly used measures following concussion in the middle school and high school aged population . Int J Sports Phys Ther . 2011 ; 6 : 150 - 7 .
5. Berz K , Divine J , Foss KB , Heyl R , Ford KR , Myer GD . Sex-specific differences in the severity of symptoms and recovery rate following sports-related concussion in young athletes . Physician Sport Med . 2013 ; 41 : 58 - 63 .
6. Bruce JM , Echemendia RJ , Meeuwisse W , Hutchison MG , Aubry M , Comper P . Development of a risk prediction model among professional hockey players with visible signs of concussion . Br J Sports Med . 2017 ; bjsports-2016 -097091
7. Chermann JF , Klouche S , Savigny A , Lefevre N , Herman S , Bohu Y. Return to rugby after brain concussion: a prospective study in 35 high level rugby players . Asian J Sports Med . 2014 ; 5 : e24042 .
8. Collins MW , Iverson GL , Lovell MR , McKeag DB , Norwig J , Maroon J . On-field predictors of neuropsychological and symptom deficit following sportsrelated concussion . Clin J Sport Med . 2003 ; 13 : 222 - 9 .
9. Davis G , Makdissi M . Use of video to facilitate sideline concussion diagnosis and management decision-making . J Sci Med Sport . 2016 ; online first .
10. Echemendia RJ , Bruce JM , Meeuwisse W , Hutchison MG , Comper P , Aubry M. Can visible signs predict concussion diagnosis in the National Hockey League? Br J Sports Med . 2017 ; bjsports-2016-097090.
11. Ellis MJ , Ritchie LJ , Koltek M , Hosain S , Cordingley D , Chu S , Selci E , Leiter J , Russell K. Psychiatric outcome after pediatric sports-related concussion . J Neurosurg Pediatr . 2015 ; 16 : 709 - 18 .
12. Gardner A , Iverson GL , Levi CR , Schofield PW , Kay-Lambkin F , Kohler RMN , Stanwell P. A systematic review of concussion in rugby league . Br J Sports Med . 2015 ; 49 : 495 - 8 .
13. Gardner AJ , Iverson GL , Quinn TN , Makdissi M , Levi CR , Shultz SR , Wright DK , Stanwell P. A preliminary video analysis of concussion in the National Rugby League . Brain Inj . 2015 ; 29 : 1182 - 5 .
14. Gardner AJ , Iverson GL , Stanwell P , Ellis J , Levi CR . A video analysis of use of the new “Concussion Interchange Rule” in the National Rugby League . Int J Sports Med . 2016 ; 37 ( 4 ): 267 - 3 .
15. Gardner AJ , Kohler RMN , Levi CR , Iverson GL . Usefulness of video review of possible concussions in National Youth Rugby League . Int J Sports Med . 2016 ; in press.
16. Guskiewicz KM , McCrea M , Marshall SW , Cantu RC , Randolph C , Barr W , Onate JA , Kelly JP . Cumulative effects associated with recurrent concussion in collegiate football players: the NCAA concussion study . JAMA . 2003 ; 290 : 2549 - 55 .
17. Hallgren KA . Computing inter-rater reliability for observational data: an overview and tutorial . Tutor Quant Methods Psychol . 2012 ; 8 : 23 - 34 .
18. Hutchison MG , Comper P , Meeuwisse WH , Echemendia RJ . A systematic video analysis of National Hockey League (NHL) concussions, part I: who, when, where and what? Br J Sports Med . 2015 ; 49 ( 8 ): 547 - 51 .
19. Hutchison MG , Comper P , Meeuwisse WH , Echemendia RJ . A systematic video analysis of National Hockey League (NHL) concussions, part II: how concussions occur in the NHL . Br J Sports Med . 2015 ; 49 ( 8 ): 552 - 5 .
20. Hutchison MG , Comper P , Meeuwisse WH , Echemendia RJ . An observational method to code concussions in the National Hockey League (NHL): the heads-up checklist . Br J Sports Med . 2014 ; 48 : 125 - 9 .
21. IBM Corp . IBM SPSS Statistics for Macintosh, Version 22 .0. Armonk, NY: IBM Corp. 2013 .
22. Kerr ZY , DeFreese JD , Marshall SW . Current physical and mental health of former collegiate athletes . Orthop J Sport Med . 2014 ; 2 : 1 - 9 .
23. King DA , Hume PA , Milburn PD , Guttenbeil D . Match and training injuries in rugby league: a review of published studies . Sport Med . 2010 ; 40 : 163 - 78 .
24. Koh JO , Watkinson EJ , Yoon Y -J. Video analysis of head blows leading to concussion in competition Taekwondo . Brain Inj . 2004 ; 18 : 1287 - 96 .
25. Kohler R , Makdissi M , McDonald W , Partridge B , Gardner AJ . A preliminary video review of in-game head injury incidents (HII) and use of the head injury assessment (HIA) from the 2015 super rugby season . Br J Sports Med . 2017 ; 51 : A78 - A79 .
26. Lau B , Lovell MR , Collins MW , Pardini J . Neurocognitive and symptom predictors of recovery in high school athletes . Clin J Sport Med . 2009 ; 19 : 216 - 21 .
27. Lau BC , Kontos AP , Collins MW , Mucha A , Lovell MR . Which on-field signs/ symptoms predict protracted recovery from sport-related concussion among high school football players? Am J Sports Med . 2011 ; 39 : 2311 - 8 .
28. Lincoln AE , Caswell SV , Almquist JL , Dunn RE , Hinton RY . Video incident analysis of concussions in boys' high school lacrosse . Am J Sports Med . 2013 ; 41 : 756 - 61 .
29. Makdissi M , Davis G. The reliability and validity of video analysis for the assessment of the clinical signs of concussion in Australian football . J Sci Med Sport . 2016 ; online first .
30. Makdissi M , Davis G . Using video analysis for concussion surveillance in Australian football . J Sci Med Sport . 2016 ; online first .
31. McCrea M , Kelly JP , Randolph C , Cisler R , Berger L. Immediate neurocognitive effects of concussion . Neurosurgery . 2002 ; 50 : 1032 - 40 .
32. McCrory P , Meeuwisse WH , Aubry M , Cantu B , Dvorák J , Echemendia RJ , Engebretsen L , Johnston K , Kutcher JS , Raftery M , Sills A , Benson BW , Davis GA , Ellenbogen RG , Guskiewicz K , Herring SA , Iverson GL , Jordan BD , Kissick J , McCrea M , McIntosh AS , Maddocks D , Makdissi M , Purcell L , Putukian M , Schneider K , Tator CH , Turner M. Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport held in Zurich, November 2012 . Br J Sports Med . 2013 ; 47 : 250 - 8 .
33. McDevitt J , Tierney RT , Phillips J , Gaughan JP , Torg JS , Krynetskiy E . Association between GRIN2A promoter polymorphism and recovery from concussion . Brain Inj . 2015 ; 29 : 1674 - 81 .
34. McHugh ML . Interrater reliability: the kappa statistic . Biochem Med . 2012 ; 22 : 276 - 82 .
35. Meehan WP , Mannix RC , Stracciolini A , Elbin RJ , Collins MW . Symptom severity predicts prolonged recovery after sport-related concussion, but age and amnesia do not . J Pediatr . 2013 ; 163 : 721 - 5 .
36. Meehan WP , Monuteaux MC . Early symptom burden predicts recovery after sport-related concussion . Nerurology . 2014 ; 83 : 2204 - 10 .
37. Miele VJ , Bailes JE . Objectifying when to halt a boxing match: a video analysis of fatalities . Neurosurgery . 2007 ; 60 : 307 - 15 -6.
38. Moor HM , Eisenhauer RC , Killian KD , Proudfoot N , Henriques AA , Congeni JA , Reneker JC . The relationship between adherence behaviors and recovery time in adolescents after a sports-related concussion: an observational study . Int J Sports Phys Ther . 2015 ; 10 : 225 - 33 .
39. Morgan CD , Zuckerman SL , Lee YM , King L , Beaird S , Sills AK , Solomon GS . Predictors of postconcussion syndrome after sports-related concussion in young athletes: a matched case-control study . J Neurosurgery Pediatr . 2015 ; 15 : 589 - 98 .
40. Nelson LD , Guskiewicz KM , Marshall SW , Hammeke T , Barr W , Randolph C , McCrea MA . Multiple self-reported concussions are more prevalent in athletes with ADHD and learning disability . Clin J Sport Med . 2016 ; 26 : 120 - 7 .
41. Savage J , Hooke C , Orchard J , Parkinson R. The incidence of concussion in a professional Australian rugby league team, 1998 - 2012 . J Sports Med . 2013 ; online first .
42. Terwilliger VK , Pratson L , Vaughan CG , Gioia GA . Additional post-concussion impact exposure may affect recovery in adolescent athletes . J Neurotrauma . 2016 ; 33 : 761 - 5 .
43. Vargas G , Rabinowitz A , Meyer J , Arnett PA . Predictors and prevalence of postconcussion depression symptoms in collegiate athletes . J Athl Train . 2015 ; 50 : 250 - 5 .
44. Zuckerman SL , Apple RP , Odom MJ , Lee YM , Solomon GS , Sills AK . Effect of sex on symptoms and return to baseline in sport-related concussion . J Neurosurg Pediatr . 2014 ; 13 : 72 - 81 .
45. Zuckerman SL , Solomon GS , Forbes JA , Haase RF , Sills AK , Lovell MR . Response to acute concussive injury in soccer players: is gender a modifying factor? J Neurosurg Pediatr . 2012 ; 10 : 504 - 10 .
46. Zuckerman SL , Yengo-Kahn AM , Buckley TA , Solomon GS , Sills AK , Kerr ZY . Predictors of postconcussion syndrome in collegiate student-athletes . Neurosurg Focus . 2016 ; 40 : E13 .