Reassessing the sensitivity to leptonic CP violation
Reassessing the sensitivity to leptonic CP violation
Mattias Blennow 0 1 3 6 7 8
Pilar Coloma 0 1 3 4 7 8
Enrique Fernandez-Martinez 0 1 2 3 5 7 8
Open Access 0 1 3 7 8
c The Authors. 0 1 3 7 8
0 850 West Campus Dr , Blacksburg, VA, 24061 U.S.A
1 Stockholm , 106 91 Sweden
2 Instituto de F sica Te orica UAM/CSIC
3 KTH Royal Institute of Technology, AlbaNova University Center
4 Center for Neutrino Physics, Physics Department , Virginia Tech
5 Departamento de F sica Te orica, Universidad Aut onoma de Madrid
6 Department of Theoretical Physics, School of Engineering Sciences
7 Calle Nicol as Cabrera 13-15, Cantoblanco , Madrid, E-28049 Spain
8 Calle Francisco Tom as y Valiente , Cantoblanco, Madrid, E-28049 Spain
We address the validity of the usual procedure to determine the sensitivity of neutrino oscillation experiments to CP violation. An explicit calibration of the test statistic is performed through Monte Carlo simulations for several experimental setups. We find that significant deviations from a 2 distribution with one degree of freedom occur for experimental setups with low sensitivity to . In particular, when the allowed region to which is constrained at a given confidence level is comparable to the whole allowed range, the cyclic nature of the variable manifests and the premises of Wilk's theorem are violated. This leads to values of the test statistic significantly lower than a bution at that confidence level. On the other hand, for facilities which can place better constraints on the cyclic nature of the variable is hidden and, as the potential of the facility improves, the values of the test statistics first become slightly higher than and then approach asymptotically a 2 distribution. The role of sign degeneracies is also discussed.
violation; Neutrino Physics; CP violation; Statistical Methods
1 Introduction 2 3 4
Summary and outlook
Distribution of the test statistic for the null hypothesis
Mixing in the lepton sector of the Standard Model is described by the unitary
PontecorvoMaki-Nakagawa-Sakata (PMNS) matrix . In the standard three family scenario, it can
and, if neutrinos are Majorana particles, two additional Majorana CP phases. Neutrino
i mj2). Unlike for the CKM matrix, the mixing angles of the PMNS matrix
Electroweak Baryogenesis, JCKM is not large enough to account for the observed Baryon
asymmetry of the Universe [13, 14], the discovery of an additional source of CP violation
Given the current knowledge on the neutrino oscillation parameters, the focus for
the next generation of neutrino oscillation experiments will be to determine the neutrino
determination of the mass ordering, there has been a lively discussion in the literature
on whether or not the common way of assessing the sensitivity of a future experiment is
applicable . As shown in ref. , the common approach does give a reasonable
approximation of the median sensitivity, although not for the reasons typically used in the
argumentation for it. The small correction was mainly based on the one-sided nature of
the hypothesis test and only to a minor degree on the non-gaussianity of the statistical
2 distribution of the test statistic should be
common approach when making sensitivity analyses is that the change in number of events
events unchanged. In addition, there is no guarantee that the predicted data without
statistical fluctuations, as used in the common approach, will be representative. In the
present work, these assumptions are tested by explicit Monte Carlo simulations in order
to find out exactly how much the sensitivity analyses are affected by these assumptions.
To do so, we start from the basic frequentist definitions and apply the Feldman-Cousins
approach  in order to determine the sensitivity of several experimental setups.
The most common way of quantifying the experimental sensitivity to leptonic CP violation
rejected. In the literature, this is typically computed by constructing the test statistic
where 2 = 2 log L, and L is the likelihood of observing the data given a particular set of
oscillation parameters. It is worth noting that this involves minimizing over all nuisance
parameters, which may be subject to external constraints (we will discuss how such external
one degree of freedom, based on the implications of Wilks theorem . In addition, the
Asimov data set1 , i.e., the event rates without statistical fluctuations, is assumed to
be representative for the experimental outcome and is thus used to estimate the expected
confidence level (CL) at which the CP conservation hypothesis would be rejected. However,
as for the case of the neutrino mass ordering [15, 17], it is not clear to what degree the
assumptions underlying Wilks theorem are violated when testing CP conservation in this
fashion, resulting in a need to explicitly test this framework.
The procedure to test the CP conservation hypothesis at a given CL can be summarized
as follows: first, the distribution of the test statistic S is found by simulating a large number
of realizations of the experiments based on the predicted event rates under the assumption
oscillation parameters, which we assume to be the true values. The value of S is then
computed for each realization, which provides the distribution of S. CP conservation will
1So named after the Franchise short story by Isaac Asimov, where an entire electorate was replaced by
values in the distribution. This automatically defines a critical value, Sc(x), such that CP
conservation is rejected at CL x if S > Sc(x). By construction, Sc(x) is the inverse of the
cumulative distribution function (CDF) of S under the CP conserving hypothesis.
The above construction is only concerned with the test of CP conservation for a given
data set, i.e., once the experiment has already taken data. The expected performance of
due to statistical fluctuations, different realizations of a given facility will lead to a different
significance at which CP conservation can be rejected. Therefore, the convention is to define
the expected sensitivity of a given experiment as the CL obtained for the median of the
referred to as the median sensitivity and it will not necessarily coincide with the significance
computed with the Asimov data set.
performed in ref. . The T2HK experiment  was used as an example, and the critical
values of sin2 213 . 5 102, and was done using a different test statistic than the one
considered in the present work.
The common approach when dealing with external constraints on the nuisance
parameters (such as previous determinations of oscillation parameters or prior constraints for the
the outcome of any external past experiment should be taken into account. This implies
must still be separately calibrated for each simple hypothesis.
would obtain the probability of reaching a given CL using the already known outcome for
the external constraints. This is the procedure that has been followed in this paper. On
the other hand, if external constraints from a hypothetical future experiment were to be
implemented instead, the outcome of such an experiment would be unknown and should
therefore still be chosen according to the expected distribution of possible outcomes.
In our simulations, we have followed the prescription described above for existing
constraints on the neutrino oscillation parameters as well as for including constraints on
the systematic errors. The calibration of the test statistic has been performed for the best fit
values of the parameters. Given our present knowledge, we do not expect the calibration of
the test statistic for CP violation to be crucially dependent on this choice with the notable
show that this can strongly reflect in the test statistic calibration . Such correlations
asses the significance of a signal in both variables simultaneously. Nevertheless, such a
study is computationally very expensive and therefore beyond the scope of this work.
Once the Asimov data sets were computed, realizations of the experiments were constructed
by applying Poisson statistics individually to the number of events in each bin based on
the expected rates. This was implemented using the MonteCUBES software . The
value of S was then computed for each realization in order to find the distribution of S
under the CP conserving and CP violating hypotheses. In all cases, 5% (10%) systematic
errors were used for the signal (background) event rates. These are bin-to-bin correlated,
but uncorrelated between signal and backgrounds and between different oscillation
channels. The true values of the oscillation parameters have been set according to the best
with present experimental uncertainties from ref. . Sign degeneracies are fully taken
into account during minimization.
in obtaining the value of S for the median of the distributions. Since this is much less
3The only modification is that in the present work the beam power of the LBNE experiment has been
increased to 1.2 MW, and the detector mass has been fixed to 34 kt.
sensitive to statistical fluctuations than the sampling of the tails, only 103 realizations of
the experiments are simulated in order to obtain the median of S for these values.
In this section we present our simulation results and we discuss the general behaviour of
the test statistic and the final sensitivities for CP violation. In section 4.1 we show the
distribution of the test statistic defined in eq. (2.1) for all facilities under consideration, as
obtained from the Monte Carlo simulations. Then, in section 4.2 we discuss the dependence
of the distributions with several factors, such as the statistics of a given experiment and/or
the presence of sign degeneracies. Finally, in section 4.3 we show the resulting sensitivities
obtained from Monte Carlo and we compare to the usual values reported in the literature
(which are obtained under the assumption that Wilks theorem is valid).
Distribution of the test statistic for the null hypothesis
Figure 1 shows our results for the distribution of the test statistic S for the experimental
setups considered in this work. In order to show the dependence with the statistics of the
experiment, results are also shown for T2HK with a factor 20 reduced statistics. Such a
setup would also be similar to T2K at the end of its planned running time, although with a
somewhat extended run and equal running times in the neutrino and antineutrino modes.
degree of freedom for almost all experiments under consideration. The notable exceptions to
are observed and the critical values corresponding to a given CL are considerably smaller
lines have been obtained after increasing the nominal exposure by factors of 1, 2, 5, 10, 20 and 100.
be understood as follows. One of the requirements for the applicability of Wilks theorem
change in the observables used to determine it should constitute a linear space. This
no change in the number of events. Nevertheless, if a given facility is able to constrain the
for the statistical fluctuations on the number of events. Therefore, one could naively expect
more apparent. This will be discussed in more detail in section 4.2.
We have checked that this extra deviation with respect to the better agreement showed by
LBNE and the ESS can be attributed to the sign degeneracies, that play a very important
situations in which the change in the number of events does not span a linear space,
implying the non-applicability of Wilks theorem. When the mass hierarchy is assumed to
be known, the distribution obtained for T2HK lies on top of those obtained for LBNE and
the ESS as expected.
In this section, we explicitly test the interpretation of figure 1 presented above with detailed
and increase its statistics by factors of 1, 2, 5, 10, 20 and 100 in order to improve the
milder and happens at higher and higher CL. Indeed, the different curves show a change
of trend developing a sharper decrease after a certain confidence level, which increases
with statistics. For an increase in statistics of one order of magnitude this deviation is no
number of experimental realizations simulated in this work. Indeed, for increases of the
statistics by factors of 10 and 20 the obtained distribution is rather consistently above a
In the right panel of figure 2, we show the correlation between the observed deviations
to take place in all cases. It can be seen that the height of the barrier separating the two
minima in the right panel seems roughly correlated with the value of the test statistic for
which a change of trend is observed in the slope of the curves in the left panel. The change
whole allowed range for this variable. Therefore, its cyclicity and hence the violation of the
requisites of Wilks theorem become manifest at the CL given by the height of the barrier
separating the two minima.
To understand all the features displayed by the left panel of figure 2 we have used a
toy model, represented in the left panel of figure 3. Let us assume that the observables
neutrinos and antineutrinos. The expected values for the number of events would span
by the assumptions of Wilks theorem). For simplicity we approximate this ellipse in
observable space by a circle,4 depicted in the left panel in figure 3. The line tangent to the
circle represents how this distribution of expected number of events should look like if the
premises of Wilks theorem were satisfied. The point belonging both to the line and to the
statistic. To perform this calibration, gaussian statistical fluctuations with a characteristic
standard deviation s (determined by the statistic and systematic errors of the experiment)
test statistic for each realization would be obtained as the square of the distance of the
distribution would be obtained as a result. Here, the distances are defined relative to the
standard deviation of the gaussian fluctuations. On the other hand, for the case where the
variable is cyclic the test statistic would rather correspond to the square of the distance
4As we will see, even with these simplifying assumptions the qualitative, and even quantitative, behaviour
observed in the left panel of figure 2 can be very well reproduced.
model depicted in the left panel. The different lines correspond to different sizes of the standard
of the distance of this point to the circle (again, in analogy to eq. (2.1)). In this case, the
will be characterized by statistical fluctuations (at this CL) which are of the same order or
even larger than the size of the circle in the left panel of figure 3. Indeed, if the expected
good fit to the data. In this case, statistical fluctuations will often be significantly larger
than the size of the circle and the distance from any fluctuation to the circle will tend to be
larger than the distance of the fluctuations to the line. Hence, the value of S reconstructed
computed the distribution of the test statistics for our toy model, for different sizes of the
standard deviation s relative to the circle radius depicted in the left panel in figure 3. The
obtained distributions are depicted in the right panel of figure 3 for different sizes of the
right panel of figure 2 is remarkable. As anticipated, the lines for which the size of the
fluctuations is of the order of the circle size (s > 0.5) translate into a distribution falling
respectively. As the relative size of the gaussian perturbations with respect to the size of
toy model to a significant decrease in the relative size of the fluctuations with respect
to the size of the circle. In this situation, perturbations can then easily fall inside the
circle. While for small size of the perturbations with respect to the circle radius there are
still more points closer to the line than to the circle, the points that fall within the circle
overcompensate for this fact since they are significantly closer to it. Therefore, the average
distance is shorter to the circle than to the line, and the distribution of S gets shifted to
times higher statistics in figure 2.
If the size of the perturbation keeps decreasing, then the distribution asymptotically
is very small the system is no longer sensitive to the curvature of the outcome space and
the effects coming from this should vanish. This result is shown by the red line obtained for
Finally, we briefly comment on the role of sign degeneracies. In presence of sign
degeneracies the above simplified model would have two, rather than one, ellipses (or
circles). In this situation, the perturbations that take place in a direction away from
the second (degenerate) ellipse are essentially unaffected by its presence. However, for
perturbations in the direction of the second ellipse, the distance to this second ellipse can
decrease with respect to the case in which no degeneracies are present. Naively, this could
mean that the values of the test statistics S would increase, and indeed this is the effect
observed for T2HK in figure 1. However, the distance between the fluctuation and the
CP-conserving points in the new ellipse may also decrease, which would tend to produce
a decrease of the obtained values of S. In practice, whether the values of S increase or
decrease when in presence of sign degeneracies depends on the relative positions of the two
ellipses in observable space as well as on the relative locations of the CP-conserving values
found examples that change the distribution of S in either direction. However, in all cases
above. Moreover, the vast majority of proposed future facilities facing the measurement
in the future that the mass hierarchy will most likely have been measured by some other
combination of experiments. A more interesting effect, though, might be caused by the
from  show. However, a detailed study of these degeneracies calls for a calibration of
of a signal in both variables simultaneously. Such a study is computationally expensive
and beyond the scope of this work.
(in degrees). The median values obtained from Monte Carlo simulations are shown as solid lines,
whereas the values corresponding to the predicted Asimov data set are shown as dashed lines.
distribution does not necessarily imply a change in the sensitivity of an experiment. This
will depend on whether the Asimov data can be taken as a good approximation of the
are compared to the values obtained from the Asimov data.
We find that the Asimov
data set is very close to the median value of S for all future experiments, while significant
a good approximation of the median test statistic when considering the more sensitive
Finally, in figure 5 we compare the computed sensitivity to CP violation using a full
analysis based on Monte Carlo simulation with the results obtained in the common
apthe Asimov data set). The results obtained in 68% and 95% of the simulated experiments
are also indicated by the yellow and green bands, respectively, to show the expected
dispersion with respect to the median result. From this figure, a reasonable agreement in the
the Asimov data set considerably overestimates the median result from the Monte Carlo
for the Asimov data set. The yellow (green) bands show the regions containing 68% (95%) of the
experimental realizations obtained from the Monte Carlo.
We also note that the Monte Carlo results do not go to zero around the CP conserving
This should be expected as the median confidence level obtained if CP is conserved should
be 50%. The Asimov data cannot reflect this since any fluctuations around it will increase
the value of the test statistic. It is therefore not a good approximation of the median in a
Summary and outlook
In this work, we have studied the validity of the common approach used to compute the
sensitivity of future long baseline experiments to leptonic CP violation. By explicit Monte
Carlo simulation we have found that the test statistic (defined in eq. (2.1)) is close to being
range, so that the cyclic nature the variable is not apparent. Even in this scenario, the
Otherwise, the distribution of the test statistic is instead significantly shifted towards lower
We have also found that the median value of the distribution is well approximated
by the Asimov data set in most cases. This results in a sensitivity which is very similar,
although slightly worse, than what is typically quoted in the literature.
of T2K (with relatively low statistics), and reactor data, it would be interesting to reassess
its significance through a calibration of the test statistic also fully taking into account the
octant degeneracy which seems to play a significant role.
We are very grateful to Thomas Schwetz for valuable discussions and encouragement. We
warmly thank Peter Ballett for pointing out an inconsistency in the treatment of the
systematic errors in the first version of the manuscript, and Pilar Hernandez for illuminating
discussions. This work was supported by the Goran Gustafsson Foundation [MB] and by the
U.S. Department of Energy under award number DE-SC0003915 [PC]. EFM acknowledges
financial support by the European Union through the FP7 Marie Curie Actions CIG
NeuProbes (PCIG11-GA-2012-321582) and the ITN INVISIBLES (PITN-GA-2011-289442),
and the Spanish MINECO through the Ramon y Cajal programme (RYC2011-07710)
and through the project FPA2009-09017. We also thank the Spanish MINECO (Centro
de excelencia Severo Ochoa Program) under grant SEV-2012-0249 as well as the Nordita
Scientific program News in Neutrino Physics, where part of this work was performed.
This article is distributed under the terms of the Creative Commons
Attribution License (CC-BY 4.0), which permits any use, distribution and reproduction in
any medium, provided the original author(s) and source are credited.
Teor. Fiz. 33 (1957) 549] [INSPIRE].
JETP 7 (1958) 172 [Zh. Eksp. Teor. Fiz. 34 (1957) 247] [INSPIRE].
Prog. Theor. Phys. 23 (1960) 1174 [INSPIRE].
Prog. Theor. Phys. 28 (1962) 870 [INSPIRE].
 Z. Maki, M. Nakagawa and S. Sakata, Remarks on the unified model of elementary particles,
 B. Pontecorvo, Neutrino Experiments and the Problem of Conservation of Leptonic Charge,
Sov. Phys. JETP 26 (1968) 984 [Zh. Eksp. Teor. Fiz. 53 (1967) 1717] [INSPIRE].
 DAYA-BAY collaboration, F.P. An et al., Observation of electron-antineutrino
disappearance at Daya Bay, Phys. Rev. Lett. 108 (2012) 171803 [arXiv:1203.1669]
 Double CHOOZ collaboration, Y. Abe et al., Reactor electron antineutrino disappearance
in the Double CHOOZ experiment, Phys. Rev. D 86 (2012) 052008 [arXiv:1207.6632]
mixing: critical look at present precision, JHEP 12 (2012) 123 [arXiv:1209.3023] [INSPIRE].
Phys. Rev. D 86 (2012) 010001 [INSPIRE].
382 [hep-ph/9406289] [INSPIRE].
113011 [arXiv:1210.3651] [INSPIRE].
oscillation parameters with one hundred thousand reactor events, Phys. Rev. D 89 (2014)
013001 [arXiv:1309.1638] [INSPIRE].
experiments to the neutrino mass ordering, JHEP 03 (2014) 028 [arXiv:1311.1822]
 O. Vitells and A. Read, A comment on estimating sensitivity to neutrino mass hierarchy in
neutrino experiments, arXiv:1311.4076 [INSPIRE].
 G.J. Feldman and R.D. Cousins, A Unified approach to the classical statistical analysis of
small signals, Phys. Rev. D 57 (1998) 3873 [physics/9711021] [INSPIRE].
 S.S. Wilks, The Large-Sample Distribution of the Likelihood Ratio for Testing Composite
Hypotheses, Ann. Math. Stat. 9 (1938) 60 [INSPIRE].
neutrino oscillation experiments?, Phys. Lett. B 648 (2007) 54 [hep-ph/0612223] [INSPIRE].
for leptonic CP-violation discovery based on the European spallation source linac, Nucl.
experiments with GLoBES (General Long Baseline Experiment Simulator), Comput. Phys.
of neutrino oscillation experiments with GLoBES 3.0: General Long Baseline Experiment
 B. Pontecorvo , Mesonium and anti-mesonium , Sov. Phys. JETP 6 ( 1957 ) 429 [Zh . Eksp.
 B. Pontecorvo , Inverse beta processes and nonconservation of lepton charge, Sov . Phys.  RENO collaboration , J.K. Ahn et al., Observation of Reactor Electron Antineutrino Disappearance in the RENO Experiment, Phys. Rev. Lett . 108 ( 2012 ) 191802  MINOS collaboration, P. Adamson et al ., Improved search for muon-neutrino to electron-neutrino oscillations in MINOS , Phys. Rev. Lett . 107 ( 2011 ) 181802  T2K collaboration , K. Abe et al., Indication of Electron Neutrino Appearance from an Accelerator-produced Off-axis Muon Neutrino Beam , Phys. Rev. Lett . 107 ( 2011 ) 041801  M.C. Gonzalez-Garcia , M. Maltoni , J. Salvado and T. Schwetz , Global fit to three neutrino  Particle Data Group collaboration , J. Beringer et al., Review of Particle Physics (RPP) ,  M.B. Gavela , P. Hernandez , J. Orloff and O. Pene , Standard model CP-violation and baryon asymmetry, Mod . Phys. Lett . A 9 ( 1994 ) 795 [hep-ph/9312215] [INSPIRE].
 M.B. Gavela , P. Hernandez , J. Orloff , O. Pene and C. Quimbay , Standard model CP-violation and baryon asymmetry . Part 2: Finite temperature, Nucl . Phys . B 430 ( 1994 )  X. Qian , A. Tan , W. Wang , J.J. Ling , R.D. McKeown and C. Zhang , Statistical Evaluation of Experimental Determinations of Neutrino Mass Hierarchy , Phys. Rev . D 86 ( 2012 )  F. Capozzi , E. Lisi and A. Marrone , Neutrino mass hierarchy and electron neutrino  M. Blennow , P. Coloma , P. Huber and T. Schwetz , Quantifying the sensitivity of oscillation  G. Cowan , K. Cranmer , E. Gross and O. Vitells , Asymptotic formulae for likelihood-based tests of new physics , Eur. Phys. J. C 71 ( 2011 ) 1554 [Erratum ibid . C 73 ( 2013 ) 2501]  T. Schwetz , What is the probability that 13 and CP-violation will be discovered in future  K. Abe et al., Letter of Intent: The Hyper-Kamiokande Experiment. Detector Design and Physics Potential, arXiv:1109 .3262 [INSPIRE].
 P. Coloma , H. Minakata and S.J. Parke , Interplay between appearance and disappearance channels for precision measurements of 23 and , Phys. Rev. D 90 ( 2014 ) 093003 [arXiv:1406.2551] [INSPIRE].
 M.C. Gonzalez-Garcia , M. Maltoni and T. Schwetz , Updated fit to three neutrino mixing: status of leptonic CP-violation , JHEP 11 ( 2014 ) 052 [arXiv:1409.5439] [INSPIRE].
 LBNE collaboration, C. Adams et al., The Long-Baseline Neutrino Experiment: Exploring Fundamental Symmetries of the Universe , arXiv:1307.7335 [INSPIRE].
 ESSnuSB collaboration , E. Baussan et al., A very intense neutrino super beam experiment  NOA collaboration , D.S. Ayres et al., NOA: Proposal to build a 30 kiloton off-axis  P. Huber , M. Lindner and W. Winter , Simulation of long-baseline neutrino oscillation  P. Huber , J. Kopp , M. Lindner , M. Rolinec and W. Winter , New features in the simulation Simulator , Comput. Phys. Commun . 177 ( 2007 ) 432 [hep-ph/0701187] [INSPIRE].
 P. Coloma , P. Huber , J. Kopp and W. Winter , Systematic uncertainties in long-baseline neutrino oscillations for large 13 , Phys. Rev . D 87 ( 2013 ) 033004 [arXiv:1209.5973]  M. Blennow and E. Fernandez-Martinez , Neutrino oscillation parameter sampling with MonteCUBES, Comput . Phys. Commun . 181 ( 2010 ) 227 [arXiv:0903.3985] [INSPIRE].