Test for non-negligible adverse shifts
Extracted source text · arXiv 2107.02990
\documentclass{article}
\usepackage{arxiv}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{lmodern} % https://github.com/rstudio/rticles/issues/343
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{graphicx} % figures
\usepackage{amsmath} % equations
\usepackage{caption} % tables
\usepackage{longtable} % tables
\usepackage[ruled, linesnumbered, commentsnumbered]{algorithm2e}
\title{Test for non-negligible adverse shifts}
\author{
Vathy M. Kamulete
\href{https://orcid.org/0000-0002-4451-3743}{\includegraphics[scale=0.06]{images/orcid.png}}
\\
Enterprise Model Risk Management \\
Royal Bank of Canada \\
Toronto, Canada \\
\texttt{\href{mailto:vathy.kamulete@rbccm.com}{\nolinkurl{vathy.kamulete@rbccm.com}}} \\
}
% Pandoc citation processing
\begin{document}
\maketitle
\def\tightlist{}
\begin{abstract}
Statistical tests for dataset shift are susceptible to false alarms:
they are sensitive to minor differences where there is in fact adequate
sample coverage and predictive performance. We propose instead a robust
framework for tests of dataset shift based on outlier scores,
\texttt{D-SOS} for short. \texttt{D-SOS} detects adverse shifts and can
identify false alarms caused by benign ones. It posits that a new (test)
sample is not substantively worse than an old (training) sample, and not
that the two are equal. The key idea is to reduce observations to
outlier scores and compare contamination rates. Beyond comparing
distributions, users can define what worse means in terms of predictive
performance and other relevant notions. We show how versatile and
practical \texttt{D-SOS} is for a wide range of real and simulated
datasets. Unlike tests of equal distribution and of goodness-of-fit, the
\texttt{D-SOS} tests are uniquely tailored to serve as robust
performance metrics to monitor model drift and dataset shift.
\end{abstract}
\keywords{
Dataset shift
\and
Model drift
\and
Robust statistics
\and
Goodness-of-fit
\and
Hypothesis testing
}
\section{Introduction}\label{sec:intro}
Suppose we fit a predictive model on a training set and predict on a
test set. Dataset shift (Quionero-Candela et al.
\protect\hyperlink{ref-quionero2009dataset}{2009}; Jose G Moreno-Torres
et al. \protect\hyperlink{ref-moreno2012unifying}{2012}; Kelly, Hand,
and Adams \protect\hyperlink{ref-kelly1999impact}{1999}), also known as
data or population drift, occurs when training and test distributions
are not alike. This is essentially a sample mismatch problem. Some
regions of the data space are either too sparse or absent during
training and gain importance at test time. We want methods that alert
users to the presence of unexpected inputs in the test set (Rabanser,
Günnemann, and Lipton
\protect\hyperlink{ref-rabanser2019failing}{2019}). To do so, a measure
of divergence between training and test set is required. Can we not
simply use the many modern off-the-shelf multivariate tests of equal
distributions for this?
One reason for moving beyond tests of equal distributions is that they
are often too strict. They require high fidelity between training and
test set everywhere in the input domain. However, not \emph{all} changes
in distribution are a cause for concern -- some changes are benign.
Practitioners distrust these tests because of false alarms. Polyzotis et
al. (\protect\hyperlink{ref-polyzotis2019data}{2019}) comment:
\begin{quote}
statistical tests for detecting changes in the data distribution
{[}\ldots{]} are too sensitive and also uninformative for the typical
scale of data in machine learning pipelines, which led us to seek
alternative methods to quantify changes between data distributions.
\end{quote}
Even when the difference is small or negligible, tests of equal
distributions reject the null hypothesis of no difference. An alarm
should only be raised if a shift warrants intervention. Retraining
models when distribution changes are benign is both costly and
ineffective. To tackle these challenges, we propose \texttt{D-SOS}
instead. \texttt{D-SOS} provides robust performance metrics for
monitoring model drift and validating data quality.
In comparing the test set to the training set, \texttt{D-SOS} pays more
attention to the regions --- typically, the outlying regions --- where
we are most vulnerable. To confront false alarms, it uses a robust test
statistic, namely the weighted area under the receiver operating
characteristic curve (WAUC). The weights in the WAUC (Li and Fine
\protect\hyperlink{ref-li2010weighted}{2010}) discount the safe regions
of the distribution. The goal of \texttt{D-SOS} is to detect
\emph{non-negligible adverse shifts}. This is reminiscent of
noninferiority tests (Wellek
\protect\hyperlink{ref-wellek2010testing}{2010}), widely used in
healthcare for determining that a new treatment is in fact not inferior
to an older one. Colloquially, the \texttt{D-SOS} null hypothesis holds
that the new sample is not substantively worse than the old sample, and
not that the two are equal. \texttt{D-SOS} moves beyond tests of equal
distributions and lets users specify which notions of outlyingness to
probe. The choice of the score function plays a central role in
formalizing what we mean by `worse'. \texttt{D-SOS} avoids the
prevailing overreliance on tests of equal distributions to detect and
assess the impact of dataset shift. We argue throughout that other
notions of shift or outlyingness are often, if not more informative, at
the very least complementary.
In this paper, we make the following contributions:
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
We derive \texttt{D-SOS}, a novel multivariate two-sample test for no
adverse shift, from tests of goodness-of-fit. \texttt{D-SOS} computes
its \(p\)-values based on sample splitting, permutation and/or
out-of-bag predictions. In our experience, the out-of-bag variant is
the most convenient. This approach improves on sample splitting, which
sacrifices calibration accuracy for inferential robustness (Rinaldo et
al. \protect\hyperlink{ref-rinaldo2019bootstrapping}{2019}).
\item
We compare \texttt{D-SOS} performance to two modern tests of equal
distributions. To do so, we transform \(p\)-values to \(s\)-values
(Greenland \protect\hyperlink{ref-greenland2019valid}{2019}).
\(s\)-values give us the means to define a region of practical
equivalence. We show via simulations that \texttt{D-SOS} matches or
exceeds the performance of these tests when the outlier score reflects
whether the observation belongs to the training or test set. These
results underscore the point that tests of equal distributions are
suboptimal when we could instead opt for tests of goodness-of-fit,
which explicitly account for the training set being the reference
distribution.
\item
Based on 62 real-world classification tasks from the OpenML-CC18
benchmark (Casalicchio et al.
\protect\hyperlink{ref-OpenMLR2017}{2017}; Bischl et al.
\protect\hyperlink{ref-bischl2017openml}{2017}), we show that
different notions of outlyingness are complementary for detecting
partition-induced dataset shift. In supervised settings,
\texttt{D-SOS} extends to notions of outlyingness such as residuals
and prediction intervals. These help to determine whether the shift
has an adverse impact on predictive performance. We estimate the
correlations between these complementary notions.
\end{enumerate}
The main takeaway is that given a generic method that assigns an outlier
score to a data point, \texttt{D-SOS} uplifts these scores and turns
them into a two-sample test for no adverse shift. The scores can come
from anomaly detection, two-sample classification, uncertainty
quantification, residual diagnostics, density estimation, dimension
reduction, and more. We have created an R package \texttt{dsos} for our
method. In addition, all code and data used in this paper are publicly
available.
\section{Statistical framework}\label{sec:test-statistic}
The theoretical framework builds on Zhang
(\protect\hyperlink{ref-zhang2002powerful}{2002}), hereafter referred to
as the the Zhang test. Take an i.i.d. training set
\(X^{tr} = \{x_{i}^{tr}\}_{i=1}^{n^{tr}}\) and a test set
\(X^{te} = \{x_{i}^{te}\}_{i=1}^{n^{te}}\). Each dataset \(X^{o}\) with
origin \(o \in \{tr(ain), te(st)\}\) lies in \(d\)-dimensional domain
\(\mathcal{X} \subseteq \mathbf{R}^d\) with sample sizes \(n^{o}\),
cumulative distribution function (CDF) \(F^{o}_{X}\) and probability
density function (PDF) \(f^{o}_{X}\). Let
\(\phi: \mathcal{X} \rightarrow \mathcal{S} \subseteq \mathbf{R}\) be a
score function and define binary instances
\(b_{i}^o(s) = \mathbf{I}(\phi(x_{i}^o) > s)\) such that \(b_{i}^o(s)\)
is 1 when the score \(\phi(x_{i}^o)\) exceeds the threshold \(s\) and 0,
otherwise. The proportion above the threshold \(s\) in dataset \(o\) is
in effect the contamination rate \(C^{o}(s)\), defined as
\(C^{o}(s) = \Pr\left(b_{i}^o(s) = 1\right)\). The contamination rate is
the complementary CDF of the scores, \(F^{o}_{S}(s) = 1 - C^{o}(s)\). As
before, \(F^{o}_{S}\) and \(f^{o}_{S}\) denote the CDF and PDF of the
scores.
Consider the null hypothesis \(\mathcal{H}_0: F^{te}_{X} = F^{tr}_{X}\)
of equal distribution against the alternative
\(\mathcal{H}_1: F^{te}_{X} \neq F^{tr}_{X}\). In tandem, consider the
null of equal contamination \(\mathcal{H}_0^{s}: C^{te}(s) = C^{tr}(s)\)
against the alternative \(\mathcal{H}_1^{s}: C^{te}(s) \neq C^{tr}(s)\)
at a given threshold score \(s\). To evaluate goodness-of-fit, Zhang
(\protect\hyperlink{ref-zhang2002powerful}{2002}) shows that testing
\(\mathcal{H}_0\) is equivalent to testing \(\mathcal{H}_0^{s}\) for
\(\forall s \in \mathcal{S}\). If \(z(s)\) is the relevant test
statistic for equal contamination, a global test statistic \(Z\) for
goodness-of-fit can be constructed from \(z(s)\), its local counterpart.
One such \(Z\) is
\begin{equation}\label{eqn:zhang}
Z = \int z(s) \cdot w(s) \cdot d(s)
\end{equation}
where \(w(s)\) are threshold-dependent weights. For concision, we
sometimes suppress the dependence on the threshold score \(s\) and
denote weights, contamination statistics and rates as \(w\), \(z\) and
\(C^{o}\) respectively. \texttt{D-SOS} differs from the \(Z\) statistic
(\ref{eqn:zhang}) in three particular aspects: the score function
\(\phi\), the weights \(w\) and the contamination statistic \(z\). We
address each in turn.
\texttt{D-SOS} scores instances from least to most \emph{abnormal}
according to a specified notion of outlyingness. To be concrete, for
density estimation\footnote{If so, the negative log density, a measure
of surprise, is a natural score for outlyingness.}, this property of
\(\phi\) can be expressed as
\begin{equation}\label{eqn:phi}
\phi(x_i) \le \phi(x_j) \Rightarrow \Pr(f^{tr}_{X}(x_i) \ge f^{tr}_{X}(x_j)) \ge 1 - \epsilon
\end{equation}
for \(x_i, x_j \in \mathcal{X}\) and \(\epsilon\), a (sufficiently
small) approximation error. Accordingly, instances in high-density
regions of the training set \(X^{tr}\) (nominal points) score low; those
in low-density regions (outliers) score high. Here, the score function
\(\phi\) can be thought of as a density-preserving projection. More
generally, higher scores, e.g.~wider prediction intervals and larger
residuals, indicate worse outcomes; the higher the score, the more
\emph{unusual} the observation. The structure in \(\phi\) is the
catalyst for adjusting the weights \(w\) and the statistic \(z\).
\texttt{D-SOS} updates the weights \(w\) to be congruent with the score
function \(\phi\). When projecting to the outlier subspace, high scores
imply unusual points, whereas both tails are viewed as extremes in the
Zhang test. Univariate tests such as the Anderson-Darling and the
Cramér-von Mises tests fit the framework in (\ref{eqn:zhang}): they make
different choices for \(\phi\), \(z\) and \(w\). These classical tests
place more weight at the tails to reflect the severity of tail
exceedances relative to deviations in the center of the distribution.
\texttt{D-SOS} corrects for outliers being projected to the upper
(right) tail of scores via the score function. The \texttt{D-SOS}
weights \(w\) are specified as
\begin{equation}\label{eqn:weights}
w(s) = \left(C^{tr}(s) - 1\right)^2
\end{equation}
The weights in (\ref{eqn:weights}) shift most of the mass from
low-threshold to high-threshold regions. As a result, \texttt{D-SOS} is
highly tolerant of negligible shifts associated with low scores, and
conversely, it is attentive to the shifts associated with high scores.
Zhang (\protect\hyperlink{ref-zhang2002powerful}{2002}) posits other
functional forms, but the quadratic relationship between weights and
contaminations gives rise to one of the most powerful variants of the
test and so, we follow suit.
\texttt{D-SOS} constructs a test statistic based on score ranks, not on
levels. The weighted area under the receiver operating curve, weighted
AUC (WAUC) for short, is a robust statistic that is invariant to changes
in levels so long as the underlying ranking is unaffected. The WAUC,
denoted \(T\), can be written as:
\begin{equation}\label{eqn:wauc}
T = \int C^{te}(s) \cdot f^{tr}_{\mathcal{S}}(s) \cdot w(s) \cdot d(s)
\end{equation}
assuming the test set is the \emph{positive} class and gets labeled as
1s (0s for the training set). See Hand
(\protect\hyperlink{ref-hand2009measuring}{2009}) for this derivation of
the AUC: the weights \(w\) are tacked on to obtain the WAUC. Formally,
\(T\) in (\ref{eqn:wauc}) is the \texttt{D-SOS} test statistic for
dataset shift. \(T\) is also clearly a member within the class of tests
in \(Z\) (see \ref{eqn:zhang}).
The null hypothesis of dataset shift \(\mathcal{H}_0^{ds}\) is that most
instances in the test set resemble the training set. The alternative
\(\mathcal{H}_1^{ds}\) is that the test set contains more outliers than
expected, if the training set is the reference distribution.
\texttt{D-SOS} specifies its null as
\(\mathcal{H}_0^{ds}: T_0 \leq T \mid \mathcal{H}_0\) against the
alternative \(\mathcal{H}_1^{ds}: T_0 > T \mid \mathcal{H}_0\), where
\(T_0\) is the observed WAUC and \(T \mid \mathcal{H}_0\), the WAUC
under the null of exchangeable samples. Note that \texttt{D-SOS} is a
one-tailed test. To have no or disproportionately fewer outliers in the
test set is desirable. These shifts do not trigger the alarm as they
would had \texttt{D-SOS} been two-tailed. Tests of goodness-of-fit and
of equal distributions reject the null if the training and test set are
different, even if the test set is less abnormal i.e.~better;
\texttt{D-SOS} does not do this. Through the mapping to the outlier
subspace, \texttt{D-SOS} simplifies a multivariate two-sample test to a
univariate representation.
\section{Motivation}\label{sec:iris}
For illustration, we apply \texttt{D-SOS} to the canonical \texttt{iris}
dataset (Anderson \protect\hyperlink{ref-anderson1935irises}{1935}). The
task is to classify the species of Iris flowers based on \(d=4\)
covariates (features) and \(n=50\) observations for each species. We
show how \texttt{D-SOS} helps diagnose false alarms. Specifically, we
highlight the following: (1) changes in distribution do not necessarily
hurt predictive performance, and (2) data points in the densest regions
of the data distribution can be the most difficult -- unsafe -- to
predict. For the subsequent tests, we split \texttt{iris} into 2/3
training and 1/3 test set. Figure \ref{fig:iris-pca} displays train-test
pairs, which are split according to different sampling strategies. The
first two principal components of \texttt{iris} show that the species
cluster together.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images/combined_pca.png}
\caption{
Principal components (PC) for split samples with different partitioning
strategies. Each subplot is a train-test pair from \texttt{iris}:
(1) random sampling, (2) stratified sampling by species,
(3) in-distribution examples in the test set, and (4) out-of-distribution
examples in the test set.}
\label{fig:iris-pca}
\end{figure}
Consider four notions of outlyingness. For two-sample classification,
define the outlier score as the probability of belonging to the training
or test set. For anomaly or out-of-distribution detection, the outlier
score is the isolation score, a proxy for local density. For residual
diagnostics, the score is the out-of-sample error. Finally, for
uncertainty quantification (resampling uncertainty), it is the standard
error of the mean prediction\footnote{This is the same as in
\texttt{RUE} (Schulam and Saria
\protect\hyperlink{ref-schulam2019can}{2019}). \texttt{RUE} depends on
pointwise confidence intervals to establish model trust.}. Only the
first notion of outlyingness -- two-sample classification -- pertains to
modern tests of equal distributions; the others capture other meaningful
notions of adverse shifts. For all these scores, higher is \emph{worse}:
higher scores indicate that the observation is diverging from the
desired outcome or that it does not conform to the training
set\footnote{Section \ref{sec:implementation} covers the implementation
details.}.
Suppose we want to test for dataset shift. How do the splits in Figure
\ref{fig:iris-pca} fare with respect to the tests for no adverse shifts
given by these notions of outlyingness? Let \(s\) and \(p\) denote
\(s-\)value and \(p-\)value. The test results are reported on the
\(s = -\, \log_{2}(p)\) scale because it is intuitive and lends itself
to comparison. We return to the advantages of using \(s-\)value for
comparison later. An \(s-\)value of \(k\) can be interpreted as seeing
\(k\) independent coin flips with the same outcome -- all heads or all
tails -- if the coin is ``fair'' (Greenland
\protect\hyperlink{ref-greenland2019valid}{2019}). This conveys how
incompatible the data is with the null hypothesis. For plotting, we
winsorize (clip) \(s-\)values to a low and high of 1 and 10
respectively. We also display a secondary y-axis with the \(p-\)value as
a cognitive bridge.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images/combined_dsos.png}
\caption{Tests of no adverse shifts for \texttt{iris}. The tests cover 4
notions of outlyingness for the sample splits in Figure \ref{fig:iris-pca}.
The dotted black line is the common -- and commonly abused -- $p-$value
$= 0.05$ threshold.}
\label{fig:iris-dsos}
\end{figure}
In Figure \ref{fig:iris-dsos}, the case with (1) random sampling
exemplifies the type of false alarms we want to avoid. Two-sample
classification, representing tests of equal distributions, is
incompatible with the null of no adverse shift (a \(s-\)value of around
8). But this shift does not carry over to the other tests. Anomaly
detection, residual diagnostics and confidence intervals are all
compatible with the view that the test set is not worse. Had we been
entirely reliant on two-sample classification, we may not have realized
that this shift is essentially benign. Tests of equal distributions
alone give a narrow perspective on dataset shift. Contrast (1) random
with (2) stratified sampling. When stratified by species, all the tests
are compatible with the null of no adverse shift.
We also expect the tests based on anomaly detection to let through the
(3) in-distribution test set and flag the (4) out-of-distribution one.
Indeed, the results in Figure \ref{fig:iris-dsos} concur. We might be
tempted to conclude that the in-distribution observations are
\emph{safe}, and yet, the tests based on residual diagnostics and
confidence intervals are fairly incompatible with this view. This is
because some of the in-distribution (densest) points are concentrated in
a region close to the decision boundary where the classifier does not
discriminate well; Figure \ref{fig:iris-pca} show where Iris versicolor
(triangles) and Iris virginica (squares) partially overlap. That is, the
densest observations are not necessarily \emph{safe}.
Out-of-distribution detection glosses over this, in that the culprits
may very well be the in-distribution points. \texttt{D-SOS} offers a
more holistic perspective of dataset shift because it borrows strength
from these complementary notions of outlyingness.
\section{Related work}\label{sec:related}
\textbf{Outlier scores}. Density ratios and class probabilities serve as
scores in comparing distributions (Menon and Ong
\protect\hyperlink{ref-menon2016linking}{2016}). These scores, however,
do not directly account for the predictive performance. Prediction
intervals and residuals, on the other hand, do. Intuitively, both
prediction intervals and residuals reflect adverse predictive
performance in some regions of the feature space. Prediction
(confidence) intervals typically widen with increasing dataset shift
(Snoek et al. \protect\hyperlink{ref-snoek2019can}{2019}). Methods like
\texttt{MD3} (Sethi and Kantardzic
\protect\hyperlink{ref-sethi2017reliable}{2017}) and \texttt{RUE}
(Schulam and Saria \protect\hyperlink{ref-schulam2019can}{2019}) lean on
this insight: they essentially track instances in the \emph{uncertain}
regions of the predictive model. Similarly, out-of-distribution examples
often have larger residuals or errors (Hendrycks and Gimpel
\protect\hyperlink{ref-hendrycks2016baseline}{2017}). This points to a
dual approach: the first is based on residual diagnostics (the classical
approach) and the second is based on out-of-distribution detection. For
recent examples of these research areas with a modern machine learning
twist, see Janková et al.
(\protect\hyperlink{ref-jankova2020goodness}{2020}) for the former and
Morningstar et al.
(\protect\hyperlink{ref-morningstar2021density}{2021}) for the latter.
Residuals traditionally underpin misspecification tests in regression.
Other approaches such as trust scores (Jiang et al.
\protect\hyperlink{ref-jiang2018trust}{2018}) can also flag unreliable
predictions. Because all these scores reflect distinct notions of
outlyingness, the contrasts are insightful even if admittedly, some are
related.\footnote{Aguinis, Gottfredson, and Joo
(\protect\hyperlink{ref-aguinis2013best}{2013}) lists ``14 unique and
mutually exclusive outlier definitions'' - see Table 1 therein.} In
this respect, \texttt{D-SOS} is in some sense a unifying framework.
Bring your own outlier scores and \texttt{D-SOS} would morph into the
corresponding two-sample test.
\textbf{Dimension reduction}. In practice, reconstruction errors from
dimension reduction can also separate inliers from outliers (Ruff et al.
\protect\hyperlink{ref-ruff2021unifying}{2021}). Recently, Rabanser,
Günnemann, and Lipton
(\protect\hyperlink{ref-rabanser2019failing}{2019}) combines dimension
reduction, as a preprocessing step, with tests of equal distributions to
detect dataset shift. Kirchler et al.
(\protect\hyperlink{ref-kirchler2020two}{2020}) pushes this further and
constructs powerful two-sample tests based on an intermediate low-rank
representation of the data using deep learning. Sticking to the theme of
projecting to a more informative subspace, some methods e.g.~(Cieslak
and Chawla \protect\hyperlink{ref-cieslak2009framework}{2009}; Lipton,
Wang, and Smola \protect\hyperlink{ref-lipton2018detecting}{2018})
gainfully rely on model predictions for their low-rank representation.
This last approach, to add a cautionary note, entangles the predictive
model, subject to its own sources of errors such as misspecification and
overfitting, with dataset shift (Wen, Yu, and Greiner
\protect\hyperlink{ref-wen2014robust}{2014}). But it also points to the
effectiveness of projecting to a lower subspace to circumvent issues
related to high dimensions. Indeed, the classifier two-sample test
(Friedman \protect\hyperlink{ref-friedman2004multivariate}{2004};
Clémençon, Depecker, and Vayatis
\protect\hyperlink{ref-ciemenccon2009auc}{2009}; Lopez-Paz and Oquab
\protect\hyperlink{ref-lopez2016revisiting}{2017}; Cai, Goggin, and
Jiang \protect\hyperlink{ref-cai2020two}{2020}) leverages
\emph{univariate} scores that discriminate between training and test set
to detect changes in distribution for \emph{multivariate} data. Inspired
by this approach, \texttt{D-SOS} effectively uses outlier scores as a
device for dimension reduction.
\textbf{Test statistic}. The area under the receiver operating
characteristic curve (AUC) has a long tradition of being used as a
robust test statistic in nonparametric tests. Within the framework of
classifier tests, Clémençon, Depecker, and Vayatis
(\protect\hyperlink{ref-ciemenccon2009auc}{2009}) pairs a high-capacity
classifier with the AUC as a test statistic, via the
Mann-Whitney-Wilcoxon test. Demidenko
(\protect\hyperlink{ref-demidenko2016p}{2016}) proposes an AUC-like test
statistic as an alternative to the classical tests at scale because the
latter ``do not make sense with big data: everything becomes
statistically significant'' while the former attenuates the strong bias
toward large sample size. As a generalization of the AUC, the weighted
AUC (WAUC) can put non-uniform weights on decision thresholds (Li and
Fine \protect\hyperlink{ref-li2010weighted}{2010}). \texttt{D-SOS}
seizes on this to give more weight to the outlying regions of the data
distribution. Weng and Poon (\protect\hyperlink{ref-weng2008new}{2008})
also advocates for the WAUC as a performance metric for imbalanced
datasets because it can capture disparate misclassification costs.
\texttt{D-SOS} duly recognizes disparate costs in other contexts and
reflects that the outlying regions suffer the most. To the best of our
knowledge, this is the first time that the WAUC is being used as a test
statistic.
\section{Implementation}\label{sec:implementation}
In this section, we turn our attention to deriving valid \(p-\)values
for inference when the score function \(\phi\) is data depedent. Without
loss of generality, let \(\mathcal{D} = [X^{tr}, X^{te}]\) be the pooled
training and test set. The score function
\(\phi = \Phi(\mathcal{D}, \lambda)\) is estimated from data
\(\mathcal{D}\) and hyperparameters \(\lambda \in \Lambda\). This
calibration procedure
\(\Phi: \mathcal{X} \times \Lambda \rightarrow \phi\) returns the
requisite score function
\(\phi: \mathcal{X} \rightarrow \mathcal{S} \subseteq \mathbb{R}\). The
asymptotic null distribution of the WAUC \(T\), however, is invalid when
the same data is used both for calibration and inference (scoring). We
circumvent this issue with permutations, sample splitting, and/or
out-of-bag predictions.
Permutations use the empirical, rather than the asymptotic, null
distribution. The maximum number of permutations \(R\) is set to
\(R = 1000\) (Marozzi \protect\hyperlink{ref-marozzi2004some}{2004}).
This procedure is outlined in Algorithm \ref{algo:dsos-1}. We refer to
this variant as \texttt{DSOS-PT} in Section \ref{sec:experiments}.
Unless stated otherwise, \texttt{DSOS-PT} is the default used in the
experiments in Section \ref{sec:experiments}. For speed, this is
implemented as a sequential Monte Carlo test (Gandy
\protect\hyperlink{ref-gandy2009sequential}{2009}), which exercises
early stopping when the result is fairly obvious. Even with
computational tricks to increase the speed, permutations can be
computationally expensive, sometimes prohibitively so.
\begin{algorithm}
\SetAlgoLined
\KwData{Pooled dataset $\mathcal{D} = [X^{tr}, X^{te}]$, calibration procedure $\Phi$,
number of permutations $R$ and hyperparameters $\lambda$}
\SetKwFunction{WeightedAUC}{WeightedAUC}
\BlankLine
\textbf{Label:} Assign labels $y = \mathbb{I}(x \in X^{te})$,
$\forall x \in \mathcal{D}$\;
\textbf{Calibrate:} Fit the score function
$\phi = \Phi(\mathcal{D}, \lambda)$\;
\textbf{Score:} Score observations $s = \phi(x)$, $\forall x \in \mathcal{D}$\;
\textbf{Test:} Compute the observed WAUC
$T_0 = \WeightedAUC(s, y)$ using (\ref{eqn:wauc})\;
\textbf{Shuffle:} Permute the $n = n^{tr} + n^{te}$ labels in
$y$ to get $y^{(r)}$ for each permutation $r = 1, 2, ..., R$\;
\textbf{Estimate null:} Repeat steps 2-4 with the matching $\mathcal{D}^{(r)}$
to get the empirical null values $T^{(r)} = \WeightedAUC(s^{(r)}, y^{(r)})$\;
\BlankLine
\KwOut{$p-value = 1 - \frac{1}{R} \sum_{r=1}^{R} \mathbb{I}(T^{(r)} \le T_0))$}
\caption{Permutation-based test (DSOS-PT)}
\label{algo:dsos-1}
\end{algorithm}
A faster alternative, based on sample splitting, relies on the
asymptotic null distribution but incurs a cost in calibration accuracy
because it sets aside some data. This tradeoff is common in classifier
two-sample tests, for example. Assume that we split each dataset in
half: \(X^{tr} = X_1^{tr} \cup X_2^{tr}\) and
\(X^{te} = X_1^{te} \cup X_2^{te}\). The first half is used for
calibration, and the second for scoring (inference). We describe this
split-sample procedure in Algorithm \ref{algo:dsos-2}, and refer to it
as \texttt{DSOS-SS} in Section \ref{sec:experiments}. Given the weights
in \ref{eqn:weights}, Li and Fine
(\protect\hyperlink{ref-li2010weighted}{2010}) shows that under some
mild regularity assumptions, the distribution of the test statistic
under the null \(T\mid \mathcal{H}_0\) is asymptotically normally
distributed as
\begin{equation}\label{eqn:null}
\begin{split}
\sigma(n^{tr}, n^{te}) = \sqrt{\frac{a_1 \cdot (\delta + 1)}{n^{te}}} \\
T \mid \mathcal{H}_0 \sim \mathcal{N}(\frac{1}{12}, \sigma(n^{tr}, n^{te}))
\end{split}
\end{equation}
with mean \(= \frac{1}{12}\) and standard deviation
\(\sigma(n^{tr}, n^{te})\), which depends on the sample sizes. The
constants in (\ref{eqn:null}) work out to
\(\delta = \frac{n^{te}}{n^{tr}}\) and \(a_1 \approx 0.008928571\). Li
and Fine (\protect\hyperlink{ref-li2010weighted}{2010}) gives the
general form of the asymptotic distribution of the WAUC, ready for use
with weights different from the chosen ones in \ref{eqn:weights}.
A third option, which is a natural extension of sample splitting, is to
use cross-validation instead. In \(V-\)fold cross-validation, the number
of folds \(V\) mediates between calibration accuracy and inferential
robustness. At the expense of refitting \(V\) times, this approach uses
most of the data for calibration and can leverage the asymptotic null
distribution, provided that the scores are out-of-sample predictions. In
other words, it combines the best of Algorithm \ref{algo:dsos-1} and
\ref{algo:dsos-2}, namely calibration accuracy from the first and
inferential speed from the second. We refer to this variant as
\texttt{DSOS-AT} in Section \ref{sec:experiments}. We show in
simulations that this approach either matches or exceeds the performance
of \texttt{DSOS-PT} and \texttt{DSOS-SS}.
\begin{algorithm}
\SetAlgoLined
\KwData{Calibration set $\mathcal{D}_1 = [X_1^{tr}, X_1^{te}]$,
Inference set $\mathcal{D}_2 = [X_2^{tr}, X_2^{te}]$, calibration procedure
$\Phi$ and hyperparameters $\lambda$}
\SetKwFunction{WeightedAUC}{WeightedAUC}
\BlankLine
\textbf{Label:} Assign labels $y = \mathbb{I}(x \in X_2^{te})$,
$\forall x \in \mathcal{D}_2$\;
\textbf{Calibrate:} Fit the score function
$\phi = \Phi(\mathcal{D}_1, \lambda)$\;
\textbf{Score:} Score observations $s = \phi(x)$, $\forall x \in \mathcal{D}_2$\;
\textbf{Test:} Compute the observed WAUC
$T_0 = \WeightedAUC(s, y)$ using (\ref{eqn:wauc})\;
\textbf{Asymptotic null:} Under the null, $T$ is asymptotically
normally distributed as in (\ref{eqn:null})\;
\BlankLine
\KwOut{$p-value = 1 - \Pr(T \le T_0)$}
\caption{Split-sample test (DSOS-SS)}
\label{algo:dsos-2}
\end{algorithm}
We make the following pragmatic choices for ease of use; typically, the
selected score functions perform well out-of-the-box with little to no
costly hyperparameter tuning. For anomaly or out-of-distribution
detection, the outlier scores are obtained with an isolation forest
using \texttt{isotree} (Cortes
\protect\hyperlink{ref-cortes2020package}{2020}). We take the default
hyperparamters from \texttt{isotree} as given. To investigate other
notions of outlyingness, we use random forests from \texttt{ranger}
(Wright and Ziegler \protect\hyperlink{ref-wright2015ranger}{2017}). We
take its default hyperparameters from Probst, Boulesteix, and Bischl
(\protect\hyperlink{ref-probst2019tunability}{2019}). As in Hediger,
Michel, and Näf (\protect\hyperlink{ref-hediger2019use}{2019}), a random
forest allows us to use the out-of-sample variant of \texttt{D-SOS}
(\texttt{DSOS-AT}) for free, so to speak, since out-of-bag predictions
are viable surrogates for out-of-sample scores. For calibration, random
forest is to two-sample classification, residual diagnostic and
resampling uncertainty (uncertainty quantification) what isolation
forest is to anomaly detection. Of course, we can plug in other
algorithms -- choose your favourites -- to obtain reasonable outlier
scores.
\section{Experiments}\label{sec:experiments}
All experiments were run on a commodity desktop computer with a 12-core
Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz processor and 64 GB RAM in R
version 3.6.1 (2019-07-05). We stress that no hyperparameter tuning was
performed - we set the hyperparameters to reasonable defaults as
previously discussed. To avoid ambiguity, we explicitly state when the
results pertain to \texttt{DSOS-SS}, \texttt{DSOS-PT} or
\texttt{DSOS-AT}.
\subsection{Simulated shifts}\label{exp:simulation}
We compare \texttt{D-SOS} to two modern tests of equal distribution
based on simulated data. The first is a classifier two-sample test,
\texttt{ctst} for short. \texttt{ctst} tests whether a classifier can
reliably distinguish training from test instances. If so, it is taken as
evidence against the null. Lopez-Paz and Oquab
(\protect\hyperlink{ref-lopez2016revisiting}{2017}) and Cai, Goggin, and
Jiang (\protect\hyperlink{ref-cai2020two}{2020}) show that \texttt{ctst}
can match the performance of kernel-based tests. The second is the
\texttt{energy} test (Székely, Rizzo, and others
\protect\hyperlink{ref-szekely2004testing}{2004}), a type of kernel test
(Sejdinovic et al.
\protect\hyperlink{ref-sejdinovic2012hypothesis}{2012}). To be
consistent with \texttt{D-SOS}, we implement \texttt{ctst} with the same
classifier and hyperparameters. Following Clémençon, Depecker, and
Vayatis (\protect\hyperlink{ref-ciemenccon2009auc}{2009}), this
\texttt{ctst} variant uses sample splitting, as does \texttt{DSOS-SS},
and the AUC as a test statistic. Both \texttt{ctst} and \texttt{D-SOS}
use the classifier's prediction (probability) as its scores. As this
score increases, so does the likelihood that an observation belongs to
the test set, as opposed to the training set.
We simulate shifts from a two-component multivariate Gaussian mixture
model (GMM). The training and test set are drawn from:
\begin{align*} \label{eq:simulated}
X^{tr} \sim \phi^{tr} \cdot \mathcal{N}_{d}(\mathbf{\mu_1^{tr}}, \mathbf{\Sigma_1^{tr}}) + (1 - \phi^{tr}) \cdot \mathcal{N}_{d}(\mathbf{\mu_2^{tr}}, \mathbf{\Sigma_2^{tr}}) \\
X^{te} \sim \phi^{te} \cdot \mathcal{N}_{d}(\mathbf{\mu_1^{te}}, \mathbf{\Sigma_1^{te}}) + (1 - \phi^{te}) \cdot \mathcal{N}_{d}(\mathbf{\mu_2^{te}}, \mathbf{\Sigma_2^{te}})
\end{align*}
Omitting subscripts and superscripts for brevity, \(\phi \in [0, 1]\),
\(\mathbf{\mu}\) and \(\mathbf{\Sigma}\) are the component weight, mean
vector and covariance matrix respectively. The baseline specifies the
training and test sample size
\(n^{tr} = n^{te} \in \{400, 800, 1600\}\), the number of dimensions
\(d \in \{4, 8, 16\}\), the component weights
\(\phi^{tr} = \phi^{te} = 0.5\), the mean vectors
\(\mathbf{\mu_1^{tr}} = \mathbf{\mu_1^{te}} = \mathbf{1}_{d}\) and
\(\mathbf{\mu_2^{tr}} = \mathbf{\mu_2^{te}} = -\mathbf{1}_{d}\) and the
covariance matrices
\(\mathbf{\Sigma_1^{tr}} = \mathbf{\Sigma_1^{te}} = \mathbf{\Sigma_2^{tr}} = \mathbf{\Sigma_2^{te}} = \mathbf{I}_{d}\).
\(\mathbf{I}_{d}\) is the \(d \times d\) identity matrix and
\(\mathbf{1}_{d}\) is the \(d\)-dimensional all-ones vector. The
baseline configurations enforce that training and test set are drawn
from the same distribution, i.e.~no shift.
We generally construct ``fair'' alternatives so that the dimension of
change \(d^{*} = 1\) is fixed as the ambient dimension \(d\) increases.
The power of multivariate tests based on kernels and distances decays
with increasing dimension when differences only exist along a few
intrinsic dimensions \(d^{*} \ll d\) (Ramdas et al.
\protect\hyperlink{ref-ramdas2015decreasing}{2015}). We vary one or more
parameters, namely \(\phi^{tr}\), \(\phi^{te}\), \(\mathbf{\mu_2^{tr}}\)
and \(\mathbf{\Sigma_2^{tr}}\), to simulate the desired shifts, all else
constant. We change the following settings to pre-set intensity levels:
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
Label shift -- We flip the weights so that
\(\phi^{tr} \in \{0.49, 0.47, 0.45\}\) goes with
\(\phi^{te} = 1 - \phi^{tr}\). The majority component in training
becomes the minority in the test sample.
\item
Corrupted sample -- We draw a fraction
\(\omega \in \{0.01, 0.02, 0.04\}\) of examples in the test set from
the component that is absent in training such that \(\phi^{tr} = 1\)
and \(\phi^{te} = 1 - \omega\).
\item
Mean shift -- We change the mean vector in the test set so that
\(\mathbf{\mu_2^{te}} = [\frac{\kappa}{10}, \mathbf{1}_{d-1}]\), where
\(\kappa \in \{11, 12, 14\}\).
\item
Noise shift -- We change the covariance matrix in the test set so that
\(diag(\mathbf{\Sigma_1^{te}}) = [\frac{\theta}{10}, \mathbf{1}_{d-1}]\),
where \(\theta\in \{11, 12, 14\}\) and \(diag(\cdot)\) is the
assignment operator for the diagonal elements of the \(d\)-by-\(d\)
covariance matrix.
\item
Dependency shift -- We induce a positive relationship between the
first two covariates. We change the covariance structure in the test
set so that
\(\mathbf{\Sigma_2^{te}}[1, 2] = \mathbf{\Sigma_2^{te}}[2, 1] = \gamma\),
where \(\gamma \in \{0.1, 0.2, 0.4\}\).
\end{enumerate}
For each type and intensity of shift, we repeat the experiment 500
times. To compare \texttt{D-SOS} to both \texttt{ctst} and
\texttt{energy}, we employ the Bayesian signed rank test (Benavoli et
al. \protect\hyperlink{ref-benavoli2017time}{2017},
\protect\hyperlink{ref-benavoli2014bayesian}{2014}). To do so, we
specify a region of practical equivalence (ROPE) on the \(s\)-value
scale and a suitable prior. We deem two statistical tests practically
equivalent if the absolute difference in \(s\)-value is \(\leq 1\). This
difference corresponds to one more coin flip turning up as heads (or
tails), keeping the streak alive. Anything greater arouses suspicion
that one test is more powerful, and that the difference is not
negligible. Specifying a ROPE on the \(p\)-value scale is a lot more
cumbersome because \(p\)-values, let alone \(p\)-value differences, are
notoriously difficult to interpret (Wasserstein, Lazar, and others
\protect\hyperlink{ref-wasserstein2016asa}{2016}). The Bayesian signed
rank test yields the posterior probability that one method is better,
practically equivalent or worse than the other. The prior for these
experiments adds one \emph{pseudo} experiment, distinct from the 500
\emph{real} simulations, where the difference is 0, i.e.~no practical
difference exists out of the gate.
\captionsetup[table]{labelformat=simple,skip=1pt}
\begin{longtable}{llrrr}
\toprule
\multicolumn{2}{c}{Contender} & \multicolumn{3}{c}{Outcome} \\
\cmidrule(lr){1-2} \cmidrule(lr){3-5}
(1) & (2) & Tie/Draw\textsuperscript{1} & Win (1) & Win (2) \\
\midrule
ctst & energy & 89 & 12 & 43 \\
ctst & DSOS-SS & 110 & 0 & 34 \\
ctst & DSOS-PT & 84 & 0 & 60 \\
ctst & DSOS-AT & 80 & 0 & 64 \\
DSOS-AT & energy & 81 & 37 & 26 \\
DSOS-SS & DSOS-AT & 108 & 0 & 36 \\
DSOS-SS & DSOS-PT & 112 & 0 & 32 \\
DSOS-AT & DSOS-PT & 144 & 0 & 0 \\
\bottomrule
\caption{Test comparisons based on simulations.}
\label{tab:simulations}
\end{longtable}
\vspace{-5mm}
\begin{minipage}{\linewidth}
\textsuperscript{1}Tied if the absolute difference in s-value (ROPE) is \textless{}= 1. \\
\end{minipage}
To remain concise, the full tables for comparison are provided as
supplementary material. Table \ref{tab:simulations} summarizes these
findings across all settings: dimension, sample size, type and intensity
of shift. For simplicity, we say that one method dominates (wins) if the
posterior probability is \(\geq 0.5\); similarly, they draw if the
posterior probability that they are tied (practically equivalent) is
\(\geq 0.5\). We make several observations:
\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
Across all settings, \texttt{DSOS-SS} is either better than or equal
to \texttt{ctst}. This forms the basis for our suggestion that tests
of goodness-of-fit, of which \texttt{D-SOS} is a member, supplants
tests of equal distributions when a shift is suspected. The latter
does not consider the training set as the reference distribution and
treats both training and test set as if they were equally important.
It ignores that the training set precedes the test set, that the
predictive model is built on the training set, and not vice versa.
Note, for example, that all the tests explored in Rabanser, Günnemann,
and Lipton (\protect\hyperlink{ref-rabanser2019failing}{2019}) for
shift detection are tests of equal distributions.
\item
Across all settings, \texttt{DSOS-PT} is either better than or equal
to \texttt{DSOS-SS}. The same applies to \texttt{DSOS-AT}.
\texttt{DSOS-SS} pays a hefty price in calibration accuracy for the
privilege of using the asymptotic null distribution. \texttt{DSOS-AT}
and \texttt{DSOS-PT} are practically equivalent. If feasible, we
recommend these two over \texttt{DSOS-SS}. Sample splitting stunts the
performance of \texttt{DSOS-SS} and \texttt{ctst} relative to the
field.
\item
\texttt{DSOS-AT} (or \texttt{DSOS-PT}) is often better than or equal
to \texttt{energy} except in settings with (1) label shift and (2)
corrupted samples. As expected, the WAUC is not unduly influenced by
outliers in the way that nonrobust statistics are. This resilience is
desirable to combat false alarms. While still attentive to outlying
regions, our approach does not break down easily.
\item
\texttt{DSOS-AT} (or \texttt{DSOS-PT}) dominates \texttt{energy} for
dependency shifts. This is consistent with Cai, Goggin, and Jiang
(\protect\hyperlink{ref-cai2020two}{2020}), but somewhat at odds with
Hediger, Michel, and Näf
(\protect\hyperlink{ref-hediger2019use}{2019}) per se. In the latter,
they induce positive association between many (all) variables, whereas
here, in the spirit of `fair' alternatives, only the correlation
between the first two variables is affected.
\end{enumerate}
\subsection{Partition-induced shifts}\label{exp:cc18}
To investigate how notions of outlyingness are correlated, we analyze 62
real-world and influential classification tasks from the OpenML-CC18
benchmark (Casalicchio et al. \protect\hyperlink{ref-OpenMLR2017}{2017};
Bischl et al. \protect\hyperlink{ref-bischl2017openml}{2017})\footnote{We
exclude datasets with more than 500 features because of long runtimes.}.
Similar to the motivating example in Section \ref{sec:iris}, we look at
tests of no adverse shifts based on two-sample classification, residual
diagnostics, and resampling uncertainty (but with no outlier detection
in this round). For each dataset in the OpenML-CC18 benchmark, we
perform stratified 10-fold cross-validation, repeated twice. We end up
with 20 train-test splits per task. In total, we run 3720 tests of no
adverse shifts (62 datasets, 20 random splits, and 3 tests). Summary
statistics and granular test results for these datasets are in the
supplementary material.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images/cc18_heatmap.png}
\caption{
Heatmap of fixed effects for 62 datasets from the OpenML-CC18 benchmark,
ordered via hierarchical clustering. Fixed effects are reported on the
$s$-value scale for 3 different \texttt{D-SOS} tests.}
\label{fig:heatmap-cc18}
\end{figure}
We expect the \texttt{D-SOS} results to be correlated within but not
across datasets. To formalize this setup, we use the following model:
\begin{equation} \label{eq:cc18}
\log s_i^j \sim \mathcal{N}(\mu_i^j, \Sigma_i^j)
\end{equation}
where for each dataset \(i = 1, 2, \cdots, 62\) and test
\(j = 1, 2, 3\), the \(s-\)value \(s_i^j\) is lognormally distributed:
positive and skewed to the right. The \(s-\)value \(s_i^j\) consists of
a dataset-specific (fixed) effect \(\mu_i^j\), subject to noise in
\(\Sigma_i^j\); \(\Sigma_i^j\) accounts for within-dataset correlation.
The specification in \ref{eq:cc18} gives rise to the so-called fixed
effects model with clustered covariance, which is widely used in
econometrics and biostatistics. We fit such a linear regression using
the \texttt{clubSandwich} package (Pustejovsky and Tipton
\protect\hyperlink{ref-pustejovsky2018small}{2018}) to obtain robust
estimates of the fixed effects \(\mu_i^j\) even with arbitrary
heteroskedasticity and autocorrelation in \(\Sigma_i^j\) left
unspecified. All fixed effects are statistically significant
(\(p \leq 0.001\)).
\begin{figure}
\centering
\includegraphics[width=\linewidth]{images/cc18_correlation.png}
\caption{
Correlation matrix of fixed effects for 62 datasets from the
OpenML-CC18 benchmark across 3 notions of outlyingness.}
\label{fig:corr-cc18}
\end{figure}
In stark contrast with worst-case (say, adversarial) shifts used to
assess or build stable models e.g.~Subbaswamy, Adams, and Saria
(\protect\hyperlink{ref-subbaswamy2021evaluating}{2021}) and Duchi and
Namkoong (\protect\hyperlink{ref-duchi2018learning}{2018}),
cross-validation generates partition-induced shifts (Jose García
Moreno-Torres, Sáez, and Herrera
\protect\hyperlink{ref-moreno2012study}{2012}). These shifts supposedly
mimic the \emph{natural} variation in the data and are less harsh or
extreme than the worst-case shifts. The fixed effects \(\mu_i^j\) in
\ref{eq:cc18} measure how sensitive or robust a dataset is to these
partition-induced shifts. In other words, these are
\textbf{interpretable} and quantitative metrics for data monitoring or
validation. Recall that data validation via statistical tests, to echo
Polyzotis et al. (\protect\hyperlink{ref-polyzotis2019data}{2019}), is
our main concern.
At a glance, the heatmap along with the dendrogram for clustering in
Figure \ref{fig:heatmap-cc18}, created with \texttt{superheat} (Barter
and Yu \protect\hyperlink{ref-barter2018superheat}{2018}), tells the
whole story quite succintly. Across 62 datasets, we see that tests of no
adverse shifts based on two-sample classification, residual diagnostics,
and resampling uncertainty are indeed highly correlated. Figure
\ref{fig:corr-cc18} shows this correlation matrix\footnote{We use the
method in Schäfer and Strimmer
(\protect\hyperlink{ref-schafer2005shrinkage}{2005}) to estimate the
correlation matrix.}. These three distinct notions of outlyingness are
distinctly not independent: they leak information about one another.
Residual diagnostics and resampling uncertainty, both of which
incorporate facets of the predictive performance, are the most
correlated. This suggests that they can be used interchangeably.
Two-sample classification is not as strongly associated with the other
two. This connects back to and further supports the point made in
Section \ref{sec:iris}; namely, tests for distributional shifts alone
are ill-equiped, by definition, to detect whether a shift is benign or
harmful for predictive tasks. If the key criterion is predictive
performance, \texttt{D-SOS} tests based on residual diagnostics and
resampling uncertainty are more informative than those solely based on
comparing distributions.
\section{Conclusion}\label{sec:conclusion}
\texttt{D-SOS} is a framework derived from goodness-of-fit testing to
detect adverse shifts. It can confront the data with more informative
hypotheses than tests of equal distributions. This works well when
mapping to the relevant subspace is based on distinguishing between safe
and unsafe regions of the data distribution. Our method accommodates
different notions of outlyingness. It lets users define, via a score
function, what is meant by `worse' off. Besides the outlier scores
explored in this paper, we stress that other sensible choices for the
score function \(\phi\) and the weights \(w\) abound. These choices can
be adjusted to reflect prior domain knowledge and serve the needs of the
data scientist. Looking ahead, future research could investigate how
different weighting schemes affect the power of the test. The functional
form of the postulated weights could be a hyperparameter worth tuning.
Moreover, \emph{composite} score functions, which would combine several
notions of outlyingness together, would enrich the types of hypotheses
that could be tested.
\section{Acknowledgements}\label{sec:acknowledgements}
The author would like to thank the reviewers for suggesting numerous
improvements. This work was supported by the Royal Bank of Canada (RBC).
The views expressed here are those of the author, not of RBC.
\section{References}
\hypertarget{refs}{}
\leavevmode\hypertarget{ref-aguinis2013best}{}%
Aguinis, Herman, Ryan K Gottfredson, and Harry Joo. 2013.
``Best-Practice Recommendations for Defining, Identifying, and Handling
Outliers.'' \emph{Organizational Research Methods} 16 (2): 270--301.
\leavevmode\hypertarget{ref-anderson1935irises}{}%
Anderson, Edgar. 1935. ``The Irises of the Gaspe Peninsula.''
\emph{Bull. Am. Iris Soc.} 59: 2--5.
\leavevmode\hypertarget{ref-barter2018superheat}{}%
Barter, Rebecca L, and Bin Yu. 2018. ``Superheat: An R Package for
Creating Beautiful and Extendable Heatmaps for Visualizing Complex
Data.'' \emph{Journal of Computational and Graphical Statistics} 27 (4):
910--22.
\leavevmode\hypertarget{ref-benavoli2017time}{}%
Benavoli, Alessio, Giorgio Corani, Janez Demšar, and Marco Zaffalon.
2017. ``Time for a Change: A Tutorial for Comparing Multiple Classifiers
Through Bayesian Analysis.'' \emph{The Journal of Machine Learning
Research} 18 (1): 2653--88.
\leavevmode\hypertarget{ref-benavoli2014bayesian}{}%
Benavoli, Alessio, Giorgio Corani, Francesca Mangili, Marco Zaffalon,
and Fabrizio Ruggeri. 2014. ``A Bayesian Wilcoxon Signed-Rank Test Based
on the Dirichlet Process.'' In \emph{International Conference on Machine
Learning}, 1026--34. PMLR.
\leavevmode\hypertarget{ref-bischl2017openml}{}%
Bischl, Bernd, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter,
Michel Lang, Rafael G Mantovani, Jan N van Rijn, and Joaquin Vanschoren.
2017. ``OpenML Benchmarking Suites.'' \emph{arXiv Preprint
arXiv:1708.03731}.
\leavevmode\hypertarget{ref-cai2020two}{}%
Cai, Haiyan, Bryan Goggin, and Qingtang Jiang. 2020. ``Two-Sample Test
Based on Classification Probability.'' \emph{Statistical Analysis and
Data Mining: The ASA Data Science Journal} 13 (1): 5--13.
\leavevmode\hypertarget{ref-OpenMLR2017}{}%
Casalicchio, Giuseppe, Jakob Bossek, Michel Lang, Dominik Kirchhoff,
Pascal Kerschke, Benjamin Hofner, Heidi Seibold, Joaquin Vanschoren, and
Bernd Bischl. 2017. ``OpenML: An R Package to Connect to the Machine
Learning Platform Openml.'' \emph{Computational Statistics} 32 (3):
1--15. \url{https://doi.org/10.1007/s00180-017-0742-2}.
\leavevmode\hypertarget{ref-cieslak2009framework}{}%
Cieslak, David A, and Nitesh V Chawla. 2009. ``A Framework for
Monitoring Classifiers' Performance: When and Why Failure Occurs?''
\emph{Knowledge and Information Systems} 18 (1): 83--108.
\leavevmode\hypertarget{ref-ciemenccon2009auc}{}%
Clémençon, Stéphan, Marine Depecker, and Nicolas Vayatis. 2009. ``AUC
Optimization and the Two-Sample Problem.'' In \emph{Proceedings of the
22nd International Conference on Neural Information Processing Systems},
360--68.
\leavevmode\hypertarget{ref-cortes2020package}{}%
Cortes, David. 2020. \emph{Isotree: Isolation-Based Outlier Detection}.
\url{https://CRAN.R-project.org/package=isotree}.
\leavevmode\hypertarget{ref-demidenko2016p}{}%
Demidenko, Eugene. 2016. ``The P-Value You Can't Buy.'' \emph{The
American Statistician} 70 (1): 33--38.
\leavevmode\hypertarget{ref-duchi2018learning}{}%
Duchi, John, and Hongseok Namkoong. 2018. ``Learning Models with Uniform
Performance via Distributionally Robust Optimization.'' \emph{arXiv
Preprint arXiv:1810.08750}.
\leavevmode\hypertarget{ref-friedman2004multivariate}{}%
Friedman, Jerome. 2004. ``On Multivariate Goodness-of-Fit and Two-Sample
Testing.'' Stanford Linear Accelerator Center, Menlo Park, CA (US).
\leavevmode\hypertarget{ref-gandy2009sequential}{}%
Gandy, Axel. 2009. ``Sequential Implementation of Monte Carlo Tests with
Uniformly Bounded Resampling Risk.'' \emph{Journal of the American
Statistical Association} 104 (488): 1504--11.
\leavevmode\hypertarget{ref-greenland2019valid}{}%
Greenland, Sander. 2019. ``Valid P-Values Behave Exactly as They Should:
Some Misleading Criticisms of P-Values and Their Resolution with
S-Values.'' \emph{The American Statistician} 73 (sup1): 106--14.
\leavevmode\hypertarget{ref-hand2009measuring}{}%
Hand, David J. 2009. ``Measuring Classifier Performance: A Coherent
Alternative to the Area Under the Roc Curve.'' \emph{Machine Learning}
77 (1): 103--23.
\leavevmode\hypertarget{ref-hediger2019use}{}%
Hediger, Simon, Loris Michel, and Jeffrey Näf. 2019. ``On the Use of
Random Forest for Two-Sample Testing.'' \emph{arXiv Preprint
arXiv:1903.06287}.
\leavevmode\hypertarget{ref-hendrycks2016baseline}{}%
Hendrycks, Dan, and Kevin Gimpel. 2017. ``A Baseline for Detecting
Misclassified and Out-of-Distribution Examples in Neural Networks.'' In
\emph{5th International Conference on Learning Representations, ICLR
2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings}.
OpenReview.net. \url{https://openreview.net/forum?id=Hkg4TI9xl}.
\leavevmode\hypertarget{ref-jankova2020goodness}{}%
Janková, Jana, Rajen D Shah, Peter Bühlmann, and Richard J Samworth.
2020. ``Goodness-of-Fit Testing in High Dimensional Generalized Linear
Models.'' \emph{Journal of the Royal Statistical Society: Series B
(Statistical Methodology)} 82 (3): 773--95.
\leavevmode\hypertarget{ref-jiang2018trust}{}%
Jiang, Heinrich, Been Kim, Melody Y Guan, and Maya R Gupta. 2018. ``To
Trust or Not to Trust a Classifier.'' In \emph{NeurIPS}, 5546--57.
\leavevmode\hypertarget{ref-kelly1999impact}{}%
Kelly, Mark G, David J Hand, and Niall M Adams. 1999. ``The Impact of
Changing Populations on Classifier Performance.'' In \emph{Proceedings
of the Fifth Acm Sigkdd International Conference on Knowledge Discovery
and Data Mining}, 367--71.
\leavevmode\hypertarget{ref-kirchler2020two}{}%
Kirchler, Matthias, Shahryar Khorasani, Marius Kloft, and Christoph
Lippert. 2020. ``Two-Sample Testing Using Deep Learning.'' In
\emph{International Conference on Artificial Intelligence and
Statistics}, 1387--98. PMLR.
\leavevmode\hypertarget{ref-li2010weighted}{}%
Li, Jialiang, and Jason P Fine. 2010. ``Weighted Area Under the Receiver
Operating Characteristic Curve and Its Application to Gene Selection.''
\emph{Journal of the Royal Statistical Society: Series C (Applied
Statistics)} 59 (4): 673--92.
\leavevmode\hypertarget{ref-lipton2018detecting}{}%
Lipton, Zachary, Yu-Xiang Wang, and Alexander Smola. 2018. ``Detecting
and Correcting for Label Shift with Black Box Predictors.'' In
\emph{International Conference on Machine Learning}, 3122--30. PMLR.
\leavevmode\hypertarget{ref-lopez2016revisiting}{}%
Lopez-Paz, David, and Maxime Oquab. 2017. ``Revisiting Classifier
Two-Sample Tests.'' In \emph{5th International Conference on Learning
Representations, ICLR 2017, Toulon, France, April 24-26, 2017,
Conference Track Proceedings}. OpenReview.net.
\url{https://openreview.net/forum?id=SJkXfE5xx}.
\leavevmode\hypertarget{ref-marozzi2004some}{}%
Marozzi, Marco. 2004. ``Some Remarks About the Number of Permutations
One Should Consider to Perform a Permutation Test.'' \emph{Statistica}
64 (1): 193--201.
\leavevmode\hypertarget{ref-menon2016linking}{}%
Menon, Aditya, and Cheng Soon Ong. 2016. ``Linking Losses for Density
Ratio and Class-Probability Estimation.'' In \emph{International
Conference on Machine Learning}, 304--13.
\leavevmode\hypertarget{ref-moreno2012study}{}%
Moreno-Torres, Jose García, José A Sáez, and Francisco Herrera. 2012.
``Study on the Impact of Partition-Induced Dataset Shift on \(k\)-Fold
Cross-Validation.'' \emph{IEEE Transactions on Neural Networks and
Learning Systems} 23 (8): 1304--12.
\leavevmode\hypertarget{ref-moreno2012unifying}{}%
Moreno-Torres, Jose G, Troy Raeder, RocíO Alaiz-RodríGuez, Nitesh V
Chawla, and Francisco Herrera. 2012. ``A Unifying View on Dataset Shift
in Classification.'' \emph{Pattern Recognition} 45 (1): 521--30.
\leavevmode\hypertarget{ref-morningstar2021density}{}%
Morningstar, Warren, Cusuh Ham, Andrew Gallagher, Balaji
Lakshminarayanan, Alex Alemi, and Joshua Dillon. 2021. ``Density of
States Estimation for Out of Distribution Detection.'' In
\emph{International Conference on Artificial Intelligence and
Statistics}, 3232--40. PMLR.
\leavevmode\hypertarget{ref-polyzotis2019data}{}%
Polyzotis, Neoklis, Martin Zinkevich, Sudip Roy, Eric Breck, and Steven
Whang. 2019. ``Data Validation for Machine Learning.'' \emph{Proceedings
of Machine Learning and Systems} 1: 334--47.
\leavevmode\hypertarget{ref-probst2019tunability}{}%
Probst, Philipp, Anne-Laure Boulesteix, and Bernd Bischl. 2019.
``Tunability: Importance of Hyperparameters of Machine Learning
Algorithms.'' \emph{Journal of Machine Learning Research} 20 (53):
1--32.
\leavevmode\hypertarget{ref-pustejovsky2018small}{}%
Pustejovsky, James E, and Elizabeth Tipton. 2018. ``Small-Sample Methods
for Cluster-Robust Variance Estimation and Hypothesis Testing in Fixed
Effects Models.'' \emph{Journal of Business \& Economic Statistics} 36
(4): 672--83.
\leavevmode\hypertarget{ref-quionero2009dataset}{}%
Quionero-Candela, Joaquin, Masashi Sugiyama, Anton Schwaighofer, and
Neil D Lawrence. 2009. \emph{Dataset Shift in Machine Learning}. The MIT
Press.
\leavevmode\hypertarget{ref-rabanser2019failing}{}%
Rabanser, Stephan, Stephan Günnemann, and Zachary Lipton. 2019.
``Failing Loudly: An Empirical Study of Methods for Detecting Dataset
Shift.'' In \emph{Advances in Neural Information Processing Systems},
1394--1406.
\leavevmode\hypertarget{ref-ramdas2015decreasing}{}%
Ramdas, Aaditya, Sashank Jakkam Reddi, Barnabás Póczos, Aarti Singh, and
Larry Wasserman. 2015. ``On the Decreasing Power of Kernel and Distance
Based Nonparametric Hypothesis Tests in High Dimensions.'' In
\emph{Twenty-Ninth Aaai Conference on Artificial Intelligence}.
\leavevmode\hypertarget{ref-rinaldo2019bootstrapping}{}%
Rinaldo, Alessandro, Larry Wasserman, Max G'Sell, and others. 2019.
``Bootstrapping and Sample Splitting for High-Dimensional,
Assumption-Lean Inference.'' \emph{The Annals of Statistics} 47 (6):
3438--69.
\leavevmode\hypertarget{ref-ruff2021unifying}{}%
Ruff, Lukas, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire
Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and
Klaus-Robert Müller. 2021. ``A Unifying Review of Deep and Shallow
Anomaly Detection.'' \emph{Proceedings of the IEEE}.
\leavevmode\hypertarget{ref-schafer2005shrinkage}{}%
Schäfer, Juliane, and Korbinian Strimmer. 2005. ``A Shrinkage Approach
to Large-Scale Covariance Matrix Estimation and Implications for
Functional Genomics.'' \emph{Statistical Applications in Genetics and
Molecular Biology} 4 (1).
\leavevmode\hypertarget{ref-schulam2019can}{}%
Schulam, Peter, and Suchi Saria. 2019. ``Can You Trust This Prediction?
Auditing Pointwise Reliability After Learning.'' In \emph{The 22nd
International Conference on Artificial Intelligence and Statistics},
1022--31. PMLR.
\leavevmode\hypertarget{ref-sejdinovic2012hypothesis}{}%
Sejdinovic, Dino, Arthur Gretton, Bharath Sriperumbudur, and Kenji
Fukumizu. 2012. ``Hypothesis Testing Using Pairwise Distances and
Associated Kernels.'' In \emph{Proceedings of the 29th International
Coference on International Conference on Machine Learning}, 787--94.
\leavevmode\hypertarget{ref-sethi2017reliable}{}%
Sethi, Tegjyot Singh, and Mehmed Kantardzic. 2017. ``On the Reliable
Detection of Concept Drift from Streaming Unlabeled Data.'' \emph{Expert
Systems with Applications} 82: 77--99.
\leavevmode\hypertarget{ref-snoek2019can}{}%
Snoek, Jasper, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan,
Sebastian Nowozin, D Sculley, Joshua Dillon, Jie Ren, and Zachary Nado.
2019. ``Can You Trust Your Model's Uncertainty? Evaluating Predictive
Uncertainty Under Dataset Shift.'' In \emph{Advances in Neural
Information Processing Systems}, 13969--80.
\leavevmode\hypertarget{ref-subbaswamy2021evaluating}{}%
Subbaswamy, Adarsh, Roy Adams, and Suchi Saria. 2021. ``Evaluating Model
Robustness and Stability to Dataset Shift.'' In \emph{International
Conference on Artificial Intelligence and Statistics}, 2611--9. PMLR.
\leavevmode\hypertarget{ref-szekely2004testing}{}%
Székely, Gábor J, Maria L Rizzo, and others. 2004. ``Testing for Equal
Distributions in High Dimension.'' \emph{InterStat} 5 (16.10): 1249--72.
\leavevmode\hypertarget{ref-wasserstein2016asa}{}%
Wasserstein, Ronald L, Nicole A Lazar, and others. 2016. ``The Asa's
Statement on P-Values: Context, Process, and Purpose.'' \emph{The
American Statistician} 70 (2): 129--33.
\leavevmode\hypertarget{ref-wellek2010testing}{}%
Wellek, Stefan. 2010. \emph{Testing Statistical Hypotheses of
Equivalence and Noninferiority}. CRC Press.
\leavevmode\hypertarget{ref-wen2014robust}{}%
Wen, Junfeng, Chun-Nam Yu, and Russell Greiner. 2014. ``Robust Learning
Under Uncertain Test Distributions: Relating Covariate Shift to Model
Misspecification.'' In \emph{ICML}, 631--39.
\leavevmode\hypertarget{ref-weng2008new}{}%
Weng, Cheng G, and Josiah Poon. 2008. ``A New Evaluation Measure for
Imbalanced Datasets.'' In \emph{Proceedings of the 7th Australasian Data
Mining Conference-Volume 87}, 27--32.
\leavevmode\hypertarget{ref-wright2015ranger}{}%
Wright, Marvin N., and Andreas Ziegler. 2017. ``ranger: A Fast
Implementation of Random Forests for High Dimensional Data in C++ and
R.'' \emph{Journal of Statistical Software} 77 (1): 1--17.
\url{https://doi.org/10.18637/jss.v077.i01}.
\leavevmode\hypertarget{ref-zhang2002powerful}{}%
Zhang, Jin. 2002. ``Powerful Goodness-of-Fit Tests Based on the
Likelihood Ratio.'' \emph{Journal of the Royal Statistical Society:
Series B (Statistical Methodology)} 64 (2): 281--94.
\bibliographystyle{apalike}
\bibliography{references.bib}
\end{document}