Neural Networks beyond explainability: Selective inference for sequence motifs
Extracted source text · arXiv 2212.12542
\documentclass[10pt]{article} % For LaTeX2e
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
%\usepackage[english]{babel}
\usepackage{dsfont}
\usepackage{natbib}
\usepackage{caption}
\usepackage{enumerate}
%\bibliographystyle{apalike}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage[algo2e,norelsize,ruled,vlined,commentsnumbered,linesnumbered]{algorithm2e}
\newcommand{\nonl}{\renewcommand{\nl}{\let\nl\oldnl}}% Remove line number for one line
\newcommand*{\Sqcup}{\sqcup}
% If accepted, instead use the following line for the camera-ready submission:
%\usepackage[accepted]{tmlr}
% To de-anonymize and remove mentions to TMLR (for example for posting to preprint servers), instead use the following:
\usepackage[preprint]{tmlr}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{tikz}
\usetikzlibrary{calc,decorations.pathreplacing}
\usepackage{pgfplots}
\pgfplotsset{width=7cm,compat=1.16}
\usepgfplotslibrary{ternary}
\usepackage{hyperref}
\definecolor{burgundy}{rgb}{0.5, 0.0, 0.13}
\definecolor{camel}{rgb}{0.76, 0.6, 0.42}
\definecolor{chamoisee}{rgb}{0.63, 0.47, 0.35}
\definecolor{grey1}{RGB}{128,128,128}
\def\yo{\textcolor{chamoisee}}
\hypersetup{colorlinks=true,linkcolor=burgundy,citecolor=chamoisee,urlcolor=burgundy,linktoc=page}
\usepackage{url}
%\usepackage[dvips]{graphicx}
\newenvironment{sloppypar*}
{\sloppy\ignorespaces}
{\par}
\title{Neural Networks beyond explainability: \\ Selective inference for sequence motifs}
% Authors must not appear in the submitted version. They should be hidden
% as long as the tmlr package is used without the [accepted] or [preprint] options.
% Non-anonymous submissions will be rejected without review.
\author{\name Antoine Villié \email antoine.villie@univ-lyon1.fr \\
\addr Université de Lyon, Université Lyon 1, CNRS, VetAgro Sup, Laboratoire de Biométrie et Biologie Evolutive, UMR5558, Villeurbanne, France
\AND
\name Philippe Veber \email philippe.veber@univ-lyon1.fr \\
\addr Université de Lyon, Université Lyon 1, CNRS, VetAgro Sup, Laboratoire de Biométrie et Biologie Evolutive, UMR5558, Villeurbanne, France
\AND
\name Yohann De Castro \email yohann.de-castro@ec-lyon.fr \\
\addr Université de Lyon, École Centrale de Lyon, CNRS UMR 5208, Institut Camille Jordan, France
\AND
\name Laurent Jacob \email laurent.jacob@univ-lyon1.fr \\
\addr Université de Lyon, Université Lyon 1, CNRS, VetAgro Sup, Laboratoire de Biométrie et Biologie Evolutive, UMR5558, Villeurbanne, France
}
% The \author macro works with any number of authors. Use \AND
% to separate the names and addresses of multiple authors.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\def\month{MM} % Insert correct month for camera-ready version
\def\year{YYYY} % Insert correct year for camera-ready version
\def\openreview{\url{https://openreview.net/forum?id=XXXX}} % Insert correct link to OpenReview for camera-ready version
\definecolor{lj-red}{HTML}{AE2012}
\definecolor{ydc-red}{HTML}{EE66AA}
\definecolor{av-blue}{HTML}{003ad6}
\newcommand{\lj}[1]{\textcolor{lj-red}{*** LJ: #1}}
\newcommand{\ydc}[1]{\textcolor{ydc-red}{*** Y: #1}}
\newcommand{\av}[1]{\textcolor{av-blue}{*** AV: #1}}
\SetKw{Inputs}{Inputs:}
\begin{document}
\maketitle
\begin{abstract}
Over the past decade, neural networks have been successful at making
predictions from biological sequences, especially in the context of
regulatory genomics. As in other fields of deep learning, tools have
been devised to extract features such as sequence motifs that can
explain the predictions made by a trained network. Here we intend to
go beyond explainable machine learning and introduce SEISM, a
selective inference procedure to test the association between these
extracted features and the predicted phenotype. In particular, we
discuss how training a one-layer convolutional network is formally
equivalent to selecting motifs maximizing some association score. We
adapt existing sampling-based selective inference procedures by
quantizing this selection over an infinite set to a large but finite
grid. Finally, we show that sampling under a specific choice of
parameters is sufficient to characterize the composite null hypothesis
typically used for selective inference---a result that goes well
beyond our particular framework. We illustrate the behavior of our
method in terms of calibration, power and speed and discuss its
power/speed trade-off with a simpler data-split strategy. SEISM paves
the way to an easier analysis of neural networks used in regulatory
genomics, and to more powerful methods for genome wide association
studies (GWAS).
%We want to make interpretable features amenable to inference. To that extent, we present a general framework, and apply it to the single example of pwms and 1 hidden layer CNN.
%Although neural networks have been successfully applied for making predictions from biological sequences, they are often criticized for their lack of explainability. In the recent years, tools inspired from the explainable deep learning literature have been adapted to extract interpretable features from such models. Despite its importance in experimental science, such as Genome Wide Association Studies, measuring the statistical significance of the association between the features and the outcome has only received little attention in the explainable machine learning framework to our knowledge. We intend to go beyond explainable machine learning with SEISM, a valid statistical inference approach for those features. For the sake of simplicity, we restrict this presentation to one-layer convolutional neural networks, whose filters can be easily interpreted as sequence motifs. We cast those networks in a \textit{de-novo} motifs discovery task by devising a stable stepwise method for selecting a small number of sequence motifs associated with the phenotype, and we then use recent advances in post-selection inference to produce a valid testing procedures for this association.
\end{abstract}
\section{Introduction}
In the recent years, neural networks have been successfully used for
making predictions from biological sequences. In particular, they have
brought significant improvements in regulatory genomics, \emph{e.g.}
to predict cell-type specific transcription factor binding, gene
expression, chromatin accessibility or histone modifications from a
DNA sequence~\citep{zhou2015predicting, kelley2018sequential,avsec2021effective, avsec2021base}. These tasks are expected to be a good proxy for
predicting the functional effect of non-coding variants, and help us
in turn make better sense of the observed human genetic variation and
its effect on various phenotypical traits including diseases. Most successful models have used convolutional neural networks~\citep[CNNs,][]{lecun1998convolutional} and more recent approaches have explored self-attention
mechanisms~\citep{vaswani2017attention}. These models have been trained from
experimental data obtained from ChIP-seq, ATAC-seq, DNase-seq, or CAGE
assays, that provide examples where both the DNA sequence and the
outcome of interest are known.
% [mention lstm, rnns, dnabert?]
% [cite a few among deepbind, deepSEA, basset, basenji, bpnet, explaiNN, enformer]
%[GIA?]
A commonly outlined limitation of neural networks is their lack of
explainability or black box aspect, \emph{i.e.}, the contrast between
their excellent prediction accuracy and the possibility to explain
these in intuitive or mechanistic terms~\citep{ras2022explainable, molnar2022interpretable}. Elementary one-layer CNNs don't face this issue, as their trained filters have a straightforward
interpretation as position weight matrices~\citep[PWMs,][]{harris1983search, schneider1990sequence}, a
historical and basic element of regulatory genomics. Nonetheless,
these simple models are notoriously too simple to capture the
complexity of the regulatory code which requires to account not only
for individual motif presence but for their long range sequence
context and mutual interactions~\citep{avsec2021base}. Multi-layer CNNs and
self-attention mechanisms model this additional complexity but are
less straightforward to interpret. Tools inspired from the
explainable deep learning literature have been adapted to extract
features beyond PWMs and one-layer CNNs to
explain the predicted regulatory behavior~\citep{novakovsky2022obtaining}. It is therefore often possible to explain the predictions of a trained neural network for biological sequences, either directly through estimates of its parameters or through features extracted post hoc.
Unfortunately, finding features somewhat associated to an outcome is
often not enough, as an observed non-zero association can be
spurious. In experimental science, it is actually common to quantify
the significance of this association, \emph{e.g.}, by testing the
hypothesis that it is zero. Genome wide association studies~\citep[GWAS,][]{visscher2017years} for
example find genetic variants correlated with a trait by building a
linear model explaining this trait by each variant and testing the
hypothesis that the weight is zero. Statistical significance has its
own limitations~\citep{wasserstein2016asa}, but often provides an intuitive scale for identifying relevant features. Quantifying the significance of associations between interpretable
features and predicted outcome is equally important in the context of
neural networks but has received little attention to our knowledge.
%These networks build approximation of functions by a finite combination of elements selected from an infinite basis~\citep{bach2017breaking, bietti2019group}. In the simple example of one-layer CNNs, learning a user-chosen number $q$ of convolution filters can also be thought of as selecting $q$ PWMs---among infinitely many others. But this selection of $q$ elements occurs even if the CNN was trained on pure noise, and offers no information on their significance.
%To our knowledge, little has been done to quantify the significance of interpretable features extracted by trained neural networks.
\begin{figure}[h!]
\centering
\includegraphics[scale=0.85]{figures/global_schema4.eps}%
\caption{Overview of our SEISM procedure. (a) The input is a set of sequences and corresponding phenotypes in some space $\mathcal{Y}$ (b) It trains a convolutional neural networks to predict a phenotype from sequences, which leads to the selection of sequence motifs. (c) Then SEISM partitions the space of motifs to quantize the selection. The selection event is the set of phenotype vectors that would lead to selecting an element in the same mesh. (d) Using a sampling strategy, SEISM builds a null distribution for the test statistic, conditional to the selection event. The $p$-values associated with a selected motif is the quantile of its score under this distribution. }
\label{fig:global_scheme}
\end{figure}
Here, we set out to go beyond explainable machine learning by
introducing SElective Inference for Sequence Motifs (SEISM), depicted in Figure \ref{fig:global_scheme}, a valid
statistical inference procedure for these features. In order to do so,
we cast commonly used CNNs in a feature selection framework, and show
that it achieves similar selection performances as existing
bioinformatics algorithms on \textit{de novo} motifs discovery
tasks. This selection needs to be accounted for when testing the
association of the features with the predicted trait. This problem has
been discussed and addressed in the growing literature on selective
inference over the past few years~\citep{taylor2015statistical,
reid2018general, slim2019kernelpsi}, but existing methods only apply
to a selection from a finite set. We work around this issue by
quantizing our selection to a very large but finite space, making it
amenable to existing sampling strategies. We show that the resulting
procedure is well calibrated and compare it to a data-split strategy
on a variety of settings.
For the sake of simplicity, we choose to restrict this presentation to
simple one-layer CNNs and sequence motifs. The procedure that we introduce, however, is by no means limited to this framework, and could be applied to any of the more expressive features proposed in the explainable machine learning literature.
Our contributions are as follows:
\begin{itemize}
\item We formally cast one-layer CNNs into a motif discovery tool, reaching similar performances as \textit{de-novo} motifs discovery tools from the bio-informatics literature (Section \ref{subsec:selection}).
\item We define a framework to perform post-selection inference dealing with selection over a continuous set of features, and thereby we make interpretable features amenable to inference (Section \ref{sec:model}).
\item The standard Gaussian framework for selective inference typically
allows several means to be under the same selective null hypothesis,
and require the variance to be known, both of which make more difficult the
sampling under this null. We provide invariance results suggesting a
practical procedure that works around these issues (Section \ref{sampling_null}). To our knowledge,
they were a blind spot in sampling-based post-selection inference
approaches beyond our specific context.
%We facilitate the use of a classical statistical model on real datasets by studying the composite null hypotheses and proving theoretical results (Section \ref{sampling_null}). \av{We provide statistical guarantees regarding the applicability of the tests performed on such null hypotheses, which to our knowledge was a blind spot in sampling-based post-selection inference approaches.}
\item We provide a PyTorch implementation of SEISM at:
\url{https://gitlab.in2p3.fr/antoine.villie1/seism}.
\end{itemize}
% Our contributions are the following:
% We cast one-layer CNNs as a feature selection process, establishing a formal link between training such a network and finding a set of sequence motifs maximally associated with the outcome for some association score.
% We introduce a selective inference procedure to test the significance of the motifs selected by training the CNN. We compare its power to the one obtained with a more rudimentary data-split procedure.
% We provide a software implementing the introduced procedure.
%Approach: CNN perform well but black boxes. Lots of paper try to identify / construct interpretable features (sequence motifs, interaction maps...). Our approach : try to give a statistical significance to those identified features.
% List of references to include:
% \begin{itemize}
% \item Peter Koo
% \begin{itemize}
% \item Uncovering motif interactions from convolutional-attention networks for genomics
% \item Global importance analysis: An interpretability method to quantify importance of genomic features in deep neural networks
% \item Improving representations of genomic sequence motifs in convolutional networks with activations
% \item Representation learning of genomic sequence motifs with convolutional neural networks
% \item Evaluating deep learning for predicting epigenomic profiles
% \end{itemize}
% \item A convolutional neural network highlights mutations relevant to antimicrobial resistance (Maha Farhat)
% \item DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome
% \item ExplaiNN: interpretable and transparent neural networks for genomics
% \item STREME: Accurate and versatile sequence motif discovery
% \item Learning important features through propagating activation differences
% \item Technical note on transcription factor motif discovery from importance scores (TF-MoDISco)
% \item Base-resolution models of transcription factor binding reveal soft motif syntax
% \end{itemize}
\section{A short overview of our SEISM procedure}
SEISM aims to detect sequence motifs associated with a biological outcome, and to test the statistical significance of this association. To this end, it performs different steps which we will briefly describe here, in order to give the reader an overview of the procedure. They are summarized in Algorithm \ref{alg:seism_general}, and more details will be given in the following sections.
\begin{enumerate}[(i)]
\item SEISM takes as input biological sequences $\bm{X}$ associated with a phenotype~$\vy$. The user must also specify the number of motifs to find, as well as a parameter controlling the meshing of the motif space, that is the precision with which the found motifs will be tested.
\item \label{enum:sel} The motif selection step corresponds to the maximisation of a so-called association score~$s(\cdot,\cdot)$, which depends on the phenotype and on the motifs~$\vz$ through their activation patterns in the biological sequences~$\bm{\varphi}^{\vz, \bm{X}}$. This step is formally equivalent to training a one hidden layer CNN. We implement a greedy procedure, optimizing each new filter over the residuals of the previously entered ones, using a gradient descent method initialized at the~$k$-mer with the best score. To that end, we enumerates the~$k$-mers contained in $\bm X$ using the DSK software \citep{rizk_dsk_2013} and compute their scores~$s(\cdot,\cdot)$.
\item \label{enum:def} SEISM splits the set of sequence motifs into meshes according to the input parameter. This step leads to the definition of a set of null hypotheses and of a selection event~$E$, \textit{i.e.} the set of outcomes~$\vy'$ that would have led to the selection of motifs within the same meshes as the ones selected in~(\ref{enum:sel}), namely the sequence of meshes~$(M_{i_1},\ldots,M_{i_q})$. Formally, the selection event reads
%\begin{equation}
% \label{e:selection_event}
% \displaystyle E(i_1,\ldots,i_q)
% :=
% \Big\{
% \vy' \in \gY\,:~\forall \ell \in [q]\,, ~\argmax_{\vz \in \gZ}s(\vz, \mP_\ell \vy')\in M_{i_\ell}
% \Big\}\,,
%\end{equation}
\begin{equation}
\label{e:selection_event}
\displaystyle E
:=
\Big\{
\vy' \in \gY\,:~\forall j \in [q]\,, ~\argmax_{\vz \in \gZ}s(\vz, \mP_j \vy')\in M_{i_j}
\Big\}\,,
\end{equation}
for some projection matrix $\mP_j$, to be defined later.
\item It approximates the conditional null distribution of the test statistics by sampling biological outcomes~$\vy'$ under the null, conditionally to the selection event. This sampling is performed using a hit-and-run strategy (according to Algorithm \ref{alg:hr_sampler}), by building a discrete time Markov chain on~$E$ whose distribution converges to the uniform one.
\item SEISM finally computes the $p$-values for the null hypotheses defined in~(\ref{enum:def}), associated with the selected motifs in~\ref{enum:sel}, using the empirical distribution of the test statistics, and returns the motifs with their association $p$-values. Given these $p$-values, one can adjust the number of selected motifs discarding the ones with non-significant $p$-values. This multiple-testing issue has not been investigated in this paper, but the practitioner can use for instance a Bonferroni bound to select the number of motifs.
\end{enumerate}
{
\normalsize%\small%\tiny
\begin{algorithm}[!h]
\captionsetup{labelfont={sc,bf}%, labelsep=newline
}
\caption{SEISM algorithm (general formulation)}
\label{alg:seism_general}
\SetAlgoLined
\BlankLine
%%% Description
{\nonl{{
{\bf \# Description:}
\begin{sloppypar*}
{\tt SEISM} {\it selects a set of sequence motifs~$(\vz_1,\ldots,\vz_q)$ based on an association score~$s(\cdot,\cdot)$, and evaluate their~$p$-values based on a partition~$\gZ=\bigsqcup M_i$.} %A sequence motif~$\vz$ is embedded into~$\R^n$ thanks to a feature function~$\vz\mapsto\mK^{\vz,\mX}$ that can be typically described by a Neural Network.} \av{Vigilance sur l'embedding de $z$ dans $\R^n$}
\end{sloppypar*}
}}}
\BlankLine
\Inputs{Response~$\vy\in\gY\subseteq\mathds R^n$, sequence samples~$\mX$, feature function~$\vz\in\mathcal Z\mapsto \bm{\varphi}^{\vz,\mX}\in\R^n$, association score~$s:\gZ\times\gY\to\R$, number of selected motifs~$q\geq1$, meshes~$\gZ=\bigsqcup\limits_{i=1}M_i$, sampling algorithm~$\mathcal{HR}$.}
\BlankLine
\KwResult{$((p_1,\vz_1),\ldots,(p_q,\vz_q))$, sequence of~$p$-values and sequence motifs.}
\BlankLine
{\nonl{{
{\bf \# Selection step:} {\it Selection of the sequence motifs $(\vz_1,\ldots,\vz_q)$ and the sequence of meshes $(M_{i_1},\dots,M_{i_q})$.}}}}
\BlankLine
\For{$j = 1,\dots,q$}
{
%Compute~$\mP_i\vy$ \tcp*{will be used to compute the association score}
$\displaystyle \vz_j \gets \argmax_{\vz \in \gZ} s(\vz, \mP_j\vy)$\;
\tcp*{
$\displaystyle \mP_k$ orthogonal projection onto $\displaystyle
\Span_{\ell<j}\big\{\bm{\varphi}^{\vz_\ell, \mX}\big\}^\perp$
}
$i_j\gets i\text{ s.t.}\,\vz_j\in M_i$\;
\tcp*{
the mesh $M_{i_j}$ is selected
}
}
\BlankLine
{\nonl{{
{\bf \# Inference step:}
\begin{sloppypar*}
{\tt SEISM} {\it provides a $p$-value $p_k$ on the~statistical influence of the selected sequence motifs~$\vz_k$ conditional on the selection event \eqref{e:selection_event} of observations~$\vy'$ that would have led to same selection of the sequence of meshes $(M_{i_1},\ldots,M_{i_q})$.}
\end{sloppypar*}
}}
}
\BlankLine
{
$\vy^{'(1)},...,\vy^{'(N)} \gets \mathcal{HR}(y, \left(M_{i_1},\dots,M_{i_q}\right))$\;
\tcp*{
Sampling outcomes under the selected null
}
}
\BlankLine
\For {$j = 1,\dots,q$}
{
$\tilde F_j(\,\cdot\,;\vy^{'(1)},...,\vy^{'(N)}) \gets$ empirical cumulative distribution function of $s(\vr M_{i_j}, \bm{\Pi_j} \vy')$ under the selected null
\tcp*{
$\vr M_{i_j}$ is a motif representing $M_{i_j}$ and $\bm{\Pi_j}$ the orthogonal projection onto $\displaystyle \Span_{\ell\neq j}\big\{\bm{\varphi}^{\vz_\ell, \mX}\big\}^\perp$
}
{
$p_j \gets \tilde F_j(s(\vr M_{i_j})\,; \vy^{'(1)},...,\vy^{'(N)}) $\;
\tcp*{output the $j^{\mathrm{th}}$ $p$-value}
}
}
\end{algorithm}
}
\section{One hidden layer CNNs select sequence motifs maximizing an association score}\label{subsec:selection}
One-layer CNNs have been at the core of the rising popularity of deep
learning over the past decade, by enabling major improvements in
computer vision tasks~\citep{krizhevsky2012ImageNet}. Although they
are formally a specialized fully connected feedforward networks with
additional constraints on the weights, CNNs are equivalent to, and more
often thought of as, a set of \emph{convolutions} of the vectorial
input with some smaller vectors referred to as filters. When applying
the network, dot products are taken between each of them and
successive windows of the vectorial input followed by some non-linear
operation, producing an activation profile for each filter. In
one-layer networks, these activations are pooled across the windows
into a single scalar for each filter and these scalars are
combined---typically through a linear or regular fully connected
network---to provide a prediction for the input. Because convolution
filters are homogeneous to the input, they easily lend themselves to
interpretation: as small image patches for image inputs, and as
sequence motifs for appropriately encoded biological sequence
inputs. Accordingly, activation profiles reflect how much each piece
of the input is similar to the filter---in the sense of the dot
product---and applying a one-layer CNN amounts to applying a
predictive function to a modified representation of the original data
by these similarity profiles. Because convolution filters are jointly
optimized with the parameterization of the predictive function, CNNs
are often described as a strategy to jointly learn a data
representation and a function acting on this representation, both
being optimized for a prediction objective. In computer vision, the
optimized filters of the first layer typically learn to detect edges
with different orientations. In biological sequences, they learn short
sequences whose presence anywhere in the input is predictive of the
output phenotype used for training.
% Use a simple figure to represent one-layer cnns for sequences?
\begin{figure}[h!]
\centering
\includegraphics[scale=0.3]{figures/motif.eps}%
\caption{A motif represented by its position weight matrix and corresponding sequence logo. The total height of the letters indicates the information content of the position (in bits), closely related to the Shannon entropy.}
\label{fig:motif}
\end{figure}
Unlike input sequences that are formed by a discrete succession of
letters in some alphabet, trained filters are continuous and therefore
account for possible variation in the predictive short sequence,
\emph{e.g.}, a T mostly followed by a C but sometimes an A or a G and so on
(Figure~\ref{fig:motif}). These probabilistic objects have also
been used for a long time in the bioinformatics literature and referred
to as position weight matrices (PWMs). Inferring PWMs either according
to their frequency in a set of sequences~\citep{bailey_meme_2006} or their
discriminating power between two sets~\citep{bailey_streme_2021} has been a major
theme over the past thirty years. Here we formalize the
training a one-layer CNN as equivalent to the selection of a set of sequence
motifs that are optimal for some association score. This formalization
will be instrumental in the definition of our hypothesis testing
procedure in Section~\ref{sec:model}.\\
%{$\bullet$ \bf Settings and notations}
\noindent
\paragraph{Notations}
Let $\displaystyle \mX$ represent a data set of $n$ one-hot encoded
sequence samples $\displaystyle \{\vx_1, \vx_2,\dots, \vx_n\}$, in a
set $\displaystyle \gX$ of biological sequences assumed to be over an
alphabet $\displaystyle \gA$---for DNA sequences,
$\gA = \{A, C, T, G\}$. One-hot encoding maps each letter in $\gA$ to
a vector in $\{0,1\}^{|\gA|}$, with all-zero entries except for a
single $1$ at the coordinate corresponding to the order of the letter
in $\gA$---for DNA sequences, $A$ is encoded as $(1, 0, 0, 0)$. Every
$\vx_i$ is therefore encoded as a matrix in
$\{0,1\}^{|\gA|\times|\vx_i|}$---although in practice, encoded
sequences are often padded with dummy columns to have the same
lengths. We denote $\displaystyle y_i\in \gY$ the measurement of a
biological property associated with sequence $\displaystyle \vx_i$,
and $\displaystyle \vy\in\gY^n$ the corresponding vector of
outcomes. We consider one-layer CNNs with a Gaussian non-linearity with scale $\omega$, a max global pooling and a linear
prediction function. These CNNs parameterize a function
$\displaystyle f:\gX \rightarrow \gY$ by $q$ filters of length $k$, namely
$\displaystyle \mZ := \{\vz_1, \dots, \vz_ q\} \in \gZ^q$, where $\gZ$
is a subset of $\sR^{|\gA|\times k}$, given by the simplex in this paper:
\begin{equation}
\label{eq:simplex}
\gZ = \bigg\{ \vz \in \R_+^{\lvert \gA \lvert \times k}\ :\ \forall j \in[k]\,,\ \sum_{i=1}^{\lvert \gA \lvert}\evz_{i,j}=1\bigg\}\,,
\end{equation}
and $q$ weights
$\beta\in\sR^q$.
%{$\bullet$ \bf Activation function}
More precisely, we define
$f(\vx_i) := \left(\bm{\varphi}^{\mZ, \mX} \bm{\beta}\right)_i$, with
$\bm{\varphi}^{\mZ, \mX} \in \sR^{n \times q}$ defined as
$\bm{\varphi}^{\mZ, \mX} = \mC_n\bm{\tilde{\varphi}}^{\mZ, \mX} $, where
$\mC_n = I_n - n^{-1}\mathbf{1}_n\mathbf{1}_n^\top$ is the centering
operator, $I_n$ the identity matrix, $\mathbf{1}_n$ the all-one
vector in $\sR^n$, and
\begin{equation}\label{eq:Kdef}
\displaystyle {\tilde{\varphi}}^{\mZ, \mX} _{i,j} := \max_{\vu \in [\vx_i]_\ell} \left\{ \exp\left(-\frac{||\vz_j-\vu||^2}{2 \omega^2} \right) \right\}\,,
\end{equation}
where $[\vx_i]_\ell$ denotes the set of $\ell$ consecutive entries of the
vector $\vx_i$ (and of its reverse-complement counterpart), and $\omega$ is a bandwidth hyperparameter whose impact and tuning is studied in Appendix \ref{app:omega}. This model differs with a typical CNN in two ways. First, it uses a Gaussian activation function instead of an exponential one; second the use of the centering operator that sets the average of the activation to zero. These adjustments were made to improve the SEISM algorithm's selection performances.
%Note that a typical CNN would have used an exponential activation function instead of this Gaussian one and would not have included the centering step. These adjustments were made to improve the SEISM algorithm's selection performances.
\subsection{From empirical risk minimization to association scores}
The function $f$ is learned in a classical penalized empirical risk
minimization framework, using the data~$\{ \mX, \vy\}$:
\begin{equation}\label{eq:erm}
\displaystyle \min_{(\mZ, \bm{\beta}) \in (\gZ \times \sR^q) }n^{-1}\big{\|}\vy-\bm{\varphi}^{\mZ, \mX} \bm{\beta}\big{\|}^2 + \lambda || \bm{\beta} || ^2,
\end{equation}
for some $\lambda>0$. Equation~\eqref{eq:erm} formalizes the idea that learning a one-layer CNN on one-hot encoded sequences amounts to learning a data-representation $\bm{\varphi}^{\mZ, \mX}$ of the sequences parameterized by a set $\mZ$ of filters---corresponding to PWMs---and a linear function with weights $\bm\beta$ acting on this representation.
%We denote $\displaystyle y_i$ the measurement of a biological property $\displaystyle \gY$ associated with sequence $\displaystyle \vx_i$, and $\displaystyle \vy$ the corresponding vector of outcomes. The aim of a convolutional neural network (CNN) is to learn a function $\displaystyle f:\gX \rightarrow \gY$ using the provided data $\displaystyle \{ \mX, \vy\}$ able to predict accurately the outcome of an unseen sequence. To this end, it typically minimizes a penalized empirical mean squared error:
%\begin{equation}
%\displaystyle \min_{f \in \gF} \frac1n \sum_{i=1}^n (y_i-f( \vx_i))^2 + \Omega_\lambda(f)\,,
%\label{eq:1}
%\end{equation}
%over some class of regression functions $\gF$, for some regularization $\Omega_\lambda$ that may depend on a tuning parameter $\lambda>0$. For a one hidden layer CNN using $q$ filters, a max-pooling step and an exponential activation (with bandwidth~$\sigma$), we can rewrite a ridge version of the objective function as:
% \begin{equation}\label{eq:2}
%\displaystyle \min_{(\mZ, \bm{\beta}) \in (\gZ \times \sR^q) } \big{\|}\vy-\mK^{\mZ, \mX} \bm{\beta}\big{\|}_n^2 + \lambda || \bm{\beta} || ^2\,,
%\end{equation}
%where $\displaystyle \mZ = \{\vz_1, \dots, \vz_ q\}$ represents the filters of the CNN, in the set of possible filters $\gZ$, and $\|\cdot\|_n^2=(1/n)\|\cdot\|_2^2$. More precisely, the one-hot encoding allows us to process the filters of the CNN as well as any $k$-mer $\vu$ of a sequence $\vx$ as matrices. We then obtain $\mK^{\mZ, \mX} \in \sR^{n \times q}$ such that $\mK^{\mZ, \mX} = \mC_n\bm{\tilde{K}}^{\mZ, \mX} $, where $\mC_n$ is the centering operator and:
%\begin{equation}\label{eq:3}
%\displaystyle \bm{\tilde{K}}^{\mZ, \mX} _{i,j} := \max_{\vu \in [\vx_i]_k} \left\{ \exp\left(-\frac{||\vz_j-\vu||^2}{2 \sigma^2} \right) \right\}\
%\end{equation}
%where $[\vx_i]_k$ denotes the set of $k$ consecutive entries of the vector $\vx_i$ (convolutional step). %We emphasize that the class of regression function $\gF$ is given by
%\[
%f(\vx)=
%\sum_{j=1}^q \bm{\beta}_j h(\vx;\vz_j )\,,
%\]
%where $h(\vx;\vz_j)=\max_{\vu \in [\vx]_k} \{ \exp(-{||\vz_j-\vu||^2}/{2 \sigma^2})\}$.
Noting that there exists a unique explicit optimal $\bm{\beta}$ for \Eqref{eq:erm}, it follows immediately that:
\begin{equation}\label{eq:equiv}
\displaystyle \argmin_{\mZ}
\Big\{
\min_{\bm{\beta}}
\big\{
n^{-1}||\vy-\bm{\varphi}^{\mZ, \mX} \bm{\beta} ||^2 + \lambda || \bm{\beta} || ^2
\big\}
\Big\}
= \argmax_{\mZ}
\Big\{
s^{\text{ridge}}_\lambda(\mZ, \vy)
\Big\}\,,
\end{equation}
where $s^{\text{ridge}}$ defines a particular quadratic association score between an outcome $\vy$ and a set of filters $\mZ$:
\begin{equation}\label{eq:ridgeScore}
s^{\text{ridge}}_\lambda(\mZ, \vy) := \vy^T \bm{\varphi}^{\mZ, \mX}\big[(\bm{\varphi}^{\mZ, \mX})^T\bm{\varphi}^{\mZ, \mX} + \lambda n \mI_q \big]^{-1}(\bm{\varphi}^{\mZ, \mX})^T\vy.
%\displaystyle s_{\text{ridge}}(\mZ, \vy) := \frac{\vy^T}{|| \vy ||} \mK^{\mZ, \mX}\big[(\mK^{\mZ, \mX})^T\mK^{\mZ, \mX} + \lambda n \mI_q \big]^{-1}(\mK^{\mZ, \mX})^T \frac{\vy}{|| \vy ||},
\end{equation}
It formalizes the training of a CNN as the selection of a set of filters whose association with~$\vy$ in the sense of~$s^{\text{ridge}}_\lambda$ is maximal. Of note, one has
\[
\lim_{\lambda\to\infty}\lambda n \times s^{\text{ridge}}_\lambda(\mZ, \vy) = \vy^T \bm{\varphi}^{\mZ, \mX}(\bm{\varphi}^{\mZ, \mX})^T\vy =: s^{\text{HSIC}}(\mZ, \vy)\,,
\]
so for large values of the regularization hyperparameter, selecting filters by learning a CNN is equivalent to selecting filters with the classical HSIC score~\citep{song2012feature}, because~$\bm{\varphi}$ already includes a centering operator. In addition to connecting~$s^{\text{ridge}}$ with~$s^{\text{HSIC}}$, we observed that the centering in the definition of~$\bm{\varphi}^{\mZ,\mX}$ led to the selection of better sequence motifs in our experiments.
% In \Eqref{eq:ridgeScore}, one may notice that the normalization step is non-standard and new as far as we know.
%Writing the objective function of the CNN as an association score depending only on the observed direction~$\vy/\|\vy\|$ and~$\mZ$ will enable us to create a framework for performing inference over the optimized filters.
%Training a one hidden layer CNN is thus equivalent to the selection of~$q$ filters, that may be interpreted as sequence motifs.
%Our choice to use this centring matrix may seem arbitrary. This choice is motivated by the empirical performance we have observed in our experiments.
%It is in no way limiting, our theoretical analysis works just as well if we do not use this centering\lj{which theoretical analysis?}.
Observe that the centering matrix is an orthogonal projection matrix onto~$\mathcal E:=\mathrm{Range}(\mC_n)$, the orthogonal of the vector line generated by the vector~$\mathbf 1$, and it holds
\begin{equation}
\label{eq:projection_y}
\big{\|}\vy-\bm{\varphi}^{\mZ, \mX} \bm{\beta}\big{\|}_n^2
=
\big{\|}\mC_n\vy-\bm{\varphi}^{\mZ, \mX} \bm{\beta}\big{\|}_n^2
+
\big{\|}\vy-\mC_n \vy{\|}_n^2\,.
\end{equation}
The solution of \eqref{eq:erm} is unchanged if $\vy$ is replaced by $\bm C_n \vy$, and so we can assume that~$\vy\in \mathcal E$ without any generality loss. Furthermore, this shows that we can work with skewed data in a classification context, since imbalanced classes will have no effect on the result.
%We deduce that the solutions to \Eqref{eq:erm} do not depend on~$\vy-\mC_n \vy$ and we can assume that~$\vy\in \mathcal E$ without any restriction. It allows us to work with skewed data in a classification context, by artificially boosting the observations from the minority class.
%This centering step also plays an important role in the inference step, as described in Section \ref{sampling_null} \av{ici c'est pas forcément la bonne réf, et il faut ajouter cette discussion dans la bonne réf}.
%\av{this paragraph has been completely reworked, the previous version is commented in the tex file}
%The effect of centring is that the regression \Eqref{eq:erm} takes place in the space~$\mathcal E$ of dimension~$n-1$.\lj{Je crois que je ne comprends pas ce qu'on veut prouver avec ce dernier point.} \av{TO DO: à décaler en partie inférence}. In addition to this, this centering step allows us to work with balanced classes in a classification context, and is standard in feature selection methods, such as the HSIC criteria. \lj{Il faudrait expliquer un peu plus précisément en quoi le centrage permet de travailler avec des classes déséquilibrées} \av{En fait le problème de ce dernier paragraphe, c'est qu'une fois qu'on a dit que $\bm{K}$ contenait un opérateur de centrage, le centrage de Y devient inutile/transparent. Donc on pourrait garder Y non centré que ça donnerait exactement le même résultat. Du coup, j'ai l'impression qu'on peut supprimer un peu cette discussion du Y centré, ou alors en parler plus tôt dans la définition de $\bm{K}$, puisque c'est là que le centrage apparaît. En fait, j'ai rajouté ce centrage forcé des Y pour la partie inférence avec le score normalisé. En effet, sans ce centrage, si le $Y_{lab}$ n'est pas centré, il va mener à des scores statistiquement plus faibles que les réplicats, qui eux seront centrés, puisque la composante portée par (1,...,1) n'augmente pas le score mais augmente la norme, ce qui mécaniquement dimiminue le score normalisé.}
%When it comes to the \textit{de-novo} motif discovery, one can implement other associations scores to achieve better performances, such as:
%\begin{equation}\label{eq:6}
%\displaystyle s_{\text{hsic}}(\mZ, \vy) = \frac{\vy^T}{|| \vy ||} \bm{\varphi}^{\mZ, \mX}(\bm{\varphi}^{\mZ, \mX})^T \frac{\vy}{||\vy ||}\,,
%\end{equation}
% In addition to the possibility of choosing other association scores, one can also improve the motif discovery performances by using a greedy approach instead of the joint optimization typically performed by CNNs. A joint approach indeed allows information sharing along several motifs, leading to good prediction performances but may suffer from less informative motifs.
%In particular, we likely select sparser solutions---smaller $q$---than what we would obtain by looking for an $f$ with the best possible prediction accuracy. \lj{This is reminiscent of what happens with Lasso (Leng et al. 2004 ``A Note on the Lasso and Related Procedures in Model Selection") but this may be too much of a digression}
\subsection{Greedy optimization}
It is common to solve~\eqref{eq:erm} by stochastic gradient descent
(SGD) jointly over the $q$ filters. More generally, this approach for
training a neural network with a single, large hidden layer is known
to find a global optimizer at the large $q$ limit under some
assumptions~\citep{soltanolkotabi2019theoretical}. Our objective here
is slightly different: we do not necessarily aim at approximating a
continuous measure with a large number of particules, but we aim at
selecting a small number of particules lending themselves to a
biological interpretation. Furthermore, the number of relevant motifs on a given dataset is generally unknown. In this context, it is known that
jointly optimizing the convolution filters leads to irrelevant PWMs,
with some actual motif split across several filters and other
duplicated~\citep{koo_representation_2019}. A possible strategy is to forego filter-level
interpretation, train an overparameterized network---with a much
larger $q$ than the expected number of motifs---and use attribution
methods to extract relevant motifs or other interpretable features
from the trained network~\citep{shrikumar2018technical}. Here we adopt a different strategy using a forward stepwise procedure, where we iteratively optimize
each of the convolution filters over the residual error left by the
previous ones.
%We want to point out that we work on a less constrained set during optimization than $\gZ$ \eqref{eq:simplex}. We indeed remove the positivity constraint, and project the motifs onto $\gZ$ at the end of the optimization.
%In our experience (Supplementary Material~\ref{app:greedy}), it is sufficient to avoid the splits and redundancies observed with joint optimization.
More precisely at each of the $q$ steps, we select
$\displaystyle \vz_j$ such that:
\begin{equation}\label{eq:greedy}
\displaystyle \vz_j = \argmax_{\vz \in \gZ} s^{\text{ridge}}(\vz, \mP_j\vy)\,,
\end{equation}
where $\displaystyle \mP_j$ is the projection operator onto the
orthogonal of the subspace
$\displaystyle \Span_{\ell <j}\big\{\bm{1},\bm{\varphi}^{\vz_\ell, \mX}\big\}$,
see line 2 of
Algorithm~\ref{alg:seism_general}. This is how $\vz_j$ is optimized over the residuals of the previous filters. The vector $\bm{1}$ enforces that we project on a subspace of $\mathcal E$, in particular
$\mP_1=\mC_n$. Without this projection, iterating~\eqref{eq:greedy} would return the same $\vz$. Of note, joint optimization procedures of the $q$ filters don't face this issue, and forward selection procedures over finite sets of features work around the problem by iteratively removing the selected elements from the set over which selection if performed~\citep{slim2019kernelpsi}. This sequential strategy combined with the testing procedure introduces in Section~\ref{sec:model} provides a data-driven mean to choose the number $q$ of relevant motifs.
In practice, we solve~\eqref{eq:greedy} with a standard gradient descent algorithm, initialized at the $k$-mer with the best
association score. The $k$-mer list is obtained using the DSK software \citep{rizk_dsk_2013}. We work on a less constrained set than $\gZ$ \eqref{eq:simplex} and don't enforce the positivity constraint during optimization. We project the optimized motifs onto the full $\gZ$ at the end of the process. Our procedure also requires to choose a motif length $k$. We proceed adaptively by choosing the length leading to the highest score, within a user-specified range.
With the one-layer CNNs training formally cast as the successive
selection of $q$ sequence motifs optimizing an association score, we
now turn to the problem of testing the significance of these
associations. Of note, what follows is only based on the definition of
an association score and could be applied to perform inference on
other features coming from the training step of any algorithm, as long
as one can define an association score between the feature and the
outcome.
%At this step, we would like to draw the reader's attention to the fact that what follows is only based on the definition of an association score. The procedure described below to perform inference on motifs selected by SEISM can also be applied to perform inference on other features coming from the training step of any algorithm, as long as one can define an association score between the feature and the outcome.
%\subsection{The association score paves the way for inference over the trained filters of the CNN} \label{model}
\section{Post-selection testing of the association between the outcome and trained convolution filters} \label{sec:model}
We now turn to the problem of testing the association between the
selected motifs $\vz$ and the trait $\vy$. In order to do so, we need
to solve three interrelated problems. First, the motifs were
specifically selected for their association with the trait, which
leads to the well known post-selection inference problem. Any
inference procedure that disregards that the hypothesis was
constructed using the same data used for testing is likely
invalid and produces deflated $p$-values. Second, we deal with a
continuous selection event, because~\eqref{eq:greedy} is performed
over a continuous set $\gZ$. By contrast, existing solutions for
post-selection inference address selections over finite sets. Third,
the null hypothesis commonly used for similar post-selection inference
problems is composite, \emph{i.e.}, it corresponds to several values
of the parameters. Existing methods work around this issue by fixing
theses parameters to arbitrary values, thereby limiting the scope
under which they are calibrated. Here we present our solutions to
these three problems.
%\lj{mention kernelpsi and previous literature (PSI framework with quadratic scores, how do we relate to that)}\av{est-ce qu'il ne faudrait pas plutôt le mentionner plus tôt, au moment où on introduit les scores d'association ? De plus, on ne se sert pas vraiment du fait que ces scores soient quadratiques (pas plus que kernelPSI en fait), donc je ne sais pas trop comment l'insérer.}
%{$\bullet$ \bf Gaussian distribution of the vector of outcomes}
%We follow~\cite{yamada_post_2018} and
Consider the Gaussian model:
\begin{equation}\label{eq:model}
\vy = \bm{\mu} + \sigma \bm{\epsilon}
\end{equation}
where $\displaystyle \bm{\mu} \in \mathcal E$ is the target deterministic signal, and
$\bm{\epsilon} \sim \mathcal{N}(\bm{0}, \mC_n) $ the standard
Gaussian distribution on~$\mathcal E$.
%\av{à enlever de cet endroit: We denote by $\mathds P_{\bm\mu}$ the resulting law, the dependence on $\sigma$ being implicit.}
% LJ: si on a besoin de dire ça je pense qu'il faut le mettre ailleurs
%, and $\vy\in\mathcal E$ almost surely in this model --- in practice, SEISM projects the input label onto $\mathcal E$.
%\av{ANCIENNE VERSION, QUE J'AI ENVIE D'ENLEVER: Recall that the selection of $\mZ$ does not depend on~$\vy-\mC_n \vy$ as depicted in \Eqref{eq:projection_y} and we can consider that this projection is zero in our analysis. In practice, it means that the practitioner needs to project $\vy$ onto~$\mathcal E$ at the first step of our methodology. Also, we could have taken an orthonormal frame of~$\mathcal E$, this would have led to the covariance of $\bm\epsilon$ being the identity. We prefer to keep explicit that $\gE:=\mathrm{Range}(\mC_n)$. For sake of readability, we consider that $\mC_n$ is the centering matrix (that can be also viewed as the orthogonnal projection onto the orthogonal of $\mathrm{Span}\{\bm 1\}$), as mentioned above; but our analysis can be directly extended to other orthogonal projection matrices.}
%\ydc{Double check above please}
%The purpose of this section is to define a hypothesis testing framework. Our test statistics are given by the association score and we would like to detect abnormally large values of these scores with respect to a family of null distributions described by a null hypothesis. We need to discuss here what we mean by null hypothesis in our framework.
%The null hypotheses and framework we will consider below are inspired from \cite{yamada_post_2018}. We want to underline that those hypotheses are composite, \textit{i.e.} several distributions $\mathds P_{\bm\mu}$ are considered under the null because several distinct $\bm{\mu}$ can have a null association score with the same $\vz$. Sections \ref{sampling_null} and \ref{sampling_null_unknown_sigma} will go into greater details on this.
%{$\bullet$ \bf Non-selective null hypothesis}
\subsection{Selective null hypothesis}
We follow~\cite{yamada_post_2018} and test the association of a
motif $\vz$ through the following null hypothesis:
\begin{equation}\label{eq:null}
\displaystyle \mathds H_{0}:‘‘s(\vz, \bm{\mu})=0",
\end{equation}
for some association score $s$. For a $\vz$ chosen independently of
the data, $\mathds H_0$ could be tested by sampling $\vy'$ under
the corresponding distribution, and using the quantile of the
$s(\vz, \vy')$ scores corresponding to $s(\vz, \vy)$ as a
$p$-value---\emph{i.e.}, the probability when sampling under $\mathds
H_0$ to observe a score as extreme as $s(\vz, \vy)$.
%{$\bullet$ \bf Selective null hypotheses}
In our case, however, the motifs $\vz$ in the trained convolution filters were specifically selected for their strong association with $\vy$, and this procedure would not produce calibrated $p$-values. This problem is known as post-selection inference, and has been discussed and addressed in a growing literature over the past few years. %\lj{Je pense que ces phrases repartent de trop loin, si ça vous va on peut les enlever: "Indeed, whether it is to lower an excessively high computational cost, to increase the generalization performances of a model, or to improve its interpretability, feature selection plays an important role in data analysis, especially in a high-dimensional data context. It can then be interesting to test the statistical significance of those features with respect to the problem at hand. But the classical inference scheme assumes that the hypotheses are chosen independently from the data on which they will be tested: if ignored, the effect of selection can substantially exaggerate the apparent strengths of the relationships."}
Although data-split strategies --- which split the data into two parts, and then perform the selection and the inference on the two different parts --- lead to valid procedures \citep{wasserman_high-dimensional_2009}, they necessarily result in a reduction of the sample size, unsatisfying when the original sample size is limited. Alternatively, selective inference frameworks were developed in the recent years to address these issues. We refer to \cite[Chapter 6]{hastie2015statistical} and references therein for a general presentation. \cite{taylor_exact_2014} and \cite{lee_exact_2016} address scenarios where the selection event, \textit{i.e.} the set of data outputs that would result in the selection of the same set of features, is polyhedral---determined by the finite intersection of linear constraints. \cite{reid_sparse_2015}, and later \cite{reid_general_2015} extend this selection to clusters or groups of features, still in the linear framework. \cite{yamada_post_2018} extended post-selection inference to the non-linear framework, by proposing a kernel-based approach, where the selection is performed through the HSIC criterion. \cite{slim2019kernelpsi} generalize this work, by allowing the selection to be carried out with a wider range of tools, making use of quadratic association scores.
%\lj{Je suggère de retirer: "Such scores meet the necessary conditions we identified to apply our post-selection inference scheme, but a wider class of score might also be compatible with our scheme". On ne voit pas encore de quoi on parle, et ça peut donner l'impression que kernelpsi serait suffisant pour ce qu'on veut faire}.
To our knowledge, post-selection inference literature only addresses the problem of selecting features from a discrete collection and does not provide a solution for selections from a continuous set like our $\gZ$. Hence, testing \eqref{eq:null} directly is not feasible and we resort to the quantization of the motif space to address this problem.
%But in our framework, $\gZ$ is continuous. We then have to work around this issue, in order to apply a similar sampling algorithm as the one described in \cite{slim2019kernelpsi}.
In addition to that, we push the analysis of the statistical model further, in order to be able to apply it with weaker assumptions on the data distribution.
%\av{A enlever ? On l'a déjà dit au dessus, et pour moi ça interrompt le flot de la lecture sur le passage continu vers discret}\lj{ok pour supprimer}
%Alternatively, \emph{selective inference} provides calibrated test of null hypotheses that were selected using the same data by conditioning the null distribution on the selection event~\citep{taylor2015statistical,reid2018general, slim2019kernelpsi}. In our setting, a valid $p$-value would therefore be obtained by assessing the probability of obtaining a value as extreme as $s(\vy,\vz)$ for a $\vy$ under $\mathds H_0$, \emph{given that this $\vy$ already led to the selection of this~$\vz$}. We now describe why our continuous selection event is not directly amenable to existing selective inference strategies and how we work around this issue to define selective null hypotheses for which we can build sampling-based testing procedures.
%We will describe below this framework, referred to as ‘‘selective null hypotheses'', for our procedure.
% LJ: a utiliser dans la partie "composite"
%In this case, the null hypothesis $\mathds H_{0}$ itself may depend on the selection event: we would like to consider null distributions $\mathds P_{\bm\mu}$ such that $\bm\mu$ satisfies (multiple) null hypotheses of the form \eqref{eq:null} where $\vz$ is given by the selected motifs or, at least, is determined by the selected motifs, as described in line 3 of Algorithm~\ref{alg:seism_general}.
%An other feature is that the selection procedure might be a forward step-wise procedure, as in our context. In this case, the selective null is a selective multiple null hypothesis scheme. We select the active support in a greedy fashion and ask ourselves if the $i$-th motifs fits some specific set of distributions. We select the active support in a greedy fashion and ask ourselves if the $i$-th motifs fits some specific set of distributions. One can formalize it by considering a sequence\lj{"a set" non? sequence suggère qu'il y a un ordre dans les hypothèses, ce qui n'est pas le cas.} of null hypotheses, denoted by $\mathds H_{0,i}$, of the form \eqref{eq:null} where $\vz$ is given by the $i^{\mathrm{th}}$-selected motif or, at least, is somehow related to it. A more precise mathematical description follows.
% LJ: a remettre plus bas
%Finally as we select several motifs, we will be testing one selective null for each of them. We formalize this procedure by considering a set of null hypotheses denoted by $\mathds H_{0,i}$, of the form \eqref{eq:null} where $\vz$ is given by the $i^{\mathrm{th}}$-selected motif or, at least depends on it, as described in \ref{sec:mesh}. A more precise mathematical description follows.
%rely on forward step-wise selection, our selective null is multiple: we select a set of motifs in a greedy fashion before testing whether each of the them fits some specific set of distributions.
% Continuity/null measure issue
% Our solution: meshes
%Recall that we have centered our observation $y$ and kernel $\bm{\varphi}^{\mZ, \mX}$, namely $y\in \mathcal E$ and $\mathrm{Span}(\bm{\varphi}^{\mZ, \mX})\subseteq \mathcal E$ where $\mathcal E=\mathrm{Span}\{\mathbf 1\}^\perp$. Let us consider the general Gaussian model:
%\begin{equation}\label{eq:8}
%\displaystyle \vy = \bm{\mu} + \sigma^2 \bm{\epsilon}
%\end{equation}
%where $\displaystyle \bm{\mu} \in \mathcal E$ stands for the true deterministic signal, and $\bm{\epsilon} \sim \mathcal{N}(\bm{0}, \mC_n) $, the standard Gaussian distribution on $\mathcal E$. For a given set of sequence motifs $\displaystyle \mZ = \{\vz_1, \dots, \vz_ q\}$, we will test the following null hypotheses: \textcolor{red}{(discuss here sequential/joint and cite paper + add clustering if similar predictors are selected)}
%\begin{equation}\label{eq:9}
%\displaystyle \mathds H_{0,i}:s(\vz_i, \bm{\Pi_i}\bm{\mu})=0
%\end{equation}
%where $\bm{\Pi_i}$ is the orthogonal projection operator onto the orthogonal of the subspace $\displaystyle \text{Span}_{j\neq i}\big\{\mathbf 1,\bm{\varphi}^{\vz_j, \mX}\big\}$.
%\subsection{Data splitting strategies to test sequence motifs} %\label{datasplit}
%Data splitting strategies consist in dividing the dataset in two subsets. %The \textit{training set} will be used to select sequence motifs as described in \ref{model}, and those motifs will be tested using the \textit{test set}. \\
%The distribution of the test statistics $V$ has no closed-form. To approach it, we propose to sample phenotypes under the null hypothesis according to \ref{eq:8}. This allows us to derive valid empirical $p$-values, which are then equal to the fraction of samples for which the association score is greater than the score calculated on the test set.
%\subsection{Using post-selection inference leads to great improvements compared to data splitting strategies in case of small datasets}
%In this section we will make use of the concept of selection event.
\subsection{Dealing with selection events over a continuous set of features}\label{sec:mesh}
Formally, our selection event $E_{\mathrm{cont.}}$ is the set of outcomes $\vy'$ that would have led to the selection of the same set of motifs $\displaystyle \mZ = \{\vz_1, \dots, \vz_ q\}$ than the one selected using $\vy$ from the real dataset, when applying the same selection procedure:
\begin{equation}\label{eq:10}
\displaystyle E_{\mathrm{cont.}} := \big\{ \vy' \in \mathcal E\ :\ \forall j \in \{1,\dots,q\} ~\argmax_{\vz \in \gZ}s(\vz, \mP_j \vy')=\vz_j \big\}\,,
\end{equation}
where $\displaystyle \mP_j$ is the orthogonal projection onto $\displaystyle\Span_{\ell<j}\big\{\bm{1},\bm{\varphi}^{\vz_\ell, \mX}\big\}^\perp$.
A simple rejection approach to sample from the null~\eqref{eq:null} conditioned to $E_{\mathrm{cont.}}$ would be to sample~$\vy$ in $\gE$ under (\ref{eq:model}, \ref{eq:null}) and only retain those in $E_{\mathrm{cont.}}$. Unfortunately, $E_{\mathrm{cont.}}$ belongs to a strictly lower-dimensional vector space of $\sR^n$ and is therefore a null set for the Lebesgue measure on $\sR^n$. For $s^{\text{HSIC}}$ and $s^{\text{ridge}}$, and noting that a maximum is also a critical point, we indeed obtain:
\begin{equation*}
\vy' \in \displaystyle E_{\mathrm{cont.}} \implies \forall j \in \left\{1,\dots,q\right\}~ \mP_j \vy' \in \Span\left\{ \nabla_{\vz} \bm{\varphi ^{z_j,X}} \right\}^\perp.
\end{equation*}
For $q=1$ and assuming that the different directions of the gradient are independent, this spans is a vector subspace with dimension $n-4\times k$.
%For $q=1$, we can then easily identify a vectorial subspace with dimension $n-4\times k$ (assuming that the different directions of the gradient are independent).
We empirically observed that sampling from this subspace produced a non-zero proportion of $\vy'$ in $E_{\mathrm{cont.}}$. Nonetheless, choosing a sampling distribution that leads to the correct conditional distribution is not straightforward---and may not even be possible---as discussed in Supplementary Material~\ref{app:null_set}. {Moreover, relying on conditional probability with respect to a null set is not well defined and may lead to the Borel-Kolmogorov paradox~\citep{bungert_lion_2022}, which further complicates its use. }
%Indeed, for the scores $s_{\text{HSIC}}$ and $s_{\text{ridge}}$, this vector space is included in the span of $\big(\left.\nabla_\mZ \bm{\varphi}^{\mZ, \mX}\right\vert_{\mZ=\mZ^*(\vy)}\big)^\perp$, where $\mZ^*(\vy)$ is the set of motifs maximizing a score $s$ for the observed $\vy$. Note that this set of equations are just the first order optimality of some critical point of the score: any $\vy'$ orthogonal to this gradient also verifies $\left.\nabla_\mZ s(\mZ, \vy')\right\vert_{\mZ=\mZ^*(\vy)} = 0$ and therefore admits $\mZ^*(\vy)$ as critical point. Empirically, we observed that sampling from this span indeed produced a non-zero proportion of $\vy'$ in $E_{\mathrm{cont.}}$. Nonetheless, choosing a sampling distribution on this span that leads to the correct conditional distribution is not straightforward---and may not even be possible---as discussed in Supplementary Material~\ref{app:null_set}.
%, but not all of them
%\ydc{Disintegration DONE, please check supplement}
%While using a data splitting strategy makes the test of pointwise motifs possible, it is not the case anymore when using a selection event.
%\subsection*{The meshes approach}
We choose to circumvent this issue using a partition of the space $\gZ$ of motifs spaces, over which our selection~\eqref{eq:greedy} operates, into a very large but finite set of meshes: $\gZ=\bigsqcup M_i$. As depicted in Figure \ref{fig:mesh}, we consider a regular partition of each coordinates into $m$ bins:
%dep\lj{On ne définit pas vraiment la partition ni m.}\ydc{On peut tenter la pirouette : we consider a regular partition, depicted in Fig 3(mettre la bonne ref), which is defined by a regular partition of each coordinate into $m$ bins}. It results in a partition of $k\times 4^m$ \ydc{d'ailleurs ce ne devrait pas être $k\times m^4$ ici?}meshes when $\mathcal A=\{A,C,G,T\}$ and $\mathcal Z$ is the simplex given by \Eqref{eq:simplex}.
\begin{figure}[!h]
\begin{center}
\includegraphics[width=0.3\linewidth]{figures/maille_3.eps}
\end{center}
\caption{Discretization of the 3-letters alphabet simplex $\left\{A, C, G \right\}$, with a binning parameter for the meshes $m=4$. }
\label{fig:mesh}
\end{figure}
Based on this partition into meshes, we define a quantized selection event~$E$ as follows. First, given an outcome $\vy$ we define the sequence of the $q$ selected meshes $(M_{i_1},\ldots,M_{i_q})$ as
\[
\forall j \in \left\{1,\dots,q\right\}\,,\ \argmax_{\vz \in \gZ}s(\vz, \mP_j \vy)\in M_{i_j}\,,
\]
Second, the selection event is given by:
\begin{equation}
\label{e:selection_event_2}
\displaystyle E(i_1,\ldots,i_q)
:=
\Big\{
\vy' \in \gY\,:~\forall j \in \left\{1,\dots,q\right\}\,,\ \argmax_{\vz \in \gZ}s(\vz, \mP_j \vy')\in M_{i_j}
\Big\}\,,
\end{equation}
the set of outcomes~$\vy'$ that would have led to the selection of motifs within the same meshes as the selected ones~$(M_{i_1},\ldots,M_{i_q})$.
We now show how quantization~\eqref{e:selection_event_2} of the selection problem make enables the definition of a valid inference procedure. We start with the simplest case where we select a single motif ($q=1$).
%Now that we have solved this discretization problem, we will go over how to test a sequence motif chosen with SEISM, in the simple case where only one motif is selected, \textit{i.e} $q=1$.
\subsection{Test with only one motif $q=1$, $\mu$ and $\sigma$ fixed}\label{sec:q_equal_1}
In this section, considering the motif $\vz_1$ was chosen by the SEISM selection procedure, selection event~\eqref{e:selection_event_2} boils down to:
\begin{equation}
\label{eq:simple_E}
\displaystyle E(i_1) := \left\{ \vy'\in \gY\,:~\argmax_{\vz \in \gZ} s(\vz,\vy') \in M_{i_1} \right\}
\end{equation}
%The null hypothesis and test statistics related to this selection event must now be defined. To that end, we have two possibilities:
We use this simplified case to introduce our null hypotheses and test statistics attached to this selection event, and consider two options:
\begin{itemize}
\item A first option consists in representing the mesh $M_{i_1}$ by its center $\vc_1$. Then the corresponding null hypothesis is the following:
\begin{equation}\label{eq:centre_q1}
\mathds H'_{0,1}\,:\,‘‘s(\vc_1, \bm{\mu})=0",
\end{equation}
It can be tested using statistic $V'_1=s(\vc_1, \vy)$.
\item A second possibility is to represent $M_{i_1}$ by the motif with the highest association score within. In this case, the null hypothesis becomes:
\begin{equation}\label{eq:nullBest_q1}
\displaystyle \mathds H''_{0,1}\,:\, ‘‘\forall \vz \in M_{i_1}\,,\ s(\vz,\bm{\mu})=0",
\end{equation}
%The test statistic is here a bit trickier: $V''_1=\max_{\vz\in M_{i_1}}s(\vz,\vy')$
We test it using statistic $\displaystyle V''_1=\max_{\vz\in M_{i_1}}s(\vz,\vy)$.
\end{itemize}
In both cases, we reject the null hypothesis if the test statistics are greater than a threshold, determined by their cumulative distributions under the nulls~\eqref{eq:centre_q1},~\eqref{eq:nullBest_q1} conditionally to $E(i_1)$ : $\mathds F'_{1, (i_1)}$ and $\mathds F''_{1, (i_1)}$. In practice, there is no closed form for these conditional cumulative distributions, and we rely on an empirical version that we build using a hit-and-run sampler algorithm, as described in Section \ref{sec:sampler}.
% LJ: je fais une tentative pour ne pas parler de ce point avant la fin du sampler, je remets si ça ne marche pas :)
%As the model described in \eqref{eq:model} depends on parameters $\bm \mu$ and $\sigma$, we will first consider them as known to facilitate the reading, before releasing this assumption in \ref{sampling_null} and \ref{sampling_null_unknown_sigma}.
Hypotheses \eqref{eq:centre_q1} and \eqref{eq:nullBest_q1} lead to very similar results when the meshes are small enough, which is easily the case in practice. \eqref{eq:centre_q1} gives us insights on one specific motif of the mesh --- the center, but \eqref{eq:nullBest_q1} tells us about whether there exists a motif within $M_{i_1}$ associated with the phenotype. To illustrate the difference, let us consider a meshing with only one bin per coordinate, that is the meshing with only one mesh, containing all the motifs:
%Although \eqref{eq:centre_q1} and \eqref{eq:nullBest_q1} lead to very similar results when the meshes are small enough, they yield different interpretations in the general case. \eqref{eq:centre_q1} gives us insights on one specific motif of the mesh --- the center, but \eqref{eq:nullBest_q1} tells us about whether there exists a motif within $M_{i_1}$ associated with the phenotype. To illustrate this difference, let us consider a meshing with only one bin per coordinate, that is the meshing with only one mesh, containing all the motifs:
\begin{itemize}
\item Testing the center-based null hypothesis~\eqref{eq:centre_q1} boils down to testing the association of~$\bm\mu$ with the motif $\vc_1$ with the same probabilities for each letter of $\gA$ at every position, and produces a $p$-value of $1$, regardless of the data, since for any $k$-mer $u$, $\lVert \vc - \vu \lVert^2 = k\times(0.75^2+3\times 0.25^2)$, which leads to $\forall \mX \in \gX,~\bm{\varphi}^{\vc, \mX}=\bm{0}$ according to the centering step, and to a zero score for any $\vy' \in \gE$ .
\item By contrast, one can obtain a strictly less than 1 $p$-value for \eqref{eq:nullBest_q1}, because different $\vy' \in \gE$ can lead to different scores, which means that there may exist a motif in $\gZ$ associated with $\vy$--- but does not inform us on which motif it is.
%not give us any information on what this motif looks like.
\end{itemize}
%\subsection{Here comes the Hypersphere Directions Hit-and-Run sampler}
\subsection{Sampling from the conditional null distribution with the Hit-and-Run algorithm}
\label{sec:sampler}
Even after reducing our selection to a finite set (Section~\ref{sec:mesh}), a rejection sampling strategy that would draw~$\vy'$ from either (\ref{eq:model}, \ref{eq:nullCenter}) or (\ref{eq:model}, \ref{eq:nullBest}) and only retain those leading to the selection of the same mesh as~$\vy$ is not tractable as the rejection rate is empirically too low. Following~\cite{slim2019kernelpsi}, we resort to a Hypersphere Direction strategy (Algorithm~\ref{alg:hr_sampler}).
{
\normalsize%\small%\tiny
\begin{algorithm}[!t]
\captionsetup{labelfont={sc,bf}%, labelsep=newline
}
\caption{Hypersphere Directions hit-and-run sampler}
\label{alg:hr_sampler}
\SetAlgoLined
{\nonl{\tcc{
{\bf Description:} The Hypersphere Directions hit-and-run sampler creates a discrete-time Markov chain on an open and bounded region and is used to approximate a uniform distribution on the selection event $E$.}}}
\BlankLine
\Inputs{Response~$\vy\in E \subseteq\mathds R^n$, $B$ and $R$ the numbers of burn-in iterations and replicates.}
\BlankLine
\KwResult{$\vy^{'(B+1)},...,\vy^{'(B+R)} \in E \subseteq\mathds R^n$ the replicates sampled under the conditional null distribution}
\BlankLine
\BlankLine
$\tilde \vy ^{'(0)} \gets \mathds L(\vy)$; \tcc{$\mathds L$ is the cumulative distribution function of $\mathcal N(\bm \mu, \sigma^2\bm C_n)$}
\For{$t = 1,\dots,B+R$}
{
Sample uniformly $\bm{\theta}^{(t)}$ from $\left\{ \bm{\theta} \in \R^n,~ \lVert \bm{\theta} \lVert =1\right\}$;
$\displaystyle a^{(t)} \gets \max\left\{ \max_{\bm{\theta}_t^{(i)} > 0} -\frac{\tilde\vy^{'(t-1)}}{\bm{\theta_t}} ;
\max_{\bm{\theta}_t^{(i)} < 0} \frac{\bm{1}-\tilde \vy^{'(t-1)}}{\bm{\theta_t}} \right\}$;
$\displaystyle b^{(t)} \gets \max\left\{ \min_{\bm{\theta}_t^{(i)} < 0} -\frac{\tilde \vy^{'(t-1)}}{\bm{\theta_t}} ;
\min_{\bm{\theta}_t^{(i)} > 0} \frac{\bm{1}-\tilde \vy^{'(t-1)}}{\bm{\theta_t}} \right\}$; \tcc{Sampling $\lambda^{(t)}$ from $\left]a^{(t)}, b^{(t)} \right[$ ensures that $\tilde \vy^{'(t-1)} + \lambda^{(t)} \bm{\theta^{(t)}} \in ]0,1[^n$}
\While{$\vy^{'(t)} \notin E $}
{
\label{alg:rejection}
{\nonl{\tcc{
This loop is parallelized on several cores until one of them discovers a replicates in the selection event.}}}
Sample uniformly $\lambda^{(t)}$ from $\left]a^{(t)}, b^{(t)} \right[$;
$\tilde\vy^{'(t)} \gets \tilde \vy^{'(t-1)} + \lambda^{(t)} \bm{\theta^{(t)}}$;
$\vy^{'(t)} \gets \mathds L^{-1}(\tilde \vy^{'(t)})$;
}
}
\end{algorithm}
}
The hit-and-run algorithm produces uniform samples from an open and bounded acceptance region---corresponding, in our case, to the selection event. It starts from any point in the acceptance region, draws a random direction from this point and samples along this direction until it finds one elements that also falls in the acceptance region. It then follows the same procedure from this new starting point. The hit-and-run sampler therefore also relies on rejection but it does so along a single dimension rather than from $\sR^n$. It explores the selection event step by step, starting from a point that belongs to this event, which guarantees a higher acceptance rate. To speed up the procedure, we parallelize the rejection step across several cores.
Because each point sampled by the hit-and-run procedure depends on the previous one, it is impossible to parallelize the whole sampling process. By contrast, the rejection step used for computing a single replicate, once a sampling direction has been fixed, can be parallelized. We draw several distances to the initial point independently, optimizing new independent points, until one of them belongs to the selection event. This parallelization provides a significant time saving, as discussed in Section \ref{sec:comp_cost}. Algorithm~\ref{alg:hr_sampler} produces uniform samples from an open and bounded acceptance region. The boundedness assumption does not hold in our case as the $\argmax$ over~$\gZ$ of the score only depends on the direction of $\vy$ and not on its norm. The openness requirement is ensured by the definition of the meshes. Following~\cite{slim2019kernelpsi} again, we use the reparameterization $\tilde{\vy} = \mathds L(\vy)$, where $\mathds L:\sR^n \rightarrow ]0,1[^n$ is defined as $\mathds L(\vy)_i = \mathds L_{\bm{\mu}, \sigma^2}(\vy_i)$ for $i=1,\dots,n$ and $\mathds L_{\bm{\mu}, \sigma^2}$ denotes the cumulative distribution function of $\mathcal N(\bm{\mu}, \sigma^2 \mC_n)$. Sampling uniform $\tilde{\vy}$ from the open bounded space $]0,1[^n$ indirectly provides normal samples from $\mathcal N(\bm{\mu}, \sigma^2 \mC_n)$.
Combining this sampling strategy with the quantization of the selection event introduced in Section~\ref{sec:mesh} and the selective null hypotheses attached to this event introduced in Section~\ref{sec:q_equal_1} provides a selective inference procedure for one selected motif $\vz_1$ ($q=1$) and a null defined by a given pair $(\bm{\mu},\sigma)$ of parameters. Our next two steps are to handle the selection of multiple motifs, and the general case where several $\bm{\mu}$ describe the same null hypothesis and $\sigma$ is not specified.
\subsection{Dealing with the selection of several motifs ($q>1$)}
%\subsubsection{Selective hypotheses and assumptions considered}
%\subsubsection*{Selective null hypotheses on meshes}\label{sec:selective_null}
%Unlike in Section \ref{sec:q_equal_1},
We now consider that we selected $q>1$ motifs with the SEISM procedure, leading to the general~\eqref{e:selection_event_2} selection event $E(i_1,\dots,i_q)$ . Generalizing our single-motif strategy of Section~\ref{sec:q_equal_1}, we propose two options for defining null hypotheses (and test statistics) related to this selection event:
\begin{itemize}
\item The first one relies on the centers of the selected meshes:
\begin{equation}\label{eq:nullCenter}
\displaystyle \mathds H_{0,j}\,:\,‘‘s(\vc_j, \bm{\Pi'_{j}}\bm{\mu})=0",
\end{equation}
where $\bm{\Pi'_j}$ is the orthogonal projector onto $\Span_{\ell \neq j}\big\{\bm{\varphi}^{\vc_\ell, \mX}\big\}^\bot$. In other words, it expresses that the center of the mesh $M_{i_j}$ is associated with $\bm \mu$ after removing its component carried by the span of the centers of the meshes corresponding the the $q-1$ other motifs.
\item And the second one takes advantages of the best motifs in each mesh:
\begin{equation}\label{eq:nullBest}
\displaystyle \mathds H_{0,j}\,:\, ‘‘\forall (\vz^*_{i_\ell})_{\ell\neq j}\in (M_{i_\ell})_{\ell\neq j}\,,\quad\forall \vz \in M_{i_j}\,,\ s(\vz, \bm{\Pi''\left( \left(\vz^*_{i_\ell}\right)_{\ell\neq j} \right) }\bm{\mu})=0",
\end{equation}
with $\bm{\Pi''\left( \left(\vz^*_{i_\ell}\right)_{\ell\neq j} \right) }$ being the projection onto $\Span_{\ell \neq j}\big\{\bm{\varphi}^{\vz^*_{i_\ell}, \mX}\big\}^\bot$.
\end{itemize}
Generalizing what we introduced for $q=1$ (Section~\ref{sec:q_equal_1}), we test those hypotheses using $V'_j = s(\bm c_j, \bm\Pi'_j\vy)$ and $V''_j = \max_{\vz\in M_{i_j}} s(\vz, \bm\Pi''_j\vy)$. To that end, we rely on their cumulative distributions under the nulls~\eqref{eq:nullCenter},~\eqref{eq:nullBest} conditionally to $E(i_1,\dots,i_q)$ : respectively $\mathds F'_{1,\dots,q (i_1,\dots,i_q)}$ and $\mathds F''_{1,\dots,q, (i_1,\dots,i_q)}$, empirically approximated with Algorithm~\ref{alg:hr_sampler}.
Following the work of~\cite{loftus2015selective} in the finite case, both versions of our null hypothesis are joint across the $q$ motifs: each of them considers the association between the $j$-th selected motif and $\bm\mu$ after projecting onto the span of all others, not just the ones that were selected before --- using $\bm \Pi'$ and $\bm \Pi''$. This is to be contrasted to our sequential selection process, which adjusts at each steps for the previously selected motifs using $\bm P$.
In order to give more insights on these null hypotheses, we derive the following proposition:%We address this issue by relying on a first assumptions, verified by $s_{\text{HSIC}}$ and $s_{\text{ridge}}$, as follows.
\begin{proposition}[Description of the selective nulls]%\lj{selective ou selected?}
\label{prop:orhtogonal_decomposition}
Let $\mZ = \{\vz_1, \dots, \vz_ q\}$ be $q$ sequence motifs. Let $s(\cdot,\cdot)$ be a score such that "nullity implies orthogonality" (for instance $s^{\text{HSIC}}$ or $s^{\text{ridge}})$:
\begin{itemize}
\item[$\bf (A_1)$] {\bf Nullity implies orthogonality: }If $\{s(\vz,\vy)=0\}$ then $\{\langle \bm{\varphi}^{\vz, \mX},\vy\rangle =0\}$, for every $(\vy,\vz)\in\mathcal E\times\mathcal Z$, and for some function $\vz\to\bm{\varphi}^{\vz, \mX}\in\mathcal E$.
\end{itemize}
Let $\bm{\mu} \in \mathcal E$ and decompose $\bm{\mu}$ as
\begin{equation}
\label{eq:decompostion_mu}
\bm{\mu}=\sum_{j=1}^q \alpha_j \bm{\varphi}^{\vz_j, \mX} + \underline{\bm{\mu}}
\end{equation}
with $\underline{\bm{\mu}}\in \mathcal E$ orthogonal to $\Span(\bm{\varphi}^{\mZ, \mX})$. %, and $c_k$ are uniquely defined.
It holds that $‘‘s(\vz_j, \bm{\Pi_j}\bm{\mu})=0"$ is equivalent to~$‘‘\alpha_j=0"$ for some decomposition \eqref{eq:decompostion_mu}.
\end{proposition}
If $\mathrm{Rank}(\bm{\varphi}^{\mZ, \mX})=q$ then the decomposition \eqref{eq:decompostion_mu} is unique, and the greedy selection procedure described in Section \ref{subsec:selection} enforces this situation. We interpret this as follows: we look at a motif $\vz_\ell$ and would like to test its significance; in view of property $\bf (A_1)$, we can eliminate the effects that are captured by the other motifs by using the orthogonal projection onto the orthogonal of $\Span(\bm{\varphi}^{\vz_j, \mX})$, given by $\bm\Pi_j$ (using $\bm \Pi_j = \bm \Pi'_j $ or $ \bm \Pi_j = \bm{\Pi''\left( \left(\vz^*_{i_\ell}\right)_{\ell\neq j} \right)} $), and consider $\bm{\Pi_j}\vy$ to test the association $‘‘s(\vz_j, \bm{\Pi_j}\bm{\mu})=0"$; equivalent to testing $‘‘\alpha_j=0"$ by the above proposition.
%We introduce two possible null hypotheses based on this quantized selection event.
%A mesh is characterized by a size $m$. \textcolor{red}{faire lien avec l'image}
%{$\bullet$ \bf Selective hypotheses defined by the centers}
%A first option is to represent a mesh $M$ by its center $\vc$. When selecting $q$ motifs, our null hypotheses are then, for $j=1,\ldots,q$:
%where $\bm{\Pi'_j}$ is the orthogonal projector onto $\displaystyle \Span_{\ell \neq j}\big\{\bm{\varphi}^{\vc_\ell, \mX}\big\}^\bot$.
%In other words, \eqref{eq:nullCenter} expresses that the center of the mesh corresponding to a motif is associated with $\bm\mu$ after removing its component carried by the span of the centers of meshes corresponging to the $q-1$ other selected motifs.
%{$\bullet$ \bf Selective hypotheses defined by the best motif}%arguments highest associations}
%A second possibility is to represent $M_{i_j}$ by its element with highest association and test
%\begin{equation}\label{eq:nullBest_}
%\displaystyle \mathds H_{0,j}\,:\, ‘‘\forall \vz \in M_{i_j}\,,\ s(\vz, \bm{\Pi''_j}\bm{\mu})=0",
%\end{equation}
%old equation
%\begin{equation}\label{eq:nullBest}
%\displaystyle \mathds H_{0,\ell}\,:\, ‘‘\forall (\vz^*_{i_j})_{j\neq \ell}\in (M_{i_j})_{j\neq k}\,,\quad\forall \vz \in M_{i_\ell}\,,\ s(\vz, \bm{\Pi''_\ell}\bm{\mu})=0",
%\end{equation}
%\av{il faut mieux décrire l'hypothèse nulle + expliquer la différence entre $P$, $\Pi'$ et $\Pi''$. Peut-être en équations 12 et 13. Corriger les indices de l'équation 15}
%where $\bm{\Pi''_j}$ is the projection onto $\displaystyle \Span_{\ell \neq j}\big\{\bm{\varphi}^{\vz^*_{i_\ell}, \mX}\big\}^\bot$ and $\displaystyle \vz^*_{i_\ell} = \argmax_{\vz \in M_{i_\ell}}s(\vz, \mP_l \vy)$.% and $\vz_j$ is the motif selected at step $j$ of the selection procedure.
% LJ: le commentaire ci-dessous serait vrai pour les deux selective nulls non?
% Here, we are testing if the selected meshes contain motifs that are associated with $\bm \mu$, once the signal carried by the other meshes has been removed.
%, as illustrated in the following toy example. %Using the center of the mesh informs us about how much $M$ as a whole is associated to the phenotype \ydc{this is not rather just a fixed point of $M$?}\av{When a mesh $M$ is represented by its center, it iprovides information on the association between motifs near to the mesh center and the phenotype}, whereas using the best motif informs us on the existence of a motif in this mesh associated with the outcome. \ydc{highest is more restrictive. As I see it: center is hypothesis for a fixed sequence $\vc_k$ while highest is hypothesis for all possible sequences $\vz_k^*$. }\av{Il faut qu'on discute de ça mardi}
% {$\bullet$ \bf Toy example: the single mesh case}
%Let us for example consider the extreme case of a single mesh containing all possible motifs. \lj{Encore un peu confus, il faudra refaire un passage pour clarifier le message de cet exemple}
%\indent $\star$
%Testing the center-based null hypothesis~\eqref{eq:nullCenter} boils down to testing the association of~$\bm\mu$ with the motif $\vc$ with the same probabilities for each letter of $\gA$ at every position, and produces a $p$-value of $1$ regardless of the data since for any $k$-mer $u$, $\lVert \vc - \vu \lVert^2 = k\times(0.75^2+3\times 0.25^2)$, which leads to $\forall \mX \in \gX,~\bm{\varphi}^{\vc, \mX}=\bm{0}$.
%\av{A GARDER ? In this particular case, the hypothesis is not selective (as it does not depend on the selection) and it can be written as
%\begin{equation*}
%\displaystyle \mathds H_{0}\,:\,‘‘s(\vc,\bm{\mu})=0",
%\end{equation*}
%for a fixed $\vc$, as defined above.}
%\indent $\star$
%By contrast, using the best motif in this mesh $\vz^*$ leads to a selective null hypothesis. A significant $p$-value means that there exists a motif in $\gZ$ associated with $\vy$---but gives no information on this specific motif.
%\av{A GARDER ? In this case, the hypothesis is selective and it can be written as
%\begin{equation*}
%\displaystyle \mathds H_{0}\,:\,‘‘s(\vz^*,\bm{\mu})=0",
%\end{equation*}
%for $\vz^*$ the best motif associated to $\vy$.}
% LJ: a placer au bon endroit quand on aura organisé l'essentiel de la structure. Il me semble aussi qu'on avait parlé d'inclure A1 dans la proposition.
%\subsubsection*{A description of the hypotheses}
%\label{sec:description}
%\av{Je n'arrive pas à aboutir à une version satisfaisante de cette section}
%We introduce a first assumption, satisfied by $s_{\text{HSIC}}$ and $s_{\text{ridge}}$, as follows.%We address this issue by relying on a first assumptions, verified by $s_{\text{HSIC}}$ and $s_{\text{ridge}}$, as follows.
%\begin{itemize}
% \item[$\bf (A_1)$] {\bf Nullity implies orthogonality: }If $\{s(\vz,\vy)=0\}$ then $\{\langle \bm{\varphi}^{\vz, \mX},\vy\rangle =0\}$, for every $(\vy,\vz)\in\mathcal E\times\mathcal Z$, and for some function $\vz\to\bm{\varphi}^{\vz, \mX}\in\mathcal E$.
%\end{itemize}
%The assumption $\bf (A_1)$ is meant to enforce the interpretation of our forward step-wise procedure as depicted by the following proposition:
%\begin{proposition}[Description of the selective nulls]%\lj{selective ou selected?}
%\label{prop:orhtogonal_decomposition}
%Let $\mZ = \{\vz_1, \dots, \vz_ q\}$ be $q$ sequence motifs. Let $s(\cdot,\cdot)$ be a score such that $\bf (A_1)$ holds. Let $\bm{\mu} \in \mathcal E$ and decompose $\bm{\mu}$ as
%\begin{equation}
% \label{eq:decompostion_mu}
% \bm{\mu}=\sum_{j=1}^q \alpha_j \bm{\varphi}^{\vz_j, \mX} + \underline{\bm{\mu}}
%\end{equation}
%with $\underline{\bm{\mu}}\in \mathcal E$ orthogonal to $\Span(\bm{\varphi}^{\mZ, \mX})$. %, and $c_k$ are uniquely defined.
%It holds that $‘‘s(\vz_j, \bm{\Pi_j}\bm{\mu})=0"$ is equivalent to~$‘‘\alpha_j=0"$ for some decomposition \eqref{eq:decompostion_mu}.
%\end{proposition}
%Remark that if $\mathrm{Rank}(\bm{\varphi}^{\mZ, \mX})=q$ then the decomposition \eqref{eq:decompostion_mu} is unique, and that the greedy selection procedure described in \ref{subsec:selection} enforces this situation. We interpret this as follows: we look at a motif $\vz_\ell$ and would like to test its significance; in view of property $\bf (A_1)$, we can eliminate the effects that are captured by the other motifs by using the orthogonal projection onto the orthogonal of $\Span(\bm{\varphi}^{\vz_j, \mX})$, given by $\bm\Pi_j$, and consider $\bm{\Pi_j}\vy$ to test the association $‘‘s(\vz_j, \bm{\Pi_j}\bm{\mu})=0"$, which is equivalent to testing $‘‘\alpha_j=0"$ by the above proposition.
%\lj{Adapter le paragraphe ci-dessous en fonction de ce qu'on aura expliqué des méthodes PSI (sampling vs distribution theorique etc) plus haut}
%The last missing step to a practical selective testing procedure is to produce a conditional null distribution of some statistic. We simply rely on the scores $V'_j=s(\vc_j, \bm{\Pi'_j}\vy)$ and $V_j'' = \max_{\vz \in M} s(\vz, \bm{\Pi''j} \vy)$ for~\eqref{eq:nullCenter} and~\eqref{eq:nullBest} respectively. We build empirical cumulative distributions $\mathds F'_{j,(i_1,\ldots, i_q)}$ and $\mathds F''_{j,(i_1,\ldots, i_q)}$ for these statistics by sampling under the conditional nulls and obtain a p-values for data $(X, \vy)$ by $\mathds F'_{j,(i_1,\ldots, i_q)}(V'_j)$ (resp. $\mathds F''_{j,(i_1,\ldots, i_q)}(V''_j)$).
%\lj{Rajouter ici si on veut en dire plus sur la zone de rejet? (commenté ci-dessous)}
%\subsubsection*{Test statistics}
%\label{sec:test_statistics}
%We propose to test hypotheses~\eqref{eq:nullCenter} %and~\eqref{eq:nullBest} using the following test statistics.
%\noindent
%{$\bullet$ \bf Selective inference defined by the centers}
%We consider the test statistics defined by
%\[
% V'_j=s(\vc_j, \bm{\Pi'_j}\vy)\,
%\]
%Let $\mathds F'_{j,(i_1,\ldots, i_q)}$ be the cumulative distribution of $s(\vc_j, \bm{\Pi'_j}\vy)$ conditional on the selection event $\vy\in E(i_1,\ldots, i_q)$. Then the conditional distribution of $\mathds F'_{j,(i_1,\ldots, i_q)}(V'_j)$ is the uniform distribution, hence it can be interpreted as significance level.
%\noindent
%{$\bullet$ \bf Selective hypotheses defined by the arguments highest %associations}
%We consider here the test statistics defined by
%\[
%V_j'' = \max_{\vz \in M} s(\vz, \bm{\Pi''j} \vy)\,,
%\]
%The same observation can be made here: $\mathds F''_{j,(i_1,\ldots, i_q)}(V''_j)$ can be interpreted as a significance level, with $\mathds F''_{j,(i_1,\ldots, i_q)}$ the conditional cumulative distribution of $\max_{\vz \in M} s(\vz, \bm{\Pi''j} \vy)$.
%Here, the choice of the test statistics does not impact the calibration, as long as the significance threshold is determined by the cumulative distribution functions of these statistics. However, this choice were made in order to maximize the statistical power of the tests, taking into consideration the description from \ref{sec:test_statistics}. For the hypothesis defined by the centers \eqref{eq:nullCenter}, the test is performed over the component of $\bm \mu$ over $\bm{\varphi^{c_j, X}}$, and for the one defined by the best motif \eqref{eq:nullBest}, the component over the best motif in the mesh is tested.
%\av{Est-ce qu'il faut vraiment préciser la forme de la zone de rejet ? On test l'hypothèse que le score est nul, sachant qu'il ne peut pas être négatif, ça me paraît trivial non ? Mais peut-être que je passe à côté de quelque chose ? En tous cas pour le moment je n'arrive pas à rédiger quelque chose de satisfaisant}
%\ydc{il y a une petite subtilité ici avec le fait que $\bm{\Pi''_\ell}$ dépend de $\vy$ et non pas que du selection event. Mais par \eqref{e:trick_translation}, il n'y a pas de problème. UPDATE : 1/ toujours calibré (Fct du $y$ et applique la cdf de cette loi image); 2/ le coix de la stat est surtout par rapport à la puissance -- pour les centres on teste la composante de mu projetée sur $c_\ell$ -- pour le meilleur score on teste la composante de mu projetee sur le meilleur kmer selectionné }\av{Oui, à discuter}
%\ydc{idées de blabla : 1/ toujours calibré (Fct du $y$ et applique la cdf de cette loi image); 2/ le coix de la stat est surtout par rapport à la puissance -- pour les centres on teste la composante de mu projetée sur $c_\ell$ -- pour le meilleur score on teste la composante de mu projetee sur le meilleur kmer selectionné } \av{proposition au-dessus}
%\ydc{mettre ici la zone de rejet ?}
\subsection{Sampling under selective multiple hypotheses with known $\sigma$}\label{sampling_null}
%With the selective null hypotheses and test statistics properly defined, we now introduce our procedure to sample under the corresponding distributions, which will allow us to produce $p$-values.
%As our model \eqref{eq:model} depends on a mean $\bm{\mu}$ and a variance $\sigma^2$, both of which are unobserved, we need to define a sampling strategy independent of those parameters. In this first section, we will consider $\sigma$ to be known.
The sampling strategy described in Section~\ref{sec:sampler} builds a conditional null distribution---therefore offering a selective inference procedure---for a given $\bm{\mu}$ and $\sigma$. In practice, $\sigma$ is not known, and several values of $\bm{\mu}$ can describe the selective null hypotheses~\eqref{eq:nullCenter} or~\eqref{eq:nullBest} for a given motif selection. Of note, this issue is not specific to our selective inference procedure. It will arise in any sampling-based post-selection inference strategy including data-split: even if the latter samples from a non-selective null hypothesis, it still needs concrete values for $\bm\mu$ and $\sigma$.
%\av{This issue also exists in data-split strategies. Indeed, the test statistics are also sampled under the non selective null hypotheses.}
We leave aside the choice of $\sigma$ for now, and describe how we can sample from any null distribution~\eqref{eq:nullCenter} or~\eqref{eq:nullBest} using $\bm{\mu}=0$ for a given $\sigma$. Our results holds for scores verifying the following assumption---this includes both $s^{\text{HSIC}}$ and $s^{\text{ridge}}$:
%Recall that our null hypothesis corresponds to a set of probability distributions of $\vy$ that may depend on $\bm{\mu}$.
%We address this issue by relying on a second assumption, verified %by $s_{\text{HSIC}}$ and $s_{\text{ridge}}$, as follows.
\begin{itemize}
\item[$\bf (A_2)$] {\bf Nullity implies translation-invariant: }If $s(\vz,\vy)=0$ then $\forall\vy'\in\mathcal E\,,\ s(\vz,\vy')=s(\vz,\vy+\vy')$, for every $(\vy,\vz)\in\mathcal E\times\mathcal Z$;
\end{itemize}
Under this assumption, the following proposition ensures that using the quantile of the empirical distribution of scores sampled under $\bm{\mu}=0$ leads to a calibrated test procedure:
\begin{proposition}\label{prop:known_sigma}
Let $s$ be an association score such that $\bf (A_2)$ holds. Let $V'_j = s(\bm c_j, \bm\Pi'_j\vy)$ and $V''_j = \max_{\vz\in M_{i_j}} s(\vz, \bm\Pi''_j\vy)$, formed from $\vy$ sampled from~\eqref{eq:model} with any mean $\bm \mu$ such that $s(\vz', \bm\mu)=0$, any known variance $\sigma>0$, and such that $\vz' = \argmax_{\vz\in \gZ}s(\vz, \vy)$. The conditional null distributions $\mathds F'_{j,(i_1,\ldots, i_q)}$ and $\mathds F''_{j,(i_1,\ldots, i_q)}$, with mean $\bm 0$ and variance $\sigma$ verify:
%Under the Gaussian model described in \eqref{eq:model}, and assuming the value of $\sigma$ is known, for any association score $s$ such that $\bf (A_2)$ holds, we have:
\begin{equation*}
\mathds F'_{j,(i_1,\ldots, i_q)}(V'_j) \sim \text{Unif}\,(0,1)~\text{and}~\mathds F''_{j,(i_1,\ldots, i_q)}(V''_j) \sim \text{Unif}\,(0,1)
%\mathds F'_{j,(i_1,\ldots, i_q)}(V'_j) \stackrel{\mathds H_0}{\sim} \text{Unif}\,(0,1)~\text{and}~\mathds F''_{j,(i_1,\ldots, i_q)}(V''_j) \stackrel{\mathds H_0}{\sim} \text{Unif}\,(0,1)
\end{equation*}
\end{proposition}
\begin{proof}
Assumption $\bf (A_2)$ under the Gaussian model \eqref{eq:model} implies the following property:
\begin{equation}
\label{e:trick_translation}
\begin{split}
\forall (\vz, \mA, \vy) \in \gZ \times \mA \times \gE \text{ such that } \vy = \bm \mu + \sigma \bm \epsilon, \\
‘‘s(\vz,\bm A\bm\mu)=0"\Longrightarrow ‘‘s(\vz,\bm A\vy)=s(\vz,\sigma\bm A\bm{\eps})",
\end{split}
\end{equation}
which implies that, for a composite null hypothesis of the form $\mathds H_0\,:\,‘‘s(\vz,\bm A\bm\mu)=0"$, the distribution of~$s(\vz,\bm A\vy)$ does not depend on the mean $\bm\mu$ that satisfies $\mathds H_0$. Hence, even if the hypothesis $\mathds H_0$ corresponds to a set of probability distributions of $\vy$ that may depend on $\bm\mu$, the distribution of the statistic $s(\vz,\bm A\vy)$ does not depend on $\bm\mu$ under this hypothesis. We can then conclude that if $\sigma$ is known, as it is assumed to be the case in this section, then a test statistic of the form $V=s(\vz,\bm\Pi\vy)$ has the same distribution as~$s(\vz,\sigma\bm\Pi\bm{\epsilon})$.
\end{proof}
%Therefore, we can study the distribution of $V'$ or $V''$ by sampling $\vy$ under \eqref{eq:model} while fixing $\bm{\mu} = \bm 0$.
\subsection{Sampling under selective multiple hypotheses with unknown $\sigma$}\label{sampling_null_unknown_sigma}
In practice, $\sigma$ is often unknown. To address this issue, we rely on the normalized versions of the test statistics~$V'$ and $V''$ introduced in Section~\ref{sec:q_equal_1}, defined by
\begin{equation}
\label{eq:normalized}
T'_j := \frac{s(\vc_j, \bm\Pi'_j \vy)}{\lVert \vy \lVert^2}
\quad\text{ and }
\quad T''_j := \max_{\vz \in M_{i_j}}\frac{s\Big(\vz,
{\bm\Pi''
\big( (\bm\vz_{\ell})_{\ell\neq j} \big)} \bm\vy\Big)}{\lVert \vy \lVert^2}\,
\end{equation}
where $\vz_\ell = \argmax_{\vz \in \gZ}s(\vz, \mP_\ell \vy)$. We will denote $\mathds G'_{j,(i_1,\dots,i_q)}$ and $\mathds G''_{j,(i_1,\dots,i_q)}$ their cumulative distribution functions under the null, conditionally to $E(i_1,\dots,i_q)$.
We will also make use of a third assumption, here again fulfilled by $s^{\text{HSIC}}$ and $s^{\text{ridge}}$:
\begin{itemize}
\item[$\bf (A_3)$] {\bf Two-homogeneity: }It holds that $s(\vz,t\vy)=t^2s(\vz,\vy)$ for all $(\vy,\vz) \in \gE \times \gZ$ and all $t>0$.
\end{itemize}
Of note, normalizing the association score with respect to the labels does not affect the selection:
\begin{equation}\label{argmax_norm}
\forall \vy \in \gY,~\argmax_{\vz \in \gZ}s(\vz, \vy) = \argmax_{\vz \in \gZ}\frac{s(\vz, \vy)}{\lVert \vy \lVert^2}
\end{equation}
If $\bm\mu=\bm 0$, the distribution of the normalized statistics does not depend on $\sigma$, and the empirical cumulative distribution functions of normalized scores obtained by sampling under $\bm\mu=\bm 0$ and any $\sigma$ still provide a valid inference procedure :
\begin{proposition}
\label{prop:variance-free} Let $s$ be an association score such that $\bf (A_2)$ and $\bf (A_3)$ hold. Let $T'_j = s(\bm c_j, \bm\Pi'_j\vy)/\lVert \bm y \lVert^2$ and $T''_j = \max_{\vz\in M_{i_j}} s(\vz, \bm\Pi''_j\vy)/\lVert \bm y \lVert^2$, formed from $\vy$ sampled from~\eqref{eq:model} with mean $\bm \mu = \bm 0$, and any variance $\sigma>0$. Then for all $\sigma'>0$, their conditional null distributions $\mathds G'_{j,(i_1,\ldots, i_q)}$ and $\mathds G''_{j,(i_1,\ldots, i_q)}$ with mean $\bm 0$ and variance $\sigma'$ verify:
%Under the Gaussian model described in \eqref{eq:model}, and assuming the value of $\sigma$ is known, for any association score $s$ such that $\bf (A_2)$ holds, we have:
\begin{equation*}
\mathds G'_{j,(i_1,\ldots, i_q)}(T'_j) \sim \text{Unif}\,(0,1)~\text{and}~\mathds G''_{j,(i_1,\ldots, i_q)}(T''_j) \sim \text{Unif}\,(0,1)
\end{equation*}
%Under the Gaussian model described in \eqref{eq:model}, and assuming $\bm \mu =0$, for any association score $s$ such that $\bf (A_2)$ and $\bf (A_3)$ hold, we have:
%\begin{equation*}
% \mathds G'_{j,(i_1,\ldots, i_q)}(T'_j) \stackrel{\mathds H_0,\bm\mu=0}{\sim} \text{Unif}\,(0,1)~\text{and}~\mathds G''_{j,(i_1,\ldots, i_q)}(T''_j) \stackrel{\mathds H_0,\bm\mu=0}{\sim} \text{Unif}\,(0,1)
%\end{equation*}
\end{proposition}
\begin{proof}
Let us consider two different normal models as defined in \eqref{eq:model} under the global null hypothesis $‘‘\bm\mu= \bm 0"$ and given by \[
y^{(1)} = \sigma^{(1)} \bm\eps^{(1)}
\quad
\text{ and }
\quad
y^{(2)} = \sigma^{(2)} \bm\eps^{(2)}
\]
Then
\[
\frac{s(\vc_j, \bm\Pi'_j \vy^{(1)})}{\lVert \vy^{(1)} \lVert^2} \sim \frac{s(\vc_j, \bm\Pi'_j \vy^{(2)})}{\lVert \vy^{(2)} \lVert^2}
\quad
\text{ and }
\quad
\frac{s(\vz, \bm \Pi''\left( \left(\vz_{\ell}\right)_{\ell\neq j} \right)\vy^{(1)})}{\lVert \vy^{(1)} \lVert^2} \sim\frac{s(\vz, \bm \Pi''\left( \left(\vz_{\ell}\right)_{\ell\neq j} \right) \vy^{(2)})}{\lVert \vy^{(2)} \lVert^2}
\,.
\]
%\ydc{le max ci dessus est déjà contenu dans $\Pi''$ non ? ne faut il pas l'enlever ? idem pour \ref{argmax_norm}}\av{Non, pour moi ça paraît OK, la stat de test inclue bien le max non ? (en revanche il y avait un problème, j'avais remplacé les z par ces $c_j$}
The proof directly follows assumption~$\bf (A_3)$ applied with $t=\lVert \vy^{(\cdot)} \lVert^2$. Proposition~\ref{prop:variance-free} is complementary to Proposition~\ref{prop:known_sigma} and provides a selective inference procedure when $\sigma$ is unknown, under the special null hypothesis $\bm\mu=0$.
\end{proof}
%However, this result no longer holds when dealing with the composite null hypothesis, that is when removing the $\bm\mu=\bm0$ assumption.
Our final result investigates the testing procedures for the general null hypotheses~\eqref{eq:nullCenter} and~\eqref{eq:nullBest}---not restricted to $\bm\mu=0$---with an unknown $\sigma$. Recall that the decision rule is to reject the null hypothesis if the observed value of the statistic is greater than a given threshold $t$. We show that choosing $t$ to be a quantile for the global null hypothesis ($\bm\mu=0$) leads to a calibrated (for the type I error) non-selective procedure, see~\eqref{eq:controlTI}.
\begin{proposition}[Global null achieves lowest observed significance]
\label{prop:composite_hypothese}
Let $\mZ = \{\vz_1, \dots, \vz_ q\}$ be $q$ sequence motifs. Let $s(\cdot,\cdot)$ be a score such that $\bf (A_1)$ and $\bf (A_2)$ hold. Let $\bm{\mu} \in \mathcal E$ be such that
\[
\mathds H_0\,:\,‘‘s(\mZ,\bm\mu)=0"
\]
Then
\begin{equation}
\label{eq:controlTI}
\forall t>0\,,\quad
\sup_{\bm\mu\in\mathds H_0}\mathds P\bigg[\frac{s(\mZ,\bm\mu+\sigma\bm{\epsilon})}{\|\bm\mu+\sigma\bm{\epsilon}\|^2}\geq t\bigg]
=
\mathds P\bigg[\frac{s(\mZ,\bm{\epsilon})}{\|\bm{\epsilon}\|^2}\geq t\bigg]
\end{equation}
%\[
%\frac{s(\mZ,\bm\mu+\sigma\bm{\epsilon})}{\|\bm\mu+\sigma\bm{\epsilon}\|^2}\preceq\frac{s(\mZ,\bm{\epsilon})}{\|\bm{\epsilon}\|^2},
%\]
%where $\preceq$ denotes the stochastic order.
\end{proposition}
We provide a proof in Appendix~\ref{proof:composite_hypothese}. This proof makes an ad-hoc use of Anderson's theorem on a symmetric convex cone (whereas it is usually devoted to symmetric convex bodies).
Proposition~\ref{prop:composite_hypothese} shows that data-split produces a calibrated procedure for testing the general null hypotheses~\eqref{eq:nullCenter} and~\eqref{eq:nullBest} when sampling under the global null ($\bm\mu=0$) the test statistics \eqref{eq:normalized}. %Proposition~\ref{prop:composite_hypothese}
We could not prove an equivalent statement for conditional null hypotheses, and Proposition~\ref{prop:composite_hypothese} therefore does not guarantee the validity of a selective inference procedure sampling under the global null ($\bm\mu=0$). Yet, we used it as a heuristic justification of SEISM and we observed that it leads to empirically calibrated procedures, see Section~\ref{sec:stat_validity_numerical}.
%selective inference but can be used as a heuristic to justify the use of a sampling the test statistics \eqref{eq:normalized} under the global null ($\bm\mu=0$).
%Unlike Propositions~\ref{prop:known_sigma} and \ref{prop:variance-free}, Proposition~\ref{prop:composite_hypothese} has two shortcomings. First, it only provides an inequality, whereas the two previous Propositions provided invariance results.
%Recalling that the decision rule is to reject the null hypothesis if the observed value of the statistic is greater than a given threshold, the stochastic ordering given in Proposition~\ref{prop:composite_hypothese} ensures that sampling replicates using model \eqref{eq:model}, $\bm \mu = \bm 0$ and any fixed value of $\sigma$ will control the type 1 error, possibly at the cost of some statistical power.
In view of Proposition~\ref{prop:composite_hypothese} and its proof, one can see that the alternatives $\bm\mu$ such that ${\|\bm P_q\bm\mu\|}/{\|\bm\mu\|}$ is large have small power. %By design, the most important power losses are achieved for signal vectors $\bm\mu$ with a high component on $\Span_{j\leq q}\big\{\bm{1},\bm{\varphi}^{\vz_j, \mX}\big\}^{\bot}$.
As the selection procedure described in Section~\ref{subsec:selection} achieves good results (Section \ref{selec_perf}), the chosen motifs $\mZ$ should capture the principal components of $\bm\mu$, and therefore are such that ${\|\bm P_q\bm\mu\|}/{\|\bm\mu\|}$ should be small.
%Second, Proposition~\ref{prop:composite_hypothese} does not strictly provide a valid selective inference procedure because it only holds under $\mathds H_0$ as opposed to the conditional null distribution. Although the proof provided in Appendix~\ref{proof:composite_hypothese} is indeed only valid for the non selective framework, the calibration curves obtained during simulations suggest that it can still hold under the selective framework.
%The construction of test statistics and sampling strategies that are independent from both $\sigma$ and the choice of a particular $\bm\mu$ under $\mathds H_0$ is necessary to apply this inference framework to real datasets, where those parameters are unknown. Assuming that Proposition~\ref{prop:composite_hypothese} holds in the conditional framework, it suggests that under these circumstances, relying on the empirical distribution of the normalized statistic leads to a calibrated procedure,
%{\color{red}{
%Réunion du 22/09
%\begin{itemize}
% \item Ordre stochastique pour prouver que la préocédure est bien calibirée sous H0 (le sup sous H0 est atteint pour m=0) DONE
% \item Sous H0 (mu orthogonal à KZX), plus la norme de mu est grande plus on est conservateur = perte de puissance pour ces hypothèses
% \item Heuristic 1: on perd en puissance pour des hypothèses dont la composante sur orthogonal de KZX est grande par rappport à sa composante sur KZX
% \item Heuristic 2: notre procédure de séleciton à l'air bonne, donc a priori la composante sur orthogonal de KZX est bien plus petite par rappport à sa composante sur KZX
% \item Conclusion, il y a une perte de puissance pour certaines hypothèses mais la procédure de sélection évite ces hypothèse (rappelons ici que les ypothèses dépendent de la sélection !)
%\end{itemize}
%}}
%\ydc{commenter ce que cela implique pour \eqref{eq:nullCenter} and \eqref{eq:nullBest}}
\section{Results}
\subsection{SEISM performs as well as state-of-the-art \textit{de novo} motif discovery methods}\label{selec_perf}
In order to compare the accuracy of our selection step with existing motif discovery algorithms, we use the 40 ENCODE Transcription Factors ChIP-seq datasets from K562 cells \citep{encode_project}, each of which contains a known TF motif, denoted $\vm^*$, derived using completely independent assays \citep{jolma_2013}.
%\lj{Décrire ici brièvement les méthods comparées. CKN devrait etre présenté comme une implémentation de CNN à une couche, dont la spécificité est l'entrainement joint des filtres.}
STREME~\citep{bailey_streme_2021} and MEME~\citep{bailey_meme_2006} are state-of-art bioinformatics methods for \textit{de-novo} motifs discovery tasks. STREME identifies motifs that maximize a Fisher score of association between the presence of the motif and the binary class of sequences. By looking for maximum likelihood estimates of the parameters of a mixture model - made up of a background distribution and a model for generating $k$-mers at some positions - that may have produced a particular dataset using an expectation maximisation technique, MEME finds enriched motifs in this dataset. Finally CKN-seq~\citep{chen_biological_2017} is a one-layer CNN tailored to small scale datasets. We set up STREME, MEME and SEISM to select 5 sequence motifs. SEISM is run with a regularization parameter $\lambda = 0.01$. CKN-seq jointly optimizes its filters, which notoriously leads to poor performances when few filters are used. We train it consequently over 128 filters. We measure these accuracy of all methods by comparing the motifs they discover with the known motif corresponding to the transcription factor $\vm^*$. We rely on the Tomtom method ~\citep{tomtom_2007}, which quantifies the probability that the euclidean distance between a random motif and $\vm^*$ is lower than the distance between the discovered motif and $\vm^*$. More precisely for each method we use the lowest Tomtom $p$-value between the known TF motif $\vm^*$ and any of those discovered by the method. The Tomtom score is then defined as $-\log_{10}$ of the Tomtom $p$-value. We define the accuracy of the method as the proportion of experiments where the Tomtom score between its best match and the true TF motif was higher than some threshold.
\begin{figure}[h!]
\centering
\includegraphics[scale=0.5]{figures/curve_2_500_samples.eps}%
\includegraphics[scale=0.5]{figures/curve_1_500_samples.eps}
\caption{\textbf{Left}: Proportion of datasets where the true motif was detected by the designated algorithm. A true motif is said to be detected if its highest Tomtom score with the discovered motifs is greater than 2. \textbf{Right}: Accuracy of motif discovery algorithms on ENCODE TF ChIP-seq datasets. The curves display the proportion of ChIP-seq datasets were the best motif identified by the designated algorithm has a Tomtom score greater than $x$. }
\label{fig:selection_performances}
\end{figure}
Figure \ref{fig:selection_performances} (left panel) demonstrates that SEISM is just as good as, if not superior to, state-of-the-art bioinformatics algorithms at detecting sequence motifs when thresholding Tomtom p-values at $0.01$. The one-layer CNN with jointly optimized filters performs poorly in this experiments, emphasising the importance of greedy optimization for selecting the right motif.
%CKN-seq, a hybrid method that combines one-hidden layer CNNs with kernel methods to predict phenotypes from unannotated sequences by finding sequence motifs,
% \av{BEGINNING OF NEW VERSION}
Figure~\ref{fig:selection_performances} (right panel) shows that SEISM performs slightly worse than STREME and MEME for high thresholds on the Tomtom scores. This suggests that the matrix $z$ that SEISM identifies is close enough to the PWM matrix of the true motif, but farther away than the matrices identified by STREME or MEME. This discrepancy reflects a different usage of $z$ to parameterize a distribution of $k$-mers. In practice, we observe that on a given dataset, the $p$-values of the best motifs discovered by SEISM and STREME are not separated by more than 2 orders of magnitude, which leads to minor differences in the motifs, as illustrated in Table \ref{tab:logo_differences}.
\begin{table}[h!]
\centering
\begin{tabular}{ccc}
\hline & & \\
Reference motif & SEISM motif & STREME motif\\
\textit{(ATF4\_DBD)} & \textit{$p$-value$=10^{-6}$} & \textit{$p$-value$=3\times 10^{-8}$} \\[0.2cm] \hline
& & \\
\includegraphics[scale=0.25]{figures/new_logo_baseline.eps} & \includegraphics[scale=0.25]{figures/new_logo_seism_1e-6.eps}& \includegraphics[scale=0.25]{figures/new_logo_streme_3e-8.eps} \\ \hline
\end{tabular}
\caption{Comparison between two discovered sequence motifs by SEISM or STREME, and the true motif \textit{(ATF4\_DBD)}}
\label{tab:logo_differences}
\end{table}
% Figure~\ref{fig:selection_performances} (right panel) shows that despite SEISM's excellent detection abilities, the motifs it discovers are less accurate than those discovered by STREME or MEME (SEISM performs worse than them for high Tomtom scores). But this difference is explicable and predictable.
% \lj{Je trouve ça trop allusif: on dit qu'on est meilleurs (paragraphe précédents), qu'on est "excellents" puis qu'on est "moins précis" que STREME/MEME et qu'on se "comporte moins bien" sur les scores hauts. Il faut dire plus précisément ce que ces différences impliquent.}
Both SEISM and MEME/STREME exploit a distribution of $k$-mers at the transcription factor binding site. MEME and STREME maximize the likelihood of a \textit{categorical model}, whereby the matrix $\bm z$ directly defines the probability to observe each letter at each of the $k$ sites:
\begin{equation} \label{eq:categorical}
\forall \big( \vu,~ \vz \in \gZ \big)\,,\quad \mathcal L_{\text{cat}}(\vu ; \vz) = \prod_{i=1}^k \vu_i ^T \vz_i
\end{equation}
SEISM on the other hand is based on a \textit{Gaussian model}. Through representation~\eqref{eq:Kdef}, $\vz$ is meant to maximize the Gaussian likelihood of a set of $k$-mers, \emph{i.e.}
\begin{equation}
\label{eq:gaussian}
\forall \big( \vu,~ \vz \in \gZ \big)\,,\quad \mathcal L_{\text{gaus}} (\vu ; \vz) = C \prod_{i=1}^k e^{-\frac{\lVert \vu_i - \vz_i \lVert ^2}{2\omega^2}}
\end{equation}
where $C$ is a constant such that the sum of probabilities over $\R^{4\times k}$ equals 1.
If we consider a binary~$\vy$ to match the setting of MEME/STREME, this set is made of one $k$-mer for each positive sequence. Importantly, the true TF motifs from~\citep{jolma_2013} that we use to assess selection accuracies are also defined through the maximum likelihood in a categorical model, which can explain why the $z$ obtained with MEME/STREME are closer to the true PWM than the one obtained with SEISM.
%A motif, in fact, is nothing more than a parametrization and representation of a distribution over the set of existent $k$-mers. However, there are other underlying models to establish this connection between a motif and a $k$-mer distribution.
%The standard model, which we will call here the \textit{categorical model}, defines the probability of a $k$-mer $\vu$ according to a motif $\vz$ as follows:
%\begin{equation} \label{eq:categorical}
% \forall \left( \vu,~ \vz \in \gZ \right) P(\vu \lvert \vz) = \prod_{i=1}^k \vu_i ^T \vz_i
%\end{equation}
% The resulting motifs must be understood in light of this paradigm since STREME and MEME employ it to determine whether a motif is present within a biological sequence. The multinomial approach used to compute the known TF motifs from \citep{jolma_2013} also completely matches this categorical model.
%On the other hand, SEISM uses a \textit{Gaussian model}, according to Equation \ref{eq:Kdef}. In this case, the $k$-mers distribution depends on a motif $\vz$ in a different way:
%\begin{equation}\label{eq:gaussian}
% \forall \left( \vu,~ \vz \in \gZ \right) P(\vu \lvert \vz) = C \prod_{i=1}^k e^{-\frac{\lVert \vu_i - \vz_i \lVert ^2}{2\omega^2}}
%\end{equation}
%where $C$ is a constant such that the sum of probabilities over the %entire set of $4^k$ $k$-mers equals 1.
% \lj{C'est encore trop allusif, et ça laisse trop de travail au lecteur. Il faut dire comment chaque méthode dépend de ces distributions, et ce que ça veut dire sur le z qu'elles sortent.}
% What matters then is identifying the appropriate $k$-mer distribution associated with the phenotype, and the motif is merely a tool for doing so. Depending on the model, the same $k$-mer distribution is represented by two slightly different sequence motifs.
We now illustrate on a simple example how the same distribution of $k$-mers is parameterized by different matrices under the two models. To build an easy example, we focus on $k$-mers of length 1, with
\begin{equation}\label{eq:true_distrib}
P(A)=0.3,~P(C)=0.4,~P(G)=0.1,~P(T)=0.2
\end{equation}
The matrix $\vz_1=(0.3, 0.4, 0.1, 0.2)^T$ used with the categorical model trivially constructs such a distribution. But using the same matrix in a Gaussian model with a parameter $\omega$ fixed as described in Appendix \ref{app:omega} leads to a slightly different distribution:
\begin{equation}
P(A)=0.28,~P(C)=0.43,~P(G)=0.11,~P(T)=0.18
\end{equation}
A distribution closer to Equation~\eqref{eq:true_distrib} can be constructed with a Gaussian model parameterized by $\vz_2=(0.315, 0.38, 0.08, 0.225)^T$.
% The true known TF motifs, as well as those discovered by STREME and MEME, use the categorical model, whereas the one optimised using SEISM must be interpreted using the Gaussian model, which explains its lower performance for higher TOMTOM scores.
%\lj{Ici on pourrait développer les implications pour les CNNs et notre approche (à quelques approximations près, z contient des log proba, et non pas directement des probas). Puis on enchaine sur la dernière remarque qui relativise la pertinence du modèle catégoriel.}\av{Il faut qu'on rediscute de ça demain, discuter de normaliser les colonnes de z, ce qui permet de passer aux produits scalaires et de faire le lien}
To clarify the relationships between those two motifs, we will rewrite \eqref{eq:categorical} considering $\vu$ is one hot encoded. That is, for each position $i$, it has only one $1$ for letter $j(i)$ and 0's elsewhere:
\begin{equation}
\label{eq:categorical_ohe}
\forall \big( \vu,~ \vz \in \gZ \big)\,,\quad \mathcal L_{\text{cat}}(\vu ; \vz) = \prod_{i=1}^k \vz_{i,j(i)}
\end{equation}
Assuming that the columns of $\vz$ are normalized and $\omega=1$, we can modify \eqref{eq:gaussian}:
\begin{equation}
\label{eq:gaussian_ohe}
\forall \big( \vu,~ \vz \in \gZ \big)\,,\quad \mathcal L_{\text{gaus}} (\vu ; \vz) = C \prod_{i=1}^k e^{-\frac{\lVert \vu_i - \vz_i \lVert ^2}{2\omega^2}} =C_2 \prod_{i=1}^k e^{\vu_i ^T \vz_i } = C_2 \prod_{i=1}^k e^{\vz_{i,j(i)} }
\end{equation}
With the Gaussian model and a few assumptions, the motifs can the be seen as defining the log probability to observe each letter at each of the $k$ sites. This gives us a new interpretation for the filters learned by CNNs and suggests that in this framework it might be interesting to constrain $e^{\vz}$ rather than $\vz$ to be in $\gZ$.
We used a Gaussian activation function since it is closer to typical CNNs approaches. Our framework is generic enough to allow other activation functions based on the categorical model, or more realistic variants~\citep{ruan_inherent_2017}.
%Finally, it should be emphasised that the non-linear relationship between binding affinity and binding probability has recently led to criticism of this categorical model for its inherent flaws \citep{ruan_inherent_2017}
%\av{END OF NEW VERSION / REMOVE WHAT COMES NEXT}
%Contrary to the traditional motifs learnt by STREME and MEME, the motifs learned by SEISM should not be employed in accordance with a categorical model and be understood as position probability matrices. \av{That is, the distribution with which the $k$-mers must be drawn is not the one defined by the following probabilistic model:
%\begin{equation}
% \forall \left( \vu \in k\text{-mer},~ \vz \in \gZ \right) P(\vu \lvert \vz) = \prod_{i=1}^k \vu_i ^T \vz_i
%\end{equation}
%}
%In fact, the SEISM motifs specify the centers of Gaussian distributions in a matrix space that contains the set of kmers. For instance, no $k$-mer created from a motif with a value $0$ for the letter A on position 1 will have an A on position 1 if the motif uses a categorical model \av{according to Equation \ref{eq:categorical}}. However, when the Gaussian model is applied to the same motif, the probability of drawing such a $k$-mer is no longer zero. The difference between the motifs detected by SEISM and those from STREME or MEME explains the Figure \ref{fig:selection_performances} (right panel). Although the motifs from SEISM are less accurate than those from STREME or MEME, this decline in performance for high Tomtom scores can be explained by the fact that the true TF motifs should be interpreted according to the categorical model, and not to the Gaussian one\lj{Il faut définir clairement ce qu'on veut dire par "interpreted according to the model" (idem plus haut avec "employed in accordance with")}\av{est-ce que l'ajout de l'équation 13 suffit ?}. Two identical $k$-mer distributions lead to two different motifs depending on the selected model. It should be emphasised that the non-linear relationship between binding affinity and binding probability has recently led to criticism of this categorical model for its inherent flaws \citep{ruan_inherent_2017}.
\subsection{Statistical validity and performances}
\label{sec:stat_validity_numerical}
In order to assess the statistical validity and of the SEISM procedure with the different strategies, we simulate datasets under the null hypothesis. To that end, we draw one sequence motif $\tilde{\vz}$ with length $k=8$ for each simulated dataset using a uniform distribution on $\gZ$ restricted to motifs with an information level fixed at $10$ bits. Then, we draw a set of $n=30$ biological sequences $X$ as follows: all sites are generated according to a uniform distribution over A, C, T, G for all sequences, and for half of the sequences one $k$-mer is drawn according to the categorical model parameterized by $\tilde{\vz}$. The phenotypes $\vy$ are drawn from $\mathcal N(\bm{0}, \sigma^2 \mC_n)$ to generate data under the null hypothesis for calibration experiments, and from $\mathcal N(\bm{\varphi}^{\tilde{\vz}, \mX}, \sigma^2 \mC_n)$ to generate data under the alternative for experiments on statistical power, with $\sigma = 0.1$ in both cases. We then run the SEISM procedure to select and test two sequence motifs. For both the data-split strategy and the hypersphere direction sampling one, the distribution from which the replicates are drawn uses the empirical variance from $\vy$ as variance parameter. Although any choice for this parameter leads to a valid procedure, as described in Section~\ref{sampling_null}, we make this choice for numerical stability considerations. For the data-split strategy, we sample 1000 replicates under the null hypothesis to compute the $p$-value. For SEISM, we sample $50,000$ replicates under the conditional null hypothesis using the hypersphere direction sampler, after $10,000$ burn-in iterations.
Figure \ref{fig:statistical_power} (top) shows the Q-Q plot of the distribution of quantiles of the uniform distribution against the $p$-values obtained across 1000 datasets under the null hypothesis for the data-split strategy and 100 datasets for the hypersphere direction sampling one. All the data points are well-aligned with the diagonal, which confirms the correct calibration of both the data-split and hypersphere direction sampling strategies, either considering the best motif or the center of the mesh and regardless of the size parameter.
Figure \ref{fig:statistical_power} (bottom) shows the same Q-Q plot on data generated under the alternative hypothesis. From this figure, we observe that on small datasets, the post-selection strategy is more powerful than the data-split one, regardless of the size of the mesh considered or the choice concerning the definition of the null hypothesis. The variance observed on the curves associated with the selective inference procedure is due to the presence of a weak residual signal after the first motif as a result of an imperfect selection step. Testing it with the best motif in the mesh captures this signal, resulting in curves under the diagonal. By contrast, focusing on the center of the meshes leads to testing motifs that do not capture this signal, placing us in the conservative situation, described at the end of Section~\ref{sampling_null_unknown_sigma}. The residual signal is not well explained by the mesh's centers, and thus its component on the orthogonal of the span of the activation vector of the second motif is important. The larger the mesh, the farther its center is to the selected motif and thus the less signal it captures, which explains the differences between the two curves.
\begin{figure}[h!]
\centering
\includegraphics[scale=0.85]{figures/inference_tmlr.eps}
\caption{Q-Q plots obtained by applying data-split and different hit-and-run sampling strategies to select two motifs and test their association with an outcome. \textbf{Top}: data simulated under the null hypothesis. The proximity between the quantiles of the obtained p-values and those of the uniform distribution confirms that all SEISM strategies presented in this article are correctly calibrated. \textbf{Bottom}: data simulated under an alternative hypothesis, where the outcome depends on the activation $\bm{\varphi}^{\tilde{\vz, \bm X}}$ of a single motif in the sequence. The distributions of the p-values computed with the post-selection inference (PSI) strategies have a larger deviation to the uniform distribution than the distributions of the p-values computed with the data-split strategy (purple).}
\label{fig:statistical_power}
\end{figure}
\subsection{Computation costs}\label{sec:comp_cost}
The section serves as an overview of how various user-specified parameters impact the computation time required by the post-selection inference procedure.
As discussed in \ref{sampling_null}, the hit-and-run algorithm is actually a rejection sampler. Its overall computation cost depends mainly on two characteristics: the cost of the selection step, that is the cost of selecting $q$ motifs for a given phenotype $\vy$, and the acceptance rate. Although some parameters affect the selection cost, the acceptance rate is primarily responsible for determining if a user-specified combination of parameters results in a tractable configuration for the post-selection method in a reasonable amount of time. This rate is high compared with a naive rejection sampler over $\gE$, as the hit-and-run strategy reduces the dimension over which the rejection step is performed: from $n$ with a naive sampler to $1$. Nonetheless some parameters may have a major impact on the rejection rate. To clarify it, we studied in Figure \ref{fig:computational_costs} the impact of several user-specified parameters --- the number of motifs to be discovered, the precision of the meshes, the regularization parameter of the ridge score and the number of computation cores allowed during the rejection step of the hit-and-run sampler.
%Since it varies on a number of variables, such as the signal intensity in the data or the CPU performances, the absolute time is not really the output of interest\lj{C'est quand même important de donner un ordre de grandeur non?}\av{Oui et non, ça dépend vraiment d'énormément de paramètres, dont certains ne sont pas inclus ici. Par exemple, dans des données avec beaucoup de signal, la proba de rejet diminue énormément, et donc le temps de calcul aussi, ça dépent de la taille des motifs (et du nombre de tailles différentes autorisées), de la machine sur laquelle on fait tourner tout ça...}. Instead, we will concentrate on how this time changes throughout the range of a given parameter.
Although the number of motifs to be found by SEISM undoubtedly affects the selection cost, we can roughly consider that this relationship is linear. The upper left figure in Figure \ref{fig:computational_costs}, however, demonstrates that the influence on the overall computing cost is superlinear, in line with the exponential growth of the number of distinct selection events one may describe with a fixed mesh size. As a result, the post-selection process quickly becomes intractable for discovering and test more than a few motifs.
%The conclusion is almost similar when it comes to mesh precision.
We make a similar observation for mesh precision:
computation time grows exponentially with the number of bins used to define the meshing. This can be explained by the exponential relationship between the number of bins and the number of different meshes (and thus the rejection rate). Of note, mesh precision has no impact on the selection time, and therefore the computation time is entirely explained by the acceptance rate.
%The relationship that seems to be exponential between the computation time and the number of bins used to define the meshing may be explained by the fact that the number of different meshes (and thus the rejection rate) increases exponentially with the number of bins. It is worth noting here that this parameter has no impact on the selection time, and therefore the computation time reflects perfectly the acceptance rate.
We observe that the greater the regularization parameter $\lambda$, the lower the computation time. This can be explained by detailing its impact on the rejection rate. To understand it, it is necessary to note that the motifs are not selected over $\gZ$, but over a less constrained set as described in \ref{subsec:selection}. They are only projected onto $\gZ$ at the end of the whole procedure, to ease their interpretation. The meshes are then defined over a vectorial space, leading to an infinite number of meshes. Compared to a small regularization parameter, a higher $\lambda$ favors motifs resulting in a $\bm{\varphi^{z,X}} $ with a higher norm. With regard to the activation function, such motifs are located closer from the $k$-mers, and thus from $\gZ$. $\lambda$ has then no effect on the number of existing meshes, but impacts the number of \textit{acceptable} ones, in the sense that they have a reasonable probability to be selected. A lower $\lambda$ leads to better selection performances, but to a higher number of acceptable meshes, and thus to a lower acceptance rate. We empirically set $\lambda=0.01$ to provide a good trade-off.
%\lj{Ici je propose de d'abord décrire ce qu'on observe puis l'analyser sinon c'est difficile de faire la part des choses.}
%The impact of the regularization parameter $\lambda$ is a bit trickier to explain. Its impact on the selection time is very limited\lj{constat empirique? Ou c'est ce à quoi on s'attend?}, and the one it has on the rejection rate is less straightforward. To understand it, it is necessary to note that the motifs are not selected over $\gZ$, but over a less constrained set as described in \ref{subsec:selection}. They are only projected onto $\gZ$ at the end of the whole procedure, to ease the interpretation of the motifs. The meshes are then not defined over $\gZ$, but over the vectorial space in which the positivity constraint is removed, which leads to a discrete but infinite number of meshes. A small regularization parameter has the effect of bringing the optimal motifs further to $\gZ$, as the selection method will try to find the $\vz$ leading to the most aligned possible activation vector $\bm{\varphi^{z,X}}$ with $\vy$, regardless to its norm. On the contrary, a high regularization coefficient will favor motifs resulting in a $\bm{\varphi^{z,X}} $ with a high norm. With regard to the activation function, such motifs can not be located far away from the $k$-mers, and thus from $\gZ$. To sum up, $\lambda$ has no impact on the number of existing meshes, but has an impact on the number of \textit{acceptable} ones, in the sense that they have a reasonable probability to be selected. A lower $\lambda$ leads to better selection performances, but to a higher number acceptable meshes, and thus to a lower acceptance rate. We empirically found $\lambda=0.01$ to provide a good trade-off.
Finally, the rejection sampling step can be parallelized over several computation cores, which accelerates the whole procedure, as described in Section \ref{sec:sampler}. As long as the acceptance rate is small enough, using $j$ cores to parallelize the rejection step should roughly divide the computation time by $j$.
We can clearly identify limitations inherent to the use of the selective inference procedure. Although it is more powerful than the data-split approach, it can not be used in every situation. This latter approach does indeed not include any rejection step, and the only factor influencing its overall computation time is the selection time, only marginally influenced by the aforementioned parameters.
%Therefore, these two procedures are complementary.\lj{Ce paragraphe ne discute pas vraiment en quoi elles sont complémentaires}
\begin{figure}
\centering
\begin{tabular}{c c}
\includegraphics[scale=0.45]{figures/nb_motifs.eps} & \includegraphics[scale=0.45]{figures/mesh.eps}\\
\includegraphics[scale=0.45]{figures/lambda.eps} & \includegraphics[scale=0.45]{figures/threads_2.eps}
\end{tabular}
\caption{Impact of different parameters on the computation time for 100 replicates for the post-selection inference procedure. \textbf{Upper left:} Impact of the number of motifs to be discovered \textbf{Upper right:} Impact of the number of bins defining the meshes. \textbf{Bottom left:} Impact of the ridge regularization parameter. \textbf{Bottom right:} (Log scale) Impact of the number of threads over which the hit-and-run sampling is parallelized. }
\label{fig:computational_costs}
\end{figure}
\section{Discussion and future works}
%\av{rajouter discussion sur comment on gère le sigma}
%This research makes a connection between two scientific fields that are still rather isolated from one another. On the one hand, the Genome Wide Association Studies aim at identifying genetic variants associated with a phenotype. They do this by using a variant list to be tested, and finding new variant classes may lead to further significant improvements. However, using very rich classes, like PWMs, which represent an infinite set, necessitates a non-trivial selection step in order to test only a sufficient number of pertinent variants. On the other hand, Neural Networks have been successfully applied to predict phenotypes from biological sequences. Since they build complex models with a very large number of parameters, they have been blamed for their black box aspect. Recent advances in interpretable deep learning, however, have been applied to networks for biological sequences, enabling the description of those trained networks in terms of genetic variants. Therefore, they provide new variant classes, which are moreover relevant since the networks achieve good prediction performances. But to our knowledge, the neural networks literature hasn't gone beyond this descriptive step, and has not developed yet tools to measure the significance of the association between the variants and the phenotype.
%We therefore propose a framework to make the variants selected by neural networks amenable to inference, especially in the context of small datasets which are particularly challenging for GWAS,and provide a proof of concept considering the one-layer CNNs and the filters they learn, which can be interpreted as sequence motif. In addition to being able to test each filter, SEISM works on the architecture of such networks, by coming up with a way to select only the filters (trained with adaptive lengths) that are significantly associated with the phenotype, although an \textit{a-priory} number of motifs must be given in the selection step.
% There is still some limitations to apply SEISM in a GWAS context. Those studies indeed use whole genomes as input data, whereas SEISM only deals with small sequences, up to a few hundred base pairs. This is due to computational issues, but also to the fact that sequence motifs are not expressive enough to represent variations at the scale of the genome, which suggests the use of richer class of variants, that may be defined by multi-layer CNNs or by other neural networks suited for predictions using whole genomes.
We have introduced a procedure to test the association between
features learned by a neural network and the outcome predicted by this
network. We did so by relying on the post-selection inference
framework and formalizing the network training as a feature selection
step. Along the way, we addressed general problems related to selective inference over
composite hypotheses, which has implications beyond testing of
features extracted by trained neural networks. In particular to our
knowledge, all previous procedures had to work under the assumption
that the variance was known. Our strategy to normalize the statistic
to make it scale-free could easily be transferred to kernelPSI for
testing the association of kernels with a trait, or to previous selective
inference frameworks for testing groups of
variables using sampling strategies~\citep{slim2019kernelpsi, reid2018general}.
Through the SEISM procedure, we are also drawing connections between
neural networks for biological sequences and two related fields:
sequence motif detection, and GWAS.
Sequence motif detection has been a major theme in bioinformatics for
the past 30 years and many methods have been proposed to identify
motifs that are over-represented in a set of sequence compared to some
control class or background distribution. The earliest CNNs for
regulatory genomics~\cite{Alipanahi2015Predicting,zhou2015predicting} already exploited the fact
that trained convolution filters of the first layer could be
interpreted as PWMs, and more recent work have sought to extract PWMs
from entire multi-layer trained networks through attribution methods.
The selection step of our procedure merely formalizes that
training a one-layer CNN is equivalent to selecting a finite set of
PWMs that have a maximal association to the outcome for some
particular score. This formalization also highlights the specific way
by which CNNs with exponential activation functions parameterize the
distribution of k-mers at a binding site. Although the PWM returned by
most bioinformatics models represents a categorical
distribution---probability to draw each letter at each site, trained
convolution matrices parameterize a Gaussian distribution. In
practice, this difference leads to discrepancies between the trained
convolution filters and PWMs learned using categorical
likelihoods---including those offered by databases and often used as
ground truth. This observation also suggests alternative sets of
constraints for convolution filters---\emph{e.g.}, each column of the
pointwise exponential of the filter should belong to the simplex.
By providing an inference procedure for features extracted by the
trained model, our work also connects neural networks for genomic
sequences to GWAS. The good predictive performances of these neural
networks is often explained by their ability to jointly learn an
appropriate data representation and a regular prediction function
acting on this representation. Nonetheless, the space from which these
representations are learned is seldom formalized and to our knowledge
the association of the extracted features with the predicted outcome
is never tested. GWAS on the other hand relies on hypothesis testing,
but commonly relies on relatively simple genomic variants such as
single nucleotide polymorphisms (SNPs) or $k$-mer
presence~\citep{jaillard2018fast, roux2022caldera}. Our framework paves the way to GWAS over
richer sets of variants, \emph{e.g.} capturing the presence of entire
polymorphic genes through large convolution filters, or the
interaction of simpler variants through multilayer or self-attention
networks~\citep{avsec2021effective}. This will require scaling to entire genomes as inputs, and making more complex networks, such as multi-layer CNNs and networks using attention mechanisms, amenable to inference. The most important step in achieving this goal is to formulate the training of these networks as a feature selection problem and formalize the association between these features and the phenotype. The inference framework might then be directly derived from this present work.
For instance, we may test motif interactions derived from convolutional-attention networks \citep{ullah_self-attention_2021}, a (motif, position) couple as selected in \citep{ditz_convolutional_2022} or motifs extracted by TF-MoDISco \citep{shrikumar2018technical}. For this latter case, we can note that the inference procedure we describe is completely independent from the selection method, and we can then apply directly this procedure to TF-MoDISco's motifs. More relevant association scores than, \emph{e.g.}, $s^{\text{ridge}}$ could be devised for such motifs, but the procedure is nonetheless valid. For other features, the definition of a relevant association score, which meets the assumptions described in Section \ref{sec:model}, is needed. A few practical problems may arise. First, the hit-and-run sampler requires the selection method to be stable, that is, running the selection method twice on the same input will lead to the same selection on features. This property is required to guarantee the theoretical convergence of the the algorithm but may not be necessary in practice. Second, some attention may be required to avoid the that computational cost become prohibitive, in particular depending on the regularity properties of the selection event leading to a higher rejection probability or to a higher number of replicates required. Granted that these technical challenges can be addressed, we are confident that extending SEISM to more general networks and corresponding features will benefit both the fields currently using these networks---such as regulatory genomics---and GWAS.
% Finally, the usage of the post-selection procedure leads to significant improvements in the context of small datasets, but is computationally expensive. The time required to optimize the motifs on a single outcome vector $\vy$, the number of replicates required to produce a decent estimate of the theoretical distribution and the rejection probability, which is involved in line \ref{alg:rejection} of Algorithm \ref{alg:hr_sampler}, all have a major role in the calculation time.
%The optimization time has already been the subject of significant investigation. The post-selection procedure makes the assumption (needed in the hit-and-run sampler) that running the selection step on the same data will result in the selection of the exact same set of motifs, so it is important to note that stochastic gradient descent methods cannot be used when using this procedure\lj{On est sûrs de ça?}\av{Je crois, c'est pour garantir que l'étape de rejet ne va pas tourner indéfiniment !}. Based on this observation, we implemented a full gradient descent approach along with a line search strategy and a best $k$-mer initialization described in \ref{subsec:selection}.
%\av{NEW: On the other hand, the rejection probability mainly depends on the volume of the selection event $E$: the larger $E$, the lower the rejection probability. While we do not have any closed form for the borders, its volume depends on the number of different meshes defined during the discretization step (Section \ref{sec:mesh}). As the meshes can be seen as hypercubes, we can specify their geometry using a single parameter. For the selection of $q$ motifs, with maximum length $k$ and meshes with edge length $\frac{1}{2^m}$, there exists roughly $2^{m \times k \times q}$ different meshes (and thus different selection event). As a first approximation, the rejection probability increases rapidly with the number of motifs, their length and the geometrical parameter used to define the meshes.}
%\av{OLD: The relative size of the selection event is the only factor that affects the rejection probability for the first level. The mesh geometrical hyperparameter supplied by the user governing the accuracy of the null hypothesis, as explained in \ref{sec:mesh}, then plainly has an impact on it. However, this probability is also influenced by the number of motifs chosen and their lengths because more and longer motifs allow for the description of a wider range of selection events.}\lj{Il faudrait rendre ces trois dernières phrase plus précises.} This computational overhead is tackled using parallel computing. The $While$ loop in line \ref{alg:rejection} is run on several cores until one of them discovers a replicates in the selection event.\lj{la parallelisation devrait aller en méthodes}\av{Passé directement dans l'algorithme du H&R sampler}
%An adequate estimate of the theoretical distribution requires a large number of replicates and burn-in iterations. This number depends on the dimension of the selection event (hence on the size of the dataset). This issue is inherent to the hit-and-run sampling strategy. The post-selection process cannot be used with datasets that have more than a few hundred samples because of this problem, which is a drawback of the hit-and-run sampling strategy.
%Here, the complementary nature of the data-split and post-selection procedures is put to use. In the setting of small datasets, data-split strategies are much less powerful than post-selection techniques, despite the fact that they can test a set of pointwise motifs with relatively modest computational costs. The precision of the null hypothesis as well as a higher computing cost are then paid for the increase in power brought about by the post-selection. The interest in adopting post-selection procedures in comparison to data-split ones becomes null as the size of the dataset increases, and this difference in power \av{should drop} dramatically.\lj{On n'illustre pas ce point dans nos expériences si?} \av{non, c'est vrai qu'on ne peut pas l'affirmer. Mais avec should ça serait OK ?}
\iffalse
\begin{itemize}
\item Ideally, construct a network whose parameters correspond to the
features of interest, rather than a two-step approach where a black
box is trained and then hacks need to be used to extract something
meaningful.
\item Missing steps for GWAS (scalability)?
\item High memory consumption when increasing the number/length of sequences - how can we scale to long sequences?
\item Scaling the number of burn-in/replicates with the number of samples?
\item At which threshold does datasplitting become better that PSI?
\item What about other selection methods (transformers), can they fit into our framework?
\item Mesh size impact on power
\end{itemize}
\fi
%\section{Conclusion}
%\lj{Ça semble assez redondant avec la discussion, peut être fusionner (supprimer la section conclusion et n'en garder que le paragraphe future direction, qui concluerait la discussion)?}
%We introduced a framework to make interpretable features learned through neural networks amenable to inference. As a proof of concept, we used one-layer CNNs trained on biological sequences, whose trained filters may be directly interpreted as sequence motifs.
%To that end, we first cast those networks into a \textit{de novo} motifs discovery problem and show that by adjusting the optimization method, they achieve similar results as state-of-the-art tools from the bioinformatics literature.
%We then specify a model and a set of null hypotheses to test the significance of the trained filters. We circumvent the composite null hypothesis and demonstrate theoretical findings on the null distribution of the test statistics in order to make this model easier to use on real data. Although this test can be carried-out using conventional data-split strategies, we take advantage of recent advancements in post-selection inference to devise a more powerful procedure, especially for small datasets.
%Future directions for our research include making more complex networks, such as multi-layer CNNs and networks using attention mechanisms, amenable to inference. The most important step in achieving this goal is to formulate the training of these networks as a feature selection problem and formalize the association between these features and the phenotype. The inference framework might then be directly derived from this present work.
%\av{For instance, we may test motif interactions derived from convolutional-attention networks \citep{ullah_self-attention_2021}, a (motif, position) couple as selected in \citep{ditz_convolutional_2022} or motifs extracted by TF-MoDISco \citep{shrikumar2018technical}. For this latter case, we can note that the inference procedure we describe is completely independent from the selection method, and we can then apply directly this procedure to TF-MoDISco's motifs. The question then arises as to whether the association being tested---in the sense of $s^{\text{ridge}}$ for instance---is relevant for such motifs, but the procedure is nonetheless valid. For other features, the definition of a relevant association score, which meets the assumptions described in Section \ref{sec:model}, is needed. A few practical problems may arise. First, the hit-and-run sampler requires the selection method to be stable, that is, running the selection method twice on the same input will lead to the same selection on features. This property is required to guarantee the theoretical convergence of the the algorithm but may not be necessary in practice. Second, the computational cost may become prohibitive, in particular depending on the regularity properties of the selection event leading to a higher rejection probability or to a higher number of replicates required.}
%\lj{Les points que je n'ai pas intégrés: la gestion du nombre de
%filtres, la désintégration de la mesure, le lien avec les noyaux.}
%\vfill
\pagebreak
\section{Acknowledgements}
This work has been supported by ANR grants (FAST-BIG project ANR-17-CE23-0011-01 and PIECES project ANR-20-CE45-0017) and was performed using the computation facilities of the LBBE/PRABI.
We thank François Gindraud, Jean-Philippe Rasigade, Lotfi Slim and Dexiong Chen for the insightful discussions and support.
\bibliography{tmlr}
\bibliographystyle{tmlr}
\vfill
%%%%%%%%%% Merge with supplemental materials %%%%%%%%%%
\pagebreak
%\widetext
\begin{center}
\textbf{\large Supplemental Materials of ‘‘Neural Networks beyond explainability: Selective inference for sequence motifs"}
\end{center}
%%%%%%%%%% Merge with supplemental materials %%%%%%%%%%
%%%%%%%%%% Prefix a "S" to all equations, figures, tables and reset the counter %%%%%%%%%%
\setcounter{equation}{0}
\setcounter{figure}{0}
\setcounter{table}{0}
\setcounter{page}{1}
\makeatletter
\renewcommand{\theequation}{S\arabic{equation}}
\renewcommand{\thefigure}{S\arabic{figure}}
\renewcommand{\bibnumfmt}[1]{[S#1]}
\renewcommand{\citenumfont}[1]{S#1}
%%%%%%%%%% Prefix a "S" to all equations, figures, tables and reset the counter %%%%%%%%%%
\appendix
%\section{Appendix}
\vspace*{2.5cm}
\section{Tuning the activation bandwidth hyperparameter}\label{app:omega}
The data representation $\bm{\varphi}^{\bm{Z}, \bm{X}}$ depends on a hyperparameter $\omega$ controlling the bandwidth of the gaussian non-linearity (Equation \ref{eq:Kdef}): $\exp\left(-\frac{||\vz_j-\vu||^2}{2 \omega^2} \right)$. Assuming that the positions are independant, we know that the expected value of the distance between a motif $\vz$ and a $k$-mer $u$ with length $k$ is proportional to $k$.
In order to get an activation that does not depend on the length of the motifs, we simply set $\omega$ to be proportional to $\sqrt{k}$. From empirical tests, we set $\omega = \frac{\sqrt{0.9*k}}{2}$ to achieve good selection results by choosing the motif that maximizes the association score among a set of possible lengths.
\section{Disintegration of the selection event given by sequence motifs}\label{app:null_set}
In this section we consider the selection event:
\begin{equation}
\label{seq:selection_event}
\displaystyle E_{\mathrm{cont.}}(\mZ) := \Big\{ \vy' \in \mathcal E,~ \forall i \in \{1,\dots,q\} ~\argmax_{\vz \in \gZ}s(\vz, \mP_i \vy')=\vz_i \Big\}\,,
\end{equation}
given by the sequence of selected motifs $\mZ=(\vz_1,\ldots,\vz_q)$. We denote by $\mu$ the law of $\vy$ as given by \Eqref{eq:model}, a Gaussian distribution on $\mathcal E$.
\subsection*{A first remark on the uniqueness of the selection}
Consider the mapping $\pi\,:\mathcal E\to \mathcal Z^q$ given by $\pi(\vy')=\mZ$ where $\mZ=(\vz_1,\ldots,\vz_q)$ is the sequence of motifs such that $\vy'\in E_{\mathrm{cont.}}(\mZ)$. It is not clear that $\pi$ is well defined as a same $\vy'$ may lead to the selection of at least two different motifs sequences $\mZ$ and $\mZ'$. As a first remark, we can see that the set of problematic $\vy'$ is exactly
\[
\mathcal P:=
\bigcup_{\mZ\neq \mZ'}E_{\mathrm{cont.}}(\mZ)\cap E_{\mathrm{cont.}}(\mZ')\,.
\]
When one assumes that $\mZ=(\vz_1,\ldots,\vz_q)$ is unique, one implicitly assumes that $\mu(\mathcal P)=0$. For sufficiently regular scores, this is however the case. For sake of readability, we will not comprehensively study this issue but we will present an argument for the scores $s^{\text{HSIC}}$ and $s^{\text{ridge}}$. In this case, we can circumvent this difficulty considering the Gaussian random field
\[
\vz\mapsto\langle \bm{\varphi}^{\vz, \mX},\vy\rangle\text{ for (HSIC)}
\quad
\text{and}
\quad
\vz\mapsto\langle \big(\|\bm{\varphi}^{\vz, \mX}\|^2 + \lambda n \big)^{-1/2}\bm{\varphi}^{\vz, \mX},\vy\rangle\text{ for (Ridge)}
\]
indexed by $\mathcal Z$ where $\vy$ is distributed with respect to a multivariate Gaussian distribution \Eqref{eq:model}. Its autocovariance function is given by $(\vz,\vz')\mapsto \sigma^2\langle \bm{\varphi}^{\vz, \mX},\bm{\varphi}^{\vz', \mX}\rangle$ from \Eqref{eq:model} (one has to multiply by $\big(\|\bm{\varphi}^{\vz, \mX}\|^2 + \lambda n \big)^{-1/2}\big(\|\bm{\varphi}^{\vz', \mX}\|^2 + \lambda n \big)^{-1/2}$ for the Ridge). The score is just the largest norm of this Gaussian random field% %(note that it is enough to consider the numerator of the aforementioned scores when dealing with the selection event, since the denominator does not depend on $\vz$) JE ME SUIS FOURVOYE sur le SCORE ;) pour HSIC on ne divise pas par y.
. It is well established in theory of Gaussian random fields that the law of this maximum is regular and the argument maximum is unique. The interested reader may consult the pioneering work of Tsirelson \citep{tsirel1976density} and Lifshits \citep{lifshits1983absolute}. In Tsirelson’s theorem, the parameter set is countable. This says that the same result holds true for separable bounded Gaussian processes, since in this case, the distribution of the supremum coincides a.s. with the one of the supremum on some countable nonrandom set. To avoid a cumbersome presentation, we will assume that almost surely the selected sequence motifs $\mZ=(\vz_1,\ldots,\vz_q)$ is uniquely defined, hence $\pi$ is well defined.
\subsection*{The disintegration steps}
To sample conditionally on \eqref{seq:selection_event}, one need to consider the conditional law with respect to this event. We will denote this law by $\mu_\mZ$, it depends only on $\mu$, $\mZ$ and $\pi$. This law is described by the theorem of disintegration, see for instance \cite[Theorem 5.3.1]{ambrosio2005gradient}. Denote $\nu$ the pushforward measure of $\mu$ by $\pi$, denoted by $\nu=\pi_\#\mu$, a probability measure on the set $\mathcal Z^q$ of $\mZ$. By the disintegration theorem, there exists a $\nu$-almost everywhere uniquely determined Borel family of probability measures $\mu_\mZ$ (the though-after conditional distributions) such that
\begin{itemize}
\item {\bf Supported by $E_{\mathrm{cont.}}(\mZ)$: } $\mu_\mZ\big\{\mathcal E\setminus \pi^{-1}(\mZ)\big\}=0$ for $\nu$-almost every $\mZ$;
\item {\bf Expectation of the conditional expectation is the expectation: }It holds that, for every Borel test map $f:\mathcal E\to[0,+\infty]$,
\begin{equation}
\label{seq:conditional_to_marginal}
\int_{\mathcal E}f\mathrm{d}\mu
=\int_{\mathcal Z^q}
\Big(
\int_{\pi^{-1}(\mZ)}f\mathrm{d}\mu_{\mZ}
\Big)\mathrm{d}\nu(\mZ)
\,,
\end{equation}
\end{itemize}
where one can remark that $\pi^{-1}(\mZ)=E_{\mathrm{cont.}}(\mZ)$ by definition of $\pi$. Let us comment on this result regarding our purposes. First, we have mentioned that we known that the support $E_{\mathrm{cont.}}(\mZ)$ is included in some subspace, say $\mathcal S$, defined by the first order conditions. Second, although one can use a rejection sampling strategy on the subspace $\mathcal S$ to draw points on the support $E_{\mathrm{cont.}}(\mZ)$ (viewed as a subset of the same Hausdorff dimension as the subspace $\mathcal S$), it is not clear at all what should be the density of $\mu_\mZ$. Indeed, the family of probability measures $\mu_\mZ$ is the unique family that satisfies \Eqref{seq:conditional_to_marginal}. It implies that a measure~$\mu_\mZ$ depends on the others measures $\mu_{\mZ'}$ and this dependency is geometrically given by the (piece-wise) topological sub-manifold given by the function $\vz\mapsto \bm{\varphi}^{\vz, \mX}$ from $\mathcal Z$ to $\mathcal E$.
From a practical view point, we tried various law for $\mu_\mZ$ such as the uniform, or a rejection sampling based on the Gaussian distribution \eqref{eq:model}, but none of them matched the condition \eqref{seq:conditional_to_marginal}. In the next subsection, we recall a toy example: the disintegration of the uniform measure on the sphere is not the uniform measure. Even in this simple geometrical example, the calculus of the conditional law might be seen as tedious. We believe that the calculus of $\mu_\mZ$ is somehow out of reach for our purposes and our analysis with selection events defined by meshes more suited.
\subsection*{A toy example on the sphere}
Let $\mathds S$ be the $2$-sphere embedded in the $3$-Euclidean space. Let $\mu$ be the uniform measure on the sphere~$\mathds S$. Let $\{\mathcal S_\theta\,:\,\theta\in[0,\pi)\}$ be a family of sub-spaces of co-dimension $1$ (hyper-planes) sharing $\mathrm{Span}\{(0,0,1)\}$ (say the north pole) as a revolution axis parameterized by $\theta$. The parameter $\theta$ can be interpreted as the longitude.
\begin{figure}[ht!]
\centering
\vspace*{-0.5cm}
\includegraphics[scale=0.45]{figures/sphere_new_2.eps}
\vspace*{-1cm}
\caption{For $\theta=0$, $S_\theta$ is the light red plan, the conditional measure $\mathrm d\mu_0(\phi)$ is depicted with a red area and is proportional to $|\sin(\phi)|$, which is not the uniform measure. }
\label{fig:toy_sphere}
\end{figure}
\noindent
Let~$\bar\pi$ be the function that maps a point to its longitude modulo $\pi$. By spherical symmetries, the pushforward measure $\nu=\bar\pi_\#\mu$ is the uniform measure on $[0,\pi)$, so that $\mathrm{d}\nu(\theta)=(1/\pi)\mathrm d\theta$. Condition \eqref{seq:conditional_to_marginal} (the lhs of the equality below) is given by the coordinate integration system (the rhs) in:
\[
\int_{\mathds S}f\mathrm{d}\mu
=\int_{0}^{\pi}
\Big(
\int_{\bar\pi^{-1}(\theta)}f\mathrm{d}\mu_{\theta}
\Big)\mathrm{d}\nu(\theta)
=\int_{0}^{\pi}
\Big(
\int_{0}^{2\pi}f(\theta,\phi)\frac{|\sin\phi|}{4\pi}\mathrm{d}\phi
\Big)\mathrm{d}\theta\,,
\]
where $\phi$ is the latitude. Note that $\bar\pi^{-1}(\theta)=\mathds S\cap \mathcal S_\theta$ and it is in bijection with $[0,2\pi)$ using the mapping that to a point maps its latitude. Using this representation, it is not hard to see that the uniform measure on~$\bar\pi^{-1}(\theta)$ is given by $(1/2\pi)\1_{[0,2\pi)}(\phi)$ while the above equality shows that the conditional measure $\mu_{\theta}$ of the uniform measure on the sphere has density $(1/4)|\sin\phi|\1_{[0,2\pi)}(\phi)$, see Figure~\ref{fig:toy_sphere}. It proves that the disintegration of the uniform measure on the sphere is not the uniform measure, but rather a distribution that will put few mass around the poles and large mass around the equator.
\section{Proof of Proposition~\ref{prop:composite_hypothese}}
\label{proof:composite_hypothese}
Consider the orthogonal decompositon
\[
\mathcal E=\mathcal R\oplus\mathcal \mathcal S\oplus \mathcal T
\]
where $\mathcal R$ is the span of $\bm{\varphi}^{\mZ,\mX}$, $\mathcal T$ is the span of $\bm{\mu}$ (orthogonal to $\mathcal R$ by Proposition~\ref{prop:orhtogonal_decomposition}), and $\mathcal S$ such that the equality holds. Consider $\vy\in\mathcal E$ and its othogonal decomposition $\vy=\vr+\vs+t\ve$ where $\ve=\bm \mu/\|\bm \mu\|_2$ is a unit norm vector that spans $\mathcal T$. Let $\tau>0$ and note that it is enough to prove that
\[
\mathds P_{\bm\mu}
\Big\{
\frac{s(\mZ,\bm Y)}{\|\bm Y\|^2}\leq\tau
\Big\}
\geq
\mathds P_{\bm 0}
\Big\{
\frac{s(\mZ,\bm Y)}{\|\bm Y\|^2}\leq\tau
\Big\}\,,
\]
where $\bm Y$ is a random variable with the same distribution as $\bm \mu+\sigma \bm \epsilon$ (resp. $\sigma \bm \epsilon$) on the probability space defined by $\mathds P_{\bm \mu}$ (resp. $\mathds P_{\bm 0}$). Not that the event decomposed as
\[
\Big\{\vy\,:\,
\frac{s(\mZ,\vy)}{\|\vy\|^2}\leq\tau
\Big\}
=
\Big\{(t,\vr,\vs)\,:\,
{s(\mZ,\vr)}\leq \tau(t^2\|\bm{\mu}\|^2+{\|\vr\|^2+\|\vs\|^2})
\Big\}
\]
By othogonality, note that $\mathcal L_{\bm \mu}(\vr,\vs)=\mathcal L_0(\vr,\vs)$ and this law is a centered Gaussian multivariate law. We deduce that the aforementioned probabilities are of the form
\[
\mathds P_\mu
\Big\{
\frac{s(\mZ,Y)}{\|Y\|^2}\leq\tau
\Big\}
=\int_0^\infty w_0(t)\varphi_\mu(t) \mathrm dt
\]
where
\begin{align*}
w_0(t)
&=\mathds P_0
\Big\{
{s(\mZ,\vr)}\leq \tau(t^2\|\bm{\mu}\|^2+{\|\vr\|^2+\|\vs\|^2})
\Big\}\\
\varphi_\mu(t)
&=
\exp\big({-(t-\mu_\ve)^2}/2\big)+\exp\big({-(t+\mu_\ve)^2}/2\big)
\end{align*}
with $\mu_\ve=\langle\ve,\bm\mu\rangle=\|\bm\mu\|_2$. Note that $w_0:\,\mathds (0,\infty)\to(0,1)$ is an increasing continuous function. It is an increasing homeomorphism and the Fubini's equality yields
\begin{align*}
\mathds P_\mu
\Big\{
\frac{s(\mZ,Y)}{\|Y\|^2}\leq\tau
\Big\}
&=\int_0^\infty w_0(t)\varphi_\mu(t) \mathrm dt\\
&=\int_0^\infty \int_0^1\bm{1}_{\{u\leq w_0(t)\}}\mathrm du\,\varphi_\mu(t)
\mathrm dt\\
&=\int_0^\infty \int_0^1\bm{1}_{\{w_0^{-1}(u)\leq t\}}\mathrm du\,\varphi_\mu(t)\mathrm dt\\
&=\int_0^1\int_{w_0^{-1}(u)}^\infty\varphi_\mu(t)\mathrm dt\mathrm du\\
\end{align*}
By Anderson's theorem, the measure of the interval $[-w_0^{-1}(u),w_0^{-1}(u)]$ for the centered Gaussian density is greater than the one for a non-centered Gaussian density with the same variance. As a result, we deduce that
\[
\int_{w_0^{-1}(u)}^\infty\varphi_\mu(t)\mathrm dt\geq
\int_{w_0^{-1}(u)}^\infty\varphi_0(t)\mathrm dt\,,
\]
which achieves the proof.
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsthm,amsfonts,bm}
\newtheorem{proposition}{Proposition}[section]
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
%\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Eq.~(\ref{#1})}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\varepsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathds{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathds{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\Span}{\mathrm{Span}}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{46}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Alipanahi et~al.(2015)Alipanahi, Delong, Weirauch, and
Frey]{Alipanahi2015Predicting}
Babak Alipanahi, Andrew Delong, Matthew~T. Weirauch, and Brendan~J. Frey.
\newblock {Predicting the sequence specificities of DNA- and RNA-binding
proteins by deep learning}.
\newblock \emph{Nature Biotechnology}, 33\penalty0 (8):\penalty0 831--838, July
2015.
\newblock ISSN 1087-0156.
\newblock \doi{10.1038/nbt.3300}.
\newblock URL \url{http://dx.doi.org/10.1038/nbt.3300}.
\bibitem[Ambrosio et~al.(2005)Ambrosio, Gigli, and
Savar{\'e}]{ambrosio2005gradient}
Luigi Ambrosio, Nicola Gigli, and Giuseppe Savar{\'e}.
\newblock \emph{Gradient flows: in metric spaces and in the space of
probability measures}.
\newblock Springer Science \& Business Media, 2005.
\bibitem[Avsec et~al.(2021{\natexlab{a}})Avsec, Agarwal, Visentin, Ledsam,
Grabska-Barwinska, Taylor, Assael, Jumper, Kohli, and
Kelley]{avsec2021effective}
{\v{Z}}iga Avsec, Vikram Agarwal, Daniel Visentin, Joseph~R. Ledsam, Agnieszka
Grabska-Barwinska, Kyle~R. Taylor, Yannis Assael, John Jumper, Pushmeet
Kohli, and David~R. Kelley.
\newblock Effective gene expression prediction from sequence by integrating
long-range interactions.
\newblock \emph{Nature Methods}, 18\penalty0 (10):\penalty0 1196--1203, Oct
2021{\natexlab{a}}.
\newblock ISSN 1548-7105.
\newblock \doi{10.1038/s41592-021-01252-x}.
\newblock URL \url{https://doi.org/10.1038/s41592-021-01252-x}.
\bibitem[Avsec et~al.(2021{\natexlab{b}})Avsec, Weilert, Shrikumar, Krueger,
Alexandari, Dalal, Fropf, McAnany, Gagneur, Kundaje, and
Zeitlinger]{avsec2021base}
{\v Z}iga Avsec, Melanie Weilert, Avanti Shrikumar, Sabrina Krueger, Amr
Alexandari, Khyati Dalal, Robin Fropf, Charles McAnany, Julien Gagneur,
Anshul Kundaje, and Julia Zeitlinger.
\newblock Base-resolution models of transcription-factor binding reveal soft
motif syntax.
\newblock \emph{Nat Genet}, 53\penalty0 (3):\penalty0 354--366, February
2021{\natexlab{b}}.
\bibitem[Bailey(2021)]{bailey_streme_2021}
Timothy~L Bailey.
\newblock {STREME}: accurate and versatile sequence motif discovery.
\newblock 37\penalty0 (18):\penalty0 2834--2840, 2021.
\newblock ISSN 1367-4803, 1460-2059.
\newblock \doi{10.1093/bioinformatics/btab203}.
\newblock URL
\url{https://academic.oup.com/bioinformatics/article/37/18/2834/6184861}.
\bibitem[Bailey et~al.(2006)Bailey, Williams, Misleh, and Li]{bailey_meme_2006}
Timothy~L. Bailey, Nadya Williams, Chris Misleh, and Wilfred~W. Li.
\newblock {MEME}: discovering and analyzing {DNA} and protein sequence motifs.
\newblock 34:\penalty0 W369--W373, 2006.
\newblock ISSN 0305-1048.
\newblock \doi{10.1093/nar/gkl198}.
\newblock URL \url{https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1538909/}.
\bibitem[Bungert \& Wacker(2022)Bungert and Wacker]{bungert_lion_2022}
Leon Bungert and Philipp Wacker.
\newblock The lion in the attic -- a resolution of the borel--kolmogorov
paradox, 2022.
\newblock URL \url{http://arxiv.org/abs/2009.04778}.
\bibitem[Chen et~al.(2017)Chen, Jacob, and Mairal]{chen_biological_2017}
Dexiong Chen, Laurent Jacob, and Julien Mairal.
\newblock Biological sequence modeling with convolutional kernel networks.
\newblock 2017.
\newblock \doi{10.1101/217257}.
\newblock URL \url{http://biorxiv.org/lookup/doi/10.1101/217257}.
\bibitem[Ditz et~al.(2022)Ditz, Reuter, and Pfeifer]{ditz_convolutional_2022}
Jonas~C. Ditz, Bernhard Reuter, and Nico Pfeifer.
\newblock Convolutional motif kernel networks, 2022.
\newblock URL \url{http://arxiv.org/abs/2111.02272}.
\bibitem[{ENCODE Project Consortium}(2004)]{encode_project}
{ENCODE Project Consortium}.
\newblock The {ENCODE} ({ENCyclopedia} of {DNA} elements) project.
\newblock 306\penalty0 (5696):\penalty0 636--640, 2004.
\newblock ISSN 1095-9203.
\newblock \doi{10.1126/science.1105136}.
\bibitem[Gupta et~al.(2007)Gupta, Stamatoyannopoulos, Bailey, and
Noble]{tomtom_2007}
Shobhit Gupta, John~A. Stamatoyannopoulos, Timothy~L. Bailey, and
William~Stafford Noble.
\newblock Quantifying similarity between motifs.
\newblock 8\penalty0 (2):\penalty0 R24, 2007.
\newblock ISSN 1474-760X.
\newblock \doi{10.1186/gb-2007-8-2-r24}.
\newblock URL \url{https://doi.org/10.1186/gb-2007-8-2-r24}.
\bibitem[Harr et~al.(1983)Harr, H{\"a}ggstr{\"o}m, and
Gustafsson]{harris1983search}
R~Harr, M~H{\"a}ggstr{\"o}m, and P~Gustafsson.
\newblock Search algorithm for pattern match analysis of nucleic acid
sequences.
\newblock \emph{Nucleic Acids Res}, 11\penalty0 (9):\penalty0 2943--2957, May
1983.
\bibitem[Hastie et~al.(2015)Hastie, Tibshirani, and
Wainwright]{hastie2015statistical}
Trevor Hastie, Robert Tibshirani, and Martin Wainwright.
\newblock Statistical learning with sparsity.
\newblock \emph{Monographs on statistics and applied probability},
143:\penalty0 143, 2015.
\bibitem[Jaillard et~al.(2018)Jaillard, Lima, Tournoud, Mahé, van Belkum,
Lacroix, and Jacob]{jaillard2018fast}
Magali Jaillard, Leandro Lima, Maud Tournoud, Pierre Mahé, Alex van Belkum,
Vincent Lacroix, and Laurent Jacob.
\newblock A fast and agnostic method for bacterial genome-wide association
studies: Bridging the gap between k-mers and genetic events.
\newblock \emph{PLOS Genetics}, 14\penalty0 (11):\penalty0 1--28, 11 2018.
\newblock \doi{10.1371/journal.pgen.1007758}.
\newblock URL \url{https://doi.org/10.1371/journal.pgen.1007758}.
\bibitem[Jolma et~al.(2013)Jolma, Yan, Whitington, Toivonen, Nitta, Rastas,
Morgunova, Enge, Taipale, Wei, Palin, Vaquerizas, Vincentelli, Luscombe,
Hughes, Lemaire, Ukkonen, Kivioja, and Taipale]{jolma_2013}
Arttu Jolma, Jian Yan, Thomas Whitington, Jarkko Toivonen, Kazuhiro R. Nitta,
Pasi Rastas, Ekaterina Morgunova, Martin Enge, Mikko Taipale, Gonghong Wei,
Kimmo Palin, Juan M. Vaquerizas, Renaud Vincentelli, Nicholas M. Luscombe,
Timothy R. Hughes, Patrick Lemaire, Esko Ukkonen, Teemu Kivioja, and Jussi
Taipale.
\newblock {DNA}-binding specificities of human transcription factors.
\newblock 152\penalty0 (1):\penalty0 327--339, 2013.
\newblock ISSN 00928674.
\newblock \doi{10.1016/j.cell.2012.12.009}.
\newblock URL
\url{https://linkinghub.elsevier.com/retrieve/pii/S0092867412014961}.
\bibitem[Kelley et~al.(2018)Kelley, Reshef, Bileschi, Belanger, McLean, and
Snoek]{kelley2018sequential}
David~R Kelley, Yakir~A Reshef, Maxwell Bileschi, David Belanger, Cory~Y
McLean, and Jasper Snoek.
\newblock Sequential regulatory activity prediction across chromosomes with
convolutional neural networks.
\newblock \emph{Genome Res}, 28\penalty0 (5):\penalty0 739--750, March 2018.
\bibitem[Koo \& Eddy(2019)Koo and Eddy]{koo_representation_2019}
Peter~K. Koo and Sean~R. Eddy.
\newblock Representation learning of genomic sequence motifs with convolutional
neural networks.
\newblock 15\penalty0 (12):\penalty0 e1007560, 2019.
\newblock ISSN 1553-7358.
\newblock \doi{10.1371/journal.pcbi.1007560}.
\newblock URL
\url{https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1007560}.
\newblock Publisher: Public Library of Science.
\bibitem[Krizhevsky et~al.(2012)Krizhevsky, Sutskever, and
Hinton]{krizhevsky2012ImageNet}
Alex Krizhevsky, Ilya Sutskever, and Geoffrey~E Hinton.
\newblock Imagenet classification with deep convolutional neural networks.
\newblock In F.~Pereira, C.J. Burges, L.~Bottou, and K.Q. Weinberger (eds.),
\emph{Advances in Neural Information Processing Systems}, volume~25. Curran
Associates, Inc., 2012.
\newblock URL
\url{https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf}.
\bibitem[LeCun \& Bengio(1998)LeCun and Bengio]{lecun1998convolutional}
Yann LeCun and Yoshua Bengio.
\newblock \emph{Convolutional Networks for Images, Speech, and Time Series},
pp.\ 255–258.
\newblock MIT Press, Cambridge, MA, USA, 1998.
\newblock ISBN 0262511029.
\bibitem[Lee et~al.(2016)Lee, Sun, Sun, and Taylor]{lee_exact_2016}
Jason~D. Lee, Dennis~L. Sun, Yuekai Sun, and Jonathan~E. Taylor.
\newblock Exact post-selection inference, with application to the lasso.
\newblock 44\penalty0 (3):\penalty0 907--927, 2016.
\newblock ISSN 0090-5364.
\newblock \doi{10.1214/15-AOS1371}.
\newblock URL \url{http://arxiv.org/abs/1311.6238}.
\bibitem[Lifshits(1983)]{lifshits1983absolute}
M.A. Lifshits.
\newblock On the absolute continuity of distributions of functionals of random
processes.
\newblock \emph{Theory of Probability \& Its Applications}, 27\penalty0
(3):\penalty0 600--607, 1983.
\bibitem[{Loftus} \& {Taylor}(2015){Loftus} and {Taylor}]{loftus2015selective}
Joshua~R. {Loftus} and Jonathan~E. {Taylor}.
\newblock {Selective inference in regression models with groups of variables}.
\newblock \emph{arXiv e-prints}, art. arXiv:1511.01478, November 2015.
\bibitem[Molnar(2022)]{molnar2022interpretable}
Christoph Molnar.
\newblock \emph{Interpretable Machine Learning}.
\newblock 2 edition, 2022.
\newblock URL \url{https://christophm.github.io/interpretable-ml-book}.
\bibitem[Novakovsky et~al.(2022)Novakovsky, Dexter, Libbrecht, Wasserman, and
Mostafavi]{novakovsky2022obtaining}
Gherman Novakovsky, Nick Dexter, Maxwell~W. Libbrecht, Wyeth~W. Wasserman, and
Sara Mostafavi.
\newblock Obtaining genetics insights from deep learning via explainable
artificial intelligence.
\newblock \emph{Nature Reviews Genetics}, Oct 2022.
\newblock ISSN 1471-0064.
\newblock \doi{10.1038/s41576-022-00532-2}.
\newblock URL \url{https://doi.org/10.1038/s41576-022-00532-2}.
\bibitem[Ras et~al.(2022)Ras, Xie, van Gerven, and Doran]{ras2022explainable}
Gabrielle Ras, Ning Xie, Marcel van Gerven, and Derek Doran.
\newblock Explainable deep learning: A field guide for the uninitiated.
\newblock \emph{J. Artif. Int. Res.}, 73, may 2022.
\newblock ISSN 1076-9757.
\newblock \doi{10.1613/jair.1.13200}.
\newblock URL \url{https://doi.org/10.1613/jair.1.13200}.
\bibitem[Reid \& Tibshirani(2013)Reid and Tibshirani]{reid_sparse_2015}
Stephen Reid and Robert Tibshirani.
\newblock Sparse regression and marginal testing using cluster prototypes.
\newblock 2013.
\newblock URL \url{http://arxiv.org/abs/1503.00334}.
\bibitem[Reid et~al.(2015)Reid, Taylor, and Tibshirani]{reid_general_2015}
Stephen Reid, Jonathan Taylor, and Robert Tibshirani.
\newblock A general framework for estimation and inference from clusters of
features.
\newblock 2015.
\newblock URL \url{http://arxiv.org/abs/1511.07839}.
\bibitem[Reid et~al.(2018)Reid, Taylor, and Tibshirani]{reid2018general}
Stephen Reid, Jonathan Taylor, and Robert Tibshirani.
\newblock A general framework for estimation and inference from clusters of
features.
\newblock \emph{Journal of the American Statistical Association}, 113\penalty0
(521):\penalty0 280--293, 2018.
\newblock \doi{10.1080/01621459.2016.1246368}.
\newblock URL \url{https://doi.org/10.1080/01621459.2016.1246368}.
\bibitem[Rizk et~al.(2013)Rizk, Lavenier, and Chikhi]{rizk_dsk_2013}
Guillaume Rizk, Dominique Lavenier, and Rayan Chikhi.
\newblock {DSK}: k-mer counting with very low memory usage.
\newblock 29\penalty0 (5):\penalty0 652--653, 2013.
\newblock ISSN 1367-4811.
\newblock \doi{10.1093/bioinformatics/btt020}.
\bibitem[Roux~de Bézieux et~al.(2022)Roux~de Bézieux, Lima, Perraudeau, Mary,
Dudoit, and Jacob]{roux2022caldera}
Hector Roux~de Bézieux, Leandro Lima, Fanny Perraudeau, Arnaud Mary, Sandrine
Dudoit, and Laurent Jacob.
\newblock {CALDERA: finding all significant de Bruijn subgraphs for bacterial
GWAS}.
\newblock \emph{Bioinformatics}, 38:\penalty0 i36--i44, 06 2022.
\newblock ISSN 1367-4803.
\newblock \doi{10.1093/bioinformatics/btac238}.
\newblock URL \url{https://doi.org/10.1093/bioinformatics/btac238}.
\bibitem[Ruan \& Stormo(2017)Ruan and Stormo]{ruan_inherent_2017}
Shuxiang Ruan and Gary~D. Stormo.
\newblock Inherent limitations of probabilistic models for protein-{DNA}
binding specificity.
\newblock 13\penalty0 (7):\penalty0 e1005638, 2017.
\newblock ISSN 1553-7358.
\newblock \doi{10.1371/journal.pcbi.1005638}.
\newblock URL
\url{https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005638}.
\newblock Publisher: Public Library of Science.
\bibitem[Schneider \& Stephens(1990)Schneider and
Stephens]{schneider1990sequence}
T~D Schneider and R~M Stephens.
\newblock Sequence logos: a new way to display consensus sequences.
\newblock \emph{Nucleic Acids Res}, 18\penalty0 (20):\penalty0 6097--6100,
October 1990.
\bibitem[Shrikumar et~al.(2018)Shrikumar, Tian, Avsec, Shcherbina, Banerjee,
Sharmin, Nair, and Kundaje]{shrikumar2018technical}
Avanti Shrikumar, Katherine Tian, Žiga Avsec, Anna Shcherbina, Abhimanyu
Banerjee, Mahfuza Sharmin, Surag Nair, and Anshul Kundaje.
\newblock Technical note on transcription factor motif discovery from
importance scores (tf-modisco) version 0.5.6.5, 2018.
\newblock URL \url{https://arxiv.org/abs/1811.00416}.
\bibitem[Slim et~al.(2019)Slim, Chatelain, Azencott, and
Vert]{slim2019kernelpsi}
Lotfi Slim, Cl{\'e}ment Chatelain, Chloe-Agathe Azencott, and Jean-Philippe
Vert.
\newblock kernel{PSI}: a post-selection inference framework for nonlinear
variable selection.
\newblock In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.),
\emph{Proceedings of the 36th International Conference on Machine Learning},
volume~97 of \emph{Proceedings of Machine Learning Research}, pp.\
5857--5865. PMLR, 09--15 Jun 2019.
\newblock URL \url{https://proceedings.mlr.press/v97/slim19a.html}.
\bibitem[Soltanolkotabi et~al.(2019)Soltanolkotabi, Javanmard, and
Lee]{soltanolkotabi2019theoretical}
Mahdi Soltanolkotabi, Adel Javanmard, and Jason~D. Lee.
\newblock Theoretical insights into the optimization landscape of
over-parameterized shallow neural networks.
\newblock \emph{IEEE Transactions on Information Theory}, 65\penalty0
(2):\penalty0 742--769, 2019.
\newblock \doi{10.1109/TIT.2018.2854560}.
\bibitem[Song et~al.(2012)Song, Smola, Gretton, Bedo, and
Borgwardt]{song2012feature}
Le~Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt.
\newblock Feature selection via dependence maximization.
\newblock \emph{Journal of Machine Learning Research}, 13\penalty0
(47):\penalty0 1393--1434, 2012.
\newblock URL \url{http://jmlr.org/papers/v13/song12a.html}.
\bibitem[Taylor \& Tibshirani(2015)Taylor and
Tibshirani]{taylor2015statistical}
Jonathan Taylor and Robert~J. Tibshirani.
\newblock Statistical learning and selective inference.
\newblock \emph{Proceedings of the National Academy of Sciences}, 112\penalty0
(25):\penalty0 7629--7634, 2015.
\newblock \doi{10.1073/pnas.1507583112}.
\newblock URL \url{https://www.pnas.org/doi/abs/10.1073/pnas.1507583112}.
\bibitem[Taylor et~al.(2014)Taylor, Lockhart, Tibshirani, and
Tibshirani]{taylor_exact_2014}
Jonathan Taylor, Richard Lockhart, Robert Tibshirani, and Ryan~J Tibshirani.
\newblock Exact post-selection inference for forward stepwise and least angle
regression.
\newblock pp.\ ~32, 2014.
\bibitem[Tsirelson(1976)]{tsirel1976density}
V.S. Tsirelson.
\newblock The density of the distribution of the maximum of a gaussian process.
\newblock \emph{Theory of Probability \& Its Applications}, 20\penalty0
(4):\penalty0 847--856, 1976.
\bibitem[Ullah \& Ben-Hur(2021)Ullah and Ben-Hur]{ullah_self-attention_2021}
Fahad Ullah and Asa Ben-Hur.
\newblock A self-attention model for inferring cooperativity between regulatory
features.
\newblock 49\penalty0 (13):\penalty0 e77, 2021.
\newblock ISSN 0305-1048.
\newblock \doi{10.1093/nar/gkab349}.
\newblock URL \url{https://doi.org/10.1093/nar/gkab349}.
\bibitem[Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez,
Kaiser, and Polosukhin]{vaswani2017attention}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, {\L}ukasz Kaiser, and Illia Polosukhin.
\newblock Attention is all you need.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Visscher et~al.(2017)Visscher, Wray, Zhang, Sklar, McCarthy, Brown,
and Yang]{visscher2017years}
P~M Visscher, N~R Wray, Q~Zhang, P~Sklar, M~I McCarthy, M~A Brown, and J~Yang.
\newblock 10 years of {GWAS} discovery: Biology, function, and translation.
\newblock \emph{Am J Hum Genet}, 101\penalty0 (1):\penalty0 5--22, July 2017.
\newblock \doi{10.1016/j.ajhg.2017.06.005}.
\newblock URL \url{https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5501872/}.
\bibitem[Wasserman \& Roeder(2009)Wasserman and
Roeder]{wasserman_high-dimensional_2009}
Larry Wasserman and Kathryn Roeder.
\newblock High-dimensional variable selection.
\newblock 37\penalty0 (5):\penalty0 2178--2201, 2009.
\newblock ISSN 0090-5364, 2168-8966.
\newblock \doi{10.1214/08-AOS646}.
\newblock Publisher: Institute of Mathematical Statistics.
\bibitem[Wasserstein \& Lazar(2016)Wasserstein and Lazar]{wasserstein2016asa}
Ronald~L. Wasserstein and Nicole~A. Lazar.
\newblock The {A}{S}{A} statement on p-values: Context, process, and purpose.
\newblock \emph{The American Statistician}, 70\penalty0 (2):\penalty0 129--133,
2016.
\newblock \doi{10.1080/00031305.2016.1154108}.
\newblock URL \url{https://doi.org/10.1080/00031305.2016.1154108}.
\bibitem[Yamada et~al.(2018)Yamada, Umezu, Fukumizu, and
Takeuchi]{yamada_post_2018}
Makoto Yamada, Yuta Umezu, Kenji Fukumizu, and Ichiro Takeuchi.
\newblock Post selection inference with kernels.
\newblock 2018.
\bibitem[Zhou \& Troyanskaya(2015)Zhou and Troyanskaya]{zhou2015predicting}
Jian Zhou and Olga~G Troyanskaya.
\newblock Predicting effects of noncoding variants with deep learning-based
sequence model.
\newblock \emph{Nat Methods}, 12:\penalty0 931--4, 2015 Oct 2015.
\newblock ISSN 1548-7105.
\newblock \doi{10.1038/nmeth.3547}.
\end{thebibliography}