On the Consistency of Average Embeddings for Item Recommendation
Extracted source text · arXiv 2308.12767
\documentclass[manuscript,screen]{acmart}
%\documentclass[sigconf]{acmart}
\usepackage[nolist,nohyperlinks]{acronym}
\usepackage[]{breqn}
\usepackage{amsmath}
\usepackage{lipsum}
\usepackage{array}
\usepackage{varwidth}
\usepackage[skip=0.333\baselineskip]{caption}
\usepackage{subcaption}
\usepackage{amsmath}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\up}[1]{\textsuperscript{#1}}
\makeatletter \renewcommand\d[1]{\ensuremath{%
\;\mathrm{d}#1\@ifnextchar\d{\!}{}}}
\makeatother
\newtheorem{theorem}{Theorem}
\newtheorem{assumption}{Assumption}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
% \newtheorem{lemma}{Lemma}%[chapter]
% \newtheorem{theorem}[lemma]{Theorem}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{example}[theorem]{Example}
\newtheorem{definition}[theorem]{Definition}
\usepackage{multirow}
\usepackage{lipsum}
\newcommand{\entiers}[2]{\llbracket #1,#2\rrbracket}
\AtBeginDocument{%
\providecommand\BibTeX{{%
\normalfont B\kern-0.5em{\scshape i\kern-0.25em b}\kern-0.8em\TeX}}}
\copyrightyear{2023}
\acmYear{2023}
\setcopyright{acmlicensed}\acmConference[RecSys '23]{Seventeenth ACM Conference on Recommender Systems}{September 18--22, 2023}{Singapore, Singapore}
\acmBooktitle{Seventeenth ACM Conference on Recommender Systems (RecSys '23), September 18--22, 2023, Singapore, Singapore}
\acmPrice{15.00}
\acmDOI{10.1145/3604915.3608837}
\acmISBN{979-8-4007-0241-9/23/09}
\DeclareMathOperator\erf{erf}
\def\Cov{{\textrm{Cov}}\,}
\def\Var{{\textrm{Var}}\,}
\def\E{\mathop{\mathbb{E}}\,}
\begin{document}
%\title{On the Precision of Item Recommendation from Average Embeddings}
\title{On the Consistency of Average Embeddings for~Item~Recommendation}
\author{Walid Bendada}
\authornote{Contact author: \href{research@deezer.com}{research@deezer.com}}
\orcid{0009-0001-7510-650X}
\affiliation{
\institution{Deezer Research \& LAMSADE, Université Paris Dauphine, PSL}
\city{}
\country{France}
}
\author{Guillaume Salha-Galvan}
\orcid{0000-0002-2452-1041}
\affiliation{
\institution{Deezer Research}
\city{}
\country{France}
}
\author{Romain Hennequin}
\orcid{0000-0001-8158-5562}
\affiliation{
\institution{Deezer Research}
\city{}
\country{France}
}
\author{Thomas Bouabça}
\orcid{0009-0001-0627-9722}
\affiliation{
\institution{Deezer Research}
\city{}
\country{France}
}
\author{Tristan Cazenave}
\orcid{0000-0003-4669-9374}
\affiliation{
\institution{LAMSADE, Université Paris Dauphine, PSL}
\city{}
\country{France}
}
\renewcommand{\shortauthors}{W. Bendada, G. Salha-Galvan, R. Hennequin, T. Bouabça, T. Cazenave}
\begin{abstract}
A prevalent practice in recommender systems consists in averaging item embeddings to represent users or higher-level concepts in the same embedding space. This paper investigates the relevance of such a practice. For this purpose, we propose an expected precision score, designed to measure the consistency of an average embedding relative to the items used for its construction. We subsequently analyze the mathematical expression of this score in a theoretical setting with specific assumptions, as well as its empirical behavior on real-world data from music streaming services. Our results emphasize that real-world averages are less consistent for recommendation, which paves the way for future research to better align real-world embeddings with assumptions from our theoretical setting.
\end{abstract}
\begin{CCSXML}
<ccs2012>
<concept>
<concept_id>10002951.10003317.10003347.10003350</concept_id>
<concept_desc>Information systems~Recommender systems</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10010147.10010257.10010293.10010319</concept_id>
<concept_desc>Computing methodologies~Learning latent representations</concept_desc>
<concept_significance>300</concept_significance>
</concept>
\end{CCSXML}
\ccsdesc[300]{Information systems~Recommender systems}
\ccsdesc[300]{Computing methodologies~Learning latent representations}
\keywords{Recommender Systems, Representation Learning, Embedding Vectors, Average Embeddings.}
\maketitle
\section{Introduction}
\label{s1}
Modern recommender systems often leverage representation learning techniques to summarize similarities between recommendable items \cite{schedl2018current,okura2017embedding,yang2016survey,zhang2019deep}. These techniques learn low-dimensional vectorial representations of~these items, also known as \textit{embedding vectors} or simply \textit{embeddings}, in a common vector space where item proximity should reflect user preferences (for a \textit{collaborative filtering} system~\cite{koren2015advances}) or resemblance of item characteristics (for a \textit{content-based} system~\cite{javed2021review}). By computing {similarity metrics} such as the inner product or Euclidean distance between embeddings, the recommender system can subsequently identify new items similar to the ones each user has interacted with~\cite{al2018similarity,jain2020survey}.
%Over the years, embedding-based recommender systems have demonstrated promising performances in various domains, and are now widely adopted in industrial applications~[cite].
%In real-world applications of embedding-based recommender systems, a prevalent practice consists in \textit{averaging item embeddings} to obtain embeddings for users or for higher-level concepts.
A prevalent practice associated with the use of such embeddings in industry-oriented research and applications consists in \textit{averaging item embeddings} to obtain embeddings for users or higher-level concepts in the same vector space~\cite{briand2021semi,hansen2020contextual,pal2020pinnersage,wang2018billion,grbovic2018real,okura2017embedding}.
As an illustration, Spotify learns embedding representations of listening sessions by averaging pre-computed embeddings of the music tracks listened to during these sessions~\cite{hansen2020contextual}.
This service also computes ``long-term'' user embeddings, used for recommendation purposes, by averaging the session embeddings associated with each user. Deezer computes embeddings for several types of recommendable music collections, such as playlists and albums, by averaging embeddings of the music tracks present in these collections~\cite{briand2021semi}.
%Pinterest considers average embeddings of ``pins'', i.e., visual bookmarks, as a candidate method to represent users, with the aim of recommending new pins close to each user in the embedding space~\cite{pal2020pinnersage}. %pinnersage, Pal et al. KDD 2020
Yahoo averages embeddings of the news articles previously browsed by each user to represent them in the same embedding space as articles and provide personalized news recommendations~\cite{okura2017embedding}.
Alibaba averages side information embeddings, including category and brand embeddings, to obtain product embeddings for cold start recommendation on the Taobao e-commerce platform~\cite{wang2018billion}.
However, despite its prevalence, this averaging practice is often adopted without explicit justification from a theoretical standpoint.
As we detail in Section~\ref{s2}, the rationale for averaging item embeddings mainly stands in the simplicity and scalability of this approach~\cite{youtube,briand2021semi}.
Yet, it is unclear to which extent averaging item embeddings guarantees to provide faithful user or higher-level concept representations for recommendation.
For instance, assuming we represent a user by the average embedding of their previously consumed items, to what degree would the neighboring items of this average constitute relevant recommendations for this user? While the impact of averaging and other pooling operations has been studied in other fields, notably natural language processing (NLP)~\cite{blog,arora2017simple}, to our knowledge, the consistency of averaging operations remains relatively understudied in the specific context of a recommender~system.
In this short paper, we propose to investigate these important considerations, making the following contributions:
\begin{itemize}
\item Firstly, we define $\text{Consistency}_k(\mathcal{X})$, a general expected precision score introduced in this study to measure the consistency of an average embedding relative to the items it summarizes, from a recommendation~standpoint.
\item Secondly, we examine the consistency of averaging operations in a \textit{theoretical} setting with general assumptions on item embeddings, providing an in-depth analysis of the expression of $\text{Consistency}_k(\mathcal{X})$ in this setting.
\item Thirdly, we analyze the \textit{empirical} behavior of this score on real-world data. Our experiments consider three variants of large-scale music track embeddings obtained from the music streaming service Deezer~\cite{briand2021semi}.
\item Lastly, we discuss the discrepancies between our theoretical and empirical results, emphasizing that real-world averages are less consistent for recommendation. This discussion paves the way for future research to better align real-world data with our theoretical assumptions. Overall, we believe this study will be insightful for researchers and practitioners aiming to improve the faithfulness of average embeddings in their recommender systems.
\end{itemize}
This paper is organized as follows. In Section~\ref{s2}, we introduce the average embedding consistency problem more precisely. We subsequently present our proposed $\text{Consistency}_k(\mathcal{X})$ score. We report our theoretical analysis of this score in Section~\ref{s3}, and discuss our experimental results on real-world data in Section~\ref{s4}. Finally, we conclude in Section~\ref{s5}.
\section{Problem Formulation and Evaluation}
\label{s2}
\subsection{Preliminaries}
\label{s21}
\subsubsection{Mathematical Notation} Throughout this paper, we consider a set $\mathcal{X} = \{X_{i}\}_{1 \leq i \leq N}$ of $N \in \mathbb{N}^*$ real-valued vectors of dimension $d \in \mathbb{N}^*$, i.e., $\forall i \in \{1, \dots, N\}, X_i = (X_{i,1}, X_{i,2}, \dots, X_{i,d})^\intercal \in \mathbb{R}^{d}$.
We denote by $\mathcal{X}_k$ the set of subsets of $\mathcal{X}$ of cardinality $k$, for any $k \in \{1,\dots,N\}$. For any vector $z \in \mathbb{R}^{d}$ and $k \in \{1,\dots,N\}$, we define $\mathcal{X}_k(z) \in \mathcal{X}_k$ as the set of the $k$~\textit{nearest~neighbors}\footnote{This set might not be unique if $z$ is equidistant to several elements of $\mathcal{X}$, in which case $\mathcal{X}_k(z)$ can be drawn uniformly from all complying sets.} of $z$ among the $N$ elements of $\mathcal{X}$, according to some \textit{similarity metric} $s: \mathbb{R}^d \times \mathbb{R}^d \rightarrow \mathbb{R}$, i.e.,
\begin{equation}
\mathcal{X}_k(z) = \argmax_{\mathcal{Y} \in \mathcal{X}_k} \sum_{y \in \mathcal{Y}} s(z, y).
\end{equation}
Moreover, for any subset $\mathcal{U} = \{u_{i}\}_{1 \leq i \leq k} \subseteq \mathcal{X}$ of $k$ vectors, we define the \textit{center} or \textit{average} of $\mathcal{U}$ as $\mu_{\mathcal{U}} = \frac{1}{k}\sum_{u_{i} \in \mathcal{U}} u_{i}$,
and we denote the complementary set of $\mathcal{U}$ within $\mathcal{X}$ as $\overline{\mathcal{U}}$, i.e., $\overline{\mathcal{U}}= \mathcal{X}\setminus\mathcal{U}=\{X \in \mathcal{X}, X \notin\mathcal{U}\}$.
\subsubsection{From Mathematics to Recommender Systems}
In the context of an embedding-based recommender system~\cite{zhang2019deep,koren2015advances}, $\mathcal{X}$ would correspond to a set of $d$-dimensional embedding representations associated with all recommendable \textit{items} in a catalog, while $\mathcal{U} \in \mathcal{X}_k$ would correspond to the embeddings of a subset of $k$ items.
For instance, $\mathcal{X}$ could represent music tracks on a music streaming service, and $\mathcal{U}$ could represent the tracks present in a playlist or album of length~$k$~\cite{hansen2020contextual,briand2021semi}. Alternatively, $\mathcal{X}$ could represent all products from an e-commerce platform and $\mathcal{U}$ the shopping cart of a user~\cite{wang2018billion,wu2017session}.
At this stage, we do not make assumptions regarding the representation learning technique used~to~learn~these~embeddings.
\subsection{Problem Formulation}
\label{s22}
\subsubsection{Averaging Embeddings} As illustrated in Section~\ref{s1}, recommender systems frequently average item embeddings.
Using the above notation, the action of averaging all embeddings of an item collection $\mathcal{U}$ to represent this collection translates to using $\mu_{\mathcal{U}}$ as the representation.
As $\mu_{\mathcal{U}} \in \mathbb{R}^d$, one can interpret this vector as a new embedding in the same vector space as items,
and, therefore, measure the similarity between $\mu_{\mathcal{U}}$ and other items using the similarity metric $s$.
\subsubsection{How Relevant is this Practice?}
\label{s222}
This paper aims to rigorously investigate the consistency of this averaging~practice. For instance, if we represent a user by the average embedding $\mu_{\mathcal{U}}$ of the items $\mathcal{U}$ this user has consumed or liked, do the items similar to $\mu_{\mathcal{U}}$ in the embedding space (according to $s$) also constitute items that the user would like? In the same way, if $\mu_{\mathcal{U}}$ summarizes an album composed of the tracks in $\mathcal{U}$, are tracks similar to $\mu_{\mathcal{U}}$ also similar to tracks~in~$\mathcal{U}$?
Intuitively, for $\mu_{\mathcal{U}}$ to be faithful to $\mathcal{U}$, we expect $\mu_{\mathcal{U}}$ to remain similar to the original items from $\mathcal{U}$. For instance, the user embedding should remain similar to the items the user has already liked, and the album embedding to the tracks contained in the album. For this reason, this paper focuses on the following specific research question: \textit{to which extent and under which conditions does $\mu_{\mathcal{U}}$ remain similar to items from $\mathcal{U}$, i.e., to the items used for its construction?}
\subsubsection{Related Work}
To our knowledge, this theoretical question remains understudied in recommendation.
Existing research predominantly relied on averages for practical reasons.
%In previous work, averaging operations were mainly motivated by practical reasons.
%Existing research mainly motivated the use of $\mu_{\mathcal{U}}$ by practical reasons.
Average embeddings are faster and simpler to compute than alternatives such as neural aggregations~\cite{he2017neural,hidasi2015session}. They have also been praised for their scalability, as they provide fixed-size representations independently of the number of items in $\mathcal{U}$~\cite{youtube,briand2021semi}. Yet, recent research pointed out some of their limitations, e.g., to represent heterogeneous or contextual preferences~\cite{hansen2020contextual,park2018collaborative,tran2021hierarchical}.
We also acknowledge that the pros and cons of average embeddings have been studied for other applications, such as NLP tasks~\cite{mishra2019survey,arora2017simple,white2015well,blog}. While being out of our scope (averaging for recommendation), these studies confirm the~importance~of~our~research~problem.
% [LATER!]
%In the remainder of this article we will show that the extent to which this property is respected depends on the distribution of the set of vectors $\mathcal{X}$.
\subsection{Problem Evaluation}
To evaluate the consistency of $\mu_{\mathcal{U}}$ in accordance with our formulation in Section~\ref{s222}, we propose the following~score:
\begin{dmath}[compact]
\label{eq:consistency}
\text{Consistency}_k(\mathcal{X}) = \mathbb{E}_{\mathcal{U}\in\mathcal{X}_k} \Big[\text{Precision}_k(\mathcal{U})\Big], \allowbreak \text{ where }
\text{Precision}_k(\mathcal{U})=\frac{|\mathcal{X}_k(\mu_{\mathcal{U}}) \bigcap \mathcal{U}|}{k} \in [0, 1],
\end{dmath}
for a given $k \in \{1,\dots,N\}$.
In essence, $\text{Precision}_k(\mathcal{U})$ measures the percentage of items from $\mathcal{U}$ among the $k$ nearest neighbors of $\mu_{\mathcal{U}}$ in $\mathcal{X}$. A perfect precision of 1 indicates that $\mathcal{X}_k(\mu_{\mathcal{U}}) = \mathcal{U}$.
%. This definition can be extended to the precision of set $\mathcal{X}$ at degree k:
%$$
% \forall k\in \entiers2N, precision@k(\mathcal{X})=$$
Therefore, higher values of $\text{Consistency}_k(\mathcal{X})$ indicate that, on expectation, average embeddings computed from $\mathcal{X}$ will comprise more items used for their constructions in their neighborhood.
%The score depends on the item collection size $k$ and the distribution of all embeddings in $\mathcal{X}$.
The remainder of this paper provides an analysis of $\text{Consistency}_k(\mathcal{X})$ in a theoretical setting with assumptions of the distribution of embeddings in $\mathcal{X}$, and studies its empirical behavior on real-world~embeddings.
% consistency of an average embedding relative to the items used for its construction.
%\section{Background}
%\label{s2}
%\subsection{Definitions}
%\label{s21}
%Let $\mathcal{X} = \{X_{i}\}_{1 \leq i \leq N}$ be a set of real-valued vectors of dimensions $d$. That is, $|\mathcal{X}| = N \in \mathbb{N}$ and
% \begin{align}
% \forall i \in \entiers1N, X_i &= \begin{bmatrix}
% X_{i,1} \\
% X_{i,2} \\
% \vdots \\
% X_{i,d}
%% \end{bmatrix}
% \in \mathbb{R}^{d}
% \end{align}
%We assume that $\mathbb{R}^{d}$ is equipped with the usual dot product, denoted $s$:
%$$\forall (X,Y) \in \mathbb{R}^{2\times d}, s(X, Y) = X \cdot Y = \sum_{j=1}^{d}X_jY_j$$
%$\forall k \in \entiers1N$, let $\mathcal{X}_k$ be the set of subsets of $\mathcal{X}$ of cardinality $k$.
%For any vector $z \in \mathbb{R}^{d}$, we can define a set $\mathcal{X}_k(z)$ as the k nearest neighbours of $z$ among the elements of $\mathcal{X}$\footnote{this set might not be unique if $z$ is equidistant to several elements of $\mathcal{X}$, in which case $\mathcal{X}_k(z)$ can be drawn uniformly from all complying sets.}. $$\sum_{y* \in \mathcal{X}_k(z)} s(z, y*) = \max_{\mathcal{Y} \in \mathcal{X}_k} \sum_{y \in \mathcal{Y}} s(z, y)$$
%Moreover, for any subset $\mathcal{U} = \{u_{i}\}_{1 \leq i \leq k} \in \mathcal{X}_k$, the center of $\mathcal{U}$ is defined as $$\mu_{\mathcal{U}} = \sum_{i =1}^{k}\frac{ u_{i}}{k}$$
%and the complementary of $\mathcal{U}$ within $\mathcal{X}$ is $\overline{\mathcal{U}} = \mathcal{X}\setminus\mathcal{U}=\{X \in \mathcal{X}, X \notin\mathcal{U}\}$.
%\subsection{Problem Formulation}
%\label{s22}
%\subsubsection{Averaging in recommender systems}
%In the context of a recommender system relying on latent representations, $\mathcal{X}$ will denote the set of embeddings of all items of the catalog, while $\mathcal{U} \in \mathcal{X}_k$ will be the embeddings of a collection of items of size $k$. For instance, $\mathcal{X}$ could represent all songs available on a music streaming platforms and $\mathcal{U}$ could represent the songs in a playlist of length k. Alternatively, $\mathcal{X}$ could represent all products available on an e-commerce platform while $\mathcal{U}$ could represent the shopping cart of a user with k products in it. Using the above definitions, the action of averaging the embeddings of a collection of item to get a single representation of this collection translates to using the vector $\mu_{\mathcal{U}}$ as the representation of $\mathcal{U}$ for all $\mathcal{U} \subseteq \mathcal{X}$. Furthermore, we argue that for this aggregate representation to be faithful to the original set $\mathcal{U}$, we expect that $\mu_{\mathcal{U}}$ be as close to elements of $\mathcal{U}$ as possible, while remaining further away from elements of $\overline{\mathcal{U}}$. In the remainder of this article we will show that the extent to which this property is respected depends on the distribution of the set of vectors $\mathcal{X}$.
%\subsubsection{Assessing the relevance: precision of the average}
%To quantify how much the aforementioned property is respected, we introduce the following metric:
%\begin{align}
% \forall k\in \entiers2N, \forall \mathcal{U}\in\mathcal{X}_k, precision(\mathcal{U})=\frac{|\mathcal{X}_k(\mu_{\mathcal{U}}) \bigcap \mathcal{U}|}{k}
%\end{align}
%with $\mathcal{X}_k(\mu_{\mathcal{U}})$ the k nearest neighbours of $\mu_{\mathcal{U}}$ as defined in section \ref{s21}. This metric, which we call the precision of set $\mathcal{U}$ within $\mathcal{X}$, quantifies how many elements of $\mathcal{U}$ are among the k nearest neighbours of $\mu_{\mathcal{U}}$. Its value ranges from 0 (when no elements of $\mathcal{U}$ are present in $\mathcal{X}_k(\mu_{\mathcal{U}})$) to 1 (when $\mathcal{X}_k(\mu_{\mathcal{U}})$ is exactly $\mathcal{U}$). This definition can be extended to the precision of set $\mathcal{X}$ at degree k:
%\begin{align}
% \forall k\in \entiers2N, precision@k(\mathcal{X})=\mathop{\mathbb{E}}_{\mathcal{U}\in\mathcal{X}_k}precision(\mathcal{U})
%\end{align}
%$precision@k(\mathcal{X})$ thus gives a measure of how relevant the average of a collection of embeddings can be as function of the size of the collection k and the overall distribution of the embeddings. In the next section we will show how, given some hypothesis on the distribution of the embeddings and their dimension d, $precision@k(\mathcal{X})$ can be approximated with an explicit analytical expression.
\section{A Theoretical Analysis of $\text{CONSISTENCY}_k(\mathcal{X})$}
\label{s3}
We dedicate Section~\ref{s3} to our theoretical analysis of $\text{Consistency}_k(\mathcal{X})$. For clarity, we only present our setting, main results, and interpretation of these results in this section. We report all mathematical~proofs in Appendices~\ref{s32}~and~\ref{s33}.
\subsection{Setting and Assumptions}
\label{s311} We focus on the setting where $\mathcal{X}$ is a set of independent and identically distributed (i.i.d.) multi-dimensional random variables (r.v.). For every $X_{i} \in \mathcal{X}$, the elements $\{X_{i,j}\}_{1 \leq j \leq d}$ form a set of i.i.d uni-dimensional r.v. and we denote by $\mu$, $\sigma^{2}$, $\gamma$, and $\kappa$ the mean, variance, skewness, and kurtosis of their distribution, respectively. We assume that these moments are finite. $s$ is the prevalent \textit{inner product similarity}: $\forall (x,y) \in \mathbb{R}^{d} \times \mathbb{R}^{d}, s(x, y) = x^\intercal y = \sum_{i=1}^{d}x_i y_i$.% We note that, if one additionally assumes that $\forall X_{i} \in \mathcal{X}, \|X_{i} \|_2 = 1$, then $s$ would correspond to the \textit{cosine similarity}, another popular similarity metric in recommender systems~\cite{gunawardana2009survey}.
\subsection{Main Results and Interpretations}
In this theoretical setting, we obtain the following results on average embeddings.
\begin{proposition}
\label{proposition1}
Let $k \in \{2,\dots,N\}$, $\mathcal{U} \in \mathcal{X}_k$, $u_{\text{in}} \in \mathcal{U}$, and $u_{\text{out}} \in \overline{\mathcal{U}}$. Then, under the hypotheses of Section~\ref{s311}, both $s_{\text{in}} = s(u_{\text{in}}, \mu_{\mathcal{U}})$ and $s_{\text{out}}=s(u_{\text{out}}, \mu_{\mathcal{U}})$ converge in probability to normal distributions as $d$ increases. Their respective means and variances are provided in the proof of Appendix~\ref{s32}.
Moreover, under such distributions, we have: %the probability that $s(u_{\text{in}}, \mu_{\mathcal{U}}) > s(u_{\text{out}}, \mu_{\mathcal{U}})$ can be \textcolor{red}{approximated (?)} by the following expression:
\begin{dmath}[compact]
\label{eq:th_result_1}
\mathbb{P}\Big(s(u_{\text{in}}, \mu_{\mathcal{U}}) > s(u_{\text{out}}, \mu_{\mathcal{U}})\Big) =
\frac{1}{2}\Big(1+\erf(\sqrt{\frac{d\sigma^{2}}{2((2(k-1)+\kappa)\sigma^{2} + 2k\gamma\mu\sigma+ 2k^{2}\mu^{2})}})\Big),
\end{dmath}
where $\erf$ denotes the Gauss error function~\cite{andrews1998special}: $\erf : x \mapsto \frac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^2}dt$.
\end{proposition}
%\begin{proposition}
%\label{proposition1}
%Let $k \in \{2,\dots,N\}$ and $\mathcal{U} \in \mathcal{X}_k$. Let $u_{\text{in}} \in \mathcal{U}$ and $u_{\text{out}} \in \overline{\mathcal{U}}$. Let $\erf$ denote the Gauss error function~\cite{andrews1998special}: $\erf : x \mapsto \frac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^2}dt$. Then, in the setting of Section~\ref{s311} and provided that $d$ \textcolor{red}{is large enough (?)}, we have: %the probability that $s(u_{\text{in}}, \mu_{\mathcal{U}}) > s(u_{\text{out}}, \mu_{\mathcal{U}})$ can be \textcolor{red}{approximated (?)} by the following expression:
%\begin{equation}
%\label{eq:th_result_1}
% \mathbb{P}\Big(s(u_{\text{in}}, \mu_{\mathcal{U}}) > s(u_{\text{out}}, \mu_{\mathcal{U}})\Big) ~\textcolor{red}{\approx}~ \frac{1}{2}\Big(1-\erf(-\sqrt{\frac{d\sigma^{2}}{2((2(k-1)+\kappa)\sigma^{2} + 2k\gamma\mu\sigma+ 2k^{2}\mu^{2}}})\Big).
%\end{equation}
%\end{proposition}
Proposition~\ref{proposition1} is a consequence of the \textit{central limit theorem}~\cite{fischer2011history} applied to inner product similarities. %between multidimensional i.i.d random variables.
Figure~\ref{figure1} illustrates that, for dimension values such as $d=128$ (a common choice in recommendation applications), the normal distribution faithfully approximates the distribution of similarities, regardless of the original embedding distribution. %We will detail in section \ref{s32} how this observation leads to Proposition~\ref{proposition1}.
Hence, Equation~\eqref{eq:th_result_1} provides a reliable approximation of the probability that an element from $\mathcal{U}$ will be closer to its center $\mu_{\mathcal{U}}$ than a random point from $\overline{\mathcal{U}}$.
%Because of the monotony of the error function,
It is worth noting that the probability is guaranteed to be greater than 0.5, as the expression within the erf function is positive.
We also observe that the probability increases with $d$,
illustrating that, as the dimension increases, the chances for a vector from $\overline{\mathcal{U}}$ to have a stronger similarity with $\mu_{\mathcal{U}}$ than elements~of~$\mathcal{U}$~diminishes.
%For a fixed $d$, the probability in Proposition \ref{proposition1} is maximized when the denominator inside of the error function is minimized. As such, the case when $\mu=0$ is particularly interesting, which is not surprising considering the symmetry of the problem at hand. In this setting, the considered probability becomes independent on both $\sigma$ and $\gamma$ and is a strictly decreasing function of both $k$ and $\kappa$.
The case of centered embeddings ($\mu=0$) is of particular interest. Indeed, for $\mu=0$ and a fixed $d$, the probability becomes independent of $\sigma$ and $\gamma$, and a decreasing function of $k$ and $\kappa$.
%The first relation can be explained by the fact that as $k$ increases, $\mu_{\mathcal{U}}$ can not be guaranteed to remain close to every element of $\mathcal{U}$ while in the same time staying away from every element of $\overline{\mathcal{U}}$.
In essence, as $k$ increases, it becomes increasingly difficult for $\mu_{\mathcal{U}}$ to remain similar to all $\mathcal{U}$ items, while remaining dissimilar to all $\overline{\mathcal{U}}$ items. The decrease with respect to the kurtosis $\kappa$ relates to its interpretation as a measure of the propensity of a distribution to produce outliers~\cite{westfall2014kurtosis}. Intuitively, the presence of outliers would impact the ability for $\mu_{\mathcal{U}}$ to remain similar to all $\mathcal{U}$ items and dissimilar to all $\overline{\mathcal{U}}$ items\footnote{On the contrary, Equation~\eqref{eq:th_result_1} would be maximized by distributions with minimal kurtosis, such as a re-centered $\text{Bernoulli}(0.5)$ or a Rademacher distribution ($X_{i,j} = 1$ or $-1$ with probability $0.5$ each). We emphasize that these distributions are \textit{discrete}, and might therefore allow for less precise similarity computations between vectors. This brings to light an interesting \textit{trade-off} between similarity precision and average embedding consistency.}. In the following, we continue to focus on $\mu = 0$, using results from Proposition~\ref{proposition1} to express~$\text{Consistency}_k(\mathcal{X})$.
%the Bernoulli distribution, having the , guarantees that all vectors are equidistant from the origin and maximizes the probability from Equation~\eqref{eq:th_result_1}. Since this distribution is discrete we might fear it allows for less precise similarity values between vectors and so this raises the question of the desired trade-off between precision of similarity values and consistency of averages as introduced in \ref{eq:consistency}. For the case when $\mu=0$, we will prove in section \ref{s33} that we can derive a mathematical expression of $\text{Consistency}_k(\mathcal{X})$ which we introduce below.}.%, even for low values of $k$.
%On the contrary, the Bernoulli distribution, having the lowest possible kurtosis, guarantees that all vectors are equidistant from the origin and maximizes the probability from Equation~\eqref{eq:th_result_1}. Since this distribution is discrete we might fear it allows for less precise similarity values between vectors and so this raises the question of the desired trade-off between precision of similarity values and consistency of averages as introduced in \ref{eq:consistency}. For the case when $\mu=0$, we will prove in section \ref{s33} that we can derive a mathematical expression of $\text{Consistency}_k(\mathcal{X})$ which we introduce below.
%The next proposition focuses on the setting $\mu=0$, which maximizes this probability,
\begin{proposition}
\label{proposition2}
Under the hypotheses of Section~\ref{s311} and approximated distributions of Proposition~\ref{proposition1} with $\mu=0$:
\begin{equation}
\label{eq:th_result_2}
\text{Consistency}_k(\mathcal{X})
= \frac{1}{k} \sum_{i=1}^{k}\int_{-\infty}^{\infty}f_{\text{in},(i)}(x)\times F_{\text{out},(k-i+1)}(x)\d x,
\end{equation}
where explicit formulas for the $f_{\text{in},(i)}$ and $F_{\text{out},(k-i+1)}$ functions are provided in the proof of Appendix~\ref{s33}.
\end{proposition}
%\begin{proposition}
%\label{proposition2}
% Under the hypotheses and of Proposition~\ref{proposition1}, and additionally assuming that $\mu=0$, then:
%\begin{equation}
%\label{eq:th_result_2}
%\text{Consistency}_k(\mathcal{X})
% ~\textcolor{red}{\approx}~ \frac{1}{k} \sum_{i=1}^{k}\int_{-\infty}^{\infty}f_{\text{in},(i)}(x)\times F_{\text{out},(k-i+1)}(x)\d x,
%\end{equation}
% where explicit forms are provided for the $f_{\text{in},(i)}$ and $F_{\text{out},(k-i+1)}$ functions in Section~\ref{s33}.
%\end{proposition}
%\begin{align*}
%f_{\text{in},(i)}(x) &= \frac{k!}{(k-i)!(i-1)!}f_{\text{in}}(x)[F_{\text{in}}(x)]^{k-i}[1-F_{\text{in}}(x)]^{i-1} \\
%F_{\text{out},(k-i+1)}(x) &= \sum_{j=N-k+i}^{N-k}\binom {N-k}{j}F_{\text{out}}(x)^{j}(1-F_{\text{out}}(x))^{N-j} \\
%f_{\text{in}}(x) &= \phi(\frac{x-\mu_{\text{in}}}{\sigma_{\text{in}}}), with \phi: x \mapsto \frac{e^{\frac{-x^2}{2\sigma^{2}}}}{\sigma\sqrt{2\pi}} \\
%F_{s_{\text{in}}}(x) &= \frac{1}{2}(1+\erf(\frac{x-\mu_{\text{in}}}{\sigma_{\text{in}}})), F_{s_{\text{out}}}(x) = \frac{1}{2}(1+\erf(\frac{x-\mu_{\text{out}}}{\sigma_{\text{out}}}))
%\end{align*}
%and $\mu_{\text{in}}=d\sigma^{2}$, $\sigma_{\text{in}}^{2} = d\sigma^{4}\frac{(\kappa+k-2)}{k^2}$, $\mu_{\text{out}}=0$, $\sigma_{\text{out}}^{2}=\frac{d\sigma^{4}}{k}$.
\begin{figure*}[t!]
\begin{center}
\begin{subfigure}{.325\linewidth}
\includegraphics[width=\linewidth]{figures/sXYnormal.pdf}
\caption{}
\label{fig1a}
\end{subfigure} \hfill %\hfill % <-- "\hfill"
\begin{subfigure}{.32\linewidth}
\includegraphics[width=\linewidth]{figures/sXYbeta.pdf}
\caption{}
\label{fig1b}
\end{subfigure}\hfill
%\medskip % create some *vertical* separation between the graphs
\begin{subfigure}{.32 \linewidth}
\includegraphics[width=\linewidth]{figures/sXYuniform.pdf}
\caption{}
\label{fig1c}
\end{subfigure}% <-- "\hfill"
\caption{Histograms of inner product similarity values between 1~000 $d$-dimensional vectors with entries randomly drawn from $\mathcal{N}(0.5,1)$ (Figure~\ref{fig1a}), $\text{Beta}(2,2)$ (Figure~\ref{fig1b}), or $\text{Uniform}(0,1)$ (Figure~\ref{fig1c}) distributions, with $d \in \{2,10,32,64,128\}$. Histograms undergo a rightward shift as $d$ increases, since similarity computations involve summing more elements (see Section~\ref{s311}). Curves correspond to normal approximations of similarity distributions. We observe that, for the largest values of $d$, the normal distribution faithfully approximates all similarity distributions, an important result to validate the approximations of Propositions~\ref{proposition1}~and~\ref{proposition2}.}
\label{figure1}
%\vspace{-0.5cm}
\end{center}
\end{figure*}
Proposition~\ref{proposition2} provides a useful approximated analytical expression of $\text{Consistency}_k(\mathcal{X})$. In
Figure \ref{figure2}, we assess the accuracy of this expression by comparing, for various $X_{i,j}$ distributions and values of $k$, the $\text{Consistency}_k(\mathcal{X})$ score obtained from Equation~\eqref{eq:th_result_2} to the one estimated via numerical simulations.
%can thus be computed by estimating the integral of a continuous function with a closed form expression.
Our computed expression systematically coincides with the simulated score, validating the correctness of Proposition~\ref{proposition2} and the relevance of our approximations.
Overall, we observe that $\text{Consistency}_k(\mathcal{X})$ decreases with $k$ and the number of items $N = |\mathcal{X}|$. Regarding $k$, this result is coherent with our above interpretation of Proposition~\ref{proposition1}. Regarding $N$, such a result is also unsurprising. Indeed, the more vectors in $\mathcal{X}$, the greater the likelihood of some unrelated item embeddings becoming similar to $\mu_{\mathcal{U}}$ by~chance.
Yet, in Figure~\ref{figure2}, all scores
%However, we can see that overall the consistency observed, while steadily decreasing,
remain quite high for $N=$~1~000 (e.g., around 0.4 for $k=50$). Importantly, for small values of $k$, the consistency of average embeddings remains close to 1 even for $N=$~1~000~000.
Therefore, even with a large catalog of millions of items, as in music streaming services, averaging item embeddings appears as a consistent way to faithfully represent collections of a few items, provided that these embeddings comply with our~theoretical~assumptions.
%This means that even when the number of items in a catalog is in the order of one million, as in music platforms, the average of their embeddings is a nearly flawless representation of a group of 2 or 3 songs, provided that song embedding are distributed as stated in proposition \ref{proposition2}. %The remainder of the section is dedicated to proving proposition \ref{proposition1} and \ref{proposition2}
\begin{figure*}[t!]
\begin{center}
\begin{subfigure}{.32\linewidth}
\includegraphics[width=\linewidth]{figures/consistency_normal.pdf}
\caption{}
\label{fig2a}
\end{subfigure} \hfill %\hfill % <-- "\hfill"
\begin{subfigure}{.32\linewidth}
\includegraphics[width=\linewidth]{figures/consistency_rademacher.pdf}
\caption{}
\label{fig2b}
\end{subfigure}\hfill
%\medskip % create some *vertical* separation between the graphs
\begin{subfigure}{.32 \linewidth}
\includegraphics[width=\linewidth]{figures/consistency_uniform.pdf}
\caption{}
\label{fig2c}
\end{subfigure}% <-- "\hfill"
\caption{Comparison of the $\text{Consistency}_k(\mathcal{X})$ scores obtained by a direct \textit{computation} of Equation~\eqref{eq:th_result_2} to the ones estimated via numerical \textit{simulations}, for $k \in \{2, \dots, 50\}$, with $d =$ 128, $N =$ 1~000 or 1~000~000, and $X_{i,j} \sim \mathcal{N}(0,1)$ (Figure~\ref{fig2a}), $X_{i,j} \sim \text{Rademacher}()$ (Figure~\ref{fig2b}), or $X_{i,j} \sim \text{Uniform}(-1,1)$ (Figure~\ref{fig2c}).
%both the curve corresponding to \ref{proposition2} (with integrals numerically computed using the python library scipy).
We used the Python library \textit{scipy} to compute integrals from Equation~\eqref{eq:th_result_2}. For numerical simulations, we sampled $N$ vectors from the above three distributions. Then, for each $k$, we randomly picked a subset of $k$ vectors and computed the precision of this subset (Equation~\ref{eq:consistency}). We repeated this operation 1~000 times for each value of $k$ and reported averaged scores on figures. %Note: while simulations were possible in these three examples as $X_{i,j}$ distributions were known, we emphasize that Equation~\eqref{eq:th_result_2} would permit computing $\text{Consistency}_k(\mathcal{X})$ in more general settings.
}
\label{figure2}
%from k going from 2 to 50, both the curve corresponding to \ref{proposition2} (with integrals numerically computed using the python library scipy) as well as the curve of precision when sampling N vectors then, for each value of k, picking a subset of k vectors at random and computing the precision of this subset as defined in \ref{eq:consistency}. This operation is repeated 1000 times for each value of k and the results are averaged to get the desired curve. For each plot on figure \ref{figure2}, the blue curves correspond to the computed consistency using \ref{proposition1} while the orange curves correspond to the simulated consistency, and the dark curves where generated with $N=1000$ while the light curves were generated with $N=1000000$.
%For each plot on figure \ref{figure2}, the blue curves correspond to the computed consistency using \ref{proposition1} while the orange curves correspond to the simulated consistency, and the dark curves where generated with $N=1000$ while the light curves were generated with $N=1000000$.
\end{center}
\end{figure*}
%%%% PREVIOUS BEGINNING OF SECTION 3
%\section{A Theoretical Analysis of $\text{CONSISTENCY}_k(\mathcal{X})$}
%\label{s3}
%\subsection{Hypotheses and Global Overview of Results}
%\label{s31}
%\textcolor{red}{note: introduce the inner product here}
%In this section, we study the case where $\mathcal{X} = \{X_{i}\}_{1 \leq i \leq N}$ is a set of i.i.d multi-dimensional random variables and, for every $X_{i} \in \mathcal{X}$, the vector elements $\{X_{i,j}\}_{1 \leq j \leq d}$ form a set of $d$ i.i.d uni-dimensional random variables. Let $\mu, \sigma^{2}, \gamma, \kappa$ be respectively the mean, variance, skewness and kurtosis of the distribution of the random variables $\{X_{i,j}\}_{1 \leq i \leq N, 1 \leq j \leq d}$. Let $k \in \entiers2N$ and $\mathcal{U} \in \mathcal{X}_k$. Let $u_{\text{in}} \in \mathcal{U}$ and $u_{\text{out}} \in \overline{mathcal{U}}$. Then, provided that $d$ is large enough, the probability that $s(u_{\text{in}}, \mu_{\mathcal{U}}) > s(u_{\text{out}}, \mu_{\mathcal{U}})$ can be approximated by the following expression:
%\begin{align}
%\label{eq:th_result_1}
% P(s(u_{\text{in}}, \mu_{\mathcal{U}}) > s(u_{\text{out}}, \mu_{\mathcal{U}})) \approx \frac{1}{2}(1-\erf(-\sqrt{\frac{d\sigma^{2}}{2((2k+\kappa)\sigma^{2}+(2k^2-k+11)\mu^{2}+(4-k)\gamma\mu\sigma)}}))
%end{align}
%with $\erf : x \mapsto \frac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^2}dt$ the error function.
%Moreover, if the distribution is centered around the origin (i.e., $\mu=0$) the value of $precision@k(\mathcal{X})$ can be approximated by:
%\begin{align}
%\label{eq:th_result_2}
% precision@k(\mathcal{X}) \approx \sum_{i=1}^{k}\int_{-\infty}^{\infty}f_{\text{in},(i)}(x)\times F_{\text{out},(k-i+1)}(x)dx
%\end{align}
%where
%\begin{align*}
%f_{\text{in},(i)}(x) &= \frac{k!}{(k-i)!(i-1)!}f_{\text{in}}(x)[F_{\text{in}}(x)]^{k-i}[1-F_{\text{in}}(x)]^{i-1} \\
%F_{\text{out},(k-i+1)}(x) &= \sum_{j=N-k+i}^{N-k}\binom {N-k}{j}F_{\text{out}}(x)^{j}(1-F_{\text{out}}(x))^{N-j} \\
%f_{\text{in}}(x) &= \phi(\frac{x-\mu_{\text{in}}}{\sigma_{\text{in}}}), with \phi: x \mapsto \frac{e^{\frac{-x^2}{2\sigma^{2}}}}{\sigma\sqrt{2\pi}} \\
%F_{s_{\text{in}}}(x) &= \frac{1}{2}(1+\erf(\frac{x-\mu_{\text{in}}}{\sigma_{\text{in}}})), F_{s_{\text{out}}}(x) = \frac{1}{2}(1+\erf(\frac{x-\mu_{\text{out}}}{\sigma_{\text{out}}}))
%\end{align*}
%and $\mu_{\text{in}}=d\sigma^{2}$, $\sigma_{\text{in}}^{2} = d\sigma^{4}\frac{(\kappa+k-2)}{k^2}$, $\mu_{\text{out}}=0$, $\sigma_{\text{out}}^{2}=\frac{d\sigma^{4}}{k}$.
%The first result is a direct consequence of the central limit theorem applied to cosine similarities between vectors in large dimensional spaces, for which each dimension is an i.i.d random variable. We will see in section \ref{s4} that for a value $d=128$ typically used by real-world systems the "large dimension" hypothesis holds.
%The remainder of this section is dedicated to proving these results.
%\subsection{Proof of Proposition~\ref{proposition1}}
%\label{s32}
%In this section, we will give the proof for equation \ref{eq:th_result_1}. We will show how both $s(u_{\text{in}}, \mu_{\mathcal{U}})$ and $s(u_{\text{out}}, \mu_{\mathcal{U}})$ can be approximated by normal distributions regardless of the type of distribution followed by the elements of $\mathcal{X}$. We will then derive an expression for the difference of these two values which also follows a normal distribution, and use the cumulative distribution function of normal distributions to finally prove equation \ref{eq:th_result_1}. But firsthand, we will require to establish some preliminary regarding dot products of random variables in high dimensions.
%\subsubsection{Preliminaries}
%Let $X, Y, Z$ be $d$-dimensional random variables. Suppose that each of them is made of $d$ i.i.d uni-dimensional random variables. The underlying law of their dimensions might differ between $X,Y$ and $Z$. Let $\mu_{X}, \sigma^{2}_{X}, \gamma_{X}, \kappa_{X}$ be respectively the mean, variance, skewness and kurtosis of the distribution of the elements of $X$. Let $\mu_{Y}, \sigma^{2}_{Y}, \mu_{Z}, \sigma^{2}_{Z}$ be the mean and variance of the dimensions of $Y$ and $Z$ respectively. Then if $d$ is large enough, the following approximations hold:
%\begin{align}
% s(X,Y) &\sim \mathcal{N}(\mu_{XY}, \sigma_{XY}^{2})\label{eq:CLT_1} \\
% s(X,X) &\sim \mathcal{N}(\mu_{XX}, \sigma_{XX}^{2})\label{eq:CLT_2}
%\end{align}
%with
%\begin{align*}
%\mu_{XY} &= d \times \mu_{X}\mu_{Y} \\
%\sigma_{XY}^{2} &= d \times ((\sigma_{X}^{2} + \mu_{X}^{2}) (\sigma_{Y}^{2} + \mu_{Y}^{2}) -\mu_{X}^{2}\mu_{Y}^{2}) \\
%\mu_{XX} &= d \times (\mu^{2}_{X} + \sigma^{2}_{X}) \\
%\sigma_{XX}^{2} &= d \times (4\mu_{X}^{2}\sigma_{X}^{2}+ 4\mu_{X}\gamma_{X}\sigma_{X}^{3}+ (\kappa_{X} - 1)\sigma_{X}^4)
%\end{align*}
%Indeed, we have by definition $s(X,Y) = \sum_{i=1}^dX_{i}Y_{i}$. Each of the $X_{i}Y_{i}$ is a random variable of expectation $\mathbb{E}[X_{i}Y_{i}] = \mu_{X}\mu_{Y}$ and variance $\Var(X_{i}Y_{i})=(\sigma_{X}^{2}+\mu_{X}^{2})(\sigma_{Y}^{2}+\mu_{Y}^{2}) - \mu_{X}^{2}\mu_{Y}^{2}$. The dot product $s(X,Y)$ being the sum of $d$ i.i.d random variables, according to the central limit theorem, for large enough values of $d$ it can be approximated by normal distribution of expectation $d \times \mathbb{E}[X_{i}Y_{i}]$ and variance $d \times \Var(X_{i}Y_{i})$, which leads to equation \ref{eq:CLT_1}.
%A similar reasoning can be performed to prove equation \ref{eq:CLT_2}.
%Additionally, covariances between the different dot products can be written as :
%\begin{align}
% \Cov(s(X,Y), s(X,Z)) &= d\times\sigma_{X}^{2}\mu_{Y}\mu_{Z} \label{eq:cov_1}\\
% \Cov(s(X,X), s(X,Y)) &= d\times\mu_{Y}(\gamma_{X}\sigma_{X}^{3}+ 2\mu_{X}\sigma_{X}^{2})\label{eq:cov_2}
%\end{align}
%The first equation can be obtained by writing
%\begin{align*}
% \Cov(s(X,Y), s(X,Z)) &= \mathbb{E}[s(X,Y)s(X,Z)] -\mathbb{E}[s(X,Y)]\mathbb{E}[s(X,Z)]\\
% &=\mathbb{E}[(\sum_{i=1}^{d}X_{i}Y_{i})(\sum_{j=1}^{d}X_{j}Y_{j})] - d \times \mu_{X}\mu_{Y} \times d \times \mu_{X}\mu_{Z} \\
% &=E[\sum_{i=1}^{d}(X_{i}Y_{i}X_{i}Z_{i} + \sum_{j=1, j\neq i}^{d}X_{i}Y_{i}X_{j}Z_{j})] - d^2 \times \mu_{X}^2\mu_{Y}\mu_{Z}\\
% &=\sum_{i=1}^{d} (E[X_{i}^2]E[Y_{i}]E[Z_{i}] + \sum_{j=1 \atop j\neq i}^{d}E[X_{i}]E[Y_{i}]E[X_{j}]E[Z_{j}])- d^2 \times \mu_{X}^2\mu_{Y}\mu_{Z} \\
% &= d\times((\sigma_{X}^2+\mu_{X}^2)\mu_{Y}\mu_{Z} + (d-1)(\mu_{X}^2\mu_{Y}\mu_{Z}))- d^2 \times \mu_{X}^2\mu_{Y}\mu_{Z}\\
% &= d \times \sigma_{X}^2\mu_{Y}\mu_{Z}
%\end{align*}
%We can obtain equation \ref{eq:cov_2} with similar computations. Notice how $\mu_{Y}=0$ or $\mu_{Z}=0$ implies that the covariances turn into zero, which means that the dot products become uncorrelated and so, since they are multivariate normal distributions, they're also independent. This will prove helpful in the second part of the proof.
%\subsubsection{Distribution of $s_{\text{in}} = s(u_{\text{in}}, \mu_{\mathcal{U}})$ and $s_{\text{out}}=s(u_{\text{out}}, \mu_{\mathcal{U}})$}
%Considering again each assumption made in section \ref{s31}, we want to use the results from the previous part to approximate the laws of $s_{\text{in}}$ and $s(u_{\text{out}}, \mu_{\mathcal{U}})$:
%\begin{align*}
% s_{\text{out}} = \frac{\sum_{i=1}^{k}s(u_{\text{out}}, u_{i})}{k}
%\end{align*} which is the sum of $k$ correlated identically distributed normal distributions. The expectation of the sum is thus $\E[s_{\text{in}}]=d\times\mu^{2}$. We can compute its variance using Bienaymé's identity which states that for a set of $k$ random variables $(X_{i})_{1 \leq i \leq k}$:
%\begin{align}
%\label{eq:bienayme}
% \Var(\sum_{i=1}^k X_i)=\sum_{i=1}^k \Var(X_i)+\sum_{i,j=1 \atop i \neq j}^k \Cov(X_i,X_j)=\sum_{i,j=1}^k\Cov(X_i,X_j)
%\end{align}
%and considering $X_{i}=s(u_{\text{in}},u_{i})$, we can leverage results from \ref{eq:CLT_1} and \ref{eq:cov_1} to compute every necessary variance and covariance:
%\begin{align}
%\Var(s_{\text{out}}) &= \frac{kd(\sigma^{4}+ 2\sigma^{2}\mu^{2})+ k(k-1)d\sigma^{2}\mu^{2}}{k^2} \\
%&= d\frac{(\sigma^{4}+ (k+1)\sigma^{2}\mu^{2})}{k}
%\end{align}
%We also have
%\begin{align*}
% s_{1} = \frac{s(u_{1}, u_{1})+\sum_{i=2}^{k}s(u_{1}, u_{i})}{k}
%\end{align*} which is the sum of $k$ correlated normal distributions, and thus a normal distribution of expectation $\E[s_{\text{in}}]=d(\mu^{2}+\frac{\sigma^{2}}{k})$. Using \ref{eq:bienayme} once more along with \ref{eq:CLT_1},\ref{eq:CLT_2},\ref{eq:cov_1} and \ref{eq:cov_2}, we get the expression of its variance:
%\begin{align}
% \Var(s_{\text{in}}) &= d \times \frac{4\mu^{2}\sigma^{2}+4\mu\gamma\sigma^{3}+(\kappa-1)\sigma^{4}+ (k-1)(\sigma^{4}+2\sigma^{2}\mu^{2})+2(k-1)\mu(\gamma\sigma^{3}+2\mu\sigma^{2})+(k-1)(k-2)\sigma^{2}\mu^2}{k^2} \\
%&=\frac{d\times(k(k+3)\mu^{2}\sigma^{2}+4\mu\gamma\sigma^{3}+ (\kappa+k-2)\sigma^{4})}{k^2}
%\end{align}
%\subsubsection{Distribution of
%$s_{diff} = s_{1} - s_{\text{out}}$}
%The stability of the normal distribution implies that $s_{diff}= s(u_{1}, \mu_{\mathcal{U}}) - s(u_{\text{out}}, \mu_{\mathcal{U}})$ also follows a normal distribution with $\E[s_{diff}]=\E[s(u_{1}, \mu_{\mathcal{U}})] -\E[s(u_{\text{out}}, \mu_{\mathcal{U}})] = d\frac{\sigma^2}{k}$. To compute its variance, we can leverage the fact that
%\begin{align*}
% s_{diff} &= s(u_{1}, \mu_{\mathcal{U}}) -s(u_{\text{out}}, \mu_{\mathcal{U}}) \\
% &= \frac{s(u_{1},u_{1}) + s(u_{1},\sum_{j=2}^{k}u_{j}) -s(u_{\text{out}},u_{1}) -s(u_{\text{out}},\sum_{j=2}^{k}u_{j})}{k}\\
% &= \frac{s(X,X)+s(X,Z)+s(Y,X)+s(Y,Z)}{k}
%\end{align*}
%with $X=u_{1}, Y=-u_{\text{out}}$ and $Z=\sum_{j=2}^{k}u_{j}$.
%Since $X,Y$ and $Z$ are three independent multidimensional normal distributions, for which each dimensions are i.i.d with expectation $\mu, -\mu, (d-1)\mu$ and variance $\sigma^{2}, \sigma^{2}, (d-1)^2\sigma^{2}$, we can apply Bienaymé's inequality again together with equations \ref{eq:CLT_1}, \ref{eq:CLT_2}, \ref{eq:cov_1} and \ref{eq:cov_2} to get the variance of $s_{diff}$:
% \begin{align*}
% \Var(s_{diff})&=d\frac{(2(k-1)+\kappa)\sigma^{4} + 2k\gamma\mu\sigma^{3}+ 2k^{2}\sigma^{2}\mu^{2}}{k^2}
% \end{align*}
%We can use the c.d.f of normal distributions to compute the probability
%\begin{align*}
% &P(s_{diff} > 0) = 1-P(s_{diff} < 0) \approx \frac{1}{2}(1-\erf(-\frac{\E[s_{diff}]}{\sqrt{2\Var(s_{diff})}}))\\
% &\iff P(s(u_{1}, \mu_{\mathcal{U}}) > s(u_{\text{out}}, \mu_{\mathcal{U}})) \approx \frac{1}{2}(1-\erf(-\sqrt{\frac{d\sigma^{2}}{2((2(k-1)+\kappa)\sigma^{2} + 2k\gamma\mu\sigma+ 2k^{2}\mu^{2}}}))
%\end{align*}
%which is exactly the result written in equation \ref{eq:th_result_1}. We have thus derived a general expression for the probability for any element of $\mathcal{U}$ to be closer to $\mu_{\mathcal{U}}$, the center of $\mathcal{U}$, than a randomly chosen point from $\overline{\mathcal{U}}$. In the remainder of this section, we will focus on the particular case where $\mu=0$. Looking closer at equation \ref{eq:th_result_1}, we can see that this maximizes the considered probability (provided that the distribution is symmetrical i.e $\gamma=0)$ and that covariances of dot product \ref{eq:cov_1} and \ref{eq:cov_2} are equal to 0. In the next part we will see that these observations will help us find an explicit analytical expression for $\E_{\mathcal{U}\in\mathcal{X}}precision(\mathcal{U})$.
%\subsection{Proof of Proposition 2}
%\label{s33}
%$\forall i \in \entiers1k$, we want to find the probability $p_{(\frac{i}{k})} = P(precision(\mathcal{U}) = \frac{i}{k})$ and then compute the expectation $\mathbb{E}[precision(\mathcal{U})] = \sum_{i=1}^{k}\frac{i}{k}\times p_{(\frac{i}{k})}$. $\forall i \in \entiers{1}{k+1}$, We denote by $p_{(\frac{i}{k})}^{+}$ the probability $P(precision(\mathcal{U}) \geq \frac{i}{k})$. Because $i$ can only take integer values, we have the recurrence relationship $p_{(\frac{i}{k})} = p_{(\frac{i}{k})}^{+} -p_{(\frac{i+1}{k})}^{+}$. The desired expectation can thus be written
%\begin{align*}
% \E[precision(\mathcal{U})] = \sum_{i=1}^{k}\frac{i}{k}\times (p_{(\frac{i}{k})}^{+} -p_{(\frac{i+1}{k})}^{+}) = \frac{\sum_{i=1}^{k}p_{(\frac{i}{k})}^{+}}{k}
%\end{align*}
%By definition, $p_{(\frac{k+1}{k})}^{+}$ always equals 0. Moreover, suppose that for any $i\in \entiers1k$, we know $s_{\text{in},(i)}$, the distribution of the $i-th$ highest value of $S_{\text{in}}=\{s(u_{\text{in}}, \mu_{\mathcal{U}}), u_{\text{in}}\in\mathcal{U}\}$ as well as $s_{\text{out},(k-i+1)}$, the $(k-i+1)-th$ highest value of $S_{\text{out}}=\{s(u_{\text{out}}, \mu_{\mathcal{U}}), u_{\text{out}}\in\overline{\mathcal{U}}\}$. Then the following relationship holds:
%\begin{align*}
% P(s_{\text{in},(i)} > s_{\text{out},(k-i+1)}) = p_{(\frac{i}{k})}^{+}
%\end{align*}
%To prove it we must show that $s_{\text{in},(i)} > s_{\text{out},(k-i+1)} \iff prec(v_{\mathcal{U}}) \geq \frac{i}{k}$:
%\begin{itemize}
% \item If $s_{\text{in},(i)} > s_{\text{out},(k-i+1)}$ then there are at most $k-i$ elements from $S_{\text{out}}$ that are greater than $s_{\text{in},(i)}$. Also, by definition, there are exactly $i-1$ elements from $S_{\text{in}}$ that are greater than $s_{\text{in},(i)}$ so that makes at most $k-1$ elements greater than $s_{\text{in},(i)}$ overall. So $s_{\text{in},(i)}$ is in the top-k and so $prec(v_{\mathcal{U}}) \geq \frac{i}{k}$
% \item If $prec(v_{\mathcal{U}}) \geq \frac{i}{k}$ then there are at least $i$ elements of $S_{\text{in}}$ in the top-k, including $s_{\text{in},(i)}$. That leaves at most $k-i$ seats available in the top-k, and since by definition there are exactly $k-i$ elements of $S_{\text{out}}$ greater than $s_{\text{out},(k-i+1)}$, it is necessarily outside of the top-k and so $s_{\text{in},(i)}> s_{\text{out},(k-i+1)}$.
%\end{itemize}
%\begin{figure}
% \includegraphics[width=0.9\linewidth]{figures/s_in_out.pdf}
% \caption{Comparing $s_{\text{in},(i)}$ and $s_{\text{out},(k-i+1)}$}
% \label{fig:normal}
%\end{figure}
%We deduce $s_{\text{in}, (i)}$ by noticing that since it is the distribution of the i-th highest value of $S_{\text{in}}$, it is also the distribution of its $k-i$-th lowest value and so $s_{\text{in}, (i)}$ is the $k-i$-th order statistics of $S_{\text{in}}$. Now that $\mu = 0$, we have that given two distinct elements of $\mathcal{U}$, let's call them $u_{1}$ and $u_{2}$, $\Cov(s(u_{1}, \mu_{\mathcal{U}}), s(u_{2}, \mu_{\mathcal{U}})) = 0$, which implies that all elements of $S_{\text{in}}$ are i.i.d and thus the p.d.f of the $k-i$-th order statistics of $S_{\text{in}}$ is (reference ici?) $f_{\text{in}, (k-i)}(x) = \frac{k!}{\sigma_{\text{in}}2^{k-1}(k-i)!(i-1)!}\phi(\frac{x-\mu_{\text{in}}}{\sigma_{\text{in}}})(1+\erf(-\frac{x-\mu_{\text{in}}}{\sqrt{2}\sigma_{\text{in}}}))^{k-i}(1-\erf(-\frac{x-\mu_{\text{in}}}{\sqrt{2}\sigma_{\text{in}}}))^{i-1}$ with $\mu_{\text{in}} = d\frac{\sigma^{2}}{k}$, $\sigma_{\text{in}}= \sigma^{2}\frac{\sqrt{d(\kappa+k-2)}}{k}$ and $\phi : x \mapsto \frac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^2}dt$. Similarly, we see that $s_{\text{out}, (k-i+1)}$ is the $N-2k+i$-th order statistics of $S_{\text{out}}$ and so its c.d.f is $F_{\text{out}, (k-i+1)}(x) = \sum_{j=N-2k+i}^{N-k} \binom{N-k}{j}(1+\erf(-\frac{x}{\sqrt{2}\sigma_{\text{out}}}))^{j}(1-\erf(-\frac{x}{\sqrt{2}\sigma_{\text{out}}}))^{N-k-j}$ with $\sigma_{\text{out}} = \sigma^{2}\sqrt{\frac{d}{k}}$. Finally, we get the desired expression by noticing that $P(s_{\text{in},(i)} > s_{\text{out},(k-i+1)}) = \int_{-\infty}^{\infty}f_{\text{in},(i)}(x)\times F_{\text{out},(k-i+1)}(x)\d x$.
\section{From Theory to Practice}
\label{s4}
\subsection{Experimental Setting}
In this Section~\ref{s4}, we analyze the empirical behavior of $\text{Consistency}_k(\mathcal{X})$ on real-world data, with the aim of discussing potential discrepancies with our results from Section~\ref{s3}. Our experiments focus on three \textit{music track embeddings} datasets. As illustrated in this paper, music recommendation is especially prone to high-order concept learning: album, playlist, session, music genre, and user embeddings can all be obtained by averaging music track embeddings~\cite{briand2021semi,hansen2020contextual,schedl2018current,bendada2023scalable}.
Firstly, we consider two variants\footnote{We release our source code on GitHub to ensure the reproducibility of our experimental analysis on these two Deezer public datasets: \href{https://github.com/deezer/consistency}{https://github.com/deezer/consistency}.} of 50~000 music track embeddings publicly released by Deezer~\cite{briand2021semi}.
The first ones, denoted \textit{TT-SVD embeddings}, consist of 128-dimensional vectors obtained by factorizing a track-track (TT) pointwise mutual information matrix computing track co-occurrences in Deezer playlists, using singular value decomposition (SVD)~\cite{briand2021semi}. The second ones, denoted \textit{UT-ALS embeddings}, are 256-dimensional vectors obtained by factorizing a user-track (UT) interaction matrix, using alternating least squares~(ALS)~\cite{briand2021semi}.
Besides, we report results on \textit{2M-TT-SVD embeddings}, a private dataset of two million 128-dimensional track embeddings, extracted from Deezer's production environment. These embeddings are computed using SVD on a co-occurrence matrix comparable to the TT-SVD one. We voluntarily omit technical details for confidentiality~reasons.
\label{s41}
\subsection{Results and Discussion}
\label{s42}
\begin{figure*}[t!]
\begin{center}
\begin{subfigure}{.32\linewidth}
\includegraphics[width=\linewidth]{figures/consistency_svd.pdf}
\caption{}
\label{fig3a}
\end{subfigure} \hfill %\hfill % <-- "\hfill"
\begin{subfigure}{.32\linewidth}
\includegraphics[width=\linewidth]{figures/consistency_als.pdf}
\caption{}
\label{fig3b}
\end{subfigure} \hfill
%\medskip % create some *vertical* separation between the graphs
\begin{subfigure}{.32\linewidth}
\includegraphics[width=\linewidth]{figures/consistency_xxx.pdf}
\caption{}
\label{fig3c}
\end{subfigure}%\hfill % <-- "\hfill"
\caption{$\text{Consistency}_k(\mathcal{X})$ scores of centered versions of TT-SVD (Figure~\ref{fig3a}), UT-ALS (Figure~\ref{fig3b}), and 2M-TT-SVD embeddings (\ref{fig3c}), for $k\in\{2,\dots,50\}$ and the inner product similarity $s$. For comparison, figures also display the $\text{Consistency}_k(\mathcal{X})$ scores of embeddings generated from normal distributions, with the same dimension and number of items as the real-world embeddings under consideration. Average embeddings of real-world data are less consistent than those of data complying with our theoretical setting~from~Section~\ref{s3}.}
\label{figure3}
\end{center}
\end{figure*}
Figure~\ref{figure3} reports our evaluation of $\text{Consistency}_k(\mathcal{X})$ scores for the three music track embeddings under consideration.
%For each embedding set, we compare the consistency of its average embeddings, as measured by $\text{Consistency}_k(\mathcal{X})$, to the one of a standard normal distribution with the same dimension $d$ and number of embeddings (computed using Equation~\eqref{eq:th_result_2}).
Our experiments show that average embeddings of real-world data are less consistent for recommendation than those computed from embedding data explicitly complying with our theoretical setting from Section~\ref{s3}.
%In each setting, we observe that the consistency of real-world embeddings is significantly lower than the one of distributions studied in section \ref{s3},
Specifically, one can not ensure that the average embedding of a collection of items will remain similar to the items present in this collection. Even for low values of $k$, the consistency of average embeddings does not surpass 14\%, 6\%, and 2 \% for TT-SVD, 2M-TT-SVD, and UT-ALS embeddings, respectively.
%This means that, for all three of these datasets, even when only two embeddings are picked at random, most of the time their average is a vector whose 2 nearest neighbours aren't the selected embeddings.
Consequently, even the average of two randomly selected embeddings would likely result in a vector whose two most similar neighbors will not be the selected embeddings~themselves.
Regarding SVD-based embeddings (TT-SVD, 2M-TT-SVD), we observe that consistency scores drop as $k$ increases, as in Figure~\ref{figure2}. On the contrary, scores remain steady at around 2\% for UT-ALS embeddings. %\footnote{One possible explanation for this latter observation is to consider the degenerate case where a set of 50000 embeddings are actually made of 1000 different embeddings each identically repeated 50 times. Because each embedding would be identical to 49 other, consistency for $k=1$ would only be of 2\% and would oscillate around this value for higher values of k.}.
This phenomenon highlights the dissimilarity in distributions of embeddings generated by different representation learning algorithms (SVD, ALS). In particular, the steady consistency of UT-ALS suggests that these embeddings might be more suitable for downstream applications involving average operations on large collections, while TT-SVD embeddings might be preferable for applications with low values of $k$,
%This raises the question of how embeddings generated using optimization algorithms such as SVD or ALS are distributed, which is something we would like to study in the future.
although more studies would be required in future work~for~confirmation.
%In particular, we would like to better understand how they differ from the hypothesis of Proposition \ref{proposition2}, and what would be the consequences, in terms of similarity between embeddings
Overall, our experiments also pave the way for future work to align real-world embeddings with the setting from Section~\ref{s3}, to improve the consistency of real-world averages.
For instance, one could consider alternating SVD or ALS matrix reconstruction optimization steps with projections within the set of distributions complying with assumptions from Section~\ref{s3}.
One could also examine adding a regularization term to the optimized loss during training, e.g., the Kullback-Leibler divergence of embeddings with a pre-selected complying distribution. In particular, these strategies could help to enforce the identical distribution of embedding dimensions.
In addition, we note that, in Figure~\ref{figure3}, we computed scores on \textit{centered} embeddings. Besides being in line with Section~\ref{s3}, our tests revealed that this centering operation slightly improves the consistency of TT-SVD, UT-ALS, and 2M-TT-SVD average embeddings (albeit modifying initial similarities). Our future research will aim to further understand the impact of centering embeddings on consistencies.
%Lastly, our future research will aim to extend our theoretical analysis to~other~similarities. %and on subsets $\mathcal{U}$ where similarities between elements of $\mathcal{U}$ are high by design, e.g., playlists of similar tracks.
%and thus provide further guidance to practitioners of recommender systems who rely on averaging to compute an aggregate representation of collections of items. and thus provide further guidance to practitioners of recommender
\section{Conclusion}
\label{s5}
This short paper proposed a rigorous study of the common practice consisting of averaging item embeddings in recommender systems. We provided a mathematical analysis of the consistency of these averaging operations in a general theoretical setting, as well as an empirical evaluation on real-world data. Our results revealed that real-world averages were less consistent than those computed in our theoretical setting. This sets the stage for future research directions, discussed in this paper, toward better aligning real-world data with our theoretical assumptions. Due to the prevalence of the embedding averaging practice in industry-oriented research and applications, we believe our study and proposed directions will be insightful for the recommendation community, and could eventually lead to the improvement of embedding-based recommender systems leveraging average embeddings for representation learning.
\clearpage
\appendix
\section*{Appendix}
In this supplementary section, we report the mathematical proofs of our Propositions~\ref{proposition1}~and~\ref{proposition2} from Section~\ref{s3}.
\section{Proof of Proposition~\ref{proposition1}}
\label{s32}
%In this proof, we show that $s(u_{\text{in}}, \mu_{\mathcal{U}})$ and $s(u_{\text{out}}, \mu_{\mathcal{U}})$ approximately follow normal distributions. Then, we use the cumulative distribution of their difference, which also approximately follows a normal distribution, to obtain~Equation~\eqref{eq:th_result_1}.
%But firsthand, we will require to establish some preliminary regarding dot products of random variables in high dimensions.
\subsection{Preliminaries}
Let $X$, $Y$, and $Z$ be $d$-dimensional random variables (r.v.) composed of $d$ independent and identically distributed (i.i.d.) uni-dimensional r.v. as elements. Distributions might differ between $X$,$Y$, and $Z$. Let $\mu_{X}, \sigma^{2}_{X}, \gamma_{X}, \kappa_{X}$ be the finite mean, variance, skewness, and kurtosis of the distribution of elements of $X$, respectively. Let $\mu_{Y}, \sigma^{2}_{Y}, \mu_{Z}, \sigma^{2}_{Z}$ be the finite mean and variance of the elements of $Y$ and $Z$, respectively. When $d$ increases, the following approximations hold:
\begin{equation}
\begin{split}
s(X,Y) &\sim \mathcal{N}(\mu_{XY}, \sigma_{XY}^{2}), \\
s(X,X) &\sim \mathcal{N}(\mu_{XX}, \sigma_{XX}^{2}),
\label{gaussian}
\end{split}
\end{equation}
with:
\begin{align*}
\mu_{XY} &= d \times \mu_{X}\mu_{Y}, \\
\sigma_{XY}^{2} &= d \times ((\sigma_{X}^{2} + \mu_{X}^{2}) (\sigma_{Y}^{2} + \mu_{Y}^{2}) -\mu_{X}^{2}\mu_{Y}^{2}), \\
\mu_{XX} &= d \times (\mu^{2}_{X} + \sigma^{2}_{X}), \\
\sigma_{XX}^{2} &= d \times (4\mu_{X}^{2}\sigma_{X}^{2}+ 4\mu_{X}\gamma_{X}\sigma_{X}^{3}+ (\kappa_{X} - 1)\sigma_{X}^4).
\end{align*}
Indeed, we have $s(X,Y) = \sum_{i=1}^dX_{i}Y_{i}$. Each r.v. $X_{i}Y_{i}$ verifies $\mathbb{E}[X_{i}Y_{i}] = \mu_{X}\mu_{Y}$ and $\Var(X_{i}Y_{i})=(\sigma_{X}^{2}+\mu_{X}^{2})(\sigma_{Y}^{2}+\mu_{Y}^{2}) - \mu_{X}^{2}\mu_{Y}^{2}.$
$s(X,Y)$ being the sum of $d$ i.i.d. r.v., according to the central limit theorem~\cite{fischer2011history}, it can be approximated by a normal distribution of expectation $d \times \E[X_{i}Y_{i}]$ and variance $d \times \Var(X_{i}Y_{i})$ for large values of $d$, leading to the approximation for $s(X,Y)$ in Equation~\eqref{gaussian}. A similar reasoning leads to the approximation~for~$s(X,X)$ in this same Equation~\eqref{gaussian}. \newline
\noindent Moreover, regarding covariances of these similarities, we have:
\begin{equation}
\begin{split}
\Cov(s(X,Y), s(X,Z)) &= d\times\sigma_{X}^{2}\mu_{Y}\mu_{Z} ,\\ \Cov(s(X,X), s(X,Y)) &= d\times\mu_{Y}(\gamma_{X}\sigma_{X}^{3}+ 2\mu_{X}\sigma_{X}^{2}).
\label{covariance}
\end{split}
\end{equation}
Indeed: \begin{align*}
\Cov(s(X,Y), s(X,Z)) &= \mathbb{E}[s(X,Y)s(X,Z)] -\mathbb{E}[s(X,Y)]\mathbb{E}[s(X,Z)] \\
&=\mathbb{E}[(\sum_{i=1}^{d}X_{i}Y_{i})(\sum_{j=1}^{d}X_{j}Y_{j})] - d \times \mu_{X}\mu_{Y} \times d \times \mu_{X}\mu_{Z} \\
&= \allowbreak \mathbb{E}[\sum_{i=1}^{d}(X_{i}Y_{i}X_{i}Z_{i} + \sum_{j=1 \atop j\neq i}^{d}X_{i}Y_{i}X_{j}Z_{j})] - d^2 \times \mu_{X}^2\mu_{Y}\mu_{Z} \\
&= \allowbreak \sum_{i=1}^{d} (\mathbb{E}[X_{i}^2]\mathbb{E}[Y_{i}]\mathbb{E}[Z_{i}] \allowbreak + \sum_{j=1 \atop j\neq i}^{d}(\mathbb{E}[X_{i}] \mathbb{E}[Y_{i}] \mathbb{E}[X_{j}] \mathbb{E}[Z_{j}])) - d^2 \times \mu_{X}^2\mu_{Y}\mu_{Z} \\ &= d \times((\sigma_{X}^2+\mu_{X}^2)\mu_{Y}\mu_{Z} + (d-1)(\mu_{X}^2\mu_{Y}\mu_{Z}))- d^2 \times \mu_{X}^2\mu_{Y}\mu_{Z} \\
&= d \times \sigma_{X}^2\mu_{Y}\mu_{Z}.\end{align*}
We obtain the second covariance of Equation~\eqref{covariance} with similar computations. Notice how $\mu_{Y}=0$ or $\mu_{Z}=0$ implies that similarities are uncorrelated and, therefore, also independent when they follow multivariate normal distributions~\cite{jacod2004probability}.
\subsection{Distribution of $s_{\text{in}} = s(u_{\text{in}}, \mu_{\mathcal{U}})$ and $s_{\text{out}}=s(u_{\text{out}}, \mu_{\mathcal{U}})$} Using the distributive property of the inner product $s$, we~have:
$$s_{\text{out}} = \frac{\sum\limits_{i=1}^{k}s(u_{\text{out}}, u_{i})}{k},$$
i.e., the sum of $k$ correlated identically distributed normal distributions. The expectation of the sum is thus:
$$\mathbb{E}[s_{\text{out}}]=d\times\mu^{2}.$$
We compute its variance using Bienaymé's identity~\cite{loeve1977probability} stating that, for $k$ r.v. $(A_{i})_{1 \leq i \leq k}$, then:
$$\Var(\sum_{i=1}^k A_i)=\sum_{i=1}^k \Var(A_i)+\sum_{i,j=1 \atop i \neq j}^k \Cov(A_i,A_j)=\sum_{i,j=1}^k\Cov(A_i,A_j).$$ Also considering $A_{i}=s(u_{\text{out}},u_{i})$, and using the first half of both Equations~\eqref{gaussian}~and~\eqref{covariance}, we obtain:
\begin{align*}
\Var(s_{\text{out}}) &= \frac{kd(\sigma^{4}+ 2\sigma^{2}\mu^{2})+ k(k-1)d\sigma^{2}\mu^{2}}{k^2} \\
&= d\frac{(\sigma^{4}+ (k+1)\sigma^{2}\mu^{2})}{k}.\end{align*} \newline
\noindent Besides, under preliminary approximations:
$$s_{\text{in}} = \frac{s(u_{\text{in}}, u_{\text{in}})+\sum_{i\neq \text{in}} s(u_{\text{in}}, u_{i})}{k}$$ is the sum of $k$ correlated normal distributions, i.e., a normal distribution with:
$$\mathbb{E}[s_{\text{in}}]=d(\mu^{2}+\frac{\sigma^{2}}{k}).$$
Using Bienaymé's identity and Equations~\eqref{gaussian}~and~\eqref{covariance}, we get:
\begin{align*}
\Var(s_{\text{in}})
&= d \times \frac{4\mu^{2}\sigma^{2}+4\mu\gamma\sigma^{3}+(\kappa-1)\sigma^{4}+ (k-1)(\sigma^{4}+2\sigma^{2}\mu^{2})+2(k-1)\mu(\gamma\sigma^{3}+2\mu\sigma^{2})+(k-1)(k-2)\sigma^{2}\mu^2}{k^2} \\
&=\frac{d\times(k(k+3)\mu^{2}\sigma^{2}+4\mu\gamma\sigma^{3}+ (\kappa+k-2)\sigma^{4})}{k^2}.
\end{align*}
\subsection{Distribution of
$s_{\text{diff}} = s_{\text{in}} - s_{\text{out}}$}
Using preliminary approximations, and since the difference of two normally distributed r.v. is also normally distributed~\cite{jacod2004probability}, we obtain that $s_{\text{diff}}= s(u_{\text{in}}, \mu_{\mathcal{U}}) - s(u_{\text{out}}, \mu_{\mathcal{U}})$ follows a normal distribution with:
\begin{align*}
\mathbb{E}[s_{\text{diff}}] &=\mathbb{E}[s(u_{\text{in}}, \mu_{\mathcal{U}})] -\mathbb{E}[s(u_{\text{out}}, \mu_{\mathcal{U}})] \\ &= d\frac{\sigma^2}{k}.\end{align*}
To compute its variance, we remark that:
\begin{align*}
s_{\text{diff}} &= s(u_{\text{in}}, \mu_{\mathcal{U}}) -s(u_{\text{out}}, \mu_{\mathcal{U}}) \\
&= \frac{1}{k}(s(u_{\text{in}},u_{\text{in}}) + s(u_{\text{in}},\sum_{j\neq \text{in}}^{k}u_{j}) -s(u_{\text{out}},u_{\text{in}}) -s(u_{\text{out}},\sum_{j\neq \text{in}}u_{j})) \\
&= \frac{s(X,X)+s(X,Z)+s(Y,X)+s(Y,Z)}{k}, \end{align*} with $X=u_{\text{in}}$, $Y=-u_{\text{out}}$ and $Z=\sum_{j\neq \text{in}}u_{j}.$
Since $X$, $Y$, and $Z$ are three independent multidimensional normal distributions, for which dimensions are i.i.d. with expectation $\mu, -\mu, (d-1)\mu$ and variance $\sigma^{2}, \sigma^{2}, (d-1)^2\sigma^{2}$, we once again use Bienaymé's identity with Equations~\eqref{gaussian}~and~\eqref{covariance} to obtain:
$$\Var(s_{\text{diff}})=d\frac{(2(k-1)+\kappa)\sigma^{4} + 2k\gamma\mu\sigma^{3}+ 2k^{2}\sigma^{2}\mu^{2}}{k^2}.$$
Finally, using the cumulative distribution function of normal distributions~\cite{jacod2004probability}, we get Equation~\eqref{eq:th_result_1}:
\begin{align*}
&\mathbb{P}(s_{\text{diff}} > 0) = 1-\mathbb{P}(s_{\text{diff}}\leq0) = \frac{1}{2}(1-\erf(-\frac{\mathbb{E}[s_{\text{diff}}]} {\sqrt{2\Var(s_{\text{diff}})}})) \\ \iff
&\mathbb{P}(s(u_{\text{in}}, \mu_{\mathcal{U}}) > s(u_{\text{out}}, \mu_{\mathcal{U}})) = \frac{1}{2}(1+\erf(\sqrt{\frac{d\sigma^{2}}{2((2(k-1)+\kappa)\sigma^{2} + 2k\gamma\mu\sigma+ 2k^{2}\mu^{2})}})).
\end{align*} \newline
%This is i.e., a general expression for the probability for any element of $\mathcal{U}$ to be closer to $\mu_{\mathcal{U}}$, the center of $\mathcal{U}$, than a random point from $\overline{\mathcal{U}}$. The next proof focuses on the setting $\mu=0$, which maximizes~this~probability.
%In the next proof, we will focus on the particular case where $\mu=0$. Looking closely at Equation~\eqref{eq:th_result_1}, we observe that this case maximizes the considered probability, provided that the distribution is symmetrical, i.e., that $\gamma=0$, and that the above covariances of inner products are equal to 0. These observations will help us compute an analytical expression for~$\text{Consistency}_k(\mathcal{X})$.
\section{Proof of Proposition~\ref{proposition2}}
\label{s33}
This proof starts by evaluating $p_{(\frac{i}{k})} = \mathbb{P}(\text{Precision}_k(\mathcal{U})) = \frac{i}{k})$ for $i \in \{1,\dots,k\}$, and then computes $\text{Consistency}_k(\mathcal{X}) = \mathbb{E}[\text{Precision}_k(\mathcal{U})] = \sum_{i=1}^{k}\frac{i}{k}\times p_{(\frac{i}{k})}$ under the studied hypotheses. To begin with, we set:
$$p_{(\frac{i}{k})}^{+} = \mathbb{P}(\text{Precision}_k(\mathcal{U}) \geq \frac{i}{k}), \text{ for } i \in \{1,\dots,k+1\}.$$
As $i$ takes integer values:
$$p_{(\frac{i}{k})} = p_{(\frac{i}{k})}^{+} -p_{(\frac{i+1}{k})}^{+}.$$
Assuming $p_{(\frac{k+1}{k})}^{+} = 0$ by definition, we have:
\begin{equation}
\mathbb{E}\Big[\text{Precision}_k(\mathcal{U})\Big] = \sum_{i=1}^{k}\frac{i}{k}\times (p_{(\frac{i}{k})}^{+} -p_{(\frac{i+1}{k})}^{+}) = \frac{1}{k}\sum_{i=1}^{k}p_{(\frac{i}{k})}^{+}.
\label{eq-proof2-1}
\end{equation}
Let us denote by $s_{\text{in},(i)}$ the distribution of the $i$\up{th} highest value of $S_{\text{in}}=\{s(u_{\text{in}}, \mu_{\mathcal{U}}), u_{\text{in}}\in\mathcal{U}\}$, and by $s_{\text{out},(k-i+1)}$ the $(k-i+1)$\up{th} highest value of $S_{\text{out}}=\{s(u_{\text{out}}, \mu_{\mathcal{U}}), u_{\text{out}}\in\overline{\mathcal{U}}\}$, for $i \in \{1,\dots,k\}$. Using this notation, we observe that:
$$p_{(\frac{i}{k})}^{+} = \mathbb{P}(s_{\text{in},(i)} > s_{\text{out},(k-i+1)}).$$ Indeed, as illustrated in Figure~\ref{fig:appendix}, we can show that $s_{\text{in},(i)} > s_{\text{out},(k-i+1)} \iff \text{Precision}_k(\mathcal{U})~\geq~\frac{i}{k}$:
\begin{itemize}
\item If $s_{\text{in},(i)} > s_{\text{out},(k-i+1)}$ then at most $k-i$ elements of $S_{\text{out}}$ are greater than $s_{\text{in},(i)}$. Also, by definition, exactly $i-1$ elements of $S_{\text{in}}$ are greater than $s_{\text{in},(i)}$. So, at most $k-1$ similarities are greater than $s_{\text{in},(i)}$ overall. Thus, $s_{\text{in},(i)}$ is one of the $k$ highest values of $S_{\text{in}} \bigcup S_{\text{out}}$ and $\text{Precision}_k(\mathcal{U}) \geq \frac{i}{k}$.
\item If $\text{Precision}_k(\mathcal{U}) \geq \frac{i}{k}$, then at least $i$ elements of $S_{\text{in}}$ are in the top-$k$ of $S_{\text{in}}\bigcup S_{\text{out}}$, including $s_{\text{in},(i)}$. That leaves at most $k-i$ slots available in the top-$k$. Since, by definition, exactly $k-i$ elements of $S_{\text{out}}$ are greater than $s_{\text{out},(k-i+1)}$, $s_{\text{out},(k-i+1)}$ is necessarily outside of the top-$k$ elements of $S_{\text{in}}\bigcup S_{\text{out}}$, and so $s_{\text{in},(i)}> s_{\text{out},(k-i+1)}$.
\end{itemize}
%\begin{figure}
% \includegraphics[width=0.9\linewidth]{figures/s_in_out.pdf}
% \caption{Comparing $s_{\text{in},(i)}$ and $s_{\text{out},(k-i+1)}$}
% \label{fig:normal}
%\end{figure}
\begin{figure}[t]
\centering
\includegraphics[width=0.4\textwidth]{figures/inout.pdf}
\caption{Ranking $s_{\text{in}}$ and $s_{\text{out}}$ statistics.}
\label{fig:appendix}
\end{figure}
\noindent In our setting, we derive $s_{\text{in}, (i)}$ by noticing that the distribution of the i\up{th} highest value of $S_{\text{in}}$ is also the distribution of its $(k-i)$\up{th} lowest value. Hence, $s_{\text{in}, (i)}$ is the $(k-i)$\up{th} order statistic of $S_{\text{in}}$. As $\mu = 0$, we have that, given two distinct elements of $\mathcal{U}$, say $u_{1}$ and $u_{2}$, $\Cov(s(u_{1}, \mu_{\mathcal{U}}), s(u_{2}, \mu_{\mathcal{U}})) = 0$, which implies that all elements of $S_{\text{in}}$ are i.i.d. and that the probability density function of the $(k-i)$\up{th} order statistic of $S_{\text{in}}$ is \cite{nagaraja2003order}:
$$f_{\text{in}, (k-i)}(x) = \frac{k!}{\sigma_{\text{in}}2^{k-1}(k-i)!(i-1)!}\phi(\frac{x-\mu_{\text{in}}}{\sigma_{\text{in}}})(1+\erf(-\frac{x-\mu_{\text{in}}}{\sqrt{2}\sigma_{\text{in}}}))^{k-i}(1-\erf(-\frac{x-\mu_{\text{in}}}{\sqrt{2}\sigma_{\text{in}}}))^{i-1},$$
with:
\begin{align*}
\mu_{\text{in}} &= d\frac{\sigma^{2}}{k}, \\ \sigma_{\text{in}} &= \sigma^{2}\frac{\sqrt{d(\kappa+k-2)}}{k}, \\ \phi &: x \mapsto \frac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^2}dt.
\end{align*}
Similarly, $s_{\text{out}, (k-i+1)}$ is the $(N-2k+i)$\up{th} order statistic of $S_{\text{out}}$ and so its cumulative density function is~\cite{nagaraja2003order}:
$$F_{\text{out}, (k-i+1)}(x) = \sum_{j=N-2k+i}^{N-k} \binom{N-k}{j}(1+\erf(-\frac{x}{\sqrt{2}\sigma_{\text{out}}}))^{j}(1-\erf(-\frac{x}{\sqrt{2}\sigma_{\text{out}}}))^{N-k-j},$$
with:
$$\sigma_{\text{out}} = \sigma^{2}\sqrt{\frac{d}{k}}.$$ Finally, we obtain:
\begin{equation}
\begin{split}
p_{(\frac{i}{k})}^{+} &= \mathbb{P}(s_{\text{in},(i)} > s_{\text{out},(k-i+1)}) \\
&= \int_{-\infty}^{\infty}f_{\text{in},(i)}(x)\times F_{\text{out},(k-i+1)}(x)\d x,
\label{eq-proof2-2}
\end{split}
\end{equation}
and, by injecting Equation~\eqref{eq-proof2-2} into Equation~\eqref{eq-proof2-1}, we retrieve the expression from Equation~\eqref{eq:th_result_2} of Proposition~\ref{proposition2}.
\clearpage
\bibliographystyle{ACM-Reference-Format}
\bibliography{references}
\end{document}
\endinput
%%% -*-BibTeX-*-
%%% Do NOT edit. File created by BibTeX with style
%%% ACM-Reference-Format-Journals [18-Jan-2012].
\begin{thebibliography}{30}
%%% ====================================================================
%%% NOTE TO THE USER: you can override these defaults by providing
%%% customized versions of any of these macros before the \bibliography
%%% command. Each of them MUST provide its own final punctuation,
%%% except for \shownote{}, \showDOI{}, and \showURL{}. The latter two
%%% do not use final punctuation, in order to avoid confusing it with
%%% the Web address.
%%%
%%% To suppress output of a particular field, define its macro to expand
%%% to an empty string, or better, \unskip, like this:
%%%
%%% \newcommand{\showDOI}[1]{\unskip} % LaTeX syntax
%%%
%%% \def \showDOI #1{\unskip} % plain TeX syntax
%%%
%%% ====================================================================
\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi
\ifx \showDOI \undefined \def \showDOI #1{#1}\fi
\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi
\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi
\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi
\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi
\ifx \shownote \undefined \def \shownote #1{#1} \fi
\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi
\ifx \showURL \undefined \def \showURL {\relax} \fi
% The following commands are used for tagged output and should be
% invisible to TeX
\providecommand\bibfield[2]{#2}
\providecommand\bibinfo[2]{#2}
\providecommand\natexlab[1]{#1}
\providecommand\showeprint[2][]{arXiv:#2}
\bibitem[Andrews(1998)]%
{andrews1998special}
\bibfield{author}{\bibinfo{person}{Larry~C Andrews}.}
\bibinfo{year}{1998}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{Special Functions of Mathematics for
Engineers}}. Vol.~\bibinfo{volume}{49}.
\newblock \bibinfo{publisher}{Spie Press}.
\newblock
\bibitem[Arora et~al\mbox{.}(2017)]%
{arora2017simple}
\bibfield{author}{\bibinfo{person}{Sanjeev Arora}, \bibinfo{person}{Yingyu
Liang}, {and} \bibinfo{person}{Tengyu Ma}.} \bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{A Simple but Tough-To-Beat Baseline for Sentence
Embeddings}. In \bibinfo{booktitle}{\emph{Proceedings of the 5th
International Conference on Learning Representations}}.
\newblock
\bibitem[Bendada et~al\mbox{.}(2023)]%
{bendada2023scalable}
\bibfield{author}{\bibinfo{person}{Walid Bendada}, \bibinfo{person}{Guillaume
Salha-Galvan}, \bibinfo{person}{Thomas Bouab\c{c}a}, {and}
\bibinfo{person}{Tristan Cazenave}.} \bibinfo{year}{2023}\natexlab{}.
\newblock \showarticletitle{A Scalable Framework for Automatic Playlist
Continuation on Music Streaming Services}. In
\bibinfo{booktitle}{\emph{Proceedings of the 46th Int. ACM SIGIR Conf. on
Research and Development in Information Retrieval}}.
\bibinfo{pages}{464--474}.
\newblock
\bibitem[Briand et~al\mbox{.}(2021)]%
{briand2021semi}
\bibfield{author}{\bibinfo{person}{L{\'e}a Briand}, \bibinfo{person}{Guillaume
Salha-Galvan}, \bibinfo{person}{Walid Bendada}, \bibinfo{person}{Mathieu
Morlon}, {and} \bibinfo{person}{Viet-Anh Tran}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{A Semi-Personalized System for User Cold Start
Recommendation on Music Streaming Apps}. In
\bibinfo{booktitle}{\emph{Proceedings of the 27th ACM SIGKDD Conference on
Knowledge Discovery and Data Mining}}. \bibinfo{pages}{2601--2609}.
\newblock
\bibitem[Coleman(2020)]%
{blog}
\bibfield{author}{\bibinfo{person}{Ben Coleman}.}
\bibinfo{year}{2020}\natexlab{}.
\newblock \bibinfo{title}{Why is it Okay to Average Embeddings?}
\newblock \bibinfo{howpublished}{Technical post on:
\url{https://randorithms.com/2020/11/17/Adding-Embeddings.html}}.
\newblock
\bibitem[Covington et~al\mbox{.}(2016)]%
{youtube}
\bibfield{author}{\bibinfo{person}{Paul Covington}, \bibinfo{person}{Jay
Adams}, {and} \bibinfo{person}{Emre Sargin}.}
\bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{Deep Neural Networks for Youtube Recommendations}.
In \bibinfo{booktitle}{\emph{Proceedings of the 10th ACM Conference on
Recommender Systems}}. \bibinfo{pages}{191--198}.
\newblock
\bibitem[Fischer(2011)]%
{fischer2011history}
\bibfield{author}{\bibinfo{person}{Hans Fischer}.}
\bibinfo{year}{2011}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{A History of the Central Limit Theorem:
from Classical to Modern Probability Theory}}.
\newblock \bibinfo{publisher}{Springer}.
\newblock
\bibitem[Grbovic and Cheng(2018)]%
{grbovic2018real}
\bibfield{author}{\bibinfo{person}{Mihajlo Grbovic} {and}
\bibinfo{person}{Haibin Cheng}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Real-Time Personalization using Embeddings for
Search Ranking at Airbnb}. In \bibinfo{booktitle}{\emph{Proceedings of the
24th ACM SIGKDD International Conference on Knowledge Discovery \& Data
Mining}}. \bibinfo{pages}{311--320}.
\newblock
\bibitem[Hansen et~al\mbox{.}(2020)]%
{hansen2020contextual}
\bibfield{author}{\bibinfo{person}{Casper Hansen}, \bibinfo{person}{Christian
Hansen}, \bibinfo{person}{Lucas Maystre}, \bibinfo{person}{Rishabh Mehrotra},
\bibinfo{person}{Brian Brost}, \bibinfo{person}{Federico Tomasi}, {and}
\bibinfo{person}{Mounia Lalmas}.} \bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{Contextual and Sequential User Embeddings for
Large-Scale Music Recommendation}. In \bibinfo{booktitle}{\emph{Proceedings
of the 14th ACM Conference on Recommender Systems}}. \bibinfo{pages}{53--62}.
\newblock
\bibitem[Hassanieh et~al\mbox{.}(2018)]%
{al2018similarity}
\bibfield{author}{\bibinfo{person}{Lamis Hassanieh}, \bibinfo{person}{Chadi
Abou~Jaoudeh}, \bibinfo{person}{Jacques~Bou Abdo}, {and}
\bibinfo{person}{Jacques Demerjian}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Similarity Measures for Collaborative Filtering
Recommender Systems}. In \bibinfo{booktitle}{\emph{Proceedings of the 2018
IEEE Middle East and North Africa Communications Conference}}. IEEE,
\bibinfo{pages}{1--5}.
\newblock
\bibitem[He et~al\mbox{.}(2017)]%
{he2017neural}
\bibfield{author}{\bibinfo{person}{Xiangnan He}, \bibinfo{person}{Lizi Liao},
\bibinfo{person}{Hanwang Zhang}, \bibinfo{person}{Liqiang Nie},
\bibinfo{person}{Xia Hu}, {and} \bibinfo{person}{Tat-Seng Chua}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Neural Collaborative Filtering}. In
\bibinfo{booktitle}{\emph{Proceedings of the 26th International Conference on
World Wide Web}}. \bibinfo{pages}{173--182}.
\newblock
\bibitem[Hidasi et~al\mbox{.}(2015)]%
{hidasi2015session}
\bibfield{author}{\bibinfo{person}{Bal{\'a}zs Hidasi},
\bibinfo{person}{Alexandros Karatzoglou}, \bibinfo{person}{Linas Baltrunas},
{and} \bibinfo{person}{Domonkos Tikk}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{Session-Based Recommendations with Recurrent Neural
Networks}.
\newblock \bibinfo{journal}{\emph{Proceedings of the 4th International
Conference on Learning Representations}} (\bibinfo{year}{2015}).
\newblock
\bibitem[Jacod and Protter(2004)]%
{jacod2004probability}
\bibfield{author}{\bibinfo{person}{Jean Jacod} {and} \bibinfo{person}{Philip
Protter}.} \bibinfo{year}{2004}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{Probability Essentials}}.
\newblock \bibinfo{publisher}{Springer Science \& Business Media}.
\newblock
\bibitem[Jain et~al\mbox{.}(2020)]%
{jain2020survey}
\bibfield{author}{\bibinfo{person}{Gourav Jain}, \bibinfo{person}{Tripti
Mahara}, {and} \bibinfo{person}{Kuldeep~Narayan Tripathi}.}
\bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{A Survey of Similarity Measures for Collaborative
Filtering-Based Recommender System}.
\newblock In \bibinfo{booktitle}{\emph{Soft Computing: Theories and
Applications}}. \bibinfo{publisher}{Springer}, \bibinfo{pages}{343--352}.
\newblock
\bibitem[Javed et~al\mbox{.}(2021)]%
{javed2021review}
\bibfield{author}{\bibinfo{person}{Umair Javed}, \bibinfo{person}{Kamran
Shaukat}, \bibinfo{person}{Ibrahim~A Hameed}, \bibinfo{person}{Farhat Iqbal},
\bibinfo{person}{Talha~Mahboob Alam}, {and} \bibinfo{person}{Suhuai Luo}.}
\bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{A Review of Content-Based and Context-Based
Recommendation Systems}.
\newblock \bibinfo{journal}{\emph{International Journal of Emerging
Technologies in Learning}} \bibinfo{volume}{16}, \bibinfo{number}{3}
(\bibinfo{year}{2021}), \bibinfo{pages}{274--306}.
\newblock
\bibitem[Koren and Bell(2015)]%
{koren2015advances}
\bibfield{author}{\bibinfo{person}{Yehuda Koren} {and} \bibinfo{person}{Robert
Bell}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{{Advances in Collaborative Filtering}}.
\newblock \bibinfo{journal}{\emph{Recommender Systems Handbook}}
(\bibinfo{year}{2015}), \bibinfo{pages}{77--118}.
\newblock
\bibitem[Lo{\`e}ve(1977)]%
{loeve1977probability}
\bibfield{author}{\bibinfo{person}{Michel Lo{\`e}ve}.}
\bibinfo{year}{1977}\natexlab{}.
\newblock \showarticletitle{Probability Theory I}.
\newblock \bibinfo{journal}{\emph{Graduate Texts in Mathematics}}
\bibinfo{volume}{4} (\bibinfo{year}{1977}).
\newblock
\bibitem[Mishra and Viradiya(2019)]%
{mishra2019survey}
\bibfield{author}{\bibinfo{person}{Mridul~K Mishra} {and}
\bibinfo{person}{Jaydeep Viradiya}.} \bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{Survey of Sentence Embedding Methods}.
\newblock \bibinfo{journal}{\emph{International Journal of Applied Science and
Computations}} \bibinfo{volume}{6}, \bibinfo{number}{3}
(\bibinfo{year}{2019}), \bibinfo{pages}{592--592}.
\newblock
\bibitem[Nagaraja and David(2003)]%
{nagaraja2003order}
\bibfield{author}{\bibinfo{person}{HN Nagaraja} {and} \bibinfo{person}{HA
David}.} \bibinfo{year}{2003}\natexlab{}.
\newblock \bibinfo{booktitle}{\emph{Order Statistics}}.
\newblock \bibinfo{publisher}{John Wiley \& Sons, Inc}.
\newblock
\bibitem[Okura et~al\mbox{.}(2017)]%
{okura2017embedding}
\bibfield{author}{\bibinfo{person}{Shumpei Okura}, \bibinfo{person}{Yukihiro
Tagami}, \bibinfo{person}{Shingo Ono}, {and} \bibinfo{person}{Akira Tajima}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Embedding-Based News Recommendation for Millions of
Users}. In \bibinfo{booktitle}{\emph{Proceedings of the 23rd ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining}}.
\bibinfo{pages}{1933--1942}.
\newblock
\bibitem[Pal et~al\mbox{.}(2020)]%
{pal2020pinnersage}
\bibfield{author}{\bibinfo{person}{Aditya Pal}, \bibinfo{person}{Chantat
Eksombatchai}, \bibinfo{person}{Yitong Zhou}, \bibinfo{person}{Bo Zhao},
\bibinfo{person}{Charles Rosenberg}, {and} \bibinfo{person}{Jure Leskovec}.}
\bibinfo{year}{2020}\natexlab{}.
\newblock \showarticletitle{PinnerSage: Multi-Modal User Embedding Framework
for Recommendations at Pinterest}. In \bibinfo{booktitle}{\emph{Proceedings
of the 26th ACM SIGKDD International Conference on Knowledge Discovery \&
Data Mining}}. \bibinfo{pages}{2311--2320}.
\newblock
\bibitem[Park et~al\mbox{.}(2018)]%
{park2018collaborative}
\bibfield{author}{\bibinfo{person}{Chanyoung Park}, \bibinfo{person}{Donghyun
Kim}, \bibinfo{person}{Xing Xie}, {and} \bibinfo{person}{Hwanjo Yu}.}
\bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Collaborative Translational Metric Learning}. In
\bibinfo{booktitle}{\emph{Proceedings of the 2018 IEEE International
Conference on Data Mining}}. IEEE, \bibinfo{pages}{367--376}.
\newblock
\bibitem[Schedl et~al\mbox{.}(2018)]%
{schedl2018current}
\bibfield{author}{\bibinfo{person}{Markus Schedl}, \bibinfo{person}{Hamed
Zamani}, \bibinfo{person}{Ching-Wei Chen}, \bibinfo{person}{Yashar Deldjoo},
{and} \bibinfo{person}{Mehdi Elahi}.} \bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{{Current Challenges and Visions in Music
Recommender Systems Research}}.
\newblock \bibinfo{journal}{\emph{International Journal of Multimedia
Information Retrieval}} \bibinfo{volume}{7}, \bibinfo{number}{2}
(\bibinfo{year}{2018}), \bibinfo{pages}{95--116}.
\newblock
\bibitem[Tran et~al\mbox{.}(2021)]%
{tran2021hierarchical}
\bibfield{author}{\bibinfo{person}{Viet-Anh Tran}, \bibinfo{person}{Guillaume
Salha-Galvan}, \bibinfo{person}{Romain Hennequin}, {and}
\bibinfo{person}{Manuel Moussallam}.} \bibinfo{year}{2021}\natexlab{}.
\newblock \showarticletitle{Hierarchical Latent Relation Modeling for
Collaborative Metric Learning}. In \bibinfo{booktitle}{\emph{Proceedings of
the 15th ACM Conference on Recommender Systems}}. \bibinfo{pages}{302--309}.
\newblock
\bibitem[Wang et~al\mbox{.}(2018)]%
{wang2018billion}
\bibfield{author}{\bibinfo{person}{Jizhe Wang}, \bibinfo{person}{Pipei Huang},
\bibinfo{person}{Huan Zhao}, \bibinfo{person}{Zhibo Zhang},
\bibinfo{person}{Binqiang Zhao}, {and} \bibinfo{person}{Dik~Lun Lee}.}
\bibinfo{year}{2018}\natexlab{}.
\newblock \showarticletitle{Billion-Scale Commodity Embedding for E-Commerce
Recommendation in Alibaba}. In \bibinfo{booktitle}{\emph{Proceedings of the
24th ACM SIGKDD International Conference on Knowledge Discovery \& Data
Mining}}. \bibinfo{pages}{839--848}.
\newblock
\bibitem[Westfall(2014)]%
{westfall2014kurtosis}
\bibfield{author}{\bibinfo{person}{Peter~H Westfall}.}
\bibinfo{year}{2014}\natexlab{}.
\newblock \showarticletitle{Kurtosis as peakedness, 1905--2014. RIP}.
\newblock \bibinfo{journal}{\emph{The American Statistician}}
\bibinfo{volume}{68}, \bibinfo{number}{3} (\bibinfo{year}{2014}),
\bibinfo{pages}{191--195}.
\newblock
\bibitem[White et~al\mbox{.}(2015)]%
{white2015well}
\bibfield{author}{\bibinfo{person}{Lyndon White}, \bibinfo{person}{Roberto
Togneri}, \bibinfo{person}{Wei Liu}, {and} \bibinfo{person}{Mohammed
Bennamoun}.} \bibinfo{year}{2015}\natexlab{}.
\newblock \showarticletitle{How Well Sentence Embeddings Capture Meaning}. In
\bibinfo{booktitle}{\emph{Proceedings of the 20th Australasian Document
Computing Symposium}}. \bibinfo{pages}{1--8}.
\newblock
\bibitem[Wu and Yan(2017)]%
{wu2017session}
\bibfield{author}{\bibinfo{person}{Chen Wu} {and} \bibinfo{person}{Ming Yan}.}
\bibinfo{year}{2017}\natexlab{}.
\newblock \showarticletitle{Session-Aware Information Embedding for E-Commerce
Product Recommendation}. In \bibinfo{booktitle}{\emph{Proceedings of the 2017
ACM on Conference on Information and Knowledge Management}}.
\bibinfo{pages}{2379--2382}.
\newblock
\bibitem[Yang et~al\mbox{.}(2016)]%
{yang2016survey}
\bibfield{author}{\bibinfo{person}{Zhe Yang}, \bibinfo{person}{Bing Wu},
\bibinfo{person}{Kan Zheng}, \bibinfo{person}{Xianbin Wang}, {and}
\bibinfo{person}{Lei Lei}.} \bibinfo{year}{2016}\natexlab{}.
\newblock \showarticletitle{A Survey of Collaborative Filtering-Based
Recommender Systems for Mobile Internet Applications}.
\newblock \bibinfo{journal}{\emph{IEEE Access}} \bibinfo{volume}{4}
(\bibinfo{year}{2016}), \bibinfo{pages}{3273--3287}.
\newblock
\bibitem[Zhang et~al\mbox{.}(2019)]%
{zhang2019deep}
\bibfield{author}{\bibinfo{person}{Shuai Zhang}, \bibinfo{person}{Lina Yao},
\bibinfo{person}{Aixin Sun}, {and} \bibinfo{person}{Yi Tay}.}
\bibinfo{year}{2019}\natexlab{}.
\newblock \showarticletitle{{Deep Learning Based Recommender System: A Survey
and New Perspectives}}.
\newblock \bibinfo{journal}{\emph{ACM Computing Surveys (CSUR)}}
\bibinfo{volume}{52}, \bibinfo{number}{1} (\bibinfo{year}{2019}),
\bibinfo{pages}{1--38}.
\newblock
\end{thebibliography}