Metric Space Magnitude for Evaluating the Diversity of Latent Representations
Extracted source text · arXiv 2311.16054
\newif\ifarxiv
\arxivtrue
\ifarxiv
\documentclass[10pt, DIV=12, enabledeprecatedfontcommands]{scrartcl}
\usepackage{authblk}
\usepackage{etoolbox}
\usepackage{natbib}
\usepackage{mathpazo}
\linespread{1.05}
\renewcommand\Authfont{\small\bfseries}
\renewcommand\Affilfont{\small\normalfont}
\renewcommand\Authsep{\quad}
\renewcommand\Authands{\quad}
\setkomafont{sectioning}{\bfseries}
\setkomafont{author}{\small}
\makeatletter
\patchcmd{\@maketitle}{\titlefont\huge}{\titlefont\small}{}{}
\makeatother
\else
\documentclass{article}
\usepackage{iclr2024_conference}
\fi
\usepackage{times}
\usepackage {amsfonts}
\usepackage {amsmath}
\usepackage {amssymb}
\usepackage {amsthm}
\usepackage {bbm}
\usepackage[british]{babel}
\usepackage {booktabs}
\usepackage {comment}
\usepackage {dsfont}
\usepackage {mathtools}
\usepackage {microtype}
\usepackage {nicefrac}
\usepackage {paralist}
\usepackage {subcaption}
\usepackage {siunitx}
\usepackage {wrapfig}
\usepackage {thm-restate}
\usepackage{rotating}
\usepackage{tikz}
\usepackage{pgfplots}
\usepgfplotslibrary{colorbrewer}
\pgfplotsset{
compat = 1.17,
}
\usepackage{hyperref}
\usepackage[capitalise]{cleveref}
\hypersetup{
colorlinks,
%
urlcolor = blue,
citecolor = blue,
linkcolor = blue,
}
% ------------ THEOREM STYLING --------------------------------------------------------
\newtheorem{theorem} {Theorem} [section]
\newtheorem{lemma} {Lemma} [section]
\newtheorem{corollary} {Corollary} [section]
\newtheorem{definition}{Definition}[section]
\newtheorem{example} {Example} [section]
\DeclareMathOperator{\magnitude}{Mag}
\newcommand{\ones} {\mathds{1}}
\newcommand{\reals}{\mathds{R}}
\newcommand{\rayna}[1]{{\bf\textcolor{blue}{ Rayna: #1}}}
\newcommand{\kat}[1]{{\bf\textcolor{violet}{ Kat: #1}}}
\newcommand{\bastian}[1]{{\bf\textcolor{red}{Bastian: #1}}}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\ifarxiv
\title{
Metric Space Magnitude for Evaluating\\
Unsupervised Representation Learning
% Was: Metric space magnitude improves unsupervised dimensionality reduction
}
\date{}
\author[1, 2, $\ast$]{Katharina Limbeck}
\author[3, $\ast$]{Rayna Andreeva}
\author[3]{Rik Sarkar}
\author[1, 2]{Bastian~Rieck}
\affil[1]{Institute of AI for Health, Helmholtz Zentrum München}
\affil[2]{School of Computation, Information and Technology, Technical University of Munich}
\affil[3]{School of Informatics, University of Edinburgh}
\affil[$\ast$]{These authors contributed equally.}
%\affil[$\dagger$]{These authors jointly supervised this work.}
\else
\author{Antiquus S.~Hippocampus, Natalia Cerebro \& Amelie P. Amygdale \thanks{ Use footnote for providing further information
about author (webpage, alternative address)---\emph{not} for acknowledging
funding agencies. Funding acknowledgements go at the end of the paper.} \\
Department of Computer Science\\
Cranberry-Lemon University\\
Pittsburgh, PA 15213, USA \\
\texttt{\{hippo,brain,jen\}@cs.cranberry-lemon.edu} \\
\And
Ji Q. Ren \& Yevgeny LeNet \\
Department of Computational Neuroscience \\
University of the Witwatersrand \\
Joburg, South Africa \\
\texttt{\{robot,net\}@wits.ac.za} \\
\AND
Coauthor \\
Affiliation \\
Address \\
\texttt{email}
}
\fi
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
%\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
\begin{abstract}
The \emph{magnitude} of a metric space was recently established as
a novel invariant, providing a measure of the `effective size' of
a space across multiple scales.
%
By capturing both geometrical and topological properties of
data, magnitude is poised to address challenges in unsupervised
representation learning tasks.
%
We formalise a novel notion of dissimilarity between magnitude
functions of finite metric spaces and use them to derive a quality
measure for dimensionality reduction tasks.
%
Our measure is provably stable under perturbations of the data, can be
efficiently calculated, and enables a rigorous multi-scale comparison of
embeddings.
%
We show the utility of our measure in an experimental suite that
comprises different domains and tasks, including the comparison of
data visualisations.
\end{abstract}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Determining suitable low-dimensional representations of complex high-dimensional data is a challenging task in numerous applications.
%
Whether its preprocessing biological datasets prior to their analysis~\citep{nguyen2019ten}, the visualisation of complex structure in single-cell sequencing data~\citep{lahnemann2020eleven}, or the comparison of different manifold representations~\citep{Barannikov22a}: an understanding of structural (dis)similarities is crucial, especially in the context of datasets that are ever-increasing in size and dimensionality.
%
The primary assumption driving such analyses is the \emph{manifold hypothesis}, which assumes that data is a~(noisy) subsample from some unknown manifold.
%
Operating under this assumption, \emph{manifold learning} methods have made large advances in detecting complex structures in data, but they typically use \emph{local} measures of the embedding quality, which are ultimately relying on local approximations of manifolds by $k$-nearest neighbour graphs.
%
However, such approximations---which require specific parameter choices and thresholds---can have a substantial negative impact on both embedding results and the interpretation of evaluation scores.
%
Moreover, countering the increasing popularity of non-linear dimensionality reduction methods that claim to preserve local and global structures, recent work~\citep{chari2023specious} sheds some doubt on the assumption that `good' embeddings should also faithfully preserve distances, while raising questions of how to measure the inevitable distortions introduced by representation learning.
%
% While, commonly used distance preservation metrics such as distance correlating might miss out on detecting the underlying manifold structure of a space, a good embedding arguably should faithfully preserve a sense of dissimilarities between both observations and clusters.
%
Thus, there is a need for novel methods in representation learning, which efficiently summarise data across varying levels of similarity, eliminating the need to rely on fixed neighbourhood graphs.
Motivated by these considerations, we adopt a more general perspective that does not rely on manifold approximations.
To this end, we propose a novel embedding quality measure based on \emph{metric space magnitude}, a recently-proposed mathematical invariant that encapsulates numerous important geometric characteristics of metric spaces.
%
Specifically, magnitude summarises and captures characteristic properties like the diameter, curvature, density, and distance distribution of a space across varying scales of distances.
%
Its computation relies on encoding dissimilarities between data points, permitting magnitude to capture relevant information at local and global scales.
%
Magnitude is thus poised to compare metric spaces %---embeddings---
at different scales, yielding a compact holistic summary of embedding quality that is inherently aware of preserving geometric structure across multiple scales.
% Magnitude can thus be used to compare , can thus be used to assess distributions of points in an embedding by considering all distances between them simultaneously and comparing the diversity of information present in the embedding to its high-dimensional version. Throughout this paper, we thus develop magnitude as a compact holistic summary of global and local embedding quality.
%Due to these properties, magnitude lends itself naturally as a compact summary of the distortion in the context of dimensionality reduction.
%fundamental properties
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Our contributions.}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
Our work is the first to formalise a notion of difference between the magnitude of two different metric spaces. We use this framework in the context of dimensionality reduction to compare learned representations in a scale-invariant manner.
%framework for comparing magnitude across different metric spaces.
%We develop a framework to formalise the
%notion of difference between the magnitude of two metric spaces in a scale-invariant manner.
Moreover, we link our proposed magnitude difference to embedding quality and empirically demonstrate its utility both on simulated and real data, while also improving the computational efficiency of magnitude calculations. Finally, we prove the stability of magnitude and empirically validate its robustness to noise.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We briefly review the literature related to dimensionality reduction and previous applications of magnitude to machine learning.
\paragraph{Dimensionality reduction.}
%
Dimensionality reduction (DR) plays an important role in representation learning that affords many different uses, including
%
\begin{inparaenum}[(i)]
\item exploring data,
\item extracting latent features, or
\item compressing data~\citep{nguyen2019ten}.
\end{inparaenum}
%
It is thus of the utmost importance that DR results carry as much information about characteristic properties of the data as possible.
%
A plethora of DR methods exist, including PCA, UMAP~\citep{mcinnes2018umap}, t-SNE~\citep{van2008visualizing}, or methods based on~(variational) autoencoders.
%
The abundance of such techniques leads to the question of how to choose an appropriate method for a given task.
%
The answer is often not straightforward, prompting practitioners to rely on embedding quality measures, i.e.\ scalar summaries designed to determine to what extent a specific structure or property of interest is preserved in lower dimensions.
%given the various practical requirements on preserving distances, minimising distortion or representing clusters. In order to choose one, practitioners often resort to embbeding quality measures, one number summaries designed to determine to what extant a particular structure or property of interest is preserved in lower dimensions.
Numerous quality measures exist~\citep{Lee09a, Rieck17a}; they can be broadly split into two groups, depending on whether they measure the preservation of \emph{local} or of \emph{global} structures.
%
However, when the goal is to preserve both of them, existing measures are not up to the task:
%
neighbourhood-based measures like \emph{trustworthiness}~\citep{Lee08a} fail to capture the global structures, whereas global measures like \emph{distance correlation}~\citep{Szekely07a} are unable to offer a sufficiently fine-grained view; as our experiments show, such measures are misleading even for toy examples like the `Swiss Roll' dataset.
%
Finally, existing methods are incapable to track properties across multiple scales, relying on fixed-scale neighbourhood graphs, for instance.
%
There is thus a crucial need for a measure that can capture both the local and the global structure of a space across multiple scales.
\paragraph{Magnitude and its applications in machine learning.}
%
Magnitude has first been introduced by \citet{solow1994measuring}
for measuring \emph{biological diversity} in a set of species, with subsequent formalisations by \citet{leinster2013magnitude} in terms of category theory.
%
Since then, the mathematical properties of magnitude have been studied more extensively, linking it to
%
\begin{inparaenum}[(i)]
\item maximum entropy,
\item the \emph{intrinsic dimension} of a space, or
\item its \emph{Euler characteristic}.
\end{inparaenum}
%
Nevertheless, despite these strong ties to characteristic properties, magnitude has only rarely been applied in a machine learning context, and its potential for studying data remains under-explored.
%
Recent publications started to bridge this gap.
%
For instance, \citet{bunch2021weighting} successfully demonstrated the use of \emph{magnitude weights} for boundary detection, while \citet{adamer2021magnitude} showed how to use magnitude for \emph{edge detection} in images.
%
In the context of deep learning, a recent work links magnitude to the \emph{generalisation error} of neural networks~\citep{andreeva2023metric}.
%
However, somewhat surprisingly, we find that the question of how to compare the behaviour of metric space magnitude across two spaces has not yet been addressed---our work tackles this aspect in the context of dimensionality reduction.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Background: The Magnitude of a Metric Space}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[b!]
\centering
\centerline{\includegraphics[clip, trim=0cm 7.4cm 11cm 0cm, width=0.7\textwidth]{Figures/weights_example_corrected.pdf}}
\caption{ \textbf{Example of magnitude weights and the magnitude function for a metric space with 5 points.}
%
When the scaling factor $t$ is very small, e.g.\ $t=0.01$, the magnitude weights of all points
sum up to approximately 1, so that magnitude is very close to 1,
and the space effectively looks like one point.
Following this, as we zoom in further, magnitude grows and at $t=0.41$, 3 distinct clusters or points are visible.
Finally, for $t = 100$, all the points are clearly separated, their magnitude weights converge to one, and the value of magnitude approaches $5$, i.e.\ the cardinality of the space.
}
\label{fig:magnitude_visualisation_5}
\end{figure}
\emph{Magnitude} is a recently-developed invariant that describes the `effective number of points' of a metric space
as a function of its scaled distances~\citep{leinster2013magnitude}.
We note that while we restrict our description to \emph{finite} metric spaces, magnitude is defined for infinite spaces as well.
%
\begin{definition}[Magnitude of a metric space]
Let $X = \{x_1, \dots, x_n\}$ be a finite metric space with an associated distance metric $d$.
%
The \emph{similarity matrix} of $X$ is defined as $\zeta_X(i,j) = e^{-d(i,j)}$ for $1 \leq i,j \leq n$.
%
If $\zeta_X$ is invertible,
the \emph{magnitude} of $X$ is defined as
%
\begin{equation}
\magnitude(X) := \sum_{ij}(\zeta_X^{-1})_{ij},
\end{equation}
%
while the \emph{magnitude weighting vector}~$w_X$ is defined as
$w_X := \zeta_X^{-1} \ones$.
%
Denoting the $i$th element of $w_X$ by $w_{x_i}$, we obtain an equivalent characterisation of magnitude as $\magnitude(X) = \sum_i w_{x_i}$.
\label{def:Magnitude}
\end{definition}
%
The existence of magnitude is contingent on the existence of $\zeta_X$.
%
Given a \emph{negative definite metric}~$d$, such as the $L_1$ and $L_2$ distance, $\zeta_X$ is invertible~\citep{Feragen15a}.
%
We will thus subsequently assume that $(X, d)$ with $X \subseteq \reals^D$ permits the calculation of magnitude; in particular $X$ must \emph{not} have any duplicate points.
While the magnitude of a metric space is expressive even at a single scale~\citep{leinster2013magnitude, leinster2021magnitude},
magnitude unleashes its full potential in a multi-scale setting, assigning to a metric space not just a scalar but a function.
%
To this end, we introduce a parameter $t \in \mathbb{R}_{+}$ and consider the metric space with distances scaled by $t$, which we denote by $tX$.
%
Intuitively, this procedure corresponds to viewing the same space through different lenses, or at different `zoom factors,' for example by converting distances from metres to centimetres.
%
Computing the magnitude for each value of $t$ then yields the \emph{magnitude function}.
%
\begin{definition}[Magnitude function]
For a metric space $(X, d)$, we define its scaled version $tX := (X, d_t)$ by $d_t(x,y) := td(x,y)$, where $t \in \mathbb{R}_+$ is the scaling factor.
%
The \emph{magnitude function} of $(X, d)$ is the function $\magnitude_X\colon t \mapsto \magnitude(tX)$.
%
\label{def:Magnitude function}
\end{definition}
%
For $t \in (0, \infty)$, the magnitude function is defined for all but finitely many values of $t$~\citep{leinster2013magnitude}.
%
In our setting, given a negative definite metric~$d$, the magnitude function is defined for \emph{all} $t \in (0, \infty)$ because the scaled distance matrix is guaranteed to remain negative definite.
%
For such metrics, the magnitude function is also \emph{continuous}~\citep[Corollary~5.5]{meckes2015magnitude}\footnote{%
Even for general metrics, the magnitude function is continuous for $t > t_{\text{crit}}$, where $t_\text{crit}$ is the supremum of its finitely many singularities.
} and for finite metric spaces, we have $\lim_{t \to \infty}\magnitude(tX) = |X| = n$, i.e.\ the \emph{cardinality} of $X$~\citep[Proposition~2.2.6]{leinster2013magnitude}.
%
This limit behaviour exemplifies to what extent the magnitude function counts the `effective number of points at scale~$t$.'
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Metric Space Magnitude to Analyse Embeddings}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%Wanting to compare the behaviour of metric space magnitude across two different spaces, for example in the context of dimensionality reduction, naturally raises the question on how to define a suitable difference between magnitude functions.
%Interestingly enough, %due to practical limitations partly related to the computational cost, a consideration we will adress in Section ???, and partly related to the
%this question has not yet been addressed (or answered) in existing literature.
%\kat{Reword the start of this.}
%\subsection{Defining a scale-invariant distance between magnitude functions}
Motivated by the characteristic properties of magnitude, we want to leverage it to assess the quality of embeddings, a common task in representation learning, which is often exacerbated by a lack of methods that are capable of comparing metric spaces.
%
When assessing embedding quality, a dataset $X$ and its representation $Y$ can either be aligned, i.e.\ points in~$X$ are in direct correspondence with points in~$Y$, or unaligned.
%
We will propose methods to address both scenarios, but our main goal is to compare \emph{aligned} spaces, as these are more often encountered in the setting of dimensionality reduction.
%However, observations are not necessarily aligned in other contexts, for example when creating word embeddings across languages with differing vocabulary or integrating multiple unpaired modalities in single cell data analysis. Thus, there still exists motivation to propose a notion of magnitude difference for unaligned spaces.
%Further, %the comparison between $X$ and $Y$ %could be invariant to the scaling of the distances in a metric space or scale-sensitive.
To address this setting, we define a novel method for comparing the magnitude functions of two (un)aligned spaces.
We aspire towards a scale-invariant notion of distance between magnitude functions, because this property is necessary when accounting for the distortion of distances, which inevitably occurs under DR.
%
As a first step towards scale invariance, we first normalise all distances by the diameter of the metric space, prior to calculating magnitude. This
re-parametrisation of the magnitude function is permissible given the desired scale-invariance and the behaviour of magnitude under linear scaling as detailed in \cref{Cor:scaling}. %Note however, that this cannot account for all aspects of distance distortion as more often than not, distances are not just distorted in a linear manner.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Comparing the Magnitude of Different Metric Spaces}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Existing work that computes magnitude functions for a specific space typically employs \emph{fixed} evaluation intervals, thus possibly missing relevant scales.
%
To address this limitation and to further incorporate knowledge on the distance distribution, we propose comparing magnitude across \emph{multiple scales}, taking the overall convergence behaviour of magnitude functions into account.
%
Our procedure automatically determines a suitable range of scaling factors, enabling us to compare magnitude functions in a \emph{scale-invariant manner}.
%
Intuitively, evaluation intervals should be related to the convergence behaviour of the magnitude function $\magnitude_{X}(t)$, which converges to $|X|$ as $t \to \infty$.
%
This convergence is monotonic in the sense that for~$t$ sufficiently large, the magnitude function is increasing in~$t$ under the condition that $tX$ is a \emph{scattered space}, i.e.\ $t > \nicefrac{\log(n - 1)}{\min_{x \neq y \in X} d(x, y)}$~\citep{leinster2013magnitude},
%
Given an error threshold $\epsilon > 0$, we thus argue that `nothing interesting' happens after a certain scale $t_{\mathrm{conv}}$ that satisfies $\magnitude_X(t_{\mathrm{conv}}) \ge |X|-\epsilon$, as we are only interested in the behaviour of magnitude until it starts converging.
%
In practice, we set $\epsilon$ to a proportion of the cardinality, e.g\ $\epsilon=0.05 n$.
%
We automate the process of finding the convergence point $t_{\mathrm{conv}}$ using numerical root-finding procedures.
Next, having determined $t_{\mathrm{conv}}$, we evaluate magnitude at evenly-spaced intervals between $(0, t_{\mathrm{conv}}]$.
This permits us to define the \emph{re-scaled magnitude function}.
%
\begin{definition}[Re-scaled magnitude function]
Let $(X,d)$ be a metric space, $\magnitude_{X}(t)$ its magnitude function, and $t_{\mathrm{conv}}$ its convergence point.
%
For $s \in [0, 1]$, we then define the \emph{re-scaled magnitude function} as $\mathcal{M}_{X}(s) := \magnitude_{X}(s \cdot t_{\mathrm{conv}})$, and its re-scaled weights as $\mathcal{W}_{X}(s) := w_{X}(s \cdot t_{\mathrm{conv}})$.
\end{definition}
%
The re-scaled magnitude functions are advantageous because they permit us to reason about different metric spaces without having to account for~(non)uniform metric scaling effects, making it possible to use the same domain for both functions.
%
Subsequently, we will be operating solely with the \emph{re-scaled magnitude functions} unless mentioned otherwise.
%
Assuming that we have approximated the convergence scales $x_{\mathrm{conv}}$ and $y_{\mathrm{conv}}$ for two similar metric spaces $X$ and $Y$ of equal cardinality, we now want to compare them via magnitude.
%
To this end, we propose the \emph{magnitude profile difference}, a novel dissimilarity measure between the re-scaled magnitude functions $\mathcal{M}_{X}(s)$ and $\mathcal{M}_{Y}(s)$.
%
\begin{definition}[Magnitude profile difference]
Let $X$ and $Y$ be two metric spaces with $|X|=n_x$ and $|Y|=n_y$. Assume the associated~(re-scaled) magnitude functions $\mathcal{M}_{X}(s)$ and $\mathcal{M}_{Y}(s)$ have been evaluated across the same scale factors $S =\{s_1, ..., s_i\}$.
%
We define the point-wise magnitude profile difference to be
$\Delta\magnitude_{XY} := \int_{0}^1|\frac{\mathcal{M}_{X}(s)}{n_x}-\frac{\mathcal{M}_{Y}(s)}{n_y}|ds \approx \sum_{j}|\frac{\mathcal{M}_{X}(s_j)}{n_x}-\frac{\mathcal{M}_{Y}(s_j)}{n_y}|$.
%
\label{def:Magnitude profile difference}
\end{definition}
%
The magnitude profile difference involves using the \emph{area} between the curves as a measure of dissimilarity.
%
The advantage of this formulation is that it affords an easy calculation while remaining interpretable.
%
In practice, the summation from \cref{def:Magnitude profile difference} could also be replaced by more accurate numerical integration methods, such as Trapezoidal or Romberg integration.
%
The difference between magnitude profiles now formalises the intuitive idea of comparing the behaviour of magnitude across two similar spaces as their magnitude approaches comparable convergence points.
%
One can interpret this distance as comparing
the behaviour of magnitude until each space effectively looks like it has almost reached its cardinality.
%
Alternatively, one may think about this comparison as starting with each space looking like one single point, then zooming in until the selected number of distinct observations are visible, and ultimately summing up the differences in magnitude at each zoom factor.
The magnitude profile difference studies the geometrical properties of each space across scales; it is thus even suitable for comparing unaligned metric spaces and spaces with different cardinalities.
%
Nevertheless, whenever possible, tracking the identity of each point from the original to the reduced space is highly informative for assessing embedding quality.
%
We thus propose another distance between $X$ and $Y$ based on the idea of \emph{magnitude weights} from \cref{def:Magnitude}.
%
\begin{definition}[Magnitude weight difference]
Let $X$ and $Y$ be two metric spaces
with cardinality $n$ whose elements are aligned.
%
Assume the associated re-scaled weight vectors $\mathcal{W}_{X}$ and $\mathcal{W}_{Y}$ have been evaluated across the same scaling factors $\{s_1, ..., s_i\}$.
%
We define the magnitude weight difference function
$\mathbf{w_{XY}}\colon j\in\{1,2,..,i\} \to \mathbb{R}$ as
$\mathbf{w_{XY}}(j) := \frac{1}{n}\sum_{k=1,2,..,n} |\mathcal{W}_X(s_j)- \mathcal{W}_Y(s_j)|_k$.
Moreover, we define the magnitude weight difference between $\mathcal{W}_X$ and $\mathcal{W}_Y$ as $\Delta \mathbf{w_{XY}} := \sum_{j\in\{1,2,..,i\}} \mathbf{w_{XY}}(j)$.
\label{def:Magnitude weight difference}
\end{definition}
%
Notice that this definition of magnitude weight difference is similar to the previously-defined point-wise magnitude profile difference.
%
Both notions assume that the magnitude values of $X$ and $Y$ evaluated at the scaling factors $s_j \cdot x_{conv}$ and $s_j \cdot y_{conv}$ respectively
are comparable.
%
The major distinction is that \cref{def:Magnitude weight difference} lends itself to a comparison on a per-point basis, whereas \cref{def:Magnitude profile difference} averages across points.
%
We find the former to be beneficial in case data is paired or aligned~(i.e.\ $x_i$ corresponds to $y_i$ under some transformation like an embedding)
because it permits us to directly identify the changes in each point's magnitude weight before and after dimensionality reduction.
%
We will thus use \cref{def:Magnitude weight difference} as the basis for an embedding quality measure.
\begin{figure}[h!]
\centering
%\centerline{\includegraphics[clip, trim=0cm 6cm 0cm 0cm, width=0.8\textwidth]{Figures/Overview_corrected.pdf}}
\centerline{\includegraphics[clip, trim=0cm 8.2cm 0cm 0cm, width=1\textwidth]{Figures/Overview_corrected.pdf}}
\caption{ \textbf{Overview of the proposed dimensionality reduction quality measures.} Magnitude profile difference calculates the area between the re-scaled magnitude functions, which have been re-parametrised to the same approximate convergence point. Similarly, magnitude weight difference sums up the pairwise difference in magnitude weights across all re-scaled evaluation values.
%
Lower values indicate a better agreement with the ground truth or original data.
}
\label{fig:magnitude_function_visualisation}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Computing magnitude.}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
A na\"ive calculation of magnitude according to \cref{def:Magnitude} requires inverting the similarity matrix~$\zeta_X$, which has a worst-case complexity of $\mathcal{O}(n^3)$ and is numerically unstable.
%
However, inverting $\zeta_X$ is not required in practice; instead, it suffices to solve certain \emph{linear equations}.
%
First, we notice that the calculation of magnitude can be written as $\magnitude(X) := \ones^\top \zeta_X^{-1} \ones$.
%
For finite metric spaces and negative definite metrics, $\zeta_X$ is a \emph{symmetric positive definite matrix}, thus affording a \emph{Cholesky decomposition}, which factorises $\zeta_X = LL^\top$, with $L$ being a \emph{lower triangular matrix}.
%
This operation is numerically stable and more efficient than matrix inversion~\citep{Higham09a}.
%
We thus have $\magnitude(X) := \ones^\top \zeta_X^{-1} \ones = \ones^\top (LL^\top)^{-1} \ones = (L^{-1} \ones)^\top (L^{-1}\ones)$.
%
This is equivalent to calculating $x^\top x$ with $x = L^{-1}\ones$, which we can efficiently obtain by solving $Lx = \ones$ since $L$ is lower triangular.
%
Likewise, we can reformulate the calculating of the \emph{magnitude weight vector} $w_X = \zeta_X^{-1}\ones$ as solving $\zeta_X w_X = \ones$, which also benefits from the Cholesky factorisation.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% ___-___ o==o====== . . . . .
% =========== ||//
% \ \ |//__
% #_______/
% TODO: Add mini table of magnitude speeds here.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Theoretical Properties of Magnitude}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Next to the theoretical properties linking magnitude to geometrical properties of a space, which we previously outlined, we prove that magnitude also satisfies properties that are advantageous in the setting of comparing different metric spaces.
%
Specifically, we prove that magnitude is an \emph{isometry invariant} and satisfies certain \emph{stability properties} in light of perturbations of metric space.
\begin{definition}[Isometry]
Let $(X, d_X)$ and $(Y, d_Y)$ be two metric spaces.
%
A map $f\colon X \to Y$ is called an \emph{isometry}, or distance-preserving, if for any $a, b \in X$, we have $d_{X}(a,b) = d_{Y}(f(a),f(b))$.
%
$X$ and $Y$ are called \emph{isometric} if there is a \emph{bijective isometry} from $X$ to $Y$.
\end{definition}
%
\begin{restatable}[Isometry invariance]{lemma}{lemisometryinvariance}
Given two isometric spaces $X, Y$, we have $\magnitude_{X} = \magnitude_{Y}$.
\end{restatable}
%
\begin{corollary}
The magnitude functions of two isometric spaces~$X, Y$ are equal for all $t > 0$.
\end{corollary}
%
Notice that the \emph{converse} of this statement is not true in general, i.e.\ there are non-isometric spaces whose magnitude functions are the same~\citep{leinster2013magnitude}.
%
\begin{corollary}
Let $X$ be a metric space and $Y=cX$ with $c \in \mathbb{R}_+$. Then the magnitude functions
of $X$ and $\frac{1}{c} Y$ are equal. Also, the magnitude functions of $\frac{1}{\mathrm{diam}_X} X$ and $\frac{1}{\mathrm{diam}_Y} Y$ are equal, where $\mathrm{diam}_X := \max(d_X)$.
\label{Cor:scaling}
\end{corollary}
Moreover, as a metric space invariant, magnitude should ideally satisfy certain stability properties.
%
By this, we mean that if two metric spaces $X, Y$ are \emph{close}, we want to obtain bounds on the differences between their magnitude values.
%
The canonical choice to measure closeness would be the Gromov--Hausdorff distance, but in the absence of strong results concerning the behaviour of magnitude under this distance~\citep{leinster2013magnitude}, we resort to a more general---but also weaker---notion of similarity in terms of \emph{continuity}.
%
More precisely, we will show that the similarity matrices used in the calculation of magnitude are well-behaved in the sense that closeness of metric spaces~(under some matrix norm) translates to a continuous bound on the variation of the similarity matrices.
%
We first prove a general result about matrices and their associated transformations.
%
\begin{restatable}{lemma}{lemspectralbound}
Let $\|A\|_2 := \sup{\{\|Ax\|_2 : x \in \reals^n \text{ with } \|x\|_2 = 1\}}$ refer to the \emph{induced $2$-norm for matrices}, and let $A, B$ be two $n \times n$ matrices with $\|A - B\|_2 \leq \epsilon$.
Moreover, let $f(M) := \ones^\top M \ones$.
Then $\|f(A) - f(B)\|_2 \leq n\epsilon$.
%
\label{lem:Magnitude matrix spectral bound}
\end{restatable}
%
Treating $A, B$ as inverse similarity matrices, the preceding statement shows that if the two inverse similarity matrices are close with respect to their spectral radius, the difference between their magnitude can be bounded.
%
The following lemma shows that the similarity matrices satisfy a general continuity condition.\footnote{
It is clear that the mapping itself is continuous because of the functions involved in its calculation.
However, we find it important to remark on the bound obtained with respect to the \emph{spectral norm} of the two similarity matrices.
}
%
\begin{restatable}{lemma}{lemsimilaritymatrixspectralbound}
Let $(X, d_X)$ and $(Y, d_Y)$ be two metric spaces with corresponding distance matrices $D_X, D_Y$ and cardinality~$n$.
%
For all $\epsilon > 0$, there exists $\delta > 0$ such that if $|D_X - D_Y| < \delta$ holds elementwise, then $\|\zeta_X - \zeta_Y\|_2 \leq \epsilon$.
%
\label{lem:Similarity matrix spectral bound}
\end{restatable}
%
As a consequence of \cref{lem:Similarity matrix spectral bound}, and the continuity of matrix inversion, we know that magnitude is well-behaved under small perturbations of the respective distance matrices.
%
Given a pre-defined threshold $\epsilon$, we can always find perturbations that preserve the magnitude difference accordingly.
%
Notice that this result does not make any assumptions about the Gromov--Hausdorff distance of the metric space and only leverages the distance matrices themselves.
%
Moreover, this result applies in case $X, Y$ are close with respect to the \emph{Hausdorff distance}.
%
If $d_H(X, Y) < \delta$, the elementwise condition $|D_X - D_Y| < \delta$ is satisfied \emph{a fortiori}.
%
Nevertheless, from a theoretical point of view, this result could be made stronger by showing bounds in terms of distances between the metric spaces.
%
We leave such a result for future work, noting in passing that such strong results remain elusive at the moment~\citep{govc2021persistent}; it is known, however, that the magnitude function is at least \emph{lower semicontinous}~\citep[Theorem~2.6]{meckes2013positive}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
In the following, we will demonstrate the utility of our \emph{magnitude profile difference} and \emph{magnitude weight difference} metrics in the context of evaluating DR methods.
%
Using both simulated and real data, we demonstrate the following results:
%
\begin{inparaenum}[(i)]
\item The magnitude profile difference is empirically stable with respect to varying levels of Laplacian noise.
%
\item In the context of non-linear DR, magnitude weight difference successfully identifies the ground truth and detects the underlying manifold.
%
\item Magnitude difference relates local and global properties and finds a good compromise between cluster, distance, and density preservation for visualising the cluster structures of a subset of MNIST.
%
\item When visualising single-cell data, magnitude difference identifies the information loss introduced by preprocessing. Further, for visualisation, it
prefers representations that best reflect the underlying biological clustering structure while preserving the global geometry of the data.
\end{inparaenum}
%While, commonly used distance preservation metrics such as distance correlating might miss out on detecting the underlying manifold structure of a space, a good embedding arguably should faithfully preserve a sense of dissimilarities between both observations and clusters.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Empirical Stability of Magnitude}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[b]
\centering
\subcaptionbox{Circles}{\includegraphics[width=0.293\textwidth]{Figures/magnitude_difference_laplace_final.png}}
\subcaptionbox{Swiss Roll}{\includegraphics[width=0.3\textwidth]{Figures/magnitude_difference_laplace_swissroll.png}}
\subcaptionbox{Gaussian blobs}{\includegraphics[width=0.3\textwidth]{Figures/magnitude_difference_laplace_blobs_final.png}}
\caption{%
\textbf{Empirical stability of magnitude.} Magnitude difference is stable across different datasets and sample sizes. The lines show the mean magnitude difference and the shaded area the standard deviation calculated across 50 repetitions.
}
\label{fig:magnitude_stability}
\end{figure}
We investigate the empirical stability of the magnitude profile difference.
%
Given the difficulty in proving strong theoretical stability results, we verify that, in practice, the magnitude profile difference remains stable when adding noise to the input space.
%
We thus sample points from a Laplace distribution with mean~$\mu = 0$ and variance~$2b^2$ with different levels of noise, i.e.\ $b \in \{0.0001, 0.001, 0.005, 0.01, 0.05\}$.
%
\Cref{fig:magnitude_stability} depicts the errors in magnitude profile difference across three different datasets~(circles, Swiss Roll, Gaussian blobs), using a different number of samples~(varying between $100$ and $5000$ across $50$ repetitions).
%
The bound of $5000$ points has been chosen given the clear downwards trend across multiple noise levels; we expect the same trend to hold for larger sample sizes.
%
We observe that the magnitude profile difference does not increase above the value of \num{1e-4} with increasing sample size. In fact, the magnitude profile difference fluctuates more for smaller number of points, but this is still within a very small range. We therefore conclude that the magnitude profile difference between the original space and its noisy version does not change much, which indicates that our measure is reliable and stable across multiple experimental conditions.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Comparing Manifold Learning Algorithms}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
To test the primary utility of our method, we analyse its behaviour in comparing embeddings arising from different manifold learning algorithms.
%
As a simple toy example, we first compute the magnitude differences between a dataset with $300$ points sampled randomly from a 3-dimensional `Swiss Roll' and different two-dimensional representations, shown in \cref{fig:not_a_duck}(a) and ranked from best to worst.
%
Magnitude weight difference clearly identifies the ground truth, which is
given by the original points sampled on a 2D plane before being rolled up,
as the best embedding.
%
Magnitude properly detects the underlying manifold structure using only Euclidean distances.
%
The reason for its success is that magnitude is more heavily influenced by close-by points rather than by the gap between the layers of the roll; magnitude is thus capable of detecting that `zooming out' of the data is very similar to zooming out of the ideal planar representation.
%
Meanwhile, other embedding quality measures based on the Euclidean distance, such as pairwise distance correlation, are observed to \emph{fail} at this task.
%
As summarised in \cref{tab:still_not_a_duck}, spectral distance correlation is the only other measure able to also detect the ground truth, but its calculation relies on the choice of $k$-NN graph, whereas magnitude weight difference arguably constitutes a more flexible, parameter-free multi-scale summary.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[b!]
\centering
\subcaptionbox{Swiss Roll}{%
\includegraphics[width=0.325\textwidth]{Figures/sr_embeddings_plot.pdf}%
}%
\subcaptionbox{MNIST}{%
\includegraphics[clip, trim=0cm 0cm 4cm 0cm, width=0.38\textwidth]{Figures/all_embeddings_fig4_mnist.pdf}%
}%
\subcaptionbox{Single-Cell PBMC}{%
\includegraphics[clip, trim=0cm 0cm 8.2cm 0cm, width=0.275\textwidth]{Figures/all_embeddings_fig4_sc.pdf}
}
\caption{%
\textbf{Examples of 2D embeddings} for three different datasets using various dimensionality reduction methods, ordered by their magnitude weight difference~($\Delta \mathbf{w_{XY}}$) from low~(top left) to high (bottom right) values.
%
As $\Delta \mathbf{w_{XY}}$ increases, there is a clearly-observable decrease in embedding quality and increase in distortion across these examples.
%confidently selects the best embedding for all three datasets.
}
\label{fig:not_a_duck}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{table}[tbp]
\centering
\caption{%
\textbf{Comparison between magnitude weight difference and other quality measures}, with RMSE being calculated between the pairwise distance matrices,
for the datasets and embeddings depicted in \cref{fig:not_a_duck}.
%
We notice that $\Delta \mathbf{w_{XY}}$, our proposed magnitude weight difference, selects the ground truth for the Swiss Roll, determines t-SNE as a good method for visualising clusters in the MNIST digits data and prefers UMAP as a good compromise for the PBMC single-cell data.
}
\label{tab:still_not_a_duck}%
\resizebox{1\columnwidth}{!}{
\sisetup{
round-mode = figures,
table-text-alignment = center,
}
\begin{tabular}{@{}lS[round-precision=3]S[round-mode=places, round-precision=3]S[round-precision=3]S[round-mode=places, round-precision=4]S[round-precision=3]S[round-mode=places, round-precision=1]S[round-mode=places, round-precision=2]S[round-mode=places, round-precision=2]@{}}
\toprule
& \textbf{$\Delta \mathbf{w_{XY}}$}~{($\downarrow$)} & \textbf{$\Delta \mathbf{\magnitude_{XY}}$}~{($\downarrow$)}& \textbf{Dist.\ corr.}~{($\uparrow$)} & \textbf{Spectral dist.\ corr.}~{($\uparrow$)} & \textbf{Density preservation}~{($\uparrow$)} & \textbf{RMSE}~{($\downarrow$)} & \textbf{Trustworthiness}~{($\uparrow$)} & \textbf{Silhouette Score}~{($\uparrow$)}\\
\midrule
& \multicolumn{8}{c}{\textsc{Swiss Roll}}\\
\midrule
Ground Truth & 0.12948561697742758 & 0.006636627451412344 & 0.4585019110476058 & 0.08128245527292584 & 0.4091057189575403 & 8.59008970328983 & 0.7973721894782799 & \\
MDS & 0.13025028496720714 & 0.020661478430234217 & 0.8827988004228312 & -0.000635018220590786 & 0.5203150595923918 & 3.214049712440246 & 0.9277672997162192 & \\
Isomap & 0.14581394415794965 & 0.027503629429951094 & 0.8437103648299701 & -0.031914355393737645 & 0.24780664529082447 & 8.660554259947336 & 0.9229246889325475 & \\
UMAP & 0.16285059715010314 & 0.04436980892839972 & 0.643281260978966 & 0.034783587720729384 & 0.4079545142892539 & 12.410570196973987 & 0.9806710325256495 & \\
PHATE & 0.1722284314302854 & 0.07501886656238443 & 0.4057260885678108 & -0.0007914356133396025 & 0.3161118790757986 & 16.192512368972 & 0.9104086444007858 & \\
Dummy & 0.2101304555474421 & 0.15206159938725503 & 0.8442099732498396 & -0.010469297512964836 & 0.42607742534163656 & 4.359292303299585 & 0.890068107400131 & \\
\midrule
& \multicolumn{8}{c}{\textsc{MNIST digits}}\\
\midrule
MDS & 0.25882703789961603 & 0.23286128121266933 & 0.7792022277293873 & 0.06382900420204893 & 0.5834187645675377 & 15.313848565706857 & 0.9074065421427168 & 0.29746595819459437 \\
t-SNE & 0.28887752357792196 & 0.2670352003815647 & 0.5713005432510415 & -0.015419259800170282 & 0.42450743157919385 & 21.342895469270104 & 0.9849751879169494 & 0.6733258366584778 \\
Isomap & 0.31226950932115205 & 0.2850309297338081 & 0.6485051877216458 & 0.035075796967476175 & 0.4843665924770535 & 41.24537495603067 & 0.9015010067972722 & 0.4393579686685604 \\
LLE & 0.3218920434576917 & 0.293393182887077 & 0.4155903111666728 & 0.07471545773453922 & 0.1437977227381689 & 49.264543191716214 & 0.8248863079279259 & 0.08436748186776016 \\
UMAP & 0.3243659756252932 & 0.30043921871542434 & 0.5212221194595449 & -0.002546588891011079 & 0.12684078781720148 & 37.667486817268845 & 0.9849481026599473 & 0.735994279384613 \\
Spectral & 0.35619973816528905 & 0.32187421692837614 & 0.5538572452713432 & -0.0008096152267080899 & 0.4300215207916755 & 49.31195227137102 & 0.9227613019019754 & 0.5496598466245219 \\
\midrule
& \multicolumn{8}{c}{\textsc{Single-cell PBMC}}\\
\midrule
UMAP ($40$ PCs) & 0.39 & 0.379 & 0.331 & 0.043 & 0.193 & 49.848 & 0.625 & 0.492 \\
tSNE ($40$ PCs) & 0.442 & 0.43 & 0.304 & 0.104 & -0.233 & 28.944 & 0.635 & 0.4 \\
PCA~($2$ PCs) & 0.437 & 0.424 & 0.447 & 0.07 & 0.407 & 48.057 & 0.611 & 0.408 \\
PHATE~($40$ PCs) & 0.454 & 0.44 & 0.334 & 0.044 & 0.322 & 56.449 & 0.618 & 0.617 \\
\bottomrule
\end{tabular}
}
\end{table}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{MNIST and Clustering}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Motivated by the results on the `Swiss Roll' dataset, we now analyse data that exhibits distinct clustering patterns. To this end, we consider a dataset of handwritten MNIST digits, which we further subset to $1,083$ greyscale images, corresponding to the digits from 0 to 5.
%
We choose these data in order to visualise and visually assess the six distinct clusters more easily.
%
From the two-dimensional representations shown in \cref{fig:not_a_duck}(b) and \cref{fig:magnitude_mnist}~(in the appendix),
magnitude weight difference identifies MDS as the best embedding, which is likely because both MDS and magnitude assess embedding quality in terms of Euclidean distances.
%
This is followed by t-SNE, which in this case is the representation that clearly separates different digits, exhibiting the highest silhouette score of $0.67$, the highest trustworthiness, the lowest neighbourhood loss,
and the second-lowest RMSE as shown in \cref{tab:still_not_a_duck} and \cref{tab:full_results} (in the appendix).
%
Magnitude thus demonstrates its ability to take cluster structure into account, finding the best compromise between cluster preservation, preservation of local neighbourhoods, and preservation of all distances.
%
By contrast, embeddings like PCA and Isomap that show separate groups but do not clearly separate between them, have lower magnitude differences than UMAP or PHATE, which exhibit distinct clusters but worse RMSE and distance correlation, because they introduce more global distortion.
%
LLE and the spectral embedding generally are ranked worse in terms of magnitude and other measures, confirming the visual suspicion that they introduce spurious structures.
%
Thus, since tracking the number of distinct points is related to clustering behaviour, we have shown that magnitude weight difference emphasises methods that preserve both individual groups and the relationship between them.
\paragraph{Magnitude finds suitable embeddings.}
%
We find that in comparison to other quality measures, magnitude weight difference is correlated with distance correlation~($r=-0.85$), RMSE~($r=0.81$) and density preservation~($r=-0.79$), lending credence to the numerous beneficial geometrical and topological properties captured by magnitude.
%
However, it still regards t-SNE as a suitable embedding, despite its comparatively low distance correlation, high local density preservation, and strong preservation of clusters.
%
This highlights that magnitude weight difference offers a more holistic view of an embedding than existing measures: in fact, although magnitude is related to ideas such as distance preservation and density preservation, magnitude in fact does \emph{not} measure the same structure as existing quality metrics, but finds a compromise between preserving the global and local geometry of data.
%
Furthermore, while distance correlation is limited to comparing individual pairwise distances, and local density preservation is limited by its choice of static local neighbourhoods, the magnitude difference compares \emph{all} distances simultaneously across multiple scales without the need to manually pick a neighbourhood size, constituting a parameter-free method~(except for the choice of convergence point, which we argue can be made canonically).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Single-cell Data Analysis and Visualisation}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Finally, we assess the utility of magnitude in a single-cell data analysis context.
%
Here, DR methods are commonly used for simplifying gene expression features prior to analysis or for explanatory data analysis.
%
However, due to the complex nature of these data and challenges in assessing the loss of global, local, or distance-related information after dimensionality reduction, visualising %these data
and interpreting the resulting representations remains an important open problem~\citep{lahnemann2020eleven}.
%
Motivated by this, we use magnitude difference to assess embedding quality for a data set of human peripheral blood mononuclear cells~(PBMCs), consisting of $2,638$ cells and $1,838$ features.
\begin{wrapfigure}{l}{0.40\textwidth}
\centering
\includegraphics[width=\linewidth]{Figures/pbm_pca.pdf}
%
\caption{%
\textbf{Re-scaled magnitude functions of the single-cell data for varying numbers of principal components~(PCs).}
%
The difference to the original magnitude function serves as a reliable indicator of information loss and distortions as the number of PCs decreases.
}
\label{fig:pca_single_cell}
\end{wrapfigure}
%
When following the common practice of using PCA for initial pre-processing~\citep{heumos2023best},
we see that the magnitude profile and weight difference consistently increases as the dimensionality decreases as plotted in \cref{fig:pca_single_cell}.
%
The first two principal components~(PCs) constitute an unsuitable representation of these data, and the corresponding magnitude weight difference is noticeably higher than for any higher-dimensional projections.
%
Meanwhile, the first 50 PCs capture the trend in the magnitude function of the original space, confirming that magnitude difference accurately encodes the loss of information introduced via preprocessing.
%
Next, we compare the visualisation of these data using the first two PCs against t-SNE, UMAP or PHATE plots each computed after an initial reduction to 30, 40, or 50 PCs, following common practice.
%
In terms of magnitude weight difference, from the full results summarised in \cref{tab:full_results_sc} in the appendix, we find that UMAP, with an average magnitude weight difference of $0.395$, consistently performs better than t-SNE, with a mean difference of $0.443$, which however achieved a lower dissimilarity than PHATE with a value of $0.456$.
%
Meanwhile, the magnitude weight difference for two PCs is higher than for t-SNE
supporting that, compared to t-SNE, this linear projection better preserves local density and pairwise distances as given in \cref{tab:still_not_a_duck}.
%
Interestingly, these results confirm existing analyses, which found that
UMAP is often more faithful to the true biological signal than t-SNE \citep{heumos2023best}, and that it better captures the
distinct cell clusters
visible in the data than PHATE, which is more suitable for describing continuous cell states~\citep{moon2019visualizing}.
%
Indeed the visualisations given in \cref{fig:not_a_duck} support the observation that PHATE introduces visibly more global distortion than the other manifold learning methods as measured by its noticeably higher magnitude weight difference.
Nevertheless, PHATE still achieves better results than UMAP in terms of local density preservation and does comparably well in terms of trustworthiness and distance correlation.
%
Magnitude can thus successfully detect that the continuous trajectory structures shown by PHATE are an unsuitable choice for representing the clusters in these data, whereas alternative measures fail to make this distinction.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussion}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Using \emph{magnitude}, a multi-scale invariant of metric spaces, we have derived a novel measure of the quality of embedding algorithms.
%
To this end, we have formalised innovative notions \emph{difference} between appropriately-scaled magnitude functions as well as magnitude weights, and demonstrated
their utility as embedding quality measures across multiple representation learning tasks, using both real and simulated data.
%
Magnitude reliably detects the underlying ground truth in the context of manifold learning and turns out to be capable of determining the most faithful visualisation of single-cell data, thus preserving both the local and the global geometry of a dataset.
%
% Supporting this, we demonstrated the stability of magnitude both theoretically and empirically and improved the computational efficiency of magnitude calculations.
%in the context of manifold learning, data visualisation and
%
For future work, we believe that the proposed magnitude profile differences exhibit a strong potential for applications to unaligned spaces, for example when integrating multimodal single-cell data.
%
There are also promising opportunities to further advance the scalability of magnitude calculations, leveraging either subsampling or numerical approximation techniques.
%
Finally, integrating magnitude into deep learning models would be beneficial to obtain novel geometry-based regularisation strategies.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection*{Acknowledgments}
K.L is supported by the Helmholtz Association under the joint research school “Munich School for Data Science - MUDS”. R.A. is supported by the United
Kingdom Research and Innovation (grant EP/S02431X/1),
UKRI Centre for Doctoral Training in Biomedical AI at the
University of Edinburgh, School of Informatics and the International Helmholtz-Edinburgh Research School for Epigenetics (EpiCrossBorders).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliography{iclr2024_conference}
\bibliographystyle{iclr2024_conference}
\newpage
\appendix
\section{Appendix}
\subsection{Proofs}
\lemisometryinvariance*
\begin{proof}
Let $(X, d_X)$ and $(Y, d_Y)$ be metric spaces with cardinality $n$ and let $f\colon X \to Y$ denote their isometry.
%
Then, the similarity matrix of $X$ is $\zeta_{ij}^{X} = e^{-d_{X}({a_i, a_j})}$.
%
Since $f$ is an isometry, we have, $d_{X}({a_i, a_j}) = d_{Y}(f(a_i), f(a_j))$.
%
Hence, $\zeta_{ij}^{X} = e^{-d_{X}({a_i, a_j})} = e^{-d_{Y}(f(a_i), f(a_j))} = \zeta_{ij}^{Y}$.
%
Since $X$ and $Y$ have the same similarity matrix, we have $\magnitude_{X} = \magnitude_{Y}$.
\end{proof}
\lemspectralbound*
\begin{proof}
Because $\|\cdot\|_2$ is a \emph{consistent} norm, we have $\|f(M)\|_2 \leq \|\ones^\top\|_2 \|M\|_2 \|\ones\|_2 = n \|M\|_2$ for all $n \times n$ matrices~$M$.
%
Without loss of generality, assume that $\|f(A)\|_2 \geq \|f(B)\|_2$ and $\|A\|_2 \geq \|B\|_2$.
%
Thus, $\|f(A)\|_2 - \|f(B)\|_2 \leq d(\|A\|_2 - \|B\|_2) \leq d(\|A - B\|_2) = n\epsilon$.
\end{proof}
\lemsimilaritymatrixspectralbound*
\begin{proof}
As a consequence of the continuity of the exponential function, we know that there is $\delta$ such that
$|\zeta_X - \zeta_Y| < n^{-1} \epsilon$.
%
The row sums of $\zeta_X - \zeta_Y$ are therefore upper-bounded by $\epsilon$.
%
We thus have $\|\zeta_X - \zeta_Y\|_2 \leq \epsilon$~\citep[Theorem~1.1, p.~24]{Minc88}.
\end{proof}
\subsection{Methods for Computing Magnitude}
When computing magnitude differences in practice, only two parameter choices are necessary - the choice of comparable convergence points and the choice of evaluation values at which to empirically evaluate magnitude.
For the Swiss Roll example, we choose to scale all magnitude functions until 95\% of points are visible and evaluate each function across 64 equally spaced intervals. For the MNIST and the PBMCs data, we again compute the magnitude difference between lower dimensional representations and the original datasets until a desired value of magnitude chosen to equal 95\% of the cardinality across 32 intervals. Across all examples, we approximate the difference in magnitude functions and difference in magnitude weight functions using Romberg integration. Any duplicate observations in the datasets or embeddings analysed throughout this study were removed to guarantee that magnitude can be computed for these spaces.
\subsection{Single Cell Data Analysis}
We first try out method on a well-known dataset on human peripheral blood mononuclear cells (PBMCs) from one healthy donor provided by 10x Genomics that is available through scanpy and has been processed following Seurat's guided tutorial \citep{satija2015spatial, wolf2018scanpy}.
After processing, the data consists of
2,638 cells with
1,838 highly variable genes and
eight annotated cell type clusters.
As given in Figure \ref{fig:pca_single_cell}, we first compare the magnitude difference between the "original" space, that is the space spanned by all genes, to the first 2, 4, 5, 10, 30, 40 and 50 principal components calculated from these data. Indeed, we see that as the number of PCs decreases, the magnitude difference consistently increases. The two dimensional representation is by far the worse projection, whereas the 50 PCs closest capture the trend in magnitude of the original space confirming our intuition.
\newpage
\subsection{Extended Results on Magnitude Differences}
For completeness, Figure \ref{fig:magnitude_swiss} and \ref{fig:magnitude_mnist} report on specific examples of the re-scaled magnitude functions and weight differences computed across multiple scales for the MNIST data and Swiss Roll simulation considered in the main body of the paper.
\begin{figure}[ht!]
\centering
\centerline{\includegraphics[width=1.2\textwidth]{Figures/sr_plot.pdf}}
\caption{ \textbf{Re-scaled magnitude functions and magnitude differences amongst different embeddings for the Swiss Roll.} Magnitude reliably detects the ground truth.
}
\label{fig:magnitude_swiss}
\end{figure}
\begin{figure}[ht!]
\centering
\centerline{\includegraphics[width=1.2\textwidth]{Figures/mnist_plot.pdf}}
\caption{ \textbf{Re-scaled magnitude functions and magnitude differences amongst different embeddings for the digits data.} Smaller magnitude differences are related to better clustering preservation.
}
\label{fig:magnitude_mnist}
\end{figure}
\newpage
\subsection{Extended Comparison Between Embedding Quality Measures}
\label{app:sec:Extended Comparison}
In the results reported throughout this paper, we choose to compare the magnitude weight difference to the Spearman correlation in pairwise Euclidean distances (DC), the Spearman correlation in pairwise spectral distances (DCS), the preservation of local density (DP) and the RMSE in Euclidean distances. Further, the tables below shows a more complete comparison between other embedding quality measures, dimensionality reduction methods and both magnitude weight and magnitude profile differences extending the results in the main texts.
For all neighbourhood-based embedding quality measures k=30 was choosen by default.
Finally, we choose to report on the embedding quality measures listed below across an extended list of dimensionality reduction methods for each of the datasets considered throughout this study as reported in \cref{tab:sr_measures_appendix}, \cref{tab:full_results} and \cref{tab:full_results_sc}:
\begin{itemize}
\item MW\_2: Magnitude weight difference using Euclidean distances
\item MW\_2: Magnitude profile difference using Euclidean distances
\item DC: Spearman correlation coefficient between pairwise Euclidean distances
\item DCS: Spearman correlation coefficient between pairwise spectral distances
\item RMSE: Root mean squared error between pairwise Euclidean distances
\item stress: Normalized stress measure
\item DP: Local density preservation
\item TW: Trustworthiness
\item NL: Neighbourhood loss
\item C: Continuity
\item M: Mean relative rank error between original and embedding
\item M1: Mean relative rank error between embedding and original
\item DG: Density preservation global (Gaussian kernel)
\item DKLG: Kullback–Leibler divergence between the density estimates (Gaussian kernel)
\item DGL: Density preservation global (Laplacian kernel)
\item DKLGL: Kullback–Leibler divergence between the density estimates (Laplacian kernel)
\item SS: Silhouette score
\end{itemize}
\begin{sidewaystable}[!ht]
\centering
\caption{
Quality measures of different embeddings for the Swiss Roll.
}
\label{tab:sr_measures_appendix}
\resizebox{\textheight}{!}{%}
\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|}
\hline
~ & Emb & MW\_2 & MF\_2 & DC & DCS & RMSE & stess & DP & TW & NL & C & M & M1 & DG & DKLG & DGL & DKLGL \\ \hline
0 & original & 0.0 & 0.0 & 1.0 & 0.026 & 0.0 & 0.0 & 1.0 & 1.0 & 0.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ \hline
1 & truth & 0.129 & 0.007 & 0.459 & 0.081 & 8.59 & 0.894 & 0.409 & 0.797 & 0.589 & 0.868 & 0.109 & 0.079 & 0.222 & 0.034 & 0.237 & 0.041 \\ \hline
2 & mds & 0.13 & 0.021 & 0.883 & -0.001 & 3.214 & 0.202 & 0.52 & 0.928 & 0.347 & 0.966 & 0.077 & 0.037 & 0.084 & 0.007 & 0.127 & 0.014 \\ \hline
3 & pca & 0.144 & 0.037 & 0.861 & -0.029 & 3.89 & 0.274 & 0.343 & 0.901 & 0.367 & 0.973 & 0.109 & 0.027 & 0.156 & 0.021 & 0.195 & 0.034 \\ \hline
4 & isomap & 0.146 & 0.028 & 0.844 & -0.032 & 8.661 & 0.366 & 0.248 & 0.923 & 0.343 & 0.972 & 0.087 & 0.029 & 0.177 & 0.027 & 0.23 & 0.045 \\ \hline
5 & d\_no\_y & 0.146 & 0.024 & 0.799 & 0.029 & 4.697 & 0.349 & 0.466 & 0.869 & 0.452 & 0.963 & 0.148 & 0.035 & 0.173 & 0.019 & 0.189 & 0.025 \\ \hline
6 & ran\_un\_9 & 0.148 & 0.044 & -0.001 & 0.001 & 15.788 & 27.714 & 0.028 & 0.522 & 0.904 & 0.52 & 0.493 & 0.495 & 0.293 & 0.065 & 0.298 & 0.07 \\ \hline
7 & d\_no\_z & 0.149 & 0.027 & 0.709 & 0.058 & 5.701 & 0.45 & 0.541 & 0.825 & 0.499 & 0.952 & 0.203 & 0.043 & 0.151 & 0.017 & 0.154 & 0.021 \\ \hline
8 & d\_r\_n & 0.152 & 0.049 & -0.008 & -0.027 & 14.631 & 7.215 & -0.073 & 0.523 & 0.896 & 0.523 & 0.491 & 0.49 & 0.284 & 0.073 & 0.368 & 0.137 \\ \hline
9 & d\_el & 0.152 & 0.037 & 0.713 & 0.025 & 68.847 & 0.819 & 0.272 & 0.862 & 0.511 & 0.933 & 0.159 & 0.055 & 0.171 & 0.019 & 0.206 & 0.032 \\ \hline
10 & lle & 0.155 & 0.034 & 0.852 & -0.006 & 16.153 & 139.889 & 0.449 & 0.897 & 0.396 & 0.971 & 0.121 & 0.03 & 0.148 & 0.018 & 0.171 & 0.026 \\ \hline
11 & tsne & 0.155 & 0.109 & 0.706 & -0.018 & 142.14 & 0.905 & 0.433 & 0.943 & 0.328 & 0.913 & 0.045 & 0.077 & 0.173 & 0.031 & 0.173 & 0.031 \\ \hline
12 & umap & 0.163 & 0.044 & 0.643 & 0.035 & 12.411 & 2.947 & 0.408 & 0.981 & 0.259 & 0.946 & 0.019 & 0.033 & 0.207 & 0.034 & 0.2 & 0.031 \\ \hline
13 & phate & 0.172 & 0.075 & 0.406 & -0.001 & 16.193 & 201.335 & 0.316 & 0.91 & 0.41 & 0.846 & 0.066 & 0.071 & 0.198 & 0.03 & 0.205 & 0.034 \\ \hline
14 & d\_no\_x & 0.21 & 0.152 & 0.844 & -0.01 & 4.359 & 0.319 & 0.426 & 0.89 & 0.445 & 0.966 & 0.137 & 0.04 & 0.134 & 0.013 & 0.135 & 0.015 \\ \hline
\end{tabular}
}
\end{sidewaystable}
\begin{sidewaystable}[!ht]
\centering
\caption{
Quality measures of different embeddings for MNIST.
}
\label{tab:full_results}
\resizebox{\textheight}{!}{%}
\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|}
\hline
~ & Emb & MW2 & MF2 & DC & DCS & RMSE & stess & DP & TW & NL & C & M & M1 & RIE & RIA & DG & DKLG & DGL & DKLGL & SS \\ \hline
0 & Or & 0.0 & 0.0 & 1.0 & 0.074 & 0.0 & 0.0 & 1.0 & 1.0 & 0.0 & 1.0 & 0.0 & 0.0 & 1.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.205 \\ \hline
1 & MDS & 0.259 & 0.233 & 0.779 & 0.064 & 15.314 & 0.327 & 0.583 & 0.907 & 0.646 & 0.93 & 0.098 & 0.069 & 0.922 & 0.854 & 0.238 & 0.042 & 0.189 & 0.029 & 0.297 \\ \hline
2 & tsne & 0.289 & 0.267 & 0.571 & -0.015 & 21.343 & 0.374 & 0.425 & 0.985 & 0.356 & 0.98 & 0.012 & 0.016 & 0.918 & 0.725 & 0.314 & 0.085 & 0.25 & 0.05 & 0.673 \\ \hline
3 & NCA & 0.294 & 0.268 & 0.687 & 0.049 & 28.25 & 0.409 & 0.1 & 0.883 & 0.674 & 0.953 & 0.118 & 0.044 & 0.923 & 0.74 & 0.366 & 0.102 & 0.351 & 0.097 & 0.374 \\ \hline
4 & pca & 0.3 & 0.275 & 0.693 & 0.047 & 23.821 & 0.808 & 0.175 & 0.875 & 0.677 & 0.95 & 0.133 & 0.046 & 0.917 & 0.854 & 0.393 & 0.118 & 0.373 & 0.106 & 0.326 \\ \hline
5 & Iso & 0.312 & 0.285 & 0.649 & 0.035 & 41.245 & 0.494 & 0.484 & 0.902 & 0.613 & 0.968 & 0.106 & 0.033 & 0.922 & 0.803 & 0.366 & 0.109 & 0.298 & 0.069 & 0.439 \\ \hline
6 & SLLE & 0.322 & 0.293 & 0.416 & 0.075 & 49.265 & 810 & 0.144 & 0.825 & 0.757 & 0.883 & 0.174 & 0.095 & 0.708 & 0.033 & 0.364 & 0.129 & 0.514 & 0.24 & 0.084 \\ \hline
7 & umap & 0.324 & 0.3 & 0.521 & -0.003 & 37.667 & 2.803 & 0.127 & 0.985 & 0.369 & 0.978 & 0.018 & 0.018 & 0.855 & 0.632 & 0.379 & 0.143 & 0.28 & 0.065 & 0.736 \\ \hline
8 & phate & 0.325 & 0.298 & 0.578 & 0.011 & 49.279 & 1190 & 0.388 & 0.966 & 0.49 & 0.974 & 0.036 & 0.022 & 0.891 & 0.642 & 0.329 & 0.086 & 0.263 & 0.055 & 0.572 \\ \hline
9 & Sp & 0.356 & 0.322 & 0.554 & -0.001 & 49.312 & 8516 & 0.43 & 0.923 & 0.628 & 0.96 & 0.084 & 0.043 & 0.815 & 0.713 & 0.456 & 0.193 & 0.431 & 0.136 & 0.55 \\ \hline
\end{tabular}
}
\end{sidewaystable}
\begin{sidewaystable}[!ht]
\centering
\caption{
Quality measures of different embeddings for the single cell data.
}
\label{tab:full_results_sc}
\resizebox{\textheight}{!}{%}
\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|}
\hline
~ & Em & MW1 & MF1 & MW2 & MF2 & DC & DCS & RMSE & stess & DP & TW & NL & C & M & M1 & RIE & RIA & DG & DKLG & DGL & DKLGL & SS \\ \hline
0 & Or & 0.0 & 0.0 & 0.0 & 0.0 & 1.0 & -0.024 & 0.0 & 0.0 & 1.0 & 1.0 & 0.0 & 1.0 & 0.0 & 0.0 & 1.0 & 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.01 \\ \hline
1 & 50 & 0.193 & 0.019 & 0.178 & 0.078 & 0.758 & 0.067 & 37.5 & 1.903 & 0.879 & 0.84 & 0.887 & 0.869 & 0.149 & 0.113 & 0.999 & 0.991 & 0.149 & 0.017 & 0.118 & 0.012 & 0.128 \\ \hline
2 & 40 & 0.198 & 0.031 & 0.186 & 0.089 & 0.739 & 0.072 & 38.5 & 2.051 & 0.873 & 0.822 & 0.9 & 0.855 & 0.168 & 0.125 & 0.999 & 0.989 & 0.153 & 0.018 & 0.125 & 0.013 & 0.146 \\ \hline
3 & 30 & 0.203 & 0.049 & 0.196 & 0.106 & 0.71 & 0.078 & 39.8 & 2.264 & 0.861 & 0.795 & 0.919 & 0.833 & 0.197 & 0.147 & 0.999 & 0.986 & 0.159 & 0.02 & 0.134 & 0.015 & 0.176 \\ \hline
4 & 20 & 0.212 & 0.077 & 0.214 & 0.131 & 0.654 & 0.084 & 41.7 & 2.594 & 0.821 & 0.747 & 0.945 & 0.795 & 0.251 & 0.186 & 0.999 & 0.975 & 0.169 & 0.022 & 0.154 & 0.02 & 0.233 \\ \hline
5 & 10 & 0.233 & 0.135 & 0.247 & 0.186 & 0.583 & 0.09 & 43.9 & 3.089 & 0.721 & 0.674 & 0.969 & 0.738 & 0.328 & 0.248 & 0.999 & 0.96 & 0.179 & 0.025 & 0.185 & 0.028 & 0.362 \\ \hline
6 & 5 & 0.276 & 0.216 & 0.303 & 0.266 & 0.529 & 0.096 & 45.6 & 3.512 & 0.583 & 0.635 & 0.977 & 0.718 & 0.365 & 0.268 & 0.996 & 0.93 & 0.186 & 0.027 & 0.216 & 0.037 & 0.458 \\ \hline
7 & 4 & 0.291 & 0.244 & 0.32 & 0.293 & 0.519 & 0.106 & 46.1 & 3.657 & 0.568 & 0.626 & 0.977 & 0.709 & 0.375 & 0.274 & 0.987 & 0.934 & 0.188 & 0.027 & 0.2 & 0.033 & 0.42 \\ \hline
8 & 40\_u & 0.35 & 0.33 & 0.39 & 0.379 & 0.331 & 0.043 & 49.8 & 6.357 & 0.193 & 0.625 & 0.974 & 0.675 & 0.377 & 0.326 & 0.257 & 0.193 & 0.262 & 0.056 & 0.27 & 0.056 & 0.492 \\ \hline
9 & 30\_u & 0.354 & 0.335 & 0.397 & 0.386 & 0.352 & 0.054 & 49.3 & 5.801 & 0.173 & 0.628 & 0.975 & 0.712 & 0.375 & 0.277 & 0.182 & -0.502 & 0.277 & 0.063 & 0.277 & 0.06 & 0.496 \\ \hline
10 & 50\_u & 0.36 & 0.339 & 0.399 & 0.388 & 0.348 & 0.053 & 50.3 & 7.027 & 0.174 & 0.626 & 0.974 & 0.719 & 0.373 & 0.262 & 0.434 & 0.07 & 0.265 & 0.057 & 0.265 & 0.055 & 0.492 \\ \hline
11 & 2 & 0.397 & 0.375 & 0.437 & 0.424 & 0.447 & 0.07 & 48.1 & 4.767 & 0.407 & 0.611 & 0.98 & 0.716 & 0.394 & 0.268 & 0.255 & 0.382 & 0.225 & 0.046 & 0.328 & 0.086 & 0.408 \\ \hline
12 & 50\_t & 0.395 & 0.376 & 0.438 & 0.427 & 0.334 & 0.065 & 29.3 & 0.781 & -0.016 & 0.643 & 0.969 & 0.735 & 0.34 & 0.248 & 0.173 & -0.413 & 0.245 & 0.052 & 0.208 & 0.034 & 0.453 \\ \hline
13 & 40\_t & 0.403 & 0.381 & 0.442 & 0.43 & 0.304 & 0.104 & 28.9 & 0.803 & -0.233 & 0.635 & 0.973 & 0.602 & 0.36 & 0.388 & 0.4 & -0.327 & 0.234 & 0.044 & 0.203 & 0.033 & 0.4 \\ \hline
14 & 30\_t & 0.407 & 0.387 & 0.449 & 0.438 & 0.371 & 0.068 & 26.3 & 0.523 & -0.041 & 0.643 & 0.971 & 0.727 & 0.341 & 0.257 & 0.179 & -0.562 & 0.229 & 0.045 & 0.204 & 0.033 & 0.431 \\ \hline
15 & 30\_p & 0.415 & 0.389 & 0.453 & 0.439 & 0.348 & 0.049 & 56 & 1786 & 0.271 & 0.62 & 0.978 & 0.702 & 0.377 & 0.292 & 0.295 & 0.814 & 0.331 & 0.09 & 0.435 & 0.156 & 0.598 \\ \hline
16 & 40\_p & 0.415 & 0.39 & 0.454 & 0.44 & 0.334 & 0.044 & 56 & 1817 & 0.322 & 0.618 & 0.978 & 0.691 & 0.384 & 0.296 & 0.277 & 0.856 & 0.323 & 0.086 & 0.428 & 0.15 & 0.617 \\ \hline
17 & 50\_p & 0.421 & 0.395 & 0.46 & 0.444 & 0.358 & 0.045 & 56 & 1821 & 0.298 & 0.622 & 0.976 & 0.714 & 0.379 & 0.273 & 0.421 & 0.558 & 0.327 & 0.088 & 0.435 & 0.156 & 0.62 \\ \hline
\end{tabular}
}
\end{sidewaystable}
% FIXME: table breaks compiler
%\include{Figures/sc_table}
\newpage
\subsection{Example of Calculating Magnitude for Planets Data}
We further decided to add another toy example on a dataset with measurements on eight planets to demonstrate the calculation of magnitude weights and the magnitude function.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[h!]
\centering
\subcaptionbox{Metric space}{%
\sisetup{%
detect-all = true,
detect-mode = true,
group-minimum-digits = 4,
}
\scriptsize%
\begin{tabular}{lS[table-format=6]S[table-format=4]S[table-format=2.1]}
\toprule
\textsc{Planet} & \textsc{Diameter} & \textsc{Density} & \textsc{Gravity}\\
\midrule
Mercury & 4879 & 5429 & 3.7\\
Venus & 12104 & 5243 & 8.9\\
Earth & 12756 & 5514 & 9.8\\
Mars & 6792 & 3934 & 3.7\\
Jupiter & 142984 & 1326 & 23.1\\
Saturn & 120536 & 687 & 9.0\\
Uranus & 51118 & 1270 & 8.7\\
Neptune & 49528 & 1638 & 11.0\\
\bottomrule
\end{tabular}
%
% This is a poor way of aligning this stuff; feel free to improve.
\vspace{1cm}
}%
\subcaptionbox{Magnitude function}{%
\begin{tikzpicture}[font = \scriptsize]
\tikzset{%
pin distance = 7ex,
%
every pin/.style ={%
font = \tiny,
draw = black,
fill = white,
rectangle,
},
small dot/.style = {
circle,
fill = black,
scale = 0.3,
},
}
\begin{axis}[
axis x line* = bottom,
axis y line* = left,
enlargelimits = false,
xlabel = {$t$},
ylabel = {Magnitude},
width = 5cm,
height = 5cm,
tick align = outside,
ytick = {2, 4, 6, 8},
ymax = 8.0,
clip = false,
]
\addplot[black] coordinates {
( 0.00, 1.00)
( 2.04, 2.20)
( 4.08, 3.20)
( 6.12, 4.00)
( 8.16, 4.50)
( 10.20, 4.90)
( 12.25, 5.10)
( 14.29, 5.30)
( 16.33, 5.50)
( 18.37, 5.60)
( 20.41, 5.70)
( 22.45, 5.90)
( 24.49, 6.00)
( 26.53, 6.10)
( 28.57, 6.20)
( 30.61, 6.20)
( 32.65, 6.30)
( 34.69, 6.40)
( 36.74, 6.50)
( 38.78, 6.60)
( 40.82, 6.60)
( 42.86, 6.70)
( 44.90, 6.80)
( 46.94, 6.80)
( 48.98, 6.90)
( 51.02, 7.00)
( 53.06, 7.00)
( 55.10, 7.10)
( 57.14, 7.10)
( 59.18, 7.20)
( 61.22, 7.20)
( 63.27, 7.20)
( 65.31, 7.30)
( 67.35, 7.30)
( 69.39, 7.40)
( 71.43, 7.40)
( 73.47, 7.40)
( 75.51, 7.50)
( 77.55, 7.50)
( 79.59, 7.50)
( 81.63, 7.60)
( 83.67, 7.60)
( 85.71, 7.60)
( 87.76, 7.60)
( 89.80, 7.60)
( 91.84, 7.70)
( 93.88, 7.70)
( 95.92, 7.70)
( 97.96, 7.70)
(100.00, 7.70)
};
\node[small dot, pin= 45:{$ 0.1, 1.05$}] at ( 0.1, 1.05) {};
\node[small dot, pin= 70:{$ 1, 1.57$}] at ( 1.0, 1.57) {};
\node[small dot, pin= 90:{$ 10, 4.87$}] at ( 10.0, 4.87) {};
\node[small dot, pin=-90:{$100, 7.78$}] at (100.0, 7.78) {};
\end{axis}
\end{tikzpicture}
}%
\subcaptionbox{Magnitude weights}{%
\begin{tikzpicture}[font = \scriptsize]
\begin{axis}[%
axis line style = {
draw = none,
},
%
colormap/Greens,
%
enlargelimits = false,
tick align = outside,
unit vector ratio* = 1 1 1,
xtick = data,
ytick = data,
xticklabels = {$0.1$, $1$, $10$, $100$},
yticklabels = {Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune},
xtick pos = bottom,
ytick pos = left,
xlabel = {$t$},
height = 5cm,
]
\addplot[%
matrix plot,
draw = gray,
mesh/ordering = y varies,
point meta = explicit
] coordinates {
(0, 0) [1.00]
(0, 1) [0.04]
(0, 2) [0.02]
(0, 3) [0.08]
(0, 4) [0.21]
(0, 5) [0.04]
(0, 6) [0.02]
(0, 7) [-0.01]
(1, 0) [1.00]
(1, 1) [0.14]
(1, 2) [0.07]
(1, 3) [0.26]
(1, 4) [0.64]
(1, 5) [0.24]
(1, 6) [0.13]
(1, 7) [-0.01]
(2, 0) [1.00]
(2, 1) [0.43]
(2, 2) [0.24]
(2, 3) [0.78]
(2, 4) [1.00]
(2, 5) [0.99]
(2, 6) [0.94]
(2, 7) [0.41]
(3, 0) [1.00]
(3, 1) [0.98]
(3, 2) [0.92]
(3, 3) [1.00]
(3, 4) [1.00]
(3, 5) [1.00]
(3, 6) [1.00]
(3, 7) [0.98]
};
\end{axis}
\end{tikzpicture}
}%
\caption{%
\textbf{Example of magnitude} for a metric space with eight points.
}
\label{fig:magnitude_visualisation_planets}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{example}
Consider the magnitude function of the metric space consisting of 5 data points%consisting of eight planets described in Example ??? and plotted in Figure \ref{fig:magnitude_visualisation}.
$(X, d)$ where each observation %$x\in X \subset \mathbb{R}^3$
corresponds to standard-scaled measurements of the mass, diameter and density of one of the eight planets in our solar system as plotted in Figure \ref{fig:magnitude_visualisation_planets}. When the zooming scale $t$ is very small, for example $t=0.01$, the magnitude weights of all points sum up to 1 %magnitude is very close to 1
and the space effectively looks like one point. Following this, magnitude grows so that at $t=1$, around $1.6$ distinct points are visible, which increases to almost five at scale $t=10$.
Finally, when $t$ is large, for example $t=100$, all the points are clearly separated, their magnitude weights converge to one and the value of magnitude approaches 8, which equals the cardinality of the space.
\end{example}
\end{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection*{Author Contributions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Everyone writes, no one quits.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection*{Acknowledgments}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Let's acknowledge everyone.
\begin{thebibliography}{26}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Adamer et~al.(2021)Adamer, De~Brouwer, O'Bray, and Rieck]{adamer2021magnitude}
Michael~F Adamer, Edward De~Brouwer, Leslie O'Bray, and Bastian Rieck.
\newblock The magnitude vector of images.
\newblock \emph{arXiv preprint arXiv:2110.15188}, 2021.
\bibitem[Andreeva et~al.(2023)Andreeva, Limbeck, Rieck, and Sarkar]{andreeva2023metric}
Rayna Andreeva, Katharina Limbeck, Bastian Rieck, and Rik Sarkar.
\newblock Metric space magnitude and generalisation in neural networks.
\newblock \emph{arXiv preprint arXiv:2305.05611}, 2023.
\bibitem[Barannikov et~al.(2022)Barannikov, Trofimov, Balabin, and Burnaev]{Barannikov22a}
Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev.
\newblock Representation topology divergence: A method for comparing neural network representations.
\newblock In Kamalika Chaudhuri, Stefanie Jegelka, Le~Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), \emph{Proceedings of the 39th International Conference on Machine Learning}, number 162 in Proceedings of Machine Learning Research, pp.\ 1607--1626. PMLR, 2022.
\bibitem[Bunch et~al.(2021)Bunch, Kline, Dickinson, Bhat, and Fung]{bunch2021weighting}
Eric Bunch, Jeffery Kline, Daniel Dickinson, Suhaas Bhat, and Glenn Fung.
\newblock Weighting vectors for machine learning: numerical harmonic analysis applied to boundary detection.
\newblock \emph{arXiv preprint arXiv:2106.00827}, 2021.
\bibitem[Chari \& Pachter(2023)Chari and Pachter]{chari2023specious}
Tara Chari and Lior Pachter.
\newblock The specious art of single-cell genomics.
\newblock \emph{PLOS Computational Biology}, 19\penalty0 (8):\penalty0 e1011288, 2023.
\bibitem[Feragen et~al.(2015)Feragen, Lauze, and Hauberg]{Feragen15a}
Aasa Feragen, Francois Lauze, and Soren Hauberg.
\newblock Geodesic exponential kernels: When curvature and linearity conflict.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition~(CVPR)}, 2015.
\bibitem[Govc \& Hepworth(2021)Govc and Hepworth]{govc2021persistent}
Dejan Govc and Richard Hepworth.
\newblock Persistent magnitude.
\newblock \emph{Journal of Pure and Applied Algebra}, 225\penalty0 (3):\penalty0 106517, 2021.
\bibitem[Heumos et~al.(2023)Heumos, Schaar, Lance, Litinetskaya, Drost, Zappia, L{\"u}cken, Strobl, Henao, Curion, {Single-cell Best Practices Consortium}, Aliee, Ansari, Badia-i Mompel, B{\"u}ttner, Dann, Dimitrov, Dony, Frishberg, He, Hediyeh-zadeh, Hetzel, Ibarra, Jones, Lotfollahi, Martens, M{\"u}ller, Nitzan, Ostner, Palla, Patro, Piran, Ram{\'i}rez-Su{\'a}stegui, Saez-Rodriguez, Sarkar, Schubert, Sikkema, Srivastava, Tanevski, Virshup, Weiler, Schiller, and Theis]{heumos2023best}
Lukas Heumos, Anna~C. Schaar, Christopher Lance, Anastasia Litinetskaya, Felix Drost, Luke Zappia, Malte~D. L{\"u}cken, Daniel~C. Strobl, Juan Henao, Fabiola Curion, {Single-cell Best Practices Consortium}, Hananeh Aliee, Meshal Ansari, Pau Badia-i Mompel, Maren B{\"u}ttner, Emma Dann, Daniel Dimitrov, Leander Dony, Amit Frishberg, Dongze He, Soroor Hediyeh-zadeh, Leon Hetzel, Ignacio~L. Ibarra, Matthew~G. Jones, Mohammad Lotfollahi, Laura~D. Martens, Christian~L. M{\"u}ller, Mor Nitzan, Johannes Ostner, Giovanni Palla, Rob Patro, Zoe Piran, Ciro Ram{\'i}rez-Su{\'a}stegui, Julio Saez-Rodriguez, Hirak Sarkar, Benjamin Schubert, Lisa Sikkema, Avi Srivastava, Jovan Tanevski, Isaac Virshup, Philipp Weiler, Herbert~B. Schiller, and Fabian~J. Theis.
\newblock Best practices for single-cell analysis across modalities.
\newblock \emph{Nature Reviews Genetics}, pp.\ 1--23, 2023.
\bibitem[Higham(2009)]{Higham09a}
Nicholas~J. Higham.
\newblock Cholesky factorization.
\newblock \emph{Wiley Interdisciplinary Reviews: Computational Statistics}, 1\penalty0 (2):\penalty0 251--254, 2009.
\newblock \doi{10.1002/wics.18}.
\bibitem[L{\"a}hnemann et~al.(2020)L{\"a}hnemann, K{\"o}ster, Szczurek, McCarthy, Hicks, Robinson, Vallejos, Campbell, Beerenwinkel, Mahfouz, Pinello, Skums, Stamatakis, Attolini, Aparicio, Baaijens, Balvert, Barbanson, Cappuccio, Corleone, Dutilh, Florescu, Guryev, Holmer, Jahn, Lobo, Keizer, Khatri, Kielbasa, Korbel, Kozlov, Kuo, Lelieveldt, Mandoiu, Marioni, Marschall, M{\"o}lder, Niknejad, Raczkowski, Reinders, Ridder, Saliba, Somarakis, Stegle, Theis, Yang, Zelikovsky, McHardy, Raphael, Shah, and Sch{\"o}nhuth]{lahnemann2020eleven}
David L{\"a}hnemann, Johannes K{\"o}ster, Ewa Szczurek, Davis~J. McCarthy, Stephanie~C. Hicks, Mark~D. Robinson, Catalina~A. Vallejos, Kieran~R. Campbell, Niko Beerenwinkel, Ahmed Mahfouz, Luca Pinello, Pavel Skums, Alexandros Stamatakis, Camille Stephan-Otto Attolini, Samuel Aparicio, Jasmijn Baaijens, Marleen Balvert, Buys~de Barbanson, Antonio Cappuccio, Giacomo Corleone, Bas~E. Dutilh, Maria Florescu, Victor Guryev, Rens Holmer, Katharina Jahn, Thamar~Jessurun Lobo, Emma~M. Keizer, Indu Khatri, Szymon~M. Kielbasa, Jan~O. Korbel, Alexey~M. Kozlov, Tzu-Hao Kuo, Boudewijn~P.F. Lelieveldt, Ion~I. Mandoiu, John~C. Marioni, Tobias Marschall, Felix M{\"o}lder, Amir Niknejad, Lukasz Raczkowski, Marcel Reinders, Jeroen~de Ridder, Antoine-Emmanuel Saliba, Antonios Somarakis, Oliver Stegle, Fabian~J. Theis, Huan Yang, Alex Zelikovsky, Alice~C. McHardy, Benjamin~J. Raphael, Sohrab~P. Shah, and Alexander Sch{\"o}nhuth.
\newblock Eleven grand challenges in single-cell data science.
\newblock \emph{Genome Biology}, 21\penalty0 (1):\penalty0 1--35, 2020.
\bibitem[Lee \& Verleysen(2008)Lee and Verleysen]{Lee08a}
John Lee and Michel Verleysen.
\newblock Quality assessment of nonlinear dimensionality reduction based on k-ary neighborhoods.
\newblock In Yvan Saeys, Huan Liu, Iñaki Inza, Louis Wehenkel, and Yves Van~de Pee (eds.), \emph{Proceedings of the Workshop on New Challenges for Feature Selection in Data Mining and Knowledge Discovery at ECML/PKDD 2008}, number~4 in Proceedings of Machine Learning Research, pp.\ 21--35. PMLR, 2008.
\bibitem[Lee \& Verleysen(2009)Lee and Verleysen]{Lee09a}
John~A. Lee and Michel Verleysen.
\newblock Quality assessment of dimensionality reduction: Rank-based criteria.
\newblock \emph{Neurocomputing}, 72\penalty0 (7):\penalty0 1431--1443, 2009.
\newblock \doi{10.1016/j.neucom.2008.12.017}.
\bibitem[Leinster(2013)]{leinster2013magnitude}
Tom Leinster.
\newblock The magnitude of metric spaces.
\newblock \emph{Documenta Mathematica}, 18:\penalty0 857--905, 2013.
\bibitem[Leinster \& Shulman(2021)Leinster and Shulman]{leinster2021magnitude}
Tom Leinster and Michael Shulman.
\newblock Magnitude homology of enriched categories and metric spaces.
\newblock \emph{Algebraic \& Geometric Topology}, 21\penalty0 (5):\penalty0 2175--2221, 2021.
\bibitem[McInnes et~al.(2018)McInnes, Healy, Saul, and Gro{\ss}berger]{mcinnes2018umap}
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Gro{\ss}berger.
\newblock Umap: Uniform manifold approximation and projection.
\newblock \emph{Journal of Open Source Software}, 3\penalty0 (29):\penalty0 861, 2018.
\bibitem[Meckes(2013)]{meckes2013positive}
Mark~W. Meckes.
\newblock Positive definite metric spaces.
\newblock \emph{Positivity}, 17\penalty0 (3):\penalty0 733--757, 2013.
\bibitem[Meckes(2015)]{meckes2015magnitude}
Mark~W. Meckes.
\newblock Magnitude, diversity, capacities, and dimensions of metric spaces.
\newblock \emph{Potential Analysis}, 42\penalty0 (2):\penalty0 549--572, 2015.
\bibitem[Minc(1988)]{Minc88}
Henryk Minc.
\newblock \emph{Nonnegative Matrices}.
\newblock Wiley, New York, NY, USA, 1988.
\bibitem[Moon et~al.(2019)Moon, van Dijk, Wang, Gigante, Burkhardt, Chen, Yim, Elzen, Hirn, Coifman, Ivanova, Wolf, and Krishnaswamy]{moon2019visualizing}
Kevin~R. Moon, David van Dijk, Zheng Wang, Scott Gigante, Daniel~B Burkhardt, William~S. Chen, Kristina Yim, Antonia van~den Elzen, Matthew~J. Hirn, Ronald~R. Coifman, Natalia~B. Ivanova, Guy Wolf, and Smita Krishnaswamy.
\newblock Visualizing structure and transitions in high-dimensional biological data.
\newblock \emph{Nature Biotechnology}, 37\penalty0 (12):\penalty0 1482--1492, 2019.
\bibitem[Nguyen \& Holmes(2019)Nguyen and Holmes]{nguyen2019ten}
Lan~Huong Nguyen and Susan Holmes.
\newblock Ten quick tips for effective dimensionality reduction.
\newblock \emph{PLoS Computational Biology}, 15\penalty0 (6):\penalty0 e1006907, 2019.
\bibitem[Rieck \& Leitte(2017)Rieck and Leitte]{Rieck17a}
Bastian Rieck and Heike Leitte.
\newblock Agreement analysis of quality measures for dimensionality reduction.
\newblock In Hamish Carr, Christoph Garth, and Tino Weinkauf (eds.), \emph{Topological Methods in Data Analysis and Visualization IV}, pp.\ 103--117. Springer, Cham, Switzerland, 2017.
\newblock \doi{10.1007/978-3-319-44684-4_6}.
\bibitem[Satija et~al.(2015)Satija, Farrell, Gennert, Schier, and Regev]{satija2015spatial}
Rahul Satija, Jeffrey~A. Farrell, David Gennert, Alexander~F. Schier, and Aviv Regev.
\newblock Spatial reconstruction of single-cell gene expression data.
\newblock \emph{Nature Biotechnology}, 33\penalty0 (5):\penalty0 495--502, 2015.
\bibitem[Solow \& Polasky(1994)Solow and Polasky]{solow1994measuring}
Andrew~R Solow and Stephen Polasky.
\newblock Measuring biological diversity.
\newblock \emph{Environmental and Ecological Statistics}, 1:\penalty0 95--103, 1994.
\bibitem[Székely et~al.(2007)Székely, Rizzo, and Bakirov]{Szekely07a}
Gábor~J. Székely, Maria~L. Rizzo, and Nail~K. Bakirov.
\newblock Measuring and testing dependence by correlation of distances.
\newblock \emph{The Annals of Statistics}, 35\penalty0 (6):\penalty0 2769--2794, 2007.
\bibitem[van~der Maaten \& Hinton(2008)van~der Maaten and Hinton]{van2008visualizing}
Laurens van~der Maaten and Geoffrey Hinton.
\newblock Visualizing data using {t-SNE}.
\newblock \emph{Journal of Machine Learning Research}, 9\penalty0 (11), 2008.
\bibitem[Wolf et~al.(2018)Wolf, Angerer, and Theis]{wolf2018scanpy}
F~Alexander Wolf, Philipp Angerer, and Fabian~J. Theis.
\newblock {SCANPY}: large-scale single-cell gene expression data analysis.
\newblock \emph{Genome Biology}, 19:\penalty0 1--5, 2018.
\end{thebibliography}