No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Extracted source text · arXiv 2502.04959
%%%%%%%% ICML 2025 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%
\documentclass{article}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{booktabs} % for professional tables
\usepackage[table]{xcolor}
\usepackage{pifont}
\usepackage{multirow}
\usepackage{algorithm}
\usepackage{algorithmic}
% \usepackage{subfig} % Instead of subcaption
\usepackage[caption=false]{subfig}
% \usepackage{subcaption}
% For theorems and such
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2025} with \usepackage[nohyperref]{icml2025} above.
\usepackage{hyperref}
% Attempt to make hyperref and algorithmic work together better:
\newcommand{\theHalgorithm}{\arabic{algorithm}}
% Use the following line for the initial blind version submitted for review:
% \usepackage[preprint]{icml2025}
% If accepted, instead use the following line for the camera-ready submission:
\usepackage[accepted]{icml2025}
\definecolor{ta}{HTML}{cc7370}
\definecolor{isoc}{HTML}{2d9666}
\definecolor{isocts}{HTML}{d9a941}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% THEOREMS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{assumption}[theorem]{Assumption}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\newcommand{\minisection}[1]{\vspace{0.0in}\noindent{\bf #1}\hspace{0.15em}}
% Todonotes is useful during development; simply uncomment the next line
% and comment out the line below the next line to turn off comments
%\usepackage[disable,textsize=tiny]{todonotes}
% \usepackage[textsize=tiny]{todonotes}
% The \icmltitle you define below is probably too long as a header.
% Therefore, a short form for the running title is supplied here:
\icmltitlerunning{No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces}
\begin{document}
\twocolumn[
\icmltitle{No Task Left Behind:\\Isotropic Model Merging with Common and Task-Specific Subspaces}
% It is OKAY to include author information, even for blind
% submissions: the style file will automatically remove it for you
% unless you've provided the [accepted] option to the icml2025
% package.
% List of affiliations: The first argument should be a (short)
% identifier you will use later to specify author affiliations
% Academic affiliations should list Department, University, City, Region, Country
% Industry affiliations should list Company, City, Region, Country
% You can specify symbols, otherwise they are numbered in order.
% Ideally, you should not use this facility. Affiliations will be numbered
% in order of appearance and this is the preferred way.
\icmlsetsymbol{equal}{*}
\begin{icmlauthorlist}
\icmlauthor{Daniel Marczak}{pw,ideas}
\icmlauthor{Simone Magistri}{uof}
\icmlauthor{Sebastian Cygert}{ideas,pg}\\
\icmlauthor{Bartłomiej Twardowski}{ideas,cvc,uab}
\icmlauthor{Andrew D. Bagdanov}{uof}
\icmlauthor{Joost van de Weijer}{cvc,uab}
\end{icmlauthorlist}
\icmlaffiliation{pw}{Warsaw University of Technology, Poland}
\icmlaffiliation{ideas}{IDEAS NCBR, Warsaw, Poland}
\icmlaffiliation{uof}{Department of Information Engineering, University of Florence, Italy}
\icmlaffiliation{pg}{Gdańsk University of Technology, Poland}
\icmlaffiliation{cvc}{Computer Vision Center, Barcelona, Spain}
\icmlaffiliation{uab}{Department of Computer Science, Universitat Autonoma de Barcelona, Spain}
\icmlcorrespondingauthor{Daniel Marczak}{daniel.marczak.dokt@pw.edu.pl}
% You may provide any keywords that you
% find helpful for describing your paper; these are used to populate
% the "keywords" metadata in the PDF but will not be shown in the document
% \icmlkeywords{Machine Learning, ICML}
\vskip 0.3in
]
% this must go after the closing bracket ] following \twocolumn[ ...
% This command actually creates the footnote in the first column
% listing the affiliations and the copyright notice.
% The command takes one argument, which is text to display at the start of the footnote.
% The \icmlEqualContribution command is standard text for equal contribution.
% Remove it (just {}) if you do not need this facility.
\printAffiliationsAndNotice{} % leave blank if no need to mention equal contribution
% \printAffiliationsAndNotice{\icmlEqualContribution} % otherwise use the standard text.
\begin{abstract}
Model merging integrates the weights of multiple task-specific models into a single multi-task model. Despite recent interest in the problem, a significant performance gap between the combined and single-task models remains. In this paper, we investigate the key characteristics of task matrices -- weight update matrices applied to a pre-trained model -- that enable effective merging. We show that alignment between singular components of task-specific and merged matrices strongly correlates with performance improvement over the pre-trained model. Based on this, we propose an isotropic merging framework that flattens the singular value spectrum of task matrices, enhances alignment, and reduces the performance gap. Additionally, we incorporate both common and task-specific subspaces to further improve alignment and performance. Our proposed approach achieves state-of-the-art performance across multiple scenarios, including various sets of tasks and model scales. This work advances the understanding of model merging dynamics, offering an effective methodology to merge models without requiring additional training. Code is available at~\url{https://github.com/danielm1405/iso-merging}.
\end{abstract}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
\begin{figure}[t!]
\centering
\includegraphics[width=1.0\linewidth]{figs/iso-teaser.png}
\caption{Spectrum of singular values for a single layer weight update matrix obtained by merging using \textcolor{ta}{\textbf{\texttt{Task Arithmetic}}} (top) compared to our approaches: \textcolor{isoc}{\textbf{\texttt{Iso-C}}} (middle) and \textcolor{isocts}{\textbf{\texttt{Iso-CTS}}} (bottom). \textcolor{ta}{\textbf{\texttt{Task Arithmetic}}} sums the task-specific matrices, which result in a spectrum with a few dominant components. \textcolor{isoc}{\textbf{\texttt{Iso-C}}} instead replaces this spectrum with a uniform one, which results in significant performance improvement. \textcolor{isocts}{\textbf{\texttt{Iso-CTS}}} enhances the common subspace with task-specific subspaces and yields state-of-the-art model merging performance.}
\label{fig:teaser}
\end{figure}
Pre-trained models are the foundation of modern machine learning systems~\cite{carion2020endtoend, radford2021learning, DINO, zhai2023sigmoid}. In practice, they are typically fine-tuned for specialization on specific tasks~\cite{wortsman2022robust, ilharco2022patching}.
Recently, a growing body of research has focused on \textit{model merging}~\cite{li2023deep}, which combines multiple task-specific experts into a single multi-task model. Many methods have been proposed to improve the effectiveness of model merging by reducing sign conflicts~\cite{yadav2023tiesmerging}, by aligning gradients~\cite{DaheimMPGK24}, or through magnitude-based selection~\cite{MarczakTTC24}.
However, a significant performance gap between the combined and single-task models remains.
A key insight from \citet{ilharco2023task} is that \textit{task vectors}, defined as the offset between the \textit{flattened} fine-tuned weights and the pre-trained checkpoint, from different tasks are typically close to orthogonal. This orthogonality has been seen as a fundamental property enabling effective merging with reduced interference and has inspired works that enforce the orthogonality by modifying the fine-tuning procedure~\cite{PoYAW24}. Most recently, \citet{stoica2024knots} and \citet{tsv} have shown that accounting for the structure of the weight update matrix, dubbed \textit{task matrix}, is a more effective strategy for improving the performance of model merging. In this paper, we investigate precisely what the characteristics of task matrices are that favor effective model merging. Different from previous works, we propose to analyze the alignment between task-specific and merged subspaces.
Specifically, to capture the similarity between task matrices, we propose to investigate the \textit{Subspace Alignment Ratio}. Through the lens of Singular Value Decomposition, our metric quantifies the similarity between subspaces spanned by the top singular vectors of task matrices. When applied to compare matrices of the merged model to the task-specific ones, this metric strongly correlates with the performance of the merged model on a given task. This allows us to identify the directions amplified by multiple tasks as well as the underrepresented directions that lead to poor performance on corresponding tasks.
Our goal is to design a model merging technique that balances directions in the weight space across different tasks. We achieve this by flattening the singular values spectrum of the merged matrix, making it more uniform. Enforcing a uniform (isotropic) spectrum significantly improves the alignment and performance of the merged model. This simple yet effective adjustment, which requires no changes to the fine-tuning procedure, leads to substantial gains in merging performance (see method \texttt{Iso-C} in \cref{fig:teaser}).
However, tasks with dominant directions of smaller intensity compared to the majority of tasks and whose directions are orthogonal to the common directions may still remain underrepresented, especially when the number of tasks increases. To address this, we enhance isotropic model merging by introducing task-specific subspaces that retain unique task features while preserving shared knowledge. Our approach begins with the top singular values of the common subspace and iteratively replaces the least significant singular vectors with task-specific directions. This strategy allows us to increase the scalability of our merging approach to more tasks (see method \texttt{Iso-CTS} in \cref{fig:teaser}).
The main contributions of this paper are:
\begin{itemize}
\item We show that the alignment between the subspace spanned by the principal directions of the task-specific matrices and that of the merged matrix positively correlates with the performance of the merged model.-
\item We demonstrate that applying an isotropic scaling to singular directions of merged task matrices improves the alignment between merged and task-specific matrices. This results in a simple yet highly effective technique for model merging that we call \texttt{Iso-C}, which outperforms most baselines.
\item We further enhance our approach by incorporating task-specific directions into the merged matrix resulting in \texttt{Iso-CTS}, a merging method that achieves state-of-the-art results, in particular for a large number of tasks.
\end{itemize}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}
\minisection{Model merging.}
Pre-trained models serve as a foundation for expert models specialized in specific downstream tasks~\cite{radford2021learning}. Recently, model merging has emerged as a promising technique to combine multiple expert models into a single multi-task model.
One of the pioneering works in the field, Task Arithmetic (TA)~\cite{ilharco2023task}, proposed to compute a \textit{task vector} as a difference between the expert and the pre-trained model and to then aggregate task vectors via scaled addition to create an expert in multiple tasks. The significant performance gap between individual experts and the combined model sparked an abundance of works with the aim of reducing interference when merging models. TIES~\cite{yadav2023tiesmerging} proposed a novel way to reduce sign conflicts between the parameters of expert models, Model Breadcrumbs~\cite{davari2023model} removed outliers from the task vectors, and Consensus Merging~\cite{wang2024localizing} removed catastrophic and selfish weights. These methods focused on per-parameter techniques to mitigate the interference, treating each parameter independently.
\minisection{Singular Value Decomposition of model weights.}
While SVD of weight matrices has been primarily used for model compression~\cite{NIPS2014_2afe4567,KimPYCYS15}, recently its effectiveness was also identified for fine-tuning of large models. LoRA~\cite{hu2021lora} uses SVD to identify the similarities of weight updates between low-rank and full-rank fine-tuning.
MiLORA~\cite{milora} identifies that the bottom singular components correspond to noisy or long-tail information, while the top singular vectors contain important knowledge. Therefore, they propose a fine-tuning approach that updates only the minor singular components of the weight matrix while keeping the top singular components frozen.
SVFT~\cite{svft} computes outer products of its singular vectors and, during fine-tuning updates, only sparse coefficients of these combinations.
\minisection{SVD for model merging.}
The structure imposed by SVD was used for model merging in KnOTS~\cite{stoica2024knots}, which proposes to concatenate the task-specific low-rank adaptation matrices (LoRA) and average the right-singular vectors before SVD reconstruction to obtain the merged weights.
The most similar work to us is the parallel work Task Singular Vectors (TSV)~\cite{tsv}, which measures task interference based on the interaction of singular vectors from different tasks and uses it to increase merging effectiveness. We share the motivation to improve model merging through SVD decomposition. However, while they focus on the orthogonalization of task-specific subspaces to reduce interference, we show that making singular values uniform in a common subspace is a surprisingly powerful method. Further, we show how to combine shared and task-specific subspaces for improved performance.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Background and Motivation}
In this section, we first describe the general framework of model merging and provide the notation used throughout the rest of the paper. We then motivate our approach via an analysis of the correlation between task similarity and performance improvement of the merged model.
\subsection{Model Merging}
Model merging integrates multiple deep neural network models, each individually trained (i.e. fine-tuned) on distinct tasks starting from the same pre-trained model, into a single merged model. Let $\theta_0$ denote the weights of the pre-trained network, and $\theta_t$ denote the fine-tuned weights for task $t$, with $t=1,\ldots, T$, where $T$ is the total number of tasks. We will use the notation $\theta_t^{(l)}$ to identify the weights of layer $l$ for task $t$ and $L$ to denote the total number of layers in a network. The objective of model merging is to find a merging function $f$, such that the model:
\begin{equation}
\theta_{\text{M}}^{(\ell)} = f(\theta_0^{(\ell)},\{\theta_t^{(\ell)}\}_{t=1}^T), \quad \forall \ell=1,\ldots, L
\end{equation}
is able to perform all tasks on which the individual models $\theta_t$ are trained.
Building upon Task Arithmetic (TA), we define the layer-wise \textit{task matrix} $\Delta^{(\ell)}_t$ as the difference between the weights of the model $\theta_t$ and the pre-trained model $\theta_0$ for layer $\ell$:
\begin{equation}
\Delta^{(\ell)}_t = \theta^{(\ell)}_{t} - \theta^{(\ell)}_{0}.
\end{equation}
In the rest of the paper, the $\ell$ superscript is omitted when not relevant to the discussion, and all definitions refer to an arbitrary layer. The authors of Task Arithmetic propose to solve the problem of model merging by defining a merging function that sums all task matrices to the pre-trained model weights:
\begin{equation}
\label{eq:ta}
\theta^{(\ell)}_{\text{TA}} = \theta^{(\ell)}_0 + \alpha \Delta^{(\ell)}_{\text{TA}},
\end{equation}
where $\alpha$ is a scaling factor determined on a held-out validation dataset and $\Delta^{(\ell)}_{\text{TA}} = \sum_{t=1}^T \Delta^{(\ell)}_t$. The advantage of this merging strategy is that it allows for the reuse and transfer of knowledge from many fine-tuned models to the pre-trained model without requiring additional training or access to the original training data~\cite{ilharco2023task}.
\subsection{Cosine Similarity and Performance Improvement are Uncorrelated}
\begin{figure}[t!]
\centering
\subfloat[Cosine similarity between pairs of task vectors.]{\includegraphics[width=0.49\linewidth]{figs/task-cossim.pdf}\label{fig:task-cossim}}
\hfill
\subfloat[NAI vs cosine similarity between task and merged vectors.]{\includegraphics[width=0.49\linewidth]{figs/ta-nai-vs-cossim.pdf}\label{fig:nai_vs_cossim_ta}}
\caption{(a) Tasks vectors are typically close to \textit{orthogonal} to each other. (b) Models with very different normalized accuracy improvements (NAI) exhibit very close cosine similarities, and the correlation between cosine similarity and NAI is low.}
\label{fig:cos_sim}
\end{figure}
Starting from the definition of Task Arithmetic (TA) in Eq.~\eqref{eq:ta}, we aim to explore the possible reasons for the improvement achieved by TA merging over the pre-trained (or zero-shot) model across multiple tasks. To empirically quantify performance gain, we propose the \textit{Normalized Accuracy Improvement (NAI)} metric, defined as:
\begin{equation}
\label{eq:nai}
\text{NAI}( \theta_{M}, \theta_t; \theta_0) = \frac{\text{Acc}(\theta_{M}) - \text{Acc}(\theta_0)}{\text{Acc}(\theta_{t}) - \text{Acc}(\theta_0)},
\end{equation}
which quantifies the improvement of the merged model $\theta_{M}$ relative to that achieved by the task-specific model $\theta_t$, both measured with respect to the zero-shot baseline $\theta_0$.\footnote{NAI differs from Normalized Accuracy~\cite{ortizjimenez2023tangent} which does not account for zero-shot performance.}
\citet{ilharco2023task} hypothesize that the effectiveness of task vector summation arises from the cosine similarity between the vectorized representations of the task matrices being close to zero, i.e. $\langle \text{vec}(\Delta_i), \text{vec}(\Delta_j) \rangle \approx 0$ for $i \neq j$, which minimizes inter-task interference. Based on this intuition, we measured the correlation between the cosine similarity of each task vector with the merged model vector and the normalized accuracy improvement $\text{NAI}(\theta_{\text{TA}}, \theta_t; \theta_0)$. However, we observe no clear correlation (see \cref{fig:cos_sim}). This suggests that the underlying reason for the performance improvement of the Task Arithmetic update over the zero-shot model likely originates from other factors, which we show below can be unveiled via spectral analysis of the Task Arithmetic and task-specific matrices.
\subsection{Performance Correlates with Subspace Alignment} \label{sec:motivation-alignment}
\begin{figure*}[t!]
\centering
\subfloat[Normalized Accuracy Improvement (NAI) vs. Average Subspace Alignment Ratio ($\text{SAR}_\text{avg}$).]{\includegraphics[width=0.5\linewidth]{figs/ta-iso-nai-vs-ar.pdf}\label{fig:motivation_nai_vs_ar}}
\hfill
\subfloat[Average Subspace Alignment Ratios ($\text{SAR}_\text{avg}$) between pairs of task vectors.]{\includegraphics[width=0.43\linewidth]{figs/inter-task-alignment-8-tasks.pdf}\label{fig:task-wise-alignment}}
\caption{
(a) NAI strongly correlates with $\text{SAR}_\text{avg}$ (Pearson correlation coefficient $\rho_{TA} = 0.88$).
(b) Note the groups of highly aligned tasks such as \{MNIST, SVHN, GTSRB\} and \{EuroSAT, RESISC45\}. By comparing (b) and (a), the mutually aligned datasets exhibit higher alignment with the merged model and consequently achieve good performance. On the other hand, tasks with low mutual alignment, such as DTD, Cars, and SUN397, are less aligned with the merged model and achieve poor performance.
}
\label{fig:nai_and_alignment}
\end{figure*}
We argue that the improvement in Task Arithmetic performance derives from the relationship between the top singular vectors of $\Delta_{\text{TA}}$ and those of each $\Delta_t$. Specifically, we hypothesize that the subspace of $\Delta_{\text{TA}}$ approximates the union of the subspaces of each $\Delta_t$, and that the overlap of this overall subspace with each task matrix correlates with the performance improvement of the merged model.
In order to empirically quantify the overlap between subspaces, we propose the \textit{Subspace Alignment Ratio (SAR)} metric. Without loss of generality, we define SAR between a source task matrix $\Delta_{\text{src}}$ and a target task matrix $\Delta_{\text{trg}}$, with respect to a generic merged task matrix $\Delta_{\text{M}}$, as:
\begin{equation}
\label{eq:sar}
\text{SAR} (\Delta_{\text{src}}, \Delta_{\text{trg}}; k_{\text{M}}) = \frac{||\Pi_{k_{\text{M}}, \text{trg}}\Delta_{\text{src}}||_{F}}{||\Delta_{\text{src}} ||_{F}},
\end{equation}
where $\Pi_{k_{\text{M}},\text{trg}} = U_{k_{\text{M}},\text{trg}} U^\top_{k_{\text{M}},\text{trg}}$ is the projection matrix onto the subspace spanned by the top $k_{\text{M}}$ left-singular vectors of $\Delta_{\text{trg}}$. The columns of $U_{k_{\text{M}}, \text{trg}}$ are obtained from the SVD decomposition of $\Delta_{\text{trg}}$, and the number of singular vectors used ($k_{M}$) is determined from the merged task matrix $\Delta_{M}$ by minimizing the approximation error:
\begin{equation}
\label{eq:rank}
k_{M} = \text{min} \{ k : ||\Delta_{\text{M}} - \Pi_{k, \text{M}} \Delta_{\text{M}}||_{F} \leq \epsilon ||\Delta_{\text{M}} ||_{F}\},
\end{equation}
with $\epsilon=0.05$. SAR quantifies the alignment between the subspaces of two task matrices as a function of the number of dominant singular vectors of the merged matrix. To provide a single score measuring the overlap between two models, we denote with $\text{SAR}_\text{avg}$ the \textit{Average Subspace Alignment Ratio} across all layers.
In \cref{fig:motivation_nai_vs_ar} (left, represented by stars), we plot the Normalized Accuracy Improvement achieved by TA on each task, given by $\text{NAI}(\theta_{\text{TA}}, \theta_t; \theta_0)$, against the Average Subspace Alignment Ratio of each task matrix $\Delta_t$ with the merged task matrix $\Delta_{\text{TA}}$, i.e. $\text{SAR}_{\text{avg}}(\Delta_t, \Delta_\text{TA}; k_\text{TA})$. First, we note that the alignment between task and merged matrices are notably high (ranging from 0.75 to 0.87), but vary significantly across datasets. This suggests that task vectors are well represented in the subspace identified by the task-arithmetic matrix but with different degrees of alignment and consistency depending on dataset characteristics. Furthermore, we highlight a strong correlation ($\rho=0.88$) between the performance improvement on individual tasks achieved by $\theta_{\text{TA}}$ and the degree of alignment of $\Delta_t$ with $\Delta_{\text{TA}}$.
In~\cref{fig:task-wise-alignment}, we report the average alignment ratios between pairs of tasks, i.e. $\text{SAR}_\text{avg}(\Delta_i,\Delta_j; k_{\text{TA}})$. Some groups of tasks exhibit higher alignment which is due to their semantic similarity, e.g. MNIST, SVHN, and GTSRB are digit recognition datasets, while EuroSAT and RESISC45 are satellite image datasets. On the other hand, datasets such as Cars, DTD or SUN397 are less aligned to other tasks. Most importantly, tasks belonging to highly aligned groups are also highly aligned with the TA model and achieve the highest accuracy improvements (see \cref{fig:motivation_nai_vs_ar}). The tasks that are not aligned are underrepresented in the dominant subspace of $\Delta_{\text{TA}}$, and the performance on them is low.
Based on the observed correlation between performance and alignment ratio, we hypothesize that a merging method that aims to achieve high alignment will also achieve strong performance. Therefore, in the next section, we propose an approach called \textit{Isotropic Merging} that improves alignment and, most importantly, the performance of the merged models.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Isotropic Merging in Common and Task-specific Subspaces}
In this section, we propose a novel model merging method we call Isotropic Merging in Common and Task-Specific Subspaces (\texttt{Iso-CTS}). First, we introduce Isotropic Merging in Common Subspace (\texttt{Iso-C}), which is able to enhance the normalized accuracy improvement and the alignment of each task matrix using common directions identified by Task Arithmetic. Then, we show how to further enhance the performance of merged models by introducing task-specific directions to improve merging performance on sets of many diverse tasks.
\subsection{Isotropic Merging in Common Subspace}
\label{sec:isoc}
In~\cref{sec:motivation-alignment}, we demonstrated the high alignment of each task matrix with the matrix obtained by Task Arithmetic. This alignment indicates that the span of dominant singular vectors of the merged matrix effectively covers the subspace of each task and provides a good approximation of the \textit{common subspace}. However, significant variability in the average alignment ratio across the dataset leads to a lower accuracy improvement for less correlated tasks compared to more correlated ones. This variability stems from the skewness of the task arithmetic spectrum (\cref{fig:teaser} and \ref{fig:combined-spectra}), which is concentrated in the first few singular values (which we call \textit{top} or \textit{dominant}), favoring more correlated tasks. Our proposed methodology, which we call \textit{\underline{Iso}tropic Merging in \underline{C}ommon Subspace} (\texttt{Iso-C}), aims to equalize the spectrum of the task arithmetic matrix in order to enhance the \textit{average subspace alignment ratio} and ensure a more balanced representation across tasks in the merged model.
Consider the sum of task matrices $\Delta_{\text{TA}} = \sum_t \Delta_t$, where $\Delta_t \in \mathbb{R}^{m \times n}$. Via Singular Value Decomposition (SVD) on $\Delta_{\text{TA}}$ we obtain $\Delta_{\text{TA}} = U \Sigma V^{\top}$, where $U \in \mathbb{R}^{m \times r}$ and $V \in \mathbb{R}^{n \times r}$ represent, respectively, the left and right singular vectors of $\Delta_{\text{TA}}$, and $\Sigma \in \mathbb{R}^{r \times r}$ is the diagonal matrix containing the singular values. We denote the vector of singular values by $\sigma = \text{diag}(\Sigma) \in \mathbb{R}^{r}$.
To reduce the skewness towards the dominant singular vectors of $\Delta_{\text{TA}}$,
we propose scaling all directions of the transformation applied by the right-singular vectors $V$ to a fixed value rather than using their corresponding singular values. This ensures that the final transformation is \textit{isotropic}, with the scaling factor set to the average singular value:
\begin{equation}
\label{eq:iso-c}
\overline{\sigma} = \frac{1}{r}\sum_{i=1}^{r} \sigma_i,
\end{equation}
and merged matrix is computed using the reconstruction:
\begin{equation}
\label{eq:final-isoc}
\Delta_{\texttt{Iso-C}} = \overline{\sigma} U V^\top.
\end{equation}
We apply this operation to all network layers, and the final merged model is defined as:
\begin{equation}
\label{eq:isoc-alpha}
\theta_{\texttt{Iso-C}}^{(\ell)} = \theta_0^{(\ell)} + \alpha \Delta^{(\ell)}_{\texttt{Iso-C}}, \,\,\, \forall \ell=1,\ldots, L
\end{equation}
where $\alpha$ is chosen on a held-out validation set.
Applying isotropic merging results in an enhancement of the normalized accuracy improvement and the alignment of each task subspace with the top singular vectors of the task arithmetic matrix (see Figure~\ref{fig:motivation_nai_vs_ar}).
In \cref{alg:method_common}, we give the \texttt{Iso-C} model merging algorithm for a single layer.
\begin{algorithm}[t]
\caption{\texttt{Iso-C}: Isotropic Merging in Common Subspace}
\label{alg:method_common}
\begin{algorithmic}[1]
\REQUIRE Task matrices $\Delta_1, \dots, \Delta_T \,\, \text{with}\,\, \Delta_t \in \mathbb{R}^{m \times n}$
%\ENSURE Merged matrix $\Delta_{\text{M}}$
\STATE Sum task matrices: $\Delta_{\text{TA}} = \sum_{t=1}^{T} \Delta_{t}$
\STATE Compute the SVD of $\Delta_{\text{TA}}$: $\Delta_{\text{TA}} = U \Sigma V^\top$, with
$U \in \mathbb{R}^{m\times r}, \Sigma \in \mathbb{R}^{r \times r}, V \in \mathbb{R}^{n \times r}, \sigma = \text{diag}(\Sigma)\!\!\in \mathbb{R}^r$
%\STATE Calculate isotropic scaling factor $\overline{\sigma} = \frac{1}{r} \sum_{j=1}^{r} \sigma_{j} $
\STATE Calculate isotropic factor: \! $\overline{\sigma}=\frac{1}{r} \sum_{i=1}^{r} \sigma_i$ \hfill (Eq.\ref{eq:iso-c})
\STATE Reconstruct the matrix: $\Delta_{\texttt{Iso-C}} = \overline{\sigma} U V^\top$ \hfill (Eq.\ref{eq:final-isoc})
\STATE \textbf{return} $\Delta_{\texttt{Iso-C}}$
\end{algorithmic}
\end{algorithm}
\subsection{Isotropic Merging in Common and Task-Specific Subspaces}
\label{sec:isocts}
\begin{algorithm}[t]
\caption{\texttt{Iso-CTS}: Isotropic Merging in Common and Task-Specific Subspaces (\textcolor{isoc}{green} -- shared with \texttt{Iso-C})}
\label{alg:method_common_and_task_specific}
\begin{algorithmic}[1]
\REQUIRE Task matrices $\Delta_1, \dots, \Delta_T \,\, \text{with}\,\, \Delta_t \in \mathbb{R}^{m \times n}$
\textcolor{isoc}{\STATE Sum task matrices $\Delta_{\text{TA}} = \sum_{t=1}^{T} \Delta_{t}$}
\textcolor{isoc}{\STATE Compute the SVD of $\Delta_{\text{TA}}$:
$\Delta_{\text{TA}} = U \Sigma V^\top$, with}
\textcolor{isoc}{$U \in \mathbb{R}^{m\times r}, \Sigma \in \mathbb{R}^{r \times r}, V \in \mathbb{R}^{n \times r},\sigma = \text{diag}(\Sigma)\!\!\in \mathbb{R}^r$}
\STATE \textbf{Retain} top-$k$ singular vectors and values from \textbf{common subspace}:
$U^{1:k} = [u_1| \ldots | u_k] \quad V^{1:k} = [v_1| \ldots | v_k]$
\quad \quad \quad \quad \quad $\sigma^{\text{cm}}=\text{diag}(\Sigma)^{1:k}$
\STATE \textbf{Accumulate} task-specific directions via projection:
\FOR{$t = 1$ to $T$}
\STATE $\overline{\Delta}_t = \Delta_t - U^{1:k} (U^{1:k})^{\top} \Delta_t$ \hfill (Eq.\ref{eq:residual})
\STATE Compute SVD: $ \overline{\Delta}_{t} = \overline{U}_{t} \overline{\Sigma}_{t} \overline{V}^\top_{t}$
\STATE Retain first $s=\frac{r-k}{T}$ components of $\overline{U}_{t}$ and $\overline{V}_{t}$:
$\overline{U}_t^{1:s} = [\overline{u}_{t,1}\vert \ldots \vert \overline{u}_{t, s}] \quad \overline{V}_t^{1:s} = [\overline{v}_{t,1}\vert \ldots \vert \overline{v}_{t, s}]$
\quad \quad \quad \quad \quad $\sigma_t^{\text{ts}}=\text{diag}(\overline{\Sigma}_t)^{1:s}$
\ENDFOR
\STATE \textbf{Combine} common and task-specific spaces: \vspace{-0.9em}\begin{align*}
U_* &= [U^{1:k} \vert \overline{U}_1^{1:s}\vert \ldots \vert \overline{U}_T^{1:s}] \in \mathbb{R}^{m \times r} \\[-3pt] V_* &= [V^{1:k} \vert \overline{V}_1^{1:s}\vert\ldots\vert \overline{V}_T^{1:s}] \in \mathbb{R}^{n \times r}
\end{align*}
\vspace{-2.0em}
\STATE \textbf{Orthogonalize} $U_{*}$ and $V_{*}$ via whitening \hfill (Eq.\ref{eq:withening})
\textcolor{isoc}{\STATE Calculate isotropic factor
$\overline{\sigma}$:}
\vspace{-0.8em}
\begin{equation*}
\overline{\sigma} = \frac{1}{r}\Big(\sum_{i=1}^k \sigma_i^\text{cm} + \sum_{t=1}^{T} \sum_{i=1}^{s} \sigma_{t,i}^{\text{ts}}\Big) \quad \quad \quad \quad \quad \quad \text{(Eq.\ref{eq:iso-cts})}
\end{equation*}
\vspace{-1.2em}
\textcolor{isoc}{\STATE \textbf{Reconstruct} the matrix
$\Delta_{\texttt{Iso-CTS}} = \overline{\sigma} U_{*} V_{*}^\top$} \hfill (Eq.\ref{eq:final-cts})
\STATE \textbf{return} $\Delta_{\texttt{Iso-CTS}}$
\end{algorithmic}
\end{algorithm}
The effectiveness of \texttt{Iso-C} depends on how well the common subspace -- identified by the dominant singular vectors of $\Delta_{\text{TA}}$ -- approximates the subspaces of the individual tasks. The approximation error arises from how these tasks interact when summed. The top singular directions of $\Delta_{\text{TA}}$ capture only the dominant common variations, while singular vectors associated with near-zero singular values provide negligible information. At the same time, tasks with dominant directions of smaller intensity compared to the majority of tasks and whose directions are orthogonal to the common directions remain underrepresented. This limitation becomes more pronounced as the number of tasks increases and the tasks become more diverse.
To address this limitation, we propose enhancing the range of directions used by \texttt{Iso-C} to ensure that the task-specific directions, which are orthogonal to those of the common subspace, are incorporated into the singular basis of the final merged matrix. We call this methodology as \textit{\underline{Iso}tropic Merging in \underline{C}ommon and \underline{T}ask-\underline{S}pecific Subspaces} (\texttt{Iso-CTS}).
Our approach starts with the top singular values of the common subspace and iteratively replaces the singular vectors associated with the lowest singular values with task-specific directions. The final goal is to find two orthonormal matrices $U_{*} \in \mathbb{R}^{m \times r}$ and $V_{*} \in \mathbb{R}^{n \times r}$ whose columns contain both common and task-specific directions. Afterward, the final matrix is reconstructed, and isotropic merging is applied. In the following, we provide a detailed explanation of our proposed algorithm.
\minisection{Retaining components from the common subspace.} We retain the top-$k$ singular vectors associated with the subspace identified by $\Delta_{\text{TA}}$:
\begin{equation*}
U^{1:k} = [u_1| \ldots | u_k] \quad V^{1:k} = [v_1| \ldots | v_k],
\end{equation*}
where $U^{1:k}$, $V^{1:k}$ are the top-$k$ left- and right-singular vectors from the SVD of $\Delta_{\text{TA}}$. We analyze the impact of selecting $k$ in~\cref{sec:analysis}.
\minisection{Accumulating task-specific directions.} We project each task-specific matrix $\Delta_t$ onto the subspace orthogonal to the common subspace, i.e. the space spanned by top left-singular directions of the common subspace $U^{1:k}$:
\begin{equation}
\label{eq:residual}
\overline{\Delta}_t = \Delta_t - U^{1:k} (U^{1:k})^T \Delta_t.
\end{equation}
We then compute the SVD of $\overline{\Delta}_t = \overline{U}_t\, \overline{\Sigma}_t \overline{V}_t$ and retain the top $s=\frac{r-k}{T}$ directions for each task $t$:
\begin{equation*}
\overline{U}_t^{1:s}\!\!=\![\overline{u}_{t,1}\vert \ldots \vert \overline{u}_{t,s}] \,\,\, \overline{V}_t^{1:s}\!\!=\![\overline{v}_{t,1}\vert \ldots \vert \overline{v}_{t,s}], \forall t=1,\!\ldots\!, T.
\end{equation*}
The orthogonal projection Eq.~\eqref{eq:residual} guarantees that both the left- and right-singular vectors of $\overline{\Delta}_t$, representing task-specific directions, are orthogonal to the subspace spanned by the common directions (given by $U^{1:k}$).
\minisection{Combining common and task-specific matrices.} After identifying the $k$ principal vectors for the common subspace and $s=\frac{r-k}{T}$ principal vectors for each task, we now combine the common and task-specific directions by concatenating them: $U_* = [U^{1:k}\vert\overline{U}_1^{1:s}\vert\ldots \vert \overline{U}_T^{1:s}] \in \mathbb{R}^{m \times r} $ and $V_* = [V^{1:k} \vert \overline{V}_1^{1:s}\vert\ldots\vert\overline{V}_T^{1:s}] \in \mathbb{R}^{n \times r}$.
\minisection{Orthogonalization.} There is no guarantee that the left- and right-singular task-specific vectors are orthogonal to each other, as we are only projecting each task matrix onto the common subspace. To reconstruct the final merged matrix, we must orthogonalize $U_*$ and $V_*$. Following~\citet{tsv}, we compute the SVD of $U_*= P_{U_*} \Sigma_{U_*} Q_{U_*}^\top $ and $V_*=P_{V_*} \Sigma_{V_*} Q_{V_*}^\top$, and whiten~\citep{schonemann1966}:
\begin{equation}
\label{eq:withening}
U_* = P_{U_*} Q_{U_*}^\top \quad V_* = P_{V_*} Q_{V_*}^\top.
\end{equation}
\minisection{Isotropic scaling and reconstruction.} Finally, we reconstruct the final merged matrix and apply isotropic merging:
\begin{equation}
\label{eq:final-cts}
\Delta_{\texttt{Iso-CTS}}=\overline{\sigma} U_{*} V_{*}^\top,
\end{equation}
where $\overline{\sigma}$ is obtained by averaging the singular values associated with the vectors selected for both common and task-specific subspaces. Specifically, defining $\sigma^{\text{cm}}= \text{diag}(\Sigma)^{1:k} \in \mathbb{R}^{k}$, the vector of singular values associated with the common subspace identified by $U_{1:k}$ and $V_{1:k}$, and $\sigma^{\text{ts}}_{t} = \text{diag}(\overline{\Sigma}_t)^{1:s}\in \mathbb{R}^{s}$, with $s=\frac{r-k}{T}$, the vector of singular values associated with each task-specific subspace $\overline{U}_t^{1:s}$ and $\overline{V}_t^{1:s}$, we define the scaling factor as:
\begin{equation}
\label{eq:iso-cts}
\overline{\sigma} = \frac{1}{r}\Big(\sum_{i=1}^k \sigma_i^\text{cm} + \sum_{t=1}^{T} \sum_{i=1}^{s} \sigma_{t,i}^{\text{ts}}\Big).
\end{equation}
Finally, similar to \texttt{ISO-C}, the merged model is defined as:
\begin{equation}
\label{eq:isocts-alpha}
\theta_{\texttt{Iso-{CTS}}}^{(\ell)} = \theta_0^{(\ell)} + \alpha \Delta^{(\ell)}_{\texttt{Iso-{CTS}}}, \quad \forall \ell=1,\ldots,L
\end{equation}
where $\alpha$ is chosen on a held-out validation set.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experimental Results}
\label{sec:exp}
\begin{table*}[t!]
\caption{\texttt{Iso-CTS} achieves state-of-the-art performance for all backbones on all evaluated scenarios. We present average absolute accuracy and average normalized accuracy (in subscript) in $\%$. The best method in \textbf{bold} and the second-best \underline{underlined}.}
\label{tab:multitask_acc}
\vspace{0.03in}
\resizebox{1\textwidth}{!}{
\begin{tabular}{cccc|ccc|ccc}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{3}{c}{ViT-B/32} & \multicolumn{3}{c}{ViT-B/16} & \multicolumn{3}{c}{ViT-L/14} \\
\cmidrule{2-10}
& 8 tasks & 14 tasks & 20 tasks & 8 tasks & 14 tasks & 20 tasks & 8 tasks & 14 tasks & 20 tasks \\
\midrule
\rowcolor{gray!15}
\multicolumn{1}{c|}{Zero-shot} & $48.3$ & $57.2$ & $56.1$ & $55.3$ & $61.3$ & $59.7$ & $64.7$ & $68.2$ & $65.2$ \\
\rowcolor{gray!15}
\multicolumn{1}{c|}{Fine-tuned} & $92.8$ & $90.9$ & $91.3$ & $94.6$ & $92.8$ & $93.2$ & $95.8$ & $94.3$ & $94.7$ \\
\midrule
\multicolumn{1}{c|}{Weight Averaging} & $66.3_{(72.1)}$ & $64.3_{(71.1)}$ & $61.0_{(67.5)}$ & $72.2_{(76.6)}$ & $69.5_{(74.8)}$ & $65.3_{(70.4)}$ & $79.6_{(83.2)}$ & $76.7_{(81.1)}$ & $71.6_{(75.6)}$ \\
\multicolumn{1}{c|}{Task Arithmetic} & $70.8_{(76.5)}$ & $65.3_{(72.1)}$ & $60.5_{(66.8)}$ & $75.4_{(79.6)}$ & $70.5_{(75.9)}$ & $65.8_{(70.8)}$ & $84.9_{(88.7)}$ & $79.4_{(84.0)}$ & $74.0_{(78.1)}$ \\
\multicolumn{1}{c|}{TIES} & $75.1_{(81.0)}$ & $68.0_{(74.8)}$ & $63.4_{(69.9)}$ & $79.7_{(84.3)}$ & $73.2_{(78.7)}$ & $68.2_{(73.3)}$ & $86.9_{(90.7)}$ & $79.5_{(84.1)}$ & $75.7_{(79.8)}$ \\
\multicolumn{1}{c|}{Consensus TA} & $75.0_{(80.8)}$ & $70.4_{(77.4)}$ & $65.4_{(72.0)}$ & $79.4_{(83.9)}$ & $74.4_{(79.9)}$ & $69.8_{(74.9)}$ & $86.3_{(90.1)}$ & $82.2_{(86.9)}$ & $79.0_{(83.2)}$ \\
\multicolumn{1}{c|}{TSV-M} & $85.9_{(92.3)}$ & $80.1_{(87.9)}$ & $\underline{77.1_{(84.3)}}$ & $89.0_{(93.9)}$ & $84.6_{(91.0)}$ & $\underline{80.6_{(86.5)}}$ & $93.0_{(97.0)}$ & $89.2_{(94.4)}$ & $\underline{87.7_{(92.5)}}$ \\
\rowcolor{isoc!20}
\multicolumn{1}{c|}{\textbf{\texttt{Iso-C} (Ours)}} & $\mathbf{86.3_{(92.9)}}$ & $\underline{80.3_{(88.1)}}$ & $75.5_{(82.5)}$ & $\underline{90.6_{(95.6)}}$ & $\underline{84.8_{(91.1)}}$ & $79.6_{(85.4)}$ & $\underline{94.2_{(98.3)}}$ & $\underline{89.3_{(94.5)}}$ & $87.6_{(92.2)}$ \\
\rowcolor{isocts!25}
\multicolumn{1}{c|}{\textbf{\texttt{Iso-CTS} (Ours)}} & $\underline{86.2_{(92.8)}}$ & $\mathbf{81.7_{(89.7)}}$ & $\mathbf{78.1_{(85.5)}}$ & $\mathbf{91.1_{(96.1)}}$ & $\mathbf{86.4_{(92.8)}}$ & $\mathbf{82.4_{(88.4)}}$ & $\mathbf{94.7_{(98.8)}}$ & $\mathbf{91.0_{(96.3)}}$ & $\mathbf{90.1_{(94.9)}}$ \\
\bottomrule
\end{tabular}
}
\end{table*}
\begin{figure*}[t!]
\vspace{-0.15in}
\centering
\subfloat[Spectra of singular values for different values of interpolation coefficient ($\beta$).]{\includegraphics[width=0.325\linewidth]{figs/ta-iso-interpolation-spectrum.pdf}\label{fig:ta-iso-interpolation-spectrum}}
\hfill
\subfloat[Average Subspace Alignment Ratio ($\text{SAR}_{\text{avg}}$) vs. interpolation coefficient ($\beta$).]{\includegraphics[width=0.32\linewidth]{figs/ta-iso-interpolation-ar-vs-beta.pdf}\label{fig:ta-iso-interpolation-ar-vs-beta}}
\hfill
\subfloat[Normalized Accuracy Improvement (NAI) vs. interpolation coefficient ($\beta$).]{\includegraphics[width=0.32\linewidth]{figs/ta-iso-interpolation-nai-vs-beta.pdf}\label{fig:ta-iso-interpolation-nai-vs-beta}}
\vspace{-0.05in}
\caption{
(a) Interpolating from $\Delta_{\text{TA}}$ ($\beta = 0$) towards $\Delta_{\texttt{Iso-C}}$ ($\beta = 1$) makes the spectrum of singular values of $\Delta_M$ more uniform and increases the number of preserved components $k_M$ (Eq.~\eqref{eq:rank}) denoted by dashed lines. (b) This results in an increased alignment between each task-specific model and merged model measured by $\text{SAR}_{\text{avg}}$. (c) As alignment increases, the performance also improves as predicted based on the strong correlation between these two properties investigated in~\cref{sec:motivation-alignment}.
}
\vspace{-0.1in}
\label{fig:ta-iso-interpolation}
\end{figure*}
\subsection{Experimental setup}
\label{sec:exp_setup}
We evaluate our approaches over sets of 8, 14, and 20 datasets, following~\citet{wang2024localizing}. We provide the details of the datasets in \cref{apdx:datasets}.
We consider three variants of CLIP~\cite{radford2021learning} with ViT-B/32, ViT-B/16 and ViT-L/14 as visual encoders~\cite{dosovitskiy2021an}. We use the checkpoints fine-tuned on the tasks above, provided in~\cite{wang2024localizing}. If not stated otherwise, we present the results using the ViT-B/16 visual encoder.
We compare our approaches with the following model merging methods: weight averaging~\cite{wortsman2022model}, Task Arithmetic~\cite{ilharco2023task}, TIES-Merging~\cite{yadav2023tiesmerging}, Consensus TA~\cite{wang2024localizing} and TSV-M~\cite{tsv}. We include the results of the zero-shot model and fine-tuned models serving as lower- and upper-bound, respectively. We compare the results based on absolute and normalized accuracy following standard practice~\cite{wang2024localizing, tsv}.
\subsection{Multi-task model merging}
\cref{tab:multitask_acc} presents our main results for multi-task model merging. \texttt{Iso-CTS} achieves state-of-the-art results in all of the settings. %It outperforms previous SoTA method TSV-M across all the evaluated scenarios.
\texttt{Iso-C} achieves very similar results to \texttt{Iso-CTS} in the 8 task scenario. However, \texttt{Iso-CTS} significantly outperforms \texttt{Iso-C} when merging 14 and 20 models, with improvements of up to 2.8\% in absolute accuracy. This suggests that it is possible to faithfully represent a small number of tasks in the common subspace. However, when the number of tasks increases, it becomes crucial to retain important directions from the task-specific subspaces in order to maximize model merging effectiveness.
\subsection{Analysis and Ablations}\label{sec:analysis}
\begin{figure*}[t!]
\centering
\subfloat[Normalized Accuracy Improvement (NAI) of a model created by retaining $k$ components of \texttt{Iso-C} (associated with top-$k$ singular vectors from $\Delta_{\text{TA}}$).]{\includegraphics[width=0.39\linewidth]{figs/nai-vs-k.pdf}\label{fig:nai-vs-k}}
\hfill
\subfloat[Average Subspace Alignment Ratios ($\text{SAR}_{\text{avg}}$) between merged and task-specific models for varying sets of tasks.]{\includegraphics[width=0.289\linewidth]{figs/alignment-across-tasks.pdf}\label{fig:alignment-across-tasks}}
\hfill
\subfloat[Distribution of accuracies of the merged models for varying sets of tasks.]{\includegraphics[width=0.285\linewidth]{figs/acc-across-tasks.pdf}\label{fig:acc-across-tasks}}
\vspace{-0.05in}
\caption{
(a) The directions associated with the least significant singular values of $\Delta_{\text{TA}}$ have a minor contribution to the performance of \texttt{Iso-C} model.
(b) Task-specific directions introduced in \texttt{Iso-CTS} improve the Average Subspace Alignment Ratio ($\text{SAR}_{\text{avg}}$) between task-specific models and the merged model compared to \texttt{Iso-C} which uses only a common subspace.
(c) Higher alignment translates to higher accuracy of \texttt{Iso-CTS} with respect to \texttt{Iso-C}.
}
\vspace{-0.1in}
\label{fig:iso-cts-motivation}
\end{figure*}
\minisection{From Task Arithmetic to Isotropic Merging.}
We analyze what happens when interpolating between the singular values obtained by Task Arithmetic (TA) and those obtained by \texttt{Iso-C}, i.e. the model with the following spectra:
\begin{equation}
\Sigma_{\beta} = (1 - \beta) \Sigma_{\text{TA}} + \beta \Sigma_{\texttt{Iso-C}},
\end{equation}
where $\beta$ is an interpolation coefficient. Firstly, \cref{fig:ta-iso-interpolation-spectrum} presents the change in singular values spectrum as we interpolate towards $\Delta_{\texttt{Iso-C}}$ ($\beta \rightarrow 1$). The skewed spectrum achieved by Task Arithmetic becomes isotropic, i.e. the scaling factor is equal along all of the singular directions. In~\cref{fig:ta-iso-interpolation-ar-vs-beta} we observe a steady increase in alignment between task-specific and merged models as measured by $\text{SAR}_{\text{avg}}$ (Eq.~\eqref{eq:sar}), and \cref{fig:ta-iso-interpolation-nai-vs-beta} shows that as alignment increases (with $\beta \rightarrow 1$), the performance of the merged model improves across all tasks. These results are consistent with our findings from~\cref{sec:motivation-alignment} that show a strong correlation between alignment and the performance of the final model.
\minisection{The impact of singular directions on performance.}
We analyze which singular directions contribute to the improvement of individual tasks. We truncate the flattened spectrum of \texttt{Iso-C}, keeping the $k$ directions associated with the leftmost singular values, i.e. $\sigma_i = \overline{\sigma}$ for $i \leq k$ and $\sigma_i = 0$ for $i > k$. Note that the leftmost $k$ directions are the ones associated with the highest singular values of $\Delta_{\text{TA}}$. We plot the task-wise Normalized Accuracy Improvement (NAI, Eq. \eqref{eq:nai}) for varying $k$ in~\cref{fig:nai-vs-k}. We observe that the first few directions are responsible for rapid improvement on several tasks. Notably, these tasks belong to the aligned groups identified in~\cref{sec:motivation-alignment} such as \{MNIST, SVHN, GTSRB\} and \{EuroSAT, RESISC45\}. Moreover, the directions associated with the least significant singular values of $\Delta_{\text{TA}}$ have a negligible contribution to the performance. This supports our intuition for replacing less significant common directions with task-specific components in \texttt{Iso-CTS} (see ~\cref{sec:isocts}). \cref{fig:alignment-across-tasks} shows that \texttt{Iso-CTS} achieves higher Average Subspace Alignment Ratio ($\text{SAR}_{\text{avg}}$, Eq.~\eqref{eq:sar}) than \texttt{Iso-C}.
Most importantly, \cref{fig:acc-across-tasks} shows that thanks to the addition of task-specific directions, \texttt{Iso-CTS} achieves better performance across tasks.
\minisection{Size of the common subspace for \texttt{Iso-CTS}.}
\begin{figure}[t!]
\centering
% \vspace{-0.1in}
\includegraphics[width=0.8\linewidth]{figs/frac-common-space.pdf}
\vspace{-0.1in}
\caption{\texttt{Iso-CTS} is robust to the selected size of the common subspace as any value leads to improvement over \texttt{Iso-C}. These results are for the 20-task scenario.}
\label{fig:frac-common-subspace}
\vspace{-0.1in}
\end{figure}
While \texttt{Iso-C} operates only in the common subspace, \texttt{Iso-CTS} enhances it with task-specific subspaces. Therefore, we must select the size of the common subspace $k$ (and consequently the size of each task-specific subspace given by $\frac{r-k}{T}$). \cref{fig:frac-common-subspace} plots the relationship between accuracy and the fraction of subspace assigned for the common subspace ($\frac{k}{r}$) when merging 20 tasks. When $\frac{k}{r}=1$ \texttt{Iso-CTS} is equivalent to \texttt{Iso-C} and suffers a 2.8\% drop in accuracy from the maximum. The optimal fraction of common subspace $\frac{k}{r} = 0.8$, and we use this as a default value for \texttt{Iso-CTS} across all settings. Moreover, note that \texttt{Iso-CTS} is quite robust to the selection of this hyperparameter -- any $\frac{k}{r} \in (0.0, 1.0)$ offers a performance improvement over \texttt{Iso-C} while the performance for $\frac{k}{r} \in [0.5, 0.9]$ varies by less than 0.5\% from the optimal one.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}
In this work, we introduced an isotropic model merging framework that enhances alignment between task-specific and merged model subspaces to significantly improve the multi-task performance of the final merged model. We proposed \texttt{Iso-C}, which leverages Singular Value Decomposition to equalize singular values and create a more balanced representation across tasks, and \texttt{Iso-CTS}, which further incorporates task-specific directions to retain unique task features while preserving shared knowledge. \texttt{Iso-CTS} achieves state-of-the-art results across multiple model scales and task sets, demonstrating that subspace alignment is a critical factor in effective model merging. These findings provide new insights into model merging and pave the way for the future development of more effective techniques for combining the knowledge of multiple models.
\minisection{Limitations.} The common subspace is determined by Task Arithmetic, which can be suboptimal, and better methods can be developed. We consider only vision tasks, and future work could extend our findings to other domains, such as natural language processing.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Impact Statement}
This paper aims to advance the field of Machine Learning, specifically the subfield focused on merging models fine-tuned on different tasks to create a more effective multi-task model. With the growing popularity of deep learning, increasingly powerful open-source models are becoming widely available and are being adopted in both research and industry. Advances in model merging could enhance the flexibility of utilizing these models by providing an efficient way to combine their specialized capabilities. Beyond this, our paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliography{main}
\bibliographystyle{icml2025}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\onecolumn
\section{Additional details}
\subsection{Datasets}
\label{apdx:datasets}
The 8-dataset benchmark consists of: Cars~\cite{cars}, DTD~\cite{dtd}, EuroSAT~\cite{eurosat}, GTSRB~\cite{gtsrb}, MNIST~\cite{MNIST}, RESISC45~\cite{cheng2017remote}, SUN397~\cite{sun397}, and SVHN~\cite{svhn}.
The 14-dataset benchmark builds on the preceding one, incorporating six additional datasets: CIFAR100~\cite{krizhevsky_learning_nodate}, STL10~\cite{coates_analysis_2011}, Flowers102~\cite{nilsback_automated_2008}, OxfordIIITPet~\cite{parkhi_cats_2012}, PCAM~\cite{veeling_rotation_2018}, and FER2013~\cite{goodfellow_challenges_2013}.
Finally, the 20-dataset benchmark includes the preceding 14 plus the following six: EMNIST~\cite{cohen_emnist_2017}, CIFAR10~\cite{krizhevsky_learning_nodate}, Food101~\cite{bossard_food-101_2014}, FashionMNIST~\cite{xiao_fashion-mnist_2017}, RenderedSST2~\cite{socher_recursive_nodate}, and KMNIST~\cite{clanuwat_deep_2018}.
\subsection{Implementation details}
Our method relies on SVD, which is defined for two-dimensional matrices $\Delta \in \mathbb{R}^{m\times n}$. However, some weights of the neural networks are represented by vectors $\delta \in \mathbb{R}^{n}$, e.g. bias vectors and parameters of layer normalization~\cite{ba2016layer}. Therefore, following~\citet{tsv}, we apply simple averaging to combine these parameters. Code to reproduce all experiments will be released upon publication of this work.
\section{Additional experiments}
\subsection{Visualization of task matrix spectra}
When visualizing spectra of singular values of task matrices (\cref{fig:teaser} and \cref{fig:ta-iso-interpolation}), we selected an output projection matrix $W^O$ from layer $\ell=4$ of ViT/B-16 as an illustrative example. In~\cref{fig:combined-spectra}, we present spectra across a variety of layers of ViT/B-16 for the task matrices of task-specific models, TA, \texttt{Iso-C} and \texttt{Iso-CTS}.
\begin{figure}[t]
\includegraphics[width=\linewidth]{figs/combined-spectra.pdf}
\caption{Visualization of singular value spectra of different task matrices for different types of layers in ViT/B-16.}
\label{fig:combined-spectra}
\end{figure}
\subsection{Selection of scaling coefficient $\alpha$}
\begin{table}[h]
\begin{center}
\caption{Optimal $\alpha$ value chosen on a held-out validation set for different model types and numbers of tasks for \texttt{Iso-C} and \texttt{Iso-CTS}.}
\vspace{0.05in}
\scalebox{0.85}{
\begin{tabular}{c|c|c|c|c}
\toprule
\textbf{Method} & \textbf{Model} & \textbf{8 tasks} & \textbf{14 tasks} & \textbf{20 tasks} \\
\midrule
\multirow{3}{*}{\texttt{Iso-C}} & ViT/32-B & $1.30$ & $1.00$ & $0.90$ \\
& ViT/16-B & $1.40$ & $1.00$ & $0.80$ \\
& ViT/14-L & $1.50$ & $1.30$ & $1.00$ \\
\midrule
\multirow{3}{*}{\texttt{Iso-CTS}} & ViT/32-B & $1.50$ & $1.20$ & $1.10$ \\
& ViT/16-B & $1.60$ & $1.20$ & $1.10$ \\
& ViT/14-L & $1.90$ & $1.50$ & $1.20$ \\
\bottomrule
\end{tabular}
}
\label{tab:alpha_table}
\end{center}
\end{table}
On \cref{fig:acc-vs-alpha}, we present the relationship between the validation accuracy and scaling factor $\alpha$. We observe that TA is very sensitive to the selection of $\alpha$, which potentially may require a more fine-grained search. On the other hand, both \texttt{Iso-C} and \texttt{Iso-CTS} are more robust to $\alpha$ selection, resembling the task-specific models. For reproducibility, In Table \ref{tab:alpha_table}, we provide the optimal $\alpha$ value chosen on the held-out validation set for each model and number of tasks.
\begin{figure}[t]
\includegraphics[width=\linewidth]{figs/acc-vs-alpha.pdf}
\caption{TA is sensitive to the selection of $\alpha$, while both \texttt{Iso-C} and \texttt{Iso-CTS} are more robust to $\alpha$ selection, resembling the task-specific models. The $\alpha$ is chosen based on the best average performance on the validation set across tasks. The bottom right subplot denotes the optimal $\alpha$ for each method (Eq. \eqref{eq:ta}, Eq. \eqref{eq:isoc-alpha} and Eq. \eqref{eq:isocts-alpha}). The model is ViT-B/16.}
\label{fig:acc-vs-alpha}
\end{figure}
\subsection{Applying \texttt{Iso} to individual task matrices}
Flattening the skewed spectrum of singular values significantly improves the performance of the merged model, as demonstrated in \cref{sec:analysis}. One may wonder if this operation might also be an effective strategy for improving single-task models. \cref{fig:ind-orig-vs-ind-iso} presents the performance of task-specific models in their original form along with their modified versions with singular value spectra of their task matrices flattened (which is equivalent to performing \texttt{Iso-C} for a single model). We observe a 3.3\% drop in average performance across tasks. Therefore, the reason for the success of \texttt{Iso-C} lies in its ability to mitigate the negative effects of summing task matrices, not in inadvertently improving the original individual task matrices.
\begin{figure}[t]
\centering
\includegraphics[width=0.8\linewidth]{figs/ind-orig-vs-ind-iso.pdf}
\caption{Validation Accuracy while scaling task matrices with $\alpha$ coefficient (Eq.~\eqref{eq:ta} applied for a single task). We observe a performance gap between the accuracy of original and modified models for the optimal values of $\alpha$ (denoted by square).}
\label{fig:ind-orig-vs-ind-iso}
\end{figure}
\end{document}
\begin{thebibliography}{45}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ba et~al.(2016)Ba, Kiros, and Hinton]{ba2016layer}
Ba, J.~L., Kiros, J.~R., and Hinton, G.~E.
\newblock Layer normalization.
\newblock \emph{arXiv preprint arXiv: 1607.06450}, 2016.
\bibitem[Bossard et~al.(2014)Bossard, Guillaumin, and Van~Gool]{bossard_food-101_2014}
Bossard, L., Guillaumin, M., and Van~Gool, L.
\newblock Food-101 – {Mining} {Discriminative} {Components} with {Random} {Forests}.
\newblock In \emph{ECCV}, 2014.
\bibitem[Carion et~al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko]{carion2020endtoend}
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S.
\newblock End-to-end object detection with transformers.
\newblock \emph{ECCV}, 2020.
\bibitem[Caron et~al.(2021)Caron, Touvron, Misra, Jegou, Mairal, Bojanowski, and Joulin]{DINO}
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., and Joulin, A.
\newblock Emerging properties in self-supervised vision transformers.
\newblock \emph{ICCV}, 2021.
\bibitem[Cheng et~al.(2017)Cheng, Han, and Lu]{cheng2017remote}
Cheng, G., Han, J., and Lu, X.
\newblock Remote sensing image scene classification: Benchmark and state of the art.
\newblock \emph{Proceedings of the IEEE}, 2017.
\bibitem[Cimpoi et~al.(2014)Cimpoi, Maji, Kokkinos, Mohamed, and Vedaldi]{dtd}
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A.
\newblock Describing textures in the wild.
\newblock In \emph{CVPR}, 2014.
\bibitem[Clanuwat et~al.(2018)Clanuwat, Bober-Irizar, Kitamoto, Lamb, Yamamoto, and Ha]{clanuwat_deep_2018}
Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A., Yamamoto, K., and Ha, D.
\newblock Deep {Learning} for {Classical} {Japanese} {Literature}.
\newblock \emph{arXiv preprint arXiv: 1607.06450}, 2018.
\bibitem[Coates et~al.(2011)Coates, Ng, and Lee]{coates_analysis_2011}
Coates, A., Ng, A., and Lee, H.
\newblock An {Analysis} of {Single}-{Layer} {Networks} in {Unsupervised} {Feature} {Learning}.
\newblock In \emph{Proceedings of the {Fourteenth} {International} {Conference} on {Artificial} {Intelligence} and {Statistics}}. JMLR Workshop and Conference Proceedings, 2011.
\bibitem[Cohen et~al.(2017)Cohen, Afshar, Tapson, and van Schaik]{cohen_emnist_2017}
Cohen, G., Afshar, S., Tapson, J., and van Schaik, A.
\newblock {EMNIST}: {Extending} {MNIST} to handwritten letters.
\newblock In \emph{IJCNN}, 2017.
\bibitem[Daheim et~al.(2024)Daheim, M{\"{o}}llenhoff, Ponti, Gurevych, and Khan]{DaheimMPGK24}
Daheim, N., M{\"{o}}llenhoff, T., Ponti, E.~M., Gurevych, I., and Khan, M.~E.
\newblock Model merging by uncertainty-based gradient matching.
\newblock In \emph{ICLR}, 2024.
\bibitem[Davari \& Belilovsky(2024)Davari and Belilovsky]{davari2023model}
Davari, M.-J. and Belilovsky, E.
\newblock Model breadcrumbs: Scaling multi-task model merging with sparse masks.
\newblock \emph{ECCV}, 2024.
\bibitem[Denton et~al.(2014)Denton, Zaremba, Bruna, LeCun, and Fergus]{NIPS2014_2afe4567}
Denton, E.~L., Zaremba, W., Bruna, J., LeCun, Y., and Fergus, R.
\newblock Exploiting linear structure within convolutional networks for efficient evaluation.
\newblock In \emph{NeurIPS}, 2014.
\bibitem[Dosovitskiy et~al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby]{dosovitskiy2021an}
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.
\newblock An image is worth 16x16 words: Transformers for image recognition at scale.
\newblock In \emph{ICLR}, 2021.
\bibitem[Gargiulo et~al.(2024)Gargiulo, Crisostomi, Bucarelli, Scardapane, Silvestri, and Rodolà]{tsv}
Gargiulo, A.~A., Crisostomi, D., Bucarelli, M.~S., Scardapane, S., Silvestri, F., and Rodolà, E.
\newblock Task singular vectors: Reducing task interference in model merging.
\newblock \emph{arXiv preprint arXiv: 2412.00081}, 2024.
\bibitem[Goodfellow et~al.(2013)Goodfellow, Erhan, Carrier, Courville, Mirza, Hamner, Cukierski, Tang, Thaler, Lee, Zhou, Ramaiah, Feng, Li, Wang, Athanasakis, Shawe-Taylor, Milakov, Park, Ionescu, Popescu, Grozea, Bergstra, Xie, Romaszko, Xu, Chuang, and Bengio]{goodfellow_challenges_2013}
Goodfellow, I.~J., Erhan, D., Carrier, P.~L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Milakov, M., Park, J., Ionescu, R., Popescu, M., Grozea, C., Bergstra, J., Xie, J., Romaszko, L., Xu, B., Chuang, Z., and Bengio, Y.
\newblock Challenges in {Representation} {Learning}: {A} {Report} on {Three} {Machine} {Learning} {Contests}.
\newblock \emph{Neural Networks}, 2013.
\bibitem[Helber et~al.(2019)Helber, Bischke, Dengel, and Borth]{eurosat}
Helber, P., Bischke, B., Dengel, A., and Borth, D.
\newblock Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.
\newblock \emph{Journal of Selected Topics in Applied Earth Observations and Remote Sensing}, 2019.
\bibitem[Hu et~al.(2021)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, and Chen]{hu2021lora}
Hu, J.~E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W.
\newblock Lora: Low-rank adaptation of large language models.
\newblock \emph{ICLR}, 2021.
\bibitem[Ilharco et~al.(2022)Ilharco, Wortsman, Gadre, Song, Hajishirzi, Kornblith, Farhadi, and Schmidt]{ilharco2022patching}
Ilharco, G., Wortsman, M., Gadre, S.~Y., Song, S., Hajishirzi, H., Kornblith, S., Farhadi, A., and Schmidt, L.
\newblock Patching open-vocabulary models by interpolating weights.
\newblock In \emph{NeurIPS}, 2022.
\bibitem[Ilharco et~al.(2023)Ilharco, Ribeiro, Wortsman, Schmidt, Hajishirzi, and Farhadi]{ilharco2023task}
Ilharco, G., Ribeiro, M.~T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A.
\newblock Editing models with task arithmetic.
\newblock In \emph{ICLR}, 2023.
\bibitem[Kim et~al.(2016)Kim, Park, Yoo, Choi, Yang, and Shin]{KimPYCYS15}
Kim, Y., Park, E., Yoo, S., Choi, T., Yang, L., and Shin, D.
\newblock Compression of deep convolutional neural networks for fast and low power mobile applications.
\newblock In \emph{ICLR}, 2016.
\bibitem[Krause et~al.(2013)Krause, Stark, Deng, and Fei-Fei]{cars}
Krause, J., Stark, M., Deng, J., and Fei-Fei, L.
\newblock {3D Object representations for fine-grained categorization}.
\newblock In \emph{ICCV Workshops}, 2013.
\bibitem[Krizhevsky \& Hinton(2009)Krizhevsky and Hinton]{krizhevsky_learning_nodate}
Krizhevsky, A. and Hinton, G.
\newblock Learning multiple layers of features from tiny images.
\newblock Technical Report~0, University of Toronto, Toronto, Ontario, 2009.
\newblock URL \url{https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf}.
\bibitem[Lecun et~al.(1998)Lecun, Bottou, Bengio, and Haffner]{MNIST}
Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P.
\newblock Gradient-based learning applied to document recognition.
\newblock \emph{Proceedings of the IEEE}, 1998.
\bibitem[Li et~al.(2023)Li, Peng, Zhang, Ding, Hu, and Shen]{li2023deep}
Li, W., Peng, Y., Zhang, M., Ding, L., Hu, H., and Shen, L.
\newblock Deep model fusion: A survey.
\newblock \emph{arXiv preprint arXiv: 2309.15698}, 2023.
\bibitem[Lingam et~al.(2024)Lingam, Tejaswi, Vavre, Shetty, Gudur, Ghosh, Dimakis, Choi, Bojchevski, and Sanghavi]{svft}
Lingam, V., Tejaswi, A., Vavre, A., Shetty, A., Gudur, G.~K., Ghosh, J., Dimakis, A., Choi, E., Bojchevski, A., and Sanghavi, S.
\newblock {SVFT:} parameter-efficient fine-tuning with singular vectors.
\newblock \emph{CoRR}, abs/2405.19597, 2024.
\bibitem[Marczak et~al.(2024)Marczak, Twardowski, Trzcinski, and Cygert]{MarczakTTC24}
Marczak, D., Twardowski, B., Trzcinski, T., and Cygert, S.
\newblock {MagMax:} {Leveraging} {Model} {Merging} for {Seamless} {Continual} {Learning}.
\newblock In \emph{ECCV}, 2024.
\bibitem[Netzer et~al.(2011)Netzer, Wang, Coates, Bissacco, Wu, and Ng]{svhn}
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A.~Y.
\newblock Reading digits in natural images with unsupervised feature learning.
\newblock In \emph{NeurIPS Workshops}, 2011.
\bibitem[Nilsback \& Zisserman(2008)Nilsback and Zisserman]{nilsback_automated_2008}
Nilsback, M.-E. and Zisserman, A.
\newblock Automated {Flower} {Classification} over a {Large} {Number} of {Classes}.
\newblock In \emph{2008 {Sixth} {Indian} {Conference} on {Computer} {Vision}, {Graphics} \& {Image} {Processing}}, 2008.
\bibitem[Ortiz{-}Jim{\'{e}}nez et~al.(2023)Ortiz{-}Jim{\'{e}}nez, Favero, and Frossard]{ortizjimenez2023tangent}
Ortiz{-}Jim{\'{e}}nez, G., Favero, A., and Frossard, P.
\newblock Task arithmetic in the tangent space: Improved editing of pre-trained models.
\newblock In \emph{NeurIPS}, 2023.
\bibitem[Parkhi et~al.(2012)Parkhi, Vedaldi, Zisserman, and Jawahar]{parkhi_cats_2012}
Parkhi, O.~M., Vedaldi, A., Zisserman, A., and Jawahar, C.~V.
\newblock Cats and dogs.
\newblock In \emph{CVPR}, 2012.
\bibitem[Po et~al.(2024)Po, Yang, Aberman, and Wetzstein]{PoYAW24}
Po, R., Yang, G., Aberman, K., and Wetzstein, G.
\newblock Orthogonal adaptation for modular customization of diffusion models.
\newblock In \emph{CVPR}, 2024.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever]{radford2021learning}
Radford, A., Kim, J.~W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.
\newblock Learning transferable visual models from natural language supervision.
\newblock In \emph{ICML}, 2021.
\bibitem[Schönemann(1966)]{schonemann1966}
Schönemann, P.~H.
\newblock A generalized solution of the orthogonal procrustes problem.
\newblock \emph{Psychometrika}, 1966.
\bibitem[Socher et~al.(2013)Socher, Perelygin, Wu, Chuang, Manning, Ng, and Potts]{socher_recursive_nodate}
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.~D., Ng, A., and Potts, C.
\newblock Recursive deep models for semantic compositionality over a sentiment treebank.
\newblock In \emph{EMNLP}, 2013.
\bibitem[Stallkamp et~al.(2011)Stallkamp, Schlipsing, Salmen, and Igel]{gtsrb}
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C.
\newblock The german traffic sign recognition benchmark: a multi-class classification competition.
\newblock In \emph{IJCNN}, 2011.
\bibitem[Stoica et~al.(2024)Stoica, Ramesh, Ecsedi, Choshen, and Hoffman]{stoica2024knots}
Stoica, G., Ramesh, P., Ecsedi, B., Choshen, L., and Hoffman, J.
\newblock Model merging with svd to tie the knots.
\newblock \emph{arXiv preprint arXiv: 2410.19735}, 2024.
\bibitem[Veeling et~al.(2018)Veeling, Linmans, Winkens, Cohen, and Welling]{veeling_rotation_2018}
Veeling, B.~S., Linmans, J., Winkens, J., Cohen, T., and Welling, M.
\newblock Rotation {Equivariant} {CNNs} for {Digital} {Pathology}.
\newblock In \emph{MICCAI}, 2018.
\bibitem[Wang et~al.(2024{\natexlab{a}})Wang, Xiao, Li, Wang, Chen, and Chen]{milora}
Wang, H., Xiao, Z., Li, Y., Wang, S., Chen, G., and Chen, Y.
\newblock Milora: Harnessing minor singular components for parameter-efficient {LLM} finetuning.
\newblock \emph{CoRR}, abs/2406.09044, 2024{\natexlab{a}}.
\bibitem[Wang et~al.(2024{\natexlab{b}})Wang, Dimitriadis, Ortiz{-}Jim{\'{e}}nez, Fleuret, and Frossard]{wang2024localizing}
Wang, K., Dimitriadis, N., Ortiz{-}Jim{\'{e}}nez, G., Fleuret, F., and Frossard, P.
\newblock Localizing task information for improved model merging and compression.
\newblock In \emph{ICML}, 2024{\natexlab{b}}.
\bibitem[Wortsman et~al.(2022{\natexlab{a}})Wortsman, Ilharco, Gadre, Roelofs, Gontijo-Lopes, Morcos, Namkoong, Farhadi, Carmon, Kornblith, et~al.]{wortsman2022model}
Wortsman, M., Ilharco, G., Gadre, S.~Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A.~S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et~al.
\newblock Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.
\newblock In \emph{ICML}, 2022{\natexlab{a}}.
\bibitem[Wortsman et~al.(2022{\natexlab{b}})Wortsman, Ilharco, Kim, Li, Kornblith, Roelofs, Lopes, Hajishirzi, Farhadi, Namkoong, and Schmidt]{wortsman2022robust}
Wortsman, M., Ilharco, G., Kim, J.~W., Li, M., Kornblith, S., Roelofs, R., Lopes, R.~G., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L.
\newblock Robust fine-tuning of zero-shot models.
\newblock In \emph{CVPR}, 2022{\natexlab{b}}.
\bibitem[Xiao et~al.(2017)Xiao, Rasul, and Vollgraf]{xiao_fashion-mnist_2017}
Xiao, H., Rasul, K., and Vollgraf, R.
\newblock Fashion-{MNIST}: a {Novel} {Image} {Dataset} for {Benchmarking} {Machine} {Learning} {Algorithms}, 2017.
\newblock URL \url{http://arxiv.org/abs/1708.07747}.
\bibitem[Xiao et~al.(2016)Xiao, Ehinger, Hays, Torralba, and Oliva]{sun397}
Xiao, J., Ehinger, K.~A., Hays, J., Torralba, A., and Oliva, A.
\newblock Sun database: Exploring a large collection of scene categories.
\newblock \emph{IJCV}, 2016.
\bibitem[Yadav et~al.(2023)Yadav, Tam, Choshen, Raffel, and Bansal]{yadav2023tiesmerging}
Yadav, P., Tam, D., Choshen, L., Raffel, C., and Bansal, M.
\newblock {TIES}-merging: Resolving interference when merging models.
\newblock In \emph{NeurIPS}, 2023.
\bibitem[Zhai et~al.(2023)Zhai, Mustafa, Kolesnikov, and Beyer]{zhai2023sigmoid}
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L.
\newblock Sigmoid loss for language image pre-training.
\newblock \emph{ICCV}, 2023.
\end{thebibliography}