Cross-Domain Ensemble Distillation for Domain Generalization
Extracted source text · arXiv 2211.14058
\begin{abstract}
Domain generalization is the task of learning models that generalize to unseen target domains. We propose a simple yet effective method for domain generalization, named cross-domain ensemble distillation (XDED), that learns domain-invariant features while encouraging the model to converge to flat minima, which recently turned out to be a sufficient condition for domain generalization. To this end, our method generates an ensemble of the output logits from training data with the same label but from different domains and then penalizes each output for the mismatch with the ensemble. Also, we present a de-stylization technique that standardizes features to encourage the model to produce style-consistent predictions even in an arbitrary target domain. Our method greatly improves generalization capability in public benchmarks for cross-domain image classification, cross-dataset person re-ID, and cross-dataset semantic segmentation. Moreover, we show that models learned by our method are robust against adversarial attacks and image corruptions.
% suha-backup
% Domain generalization is the task of learning models that generalize to unseen target domains. One of the mainstream approaches to domain generalization is to align multiple source domains in a feature space while learning a model to predict labels of the source data. This approach can learn domain-invariant features but has a risk of overfitting to the source domains. To overcome We propose a simple yet effective method, named cross-domain ensemble distillation (XDED), that learns domain-invariant features while encouraging the model to converge on flat minima, which recently turned out to be a sufficient condition for domain generalization. To this end, our method generates an ensemble of the output distributions for the data with the same label but from different domains and then penalizes each output distribution for the mismatch with the ensemble. Also, we present a de-stylization technique that standardizes features to encourage the model to produce style-consistent predictions even in an arbitrary target domain. Our method greatly improves generalization capability in two benchmarks for cross-domain image classification, cross-dataset person re-ID, and cross-dataset semantic segmentation. Moreover, we show that models learned by our method are also robust against adversarial attacks.
% suha-backup
%Domain generalization aims to generalize to unseen target domains by learning the model on the multiple source domains. One of the mainstream approaches is to align multiple source domains and learn a classifier to predict labels of the source data. This approach can learn domain-invariant features but has a risk of overfitting on the source domains. Inspired by the recent analysis result that flat minima should be considered for domain generalization, we propose a simple yet novel approach, named cross-domain ensemble distillation (XDED), that aims to learn domain-invariant features while helping the model converge on flat minima. Specifically, our approach generates an ensemble of the output distributions for the data with the same label but from different domains and then penalizes each output distribution for the mismatch with the ensemble. Furthermore, we present a de-stylization technique that standardizes feature maps of images to encourage the model to produce style-consistent predictions even in the target domain. We show that our method greatly improves generalization capability in several benchmarks for domain generalization and cross-dataset person re-ID. Lastly, we show that our framework is also robust against adversarial attacks and unseen corruptions.
\keywords{domain generalization, knowledge distillation, flat minima}
\end{abstract}
\section{Introduction}
\begin{figure*}[t]
\centering
\includegraphics[width=0.9\textwidth]{CameraReady/figure/cr_fig1.pdf}
\caption{Illustration of cross-domain ensemble distillation (XDED). Although the four images share the same class label, their predictions manifest different inter-class relations due to the visual gap between domains. XDED constructs an ensemble by averaging all predictions and matches it with each prediction.
% (\eg, the predictive distribution of Cartoon has a high probability on class Person, but that of Sketch has a high probability on class Dog.)
}
\label{fig:teaser}
\end{figure*}
Deep neural networks (DNNs) have brought remarkable advances in a number of research areas such as image classification~\cite{Alexnet}, image synthesis~\cite{goodfellow2014generative}, and reinforcement learning~\cite{mnih2013playing}. The huge success of DNNs depends heavily on the assumption that training and test data are sampled under the independent and identically distributed (i.i.d.)~condition. However, this assumption often does not hold in real-world scenarios; a large error occurs due to the discrepancy between training and test data, also known as the domain shift problem. As a solution to this problem, domain generalization, the task of learning models that generalize to unseen target domains, is in the spotlight.
A key to the success of domain generalization is to learn invariant features across domains. To this end, most previous methods align feature distributions of multiple domains by adversarial training~\cite{li2018domain,li2018deep}, minimizing the dissimilarity between the distributions of source domains~\cite{muandet2013domain}, or contrastive learning~\cite{kim2021selfreg}. Then, a classifier is trained to predict the labels for the aligned source features in hopes that it will also generalize well for any target domain.
However, this approach often drops performance when the target domain differs substantially from the source domains as the model is prone to overfit to the source domains.
% However, performance drop becomes inevitable when the target differs from the source as the model is often overfitted to the source domains.
Meanwhile, the relationship between the geometry of loss landscapes and generalization ability has attracted increasing attention~\cite{dziugaite2017computing,foret2021sharpness,jiang2020fantastic,keskar2016large}.
In particular, converging to flat minima in loss landscapes is known as a key to achieve robustness against the
loss landscape shift between training and test datasets.
Inspired by the observation that higher posterior entropy helps a model converge to flat minima~\cite{chaudhari2017entropy,pereyra2017regularizing,zhang2018deep}, entropy regularization techniques like self-knowledge distillation~\cite{zhang2019your} and entropy maximization~\cite{cha2020cpr} have been proposed to increase entropy rather than forcing a model to completely fit training data (\ie, one-hot labels) to induce low entropy.
% and demonstrate their efficacy empirically.
Since the degree
of loss landscape shift is generally expected to be bigger in the case of domain generalization, it is more important to converge to flat minima
% in the case of the i.i.d. condition.
in domain generalization.
However, the benefit of flat minima in terms of domain generalization has not been actively studied yet.
In this paper, we propose a novel method, named \emph{cross-domain ensemble distillation} (XDED), that learns domain-invariant features while encouraging convergence to flat minima for domain generalization.
Specifically, XDED generates an ensemble of the output logits for the data with the same label but from different domains, and then penalizes each output for the mismatch with the ensemble (Fig.~\ref{fig:teaser}).
% XDED naturally achieves two main goals.
% First, it
By doing so, it enables a model to learn domain-invariant features by enforcing prediction consistency between the data with the same label but from different domains.
Also,
% Due to the information discrepancy between domains as shown in Fig.~\ref{fig:teaser},
XDED increases the posterior entropy of each output distribution, which helps the model converge to flat minima as the entropy regularization does.
% for domain generalization.
To the best of our knowledge, XDED is the first to achieve these two objectives simultaneously for domain generalization, and this contribution leads to significant performance improvement.
Since XDED is still limited to exploiting the information of only source domains, there is further room to reduce the domain gap with the target domain. Hence, we also introduce a de-stylization technique well-suited to domain generalization, called UniStyle. UniStyle suppresses domain-specific style bias simply by standardizing intermediate feature maps of input image during both training and test time. Thanks to UniStyle, our model produces style-consistent predictions not only for the source domains but also for the target domain, which greatly reduces the domain gap and boosts the effect of XDED.
Based on the recent theoretical result on the relationship between the domain generalization and the flatness of local minima~\cite{cha2021swad}, we first empirically show that the proposed framework can improve generalization capability by achieving two goals: promoting flat minima and reducing the domain gap. Next, we further demonstrate the superiority of our method through extensive experimental results.
On the standard public benchmarks for cross-domain image classification, XDED significantly enhances generalization ability in both multi-source and single-source settings.
We also validate the effectiveness of our method in various domain generalization scenarios by showing the non-trivial improvement on the DomainBed~\cite{gulrajani2020search}, cross-dataset person re-ID~\cite{zheng2015scalable,zheng2017unlabeled}, and cross-dataset semantic segmentation experiments. Moreover, we demonstrate that models learned by our method also help achieve robustness against adversarial attacks and unseen image corruptions.
\section{Related Work}
\label{relatedwork}
\noindent \textbf{Domain generalization.}
The goal of domain generalization is to learn domain-invariant features that well generalize to unseen target domains.
For the purpose, existing methods match feature distributions of different domains by adversarial feature alignment~\cite{li2018domain,li2018deep} or reducing the difference between feature distributions of diverse source domains~\cite{muandet2013domain}.
Recently, meta-learning frameworks~\cite{balaji2018metareg,dou2019domain,li2019episodic} have been introduced to simulate the domain shift by dividing the meta-train and meta-test domains from source domains.
Also, data augmentation methods have been proposed to generate more diverse data beyond those of given source domains~\cite{StyleNeophile,kim2021wedge,shankar2018generalizing,xu2021fourier,zhou2020learning}.
%
% For instance, CrossGrad~\cite{shankar2018generalizing} perturbs images according to adversarial gradients induced by a domain classifier. L2A-OT~\cite{zhou2020learning} learns a generator to map source data to synthetic domains by maximizing a divergence measure. FACT~\cite{xu2021fourier} mixes the amplitude spectrums of two images from a Fourier-based perspective.
%
Most similar to our framework, ensemble methods for domain generalization have been proposed~\cite{xu2014exploiting,seo2020learning,zhou2021dael}. They all train multiple modules such as exemplar SVMs~\cite{xu2014exploiting}, domain-specific BN~\cite{Batchnorm} layers~\cite{seo2020learning} or classifiers~\cite{zhou2021dael}, and exploit the ensemble of learned modules for prediction in testing.
% when predicting the test data.
However, we remark that our XDED utilizes the ensemble of model predictions as the soft label and transfers it to the model itself. Therefore, it does not demand any additional module during both training and testing.
% Further comparative analysis is presented in the supplementary material.
% Sec.~\ref{sec:experiment_ablation}.
\noindent \textbf{Knowledge distillation (KD).}
KD
% , which is mainly devised for model compression,
was originally studied to transfer the knowledge of a deep model to a shallow model for model compression~\cite{hinton2015distilling}.
% For example, a seminal work by Hinton \etal~\cite{hinton2015distilling} encourages a student model to imitate class logits of a teacher model, which contain richer information than one-hot labels.
%
% KD has been applied for other purposes such as cross-modality learning~\cite{tian2019contrastive} and metric learning~\cite{park2019relational,kim2021embedding}.
% In addition, it has been used for network regularization~\cite{xu2019data,zhang2019your,yun2020regularizing}, which is named self-knowledge distillation (self-KD).
% Unlike the conventional teacher-student framework, self-KD mainly distills the knowledge from the model itself and enforce consistency between the query and other data such as perturbed ones or other samples.
It has been also used for other purposes such as metric learning~\cite{park2019relational,kim2021embedding} and network regularization~\cite{xu2019data,zhang2019your,yun2020regularizing}.
In particular for network regularization, self-knowledge distillation (self-KD) has been studied; it distills knowledge from the model itself and enforces prediction consistency between a sample and its perturbed one or other samples.
KD has been used for domain adaptation~\cite{meng2018adversarial,feng2020kd3a}, and such method trains several teacher models from the source domains and distills the ensemble of their predictions to the student model.
It unfortunately requires large memory due to multiple teachers, and are difficult to be extended to domain generalization as they demand target images in training.
% along with the strategy of domain adaptation.
In contrast, our method improves generalization capability of a model on unseen domains without the need for target images and additional teacher models.
\noindent \textbf{Flat minima in loss landscapes.}
Recent analyses have revealed that finding flat minima is crucial for model generalization~\cite{keskar2016large,dziugaite2017computing,foret2021sharpness}.
% Recently, finding flat minima has been in spotlight in the context of model generalization~\cite{keskar2016large,dziugaite2017computing,foret2021sharpness}.
In this context,
multiple methods have been proposed to promote flat minima in loss landscapes since flat minima have an advantage over sharp minima in robustness against the loss landscape shift between training and test data.% A flat minimum has an advantage over a sharp minimum in robustness to changes in the loss environment between the training data and the test data.
Among literature on ways of promoting flat minima (\eg, weight averaging~\cite{izmailov2018averaging,cha2021swad} and training strategies~\cite{foret2021sharpness,chaudhari2017entropy}), we focus on the high entropy-seeking approaches, on which XDED is based.
Maximum Entropy~\cite{pereyra2017regularizing,cha2020cpr} maximizes the entropy of an output distribution from a classifier.
% by following the uniform distribution.
Similarly, KD-based methods also aim at inducing high entropy of the output distribution by penalizing the mismatch with the output distribution from that of another classifier such as differently initialized peer networks~\cite{zhang2018deep} or subnetworks within a network itself~\cite{zhang2019your}. Although SWAD~\cite{cha2021swad} has introduced the importance of flat minima in the area of domain generalization, we remark that SWAD belongs to weight averaging but does not focus on learning domain-invariant features, whereas XDED belongs to entropy regularization as well as is designed for learning domain-invariant features.
%Although SWAD~\cite{cha2021swad} has been proposed to promote flat minima in the area of domain generalization, we remark that SWAD belongs to weight averaging and does not focus on learning domain-invariant features, whereas XDED belongs to entropy regularization as well as being designed for learning domain-invariant features.
\noindent \textbf{Bias towards styles.}
Recent studies~\cite{brendel2019approximating,geirhos2018imagenet} revealed that DNNs overly depend on a strong bias towards styles, and it is also confirmed in the domain generalization literature~\cite{choi2021robustnet,zhou2021domain,StyleNeophile} that a visual domain is highly correlated to feature statistics.
Hence, previous work defines image styles as the bias and attempts to remove the bias by style augmentation in the space of feature statistics~\cite{zhou2021domain,StyleNeophile}, using another model that is intentionally biased to styles~\cite{nam2021reducing}, or minimizing a whitening loss~\cite{choi2021robustnet}.
Distinct from these techniques, we show that a simple yet effective de-stylization technique leads to a smaller divergence measure between target and source domains without bells and whistles.
% As recent studies~\cite{brendel2019approximating,geirhos2018imagenet} demonstrate that deep neural networks overly depend on a strong bias towards styles, it is also confirmed that the visual domain is highly correlated to the feature-level statistics in the domain generalization community~\cite{choi2021robustnet,zhou2021domain}. Therefore, previous approaches have been proposed to define image styles as the bias and attempt to remove the dependency through the augmentation in feature-level statistics space~\cite{zhou2021domain}, the adversarial training against another classifier which is intentionally biased to image styles~\cite{nam2021reducing}, or minimizing the whitening loss which addresses higher-order statistics (\ie, feature covariance)~\cite{choi2021robustnet}. Distinct from them, we find that a simple yet effective de-stylization technique leads to a smaller divergence measure between target and source domains without bells and whistles.
\section{Our Method}
\label{method}
\subsection{Cross-Domain Ensemble Distillation}
\noindent \textbf{Review of knowledge distillation (KD).}
The goal of KD~\cite{hinton2015distilling} is to transfer knowledge of a teacher model $t$ to a student model $s$, usually a wide and deep model to a smaller one, for the purposes of model compression or model regularization. Given input data point $x$ and its label $y \in \{1, \cdots, C \} $, we denote the output logit of model as $z(x) = [z_{1}(x), \cdots, z_{C}(x)]$. The posterior predictive distribution of $x$ is then formulated as:
\begin{align}
\begin{split}
P(y |x; \theta, \tau) = \frac{\textrm{exp}( z_{y}(x) / \tau ) }
{\sum^{C}_{i=1}\textrm{exp}(z_{i}(x)/\tau)},
\end{split}
\label{eq:softmax}
\end{align}
where the model is parameterized by $\theta$ and $\tau$ is a temperature scaling parameter. KD enforces to match the predictive distributions of $s$ and $t$. Specifically, it is achieved by minimizing the Kullback-Leibler (KL) divergence between their predictive distributions as follows:
\begin{align}
\begin{split}
\mathcal{L}_{\textrm{KD}}(X; \theta_{s},\tau) = \sum_{x_{i} \in X}
\sum^{C}_{c=1}
D_{KL}(
P(c|x_{i}; \theta_{t}, \tau) ||
P(c|x_{i}; \theta_{s}, \tau)
),
\end{split}
\label{eq:original_kd}
\end{align}
where $X$ is a batch of input data, $\theta_{t}$ and $\theta_{s}$ are the parameters of a teacher and a student, respectively.
%\noindent \textbf{Cross-domain ensemble distillation (XDED).} We propose a new KD method for domain generalization, called XDED.
% \ready{which aims to transfer domain-integrated and class-specific knowledge obtained from various domains to model.
%It aims to create domain-invariant knowledge by integrating class-wise information extracted from the data of multiple domains, and consistently distills it to all data points of the same class regardless of their domain membership. XDED penalizes predictions between the data of the same class in multiple domains, which encourages learning domain-invariant features that are not biased toward specific domain.
%Specifically, it constructs an ensemble by averaging multiple logits of the data with the same class labels in a mini-batch and utilized it as knowledge.
% XDED defines that knowledge as an ensemble constructed by averaging the logits from the data with the same label but from different domains.
% Thus, the ensemble integrates various domain-specific knowledge and the model learned with it would produce a consistent prediction per class from the data of different domains.
%Unlike the conventional KD, XDED does not require an additional network that increases training complexity (\eg, extra parameters and training time) but distills the ensemble constructed by multiple samples to the model itself in the form of self-KD.
% More specifically, XDED constructs the ensemble by averaging multiple logits of data with the same class labels in a mini-batch. % \ready{The driving rationale behind the usage of the ensemble is to enforce consistency at a meaningful distribution where complementary knowledge from multiple domains is encoded, which is motivated by the observation shown in Fig.~\ref{fig:teaser}.} % (as shown in Figure~\ref{fig:teaser}, the predictive distribution of Cartoon has a high probability on class Person, but that of Sketch has a high probability on class Dog even though they belong to the same class label.)
% \ready{As a result, each sample contributes to constructing the complementary knowledge by providing its own or its domain-specific information. At the same time, it is supervised with that knowledge, which is to learn domain-invariant information.}
\noindent \textbf{Cross-domain ensemble distillation.} We propose a new KD method for domain generalization, called cross-domain ensemble distillation (XDED). XDED aims to construct the domain-invariant knowledge from the data of multiple domains. Specifically, XDED generates an ensemble of logits from the data with the same label but from different domains. Next, XDED penalizes each logit for the mismatch with the ensemble which is not biased towards a specific domain, which encourages learning domain-invariant features.
% \ready{which aims to transfer domain-integrated and class-specific knowledge obtained from various domains to model.
%It aims to create domain-invariant knowledge by integrating class-wise information extracted from the data of multiple domains, and consistently distills it to all data points of the same class regardless of their domain membership.
%XDED penalizes predictions between the data of the same class in multiple domains, which encourages learning domain-invariant features that are not biased toward specific domain.
%Specifically, it constructs an ensemble by averaging multiple logits of the data with the same label in a mini-batch and utilized it as knowledge.
% XDED defines that knowledge as an ensemble constructed by averaging the logits from the data with the same label but from different domains.
% Thus, the ensemble integrates various domain-specific knowledge and the model learned with it would produce a consistent prediction per class from the data of different domains.
Unlike the conventional KD, XDED does not require an additional network that increases training complexity (\eg, extra parameters and training time) but distills the ensemble constructed by multiple samples to the model itself in the form of self-KD.
% More specifically, XDED constructs the ensemble by averaging multiple logits of data with the same class labels in a mini-batch. % \ready{The driving rationale behind the usage of the ensemble is to enforce consistency at a meaningful distribution where complementary knowledge from multiple domains is encoded, which is motivated by the observation shown in Fig.~\ref{fig:teaser}.} % (as shown in Figure~\ref{fig:teaser}, the predictive distribution of Cartoon has a high probability on class Person, but that of Sketch has a high probability on class Dog even though they belong to the same class label.)
% \ready{As a result, each sample contributes to constructing the complementary knowledge by providing its own or its domain-specific information. At the same time, it is supervised with that knowledge, which is to learn domain-invariant information.}
Formally, let $X_{y}$ denote the set of samples that have the same class label $y$ in a mini-batch. Then, we obtain an ensemble of logits from $X_{y}$ by simply taking an average as:
\begin{align}
\begin{split}
\bar{z}(X_{y}) = \sum_{x_{i} \in X_{y}} \frac{ z(x_{i}) }{|X_{y}|}.
\end{split}
\label{eq:ensemble_logit}
\end{align}
%Then, the predictive distribution from $X_{y}$ can be defined as:
Then, the predictive distribution for the ensemble created from data $X_{y}$ is as:
\begin{align}
\begin{split}
\bar{P}(c | X_{y} ; \theta, \tau) = \frac{\textrm{exp}( \bar{z}_{c}(X_{y}) / \tau ) }
{\sum^{C}_{i=1}\textrm{exp}( \bar{z}_{i}(X_{y})/\tau)},
\end{split}
\label{eq:softmax}
\end{align}
The loss function of XDED is defined as follows:
\begin{align}
\begin{split}
\mathcal{L}_{\textrm{XDED}}(X_{y}; \theta, \tau) = \sum_{x_{i} \in X_{y}}
\sum^{C}_{c=1}
D_{KL}(
\bar{P}(c|X_{y}; \hat{\theta}, \tau) ||
P(c|x_{i}; \theta, \tau)
),
\end{split}
\label{eq:xe_ed}
\end{align}
where $\hat{\theta}$ is a fixed copy of the parameter $\theta$. Following \cite{miyato2018virtual,yun2020regularizing}, we stop the gradient to be propagated through $\hat{\theta}$ to prevent the model from falling into some trivial solutions. To sum up, we set our objective function as
\begin{align}
\begin{split}
\min_{\theta}L_{\theta} =
\mathcal{L}_{\textrm{CE}}(X,Y; \theta) + \lambda \sum_{y \in \{Y\} } \mathcal{L}_{\textrm{XDED}}(X_{y}; \theta, \tau),
\end{split}
\label{eq:total_loss}
\end{align}
where $X$ is a batch of input images, $Y$ is a batch of corresponding class labels, $\mathcal{L}_{\textrm{CE}}$ denotes the vanilla cross-entropy loss, and $\lambda$ is a hyperparameter to balance $\mathcal{L}_{\textrm{CE}}$ and $\mathcal{L}_{\textrm{XDED}}$. $\lambda$ and $\tau$ are 5.0 and 4.0 throughout this paper.
\subsection{UniStyle: removing and unifying style bias} To further regularize the model to produce style-consistent predictions, we propose a de-stylization technique that is well-suited to domain generalization. As source domain styles are not expected to appear at test time, we propose UniStyle to prevent the model from being biased towards the domain-specific styles, which reduces the domain gap with the target domain.
More specifically, following existing methods based on style transfer~\cite{dumoulin2016learned,huang2017arbitrary,ulyanov2016instance}, we first represent a neural style as statistics of intermediate feature maps from the feature extractor. Formally, let $ F \in \mathbb{R}^{C \times H \times W} $ denote an intermediate feature map of an image. Then, a neural style of the image is represented as the combination of channel-wise mean $\mu(F) \in \mathbb{R}^{C}$ and standard deviation $ \sigma(F) \in \mathbb{R}^{C} $ of $F$ as:
\begin{equation}
\mu_{c}(F) = \frac{1}{HW} \sum^{H}_{h=1}\sum^{W}_{w=1} F_{c,h,w},
\end{equation}
and
\begin{equation}
\sigma_{c}(F) = \sqrt{ \frac{1}{HW} \sum^{H}_{h=1}\sum^{W}_{w=1} ( F_{c,h,w} - \mu_{c}(F) )^{2} },
\end{equation}
where $\mu(F) = [ \mu_{1}(F), \cdots, \mu_{C}(F) ] $ and $\sigma(F) = [ \sigma_{1}(F), \cdots, \sigma_{C}(F) ] $. Next, we simply standardize each feature to have constant channel-wise statistics, $ \mu_{W} $ and $\sigma_{W} $ as:
\begin{equation}
\textrm{UniStyle}(F) = \sigma_{W} \frac{F-\mu(F)}{\sigma(F)} + \mu_{W},
\end{equation}
where $\mu_{W} = \mathbf{0} $ and $ \sigma_{W}= \mathbf{1} $ (\ie, zero-mean standardization).
% Although UniStyle is a special case of InstanceNorm (IN)~\cite{ulyanov2016instance}, they have clear differences in that UniStyle aims to remove instance-specific information to reduce domain gap without any learnable parameter while IN learns channel-wise scaling and bias parameters for style transfer\footnote{See the supplementary material for further comparative analysis.}.
Technically, UniStyle is a special case of InstanceNorm (IN)~\cite{ulyanov2016instance}. Nevertheless, we remark that UniStyle aims to remove domain-specific information without any learnable parameters to reduce the domain gap while IN learns channel-wise scaling and bias parameters for style transfer.
%unlike IN which learns channel-wise scaling and bias parameters for style transfer
%Unistyle is a specific case that does not learn any learnable parameter of InstanceNorm (IN)~\cite{ulyanov2016instance}. Unlike IN, which learns channel-wise scaling and bias parameters for style transfer, it only removes instance-specific information to reduce the domain gap.\
Also, note that we empirically observed that UniStyle is effective when being applied at multiple early layers, which is aligned with recent studies~\cite{dumoulin2016learned,huang2017arbitrary} suggesting that the style information is usually captured at the early layers.\footnote{See the supplementary material for further analyses.}
%\vspace{1mm}
%\noindent \textbf{Comparison to neural style techniques.} Despite its similarity to InstanceNorm (IN)~\cite{ulyanov2016instance} (\ie, UniStyle is a special case of IN), they have clear differences in that UniStyle aims to remove instance-specific information to reduce domain gap without any learnable parameter while IN learns channel-wise scaling and bias parameters for style transfer. Next, in the context of domain generalization, MixStyle~\cite{zhou2021domain} has been proposed. However, they have exactly the opposite motivations. While MixStyle augments image styles by mixing the scaling and bias parameters of two instances, UniStyle standardizes the feature maps to remove domain-specific styles. Even, at test time, UniStyle works the same for test images as during training but MixStyle can not be applied. Further analysis on UniStyle is supplemented in the supplementary material.
\subsection{Analysis of Our Method}
In this section, we analyze the effectiveness of XDED, especially through the link to the theoretical result and the supporting empirical evidences. We first begin with a theorem related to domain adaptation~\cite{ben2010theory,ben2007analysis}, which shows that the expected risk on the target domain is bounded by that on the source domain and the divergence between these domains.
To find a model parameter $\theta \in \Theta$ for domain generalization, Cha \etal~\cite{cha2021swad} considered a robust empirical loss:
\begin{align}
\begin{split}
\hat{\varepsilon}^{\gamma}_{S}(\theta) := \max_{|| \Delta || \leq \gamma} \hat{\varepsilon}_{S}(\theta+\Delta)
\end{split}
\label{eq:robust_risk}
\end{align}
where $\hat{\varepsilon}_{S}(\theta)$ is an empirical risk over source domains $S$ and $\gamma$ is a radius which defines neighbor parameters of $\theta$. Then, Cha \etal~\cite{cha2021swad} proved that finding flat minima reduces the domain gap through the theorem below:
\begin{theorem}
Consider a set of N covers $ \{ \Theta_{k} \}^{N}_{k=1} $ such that the hypothesis space $ \Theta \subset \cup^{N}_{k}\Theta_{k}$ where $ diam(\Theta) := \sup_{\theta,\theta' \in \Theta} || \theta - \theta' ||_{2}, N:= \lceil (diam(\Theta)/\gamma)^{d} \rceil $ and $d$ is dimension of $\Theta$. Let $v_{k}$ be a VC dimension of each $\Theta_{k}$. Then, for any $\theta \in \Theta$, the following bound holds with probability at least $1-\delta$,
\begin{align}
\varepsilon_{T}(\theta) < & \hat{\varepsilon}^{\gamma}_{S}(\theta) + \frac{1}{2I} \sum^{I}_{i=1} \textrm{Div}(S_{i},T) + \max_{k\in[1,N]} \sqrt{ \frac{v_{k}\ln{(m/v_{k})} + \ln{(N/\delta)}}{m}},
\label{eq:swad}
\end{align}
where $m=nI$ is the number of training samples and $ \textrm{Div}(S_{i},T) $ is the divergence between the source domain $S_{i}$ and the target domain $T$.
\label{theorem:swad}
\end{theorem}
We remark that, in Eq.~(\ref{eq:swad}), the test loss $\varepsilon_{T}(\theta)$ is bounded by three terms: (1) the robust empirical loss $\hat{\varepsilon}^{\gamma}_{S}(\theta)$, (2) the divergence $\textrm{Div}(S_{i}, T)$, and (3) a confidence bound depending on the radius $\gamma$ and the number of training samples $m$. In the rest of this section, according to the above theorem, we provide a theoretical interpretation that our method enhances the generalization ability by lowering both $\hat{\varepsilon}^{\gamma}_{S}(\theta)$ and $\textrm{Div}(S_{i}, T)$ with the empirical evidences.
\input{table/comparison_entropy}
\begin{figure*}[t]
\centering
%\includegraphics[width=1.0\textwidth]{figure/fig2_2.pdf}
\includegraphics[width=0.85\textwidth]{CameraReady/figure/cr_fig2_4_ver2.pdf}
\caption{
Train/Test losses versus the weight perturbation while varying the standard deviation of the added Gaussian noise. Note that the results are produced with the target domain (Art of PACS) and the rest source domains, and the loss values are log-scaled.
%\textbf{Left}: Train/Test losses versus the weight perturbation while varying the standard deviation of the added Gaussian noise. Note that the loss values are log-scaled, \textbf{Right}: The divergence ($\mathcal{A}$-distance) between the source domains and the target domain.
}
\label{fig:a_dist}
\end{figure*}
\begin{figure*}[t]
\centering
\includegraphics[width=0.9\textwidth]{CameraReady/figure/cr_fig3_ver2.pdf}
% {supple/supple_figure/supple_a_dist_2.pdf}
\caption{
Comparison to existing methods promoting flat minima. Each model is evaluated on Cartoon of PACS after being trained on the rest source domains. \textbf{Left}: The divergence ($\mathcal{A}$-distance) between the source domains and the target domain, \textbf{Right}: Generalization performance on the target domain.
}
\label{fig:supple_a_dist}
\end{figure*}
\noindent \textbf{Promoting flat minima.}
We remark that XDED is motivated by recent entropy regularization methods~\cite{cha2020cpr,zhang2019your,zhang2018deep} in pursuit of flat minima. It has been empirically demonstrated that these methods promote flat minima by inducing higher posterior entropy. It can be interpreted as relaxing the training procedure to learn richer information encoded in soft labels, which helps the model converge to flat minima more than forcing the model to completely fit one-hot labels. In this context, we also demonstrate that XDED clearly induces higher entropy as shown in Table~\ref{tab:supple_comparison_entropy}. Considering that XDED is motivated by the observation that different domains manifest different inter-class relations due to the domain gap (Fig.~\ref{fig:teaser}), this is natural since our ensembles would integrate meaningful inter-class relations from multiple domains and the model learned with them would be led towards high entropy.
Next, to investigate whether the model learned with XDED converges to flat minima indeed, we quantify the flatness of the local minima where the model converged by measuring the increase of loss values between $\theta$ and its neighborhoods, assuming that the model converged in flat minima would have smaller increases. Following \cite{cha2021swad,cha2020cpr,zhang2019your,zhang2018deep}, we measure the losses of the learned models before and after adding Gaussian noises to model parameters while varying the standard deviation of the noise $\sigma{}_{\epsilon}$ (\ie, $\mathcal{L}_{\textrm{CE}}(X,Y; \theta + \epsilon )$ where $ \epsilon \sim N(0, \sigma{}_{\epsilon}) $) with 100 runs. As a result, XDED demonstrates its robustness against the weight perturbation with smaller loss increases as shown in Fig.~\ref{fig:a_dist}.
% \ready{It can be said that XDED penalizes the cases where only certain domains are well optimized and another one is relatively overlooked during training, which in turn leads to flat minima.}
\noindent \textbf{Domain-invariant feature learning.}
Here, we highlight that XDED also learns domain-invariant features via regularizing the consistency between the predictions from the data with the same label but from different domains and their ensemble. Thus, we compare XDED with existing methods promoting flat minima, which are dedicated to the flatness of local minima only. Specifically, to examine the effectiveness in reducing the divergence $\textrm{Div}(S_{i},T)$, we measure $\mathcal{A}$-distance~\cite{ben2010theory,kifer2004detecting}. Due to the computational intractability, we calculated an approximated one~\cite{long2015learning,nam2021reducing}~\footnote{It is defined as $ \hat{d}_{\mathcal{A}} = 2(1-2 \epsilon_{\textrm{svm}})$ where $\epsilon_{\textrm{svm}}$ is the generalization error of a SVM-based two-class classifier trained to distinguish between target and source domains.}
% the domain label of features from target and source domains.}.
As shown in Fig.~\ref{fig:supple_a_dist} (Left), we observe that the existing methods promoting flat minima fail to reduce the distance while XDED clearly lowers the distance and UniStyle further enhances the result. Naturally, that result is connected to the quantitative superiority of our framework over existing flat minima-promoting methods (Fig.~\ref{fig:supple_a_dist} (Right)).
% It is because ours is designed to learn domain-invariant features as well as promote flat minima, which differentiates our framework from existing approaches dedicated to domain-invariant features only (Table~\ref{tab:supple_comparison_entropy}) or flat minima only.
\section{Experiments}
\label{experiments}
%\input{table/pacs}
\subsection{Generalization in image classification}
\label{dg_cls}
\noindent \textbf{Multi-source domain generalization.}
Specifically, for a fair comparison, we follow the leave-one-domain-out protocol~\cite{li2017deeper} where we train a model on three domains and evaluate it on the remaining domain. For the benchmark datasets, we employ the PACS~\cite{li2017deeper} and OfficeHome~\cite{venkateswara2017deep} that are widely-used benchmarks for domain generalization in image classification. PACS contains 9,991 images of 7 classes over 4 domains: Art Painting, Cartoon, Photo, and Sketch. OfficeHome includes 15,500 images of 65 classes over 4 domains: Artistic, Clipart, Product, and Real. We use ResNet-18~\cite{resnet} as the backbone, and our UniStyle is applied to output feature maps of the first and second residual blocks for PACS and the first one only for OfficeHome.
%\vspace{1mm}
\noindent \textbf{Results.}
As summarized in Table.~\ref{tab:cr_pacs_officehome}, we observe that our method not only significantly enhances the vanilla but also outperforms the latest competing methods. In particular, our method outperforms the second-best method on Cartoon of PACS and Clipart of OfficeHome by about 4.0\% and 2.0\%, respectively. these results justify the superiority of our method, which is simple yet effective.
\input{table/CR_PACS_OfficeHome}
%\input{table/officeHome}
\noindent \textbf{Single-source domain generalization}
\label{single_src_dg}
Thanks to the simple design of our proposed method, which does not explicitly require domain labels, our method can be transparently incorporated with single-source domain generalization where we only have access to a single source domain during training. Therefore, to further evaluate the impact of our method on single-source domain generalization, our model is trained on each single domain of PACS and evaluated on the remaining target domains.
\noindent \textbf{Results.} As shown in Table.~\ref{tab:single_source}, our model, on average, significantly outperforms other baselines by 8.7\% in average accuracy. Besides, in all cases except for the case of $C \rightarrow S$, our model shows its superiority in performance. We believe this interesting result stems from the fact that our method is still able to help the model converge to flat minima and exploit the fine-grained relations between intra-domain samples even if only a single source domain is given.
\input{table/single_src_dg}
%Even though only a single domain is provided during training, we argue that this interesting result is attributed to the ability of our method to exploit the fine-grained relations between samples and the existence of inherent variation between intra-domain samples, which is also aligned with the observation of multiple sub-domains in a single domain~\cite{zhou2021domain}.}
\input{table/domainbed_criteria2}
%\input{table/domainbed_criteria2_2}
\noindent \textbf{DomainBed.}
We also conduct extensive experiments on the DomainBed~\cite{gulrajani2020search} which is a testbed for domain generalization to compare state-of-the-art methods across several benchmark datasets. The rationale behind the DomainBed is that the domain generalization performances are too much dependent on the hyperparameter tuning. For a fair comparison, we follow its standard protocols for training and evaluation.
%\vspace{1mm}
\noindent \textbf{Results.} As shown in Table.~\ref{tab:domainbed}, our method generally shows competitive performances and ranks second out of 15 methods on average accuracy. In particular, on CMNIST, our method substantially outperforms other competing methods. Since CMNIST is designed to simulate the domain shift by correlating the digit colors with the class labels, we conjecture that our improvement on CMNIST is attributed to the de-stylization effect of UniStyle, which would help the model decorrelate between the colors and labels.
\subsection{Generalization in person re-ID}
\label{dg_reid}
In this section, we further evaluate our method on person re-identification (re-ID), which is to match pedestrians across non-overlapping camera views.
% In this section, our method is further evaluated on person re-identification (re-ID), which is the task of matching pedestrians across non-overlapping camera views.
%\vspace{1mm}
\noindent \textbf{Experimental setup.}
Here, we address domain generalization for person re-ID, where the test data is collected from cameras of the unseen dataset rather than from those of the training dataset. Specifically, the model trained to match people in the source dataset is evaluated by how well it matches pedestrian data of the unseen test set, which are disjoint from the source dataset. For datasets, we adopt two widely-used benchmarks: Market1501 (Market)~\cite{zheng2015scalable} and DukeMTMC-reID (Duke)~\cite{ristani2016performance,zheng2017unlabeled}. We use 32,668 images of 1,501 identities collected from 6 cameras and 36,411 images of 1,812 identities from 8 cameras for Market1501 and Duke, respectively. As for performance measures, we adopt mean average precision (mAP) and Recall@K (R@K). Following the prior work~\cite{zhou2021domain}, we adopt ResNet-50~\cite{resnet} as a backbone architecture. In these experiments, we apply UniStyle to the 1st, 2nd, and 3rd residual blocks of a model.
\input{table/person_reid}
%\vspace{1mm}
\noindent \textbf{Comparison to other regularization methods.}
As shown in Table.~\ref{tab:person_reid}, our method substantially outperforms other methods in mAP and Recall@1. Although RandomErase and Dropblock are effective for learning discriminative features, they fail to improve performance when encountering unseen domain data. Furthermore, by exploiting inter-class relations provided by different cameras, our method shows its superiority over MixStyle and StyleNeophile which are designed for domain generalization but utilizes one-hot labels only.
\begin{table}[!t]
\centering
\caption{
mIoU (\%) results on the cross-dataset semantic segmentation. GTA5 is for training, and Cityscapes, SYNTHIA, BDD, and Mapillary are test sets.
% ResNet-50 is used as a backbone network.}
}
\fontsize{8}{10}\selectfont
\begin{tabularx}{0.9 \textwidth}{
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
\hline
\multicolumn{1}{l|}{Methods (GTA5)} & Cityscapes & BDD & Mapillary & SYNTHIA \\
\hline
\multicolumn{1}{l|}{DeepLabV3+~\cite{chen2018encoder}} & 28.9 & 25.1 & 28.1 & 26.2 \\
\multicolumn{1}{l|}{SW~\cite{pan2019switchable}} & 29.9 & 27.4 & 29.7 & 27.6 \\
\multicolumn{1}{l|}{DRPC~\cite{yue2019domain}} & \underline{37.4} & 32.1 & 34.1 & \underline{28.0} \\
%\multicolumn{1}{l|}{ItermNorm} & 31.8 & 27.0 & 32.7 & 33.8 \\
\multicolumn{1}{l|}{RobustNet~\cite{choi2021robustnet}} & 36.5 & \textbf{35.2} & \textbf{40.3} & \textbf{28.3} \\
\multicolumn{1}{l|}{\ccol Ours} & \ccol \textbf{39.2} & \ccol \underline{32.4} & \ccol \underline{37.1} & \ccol \underline{28.0} \\
\hline
\end{tabularx}
\label{tab:dg_segmentation}
%\vspace{-4mm}
\end{table}
\subsection{Generalization in semantic segmentation}
\label{dg_segmentation}
%\vspace{1mm}
\noindent \textbf{Experimental setup.} Lastly, to investigate whether our method can be extended to the dense prediction task, evaluation on semantic segmentation is addressed here. Following the mainstream protocol, we train models on a synthetic dataset and evaluate them on several datasets which mainly belong to real-world. Specifically, we adopt GTA5~\cite{richter2016playing} as a source dataset which consists of 24,966 images. For target datasets, Cityscapes~\cite{cityscapes}, BDD~\cite{yu2018bdd100k}, and Mapillary~\cite{neuhold2017mapillary} are real-world datasets whose image sizes are 5,000, 10,000, and 25,000, respectively. Lastly, SYNTHIA~\cite{ros2016synthia} has 9,400 images. Note that ResNet-50 is used as the backbone and the common 19 classes are used across all datasets.
%\vspace{1mm}
\noindent \textbf{Results.}
We remark that XDED constructs an ensemble by simply averaging all the logits from the pixels whose gt is the same in a mini-batch. As shown in Table~\ref{tab:dg_segmentation}, ours outperforms the competing methods overall, even if those are dedicated to this task only. We show that our method can be extended to the pixel-wise classification with little modification on XDED. Also, the results support our claim that our method is simple yet effective in a wide range of tasks.
\subsection{In-depth Analysis}
\label{sec:experiment_ablation}
\input{table/ablation_components}
\noindent \textbf{Ablation study.} To investigate the impact of each component in our method, we conduct an ablation study which is summarized in Table~\ref{tab:ablation_components}. The result reveals that two components are complementary and consistently help the model improve the generalization ability. For image classification, XDED contributes most to the performance, and UniStyle boosts the effect of XDED. On both domains, XDED uniformly improves the vanilla method by about 6\%, whereas UniStyle shows different degrees of improvement. It is because the image style discrepancy between domains in OfficeHome is less severe than that in PACS. Interestingly, for the task of person re-ID, UniStyle reveals more impact than does XDED. Due to the inherent characteristics of the task itself, the effect of XDED on collecting meaningful knowledge of the same pedestrian from different cameras may become less significant.
\begin{figure}[t]
\centering
\includegraphics[width=0.92\textwidth]{CameraReady/figure/cr_fig4.pdf}
\caption{Visualization results of the loss landscapes incorporating the vanilla method and XDED on the PACS dataset. Note that each loss landscape is visualized on the data of source domains, not the data of the marked target domain. Blue and red surfaces are from the vanilla method and XDED, respectively.
}
%\vspace{-2mm}
\label{fig:loss_surface}
\end{figure}
\noindent \textbf{Loss surface visualization.}
To further illustrate how XDED leads to flat minima in the loss landscapes, we provide qualitative results that visualize the loss landscapes. Following \cite{cha2020cpr}, we plot the loss landscapes on data of source domains per each case by perturbing the model parameters across the first and second Hessian eigenvectors which are provided by PyHessian~\cite{yao2020pyhessian} which is a framework for Hessian-based analysis of neural networks. As shown in Fig.~\ref{fig:loss_surface}, we observe that the loss landscapes incorporating XDED clearly become flatter than those incorporating the vanilla method for all cases. We argue that these qualitative results also consistently support that XDED promotes flat minima.
%\input{table/comparison_flat_minima}
%\noindent \textbf{Comparison to existing flat minima-promoting methods.} To investigate the ability of XDED to learn domain-invariant features, we compare XDED with existing flat minima-promoting methods in terms of the robustness on the unseen domain. As shown in Table.~\ref{tab:comparison_flat_minima}, while all methods show consistent superiority over the vanilla method thanks to the advantage of promoting flat minima, XDED substantially outperforms other baseline methods with the largest margin over the vanilla method. We argue that this result is attributed to the ability of XDED to learn domain-invariant features as well as promote flat minima.
\input{table/comparison_adversarial_attack}
\noindent \textbf{Robustness to adversarial examples.}
Recent studies have demonstrated that convergence on flat minima strengthens the adversarial robustness~\cite{wu2020adversarial,stutz2021relating}. To revalidate that our method promotes flat minima, we evaluate the adversarial robustness of learned models. Specifically, we trained models on source domains and added adversarial perturbations on images of the unseen target domain by using existing adversarial attack methods: FGSM~\cite{goodfellow2014explaining} and PGD~\cite{madry2018towards}. Table~\ref{tab:comparison_adversarial_attack} shows that our method outperforms other regularization methods in terms of robustness against both unseen data and adversarial attacks. Considering that adversarial attacks are made to maximize the loss value, we argue that our superiority in adversarial robustness is also attributed to the capability of promoting flat minima as desired, even though our method has no direct connection to adversarial training.
\input{table/comparison_corruption}
\noindent \textbf{Results on corruption benchmarks.} We further measure the resilience of learned models to image corruptions. Following the protocol provided by \cite{hendrycks2019benchmarking}, we trained models on the original training dataset, and evaluated them on the test dataset constructed by corrupting the original test dataset through predefined corruption types. Table.~\ref{tab:comparison_corruption} shows that our method outperforms all regularization methods except AugMix~\cite{hendrycks2020augmix}. Considering AugMix is a state of the art that is dedicated to corruption robustness while ours is not, we argue that our method still shows its significant robustness against image corruptions.
\section{Conclusion}
We have presented a simple yet effective framework for domain generalization. XDED first generates an ensemble of output distributions for the data with the same label but from different domains, and then penalizes each output distribution for the mismatch with the ensemble in the form of self-knowledge distillation. With this approach, our model can learn domain-invariant features and also easily converges to flat minima. Besides, the proposed UniStyle suppresses domain-specific style bias to boost the effect of XDED and encourage style-consistent predictions. Furthermore, we empirically validate the generalization ability of the proposed method from the perspective of flat minima and reduced divergence between source and target. Through extensive experimental results, we demonstrate the superiority of the proposed framework. %\kmc{REMOVE? For a limitation, we remark that our method is not fully free from domain labels in multi-source domain generalization since useful batch sampling for models would be possible by using domain labels. Nevertheless, we respectfully remark that our method is still executable without them and also achieves significant performance in single-source domain generalization where domain label is non-existent.}
% \section*{Acknowledgements}
% \ready{This work was supported by the NRF grant, the IITP grant, and R\&D program for Advanced Integrated-intelligence for IDentification, funded by Ministry of Science and ICT, Korea (No.2019-0-01906 Artificial Intelligence Graduate School Program--POSTECH, NRF-2021R 1A2C3012728--50\%, NRF-2018M3E3A1057306--50\%).
% }
\vfill
{\small
\noindent \textbf{Acknowledgement.}
% \section*{Acknowledgements}
This work was supported by
the NRF grant and % 중견
the IITP grant % AI대학원, 자기주도, 시각상식
funded by Ministry of Science and ICT, Korea
(NRF-2021R1A2C3012728, % 중견
IITP-2019-0-01906, % AI대학원
IITP-2022-0-00926, % 자기주도
IITP-2022-0-00290). % 시각상식
}
% updated April 2002 by Antje Endemann
% Based on CVPR 07 and LNCS, with modifications by DAF, AZ and elle, 2008 and AA, 2010, and CC, 2011; TT, 2014; AAS, 2016; AAS, 2020; TH, 2022
\documentclass[runningheads]{llncs}
\usepackage{graphicx}
% DO NOT USE \usepackage{times}, it will be removed by typesetters
%\usepackage{times}
\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{color}
\usepackage{orcidlink}
% kyungmoon
\usepackage{soul}
\usepackage{makecell}
\usepackage{tabularx}
\usepackage{xcolor, colortbl}
\usepackage{enumitem}
\usepackage{subcaption}
\usepackage{adjustbox}
\usepackage{diagbox}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{pifont}
%\theoremstyle{theorem}
%\newtheorem{theorem}{Theorem}
\def\ie{\emph{i.e.}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al.}}
\def\wrt{\emph{w.r.t.}}
\definecolor{grey}{rgb}{0.9, 0.9, 0.9}
\newcommand{\ccol}{\cellcolor{grey}}
\definecolor{brown}{rgb}{0.65, 0.16, 0.16}
\newcommand{\ready}[1]{{\color{brown}{#1}}}
\newcommand{\sy}[1]{{\color{purple}{#1}}}
% The "axessiblity" package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
% \usepackage{ruler}
% \usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\begin{document}
% \renewcommand\thelinenumber{\color[rgb]{0.2,0.5,0.8}\normalfont\sffamily\scriptsize\arabic{linenumber}\color[rgb]{0,0,0}}
% \renewcommand\makeLineNumber {\hss\thelinenumber\ \hspace{6mm} \rlap{\hskip\textwidth\ \hspace{6.5mm}\thelinenumber}}
% \linenumbers
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{4593} % Insert your submission number here
\title{Cross-Domain Ensemble Distillation \\for Domain Generalization} % Replace with your title
% INITIAL SUBMISSION
\begin{comment}
\titlerunning{ECCV-22 submission ID \ECCVSubNumber}
\authorrunning{ECCV-22 submission ID \ECCVSubNumber}
\author{Anonymous ECCV submission}
\institute{Paper ID \ECCVSubNumber}
\end{comment}
%******************
% CAMERA READY SUBMISSION
%\begin{comment}
\titlerunning{Cross-Domain Ensemble Distillation for Domain Generalization}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%
\author{Kyungmoon Lee\thanks{This work was done when Kyungmoon Lee was at POSTECH.}\inst{1} \and
Sungyeon Kim\inst{2} \and
Suha Kwak\inst{2}}
%
\authorrunning{Kyungmoon Lee, Sungyeon Kim, Suha Kwak.}
% First names are abbreviated in the running head.
% If there are more than two authors, 'et al.' is used.
%
\institute{NALBI Inc, Seoul, Korea \and POSTECH, Pohang, Korea \\
\email{kyungmoon@nalbi.ai, \{sungyeon.kim, suha.kwak\}@postech.ac.kr }\\
{\tt\small
\url{http://cvlab.postech.ac.kr/research/XDED}
}
}
%\end{comment}
%******************
\maketitle
\input{CameraReady/CR_0_abstract}
\input{CameraReady/CR_1_introduction}
\input{CameraReady/CR_2_relatedwork}
\input{CameraReady/CR_3_method}
\input{CameraReady/CR_4_experiments}
\input{CameraReady/CR_5_conclusion}
\clearpage
% ---- Bibliography ----
%
% BibTeX users should specify bibliography style 'splncs04'.
% References will then be sorted and formatted in the correct style.
%
\bibliographystyle{splncs04}
\bibliography{cvlab_kwak}
\end{document}
This supplementary material presents additional experimental results, further analyses, and implementation details, all of which are omitted from the main paper due to the space limit. In Sec.~\ref{sec:cr_supple_exp}, additional experimental results of our method are provided. Sec.~\ref{sec:cr_supple_xded} and Sec.~\ref{sec:cr_supple_unistyle} present further analyses of the proposed components, respectively. Lastly, implementation details are in Sec~\ref{sec:cr_supple_implementation}.
% the typo correction in our main paper, additional analysis results of the proposed components and other experimental materials, all of which are omitted from the main paper due to the space limit. First, Sec.~\ref{sec:supple_xded} and Sec.~\ref{sec:supple_unistyle} present additional analysis results of XDED and UniStyle, respectively. Lastly, Sec.~\ref{sec:supple_other_materials} includes implementation details.
%First, Sec.~\ref{sec:supple_type_correction} corrects our typo in the main paper to avoid misunderstanding. Sec.~\ref{sec:supple_xded} and Sec.~\ref{sec:supple_unistyle} present additional analysis results of XDED and UniStyle, respectively. Sec.~\ref{sec:supple_other_materials} includes implementation details and and instructions for our source code.
% Finally, we attach the partial source codes implemented using Pytorch. The full implementation will be released on publication.
\section{Additional experimental results}
\label{sec:cr_supple_exp}
%\subsection{Results on corrruption benchmarks}
%We further measure the resilience of learned models to unseen corruptions. Following the protocol provided by \cite{hendrycks2019benchmarking}, we trained models on the original training dataset, and evaluated them on the test dataset constructed by corrupting the original test dataset through predefined corruption types. Table.~\ref{tab:comparison_corruption} shows that our method outperforms all regularization methods except AugMix~\cite{hendrycks2020augmix}. Considering AugMix is a state of the art that is dedicated to corruption robustness while ours is not, we argue that our method still shows its significant robustness against corruption.
\subsection{Comparison to existing self-KD methods.}
To further assess the impact of the ensemble distillation, we also compare XDED with state-of-the-art self-knowledge distillation (self-KD) methods in spite of sharing a similar methodology. Furthermore considering that our method does not explicitly depend on domain labels, we investigate the impact of XDED on the standard benchmark datasets for the image recognition task, which assume the independent and identically distributed (i.i.d.) condition. As shown in Table~\ref{tab:comparison_iid}, XDED outperforms not only the vanilla method but also other self-KD methods for all datasets. Considering that XDED is not dedicated to the case under the i.i.d. condition whereas other methods are dedicated to the case under the i.i.d. condition, these experimental results support our view that XDED can provide more meaningful supervisory signals thanks to the use of ensemble which encodes the complementary knowledge from the data with the same label.
%\input{table/comparison_corruption}
\subsection{Results on input deformations}
To further assess the quality of the robustness to unseen domains, we measure the changes in test accuracy as the domain gap becomes larger. We simulate larger domain gaps by increasing the degree of input deformations. Specifically, we applied multiple augmentations defined in RandAugment~\cite{cubuk2020randaugment} to images of the unseen target domain (\ie, Cartoon of PACS), and gradually increased the number of deformations\footnote{For a fair comparison, we use the same operations for each setting.}. Similar to the result of table 9 in our main paper, Fig.~\ref{fig:supple_deformation} shows that our method outperforms baseline methods against pixel-level deformations on unseen domains.
\input{table/comparison_iid}
\begin{figure*}[t]
\centering
\includegraphics[width=1.0\textwidth]{supple/supple_figure/deformation_sero.pdf}
\caption{Test accuracy (\%) on target domains with input deformations. All models are trained under the multi-source domain generalization setting.
}
\label{fig:supple_deformation}
\end{figure*}
\subsection{Learning acceleration}
\label{sec:supple_convergence}
Our method also enables faster convergence of the learning process. In generalization on PACS, our effectiveness of learning acceleration is demonstrated. As shown in Fig.~\ref{fig:supple_convergence}, our method reaches highest performance with less epochs (\ie, 24 epochs) compared to both vanilla method (40 epochs) and MixStyle~\cite{zhou2021domain} (39 epochs). Although the vanilla method requires many iterations to address the domain gaps between source domains, our method accelerates the learning process via promoting flat minima which produce lower errors in not only source domains but also the target domain. Since MixStyle~\cite{zhou2021domain}, as an augmentation method, synthesizes novel styles via mixing statistics at the feature level, it is inherently limited to requiring as many epochs as possible to address many augmented styles for its best performance. Also, in the context of the actual training time in total, compared to MixStyle, we remark that our method significantly reduces the training complexity. In detail, our method increases training time per iteration by 5\% but requires only 60\% of training epochs and achieves about 1.6 times more performance boost averaged over every target domain of the PACS dataset. The results are summarized in Table~\ref{tab:supple_convergence}.
\begin{table}[!t]
\centering
\caption{ Performance boosts over training complexity.
}
%\fontsize{9}{11}\selectfont
\begin{tabularx}{0.7\textwidth}{
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X}
\hline
%\multicolumn{1}{l|}{\multirow{2}{*}[0mm]{Methods}} & Classification & Person re-ID \\ \cline{2-3}
\multicolumn{1}{l|}{Methods} & Millisec / iter & Accuracy boost (\%) \\
\hline
%\multicolumn{1}{l|}{IN~\cite{ulyanov2016instance}} & \checkmark & \checkmark & 74.0\\
\multicolumn{1}{l|}{ResNet-18} & 34 & -\\
\multicolumn{1}{l|}{MixStyle~\cite{zhou2021domain}} & 35 & 4.2 \\
\multicolumn{1}{l|}{\ccol Ours} & \ccol 37 & \ccol 6.9\\
\hline
\end{tabularx}
\label{tab:supple_convergence}
% \vspace{-4mm}
\end{table}
\begin{figure}[!t]
\centering
\includegraphics[width=0.8\columnwidth]{supple/supple_figure/learning_acceleration.pdf}
\caption{Multi-source domain generalization accuracy (\%) versus the number of epochs on the unseen target domain, Art Painting of PACS. Note that all models were trained on the rest three domains of PACS (\ie, Cartoon, Sketch, and Photo) and all methods were trained with the same hyperparameter setting (\eg, batch size and learning rate) on a single RTX 2080 GPU.
}
\label{fig:supple_convergence}
\end{figure}
\begin{figure}[t]
\centering
% \includegraphics[width=1.\textwidth]{figure/fig1.pdf}
\includegraphics[width=0.9\textwidth]{supple/supple_figure/sketch_ablation_hyperparam.pdf}
\caption{Multi-source domain generalization accuracy (\%) versus hyperparameters $\lambda$ and $\tau$ on Sketch of the PACS dataset.
}
\label{fig:supple_hyperparam}
\end{figure}
\subsection{Impact of hyperparameters}
We empirically investigated the effect of the two hyperparameters of XDED: $\lambda$ and $\tau$. We examine accuracy of multi-source domain generalization on Sketch of the PACS dataset by varying the values of $\lambda \in \{ 0.5, 1.0, 2.5, 5.0, 10.0 \} $ and $\tau \in \{ 1.0, 2.0, 4.0, 6.0, 8.0 \} $. The results are summarized in Fig.~\ref{fig:supple_hyperparam}. First, the performance boost is the highest when $ \tau $ is increased from 1.0 to 2.0. However, if $\tau$ exceeds 2.0, the performance boost is somewhat reduced. Next, $\lambda$ also affects the performance. The performance boost commonly increases with the value of $\lambda$. However, the degree of boost is also reduced when $\tau$ is 1.0. To sum it up, the accuracy of our method is generally high and stable when $\lambda$ and $\tau$ are greater than 5.0 and 2.0, and we argue that this result supports our view that our method does not strongly depend on the specific hyperparameter setting.
\section{Additional analysis results on XDED}
\label{sec:cr_supple_xded}
In this section, we provide further analyses of XDED. Also, we provide additional empirical results on the flatness of local minima (Fig.~\ref{fig:supple_weight_perturbation}).
\begin{table}[!t]
\centering
\caption{
Generalization results on the target domain (Clipart of OfficeHome) and required overhead during training.
}
%\fontsize{9}{11}\selectfont
\begin{tabularx}{0.7 \textwidth}{
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
\hline
\multicolumn{1}{l|}{Methods} & Vanilla & SAM~\cite{foret2021sharpness} & \ccol XDED \\
\hline
\multicolumn{1}{l|}{Accuracy (\%) ($\uparrow$)} & 49.4 & \underline{51.9} & \ccol \textbf{55.2} \\
% sec/iter
\multicolumn{1}{l|}{Time / iter (ms) ($\downarrow$)} & \textbf{32} & 72 & \ccol \underline{33} \\
\hline
\end{tabularx}
\label{tab:supple_flat_minima_cost}
\end{table}
\subsection{Convergence into flat minima at a low cost}
Since there are several ways for convergence into flat minima, we believe that the required overhead to achieve that goal should be considered as another criterion of valuation for a practical usage. Therefore, according to this point of view, we compare XDED with SAM~\cite{foret2021sharpness} which has been proposed to optimize loss sharpness more directly. As shown in Table.~\ref{tab:supple_flat_minima_cost}, XDED shows superior robustness on the unseen domain over SAM. Furthermore, XDED requires negligible overhead while SAM requires the training time more than 2 times longer than the vanilla method and XDED. This is because SAM must additionally compute gradient approximation for the loss sharpness to be optimized, which exposes a trade-off between training complexity and the optimization of the loss sharpness.
On the other hand, SAM's performance is inferior to that of XDED, since it directly promotes a flat minima but cannot learn domain-invariant features, unlike our method. To sum it up, we argue that XDED offers more appealing option for domain generalization in the context of both performance improvement and required overhead.
\subsection{Ensemble matters for generalization ability}
\label{sec:supple_standard_benchmarks}
\begin{figure}[!t]
\centering
\includegraphics[width=0.82\textwidth]{supple/supple_figure/supple_qualitative1_sero.pdf}
\caption{Prediction scores for misclassified samples from the vanilla method (\ie, Cross-entropy). \textbf{Top}: Well-classified results from XDED, \textbf{Bottom}: Misclassified results from XDED.
}
\label{fig:supple_qualitative}
\end{figure}
To examine how XDED affects the predictions of the learned model in detail, we investigate the softmax scores from the model which is trained on the source domains (\ie, Cartoon, Sketch and Photo of PACS) and evaluated on the target domain (\ie, Art Painting). Specifically, for the misclassified samples from the vanilla method, we assess the differences in the softmax scores from the vanilla method and XDED. First, as shown in Fig.~\ref{fig:supple_qualitative} (Top), XDED can lead to accurate predictions on the samples which the vanilla method confidently misclassifies. For the giraffe sample, the model learned with XDED not only predicts the ground truth successfully but also puts relatively high probability on the similar class (\ie, Dog) while lowering the probabilities of the irrelevant classes (\ie, Guitar and House). Similarly, even in its failure cases, XDED encourages the model to predict with consideration of inter-class relations (\ie, increasing scores for similar classes while suppressing those for irrelevant classes) as shown in Fig.~\ref{fig:supple_qualitative} (Bottom). By exploiting the ensemble incorporating information of multiple domains, XDED encourages the model to consider richer inter-class relations which may be exploited on the target domain and lead to meaningful predictions even in failure cases. Note that UniStyle is excluded from the results discussed above.
\subsection{Comparison with other ensemble methods.}
\label{sec:supple_comparison_ensemble}
Lastly, we compare XDED with other domain generalization methods which adopt ensemble techniques. The comparison analysis is summarized in Table~\ref{tab:comparison_ensemble}. We remark that the proposed method does not require any additional module and is also free from domain labels (Note that our method, thus, seamlessly extends to the single-source domain generalization setting, which is discussed in Sec.4.1 of our main paper.). Nevertheless, our method clearly outperforms other methods in the context of generalization ability. In contrast, DAEL~\cite{zhou2021dael} and DSON~\cite{seo2020learning} show inferior results even though they explicitly exploit domain labels and multiple domain-specific modules (\ie, classifiers and BN layers~\cite{Batchnorm}, respectively.). DAEL is the most similar to our method but clearly different in the sense that DAEL exploits the predictions of an expert as supervisory signals for non-experts on the same instance, but our method constructs the ensemble using multiple instances and exploits it as supervision to model itself. In other words, by learning domain-invariant information through the ensemble of meaningful knowledge contained in different domains, better performance could be achieved even with limited resources.
\input{table/comparison_ensemble}
\begin{figure*}%[t]
\centering
\includegraphics[width=1.0\textwidth]{supple/supple_figure/weight_perturb.pdf}
\caption{Train/Test losses versus the weight perturbation while varying the standard deviation of the added Gaussian noise. Note that train losses are calculated over the remaining three source domains for each target domain and the loss values are log-scaled.
}
\label{fig:supple_weight_perturbation}
\end{figure*}
\section{Additional analysis results on UniStyle}
\label{sec:cr_supple_unistyle}
\input{table/ablation_where_to_whiten}
\subsection{Where to apply UniStyle}
We remark that UniStyle can be applied after arbitrary intermediate layers of the backbone (\ie, ResNet~\cite{resnet}) as a plug-and-play module. Therefore, to investigate the impact of where UniStyle is applied, we evaluate the generalization performance in image classification and person re-ID while varying the locations of where the operation is applied. For a baseline, MixStyle~\cite{zhou2021domain} is adopted and compared with our UniStyle since they share the commonality of being applied to multiple intermediate feature maps.
For brevity, let \textsc{res\#} denote the indexes of residual blocks where the specified operation is applied (\eg, \textsc{res12} means the operation is applied after both the first and second residual blocks). As shown in Table.~\ref{tab:ablation_where_to_whiten}, we observe that UniStyle and MixStyle have a similar trend in both tasks. First, since early layers are known to capture low-level features such as texture or edge information, it is pertinent for both UniStyle and MixStyle to be applied after early layers to remove style bias and synthesize novel styles, respectively. However, UniStyle achieves the best performances on both tasks, which indicates the importance of removing and unifying style bias rather than augmenting novel styles. Next, on the contrary, both operations lead critical performance drop when incorporated with the last layer, \textsc{res4}, since late layers are known to address semantic information (\ie, their statistics would be highly correlated with target labels). In detail, MixStyle perturbs the statistics by interpolating those of two different instances that may have different labels, whereas UniStyle normalizes the feature map of all samples regardless of their labels, resulting in a more critical performance drop. Note that XDED is excluded in results of Table.~\ref{tab:ablation_where_to_whiten}.
\subsection{Comparison with other methods}
Even though UniStyle is not the first that addresses feature-level statistics (\ie, style information), we take notice of its appealing trade-off between performance boost and training complexity and compare it with other related methods. The results are summarized in Table~\ref{tab:supple_comparison_unistyle}. First, UniStyle outperforms SagNet~\cite{nam2021reducing} and MixStyle~\cite{zhou2021domain}, all of which are dedicated to domain generalization. This indicates that directly removing and unifying the style bias is more effective for generalization ability than augmenting feature statistics or learning style-agnostic features.
Next, although BIN~\cite{nam2018batch} and SRM~\cite{lee2019srm} adaptively recalibrate feature statistics and achieve state-of-the-art performance in image recognition tasks, they show inferior results on unseen domains. Last but not least, Deep Whitening Transformation (DWT)~\cite{cho2019image} which is the most similar to UniStyle shows the second-best performance. DWT has been proposed to encourage the feature extractor to naturally encode the whitened feature by enforcing consistency regularization between the covariance matrix of the feature and identity matrix. Nevertheless, as a constraint that is too strong, DWT weakens feature discrimination and excessively throws away content information, resulting in performance that lags behind UniStyle.
\input{table/comparison_unistyle}
\section{Implementation details}
\label{sec:cr_supple_implementation}
For the task of domain generalization in image classification, we train the models using the sgd optimizer with the cosine learning decay~\cite{loshchilov2016sgdr} and initial learning rate of $10^{-3}$. They are learned for 100 epochs. For batch construction, we use the batch size of 64, and sample 16 instances per class for the proposed XDED. For the task of person re-ID, we also train the models using the sgd optimizer with initial learning rate of 0.05.
% updated April 2002 by Antje Endemann
% Based on CVPR 07 and LNCS, with modifications by DAF, AZ and elle, 2008 and AA, 2010, and CC, 2011; TT, 2014; AAS, 2016; AAS, 2020; TH, 2022
\documentclass[runningheads]{llncs}
\usepackage{graphicx}
% DO NOT USE \usepackage{times}, it will be removed by typesetters
%\usepackage{times}
\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{color}
\usepackage{epsfig}
\usepackage{lipsum}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{url}
\usepackage{xcolor, colortbl}
\usepackage{mathtools}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{enumitem}
\usepackage{bbm}
\usepackage{wrapfig}
\usepackage{setspace}
\usepackage{rotating}
\RequirePackage{fix-cm}
\usepackage{booktabs}
\usepackage{kotex}
% \documentclass{article}
\usepackage{pifont}
\usepackage{blindtext}
\usepackage{xr}
\usepackage[toc]{multitoc}
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
% \hypersetup{colorlinks=true,allcolors=[rgb]{0,1,0}}
% \hypersetup{colorlinks=true}
% \newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}p{#1}}
\newcolumntype{P}[1]{>{\centering\arraybackslash}p{#1}}
%\input{sec/9_custom_cmds}
% The “axessiblity” package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
% \usepackage{ruler}
% \usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\usepackage[capitalize]{cleveref}
\newcommand{\sftype}[1]{{\textsf{\small #1}}}
\newcommand{\sftypes}[1]{{\textsf{\tiny #1}}}
\newcommand{\expnum}[2]{{#1}\mathrm{e}{-#2}}
%\renewcommand{\Tbl}[1]{Table~\ref{tab:#1}}
%\end{comment}
%******************
% kyungmoon
\usepackage{soul}
\usepackage{makecell}
\usepackage{tabularx}
\usepackage{xcolor, colortbl}
\usepackage{enumitem}
\usepackage{subcaption}
\usepackage{adjustbox}
\usepackage{diagbox}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{pifont}
\usepackage{orcidlink}
\usepackage{pict2e,picture}
%\theoremstyle{theorem}
%\newtheorem{theorem}{Theorem}
\def\ie{\emph{i.e.}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al.}}
\def\wrt{\emph{w.r.t.}}
\definecolor{grey}{rgb}{0.9, 0.9, 0.9}
\newcommand{\ccol}{\cellcolor{grey}}
\definecolor{brown}{rgb}{0.65, 0.16, 0.16}
\newcommand{\ready}[1]{{\color{brown}{#1}}}
\newcommand{\sy}[1]{{\color{purple}{#1}}}
\usepackage{pict2e,picture}
\makeatletter
\DeclareRobustCommand{\shortarrow}[1][]{%
\check@mathfonts
\if\relax\detokenize{#1}\relax
\settowidth{\dimen@}{$\m@th\rightarrow$}%
\else
\setlength{\dimen@}{#1}%
\fi
\sbox\z@{\usefont{U}{lasy}{m}{n}\symbol{41}}%
\begin{picture}(\dimen@,\ht\z@)
\roundcap
\put(\dimexpr\dimen@-.7\wd\z@,0){\usebox\z@}
\put(0,\fontdimen22\textfont2){\line(1,0){\dimen@}}
\end{picture}%
}
\makeatother
% The "axessiblity" package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
%\usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\begin{document}
% \renewcommand\thelinenumber{\color[rgb]{0.2,0.5,0.8}\normalfont\sffamily\scriptsize\arabic{linenumber}\color[rgb]{0,0,0}}
% \renewcommand\makeLineNumber {\hss\thelinenumber\ \hspace{6mm} \rlap{\hskip\textwidth\ \hspace{6.5mm}\thelinenumber}}
% \linenumbers
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{4593} % Insert your submission number here
\title{Cross-Domain Ensemble Distillation \linebreak for Domain Generalization} % Replace with your title
% INITIAL SUBMISSION
\begin{comment}
\titlerunning{ECCV-22 submission ID \ECCVSubNumber}
\authorrunning{ECCV-22 submission ID \ECCVSubNumber}
\author{Anonymous ECCV submission}
\institute{Paper ID \ECCVSubNumber}
\end{comment}
%******************
% CAMERA READY SUBMISSION
%\begin{comment}
\titlerunning{Cross-Domain Ensemble Distillation for Domain Generalization}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%
\author{Kyungmoon Lee%
% \thanks{This work was done when Kyungmoon Lee was at POSTECH.}
\inst{1,2}\orcidlink{0000-0001-6597-3928} \qquad
Sungyeon Kim\inst{1}\orcidlink{0000-0002-6919-4822} \qquad
Suha Kwak\inst{1}\orcidlink{0000-0002-4567-9091}}%\orcidID{2222--3333-4444-5555}}
%
\authorrunning{Kyungmoon Lee, Sungyeon Kim, Suha Kwak}
\institute{$^{1}$POSTECH, Pohang, Korea \qquad $^{2}$NALBI Inc, Seoul, Korea \\
\email{kyungmoon@nalbi.ai, \{sungyeon.kim, suha.kwak\}@postech.ac.kr }\\
{\tt\small
\url{http://cvlab.postech.ac.kr/research/XDED/}
}
}
\maketitle
\input{CameraReady/CR_0_abstract}
\input{CameraReady/CR_1_introduction}
\input{CameraReady/CR_2_relatedwork}
\input{CameraReady/CR_3_method}
\input{CameraReady/CR_4_experiments}
\input{CameraReady/CR_5_conclusion}
%\clearpage
%{\small
\bibliographystyle{splncs04}
\bibliography{cvlab_kwak}
%}
\clearpage
\appendix
% \title{Cross-Domain Ensemble Distillation \linebreak for Domain Generalization\\{\it ---Supplementary Material---} }
\titlerunning{Cross-Domain Ensemble Distillation for Domain Generalization}
\author{Kyungmoon Lee
\inst{1,2}
Sungyeon Kim\inst{2}
Suha Kwak\inst{2}}
\authorrunning{Kyungmoon Lee, Sungyeon Kim, Suha Kwak}
\institute{$^{1}$NALBI Inc, Seoul, Korea \qquad $^{2}$POSTECH, Pohang, Korea \\
}
\end{document}
% updated April 2002 by Antje Endemann
% Based on CVPR 07 and LNCS, with modifications by DAF, AZ and elle, 2008 and AA, 2010, and CC, 2011; TT, 2014; AAS, 2016; AAS, 2020; TH, 2022
\documentclass[runningheads]{llncs}
\usepackage{graphicx}
% DO NOT USE \usepackage{times}, it will be removed by typesetters
%\usepackage{times}
\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{color}
\usepackage{epsfig}
\usepackage{lipsum}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{url}
\usepackage{xcolor, colortbl}
\usepackage{mathtools}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{enumitem}
\usepackage{bbm}
\usepackage{wrapfig}
\usepackage{setspace}
\usepackage{rotating}
\RequirePackage{fix-cm}
\usepackage{booktabs}
\usepackage{kotex}
% \documentclass{article}
\usepackage{pifont}
\usepackage{blindtext}
\usepackage{xr}
\usepackage[toc]{multitoc}
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
% \hypersetup{colorlinks=true,allcolors=[rgb]{0,1,0}}
% \hypersetup{colorlinks=true}
% \newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}p{#1}}
\newcolumntype{P}[1]{>{\centering\arraybackslash}p{#1}}
%\input{sec/9_custom_cmds}
% The “axessiblity” package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
% \usepackage{ruler}
% \usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\usepackage[capitalize]{cleveref}
\newcommand{\sftype}[1]{{\textsf{\small #1}}}
\newcommand{\sftypes}[1]{{\textsf{\tiny #1}}}
\newcommand{\expnum}[2]{{#1}\mathrm{e}{-#2}}
%\renewcommand{\Tbl}[1]{Table~\ref{tab:#1}}
%\end{comment}
%******************
% kyungmoon
\usepackage{soul}
\usepackage{makecell}
\usepackage{tabularx}
\usepackage{xcolor, colortbl}
\usepackage{enumitem}
\usepackage{subcaption}
\usepackage{adjustbox}
\usepackage{diagbox}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{pifont}
\usepackage{orcidlink}
\usepackage{pict2e,picture}
%\theoremstyle{theorem}
%\newtheorem{theorem}{Theorem}
\def\ie{\emph{i.e.}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al.}}
\def\wrt{\emph{w.r.t.}}
\definecolor{grey}{rgb}{0.9, 0.9, 0.9}
\newcommand{\ccol}{\cellcolor{grey}}
\definecolor{brown}{rgb}{0.65, 0.16, 0.16}
\newcommand{\ready}[1]{{\color{brown}{#1}}}
\newcommand{\sy}[1]{{\color{purple}{#1}}}
\usepackage{pict2e,picture}
\makeatletter
\DeclareRobustCommand{\shortarrow}[1][]{%
\check@mathfonts
\if\relax\detokenize{#1}\relax
\settowidth{\dimen@}{$\m@th\rightarrow$}%
\else
\setlength{\dimen@}{#1}%
\fi
\sbox\z@{\usefont{U}{lasy}{m}{n}\symbol{41}}%
\begin{picture}(\dimen@,\ht\z@)
\roundcap
\put(\dimexpr\dimen@-.7\wd\z@,0){\usebox\z@}
\put(0,\fontdimen22\textfont2){\line(1,0){\dimen@}}
\end{picture}%
}
\makeatother
% The "axessiblity" package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
%\usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\begin{document}
% \renewcommand\thelinenumber{\color[rgb]{0.2,0.5,0.8}\normalfont\sffamily\scriptsize\arabic{linenumber}\color[rgb]{0,0,0}}
% \renewcommand\makeLineNumber {\hss\thelinenumber\ \hspace{6mm} \rlap{\hskip\textwidth\ \hspace{6.5mm}\thelinenumber}}
% \linenumbers
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{4593} % Insert your submission number here
\title{Cross-Domain Ensemble Distillation \linebreak for Domain Generalization} % Replace with your title
% INITIAL SUBMISSION
\begin{comment}
\titlerunning{ECCV-22 submission ID \ECCVSubNumber}
\authorrunning{ECCV-22 submission ID \ECCVSubNumber}
\author{Anonymous ECCV submission}
\institute{Paper ID \ECCVSubNumber}
\end{comment}
%******************
% CAMERA READY SUBMISSION
%\begin{comment}
\titlerunning{Cross-Domain Ensemble Distillation for Domain Generalization}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%
\author{Kyungmoon Lee%
% \thanks{This work was done when Kyungmoon Lee was at POSTECH.}
\inst{1,2}\orcidlink{0000-0001-6597-3928} \qquad
Sungyeon Kim\inst{1}\orcidlink{0000-0002-6919-4822} \qquad
Suha Kwak\inst{1}\orcidlink{0000-0002-4567-9091}}%\orcidID{2222--3333-4444-5555}}
%
\authorrunning{Kyungmoon Lee, Sungyeon Kim, Suha Kwak}
% First names are abbreviated in the running head.
% If there are more than two authors, 'et al.' is used.
%
\institute{$^{1}$POSTECH, Pohang, Korea \qquad $^{2}$NALBI Inc, Seoul, Korea \\
\email{kyungmoon@nalbi.ai, \{sungyeon.kim, suha.kwak\}@postech.ac.kr }\\
{\tt\small
\url{http://cvlab.postech.ac.kr/research/XDED/}
}
}
\maketitle
\input{CameraReady/CR_0_abstract}
\input{CameraReady/CR_1_introduction}
\input{CameraReady/CR_2_relatedwork}
\input{CameraReady/CR_3_method}
\input{CameraReady/CR_4_experiments}
\input{CameraReady/CR_5_conclusion}
%\clearpage\mbox{}Page \thepage\ of the manuscript.
%\clearpage\mbox{}Page \thepage\ of the manuscript.
%This is the last page of the manuscript.
%\par\vfill\par
%Now we have reached the maximum size of the ECCV 2022 submission (excluding references).
%References should start immediately after the main text, but can continue on p.15 if needed.
%\clearpage
% ---- Bibliography ----
%
% BibTeX users should specify bibliography style 'splncs04'.
% References will then be sorted and formatted in the correct style.
%
\bibliographystyle{splncs04}
\bibliography{cvlab_kwak}
\end{document}
% updated April 2002 by Antje Endemann
% Based on CVPR 07 and LNCS, with modifications by DAF, AZ and elle, 2008 and AA, 2010, and CC, 2011; TT, 2014; AAS, 2016; AAS, 2020; TH, 2022
\documentclass[runningheads]{llncs}
\usepackage{graphicx}
% DO NOT USE \usepackage{times}, it will be removed by typesetters
%\usepackage{times}
\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{color}
\usepackage{epsfig}
\usepackage{lipsum}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{url}
\usepackage{xcolor, colortbl}
\usepackage{mathtools}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{enumitem}
\usepackage{bbm}
\usepackage{wrapfig}
\usepackage{setspace}
\usepackage{rotating}
\RequirePackage{fix-cm}
\usepackage{booktabs}
\usepackage{kotex}
% \documentclass{article}
\usepackage{pifont}
\usepackage{blindtext}
\usepackage{xr}
\usepackage[toc]{multitoc}
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
% \hypersetup{colorlinks=true,allcolors=[rgb]{0,1,0}}
% \hypersetup{colorlinks=true}
% \newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}p{#1}}
\newcolumntype{P}[1]{>{\centering\arraybackslash}p{#1}}
%\input{sec/9_custom_cmds}
% The “axessiblity” package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
% \usepackage{ruler}
% \usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\usepackage[capitalize]{cleveref}
\newcommand{\sftype}[1]{{\textsf{\small #1}}}
\newcommand{\sftypes}[1]{{\textsf{\tiny #1}}}
\newcommand{\expnum}[2]{{#1}\mathrm{e}{-#2}}
%\renewcommand{\Tbl}[1]{Table~\ref{tab:#1}}
%\end{comment}
%******************
% kyungmoon
\usepackage{soul}
\usepackage{makecell}
\usepackage{tabularx}
\usepackage{xcolor, colortbl}
\usepackage{enumitem}
\usepackage{subcaption}
\usepackage{adjustbox}
\usepackage{diagbox}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{pifont}
\usepackage{orcidlink}
\usepackage{pict2e,picture}
%\theoremstyle{theorem}
\newtheorem{theorem}{Theorem}
\def\ie{\emph{i.e.}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al.}}
\def\wrt{\emph{w.r.t.}}
\definecolor{grey}{rgb}{0.9, 0.9, 0.9}
\newcommand{\ccol}{\cellcolor{grey}}
\definecolor{brown}{rgb}{0.65, 0.16, 0.16}
\newcommand{\ready}[1]{{\color{brown}{#1}}}
\newcommand{\sy}[1]{{\color{purple}{#1}}}
\usepackage{pict2e,picture}
\makeatletter
\DeclareRobustCommand{\shortarrow}[1][]{%
\check@mathfonts
\if\relax\detokenize{#1}\relax
\settowidth{\dimen@}{$\m@th\rightarrow$}%
\else
\setlength{\dimen@}{#1}%
\fi
\sbox\z@{\usefont{U}{lasy}{m}{n}\symbol{41}}%
\begin{picture}(\dimen@,\ht\z@)
\roundcap
\put(\dimexpr\dimen@-.7\wd\z@,0){\usebox\z@}
\put(0,\fontdimen22\textfont2){\line(1,0){\dimen@}}
\end{picture}%
}
\makeatother
% The "axessiblity" package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
%\usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\begin{document}
% \renewcommand\thelinenumber{\color[rgb]{0.2,0.5,0.8}\normalfont\sffamily\scriptsize\arabic{linenumber}\color[rgb]{0,0,0}}
% \renewcommand\makeLineNumber {\hss\thelinenumber\ \hspace{6mm} \rlap{\hskip\textwidth\ \hspace{6.5mm}\thelinenumber}}
% \linenumbers
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{4593} % Insert your submission number here
\title{Cross-Domain Ensemble Distillation \linebreak for Domain Generalization} % Replace with your title
% INITIAL SUBMISSION
\begin{comment}
\titlerunning{ECCV-22 submission ID \ECCVSubNumber}
\authorrunning{ECCV-22 submission ID \ECCVSubNumber}
\author{Anonymous ECCV submission}
\institute{Paper ID \ECCVSubNumber}
\end{comment}
%******************
% CAMERA READY SUBMISSION
%\begin{comment}
\titlerunning{Cross-Domain Ensemble Distillation for Domain Generalization}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%
\author{Kyungmoon Lee%
% \thanks{This work was done when Kyungmoon Lee was at POSTECH.}
% 0000-0001-6597-3928
\inst{1,2}\orcidlink{0000-0001-6597-3928} \qquad
Sungyeon Kim\inst{2}\orcidlink{0000-0002-6919-4822} \qquad
Suha Kwak\inst{2}\orcidlink{0000-0002-4567-9091}}%\orcidID{2222--3333-4444-5555}}
%
\authorrunning{Kyungmoon Lee, Sungyeon Kim, Suha Kwak}
% First names are abbreviated in the running head.
% If there are more than two authors, 'et al.' is used.
%
\institute{$^{1}$NALBI Inc, Seoul, Korea \qquad $^{2}$POSTECH, Pohang, Korea \\
\email{kyungmoon@nalbi.ai, \{sungyeon.kim, suha.kwak\}@postech.ac.kr }\\
{\tt\small
\url{http://cvlab.postech.ac.kr/research/XDED/}
}
}
%\end{comment}
%******************
\maketitle
%%%%%%%%% ABSTRACT
%\input{_0_abstract}
%%%%%%%%% BODY TEXT
%\input{_1_introduction}
%\input{_2_relatedwork}
%\input{_3_method}
%\input{_4_experiments}
\input{CameraReady/CR_0_abstract}
\input{CameraReady/CR_1_introduction}
\input{CameraReady/CR_2_relatedwork}
\input{CameraReady/CR_3_method}
\input{CameraReady/CR_4_experiments}
\input{CameraReady/CR_5_conclusion}
%\clearpage\mbox{}Page \thepage\ of the manuscript.
%\clearpage\mbox{}Page \thepage\ of the manuscript.
%This is the last page of the manuscript.
%\par\vfill\par
%Now we have reached the maximum size of the ECCV 2022 submission (excluding references).
%References should start immediately after the main text, but can continue on p.15 if needed.
%\clearpage
% ---- Bibliography ----
%
% BibTeX users should specify bibliography style 'splncs04'.
% References will then be sorted and formatted in the correct style.
%
\bibliographystyle{splncs04}
\bibliography{cvlab_kwak}
\end{document}
% updated April 2002 by Antje Endemann
% Based on CVPR 07 and LNCS, with modifications by DAF, AZ and elle, 2008 and AA, 2010, and CC, 2011; TT, 2014; AAS, 2016; AAS, 2020; TH, 2022
\documentclass[runningheads]{llncs}
\usepackage{graphicx}
% DO NOT USE \usepackage{times}, it will be removed by typesetters
%\usepackage{times}
\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{color}
\usepackage{epsfig}
\usepackage{lipsum}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{url}
\usepackage{xcolor, colortbl}
\usepackage{mathtools}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{enumitem}
\usepackage{bbm}
\usepackage{wrapfig}
\usepackage{setspace}
\usepackage{rotating}
\RequirePackage{fix-cm}
\usepackage{booktabs}
\usepackage{kotex}
% \documentclass{article}
\usepackage{pifont}
\usepackage{blindtext}
\usepackage{xr}
\usepackage[toc]{multitoc}
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
% \hypersetup{colorlinks=true,allcolors=[rgb]{0,1,0}}
% \hypersetup{colorlinks=true}
% \newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}p{#1}}
\newcolumntype{P}[1]{>{\centering\arraybackslash}p{#1}}
%\input{sec/9_custom_cmds}
\def\ie{\emph{i.e.}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al.}}
\def\wrt{\emph{w.r.t.}}
% The “axessiblity” package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
% \usepackage{ruler}
% \usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\usepackage[capitalize]{cleveref}
\newcommand{\sftype}[1]{{\textsf{\small #1}}}
\newcommand{\sftypes}[1]{{\textsf{\tiny #1}}}
\newcommand{\expnum}[2]{{#1}\mathrm{e}{-#2}}
%\renewcommand{\Tbl}[1]{Table~\ref{tab:#1}}
% kyungmoon
\usepackage{soul}
\usepackage{makecell}
\usepackage{tabularx}
\usepackage{xcolor, colortbl}
\usepackage{enumitem}
\usepackage{subcaption}
\usepackage{adjustbox}
\usepackage{diagbox}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{pifont}
\usepackage{orcidlink}
\usepackage{pict2e,picture}
%\theoremstyle{theorem}
\def\ie{\emph{i.e.}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al.}}
\def\wrt{\emph{w.r.t.}}
\definecolor{grey}{rgb}{0.9, 0.9, 0.9}
\newcommand{\ccol}{\cellcolor{grey}}
\definecolor{brown}{rgb}{0.65, 0.16, 0.16}
\newcommand{\ready}[1]{{\color{brown}{#1}}}
\newcommand{\sy}[1]{{\color{purple}{#1}}}
\usepackage{pict2e,picture}
\makeatletter
\DeclareRobustCommand{\shortarrow}[1][]{%
\check@mathfonts
\if\relax\detokenize{#1}\relax
\settowidth{\dimen@}{$\m@th\rightarrow$}%
\else
\setlength{\dimen@}{#1}%
\fi
\sbox\z@{\usefont{U}{lasy}{m}{n}\symbol{41}}
\begin{picture}(\dimen@,\ht\z@)
\roundcap
\put(\dimexpr\dimen@-.7\wd\z@,0){\usebox\z@}
\put(0,\fontdimen22\textfont2){\line(1,0){\dimen@}}
\end{picture}%
}
\makeatother
\begin{document}
\renewcommand{\theequation}{a\arabic{equation}}
\renewcommand{\thetable}{a\arabic{table}}
\renewcommand{\thefigure}{a\arabic{figure}}
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{4593}
\title{Cross-Domain Ensemble Distillation \linebreak for Domain Generalization}
\begin{comment}
\titlerunning{ECCV-22 submission ID \ECCVSubNumber}
\authorrunning{ECCV-22 submission ID \ECCVSubNumber}
\author{Anonymous ECCV submission}
\institute{Paper ID \ECCVSubNumber}
\end{comment}
%******************
% CAMERA READY SUBMISSION
%\begin{comment}
\titlerunning{Cross-Domain Ensemble Distillation for Domain Generalization}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%
\author{Kyungmoon Lee
\inst{1,2}
Sungyeon Kim\inst{1}
Suha Kwak\inst{1}}%\orcidID{2222--3333-4444-5555}}
%
\authorrunning{Kyungmoon Lee, Sungyeon Kim, Suha Kwak}
% First names are abbreviated in the running head.
% If there are more than two authors, 'et al.' is used.
%
\institute{$^{1}$POSTECH, Pohang, Korea \qquad $^{2}$NALBI Inc, Seoul, Korea \\
%\email{kyungmoon@nalbi.ai, \{sungyeon.kim, suha.kwak\}@postech.ac.kr }\\
%{\tt\small
%\url{http://cvlab.postech.ac.kr/research/XDED%/}
%}
}
%\end{comment}
%******************
\maketitle
\appendix
\input{CameraReady/supp/CR_1_introduction}
\input{CameraReady/supp/CR_2_additional_exp}
\input{CameraReady/supp/CR_3_xded}
\input{CameraReady/supp/CR_4_unityle}
\input{CameraReady/supp/CR_5_implementation}
%\input{supple/_3_xded}
%\input{supple/_4_unistyle}
%\input{supple/_5_other_experiment_materials}
\clearpage
{\small
\bibliographystyle{splncs04}
\bibliography{cvlab_kwak}
}
\end{document}
% updated April 2002 by Antje Endemann
% Based on CVPR 07 and LNCS, with modifications by DAF, AZ and elle, 2008 and AA, 2010, and CC, 2011; TT, 2014; AAS, 2016; AAS, 2020; TH, 2022
\documentclass[runningheads]{llncs}
\usepackage{graphicx}
% DO NOT USE \usepackage{times}, it will be removed by typesetters
%\usepackage{times}
\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{color}
% kyungmoon
\usepackage{soul}
\usepackage{makecell}
\usepackage{tabularx}
\usepackage{xcolor, colortbl}
\usepackage{enumitem}
\usepackage{subcaption}
\usepackage{adjustbox}
\usepackage{diagbox}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{pifont}
%\theoremstyle{theorem}
\newtheorem{theorem}{Theorem}
\def\ie{\emph{i.e.}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al.}}
\def\wrt{\emph{w.r.t.}}
\definecolor{grey}{rgb}{0.9, 0.9, 0.9}
\newcommand{\ccol}{\cellcolor{grey}}
\definecolor{brown}{rgb}{0.65, 0.16, 0.16}
\newcommand{\kmc}[1]{{\color{brown}{#1}}}
\newcommand{\sy}[1]{{\color{purple}{#1}}}
% The "axessiblity" package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
%\usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\begin{document}
% \renewcommand\thelinenumber{\color[rgb]{0.2,0.5,0.8}\normalfont\sffamily\scriptsize\arabic{linenumber}\color[rgb]{0,0,0}}
% \renewcommand\makeLineNumber {\hss\thelinenumber\ \hspace{6mm} \rlap{\hskip\textwidth\ \hspace{6.5mm}\thelinenumber}}
% \linenumbers
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{4593} % Insert your submission number here
\title{Cross-Domain Ensemble Distillation \linebreak for Domain Generalization} % Replace with your title
% INITIAL SUBMISSION
\begin{comment}
\titlerunning{ECCV-22 submission ID \ECCVSubNumber}
\authorrunning{ECCV-22 submission ID \ECCVSubNumber}
\author{Anonymous ECCV submission}
\institute{Paper ID \ECCVSubNumber}
\end{comment}
%******************
% CAMERA READY SUBMISSION
%\begin{comment}
\titlerunning{Cross-Domain Ensemble Distillation for Domain Generalization}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%
\author{Kyungmoon Lee\thanks{This work was done when Kyungmoon Lee was at POSTECH.}\inst{1} \and
Sungyeon Kim\inst{2} \and
Suha Kwak\inst{2}}%\orcidID{2222--3333-4444-5555}}
%
\authorrunning{Kyungmoon Lee, Sungyeon Kim, Suha Kwak}
% First names are abbreviated in the running head.
% If there are more than two authors, 'et al.' is used.
%
\institute{NALBI Inc, Seoul, Korea \and POSTECH, Pohang, Korea \\
\email{kyungmoon@nalbi.ai, \{sungyeon.kim, suha.kwak\}@postech.ac.kr }\\
{\tt\small
\url{http://cvlab.postech.ac.kr/research/XDED}
}
}
%\end{comment}
%******************
\maketitle
%%%%%%%%% ABSTRACT
%\input{_0_abstract}
%%%%%%%%% BODY TEXT
%\input{_1_introduction}
%\input{_2_relatedwork}
%\input{_3_method}
%\input{_4_experiments}
%\input{_5_conclusion}
%\clearpage\mbox{}Page \thepage\ of the manuscript.
%\clearpage\mbox{}Page \thepage\ of the manuscript.
%This is the last page of the manuscript.
%\par\vfill\par
%Now we have reached the maximum size of the ECCV 2022 submission (excluding references).
%References should start immediately after the main text, but can continue on p.15 if needed.
%\clearpage
% ---- Bibliography ----
%
% BibTeX users should specify bibliography style 'splncs04'.
% References will then be sorted and formatted in the correct style.
%
\bibliographystyle{splncs04}
\bibliography{cvlab_kwak}
\end{document}
% updated April 2002 by Antje Endemann
% Based on CVPR 07 and LNCS, with modifications by DAF, AZ and elle, 2008 and AA, 2010, and CC, 2011; TT, 2014; AAS, 2016; AAS, 2020; TH, 2022
\documentclass[runningheads]{llncs}
\usepackage{graphicx}
% DO NOT USE \usepackage{times}, it will be removed by typesetters
%\usepackage{times}
\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{color}
\usepackage{epsfig}
\usepackage{lipsum}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{url}
\usepackage{xcolor, colortbl}
\usepackage{mathtools}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{enumitem}
\usepackage{bbm}
\usepackage{wrapfig}
\usepackage{setspace}
\usepackage{rotating}
\RequirePackage{fix-cm}
\usepackage{booktabs}
\usepackage{kotex}
% \documentclass{article}
\usepackage{pifont}
\usepackage{blindtext}
\usepackage{xr}
\usepackage[toc]{multitoc}
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
% \hypersetup{colorlinks=true,allcolors=[rgb]{0,1,0}}
% \hypersetup{colorlinks=true}
% \newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}p{#1}}
\newcolumntype{P}[1]{>{\centering\arraybackslash}p{#1}}
%\input{sec/9_custom_cmds}
% The “axessiblity” package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
% \usepackage{ruler}
% \usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\usepackage[capitalize]{cleveref}
\newcommand{\sftype}[1]{{\textsf{\small #1}}}
\newcommand{\sftypes}[1]{{\textsf{\tiny #1}}}
\newcommand{\expnum}[2]{{#1}\mathrm{e}{-#2}}
%\renewcommand{\Tbl}[1]{Table~\ref{tab:#1}}
%\end{comment}
%******************
% kyungmoon
\usepackage{soul}
\usepackage{makecell}
\usepackage{tabularx}
\usepackage{xcolor, colortbl}
\usepackage{enumitem}
\usepackage{subcaption}
\usepackage{adjustbox}
\usepackage{diagbox}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{pifont}
%\usepackage{orcidlink}
\usepackage{pict2e,picture}
%\theoremstyle{theorem}
%\newtheorem{theorem}{Theorem}
\def\ie{\emph{i.e.}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al.}}
\def\wrt{\emph{w.r.t.}}
\definecolor{grey}{rgb}{0.9, 0.9, 0.9}
\newcommand{\ccol}{\cellcolor{grey}}
\definecolor{brown}{rgb}{0.65, 0.16, 0.16}
\newcommand{\ready}[1]{{\color{brown}{#1}}}
\newcommand{\sy}[1]{{\color{purple}{#1}}}
\usepackage{pict2e,picture}
\makeatletter
\DeclareRobustCommand{\shortarrow}[1][]{%
\check@mathfonts
\if\relax\detokenize{#1}\relax
\settowidth{\dimen@}{$\m@th\rightarrow$}%
\else
\setlength{\dimen@}{#1}%
\fi
\sbox\z@{\usefont{U}{lasy}{m}{n}\symbol{41}}%
\begin{picture}(\dimen@,\ht\z@)
\roundcap
\put(\dimexpr\dimen@-.7\wd\z@,0){\usebox\z@}
\put(0,\fontdimen22\textfont2){\line(1,0){\dimen@}}
\end{picture}%
}
\makeatother
% The "axessiblity" package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
%\usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\begin{document}
% \renewcommand\thelinenumber{\color[rgb]{0.2,0.5,0.8}\normalfont\sffamily\scriptsize\arabic{linenumber}\color[rgb]{0,0,0}}
% \renewcommand\makeLineNumber {\hss\thelinenumber\ \hspace{6mm} \rlap{\hskip\textwidth\ \hspace{6.5mm}\thelinenumber}}
% \linenumbers
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{4593} % Insert your submission number here
\title{Cross-Domain Ensemble Distillation \linebreak for Domain Generalization} % Replace with your title
% INITIAL SUBMISSION
\begin{comment}
\titlerunning{ECCV-22 submission ID \ECCVSubNumber}
\authorrunning{ECCV-22 submission ID \ECCVSubNumber}
\author{Anonymous ECCV submission}
\institute{Paper ID \ECCVSubNumber}
\end{comment}
%******************
% CAMERA READY SUBMISSION
%\begin{comment}
\titlerunning{Cross-Domain Ensemble Distillation for Domain Generalization}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%
\author{Kyungmoon Lee%
% \thanks{This work was done when Kyungmoon Lee was at POSTECH.}
\inst{1,2}
%\orcidlink{0000-0001-6597-3928} \qquad
Sungyeon Kim\inst{1}
%\orcidlink{0000-0002-6919-4822} \qquad
Suha Kwak\inst{1}
%\orcidlink{0000-0002-4567-9091}
}%\orcidID{2222--3333-4444-5555}}
%
\authorrunning{Kyungmoon Lee, Sungyeon Kim, Suha Kwak}
% First names are abbreviated in the running head.
% If there are more than two authors, 'et al.' is used.
%
\institute{$^{1}$POSTECH, Pohang, Korea \qquad $^{2}$NALBI Inc., Seoul, Korea \\
\email{kyungmoon@nalbi.ai, \{sungyeon.kim, suha.kwak\}@postech.ac.kr }\\
{\tt\small
\url{http://cvlab.postech.ac.kr/research/XDED/}
}
}
\maketitle
\input{CameraReady/CR_0_abstract}
\input{CameraReady/CR_1_introduction}
\input{CameraReady/CR_2_relatedwork}
\input{CameraReady/CR_3_method}
\input{CameraReady/CR_4_experiments}
\input{CameraReady/CR_5_conclusion}
%\clearpage\mbox{}Page \thepage\ of the manuscript.
%\clearpage\mbox{}Page \thepage\ of the manuscript.
%This is the last page of the manuscript.
%\par\vfill\par
%Now we have reached the maximum size of the ECCV 2022 submission (excluding references).
%References should start immediately after the main text, but can continue on p.15 if needed.
%\clearpage
% ---- Bibliography ----
%
% BibTeX users should specify bibliography style 'splncs04'.
% References will then be sorted and formatted in the correct style.
%
\bibliographystyle{splncs04}
\bibliography{cvlab_kwak}
\end{document}
% updated April 2002 by Antje Endemann
% Based on CVPR 07 and LNCS, with modifications by DAF, AZ and elle, 2008 and AA, 2010, and CC, 2011; TT, 2014; AAS, 2016; AAS, 2020; TH, 2022
\documentclass[runningheads]{llncs}
\usepackage{graphicx}
% DO NOT USE \usepackage{times}, it will be removed by typesetters
%\usepackage{times}
\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} % define this before the line numbering.
\usepackage{color}
\usepackage{epsfig}
\usepackage{lipsum}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{url}
\usepackage{xcolor, colortbl}
\usepackage{mathtools}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{enumitem}
\usepackage{bbm}
\usepackage{wrapfig}
\usepackage{setspace}
\usepackage{rotating}
\RequirePackage{fix-cm}
\usepackage{booktabs}
\usepackage{kotex}
% \documentclass{article}
\usepackage{pifont}
\usepackage{blindtext}
\usepackage{xr}
\usepackage[toc]{multitoc}
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
% \hypersetup{colorlinks=true,allcolors=[rgb]{0,1,0}}
% \hypersetup{colorlinks=true}
% \newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}p{#1}}
\newcolumntype{P}[1]{>{\centering\arraybackslash}p{#1}}
%\input{sec/9_custom_cmds}
\def\ie{\emph{i.e.}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al.}}
\def\wrt{\emph{w.r.t.}}
% The “axessiblity” package can be found at: https://ctan.org/pkg/axessibility?lang=en
\usepackage[accsupp]{axessibility} % Improves PDF readability for those with disabilities.
% INITIAL SUBMISSION - The following two lines are NOT commented
% CAMERA READY - Comment OUT the following two lines
% \usepackage{ruler}
% \usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\usepackage[capitalize]{cleveref}
\newcommand{\sftype}[1]{{\textsf{\small #1}}}
\newcommand{\sftypes}[1]{{\textsf{\tiny #1}}}
\newcommand{\expnum}[2]{{#1}\mathrm{e}{-#2}}
%\renewcommand{\Tbl}[1]{Table~\ref{tab:#1}}
% kyungmoon
\usepackage{soul}
\usepackage{makecell}
\usepackage{tabularx}
\usepackage{xcolor, colortbl}
\usepackage{enumitem}
\usepackage{subcaption}
\usepackage{adjustbox}
\usepackage{diagbox}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{pifont}
\usepackage{orcidlink}
\usepackage{pict2e,picture}
%\theoremstyle{theorem}
\def\ie{\emph{i.e.}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al.}}
\def\wrt{\emph{w.r.t.}}
\definecolor{grey}{rgb}{0.9, 0.9, 0.9}
\newcommand{\ccol}{\cellcolor{grey}}
\definecolor{brown}{rgb}{0.65, 0.16, 0.16}
\newcommand{\ready}[1]{{\color{brown}{#1}}}
\newcommand{\sy}[1]{{\color{purple}{#1}}}
\usepackage{pict2e,picture}
\makeatletter
\DeclareRobustCommand{\shortarrow}[1][]{%
\check@mathfonts
\if\relax\detokenize{#1}\relax
\settowidth{\dimen@}{$\m@th\rightarrow$}%
\else
\setlength{\dimen@}{#1}%
\fi
\sbox\z@{\usefont{U}{lasy}{m}{n}\symbol{41}}%
\begin{picture}(\dimen@,\ht\z@)
\roundcap
\put(\dimexpr\dimen@-.7\wd\z@,0){\usebox\z@}
\put(0,\fontdimen22\textfont2){\line(1,0){\dimen@}}
\end{picture}%
}
\makeatother
\begin{document}
\renewcommand{\theequation}{a\arabic{equation}}
\renewcommand{\thetable}{a\arabic{table}}
\renewcommand{\thefigure}{a\arabic{figure}}
\pagestyle{headings}
\mainmatter
\def\ECCVSubNumber{4593} % Insert your submission number here
\title{Cross-Domain Ensemble Distillation \linebreak for Domain Generalization} % Replace with your title
% INITIAL SUBMISSION
\begin{comment}
\titlerunning{ECCV-22 submission ID \ECCVSubNumber}
\authorrunning{ECCV-22 submission ID \ECCVSubNumber}
\author{Anonymous ECCV submission}
\institute{Paper ID \ECCVSubNumber}
\end{comment}
%******************
% CAMERA READY SUBMISSION
%\begin{comment}
\titlerunning{Cross-Domain Ensemble Distillation for Domain Generalization}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%
\author{Kyungmoon Lee%
% \thanks{This work was done when Kyungmoon Lee was at POSTECH.}
\inst{1,2}
Sungyeon Kim\inst{2}
Suha Kwak\inst{2}}%\orcidID{2222--3333-4444-5555}}
%
\authorrunning{Kyungmoon Lee, Sungyeon Kim, Suha Kwak}
% First names are abbreviated in the running head.
% If there are more than two authors, 'et al.' is used.
%
\institute{$^{1}$NALBI Inc, Seoul, Korea \qquad $^{2}$POSTECH, Pohang, Korea \\
%\email{kyungmoon@nalbi.ai, \{sungyeon.kim, suha.kwak\}@postech.ac.kr }\\
%{\tt\small
%\url{http://cvlab.postech.ac.kr/research/XDED%/}
%}
}
%\end{comment}
%******************
\maketitle
\appendix
\input{CameraReady/supp/CR_1_introduction}
\input{CameraReady/supp/CR_2_additional_exp}
\input{CameraReady/supp/CR_3_xded}
\input{CameraReady/supp/CR_4_unityle}
\input{CameraReady/supp/CR_5_implementation}
%\input{supple/_3_xded}
%\input{supple/_4_unistyle}
%\input{supple/_5_other_experiment_materials}
\clearpage
{\small
\bibliographystyle{splncs04}
\bibliography{cvlab_kwak}
}
\end{document}
This supplementary material presents additional experimental results, further analyses, and implementation details, all of which are omitted from the main paper due to the space limit.
% the typo correction in our main paper, additional analysis results of the proposed components and other experimental materials, all of which are omitted from the main paper due to the space limit. First, Sec.~\ref{sec:supple_xded} and Sec.~\ref{sec:supple_unistyle} present additional analysis results of XDED and UniStyle, respectively. Lastly, Sec.~\ref{sec:supple_other_materials} includes implementation details.
%First, Sec.~\ref{sec:supple_type_correction} corrects our typo in the main paper to avoid misunderstanding. Sec.~\ref{sec:supple_xded} and Sec.~\ref{sec:supple_unistyle} present additional analysis results of XDED and UniStyle, respectively. Sec.~\ref{sec:supple_other_materials} includes implementation details and and instructions for our source code.
% Finally, we attach the partial source codes implemented using Pytorch. The full implementation will be released on publication.
\section{Typo correction for Table.9 of our main paper}
\label{sec:supple_type_correction}
Before diving into additional results of our paper, we sincerely apologize for our typo in table 9 of our main paper. Even though we adopted the 40-2 Wide ResNet~\cite{zagoruyko2016wide} as the backbone network for the results of table 9 in our main paper, we wrote it wrong as WRN-28-10. Again, we are respectfully sorry for the typo, and remark that the experiment of table 9 in our main paper is conducted using the same backbone architecture (\ie, WRN-40-2) for a fair comparison, exactly following the protocol~\cite{hendrycks2019benchmarking,hendrycks2019augmix} to measure generalization to unseen corruptions.
%we exactly follow the protocol to measure generalization to unseen corruptions using \emph{the same backbone architecture} for a fair comparison, which is provided by \cite{hendrycks2019benchmarking}.
\section{Additional analysis results on XDED}
\label{sec:supple_xded}
In this section, we provide in-depth comparative analysis results between XDED and existing flat minima-promoting methods, qualitative/quantitative analysis results delving into the effect of using the ensemble, and additional ablation studies related to XDED. Lastly, we provide additional empirical results on the flatness of local minima (Fig.~\ref{fig:supple_weight_perturbation}).
% to-do:
% \subsection{on How XDED promotes flat minima.}
% \subsubsection{XDED induces high entropy indeed.}
% \subsection{Ensembles matter for generalization ability.}
% high-entropy,
% Ensemble matters: qualitative (from CSKD), comparison XDED with (DWED, CSKD), maybe? standard benchmarks (cifar-10, cifar-100, imagenet)
% Comparison to flat minima methods (e.g., A-distance, t-sne, overhead against SAM)
% number of classes in a batch
% standard benchmarks (cifar-10, cifar-100, imagenet)
% learning acceleration
% comparison to ensemble methods
% hyperparam-ablation
% input deformation results
% implementation details
% done: implementation details, input deformation results, hyperparam-ablation, comparison to ensemble methods, learning acceleration, standard benchmarks (cifar-10, cifar-100, imagenet)
%\subsection{The Perspective of Flat Minima}
\subsection{Comparison to existing flat minima-promoting methods}
\begin{figure*}[t]
\centering
\includegraphics[width=1.0\textwidth]{supple/supple_figure/supple_a_dist_2.pdf}
\caption{
Comparison to existing flat minima-promoting methods. Each model is trained on source domains (\ie, Art Painting, Sketch and Photo of PACS) and evaluated on the target domain (\ie, Cartoon of PACS). \textbf{Left}: The divergence ($\mathcal{A}$-distance) between the source domains and the target domain, \textbf{Right}: Generalization performance on the target domain.
}
\label{fig:supple_a_dist}
\end{figure*}
%\subsubsection{XDED learns domain-invariant features as well.}
\subsubsection{Domain-invariant feature learning.}
According to Theorem 1 in our main paper, the test loss on the target domain is bounded by three terms: (1) the robust empirical loss, (2) the domain discrepancy between multiple domains, and (3) confidence bound. While our main paper primarily focuses on the empirical evidences verifying that XDED can effectively reduce the first term (\ie, elevate the flatness of local minima where the model converges), this section highlights that XDED also can reduce the second term (\ie, domain discrepancy) as well as the first term. Therefore, we compare XDED with existing flat minima-promoting methods which are dedicated to reducing the first term only. To investigate the effectiveness of XDED in reducing the domain discrepancy, we measure $\mathcal{A}$-distance~\cite{ben2010theory,kifer2004detecting}, following the protocol stated in our main paper. As shown in Fig.~\ref{fig:supple_a_dist} (Left), although existing flat minima-promoting methods can reduce the robust empirical loss term, they fail to reduce domain discrepancy compared to the vanilla method. In contrast, XDED clearly demonstrates its efficacy to lower the distance between source domains and the target domain. This result demonstrates that XDED learns domain-invariant features by enforcing predictive distribution-level consistency between the data with the same label but from different domains, similar to previous approaches to aligning the feature-level distributions of multiple domains. Next, our investigation also reveals that the result of Fig.~\ref{fig:supple_a_dist} (Left) is exactly reflected in the generalization ability of each method. In detail, as shown in Fig.~\ref{fig:supple_a_dist} (Right), XDED clearly outperforms all the flat minima-promoting methods since only it can learn domain-invariant features unlike them, and this difference leads to significant generalization ability as a result. In addition, when equipped with our UniStyle proposed to reduce the domain gap with the target domain, our framework further reduces the domain discrepancy and enhances the generalization ability. With the above results, we argue that both promoting flat minima and learning domain-invariant features are important for better domain generalization ability, and XDED simultaneously achieves these two goals.
% The result of Fig.~\ref{fig:supple_a_dist} (Left) is exactly reflected to the generalization results of each method, which is shown in Fig.~\ref{fig:supple_a_dist} (Right).
% As shown in Fig.~\ref{fig:supple_a_dist} (Right), by promoting flat minima, all the methods including XDED clearly outperform the vanilla method. However, since XDED can learn domain-invariant features, it shows superiority over other baseline methods.
% With the above results, we argue that both planar minima facilitation and domain invariant feature learning are necessary for better domain generalization, and that XDED achieves these two goals simultaneously.
% not only promote flat minima but also learn domain-invariant features for better domain generalization and XDED achieves these two goals at the same time.
% This result supports our view that XDED leads to better generalization ability by not only promoting flat minima but also reducing domain discrepancy.
% To sum it up, we demonstrate that XDED is not limited to encouraging the convergence into flat minima but also helps the model learn domain-invariant features by penalizing the mismatch of the predictive distributions between the data with the same label but \emph{from different domains}.
\iffalse
\begin{table}[!t]
\centering
\caption{
Comparison of the entropy values which each method induces. Note that the entropy values are calculated by averaging over all training samples when the marked domain is the target domain and the maximum of entropy increases as the number of classes increases.
}
%\fontsize{9}{11}\selectfont
\begin{tabularx}{0.8 \textwidth}{
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
\hline
\multicolumn{1}{l|}{Methods} & Vanilla & MixStyle~\cite{zhou2021domain} & XDED \\
\hline
\multicolumn{1}{l|}{OfficeHome (Clipart)} & 0.25 & 0.35 & \textbf{0.92} \\
% sec/iter
\multicolumn{1}{l|}{PACS (Cartoon)} & 0.01 & 0.03 & \textbf{0.38} \\
\hline
\end{tabularx}
\label{tab:supple_comparison_entropy}
\end{table}
\fi
\subsubsection{High entropy induced by XDED.} To investigate whether XDED induces high entropy in practice, we calculate the prediction entropy values of the learned model and compare the results of XDED with those of baseline methods. As shown in Table.~\ref{tab:supple_comparison_entropy}, XDED induces higher entropy than other methods for all cases. Specifically, in the case of OfficeHome, the entropy of XDED is about 4 times larger than that of the vanilla method. In the case of PACS, entropy values of other methods are close to zero, while XDED induces higher entropy. In addition, we argue that XDED induces high entropy meaningfully, not blindly. Namely, compared to the entropy value of uniform distribution, the entropy values from XDED are reasonable. Thanks to the use of the ensemble which encodes the meaningful information, XDED is not encouraged to pursue blindly high entropy. Also, note that MixStyle has shown its effectiveness in domain generalization, but it cannot make a difference compared to the vanilla method in inducing high entropy since it has been proposed from the perspective of the data augmentation rather than entropy regularization like our approach. This also supports our view that XDED can further enhance generalization ability as an entropy regularization approach by helping the model converge to flat minima.
\input{table/comparison_entropy}
\begin{table}[!t]
\centering
\caption{
Generalization results on the target domain (Clipart of OfficeHome) and required overhead during training.
}
%\fontsize{9}{11}\selectfont
\begin{tabularx}{0.7 \textwidth}{
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
\hline
\multicolumn{1}{l|}{Methods} & Vanilla & SAM~\cite{foret2021sharpness} & XDED \\
\hline
\multicolumn{1}{l|}{Accuracy (\%) ($\uparrow$)} & 49.4 & \underline{51.9} & \textbf{55.2} \\
% sec/iter
\multicolumn{1}{l|}{Time / iter (ms) ($\downarrow$)} & \textbf{32} & 72 & \underline{33} \\
\hline
\end{tabularx}
\label{tab:supple_flat_minima_cost}
\end{table}
\subsubsection{Convergence into flat minima at a low cost.}
Since there are several ways for convergence into flat minima, we believe that the required overhead to achieve that goal should be considered as another criterion of valuation for a practical usage. Therefore, according to this point of view, we compare XDED with SAM~\cite{foret2021sharpness} which has been proposed to optimize loss sharpness more directly. As shown in Table.~\ref{tab:supple_flat_minima_cost}, XDED shows superior robustness on the unseen domain over SAM. Furthermore, XDED requires negligible overhead while SAM requires the training time more than 2 times longer than the vanilla method and XDED. This is because SAM must additionally compute gradient approximation for the loss sharpness to be optimized, which exposes a trade-off between training complexity and the optimization of the loss sharpness.
% Although XDED has relatively weak methodological link to convergence to flat minima compared to SAM, we propose it based on the inspiring observation~\cite{zhang2018deep,zhang2019your} that entropy regularization approaches empirically have shown their efficacy in the convergence into flat minima and provide the empirical evidences that it helps the model converge to flat minima indeed.
On the other hand, SAM's performance is inferior to that of XDED, since it directly promotes a flat minima but cannot learn domain-invariant features, unlike our method. To sum it up, we argue that XDED offers more appealing option for domain generalization in the context of both performance improvement and required overhead.
% We remark that SAM demands additional overhead since it must compute gradient approximation for the loss sharpness to be optimized.
%\subsubsection{Additional Comparison to Existing Flat Minima-Promoting Methods.? Comparison to SAM?}
%\subsubsubsection{A-distance}
%\subsubsubsection{t-SNE}
%\subsubsubsection{Overhead}
\subsection{Ensemble matters for generalization ability.}
\label{sec:supple_standard_benchmarks}
\begin{figure}[!t]
\centering
\includegraphics[width=0.82\textwidth]{supple/supple_figure/supple_qualitative1_sero.pdf}
\caption{Prediction scores for misclassified samples from the vanilla method (\ie, Cross-entropy). \textbf{Up}: Well-classified results from XDED, \textbf{Down}: Misclassified results from XDED.
}
\label{fig:supple_qualitative}
\end{figure}
\subsubsection{Qualitative analysis.}
To examine how XDED affects the predictions of the learned model in detail, we investigate the softmax scores from the model which is trained on the source domains (\ie, Cartoon, Sketch and Photo of PACS) and evaluated on the target domain (\ie, Art Painting). Specifically, for the misclassified samples from the vanilla method, we assess the differences in the softmax scores from the vanilla method and XDED. First, as shown in Fig.~\ref{fig:supple_qualitative} (Up), XDED can lead to accurate predictions on the samples which the vanilla method confidently misclassifies. For the giraffe sample, the model learned with XDED not only predicts the ground truth successfully but also puts relatively high probability on the similar class (\ie, Dog) while lowering the probabilities of the irrelevant classes (\ie, Guitar and House). Similarly, even in its failure cases, XDED encourages the model to predict with consideration of inter-class relations (\ie, increasing scores for similar classes while suppressing those for irrelevant classes) as shown in Fig.~\ref{fig:supple_qualitative} (Down). By exploiting the ensemble incorporating information of multiple domains, XDED encourages the model to consider richer inter-class relations which may be exploited on the target domain and lead to meaningful predictions even in failure cases. Note that UniStyle is excluded from the results discussed above.
\input{table/comparison_iid}
\subsubsection{Comparison to existing self-KD methods.}
To further assess the impact of the ensemble distillation, we also compare XDED with state-of-the-art self-knowledge distillation (self-KD) methods in spite of sharing a similar methodology. Furthermore considering that our approach does not explicitly depend on domain labels, we investigate the impact of XDED on the standard benchmark datasets for the image recognition task, which assume the independent and identically distributed (i.i.d.) condition. As shown in Table~\ref{tab:comparison_iid}, XDED outperforms not only the vanilla method but also other self-KD methods for all datasets. Considering that XDED is not dedicated to the case under the i.i.d. condition whereas other methods are dedicated to the case under the i.i.d. condition, these experimental results support our view that XDED can provide more meaningful supervisory signals thanks to the use of ensemble which encodes the complementary knowledge from the data with the same label.
\subsection{Additional ablation studies.}
\subsubsection{Learning acceleration.}
\label{sec:supple_convergence}
Our framework also enables faster convergence of the learning process. In generalization on PACS, our effectiveness of learning acceleration is demonstrated. As shown in Fig.~\ref{fig:supple_convergence}, our framework reaches highest performance with less epochs (\ie, 24 epochs) compared to both vanilla method (40 epochs) and MixStyle~\cite{zhou2021domain} (39 epochs). Although the vanilla method requires many iterations to address the domain gaps between source domains, our framework accelerates the learning process via promoting flat minima which produce lower errors in not only source domains but also the target domain. Since MixStyle~\cite{zhou2021domain}, as an augmentation method, synthesizes novel styles via mixing statistics at the feature level, it is inherently limited to requiring as many epochs as possible to address many augmented styles for its best performance. Also, in the context of the actual training time in total, compared to MixStyle, we remark that our method significantly reduces the training complexity. In detail, our method increases training time per iteration by 5\% but requires only 60\% of training epochs and achieves about 1.6 times more performance boost averaged over every target domain of the PACS dataset. The results are summarized in Table~\ref{fig:supple_convergence}.
\begin{figure}[!t]
\centering
\includegraphics[width=0.8\columnwidth]{supple/supple_figure/learning_acceleration.pdf}
\caption{Multi-source domain generalization accuracy (\%) versus the number of epochs on the unseen target domain, Art Painting of PACS. Note that all models were trained on the rest three domains of PACS (\ie, Cartoon, Sketch, and Photo) and all methods were trained with the same hyperparameter setting (\eg, batch size and learning rate) on a single RTX 2080 GPU.
}
\label{fig:supple_convergence}
\end{figure}
\subsubsection{Comparison with other ensemble methods.}
\label{sec:supple_comparison_ensemble}
Lastly, we compare our framework with other domain generalization methods which adopt ensemble techniques. The comparison analysis is summarized in Table~\ref{tab:comparison_ensemble}. We remark that the proposed method does not require any additional module and is also free from domain labels (Note that our method, thus, seamlessly extends to the single-source domain generalization setting, which is discussed in Sec.4.1 of our main paper.). Nevertheless, our method clearly outperforms other methods in the context of generalization ability. In contrast, DAEL~\cite{zhou2021dael} and DSON~\cite{seo2020learning} show inferior results even though they explicitly exploit domain labels and multiple domain-specific modules (\ie, classifiers and BN layers~\cite{Batchnorm}, respectively.). DAEL is the most similar to our method but clearly different in the sense that DAEL exploits the predictions of an expert as supervisory signals for non-experts on the same instance, but our method constructs the ensemble using multiple instances and exploits it as supervision to model itself. In other words, by learning domain-invariant information through the ensemble of meaningful knowledge contained in different domains, better performance could be achieved even with limited resources.
\begin{table}[!t]
\centering
\caption{ Performance boosts over training complexity.
}
%\fontsize{9}{11}\selectfont
\begin{tabularx}{0.7\textwidth}{
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X}
\hline
%\multicolumn{1}{l|}{\multirow{2}{*}[0mm]{Methods}} & Classification & Person re-ID \\ \cline{2-3}
\multicolumn{1}{l|}{Methods} & Millisec / iter & Accuracy boost (\%) \\
\hline
%\multicolumn{1}{l|}{IN~\cite{ulyanov2016instance}} & \checkmark & \checkmark & 74.0\\
\multicolumn{1}{l|}{ResNet-18} & 34 & -\\
\multicolumn{1}{l|}{MixStyle~\cite{zhou2021domain}} & 35 & 4.2 \\
\multicolumn{1}{l|}{\ccol Ours} & \ccol 37 & \ccol 6.9\\
\hline
\end{tabularx}
\label{tab:supple_convergence}
% \vspace{-4mm}
\end{table}
\subsubsection{Impact of hyperparameters.}
We empirically investigated the effect of the two hyperparameters of XDED: $\lambda$ and $\tau$. We examine accuracy of multi-source domain generalization on Sketch of the PACS dataset by varying the values of the hyperparameters $\lambda \in \{ 0.5, 1.0, 2.5, 5.0, 10.0 \} $ and $\tau \in \{ 1.0, 2.0, 4.0, 6.0, 8.0 \} $. The results are summarized in Fig.~\ref{fig:supple_hyperparam}. First, the performance boost is the highest when $ \tau $ is increased from 1.0 to 2.0. However, if $\tau$ exceeds 2.0, the performance boost is somewhat reduced. Next, $\lambda$ also affects the performance. The performance boost commonly increases with the value of $\lambda$. However, the degree of boost is also reduced when $\tau$ is 1.0. To sum it up, the accuracy of our method is generally high and stable when $\lambda$ and $\tau$ are greater than 5.0 and 2.0, and we argue that this result supports our view that our framework does not strongly depend on the specific hyperparameter setting.
\input{table/comparison_ensemble}
\begin{figure}[t]
\centering
% \includegraphics[width=1.\textwidth]{figure/fig1.pdf}
\includegraphics[width=0.9\textwidth]{supple/supple_figure/sketch_ablation_hyperparam.pdf}
\caption{Multi-source domain generalization accuracy (\%) versus hyperparameters $\lambda$ and $\tau$ on Sketch of the PACS dataset.
}
\label{fig:supple_hyperparam}
\end{figure}
\begin{figure*}[t]
\centering
\includegraphics[width=1.0\textwidth]{supple/supple_figure/deformation_sero.pdf}
\caption{Test accuracy (\%) on target domains with input deformations. All models are trained under the multi-source domain generalization setting.
}
\label{fig:supple_deformation}
\end{figure*}
\subsubsection{Robustness to input deformations.}
\label{sec:supple_input_deformations}
To further assess the quality of the robustness to unseen domains, we measure the changes in test accuracy as the domain gap becomes larger. We simulate larger domain gaps by increasing the degree of input deformations. Specifically, we applied multiple augmentations defined in RandAugment~\cite{cubuk2020randaugment} to images of the unseen target domain (\ie, Cartoon of PACS), and gradually increased the number of deformations\footnote{For a fair comparison, we use the same operations for each setting.}. Similar to the result of table 9 in our main paper, Fig.~\ref{fig:supple_deformation} shows that our method outperforms baseline methods against pixel-level deformations on unseen domains.
% Additional results are presented in the supplementary material.
\section{Additional analysis results on UniStyle}
\label{sec:supple_unistyle}
\input{table/ablation_where_to_whiten}
\subsection{Where to apply UniStyle}
% all the quantitative results above are produced with the ResNet~\citep{resnet} as a backbone network and
We remark that UniStyle can be applied after arbitrary intermediate layers of the backbone (\ie, ResNet~\cite{resnet}) as a plug-and-play module. Therefore, to investigate the impact of where UniStyle is applied, we evaluate the generalization performance in image classification and person re-ID while varying the locations of where the operation is applied. For a baseline, MixStyle~\cite{zhou2021domain} is adopted and compared with our UniStyle since they share the commonality of being applied to multiple intermediate feature maps.
For brevity, let \textsc{res\#} denote the indexes of residual blocks where the specified operation is applied (\eg, \textsc{res12} means the operation is applied after both the first and second residual blocks). As shown in Table.~\ref{tab:ablation_where_to_whiten}, we observe that UniStyle and MixStyle have a similar trend in both tasks. First, since early layers are known to capture low-level features such as texture or edge information, it is pertinent for both UniStyle and MixStyle to be applied after early layers to remove style bias and synthesize novel styles, respectively. However, UniStyle achieves the best performances on both tasks, which indicates the importance of removing and unifying style bias rather than augmenting novel styles. Next, on the contrary, both operations lead critical performance drop when incorporated with the last layer, \textsc{res4}, since late layers are known to address semantic information (\ie, their statistics would be highly correlated with target labels). In detail, MixStyle perturbs the statistics by interpolating those of two different instances that may have different labels, whereas UniStyle normalizes the feature map of all samples regardless of their labels, resulting in a more critical performance drop. Note that XDED is excluded in results of Table.~\ref{tab:ablation_where_to_whiten}.
\subsection{Comparison with other methods}
Even though UniStyle is not the first that addresses feature-level statistics (\ie, style information), we take notice of its appealing trade-off between performance boost and training complexity and compare it with other related methods. The results are summarized in Table~\ref{tab:supple_comparison_unistyle}. First, UniStyle outperforms SagNet~\cite{nam2021reducing} and MixStyle~\cite{zhou2021domain}, all of which are dedicated to domain generalization. This indicates that directly removing and unifying the style bias is more effective for generalization ability than augmenting feature statistics or learning style-agnostic features.
Next, although BIN~\cite{nam2018batch} and SRM~\cite{lee2019srm} adaptively recalibrate feature statistics and achieve state-of-the-art performance in image recognition tasks, they show inferior results on unseen domains. Last but not least, Deep Whitening Transformation (DWT)~\cite{cho2019image} which is the most similar to UniStyle shows the second-best performance. DWT has been proposed to encourage the feature extractor to naturally encode the whitened feature by enforcing consistency regularization between the covariance matrix of the feature and identity matrix. Nevertheless, as a constraint that is too strong, DWT weakens feature discrimination and excessively throws away content information, resulting in performance that lags behind UniStyle.
\input{table/comparison_unistyle}
\section{Implementation details}
\label{sec:supple_other_materials}
\label{sec:supple_implementation}
For the task of domain generalization in image classification, we train the models using the sgd optimizer with the cosine learning decay~\cite{loshchilov2016sgdr} and initial learning rate of $10^{-3}$. They are learned for 100 epochs. For batch construction, we use the batch size of 64, and sample 16 instances per class for the proposed XDED. For the task of person re-ID, we also train the models using the sgd optimizer with initial learning rate of 0.05.
%\subsection{Source code}
%Our code materials to reproduce XDED are located in \q{code\_xded} of \q{supple\_ECCV22\_Cross\_Domain\_Ensemble\_Distillation.zip}. We hope that \q{README.md} helps understand to execute the submitted script.
\begin{table}[!t]
\centering
\caption{
Leave-one-domain-out generalization results on PACS and OfficeHome.
}
% \fontsize{8}{10}\selectfont
% \begin{tabularx}{0.75 \textwidth}{
\fontsize{8}{10}\selectfont
\begin{tabularx}{1.0 \textwidth}{
>{\centering\arraybackslash}X|
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
}
\hline
\multicolumn{1}{l|}{} &
\multicolumn{5}{c|}{PACS} & \multicolumn{5}{c}{OfficeHome} \\
\hline
%\multicolumn{1}{l|}{Methods} & {Entropy} & {Accuracy} & {Entropy} & {Accuracy} \\
\multicolumn{1}{l|}{Methods} & {Art} & {Cartoon} & {Photo} & {Sketch} & \multicolumn{1}{|c|}{Avg.} & {Artistic} & {Clipart} & {Product} & {Real} & \multicolumn{1}{|c}{Avg.} \\
\hline
\multicolumn{1}{l|}{ResNet-18} & {77.0} & {75.9} & {96.0} & {69.2} & \multicolumn{1}{|c|}{79.5} & {58.9} & {49.4} & {74.3} & {76.2} & \multicolumn{1}{|c}{64.7} \\
\multicolumn{1}{l|}{MMD-AE~\cite{li2018domain}} & {75.2} & {72.7} & {96.0} & {64.2} & \multicolumn{1}{|c|}{77.0} & {56.5} & {47.3} & {72.1} & {74.8} & \multicolumn{1}{|c}{62.7} \\
\multicolumn{1}{l|}{JiGen~\cite{carlucci2019domain}} & {79.4} & {75.3} & {96.0} & {71.6} & \multicolumn{1}{|c|}{80.5} & {53.0} & {47.4} & {71.4} & {72.7} & \multicolumn{1}{|c}{61.2} \\
\multicolumn{1}{l|}{CrossGrad~\cite{shankar2018generalizing}} & {79.8} & {76.8} & {96.0} & {70.2} & \multicolumn{1}{|c|}{80.7} & {58.4} & {49.4} & {73.9} & {75.8} & \multicolumn{1}{|c}{64.4} \\
\multicolumn{1}{l|}{MASF~\cite{dou2019domain}} & {80.2} & {77.1} & {94.9} & {71.6} & \multicolumn{1}{|c|}{81.0} & {-} & {-} & {-} & {-} & \multicolumn{1}{|c}{-} \\
\multicolumn{1}{l|}{Epi-FCR~\cite{li2019episodic}} & {82.1} & {77.0} & {93.9} & {73.0} & \multicolumn{1}{|c|}{81.5} & {-} & {-} & {-} & {-} & \multicolumn{1}{|c}{-} \\
\multicolumn{1}{l|}{EISNet~\cite{wang2020learning}} & {81.8} & {76.4} & {95.9} & {74.3} & \multicolumn{1}{|c|}{82.1} & {-} & {-} & {-} & {-} & \multicolumn{1}{|c}{-} \\
\multicolumn{1}{l|}{L2A-OT~\cite{zhou2020learning}} & {83.3} & {78.2} & {\underline{96.2}} & {73.6} & \multicolumn{1}{|c|}{82.8} & {\underline{60.6}} & {50.1} & {\underline{74.8}} & {\textbf{77.0}} & \multicolumn{1}{|c}{65.6} \\
\multicolumn{1}{l|}{SagNet~\cite{nam2021reducing}} & {83.5} & {77.6} & {95.4} & {76.3} & \multicolumn{1}{|c|}{83.2} & {60.2} & {45.3} & {70.4} & {73.3} & \multicolumn{1}{|c}{62.3} \\
\multicolumn{1}{l|}{SelfReg~\cite{kim2021selfreg}} & {82.3} & {78.4} & {\underline{96.2}} & {77.4} & \multicolumn{1}{|c|}{83.6} & {-} & {-} & {-} & {-} & \multicolumn{1}{|c}{-} \\
\multicolumn{1}{l|}{MixStyle~\cite{zhou2021domain}} & {84.1} & {78.8} & {96.1} & {75.9} & \multicolumn{1}{|c|}{83.7} & {58.7} & {53.4} & {74.2} & {75.9} & \multicolumn{1}{|c}{65.5} \\
\multicolumn{1}{l|}{L2D~\cite{wang2021learning}} & {81.4} & {79.5} & {95.5} & {80.5} & \multicolumn{1}{|c|}{84.2} & {-} & {-} & {-} & {-} & \multicolumn{1}{|c}{-} \\
\multicolumn{1}{l|}{FACT~\cite{xu2021fourier}} & {\underline{85.3}} & {78.3} & {95.1} & {79.1} & \multicolumn{1}{|c|}{84.5} & {60.3} & {54.8} & {74.4} & {\underline{76.5}} & \multicolumn{1}{|c}{\underline{66.5}} \\
\multicolumn{1}{l|}{DSON~\cite{seo2020learning}} & {84.6} & {77.6} & {95.8} & {\underline{82.2}} & \multicolumn{1}{|c|}{85.1} & {59.3} & {45.7} & {71.8} & {74.6} & \multicolumn{1}{|c}{62.9} \\
\multicolumn{1}{l|}{RSC~\cite{huang2020self}} & {83.4} & {\underline{80.3}} & {95.9} & {80.8} & \multicolumn{1}{|c|}{85.1} & {58.4} & {47.9} & {71.6} & {74.5} & \multicolumn{1}{|c}{63.1} \\
\multicolumn{1}{l|}{StyleNeophile~\cite{StyleNeophile}} & {84.4} & {79.2} & {94.9} & {\textbf{83.2}} & \multicolumn{1}{|c|}{\underline{85.4}} & {59.5} & {\underline{55.0}} & {73.5} & {75.5} & \multicolumn{1}{|c}{65.8} \\
\hline
\multicolumn{1}{l|}{\ccol Ours} & {\ccol \textbf{85.6}} & {\ccol \textbf{84.2}} & {\ccol \textbf{96.5}} & {\ccol 79.1} & \multicolumn{1}{|c|}{\ccol \textbf{86.4}} & {\ccol \textbf{60.8}} & {\ccol \textbf{57.1}} & {\ccol \textbf{75.3}} & {\ccol \underline{76.5}} & \multicolumn{1}{|c}{\ccol \textbf{67.4}} \\
\hline
\end{tabularx}
%\vspace{-3mm}
\label{tab:cr_pacs_officehome}
\end{table}
\begin{table}[!t]
\centering
\caption{
Ablation study of the proposed components on cross-domain tasks of image classification (Accuracy) and person re-ID (mAP).
}
\fontsize{8}{10}\selectfont
\begin{tabularx}{0.8 \textwidth}{
>{\centering\arraybackslash}X
>{\centering\arraybackslash\hsize=.5\hsize}X
>{\centering\arraybackslash\hsize=.5\hsize}X|
>{\centering\arraybackslash}X}
%\toprule
\hline
\multicolumn{1}{l|}{\multirow{1}{*}{Methods}}& \multicolumn{1}{c}{Art} & \multicolumn{1}{c|}{Clipart} & {Market $\rightarrow$ Duke}\\
% \multicolumn{1}{l|}{}& \multicolumn{1}{c}{Accuracy}&
% \multicolumn{1}{c|}{Accuracy} & {mAP} \\
%\cline{2-4}
% \cmidrule(l){2-3} \cmidrule(l){4-4}
% \hline
% \midrule
\hline
\multicolumn{1}{l|}{Vanilla} & 77.0 & 49.4 & 19.3 \\
%\hline
\multicolumn{1}{l|}{w/ UniStyle} & 81.2 & 50.4 & \underline{26.2} \\
%\hline
\multicolumn{1}{l|}{w/ XDED} & \underline{83.3} & \underline{55.2} & 24.2 \\
%\hline
\multicolumn{1}{l|}{\ccol Ours} & \ccol \textbf{85.6} & \ccol \textbf{57.1} & \ccol \textbf{27.4} \\
%\bottomrule
\hline
\end{tabularx}
\label{tab:ablation_components}
%\label{tab:dml_scratch}
%\vspace{-2mm}
\end{table}
\begin{table}
\centering
\fontsize{9}{11}\selectfont
\begin{tabularx}{0.48 \textwidth}{
p{0.11\textwidth}>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
\toprule
%\multicolumn{1}{l}{\multirow{2}{*}[-3.5mm]{Methods}}&
%\multicolumn{1}{c}{PACS} &
%\multicolumn{1}{c}{Office-Home} &
%\multicolumn{1}{c}{Person Re-ID} \\
\multicolumn{1}{l}{\multirow{2}{*}[-3.5mm]{Methods}}&
\multicolumn{2}{c}{Accuracy} &
\multicolumn{1}{c}{mAP} \\
%\cline{2-4}
\cmidrule(l){2-3} \cmidrule(l){4-4}
& \multicolumn{1}{c}{PACS} & \multicolumn{1}{c}{Office-Home} & \multicolumn{1}{c}{M$\rightarrow$D}\\
\midrule
\multicolumn{1}{l}{Ours} & 86.4 & (66.9) & 27.4 \\
\multicolumn{1}{l}{w/o UniStyle} & (82.8) & 66.9 & (24.2) \\
\multicolumn{1}{l}{w/o XDED} & 82.9 & (64.5) & 26.2 \\
\bottomrule
\end{tabularx}
\vspace{-2mm}
\caption{
asdf Ablation study of the each component in our framework. (M: Market1501, D:Duke)
}
\vspace{-2mm}
\end{table}
%\begin{table}[!htb]
\begin{table*}[!t]
\caption{Ablation study on where to apply UniStyle in the ResNet architecture.}
%\vspace{-2mm}
\footnotesize
%\begin{tabularx}{0.6\textwidth}{
\begin{subtable}{.5\linewidth}
\centering
\caption{Image classification on PACS}
\begin{tabular}{l c}
\hline
%\backslashbox{Method}{Model} & ReNet-18 & + res1
Model & Accuracy (\%) \\
\hline
ResNet-18 & 79.5 \\
\hline
+ MixStyle (\textsc{res1}) & 80.1 \\
+ UniStyle (\textsc{res1}) & 81.5 \\
\hline
+ MixStyle (\textsc{res12}) & 81.6 \\
+ UniStyle (\textsc{res12}) & \textbf{82.9} \\
\hline
+ MixStyle (\textsc{res123}) & 82.8 \\
+ UniStyle (\textsc{res123}) & 82.4 \\
\hline
+ MixStyle (\textsc{res1234}) & 75.6 \\
+ UniStyle (\textsc{res1234}) & 12.8 \\
%\hline
%+ MixStyle (\textsc{res14) & 76.3 \\
%+ UniStyle (\textsc{res14) & - \\
%\hline
%+ MixStyle (\textsc{res23) & 81.7 \\
%+ UniStyle (\textsc{res23) & - \\
\hline
\end{tabular}
\end{subtable}%
\begin{subtable}{.5\linewidth}
\centering
\caption{Person re-ID from Market1501$\rightarrow$Duke}
\begin{tabular}{ll}
\hline
%\backslashbox{Method}{Model} & ReNet-18 & + res1
Model & mAP (\%) \\
\hline
ResNet-50 & 19.3 \\
\hline
+ MixStyle (\textsc{res1}) & 22.6 \\
+ UniStyle (\textsc{res1}) & 23.3 \\
\hline
+ MixStyle (\textsc{res12}) & 23.8 \\
+ UniStyle (\textsc{res12}) & 24.1 \\
\hline
+ MixStyle (\textsc{res123}) & 22.0 \\
+ UniStyle (\textsc{res123}) & \textbf{26.2} \\
\hline
+ MixStyle (\textsc{res1234}) & 10.2 \\
+ UniStyle (\textsc{res1234}) & 0.2 \\
%\hline
%res14 + MixStyle & 11.1 \\
%res14 + UniStyle & - \\
%\hline
%res23 + MixStyle & 20.6 \\
%res23 + UniStyle & - \\
\hline
\end{tabular}
\end{subtable}
\label{tab:ablation_where_to_whiten}
\end{table*}
\begin{table}[!t]
\centering
\caption{
Multi-source domain generalization accuracy (\%) on Photo of PACS before and after applying given adversarial attacks.
}
\fontsize{8}{10}\selectfont
\begin{tabularx}{0.75 \textwidth}{
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
\hline
\multicolumn{1}{p{33mm}|}{Methods} & Photo & w/ FGSM & w/ PGD \\ % \multicolumn{1}{|c}{Avg.} \\
\hline
%\multicolumn{1}{l|}{FGSM training} & 76.2 & \textbf{67.7} & \multicolumn{1}{|c}{71.9}\\ \hline
\multicolumn{1}{l|}{ResNet-18} & 96.0 & 39.6 & 16.3\\
\multicolumn{1}{l|}{Label smoothing~\cite{szegedy2016rethinking}} & 95.6 & 43.5 & 20.2\\
%\multicolumn{1}{l|}{Entropy maximization~\cite{pereyra2017regularizing}} & 94.8 & 55.1 & \multicolumn{1}{|c}{74.9}\\
\multicolumn{1}{l|}{Mixup~\cite{zhang2017mixup}} & 95.8 & 46.5 & 21.9\\
\multicolumn{1}{l|}{Manifold mixup~\cite{verma2019manifold}} & 93.5 & \underline{46.6} & \underline{23.8}\\
\multicolumn{1}{l|}{MixStyle~\cite{zhou2021domain}} & \underline{96.1} & 41.4 & 22.7\\
%\multicolumn{1}{l|}{PGD training} & - & - & \multicolumn{1}{|c}{-}\\
\multicolumn{1}{l|}{ \ccol Ours} & \ccol \textbf{96.5} & \ccol \textbf{55.4} & \ccol \textbf{30.4}\\
\hline
\end{tabularx}
\label{tab:comparison_adversarial_attack}
\vspace{-2mm}
\end{table}
\begin{table}[!t]
%\vspace{-1.0mm}
\centering
\caption{
Average classification error (\%) on the corruption benchmarks.
%Corruption robustness results. Mean corruption error (mCE) values are calculated over various corruption types.
}
\fontsize{8}{10}\selectfont
\begin{tabularx}{0.7 \textwidth}{
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
\hline
\multicolumn{1}{p{21mm}|}{Methods} & CIFAR-10-C & CIFAR-100-C \\
\hline
\multicolumn{1}{l|}{40-2 WRN~\cite{zagoruyko2016wide}} & 26.9 & 53.3 \\
\multicolumn{1}{l|}{Cutout~\cite{devries2017improved}} & 26.8 & 53.5 \\
\multicolumn{1}{l|}{Mixup~\cite{zhang2017mixup}} & 22.3 & 50.4 \\
\multicolumn{1}{l|}{CutMix~\cite{yun2019cutmix}} & 27.1 & 52.9 \\
\multicolumn{1}{l|}{AutoAug~\cite{cubuk2018autoaugment}} & 23.9 & 49.6 \\
\multicolumn{1}{l|}{AugMix~\cite{hendrycks2019augmix}} & \textbf{11.2} & \textbf{35.9} \\
\multicolumn{1}{l|}{ \ccol Ours} & \ccol \underline{18.5} & \ccol \underline{46.6} \\
\hline
\end{tabularx}
\label{tab:comparison_corruption}
%\vspace{-2mm}
\end{table}
\begin{table}[!t]
\centering
\fontsize{9}{11}\selectfont
\caption{
% Comparison of source embedding and ablations of our method on the CUB-200-2011 (CUB) and Cars-196 (Cars) datasets.
Comparison with other ensemble methods which are dedicated to domain generalization. Extra $\theta$ and Domain ID denote whether each method requires additional modules and domain labels are explicitly bound to be given, respectively. Accuracy denotes the average accuracy of each method over PACS and Office-Home.
}
\begin{tabularx}{0.8\textwidth}{
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X}
\hline
%\multicolumn{1}{l|}{\multirow{2}{*}[0mm]{Methods}} & Classification & Person re-ID \\ \cline{2-3}
\multicolumn{1}{l|}{Methods} & Extra $\theta$ & Domain ID & Accuracy \\
\hline
%\multicolumn{1}{l|}{DE~\cite{lakshminarayanan2017simple}} & \ding{51} & \ding{55} & -\\
\multicolumn{1}{l|}{DAEL~\cite{zhou2021dael}} & \ding{51} & \ding{51} & 74.7\\
\multicolumn{1}{l|}{DSON~\cite{seo2020learning}} & \ding{51} & \ding{51} & 74.0\\
\multicolumn{1}{l|}{\ccol Ours} & \ccol \ding{55} & \ccol \ding{55} & \ccol 76.9\\
%& Accuracy(\%) & mAP(\%)\\
% \multicolumn{1}{l|}{\emph{Source}: PA} & 69.1 & 86.4\\ %\hline
%& Accuracy & mAP \\ \hline
%\multicolumn{1}{l|}{Vanilla} & 70.4 & 88.2\\
%\multicolumn{1}{l|}{+ UniStyle} & 71.5 & 89.0\\
%\multicolumn{1}{l|}{+ XDED} & 72.1 & 89.6\\
\hline
\end{tabularx}
\label{tab:comparison_ensemble}
% \vspace{-4mm}
\end{table}
\begin{table}%[!t]
\centering
\fontsize{9}{11}\selectfont
\begin{tabularx}{0.48\textwidth}{
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X}
\hline
%\multicolumn{1}{l|}{\multirow{2}{*}[0mm]{Methods}} & Classification & Person re-ID \\ \cline{2-3}
\multicolumn{1}{l|}{Methods} & Extra $\theta$ & Domain ID & Accuracy \\
\hline
%\multicolumn{1}{l|}{DE~\cite{lakshminarayanan2017simple}} & \checkmark & \ding{56} & -\\
\multicolumn{1}{l|}{DAEL~\cite{zhou2021dael}} & \checkmark & \checkmark & 74.7\\
\multicolumn{1}{l|}{DSON~\cite{seo2020learning}} & \checkmark & \checkmark & 74.0\\
\multicolumn{1}{l|}{\ccol Ours} & \ccol \ding{56} & \ccol \ding{56} & \ccol 76.9\\
%& Accuracy(\%) & mAP(\%)\\
% \multicolumn{1}{l|}{\emph{Source}: PA} & 69.1 & 86.4\\ %\hline
%& Accuracy & mAP \\ \hline
%\multicolumn{1}{l|}{Vanilla} & 70.4 & 88.2\\
%\multicolumn{1}{l|}{+ UniStyle} & 71.5 & 89.0\\
%\multicolumn{1}{l|}{+ XDED} & 72.1 & 89.6\\
\hline
\end{tabularx}
\vspace{0.1mm}
\caption{
% Comparison of source embedding and ablations of our method on the CUB-200-2011 (CUB) and Cars-196 (Cars) datasets.
Comparison with other ensemble methods which are dedicated to domain generalization. Extra $\theta$ and Domain ID denote whether each method requires additional networks and domain labels are bound to be given, respectively. Accuracy denotes the average accuracy of each method over PACS and Office-Home.
}
% \vspace{-4mm}
\end{table}
\begin{table}[!t]
\centering
\caption{
Comparison of the entropy values. When each model is converged, the entropy value is calculated by averaging over all training samples.
}
\fontsize{8.5}{10.5}\selectfont
\begin{tabularx}{0.7 \textwidth}{
>{\centering\arraybackslash}X|
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X|
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
\hline
\multicolumn{1}{l|}{} &
\multicolumn{2}{c|}{OfficeHome (Clipart)} & \multicolumn{2}{c}{PACS (Cartoon)} \\
\multicolumn{1}{l|}{Methods} & {Entropy} & {Accuracy} & {Entropy} & {Accuracy} \\
\hline
\multicolumn{1}{l|}{ResNet-18} & 0.25 & 49.4 & 0.01 & 75.9\\
\multicolumn{1}{l|}{MixStyle~\cite{zhou2021domain}} & 0.35 & 53.4 & 0.03 & 78.8 \\
\multicolumn{1}{l|}{\ccol XDED} & \ccol \textbf{0.92} & \ccol \textbf{55.2} & \ccol \textbf{0.38} & \ccol \textbf{81.7} \\
\hline
\end{tabularx}
\label{tab:supple_comparison_entropy}
\end{table}
\begin{table}[!t]
\centering
\caption{
Comparison to existing flat minima-promoting methods on Clipart of OfficeHome in the leave-one-domain-out setting. Note that UniStyle is not applied to the result of XDED.
% Domain generalization results on Clipart of OfficeHome.
}
\fontsize{7}{9}\selectfont
\begin{tabularx}{1.0 \textwidth}{
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
\hline
\multicolumn{1}{l|}{Methods} & Vanilla & MaxEnt~\cite{pereyra2017regularizing} & DML~\cite{zhang2018deep} & SAM~\cite{foret2021sharpness} & SWA~\cite{izmailov2018averaging} & SWAD~\cite{cha2021swad} & \ccol XDED \\
\hline
\multicolumn{1}{l|}{Accuracy (\%)} & 49.4 & 51.7 & 51.8 & 51.9 & 52.3 & \underline{52.9} & \ccol \textbf{55.2} \\
% sec/iter
\hline
\end{tabularx}
\label{tab:comparison_flat_minima}
\vspace{-4mm}
\end{table}
\begin{table}[!t]
\centering
\caption{
Accuracy (\%) on general image recognition benchmarks.
}
%\fontsize{9}{11}\selectfont
\begin{tabularx}{0.8 \textwidth}{
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
\hline
\multicolumn{1}{l|}{Methods} & CIFAR-10 & CIFAR-100 \\
\hline
\multicolumn{1}{l|}{ResNet-18} & 94.6 & 75.2 \\
\multicolumn{1}{l|}{CS-KD~\cite{yun2020regularizing}} & 94.6 & 78.0 \\
\multicolumn{1}{l|}{BYOT~\cite{zhang2019your}} & \underline{95.1} & \underline{78.6} \\
\multicolumn{1}{l|}{ \ccol XDED} & \ccol \textbf{95.3} & \ccol \textbf{78.7} \\
\hline
\end{tabularx}
\label{tab:comparison_iid}
%\vspace{-2mm}
\end{table}
\begin{table}%[!t]
\centering
\begin{tabularx}{0.5\textwidth}{
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X}
\hline
%\multicolumn{1}{l|}{\multirow{2}{*}[0mm]{Methods}} & Classification & Person re-ID \\ \cline{2-3}
%Methods & Top-1 \downarrow & Top-5 \downarrow & NLL \downarrow & ECE \downarrow \\
Methods & NLL & OE (\%) & ECE (\%) & TCOM (\%) \\
\hline
%\multicolumn{1}{l|}{DE~\cite{lakshminarayanan2017simple}} & \checkmark & \ding{56} & -\\
\multicolumn{1}{l|}{Cross-Entropy} & 0.92 & 8.0 & 10.43 & 13.89 \\
\multicolumn{1}{l|}{Label smoothing} & 0.92 & 0.19 & \textbf{1.68} & 17.37 \\
\multicolumn{1}{l|}{Mixup} & 1.12 & \textbf{0.0} & 11.53 & 17.25 \\
\multicolumn{1}{l|}{Manifold} & 1.02 & 9.24 & 11.37 & 12.84 \\
\multicolumn{1}{l|}{Entropy-Maximization} &1.75 & \textbf{0.0} & 37.87 & 15.13 \\
\multicolumn{1}{l|}{Ours} & \textbf{0.70} & \textbf{0.0} & 15.79 & \textbf{20.95} \\
\hline
\end{tabularx}
\vspace{0.1mm}
\caption{
Top-K error, negative log-likelihood (NLL) and expected calibration error (ECE) on Art Painting of PACS.%
-Home.
}
\label{tab:comparison_soft_targets}
% \vspace{-4mm}
\end{table}
%\caption{Top-K error, negative log-likelihood (NLL) and expected calibration error (ECE) on the Office-Home.}
%\label{tab:comparison_soft_targets}
\begin{table}[!t]
\centering
\caption{
Comparison between UniStyle and other related methods. Extra $\theta$ and Extra $\mathcal{L}$ denote whether each method requires additional modules and loss function, respectively, which both increase training complexity and resources. Accuracy denotes the multi-source domain generalization accuracy (\%) of each method on the target domain, Cartoon of the PACS dataset.
}
%\fontsize{9}{11}\selectfont
\begin{tabularx}{0.85\textwidth}{
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X |
>{\centering\arraybackslash}X}
\hline
%\multicolumn{1}{l|}{\multirow{2}{*}[0mm]{Methods}} & Classification & Person re-ID \\ \cline{2-3}
\multicolumn{1}{l|}{Methods} & Extra $\theta$ & Extra $\mathcal{L}$ & Accuracy \\
\hline
%\multicolumn{1}{l|}{IN~\cite{ulyanov2016instance}} & \ding{51} & \ding{51} & 74.0\\
\multicolumn{1}{l|}{ResNet-18} & \ding{55} & \ding{55} & 75.9\\
\hline
\multicolumn{4}{c}{Domain Generalization}\\
\hline
\multicolumn{1}{l|}{SagNet~\cite{nam2021reducing}} & \ding{51} & \ding{51} & 77.6\\
\multicolumn{1}{l|}{MixStyle~\cite{zhou2021domain}} & \ding{55} & \ding{55} & 78.8\\
\hline
\multicolumn{4}{c}{Style Transfer}\\
\hline
\multicolumn{1}{l|}{BIN~\cite{nam2018batch}} & \ding{51} & \ding{55} & 78.7\\
\multicolumn{1}{l|}{SRM~\cite{lee2019srm}} & \ding{51} & \ding{55} & 78.9\\
\multicolumn{1}{l|}{DWT~\cite{cho2019image}} & \ding{55} & \ding{51} & \underline{79.0}\\
% \multicolumn{1}{l|}{RobustNet~\cite{choi2021robustnet}} & \ding{51} & \ding{51} & 74.0\\
\hline
\multicolumn{1}{l|}{\ccol UniStyle} & \ccol \ding{55} & \ccol \ding{55} & \ccol \textbf{79.2}\\
\hline
\end{tabularx}
\label{tab:supple_comparison_unistyle}
% \vspace{-4mm}
\end{table}
\begin{table*}[!t]
\caption{Ablation study on the proposed components.}
\vspace{-2mm}
\footnotesize
\begin{subtable}{.5\linewidth}
\centering
\caption{Image classification on PACS}
\begin{tabular}{l c}
\hline
%\backslashbox{Method}{Model} & ReNet-18 & + res1
Model & Accuracy (\%) \\
\hline
ResNet-18 & 79.5 \\
+ UniStyle & 82.9 \\
+ UniStyle + XDED & 86.4 \\
\hline
\end{tabular}
\end{subtable}%
\begin{subtable}{.5\linewidth}
\centering
\caption{Person re-ID from Market1501$\rightarrow$Duke}
\begin{tabular}{ll}
\hline
%\backslashbox{Method}{Model} & ReNet-18 & + res1
Model & mAP (\%) \\
\hline
ResNet-50 & 19.3 \\
+ UniStyle & 24.0 \\
+ UniStyle $\&$ XDED & 27.4 \\
\hline
\end{tabular}
\end{subtable}
\label{tab:ablation_component}
\end{table*}
\begin{table*}
\centering
%\resizebox{\textwidth}{!}{
\adjustbox{max width=\textwidth}{
\begin{tabular}{lcccccccc}
%\toprule
\hline
\textbf{Algorithm} & \textbf{ColoredMNIST} & \textbf{RotatedMNIST} & \textbf{VLCS} & \textbf{PACS} & \textbf{OfficeHome} & \textbf{TerraIncognita} & \textbf{DomainNet} & \textbf{Avg} \\
%\midrule
%\midrule
\hline
ERM & 57.8 $\pm$ 0.2 & 97.8 $\pm$ 0.1 & 77.6 $\pm$ 0.3 & 86.7 $\pm$ 0.3 & 66.4 $\pm$ 0.5 & 53.0 $\pm$ 0.3 & 41.3 $\pm$ 0.1 & 68.7 \\
IRM & 67.7 $\pm$ 1.2 & 97.5 $\pm$ 0.2 & 76.9 $\pm$ 0.6 & 84.5 $\pm$ 1.1 & 63.0 $\pm$ 2.7 & 50.5 $\pm$ 0.7 & 28.0 $\pm$ 5.1 & 66.9 \\
GroupDRO & 61.1 $\pm$ 0.9 & 97.9 $\pm$ 0.1 & 77.4 $\pm$ 0.5 & 87.1 $\pm$ 0.1 & 66.2 $\pm$ 0.6 & 52.4 $\pm$ 0.1 & 33.4 $\pm$ 0.3 & 67.9 \\
Mixup & 58.4 $\pm$ 0.2 & 98.0 $\pm$ 0.1 & 78.1 $\pm$ 0.3 & 86.8 $\pm$ 0.3 & 68.0 $\pm$ 0.2 & 54.4 $\pm$ 0.3 & 39.6 $\pm$ 0.1 & 69.0 \\
MLDG & 58.2 $\pm$ 0.4 & 97.8 $\pm$ 0.1 & 77.5 $\pm$ 0.1 & 86.8 $\pm$ 0.4 & 66.6 $\pm$ 0.3 & 52.0 $\pm$ 0.1 & 41.6 $\pm$ 0.1 & 68.7 \\
CORAL & 58.6 $\pm$ 0.5 & 98.0 $\pm$ 0.0 & 77.7 $\pm$ 0.2 & 87.1 $\pm$ 0.5 & 68.4 $\pm$ 0.2 & 52.8 $\pm$ 0.2 & 41.8 $\pm$ 0.1 & 69.2 \\
MMD & 63.3 $\pm$ 1.3 & 98.0 $\pm$ 0.1 & 77.9 $\pm$ 0.1 & 87.2 $\pm$ 0.1 & 66.2 $\pm$ 0.3 & 52.0 $\pm$ 0.4 & 23.5 $\pm$ 9.4 & 66.9 \\
DANN & 57.0 $\pm$ 1.0 & 97.9 $\pm$ 0.1 & 79.7 $\pm$ 0.5 & 85.2 $\pm$ 0.2 & 65.3 $\pm$ 0.8 & 50.6 $\pm$ 0.4 & 38.3 $\pm$ 0.1 & 67.7 \\
CDANN & 59.5 $\pm$ 2.0 & 97.9 $\pm$ 0.0 & 79.9 $\pm$ 0.2 & 85.8 $\pm$ 0.8 & 65.3 $\pm$ 0.5 & 50.8 $\pm$ 0.6 & 38.5 $\pm$ 0.2 & 68.2 \\
MTL & 57.6 $\pm$ 0.3 & 97.9 $\pm$ 0.1 & 77.7 $\pm$ 0.5 & 86.7 $\pm$ 0.2 & 66.5 $\pm$ 0.4 & 52.2 $\pm$ 0.4 & 40.8 $\pm$ 0.1 & 68.5 \\
SagNet & 58.2 $\pm$ 0.3 & 97.9 $\pm$ 0.0 & 77.6 $\pm$ 0.1 & 86.4 $\pm$ 0.4 & 67.5 $\pm$ 0.2 & 52.5 $\pm$ 0.4 & 40.8 $\pm$ 0.2 & 68.7 \\
ARM & 63.2 $\pm$ 0.7 & 98.1 $\pm$ 0.1 & 77.8 $\pm$ 0.3 & 85.8 $\pm$ 0.2 & 64.8 $\pm$ 0.4 & 51.2 $\pm$ 0.5 & 36.0 $\pm$ 0.2 & 68.1 \\
VREx & 67.0 $\pm$ 1.3 & 97.9 $\pm$ 0.1 & 78.1 $\pm$ 0.2 & 87.2 $\pm$ 0.6 & 65.7 $\pm$ 0.3 & 51.4 $\pm$ 0.5 & 30.1 $\pm$ 3.7 & 68.2 \\
RSC & 58.5 $\pm$ 0.5 & 97.6 $\pm$ 0.1 & 77.8 $\pm$ 0.6 & 86.2 $\pm$ 0.5 & 66.5 $\pm$ 0.6 & 52.1 $\pm$ 0.2 & 38.9 $\pm$ 0.6 & 68.2 \\
%\bottomrule
\hline
\end{tabular}
}
\vspace{-2mm}
\caption{
DomainBed
}
\vspace*{-2mm}
\end{table*}
% \begin{table}
% \centering
% %\resizebox{\textwidth}{!}{
% %\adjustbox{max width=\textwidth}
% \resizebox{\columnwidth}{!}
% {
% \begin{tabular}{lccccccc}
% \hline
% \multicolumn{8}{c}{Model selection: leave-one-domain-out cross-validation} \\
% %\toprule
% \hline
% {Methods} & {CMNIST} & {RMNIST} & {VLCS} & {PACS} & {OfficeHome} & {TerraIncognita} & {Avg.} \\
% \hline
% %\midrule
% ERM & 36.7 & 97.7 & 77.2 & 83.0 & 65.7 & 41.4 & 66.9 \\
% IRM & 40.3 & 97.0 & 76.3 & 81.5 & 64.3 & 41.2 & 66.7 \\
% GroupDRO & 36.8 & 97.6 & \underline{77.9} & 83.5 & 65.2 & 44.9 & 66.7 \\
% Mixup & 33.4 & \underline{97.8} & 77.7 & 83.2 & 67.0 & \textbf{48.7} & 67.9 \\
% MLDG & 36.7 & 97.6 & 77.2 & 82.9 & 66.1 & 46.2 & 67.7 \\
% CORAL & 39.7 & \underline{97.8} & \textbf{78.7} & 82.6 & \textbf{68.5} & 46.3 & \textbf{68.9} \\
% MMD & 36.8 & \underline{97.8} & 77.3 & 83.2 & 60.2 & 46.5 & 66.9 \\
% DANN & \underline{40.7} & 97.6 & 76.9 & 81.0 & 64.9 & 44.4 & 67.5 \\
% CDANN & 39.1 & 97.5 & 77.5 & 78.8 & 64.3 & 39.9 & 66.1 \\
% MTL & 35.0 & \underline{97.8} & 76.6 & \underline{83.7} & 65.7 & 44.9 & 67.2 \\
% SagNet & 36.5 & 94.0 & 77.5 & 82.3 & \underline{67.6} & \underline{47.2} & 67.5 \\
% ARM & 36.8 & \textbf{98.1} & 76.6 & 81.7 & 64.4 & 42.6 & 66.7 \\
% VREx & 36.9 & 93.6 & 76.7 & 81.3 & 64.9 & 37.3 & 65.1 \\
% RSC & 36.5 & 97.6 & 77.5 & 82.6 & 65.8 & 40.0 & 66.6 \\
% \hline
% \ccol Ours & \ccol \textbf{46.5} & \ccol 97.7 & \ccol 74.8 & \ccol \textbf{83.8} & \ccol 65.0 & \ccol 42.5 & \ccol \underline{68.4} \\
% \hline
% \end{tabular}
% }
% \vspace{-2mm}
% \caption{
% Domain generalization accuracy (\%) on DomainBed. The results compare fifteen methods including ours across six domain generalization benchmark datasets. Note that we adopt leave-one-domain-out cross-validation as a model selection criteria.
% }
% \vspace*{-2mm}
% \label{tab:domainbed}
% \end{table}
\begin{table*}[!t]
\centering
%\resizebox{\textwidth}{!}{
%\adjustbox{max width=\textwidth}
% \resizebox{\columnwidth}{!}
% {
\caption{Domain generalization accuracy (\%) on DomainBed.
% The results compare fifteen methods including ours across six domain generalization benchmark datasets.
The column ``Terra'' stands for TerraIncognita dataset. Note that we adopt leave-one-domain-out cross-validation as a model selection criteria.
}
% \fontsize{8.7}{9.7}\selectfont
%\fontsize{6}{7}\selectfont
\adjustbox{max width=1.0\textwidth}
{
\fontsize{8}{10}\selectfont
\begin{tabularx}{1.\textwidth} {
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X|
>{\centering\arraybackslash}X
}
\hline
\multicolumn{8}{c}{Model selection: leave-one-domain-out cross-validation} \\
%\toprule
\hline
%\multicolumn{1}{l|}{Methods} & {CMNIST~\cite{arjovsky2019invariant}} & {RMNIST~\cite{ghifary2015domain}} & {VLCS~\cite{fang2013unbiased}} & {PACS~\cite{li2017deeper}} & {OfficeHome~\cite{venkateswara2017deep}} & {TerraIncognita~\cite{beery2018recognition}} & {Avg.} \\
\multicolumn{1}{l|}{Methods}
& {CMNIST} & {RMNIST} & {VLCS} & {PACS} & {OfficeHome} & {Terra} & {Avg.} \\
\hline
%\midrule
\multicolumn{1}{l|}{ERM~\cite{vapnik1998statistical}} & 36.7 & 97.7 & 77.2 & 83.0 & 65.7 & 41.4 & 66.9 \\
\multicolumn{1}{l|}{IRM~\cite{arjovsky2019invariant}} & 40.3 & 97.0 & 76.3 & 81.5 & 64.3 & 41.2 & 66.7 \\
\multicolumn{1}{l|}{GroupDRO~\cite{sagawa2020distributionally}} & 36.8 & 97.6 & \underline{77.9} & 83.5 & 65.2 & 44.9 & 66.7 \\
\multicolumn{1}{l|}{Mixup~\cite{zhang2017mixup}} & 33.4 & \underline{97.8} & 77.7 & 83.2 & 67.0 & \textbf{48.7} & 67.9 \\
\multicolumn{1}{l|}{MLDG~\cite{li2018mldg}} & 36.7 & 97.6 & 77.2 & 82.9 & 66.1 & 46.2 & 67.7 \\
\multicolumn{1}{l|}{CORAL~\cite{sun2016coral}} & 39.7 & \underline{97.8} & \textbf{78.7} & 82.6 & \textbf{68.5} & 46.3 & \textbf{68.9} \\
\multicolumn{1}{l|}{MMD~\cite{li2018domain}} & 36.8 & \underline{97.8} & 77.3 & 83.2 & 60.2 & 46.5 & 66.9 \\
\multicolumn{1}{l|}{DANN~\cite{ganin2016dann}} & \underline{40.7} & 97.6 & 76.9 & 81.0 & 64.9 & 44.4 & 67.5 \\
\multicolumn{1}{l|}{CDANN~\cite{li2018deep}} & 39.1 & 97.5 & 77.5 & 78.8 & 64.3 & 39.9 & 66.1 \\
\multicolumn{1}{l|}{MTL~\cite{blanchard2021domain}} & 35.0 & \underline{97.8} & 76.6 & \underline{83.7} & 65.7 & 44.9 & 67.2 \\
\multicolumn{1}{l|}{SagNet~\cite{nam2021reducing}} & 36.5 & 94.0 & 77.5 & 82.3 & \underline{67.6} & \underline{47.2} & 67.5 \\
\multicolumn{1}{l|}{ARM~\cite{zhang2020adaptive}} & 36.8 & \textbf{98.1} & 76.6 & 81.7 & 64.4 & 42.6 & 66.7 \\
\multicolumn{1}{l|}{VREx~\cite{krueger2021out}} & 36.9 & 93.6 & 76.7 & 81.3 & 64.9 & 37.3 & 65.1 \\
\multicolumn{1}{l|}{RSC~\cite{huang2020self}} & 36.5 & 97.6 & 77.5 & 82.6 & 65.8 & 40.0 & 66.6 \\
\hline
\multicolumn{1}{l|}{\ccol Ours} & \ccol \textbf{46.5} & \ccol 97.7 & \ccol 74.8 & \ccol \textbf{83.8} & \ccol 65.0 & \ccol 42.5 & \ccol \underline{68.4} \\
\hline
\end{tabularx}
}
%\vspace{-2mm}
\label{tab:domainbed}
\end{table*}
%\label{tab:domainbed}
% \begin{table}
% \centering
% %\resizebox{\textwidth}{!}{
% %\adjustbox{max width=\textwidth}
% \resizebox{\columnwidth}{!}
% {
% \begin{tabular}{lccccccc}
% \hline
% \multicolumn{8}{c}{Model selection: leave-one-domain-out cross-validation} \\
% %\toprule
% \hline
% {Methods} & {CMNIST} & {RMNIST} & {VLCS} & {PACS} & {OfficeHome} & {TerraIncognita} & {Avg.} \\
% \hline
% %\midrule
% ERM & 36.7 & 97.7 & 77.2 & 83.0 & 65.7 & 41.4 & 66.9 \\
% IRM & 40.3 & 97.0 & 76.3 & 81.5 & 64.3 & 41.2 & 66.7 \\
% GroupDRO & 36.8 & 97.6 & \underline{77.9} & 83.5 & 65.2 & 44.9 & 66.7 \\
% Mixup & 33.4 & \underline{97.8} & 77.7 & 83.2 & 67.0 & \textbf{48.7} & 67.9 \\
% MLDG & 36.7 & 97.6 & 77.2 & 82.9 & 66.1 & 46.2 & 67.7 \\
% CORAL & 39.7 & \underline{97.8} & \textbf{78.7} & 82.6 & \textbf{68.5} & 46.3 & \textbf{68.9} \\
% MMD & 36.8 & \underline{97.8} & 77.3 & 83.2 & 60.2 & 46.5 & 66.9 \\
% DANN & \underline{40.7} & 97.6 & 76.9 & 81.0 & 64.9 & 44.4 & 67.5 \\
% CDANN & 39.1 & 97.5 & 77.5 & 78.8 & 64.3 & 39.9 & 66.1 \\
% MTL & 35.0 & \underline{97.8} & 76.6 & \underline{83.7} & 65.7 & 44.9 & 67.2 \\
% SagNet & 36.5 & 94.0 & 77.5 & 82.3 & \underline{67.6} & \underline{47.2} & 67.5 \\
% ARM & 36.8 & \textbf{98.1} & 76.6 & 81.7 & 64.4 & 42.6 & 66.7 \\
% VREx & 36.9 & 93.6 & 76.7 & 81.3 & 64.9 & 37.3 & 65.1 \\
% RSC & 36.5 & 97.6 & 77.5 & 82.6 & 65.8 & 40.0 & 66.6 \\
% \hline
% \ccol Ours & \ccol \textbf{46.5} & \ccol 97.7 & \ccol 74.8 & \ccol \textbf{83.8} & \ccol 65.0 & \ccol 42.5 & \ccol \underline{68.4} \\
% \hline
% \end{tabular}
% }
% \vspace{-2mm}
% \caption{
% Domain generalization accuracy (\%) on DomainBed. The results compare fifteen methods including ours across six domain generalization benchmark datasets. Note that we adopt leave-one-domain-out cross-validation as a model selection criteria.
% }
% \vspace*{-2mm}
% \label{tab:domainbed}
% \end{table}
\begin{table*}
\centering
%\resizebox{\textwidth}{!}{
%\adjustbox{max width=\textwidth}
% \resizebox{\columnwidth}{!}
% {
%\fontsize{8.7}{9.7}\selectfont
\fontsize{6.7}{7.7}\selectfont
\begin{tabularx}{1.0\textwidth} {
>{\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
%>{\centering\arraybackslash}X|
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
}
\hline
\multicolumn{16}{c}{Model selection: leave-one-domain-out cross-validation} \\
\hline
{} & {ERM} & {IRM} & {GroupDRO} & {Mixup} & {MLDG} & {CORAL} & {MMD} & {DANN} & {CDANN} & {MTL} & {SagNet} & {ARM} & {VREx} & {RSC} & \ccol {Ours} & \\
\hline
\multicolumn{1}{l}{CMNIST} & 36.7 & 40.3 & 36.8 & 33.4 & 36.7 & 39.7 & 36.8 & \underline{40.7} & 39.1 & 35.0 & 36.5 & 36.8 & 36.9 & 36.5 & \ccol \textbf{46.5} \\
\hline
\multicolumn{1}{l}{RMNIST} & 97.7 & 97.0 & 97.6 & \underline{97.8} & 97.6 & \underline{97.8} & \underline{97.8} & 97.6 & 97.5 & \underline{97.8} & 94.0 & \textbf{98.1} & 93.6 & 97.6 & \ccol 97.7 \\
\iffalse
%\toprule
\hline
{Methods} & {CMNIST} & {RMNIST} & {VLCS} & {PACS} & {OfficeHome} & {TerraIncognita} & {Avg.} \\
\hline
%\midrule
ERM & 36.7 & 97.7 & 77.2 & 83.0 & 65.7 & 41.4 & 66.9 \\
IRM & 40.3 & 97.0 & 76.3 & 81.5 & 64.3 & 41.2 & 66.7 \\
GroupDRO & 36.8 & 97.6 & \underline{77.9} & 83.5 & 65.2 & 44.9 & 66.7 \\
Mixup & 33.4 & \underline{97.8} & 77.7 & 83.2 & 67.0 & \textbf{48.7} & 67.9 \\
MLDG & 36.7 & 97.6 & 77.2 & 82.9 & 66.1 & 46.2 & 67.7 \\
CORAL & 39.7 & \underline{97.8} & \textbf{78.7} & 82.6 & \textbf{68.5} & 46.3 & \textbf{68.9} \\
MMD & 36.8 & \underline{97.8} & 77.3 & 83.2 & 60.2 & 46.5 & 66.9 \\
DANN & \underline{40.7} & 97.6 & 76.9 & 81.0 & 64.9 & 44.4 & 67.5 \\
CDANN & 39.1 & 97.5 & 77.5 & 78.8 & 64.3 & 39.9 & 66.1 \\
MTL & 35.0 & \underline{97.8} & 76.6 & \underline{83.7} & 65.7 & 44.9 & 67.2 \\
SagNet & 36.5 & 94.0 & 77.5 & 82.3 & \underline{67.6} & \underline{47.2} & 67.5 \\
ARM & 36.8 & \textbf{98.1} & 76.6 & 81.7 & 64.4 & 42.6 & 66.7 \\
VREx & 36.9 & 93.6 & 76.7 & 81.3 & 64.9 & 37.3 & 65.1 \\
RSC & 36.5 & 97.6 & 77.5 & 82.6 & 65.8 & 40.0 & 66.6 \\
\hline
\ccol Ours & \ccol \textbf{46.5} & \ccol 97.7 & \ccol 74.8 & \ccol \textbf{83.8} & \ccol 65.0 & \ccol 42.5 & \ccol \underline{68.4} \\
\hline
\fi
\end{tabularx}
% }
\vspace{-2mm}
\caption{
Domain generalization accuracy (\%) on DomainBed. The results compare fifteen methods including ours across six domain generalization benchmark datasets. Note that we adopt leave-one-domain-out cross-validation as a model selection criteria.
}
\vspace*{-2mm}
\end{table*}
\begin{table*}
\centering
%\resizebox{\textwidth}{!}{
\adjustbox{max width=\textwidth}
{
\begin{tabular}{lccccccc}
\hline
\multicolumn{8}{c}{Model selection: leave-one-domain-out cross-validation} \\
%\toprule
\hline
\textbf{Algorithm} & \textbf{CMNIST} & \textbf{RMNIST} & \textbf{VLCS} & \textbf{PACS} & \textbf{OfficeHome} & \textbf{TerraIncognita} & \textbf{Avg} \\
\midrule
ERM & 36.7 & 97.7 & 77.2 & 83.0 & 65.7 & 41.4 & 66.9 \\
IRM & 40.3 & 97.0 & 76.3 & 81.5 & 64.3 & 41.2 & 66.7 \\
GroupDRO & 36.8 & 97.6 & 77.9 & 83.5 & 65.2 & 44.9 & 66.7 \\
Mixup & 33.4 & 97.8 & 77.7 & 83.2 & 67.0 & \textbf{48.7} & 67.9 \\
MLDG & 36.7 & 97.6 & 77.2 & 82.9 & 66.1 & 46.2 & 67.7 \\
CORAL & 39.7 & 97.8 & \textbf{78.7} & 82.6 & \textbf{68.5} & 46.3 & \textbf{68.9} \\
MMD & 36.8 & 97.8 & 77.3 & 83.2 & 60.2 & 46.5 & 66.9 \\
DANN & 40.7 & 97.6 & 76.9 & 81.0 & 64.9 & 44.4 & 67.5 \\
CDANN & 39.1 & 97.5 & 77.5 & 78.8 & 64.3 & 39.9 & 66.1 \\
MTL & 35.0 & 97.8 & 76.6 & 83.7 & 65.7 & 44.9 & 67.2 \\
SagNet & 36.5 & 94.0 & 77.5 & 82.3 & 67.6 & 47.2 & 67.5 \\
ARM & 36.8 & \textbf{98.1} & 76.6 & 81.7 & 64.4 & 42.6 & 66.7 \\
VREx & 36.9 & 93.6 & 76.7 & 81.3 & 64.9 & 37.3 & 65.1 \\
RSC & 36.5 & 97.6 & 77.5 & 82.6 & 65.8 & 40.0 & 66.6 \\
\hline
\ccol Ours & \ccol \textbf{46.5} & \ccol 97.7 & \ccol 74.8 & \ccol \textbf{83.8} & \ccol 65.0 & \ccol 42.5 & \ccol 68.4 \\
\hline
\end{tabular}
}
\vspace{-2mm}
\caption{
Domain generalization accuracy (\%) on DomainBed. The results compare fifteen methods including ours across six domain generalization benchamark datasets. Note that we adopt leave-one-domain-out cross-validation as a model selection criteria.
}
\vspace*{-2mm}
\label{tab:domainbed}
\end{table*}
%\begin{center}
%\adjustbox{max width=\textwidth}{%
\begin{table}[!t]
%\vspace{-2mm}
\fontsize{8}{10}\selectfont
\centering
%\adjustbox{max width=\textwidth}{%
% \footnotesize
\caption{
Leave-one-domain-out generalization results on OffceHome.
}
%\fontsize{8.5}{10.5}\selectfont
\begin{tabularx}{0.75\textwidth}{
>{\arraybackslash}l|
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X|
>{\centering\arraybackslash}X}
%\toprule
\hline
{Methods} & {Artistic} & {Clipart} & {Product} & {Real} & {Avg.} \\
%\midrule
\hline
ResNet-18 & 58.9 & 49.4 & 74.3 & 76.2 & 64.7 \\
MMD-AE~\cite{li2018domain} & 56.5 & 47.3 & 72.1 & 74.8 & 62.7 \\
%CCSA~\cite{motiian2017unified} & 59.9 & 49.9 & 74.1 & 75.7 & 64.9 \\
JiGen~\cite{carlucci2019domain} & 53.0 & 47.4 & 71.4 & 72.7 & 61.2 \\
CrossGrad~\cite{shankar2018generalizing} & 58.4 & 49.4 & 73.9 & 75.8 & 64.4 \\
% DDAIG~\cite{zhou2020deep} & 59.2 & 52.3 & 74.6 & \underline{76.6} & 65.5 \\
%Epi-FCR~\cite{li2019episodic} & 82.1 & 77.0 & 93.9 & 73.0 & 81.5 \\
%MetaReg~\cite{balaji2018metareg} & 83.7 & 77.2 & 95.5 & 70.3 & 81.7 \\
%EISNet~\cite{wang2020learning} & 81.8 & 76.4 & 95.9 & 74.3 & 82.1 \\
L2A-OT~\cite{zhou2020learning} & \underline{60.6} & 50.1 & \underline{74.8} & \textbf{77.0} & 65.6 \\
SagNet~\cite{nam2021reducing} & 60.2 & 45.3 & 70.4 & 73.3 & 62.3 \\
%SagNet~\cite{nam2021reducing} & 83.5 & 77.6 & 95.4 & 76.3 & 83.2 \\
%SelfReg~\cite{kim2021selfreg} & 82.3 & 78.4 & 96.2 & 77.4 & 83.6 \\
MixStyle~\cite{zhou2021domain} & 58.7 & 53.4 & 74.2 & 75.9 & 65.5 \\
%L2D~\cite{wang2021learning} & 81.4 & 79.5 & 95.5 & 80.5 & 84.2 \\
FACT~\cite{xu2021fourier} & 60.3 & 54.8 & 74.4 & \underline{76.5} & \underline{66.5} \\
DSON~\cite{seo2020learning} & 59.3 & 45.7 & 71.8 & 74.6 & 62.9 \\
%DAEL~\cite{zhou2021dael} & 59.4 & \underline{55.1} & 74.0 & 75.7 & 66.0 \\
RSC~\cite{huang2020self} & 58.4 & 47.9 & 71.6 & 74.5 & 63.1 \\
\hline
%\cite{pandey2021generalization} & 86.3 & 81.2 & 97.1 & 81.7 & 86.6 \\
\ccol Ours & \ccol \textbf{60.8} & \ccol \textbf{57.1} & \ccol \textbf{75.3} & \ccol \underline{76.5} & \ccol \textbf{67.4} \\
\hline
%DeepAll+Mixup & 76.8 & 74.9 & 95.8 & 66.6 & 78.5 \\
%Ours+SoftMixup & 84.8 & 83.7 & 95.8 & 79.6 & 86.0 \\
%\bottomrule
\end{tabularx}
%}
\label{tab:officeHome}
%\vspace{-6mm}
\end{table}
%}
%\end{center}
% %\begin{center}
% %\adjustbox{max width=\textwidth}{%
% \begin{table}%[!t]
% % \vspace{-2mm}
% \fontsize{8.3}{9.8}\selectfont
% \centering
% %\adjustbox{max width=\textwidth}{%
% % \footnotesize
% \begin{tabular}{lccccc}
% %\toprule
% \hline
% \textbf{Methods} & \textbf{Art} & \textbf{Cartoon} & \textbf{Photo} & \textbf{Sketch} & \textbf{Avg.} \\
% %\midrule
% \hline
% DeepAll & 77.0 & 75.9 & 96.0 & 69.2 & 79.5 \\
% MMD-AE~\cite{li2018domain} & 75.2 & 72.7 & 96.0 & 64.2 & 77.0 \\
% CCSA~\cite{motiian2017unified} & 80.5 & 76.9 & 93.6 & 66.8 & 79.4 \\
% JiGen~\cite{carlucci2019domain} & 79.4 & 75.3 & 96.0 & 71.6 & 80.5 \\
% CrossGrad~\cite{shankar2018generalizing} & 79.8 & 76.8 & 96.0 & 70.2 & 80.7 \\
% MASF~\cite{dou2019domain} & 80.2 & 77.1 & 94.9 & 71.6 & 81.0 \\
% Epi-FCR~\cite{li2019episodic} & 82.1 & 77.0 & 93.9 & 73.0 & 81.5 \\
% MetaReg~\cite{balaji2018metareg} & 83.7 & 77.2 & 95.5 & 70.3 & 81.7 \\
% EISNet~\cite{wang2020learning} & 81.8 & 76.4 & 95.9 & 74.3 & 82.1 \\
% L2A-OT~\cite{zhou2020learning} & 83.3 & 78.2 & 96.2 & 73.6 & 82.8 \\
% SagNet~\cite{nam2021reducing} & 83.5 & 77.6 & 95.4 & 76.3 & 83.2 \\
% DAEL~\cite{zhou2021dael} & 84.6 & 74.4 & 95.6 & 78.9 & 83.4 \\
% SelfReg~\cite{kim2021selfreg} & 82.3 & 78.4 & 96.2 & 77.4 & 83.6 \\
% MixStyle~\cite{zhou2021domain} & 84.1 & 78.8 & 96.1 & 75.9 & 83.7 \\
% L2D~\cite{wang2021learning} & 81.4 & 79.5 & 95.5 & 80.5 & 84.2 \\
% FACT~\cite{xu2021fourier} & 85.3 & 78.3 & 95.1 & 79.1 & 84.5 \\
% DSON~\cite{seo2020learning} & 84.6 & 77.6 & 95.8 & \textbf{82.2} & 85.1 \\
% RSC~\cite{huang2020self} & 83.4 & 80.3 & 95.9 & 80.8 & 85.1 \\
% \hline
% %\cite{pandey2021generalization} & 86.3 & 81.2 & 97.1 & 81.7 & 86.6 \\
% \ccol Ours & \ccol \textbf{85.6} & \ccol \textbf{84.2} & \ccol \textbf{96.5} & \ccol 79.1 & \ccol \textbf{86.4} \\
% \hline
% %DeepAll+Mixup & 76.8 & 74.9 & 95.8 & 66.6 & 78.5 \\
% %Ours+SoftMixup & 84.8 & 83.7 & 95.8 & 79.6 & 86.0 \\
% %\bottomrule
% \end{tabular}
% \caption{
% Leave-one-domain-out generalization results on PACS.
% }
% %}
% \label{tab:pacs}
% \end{table}
% %}
% %\end{center}
%\begin{center}
%\adjustbox{max width=\textwidth}{%
\begin{table}[!t]
% \vspace{-2mm}
% \fontsize{8.3}{9.5}\selectfont
\centering
%\adjustbox{max width=\textwidth}{%
% \footnotesize
\caption{
Leave-one-domain-out generalization results on PACS.
}
\fontsize{8}{10}\selectfont
\begin{tabularx}{0.75 \textwidth}{
>{\centering\arraybackslash}X| >{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X|
>{\centering\arraybackslash}X}
%\toprule
\hline
\multicolumn{1}{l|}{{Methods}} & {Art} & {Cartoon} & {Photo} & {Sketch} & {Avg.} \\
%\midrule
\hline
\multicolumn{1}{l|}{ResNet-18} & 77.0 & 75.9 & 96.0 & 69.2 & 79.5 \\
\multicolumn{1}{l|}{MMD-AE~\cite{li2018domain}} & 75.2 & 72.7 & 96.0 & 64.2 & 77.0 \\
%\multicolumn{1}{l|}{CCSA~\cite{motiian2017unified}} & 80.5 & 76.9 & 93.6 & 66.8 & 79.4 \\
\multicolumn{1}{l|}{JiGen~\cite{carlucci2019domain}} & 79.4 & 75.3 & 96.0 & 71.6 & 80.5 \\
\multicolumn{1}{l|}{CrossGrad~\cite{shankar2018generalizing}} & 79.8 & 76.8 & 96.0 & 70.2 & 80.7 \\
\multicolumn{1}{l|}{MASF~\cite{dou2019domain}} & 80.2 & 77.1 & 94.9 & 71.6 & 81.0 \\
\multicolumn{1}{l|}{Epi-FCR~\cite{li2019episodic}} & 82.1 & 77.0 & 93.9 & 73.0 & 81.5 \\
%\multicolumn{1}{l|}{MetaReg~\cite{balaji2018metareg}} & 83.7 & 77.2 & 95.5 & 70.3 & 81.7 \\
\multicolumn{1}{l|}{EISNet~\cite{wang2020learning}} & 81.8 & 76.4 & 95.9 & 74.3 & 82.1 \\
\multicolumn{1}{l|}{L2A-OT~\cite{zhou2020learning}} & 83.3 & 78.2 & \underline{96.2} & 73.6 & 82.8 \\
\multicolumn{1}{l|}{SagNet~\cite{nam2021reducing}} & 83.5 & 77.6 & 95.4 & 76.3 & 83.2 \\
%\multicolumn{1}{l|}{DAEL~\cite{zhou2021dael}} & 84.6 & 74.4 & 95.6 & 78.9 & 83.4 \\
\multicolumn{1}{l|}{SelfReg~\cite{kim2021selfreg}} & 82.3 & 78.4 & \underline{96.2} & 77.4 & 83.6 \\
\multicolumn{1}{l|}{MixStyle~\cite{zhou2021domain}} & 84.1 & 78.8 & 96.1 & 75.9 & 83.7 \\
\multicolumn{1}{l|}{L2D~\cite{wang2021learning}} & 81.4 & 79.5 & 95.5 & 80.5 & 84.2 \\
\multicolumn{1}{l|}{FACT~\cite{xu2021fourier}} & \underline{85.3} & 78.3 & 95.1 & 79.1 & 84.5 \\
\multicolumn{1}{l|}{DSON~\cite{seo2020learning}} & 84.6 & 77.6 & 95.8 & \textbf{82.2} & \underline{85.1} \\
\multicolumn{1}{l|}{RSC~\cite{huang2020self}} & 83.4 & \underline{80.3} & 95.9 & \underline{80.8} & \underline{85.1} \\
\hline
%\cite{pandey2021generalization} & 86.3 & 81.2 & 97.1 & 81.7 & 86.6 \\
\multicolumn{1}{l|}{\ccol Ours} & \ccol \textbf{85.6} & \ccol \textbf{84.2} & \ccol \textbf{96.5} & \ccol 79.1 & \ccol \textbf{86.4} \\
\hline
%DeepAll+Mixup & 76.8 & 74.9 & 95.8 & 66.6 & 78.5 \\
%Ours+SoftMixup & 84.8 & 83.7 & 95.8 & 79.6 & 86.0 \\
%\bottomrule
\end{tabularx}
%}
\label{tab:pacs}
\vspace{-4mm}
\end{table}
%}
%\end{center}
% %\begin{center}
% %\adjustbox{max width=\textwidth}{%
% \begin{table}[]%[!t]
% \centering
% % \adjustbox{max width=\textwidth}
% %\footnotesize
% %\setlength{\tabcolsep}{1.0mm}
% \resizebox{\columnwidth}{!}
% {
% \begin{tabular}{l | cccc | cccc }
% %\toprule
% \hline
% \multicolumn{1}{l|}{} &
% \multicolumn{4}{c|}{Market1501$\rightarrow$Duke} & \multicolumn{4}{c}{Duke$\rightarrow$Market1501} \\
% \textbf{Methods} & \textbf{mAP} & \textbf{R@1} & \textbf{R@5} & \textbf{R@10} & \textbf{mAP} & \textbf{R@1} & \textbf{R@5} & \textbf{R@10} \\
% \hline
% %\multicolumn{1}{l}{} &
% %\multicolumn{8}{c}{Domain Adaptation}\\
% %\hline
% %ATNet~\cite{liu2019adaptive} & 24.9 & 45.1 & 59.5 & 64.2 & 25.6 & 55.7 & 73.2 & 79.4 \\
% %SPGAN~\cite{deng2018image} & - & - & - & - & - & - & - & - \\
% %TJ-AIDL~\cite{wang2018transferable} & 23.0 & 44.3 & \underline{59.6} & \underline{65.0} & 26.5 & 58.2 & \underline{74.8} & \underline{81.1} \\
% %CamStyle & 25.1 & 48.4 & 62.5 & 68.9 & 27.4 & 58.8 & 78.2 & 84.3 \\
% %HHL~\cite{zhong2018generalizing} & \underline{27.2} & \underline{46.9} & \textbf{61.0} & \textbf{66.7} & \textbf{31.4} & \textbf{62.2} & \textbf{78.8} & \textbf{84.0} \\
% %\hline
% %\midrule
% %\multicolumn{1}{l}{} &
% %\multicolumn{8}{c}{Domain Generalization}\\
% %\hline
% %\cline{3-14}
% %\midrule
% ResNet-50 & 19.3 & 35.4 & 50.3 & 56.4 & 20.4 & 45.2 & 63.6 & 70.9 \\
% RandomErase~\cite{zhong2020random} & 14.3 & 27.8 & 42.6 & 49.1 & 16.1 & 38.5 & 56.8 & 64.5 \\
% DropBlock~\cite{ghiasi2018dropblock} & 18.2 & 33.2 & 49.1 & 56.3 & 19.7 & 45.3 & 62.1 & 69.1 \\
% %MixStyle~\cite{zhou2021domain} w/ random shuffle & 23.8 & 42.2 & 58.8 & 64.8 & 24.1 & 51.5 & 69.4 & 76.2 \\
% %MixStyle~\cite{zhou2021domain} w/ domain label & 23.4 & 43.3 & 58.9 & 64.7 & 24.7 & 53.0 & 70.9 & 77.8 \\
% %MixStyle w/ random shuffle~\cite{zhou2021domain} & \underline{23.8} & \underline{42.2} & \textbf{58.8} & \textbf{64.8} & \underline{24.1} & \underline{51.5} & \textbf{69.4} & \textbf{77.8} \\
% MixStyle~\cite{zhou2021domain} & \underline{23.4} & \underline{43.3} & \textbf{58.9} & \textbf{64.7} & \underline{24.7} & \underline{53.0} & \textbf{70.9} & \textbf{77.8} \\
% %Ours w/o KD & 26.2 & 46.3 & 61.8 & 67.0 & 28.0 & 57.5 & 65.7 & 69.7 \\
% %Ours w/ KD & \textbf{27.4} & \textbf{49.3} & 56.0 & 59.5 & \underline{30.1} & \underline{59.0} & 67.0 & 71.5 \\
% \hline
% \ccol Ours & \ccol \textbf{27.4} & \ccol \textbf{49.3} & \ccol \underline{56.0} & \ccol \underline{59.5} & \ccol \textbf{30.1} & \ccol \textbf{59.0} & \ccol \underline{67.0} & \ccol \underline{71.5} \\
% %Ours w/ KD & 26.9 & 46.6 & 53.2 & 56.9 & - & - & - & - \\
% %\bottomrule
% \hline
% \end{tabular}
% }
% \vspace{-2mm}
% \caption{
% Generalization results on the cross-dataset person re-ID.
% }
% \vspace*{-2mm}
% \label{tab:person_reid}
% \end{table}
%\begin{center}
%\adjustbox{max width=\textwidth}{%
\begin{table}[!t]
\centering
% \adjustbox{max width=\textwidth}
%\footnotesize
%\setlength{\tabcolsep}{1.0mm}
\caption{
Generalization results on the cross-dataset person re-ID.
}
\fontsize{8}{10}\selectfont
\begin{tabularx}{0.75 \textwidth}{
>{\centering\arraybackslash}X|
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X|
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X}
%\toprule
\hline
\multicolumn{1}{l|}{} &
\multicolumn{2}{c|}{Market $\rightarrow$ Duke} & \multicolumn{2}{c}{Duke $\rightarrow$ Market} \\
\multicolumn{1}{l|}{Methods} & {mAP} & {R@1} & {mAP} & {R@1} \\
\hline
%\multicolumn{1}{l}{} &
%\multicolumn{8}{c}{Domain Adaptation}\\
%\hline
%ATNet~\cite{liu2019adaptive} & 24.9 & 45.1 & 59.5 & 64.2 & 25.6 & 55.7 & 73.2 & 79.4 \\
%SPGAN~\cite{deng2018image} & - & - & - & - & - & - & - & - \\
%TJ-AIDL~\cite{wang2018transferable} & 23.0 & 44.3 & \underline{59.6} & \underline{65.0} & 26.5 & 58.2 & \underline{74.8} & \underline{81.1} \\
%CamStyle & 25.1 & 48.4 & 62.5 & 68.9 & 27.4 & 58.8 & 78.2 & 84.3 \\
%HHL~\cite{zhong2018generalizing} & \underline{27.2} & \underline{46.9} & \textbf{61.0} & \textbf{66.7} & \textbf{31.4} & \textbf{62.2} & \textbf{78.8} & \textbf{84.0} \\
%\hline
%\midrule
%\multicolumn{1}{l}{} &
%\multicolumn{8}{c}{Domain Generalization}\\
%\hline
%\cline{3-14}
%\midrule
\multicolumn{1}{l|}{ResNet-50} & 19.3 & 35.4 & 20.4 & 45.2\\
\multicolumn{1}{l|}{RandomErase~\cite{zhong2020random}} & 14.3 & 27.8 & 16.1 & 38.5\\
\multicolumn{1}{l|}{DropBlock~\cite{ghiasi2018dropblock}} & 18.2 & 33.2 & 19.7 & 45.3\\
%MixStyle~\cite{zhou2021domain} w/ random shuffle & 23.8 & 42.2 & 58.8 & 64.8 & 24.1 & 51.5 & 69.4 & 76.2 \\
%MixStyle~\cite{zhou2021domain} w/ domain label & 23.4 & 43.3 & 58.9 & 64.7 & 24.7 & 53.0 & 70.9 & 77.8 \\
%MixStyle w/ random shuffle~\cite{zhou2021domain} & \underline{23.8} & \underline{42.2} & \textbf{58.8} & \textbf{64.8} & \underline{24.1} & \underline{51.5} & \textbf{69.4} & \textbf{77.8} \\
\multicolumn{1}{l|}{MixStyle~\cite{zhou2021domain}} & 23.4 & 43.3 & 24.7 & 53.0 \\
\multicolumn{1}{l|}{StyleNeophile~\cite{StyleNeophile}} & \underline{26.3} & \underline{46.5} & \underline{27.2} & \underline{55.0} \\
%Ours w/o KD & 26.2 & 46.3 & 61.8 & 67.0 & 28.0 & 57.5 & 65.7 & 69.7 \\
%Ours w/ KD & \textbf{27.4} & \textbf{49.3} & 56.0 & 59.5 & \underline{30.1} & \underline{59.0} & 67.0 & 71.5 \\
% \hline
\multicolumn{1}{l|}{\ccol Ours} & \ccol \textbf{27.4} & \ccol \textbf{49.3} & \ccol \textbf{30.1} & \ccol \textbf{59.0} \\
%Ours w/ KD & 26.9 & 46.6 & 53.2 & 56.9 & - & - & - & - \\
%\bottomrule
\hline
\end{tabularx}
%\vspace*{-2mm}
\label{tab:person_reid}
\end{table}
%\begin{center}
%\adjustbox{max width=\textwidth}{%
\begin{table*}[!t]
\centering
% \adjustbox{max width=\textwidth}
% \tiny
% {%
\caption{
Single-source domain generalization accuracy (\%) on PACS with a ResNet-18. (A: Art Painting, C: Cartoon, S:Sketch, P:Photo).
}
\fontsize{7}{9}\selectfont
%\begin{tabularx}{0.7 \textwidth}{
\begin{tabularx}{1.0\textwidth}{
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X
>{\centering\arraybackslash}X|
>{\centering\arraybackslash}X
}
\hline
%\toprule
\multicolumn{1}{l|}{Methods} & {A }\shortarrow[.12cm] {C}&
{A }\shortarrow[.12cm] {S} & {A }\shortarrow[.12cm] {P} & {C }\shortarrow[.12cm] {A} & {C }\shortarrow[.12cm] {S} & {C }\shortarrow[.12cm] {P} & {S }\shortarrow[.12cm] {A} & {S }\shortarrow[.12cm] {C} & {S }\shortarrow[.12cm] {P} & {P }\shortarrow[.12cm] {A} & {P }\shortarrow[.12cm] {C} & {P }\shortarrow[.12cm] {S} & {Avg.} \\
\hline
%\midrule
\multicolumn{1}{l|}{ResNet-18} & 62.3 & 49.0 & 95.2 & 65.7 & 60.7 & 83.6 & 28.0 & 54.5 & 35.6 & 64.1 & 23.6 & 29.1 & 54.3 \\
%ResNet-18^{J} & 59.1 & 49.2 & 96.1 & 65.1 & 65.6 & 84.3 & 27.2 & 37.2 & 39.7 & 66.3 & 26.8 & 34.7 & 54.3 \\
\multicolumn{1}{l|}{JiGen~\cite{carlucci2019domain}} & 57.0 & 50.0 & 96.1 & 65.3 & 65.9 & 85.5 & 26.6 & 41.1 & 42.8 & 62.4 & 27.2 & 35.5 & 54.6 \\
\multicolumn{1}{l|}{MixStyle~\cite{zhou2021domain}} & 65.5 & 49.8 & \underline{96.7} & 69.9 & 64.5 & 85.3 & 27.1 & 50.9 & 32.6 & 67.7 & \underline{38.9} & 39.1 & 57.4 \\
\multicolumn{1}{l|}{RSC~\cite{huang2020self}} & 62.5 & 53.1 & 96.2 & 68.9 & \textbf{70.3} & 85.8 & 37.9 & 56.3 & \underline{47.4} & 66.3 & 26.4 & 32.0 & 58.6 \\
\multicolumn{1}{l|}{SelfReg~\cite{kim2021selfreg}} & 65.2 & 55.9 & 96.6 & 72.0 & \underline{70.0} & \underline{87.5} & 37.1 & 54.0 & 46.0 & 67.7 & 28.9 & 33.7 & 59.5 \\
\multicolumn{1}{l|}{SagNet~\cite{nam2021reducing}} & \underline{67.1} & \underline{56.8} & 95.7 & \underline{72.1} & 69.2 & 85.7 & \underline{41.1} & \underline{62.9} & 46.2 & \underline{69.8} & 35.1 & \underline{40.7} & \underline{61.9} \\
%Ours w.o. SoftMixup & \textbf{74.6} & \textbf{58.1} & \textbf{96.8} & \textbf{74.4} & 69.6 & \textbf{87.6} & \textbf{43.3} & \textbf{65.6} & \textbf{50.3} & \textbf{71.4} & \textbf{54.3} & \textbf{51.5} & \textbf{66.5} \\
\hline
\multicolumn{1}{l|}{\ccol Ours} & \ccol \textbf{74.6} & \ccol \textbf{58.1} & \ccol \textbf{96.8} & \ccol \textbf{74.4} & \ccol 69.6 & \ccol \textbf{87.6} & \ccol \textbf{43.3} & \ccol \textbf{65.6} & \ccol \textbf{50.3} & \ccol \textbf{71.4} & \ccol \textbf{54.3} & \ccol \textbf{51.5} & \ccol \textbf{66.5} \\
\hline
%\bottomrule
\end{tabularx}
% }
%\vspace{-8mm}
\label{tab:single_source}
\end{table*}
%\begin{center}
%\adjustbox{max width=\textwidth}{%
\begin{table*}[!t]
\centering
%\adjustbox{max width=\textwidth}{%
\footnotesize
\begin{tabular}{lccccc}
%\toprule
\hline
\textbf{Methods} & \textbf{Art} & \textbf{Cartoon} & \textbf{Photo} & \textbf{Sketch} & \textbf{Average} \\
%\midrule
\hline
JiGen~\cite{carlucci2019domain} & 79.4 & 75.3 & 96.0 & 71.6 & 80.5 \\
%JiGen~\cite{carlucci2019domain} (ours) & 80.5 & 74.7 & 96.0 & 72.1 & - \\
Ours + JiGen & 85.3 & 79.2 & 95.9 & 79.2 & 84.9 \\
%\hline
%SagNet~\cite{nam2021reducing} & 83.5 & 77.6 & 95.4 & 76.3 & 83.2 \\
%Ours + SagNet~\cite{nam2021reducing} & - & - & - & - & - \\
%\hline
%SelfReg~\cite{kim2021selfreg} & 82.3 & 78.4 & 96.2 & 77.4 & 83.6 \\
%Ours+SelfReg & - & - & - & - & - \\
\hline
%RSC~\cite{huang2020self} & 83.4 & 80.3 & 95.9 & 80.8 & 85.1 \\
RSC~\cite{huang2020self} (\textit{our imple}) & 82.8 & 77.6 & 95.7 & 78.6 & 83.7 \\
Ours+RSC & 84.7 & 81.3 & 96.2 & 82.2 & 86.1 \\
\hline
\end{tabular}
%}
\vspace{-2mm}
\caption{
Universality of our proposed framework. Generalization accuracy (\%) on PACS
}
\vspace*{-2mm}
\label{tab:universality_all}
\end{table*}
%}
%\end{center}
%\begin{center}
%\adjustbox{max width=\textwidth}{%
\begin{table*}[!t]
\caption{
Universality of the proposed whitening. Generalization accuracy (\%) on PACS
}
\vspace{-2mm}
\centering
%\adjustbox{max width=\textwidth}{%
\footnotesize
\begin{tabular}{lccccc}
%\toprule
\hline
\textbf{Methods} & \textbf{Art} & \textbf{Cartoon} & \textbf{Photo} & \textbf{Sketch} & \textbf{Average} \\
%\midrule
\hline
JiGen~\cite{carlucci2019domain} & 79.4 & 75.3 & 96.0 & 71.6 & 80.5 \\
%JiGen~\cite{carlucci2019domain} (ours) & 80.5 & 74.7 & 96.0 & 72.1 & - \\
Whitening (proposed) + JiGen & 81.8 & 78.5 & 95.7 & 76.8 & 83.2 \\
%\hline
%SagNet~\cite{nam2021reducing} & 83.5 & 77.6 & 95.4 & 76.3 & 83.2 \\
%Ours + SagNet~\cite{nam2021reducing} & - & - & - & - & - \\
\hline
SelfReg~\cite{kim2021selfreg} & 82.3 & 78.4 & 96.2 & 77.4 & 83.6 \\
Whitening (proposed) + SelfReg & - & - & - & - & - \\
\hline
%RSC~\cite{huang2020self} & 83.4 & 80.3 & 95.9 & 80.8 & 85.1 \\
%\footnote{We remark that the performance of RSC was not reproduced even though we adopt the official code of the authors. Also, note that this irreproduceability was confirmed at other methods such as FACT~\cite{xu2021fourier} and SelfReg\cite{kim2021selfreg}}
RSC~\cite{huang2020self} (\textit{our imple}) & 82.8 & 77.6 & 95.7 & 78.6 & 83.7 \\
Whitening (proposed) + RSC & 83.5 & 78.8 & 95.0 & 79.6 & 84.2\footnote{asdffdsaasdffdsaasdf} \\
\hline
\end{tabular}
%}
\label{tab:universality_whitening}
\end{table*}
%}
%\end{center}
\begin{thebibliography}{10}
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\urlprefix}{URL }
\providecommand{\doi}[1]{https://doi.org/#1}
\bibitem{arjovsky2019invariant}
Arjovsky, M., Bottou, L., Gulrajani, I., Lopez-Paz, D.: Invariant risk
minimization. arXiv preprint arXiv:1907.02893 (2019)
\bibitem{balaji2018metareg}
Balaji, Y., Sankaranarayanan, S., Chellappa, R.: Metareg: Towards domain
generalization using meta-regularization. In: Proc. Neural Information
Processing Systems (NeurIPS) (2018)
\bibitem{ben2010theory}
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan,
J.W.: A theory of learning from different domains. Machine learning (2010)
\bibitem{ben2007analysis}
Ben-David, S., Blitzer, J., Crammer, K., Pereira, F., et~al.: Analysis of
representations for domain adaptation. In: Proc. Neural Information
Processing Systems (NeurIPS) (2007)
\bibitem{blanchard2021domain}
Blanchard, G., Deshmukh, A.A., Dogan, U., Lee, G., Scott, C.: Domain
generalization by marginal transfer learning. Journal of Machine Learning
Research (JMLR) (2021)
\bibitem{brendel2019approximating}
Brendel, W., Bethge, M.: Approximating cnns with bag-of-local-features models
works surprisingly well on imagenet. In: Proc. International Conference on
Learning Representations (ICLR) (2019)
\bibitem{carlucci2019domain}
Carlucci, F.M., D'Innocente, A., Bucci, S., Caputo, B., Tommasi, T.: Domain
generalization by solving jigsaw puzzles. In: Proc. IEEE Conference on
Computer Vision and Pattern Recognition (CVPR) (2019)
\bibitem{cha2021swad}
Cha, J., Chun, S., Lee, K., Cho, H.C., Park, S., Lee, Y., Park, S.: Swad:
Domain generalization by seeking flat minima. In: Proc. Neural Information
Processing Systems (NeurIPS) (2021)
\bibitem{cha2020cpr}
Cha, S., Hsu, H., Hwang, T., Calmon, F.P., Moon, T.: Cpr: Classifier-projection
regularization for continual learning. In: Proc. International Conference on
Learning Representations (ICLR) (2021)
\bibitem{chaudhari2017entropy}
Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C.,
Chayes, J., Sagun, L., Zecchina, R.: Entropy-sgd: Biasing gradient descent
into wide valleys. In: Proc. International Conference on Learning
Representations (ICLR) (2017)
\bibitem{chen2018encoder}
Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder
with atrous separable convolution for semantic image segmentation. In: Proc.
European Conference on Computer Vision (ECCV) (2018)
\bibitem{choi2021robustnet}
Choi, S., Jung, S., Yun, H., Kim, J.T., Kim, S., Choo, J.: Robustnet: Improving
domain generalization in urban-scene segmentation via instance selective
whitening. In: Proc. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) (2021)
\bibitem{cityscapes}
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R.,
Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban
scene understanding. In: Proc. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) (2016)
\bibitem{cubuk2018autoaugment}
Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: Autoaugment: Learning
augmentation policies from data (2019)
\bibitem{devries2017improved}
DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural
networks with cutout. arXiv preprint arXiv:1708.04552 (2017)
\bibitem{dou2019domain}
Dou, Q., Coelho~de Castro, D., Kamnitsas, K., Glocker, B.: Domain
generalization via model-agnostic learning of semantic features. In: Proc.
Neural Information Processing Systems (NeurIPS) (2019)
\bibitem{dumoulin2016learned}
Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic
style. In: Proc. International Conference on Learning Representations (ICLR)
(2017)
\bibitem{dziugaite2017computing}
Dziugaite, G.K., Roy, D.M.: Computing nonvacuous generalization bounds for deep
(stochastic) neural networks with many more parameters than training data.
In: Proc. The Conference on Uncertainty in Artificial Intelligence (UAI)
(2017)
\bibitem{feng2020kd3a}
Feng, H.Z., You, Z., Chen, M., Zhang, T., Zhu, M., Wu, F., Wu, C., Chen, W.:
Kd3a: Unsupervised multi-source decentralized domain adaptation via knowledge
distillation. In: Proc. International Conference on Machine Learning (ICML)
(2021)
\bibitem{foret2021sharpness}
Foret, P., Kleiner, A., Mobahi, H., Neyshabur, B.: Sharpness-aware minimization
for efficiently improving generalization. In: Proc. International Conference
on Learning Representations (ICLR) (2021)
\bibitem{ganin2016dann}
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette,
F., Marchand, M., Lempitsky, V.: Domain-adversarial training of neural
networks. Journal of Machine Learning Research (JMLR) (2016)
\bibitem{geirhos2018imagenet}
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F.A., Brendel,
W.: Imagenet-trained cnns are biased towards texture; increasing shape bias
improves accuracy and robustness. In: Proc. International Conference on
Learning Representations (ICLR) (2019)
\bibitem{ghiasi2018dropblock}
Ghiasi, G., Lin, T.Y., Le, Q.V.: Dropblock: A regularization method for
convolutional networks. In: Proc. Neural Information Processing Systems
(NeurIPS) (2018)
\bibitem{goodfellow2014generative}
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair,
S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Proc. Neural
Information Processing Systems (NeurIPS) (2014)
\bibitem{goodfellow2014explaining}
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing
adversarial examples. In: Proc. International Conference on Learning
Representations (ICLR) (2014)
\bibitem{gulrajani2020search}
Gulrajani, I., Lopez-Paz, D.: In search of lost domain generalization. In:
Proc. International Conference on Learning Representations (ICLR) (2021)
\bibitem{resnet}
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image
recognition. In: Proc. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) (June 2016)
\bibitem{hendrycks2019benchmarking}
Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common
corruptions and perturbations. In: Proc. International Conference on Learning
Representations (ICLR) (2019)
\bibitem{hendrycks2019augmix}
Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.:
Augmix: A simple data processing method to improve robustness and uncertainty
(2020)
\bibitem{hendrycks2020augmix}
Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.:
Augmix: A simple data processing method to improve robustness and
uncertainty. In: Proc. International Conference on Learning Representations
(ICLR) (2020)
\bibitem{hinton2015distilling}
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural
network. arXiv preprint arXiv:1503.02531 (2015)
\bibitem{huang2017arbitrary}
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive
instance normalization. In: Proc. IEEE International Conference on Computer
Vision (ICCV) (2017)
\bibitem{huang2020self}
Huang, Z., Wang, H., Xing, E.P., Huang, D.: Self-challenging improves
cross-domain generalization. In: Proc. European Conference on Computer Vision
(ECCV) (2020)
\bibitem{Batchnorm}
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training
by reducing internal covariate shift. In: Proc. International Conference on
Machine Learning (ICML) (2015)
\bibitem{izmailov2018averaging}
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G.: Averaging
weights leads to wider optima and better generalization. In: Proc. The
Conference on Uncertainty in Artificial Intelligence (UAI) (2018)
\bibitem{jiang2020fantastic}
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., Bengio, S.: Fantastic
generalization measures and where to find them. In: Proc. International
Conference on Learning Representations (ICLR) (2020)
\bibitem{StyleNeophile}
Kang, J., Lee, S., Kim, N., Kwak, S.: Style neophile: Constantly seeking novel
styles for domain generalization. In: Proc. IEEE Conference on Computer
Vision and Pattern Recognition (CVPR) (2022)
\bibitem{keskar2016large}
Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M., Tang, P.T.P.: On
large-batch training for deep learning: Generalization gap and sharp minima.
In: Proc. International Conference on Learning Representations (ICLR) (2017)
\bibitem{kifer2004detecting}
Kifer, D., Ben-David, S., Gehrke, J.: Detecting change in data streams. In: In
Very Large Databases (VLDB) (2004)
\bibitem{kim2021selfreg}
Kim, D., Park, S., Kim, J., Lee, J.: Selfreg: Self-supervised contrastive
regularization for domain generalization. In: Proc. IEEE International
Conference on Computer Vision (ICCV) (2021)
\bibitem{kim2021wedge}
Kim, N., Son, T., Lan, C., Zeng, W., Kwak, S.: Wedge: Web-image assisted domain
generalization for semantic segmentation. arXiv preprint arXiv:2109.14196
(2021)
\bibitem{kim2021embedding}
Kim, S., Kim, D., Cho, M., Kwak, S.: Embedding transfer with label relaxation
for improved metric learning. In: Proc. IEEE Conference on Computer Vision
and Pattern Recognition (CVPR) (2021)
\bibitem{Alexnet}
Krizhevsky, A., Sutskever, I., Hinton, G.E.: {ImageNet} classification with
deep convolutional neural networks. In: Proc. Neural Information Processing
Systems (NeurIPS) (2012)
\bibitem{krueger2021out}
Krueger, D., Caballero, E., Jacobsen, J.H., Zhang, A., Binas, J., Zhang, D.,
Le~Priol, R., Courville, A.: Out-of-distribution generalization via risk
extrapolation (rex). In: Proc. International Conference on Machine Learning
(ICML) (2021)
\bibitem{li2017deeper}
Li, D., Yang, Y., Song, Y.Z., Hospedales, T.M.: Deeper, broader and artier
domain generalization. In: Proc. IEEE International Conference on Computer
Vision (ICCV) (2017)
\bibitem{li2018mldg}
Li, D., Yang, Y., Song, Y.Z., Hospedales, T.M.: Learning to generalize:
Meta-learning for domain generalization. In: Proc. AAAI Conference on
Artificial Intelligence (AAAI) (2018)
\bibitem{li2019episodic}
Li, D., Zhang, J., Yang, Y., Liu, C., Song, Y.Z., Hospedales, T.M.: Episodic
training for domain generalization. In: Proc. IEEE International Conference
on Computer Vision (ICCV) (2019)
\bibitem{li2018domain}
Li, H., Pan, S.J., Wang, S., Kot, A.C.: Domain generalization with adversarial
feature learning. In: Proc. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) (2018)
\bibitem{li2018deep}
Li, Y., Tian, X., Gong, M., Liu, Y., Liu, T., Zhang, K., Tao, D.: Deep domain
generalization via conditional invariant adversarial networks. In: Proc.
European Conference on Computer Vision (ECCV) (2018)
\bibitem{long2015learning}
Long, M., Cao, Y., Wang, J., Jordan, M.: Learning transferable features with
deep adaptation networks. In: Proc. International Conference on Machine
Learning (ICML) (2015)
\bibitem{madry2018towards}
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep
learning models resistant to adversarial attacks. In: Proc. International
Conference on Learning Representations (ICLR) (2018)
\bibitem{meng2018adversarial}
Meng, Z., Li, J., Gong, Y., Juang, B.H.: Adversarial teacher-student learning
for unsupervised domain adaptation. In: 2018 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP) (2018)
\bibitem{miyato2018virtual}
Miyato, T., Maeda, S.i., Koyama, M., Ishii, S.: Virtual adversarial training: a
regularization method for supervised and semi-supervised learning. IEEE
Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2018)
\bibitem{mnih2013playing}
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra,
D., Riedmiller, M.: Playing atari with deep reinforcement learning. In:
NeurIPS Deep Learning Workshop (2013)
\bibitem{muandet2013domain}
Muandet, K., Balduzzi, D., Sch{\"o}lkopf, B.: Domain generalization via
invariant feature representation. In: Proc. International Conference on
Machine Learning (ICML) (2013)
\bibitem{nam2021reducing}
Nam, H., Lee, H., Park, J., Yoon, W., Yoo, D.: Reducing domain gap by reducing
style bias. In: Proc. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) (2021)
\bibitem{neuhold2017mapillary}
Neuhold, G., Ollmann, T., Rota~Bulo, S., Kontschieder, P.: The mapillary vistas
dataset for semantic understanding of street scenes. In: Proc. IEEE
International Conference on Computer Vision (ICCV) (2017)
\bibitem{pan2019switchable}
Pan, X., Zhan, X., Shi, J., Tang, X., Luo, P.: Switchable whitening for deep
representation learning. In: Proc. IEEE International Conference on Computer
Vision (ICCV) (2019)
\bibitem{park2019relational}
Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In:
Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2019)
\bibitem{pereyra2017regularizing}
Pereyra, G., Tucker, G., Chorowski, J., Kaiser, {\L}., Hinton, G.: Regularizing
neural networks by penalizing confident output distributions. In: ICLR
Workshop (2017)
\bibitem{richter2016playing}
Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: Ground truth
from computer games. In: Proc. European Conference on Computer Vision (ECCV)
(2016)
\bibitem{ristani2016performance}
Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C.: Performance
measures and a data set for multi-target, multi-camera tracking. In: Proc.
European Conference on Computer Vision (ECCV) (2016)
\bibitem{ros2016synthia}
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The synthia
dataset: A large collection of synthetic images for semantic segmentation of
urban scenes. In: Proc. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) (2016)
\bibitem{sagawa2020distributionally}
Sagawa, S., Koh, P.W., Hashimoto, T.B., Liang, P.: Distributionally robust
neural networks for group shifts: On the importance of regularization for
worst-case generalization. In: Proc. International Conference on Learning
Representations (ICLR) (2020)
\bibitem{seo2020learning}
Seo, S., Suh, Y., Kim, D., Kim, G., Han, J., Han, B.: Learning to optimize
domain specific normalization for domain generalization. In: Proc. European
Conference on Computer Vision (ECCV) (2020)
\bibitem{shankar2018generalizing}
Shankar, S., Piratla, V., Chakrabarti, S., Chaudhuri, S., Jyothi, P., Sarawagi,
S.: Generalizing across domains via cross-gradient training. In: Proc.
International Conference on Learning Representations (ICLR) (2018)
\bibitem{stutz2021relating}
Stutz, D., Hein, M., Schiele, B.: Relating adversarially robust generalization
to flat minima. In: Proc. IEEE International Conference on Computer Vision
(ICCV) (2021)
\bibitem{sun2016coral}
Sun, B., Saenko, K.: Deep coral: Correlation alignment for deep domain
adaptation. In: Proc. European Conference on Computer Vision (ECCV) (2016)
\bibitem{szegedy2016rethinking}
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the
inception architecture for computer vision. In: Proc. IEEE Conference on
Computer Vision and Pattern Recognition (CVPR) (2016)
\bibitem{ulyanov2016instance}
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing
ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
\bibitem{vapnik1998statistical}
Vapnik, V.: Statistical learning theory. NY: Wiley (1998)
\bibitem{venkateswara2017deep}
Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing
network for unsupervised domain adaptation. In: Proc. IEEE Conference on
Computer Vision and Pattern Recognition (CVPR) (2017)
\bibitem{verma2019manifold}
Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D.,
Bengio, Y.: Manifold mixup: Better representations by interpolating hidden
states. In: Proc. International Conference on Machine Learning (ICML) (2019)
\bibitem{wang2020learning}
Wang, S., Yu, L., Li, C., Fu, C.W., Heng, P.A.: Learning from extrinsic and
intrinsic supervisions for domain generalization. In: Proc. European
Conference on Computer Vision (ECCV) (2020)
\bibitem{wang2021learning}
Wang, Z., Luo, Y., Qiu, R., Huang, Z., Baktashmotlagh, M.: Learning to
diversify for single domain generalization. In: Proc. IEEE International
Conference on Computer Vision (ICCV) (2021)
\bibitem{wu2020adversarial}
Wu, D., Xia, S.T., Wang, Y.: Adversarial weight perturbation helps robust
generalization. In: Proc. Neural Information Processing Systems (NeurIPS)
(2020)
\bibitem{xu2021fourier}
Xu, Q., Zhang, R., Zhang, Y., Wang, Y., Tian, Q.: A fourier-based framework for
domain generalization. In: Proc. IEEE Conference on Computer Vision and
Pattern Recognition (CVPR) (2021)
\bibitem{xu2019data}
Xu, T.B., Liu, C.L.: Data-distortion guided self-distillation for deep neural
networks. In: Proc. AAAI Conference on Artificial Intelligence (AAAI) (2019)
\bibitem{xu2014exploiting}
Xu, Z., Li, W., Niu, L., Xu, D.: Exploiting low-rank structure from latent
domains for domain generalization. In: Proc. European Conference on Computer
Vision (ECCV) (2014)
\bibitem{yao2020pyhessian}
Yao, Z., Gholami, A., Keutzer, K., Mahoney, M.W.: Pyhessian: Neural networks
through the lens of the hessian. In: 2020 IEEE International Conference on
Big Data (Big Data) (2020)
\bibitem{yu2018bdd100k}
Yu, F., Xian, W., Chen, Y., Liu, F., Liao, M., Madhavan, V., Darrell, T.:
Bdd100k: A diverse driving video database with scalable annotation tooling.
arXiv preprint arXiv:1805.04687 (2018)
\bibitem{yue2019domain}
Yue, X., Zhang, Y., Zhao, S., Sangiovanni-Vincentelli, A., Keutzer, K., Gong,
B.: Domain randomization and pyramid consistency: Simulation-to-real
generalization without accessing target domain data. In: Proc. IEEE
International Conference on Computer Vision (ICCV) (2019)
\bibitem{yun2019cutmix}
Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization
strategy to train strong classifiers with localizable features. In: Proc.
IEEE International Conference on Computer Vision (ICCV) (2019)
\bibitem{yun2020regularizing}
Yun, S., Park, J., Lee, K., Shin, J.: Regularizing class-wise predictions via
self-knowledge distillation. In: Proc. IEEE Conference on Computer Vision and
Pattern Recognition (CVPR) (2020)
\bibitem{zagoruyko2016wide}
Zagoruyko, S., Komodakis, N.: Wide residual networks. In: Proc. British Machine
Vision Conference (BMVC) (2016)
\bibitem{zhang2017mixup}
Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical
risk minimization. In: Proc. International Conference on Learning
Representations (ICLR) (2018)
\bibitem{zhang2019your}
Zhang, L., Song, J., Gao, A., Chen, J., Bao, C., Ma, K.: Be your own teacher:
Improve the performance of convolutional neural networks via self
distillation. In: Proc. IEEE International Conference on Computer Vision
(ICCV) (2019)
\bibitem{zhang2020adaptive}
Zhang, M.M., Marklund, H., Dhawan, N., Gupta, A., Levine, S., Finn, C.:
Adaptive risk minimization: A meta-learning approach for tackling group
shift. In: Proc. Neural Information Processing Systems (NeurIPS) (2021)
\bibitem{zhang2018deep}
Zhang, Y., Xiang, T., Hospedales, T.M., Lu, H.: Deep mutual learning. In: Proc.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
\bibitem{zheng2015scalable}
Zheng, L., Shen, L., Tian, L., Wang, S., Wang, J., Tian, Q.: Scalable person
re-identification: A benchmark. In: Proc. IEEE International Conference on
Computer Vision (ICCV) (2015)
\bibitem{zheng2017unlabeled}
Zheng, Z., Zheng, L., Yang, Y.: Unlabeled samples generated by gan improve the
person re-identification baseline in vitro. In: Proc. IEEE International
Conference on Computer Vision (ICCV) (2017)
\bibitem{zhong2020random}
Zhong, Z., Zheng, L., Kang, G., Li, S., Yang, Y.: Random erasing data
augmentation. In: Proc. AAAI Conference on Artificial Intelligence (AAAI)
(2020)
\bibitem{zhou2020learning}
Zhou, K., Yang, Y., Hospedales, T., Xiang, T.: Learning to generate novel
domains for domain generalization. In: Proc. European Conference on Computer
Vision (ECCV) (2020)
\bibitem{zhou2021dael}
Zhou, K., Yang, Y., Qiao, Y., Xiang, T.: Domain adaptive ensemble learning.
IEEE Transactions on Image Processing (TIP) (2021)
\bibitem{zhou2021domain}
Zhou, K., Yang, Y., Qiao, Y., Xiang, T.: Domain generalization with mixstyle.
In: Proc. International Conference on Learning Representations (ICLR) (2021)
\end{thebibliography}