SelfReplay: Adapting Self-Supervised Sensory Models via Adaptive Meta-Task Replay
Extracted source text · arXiv 2404.15305
\documentclass[conference]{IEEEtran}
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
% \usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\input{packages}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}
\input{project_commands}
% \title{Conference Paper Title*\\
% {\footnotesize \textsuperscript{*}Note: Sub-titles are not captured in Xplore and
% should not be used}
% \thanks{Identify applicable funding agency here. If none, delete this.}
% }
% \title{\proj{}: Adapting Self-Supervised Models for Edge Devices via Meta-Learned Pretext Replay}
\title{\proj{}: Adapting Pre-Trained Sensing Models\\ to End-Users via Self-Supervision Replay}
% \title{\proj{}: Adapting Pre-Trained Models by Replaying Self-Supervision for Mobile Sensing}
% \title{\proj{}: Adapting Pre-Trained Models by Replaying Self-Supervision on Edge Devices}
% \title{\proj{}: Adapting Pre-Trained Models by Replaying Self-Supervision for Mobile Sensing}
% \title{\proj{}: Few-Shot Domain Adaptation for Self-Supervised Models via Pretext Replay}
% \title{\proj{}: Few-Shot Domain Adaptation for Self-Supervised Models via Pretext Replay in Mobile Sensing}
\author{\IEEEauthorblockN{Hyungjun Yoon\IEEEauthorrefmark{1},
Jaehyun Kwak\IEEEauthorrefmark{1}, Biniyam Aschalew Tolera\IEEEauthorrefmark{1}, Gaole Dai\IEEEauthorrefmark{2}, \\
Mo Li\IEEEauthorrefmark{3}, Taesik Gong\IEEEauthorrefmark{4}, Kimin Lee\IEEEauthorrefmark{1} and
Sung-Ju Lee\IEEEauthorrefmark{1}}
\IEEEauthorblockA{\IEEEauthorrefmark{1}KAIST, \IEEEauthorrefmark{2}Nanyang Technological University, \\
\IEEEauthorrefmark{3}Hong Kong University of Science and Technology, \IEEEauthorrefmark{4}Nokia Bell Labs\\
Email: \{hyungjun.yoon, jaehyun98, biasc, kiminlee, profsj\}@kaist.ac.kr, \\
gaole001@e.ntu.edu.sg, lim@cse.ust.hk, taesik.gong@nokia-bell-labs.com}}
% \author{\IEEEauthorblockN{Hyungjun Yoon}
% \IEEEauthorblockA{\textit{KAIST} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% hyungjun.yoon@kaist.ac.kr}
% \and
% \IEEEauthorblockN{2\textsuperscript{nd} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% \and
% \IEEEauthorblockN{3\textsuperscript{rd} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% \and
% \IEEEauthorblockN{4\textsuperscript{th} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% \and
% \IEEEauthorblockN{5\textsuperscript{th} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% \and
% \IEEEauthorblockN{6\textsuperscript{th} Given Name Surname}
% \IEEEauthorblockA{\textit{dept. name of organization (of Aff.)} \\
% \textit{name of organization (of Aff.)}\\
% City, Country \\
% email address or ORCID}
% }
\maketitle
\input{src_submission/00_abstract}
\begin{IEEEkeywords}
Human Activity Recognition, Domain adaptation, Self-supervised learning, Meta-learning
\end{IEEEkeywords}
\input{src_submission/01_introduction}
\input{src_submission/02_related_work}
\input{src_submission/03_methodology}
\input{src_submission/04_evaluation}
\input{src_submission/05_discussion}
\input{src_submission/06_conclusion}
\input{references}
\end{document}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{bm}
\usepackage{multirow}
\usepackage{tabularx}
\usepackage{array}
% \usepackage{nth}
\usepackage{color}
\usepackage{enumitem}
\usepackage{makecell}
\usepackage{xcolor}
% \usepackage{tabularray}
\usepackage{colortbl}
\usepackage{booktabs}
% \usepackage{hhline}
\usepackage{algorithm}
\usepackage{algpseudocode}
% \usepackage{algcompatible}
\definecolor{my_green}{RGB}{227, 254, 223}
\definecolor{my_blue}{RGB}{237, 238, 254}
\newcolumntype{C}{>{\centering\arraybackslash}X}
\newcolumntype{B}{>{\columncolor{my_blue}}C}
\newcommand{\proj}{\emph{ADAPT$^2$}}
\definecolor{camouflagegreen}{rgb}{0.47, 0.53, 0.42}
\newcommand{\revision}[1]{{\leavevmode\color{camouflagegreen}{#1}}}
\newcommand{\hj}[1]{{\leavevmode\color{green}{[HJ: #1]}}}
\newcommand{\sj}[1]{{\leavevmode\color{magenta}{[SJ: #1]}}}
\newcommand{\moli}[1]{{\leavevmode\color{purple}{[Mo Li: #1]}}}
\newcommand{\km}[1]{{\leavevmode\color{blue}{[KM: #1]}}}
\newcommand{\ts}[1]{{\leavevmode\color{red}{[TS: #1]}}}
\newcommand{\jh}[1]{{\leavevmode\color{pink}{[JH: #1]}}}
\newcommand{\gaole}[1]{{\leavevmode\color{plum}{[Gaole: #1]}}}
\newcommand{\bn}[1]{{\leavevmode\color{yellow}{[BN: #1]}}}
\newcommand{\etal}{\textit{et al.}}
\newcommand{\eg}{\textit{e.g.},}
\newcommand{\ie}{\textit{i.e.},}
\newcommand{\cf}{\textit{c.f.},}
\definecolor{LightCyan}{rgb}{0.88,1,1}
\begin{abstract}
% Self-supervised learning has emerged as a method for utilizing a massive amount of unlabeled data, addressing the challenge of obtaining annotations for Human Activity Recognition (HAR). Yet, when these models are deployed to real-world users in mobile sensing, they encounter significant domain shifts due to the variability in activities across different users and devices. We investigate the performance degradation when self-supervised models are distributed to heterogeneous domains. To address the issue, we propose \proj{}, a domain adaptation framework to tune self-supervised models using very few data on edge devices. \proj{} employs meta-learning with a self-supervised learning objective during pre-training, followed by a target-side pretext replay for domain adaptation. This allows pre-trained models to adjust their pre-trained representations to the target domain with only a few samples. Evaluation of four real-world datasets demonstrates \proj{} outperforming existing baselines by an average F1-score improvement of 8.8\%. Moreover, our on-device computational overhead analysis on a commercial smartphone shows that \proj{} completes adaptation within three minutes with only a 9.54\% memory consumption. This demonstrates the computational efficiency diof the proposed method while improving model performance.
Self-supervised learning has emerged as a method for utilizing massive unlabeled data for pre-training models, providing an effective feature extractor for various mobile sensing applications. However, when deployed to end-users, these models encounter significant domain shifts attributed to user diversity. We investigate the performance degradation that occurs when self-supervised models are fine-tuned in heterogeneous domains. To address the issue, we propose \proj{}, a few-shot domain adaptation framework for personalizing self-supervised models. \proj{} proposes self-supervised meta-learning for initial model pre-training, followed by a user-side model adaptation by replaying the self-supervision with user-specific data. This allows models to adjust their pre-trained representations to the user with only a few samples. Evaluation with four benchmarks demonstrates that \proj{} outperforms existing baselines by an average F1-score of 8.8\%p. Our on-device computational overhead analysis on a commodity off-the-shelf (COTS) smartphone shows that \proj{} completes adaptation within an unobtrusive latency (in three minutes) with only a 9.54\% memory consumption, demonstrating the computational efficiency of the proposed method.
\end{abstract}
\section{Introduction}
\label{sec:intro}
The integration of deep learning into mobile sensing has broadened the scope of ubiquitous applications, such as contactless authentication~\cite{wang2021ear, liu2022recent}, sign language translation~\cite{hou2019signspeaker, park2021enabling}, and mobile health applications~\cite{zhang2019pdmove, song2020spirosonic}. Nevertheless, a major challenge in mobile sensing is the scarcity of labeled data, which is often expensive to acquire. As a breakthrough, self-supervised learning~\cite{haresamudram2022assessing} has been explored for its ability to train models with unlabeled data and transfer knowledge to downstream tasks. Self-supervised learning methods, such as Contrastive Predictive Coding~(CPC)~\cite{haresamudram2021contrastive, haresamudram2023investigating}, similarity-based contrastive learning~(SimCLR)~\cite{tang2020exploring}, and Multi-Task Learning~\cite{saeed2019multi}, have been showing their effectiveness in sensory applications without needing explicit labels.
% The integration of deep learning into mobile sensing has broadened the scope of applications, such as contactless authentication~\cite{wang2021ear}, sign language translation~\cite{hou2019signspeaker}, and health applications~\cite{song2020spirosonic, li2023neuralgait}. To meet the growing demands for the applications, a paradigm of pre-training models with extensive data and fine-tuning them for various downstream tasks has emerged~\cite{zhuang2020comprehensive}. In particular, self-supervised learning~\cite{haresamudram2022assessing} has been actively explored recently for training models with unlabeled data and transferring knowledge to downstream tasks. Self-supervised learning methods, such as Contrastive Predictive Coding~(CPC)~\cite{haresamudram2023investigating}, similarity-based contrastive learning~(SimCLR)~\cite{chen2020simple}, and Multi-Task Learning~\cite{doersch2017multi}, have been showing the their effectiveness in sensory applications~\cite{haresamudram2021contrastive, haresamudram2023investigating, tang2020exploring, saeed2019multi}. %Specifically, we target a scenario where individuals fine-tune the pre-trained models to align them with their own data.% their specific needs.
%A challenge arises in transitioning the pre-trained model to the end-user's fine-tuning.
A challenge arises when fine-tuning is performed by an end-user showing heterogeneity from the pre-training data. In mobile sensing, data collected in distinct environments varies significantly across individual users, devices, and settings (e.g., position and sampling rate)~\cite{ustev2013user}. Thus, the end-user data might have heterogeneous characteristics from those used for pre-training.
%Thus, the data used for pre-training might have heterogeneous characteristics from those observed by end-users.
This gap results in \textit{domain shift}, where models trained in one domain underperform when applied to another~\cite{stisen2015smart}. To illustrate this challenge within the self-supervised learning setting, we conducted an empirical analysis using CPC pre-trained models~\cite{haresamudram2023investigating} (experimental details in \S\ref{sec:domain_shift_effect}). In Fig.~\ref{fig:motivation_study}, the results show CPC is useful in downstream tasks when models are pre-trained in the same domain as the target domain (in-domain). However, there is a substantial decline in performance when models are pre-trained with different domains from the target domain (out-of-domain). This result emphasizes the challenges of deploying self-supervised models to diverse mobile sensing environments. %Based on the observation, we delve into the domain shift problem stemming from the self-supervised pre-trained models.
%Although fine-tuning was performed in domain, the results show a substantial decline in performance when pre-training occurred out of domain.
\begin{figure}
% \setlength{\belowcaptionskip}{-6pt}
\centering
\includegraphics[width=0.99\columnwidth]{figures/motivation_study_user.pdf}
\vspace{-5pt}
\caption{Efficacy of CPC~\cite{haresamudram2023investigating} pre-trained models on ICHAR~\cite{gong2019metasense} dataset, comparing F1-scores between in-domain and out-of-domain pre-training scenarios. Ten participants with different devices are shown as domains. Domain shifts deteriorate the performance of self-supervised models in heterogeneous environments.}
\vspace{-5pt}
\label{fig:motivation_study}
\end{figure}
Training a domain-specific model with data from the end user could be a straightforward solution, but this is infeasible considering the efforts and cost of gathering enough data from an individual user. To address the challenge, existing research encompasses domain generalization methods~\cite{qian2021latent, qin2022domain, lu2022semantic}, which strive to train domain-invariant features, or domain adaptation techniques~\cite{chang2020systematic, gong2019metasense, zhou2020xhar, wang2018stratified} that leverage a small portion of target domain data to achieve domain-specific performance. However, they primarily rely on labeled data for training, making them difficult to be applied to self-supervised learning which utilizes only unlabeled data for pre-training. %Additionally, our focus is on the domain shift between pre-training and fine-tuning stages, which was not investigated in the existing literature. %In our scenario, testing and fine-tuning occur in the same target domain. However, the domain gap originating from the pre-training phase can still pose significant challenges.
%differs from existing methods in perceiving domain shift. While most existing methods focus on the gap between training and testing domains, we focus
To mitigate the issue, we propose \proj{} as a few-shot domain adaptation framework to refine the self-supervised model into a personalized model specific to the end user. Inspired by the capability of meta-learning to ``learn to learn'' from few-shot data, we adapt the concept to our unsupervised pre-training setting. We introduce \textit{self-supervised meta-learning} for pre-training, which enables the model to ``learn to self-supervise'' with only a few data. This results in the model being adaptable to the learning task of self-supervised pre-training, \ie{} \textit{pretext task}. Next, we synergistically harness the adaptability by involving \textit{pretext replay} as a pivotal domain adaptation step. The end-user engages in adapting the pre-trained model by replaying the pretext task using its own few-shot data. This process effectively aligns the model to the representation reflecting the end user's domain property with very few data. %, thereby mitigating the impact of domain shifts.
It is noteworthy that our framework is designed to be agnostic to the self-supervision algorithm, and thus, it can be integrated into existing self-supervision methods. Fig.~\ref{fig:intro_overview} outlines a comparative overview of \proj{}'s operation against the standard self-supervised learning.
\begin{figure}
% \setlength{\belowcaptionskip}{-6pt}
\centering
\includegraphics[width=0.99\columnwidth]{figures/intro_overview.pdf}
\vspace{-15pt}
\caption{A high-level comparison between the standard self-supervised learning (left) and \proj{}(right).}
\vspace{-10pt}
\label{fig:intro_overview}
\end{figure}
% % original
% To address the domain shift challenges, we propose \proj{}, a novel framework that facilitates few-shot domain adaptation for self-supervised learning. First, \proj{} pre-trains models via \textit{self-supervised meta-learning}. This technique integrates self-supervised learning into meta-learning optimization, allowing for rapid adaptation to tasks with very few data. Second, \proj{} further refines the pre-trained model on the end-user side via a process named \textit{pretext replay}. Specifically, in the \textit{pretext replay} step, we adapt the pre-trained model for the target domain with the same pretext task but with a small amount of target domain data. This adapted model is then used for fine-tuning towards a specific downstream task. The rationale behind pretext replay is to align the model to the target domain as if it was pre-trained with target domain data, thereby mitigating the impact of domain shifts. In addition, our meta-learning component is a key enabler of effective pretext replay with very few samples from the target domain.
%We find that this additional adapting step is useful in improving transferability towards fine-tuning.
%Pre-training is followed by a few-shot domain adaptation to refine the pre-trained model on the end-user side. This is achieved by training the model via the pretext task used for pre-training, using few-shot target-domain data. This \textit{pretext replay} step is reinforced by the pre-trained model's adaptability to the pretext task, simulated during the self-supervised meta-learning.
% This adapted representation is then used for fine-tuning towards the downstream tasks, tailored to user applications.
% Our intuition behind employing pretext replay is straightforward: inadequately pre-trained models lead to suboptimal performance after fine-tuning. As models pre-trained out-of-domain function inadequately on the target side (Figure~\ref{fig:motivation_study}), we align the model as if to be pre-trained on the target domain, by additionally training the same pretext task using the target domain data. Note that target-side end-users lack data, which may lead to overfitting. To facilitate this adaptation in data-scarce environments, meta-learning is adopted as our core component.
% Our approach is grounded in the observation (\S\ref{sec:domain_shift_effect}) that a model pre-trained within the same domain shows better transferability towards fine-tuning. %For instance, a model pre-trained on motion data from a younger user group may struggle to extract relevant features from elderly groups, potentially misinterpreting characteristics like decreased activity level, while performing well in the younger group.
% Our pretext replay is designed to adjust the model to learn end-user-specific features within the same pretext task, and meta-learning is employed to facilitate few-shot adaptation considering the data scarcity.
% We evaluate \proj{} under domain shifts between the unlabeled pre-training and few-shot fine-tuning data. %On four real-world datasets with domain shifts, ICHAR~\cite{gong2019metasense}, HHAR~\cite{stisen2015smart}, PAMAP2~\cite{reiss2012introducing}, and DSA~\cite{altun2010comparative},
For evaluation, we used four Human Activity Recognition datasets~\cite{gong2019metasense, stisen2015smart, reiss2012introducing, altun2010comparative} that are representative datasets used for evaluating the performance under domain shifts. Our experiments demonstrate the superior performance of \proj{} to %three state-of-the-art self-supervised learning baselines~\cite{oord2018representation, chen2020simple, saeed2019multi} and two
domain generalization and adaptation baselines~\cite{zhang2022towards, lee2023self}, exhibiting a high F1-score that shows an $8.83\%$p improvement on average.
%. \proj{} exhibits a high F1-score that shows an $8.83\%$p improvement on average compared to the baselines.
To further assess the practical viability of executing fine-tuning with domain adaptation on mobile devices, we measure the computational overhead of \proj{} on an off-the-shelf smartphone. Our evaluation reveals that the adaptation step with pretext replay can be completed within three minutes, indicating marginal user-side computational overhead while achieving improved performance.
We summarize our main contributions as follows:
\begin{itemize}[left=1em, topsep=0pt, partopsep=0pt]
\item We investigate the domain shift problem arising when self-supervised models are deployed to diverse users for fine-tuning. %in widely used self-supervised learning on mobile sensing applications.
%\item We explore a practical domain shift scenario, where domain shift occurs between the unlabeled pre-training and labeled fine-tuning data.
Our research reveals that even after fine-tuning the pre-trained model with target domain data, the domain shift from the pre-training data can lead to significant performance degradation.
\item We introduce \proj{}, a few-shot domain adaptation framework that can be integrated into existing self-supervised learning methods.
%a self-supervised learning method-agnostic framework designed for few-shot domain adaptation of models pre-trained without labels.
\proj{} incorporates self-supervised meta-learning for the pre-training and target-side pretext replay, enabling the adaptation of pre-trained representations to the target domain using only a few target samples.
\item We perform rigorous evaluations using four mobile sensing datasets %~\cite{gong2019metasense, stisen2015smart, reiss2012introducing, altun2010comparative}
with diverse domain characteristics. We compare \proj{} against %three self-supervised learning baselines and two
domain generalization/adaptation baselines%~\cite{zhang2022towards, lee2023self}, %, oord2018representation, chen2020simple, saeed2019multi}. Our results show
, that shows \proj{} achieves an average F1-score improvement of $8.83\%$p.
\item We assess the computational overhead of deploying \proj{} on real-world mobile applications and demonstrate that the one-time target-side adaptation can be executed within three minutes %by utilizing 3.4 CPU cores
by consuming a mere $9.54\%$ of the device's memory on a COTS smartphone. %, making it practical for real-world mobile applications.
\end{itemize}
% \begin{figure}
% \vspace{0.8cm}
% \centering
% \includegraphics[width=0.4\columnwidth]{figures/intro.pdf}
% \caption{\hj{Just the concept drawing. Will be replaced with a pretty icons and drawings.}}
% \label{fig:problem_scenario}
% \end{figure}
\section{Related Work}
% We tackle the challenges arising from the domain shift when unsupervised pre-training is performed in domains heterogeneous from the fine-tuning. We present relevant prior work, targeting the same challenge, applicable to the unsupervised pre-training scenarios. Prior work is categorized into self-supervised learning, domain generalization and adaptation methods, and unsupervised meta-learning.
\subsection{Self-Supervised Learning}
Self-supervised learning trains models using an auxiliary task that can be defined without labels, which enables learning generalized features of the data. %The task for this self-supervision is called a pretext task, which enables learning generalized features of the data~\cite{}. %The model pre-trained through self-supervised learning works as a robust feature extractor across diverse downstream tasks.
Among numerous approaches, we focus on the methods applied for mobile sensing~\cite{haresamudram2022assessing}. Multi-Task Learning~\cite{saeed2019multi} utilized multiple types of synthetic augmentations on the data and trained task prediction networks to infer the occurrence of the augmentation. Sensor-specific augmentations were selected to make the model learn sensory properties. Recent work focuses on using contrastive learning~\cite{jaiswal2020survey}, which generates augmented views of data and trains the model with the objective of maximizing the similarity between the augmented views. Existing methods such as MoCo~\cite{he2020momentum} and SimCLR~\cite{chen2020simple} were applied to mobile sensing~\cite{wang2022sensor, tang2020exploring} by using sensory augmentations to generate views. The temporal property of time-series data is utilized for generating views in a recent work~\cite{eldele2023self}. Taking into account the multi-modality, Cosmo~\cite{ouyang2022cosmo}, COCOA~\cite{deldari2022cocoa}, and ColloSSL~\cite{jain2022collossl} utilized contrastive learning to maximize the similarity between the embeddings driven from different modalities in the same context. Contrastive predictive coding (CPC)~\cite{haresamudram2021contrastive, haresamudram2023investigating} defined another type of task, predicting the embedding of future segments within the data based on the previously aggregated embeddings. In a similar context, masked-reconstruction-based methods~\cite{haresamudram2020masked, xu2021limu} have been explored for mobile sensing, using the task of reconstructing the synthetically masked segment within the data.
While models trained through self-supervised learning are known to be generalizable across diverse tasks, %our research discovered that they suffer from
the potential performance decline when applied to the task of different domains~(shown in \S\ref{sec:domain_shift_effect}) is overlooked. Our work differs from the prior research in exploring the domain shift problem between pre-training and fine-tuning data.
\subsection{Domain Generalization and Domain Adaptation}
Domain generalization (DG)~\cite{wang2022generalizing} has been explored as a solution to mitigate domain shifts. DG includes approaches to learn domain-invariant features by adjusting the optimization objective~\cite{muandet2013domain, li2019episodic}, performing adversarial training to discriminate domains~\cite{li2018domain}, and processing the data to minimize the domain information from itself~\cite{wang2020heterogeneous, zhou2021domain}.
Recent research incorporated meta-learning~\cite{li2018learning, qiao2020learning} and self-supervised learning~\cite{kim2021selfreg, yao2022pcl} to define domain-invariant training objectives. In mobile sensing, %Generalizable Independent Latent Excitation (GILE)
GILE~\cite{qian2021latent} introduced a training scheme disentangling domain-specific information, while %Semantic-Discriminative Mixup (SDMix)
SDMix~\cite{lu2022semantic} employed semantic-aware augmentations to achieve DG tailored to activity recognition. Domain adaptation (DA) methods~\cite{wilson2020survey} offer a more suitable fit for our scenario, as they allow the utilization of fine-tuning data collected by end-users. Most approaches target to utilize unlabeled or a limited number of labeled target domain data~\cite{ganin2015unsupervised, rahman2020correlation} to adapt the model to the target domain. In activity recognition, DA has been approached as an efficient transfer learning problem~\cite{wang2018stratified, lu2021cross, khan2018scaling}, and methods employing feature matching and confusion maximization~\cite{chang2020systematic} have been proposed. MetaSense~\cite{gong2019metasense} introduced a meta-learning-based model training approach followed by few-shot adaptation to create domain-specific models. DAPPER~\cite{gong2021dapper} is proposed as another line of research for estimating the expected performance of DA in mobile sensing.
However, these approaches assume the availability of labels in the source domain, making them incompatible with our unsupervised pre-training scenario. DARLING~\cite{zhang2022towards} addresses the domain shift from unsupervised learning and covers the problem by integrating conditional optimization that optimizes the contrastive loss per domain. However, our approach differs from DARLING in that our method uses the available target domain data (\ie fine-tuning data) to train domain-specific models. In our evaluation (\S\ref{sec:overall_results}), we demonstrate that this utilization results in superior performance.
\begin{figure*}
% \setlength{\belowcaptionskip}{-8pt}
\centering
\includegraphics[width=0.95\textwidth]{figures/overview.pdf}
\vspace{-6pt}
\caption{Overview of \proj{} framework.}% The components within the white box represent the core components of \proj{}.}
\vspace{-10pt}
\label{fig:project_overview}
\end{figure*}
\subsection{Unsupervised Meta-Learning}
We consider unsupervised meta-learning (UML)~\cite{khodadadeh2019unsupervised, khodadadeh2020unsupervised, lee2023self} methods due to their effectiveness in few-shot adaptation, which is also applicable to our unsupervised pre-training scenario. Traditional methods~ employ pseudo-labeling data through augmentation~\cite{khodadadeh2019unsupervised} or generative methods~\cite{khodadadeh2020unsupervised}, followed by supervised meta-learning~\cite{andrychowicz2016learning} using the generated labels. Set-SimCLR~\cite{lee2023self}, during pre-training, trains a set encoder by creating sets of augmented samples from the same data, employing contrastive learning to maximize agreement between set embeddings. In fine-tuning, it composes sets of data by classes, generating class prototypes using the set encoder to initialize the classifier's parameters. These prototypes enable rapid adaptation for further few-shot fine-tuning. However, our approach differs in that we perform the adaptation to refine the encoder for the target domain, while Set-SimCLR primarily focuses on making the following classifier adaptable to few-shot fine-tuning. Our evaluation (\S\ref{sec:overall_results}) demonstrates the superior performance of our approach in mobile sensing scenarios.
\section{\proj{}:
Few-Shot Adaptation With Meta-Learned Pretext Replay
% Few-Shot Domain Adaptation for Self-Supervised Models
}
\subsection{Overview}
We present \proj{} as a framework designed to effectively tune a deployed self-supervised model into a personalized model on an end user's device. %Our approach aims to address the domain shift between the data used for self-supervised pre-training and the end user's data. %To achieve this, \proj{} modifies the typical pipeline of pre-training and fine-tuning self-supervised models (Fig.~\ref{}).
% utilizes the end user's data, facilitating additional self-supervised training on the representation model. However, considering the practical scenario in which end users have access to only a few samples, the scarcity of training samples could lead to overfitting. To address the challenge, we introduce a novel approach designed to enable the models to adapt effortlessly to the few-shot self-supervised learning. We propose self-supervised meta-learning, optimizing models via meta-learning during the unsupervised pre-training. Note that \proj{} is designed to be agnostic to both model architecture and the self-supervised learning method used for pre-training.
% We address the challenge by introducing a novel approach to pre-train models via self-supervised meta-learning. This method is designed to enable the models to adapt effortlessly to the few-shot self-supervised learning of the end user. Note that \proj{} is designed to be agnostic to both model architecture and the self-supervised learning method used for pre-training.
Fig.~\ref{fig:project_overview} shows \proj{} unfolds through two core stages: (1) pre-training models via \textit{self-supervised meta-learning} and (2) adapting pre-trained models via \textit{pretext replay}. The process begins with a model provider, utilizing a large amount of unlabeled data to train the model with a self-supervised objective optimized via meta-learning. This method is designed to enable the models to adapt effortlessly to the few-shot self-supervised learning of the end user. Once trained, these models are deployed to the edge devices of end users. Each user then engages in a domain adaptation process, replaying the learning task of the self-supervised pre-training (\ie{} pretext task) with the user's own few-shot data. This results in the model achieving a personalized representation, akin to having been pre-trained with the same user's data. The tuned models are subsequently fine-tuned again with few-shot labeled data, this time for downstream tasks, similar to the original self-supervised learning. Further details about these processes will be elaborated in the following sections.
% % original
% We aim to address the domain shift problem when pre-trained models are fine-tuned in heterogeneous target domains. To achieve our goal, we present \proj{}, a few-shot domain adaptation method to refine pre-trained representations to align with the target domain. Figure~\ref{fig:project_overview} shows \proj{} unfolds through two core stages. First, we perform \textit{self-supervised learning via meta-learning} for pre-training models without labels. \proj{} defines meta-learning tasks composed of only a few domain-specific data, optimizing for self-supervised learning objective (\ie{} pretext task). This approach results in a pre-trained model that is highly adaptable to the pretext task with few-shot examples from a single domain. Next, target-side few-shot fine-tuning is performed. We split this fine-tuning into two adaptation stages: domain adaptation using \textit{pretext replay} and adaptation to the downstream task objective. Our domain adaptation refines the model while replaying the same pretext task used for pre-training but with target domain data in this stage. Thanks to the adaptability achieved through meta-learning, the model is effectively refined as a pre-trained model tailored for the target domain. Afterward, we fine-tune the domain-adapted model for the end-user application, optimizing it for the downstream task objective with labels.
%Our goal is to optimize the performance of pre-train models when they are fine-tuned for a distinct target domain.
% We present \proj{}, a domain adaptation approach to refine pre-trained representations for effective application in heterogeneous target domains. As depicted in Figure~\ref{fig:project_overview}, \proj{} unfolds through two core stages. First, model providers perform self-supervised pre-training via meta-learning. \proj{} defines meta-learning tasks composed of only a few data divided by domains to be optimized with self-supervised objectives. This approach generates a highly adaptable model to the pretext task based on limited data from a single domain. Next, we fine-tune the pre-trained model using few-shot examples collected from end-users. We divide the fine-tuning step into two adaptation stages: %(1) first, \proj{} fine-tunes the model using end-users' data while replaying the same pretext task used for pre-training and (2) then we further update the model using labeled data based on downstream task objective.
% the first \textit{ADAPT} towards the target domain via \textit{pretext replay} and the second \textit{ADAPT} towards the downstream task. To tailor the pre-trained model to their specific domain, \proj{} fine-tunes the model using end-users' data while replaying the same pretext task used for pre-training. This step outputs an adapted user-specific model, which is finally fine-tuned for downstream tasks.
%This adaptation makes the model face the simulated training environment of meta-learning, effectively fitting it to the target domain using only a few examples. The domain-adapted model trains the downstream tasks specific to the application.
\subsection{Problem Formulation}
Our target scenario is the same as typical self-supervised learning, where a model is pre-trained through self-supervision with unlabeled data, followed by fine-tuning on a target task (in our case, this is an end user's edge device) using very few labeled data. This approach is designed to reflect two key challenges: the difficulty of obtaining costly annotations for pre-training, and the constraints faced by end users in gathering enough data and executing large-scale training on resource-limited edge devices. We explore a setup where data for self-supervised pre-training originate from domains that are completely separate from that of the end user. We assume the availability of domain labels in the pre-training data. Domain labels involve the type of measurement device or anonymized user annotation, which are generally more accessible to acquire as metadata.
\textbf{Notations.} We use $x$, $y$, and $d$ to indicate the sensory feature, label, and domain information (\eg{} anonymized user or device), respectively. We denote the unlabeled pre-training dataset with $N^{\text{pt}}$ samples as $S^{\text{pt}}=\left\{(x_i, d_i) \,|\, i=1...N^{\text{pt}}\right\}$. The set of $d_i$ in $S^{\text{pt}}$ composes the source domain, referred as $D^s$. The fine-tuning dataset with $N^{\text{ft}}$ samples is defined as $S^{\text{ft}}=\left\{(x_i, y_i) \,|\, i=1...N^{\text{ft}}\right\}$, where label $y_i$ is available. Data $x_i$ in $S^{\text{ft}}$ originates from a single target domain $D^t$, which is distinct from $D^s$, \ie{} $D^t \cap D^s =
\emptyset$. The fine-tuned model is tested in the same target domain $D^t$.
% % original
% We target a setup where (1) pre-training is performed using large, unlabeled data and (2) fine-tuning using very few labeled data. This setup reflects the difficulties of acquiring privacy-sensitive label information from the model provider side and collecting enough training data from the end-user side. We assume model providers have access to the domain information indicating distinct user or device characteristics, considering the practicality of acquiring those data as metadata, even in anonymized form.
% We denote the pre-training dataset with $N^{\text{pt}}$ samples as $S^{\text{pt}}=\left\{(x_i, d_i) \,|\, i=1...N^{\text{pt}}\right\}$ where $x_i$ represents sensor data and $d_i$ represents the domain information (\eg{} anonymized user or device). The source domain $D^s$ is composed of the set of domain labels, \ie{} $d_i \in D^s$. %for all domain labels.
% % The superset of $d_i$ comprises the source domain $D^s$, \ie{} $D^s=\cup_{i=1}^{N^{\text{pt}}}d_i$.
% The fine-tuning dataset with $N^{\text{ft}}$ samples is defined as $S^{\text{ft}}=\left\{(x_i, y_i) \,|\, i=1...N^{\text{ft}}\right\}$, where label $y_i$ of downstream task is available with sensor data $x_i$. Data $x_i$ in $S^{\text{ft}}$ originates from a single target domain $D^t$, which is distinct from $D^s$, \ie{} $D^t \cap D^s =
% \emptyset$. The fine-tuned model is tested in the same target domain $D^t$.
\subsection{Self-Supervised Meta-Learning}
In \proj{}, we pre-train an \textit{adaptable} self-supervised model through the integration of meta-learning. Meta-learning~\cite{andrychowicz2016learning}, referred to as the method for ``learning to learn,'' is renowned for its efficacy in fine-tuning models for unseen environments using only a few data. However, the incorporation of meta-learning into our framework presents a challenge, as it traditionally relies on supervised training, whereas our approach assumes the absence of labeled data. \proj{} redefines meta-learning by shifting from a supervised to a self-supervised training objective. This shift implies that the model %, now trained under a self-supervised regime, is not suited for conventional supervised fine-tuning: it
transitions from ``learning to supervise'' to ``learning to self-supervise.'' %Consequently, our meta-learned model is tailored for few-shot adaptation aligning with the self-supervised learning objective, namely the pretext task, employed during the meta-learning.
Our design is grounded on the intuition that meta-learning, effective for enhancing supervised learning with few-shot data, can be equally potent in a self-supervised context, allowing our pre-trained models to adapt effectively in a self-supervised manner, even with only a few data.
Therefore, we devise a unique adaptation approach for our meta-learned model; we replay the self-supervised training with the end user's few-shot data by performing the same pretext task. We detail this \textit{pretext replay} in \S\ref{sec:pretext_replay}.
% Therefore, we adopt a unique adaptation approach for our meta-learned model. We replay the self-supervised training but integrate the end user's few-shot data into the same pretext task. This strategy is grounded on the intuition that meta-learning, effective for enhancing supervised learning with limited data, can be equally potent in a self-supervised context. This allows our pre-trained models to adapt effectively in a self-supervised manner, even with only a few data. We detail this \textit{pretext replay} in \S\ref{sec:pretext_replay}.
Our meta-learning implementation is based on Model-Agnostic Meta-Learning (MAML)~\cite{finn2017model}. We aim to identify the model parameters that effectively perform pretext tasks within new domains in minimal gradient steps. Our objective closely aligns with the core idea of MAML, training initial parameters for optimal performance on new tasks with minimal gradient updates. It is noteworthy that the model-agnostic nature of MAML enables \proj{} to be adapted flexibly, regardless of the type of self-supervised learning methods.
MAML structures a set of tasks, $\mathcal{T}$, to emulate the process of few-shot training and subsequent testing. Each task, $\mathcal{T}_i$, involves training the model using a small number of examples to optimize task-specific weights, $\theta_i$. The dataset for training these weights is denoted as the \textit{support set}, \ie{} $\mathcal{S}_i$. The trained task-specific weights are then evaluated on a distinct dataset termed the \textit{query set}, \ie{} $\mathcal{Q}_i$. The evaluation results from each task are aggregated over all tasks and summed into a loss value to optimize the actual model weights.% Through this iteration, the model learns \textit{how to learn over few-shot samples} for higher performance by mimicking the few-shot adaptation within a task.
\textbf{Task Generation.} We design the meta-learning task $\mathcal{T}_i$ to facilitate domain-specific few-shot adaptation. As we aim to adapt the model to data from a single target domain, we configure each task to compose the data samples in $\mathcal{S}_i$ and $\mathcal{Q}_i$ from a single domain $d$. We call these composed tasks as domain-specific tasks. $d$ is randomly selected from $D^s$. We further mitigate the risk of overfitting to domains based solely on $D^s$ by leveraging multi-conditioned tasks. Drawing on a prior study~\cite{gong2019metasense2}, we note that incorporating a small proportion (\eg{} 30\%) of tasks comprised of samples from various random domains can enhance the robustness of the meta-learned model. These multi-conditioned tasks, formed from domain-agnostic random samples, function as new synthetic domains. While not representing real domains, we ensure they are not the primary, accounting for only a small fraction of the total task set $\mathcal{T}$. Algorithm~\ref{alg:task_generation} summarizes our meta-learning task generation.
% % original
% Given the data scarcity from end-users, \proj{} employs meta-learning~\cite{andrychowicz2016learning} for pre-training for few-shot adaptation in the later stage. Our meta-learning implementation is based on Model-Agnostic Meta-Learning (MAML)~\cite{finn2017model}. We aim to identify the model parameters that effectively perform pretext tasks within new domains in minimal gradient steps. Our objective closely aligns with the core idea of training initial parameters for optimal performance on new tasks with minimal gradient updates. Furthermore, the model-agnostic nature of MAML ensures \proj{}'s adaptability to different types of self-supervised learning methods (\eg{} CPC, SimCLR, and Multi-Task Learning).%, expanding its utility across a wider spectrum of applications.
% %to facilitate applicability across various training objectives (\eg{} CPC, SimCLR, and Multi-Task Learning). In contrast to typical training,
% In MAML, meta-learning composes a set of tasks (denoted as $\mathcal{T}$) that simulates few-shot learning and the corresponding evaluation. Using only a few examples, the model trains task-specific weights, $\theta_i$, within each task, $\mathcal{T}_i$. The dataset used for training the task-specific weights is denoted as a \textit{support set}, \ie{} $\mathcal{S}_i$. The trained task-specific weights are then evaluated on a distinct dataset named a \textit{query set}, \ie{} $\mathcal{Q}_i$. The evaluation results %for each task-specific weight
% from each task are aggregated over all tasks and generate a loss value to optimize the actual model weights. Through this iteration, the model learns \textit{how to learn over few-shot samples} for higher performance by mimicking the few-shot adaptation within a task.
% We carefully design the meta-learning task $\mathcal{T}$ to facilitate domain-specific few-shot adaptation (stage~2 in Figure~\ref{fig:project_overview}). As we aim to adapt the model to %the environment of
% a single target domain, we define each task to compose $\mathcal{S}_i$ and $\mathcal{Q}_i$ from a single domain by replicating the later adaptation. We call these composed tasks domain-specific tasks. %Besides, the pre-trained model should be able to adapt to diverse, unseen domains.
% For each $\mathcal{T}_i$, a single domain is randomly selected from the source domain set $D^s$.
% % Besides, for pre-training the model to learn adaptable weights towards diverse and unseen domains, we compose %the dataset of
% % each task $\mathcal{T}_i$ by different, randomly selected domains.
% We further mitigate the risk of overfitting to domains based solely on $D^s$ by leveraging multi-conditioned tasks~\cite{gong2019metasense}. The multi-conditioned tasks are formed by random samples selected regardless of the domain, thus working as new types of domains. With a small portion of randomly sampled tasks, we avoid the potential overfitting to the source-domain-specific features. Algorithm~\ref{alg:task_generation} summarizes our task generation.
% \begin{algorithm}
% \caption{Task Generation}
% \label{alg:task_generation}
% \begin{flushleft}
% \textbf{Output:} Support and query sets composed for a set of tasks $\mathcal{T}$
% \end{flushleft}
% \begin{algorithmic}[1]
% \For{$i \in \{1, 2, ..., N\}$} \Comment{$N$ representing number of tasks}
% \If{$i \leq N_\text{dom}$} \Comment{$N_\text{dom}$ representing number of domain-specific tasks}
% \State $d\sim \text{RandomSample}(D_s)$
% \State $\mathcal{S}_{\mathcal{T}_i}\sim \text{RandomSample}(S^{(d)}_{\text{pt}}, K)$ \Comment{Where $K$ denotes the task size, $S^{(d)}_{\text{pt}}$ denotes dataset with domain $d$}
% \State $\mathcal{Q}_{\mathcal{T}_i}\sim \text{RandomSample}(S^{(d)}_{\text{pt}}, K)$
% \Else
% \State $\mathcal{S}_{\mathcal{T}_i}\sim \text{RandomSample}(S_{\text{pt}}, K)$
% \State $\mathcal{Q}_{\mathcal{T}_i}\sim \text{RandomSample}(S_{\text{pt}}, K)$
% \EndIf
% \State Assign$(\mathcal{S}_{\mathcal{T}_i}, \mathcal{Q}_{\mathcal{T}_i})$ to $\mathcal{T}_i$
% \EndFor
% \end{algorithmic}
% \end{algorithm}
\begin{algorithm}
\caption{\proj{}'s Task Generation for Meta-Learning}
\label{alg:task_generation}
\begin{flushleft}
\textbf{Inputs:} Pre-train dataset $S^{\text{pt}}$, number of tasks $M$ and number of domain-specific tasks $M^{\tt dom}<M$
\end{flushleft}
\begin{algorithmic}[1]
\For{$j \in \{1, 2, ..., M\}$}
\If{$j \leq M^{\tt dom}$}
\State Select domain $d$ from $D^s$ uniformly at random
\State Select $K$ samples with domain $d$ randomly:
\Statex \hspace{\algorithmicindent}\hspace{\algorithmicindent} $\mathcal{S}=\{x_i \in S^{\text{pt}} |d_i = d \}$ such that $|\mathcal{S}| = K$
\State Select another set of $K$ samples with $d$ randomly:
\Statex \hspace{\algorithmicindent}\hspace{\algorithmicindent} $\mathcal{Q}=\{x_i \in S^{\text{pt}} |d_i = d \}$ such that $|\mathcal{Q}| = K$
\Else
\State Select $K$ samples randomly:
\Statex\hspace{\algorithmicindent}\hspace{\algorithmicindent} $\mathcal{S}=\{x_i \in S^{\text{pt}}\}$ such that $|\mathcal{S}| = K$
\State Select another set of $K$ samples randomly:
\Statex\hspace{\algorithmicindent}\hspace{\algorithmicindent} $\mathcal{Q}=\{x_i \in S^{\text{pt}} \}$ such that $|\mathcal{Q}| = K$
\EndIf
\State Update a set of task $\mathcal{T} \leftarrow \mathcal{T} \cup (\mathcal{S},\mathcal{Q})$
\EndFor \\
\Return Task set $\mathcal{T}$
\end{algorithmic}
\end{algorithm}
\textbf{Parameter Updates.} Our framework trains adaptable parameters to optimize for minimal evaluation loss in a single domain, through simulating few-shot self-supervised learning tasks. For each task $\mathcal{T}_i$, composed from a random single domain. It trains task-specific weights, $\theta_i$, using $\mathcal{S}_i$ and evaluates the corresponding task-specific performance using $\mathcal{Q}_i$. The loss function $\mathcal{L}_\texttt{SSL}$, used for both training and evaluation, is tailored to self-supervised learning. Importantly, our framework is designed to be agnostic to self-supervised learning methods, allowing for a versatile loss function application. During the training, task-specific losses are aggregated across all tasks and then utilized to optimize the model parameters, $\theta$. The iteration of this process results in model parameters that are highly adaptable for few-shot self-supervised learning across different domains. Algorithm~\ref{alg:ssml} outlines the process of parameter updates via self-supervised meta-learning. $\alpha$ and $\beta$ denote the learning rates for task-specific training and model parameter update.
% % original
% \proj{} performs self-supervised learning to pre-train models. % via meta-learning.
% As label information is not accessible in $S^{\text{pt}}$, parameter update with supervised objectives of the traditional meta-learning cannot be applied. Instead, we replace the supervised objectives of task-specific weight optimization with a pretext (\ie{} self-supervised) task that can be defined without labels.
% Integrating self-supervised objectives into meta-learning is agnostic to the type of the pretext task. In this study, we focus on the three most widely-used self-supervised learning methods in mobile sensing: CPC~\cite{oord2018representation}, SimCLR~\cite{chen2020simple}, and Multi-Task Learning~\cite{saeed2019multi}.
% With the integration of self-supervised learning objectives, meta-learned models learn \textit{how to learn with the pretext task}. This implies its adaptability towards few-shot fine-tuning is not on the downstream task but on the pretext task. To fully utilize this adaptability, we perform a few-shot adaptation via the same pretext task in the following step. We detail this target-side \textit{pretext replay} in \S\ref{sec:pretext_replay}.
% Integrating self-supervised objectives into meta-learning is agnostic to the type of the pretext task. While \proj{} can accommodate a variety of self-supervised learning methods, this study focuses on the implementation based on the pretext tasks of the most popular self-supervised learning methods in mobile sensing: CPC~\cite{oord2018representation}, SimCLR~\cite{chen2020simple}, and Multi-Task Learning~\cite{saeed2019multi}.
% \begin{algorithm}
% \caption{Self-Supervised Meta-Learning}
% \label{alg:ssml}
% \begin{flushleft}
% \textbf{Output:} Adaptable model weights,
% \end{flushleft}
% \begin{algorithmic}[1]
% \State $f \gets \text{one of } f_{\text{CPC}}, f_{\text{SimCLR}}, \text{ or } f_{\text{Multi}}$
% \State $\mathcal{L} \gets \text{one of } \mathcal{L}_{\text{CPC}}, \mathcal{L}_{\text{SimCLR}}, \text{ or } \mathcal{L}_{\text{Multi}}$ \Comment{Set optimization objective as a self-supervised pretext task}
% \For{epochs}
% \State $\mathcal{T} =$ TaskGeneration($S^{\text{pt}}$)
% \For{$\mathcal{T}_i \in \mathcal{T}$}
% \State $\theta’_i \gets \theta-\alpha \nabla_\theta \mathcal{L}_{\mathcal{S}_{\mathcal{T}_i}}(f_\theta)$ \Comment{Optimize task-specific weights using $\mathcal{S}_{\mathcal{T}_i}$}
% \State Evaluate $\mathcal{L}_{\mathcal{Q}_{\mathcal{T}_i}}(f_{\theta'_i})$ \Comment{Evaluate trained task-specific weights using $\mathcal{Q}_{\mathcal{T}_i}$}
% \EndFor
% \State $\theta \gets \theta-\beta\nabla _\theta \sum_{}^{}\mathcal{L}_{\mathcal{Q}_{\mathcal{T}_i}}(f_{\theta'_i})$
% \EndFor
% \end{algorithmic}
% \end{algorithm}
\begin{algorithm}
\caption{\proj{}'s Self-Supervised Meta-Learning}
\label{alg:ssml}
\begin{flushleft}
\textbf{Inputs:} Parameters $\theta$ and self-supervised loss $\mathcal{L}_\texttt{SSL}$%: CPC~(\eqref{eq:cpc}), SimCLR~(\eqref{eq:simclr}) or Multi~(\eqref{eq:multi})
\end{flushleft}
\begin{algorithmic}[1]
\For{epochs}
\State $\mathcal{T} =$ TaskGeneration($S^{\text{pt}}$)
\For{$\mathcal{T}_i =(\mathcal{S}_i, \mathcal{Q}_i) \in \mathcal{T}$}
\State Optimize task-specific weights using $\mathcal{S}_i$:
\Statex\hspace{\algorithmicindent}\hspace{\algorithmicindent} $\theta_i \gets \theta-\alpha \nabla_\theta \mathcal{L}_\texttt{SSL}(\theta; \mathcal{S}_i)$
\State Evaluate loss of updated model on $\mathcal{Q}_i$:
\Statex\hspace{\algorithmicindent}\hspace{\algorithmicindent} Compute $\mathcal{L}_\texttt{SSL}(\theta_i; \mathcal{Q}_i)$
\EndFor
\State Update parameters $\theta \gets \theta-\beta\nabla _\theta \sum_{i} \mathcal{L}_\texttt{SSL}(\theta_i; \mathcal{Q}_i)$
\EndFor
\end{algorithmic}
\end{algorithm}
% % original
% Algorithm~\ref{alg:ssml} describes the parameter update conducted with self-supervised meta-learning. For each generated task, $\mathcal{T}_i$, we train task-specific weights $\theta_i$ with a task-specific learning rate, denoted as $\alpha$ (Line 4). The task-specific weights are then evaluated using the query set (Line 5), and the evaluated losses are aggregated over all tasks to update the model parameter $\theta$ (Line 7), where $\beta$ represents the meta-learning rate. For all optimizations, we define the parameterized function $f_\theta$ and the corresponding loss function $\mathcal{L}$ to align with the self-supervised pretext task.
% For instance, if the pretext task is CPC, the loss function is designed to enhance distinguishability by encouraging the model to differentiate the true future embedding ($g_\theta({x_i^{t+j}})$) from other candidate embeddings ($g_\theta({x_i^l})$). This is achieved by comparing the similarity with the prediction from aggregated embedding before the time step ($f_\theta(x_i^{t})$). Denoting the parameterized encoder as $g_\theta$, predictor as $f_\theta$, and segmented data of $x_i$ at time step $t$ as $x_i^t$, we use the loss function defined as
% % enhance distinguishability by encouraging the model to differentiate the true future embedding from other candidate embeddings by comparing the similarity with the aggregated embedding before the time step. $f_\theata$, $g_\theata$, $h_\theata$ are parameterized functions for aggregator, encoder, and predictor
% % \begin{equation}
% % \mathcal{L}_{\text{CPC}} = -\log\left(\frac{e^{\text{sim}(\mathbf{z}_{t+j}, \mathbf{c_t})}}{e^{\text{sim}(\mathbf{z}_{t+j}, \mathbf{c_t})} + \sum_{k=1}^{K}e^{\text{sim}(\mathbf{z}_k, \mathbf{c_t})}}\right), \label{eq:cpc}
% % \end{equation}
% % \vspace{-2mm}
% \begin{equation}
% \mathcal{L}^{\text{CPC}}(\theta;S) = - \sum_{x_i\in S} \log\left(\frac{e^{\text{sim}(g_\theta(x_i^{t+j}), f_\theta(x_i^t))}}{e^{\text{sim}(g_\theta(x_i^{t+j}), f_\theta(x_i^{t}))} + \sum_{l\neq j}e^{\text{sim}(g_\theta(x_i^{t+l}), f_\theta(x_i^{t}))}}\right). \label{eq:cpc}
% \end{equation}
% % \begin{equation}
% % \mathcal{L}^{\text{CPC}}(\theta;S) = - \sum_{x_i\in S} \sum_{t=1}^{T} \sum_{j=1}^{J} \log\left(\frac{e^{\text{sim}(g_\theta(x_i^{t+j}), h_\theta(f_\theta(x_i^t)))}}{e^{\text{sim}(g_\theta(x_i^{t+j}), h_\theta(f_\theta(x_i^{t})))} + \sum_{l\neq j}e^{\text{sim}(g_\theta(x_i^{t+l}), h_\theta(f_\theta(x_i^{t})))}}\right), \label{eq:cpc}
% % \end{equation}
% For SimCLR, the loss function is set to minimize the similarity between the embeddings of two augmented views of the same data ($x'_i$ and $x''_i$) while ensuring dissimilarity with other data points ($x_k$). With the parameterized encoder $f_\theta$, the loss becomes
% % \begin{equation}
% % \mathcal{L}_{\text{SimCLR}} = -\log\left(\frac{e^{\text{sim}(\mathbf{z}_i, \mathbf{z}_j)}}{e^{\text{sim}(\mathbf{z}_i, \mathbf{z}_j)} + \sum_{k=1}^{K}e^{\text{sim}(\mathbf{z}_k, \mathbf{z}_j)}}\right),
% % \end{equation}
% % \vspace{-3mm}
% \begin{equation}
% \mathcal{L}^{\text{SimCLR}} (\theta; S) = - \sum_{x_i\in S} \log\left(\frac{e^{\text{sim}(f_\theta(x'_i), f_\theta(x''_i))}}{e^{\text{sim}(f_\theta(x'_i), f_\theta(x''_i))} + \sum_{k\neq i} e^{\text{sim}(f_\theta(x'_i), f_\theta(x_k))}}\right). \label{eq:simclr}
% \end{equation}
% Multi-Task Learning loss is assessed by comparing the probability %$h_\theta^t(f_\theta(x'_i))$
% $p_\theta^t(x'_i)$ of an internal prediction task $t$, which involves predicting whether an augmentation is applied to $x'_i$, to an automatically generated label $y_t$ indicating the occurrence of augmentation. Denoting $T$ as the set of prediction task and $p_\theta^t$ as the task-specific classifier composed of an encoder $f_\theta$ and task-specific prediction head $h_\theta^t$ of task $t$, \ie{} $p_\theta^t=h_\theta^t\circ f_\theta$, the loss becomes
% % \begin{equation}
% % \mathcal{L}_{\text{MTL}} = -\sum_{t \in T} \left[y_t \cdot \log(p_t) + (1 - y_t) \cdot \log(1 - p_t)\right], \label{eq:multi}
% % \end{equation}
% % \vspace{-3mm}
% % \begin{equation}
% % \mathcal{L}^{\text{MTL}} (\theta; S) = - \sum_{x_i\in S} \sum_{t \in T} \left[y_t \cdot \log(h_\theta^t(f_\theta(x'_i))) + (1 - y_t) \cdot \log(1 - h_\theta^t(f_\theta(x'_i)))\right]. \label{eq:multi}
% % \end{equation}
% \begin{equation}
% \mathcal{L}^{\text{MTL}} (\theta; S) = - \sum_{x_i\in S} \sum_{t \in T} \left[y_t \cdot \log(p_\theta^t(x'_i)) + (1 - y_t) \cdot \log(1 - p_\theta^t(x'_i)))\right]. \label{eq:multi}
% \end{equation}
% Based on the replaced optimization, our meta-learning procedure produces a model adaptive to a specific domain's data for the \textit{self-supervised pretext task}. This enhanced adaptability simplifies the subsequent design of the \textit{pretext replay} step, further facilitating domain adaptation to the target domain.
\subsection{Adapting Pre-trained Models via Pretext Replay}
\label{sec:pretext_replay}
We introduce \textit{pretext replay} as the core methodology for our few-shot domain adaptation. This adaptation involves refining the model using the same self-supervised learning objective (\ie{} pretext task) as in the pre-training step but with the end user’s data. This step serves as an additional training stage on the user's edge device, facilitating domain adaptation prior to the actual fine-tuning for the subsequent supervised downstream task (e.g., Human Activity Recognition).
Importantly, pretext replay demonstrates its effectiveness when coupled with self-supervised meta-learning. On the contrary, typical self-supervised models are prone to overfitting when exposed to few-shot training through pretext replay. Similarly, relying solely on self-supervised meta-learning without the subsequent pretext replay step does not yield optimal adaptability for supervised fine-tuning. We propose pretext replay as the unique key bridging the self-supervised pre-training and the subsequent supervised fine-tuning, enabled by the integration of meta-learning in self-supervised pre-training. This synergy between the self-supervised meta-learning and pretext replay is the core aspect of our approach, and its efficacy is validated in an ablation study detailed in \S\ref{sec:ablation}.
The model weights, $\theta$, are updated using the parameterized function and the corresponding loss $\mathcal{L}_\texttt{SSL}$, originating from the self-supervised meta-learning. Pretext replay uses the fine-tuning data set $S^\text{ft}$, without labels, and is performed in a few steps (e.g., 10) with a fixed learning rate, as formulated in the equation:
\begin{equation}
\theta \gets \theta-\alpha \nabla_{\theta} \mathcal{L}_\texttt{SSL}(\theta;S^\text{ft}) \text{.}
\end{equation}
After the adaptation, the model undergoes fine-tuning for the downstream task, which is similar to the normal self-supervised learning setup. $S^\text{ft}$ with the label information is employed to fine-tune $\theta$ towards the downstream task. We utilize the popular linear evaluation protocol~\cite{wu2018unsupervised} for fine-tuning, only training a simple classification head attached after the frozen encoder. The final model trained through this fine-tuning is used for the end-user application.
% % original
% After pre-training, \proj{} conducts domain adaptation with the target domain data, using the meta-learned pretext task (\eg{} CPC, SimCLR, or Multi-Task Learning) as the training objective. This process, represented as \textit{pretext replay}, functions as an additional pre-training step within the target domain. Its goal is to develop a domain-specific representation that can be effectively applied to the downstream tasks in the target domain. Our intuition behind pretext replay is straightforward: models pre-trained out-of-domain tend to yield sub-optimal performance for fine-tuning in the target domain. To address this issue, we align the model as if it were pre-trained on the target domain by additionally training the same pretext task using the target domain data. %Note that target-side end-users lack data, which may lead to ineffective training. We highlight that this pretext replay is enabled in data-scarce environments by meta-learning that trains adaptability toward the pretext task.
% % While pretext tasks in self-supervised learning requires a massive number of data to learn effective representations, our self-supervised meta-learning allows pretext replay to fast adapt to target domain with a limited number of samples.
% % Our intuition behind employing pretext replay is that inadequately pre-trained models lead to suboptimal performance after fine-tuning. As models pre-trained out-of-domain function inadequately on the target side, we align the model as if to be pre-trained on the target domain, by additionally training the same pretext task using the target domain data. Note that target-side end-users lack data, which may lead to overfitting. To facilitate this adaptation in data-scarce environments, meta-learning is adopted in the previous stage.
% % Our intuition behind this adaptation is based on our empirical observations (detailed in \S\ref{sec:domain_shift_effect}) indicating that pre-trained models from different domains often suffer a significant performance drop in downstream tasks, even after fine-tuning with the target domain data. Recognizing that a domain shift from the pre-training domain leads to this performance drop, our approach refines the pre-trained model, aligning it more closely with a model pre-trained on the target domain.
% Pretext replay uses few-shot samples from $S^{\text{ft}}$. While pretext tasks in self-supervised learning require massive data to learn effective representations, our self-supervised meta-learning allows pretext replay to adapt fast to the target domain with a limited number of samples. Combining self-supervised meta-learning with pretext replay makes \proj{} an effective solution for end-users with limited data.
% We update the pre-trained model weights $\theta$ by reusing the parameterized function and the corresponding loss $\mathcal{L}$ originated from the pretext task (\eg{} $\mathcal{L}^\text{CPC}$, $\mathcal{L}^\text{SimCLR}$, or $\mathcal{L}^\text{MTL}$). Here, the data used for the training is $S^\text{ft}$, but without labels. The adaptation is performed in a few steps (\eg{} 10), with a fixed learning rate. The parameter update is formulated as
% %\vspace{-3mm}
% % \begin{equation}
% % \theta \gets \theta-\alpha \nabla_{\theta} \mathcal{L}(\theta;S^\text{PR}) \text{, where } S^\text{PR} = \{x_i \,|\, (x_i, y_i) \in S^{\text{ft}}, i=1...N^\text{ft}\} \text{.}
% % \end{equation}
% \begin{equation}
% \theta \gets \theta-\alpha \nabla_{\theta} \mathcal{L}_\texttt{SSL}(\theta;S^\text{ft}) \text{.}
% \end{equation}
% After the adaptation, the model is finally fine-tuned to the specific downstream task for the end-user application (\eg{} human activity recognition or gesture recognition). $S^\text{ft}$ with the label information is used for the fine-tuning of $\theta$ towards downstream tasks. We utilize the linear evaluation protocol~\cite{wu2018unsupervised} for fine-tuning, only training a simple classification head attached after the frozen encoder. The final model trained through this fine-tuning is used for the end-user application.
\section{Experiments}
\label{sec:experiments}
% \subsection{Setup}
% We evaluate \proj{} on four different datasets, covering human activity recognition through sensory data measured in the real world. Datasets are selected to assess the effectiveness of \proj{} across various domain types. We benchmark its performance against three state-of-the-art self-supervised learning methods and two domain generalization and adaptation methods.
\subsection{Datasets}
We set Human Activity Recognition (HAR) as the representative benchmark for evaluating \proj{}. HAR is known for its heterogeneity in users and devices~\cite{ustev2013user, stisen2015smart}, making it an effective benchmark for demonstrating the challenges associated with domain shifts. Datasets are selected to assess the effectiveness of \proj{} across various domain types. %We set two datasets dividing domains by user-device combination and two datasets dividing domains by device position.
\noindent\textbf{ICHAR~\cite{gong2019metasense}} comprises inertial measurement unit (IMU) data for classifying nine types of daily activities, such as walking, running, and stair climbing. Data is measured from ten participants using different models of mobile devices (seven smartphones and three watches). %, resulting in varied sampling rates corresponding to each device's specifications (100~Hz-500~Hz).
Each participant is treated as a unique domain.
\noindent\textbf{HHAR~\cite{stisen2015smart}} is designed for classifying six daily activities collected from nine users with a combination of four smartwatches and eight smartphones. %The activities were recorded with the devices operating at their maximum sampling capabilities, which ranged between 50~Hz and 250~Hz.
We define domains based on distinct user-device pairs.
\noindent\textbf{PAMAP2~\cite{reiss2012introducing}} classifies 12 different activity types using data collected from IMUs placed on three body locations: the wrist, chest, and ankle. %Data from nine subjects were recorded at a consistent sampling rate of 100~Hz.
Domains are divided by device positions.
\noindent\textbf{DSA~\cite{altun2010comparative}} encompasses a wide array of 19 daily and sports activities data, gathered from eight participants wearing five IMUs on the torso, arms, and legs. %The sensors recorded data at a frequency of 25~Hz.
We used device positions as domains.
\subsection{Data Preprocessing}
We preprocess all datasets using a fixed window size of 256 and an overlap of 128, following a prior work~\cite{gong2019metasense}. Additionally, we standardize the entire dataset to fall within the range of -1 to 1. We focus exclusively on the 3-channel accelerometer data from all available sources. We exclude data from domains with fewer than 500 samples to ensure enough data for training. As a result, we use 20 domains for HHAR, representing combinations of five users and four devices. For all datasets, we first split $70\%$ of the data and use $90\%$ of the splitted data for pre-training and the $10\%$ as the corresponding validation. With the remaining $30\%$ split, we use only a few samples for the few-shot fine-tuning and divide the remaining data with a $5:5$ ratio to compose the validation and test sets. We ensure that the pre-training, fine-tuning, and testing data are completely separated.
\subsection{Baselines}
We benchmark \proj{} against baselines chosen for their efficacy in mitigating domain shift between the unsupervised pre-training and the following fine-tuning. Note that most existing domain generalization~\cite{qian2021latent, qin2022domain, lu2022semantic} and adaptation~\cite{chang2020systematic, gong2019metasense, zhou2020xhar, wang2018stratified} methods assume labeled data for pre-training and thus they are not applicable to our scenario. We found two approaches that fit our scenario. % among the methods. %These methods are tailored for contrastive learning~\cite{} and meta-learning in unsupervised scenarios~\cite{}, thereby aligning with our research context.
\noindent\textbf{DARLING~\cite{zhang2022towards}} is a domain generalization method that uses contrastive learning. It optimizes the contrastive loss by composing intra-domain negative samples. Encouraging the intra-domain discrimination for every sample during the training, the model learns domain-invariant features. %DARLING's domain-specific optimization of the contrastive loss allows for the creation of more robust domain-invariant representations during the self-supervised pre-training phase.
\noindent\textbf{Set-SimCLR~\cite{lee2023self}} %We consider unsupervised meta learning (UML) methods~\cite{} to be compared with our approach, as they enable effective adaptation using a few examples, in the scenario whereas the label is unavailable during pre-training. Set-SimCLR
%stands out as a state-of-the-art
is an unsupervised meta-learning method that employs a set encoder to enhance the agreement between sets of augmented samples originating from identical source data. Both an instance encoder and the set encoder are trained through contrastive learning. In fine-tuning, the set encoder generates class prototypes from sample sets by class and is used to set the initial weights of the classifier. The resulting classifier, adjusted by the prototypes, facilitates rapid adaptation to novel conditions with the initial weights.
% \noindent\textbf{CPC~\cite{haresamudram2023investigating}} serves as a self-supervised learning baseline. CPC facilitates the acquisition of meaningful representations by training models to predict forthcoming embeddings generated by aggregating prior embeddings within a data sample. This method empowers the model to extract temporal relationships from the data's structure.
% \noindent\textbf{SimCLR~\cite{chen2020simple}} generates augmented views from the original data sample and trains the model to generate closer embeddings from each other while making other data samples dissimilar. We utilize SimCLR designed for wearables~\cite{tang2020exploring}, which provides augmentations tailored to sensory data.
% \noindent\textbf{Multi-Task Learning~\cite{saeed2019multi}} trains the mutual information between various tasks to learn shared knowledge. In the Multi-Task Learning tailored for wearables, diverse sensory augmentations are applied to the original data, defining a distinct prediction task for each augmentation. A common encoder is employed across all prediction tasks, facilitating the training of shared information and representations.
\subsection{Implementation Details}
\proj{} serves as a model- and method-agnostic framework applicable to different self-supervised learning approaches. For a fair comparison with baselines, we selected SimCLR, designed for wearables~\cite{tang2020exploring}, as the primary self-supervised method. This selection aligns with DARLING and Set-SimCLR, which are also presented based on contrastive learning. To maintain consistency, all baseline models utilize a network architecture identical to \proj{}, ensuring compatibility.
We implement \proj{} based on additional self-supervised learning methods, further showing its generalizability. We incorporate two other self-supervised learning approaches popular in HAR: CPC~\cite{haresamudram2023investigating} and Multi-Task Learning~\cite{saeed2019multi}. Note that we do not extend \proj{} to be built on DARLING and Set-SimCLR. DARLING mainly offers an intra-domain negative sampling method, a concept we already integrate within \proj{} but with a different approach, focusing on creating small tasks for meta-learning. As for Set-SimCLR, it requires the training of a separate set encoder, which requires an additional training component to \proj{}, deviating from our framework's design.
% \noindent\textbf{CPC~\cite{haresamudram2023investigating}} serves as a self-supervised learning baseline. CPC facilitates the acquisition of meaningful representations by training models to predict forthcoming embeddings generated by aggregating prior embeddings within a data sample. This method empowers the model to extract temporal relationships from the data's structure.
% \noindent\textbf{SimCLR~\cite{chen2020simple}} generates augmented views from the original data sample and trains the model to generate closer embeddings from each other while making other data samples dissimilar. We utilize SimCLR designed for wearables~\cite{tang2020exploring}, which provides augmentations tailored to sensory data.
% \noindent\textbf{Multi-Task Learning~\cite{saeed2019multi}} trains the mutual information between various tasks to learn shared knowledge. In the Multi-Task Learning tailored for wearables, diverse sensory augmentations are applied to the original data, defining a distinct prediction task for each augmentation. A common encoder is employed across all prediction tasks, facilitating the training of shared information and representations.
To implement the backbone networks corresponding to the self-supervised learning methods, we used 1D convolutional neural networks (CNN) followed by a projection head of a fully connected layer. Our network design, including the architectures and hyperparameters, aligns with established practices in the prior assessment study~\cite{haresamudram2022assessing}. Specifically for CPC, we utilized the latest state-of-the-art version designed for HAR~\cite{haresamudram2023investigating}. We conducted a grid search to optimize the hyperparameters for each baseline. For pre-training, we explored learning rates from \{1e-4, 5e-4, 1e-3, 5e-3\}, batch sizes from \{64, 128, 256\}, with specific values of \{1024, 2048, 4096\} for SimCLR-based methods, and weight decays from \{0, 1e-4\}. During fine-tuning, we used a fixed learning rate of $0.005$ for the linear evaluation protocol and a reduced learning rate of $0.001$ for end-to-end fine-tuning, which exhibited superior performance. We trained models with 100 epochs for pre-training and 20 epochs for fine-tuning using Adam optimizer. We utilized parameters within the same range as the baselines for learning rate and weight decay for meta-learning. We searched for the task-specific learning rate from \{1e-3, 5e-3, 1e-2\} and the inner iteration steps for pretext replay from \{10, 20, 30\}. We kept the number of domain-specific tasks fixed at eight and multi-conditioned tasks at four. The task size was set to 128 for our evaluation. Note that meta-learning requires larger epochs for convergence than conventional training as fewer parameter updates happen inside an epoch; we set 5K epochs for meta-learning training. We implemented the methods using PyTorch, and the training was performed on NVIDIA TITAN Xp GPUs.
\subsection{Evaluation Protocol and Metric}
We employed the leave-one-domain-out setting~\cite{qian2021latent}. For each domain in the dataset, we designate it as the target domain for fine-tuning and testing purposes while utilizing all other domains for pre-training. We conduct evaluations in this manner for each domain, rotating through each one as the target domain and then averaging the results. We select few-shot (\eg{} 1, 2, 5, 10) samples for fine-tuning, and then evaluate the performance in the same target domain. We set two fine-tuning protocols: linear evaluation and end-to-end fine-tuning. We consider linear evaluation the basic method for fine-tuning, which uses the pre-trained encoder as a frozen feature extractor and exclusively trains a following linear classification layer. All experiments using \proj{} used the linear evaluation protocol. As \proj{} involves the pretext replay step refining the encoder parameters, we further use end-to-end fine-tuning protocol for baseline methods by fine-tuning the entire network without freezing the encoder. It is for a comparison in a fair setting when encoder parameter update is allowed in both cases. The evaluation is performed with five random seeds, and we report the average value and standard deviations. To measure performance, we utilize the macro-averaged F1-score as our evaluation metric. This metric is chosen due to its ability to handle the class imbalance present in the data.
\begin{table*}[]
\centering
\caption{F1-scores of \proj{} built upon different self-supervised learning methods. \\Results are compared with the base methods. The highest and comparable scores are highlighted in bold.}
\label{tab:eval_ssl}
{\renewcommand{\arraystretch}{1.1}
\begin{tabularx}{0.85\textwidth}{l *{6}{C}}
\Xhline{2\arrayrulewidth}
& & \multicolumn{2}{c}{Domain: User} & \multicolumn{2}{c}{Domain: Position} & \\ \cmidrule(lr){3-4} \cmidrule(lr){5-6}
Pre-train & Fine-tune & ICHAR & HHAR & PAMAP2 & DSA & \cellcolor{my_blue}Avg. \\ \Xhline{2\arrayrulewidth}
\multirow{2}{*}{SimCLR \cite{tang2020exploring}} & Linear eval. & 0.745\scalebox{0.7}{ $\pm$ 0.024} & 0.866\scalebox{0.7}{ $\pm$ 0.008} & 0.549\scalebox{0.7}{ $\pm$ 0.016} & 0.391\scalebox{0.7}{ $\pm$ 0.006} & \cellcolor{my_blue}0.638\scalebox{0.7}{ $\pm$ 0.014}\\
& End-to-end & 0.663\scalebox{0.7}{ $\pm$ 0.028} & 0.836\scalebox{0.7}{ $\pm$ 0.029} & \textbf{0.589\scalebox{0.7}{ $\pm$ 0.046}} & 0.253\scalebox{0.7}{ $\pm$ 0.022} & \cellcolor{my_blue}0.585\scalebox{0.7}{ $\pm$ 0.031}\\
\rowcolor{my_green}\multicolumn{2}{c}{SimCLR + \proj{} (ours)} & \textbf{0.836\scalebox{0.7}{ $\pm$ 0.011}} & \textbf{0.903\scalebox{0.7}{ $\pm$ 0.004}} & \textbf{0.639\scalebox{0.7}{ $\pm$ 0.030}} & \textbf{0.526\scalebox{0.7}{ $\pm$ 0.019}} & \textbf{0.726\scalebox{0.7}{ $\pm$ 0.016}} \\ \hline
\multirow{2}{*}{CPC \cite{haresamudram2023investigating}} & Linear eval. & 0.765\scalebox{0.7}{ $\pm$ 0.016} & 0.846\scalebox{0.7}{ $\pm$ 0.005} & 0.379\scalebox{0.7}{ $\pm$ 0.017} & 0.371\scalebox{0.7}{ $\pm$ 0.005} & \cellcolor{my_blue}0.590\scalebox{0.7}{ $\pm$ 0.011} \\
& End-to-end & \textbf{0.816\scalebox{0.7}{ $\pm$ 0.013}} & \textbf{0.849\scalebox{0.7}{ $\pm$ 0.021}} & 0.484\scalebox{0.7}{ $\pm$ 0.026} & 0.352\scalebox{0.7}{ $\pm$ 0.017} & \cellcolor{my_blue}0.625\scalebox{0.7}{ $\pm$ 0.019}\\
\rowcolor{my_green}\multicolumn{2}{c}{CPC + \proj{} (ours)} & \textbf{0.826\scalebox{0.7}{ $\pm$ 0.008}} & \textbf{0.871\scalebox{0.7}{ $\pm$ 0.005}} & \textbf{0.527\scalebox{0.7}{ $\pm$ 0.017}} & \textbf{0.419\scalebox{0.7}{ $\pm$ 0.008}} & \textbf{0.661\scalebox{0.7}{ $\pm$ 0.009}} \\ \hline
\multirow{2}{*}{Multi-task \cite{saeed2019multi}} & Linear eval. & 0.716\scalebox{0.7}{ $\pm$ 0.010} & 0.877\scalebox{0.7}{ $\pm$ 0.003} & 0.630\scalebox{0.7}{ $\pm$ 0.003} & 0.456\scalebox{0.7}{ $\pm$ 0.004} & \cellcolor{my_blue}0.670\scalebox{0.7}{ $\pm$ 0.005} \\
& End-to-end & 0.718\scalebox{0.7}{ $\pm$ 0.019} & \textbf{0.865\scalebox{0.7}{ $\pm$ 0.030}} & \textbf{0.636\scalebox{0.7}{ $\pm$ 0.015}} & 0.378\scalebox{0.7}{ $\pm$ 0.021} & \cellcolor{my_blue}0.649\scalebox{0.7}{ $\pm$ 0.021} \\
\rowcolor{my_green}\multicolumn{2}{c}{Multi-Task + \proj{} (ours)} & \textbf{0.794\scalebox{0.7}{ $\pm$ 0.015}} & \textbf{0.891\scalebox{0.7}{ $\pm$ 0.005}} & \textbf{0.659\scalebox{0.7}{ $\pm$ 0.016}} & \textbf{0.578\scalebox{0.7}{ $\pm$ 0.011}} & \textbf{0.731\scalebox{0.7}{ $\pm$ 0.012}} \\ \Xhline{2\arrayrulewidth}
\end{tabularx}}
\vspace{-4pt}
\end{table*}
\begin{table*}[]
\centering
\caption{F1-scores of \proj{} and baseline methods for 10-shot fine-tuning across four datasets, \\with the highest and comparable scores highlighted in bold.}
\label{tab:eval_baselines}
{\renewcommand{\arraystretch}{1.1}
\begin{tabularx}{0.85\textwidth}{{l}*{6}{C}}
\Xhline{2\arrayrulewidth}
& & \multicolumn{2}{c}{Domain: User} & \multicolumn{2}{c}{Domain: Position} & \\ \cmidrule(lr){3-4} \cmidrule(lr){5-6}
Pre-train & Fine-tune & ICHAR & HHAR & PAMAP2 & DSA & \cellcolor{my_blue}Avg. \\ \Xhline{2\arrayrulewidth}
% \multicolumn{2}{c}{Random} & 0.223\scalebox{0.7}{ $\pm$ 0.065} & 0.337\scalebox{0.7}{ $\pm$ 0.067} & 0.234\scalebox{0.7}{ $\pm$ 0.026} & 0.043\scalebox{0.7}{ $\pm$ 0.016} & \cellcolor{my_blue}0.209\scalebox{0.7}{ $\pm$ 0.044} \\ \hline
\multirow{2}{*}{SimCLR \cite{tang2020exploring}} & Linear eval. & 0.745\scalebox{0.7}{ $\pm$ 0.024} & 0.866\scalebox{0.7}{ $\pm$ 0.008} & 0.549\scalebox{0.7}{ $\pm$ 0.016} & 0.391\scalebox{0.7}{ $\pm$ 0.006} & \cellcolor{my_blue}0.638\scalebox{0.7}{ $\pm$ 0.014} \\
& End-to-end & 0.663\scalebox{0.7}{ $\pm$ 0.028} & 0.836\scalebox{0.7}{ $\pm$ 0.029} & \textbf{0.589\scalebox{0.7}{ $\pm$ 0.046}} & 0.253\scalebox{0.7}{ $\pm$ 0.022} & \cellcolor{my_blue}0.585\scalebox{0.7}{ $\pm$ 0.031} \\ \hline
\multirow{2}{*}{Set-SimCLR \cite{lee2023self}} & Linear eval. & 0.758\scalebox{0.7}{ $\pm$ 0.010} & 0.814\scalebox{0.7}{ $\pm$ 0.004} & 0.487\scalebox{0.7}{ $\pm$ 0.011} & 0.283\scalebox{0.7}{ $\pm$ 0.007} & \cellcolor{my_blue}0.585\scalebox{0.7}{ $\pm$ 0.008} \\
& End-to-end & 0.747\scalebox{0.7}{ $\pm$ 0.029} & 0.848\scalebox{0.7}{ $\pm$ 0.016} & 0.573\scalebox{0.7}{ $\pm$ 0.015} & 0.165\scalebox{0.7}{ $\pm$ 0.012} & \cellcolor{my_blue}0.583\scalebox{0.7}{ $\pm$ 0.018} \\ \hline
\multirow{2}{*}{DARLING \cite{zhang2022towards}} & Linear eval. & 0.749\scalebox{0.7}{ $\pm$ 0.019} & 0.831\scalebox{0.7}{ $\pm$ 0.003} & 0.551\scalebox{0.7}{ $\pm$ 0.012} & 0.399\scalebox{0.7}{ $\pm$ 0.008} & \cellcolor{my_blue}0.633\scalebox{0.7}{ $\pm$ 0.011} \\
& End-to-end & 0.656\scalebox{0.7}{ $\pm$ 0.019} & 0.844\scalebox{0.7}{ $\pm$ 0.026} & \textbf{0.580\scalebox{0.7}{ $\pm$ 0.042}} & 0.258\scalebox{0.7}{ $\pm$ 0.024} & \cellcolor{my_blue}0.584\scalebox{0.7}{ $\pm$ 0.028} \\ \hline
\rowcolor{my_green}\multicolumn{2}{c}{SimCLR + \proj{} (ours)} & \textbf{0.836\scalebox{0.7}{ $\pm$ 0.011}} & \textbf{0.903\scalebox{0.7}{ $\pm$ 0.004}} & \textbf{0.639\scalebox{0.7}{ $\pm$ 0.030}} & \textbf{0.526\scalebox{0.7}{ $\pm$ 0.019}} & \textbf{0.726\scalebox{0.7}{ $\pm$ 0.016}} \\ \Xhline{2\arrayrulewidth}
\end{tabularx}}
\vspace{-4pt}
\end{table*}
\begin{figure*}
\centering
\includegraphics[width=0.8\textwidth]{figures/baseline_shots.pdf}
\vspace{-8pt}
\caption{Average F1-scores of \proj{} and the baselines across different shot numbers (1, 2, 5, 10).}
\vspace{-10pt}
\label{fig:baseline_shots}
\end{figure*}
\subsection{Results}
\subsubsection{Integration With Different Self-Supervised Learning Methods}
\label{sec:ssl_results}
Table~\ref{tab:eval_ssl} depicts the evaluation against various self-supervised learning methods, SimCLR, CPC, and Multi-Task Learning. Fine-tuning is performed with ten-shot samples, and \proj{} frameworks are implemented on each baseline for a fair comparison. \proj{} consistently enhances F1-scores across different self-supervised learning methods: 8.8\%p for SimCLR, 7.1\%p for CPC, and 4.1\%p for Multi-Task Learning on average. The level of improvement varies depending on the self-supervised learning method, indicating that the impact of domain shift is different among self-supervised learning methods. We explore this topic further in our subsequent analysis (see \S\ref{sec:domain_shift_effect}). Nevertheless, \proj{} consistently outperforms all methods as a flexible framework. Furthermore, we observe that the best-working base method varies across datasets. For example, SimCLR excels in ICHAR, while Multi-Task Learning performs better in DSA. This variability highlights \proj{}'s method-agnostic versatility, allowing flexible integration with the most effective self-supervised learning method for specific applications.
\subsubsection{Comparison with Baselines}
\label{sec:overall_results}
Table~\ref{tab:eval_baselines} shows the performance of \proj{} compared with the domain generalization and adaptation baselines. The experiment was based on ten-shot fine-tuning. %Set-SimCLR and DARLING are implemented based on SimCLR, thus, we report the results from SimCLR pre-training. \proj{} is also implemented based on SimCLR for a fair comparison.
The bold font values indicate the highest value in the same column. Overall, \proj{} consistently shows the highest F1 scores for all datasets, regardless of the domain type. Set-SimCLR and DARLING occasionally produce comparable scores; however, their enhancements are inconsistent and lack overall improvement. This outcome suggests that the baselines struggle to capture the domain-specific features, given their reliance on a fixed encoder pre-trained out-of-domain. End-to-end fine-tuning sometimes boosts performance; its efficacy depends on the fine-tuning dataset, leading to unpredictable performance enhancements. We attribute this to the sensitivity of few-shot fine-tuning, which is highly influenced by the data quality. End-to-end fine-tuning does not help the baselines, as their design lacks the adaptability of the encoder for few-shot adaptation. With an average increase of 8.8\%p in the F1-score, \proj{} works as a robust domain adaptation method.
\begin{figure*}
\centering
\includegraphics[width=0.7\textwidth]{figures/ssl_shots.pdf}
\vspace{-6pt}
\caption{Average F1-scores of \proj{}, built upon different self-supervised learning methods across different shot numbers (1, 2, 5, 10).}
\vspace{-10pt}
\label{fig:ssl_shots}
\end{figure*}
\subsubsection{Generalization across Different Shot Numbers}
We evaluate the generalizability of \proj{} across different fine-tuning shot numbers. Fig.~\ref{fig:baseline_shots} illustrates the F1-scores for \proj{} and the baseline methods, showcasing average improvements in the F1-score of 4.4\%p, 15.5\%p, 11.4\%p, and 8.8\%p across varying shot numbers ($k\in \{1, 2, 5, 10\}$). Fig.~\ref{fig:ssl_shots} presents results when \proj{} is built upon various self-supervised learning methods, consistently demonstrating superior performance. This indicates the efficacy of our approach, making it suitable for extreme data-scarce scenarios, such as one-shot learning.
\begin{figure*}
\centering
\includegraphics[width=0.8\textwidth]{figures/ablation.pdf}
\vspace{-6pt}
\caption{F1-scores of the ablation study. We compare \proj{} against 1) the baseline self-supervised learning method, 2) the baseline only with pretext replay, and 3) the baseline only with meta-learning. Ten-shot fine-tuning is performed.}
\vspace{-10pt}
\label{fig:ablation}
\end{figure*}
% \subsubsection{Ablation Study}
\subsubsection{Effect of Self-Supervised Meta-Learning and Pretext Replay}
\label{sec:ablation}
We conduct an ablation study to investigate the impact of technical components within \proj{}. These components include \textit{self-supervised meta-learning} for pre-training and \textit{pretext replay} for domain adaptation. We compare the performance of \proj{} with and without the components to assess their contributions. We build \proj{} upon SimCLR, CPC, and Multi-Task Learning and compare it to three variants: (1) self-supervised learning without any \proj{} components (baseline), (2) self-supervised learning with pretext replay but no meta-learning, (3) self-supervised learning with meta-learning but no pretext replay.
Fig.~\ref{fig:ablation} illustrates the results of our ablation study, revealing that each component has a varying impact depending on the self-supervised learning method. For SimCLR, applying only pretext replay to the base model results in a significant performance drop. However, when meta-learning is incorporated, \proj{} outperforms the setting without pretext replay. This suggests that \textit{meta-learning is the key enabler for the domain adaptation through pretext replay} for SimCLR. Additionally, SimCLR with meta-learning achieves outstanding results compared with the baseline, indicating that self-supervised meta-learning alone results in learning meaningful features. We attribute this to the feature learning capability of meta-learning~\cite{raghu2019rapid}, enabling the acquisition of meaningful common features across domains for rapid adaptation. %In summary, the adaptation step with pretext replay improves performance, and both components should be used together for SimCLR.
In the case of CPC, pretext replay enhances the baseline without meta-learning, while using only meta-learning yields poor results. Yet, when pretext replay is applied to the meta-learned model, performance significantly increases, exceeding the improvement observed from the baseline. This suggests that the adaptation step with pretext replay generally aids in learning meaningful features, and our meta-learning significantly amplifies this effect.
For Multi-Task Learning, pretext replay results in performance degradation in the baseline, while meta-learning improves it. Unlike SimCLR or CPC, the impact of pretext replay on the meta-learned model is minimal. Pretext replay slightly improves performance in HHAR and PAMAP2 while even slightly decreasing performance in DSA and ICHAR. This indicates that user-specific adaptation does not provide substantial benefits in the case of Multi-Task Learning. Importantly, meta-learning prevents pretext replay from significantly decreasing performance due to overfitting.
\begin{figure}
% \setlength{\belowcaptionskip}{-8pt}
\centering
\includegraphics[width=\columnwidth]{figures/umap.pdf}
\caption{2D UMAP visualizations of embeddings from CPC pre-trained models. Embeddings are color-coded by class.}% Results of before and after applying \proj{}'s pretext replay are shown.}
\vspace{-10pt}
\label{fig:umap}
\end{figure}
Fig.~\ref{fig:umap} visualizes the effectiveness of pretext replay when combined with meta-learning, on a domain of ICHAR. We plot 2D UMAP visualizations~\cite{mcinnes2018umap} of embeddings from CPC pre-trained models, color-coded by class. Both CPC baseline and \proj{} before pretext replay lack distinct class-based clustering. It suggests that the models pre-trained out-of-domain fail to grasp meaningful features in the target domain. Meanwhile, \proj{} after pretext replay creates well-defined clusters aligned with class boundaries, demonstrating that pretext replay significantly enhances the model's ability to capture meaningful features within the target domain.
In summary, our self-supervised meta-learning effectively supports pretext replay, enhancing target-domain performance by training the model to learn general features that can be rapidly refined into domain-specific features. Another finding is that the level of improvement varies depending on the self-supervised learning method. This observation leads us to hypothesize that self-supervised learning methods are differently affected by domain shifts, which we analyze further in the following section (\S\ref{sec:domain_shift_effect}).
\subsubsection{Domain Shift Effect of Different Self-Supervised Learning Methods} \label{sec:domain_shift_effect}
\begin{figure*}
% \setlength{\belowcaptionskip}{-10pt}
\centering
\includegraphics[width=0.75\textwidth]{figures/domain_effect.pdf}
\vspace{-6pt}
\caption{Fine-tuning performance comparison between models pre-trained in-domain and out-of-domain settings. Ten-shot fine-tuning is performed for all settings. Performance drops between the settings are shown in red.}
\vspace{-10pt}
\label{fig:domain_effect}
\end{figure*}
Our ablation study demonstrated varying effectiveness of pretext replay across different self-supervised learning methods. To understand the discrepancies, we delved deeper into how domain shifts affect different self-supervised learning methods. We pre-trained models using CPC, SimCLR, and Multi-Task Learning in a leave-one-domain-out setup and then fine-tuned them in a novel domain. We term it as an out-of-domain setting. For comparison, we established an in-domain setting for baseline, where pre-training and fine-tuning occur within the same domain. When moving from in-domain to out-of-domain, the performance drop quantifies each method's vulnerability to domain shift. While our initial motivational analysis (in \S\ref{sec:intro}) compared models pre-trained with different data sizes (the out-of-domain setting had larger data), this study ensures a fair comparison by equalizing the data size across both in-domain and out-of-domain settings.
Fig.~\ref{fig:domain_effect} illustrates the impact of domain shift on various self-supervised learning methods. When models are fine-tuned and tested on data that differ from their pre-training domain, a common trend of performance decline is observed, highlighting the negative effects of domain shifts. However, the level of this decline varies with the self-supervised learning method. Specifically, models pre-trained with CPC show a notable average drop of 19.15\%p F1-score, whereas SimCLR exhibits a 6.7\%p decrease and Multi-Task Learning with a more modest decline of 4.95\%p.
% The degree to which domain shifts influence model performance is closely tied to the nature of the pretext task utilized during pre-training. For instance, the predictive task in CPC, which involves forecasting future data segments from previous ones, is markedly sensitive to domain specifics. Take, for example, a CPC model trained on data from a younger demographic with a pattern of increasing activity over time. This model might falter when applied to an older demographic characterized by a decreasing activity trend, as the temporal features it has learned do not carry over to the different activity patterns of the new domain. On the other hand, the pretext task in Multi-Task Learning, centered on identifying applied data augmentations, exhibits a more domain-agnostic quality. Within this framework, a model trained to recognize data manipulations such as rotations or scalings in one demographic will likely perform competently in another as such augmentation features are less tied to the specific activity patterns of different user groups.
This variability suggests that the impact of domain shift is dependent on the self-supervised learning method employed during pre-training. CPC's predictive task—forecasting future segments based on past segments—tends to be highly domain-specific. For example, a model pre-trained with CPC on data from a younger user group showing increasing activity levels over time might not perform well on data from old users who exhibit a decline in activity levels. With a different temporal trend, the model's predictive features may not generalize across these domain variations. Conversely, Multi-Task Learning's pretext task involves identifying the type of augmentation applied to the data and is less domain-specific. In the previous example, knowledge acquired from detecting augmentations in one user group's data could transfer more effectively to another group, since recognizing augmented data characteristics (\eg{} rotation) does not necessarily depend on user-specific trends.
These findings teach us two insights: (1) domain shift negatively impacts the fine-tuning performance of pre-trained models, and (2) the degree of this impact depends on the pretext task's sensitivity to domains. Our ablation study verifies this, showing that the effectiveness of our pretext replay varies across different self-supervised learning methods. While we have assessed the domain shift effects on three popular methods, our results underscore the necessity to investigate various self-supervised learning methods. %Such investigation is essential for ensuring the effective deployment of pre-trained models across highly heterogeneous mobile environments.
\begin{table}[]
\centering
\caption{Computational overhead of few-shot adaptation (Adapt) and fine-tuning (Tune) for self-supervised learning methods.}
\vspace{-4pt}
\label{tab:overhead}
{\renewcommand{\arraystretch}{1.1}
\begin{tabularx}{0.9\columnwidth}{l *{6}{C}}
\Xhline{2\arrayrulewidth}
& \multicolumn{2}{c}{CPC} & \multicolumn{2}{c}{SimCLR} & \multicolumn{2}{c}{Multi-Task} \\ \cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7}
Metric & Adapt & Tune & Adapt & Tune & Adapt & Tune \\ \hline
Time (sec) & 179.20 & 84.03 & 36.05 & 61.80 & 18.76 & 62.33 \\
CPU (\%) & 44.53 & 13.86 & 31.86 & 10.57 & 32.15 & 10.22 \\
Mem (\%) & 9.54 & 16.28 & 4.84 & 7.26 & 4.28 & 6.16 \\ \Xhline{2\arrayrulewidth}
\end{tabularx}}
\vspace{-10pt}
\end{table}
\subsubsection{Computational Overhead}
\label{sec:computational_ovehread}
Our primary goal is to enable the practical deployment of pre-trained models to end-users, particularly those with resource-constrained edge devices. To this end, we assess the computational feasibility of \proj{} for mobile devices, focusing on its user-side operations: pretext replay and fine-tuning to downstream tasks. %, which are crucial for practical deployment on resource-constrained mobile devices.
Our few-shot adaptation strategy is designed to minimize the computational load during these stages. We performed on-device training with a Samsung Galaxy S20 Ultra device equipped with 8 CPU cores and 12GB RAM to evaluate its effectiveness. Our on-device training is implemented via Termux~\cite{termux}, a Linux terminal emulator for Android, to execute PyTorch-based training code. The training protocol follows the same setting of our main evaluation in \S\ref{sec:overall_results}. The measurement metrics are execution time in seconds and CPU and memory consumption in percentage.
Table~\ref{tab:overhead} presents the overhead from our pretext replay (Adapt) and fine-tuning (Tune), measured independently across different self-supervised learning methods. Supported by the few-shot setting, operations take minute-level overhead inside the device. Notably, adaptation for SimCLR and Multi-Task Learning was completed in under 40 seconds, consuming less than 5\% of memory, even smaller than the overhead of fine-tuning. As a result, all user-side with \proj{} can be performed within 2 minutes. While CPC's adaptation is more resource-intensive due to its complex network architecture, it can still be completed within three minutes and use less than 10\% of memory. Our findings confirm that the end-users can conduct all necessary operations of \proj{} on the device—within a 2-minute window for SimCLR and Multi-Task Learning, and under five minutes for CPC, which we believe is manageable computational overhead.
% The computational overhead of the pretext replay (noted as Adapt) and the fine-tuning (noted as Tune) were measured independently across different self-supervised learning methods. Results, presented in Table~\ref{tab:overhead}, indicate that our approach facilitates user-side operations on mobile devices in just minutes. Notably, adaptation for SimCLR and Multi-Task Learning was completed in under 40 seconds, consuming less than 5\% of memory, smaller even compared to the fine-tuning phase.
% While CPC's adaptation is more resource-intensive due to its complex network architecture, it still remains within practical limits, completing in under three minutes and using less than 10\% of memory. Our findings confirm that the end-users can conduct all necessary operations of \proj{} on the device—within a 2-minute window for SimCLR and Multi-Task Learning, and under five minutes for CPC—ensuring manageable computational overhead.
% Our findings confirm that end-users can perform all necessary operations of \proj{} on a standard mobile device swiftly—within a 2-minute window for SimCLR and Multi-Task Learning, and under five minutes for CPC—ensuring a user-friendly experience with manageable computational costs.
% Table~\ref{tab:overhead} shows measured overheads. The computational overhead of our pretext replay is noted as Adapt, and the overhead for the fine-tuning towards the downstream task is noted as Tune. Supported by the small few-shot training, all user-side operations take minute-level overhead inside the device. Except for CPC, the adaptation overhead takes less computational overhead compared to the fine-tuning overhead. For SimCLR and Multi-Task Learning, adaptation takes less than 40 seconds, with less than 5\% of memory consumption. This implies that all user-side operations with \proj{} can be performed within 2 minutes if we consider fine-tuning. For CPC, it takes more computational overhead compared to the others. This is due to its relatively complex network structure and the corresponding forward operations within the networks compared to the other methods. But still, it shows an acceptable execution time less than three minutes, with the memory consumption under 10\%. Based on that, we conclude that the end-users can conduct the user-side operations of \adapt{} on the device, by only taking less than five minutes with acceptable computational overhead.
\section{Discussion}
\subsection{Expansion for Varied Self-Supervised Learning Methods}
Our findings underscore the impact of domain shift on different self-supervised learning methods. We observed that the improvement from our domain adaptation varies with the type of self-supervised learning method applied. This suggests a need for a deeper understanding of how domain shifts affect various self-supervised learning approaches. Although our results shed light on the domain shift effects for well-established methods such as SimCLR, CPC, and Multi-Task Learning, the behavior of numerous other self-supervised learning methods~\cite{haresamudram2020masked, xu2021limu, deldari2022cocoa, ouyang2022cosmo, jain2022collossl, eldele2023self} under domain shifts remains unexplored. Addressing this as future work is an essential step in the field.
\subsection{Covering Wider Range of Domains}
Our meta-learning task generation is currently crafted to reflect end-user environmental conditions as closely as possible, by composing tasks by user and device domains. However, the variability of real-world data calls for a more complicated approach to task generation. For instance, pre-trained models can be applied to an application with a different type of modality. Possible future work could involve integrating diverse data modalities and user-specific contexts as domains, thus creating meta-learning tasks that better capture the nature of real-world applications. Moving forward, refining task generation will be a priority to ensure our meta-learning approach remains robust across real-world applications.
\subsection{Application to Continuously Changing Environment}
We designed our framework with a single domain adaptation step once pre-trained models are deployed to end-users. However, data characteristics within a single domain can change over time due to changing environments. This implies that the adapted model might not perform optimally as domain characteristics change continuously. We anticipate the potential for domain adaptation through pretext replay in such scenarios, as our adaptation step does not require user labels. This allows us to continuously adapt the model using the ongoing data stream from the user. Enhancing the efficiency and effectiveness of this approach in such dynamic scenarios is a direction for future work. %Improvements are needed, such as determining when to perform the pretext task to effectively adapt the model in a changing environment.
\section{Conclusion}
We investigate the domain shift challenge in mobile sensing, where models pre-trained via self-supervised learning are fine-tuned to heterogeneous domains. To address the challenge, we propose \proj{}, a framework that enables few-shot domain adaptation for self-supervised models. Inspired by the observation that models pre-trained on homogeneous domains show superior performance, we refine the pre-trained model to better fit the target domain by replaying the pretext task on the target side. To facilitate few-shot adaptation in this step, our self-supervised pre-training is performed via meta-learning. Our evaluations, conducted across four Human Activity Recognition datasets, indicate that \proj{} consistently outperforms established self-supervised learning and domain generalization methods, achieving an average F1-score improvement of 8.8\%p. Moreover, \proj{} proves to be computationally efficient, with the adaptation process being able to be completed on a COTS smartphone in under a few minutes. The findings validate \proj{} as a practical framework that boosts the performance of pre-trained models for end-users while ensuring minimal computational burden.
\begin{thebibliography}{00}
\bibitem{wang2021ear}Z. Wang, S. Tan, L. Zhang, Y. Ren, Z. Wang, and J. Yang, “An ear canal deformation based continuous user authentication using earables,” in {\em Proceedings of the 27th Annual International Conference on Mobile Computing and Networking}, 2021, pp. 819–821.
\bibitem{liu2022recent}S. Liu, W. Shao, T. Li, W. Xu, and L. Song, “Recent advances in biometrics-based user authentication for wearable devices: A contemporary survey,” {\em Digital Signal Processing}, vol. 125, p. 103120, 2022.
\bibitem{hou2019signspeaker}J. Hou et al., “Signspeaker: A real-time, high-precision smartwatch-based sign language translator,” in {\em The 25th Annual International Conference On Mobile Computing And Networking}, 2019, pp. 1–15.
\bibitem{park2021enabling}H. Park, Y. Lee, and J. Ko, “Enabling real-time sign language translation on mobile platforms with on-board depth cameras,” {\em Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, vol. 5, no. 2, pp. 1–30, 2021.
\bibitem{zhang2019pdmove}H. Zhang et al., “Pdmove: Towards passive medication adherence monitoring of parkinson’s disease using smartphone-based gait assessment,” {\em Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies}, vol. 3, no. 3, pp. 1–23, 2019.
\bibitem{song2020spirosonic}X. Song et al., “SpiroSonic: monitoring human lung function via acoustic sensing on commodity smartphones,” in {\em Proceedings Of The 26th Annual International Conference On Mobile Computing And Networking}, 2020, pp. 1–14.
\bibitem{haresamudram2022assessing}H. Haresamudram, I. Essa, and T. Plötz, “Assessing the state of self-supervised human activity recognition using wearables,” {\em Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, vol. 6, no. 3, pp. 1–47, 2022.
\bibitem{haresamudram2021contrastive}H. Haresamudram, I. Essa, and T. Plötz, “Contrastive predictive coding for human activity recognition,” {\em Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, vol. 5, no. 2, pp. 1–26, 2021.
\bibitem{haresamudram2023investigating}H. Haresamudram, I. Essa, and T. Plötz, “Investigating enhancements to contrastive predictive coding for human activity recognition,” in {\em 2023 IEEE International Conference on Pervasive Computing and Communications (PerCom)}, 2023, pp. 232–241.
\bibitem{tang2020exploring}C. I. Tang, I. Perez-Pozuelo, D. Spathis, and C. Mascolo, “Exploring contrastive learning in human activity recognition for healthcare,” {\em arXiv preprint arXiv:2011.11542}, 2020.
\bibitem{saeed2019multi}A. Saeed, T. Ozcelebi, and J. Lukkien, “Multi-task self-supervised learning for human activity detection,” {\em Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, vol. 3, no. 2, pp. 1–30, 2019.
\bibitem{ustev2013user}Y. E. Ustev, O. Durmaz Incel, and C. Ersoy, “User, device and orientation independent human activity recognition on mobile phones: Challenges and a proposal,” in {\em Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication}, 2013, pp. 1427–1436.
\bibitem{stisen2015smart}A. Stisen et al., “Smart devices are different: Assessing and mitigating mobile sensing heterogeneities for activity recognition,” in {\em Proceedings of the 13th ACM conference on embedded networked sensor systems}, 2015, pp. 127–140.
\bibitem{qian2021latent}H. Qian, S. J. Pan, and C. Miao, “Latent independent excitation for generalizable sensor-based cross-person activity recognition,” in {\em Proceedings of the AAAI Conference on Artificial Intelligence}, 2021, vol. 35, no. 13, pp. 11921–11929.
\bibitem{qin2022domain}X. Qin, J. Wang, Y. Chen, W. Lu, and X. Jiang, “Domain generalization for activity recognition via adaptive feature fusion,” {\em ACM Transactions on Intelligent Systems and Technology}, vol. 14, no. 1, pp. 1–21, 2022.
\bibitem{lu2022semantic}W. Lu, J. Wang, Y. Chen, S. J. Pan, C. Hu, and X. Qin, “Semantic-discriminative mixup for generalizable sensor-based cross-domain activity recognition,” {\em Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, vol. 6, no. 2, pp. 1–19, 2022.
\bibitem{chang2020systematic}Y. Chang, A. Mathur, A. Isopoussu, J. Song, and F. Kawsar, “A systematic study of unsupervised domain adaptation for robust human-activity recognition,” {\em Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, vol. 4, no. 1, pp. 1–30, 2020.
\bibitem{gong2019metasense}T. Gong, Y. Kim, J. Shin, and S.-J. Lee, “Metasense: few-shot adaptation to untrained conditions in deep mobile sensing,” in {\em Proceedings of the 17th Conference on Embedded Networked Sensor Systems}, 2019, pp. 110–123.
\bibitem{zhou2020xhar}Z. Zhou et al., “Xhar: Deep domain adaptation for human activity recognition with smart devices,” in {\em 2020 17th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON)}, 2020, pp. 1–9.
\bibitem{wang2018stratified}J. Wang, Y. Chen, L. Hu, X. Peng, and S. Y. Philip, “Stratified transfer learning for cross-domain activity recognition,” in {\em 2018 IEEE international conference on pervasive computing and communications (PerCom)}, 2018, pp. 1–10.
\bibitem{reiss2012introducing}A. Reiss and D. Stricker, “Introducing a new benchmarked dataset for activity monitoring,” in {\em 2012 16th international symposium on wearable computers}, 2012, pp. 108–109.
\bibitem{altun2010comparative}K. Altun, B. Barshan, and O. Tunçel, “Comparative study on classifying human activities with miniature inertial and magnetic sensors,” {\em Pattern Recognition}, vol. 43, no. 10, pp. 3605–3620, 2010.
\bibitem{zhang2022towards}X. Zhang, L. Zhou, R. Xu, P. Cui, Z. Shen, and H. Liu, “Towards unsupervised domain generalization,” in {\em Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 2022, pp. 4910–4920.
\bibitem{lee2023self}D. B. Lee, S. Lee, J. Ko, K. Kawaguchi, J. Lee, and S. J. Hwang, “Self-Supervised Set Representation Learning for Unsupervised Meta-Learning,” {\em International Conference On Learning Representations}, 2023.
\bibitem{jaiswal2020survey}A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon, “A survey on contrastive self-supervised learning,” {\em Technologies}, vol. 9, no. 1, p. 2, 2020.
\bibitem{he2020momentum}K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in {\em Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, 2020, pp. 9729–9738.
\bibitem{chen2020simple}T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in {\em International conference on machine learning}, 2020, pp. 1597–1607.
\bibitem{wang2022sensor}J. Wang, T. Zhu, J. Gan, L. L. Chen, H. Ning, and Y. Wan, “Sensor data augmentation by resampling in contrastive learning for human activity recognition,” {\em IEEE Sensors Journal}, vol. 22, no. 23, pp. 22994–23008, 2022.
\bibitem{eldele2023self}E. Eldele et al., “Self-supervised contrastive representation learning for semi-supervised time-series classification,” {\em IEEE Transactions on Pattern Analysis and Machine Intelligence}, 2023.
\bibitem{ouyang2022cosmo}X. Ouyang et al., “Cosmo: contrastive fusion learning with small data for multimodal human activity recognition,” in {\em Proceedings of the 28th Annual International Conference on Mobile Computing And Networking}, 2022, pp. 324–337.
\bibitem{deldari2022cocoa}S. Deldari, H. Xue, A. Saeed, D. V. Smith, and F. D. Salim, “Cocoa: Cross modality contrastive learning for sensor data,” {\em Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, vol. 6, no. 3, pp. 1–28, 2022.
\bibitem{jain2022collossl}Y. Jain, C. I. Tang, C. Min, F. Kawsar, and A. Mathur, “Collossl: Collaborative self-supervised learning for human activity recognition,” {\em Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, vol. 6, no. 1, pp. 1–28, 2022.
\bibitem{haresamudram2020masked}H. Haresamudram et al., “Masked reconstruction based self-supervision for human activity recognition,” in {\em Proceedings of the 2020 ACM International Symposium on Wearable Computers}, 2020, pp. 45–49.
\bibitem{xu2021limu}H. Xu, P. Zhou, R. Tan, M. Li, and G. Shen, “Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications,” in {\em Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems}, 2021, pp. 220–233.
\bibitem{wang2022generalizing}J. Wang et al., “Generalizing to unseen domains: A survey on domain generalization,” {\em IEEE Transactions on Knowledge and Data Engineering}, 2022.
\bibitem{muandet2013domain}K. Muandet, D. Balduzzi, and B. Schölkopf, “Domain generalization via invariant feature representation,” in {\em International conference on machine learning}, 2013, pp. 10–18.
% \bibitem{shankar2018generalizing}S. Shankar, V. Piratla, S. Chakrabarti, S. Chaudhuri, P. Jyothi, and S. Sarawagi, “Generalizing across domains via cross-gradient training,” {\em arXiv preprint arXiv:1804.10745}, 2018.
\bibitem{li2019episodic}D. Li, J. Zhang, Y. Yang, C. Liu, Y.-Z. Song, and T. M. Hospedales, “Episodic training for domain generalization,” in {\em Proceedings of the IEEE/CVF International Conference on Computer Vision}, 2019, pp. 1446–1455.
\bibitem{li2018domain}H. Li, S. J. Pan, S. Wang, and A. C. Kot, “Domain generalization with adversarial feature learning,” in {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, 2018, pp. 5400–5409.
\bibitem{wang2020heterogeneous}Y. Wang, H. Li, and A. C. Kot, “Heterogeneous domain generalization via domain mixup,” in {\em ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 2020, pp. 3622–3626.
\bibitem{zhou2021domain}K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, “Domain generalization with mixstyle,” {\em International Conference On Learning Representations}, 2021.
\bibitem{li2018learning}D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales, “Learning to generalize: Meta-learning for domain generalization,” in {\em Proceedings of the AAAI conference on artificial intelligence}, 2018, vol. 32, no. 1.
\bibitem{qiao2020learning}F. Qiao, L. Zhao, and X. Peng, “Learning to learn single domain generalization,” in {\em Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 2020, pp. 12556–12565.
\bibitem{kim2021selfreg}D. Kim, Y. Yoo, S. Park, J. Kim, and J. Lee, “Selfreg: Self-supervised contrastive regularization for domain generalization,” in {\em Proceedings of the IEEE/CVF International Conference on Computer Vision}, 2021, pp. 9619–9628.
\bibitem{yao2022pcl}X. Yao et al., “Pcl: Proxy-based contrastive learning for domain generalization,” in {\em Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 2022, pp. 7097–7107.
\bibitem{wilson2020survey}G. Wilson and D. J. Cook, “A survey of unsupervised deep domain adaptation,” {\em ACM Transactions on Intelligent Systems and Technology (TIST)}, vol. 11, no. 5, pp. 1–46, 2020.
\bibitem{ganin2015unsupervised}Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in {\em International conference on machine learning}, 2015, pp. 1180–1189.
\bibitem{rahman2020correlation}M. M. Rahman, C. Fookes, M. Baktashmotlagh, and S. Sridharan, “Correlation-aware adversarial domain adaptation and generalization,” {\em Pattern Recognition}, vol. 100, p. 107124, 2020.
\bibitem{lu2021cross}W. Lu, Y. Chen, J. Wang, and X. Qin, “Cross-domain activity recognition via substructural optimal transport,” {\em Neurocomputing}, vol. 454, pp. 65–75, 2021.
\bibitem{khan2018scaling}M. A. A. H. Khan, N. Roy, and A. Misra, “Scaling human activity recognition via deep learning-based domain adaptation,” in {\em 2018 IEEE international conference on pervasive computing and communications (PerCom)}, 2018, pp. 1–9.
\bibitem{gong2021dapper}T. Gong et al., “DAPPER: Label-Free Performance Estimation after Personalization for Heterogeneous Mobile Sensing,” {\em Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, vol. 7, no. 2, Jun. 2023, doi: 10.1145/3596256.
\bibitem{khodadadeh2019unsupervised}S. Khodadadeh, L. Boloni, and M. Shah, “Unsupervised meta-learning for few-shot image classification,” {\em Advances in neural information processing systems}, vol. 32, 2019.
\bibitem{khodadadeh2020unsupervised}S. Khodadadeh, S. Zehtabian, S. Vahidian, W. Wang, B. Lin, and L. Bölöni, “Unsupervised meta-learning through latent-space interpolation in generative models,” {\em arXiv preprint arXiv:2006.10236}, 2020.
\bibitem{andrychowicz2016learning}M. Andrychowicz et al., “Learning to learn by gradient descent by gradient descent,” {\em Advances in neural information processing systems}, vol. 29, 2016.
\bibitem{finn2017model}C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in {\em International conference on machine learning}, 2017, pp. 1126–1135.
\bibitem{gong2019metasense2}T. Gong, Y. Kim, R. Choi, J. Shin, and S.-J. Lee, “Adapting to unknown conditions in learning-based mobile sensing,” {\em IEEE Transactions on Mobile Computing}, vol. 21, no. 10, pp. 3470–3485, 2021.
\bibitem{wu2018unsupervised}Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in {\em Proceedings of the IEEE conference on computer vision and pattern recognition}, 2018, pp. 3733–3742.
\bibitem{raghu2019rapid}A. Raghu, M. Raghu, S. Bengio, and O. Vinyals, “Rapid learning or feature reuse? towards understanding the effectiveness of maml,” {\em International Conference On Learning Representations}, 2020.
\bibitem{mcinnes2018umap}L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” {\em arXiv preprint arXiv:1802.03426}, 2018.
\bibitem{termux}“Microsoft Termux,” https://termux.dev/en/
% \bibitem{li2019episodic}Li, D., Zhang, J., Yang, Y., Liu, C., Song, Y. \& Hospedales, T. Episodic training for domain generalization. {\em Proceedings Of The IEEE/CVF International Conference On Computer Vision}. pp. 1446-1455 (2019)
% \bibitem{lane2010survey}Lane, N., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T. \& Campbell, A. A survey of mobile phone sensing. {\em IEEE Communications Magazine}. \textbf{48}, 140-150 (2010)
% \bibitem{weiss2012impact}Weiss, G. \& Lockhart, J. The impact of personalization on smartphone-based activity recognition. {\em Workshops At The Twenty-Sixth AAAI Conference On Artificial Intelligence}. (2012)
% \bibitem{kwon2020imutube}Kwon, H., Tong, C., Haresamudram, H., Gao, Y., Abowd, G., Lane, N. \& Ploetz, T. Imutube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition. {\em Proceedings Of The ACM On Interactive, Mobile, Wearable And Ubiquitous Technologies}. \textbf{4}, 1-29 (2020)
% \bibitem{zhuang2020comprehensive}Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H. \& He, Q. A comprehensive survey on transfer learning. {\em Proceedings Of The IEEE}. \textbf{109}, 43-76 (2020)
% \bibitem{ganin2016domain}Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M. \& Lempitsky, V. Domain-adversarial training of neural networks. {\em The Journal Of Machine Learning Research}. \textbf{17}, 2096-2030 (2016)
% \bibitem{oord2018representation}Oord, A., Li, Y. \& Vinyals, O. Representation learning with contrastive predictive coding. {\em ArXiv Preprint ArXiv:1807.03748}. (2018)
% \bibitem{lee2020meta}Lee, D., Min, D., Lee, S. \& Hwang, S. Meta-gmvae: Mixture of gaussian vae for unsupervised meta-learning. {\em International Conference On Learning Representations}. (2020)
% \bibitem{hsu2018unsupervised}Hsu, K., Levine, S. \& Finn, C. Unsupervised learning via meta-learning. {\em ArXiv Preprint ArXiv:1810.02334}. (2018)
% \bibitem{snell2017prototypical}Snell, J., Swersky, K. \& Zemel, R. Prototypical networks for few-shot learning. {\em Advances In Neural Information Processing Systems}. \textbf{30} (2017)
% \bibitem{wang2018deep}Wang, M. \& Deng, W. Deep visual domain adaptation: A survey. {\em Neurocomputing}. \textbf{312} pp. 135-153 (2018)
% \bibitem{doersch2017multi}Doersch, C. \& Zisserman, A. Multi-task self-supervised visual learning. {\em Proceedings Of The IEEE International Conference On Computer Vision}. pp. 2051-2060 (2017)
% \bibitem{dou2019domain}Dou, Q., Castro, D., Kamnitsas, K. \& Glocker, B. Domain generalization via model-agnostic learning of semantic features. {\em Advances In Neural Information Processing Systems}. \textbf{32} (2019)
% \bibitem{carlucci2019domain}Carlucci, F., D'Innocente, A., Bucci, S., Caputo, B. \& Tommasi, T. Domain generalization by solving jigsaw puzzles. {\em Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition}. pp. 2229-2238 (2019)
% \bibitem{zhang2017mixup}Zhang, H., Cisse, M., Dauphin, Y. \& Lopez-Paz, D. mixup: Beyond empirical risk minimization. {\em ArXiv Preprint ArXiv:1710.09412}. (2017)
% \bibitem{dou2019domain}Dou, Q., Castro, D., Kamnitsas, K. \& Glocker, B. Domain generalization via model-agnostic learning of semantic features. {\em Advances In Neural Information Processing Systems}. \textbf{32} (2019)
% \bibitem{balaji2018metareg}Balaji, Y., Sankaranarayanan, S. \& Chellappa, R. Metareg: Towards domain generalization using meta-regularization. {\em Advances In Neural Information Processing Systems}. \textbf{31} (2018)
% \bibitem{xu2021fourier}Xu, Q., Zhang, R., Zhang, Y., Wang, Y. \& Tian, Q. A fourier-based framework for domain generalization. {\em Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition}. pp. 14383-14392 (2021)
% \bibitem{singapuram2022swan}Singapuram, S., Lai, F., Hu, C. \& Chowdhury, M. Swan: A neural engine for efficient dnn training on smartphone socs. {\em ArXiv Preprint ArXiv:2206.04687}. (2022)
% \bibitem{kingma2014adam}Kingma, D. \& Ba, J. Adam: A method for stochastic optimization. {\em ArXiv Preprint ArXiv:1412.6980}. (2014)
% \bibitem{paszke2019pytorch}Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L. \& Others Pytorch: An imperative style, high-performance deep learning library. {\em Advances In Neural Information Processing Systems}. \textbf{32} (2019)
% \bibitem{nichol2018first}Nichol, A., Achiam, J. \& Schulman, J. On first-order meta-learning algorithms. {\em ArXiv Preprint ArXiv:1803.02999}. (2018)
% \bibitem{wu2022safegait}Wu, Y., Hassan, M. \& Hu, W. Safegait: Safeguarding gait-based key generation against vision-based side channel attack using generative adversarial network. {\em Proceedings Of The ACM On Interactive, Mobile, Wearable And Ubiquitous Technologies}. \textbf{6}, 1-27 (2022)
% \bibitem{motiian2017unified}Motiian, S., Piccirilli, M., Adjeroh, D. \& Doretto, G. Unified deep supervised domain adaptation and generalization. {\em Proceedings Of The IEEE International Conference On Computer Vision}. pp. 5715-5725 (2017)
% \bibitem{soleimani2021cross}Soleimani, E. \& Nazerfard, E. Cross-subject transfer learning in human activity recognition systems using generative adversarial networks. {\em Neurocomputing}. \textbf{426} pp. 26-34 (2021)
% \bibitem{chen2020big}Chen, T., Kornblith, S., Swersky, K., Norouzi, M. \& Hinton, G. Big self-supervised models are strong semi-supervised learners. {\em Advances In Neural Information Processing Systems}. \textbf{33} pp. 22243-22255 (2020)
% \bibitem{li2023neuralgait}Li, H., Chen, H., Xu, C., Li, Z., Zhang, H., Qian, X., Li, D., Huang, M. \& Xu, W. NeuralGait: Assessing Brain Health Using Your Smartphone. {\em Proceedings Of The ACM On Interactive, Mobile, Wearable And Ubiquitous Technologies}. \textbf{6}, 1-28 (2023)
\end{thebibliography}