Select before Act: Spatially Decoupled Action Repetition for Continuous Control
Extracted source text ยท arXiv 2502.06919
\documentclass{article} %
\usepackage{iclr2025_conference,times}
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{amsthm}
\usepackage{dsfont}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{subfigure}
\usepackage{bm}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{footnote}
\usepackage{wrapfig}
\usepackage{soul,color,xcolor}
\usepackage{enumitem}
\newcommand{\alg}[1]{\textbf{\texttt{#1}}}
\newcommand{\nalg}[1]{\texttt{#1}}
\newcommand{\salg}[1]{{\small\textbf{\texttt{#1}}}}
\newcommand{\env}[1]{{{\texttt{#1}}}}
\newcommand{\senv}[1]{{\small{\textbf{\texttt{#1}}}}}
\newcommand{\textenv}[1]{{\small{\texttt{#1}}}}
\newcommand{\vect}[1]{\small{\mathbf{#1}}}
\newcommand*{\revise}[1]{{\textcolor{orange}{#1}}}
\title{Select before Act: Spatially Decoupled Action Repetition for Continuous Control}
\author{Buqing Nie, Yangqing Fu, Yue Gao\thanks{Corresponding author.} \\
MoE Key Lab of Artificial Intelligence, AI Institute \\
Shanghai Jiao Tong University\\
\texttt{\{niebuqing,frank79110,yuegao\}@sjtu.edu.cn} \\
}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy %
\begin{document}
\maketitle
\begin{abstract}
Reinforcement Learning (RL) has achieved remarkable success in various continuous control tasks, such as robot manipulation and locomotion.
Different to mainstream RL which makes decisions at individual steps, recent studies have incorporated action repetition into RL, achieving enhanced action persistence with improved sample efficiency and superior performance.
However, existing methods treat all action dimensions as a whole during repetition, ignoring variations among them.
This constraint leads to inflexibility in decisions, which reduces policy agility with inferior effectiveness.
In this work, we propose a novel repetition framework called \textbf{SDAR}, which implements \textbf{S}patially \textbf{D}ecoupled \textbf{A}ction \textbf{R}epetition through performing closed-loop \emph{act-or-repeat} selection for each action dimension individually.
SDAR achieves more flexible repetition strategies, leading to an improved balance between action \emph{persistence} and \emph{diversity}.
Compared to existing repetition frameworks, SDAR is more sample-efficient with higher policy performance and reduced action fluctuation.
Experiments are conducted on various continuous control scenarios,
demonstrating the effectiveness of spatially decoupled repetition design proposed in this work.
\end{abstract}
\section{Introduction}
{Recently}, Deep Reinforcement Learning (DRL)~\citep{sutton2018reinforcement} has achieved remarkable success in continuous control domains, such as robot manipulation~\citep{gu2017deep}, locomotion~\citep{lee2020learning,zhang2024robust}, and autonomous driving~\citep{kiran2021deep}.
Although conventional Reinforcement Learning (RL) algorithms have demonstrated significant potential across various applications, they typically make decisions at individual time steps, neglecting higher-level decision-making mechanisms and the temporal consistency of action sequences
~\citep{silver2014deterministic,schulman2017proximal,haarnoja2018soft}.
This leads to inefficient exploration during training, challenging credit assignment tasks over long horizons, and poor sample efficiency~\citep{dabney2021temporallyextended,yu2021taac,biedenkapp2021temporl,zhang2022generative,lee2024learning}.
One mainstream solution is Hierarchical RL (HRL) based on temporal abstraction~\citep{sutton1999between,precup2000temporal}, where the high-level policy decomposes the task into simpler subgoals, and low-level policies are designed to solve isolated subtasks~\citep{vezhnevets2017feudal,pateria2021hierarchical}.
However, most HRL methods are task specific, which requires expert knowledge and handcraft design given different applications, such as pre-defining options in the option framework~\citep{bacon2017option,nachum2018data,zhang2021hierarchical}.
A simple yet effective strategy for low-level policies involves executing an action repeatedly for a number of steps, which has been actively explored and proven to be effective in various applications~\citep{lakshminarayanan2017dynamic,sharma2017learning,dabney2021temporallyextended,lee2024learning}.
Action repetition methods improve
\emph{action persistence} through \emph{act-or-repeat}
selections, making exploration trajectories more temporally consistent, leading to higher training efficiency and reduced fluctuations~\citep{chen2021addressing,yu2021taac,biedenkapp2021temporl}.
\begin{wrapfigure}{r}{0.55\textwidth}
\begin{center}
\includegraphics[width=0.54\textwidth]{figures/choices.pdf}
\end{center}
\vspace{-0.2cm}
\caption{
Difference between repetition strategies of previous methods (left) and SDAR (right).
Our method achieves a more flexible strategy through the spatially decoupled repetition design.
}
\vspace{-0.4cm}
\label{fig:action_choices_each_repetition_method}
\end{wrapfigure}
However, as illustrated in Fig.~\ref{fig:action_choices_each_repetition_method},
existing action-repetition approaches treat all action dimensions as a whole during \emph{act-or-repeat} selection, disregarding differences among them.
This constrains all action dimensions to repeat previous actions or make new decisions simultaneously at each step.
This design reduces the effectiveness of action repetition, because different dimensions may be required to select repetition at different time steps, which is quite common in continuous control tasks~\citep{kalyanakrishnan2021analysis}.
For example, robotic systems typically consist of multiple controllers operating at different frequencies due to their system specification, corresponding to different repetition schema for each actuator~\citep{raman2017modular,lee2020reinforcement}.
To address this issue, we propose \textbf{S}patially \textbf{D}ecoupled \textbf{A}ction \textbf{R}epetition (SDAR), a new flexible action repetition framework for continuous control tasks.
SDAR conducts closed-loop repetition for each action dimension individually, which is
composed of two stages: \emph{selection} and \emph{action}.
During \emph{selection}, the agent selects whether the previous actions should be repeated for each dimension.
Afterwards, the agent generates new decisions for dimensions where \emph{act} is chosen in the previous stage.
As shown in Fig.~\ref{fig:action_choices_each_repetition_method}, different to previous methods, \emph{act-or-repeat} selections in SDAR are performed in a decoupled manner, leading to more flexible repetition strategies.
Compared to exsiting repetition frameworks, SDAR achieves an improved balance between action \emph{persistence} and \emph{diversity}, resulting in improved efficiency and superior performance with reduced action fluctuation.
The main contributions of this work are summarized as follows:
\begin{itemize}[leftmargin=2em]
\item We propose a novel action repetition framework called \textbf{SDAR} for continuous control tasks.
SDAR implements \textbf{S}patially \textbf{D}ecoupled \textbf{A}ction \textbf{R}epetition through performing closed-loop \emph{act-or-repeat} selection for each action dimension individually.
\item Compared to previous repetition methods, SDAR offers more flexible repetition strategies, leading to improved balance between action \emph{persistence} and \emph{diversity}.
This results in a higher sample efficiency with superior performance and reduced action fluctuation simultaneously.
\item Experiments are conducted on various continuous control tasks, demonstrating the superior performance of our method on training efficiency and final performance.
This demonstrates the effectiveness of the spatially decoupled framework for action repetition.
\end{itemize}
\section{Related Work}
\subsection{Temporal Abstraction}
Temporal Abstraction is proposed in the semi-MDP formulation~\citep{sutton1999between,precup2000temporal} and commonly implemented based on the \emph{options} framework~\citep{stolle2002learning,bacon2017option,harutyunyan2018learning}.
Each option describes a low-level policy and is defined as $\langle\mathcal{I},\pi,\beta\rangle $, where $\mathcal{I}$ denotes the admissible states for the option initialization, $\pi$ is the policy that the option follows, and $\beta$ determines when the option is terminated.
High-level policies are trained to solve tasks utilizing temporally extended actions provided in the options, rather than one-step actions without action persistence.
Plenty of Hierarchical RL methods are proposed based on temporal abstraction, achieving faster exploration and higher sample efficiency in various sequential decision tasks~\citep{lin2021juewu,yang2021hierarchical}.
Some works are proposed to learn to design options through various techniques, including discovering state connectedness~\citep{chaganty2012learning}, replay buffer analysis~\citep{eysenbach2019search}, and learning termination criteria~\citep{vezhnevets2016strategic,harutyunyan2019termination}.
However, designing options is still a challenging task, which requires prior knowledge and handcraft tunning~\citep{pateria2021hierarchical,yu2021taac,lee2024learning}.
\subsection{Action Repetition}
One simple option strategy is repeating a primitive action for a number of steps, which is similar to the frame-skipping utilized in RL for video games~\citep{bellemare2013arcade,braylan2015frame}.
Recently, action repetition has been actively researched and widely adopted in RL, which can achieve deeper exploration~\citep{dabney2021temporallyextended}, improve sample efficiency by reducing control granularity~\citep{biedenkapp2021temporl}, and reduce action oscillations~\citep{chen2021addressing}.
Existing repetition works are classified as two categories: \emph{open-loop} and \emph{closed-loop} manners.
Open-loop methods force the agent to repeat actions for a predicted number of steps without opportunity of early terminations, such as DAR~\citep{lakshminarayanan2017dynamic}, FiGAR~\citep{sharma2017learning}, TempoRL~\citep{biedenkapp2021temporl}, and UTE~\citep{lee2024learning}.
In contrast, closed-loop methods conduct \emph{act-or-repeat} binary decision to decide if the previous action should be repeated, such as PIC~\citep{chen2021addressing} and TAAC~\citep{yu2021taac}.
Compared to open-loop methods, closed-loop methods examine whether to repeat based on the current state, which is more flexible and improves performance in emergency situations.
In this work, we propose a new closed-loop method to conduct act-or-repeat selections for each actuator individually given current state, which is more flexible and achieves higher action persistence with sufficient action diversity.
\section{Preliminaries}
\subsection{Problem Formulation}
In this work, we focus on model-free RL with continuous action space.
The interaction process in RL is formulated as a Markov Decision Process (MDP), denoted as a tuple $\mathcal{M} = \langle\mathcal{S}, \mathcal{A}, {P}, {R}, \gamma\rangle$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $P(s'|s,a)$ is the transition probability of the environment,
$R:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ denotes the reward function, and $\gamma \in [0,1)$ denotes the discount factor.
The agent takes actions according to its policy, i.e. $a\sim \pi(\cdot | s)$.
Our objective is to find a policy $\pi^*$ that maximizes the expected discounted return, i.e.
$\pi^* = \arg\max_{\pi} \mathbb{E}_{a_t\sim\pi, s_{t+1}\sim P}\left[ \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \right]$.
\subsection{Model-free RL for Continuous Control}
In order to conduct policy evaluation, we define state-action value function $Q(s_t,a_t) = R(s_t, a_t) + \gamma \mathbb{E}_{\pi,P}\left[ \sum_{t'=t}^{\infty} \gamma^{t'}R(s_{t'}, a_{t'}) \right]$ as the discounted return starting from $s_t$, given that $a_t$ is taken and then $\pi$ is followed.
The value function $V(s_t) = \mathbb{E}_{a_t\sim\pi} \left[Q(s_t,a_t)\right]$ denotes the discounted return starting from $s_t$ following $\pi$.
Typically, both $Q$ and $V$ functions are modeled as neural networks, which are optimized using the Mean Square Error (MSE) loss, with target values obtained based on the Bellman equation.
Take Soft Actor Critic (SAC)~\citep{haarnoja2018soft} as an example, the policy is trained through the entropy augmented objective:
\begin{equation}
\pi' \leftarrow \arg\max_{\pi} \mathbb{E}_{s\sim\mathcal{D}, a\sim\pi} \left[ Q\left(s,a\right) - \alpha \log{\pi\left(a|s\right)} \right],
\end{equation}
where $\mathcal{D}$ denotes the replay buffer, $\alpha$ is the temperature parameter, and $Q$ represents accumulated discounted reward augmented by the entropy.
\section{Methodology}
\subsection{Two-Stage Policy: Selection and Action}
\label{sec:two_stage_policy_arch}
As described in Fig.~\ref{fig:whole_arch}, the decision process of SDAR is composed of two stages.
\textbf{(1) Selection:} choose which action dimensions to change previous decisions utilizing the selection policy $\beta$.
\textbf{(2) Action:} generate new actions for action dimensions that choose \emph{act} in the previous stage utilizing action policy $\pi$.
Both two stages are described as follows.
\begin{figure}[tbp]
\begin{center}
\includegraphics[width=1.0\linewidth]{figures/whole_arch.pdf}
\end{center}
\caption{The two-stage decision process of SDAR algorithm.
In the first stage (gray region), the selection policy $\beta$ makes \emph{act-or-repeat} decision for each action dimension, determining whether the previous action $a^-$ (yellow blocks) should be repeated.
In the second stage (blue region), the action policy $\pi$ generates new actions (green blocks) for dimensions that choose \emph{act} in the first stage.
}
\label{fig:whole_arch}
\end{figure}
\subsubsection{Stage 1: Selection Policy}
As illustrated in the gray region of Fig.~\ref{fig:whole_arch}, given current state $s$ and the action $a^-$ at the previous step, we conduct \emph{act-or-repeat} selection for each action dimension individually.
Formally, $\beta(b|s,a^-)\in [0,1]^{|A|}$, where $\beta_i$ denotes the probability of choosing \emph{act} in the $i$-th action dimension.
Afterwards, the repetition schema ${b} \in \{0,1\}^{|A|}$ at the current state is obtained by sampling on the Bernoulli distribution, i.e. ${b_i}\sim \beta_i(\cdot|s,a^-)$, where $b_i=0$ and $b_i=1$ indicate \emph{repetition} and \emph{action} of the $i$-th action dimension correspondingly.
\subsubsection{Stage 2: Action Policy}
As shown in the blue region of Fig.~\ref{fig:whole_arch}, the agent generates new actions for dimensions that choose \emph{act} based on the current state $s$, the previous action $a^-$, and the repetition schema $b$.
Firstly, we reduce redundant input dimensions through introducing the following $\operatorname{Mix}$ operation:
\begin{equation}
\begin{aligned}
& a^{\operatorname{mix}} = \operatorname{Mix}(b, a^-, \xi) = (1-b) \odot a^- + b\odot \xi,
\label{eq:action_stage_pre_mix}
\end{aligned}
\end{equation}
where $\odot$ denotes element-wise hadamard product and $\xi$ denotes the action mask, which is a constant mask vector filled with meaningless values.
In this work, $\xi_i=-2$ for $ 1 \leq i \leq |A|$ is utilized with $A=[-1,1]^{|A|}$.
This operation eliminates the information of previous actions $a^-$ at dimensions where \emph{act} was chosen in the previous stage.
This information is redundant and may disrupt the training process of $\pi$.
Afterwards, we compute $\hat{a}$ utilizing action policy $\pi$:
\begin{equation}
\begin{aligned}
& \mu_{\operatorname{mean}}, \, \sigma_{\operatorname{std}} = \operatorname{MLP}(s, a^{\operatorname{mix}}),
\quad
\hat{a} = \mu_{\operatorname{mean}} + \sigma_{\operatorname{std}} \cdot n, \;\; n\sim \mathcal{N}(0,\mathcal{I}),
\label{eq:action_stage_MLP}
\end{aligned}
\end{equation}
where $\mu_{\operatorname{mean}}$ and $\sigma_{\operatorname{std}}$ are the mean and standard derivation of the action distribution predicted by action policy $\pi$.
The reparameterization trick is utilized to ensure that this process remains differentiable~\citep{haarnoja2018soft}.
Finally, we obtain the output action $a$ through the $\operatorname{Mix}$ operation formulated as follows:
\begin{equation}
\begin{aligned}
& a = \operatorname{Mix}(b,a^-,\hat{a}) = (1-b) \odot a^- + b\odot \hat{a}.
\label{eq:action_stage_post_mix}
\end{aligned}
\end{equation}
This procedure guarantees that the final decision $a$ replicates the same actions as the previous action $a^-$ in dimensions where \emph{repetition} is selected, i.e. $(a - a^-) \odot (1-b) = 0$.
The detailed proof is straightforward and can be found in Appendix~\ref{sec:app_proof_same_previous_action}.
Above all, the decision of the whole two-stage policy $\pi^{\operatorname{all}}$ can be described as follows:
\begin{equation}
\pi^{\operatorname{all}}(a|s,a^-) = \sum_{b} \beta(b|s,a^-) \int_{\hat{a}} \pi(\hat{a}|s,a^-,b) \cdot \delta\left( \operatorname{Mix}\left(b,a^-,\hat{a}\right) - a \right) \mathrm{d} \hat{a}.
\label{eq:pi_all_formula}
\end{equation}
\subsection{Policy Evaluation}
\label{sec:policy_evaluation}
In this work, we employ two identical $Q$ functions to evaluate the performance of $\pi^{\operatorname{all}}$.
Each $Q$ is trained utilizing Mean Square Error (MSE) loss with targets obtained based on the Bellman operator $\mathcal{T}$ formulated as follows:
\begin{equation}
\min \mathbb{E}_{(s,a)\sim\mathcal{D}} \left[ Q\left(s,a\right) - \mathcal{T}Q\left(s,a\right) \right]^2 ,
\; \text{with} \;
\mathcal{T} Q(s,a) = R(s,a) + \gamma\mathbb{E}_{P, \beta, \pi}\left[ Q\left(s', a'\right)\right],
\label{eq:optimize_Q}
\end{equation}
where $\mathcal{D}$ is the replay buffer.
$\mathcal{T}$ in this work is similar to that in standard off-policy RL algorithms~\citep{silver2014deterministic,lillicrap2015continuous}, which is demonstrated to converge to the optimal $Q$ after sufficient iterations.
In addition, we formulate two $Q$ functions based on neural networks, and train using clipped double-Q learning to alleviate the overestimation problem\citep{fujimoto2018addressing}.
More details are given in Algorithm~\ref{alg:sdar_algo}.
\subsection{Policy Improvement}
We propose to optimize the policy $\beta$ and $\pi$ through maximizing the objective $J(\theta^\beta, \theta^\pi)$ formulated as follows, where $\theta^\beta$ and $\theta^\pi$ are learnable parameters of $\beta$ and $\pi$ correspondingly.
\begin{equation}
J(\theta^\beta, \theta^\pi) =
\underbrace{
\mathbb{E}_{(s,a^-)\sim\mathcal{D}} \mathbb{E}_{b\sim\beta, \hat{a}\sim\pi}
}_{\text{make decisions on samples}}
\Bigg[
\underbrace{Q\left(s,a\right)}_{\text{max } Q \text{ values}}
\underbrace{-\alpha_{\beta}\log{\beta\left(b|s,a^{-}\right)} - \alpha_{\pi}\log{\pi\left(\hat{a}|s,a^-,b\right)}}_{\text{entropy-based exploration}}
\Bigg],
\label{eq:optimize_pi}
\end{equation}
where entropy terms $\mathbb{E}\left[ -\log \beta(b|s,a^-) \right]$ and $\mathbb{E}\left[ -\log \pi(a|s,a^-,b) \right]$ are utilized to encourage exploration during training, which is widely utilized in prior works~\citep{haarnoja2018soft,yu2021taac}.
$\alpha_{\beta}$ and $\alpha_{\pi}$ are temperature parameters to adjust exploration strategy, both of which are tuned automatically during training.
As illustrated in Sec.~\ref{sec:two_stage_policy_arch} and Sec.~\ref{sec:policy_evaluation}, the computation process of $\pi$ and $Q$ functions are both differentiable.
Therefore, the action policy $\pi$ can be directly optimized using gradient descent with $\nabla_{\theta^\pi} J$.
In order to optimize selection policy $\beta$, we can transform objective $J$ into the following formulations:
\begin{equation}
\max_{\theta^\beta}
\mathbb{E}_{(s,a^-)\sim \mathcal{D}} \sum_{b\in \mathcal{B}}\beta(b|s,a^-) \mathbb{E}_{\hat{a}\sim\pi} \left[Q\left(s,a\right) - \alpha_{\beta}\log{\beta\left(b|s,a^{-}\right)} - \alpha_{\pi}\log{\pi\left(\hat{a}|s,a^-,b\right)}\right],
\label{eq:optimize_beta_1}
\end{equation}
where $\mathcal{B} = \{0,1\}^{|A|}$ is the action space for selection policy $\beta$.
In contrast to Eq.~(\ref{eq:optimize_pi}), the new objective in Eq.~(\ref{eq:optimize_beta_1}) substitutes the expectation operator $\mathbb{E}_{b\sim\beta}$ with the summation operator $\sum_{b\in\mathcal{B}}$ by listing all possible situations $b\in\mathcal{B}$.
The new objective is differentiable for $\theta^{\beta}$, thus can be used to optimize $\beta$ through gradient descent directly.
\begin{algorithm}[tb]
\caption{Spatially Decoupled Action Repetition (SDAR) Algorithm}
\label{alg:sdar_algo}
\begin{algorithmic}[1]
\STATE {\bfseries Initialize:} Selection policy $\beta$, action policy $\pi$, buffer $\mathcal{D}$, critics $Q_{\{1,2\}}$, $Q_{\operatorname{targ},\{1,2\}}$.
\FOR{each environment step $t$}
\IF{a new episode starts}
\STATE $b = [1,1,...,1]_{|A|}$
\ELSE
\STATE $b\sim \beta(\cdot|s_t,a_{t-1})$
\ENDIF
\STATE Obtain $a_t$ through action policy, i.e. Eq.~(\ref{eq:action_stage_pre_mix})-(\ref{eq:action_stage_post_mix})
\STATE Execute $a_t$ and observe $s_{t+1}$ with reward $r_t$.
\STATE Store transition $(s_t, a_{t-1}, a_t, r_t, s_{t+1},d_t)$ into buffer $\mathcal{D}$
\STATE Sample a batch of data $\left\{ \left( s,a^-,a,r,s',d \right) \right\}$ from $\mathcal{D}$
\STATE Compute $a'$ for state $s'$ using $\beta$ and $\pi$.
\STATE Update $Q_{\{1,2\}}$ through Eq.~(\ref{eq:optimize_Q})
\IF{ $t\mod\operatorname{policy\_delay}$}
\STATE Update $\theta^\pi$ through Eq.~(\ref{eq:optimize_pi})
\STATE Update $\theta^\beta$ through Eq.~(\ref{eq:optimize_beta_1}) or Eq.~(\ref{eq:optimize_beta_2})
\STATE Update $\alpha_\beta$ and $\alpha_\pi$ through Eq.~(\ref{eq:adjust_temperature})
\STATE Update target networks $Q_{\operatorname{targ},\{1,2\}}$ based on soft updates.
\ENDIF
\ENDFOR
\end{algorithmic}
\end{algorithm}
However, Eq.~(\ref{eq:optimize_beta_1}) is quite computationally expensive, because we are required to compute the score for all $b\in\mathcal{B}$, which needs to calculate $Q$ and $\pi$ for $|\mathcal{B}| = 2^{|A|}$ times.
Thus, this method is only practical for tasks with small action spaces, such as LunarLander with $|A|=2$.
For tasks with large action spaces such as Humanoid ($|A|=17$), the selection policy $\beta$ can be optimized by sampling several $b\in\mathcal{B}$, which is formulated as follows:
\begin{equation}
\begin{aligned}
\max_{\theta^\beta} \mathbb{E}&_{\mathcal{D}}
\mathbb{E}_{b\sim\beta_{\operatorname{old}}, \hat{a}\sim\pi} \Big[Q\left(s,a\right) - \alpha_{\beta}\log{\beta_{\operatorname{old}}\left(b|s,a^-\right)}
- \alpha_{\pi}\log{\pi\left(\hat{a}|s,a^-,b\right)}\Big] \cdot
\frac{\beta\left(b|s,a^-\right)}{\beta_{\operatorname{old}}\left(b|s,a^-\right)},
\end{aligned}
\label{eq:optimize_beta_2}
\end{equation}
where $\beta_{\operatorname{old}}$ denotes the old selection policy before optimization.
This objective finds the expectation value $\mathbb{E}_{b\sim\beta}[\cdot]$ in Eq.~(\ref{eq:optimize_pi}) utilizing importance sampling.
Compared to uniform sampling, sampling with $\beta_{\operatorname{old}}$ increases the probability of sampling $b$ with high $Q$ values during training, leading to higher sampling efficiency and training stability.
As illustrated in Eq.~(\ref{eq:optimize_beta_2}), we are only required to compute $Q$ and $\pi$ given several $b$ sampled from $\beta$, thus is more computationally efficient and practical than Eq.~(\ref{eq:optimize_beta_1}) in applications.
In addition, temperature parameters $\alpha_\beta$ and $\alpha_\pi$ are tuned automatically following the objective:
\begin{equation}
\min_{ \substack{ \log{(\alpha_\beta)}, \log{\alpha_\pi} } } {\mathbb{E}}_{\beta, \pi}\left[
-\log{ (\alpha_\beta) }\left(
\log \beta(b|s,a^-) + \mathcal{H}_{\beta}
\right) -
\log{ \alpha_\pi }\left(
\log \pi(\hat{a}|s,a^-,b) + \mathcal{H}_{\pi}
\right)
\right],
\label{eq:adjust_temperature}
\end{equation}
where $\mathcal{H}_{\beta}$ and $\mathcal{H}_{\pi}$ are target entropies.
During training, $\log\alpha_\beta$ and $\log\alpha_\pi$ are updated automatically through solving Eq.~(\ref{eq:adjust_temperature}) with gradient descent, which is similar to prior works~\citep{haarnoja2018soft}.
The detailed settings of the target entropies are described in Appendix~\ref{app:exp_detail_hyper_para}.
The whole training process of SDAR is illustrated in Algorithm~\ref{alg:sdar_algo}, which iteratively collects data, trains critics, and updates policies.
As illustrated in line 10, different to typical DRL algorithms, the previous action $a_{t-1}$ is stored in the buffer $\mathcal{D}$, which is utilized to improve $\theta^\pi$ and $\theta^\beta$ described in line 15-16.
Besides, the agent is forced to choose \emph{act} at the initial step of an episode, i.e. $b=[1,...,1]_{|A|}$, because $a^{-}$ is absent at this state.
\section{Experiments}
\subsection{Experimental Setup}
\paragraph{Tasks:}
In this work, we conduct experiments on multiple continuous control tasks, which are categorized into the following three types of scenarios.
More details are given in Appendix~\ref{app:exp_detail_tasks}.
\begin{enumerate}[leftmargin=2em]
\item[(a)] \textbf{Classic Control}: Several control tasks with small observation and action spaces, including \emph{MountainCarContinuous}, \emph{LunarLanderContinuous}, and \emph{BipedalWalker}.
\item[(b)] \textbf{Locomotion}: Locomotion tasks based on the MuJoCo~\citep{todorov2012mujoco} simulation environment: \emph{Walker2d}, \emph{Hopper}, \emph{HalfCheetah}, \emph{Humanoid}, and \emph{Ant}.
\item[(c)] \textbf{Manipulation} tasks including \emph{Pusher}, \emph{Reacher}, and \emph{FetchReach}.
\end{enumerate}
\paragraph{Baseline Methods.}
In this experiment, we compare the performance of SDAR with the following baseline methods:
(a) \alg{SAC}: vanilla Soft Actor Critic~\citep{haarnoja2018soft} algorithm;
(b) \alg{N-Rep}: repeat policy actions for $n$ times, where $n$ is a \emph{fixed} number~\citep{mnih2015human};
(c) \alg{TempoRL}: repeat policy actions dynamically in an \emph{open-loop} manner, using a \emph{skip-policy} to predict repetition steps~\citep{biedenkapp2021temporl};
(d) \alg{UTE}: Uncertainty-aware Temporal Extension~\citep{lee2024learning} enhances training efficiency through incorporating uncertainty estimation of repeated action values into the \emph{open-loop} repetition framework.
(e) \alg{TAAC}: Temporally Abstract Actor-Critic~\citep{yu2021taac}, which repeats actions dynamically in a \emph{closed-loop} manner.
TAAC determines whether to repeat the previous action at each step utilizing a switch policy.
More details of the baseline methods are illustrated in Appendix~\ref{app:exp_detail_baselines}.
This study trains each method on various tasks using multiple random seeds over a range of 100K to 3M steps, depending on the complexity of the task.
More settings including hyperparameters settings are described in Appendix~\ref{app:exp_detail_hyper_para}.
\begin{figure}[t]
\begin{center}
\hspace{-4.1mm}
\subfigure[\emph{MountainCar}]{
\includegraphics[width=0.249\linewidth]{learning_curve/mountaincarcontinuous_v2.pdf}
\label{fig:learning_curve_mountaincar}
}
\hspace{-4.1mm}
\subfigure[\emph{LunarLander}]{
\includegraphics[width=0.249\linewidth]{learning_curve/lunarlander_v2.pdf}
}
\hspace{-4.1mm}
\subfigure[\emph{BipedalWalker}]{
\includegraphics[width=0.249\linewidth]{learning_curve/bipedalwalker_v2.pdf}
\label{fig:learning_curve_bipedalwalker}
}
\hspace{-4.1mm}
\subfigure[\emph{Humanoid}]{
\includegraphics[width=0.249\linewidth]{learning_curve/humanoid_v2.pdf}
\label{fig:learning_curve_humanoid}
}
\hspace{-4.1mm}
\subfigure[\emph{Ant}]{
\includegraphics[width=0.249\linewidth]{learning_curve/ant_v2.pdf}
\label{fig:learning_curve_ant}
}
\hspace{-4.1mm}
\subfigure[\emph{Walker2d}]{
\includegraphics[width=0.249\linewidth]{learning_curve/walker2d_v2.pdf}
\label{fig:learning_curve_walker2d}
}
\hspace{-4.1mm}
\subfigure[\emph{HalfCheetah}]{
\includegraphics[width=0.249\linewidth]{learning_curve/halfcheetah_v2.pdf}
\label{fig:learning_curve_halfcheetah}
}
\hspace{-4.1mm}
\subfigure[\emph{Pusher}]{
\includegraphics[width=0.249\linewidth]{learning_curve/pusher_v2.pdf}
\label{fig:learning_curve_pusher}
}
\end{center}
\caption{
Learning curves of SDAR (red) in various tasks against baseline methods.
Each method is trained with at least 10 random seeds.
The lines denote the mean episode return, while shaded regions denote the standard error during training.
As shown in the figures, our method generally achieves higher sample efficiency in various tasks compared to previous methods.
More learning curves are given in Appendix.~\ref{app:exp_res_learning_curves}.
}
\label{fig:learning_curves_sdar}
\end{figure}
\subsection{Experiment Results on Sample Efficiency}
\begin{table}[t]
\centering
\caption{
Experiment results on AUC scores (area under the learning curves) in different types of environments, where higher AUC scores indicate higher sample efficiency of corresponding DRL algorithms.
The average scores along with the standard errors are calculated across various tasks within the respective category.
The results are normalized to evaluate performance across different tasks, where $1.0$ denotes the best performance.
The best results are boldfaced.
}
\begin{center}
\begin{tabular}{lcccccc}
\toprule
\multirow{2}{*}{Env. Category} & \multicolumn{6}{c}{Normalized AUC Score} \\
\cmidrule{2-7}
& SAC & N-Rep & TempoRL & UTE & TAAC & SDAR \\
\midrule
Classic Control & 0.60$\pm$0.13 & 0.89$\pm$0.01 & 0.66$\pm$0.02 & 0.55$\pm$0.03 & 0.92$\pm$3E-3 & \textbf{1.0$\pm$0.0} \\
Locomotion & 0.78$\pm$2E-3 & 0.71$\pm$7E-3 & 0.35$\pm$0.02 & 0.43$\pm$0.01 & 0.80$\pm$0.02 & \textbf{1.0$\pm$0.0} \\
Manipulation & 0.91$\pm$4E-5 & 0.90$\pm$8E-4 & 0.77$\pm$0.02 & 0.79$\pm$0.02 & 0.95$\pm$6E-3 & \textbf{1.0$\pm$0.0} \\
\textbf{Average} & 0.76$\pm$0.02 & 0.83$\pm$0.01 & 0.59$\pm$0.05 & 0.59$\pm$0.03 & 0.90$\pm$6E-3 & \textbf{1.0$\pm$0.0} \\
\bottomrule
\end{tabular}
\end{center}
\label{tab:exp_results_auc}
\end{table}
The training curves are illustrated in Fig.~\ref{fig:learning_curves_sdar}, and the AUC scores are shown in Table~\ref{tab:exp_results_auc}.
The AUC scores are normalized into $[0, 1]$ to evaluate the sample efficiency across different tasks, where $0.0$ denotes the performance of random policies without effective training, while $1.0$ denotes the performance of the best method.
See Appendix~\ref{app:exp_res_normalization_process} for the detailed computation process of the normalization.
More experiment results are given in the Appendix~\ref{app:exp_res_learning_curves}.
As illustrated in Fig.~\ref{fig:learning_curves_sdar}, our method \alg{SDAR} (red lines) achieves higher sample efficiency than other baseline methods in various continuous control environments, corresponding to higher AUC scores shown in Table~\ref{tab:exp_results_auc}.
This demonstrate effectiveness of performing \emph{act-or-repeat} selection for each action dimension individually.
Such spatially decoupled repetition framework enhances action persistence while maintaining policy flexibility, leading to efficient exploration and training.
In the following, we analyze the performance of each method individually:
\begin{itemize}[leftmargin=1.1em]
\item Naive action repetition \alg{N-Rep} achieves unstable improvement compared to vanilla \alg{SAC}, but underperforms closed-loop repetition approaches generally.
As described in Table~\ref{tab:exp_results_auc}, \alg{N-Rep} achieves higher AUC scores than \salg{SAC} in classic control tasks, such as \emph{MountainCar} shown in Fig.~\ref{fig:learning_curve_mountaincar}.
Naive repetition improves the persistence of actions during exploration, accelerating the training progress effectively in simple tasks.
Nevertheless, this approach lacks flexibility, which could negatively impact the performance in tasks requiring agile movements, such as \emph{HalfCheetah} (Fig.~\ref{fig:learning_curve_halfcheetah}) in locomotion domains.
\item Open-loop methods (\alg{TempoRL} and \alg{UTE}) show minor improvements over \alg{SAC} in classic control tasks, while performing worse performance in other domains.
These approaches force the agent to repeat actions for a predicted number of steps, eliminating the possibility of early termination.
This inflexibility makes such methods
difficult to be utilized in tasks requiring agile movements, such as \emph{Humanoid} shown in Fig.~\ref{fig:learning_curve_humanoid}.
\item The closed-loop method \alg{TAAC} outperforms \alg{SAC} and open-loop approaches generally in various classes of tasks.
\alg{TAAC} introduces a switch policy to check whether to repeat previous actions at each step, which solves the lack of a termination mechanism in open-loop repetition methods.
This leads to a more flexible repetition, which is suitable for both simple tasks and locomotion tasks requiring agile motions, such as \emph{Ant} shown in Fig.~\ref{fig:learning_curve_ant}.
However, \alg{TAAC} treats all action dimensions as a whole during \emph{act-or-repeat} selection, which downgrades the effectiveness of action repetition.
Take \emph{Humanoid} and \emph{HalfCheetah} shown in Fig.~\ref{fig:learning_curve_humanoid} and Fig.~\ref{fig:learning_curve_halfcheetah} respectively as examples, \alg{TAAC} (brown lines) demonstrates superior sample efficiency during the initial training phase, specifically between $0$ and $1$M steps.
However, in later stages, the agent's learning speed diminishes compared to \alg{SDAR}, which even results in performance decline in the \emph{Humanoid} task depicted in Fig.~\ref{fig:learning_curve_humanoid}.
This is because the \emph{Humanoid} is composed of multiple actuators, where some actuators are required to select \emph{act} for action diversity, while others choosing \emph{repeat} for action persistence at the same time.
However, \alg{TAAC} is constrained to repeat actions of all actuators simultaneously, leading to either inadequate repetition for inefficient exploration, or excessive repetition to damage the policy performance.
\item Our method \alg{SDAR} outperforms other methods in various tasks, including \alg{TAAC} in \emph{locomotion} tasks.
\alg{SDAR} decouples action dimensions during closed-loop \emph{act-or-repeat} selection, which is more flexible and suitable for various types of tasks.
Taking the \emph{Humanoid} task illustrated in Fig.~\ref{fig:learning_curve_humanoid} as an instance, \alg{SDAR} compromises marginal efficiency in the initial stages to achieve greater overall efficiency and stability throughout the entire training process.
Above all, \alg{SDAR} achieves a better balance between action persistence for efficient exploration, and action diversity for high policy performance.
\end{itemize}
\subsection{Policy Performance and Fluctuation}
\begin{table}[t]
\caption{
Experiment results on \textbf{Episode Return}, \textbf{APR (higher is better)}, and \textbf{AFR (lower is better)} in different tasks.
The best results on episode returns are boldfaced.
Our method SDAR outperforms baseline methods on episode returns generally, achieving excellent action persistence with fewer fluctuation than vanilla methods.
}
\begin{center}
\begin{small}
\begin{tabular}{lcccccc}
\toprule
\multirow{3}{*}{Tasks} & \multicolumn{6}{c}{Episode Return (Mean $\pm$ Standard Error)} \\
\cmidrule{3-6}
& \multicolumn{6}{c}{Action Persistence Rate (APR) / Action Fluctuation Rate (AFR)} \\
\cmidrule{2-7}
& SAC & N-Rep & TempoRL & UTE & TAAC & SDAR (Ours) \\
\midrule
\multirow{2}{*}{LunarLander} & 275.9$\pm$6.83 & 280.8$\pm$1.21 & 281.5$\pm$7.22 & \textbf{282.8$\pm$8.74} & 261.4$\pm$18.9 & 282.2$\pm$5.84 \\
\cmidrule{2-7}
& 1.00 / 0.09 & 4.00 / 0.08 & 1.43 / 0.18 & \textbf{1.38 / 0.36} & 3.05 / 0.11 & 11.18 / 0.10 \\
\midrule
\multirow{2}{*}{Walker2d} & 5305$\pm$367 & 4724$\pm$163 & 2866$\pm$897 & 2986$\pm$836 & 5660$\pm$394 & \textbf{6028$\pm$406} \\
\cmidrule{2-7}
& 1.00 / 0.15 & 4.00 / 0.09 & 5.74 / 0.26 & 7.81 / 0.24 & 1.30 / 0.22 & \textbf{2.96 / 0.12} \\
\midrule
\multirow{2}{*}{HalfChee.} & 13122$\pm$2877 & 8378$\pm$1753 & 8065$\pm$1799 & 7917$\pm$293 & 11148$\pm$3921 & \textbf{15131$\pm$1279} \\
\cmidrule{2-7}
& 1.00 / 0.68 & 2.00 / 0.47 & 2.10 / 0.66 & 2.69 / 0.58 & 1.02 / 0.61 & \textbf{1.22 / 0.62} \\
\midrule
\multirow{2}{*}{Humanoid} & 6184$\pm$717 & 5074$\pm$310 & 1022$\pm$397 & 2595$\pm$334 & 7308$\pm$244 & \textbf{7483$\pm$288} \\
\cmidrule{2-7}
& 1.00 / 0.28 & 4.00 / 0.09 & 5.27 / 0.15 & 5.69 / 0.18 & 1.21 / 0.25 & \textbf{1.67 / 0.19} \\
\midrule
\multirow{2}{*}{Pusher} & -22.5$\pm$1.40 & \textbf{-21.2$\pm$1.08} & -48.2$\pm$2.05 & -41.3$\pm$5.56 & -30.5$\pm$1.85 & -21.3$\pm$1.26 \\
\cmidrule{2-7}
& 1.00 / 0.022 & \textbf{4.00 / 0.019} & 1.15 / 0.032 & 1.01 / 0.018 & 1.75 / 0.031 & 1.69 / 0.015 \\
\midrule
\multirow{2}{*}{\textbf{Average}} & 0.89$\pm$0.07 & 0.81$\pm$0.19 & 0.59$\pm$0.33 & 0.64$\pm$0.27 & 0.91$\pm$0.10 & \textbf{1.00$\pm$0.001} \\
\cmidrule{2-7}
& 1.00 / 0.245 & 3.60 / 0.150 & 3.12 / 0.257 & 3.71 / 0.276 & 1.66 / 0.244 & \textbf{3.75 / 0.208} \\
\bottomrule
\end{tabular}
\end{small}
\end{center}
\label{tab:exp_results_return_fluctuation}
\end{table}
In this work, we assess the policies trained through each method on the following metrics:
\textbf{(1) Episode Return}: the average cumulative reward acquired by the policy in an episode, assessing the policy's capability to solve the task successfully.
\textbf{(2) Action Persistence Rate (APR)}
and \textbf{(3) Action Fluctuation Rate (AFR)} formulated as follows:
\begin{equation*}
\operatorname{APR} = \frac{1}{1-p}, \;
p = \mathbb{E}_{\pi} \left[ \frac{1}{T\cdot |A|} \sum_{t=1}^{T}\sum_{i=1}^{|A|} \delta\left( a_t^{(i)} - a_{t-1}^{(i)} \right) \right]
; \;
\operatorname{AFR} = \mathbb{E}_{\pi}\left[ \frac{1}{T} \sum_{t=1}^{T} \left\| a_t - a_{t-1} \right\| \right],
\label{eq:apr_and_afr_metrics}
\end{equation*}
where $T$ denotes the trajectory length, $a_t^{\left(i\right)}$ denotes the action at the $t$-th step in the $i$-th dimension, and $p$ denotes the average repetition probability of $a_t^{(i)}$ at each step.
In this work, APR represents the average interval that two new decisions are made.
APR evaluates \emph{action persistenc} of the policy,
where a larger APR denotes more repetition times during interactions.
AFR evaluates the mean amplitude of fluctuations between successive steps, where a smaller AFR indicates smoother actions with fewer fluctuations.
The results of episode return, APR, and AFR are shown in Table~\ref{tab:exp_results_return_fluctuation}.
As illustrated in the tables, although the baseline method such as \alg{N-Rep} achieves excellent results in both APR and AFR, it suffers a considerable reduction on policy effectiveness, corresponding to lower episode return compared to \alg{SAC}.
This is unacceptable and makes such methods impractical for real-world applications.
In contrast, our method \alg{SDAR} outperforms baseline methods on episode return, while achieving a higher APR and a lower AFR than the vanilla DRL, demonstrating the effectiveness of our method.
In addition, we can also observe that:
\begin{itemize}[leftmargin=1.5em]
\item Naive repetition method \alg{N-Rep} achieves high APR and low AFR, at the expense of performance reduction on episode returns.
This is because the actions in \alg{N-Rep} are repeated for frozen times, where repeated actions may be unreasonable in some states.
This inflexible repetition strategy cannot be agilely adjusted during interaction, leading to performance reduction, especially in \emph{locomotion} tasks such as \emph{HalfCheetah}.
\item Open-loop methods achieve high APR, indicating a lot of action repetitions during interaction, at the expense of more fluctuations and superior episode return performance.
For instance, \alg{UTE} achieves higher APR (3.71) than \alg{SAC} (1.0), with large fluctuation and low episode return.
This suggests that \emph{the increment of repetition times in an inflexible manner may be harmful to policy effectiveness and smoothness}.
In contrast, \alg{TAAC} solves this problem, achieving a moderate AFR (1.66), with a comparable AFR (0.244) and high episode return (0.91).
\item Different to \alg{UTE} and \alg{SDAR}, although \alg{SDAR} also selects a lot of \emph{repeat} during interaction with a high APR (3.69), \alg{SDAR} achieves excellent performance on fluctuation and episode returns simultaneously.
This is because \alg{SDAR} performs repetitions in a more flexible manner.
The decoupling design is consistent with the requirement of continuous control tasks, thus can conduct repetition without harms to episode returns and fluctuation performance.
\end{itemize}
\subsection{Visualization of Act-or-repeat Selection}
\begin{figure}[t]
\begin{center}
\subfigure[LunarLander]{
\label{fig:act_or_repeat_lunarlander}
\includegraphics[width=0.96\linewidth]{act-or-repeat/lunarlander.pdf}
}
\subfigure[Walker2d]{
\label{fig:act_or_repeat_walker}
\includegraphics[width=0.96\linewidth]{act-or-repeat/walker2d.pdf}
}
\end{center}
\caption{
Visualization of \emph{act-or-repeat} selections of \alg{SDAR} and \alg{TAAC} algorithms in \emph{LunarLander} and \emph{Walker2d} tasks.
The $x$-axis denotes timesteps, and the $y$-axis denotes different action dimensions.
The light blue blocks indicate \emph{repetition}, while dark blue blocks represent \emph{act}, i.e. change actions in the corresponding action dimensions.
}
\label{fig:act_or_repeat_visualization}
\end{figure}
\begin{wraptable}{r}{0.38\textwidth}
\vspace{-0.75cm}
\caption{
Experiment results of each action dimensions in \emph{Walker2d}.
}
\begin{center}
\begin{tabular}{l c c }
\toprule
APR Val. & TAAC & SDAR \\
\midrule
Thigh ($a_0$) & 1.30 & \textbf{3.70} \\
\midrule
Leg ($a_1$) & 1.30 & \textbf{2.17} \\
\midrule
Foot ($a_2$) & 1.30 & \textbf{3.44} \\
\midrule
\small Episode Ret. & 5660 & \textbf{6028} \\
\midrule
AFR & 0.22 & \textbf{0.12} \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-0.70cm}
\label{tab:res_each_dim_walker2d}
\end{wraptable}
In order to analyze the difference between repetition behaviors of \alg{SDAR} and previous methods, we perform visualization of \emph{act-or-repeat} selections in \emph{LunarLander} and \emph{Walker2d}.
The results are illustrated in Fig.~\ref{fig:act_or_repeat_lunarlander} and Fig.~\ref{fig:act_or_repeat_walker}, where dark blue and light blue blocks denote \emph{act} and \emph{repeat} respectively.
As shown in the figures, different dimensions require different decision frequencies, corresponding to actions being repeated at distinct steps for each dimension.
Take \emph{LunarLander} in Fig.~\ref{fig:act_or_repeat_lunarlander} for example, the lateral boosters ($a_1$) change actions frequently to adjust the rocket pose, while the main engine ($a_0$) adjusts decisions occasionally.
\alg{SDAR} can adjust action persistence for each dimension automatically through spatially decoupled framework, offering more flexible repetition strategies compared to \alg{TAAC}.
As described in Fig.~\ref{fig:act_or_repeat_walker} and Table~\ref{tab:res_each_dim_walker2d}, \alg{SDAR} controls the walker with higher persistence for the leg joints, while lower persistence for thighs and feet.
In contrast, all joints in \alg{TAAC} are required to repeat simultaneously, resulting in same persistence for each joint with lower repetitions behaviors.
\alg{SDAR} achieves high action persistence, while maintaining agility required by the task, leading to higher policy performance with fewer action fluctuations.
As shown in Table~\ref{tab:res_each_dim_walker2d}, \alg{SDAR} obtains higher APR than \alg{TAAC} in \emph{Walker2d}, while achieving higher episode return with lower AFR, demonstrating the effectiveness of our framework in continuous control tasks.
\section{Conclusion}
In this work, we propose a novel action repetition framework for continuous control tasks called SDAR, which implements closed-loop \emph{act-or-repeat} selection for each action dimension individually.
Such a spatially decoupled design improves the flexibility of repetition strategies, leading to improved balance between action \emph{persistence} and \emph{diversity}, while maintaining action agility for continuous control tasks.
Experiments are conducted in various task scenarios, where SDAR achieves higher sample efficiency, superior policy performance, and reduced action fluctuation, demonstrating the effectiveness of our method.
This work provides insights into spatially decoupled framework for action repetition and temporal abstraction.
In this study, the selection policy $\beta$ is designed to decide \emph{act-or-repeat} for each dimension independently, without accounting for inter-dimensional correlations.
For instance, actuators in humanoid robots can be divided into multiple categories,
with each category typically requiring same decisions during \emph{act-or-repeat} selections.
This issue will be further researched in future works.
\subsubsection*{Acknowledgments}
This work was supported by the National Natural Science Foundation of China (Grant No. 62373242 and No.92248303), Shanghai Municipal Science and Technology Major Project (Grant No. 2021SHZDZX0102).
\bibliography{iclr2025_conference}
\bibliographystyle{iclr2025_conference}
\clearpage
\appendix
\section{Proof of the Statement in Section~\ref{sec:two_stage_policy_arch}}
\label{sec:app_proof_same_previous_action}
\paragraph{Statement.}
Given $\forall s\in \mathcal{S}, a^-\in \mathcal{A}, b\in \{0,1\}^{|A|}$, based on the two-stage policy described in Sec.~\ref{sec:two_stage_policy_arch}, the output action $a$ replicates the same actions as $a^-$ in \emph{repetition} dimensions $\{i| b_i = 0, 1\leq i \leq |A| \}$.
\begin{proof}
As described in Sec.~\ref{sec:two_stage_policy_arch}, $a = (1-b) \odot a^- + b\odot \hat{a}$.
Thus, for $\forall b, a^-$, we can obtain that:
\begin{equation}
\begin{aligned}
(a - a^-) \odot (1-b) &= \left( \left(1-b\right) \odot a^- + b\odot \hat{a} - a^- \right) \odot (1-b) \\
&= b\odot \left(\hat{a} - a^- \right) \odot (1-b) \\
&= b \odot (1-b) \odot \left(\hat{a} - a^- \right) \\
&= \bm{0}
\end{aligned}
\end{equation}
For $\forall i \in \{i| b_i = 0 \}$, we have
$[(a - a^-) \odot (1-b)]_i = a^{(i)} - a^{-,(i)} = 0$,
where $a^{\left(i\right)}$ denotes the action in the $i$-th dimension.
Thus, $a$ replicates the same actions as $a^-$ in dimensions choosing \emph{repetition}.
\end{proof}
\section{Experiment Details}
\subsection{Algorithm Settings}
\label{app:exp_detail_hyper_para}
\begin{table}[htbp]
\caption{Hyper-parameter settings for SDAR algorithm.}
\begin{center}
\begin{tabular}{lc|lc}
\toprule
Parameter & Setting & Parameter & Setting\\
\midrule
Learning rate ($\pi$) & $3\times 10^{-4}$ &
Learning rate ($\beta$) & $3\times 10^{-4}$\\
Learning rate ($Q$) & $1\times 10^{-3}$ &
Learning rate ($\alpha$) & $1\times 10^{-3}$\\
Optimizer & Adam &
Discount factor $\gamma$ & 0.99\\
Batch size & 256 &
Policy delay & 2\\
Soft update $\tau$ & 0.005 &
Sample number ($b$) & 10 \\
\bottomrule
\end{tabular}
\end{center}
\label{tab:hyper_para_SDAR}
\end{table}
The hyper-parameter settings ar shown in Table.~\ref{tab:hyper_para_SDAR}.
In addition, we need to tune the target entropies $\mathcal{H}_{\beta}$ and $\mathcal{H}_{\pi}$ to improve the efficiency of the entropy-based exploration described in Eq.~(\ref{eq:adjust_temperature}).
In this work, we utilize $\mathcal{H}_{\pi} = -|A|$, where $|A|$ denotes the size of the action space, corresponding to the recommendation given in \citep{haarnoja2018soft}.
Besides, $\mathcal{H}_{\beta} = \lambda\cdot|A|\log{2}$, where $\lambda$ is a hyper-parameter tuned according to the tasks.
In this work, we set $\lambda\in [0.4,0.6]$, which achieves excellent performance on sample efficiency and policy effectiveness.
As illustrated in Sec.~\ref{sec:two_stage_policy_arch}, SDAR is composed of two policy networks $\beta$ and $\pi$ based on MLP.
In this work, we design two MLPs with the same structures:
$\left(|S|+|A| \to 256 \to 256 \to |A| \right)$ with ReLU activation functions.
\alg{SDAR} policies for tasks with $|A|\leq 3$ are trained through Eq.~(\ref{eq:optimize_beta_1}), such as \emph{Reacher} and \emph{LunarLanderContinuous}.
Policies in other environments such as \emph{HalfCheetah} and \emph{Humanoid} are optimized through Eq.~(\ref{eq:optimize_beta_2}) for lower computation costs.
\subsection{Experiment Tasks}
\label{app:exp_detail_tasks}
The experiment tasks are listed in Table~\ref{tab:exp_envs}, where
the action spaces of all tasks are normalized as $[-1, 1]$ for convenience.
All tasks are constructed based on Gymnasium~\citep{plappert2018multi}.
The \emph{FetchPickandPlace} and \emph{FetchReach} tasks are implemented by Gymnasium-Robotics~\citep{plappert2018multi}, where \emph{observation} and \emph{desired\_goal} are concatenated as the observation in this experiment.
\begin{table}[t]
\centering
\caption{
Descriptions of experiment tasks in this work.
}
\begin{center}
\begin{tabular}{llcc}
\toprule
Caterogy & Task Name & Obervation Space & Action Space \\
\midrule
\multirow{3}{*}{Classic Control}
& MountainCarcontinuous & $\mathbb{R}^{2}$ & $[-1,1]^{1}$ \\
& LunarLanderContinuous & $\mathbb{R}^{8}$ & $[-1,1]^{2}$ \\
& BipedalWalker & $\mathbb{R}^{24}$ & $[-1,1]^{6}$ \\
\midrule
\multirow{5}{*}{Locomotion}
& HalfCheetah & $\mathbb{R}^{17}$ & $[-1,1]^{6}$ \\
& Humanoid & $\mathbb{R}^{376}$ & $[-1,1]^{17}$ \\
& Walker2d & $\mathbb{R}^{17}$ & $[-1,1]^{6}$ \\
& Hopper & $\mathbb{R}^{11}$ & $[-1,1]^{3}$ \\
& Ant & $\mathbb{R}^{27}$ & $[-1,1]^{8}$ \\
\midrule
\multirow{3}{*}{Manipulation}
& FetchReach & $\mathbb{R}^{13}$ & $[-1,1]^{4}$ \\
& Pusher & $\mathbb{R}^{23}$ & $[-1,1]^{7}$ \\
& Reacher & $\mathbb{R}^{11}$ & $[-1,1]^{2}$ \\
\bottomrule
\end{tabular}
\end{center}
\label{tab:exp_envs}
\end{table}
\subsection{Baselines}
\label{app:exp_detail_baselines}
\begin{enumerate}[leftmargin=1.5em]
\item[(1)] \alg{SAC}~\citep{haarnoja2018soft} is a famous model-free RL in continuous control domains, which trains policies efficiently with entropy-based exploration strategies.
In this work, we utilize SAC implementation and hyper-parameter settings proposed in CleanRL~\citep{huang2022cleanrl}\footnote{https://github.com/vwxyzjn/cleanrl}.
\item[(2)] \alg{N-Rep} forces the agent to repeat the actions output by the policy for $n$ times, where $n$ is a hyper-parameter.
In this work, this algorithm is implemented based on SAC, where $n$ is tuned in $[2,5]$ to achieve the balance between persistence of action and diversity.
\item[(3)] \alg{TempoRL}~\citep{biedenkapp2021temporl} repeats policy actions dynamically in an \emph{open-loop} manner, using a \emph{skip-policy} to predict when to make the next decision.
In this work, this method is implemented based on the official repository\footnote{https://github.com/automl/TempoRL}.
\item[(4)] \alg{UTE}~\citep{lee2024learning} is improved based on \alg{TempoRL}, which performs repetition while incorporating the estimated uncertainty of the repeated action values.
In this work, this method is implemented based on the official repository \footnote{https://github.com/oh-lab/UTE-Uncertainty-aware-Temporal-Extension-/}.
\item[(4)] \alg{TAAC}~\citep{yu2021taac} is a closed-loop repetition framework, which determines whether to repeat the previous action at each step utilizing a switch policy.
In this work, this experiment is conducted utilizing the official implemenatation\footnote{https://github.com/HorizonRobotics/alf}.
\end{enumerate}
\subsection{Computation Resources}
\label{app:exp_res_computation_resource}
In this work, we conduct all experiment utilizing NVIDIA RTX 3090 GPU and Pytorch 2.1 with CUDA 12.2.
The training time of our method compared to vanilla SAC is shown in Table~\ref{tab:time_costs}.
As shown in Table~\ref{tab:time_costs}, our method requires less training times than vanilla \alg{SAC} to achieve the same policy performance, because of the higher sample efficiency of \alg{SDAR}, requiring less training steps in this experiment.
In addition, there exist two policy networks $\beta$ and $\pi$ in the SDAR, both of which need to be optimized through gradient descent.
However, $\beta$ and $\pi$ can be optimized in parallel during training, achieving an improved time efficiency, which will be researched in future works.
\begin{table}[htbp]
\caption{The computational cost for the training of SAC and SDAR algorithms.
Both two methods are trained to achieve the same performance as SAC with sufficient steps.
}
\label{tab:time_costs}
\begin{center}
\begin{tabular}{lcc|cc}
\toprule
\multirow{2}{*}{Tasks} & \multicolumn{2}{c}{Steps Cost} & \multicolumn{2}{c}{Time Cost} \\
\cmidrule{2-5}
& SAC & SDAR & SAC & SDAR \\
\midrule
LunarLander & 300K & 150K & 60min & 36min \\
\midrule
BipedalWalker & 600K & 300K & 108min & 63min \\
\midrule
Walker2d & 1.5M & 700K & 282min & 167min \\
\midrule
Humanoid & 3.0M & 1.5M & 595min & 365min \\
\bottomrule
\end{tabular}
\end{center}
\end{table}
\section{Normalization of Performance}
\label{app:exp_res_normalization_process}
Different tasks can have vastly different reward scales, thus cannot be directly averaged.
In order to depict the average episode return and AUC scores across various tasks, we compute the normalized normalized score (n-score) of each method, which is widely utilized in prior works~\cite{Hessel2017RainbowCI,yu2021taac}.
Take normalized episode return as an example,
given an episode return $Z$, its n-score is calculated as $Z_{norm} = \frac{Z-Z_0}{Z_1-Z_0}$, where
$Z_0$ and $Z_1$ denote the episode return of random policies and vanilla DRL policies respectively.
The n-score $Z_{norm}\in [0,1]$, thus can be averaged across different tasks.
\section{Additional Training Curves}
\label{app:exp_res_learning_curves}
Additional training curves of each method in additional tasks are illustrated in Fig.~\ref{fig:learning_curves_addition}.
\begin{figure}[htbp]
\begin{center}
\subfigure[\emph{Reacher}]{
\includegraphics[width=0.249\linewidth]{learning_curve/reacher_v2.pdf}
}
\subfigure[\emph{Hopper}]{
\includegraphics[width=0.249\linewidth]{learning_curve/hopper_v2.pdf}
}
\subfigure[\emph{FetchReach}]{
\includegraphics[width=0.249\linewidth]{learning_curve/fetchreach_v2.pdf}
}
\end{center}
\caption{
Learning curves of SDAR (red) in additional tasks against baseline methods.
}
\label{fig:learning_curves_addition}
\end{figure}
\end{document}
\usepackage{amsmath,amsfonts,bm}
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
\newcommand{\newterm}[1]{{\bf #1}}
\def\figref#1{figure~\ref{#1}}
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
\def\secref#1{section~\ref{#1}}
\def\Secref#1{Section~\ref{#1}}
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
\def\eqref#1{equation~\ref{#1}}
\def\Eqref#1{Equation~\ref{#1}}
\def\plaineqref#1{\ref{#1}}
\def\chapref#1{chapter~\ref{#1}}
\def\Chapref#1{Chapter~\ref{#1}}
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
\def\algref#1{algorithm~\ref{#1}}
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
\def\partref#1{part~\ref{#1}}
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
\newcommand{\pdata}{p_{\rm{data}}}
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} %
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} %
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{43}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Bacon et~al.(2017)Bacon, Harb, and Precup]{bacon2017option}
Pierre-Luc Bacon, Jean Harb, and Doina Precup.
\newblock The option-critic architecture.
\newblock In \emph{Proceedings of the AAAI conference on artificial intelligence}, volume~31, 2017.
\bibitem[Bellemare et~al.(2013)Bellemare, Naddaf, Veness, and Bowling]{bellemare2013arcade}
Marc~G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.
\newblock The arcade learning environment: An evaluation platform for general agents.
\newblock \emph{Journal of Artificial Intelligence Research}, 47:\penalty0 253--279, 2013.
\bibitem[Biedenkapp et~al.(2021)Biedenkapp, Rajan, Hutter, and Lindauer]{biedenkapp2021temporl}
Andr{\'e} Biedenkapp, Raghu Rajan, Frank Hutter, and Marius Lindauer.
\newblock Temporl: Learning when to act.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 914--924. PMLR, 2021.
\bibitem[Braylan et~al.(2015)Braylan, Hollenbeck, Meyerson, and Miikkulainen]{braylan2015frame}
Alex Braylan, Mark Hollenbeck, Elliot Meyerson, and Risto Miikkulainen.
\newblock Frame skip is a powerful parameter for learning to play atari.
\newblock In \emph{Workshops at the twenty-ninth AAAI conference on artificial intelligence}, 2015.
\bibitem[Chaganty et~al.(2012)Chaganty, Gaur, and Ravindran]{chaganty2012learning}
Arun~Tejasvi Chaganty, Prateek Gaur, and Balaraman Ravindran.
\newblock Learning in a small world.
\newblock In \emph{Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1}, pp.\ 391--397, 2012.
\bibitem[Chen et~al.(2021)Chen, Tang, Hao, Liu, and Meng]{chen2021addressing}
Chen Chen, Hongyao Tang, Jianye Hao, Wulong Liu, and Zhaopeng Meng.
\newblock Addressing action oscillations through learning policy inertia.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~35, pp.\ 7020--7027, 2021.
\bibitem[Dabney et~al.(2021)Dabney, Ostrovski, and Barreto]{dabney2021temporallyextended}
Will Dabney, Georg Ostrovski, and Andre Barreto.
\newblock Temporally-extended {\ensuremath{\varepsilon}}-greedy explocxoration.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\newblock URL \url{https://openreview.net/forum?id=ONBPHFZ7zG4}.
\bibitem[Eysenbach et~al.(2019)Eysenbach, Salakhutdinov, and Levine]{eysenbach2019search}
Ben Eysenbach, Russ~R Salakhutdinov, and Sergey Levine.
\newblock Search on the replay buffer: Bridging planning and reinforcement learning.
\newblock \emph{Advances in neural information processing systems}, 32, 2019.
\bibitem[Fujimoto et~al.(2018)Fujimoto, Hoof, and Meger]{fujimoto2018addressing}
Scott Fujimoto, Herke Hoof, and David Meger.
\newblock Addressing function approximation error in actor-critic methods.
\newblock In \emph{International conference on machine learning}, pp.\ 1587--1596. PMLR, 2018.
\bibitem[Gu et~al.(2017)Gu, Holly, Lillicrap, and Levine]{gu2017deep}
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine.
\newblock Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates.
\newblock In \emph{2017 IEEE international conference on robotics and automation (ICRA)}, pp.\ 3389--3396. IEEE, 2017.
\bibitem[Haarnoja et~al.(2018)Haarnoja, Zhou, Abbeel, and Levine]{haarnoja2018soft}
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
\newblock Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.
\newblock In \emph{International conference on machine learning}, pp.\ 1861--1870. PMLR, 2018.
\bibitem[Harutyunyan et~al.(2018)Harutyunyan, Vrancx, Bacon, Precup, and Nowe]{harutyunyan2018learning}
Anna Harutyunyan, Peter Vrancx, Pierre-Luc Bacon, Doina Precup, and Ann Nowe.
\newblock Learning with options that terminate off-policy.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~32, 2018.
\bibitem[Harutyunyan et~al.(2019)Harutyunyan, Dabney, Borsa, Heess, Munos, and Precup]{harutyunyan2019termination}
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina Precup.
\newblock The termination critic.
\newblock In \emph{The 22nd International Conference on Artificial Intelligence and Statistics}, pp.\ 2231--2240. PMLR, 2019.
\bibitem[Hessel et~al.(2017)Hessel, Modayil, Hasselt, Schaul, Ostrovski, Dabney, Horgan, Piot, Azar, and Silver]{Hessel2017RainbowCI}
Matteo Hessel, Joseph Modayil, H.~V. Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad~Gheshlaghi Azar, and David Silver.
\newblock Rainbow: Combining improvements in deep reinforcement learning.
\newblock In \emph{AAAI Conference on Artificial Intelligence}, 2017.
\bibitem[Huang et~al.(2022)Huang, Dossa, Ye, Braga, Chakraborty, Mehta, and Ara{\~A}{\v{s}}jo]{huang2022cleanrl}
Shengyi Huang, Rousslan Fernand~Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and Jo{\~A}{\c{G}}o~GM Ara{\~A}{\v{s}}jo.
\newblock Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.
\newblock \emph{Journal of Machine Learning Research}, 23\penalty0 (274):\penalty0 1--18, 2022.
\bibitem[Kalyanakrishnan et~al.(2021)Kalyanakrishnan, Aravindan, Bagdawat, Bhatt, Goka, Gupta, Krishna, and Piratla]{kalyanakrishnan2021analysis}
Shivaram Kalyanakrishnan, Siddharth Aravindan, Vishwajeet Bagdawat, Varun Bhatt, Harshith Goka, Archit Gupta, Kalpesh Krishna, and Vihari Piratla.
\newblock An analysis of frame-skipping in reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2102.03718}, 2021.
\bibitem[Kiran et~al.(2021)Kiran, Sobh, Talpaert, Mannion, Al~Sallab, Yogamani, and P{\'e}rez]{kiran2021deep}
B~Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad~A Al~Sallab, Senthil Yogamani, and Patrick P{\'e}rez.
\newblock Deep reinforcement learning for autonomous driving: A survey.
\newblock \emph{IEEE Transactions on Intelligent Transportation Systems}, 23\penalty0 (6):\penalty0 4909--4926, 2021.
\bibitem[Lakshminarayanan et~al.(2017)Lakshminarayanan, Sharma, and Ravindran]{lakshminarayanan2017dynamic}
Aravind Lakshminarayanan, Sahil Sharma, and Balaraman Ravindran.
\newblock Dynamic action repetition for deep reinforcement learning.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~31, 2017.
\bibitem[Lee et~al.(2020{\natexlab{a}})Lee, Lee, and Kim]{lee2020reinforcement}
Jongmin Lee, Byung-Jun Lee, and Kee-Eung Kim.
\newblock Reinforcement learning for control with multiple frequencies.
\newblock \emph{Advances in Neural Information Processing Systems}, 33:\penalty0 3254--3264, 2020{\natexlab{a}}.
\bibitem[Lee et~al.(2024)Lee, Park, Tang, and Oh]{lee2024learning}
Joongkyu Lee, Seung~Joon Park, Yunhao Tang, and Min-hwan Oh.
\newblock Learning uncertainty-aware temporally-extended actions.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~38, pp.\ 13391--13399, 2024.
\bibitem[Lee et~al.(2020{\natexlab{b}})Lee, Hwangbo, Wellhausen, Koltun, and Hutter]{lee2020learning}
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter.
\newblock Learning quadrupedal locomotion over challenging terrain.
\newblock \emph{Science robotics}, 5\penalty0 (47):\penalty0 eabc5986, 2020{\natexlab{b}}.
\bibitem[Lillicrap(2015)]{lillicrap2015continuous}
TP~Lillicrap.
\newblock Continuous control with deep reinforcement learning.
\newblock \emph{arXiv preprint arXiv:1509.02971}, 2015.
\bibitem[Lin et~al.(2021)Lin, Li, Shi, Ye, Fu, and Yang]{lin2021juewu}
Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang.
\newblock Juewu-mc: Playing minecraft with sample-efficient hierarchical reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2112.04907}, 2021.
\bibitem[Mnih et~al.(2015)Mnih, Kavukcuoglu, Silver, Rusu, Veness, Bellemare, Graves, Riedmiller, Fidjeland, Ostrovski, et~al.]{mnih2015human}
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei~A Rusu, Joel Veness, Marc~G Bellemare, Alex Graves, Martin Riedmiller, Andreas~K Fidjeland, Georg Ostrovski, et~al.
\newblock Human-level control through deep reinforcement learning.
\newblock \emph{nature}, 518\penalty0 (7540):\penalty0 529--533, 2015.
\bibitem[Nachum et~al.(2018)Nachum, Gu, Lee, and Levine]{nachum2018data}
Ofir Nachum, Shixiang~Shane Gu, Honglak Lee, and Sergey Levine.
\newblock Data-efficient hierarchical reinforcement learning.
\newblock \emph{Advances in neural information processing systems}, 31, 2018.
\bibitem[Pateria et~al.(2021)Pateria, Subagdja, Tan, and Quek]{pateria2021hierarchical}
Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek.
\newblock Hierarchical reinforcement learning: A comprehensive survey.
\newblock \emph{ACM Computing Surveys (CSUR)}, 54\penalty0 (5):\penalty0 1--35, 2021.
\bibitem[Plappert et~al.(2018)Plappert, Andrychowicz, Ray, McGrew, Baker, Powell, Schneider, Tobin, Chociej, Welinder, et~al.]{plappert2018multi}
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et~al.
\newblock Multi-goal reinforcement learning: Challenging robotics environments and request for research.
\newblock \emph{arXiv preprint arXiv:1802.09464}, 2018.
\bibitem[Precup(2000)]{precup2000temporal}
Doina Precup.
\newblock \emph{Temporal abstraction in reinforcement learning}.
\newblock University of Massachusetts Amherst, 2000.
\bibitem[Raman et~al.(2017)Raman, Cvetkovic, and Bashir]{raman2017modular}
Ritu Raman, Caroline Cvetkovic, and Rashid Bashir.
\newblock A modular approach to the design, fabrication, and characterization of muscle-powered biological machines.
\newblock \emph{Nature protocols}, 12\penalty0 (3):\penalty0 519--533, 2017.
\bibitem[Schulman et~al.(2017)Schulman, Wolski, Dhariwal, Radford, and Klimov]{schulman2017proximal}
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
\newblock Proximal policy optimization algorithms.
\newblock \emph{arXiv preprint arXiv:1707.06347}, 2017.
\bibitem[Sharma et~al.(2017)Sharma, Lakshminarayanan, and Ravindran]{sharma2017learning}
Sahil Sharma, Aravind~S. Lakshminarayanan, and Balaraman Ravindran.
\newblock Learning to repeat: Fine grained action repetition for deep reinforcement learning.
\newblock In \emph{International Conference on Learning Representations}, 2017.
\newblock URL \url{https://openreview.net/forum?id=B1GOWV5eg}.
\bibitem[Silver et~al.(2014)Silver, Lever, Heess, Degris, Wierstra, and Riedmiller]{silver2014deterministic}
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.
\newblock Deterministic policy gradient algorithms.
\newblock In \emph{International conference on machine learning}, pp.\ 387--395. Pmlr, 2014.
\bibitem[Stolle \& Precup(2002)Stolle and Precup]{stolle2002learning}
Martin Stolle and Doina Precup.
\newblock Learning options in reinforcement learning.
\newblock In \emph{Abstraction, Reformulation, and Approximation: 5th International Symposium, SARA 2002 Kananaskis, Alberta, Canada August 2--4, 2002 Proceedings 5}, pp.\ 212--223. Springer, 2002.
\bibitem[Sutton \& Barto(2018)Sutton and Barto]{sutton2018reinforcement}
Richard~S Sutton and Andrew~G Barto.
\newblock \emph{Reinforcement learning: An introduction}.
\newblock MIT press, 2018.
\bibitem[Sutton et~al.(1999)Sutton, Precup, and Singh]{sutton1999between}
Richard~S Sutton, Doina Precup, and Satinder Singh.
\newblock Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.
\newblock \emph{Artificial intelligence}, 112\penalty0 (1-2):\penalty0 181--211, 1999.
\bibitem[Todorov et~al.(2012)Todorov, Erez, and Tassa]{todorov2012mujoco}
Emanuel Todorov, Tom Erez, and Yuval Tassa.
\newblock Mujoco: A physics engine for model-based control.
\newblock In \emph{2012 IEEE/RSJ International Conference on Intelligent Robots and Systems}, pp.\ 5026--5033. IEEE, 2012.
\newblock \doi{10.1109/IROS.2012.6386109}.
\bibitem[Vezhnevets et~al.(2016)Vezhnevets, Mnih, Osindero, Graves, Vinyals, Agapiou, et~al.]{vezhnevets2016strategic}
Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et~al.
\newblock Strategic attentive writer for learning macro-actions.
\newblock \emph{Advances in neural information processing systems}, 29, 2016.
\bibitem[Vezhnevets et~al.(2017)Vezhnevets, Osindero, Schaul, Heess, Jaderberg, Silver, and Kavukcuoglu]{vezhnevets2017feudal}
Alexander~Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu.
\newblock Feudal networks for hierarchical reinforcement learning.
\newblock In \emph{International conference on machine learning}, pp.\ 3540--3549. PMLR, 2017.
\bibitem[Yang et~al.(2021)Yang, Ji, Wu, Lai, Wei, Liu, and Setchi]{yang2021hierarchical}
Xintong Yang, Ze~Ji, Jing Wu, Yu-Kun Lai, Changyun Wei, Guoliang Liu, and Rossitza Setchi.
\newblock Hierarchical reinforcement learning with universal policies for multistep robotic manipulation.
\newblock \emph{IEEE Transactions on Neural Networks and Learning Systems}, 33\penalty0 (9):\penalty0 4727--4741, 2021.
\bibitem[Yu et~al.(2021)Yu, Xu, and Zhang]{yu2021taac}
Haonan Yu, Wei Xu, and Haichao Zhang.
\newblock Taac: Temporally abstract actor-critic for continuous control.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 29021--29033, 2021.
\bibitem[Zhang et~al.(2022)Zhang, Xu, and Yu]{zhang2022generative}
Haichao Zhang, Wei Xu, and Haonan Yu.
\newblock Generative planning for temporally coordinated exploration in reinforcement learning.
\newblock In \emph{International Conference on Learning Representations}, 2022.
\newblock URL \url{https://openreview.net/forum?id=YZHES8wIdE}.
\bibitem[Zhang et~al.(2021)Zhang, Yu, and Xu]{zhang2021hierarchical}
Jesse Zhang, Haonan Yu, and Wei Xu.
\newblock Hierarchical reinforcement learning by discovering intrinsic options.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\newblock URL \url{https://openreview.net/forum?id=r-gPPHEjpmw}.
\bibitem[Zhang et~al.(2024)Zhang, Nie, and Gao]{zhang2024robust}
Yang Zhang, Buqing Nie, and Yue Gao.
\newblock Robust locomotion policy with adaptive lipschitz constraint for legged robots.
\newblock \emph{IEEE Robotics and Automation Letters}, 2024.
\end{thebibliography}