General Dynamic Goal Recognition using Goal-Conditioned and Meta Reinforcement Learning
Extracted source text · arXiv 2505.09737
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% LaTeX Template for AAMAS-2026 (based on sample-sigconf.tex)
%%% Prepared by the AAMAS-2026 Publication Chairs based on the version from AAMAS-2025.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Start your document with the \documentclass command.
%%% == IMPORTANT ==
%%% Use the first variant below for the final paper (including author information).
%%% Use the second variant below to anonymize your submission (no author information shown).
%%% For further information on anonymity and double-blind reviewing,
%%% please consult the call for paper information
%%% https://cyprusconferences.org/aamas2026/submission-instructions/
%%%% For anonymized submission, use this
\documentclass[sigconf]{aamas}
%%%% For camera-ready, use this
%\documentclass[sigconf]{aamas}
%%% Load required packages here (note that many are included already).
\usepackage{balance} % for balancing columns on the final page
\usepackage{algorithm}
% Osher: Is that allowed?
% \usepackage{algorithmic}
\usepackage{booktabs}
\usepackage{amsmath}
\usepackage{xcolor}
\usepackage{enumitem}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
% Ours
\usepackage{comment}
\usepackage{amsmath}
\usepackage{algorithmicx}
\usepackage[noend]{algpseudocode}
\usepackage[dvipsnames]{xcolor}
\usepackage{subcaption}
\usepackage{graphicx}
\usepackage{wrapfig}
\usepackage{array}
\DeclareRobustCommand{\reuth}[1]{{\color{orange}{Reuth: #1}}}
\DeclareRobustCommand{\osher}[1]{{\color{green}{Osher: #1}}}
\DeclareRobustCommand{\owen}[1]{{\color{red}{Owen: #1}}}
\DeclareRobustCommand{\matan}[1]{{\color{blue}{matan: #1}}}
\DeclareRobustCommand{\ben}[1]{{\color{purple}{ben: #1}}}
\definecolor{goalAdapt_color}{HTML}{6121ef}
\DeclareRobustCommand{\goalAdapt}[1]{{\color{goalAdapt_color}{#1}}}
\definecolor{domainAdapt_color}{HTML}{f7934b}
\DeclareRobustCommand{\domainAdapt}[1]{{\color{domainAdapt_color}{#1}}}
\definecolor{inference_color}{HTML}{3ec163}
\DeclareRobustCommand{\inference}[1]{{\color{inference_color}{#1}}}
\definecolor{customgray}{rgb}{0.65, 0.65, 0.65} % Adjust the RGB values (0 to 1)
\newcommand{\ColorComment}[1]{\hfill\textcolor{customgray}{\(\triangleright\) #1}}
\newtheorem{example}{Example}
\newtheorem{theorem}{Theorem}
\newtheorem{definition}{Definition}
% These are are recommended to typeset listings but not required. See the subsubsection on listing. Remove this block if you don't have listings in your paper.
\usepackage{newfloat}
\usepackage{listings}
\DeclareCaptionStyle{ruled}{labelfont=normalfont,labelsep=colon,strut=off} % DO NOT CHANGE THIS
\lstset{%
basicstyle={\footnotesize\ttfamily},% footnotesize acceptable for monospace
numbers=left,numberstyle=\footnotesize,xleftmargin=2em,% show line numbers, remove this entire line if you don't want the numbers.
aboveskip=0pt,belowskip=0pt,%
showstringspaces=false,tabsize=2,breaklines=true}
\floatstyle{ruled}
\newfloat{listing}{tb}{lst}{}
\floatname{listing}{Listing}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% AAMAS-2026 copyright block (do not change!)
\setcopyright{ifaamas}
\acmConference[AAMAS '26]{Proc.\@ of the 25th International Conference
on Autonomous Agents and Multiagent Systems (AAMAS 2026)}{May 25 -- 29, 2026}
{Paphos, Cyprus}{C.~Amato, L.~Dennis, V.~Mascardi, J.~Thangarajah (eds.)}
\copyrightyear{2026}
\acmYear{2026}
\acmDOI{}
\acmPrice{}
\acmISBN{}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% == IMPORTANT ==
%%% Use this command to specify your submission number.
%%% In anonymous mode, it will be printed on the first page.
\acmSubmissionID{94}
%%% Use this command to specify the title of your paper.
\title{General Dynamic Goal Recognition using \\ Goal-Conditioned and Meta Reinforcement Learning}
% %%% Provide names, affiliations, and email addresses for all authors.
\author{Osher Elhadad}
\affiliation{
\institution{Department of Computer Science, Bar Ilan University}
\city{Ramat Gan}
\country{Israel}}
\email{oshereli2@gmail.com}
\author{Owen Morrissey}
\affiliation{
\institution{Department of Computer Science, Tufts University}
\city{Medford}
\country{MA, USA}}
\email{morrisseyowen3@gmail.com}
\author{Reuth Mirsky}
\affiliation{
\institution{Department of Computer Science, Tufts University}
\city{Medford}
\country{MA, USA}}
\email{reuth.mirsky@tufts.edu}
% %%% Use this environment to specify a short abstract for your paper.
\begin{abstract}
Understanding an agent's goal through its behavior is a common AI problem called Goal Recognition (GR).
This task becomes particularly challenging in dynamic environments where goals are numerous and ever-changing.
% Traditional GR methods, typically designed for fixed sets of goals within static environments, struggle to adapt to these complexities.
We introduce the \textbf{General Dynamic Goal Recognition (GDGR)} problem, a broader definition of GR aimed at real-time adaptation of GR systems. This paper presents two novel approaches to tackle GDGR: \textbf{(1)} GC-AURA, generalizing to new goals using Model-Free Goal-Conditioned Reinforcement Learning, and \textbf{(2)} Meta-AURA, adapting to novel environments with Meta-Reinforcement Learning.
We evaluate these methods across diverse environments, demonstrating their ability to achieve rapid adaptation and high GR accuracy under dynamic and noisy conditions. This work is a significant step forward in enabling GR in dynamic and unpredictable real-world environments.
\end{abstract}
%%% Use this command to specify a few keywords describing your work.
%%% Keywords should be separated by commas.
\keywords{Goal Recognition, Agent Modeling, Meta-Reinforcement Learning}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Include any author-defined commands here.
\newcommand{\BibTeX}{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em\TeX}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
%%% The following commands remove the headers in your paper. For final
%%% papers, these will be inserted during the pagination process.
\pagestyle{fancy}
\fancyhead{}
%%% The next command prints the information defined in the preamble.
\maketitle
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
Goal Recognition (GR) is a subfield of artificial intelligence (AI) that focuses on inferring agents' goals based on observed actions. This task is essential in a variety of fields, particularly for applications in human-robot interaction \citep{massardi2020parc,trick2019multimodal,scassellati2002theory} and multi-agent systems \citep{rabkina2019analogical,kaminka2001new,sukthankar2011activity,bansal2019beyond}, where predicting the goals and actions of other agents enables a system to respond effectively and appropriately.
\begin{figure}[t]
\begin{center}
\includegraphics[width=0.50\textwidth]{Figures/gdgr.png}
\caption{The General Dynamic Goal Recognition Problem}
\label{real_examples}
\end{center}
\end{figure}
GR is an online inference problem, yet recent approaches rely on training models to learn patterns associated with a pre-defined set of candidate goals in a single-environment GR problem. While effective in limited settings, such approaches struggle to adapt to new sets of goals within the same or entirely novel environments, requiring them to restart their process from scratch
%While GR is an online inference problem, recent approaches leverage training to reach the inference phase with learned distributions of how likely it is for an agent to pursue each goal in a potential set of goals. Still, most existing GR solutions are primarily aimed at solving a single GR task with a specific set of goals within a single environment. When presented with a new set of goals
(e.g., reapplying planners or Reinforcement Learning (RL) for each new goal). This restart adds significant overhead time, rendering these approaches impractical in real-time scenarios where the goal space is continuous or there is a need for rapid adaptation to dynamically changing GR problems with diverse goals and environments. For example, in assistive technologies for the elderly or those with motor disabilities \citep{haigh2002open,zhang2017intent,massardi2020parc}, robotic assistants must adapt to changing goals across domains: inferring needed baking ingredients from gestures and context in one task, and identifying a preferred book using similar cues and prior knowledge in another.
% As illustrated in Figure \ref{real_examples}, on the left, the robot must infer which ingredients the elderly individual needs based on previous steps and current reaching or pointing gestures and other signals. On the right, the robot must determine the elderly individual's preferred book using a similar process involving past reading preferences and contextual observations.
These scenarios highlight the need for an adaptive GR system, one that can generalize knowledge across tasks and transfer it to new contexts. In real-time applications, such a system cannot afford to relearn from scratch each time.
% Similarly, in autonomous vehicle systems \cite{brewitt2023verifiable}, vehicles must continuously adapt and infer the objectives of surrounding traffic agents, such as pedestrians, other vehicles, and drivers' behaviors.
This paper takes a first step towards addressing these limitations by \textbf{(1)} providing a definition for a new problem: \textbf{General Dynamic Goal Recognition (GDGR)}, a generalization of GR to account for changing goals and domains \textbf{(2)} outlining a solution paradigm for GDGR, named \textbf{Adaptive Universal Recognition Algorithm (AURA)}, and \textbf{(3)} present two implementations of AURA using Model-Free Goal-Conditioned RL (\textbf{GC-AURA}) and Meta-RL (\textbf{Meta-AURA}) to enable real-time GR across multiple dynamically changing tasks within multiple domains.
Figure~\ref{real_examples} highlights the three incremental phases that distinguish GDGR from classical GR: \textcolor{orange}{(a) \textit{Domain Adaptation}}, where the recognizer adapts to a newly introduced domain (e.g., an empty MiniGrid); \textcolor{RoyalPurple}{(b) \textit{Goal Adaptation}}, where new candidate goals are introduced within the domain; and \textcolor{green}{(c) \textit{Recognition Inference}}, where the system is required to infer the most likely goal based on partial observations. This decomposition emphasizes the dynamic nature of GDGR: unlike traditional GR, which assumes a fixed domain and goal set, GDGR requires continual adaptation across domains, goals, and observation sequences.
%
We present results in three environments, varying in their properties: Minigrid \citep{chevalier2024minigrid}, Point Maze \citep{gymnasium_robotics2023github}, and Panda-Gym \citep{gallouedec2021panda}. The results demonstrate that AURA significantly reduces adaptation times for new goals and environmental changes compared to existing methods, showcasing the potential of AURA to advance GR in dynamic real-world scenarios. %By addressing the run-time limitations of traditional GR systems, this research lays the groundwork for more adaptable, efficient, automated, and accurate GR applications in a wide range of changing environments.
\section{Theoretical Background}
\begin{figure*}[h]
\begin{center}
\includegraphics[width=0.85\textwidth]{Figures/Univeral_Dynamic_Goal_Recognition_Nurips.jpeg}
\caption{Adaptive Universal Recognition Algorithm (AURA).
}
\label{fig:DGRLL}
\end{center}
\end{figure*}
\noindent\textbf{Markov Decision Process (MDP)}
is a mathematical framework to model an agent’s decision-making under uncertainty. It is defined as a tuple \( \mathcal{M} = (S, A, \tau, R, \gamma) \), where: \( S \) is the set of states; \( A \) is the set of actions; \( \tau: S \times A \times S \to [0, 1] \) is the state transition probability function; \( R: S \times A \to \mathbb{R} \) is the reward function; \( \gamma \in [0,1] \) is the discount factor.
Reinforcement Learning (RL) methods aim to find a policy \( \pi: S \to A \) that maximizes the expected sum of discounted rewards \citep{kaelbling1996reinforcement, sutton2018reinforcement}.
% \paragraph{Trust Region Policy Optimization (TRPO)}
% TRPO \citep{schulman2015trust} improves policy gradient RL methods by limiting large updates, which ensures stability in optimization. The objective is to maximize the expected return \( \eta(\pi) \), subject to a constraint on the Kullback-Leibler (KL) divergence between the old and new policies:
% \begin{equation}
% \max_\theta \mathbb{E}_{s \sim \rho_{\pi_{\text{old}}}, a \sim \pi_{\text{old}}} \left[ \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} A_{\theta_{\text{old}}}(s,a) \right], \quad \text{s.t.} \quad D_{\text{KL}}(\pi_{\text{old}} \,||\, \pi_\theta) \leq \delta
% \end{equation}
% Here, \( \theta \) represents the policy parameters, \( \rho_\pi \) is the state visitation frequency, and \( A_\theta(s,a) \) is the advantage function, which measures how much better action \( a \) in state \( s \) is compared to the expected action. The constraint \( D_{\text{KL}} \) limits policy changes, ensuring gradual improvement.
\noindent\textbf{Goal-conditioned Reinforcement Learning (GCRL)} \citep{liu2022goal} adapts RL to environments where agents must achieve specific goals. A GA-MDP extends the standard MDP with an additional tuple $\langle \mathcal{G}, p_g, \phi \rangle$, where:
\( \mathcal{G} \) is the goal space; \( p_g \) is the distribution of goals; \( \phi: S \to \mathcal{G} \) maps states to goals.
The reward function depends on the state and the goal, \( R: S \times \mathcal{G} \times A \to \mathbb{R} \), and policy \( \pi \) aims to maximize the expected return for goal-directed actions:
\begin{equation}
J(\pi) = \mathbb{E}_{a_t \sim \pi(\cdot|s_t, g), g \sim p_g, s_{t+1} \sim \tau(\cdot|s_t, a_t)} \left[\sum_{t} \gamma^t R(s_t, a_t, g)\right]
\end{equation}
% \paragraph{Transfer Learning in RL}
% Transfer Learning in RL enables an agent to apply knowledge from previous tasks to new ones. This is useful in scenarios like GDGR, where knowledge from source domains \( M_s \) is transferred to the target domain \( M_t \). The optimal policy \( \pi^* \) for \( M_t \) is:
% \begin{equation}
% \pi^* = \arg\max_{\pi} \mathbb{E}_{s \sim \mu^t_0, a \sim \pi} [Q^\pi_{M_t}(s, a)],
% \end{equation}
% where \( \mu_0^t \) is the state distribution in \( M_t \), and \( Q^\pi_{M_t}(s, a) \) is the utility of taking action \( a \) in state \( s \) under policy \( \pi \). \reuth{Matan also mentioned this in the previous version -- not sure that the equation here is useful if it is not used later on. Also, unclear where $M_S$ comes into play in the equation, so I'm in favor of removing it completely.}
% \paragraph{Meta-Reinforcement Learning (Meta-RL)}
% Meta-RL aims to train agents that can quickly adapt to new tasks by leveraging experience from a distribution of related tasks. The agent learns a shared initialization of policy parameters that can be efficiently adapted to new tasks using a small amount of additional data.
% A common approach is Model-Agnostic Meta-Learning (MAML) \citep{finn2017model}, which performs a two-step learning process. First, in the inner loop, the agent adapts its policy to a specific task using a gradient update. Then, in the outer loop, the initial policy parameters are optimized such that this adaptation step leads to good performance across many tasks. The objective is to find initial parameters \( \theta \) that minimize the average loss after one or more gradient steps:
% \begin{equation}
% J(\theta) = \mathbb{E}_{\text{task}} \left[ \text{Loss}_{\text{task}}(\theta - \alpha \nabla_\theta \text{Loss}_{\text{task}}(\theta)) \right]
% \end{equation}
% Here, \( \alpha \) is the learning rate for the inner adaptation step, and \( \text{Loss}_{\text{task}} \) measures task-specific performance. This setup enables the agent to adapt quickly when faced with new, unseen tasks. \matan{still unclear to me \textbf{where} meta-RL is useful. What could the source and target domains share and what couldn't they, by this part of the paper I feel like I should have more clues as to which sort of differences between $M_S$ and $M_T$ GDGR allows. Also, I'd probably change $M_T$ to $m_t$ since it's a single domain and not a set of domains.}
\noindent\textbf{Meta-Reinforcement Learning (Meta-RL)}
is a problem formulation, which aims to train agents that can quickly adapt to new tasks by leveraging experience from a distribution of related tasks. Formally, consider a distribution over tasks \( p(\mathcal{T}) \), where each task \( \mathcal{T}_i \) is modeled as an MDP. The objective is to learn a policy \( \pi_\theta(a \mid s) \), parameterized by \( \theta \), that can be efficiently adapted to any task \( \mathcal{T}_i \) with minimal data. Concretely, we assume all tasks share the same state and action spaces and policy parameterization, but may differ in their transition dynamics \(\tau\) or reward functions \(R\). Meta-RL methods learn a common initialization of policy parameters \(\theta\) that -- after a small number of additional gradient steps -- yields strong performance on any new task drawn from this distribution.
%
A canonical algorithm is Model-Agnostic Meta-Learning (MAML) \citep{finn2017model}.% , which alternates between two nested loops over tasks:
%\begin{itemize}
%\item \emph{Inner loop:} For each task \( \mathcal{T}_i \), adapt the initial parameters \( \theta \) to task-specific parameters \( \theta'_i \) using one or more gradient descent steps on the task's loss:
%\[
%\theta'_i = \theta - \alpha \nabla_\theta \mathcal{L}_{\mathcal{T}_i}(\theta),
%\]
%where \( \alpha \) is the inner-loop learning rate, and \( \mathcal{L}_{\mathcal{T}_i}(\theta) \) denotes the loss function for task \( \mathcal{T}_i \), typically defined as the negative expected return:
%\[
%\mathcal{L}_{\mathcal{T}_i}(\theta) = -\mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^H \gamma^t R_i(s_t, a_t) \right],
%\]
%with \( \tau = (s_0, a_0, \ldots, s_H, a_H) \) representing a trajectory sampled under policy \( \pi_\theta \).
%\item \emph{Outer loop:} Update the initial parameters \( \theta \) to minimize the loss across tasks after adaptation:
%\[
%\min_{\theta} \mathbb{E}_{\mathcal{T}_i \sim p(\mathcal{T})} \left[ \mathcal{L}_{\mathcal{T}_i}(\theta'_i) \right].
%\]
%\end{itemize}
%This bi-level optimization trains the model to be sensitive to task-specific updates, enabling efficient adaptation to new tasks with limited data.
\noindent\textbf{Goal Recognition (GR)} is the task of inferring the likely goal of an observed % \ben{Did you mean observed?}
agent according to a series of observations \citep{sukthankar2011activity,meneguzzi2021survey}. %If accepted, add mirsky, keren and Geib's survey
\begin{definition}
A GR problem is defined as a tuple \( T = (D, G, O) \), where \( D \) is the domain theory, \( G \) is the set of potential goals, and \( O \) is a sequence of observations. The objective of GR is to identify a goal \( g \in G \) that best explains the observation sequence \( O \).
\end{definition}
%
Different approaches to GR primarily differ in how they formulate the domain theory \( D \), including planning domains \citep{ramirez2009plan}, grammars \citep{geib2018learning}, process graphs \citep{ko2023plan}, or GA-MDPs \citep{ramirez2011goal, amado2022goal}.
While GR focuses on identifying the specific goal an agent is pursuing based on a partial execution of its behavior, Inverse Reinforcement Learning (IRL) \citep{arora2021survey} seeks to recover the underlying reward function that explains the agent’s behavior across multiple trajectories. GR is typically formulated as an online, one-shot inference task, often under partial observability, where the aim is to determine the most likely goal from a limited observation sequence. In contrast, IRL assumes access to multiple trajectories and aims to generalize the agent’s preferences. % through a learned reward function.
% \paragraph{Goal Recognition as Reinforcement Learning}
% \citet{amado2022goal} define a GR as RL problem as: \matan{Unless you're using this definition to GR explicitly in this paper, in the light of the last statement before this definition, why is it worth mentioning/defining? there are other formulations for GR that aren't used here and aren't mentioned in this section.}
% \( T = (D, G, O) \) where: \( G \) - the set of potential goals; \( O \) - a sequence of observations; \( D \) - the domain theory, which is represented as a set of policies, one per goal \(D= \{\pi_g | g \in G\} \).
% divided into two types:
% \begin{itemize}
% \item Utility-based domain theory \( D_Q(G) \), represented as \( (S, A, Q) \) where \( Q \) is a set of Q-functions \( \{Q_g | g \in G\} \).
% \item Policy-based domain theory \( D_\pi(G) \), represented as \( (S, A, \pi) \) where \( \pi \) is a set of policies \( \{\pi_g | g \in G\} \).
% \end{itemize}
\section{The General Dynamic Goal Recognition (GDGR) Problem}
%\reuth{Start with a motivating example for changing goals. Explain why existing work can't handle it and then show GDGR.}
% We introduce a generalization to the GR problem as a continuous transfer learning task \matan{I'd remove the 'continuous', seems unrelated to the term, but rather to the environments supported.}, referred to as General Dynamic Goal Recognition (GDGR).
We introduce a transfer‐learning formulation of the GR problem - General Dynamic Goal Recognition (GDGR) - which handles sequences of GR problems with changing goals and domains. Intuitively, GDGR consists of a series of GR problems, where each problem may involve a new observation sequence and potentially introduce previously unseen goals or changes in the underlying domain. The colors throughout the paper match the colors in Figure \ref{fig:DGRLL} to provide visual assistance in decomposing GDGR into single recognition tasks.
%
\begin{definition}
\label{def:gdgr} \textbf{General Dynamic Goal Recognition (GDGR)} is
a tuple representing the prior distribution of domain theories \( p_D \) and a sequence of GR problems:
\[
\langle p_D, (T_1 = \langle \textcolor{orange}{D}_1, \textcolor{RoyalPurple}{DG}_1, \textcolor{green}{O}_1 \rangle, T_2, \ldots, T_n) \rangle,
\]
where each input is provided at an increasing time step, starting with \( p_D \) at time step \( t = 0 \), and each GR problem corresponds to a distinct time step \( t \in \{1, \ldots, n\} \).
\end{definition}
For a given series of time steps \( t \in \{1, \ldots, n\} \), each time step consists of three stages of inputs, which can be given incrementally: The first input is the domain theory \( \textcolor{orange}{D}_t \); The second input is the set of dynamic goals used for recognition \( \textcolor{RoyalPurple}{DG}_t\); and the last input is an observation sequence
$
\textcolor{green}{O}_t.
$
% that can have gaps between states and actions. %\osher{Does every GR is an online problem?} Note: as GR is an online setting where state and action pairs appear incrementally. % and conform with the order in which they appear within the sequence.
% \end{itemize}
\begin{algorithm*}[h]
\caption{Adaptive Universal Recognition Algorithm (AURA)}
\begin{algorithmic}[1]
\Require{$p_D$ - prior distribution of domain theories}
\State Init Memory $M$
\State $M \gets \text{InitMemoryPhase}(p_D, M)$ \ColorComment{Initialized $M$ after adaptation to $p_D$}
\For{all $T_i$ in $\text{GetGoalRecognitionProblem}()$}
\State Get Domain theory $\textcolor{orange}{D}_i$ from $T_i$
\State $M_{\textcolor{orange}{D}_i} \gets \text{DomainAdaptationPhase}(\textcolor{orange}{D}_i, M)$ \ColorComment{Domain Memory $M_{\textcolor{orange}{D}_i}$ after domain adaptation}
\State Get the set of new dynamic goals $\textcolor{RoyalPurple}{DG}_i$ from $T_i$
\State $\{M_g\}_{g\in \textcolor{RoyalPurple}{DG}_i} \gets \text{GoalsAdaptationPhase}(\textcolor{RoyalPurple}{\textcolor{RoyalPurple}{DG}_i}, M_{\textcolor{orange}{D}_i})$ \ColorComment{Goals Memory $\{M_g\}_{g\in \textcolor{RoyalPurple}{DG}_i}$ after goal adaptation}
\State Get the Observation sequence $\textcolor{green}{O}_i$ from $T_i$ \ColorComment{In the online GR setting, each state and action tuple is provided in a different time step and has its own recognition inference phase}
\State $g^* \gets \text{RecognitionInferencePhase}(\{M_g\}_{g\in \textcolor{RoyalPurple}{DG}_i}, \textcolor{green}{O_i})$ \ColorComment{$g^* \gets \text{arg max}_{g\in \textcolor{RoyalPurple}{DG}_i}[\text{DISTANCE}(\textcolor{green}{O}_i, M_g)]$, where DISTANCE calculates the similarity between the observation sequence and the Goals Memory}
\State Save and return $g^*$
\State $M \gets \text{UpdateMemoryPhase}(M, M_{\textcolor{orange}{D}_i}, \{M_g\}_{g\in \textcolor{RoyalPurple}{DG}_i}, T_i, g^*)$ \ColorComment{Updated $M$ using the previous $M$, and the current GR problem, adaptations and inference}
\EndFor
\end{algorithmic}
\label{alg:GDGR}
\end{algorithm*}
This work focuses on GA-MDPs as the domain theory, specifically \( \textcolor{orange}{D} = (S, A, \tau) \), where \( S \) is the set of states, \( A \) is the set of actions, and \( \tau \) is the state transition probability.
%
%The definitions of \( G \) and \( O \) also vary across GR approaches.
In GA-MDP-based formulations of GR, the observation sequence \( \textcolor{green}{O} \) is typically defined as $\textcolor{green}{O} = (\langle s_{1}, a_{1} \rangle, \langle s_{2}, a_{2} \rangle, \ldots)
$, a sequence of state-action pairs, which may be consecutive or non-consecutive, where each pair denotes a state-action transition observed over time. The goal set \( \textcolor{RoyalPurple}{DG} \subseteq \mathcal{G} \) is assumed to be a subset of the broader goal space \( \mathcal{G} \) defined by the GA-MDP.
The solution of a GDGR problem $\langle p_D, (T_1, \ldots, T_n) \rangle$ is a sequence of goals \( (g^*_1, \ldots, g^*_n) \) for each of the single GR problems \( t \in \{ 1, \ldots, n \} \), such that
$g_t^* = \arg\max_{g \in \textcolor{RoyalPurple}{DG}_t} P(g|\textcolor{green}{O}_t, M_{t-1})$. \( g^*_t \) is the recognized goal within the set of dynamic goals \( \textcolor{RoyalPurple}{DG}_t \) based on the given observations \( \textcolor{green}{O}_t \) and the \textit{Memory} \( M_{t-1} \) carried over from the previous time step. Memory refers to the structured system or set of mechanisms responsible for storing, maintaining, and retrieving information that helps to transfer knowledge between iterations.
% Going back to the assistive robot example from the introduction, we can apply the concept of GDGR to its behavior. In the kitchen domain \( \textit{\textcolor{orange}{D}}_1 \), the goals \( \textcolor{RoyalPurple}{DG}_{1,1} \) may include making tea or preparing a sandwich, with observation sequences \( \textcolor{green}{O}_{1,1,1} \) reflecting gestures like reaching for a kettle or pointing at bread. Later, in the reading room domain \( \textit{\textcolor{orange}{D}}_2 \), the set of possible goals shifts to selecting books, influenced by prior reading history. For example, \( \textcolor{RoyalPurple}{DG}_{2,1} \) includes a certain book, and once it’s read, the set changes to \( \textcolor{RoyalPurple}{DG}_{2,2} \), which no longer includes that book. The observations \( \textcolor{green}{O}_{2,2,1} \) consist of gaze or pointing gestures toward bookshelves. Each \( \langle \textit{\textcolor{orange}{D}}, \textcolor{RoyalPurple}{DG}, \textcolor{green}{O} \rangle \) represents a distinct GR problem, requiring the system to adapt without retraining. \matan{without \textbf{full} training? because there is some transfer RL, right? Another general term unused here is GR \textbf{instance}: in the ODGR work, the usage of the term \textbf{GR problem} means the unique <D, DG, O>, while GR instance includes all <G, DG, O>'s that share the same <D, DG>'s.}
% For a given series of time steps \( t \in \{1, \ldots, n\} \), each time step consists of three stages of inputs, which can be given incrementally. For each time step $t$:
% \begin{itemize}
% \item The first input is the domain theory \( \textcolor{orange}{D}_t \), formally defined as \( \textcolor{orange}{D}_t = (S_t, A_t, G_t) \), where: \( S_t \) is the state space; \( A_t \) is the action space; and \( G_t \) is the goal space.
% \item The second input is the set of dynamic goals used for recognition \( \textcolor{RoyalPurple}{DG}_t \subseteq G_t\).
% \item The last input is an observation sequence
% \[
% \textcolor{green}{O}_t = (\langle s_{t_1}, a_{t_1} \rangle, \langle s_{t_2}, a_{t_2} \rangle, \ldots).
% \]
% % that can have gaps between states and actions. %\osher{Does every GR is an online problem?} Note: as GR is an online setting where state and action pairs appear incrementally. % and conform with the order in which they appear within the sequence.
% \end{itemize}
% Note that at time step \( t=1 \), the given information state \( IS_0 \) is the initialized information state, acquired based on the prior distribution of domain theories \( p_D \) created at time step \( t=0 \) (before receiving the first GR task).
%\subsection{GDGR Algorithm}
% \subsection{Empirical Setup}
\begin{table*}[H]
\centering
\renewcommand{\arraystretch}{1.2} % slightly increase row height for readability
\begin{tabular}{m{2cm} m{2cm} m{9cm}}
\toprule
\textbf{Figure} & \textbf{Domain} & \textbf{Details} \\
\midrule
\includegraphics[width=1.5cm]{Figures/grid3.PNG}
& \raggedright\arraybackslash \textbf{MiniGrid}
& \raggedright\arraybackslash \textit{Environments}: Empty-MiniGrid-9x9 environment. \newline \textit{Motivation}: Discrete navigation; comparison with GR baselines. \newline
\textit{States}: Image + direction of agent (Discrete). \newline \textit{Actions}: Turn, move forward, stay in place (Discrete). \newline
\textit{Reward}: $1 - 0.9 * (step_{count} / max_{steps})$ for success or 0 for failure (Sparse). \\
\midrule
\includegraphics[width=1.5cm]{Figures/panda_gym_goals.png}
& \raggedright\arraybackslash \textbf{Panda-Gym}
& \raggedright\arraybackslash \textit{Environments}: PandaReach environment. \newline \textit{Motivation}: Realistic 3D robotic control scenario. \newline
\textit{States}: Robotic arm positions, velocities (Continuous). \newline \textit{Actions}: Joint torques for robotic arm (Continuous). \newline
\textit{Reward}: Gradual reward based on distance to goal (Dense). \\
\midrule
\includegraphics[width=1.5cm]{Figures/point_maze_4_rooms_1x9_9x1_9x9_1x6_4x9.png}
& \raggedright\arraybackslash \textbf{PointMaze}
& \raggedright\arraybackslash \textit{Environments}: 4-Rooms-11x11 environment. \newline \textit{Motivation}: Continuous navigation; path complexity. \newline
\textit{States}: Agent positions, velocities (Continuous). \newline \textit{Actions}: Force applied to agent (Continuous). \newline
\textit{Reward}: -1 if not reaching goal; 0 when near goal ($<$ 0.5m) (Sparse). \\
\bottomrule
\end{tabular}
\caption{Comparison of domains, visualizations, and their specific characteristics. Each domain is chosen to highlight different capabilities of the AURA framework.} % across discrete and continuous spaces.}
\label{tab:domains}
\end{table*}
\section{Adaptive Universal Recognition Algorithm (AURA)}
To solve GDGR problems, we introduce the \textit{Adaptive Universal Recognition Algorithm (AURA)} -- a generic algorithm that leverages the fact that domain theories, goal specifications, and online observations may arrive at different times. AURA operates in three main phases. First, it initializes a global \textit{Memory}, which can later reduce the computational cost of adapting to new problems. Then, for each \textit{GR problem}, AURA performs task-specific \textit{domain and goal adaptation} to tailor its knowledge to the new setting. Finally, once an observation sequence is received, AURA infers the most likely goal and updates the memory. %, enabling continual learning through knowledge transfer.
%
Figure~\ref{fig:DGRLL} and Algorithm~\ref{alg:GDGR} outline AURA's core components:
\noindent \textbf{1. Initial Memory Phase} (Algorithm~\ref{alg:GDGR}, line~2 -- \texttt{InitMemoryPhase}; Figure~\ref{fig:DGRLL}, component (a))
Given the prior distribution \( P_D \), this phase initializes the memory before any GR problem is received. Its objective is to pre-process reusable domain or goal-related knowledge to reduce adaptation and inference costs in later phases.
\noindent \textbf{2. GR Problem Processing} (Algorithm~\ref{alg:GDGR}, lines~3–10; Figure~\ref{fig:DGRLL}, component (b))
For each GR problem, AURA processes the domain theory, dynamic goals, and observation sequence, which may arrive at different times.
\begin{itemize}[leftmargin=*, itemsep=1pt, topsep=0pt]
\item \textbf{Domain Adaptation Phase:}
Upon receiving the domain theory \textcolor{orange}{D}, AURA applies the \texttt{DomainAdaptationPhase} to retrieve or refine domain-specific knowledge from memory. This adaptation supports subsequent goal reasoning and recognition.
\item\textbf{Goal Adaptation Phase:}
When the dynamic goals \textcolor{RoyalPurple}{DG} become available, AURA invokes the \texttt{GoalsAdaptationPhase} to tailor internal representations to the specific goal set. This enables efficient and accurate recognition later.
\item \textbf{Recognition Inference Phase:}
Once the sequence \textcolor{green}{O} is available, AURA performs inference using the \texttt{RecognitionInferencePhase} to identify the most likely goal. % based on the adapted knowledge.
\end{itemize}
\noindent \textit{Note:} The components of a GR problem -- domain, goals, and observations -- are not assumed to arrive simultaneously. AURA is designed to conceptually take advantage of any time gaps between their arrivals. This enables early adaptation steps, such as beginning domain or goal learning before observations are received.
\noindent \textbf{3. Memory Update Phase} (Algorithm~\ref{alg:GDGR}, line~11 -- \texttt{UpdateMemoryPhase}) \\
After solving each GR problem, AURA updates its memory with the new results and learned patterns. This formulation supports continual learning and improves future performance by transferring accumulated knowledge across problems.
%\subsection{GDGR Abstraction Levels}
\noindent\textbf{AURA Abstraction Levels} %\reuth{generalizability?}
Algorithm \ref{alg:GDGR} is a generic algorithm. It enables generalization at various levels of the problem, from basic GR problems with new observations, through changing goals, or new domain dynamics. AURA can be implemented in various ways to address each type of these \textit{abstractions} that can be classified into three levels:
\textbf{(1)} GR problems with a fixed set of goals within a single domain (\textcolor{green}{O} is changing between time steps, but \textcolor{orange}{D} and \textcolor{RoyalPurple}{DG} are fixed). \textbf{(2)} GR problems with many possible sets of goals within a single domain (\textcolor{green}{O} and \textcolor{RoyalPurple}{DG} are changing between time steps, while \textcolor{orange}{D} is fixed). \textbf{(3)} GR problems with many possible goals across multiple domains (changing \textcolor{green}{O}, \textcolor{RoyalPurple}{DG} and \textcolor{orange}{D}).
% \begin{enumerate}[leftmargin=*, itemsep=2pt, topsep=2pt]
% \item GR tasks with a fixed set of goals within a single domain (\textcolor{green}{O} is changing between time steps, but \textcolor{orange}{D} and \textcolor{RoyalPurple}{DG} are fixed). %For a single GR task, as in the GR as RL \cite{amado2022goal} setting, there will only be a Goal Adaptation Phase, where a policy is learned for each goal, and the most likely goal is identified based on the similarity between the policy and the observation sequence.
% \item GR tasks with many possible sets of goals within a single domain (\textcolor{green}{O} and \textcolor{RoyalPurple}{DG} are changing between time steps, while \textcolor{orange}{D} is fixed). %For a sequence of goals within the same domain theory, as in ODGR \cite{shamir2024odgr} (a heuristic implementation for discrete simple cases), the Information State Initialization Phase involves learning a specific domain representation (a set of policies for heuristically chosen base goals). Then, during the Goal Adaptation Phase, heuristically crafted policies are derived using the domain representation for each new dynamic goal, followed by the same inference mechanism as in GR as RL.
% \item GR tasks with many possible goals across multiple domains (changing \textcolor{green}{O}, \textcolor{RoyalPurple}{DG} and \textcolor{orange}{D}).
% \matan{I think you mean: "there can be GDGR problems where all GR problems share the same X but might differ in Y"}
% \matan{what are the time-steps? does every tuple differ in it's time-step, or could the inner components of every tuple arrive at different time-steps? I think it's important to clarify it}
% \end{enumerate}
Next, we introduce two RL-based implementations of AURA, each targeting a different level of abstraction within the framework. The first, GC-AURA, builds on GCRL to enable fast adaptation to a new set of goals, addressing AURA’s second level of abstraction. The second, Meta-AURA, is based on Meta-RL and targets AURA’s third level of abstraction by enabling adaptation to new domains that share the same state and action spaces and can vary in their transition and reward functions.
% \subsection{GC-AURA}
% This approach focuses on cases where all GR problems in the GDGR problem are within the same domain theory \( \textcolor{orange}{D} \). In this case, the Memory $M$ is the GCRL policy trained for all goals in the goal space of the domain $\textcolor{orange}{D}$.
% %, and the \texttt{cache} contains the buffered experiences accumulated up to this GR task at time step \( t \) (inclusive).
% %
% The challenge in such cases lies in adapting the goal-conditioned policy to new goals \( \textcolor{RoyalPurple}{DG} \) with minimal additional learning. This iterative process improves policy adaptation mechanisms with each recognition task and calls for a lifelong learning system that adapts to new goals.
% During the \texttt{DomainAdaptationPhase}, we train a GCRL policy for the specific domain. This policy is utilized for all problems. When a new set of goals is received, the \texttt{GoalsAdaptationPhase} determines the following for each goal: (1) Fine-tune the GCRL policy with additional training iterations on the specific goal (few-shot transfer learning). (2) Use the GCRL policy without modification (zero-shot transfer learning). (3) Retrieve a goal-specific policy from past experiences (when the exact same goal was already considered in one of the previous $\textcolor{RoyalPurple}{DG}$). In Section \ref{sec:gc_aura_exp}, we utilized TRPO to train a GCRL policy over a continuous goal space with zero-shot transfer learning.
% \matan{I don't understand the need for the few-shot transfer learning, and not sure I really like the term selection here (fine-tune and few-shot and transfer learning. what do we transfer here?). Why do we need to fine-tune the policy? what do you mean by "on the specific goal" - which specific goal? every goal in the goals adaptation phase? Is there a suggested trade-off such that the framework would prefer speed over accuracy, without additional training? could it still succeed without the fine-tune? In that case, why fine-tune at all?}
% \ben{Can you add the motivation for "few-shot transfer learning"? Why offline "domain learning"
% is not sufficient?}
\subsection{GC-AURA}
This approach targets GDGR problems in which all GR problems share the same domain theory \( \textcolor{orange}{D} \). In this setting, the memory \( M \) consists of a GCRL policy that can be trained over the entire goal space of \( \textcolor{orange}{D} \) or on a subset of the goal space. This enables AURA to reuse prior domain knowledge for new recognition problems without retraining from scratch.
The primary challenge here is to adapt this policy to newly introduced goals \( \textcolor{RoyalPurple}{DG} \) with minimal additional training. This requires a mechanism for lifelong goal adaptation, where the system continuously improves its ability to generalize across goals within the same domain.
During the \texttt{DomainAdaptationPhase}, a GCRL policy is trained on the domain goal space, or a goal subspace. This shared policy serves as a reusable basis for all subsequent recognition problems in that domain. Once a new set of goals is introduced, the \texttt{GoalsAdaptationPhase} uses transfer learning, for example, can use the following strategies:
\begin{enumerate}[leftmargin=*, itemsep=2pt, topsep=2pt]
\item \textbf{Zero-shot transfer:} Directly use the existing GCRL policy without any additional training.
\item \textbf{Few-shot adaptation:} Fine-tune the policy on a specific goal using a small number of gradient steps or episodes. Useful when the goal is substantially different from those seen in earlier steps.
\item \textbf{Goal recall:} Retrieve a previously stored goal-specific policy if the exact goal has been encountered in a past GR problem.
\end{enumerate}
Zero-shot and recall are generally preferred for their efficiency. However, some goals may require minor updates of the policy to ensure accurate recognition. Few-shot adaptation is motivated by the need to handle goals that lie in underrepresented regions of the goal space or whose reward dynamics significantly differ from previously seen goals. %This provides a trade-off between recognition speed and accuracy, depending on the task constraints.
In Section~\ref{sec:gc_aura_exp}, we demonstrate GC-AURA using TRPO \citep{schulman2015trust} to train a GCRL policy over a continuous goal space, where zero-shot transfer was found to be effective in the Panda-Gym domain.
% \begin{figure}[t]
% \begin{center}
% % Top row: MiniGrid plots
% % \begin{subfigure}[b]{0.2\columnwidth}
% % \centering
% % \includegraphics[width=\textwidth]{Figures/grid2.PNG}
% % \caption{2-Goal MiniGrid. }%: The agent, represented as a red triangle, (begins at (1x4)) navigates the grid to reach one of the two green goal cells: (7x1), or (7x7).}
% % \label{fig:2goal}
% % \end{subfigure}
% % \hfill
% \begin{subfigure}[b]{0.25\columnwidth}
% \centering
% \includegraphics[width=\textwidth]{Figures/grid3.PNG}
% \caption{MiniGrid. }%: The agent, represented as a red triangle, (begins at (1x4)) navigates the grid to reach one of the three green goal cells: (7x1), (7x7), or (4x7).}
% \label{fig:3goal}
% \end{subfigure}
% \hfill
% % Bottom row: Point Maze and Panda-Gym plots
% % \vspace{0.3cm}
% \begin{subfigure}[b]{0.25\columnwidth}
% \centering
% \includegraphics[width=\textwidth]{Figures/point_maze_4_rooms_1x9_9x1_9x9_1x6_4x9.png}
% \caption{Point Maze. }%: This environment features continuous state, goal, and action spaces within a 4-room maze of size 11x11. The agent (green ball) begins at the top-left corner (1x1) and aims to reach one of the three designated goal positions (red balls): (1x9), (9x1), or (9x9).}
% \label{fig:point_maze}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.25\columnwidth}
% \centering
% \includegraphics[width=\textwidth]{Figures/panda_gym_goals.png}
% \caption{Panda-Gym.}%: A robotic arm manipulates objects to reach designated goal positions (green ball). The predefined goal positions in this example are (-0.2, -0.2, 0.1), (0, 0, 0), and (0.2, 0.2, 0.1).}
% \label{fig:pandagym_goals}
% \end{subfigure}
% \caption{Visualization of the evaluation environments.
% %Each environment highlights distinct state, action, and goal space properties.
% }
% \label{fig:env_overview}
% \end{center}
% \end{figure}
% Upon receiving the observation sequence, the \texttt{RecognitionInferencePhase} measures the \texttt{DISTANCE} between each goal-specific policy and the observation sequence. For instance, using the MaxUtils similarity metric from Amado \textit{et al.} \shortcite{amado2022goal},
% \[
% \text{MaxUtil}(Q^{\pi_g}, O) = \sum_{i \in |O|} Q^{\pi_g}(s_i, a_i),
% \]
% where \( Q^{\pi_g}(s, a) \) is the Q-value for the state-action pair, the observation sequence \( \textcolor{green}{O} \) is evaluated to find the goal whose policy accumulated the highest utility under the observed sequence. %most likely goal. This goal is returned as the recognized goal.
% Finally, the \texttt{UpdateInformationStatePhase} caches the task, updated metrics, and policies for future use.
% \subsection{Meta-AURA}
% This approach focuses on cases where the domain theory \( \textcolor{orange}{D}_t \) is expected to change between timesteps \( t \in \{0, \ldots, n\} \).
% We employed Model-Agnostic Meta-Learning (MAML) \cite{finn2017model} to train a meta-policy capable of rapid adaptation to new tasks. MAML optimizes for initial policy parameters such that a few gradient updates on task-specific data lead to significant performance improvements. For the \texttt{DomainAdaptationPhase}, we used Trust Region Policy Optimization (TRPO), a policy gradient method designed to optimize policies while maintaining trust regions to ensure stable updates.
% %
% The meta-policy is trained using a distribution of tasks drawn from the possible set of tasks. For example, in our experiments in the Point Maze environment, we used empty mazes, two-room mazes, five-room mazes, and random-setup mazes. The training process involves alternating between meta-updates and task-specific updates, following the same protocol used in the MAML framework.
\subsection{Meta-AURA}
In RL-based GR approaches, new domains can differ in all MDP characteristics. For our Meta-AURA implementation using MAML-TRPO, the state and action spaces are assumed to be the same, and the transitions and rewards can change between different domains.
This approach focuses on cases where GR problems span across multiple domain theories \(\textcolor{orange}{D}\). The Memory $M$ is represented by a meta-policy trained to generalize over diverse domains with varying transitions and rewards. This meta-policy %is designed to
improves adaptation times to novel goals and domains by leveraging prior experience.
The primary challenge is to enable the meta-policy to efficiently adapt to a new domain \(\textcolor{orange}{D}\) and its associated dynamic goals \(\textcolor{RoyalPurple}{DG}\). This process demands a robust initialization strategy and domain adaptation mechanism to minimize adaptation time while maintaining high recognition accuracy.
During the \texttt{InitMemoryPhase} we train a meta-policy for all domains from the prior distributions $p_D$. The \texttt{GoalsAdaptationPhase} evaluates each goal in the dynamic goal set \(\textcolor{RoyalPurple}{DG}\) and decides between: (1) Few-shot adaptation of the meta-policy to specific goals, (2) Few-shot adaptation of the meta-policy to GCRL policy over all goals in \(\textcolor{RoyalPurple}{DG}\), or (3) Retrieval of previously adapted goal-specific or GC policies for recurrent goals from earlier GR problems.
In Sections \ref{sec:meta_discrete} and \ref{sec:meta_continuous}, we leveraged the MAML-TRPO algorithm to train a meta-policy capable of adapting to a diverse set of domains and goals. This meta-policy was later fine-tuned using TRPO to specific goals and domains.
\section{Experimental Setup}
\label{sec:experiments}
We start by detailing the general setup configurations used across all experiments.
\begin{table*}[t]
\begin{tabular}{m{2cm} m{2cm} m{9cm}}
\toprule
\textbf{Figure} & \textbf{Domain} & \textbf{Details} \\
\midrule
\includegraphics[width=1.5cm]{Figures/grid3.PNG}
& \raggedright\arraybackslash \textbf{MiniGrid}
& \raggedright\arraybackslash \textit{Environments}: Empty-MiniGrid-9x9 environment. \newline \textit{Motivation}: Discrete navigation; comparison with GR baselines. \newline
\textit{States}: Image + direction of agent (Discrete). \newline \textit{Actions}: Turn, move forward, stay in place (Discrete). \newline
\textit{Reward}: $1 - 0.9 * (step_{count} / max_{steps})$ for success or 0 for failure (Sparse). \\
\midrule
\includegraphics[width=1.5cm]{Figures/panda_gym_goals.png}
& \raggedright\arraybackslash \textbf{Panda-Gym}
& \raggedright\arraybackslash \textit{Environments}: PandaReach environment. \newline \textit{Motivation}: Realistic 3D robotic control scenario. \newline
\textit{States}: Robotic arm positions, velocities (Continuous). \newline \textit{Actions}: Joint torques for robotic arm (Continuous). \newline
\textit{Reward}: Gradual reward based on distance to goal (Dense). \\
\midrule
\includegraphics[width=1.5cm]{Figures/point_maze_4_rooms_1x9_9x1_9x9_1x6_4x9.png}
& \raggedright\arraybackslash \textbf{PointMaze}
& \raggedright\arraybackslash \textit{Environments}: 4-Rooms-11x11 environment. \newline \textit{Motivation}: Continuous navigation; path complexity. \newline
\textit{States}: Agent positions, velocities (Continuous). \newline \textit{Actions}: Force applied to agent (Continuous). \newline
\textit{Reward}: -1 if not reaching goal; 0 when near goal ($<$ 0.5m) (Sparse). \\
\bottomrule
\end{tabular}
\caption{Comparison of domains, visualizations, and their specific characteristics. Each domain is chosen to highlight different capabilities of the AURA framework.} % across discrete and continuous spaces.}
\label{tab:domains}
\end{table*}
\subsection{Domains}
\label{app:domains}
AURA supports both discrete and continuous environments, making it suitable for evaluation under a variety of conditions. We evaluate its variants across three benchmark domains from OpenAI Gymnasium \citep{towers2024gymnasium}: MiniGrid, Point-Maze, and Panda-Gym. Table~\ref{tab:domains} summarizes the key differences between these domains.
\paragraph{MiniGrid}
\label{sec:minigrid}
The MiniGrid domain was chosen due to its flexibility in representing environments in multiple formats, which makes it well-suited for modeling complex GR problems. This capability allows us to compare diverse approaches under consistent conditions, thereby offering insights into algorithmic performance across different representations. Our study focuses MiniGrid Empty 9x9 environment (shown in Table \ref{tab:domains}). Specifically, we compared AURA to three baseline algorithms: PDDL for reasoning and GR (R\&G) using a complete environment model, GRAQL with symbolic state representations (coordinates and angle), and DRACO with a visual image-based representation (same as AURA visual image-based representation). Each representation provides the algorithms with varying degrees of information, enabling a comprehensive comparison of their strengths and limitations when tackling the same GR problem. We used Minigrid Empty 9x9 with different goal locations and lava locations as the training environments for MAML training. The adaptation rewards for MAML in this domain are shown in the Appendix, in Figure \ref{fig:minigrid_maml_adaptation_rewards}.
\paragraph{Point-Maze}
\label{sec:pointmaze}
The Point-Maze domain presents a continuous environment where the agent navigates through a maze, starting from a fixed position and aiming to reach one of three predefined goal locations. This domain is particularly valuable for testing algorithms in continuous state, goal, and action spaces, as it introduces challenges such as path complexity, exploration efficiency, and obstacle avoidance. The environment's structure and continuous nature make it a suitable testbed for evaluating Meta-AURA. By assessing performance in this domain, we gain insights into the adaptability and effectiveness of different approaches in navigating and achieving goals within complex spatial constraints. We used Mazes with different sizes: 6x6 - 9x9, and different start goal states, and obstacles locations as the training environments for MAML training. The adaptation rewards for MAML are shown in Figure \ref{fig:point_maze_maml_adaptation_rewards}. %Fine-tuning the MAML policy on 4-Rooms 11x11 environment is reported in Table \ref{tab:domains}.
\paragraph{Panda-Gym}
\label{sec:pandagym}
The Panda-Gym domain features a robotic arm tasked with manipulating objects to reach predefined goal positions within a continuous 3D space. This environment poses unique challenges by combining high-dimensional state and action spaces. The robotic arm begins at a default position and must plan and execute precise movements to achieve one of three target goals. Panda-Gym provides a realistic simulation for evaluating reinforcement learning algorithms, especially in robotics applications, as it tests spatial reasoning, motion control, and goal-oriented behavior. This domain highlights the robustness of algorithms when applied to real-world-inspired scenarios involving intricate control dynamics and the need for precision.
%generated GR problems for each domain by creating diverse configurations with shared initial setups and strategically placed obstacles.
\subsection{Baselines.}
(1) \textit{Plan Recognition as Planning (R\&G):} This classic symbolic GR baseline, proposed by \citet{ramirez2009plan}, leverages planner executions to infer the likelihood of goals.
% \cite{ramirez2011goal}
It relies on an underlying PDDL domain representation, meaning
%, which can either be manually crafted \cite{vered2017online} or derived from labeled data \cite{granada2020object}.
without significant and non-trivial modifications, it is limited to discrete domains, making it unsuitable for continuous setups. Consequently, we use it exclusively for evaluation in MiniGrid (Section \ref{sec:meta_discrete}). % by referring to the results reported in Nageris \textit{et al}. \shortcite{nageris2024goal}.
(2) \textit{GRAQL:} A Q-learning-based approach that learns a tabular policy for each goal and infers the most likely goal based on these \citep{amado2022goal}. %to compute the likelihood of each goal hypothesis.
As GRAQL is inherently tabular, it is also restricted to discrete domains. Thus, as with R\&G, we compare it to AURA only on the MiniGrid domain (Section \ref{sec:meta_discrete}). %, using the results reported in Nageris et al. \shortcite{nageris2024goal}.
(3) \textit{DRACO:} GR method that supports discrete and continuous states, goals, and action spaces using deep RL \citep{nageris2024goal}. In the continuous domains: Panda-Gym and Point-Maze (Sections \ref{sec:gc_aura_exp} and \ref{sec:meta_continuous} respectively),
our Goal-Conditioned and Meta-RL algorithms are based on TRPO, while DRACO's original implementation uses PPO. To better isolate recognition performance and ensure a fair comparison with our approach, we modified DRACO to use TRPO as its underlying RL algorithm. This change avoids confounding effects from differences in RL strategy. %In different settings, DRACO training was referred to as Goal-Directed RL policy trained from scratch (compared to GCRL with Goal-Conditioned RL or Meta-RL with Goal-Directed RL policy trained from Meta-policy).
% we use Trust Region Policy Optimization (TRPO) instead of the algorithm's original implementation using PPO, as our goal-conditioned policies and meta-RL algorithms are based on TRPO. We made this modification so all algorithms use the same underlying RL algorithm.
\subsection{Policy Learning Algorithms.}
We implemented TRPO and MAML-TRPO using the Learn2Learn Library \citep{Arnold2020-ss}.
% \reuth{Discuss parallel being orthogonal to learning.}
%MAML enables efficient adaptation to new tasks by learning a meta-policy that can quickly fine-tune to new goals \cite{finn2017model}. TRPO, a policy gradient method, optimizes policies while maintaining trust regions to ensure stable and reliable updates, making it particularly suitable for goal-conditioned tasks in continuous domains.
All information on computational resources and efficiency is provided in Appendix~\ref{sec:compute}, information on hyperparameters and training details are presented in Appendix~\ref{appendix:hyperparameters},, and details about licenses and external packages can be found in Appendix~\ref{sec:licenses}.
\subsection{GR Methods.} We evaluate recognition performance, framed as a classification task, using standard metrics: Accuracy, Recall, Precision, and F-score. In addition, we employ two distinct distance metrics between observed trajectories and RL policies that are used by the GR algorithm for recognition inference (\texttt{DISTANCE} in Algorithm \ref{alg:GDGR}). In discrete state spaces (Section \ref{sec:meta_discrete}, Algorithms: Meta-AURA, DRACO, and GRAQL), we implement the recognition inference using Kullback–Leibler (KL) divergence as the recognition metric:
\begin{equation}
D_{\text{KL}}(\pi_g \| \pi_O) = \sum_{i \in |O|} \pi_g(a_i|s_i) \log \frac{\pi_g(a_i|s_i)}{\pi_O(a_i|s_i)}
\end{equation}
where $\pi_g$ is a goal-dependent softmax policy and $\pi_O$ represents a pseudo-policy derived from observations $O$. For continuous goal spaces (Sections: \ref{sec:gc_aura_exp} and \ref{sec:meta_continuous}, Algorithms: GC-AURA, Meta-AURA, and DRACO), we utilize the Wasserstein distance metric, as introduced in Nageris et al. for recognition purposes \citep{nageris2024goal}:
\begin{equation}
W(O, \pi) = \mathbb{E}_{(s,a) \in O}[\|a - \tilde{a}\|_{L_1}], \quad \tilde{a} \sim \pi(s)
\end{equation}
where $W(O, \pi)$ is the Wasserstein distance between observation sequence $O$ and policy $\pi$; $\mathbb{E}{(s,a) \in O}$ is the expected value over state-action pairs in observation sequence; $|a - \tilde{a}|_{L_1}$ is the L1 norm (absolute difference) between observed action $a$ and sampled action $\tilde{a}$; $\tilde{a} \sim \pi(s)$ is an action $\tilde{a}$ sampled from policy $\pi$ given state $s$.
% \noindent \textbf{Metrics.}
% We adapt standard machine learning metrics to the context of goal recognition. A goal is considered correctly classified if it has the highest probability among all goal hypotheses.
% - \textit{Accuracy} measures the proportion of correctly classified goals out of all predictions.
% - \textit{Precision} evaluates how often a predicted goal is correct, reflecting the method's ability to avoid false positives.
% - \textit{Recall} quantifies the frequency with which the correct goal is identified as the most probable.
% - \textit{F1 Score} provides a harmonic mean of precision and recall, offering a balanced measure of a model's performance, especially when precision and recall exhibit a trade-off.
% For recognition quality, we use standard machine learning metrics: Accuracy, Recall, Precision, and F-score.
% Additionally, we assess the framework's confidence in its predictions. Confidence is defined as the difference between the probabilities of the two most likely goals, normalized by the probability of the most likely goal. If two or more goals are equiprobable, the confidence score is zero. This metric provides insight into the model's certainty in distinguishing between competing hypotheses.
% \noindent\textbf{Domains:} We took three different domains, all vary greatly in their properties --- from the state and action representation, to the type of reward given and potential baselines applicable in each domain.
% Next, we provide three sets of experiments meant to evaluate the capabilities of GC-AURA in terms of recognition when introduced with new goals (Subsection \ref{sec:gc_aura_exp}); Meta-AURA's learning and recognition when introduced with new goals and new domains in discrete environments (Subsection \ref{sec:meta_discrete}); and Meta-AURA's adaptation to new goals and domains in continuous environments (Subsection \ref{sec:meta_continuous}).
% \begin{table*}[t]
% \footnotesize
% \small
% \setlength{\tabcolsep}{2pt}
% \centering
% \begin{tabular}{|c|c|c|c|c|c|}
% \hline
% & &
% \multicolumn{1}{|c|}{Meta-AURA after 100 iterations} &
% \multicolumn{1}{|c|}{DRACO after 350 iterations} &
% \multicolumn{1}{|c|}{GRAQL after 700 iterations} &
% \multicolumn{1}{|c|}{Ramirez and Geffner} \\
% \hline
% OBS & Problem & F-Score & F-Score & F-Score & F-Score\\
% \hline
% 10\% & 2-Goals &
% % \multicolumn{1}{c|}{0.9\textpm\ 0.31} &
% % \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.31}$} &
% % \multicolumn{1}{c|}{0.9\textpm\ 0.31} &
% \multicolumn{1}{c|}{0.9 \textpm\ 0.31} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{0.70 \textpm\ 0.35} &
% % \multicolumn{1}{c|}{0.64 \textpm\ 0.35} &
% % \multicolumn{1}{c|}{0.9 \textpm\ 0.31} &
% \multicolumn{1}{c|}{0.75 \textpm\ 0.32} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{0.5 \textpm\ 0.0} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{0.67 \textpm\ 0.00}
% \\
% & 3-Goals &
% % \multicolumn{1}{c|}{0.93\textpm\ 0.21} &
% % \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.31}$} &
% % \multicolumn{1}{c|}{0.9\textpm\ 0.31} &
% \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.31}$} &
% \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.31}$} &
% % \multicolumn{1}{c|}{0.66 \textpm\ 0.35} &
% % \multicolumn{1}{c|}{0.56 \textpm\ 0.40} &
% % \multicolumn{1}{c|}{0.9 \textpm\ 0.32} &
% \multicolumn{1}{c|}{0.64 \textpm\ 0.35} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{0.36 \textpm\ 0.0} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{0.52 \textpm\ 0.00}\\
% \hline
% 30\% & 2-Goals &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{0.71 \textpm\ 0.49} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{0.83 \textpm\ 0.48}
% \\
% & 3-Goals &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$}&
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% % \multicolumn{1}{c|}{0.62 \textpm\ 0.49} &
% % \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{0.77 \textpm\ 0.48}
% \\
% \hline
% \end{tabular}
% \caption{
% % Impact of partial observability, KL-Divergence is used to measure the distance between the optimal paths and the given observation in both approaches. The results are the average (± standard deviation) of 10 different executions.
% % Performance in the Minigrid domain under 10\% and 30\% partial observability.
% % Impact of partial observability on the performance of the three approaches in the MiniGrid domain.
% % The table reports the average performance (± standard deviation) of the algorithms over 30 different GR settings under 10\% and 30\% partial observability
% Performance in MiniGrid over 20 GR settings under 10\% and 30\% partial observability (using KL-divergence GR distance metric).
% % The table reports the average performance (± standard deviation) of the algorithms over 30 different GR settings under 10\% and 30\% partial observability.
% % \textit{Con.} stands for Confidence.}
% }
% \label{tab:minigrid}
% \end{table*}
\begin{figure}[t]
\centering
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/panda_gc_compared_to_gd_v2.png}
\caption{Training curves for the PandaReach-v3 domain. The \textbf{orange curve} represents the Goal-Conditioned TRPO policy (for GC-AURA) trained across all goals in the continuous goal space.
% This policy, while slower to train, allows for immediate adaptation to any goal without requiring additional training \reuth{this sentence should be in the main text, not the caption}.
The \textbf{blue curve} shows the goal-directed TRPO policy (for DRACO) trained for a specific goal. The shaded regions indicate variations across changing seeds and goals.}
\label{fig:gc_aura_train}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/gr_panda_gd_against_gc.png}
% \caption{F-Score for PandaReach environment with 10\% observability rate and 0-90\% noise rate, evaluated using the Wasserstein GR distance metric across 20 different GR tasks. The \textbf{orange curve} represents the F-Score of the GC-TRPO policy across different iterations. The \textbf{blue curve} shows the F-Score of the goal-directed TRPO policy across different iterations. The shaded regions indicate the standard deviations across the 20 GR tasks.}
\caption{F-Score for PandaReach environment with 10\% observability rate, evaluated across 20 different GR problems. The \textbf{orange curve} represents the F-Score of the GC-TRPO policy (for GC-AURA) across different iterations. The \textbf{blue curve} shows the F-Score of the goal-directed TRPO policy (for DRACO) across different iterations. The shaded regions indicate the standard deviations across the 20 GR problems.}
\label{fig:gc_aura_rec}
\end{minipage}
\end{figure}
\begin{table*}[t]
\centering
\begin{tabular}{llrrrr}
\toprule
\textbf{OBS} & \textbf{Problem} & \textbf{Meta-AURA (100 iters)} & \textbf{DRACO (350 iters)} & \textbf{GRAQL (700 iters)} & \textbf{Ramirez \& Geffner} \\
\midrule
10\% & 2-Goals & 0.90 ± 0.31 & \textbf{1.00 ± 0.00} & 0.75 ± 0.32 & 0.67 ± 0.00 \\
& 3-Goals & \textbf{0.90 ± 0.31} & \textbf{0.90 ± 0.31} & 0.64 ± 0.35 & 0.52 ± 0.00 \\
\midrule
30\% & 2-Goals & \textbf{1.00 ± 0.00} & \textbf{1.00 ± 0.00} & \textbf{1.00 ± 0.00} & 0.83 ± 0.48 \\
& 3-Goals & \textbf{1.00 ± 0.00} & \textbf{1.00 ± 0.00} & \textbf{1.00 ± 0.00} & 0.77 ± 0.48 \\
\bottomrule
\end{tabular}
\caption{Performance in MiniGrid over 20 GR settings under 10\% and 30\% partial observability (using KL-divergence as the GR distance metric). The table reports the average F-Score (± standard deviation) of the algorithms across different GR problems.}
\label{tab:minigrid}
\end{table*}
\begin{table*}[t]
\centering
\begin{tabular}{llrrrrrrrr}
\toprule
\textbf{OBS} & \textbf{G} & \multicolumn{4}{c}{\textbf{Meta-AURA (500 iterations)}} & \multicolumn{4}{c}{\textbf{DRACO (1000 iterations)}} \\
\cmidrule(lr){3-6} \cmidrule(lr){7-10}
& & \textbf{Accuracy} & \textbf{Precision} & \textbf{Recall} & \textbf{F-Score} & \textbf{Accuracy} & \textbf{Precision} & \textbf{Recall} & \textbf{F-Score} \\
\midrule
1\% & 2 & \textbf{0.90 ± 0.32} & \textbf{0.90 ± 0.32} & \textbf{0.90 ± 0.32} & \textbf{0.90 ± 0.32} & 0.40 ± 0.52 & 0.40 ± 0.52 & 0.40 ± 0.52 & 0.40 ± 0.52 \\
& 3 & \textbf{0.99 ± 0.21} & \textbf{0.90 ± 0.32} & \textbf{0.90 ± 0.32} & \textbf{0.90 ± 0.32} & 0.60 ± 0.34 & 0.40 ± 0.52 & 0.40 ± 0.52 & 0.40 ± 0.52 \\
\midrule
3\% & 2 & \textbf{1.00 ± 0.00} & \textbf{1.00 ± 0.00} & \textbf{1.00 ± 0.00} & \textbf{1.00 ± 0.00} & 0.90 ± 0.32 & 0.90 ± 0.32 & 0.90 ± 0.32 & 0.90 ± 0.32 \\
& 3 & \textbf{1.00 ± 0.00} & \textbf{1.00 ± 0.00} & \textbf{1.00 ± 0.00} & \textbf{1.00 ± 0.00} & 0.93 ± 0.21 & 0.90 ± 0.32 & 0.90 ± 0.32 & 0.90 ± 0.32 \\
\bottomrule
\end{tabular}
\caption{
Performance in PointMaze over 20 GR settings under 1\% and 3\% partial observability (OBS) using Wasserstein GR distance metric in domains with 2 or 3 goals (G). The table reports the average performance (± standard deviation) for Meta-AURA and DRACO across different GR problems.
}
\label{tab:pointmaze}
\end{table*}
\section{Results}
We evaluate AURA across two core generalization constructs: adaptation to new goals, and adaptation to new domains. The latter is further split into adaptation in discrete vs. in continuous domains. In Section \ref{sec:gc_aura_exp}, we test goal generalization in a fixed domain, comparing GC-AURA to DRACO in continuous goal spaces without additional fine-tuning. Section \ref{sec:meta_discrete} examines domain generalization in discrete environments, using Meta-AURA to adapt across changing transitions and goals in MiniGrid. Section \ref{sec:meta_continuous} extends this evaluation to continuous domains, assessing Meta-AURA’s ability to generalize across structurally varied PointMaze environments. Together, these experiments test AURA's capacity to learn transferable policies, recognize goals under uncertainty, and adapt across diverse settings.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Experiment 1: GC-AURA
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Adaptation to New Goals - \textcolor{RoyalPurple}{DG} (Fixed Domains)}
\label{sec:gc_aura_exp}
% \subsubsection{Experimental Setup}
To evaluate AURA's generalizability to new goals, we compare GC-AURA to DRACO in the PandaReach environment with dense rewards. GC-AURA policies, did not require any additional fine-tuning after the initial training, while DRACO is trained from scratch for each new goal introduced. For Goal-Directed training (DRACO), we selected three settings with specific goals. %, as shown in Table~\ref{tab:domains}.
During the Goal-Conditioned training process, the goal locations were chosen randomly from the continuous state space in each episode. TRPO was used to train all policies. Each instance consisted of 4 possible goals, and was tested with 3 different seeds.
For recognition performance, 20 different GR problems were evaluated. In addition, we used observation sequences with 10\% observability and various noise levels (0-90\%). $x\%$ noise implies that for each action of the agent that generated this observation sequence, with a probability of $\frac{x}{100}$, the action was chosen based on the trained policy, and with a probability of $1 - \frac{x}{100}$, the action was chosen randomly.
%of the time, the policy acted randomly, and . %The Wasserstein distance metric, as introduced in Nageris \textit{et al}. \cite{nageris2024goal}, was used to compute the distance between observations and the policy.
% \subsubsection{Results}
In terms of \textbf{learning}, as shown in Figure \ref{fig:gc_aura_train}, DRACO reached an approximate optimal policy within $\approx 400$ iterations, while GC-AURA required $\approx 1400$ iterations (approximately 3.5 times slower). However, GC-AURA trained a policy applicable across all goals in a continuous goal space, whereas the DRACO was limited to individual goals.
In terms of \textbf{recognition quality}, Figure \ref{fig:gc_aura_rec} highlights that the GC policy converged to a perfect F-Score during recognition after 600 iterations, demonstrating robust performance even with 10\% partial observability and high noise. In contrast, despite training for 400 iterations per new goal, DRACO policies struggled to achieve such performance under the same conditions. %\matan{You say that TRPO reaching recognition quality, and that TRPO is measured on GR tasks. But TRPO is an RL algorithm, not GR. Maybe you meant GC-AURA with GC-TRPO, and DRACO with TRPO? I think it's worth using the full names since it is not clear what s measured against what.}
%
Thus, while GC-AURA requires longer initial training, it has two key advantages: (1) it can adapt to new goals in a continuous space without requiring additional fine-tuning, and (2) it exhibits less noise-susceptibility than DRACO's goal-directed policies.
% \begin{table*}[t]
% \footnotesize
% \small
% \setlength{\tabcolsep}{2pt}
% \centering
% \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
% \hline
% & &
% \multicolumn{4}{|c|}{DRACO after 1000 iterations} & \multicolumn{4}{|c|}{Meta-AURA after 500 iterations} \\
% \hline
% OBS & Problem & Accuracy & Precision & Recall & F-Score & Accuracy & Precision & Recall & F-Score \\
% \hline
% 1\% & 2-Goals &
% \multicolumn{1}{c|}{0.4\textpm\ 0.52} &
% \multicolumn{1}{c|}{0.4\textpm\ 0.52} &
% \multicolumn{1}{c|}{0.4\textpm\ 0.52} &
% \multicolumn{1}{c|}{0.4\textpm\ 0.52} &
% \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.32}$} &
% \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.32}$} &
% \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.32}$} &
% \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.32}$}
% \\
% & 3-Goals &
% \multicolumn{1}{c|}{0.6\textpm\ 0.34} &
% \multicolumn{1}{c|}{0.4\textpm\ 0.52} &
% \multicolumn{1}{c|}{0.4\textpm\ 0.52} &
% \multicolumn{1}{c|}{0.4\textpm\ 0.52} &
% \multicolumn{1}{c|}{$\textbf{0.99}\boldsymbol{\pm} \textbf{0.21}$} &
% \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.32}$} &
% \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.32}$} &
% \multicolumn{1}{c|}{$\textbf{0.9}\boldsymbol{\pm} \textbf{0.32}$} \\
% \hline
% 3\% & 2-Goals &
% \multicolumn{1}{c|}{0.9\textpm\ 0.32} &
% \multicolumn{1}{c|}{0.9\textpm\ 0.32} &
% \multicolumn{1}{c|}{0.9\textpm\ 0.32} &
% \multicolumn{1}{c|}{0.9\textpm\ 0.32} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$}
% \\
% & 3-Goals &
% \multicolumn{1}{c|}{0.93\textpm\ 0.21} &
% \multicolumn{1}{c|}{0.9\textpm\ 0.32} &
% \multicolumn{1}{c|}{0.9\textpm\ 0.32} &
% \multicolumn{1}{c|}{0.9\textpm\ 0.32}&
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} &
% \multicolumn{1}{c|}{$\textbf{1.0}\boldsymbol{\pm} \textbf{0.0}$} \\
% \hline
% \end{tabular}
% \caption{
% Performance in PointMaze over 20 GR settings under 1\% and 3\% partial observability (using Wasserstein GR distance metric).
% % \reuth{Need to put Meta-AURA's values in bold. Both tables should have their font size bigger}
% }
% \label{tab:pointmaze}
% \end{table*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Experiment 2: META-AURA in Discrete Domains
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Adaptation to New Discrete Domains - \textcolor{orange}{D} (Changing Transitions and Goals)}
\label{sec:meta_discrete}
To evaluate AURA’s ability to generalize across new discrete settings, we compare Meta-AURA, GRAQL, DRACO, and the Ramirez \& Geffner planner-based method (R\&G). We tested each approach in three different goal configurations (Table \ref{tab:domains}) on the MiniGrid Empty-9×9 domain. Note that each method relies on a distinct input format: R\&G operates over a hand-crafted PDDL domain and problem file; GRAQL uses a symbolic MDP representation of the grid (discrete states and actions); DRACO and Meta-AURA consume raw image observations (pixel vectors) as input, without any additional symbolic context.
\begin{figure}[t]
\begin{center}
% Top row: MiniGrid plots
\begin{subfigure}[b]{0.45\columnwidth}
\centering
\includegraphics[width=\textwidth]{Figures/point_maze_train1.png}
\caption{Training maze example 1}
\label{fig:point_maze_train1}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\columnwidth}
\centering
\includegraphics[width=\textwidth]{Figures/point_maze_train2.png}
\caption{Training maze example 2}
\label{fig:point_maze_train2}
\end{subfigure}
% Bottom row: Point Maze and Panda-Gym plots
% \vspace{0.3cm}
\begin{subfigure}[b]{0.45\columnwidth}
\centering
\includegraphics[width=\textwidth]{Figures/point_maze_train3.png}
\caption{Training maze example 3}
\label{fig:point_maze_train3}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\columnwidth}
\centering
\includegraphics[width=\textwidth]{Figures/point_maze_train4.png}
\caption{Training maze example 4}
\label{fig:point_maze_train4}
\end{subfigure}
\caption{Examples from the train set of Point-Maze environments for MAML-RL training}
\label{fig:point_maze_train_set}
\end{center}
\end{figure}
% We compare AURA to DRACO trained using TRPO, using a similar protocol as the previous experiment. %For Goal-Directed training (GD-AURA), we selected three environments with specific goals, as shown in Figure~\ref{fig:3goal}, and trained a policy for each goal using TRPO with multiple seeds.
For Meta-AURA, the MAML-TRPO training involved randomly changing the goal location after each episode to vary the rewards, and placing 0–4 lava tiles in different locations within the domain to vary the transition dynamics. The meta-policy was trained using MAML-TRPO with 3 different seeds. The trained meta-policy (iteration 150 of MAML-TRPO)
% \reuth{Did the training of the meta policy take only 150 iterations, or was that just the refinement of the meta-policy for a specific goal?}
was then fine-tuned using TRPO for every goal, while GRAQL and DRACO policies were trained from scratch for every goal. % as the initial policy for the same three goals used in GD-AURA.
GR performance was evaluated under low observability rates, where only 10\% and 30\% of the sequence was observed. %partial observabili.
%using four metrics (accuracy, precision, recall, and F-Score)
Ten GR problems were evaluated for two-goal and ten for three-goal scenarios across DRACO policies (trained for $\approx 350$ iterations), Meta-AURA policies (trained for $\approx 100$ iterations), GRAQL policies (trained for $\approx 700$ iterations), and R\&G. %KL-divergence was used as the distance metric for all GR tasks.
Figure \ref{fig:minigrid_adaptation} shows a comparison of Meta-AURA and DRACO in terms of goal adaptation (training) time for a novel goal in a new domain. Meta-AURA achieved high rewards after $\approx 10$ fine-tuning TRPO iterations, while DRACO required $\approx 100$ TRPO iterations to reach similar rewards for that goal. Table \ref{tab:minigrid} indicates that Meta-AURA outperformed GRAQL and R\&G under low observability (10\% and 30\%), even with a more challenging visual representation.
% \osher{Add GPU vs. CPU, representations, offline/online}
Regarding runtime, the premise of using AURA is reducing goal adaptation and inference times.
Meta-AURA's Init Memory Phase required $150$ iterations of MAML-TRPO training, and the goal adaptation time was $\approx 100$ fine-tuning TRPO iterations. DRACO's goal adaptation time was approximately $\approx 350$ TRPO iterations, meaning that DRACO required $\approx 3.5$ times more iterations to achieve similar or higher recognition rates. Domain adaptation times are constant, as they are not used in Meta-AURA and DRACO. R\&G did not require training time but had significantly higher inference times ($\approx 2$ seconds) compared to Meta-AURA and DRACO (both with $\approx 0.13$ seconds) and GRAQL (approximately $\approx 0.1$ seconds).
% Meta-AURA's Init Information State phase time was 150 iterations of MAML-TRPO training, and the goal adaptation time was $\approx 100$ fine-tuning TRPO iterations. DRACO's goal adaptation time was $\approx 350$ TRPO iterations. Means that in order to reach similar or higher GR performance DRACO needed using $\approx 3.5$ more iterations. Note that All domain adaptation times are constant, as they are not used in Meta-AURA and DRACO. R\&G did not require training time but had significantly higher inference times ($\approx 2$ seconds) compared to Meta-AURA, DRACO (both with $\approx 0.13$ seconds) and GRAQL ($\approx 0.1$ seconds).
% Meta-AURA's domain adaptation time was $38.06 \pm 10.75$ minutes, and goal adaptation time was $64.71 \pm 2.34$ minutes. DRACO's average goal adaptation time was $77 \pm 2.29$ minutes. %, whereas GRAQL required $1108 \pm 463$ minutes.
% R\&G did not require training time but had significantly higher inference times ($\approx 2$ seconds) compared to Meta-AURA and DRACO ($\approx 0.12$ seconds) and GRAQL ($\approx 0.1$ seconds).
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Experiment 3: META-AURA in Continuous Domains
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Adaptation to New Continuous Domains (\textcolor{orange}{D}) (Changing Transitions and Goals)}
\label{sec:meta_continuous}
To evaluate AURA's generalization across new continuous settings, we used the PointMaze Four-Rooms 11x11 environment with sparse rewards (goals at 1x9, 1x6 and 4x9 locations). For DRACO, we followed a similar protocol to the previous experiment. We selected three domains with specific goals and trained a policy for each goal using TRPO with multiple seeds for $\approx 1000$ iterations.
% For GC-AURA, the environment replaced the goal location randomly across the continuous goal space after each episode, and a single policy was trained using SAC with HER for faster sparse goal-conditioned training.
For Meta-AURA, the MAML-TRPO training used domains with varying goal locations, rewards, sizes (6x6 to 9x9) and obstacles (see figure~\ref{fig:point_maze_train_set} and more details on MAML's performance in Appendix~\ref{sec:compute}). The trained meta-policy (iteration 50)
% \reuth{Did the training of the meta policy take only 150 iterations, or was that just the refinement of the meta-policy for a specific goal?}
was then fine-tuned using TRPO for the same three goals used in DRACO for $\approx 500$ iterations. %Unlike Minigrid, the meta-policy here demonstrated generalization to a new domain not included in the meta-training set.
GR performance was evaluated under 1\% and 3\% partial observability levels. Ten GR problems were evaluated for two-goal and ten for three-goal scenarios using DRACO (trained for $\approx 1000$ iterations) and Meta-AURA policies (trained for $\approx 500$ iterations). %, and GC-AURA policies (trained after 1500 iterations).
\begin{figure}[t]
\centering
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/minigrid_gd_from_scratch_vs_from_maml_7x7.png}
\caption{Adaptation to new goals (training) from a Meta-Policy and from scratch in Minigrid Empty 9x9 environment for goal 7x7.}
\label{fig:minigrid_adaptation}
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[width=\textwidth]{Figures/point_maze_gd_from_maml_vs_from_scratch_1x9.png}
\caption{Adaptation to new goals (training) from a Meta-Policy and from scratch in Point-Maze 4 Rooms 11x11 with goal 1x9.}
\label{fig:meta_aura_train}
\end{minipage}
\end{figure}
Figure \ref{fig:meta_aura_train} shows that in the goal adaptation phase, Meta-AURA policies achieved high rewards within fewer iterations ($\approx 10$), while DRACO's policies that were trained from scratch required approximately $\approx 200$ or more iterations until reaching similar rewards. %GC-AURA required 1500 iterations to train the policy.
%
Table \ref{tab:pointmaze} shows that the fine-tuned Meta-AURA policies (after $\approx 500$ iterations) outperformed DRACO (after $\approx 1000$ iterations) in terms of GR quality with a shorter goal adaptation phase. % while GC-AURA policies (after 1000 iterations) achieved perfect GR performance, even under 1\% partial observability. GD-AURA, despite near-optimal policies, failed to achieve perfect GR performance.
%
% GC-AURA's primary advantages include covering all goals in the continuous space without requiring additional fine-tuning, resulting in fast goal adaptation times. Meta-AURA demonstrated the lowest goal adaptation time due to the meta-policy initialization. GD-AURA, while simpler, incurred the highest training time for each new GR task. These results highlight the benefits of abstraction in runtime scenarios, where goal recognizers must prioritize real-time inference and adaptation.
% \section{GC-AURA Evaluation}
% \subsection{Evaluation}
% In these experiments, we focused on the PandaReach-v3 domain with dense rewards to evaluate the performance and runtime efficiency of goal-conditioned reinforcement learning (GCRL) compared to goal-directed reinforcement learning (GR as RL). The experiments utilized TRPO for both approaches. For GCRL, a single goal-conditioned policy was trained across all goals in the continuous goal space, while the goal-directed policies were trained for individual goals. The dense reward setting ensured smooth learning progress, while the continuous goal space presented significant challenges to the adaptation and scalability of the methods.
% The training process for the goal-conditioned TRPO policy was approximately 3.5 times slower than training a goal-directed TRPO policy for a single goal. However, the key advantage of GCRL lies in its ability to adapt to any new goal within the goal space without requiring additional training, making it highly efficient for dynamic tasks. In contrast, goal-directed policies need to be retrained for every new goal, which is computationally expensive and time-consuming.
% \subsection{Performance Comparison}
% Figure \ref{fig:Experiment} illustrates the training curves for the goal-conditioned and goal-directed policies in the PandaReach-v3 domain. The x-axis represents the number of iterations, while the y-axis shows the iteration rewards:
% \begin{itemize}
% \item The \textbf{orange curve} represents the goal-conditioned TRPO policy trained across all goals in the continuous goal space. While the training time is longer, this policy provides instant adaptation to any goal, eliminating the need for additional training during runtime.
% \item The \textbf{blue curve} represents goal-directed TRPO policies trained for specific goals. These policies converge faster during training for individual goals but require retraining for each new goal, limiting their scalability.
% \end{itemize}
% These results demonstrate the trade-off between training time and runtime efficiency for the two approaches:
% \begin{enumerate}
% \item \textbf{Efficiency of GCRL:} The goal-conditioned TRPO policy requires a longer initial training phase but eliminates the need for further training for new goals. This is a significant advantage in real-time applications where quick adaptation is critical.
% \item \textbf{Challenges in Goal-Directed RL:} Goal-directed TRPO policies converge faster for individual goals but require retraining from scratch for each new goal. This approach becomes impractical in domains with large or continuous goal spaces.
% \item \textbf{Scalability:} GCRL provides a scalable solution by learning a single policy capable of generalizing across all goals in the continuous goal space. In contrast, goal-directed RL lacks this scalability, making it unsuitable for complex, real-world environments.
% \end{enumerate}
% \subsection{Results}
% The results of our experiments are summarized as follows:
% \begin{itemize}
% \item The goal-conditioned TRPO policy achieved robust performance across all goals in the continuous goal space, demonstrating its adaptability and efficiency.
% \item The runtime of goal-conditioned policies was approximately 3.5 times slower than goal-directed TRPO policies during training. However, the ability to adapt instantly to any goal without additional training provides a clear advantage for dynamic tasks.
% \item Goal-directed TRPO policies, while faster to train for individual goals, were limited by the need for retraining for every new goal, making them less practical for environments with diverse goal spaces.
% \end{itemize}
% These findings highlight the superiority of goal-conditioned policies for tasks requiring scalability and quick adaptation, particularly in dynamic or continuous goal-space domains.
% \noindent\textbf{Performance}
% Table \ref{tab:minigrid} shows results for MiniGrid.
% All approaches perform overwhelmingly well, with perfect scores on all measures when the observability is $0.5$ or higher.
% However, this result misses part of the story:
% while GRAQL's and R\&G's true goal confidence score was close to the other goals' score ($9\%$ and $13\%$ difference on average respectively), DRACO had very different scores between the true goal and the other candidates ($71\%$ on average), making DRACO more confident in its predictions.
% Moreover, DRACO's results dominate GRAQL's and R\&G results in the more challenging, low observability ($10\%$, $30\%$), problem variants.
% It is a reasonable outcome because DL can make significant inroads to recognizing patterns and variance instead of valuing the current state.
% Moreover, our use of NN-based policies makes DRACO more forgiving to missing observations, as the likelihood of these missing actions is $\epsilon$ rather than zero, and this difference is accumulated with the number of missing observations.
% Lastly, as this table shows, DRACO and GRAQL both outperform the R\&G approach in most of these discrete domains.
% Importantly, DRACO dominates both approaches in the confidence metric, inferring higher probabilities for the correct goal, whereas GRAQL and R\&G often disambiguate poorly.
% \noindent\textbf{Scalability.}
% Scalability evaluation has three main factors: storage, training, and inference time.
% First, in terms of storage consumption, DRACO uses a constant $8 \pm 0$ MB while GRAQL $3190 \pm 700$ MB.
% This difference is primarily due to the huge storage needs of the Q-table used by GRAQL.
% By contrast, DRACO uses constant memory because it stores only the network architecture and neurons' values.
% Second, DRACO's average training time was $13.52 \pm 4.3$ minutes while GRAQL's was $1108 \pm 463$ minutes. We trained both approaches on the same commodity Tesla K40 server.
% This difference is due to two factors: (1) GRAQL relies on the CPU for its computation, whereas DRACO uses a GPU; and (2) GRAQL's storage-intensive operations are more expensive.
% Third, R\&G does not require training time, but its inference time is significantly higher than that of the other algorithms, as it runs planners during inference.
% Specifically, DRACO and GRAQL's inference runtime takes $\approx0.12$ and $\approx0.1$ seconds on average respectively while R\&G takes $\approx2$ seconds.
% To conclude, we evaluate the paper's two main contributions: the DRACO framework and the proposed distance metrics.
% DRACO scaled better than the state-of-the-art GR algorithms in storage consumption, computation time, and performance.
% DRACO requires no domain expert to craft an elaborate model like R\&G and instead learns via interactions.
% It achieves better or equal performance results compared to GRAQL and R\&G, with both metrics performing overwhelmingly well.
% Wasserstein was slightly better in less observable spaces and under noisy observations.
% \subsection{Evaluation}
% The evaluation compared two adaptation strategies tested on an 11x11 PointMazeFourRoomsEnv. We evaluated the performance adapting to three goal locations within this environment: 1x9, 9x1, and 9x9.
% \begin{itemize}
% \item \textbf{Meta-Policy Adaptation}: Starting with the meta-policy trained via MAML, the agent adapted to new goal locations using TRPO.
% \item \textbf{From Scratch}: The agent was initialized with a randomly initialized policy and trained from scratch on the same goal locations using TRPO.
% \end{itemize}
% For both strategies, the same hyperparameters and number of iterations were used to ensure comparability. To account for stochasticity, experiments were repeated with multiple random seeds for each goal location.
% During training, the agent's performance was measured as the cumulative rewards achieved at each iteration. For each goal location and condition (meta-policy vs. from scratch), performance data were aggregated across random seeds.
% The mean reward and individual seed performance are visualized to illustrate the variability and overall trends (Figure 6). Individual seeds are plotted with transparency to highlight variability, while the mean performance is plotted as a darker, opaque line for clarity.
\section{Related Work}
Over the past two decades, GR has been addressed primarily through symbolic approaches, such as planning \citep{ramirez2009plan,sohrabi2016plan,meneguzzi2021survey}. These studies introduced the concept of GR by employing planners to infer the most likely goal based on given observations. %However, planning-based methods require detailed domain knowledge and often rely heavily on expert inputs, which can be impractical in dynamic or unfamiliar environments. Furthermore,
In stochastic and continuous domains GR required a shift from traditional methods toward model-free approaches. Recent advancements have explored the integration of machine learning techniques into GR problems. Notably, the GR as RL framework \citep{amado2022goal} has shown promise by leveraging RL to infer goals in such environments for discrete goal spaces, and DRACO framework \citep{fang2023real,nageris2024goal} for continuous goal spaces. Other studies reframed GR as a supervised learning problem, leveraging deep learning for goal classification \citep{min2014deep,amado2018goal,chiari2023goal}. Further approaches utilize process-mining to learn the expected processes that are likely for each goal \citep{ko2023plan,su2023fast}.
%
These methods are all designed for static, single GR problems, limiting their applicability in real-world settings where agents must handle continuous and changing tasks. Recent work introduced Online Dynamic Goal Recognition for grid-based domains \citep{shamir2024odgr}. This definition encompasses multiple GR tasks within the same domain and provides a proof-of-concept for dynamic GR with changing goals in a simple, discrete navigational domain. % While this work conceptualized Dynamic GR, it was limited to empty navigational domains and GR tasks within the same domain with discrete state and goal spaces. %However, these approaches depend heavily on the availability of extensive supervised datasets and often lack interpretability, which poses challenges in critical applications where understanding the rationale behind decisions is essential.
%
%Add the paragraph below if accepted:
% Recent work introduced Online Dynamic Goal Recognition \citep{shamir2024odgr}. This definition encompasses multiple GR tasks within the same domain and provides a proof-of-concept for implementing dynamic GR in a discrete, simple navigational domain. % While this work conceptualized Dynamic GR, it was limited to empty navigational domains and GR tasks within the same domain with discrete state and goal spaces.
% Fang et al. \shortcite{fang2023real} expanded the scope of GR as RL using function approximations, which enabled some generalizability to new goals. However, it primarily focused on handling multiple GR tasks within the same domain without emphasizing generalizability among different tasks.
% This research defines the GDGR problem, proposes AURA -- algorithm for GDGR, and presents a specific application of this algorithm. Preliminary results demonstrate its time efficiency in adapting to new GR tasks, highlighting its potential to advance GR in dynamic environments.
\section{Conclusion}
\label{sec:discussion}
In this paper, we introduced the \textbf{General Dynamic Goal Recognition (GDGR)} problem, a generalization of the GR problem for dynamic environments where goals and settings change over time. To address this, we proposed the \textbf{Adaptive Universal Recognition Algorithm (AURA)}, an algorithmic framework for GDGR with three abstraction levels, each of which could benefit from mechanisms that support faster adaptation, such as caching to reuse past experiences for previously seen tasks. We further provide two example implementations: Meta RL (Meta-AURA) and GCRL (GC-AURA). These algorithms were evaluated against traditional GR baselines across multiple domains to showcase AURA's ability to adapt quickly to new goals and new domains.
Experimental results show AURA enables faster GDGR, which is essential for applications like autonomous vehicles and assistive robots. GC-AURA demonstrated scalability and adaptability to new goals without extensive retraining, while the analysis of both implementations highlighted trade-offs across abstraction levels. GC-AURA handles noise and continuous goal spaces well within a domain, whereas Meta-AURA facilitates rapid adaptation across domains. These algorithms support tailoring GR frameworks to application needs, balancing training time and inference efficiency.
\paragraph{Limitations and Assumptions.} Meta-AURA assumes GR domains share state and action spaces, varying only in transitions and rewards. GC-AURA assumes effective training across continuous goal spaces, though high-dimensional goals may require restricting the space for reliable learning. While current implementations of AURA are RL-based, AURA is defined as a general algorithm, and future work will explore symbolic or hybrid implementations to enhance generality and efficiency.
\paragraph{Future Work} We plan to further integrate Meta-RL and Goal-Conditioned RL to improve GDGR and reach real-time GR when new goals and domains are introduced on-the-fly, and extend AURA to more realistic domains. By bridging theoretical advances with practical needs, we envision AURA as a foundation for next-generation GR systems in complex, dynamic settings.
%% The file named.bst is a bibliography style file for BibTeX 0.99c
\bibliographystyle{named}
\bibliography{ijcai25}
\clearpage
\appendix
% \section{GC-AURA Implementation}
% \label{app:gc_aura}
% \begin{algorithm*}[h]
% \caption{GDGR - Implemented with Goal Conditioned Reinforcement Learning within a specific domain theory}
% \begin{algorithmic}[1]
% \Require{$p_D$ - prior distribution of domain theories, {\color{ForestGreen} assumed to be distribution with a single domain theory $D=\langle S, A, G \rangle$}}
% \State Init $IS=\langle GCP_D, cache \rangle$
% \State $IS \gets \text{InitInformationStatePhase}(p_D, IS)$ \Comment{ {\color{ForestGreen}Train a Goal Conditioned RL policy $GCP_D$ for the domain theory $D$}}
% \For{all $T_i$ in $\text{GetGoalRecognitionTask}()$}
% \State Get Domain theory {\color{ForestGreen}$D_i=D$}
% \State $IS_D \gets \text{DomainAdaptationPhase}(D_i, IS)$ \Comment{{\color{ForestGreen}Return the trained $GCP_D$ from InitInformationStatePhase and the $cache$}}
% \State Get the set of new dynamic goals $DG_i$ from $T_i$
% \State $\{\pi_g\}_{g\in GD_i} \gets \text{GoalsAdaptationPhase}(DG_i, IS_D)$ \Comment{{\color{ForestGreen}On each goal $g \in DG_i$ it evaluates the goal specific RL policy $GCP_D(g)$ to decide whether to fine-tune it with few-shot transfer learning, or without modification (zero-shot transfer learning) $GCP_D(g)$, or taking a goal specific RL policy from the cache}}
% \State Get the Observation sequence $O_i=(\langle s_{i_0}, a_{i_0} \rangle, \ldots)$ from $T_i$ \Comment{In the online GR setting, each state and action tuple is provided in a different time step and has its own recognition inference phase}
% \State $g^* \gets \text{RecognitionInferencePhase}(\{\pi_g\}_{g\in GD_i}, O_i)$ \Comment{$g^* \gets \text{arg max}_{g\in DG_i}[\text{DISTANCE}(O_i, \pi_g)]$, {\color{ForestGreen}where DISTANCE Calculates the similarity between the observation sequence and the goal specific RL policy}}
% \State Save and return $g^*$
% \State $IS \gets \text{UpdateInformationStatePhase}(IS, GCP_D, \{\pi_g\}_{g\in GD_i}, T_i, g^*)$ \Comment{\color{ForestGreen}{Caching the $\pi_g$ and }}
% \EndFor
% \end{algorithmic}
% \label{alg:GDGR_GCP}
% \end{algorithm*}
% \paragraph{Point Maze Domain Settings}
% \label{appendix:point_maze_setting}
% In our experiments, we focused on the Point Maze Four Rooms Domain (Figure \ref{point_maze}, right image) with 1000 maximum steps in an episode, with sparse rewards:
% \begin{itemize}
% \item Reward \( = 0 \) if the agent's Euclidean distance to the goal is less than 0.5 meters.
% \item Reward \( = -1 \) if the distance is greater than 0.5 meters.
% \end{itemize}
% The state space is as follows:
% \begin{itemize}
% \item \( x \)-coordinate of the green ball (\texttt{position (m)}).
% \item \( y \)-coordinate of the green ball (\texttt{position (m)}).
% \item Linear velocity in the \( x \)-direction (\texttt{velocity (m/s)}).
% \item Linear velocity in the \( y \)-direction (\texttt{velocity (m/s)}).
% \end{itemize}
% The action space includes:
% \begin{itemize}
% \item Linear force in the \( x \)-direction (\texttt{force (N)}).
% \item Linear force in the \( y \)-direction (\texttt{force (N)}).
% \end{itemize}
% The goal space comprises:
% \begin{itemize}
% \item Current goal position in the \( x \)-coordinate (\texttt{position (m)}).
% \item Current goal position in the \( y \)-coordinate (\texttt{position (m)}).
% \end{itemize}
\section{Computational Resources and Efficiency}
\label{sec:compute}
This section provides details on the computational requirements of the different methods evaluated in our experiments. All experiments except for the R\&G baseline (which uses symbolic planners with their code) involve training RL policies, which require significant computational resources. In contrast, GR inference with trained RL policies is computationally light and can be executed on a CPU within seconds by simply loading the trained models.
\paragraph{Compute Settings.} All training procedures were conducted on a single Tesla K40 GPU machine, except for GRAQL and R\&G, which ran on CPU. The R\&G planner-based method completed in a few seconds per task, as it only involves symbolic planning rather than learning-based training.
\paragraph{Storage Requirements.}
% The storage footprint varies significantly across methods:
% \begin{itemize}
% \item \textbf{MAML-TRPO (Meta-AURA):} $\approx 150$ MB
% \item \textbf{GC-TRPO (GC-AURA):} $\approx 40$ MB
% \item \textbf{DRACO:} $\approx 15$ MB
% \item \textbf{GRAQL:} $\approx 3000$ MB
% \end{itemize}
GRAQL's high storage demand stems from its reliance on tabular Q-learning, which stores a large Q-table. In contrast, DRACO, GC-TRPO and MAML-TRPO store compact neural network weights, requiring significantly less storage.
% \paragraph{Training Time.} The average training time per approach was:
% \begin{itemize}
% \item \textbf{MAML-TRPO (Meta-AURA):} $\approx 5-8$ hours and $\approx 3-20$ minutes for the TRPO training using the MAML policy for one goal.
% \item \textbf{GC-TRPO (GC-AURA):} $\approx 1-3$ hours
% \item \textbf{DRACO:} $\approx 20-50$ minutes
% \item \textbf{GRAQL:} $\approx 10$ hours
% \end{itemize}
% The large difference in training times is primarily due to two factors: (1) GRAQL is CPU-based and lacks GPU acceleration, and (2) its tabular Q-learning strategy requires intensive storage operations. In contrast, both DRACO, GC-TRPO and MAML-TRPO benefit from GPU acceleration and train compact neural models.
\paragraph{Inference Efficiency.} After training, all GR inference tasks are fast. For RL-based methods (DRACO, GC-AURA and Meta-AURA), inference only involves a forward pass through a policy network, and thus can be executed in a few seconds even on a CPU. This makes these methods suitable for real-time applications, despite their higher training cost. Figures \ref{fig:minigrid_71_47}, \ref{fig:point_maze_16_49}, and \ref{fig:point_maze_91_99}, show the adaptation process in terms of iteration rewards, when training on new goals from scratch vs. when utilizing a meta policy. The First shows this adaptation in MiniGrid, and the latter two show this adaptation in Point-Maze settings.
\begin{figure}[t]
\begin{center}
\includegraphics[width=0.5\textwidth]{Figures/minigrid_maml_adaptation_rewards_v2.png}
\caption{Minigrid MAML-TRPO training}
\label{fig:minigrid_maml_adaptation_rewards}
\end{center}
\end{figure}
\begin{figure}[t]
\begin{center}
\includegraphics[width=0.5\textwidth]{Figures/point_maze_maml_adaptation_rewards_v2.png}
\caption{Point-Maze MAML-TRPO training}
\label{fig:point_maze_maml_adaptation_rewards}
\end{center}
\end{figure}
\begin{figure}[htbp!]
\centering
\includegraphics[width=1\linewidth]{Figures/minigrid_gd_from_scratch_vs_from_maml_7x1_4x7.png}
\caption{Adaptation to new goals (training) from a Meta-Policy and from scratch in Minigrid 9x9 Empty environment across 3 goals (7x1, 4x7).}
\label{fig:minigrid_71_47}
\end{figure}
\begin{figure}[htbp!]
\centering
\includegraphics[width=1\linewidth]{Figures/point_maze_gd_from_maml_vs_from_scratch_1x6_4x9.png}
\caption{Adaptation to new goals (training) from a Meta-Policy and from scratch in Point-Maze 4 Rooms 11x11 environment across 2 goals (1x6, 4x9).}
\label{fig:point_maze_16_49}
\end{figure}
\begin{figure}[htbp!]
\centering
\includegraphics[width=1\linewidth]{Figures/point_maze_gd_from_maml_vs_from_scratch_9x1_9x9.png}
\caption{Adaptation to new goals (training) from a Meta-Policy and from scratch in Point-Maze 4 Rooms 11x11 environment across 2 goals (9x1, 9x9).}
\label{fig:point_maze_91_99}
\end{figure}
\section{Hyperparameters}
A reasonable number of hyperparameter configurations were tested (over multiple seeds), and the results were found to be similar or less efficient than those reported here.
\label{appendix:hyperparameters}
\subsection{MAML-TRPO}
Below is an explanation of each hyperparameter used for training with MAML-TRPO:
\begin{itemize}
\item \textbf{Adaptation batch size (\texttt{adapt\_bsz}):} The number of trajectories sampled for adaptation during the inner loop.
\item \textbf{Adaptation learning rate (\texttt{adapt\_lr}):} The learning rate used for updating parameters during the inner loop for task-specific adaptation.
\item \textbf{Adaptation steps (\texttt{adapt\_steps}):} The number of gradient steps performed in the inner loop for task-specific adaptation.
\item \textbf{Backtracking factor (\texttt{backtrack\_factor}):} The factor by which the step size is reduced during line search to ensure the KL-divergence constraint is satisfied.
\item \textbf{Discount factor (\( \gamma \)):} The discount factor for future rewards in the reinforcement learning algorithm.
\item \textbf{Line search maximum steps (\texttt{ls\_max\_steps}):} The maximum number of steps for line search to satisfy the KL-divergence constraint.
\item \textbf{Maximum KL divergence (\texttt{max\_kl}):} The maximum allowed KL divergence between the new and old policy distributions during updates.
\item \textbf{Meta batch size (\texttt{meta\_bsz}):} The number of tasks sampled per iteration for meta-learning.
\item \textbf{Meta learning rate (\texttt{meta\_lr}):} The learning rate for updating meta-parameters across tasks.
\item \textbf{Seeds (\texttt{seed}):} Seeds used for ensuring reproducibility in experiments.
\item \textbf{Soft update coefficient (\( \tau \)):} The coefficient for updating the policy parameters during soft updates.
\end{itemize}
Below are the hyperparameters specific to each environment:
\subsection*{MAML-TRPO Hyperparameters for MiniGrid}
\begin{itemize}
\item Adaptation batch size (\texttt{adapt\_bsz}): 5
\item Adaptation learning rate (\texttt{adapt\_lr}): 0.01
\item Adaptation steps (\texttt{adapt\_steps}): 3
\item Backtracking factor (\texttt{backtrack\_factor}): 0.5
\item Discount factor (\( \gamma \)): 0.99
\item Line search maximum steps (\texttt{ls\_max\_steps}): 15
\item Maximum KL divergence (\texttt{max\_kl}): 0.01
\item Meta batch size (\texttt{meta\_bsz}): 5
\item Meta learning rate (\texttt{meta\_lr}): 1
\item Seeds (\texttt{seed}): 111, 222, 333, 444
\item Soft update coefficient (\( \tau \)): 1
\end{itemize}
\subsection*{MAML-TRPO Hyperparameters for PointMaze}
\begin{itemize}
\item Adaptation batch size (\texttt{adapt\_bsz}): 10
\item Adaptation learning rate (\texttt{adapt\_lr}): 0.1
\item Adaptation steps (\texttt{adapt\_steps}): 3
\item Backtracking factor (\texttt{backtrack\_factor}): 0.5
\item Discount factor (\( \gamma \)): 0.95
\item Line search maximum steps (\texttt{ls\_max\_steps}): 15
\item Maximum KL divergence (\texttt{max\_kl}): 0.01
\item Meta batch size (\texttt{meta\_bsz}): 10
\item Meta learning rate (\texttt{meta\_lr}): 1
\item Seeds (\texttt{seed}): 111, 222, 333
\item Soft update coefficient (\( \tau \)): 1
\end{itemize}
\subsection*{Hyperparameters for TRPO Training}
Below is the set
of hyperparameters used for TRPO training across different environments.
\subsubsection*{MiniGrid (Trained from Scratch and from MAML-TRPO Policy)}
\begin{itemize}
\item Learning rate (\texttt{lr}): 0.01
\item Batch size (\texttt{batch\_size}): 20
\item Discount factor (\( \gamma \)): 0.99
\item Soft update coefficient (\( \tau \)): 1.0
\item Random seeds (\texttt{seeds}): 100, 200, 300
\item Maximum KL divergence (\texttt{max\_kl}): 0.01
\item Backtracking factor (\texttt{backtrack\_factor}): 0.5
\item Line search maximum steps (\texttt{ls\_max\_steps}): 15
\end{itemize}
\subsubsection*{PandaGym (Trained for Goal Conditioned and Goal Directed TRPO Policies)}
\begin{itemize}
\item Learning rate (\texttt{lr}): 0.01
\item Number of iterations (\texttt{num\_iterations}): 400
\item Batch size (\texttt{batch\_size}): 20
\item Discount factor (\( \gamma \)): 0.99
\item Soft update coefficient (\( \tau \)): 1.0
\item Random seeds (\texttt{seeds}): 100, 200, 300
\item Maximum KL divergence (\texttt{max\_kl}): 0.01
\item Backtracking factor (\texttt{backtrack\_factor}): 0.5
\item Line search maximum steps (\texttt{ls\_max\_steps}): 15
\end{itemize}
\subsubsection*{PointMaze (Trained from Scratch and from MAML-TRPO Policy)}
\begin{itemize}
\item Learning rate (\texttt{lr}): 0.001
\item Number of iterations (\texttt{num\_iterations}): 100
\item Batch size (\texttt{batch\_size}): 20
\item Discount factor (\( \gamma \)): 0.99
\item Soft update coefficient (\( \tau \)): 1.0
\item Random seeds (\texttt{seeds}): 2, 3, 4
\item Maximum KL divergence (\texttt{max\_kl}): 0.01
\item Backtracking factor (\texttt{backtrack\_factor}): 0.5
\item Line search maximum steps (\texttt{ls\_max\_steps}): 15
\end{itemize}
\subsection*{Policies for TRPO Training}
Different policies were used based on the action space of each environment:
\subsubsection*{MiniGrid (Discrete Action Space)}
For MiniGrid, we used a TRPO policy with a critic network based on a categorical policy. This network employs two hidden layers, each with 100 units, and uses ReLU activation. The policy outputs a categorical distribution over actions. The key hyperparameters are:
\begin{itemize}
\item Input size: Dimension of the state space.
\item Output size: Number of discrete actions.
\item Hidden layers: [100, 100]
\item Activation function: ReLU
\end{itemize}
\subsubsection*{PandaGym and PointMaze (Continuous Action Space)}
For PandaGym and PointMaze, we used a TRPO policy with a critic network based on a diagonal normal policy. This network includes two hidden layers, each with 100 units, and supports either ReLU or Tanh activation functions. It models actions as a normal distribution with learned mean and variance. The key hyperparameters are:
\begin{itemize}
\item Input size: Dimension of the state space.
\item Output size: Dimension of the continuous action space.
\item Hidden layers: [100, 100]
\item Activation function: ReLU
\item Initial standard deviation (\( \sigma \)): \( \log(1) \)
\end{itemize}
\subsubsection*{Value Network for All TRPO Policies}
The value network for all TRPO policies is a linear state-value function that minimizes a least-squares loss. It uses features derived from the state, including polynomial terms and additional regularization to improve stability. The key hyperparameters are:
\begin{itemize}
\item Input size: Dimension of the state space.
\item Regularization coefficient (\texttt{reg}): \( 10^{-2} \)
\item Features: Polynomial transformations of the input state.
\end{itemize}
\section{Licenses and External Packages}
\label{sec:licenses}
Our experimental framework makes use of several open-source environments and libraries. Below we provide license details, package versions, and citations for reproducibility:
\paragraph{MiniGrid}
We used version 2.5.0 of the \texttt{minigrid} package \citep{chevalier2024minigrid}, which is licensed under the \textbf{Apache License 2.0}. This license allows for free use, modification, and distribution, provided compliance with its terms.
\paragraph{PointMaze}
We used version 1.3.1 of the \texttt{gymnasium-robotics} package \citep{gymnasium_robotics2023github}, which provides the PointMaze environment. It is licensed under the \textbf{MIT License}, permitting unrestricted reuse in source and binary forms with proper attribution.
\paragraph{Panda-Gym}
The Panda-Gym environment \citep{gallouedec2021panda} was used via version 3.0.7 of the \texttt{panda-gym} package. This package is distributed under the \textbf{MIT License}, allowing flexible use and distribution.
\paragraph{Learn2Learn}
Meta-RL experiments (Meta-AURA) were implemented using version 0.2.1 of the \texttt{learn2learn} Python library \citep{Arnold2020-ss}. The library is released under the \textbf{MIT License}.
\paragraph{DRACO, GR-as-RL, and R\&G Implementations}
The DRACO \citep{nageris2024goal} and GRAQL \citep{amado2022goal} baselines, as well as the R\&G planner-based GR algorithm \citep{ramirez2009plan}, were either reimplemented following the original papers or executed using original codebases when granted by their respective authors. Where applicable, we contacted the authors and obtained permission to use their implementations to ensure experimental accuracy and comparability.
All used tools and environments are either licensed for academic use or reproduced from cited research. These resources enable the reproducibility of our GDGR framework and its associated baselines.
\end{document}