LNS2+RL: Combining Multi-Agent Reinforcement Learning with Large Neighborhood Search in Multi-Agent Path Finding

Extracted source text · arXiv 2405.17794

\documentclass{article}

\usepackage{multirow}

\usepackage[preprint]{neurips_2024}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}    
\usepackage{hyperref}       
\usepackage{url}           
\usepackage{booktabs}      
\usepackage{amsfonts}      
\usepackage{nicefrac}      
\usepackage{microtype}     
\usepackage{xcolor}        
\usepackage{graphicx}
\usepackage{caption} 
\usepackage{subcaption}
\usepackage{balance} 
\usepackage{float}
\usepackage{algorithmic}
\usepackage{algorithm}
\usepackage{rotating}
\usepackage{wrapfig}
\usepackage{caption} 
\usepackage{amsmath}

\captionsetup{font=small}

\title{LNS2+RL: Combining Multi-agent Reinforcement Learning with Large Neighborhood Search in Multi-agent Path Finding}


\author{
  Yutong Wang\\
  National University of Singapore\\
  Singapore, Republic of Singapore\\
  \texttt{e0576114@u.nus.edu}
\And
  Tanishq Duhan\\
  National University of Singapore\\
  Singapore, Republic of Singapore\\
  \texttt{e1280621@u.nus.edu}
\And
 Jiaoyang Li\\
 Carnegie Mellon University\\
 Pittsburgh, USA\\
 \texttt{jiaoyangli@cmu.edu}
\And
  Guillaume Sartoretti\\
  National University of Singapore\\
  Singapore, Republic of Singapore\\
  \texttt{guillaume.sartoretti@nus.edu.sg}\\
  }

\begin{document}


\maketitle

\begin{abstract} 
Multi-Agent Path Finding (MAPF) is a critical component of logistics and warehouse management, which focuses on planning collision-free paths for a team of robots in a known environment.
Recent work introduced a novel MAPF approach, LNS2, which proposed to repair a quickly-obtainable set of infeasible paths via iterative re-planning, by relying on a fast, yet lower-quality, priority-based planner.
At the same time, there has been a recent push for Multi-Agent Reinforcement Learning (MARL) based MAPF algorithms, which let agents learn decentralized policies that exhibit improved cooperation over such priority planning, although inevitably remaining slower.
In this paper, we introduce a new MAPF algorithm, LNS2+RL, which combines the distinct yet complementary characteristics of LNS2 and MARL to effectively balance their individual limitations and get the best from both worlds.
During early iterations, LNS2+RL relies on MARL for low-level re-planning, which we show eliminates collisions much more than a priority-based planner.
There, our MARL-based planner allows agents to reason about past and future/predicted information to gradually learn cooperative decision-making through a finely designed curriculum learning.
At later stages of planning, LNS2+RL adaptively switches to priority-based planning to quickly resolve the remaining collisions, naturally trading-off solution quality and computational efficiency.
Our comprehensive experiments on challenging tasks across various team sizes, world sizes, and map structures consistently demonstrate the superior performance of LNS2+RL compared to many MAPF algorithms, including LNS2, LaCAM, and EECBS.
In maps with complex structures, the advantages of LNS2+RL are particularly pronounced, with LNS2+RL achieving a success rate of over 50\% in nearly half of the tested tasks, while that of LaCAM and EECBS falls to 0\%.
We finally experimentally validate our algorithm in a hybrid simulation of a warehouse mockup involving a team of 100 (real-world and simulated) robots.
\end{abstract}


\section{Introduction}

Multi-Agent Path Finding (MAPF) focuses on the problem of generating a set of shortest collision-free paths for a team of agents from their initial positions to designated goals in a shared, known environment.
It is NP-hard to solve optimally and is a core problem in many applications that may involve thousands of agents, such as warehouse automation and traffic management, where balancing solution quality and speed is crucial.
Recently, \cite{li2022mapf} introduced LNS2, a large neighborhood search-based MAPF algorithm that iteratively reuse/fix currently infeasible paths via local re-planning, differing from mainstream MAPF algorithms that rely on more expensive search/backtracking.
LNS2 exhibits greater scalability than systematic search planners~\cite{sharon2015conflict} and higher solution quality than priority-based algorithms~\cite{erdmann1987multiple}.
LNS2 first quickly generates an initial solution that may contain collisions, and then iteratively replans for carefully selected subsets of agents, using a fast, priority-based low-level planner (Safe Interval Path Planning with Soft Constraints (SIPPS)~\cite{li2022mapf} for single agent pathfinding), to finally yield a collision-free solution.
However, in high-agent-density tasks, the paths generated by this priority-based planner still often contain a large number of collisions, inevitably increasing the cumulative time needed to address the overall MAPF problem.
In parallel to these developments, recent years have seen the development of many (slower) MAPF planners based on Multi-Agent Reinforcement Learning (MARL), which exhibit enhanced cooperation compared to priority-based planners.
However, MARL-based planners inevitably exhibit performance degradation when deviating from their training setup (usually only 8 agents), which limits their performance on tasks containing a high density of agents.
Moreover, the one-shot Success Rate (SR) of MARL-based planners often drops sharply due to a few agents failing to reach their goals within the time(step) limit, thereby wasting high-quality paths for the majority that succeeded.

In this paper, we propose to combine large neighborhood search and MARL, and introduce a new algorithm called LNS2+RL.
Our approach aims to balance the drawbacks of both LNS2 and MARL-based planners, showcasing enhanced performance across diverse scenarios.
The overall framework of LNS2+RL follows that of LNS2, iteratively updating the overall solution with re-planned paths.
However, it replaces the priority-based planner with a MARL-based planner during the challenging early stages of iterative re-planning to further improve solution quality (i.e., reduce collisions).
Our MARL-based planner allows agents to access and reason about historical and predictive information, progressively mastering cooperative decision-making through a carefully crafted curriculum learning strategy.
There, we rely on SIPPS to fix the paths planned by agents unable to reach their goals within a limited number of timesteps, thus ensuring the completeness of our MARL-based planner and maximizing the utilization of existing high-quality paths.
LNS2+RL then adaptively switches to the original priority-based planner (according to the relative solution quality of the two algorithms in the previous re-planning step), to swiftly clean up remaining collisions, thus naturally trading-off solution quality and re-planning speed.
Moreover, since the number of agents involved in each re-planning task (i.e., neighborhood size) is fixed to 8, LNS2+RL can naturally scale the overall planning to thousands of agents while keeping encountered tasks close to MARL's training distribution.

\begin{wrapfigure}{h}{0.4\textwidth}
    \vspace{-0.4cm}
    \centering
    \includegraphics[width=\linewidth]{exp__90+10__Agent+Goal.jpg}
    \caption{Real portion of the warehouse with 10 robots (left), and complete rviz visualization with 100 robots (right). Real robots are circled in different colors (both in the real and simulated views), and crosses of the same colors represent their goal positions.}
    \label{fig:exp_start}
    \vspace{-0.5cm}
\end{wrapfigure}

We tested LNS2+RL on an extensive set of high-difficulty tasks with various team sizes, world sizes, and map structures.
Our results indicate LNS2+RL consistently outperforms LNS2 in SR across nearly all tasks, maintaining a comparable Sum of Costs (SoC), and notably reducing the number of remaining Collision Pairs (CP) in the overall solution.
Additionally, LNS2+RL surpasses EECBS~\cite{li2021eecbs} in SR across all tasks.
Compared to LaCAM~\cite{okumura2023lacam}, LNS2+RL exhibits a substantial advantage in SR on highly structured maps (achieving 100\% SR compared to LaCAM's 0\%), while also achieving significantly lower SoC on less structured maps. 
Finally, as shown in Figure~\ref{fig:exp_start}, we experimentally validated our planner in a hybrid simulation of a warehouse environment under real-world conditions, where we let a team of 100 agents (10 real-world, and 90 simulated ones) track the paths obtained by LNS2+RL, using an online execution framework~\cite{honig2019persistent}.


\section{Prior work}

MAPF algorithms developed in recent decades fall into three main categories based on solution optimality: optimal, bounded-suboptimal and unbounded-suboptimal. 
Common optimal algorithms include Conflict-Based Search (CBS)~\cite{sharon2015conflict}, joint-state A*, and their variants with additional speed-up techniques such as ICBS~\cite{boyarski2015icbs}, M*~\cite{wagner2011m}, EPEA*~\cite{goldenberg2014enhanced}, among others.
Bounded-suboptimal algorithms trade off solution quality and runtime by finding a solution whose cost is at most $w$ times the optimal cost, where $w$ is a user-specified suboptimality factor.
Such MAPF algorithms are usually variants of optimal MAPF algorithms, including EECBS~\cite{li2021eecbs}, weighted EPEA*~\cite{barer2014suboptimal}, and inflated M*~\cite{wagner2015subdimensional}, etc.
Both optimal and bounded-suboptimal algorithms are able to ensure high solution quality but face scalability challenges when handling thousands of agents, often encountering timeouts or memory overflows.
Conversely, unbounded-suboptimal algorithms (such as priority~\cite{erdmann1987multiple} and rule-based planning~\cite{luna2011push}) prioritize speed over optimality, scaling to thousands of agents easily, but may yield substantially suboptimal solutions in complex tasks and often do not guarantee completeness. More recently, learning-based approaches have begun to emerge. 
One class of approaches, exemplified by PRIMAL~\cite{sartoretti2019primal}, directly learns a MAPF planner using MARL~\cite{ma2021distributed, wang2020mobile}, while another combines conventional algorithms with learning, exemplified by switching between two types of algorithms based on certain conditions~\cite{skrynnik2023switch} or by enhancing conventional algorithms' components with learned strategies~\cite{huang2022anytime}.


\section{Background}

\subsection{Multi-agent path finding}
In this paper, we consider a common formulation of the MAPF problem, defined within a 2d 4-edge grid $G = (V, E)$, encompassing a set of $m$ agents $A = \{a_1, \ldots, a_m\}$ and a finite set of static obstacles. 
Each agent $a_i$ initiates from a distinct starting position $s_i \in V$, with the objective of reaching its designated goal $g_i \in V$.  
At each discretized timestep, an agent can either move to an adjacent vertex or wait at its current vertex.
The \emph{path} $p_i$ for each agent is a sequence of adjacent or identical vertices $(v_1^{i}, \ldots, v_k^{i})$ satisfying $v_1^{i} = s_i$, $v_k^{i} = g_i$.
We assume that the agents wait at their goal vertices until all agents have reached their goals.
A \emph{solution} of MAPF $P=\left\{p_i \mid a_i \in A\right\}$ is a path set for all agents, ensuring all agents reach their goals at a certain timestep without any vertex (agents occupy the same vertex) or swapping collisions (agents traverse the same edge simultaneously).
This paper evaluates solution quality using a \emph{sum of costs} metric, defined as the sum of timesteps required by each agent to reach its goal $\sum_{1 \leq i \leq m}\left|p_i\right|$.

\subsection{LNS2}

LNS2 is an unbounded-suboptimal and memory-efficient MAPF algorithm based on large neighborhood search.
It begins by planning a set of paths that may contain collisions, iteratively selecting subsets for re-planning through an Adaptive Large Neighborhood Selection (ALNS) method based on previous experience.
A replanning solver then computes new paths for these selected agents and evaluates whether to adopt the new paths depending on whether they reduce or maintain the number of CP within the overall solution.
This replanning process continues iteratively until the overall path set is collision-free or the time limit is exceeded. 

The low-level, re-planning solver employed in LNS2 is a variant of priority planning (termed PP+SIPPS), which utilizes SIPPS algorithm to plan individual paths for agents in a random priority order.
SIPPS aims to find a path for an agent to avoid collisions with hard obstacles (static obstacles) while minimizing collisions with soft obstacles (pre-existing paths).
It runs five times faster than Space-Time A*, ensuring completeness and returning the shortest path with zero soft collisions if one exists. 
However, in dense-obstacle scenarios, SIPPS may produce a path with significantly more soft collisions than the minimum due to its heuristic function ignoring soft collisions that occur when the agent waits within a safe interval that contains soft obstacles.
Despite the fast speed of SIPPS, this reduced solution quality necessitates a higher number of iterations to fully solve high-difficulty MAPF instances, resulting in substantial cumulative time expenditure.
Therefore, there is an urgent need to develop a new re-planning solver that can generate high-quality solutions (fewer collisions) while potentially sacrificing speed, to find a better balance between quality and time throughout the overall planning.


\section{Re-planning tasks as a MARL problem}

\subsection{Path finding with mixed dynamic obstacles}

The re-planning task can be defined as the Path Finding with Mixed Dynamic Obstacles (PMDO) problem. 
In the PMDO problem, two finite sets of obstacles are considered. 
The first category, hard obstacles, aligns with the immovable obstacles in conventional MAPF. 
The second category, soft obstacles, refers to other unselected agents and is characterized by the ability to move to any vertex not occupied by hard obstacles and remain there for an arbitrary duration.
Notably, the complete trajectory of each soft obstacle is fully disclosed and remains constant throughout the re-planning task.
Any occupied vertex at a given timestep $t$ can contain either a single hard obstacle or multiple soft obstacles/agents.
The objective of the re-planning task is to find the shortest path for each selected agent from their start vertex to the goal vertex while avoiding collisions with hard obstacles and minimizing collisions with both soft obstacles and other agents.

\subsection{RL environment setup}
\label{sec:env_setting}
Aligned with previously outlined MAPF and PMDO problems, we consider a discrete 2D 4-connected gridworld environment where the size of each agent, goal, and obstacle is one grid cell. 
Each agent can take a discrete action at every timestep, which includes moving one unit towards any of the four cardinal directions or staying still. 
The agents' actions are executed synchronously, with collision checks conducted subsequently to the execution of all actions. 
Actions resulting in collisions with hard obstacles or exceeding world boundaries are classified as invalid.
Only valid actions are sampled during training and evaluation, and an additional supervised loss is added to the overall MARL loss.

We initially establish a MAPF task comprising $m$ agents, assigning each a unique start and a corresponding goal position. 
Using PP+SIPPS to compute the initial path set $P$ for the $m$ agents. 
Subsequently, a subset of agents is selected, and during the training phase of the MARL model, we keep the subset size $n$ (i.e., the neighborhood size) to 8.
The PMDO re-planning task is then formulated by designating the chosen subset as the controlled agents of the MARL model, while the remaining agents act as soft obstacles.
A training episode terminates either when all agents are on goal at the end of a timestep (the timestep is higher than the maximum length of the soft obstacle trajectories) or when the episode reaches the pre-defined timestep limit.

\subsection{Observation and reward}

Aligned with previous works~\cite{sartoretti2019primal}, the environment is partially observable, restricting each agent to only access information within its self-centered field of view (FoV).
This assumption enables the trained policy to scale to arbitrary world sizes while fixing the input dimensions of the neural network.
The FoV contains global spatial and temporal information derived from the complete map structure and past/predicted trajectories of all soft obstacles and other agents.

Each agent's current observation at timestep $t$ is divided into two parts.
Here, we take agent $i$ as an example.
The first part of the observation consists of thirty-one 2D matrices.
Following methodologies from previous research~\cite{wang2023scrimp}, where four matrices represent the actual/projected positions of hard obstacles, other agents, observable agents' goals, and agent $i$'s own goal. 
Additionally, there are four binary matrices representing the shortest paths of agent $i$, calculated by the Dijkstra algorithm without considering other agents and soft obstacles.
Another map that helps guide agent $i$ closer to its goal is the \textit{SIPPS map}, which represents all vertices traversed by agent $i$'s SIPPS path $\hat{p}_i$ (computed only at the beginning of the re-planning task) during $t-15$ to $t+15$.
To motivate MARL agents to refer to the SIPPS paths while allowing necessary detours to avoid collisions, we introduce an off-route penalty $ro$, that is 
\vspace{-0.3cm}
\begin{equation}
\small
\label{eq:off_reward}
   ro_t^{i}=-\alpha \min \left\|v_t^i-v\right\|_2 , {v \in \left\{\left(\hat{p}_i\right)_j \mid t-15 \leq j < t+15\right\}},
\end{equation}
where $\alpha$ is a predefined hyperparameter,  $v_t^{i}$ is the position of agent $i$ at timestep $t$, $v$ represents all vertices contained in the SIPPS map of agent $i$.

To avoid collisions with hard and soft obstacles, we further introduce an \textit{occupancy duration map}, a \textit{blank duration map}, \textit{future trajectory maps}, and \textit{predicted trajectory maps}.
The occupancy duration map denotes the ratio of time a vertex will be occupied by obstacles to the predefined episode length, while the blank duration map reflects the ratio of time a vertex will remain obstacle-free.
The future trajectory maps, comprising nine matrices, represent the positions of soft obstacles from $t$ to $t+8$. 
The future trajectories of other agents are crucial to enhancing cooperation, but these trajectories are unknown in advance. 
Hence, we employ a straightforward yet efficient method for forecasting them.
To predict the future trajectory of agent $j$, we first compute the spatial Euclidean distance between $v_t^{j}$ and all vertices on $\hat{p}_j$, and then determine the temporal distance as the absolute difference between $t$ and the timesteps of vertices on $\hat{p}_j$.
The predicted position of agent $j$ at $t+1$ is then derived from a weighted sum of these spatial and temporal distances, with coefficients of 0.9 for spatial and 0.1 for temporal distances.
If the vertex with the minimal sum distance is reachable in one action from $v_t^{j}$, it is considered the predicted position for $t+1$; otherwise, agent $j$ is assumed to remain stationary, with its actual position $v_t^{j}$ as the predicted position for $t+1$. 
The subsequent four vertices on the SIPPS path of the vertex with the minimal sum distance are designated as predicted positions from $t+2$ to $t+5$.

In path planning tasks involving a large number of agents, congestion often arises in bottleneck areas of the map, where all agents attempt to traverse the same set of vertices, leading to under-utilization of other vertices and significantly increased collision risks.
To alleviate congestion, we introduce the \textit{vertex utilization map} and the \textit{edge utilization maps}, which visualize congestion and assist agents in identifying and avoiding high-density areas, promoting more evenly dispersed paths.
Specifically, denote $U (v, t-\beta_l, t+\beta_h)$ and $U ((v_1, v_2), t-\beta_l, t+\beta_h)$ as the usage of vertex $v$ and edge $(v_1, v_2)$ during the time period from $t-\beta_l$ to $t+\beta_h$ , respectively.
They are the sum of the number of times the soft obstacles traverse through the vertex/edge from $t-\beta_l$ to $t+\beta_h$, and the number of times agents traverse through the vertex/edge from $t-\beta_l$ to $t$. 
In our experiments, we set $\beta_l$ to 2 and $\beta_h$ to 15.
The normalized value $\delta \frac{U (v, t-\beta_l, t+\beta_h)}{m}$ and $\delta \frac{U ((v_1, v_2), t-\beta_l, t+\beta_h)}{m}$ are then used as the value of each grid in the vertex utilization map and four edge utilization maps ($\delta= 10 $).
Moreover, we further encourage agents to avoid these congested areas by giving them a small penalty $rc$ calculated by  
\vspace{-0.1cm}
\begin{equation}
\small
\label{eq:uti_reward}
   rc_t^{i} = \delta (\delta_v \frac{U (v, t-\beta_l, t+\beta_h)}{m} +\delta_e \frac{U ((v_1, v_2), t-\beta_l, t+\beta_h)}{m}) .
\end{equation}
Here $\delta_v = 0.225$, $\delta_e = 0.075$ are used to balance between vertex and edge information.

The second part of the observation is an eight-element vector consisting of the normalized distance between $v_t^i$ and $g_i$ along the x-axis and y-axis, as well as the total Euclidean distance, the ratio of the number of CP in path planned by the MARL-planner compared to these in $\hat{p_i}$, the ratio of $t$ to the predefined episode length, the ratio of $t$ to the maximum length of $\hat{P}$, the ratio of agents who have reached goal, and agent $i$'s action at $t-1$.

Our reward structure is shown in Table \ref{table:reward}.
To expedite task completion, agents currently off-goal are penalized at each step, with its magnitude scaling up according to the task's difficulty within the curriculum learning framework (refer Section \ref{sec: training} for details).
Agents receive an extra penalty if the current timestep exceeds the maximum length of $\hat{P}$.
Additionally, to foster exploration and learn effective policies, agents are penalized for revisiting their previous timestep's location.
Furthermore, we employ reward shaping based on the distance from the goal to guide learning while preserving the invariance of the optimal policy \cite{ng1999policy}.

\begin{table}[]
\centering
\caption{Reward structure}
\label{table:reward}
\vspace{-0.05cm}
\scalebox{0.7}{
\begin{tabular}{c|c|c|c|c|c|c|c}
\hline
Action & \begin{tabular}[c]{@{}c@{}}Move/Stay \\ (not on goal)\end{tabular} & \begin{tabular}[c]{@{}c@{}}Stay \\ (on goal)\end{tabular} & \begin{tabular}[c]{@{}c@{}}Exceed the \\maximum length of $\hat{P}$\end{tabular} & Return & \begin{tabular}[c]{@{}c@{}}Collision \\ with obstacles\end{tabular} & \begin{tabular}[c]{@{}c@{}}Avoid\\ congestion\end{tabular} & Off-route \\ \hline
Reward & {[}-0.4,-0.5,-0.6{]} & 0.0 & -0.2 & -0.4 & -1.5 & detailed in equation \ref{eq:uti_reward}& detailed in equation \ref{eq:off_reward} \\ \hline
\end{tabular}}
\vspace{-0.6cm}
\end{table}

\subsection{Network structure}

\begin{wrapfigure}{r}{0.7\textwidth}
  \vspace{-0.45cm}
  \centering
  \setlength{\belowcaptionskip}{-0.3cm}  \includegraphics[width=1\linewidth]{net.pdf}
  \vspace{-0.5cm}
  \caption{Network structure.
  Green, purple, and red modules represent the memory encoder, observation encoder, and output heads, respectively.
$h_{t-1}^i$ represents the hidden state output by the LSTM unit at the previous time step.}
  \label{fig:net}
\vspace{-0.3cm}
\end{wrapfigure}
Figure~\ref{fig:net} illustrates the network architecture of our MARL model, encompassing three primary components: memory encoder, observation encoder, and output heads.
Given our assumption of a homogeneous team, we rely on weight sharing across all agents, enabling the model to scale effectively with the number of agents during evaluation.

Understanding the underlying dynamics of the environment is crucial in PMDO tasks.
Therefore, in addition to utilizing occupied duration map, blank duration map, future trajectory maps, and predicted trajectory maps to capture future temporal information, we incorporate observations from the past three time steps as input to the neural network to provide a more comprehensive understanding of how the environment changes over time. 
The aggregated observation first passes through a Convolutional Long Short-Term Memory (ConvLSTM)~\cite{shi2015convolutional} unit capable of capturing spatial and temporal relationships and then concatenates (Cat) its last hidden state with the current observation $o_t^i$.
The concatenated data then goes through an observation encoder with two ResNet blocks~\cite{he2016deep}, followed by an average pooling and Layer Normalization (LN) to produce a one-dimensional feature vector.
In parallel, the eight-element vector $v_t^i$ is processed through a Fully Connected (FC) layer, accompanied by two LN. 
The processed inputs are then concatenated and passed through two FC layers equipped with a residual shortcut, followed by a LN, and ultimately fed into a Long Short-Term Memory (LSTM) cell. 
The LSTM's output is further processed by a FC layer, producing a vector $\dot{o}_t^i$ for imitation learning. 
Subsequently, $\dot{o}_t^i$ passes through a softmax activation to derive the agent's policy and a sigmoid activation to yield a vector $\ddot{o}_t^i$ for supervised invalid action loss.
Another FC layer also utilizes the LSTM's output as input to generate the predicted state value.

\subsection{Training}
\label{sec: training}

Our MARL model is trained by Proximal Policy Optimization (PPO)~\cite{schulman2017proximal}.
The total loss of the MARL model is $L_{final}=\omega_v \cdot L_v+\omega_{\pi} \cdot L_\pi +\omega_{invalid} \cdot L_{invalid} - \omega_e \cdot L_{e} $, where $L_v$ and $L_\pi$ are the standard critic and actor loss in PPO.
$L_{e}$ is the entropy of the agent's policy, suppressing its decline shown to encourage exploration and inhibit premature convergence.
$L_{invalid}$ is a binary cross-entropy loss (supervised invalid action loss) designed to reduce the logarithmic likelihood of selecting an invalid action. 
$\omega_v$, $\omega_{\pi}$, $\omega_{invalid}$, and $\omega_e$ are all manually set hyperparameters.

The training of the MARL model is divided into two stages.
As described in Section~\ref{sec:env_setting}, both stages commence by generating a high-level MAPF task and obtaining the initial path set $P$ through PP+SIPPS.
Subsequently, a series of PMDO re-planning tasks are generated in different manners and used to train the MARL model.
To enhance the agents' generalization ability, we expose them to diverse scenarios during training by uniformly sampling the world size, hard obstacle rate, and the total number of agents for the MAPF task from a variety of values.

In the first training stage, when constructing each PMDO task, the MARL-controlled agents are chosen randomly, bypassing ALNS.
After an MARL episode ends, the path set $P$ is updated with the SIPPS paths $\hat{P}$ calculated at the beginning of the episode, regardless of whether these new paths reduce CP in $P$.
PMDO re-planning tasks are continually generated and only switch to the next new high-level MAPF task when the the maximum iteration limit is reached.
Curriculum learning~\cite{bengio2009curriculum} is adopted during the first stage of training, allowing agents to rapidly acquire fundamental skills from simpler tasks before progressively facing more challenging scenarios.
Specifically, we increase the task difficulty by increasing the hard obstacles rate and the total number of agents in the high-level MAPF tasks and advancing to greater difficulties once the predetermined volume of training data is accumulated.
As task difficulty increases, the advantage of SIPPS weakens, and the agents' preference for safely wandering in a certain area increases.
Therefore, we gradually lower the value of $\alpha$ in the off-route penalty to reduce references to SIPPS paths and gradually increase the action cost to incentivize agents to move toward their goal. 
Additionally, we employ imitation learning (behavior cloning based on the centralized planner ODrM*) at the beginning of each task difficulty in the first stage to explore high-quality regions of the agents' state-action space.

Upon reaching the manually predefined number of training steps, the training transitions to the second stage. 
This stage aims to acclimate the MARL policy to scenarios it will face when inserted into the LNS2+RL algorithm.
Therefore, both the selection of MARL-controlled agents and the method for updating path set $P$ are exactly the same as the LNS2+RL algorithm's execution (refer to Section \ref{sec: imp} for specifics). 
A new high-level MAPF task is generated either when $P$ achieves a collision-free state or upon hitting the maximum iteration limit.

\vspace{-0.1cm}
\section{LNS2+RL}
\label{sec: imp}

\newcommand{\selectDestroyHeuristic}{\texttt{selectDestroyHeuristic}}
\newcommand{\selectAgentSet}{\texttt{selectAgentSet}}
\newcommand{\runMARLPlaner}{\texttt{runMARLPlaner}}
\newcommand{\runPriorityPlaner}{\texttt{runPriorityPlaner}}
\newcommand{\runlnitialSolver}{\texttt{runlnitialSolver}}

\setlength{\textfloatsep}{0.3cm}
\begin{algorithm}[tb]
\caption{LNS2+RL}
\label{alg:LNS2+RL}
\scalebox{0.65}{
\begin{minipage}{1\linewidth}
\begin{algorithmic}
\setlength{\itemsep}{0pt}
\setlength{\parsep}{0pt}
   \STATE {\bfseries Input:} A MAPF instance $I$
   \STATE $P =\left\{p_i \mid a_i \in A\right\} \leftarrow$ \runlnitialSolver $(I)$ 
   \STATE $\forall w_h \in W, w_h=1$; build collision graph $G_c= (V_c, E_c)$; $Q = \emptyset$ and $|Q| \leq \mu$
   \STATE $t_m = \max\{|p_i| : p_i \in P\} $, $t_l = d_l \cdot t_m$, $t_h = d_h \cdot t_m$
   \WHILE{runtime limit not exceeded and $CP(P)> 0$}
        \STATE $h \leftarrow$ \selectDestroyHeuristic $(W)$
        \STATE $A_s, P^{-} \leftarrow$  \selectAgentSet$ (h, I, P, G_c)$
        \IF {$|Q| < \mu$ or $\bar{Q} > \rho$}
            \STATE $P^{rl}, \hat{P} \leftarrow$ \runMARLPlaner $\left(I, P \backslash P^{-}, A_s, t_l \right)$; $A_r \leftarrow \{a_i \in A_s : p_{i, \text{end}}^{rl}  \neq g_i\}$
            \STATE $P^{rl} \leftarrow$ \runPriorityPlaner $\left(I, P \backslash P^{-}, P^{rl}, A_r, t_l\right)$; $A_h \leftarrow \{a_i \in A_s : |p_i^{rl}| \geq t_h\}$
            \STATE $P^{rl} \leftarrow$ \runPriorityPlaner $\left(I, P \backslash P^{-}, P^{rl}, A_h, 0\right)$
            \STATE Update $Q$ accoridng to if $CP(P^{rl})$ $\geq$ $CP(\hat{P})$
            \IF{$ CP(\left(P \backslash P^{-}\right) \cup P^{rl})<= CP(P)$}
                \STATE $P \leftarrow\left(P \backslash P^{-}\right) \cup P^{rl}$
                 \STATE Update the collision graph $G_c$
                 \STATE Update $w_h$
                 \STATE \textbf{continue}
            \ENDIF
        \ELSE
            \STATE $\hat{P} \leftarrow$ \runPriorityPlaner $\left(I, P \backslash P^{-}, \emptyset, A_s, 0\right)$
        \ENDIF    
    \IF{$ CP(\left(P \backslash P^{-}\right) \cup \hat{P})<= CP(P)$}
        \STATE $P \leftarrow\left(P \backslash P^{-}\right) \cup \hat{P}$
        \STATE Update the collision graph $G_c$
        \ENDIF  
    \STATE Update $w_h$
   \ENDWHILE
\STATE \textbf{return} $P$
\end{algorithmic}
\end{minipage}}
\end{algorithm}
\setlength{\textfloatsep}{20pt}

The strengths and weaknesses of PP+SIPPS and the MARL-based planner are distinct yet complementary. 
PP+SIPPS is fast and effective for simple tasks but tends to generate paths with more soft collisions in complex tasks.
Conversely, our MARL-based planner, although slower, excels at planning paths with fewer collisions in difficult tasks. 
Therefore, our LNS2+RL algorithm retains the overarching framework of LNS2 but uses the MARL-based planner in the initial, more challenging stages of iterative planning for better solution quality (fewer collisions). 
It then adaptively switches to PP+SIPPS to quickly clean up the remaining collisions, thus balancing solution quality with the speed of the re-planning task solvers. 
Moreover, the key limitation in the one-shot SR of MARL-based planners is that while most agents have found high-quality paths, a few fail to reach their goals within the constrained episode length. 
To address this, we employ PP+SIPPS to supplement paths for these few agents, ensuring the high quality and completeness of the final solution.

The pseudocode for the LNS2+RL algorithm is presented in Algorithm~\ref{alg:LNS2+RL}.
Specifically, for solving a MAPF instance $I$, LNS2+RL starts by invoking the initial solver (PP+SIPPS) to generate a path set $P$ containing collisions (lines 2), initializing the weights $W$ for the three neighborhood selection methods, and constructing the collision graph $G_c$ (lines 3).
Furthermore, LNS2+RL introduces a queue $Q$ with a maximum capacity of $\mu$ to store the relative solution quality from PP+SIPPS and the MARL-planner (line 3).
Subsequently, LNS2+RL counts the maximum length of individual paths in the initial solution ($t_m$), sets the timestep threshold $t_l$ for pausing the MARL-planner, along with the maximum length $t_h$ of each hybrid path, where the constant value $d_h > d_l$ (line 4). 
If the termination criteria are not met, LNS2+RL iteratively selects a subset of agents $A_s \subseteq A$ using the ALNS, and removes their paths $P^{-} = \left\{p_i \in P: a_i \in A_s\right\}$ from $P$, treating the remaining part of the path set $P \backslash P^{-}$ as fixed to construct a PMDO re-planning task (lines 6-7).
The neighborhood size (the size of  $A_s$) is fixed at 8 (equivalent to the number of agents used in MARL training) or 16.
If the size of $Q$ is below than $\mu$, or its average exceeds another threshold $\rho$, LNS2+RL infers that the solution quality of the MARL-planner significantly exceeds that of PP+SIPPS at this stage (line 8).
Therefore, LNS2+RL chooses the trained MARL-planner as the re-planning solver. 
The function in line 9 uses the MARL-planner to predict actions until timestep $t_l$ (or until the re-planning task is fully resolved, i.e., all agents have reached their goal without future soft collisions).
It yields two path sets: $\hat{P}$, derived by PP+SIPPS at the onset of the re-planning task, acting as the reference paths for the MARL policy, and path set $P^{rl}$, computed by the MARL policy.
For incomplete paths in $P^{rl}$ where agents haven't reached their goals, we employ PP+SIPPS to plan the remaining paths from $t_l$ and merge these new paths with the existing ones (Line 10).
Subsequently, all path lengths in $P^{rl}$ are reviewed, and those exceeding $t_h$ are directly replaced with paths re-planned by PP+SIPPS from timestep zero (lines 11).
This operation aims to remove non-contributory segments planned by the MARL-planner, thus reducing the SoC of the final solution $P$ and planning time for subsequent iterations.
Next, LNS2+RL updates $Q$ by appending 0 if $CP(P^{rl}) \geq CP(\hat{P})$, and 1 otherwise (lines 12).
Finally, LNS2+RL first checks whether $P^{rl}$ meets the update condition (line 13), namely if the CP of the repaired path set $\left(P \backslash P^{-}\right) \cup P^{rl}$ is no larger than that of the old path set.
If not, LNS2+RL continues to check whether $\hat{P}$ meets the update condition (line 22).
The MARL-planner continues to re-plan until either the switching condition or termination condition is met.
Once the re-planning solver switches to PP+SIPPS, it cannot switch back to the MARL-planner and always addresses remaining collisions using the same procedure as LNS2.


\section{Experiments}
\label{sec:result}

We compare LNS2+RL with vanilla LNS2~\cite{li2022mapf} as well as a representative set of scalable
high-quality MAPF algorithms: the bounded-suboptimal algorithm EECBS~\cite{li2021eecbs} (with a suboptimality factor $w = 5$), the complete but unbounded-suboptimal algorithm LACAM~\cite{okumura2023lacam}, the MARL-based algorithm SCRIMP~\cite{wang2023scrimp}, and the priority-based planner (PP) with random restarts (using SIPP~\cite{phillips2011sipp} to plan paths for single agents)~\cite{cohen2018rapid}.
Here, we only show the results of LNS2, EECBS, and LACAM because the SR of SCRIMP and PP are 0\% in most tasks, and they cannot solve even the simplest tasks in the random-small map.
Both LNS2+RL and LNS2 use PP+SIPPS to find the initial solution and ALNS to generate neighborhoods.
Code for LNS2, EECBS, LACAM was written in C++, and LNS2+RL was written in both C++ and Python and connected using pybind (\textbf{Our full training and testing code is available at \url{https://github.com/marmotlab/LNS2-RL}}).
All experiments were conducted on a server equipped with one Nvidia GeForce RTX 3090 GPU and an AMD Ryzen 9 7950X3D 16-Core Processor.
To obtain results faster, LNS2+RL, LNS2, EECBS used Ray to test multiple MAPF tasks simultaneously.
However, due to high memory demands (we limited the maximum memory to 100GB per task), LACAM was tested using a single process.

\subsection{Results for multiple map types}
\label{sec:map_type}


\begin{figure*}[t]
\vspace{-0.4cm}
\captionsetup[subfigure]{aboveskip=-1pt}
\centering
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{random_10.pdf}
    \caption{Random-small}
    \label{fig:Random-small}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{random_25.pdf}
    \caption{Random-medium}
    \label{fig:Random-medium}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{random_50.pdf}
    \caption{Random-large}
    \label{fig:Random-large}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{empty_25.pdf}
    \caption{Empty}
    \label{fig:Empty}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{maze_25.pdf}
    \caption{Maze}
    \label{fig:Maze}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{room_25.pdf}
    \caption{Room}
    \label{fig:Room}
\end{subfigure} 
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{warehouse_25.pdf}
    \caption{Warehouse}
    \label{fig:Warehouse}
\end{subfigure}
\includegraphics[width=0.45\textwidth]{label.pdf}
\setlength{\belowcaptionskip}{-0.5cm}
\vspace{-0.2cm}
\caption{Result on different map types.
The success rate is the percentage of tasks fully solved within the planning time constraints (i.e., where all agents reached their goal without collisions).
The remaining collision pairs metric represents the number of CP remaining in the overall solution after reaching the time constraint and is only available for LNS2+RL and LNS2.
The confidence interval (shaded area) represents one standard deviation of the remaining CP.
All three random maps have a 17.5\% obstacle density. 
Except for the random-small map with a size of $10 \times 10$ and the random-large map with a size of $50 \times 50$, all other five maps have a size of $25 \times 25$.
For maps with sizes $10 \times 10$, $25 \times 25$, and $50 \times 50$, the time constraints for each task are 100 seconds, 600 seconds, and 5000 seconds, respectively, with test counts of 100, 50, and 20.
The neighborhood size of LNS2+RL is 16 on random-large maps and 8 on all other maps.}
\label{fig:all_map}
\end{figure*}

The evaluation results presented in Figure~\ref{fig:all_map} demonstrate that LNS2+RL consistently achieves equal or higher SR compared to LNS2 across all tasks, except for those on the empty map with 77.5\% agent density. 
It is noteworthy that the improvement of LNS2+RL on CP is more pronounced.
In tasks with 65\% agent density on the random-small map, despite both algorithms achieving the same SR, LNS2+RL remarkably reduces the remaining CP by 26.3\% compared to LNS2.

\begin{wraptable}{H}{0.4\textwidth}
% \vspace{-0.45cm}
\centering
\caption{
Sum of costs on random-small map with 50\%, 55\%, 60\%, 65\% agent density (accounting only for successful tasks).}
\label{table:SoC}
\vspace{-0.2cm}
\scalebox{0.7}{
\begin{tabular}{|c|cccc|}
\hline
Algorithms & \multicolumn{4}{c|}{Sum of Cost} \\ \hline
LNS2+RL & 852.7 & 942.6 & 994.9 & 1052.8 \\
LNS2 & 831.0 & 939.4 & 972.4 & 1064.2 \\
LACAM & 2395.2 & 5288.3 & 5035.1 & 306438.3 \\
EECBS & 825.1 & 912.7 & 929.7 & 1030.0 \\ \hline
\end{tabular}}
\vspace{-0.3cm}
\end{wraptable}

The structural complexity of maps significantly influences algorithm performance. 
Despite being trained on random maps without prior exposure to highly structured environments like mazes, rooms, and warehouses, LNS2+RL demonstrates remarkable improvements in SR on these three types of maps.
Conversely, LACAM excels on less structured maps, achieving a 100\% SR on empty maps, but close to 0\% SR on highly structured maps.
EECBS consistently ranks as the poorest performing algorithm, with SR often remaining at 0\% due to the high difficulty of the selected tasks.

Furthermore, taking the tasks in Table~\ref{table:SoC} as an example, the SoC for LNS2+RL only averages 1.03\% higher than that of LNS2.
Despite LACAM displaying higher SR on the random-small map, its SoC notably lags behind.
Conversely, EECBS consistently achieves the best SoC but at the expense of obtaining the lowest SR. 
Moreover, LNS2+RL and LNS2 offer the added benefit of maintaining consistent memory usage, in contrast to LACAM and EECBS, whose memory usage escalates rapidly over time due to the generation of longer paths or larger search frontiers.

\vspace{-0.2cm}
\subsection{Performance analysis}

\begin{table*}[]
\centering
\caption{Comparing MARL policy against PP+SIPPS on low-level re-planning tasks of MAPF tasks on random-medium map with 65\% agent density.
The difference between collisions and CP is that for $n$ collisions involving the same agents, the number of collisions is counted as $n$, but CP is counted as $1$.
The percentages in parentheses represent the performance ratio of the solution obtained by the MARL policy compared to that of PP+SIPPS.}
\label{table:single}
\vspace{-0.2cm}
\scalebox{0.7}{
\begin{tabular}{|c|cc|cc|cc|cc|}
\hline
\multirow{2}{*}{Iterations} & \multicolumn{2}{c|}{Collisions} & \multicolumn{2}{c|}{Collision Pairs} & \multicolumn{2}{c|}{Sum of Cost} & \multicolumn{2}{c|}{Time} \\ \cline{2-9} 
 & PP+SIPPS & MARL & PP+SIPPS & MARL & PP+SIPPS & MARL & PP+SIPPS & MARL \\ \hline
1 & 406.9 & 106.9(-73.7\%) & 109.9 & 93.9(-14.6\%) & 439.0 & 592.5(+35.0\%) & 0.012 & 0.153(+1175\%) \\
50 & 471.5 & 129.9(-72.5\%) & 107.9 & 106.4(-1.2\%) & 564.1 & 675.3(+19.7\%) & 0.019 & 0.159(+737\%) \\
100 & 488.3 & 130.8(-73.2\%) & 101.5 & 103.8(+2.3\%) & 597.2 & 689.9(+15.5\%) & 0.024 & 0.176(+633\%) \\ \hline
\end{tabular}}
\vspace{-0.7cm}
\end{table*}


\begin{wrapfigure}{r}{0.25\textwidth}
  \vspace{-0.5cm}
  \centering
  \includegraphics[width=1\linewidth]{iter_curve.pdf}
  \vspace{-0.6cm}
  \caption{ 
Number of remaining collision pairs through the iterative re-planning process.
The dashed blue line shows the switch from MARL to PP+SIPPS.}
  \label{fig:curve}
  \vspace{-0.3cm}
\end{wrapfigure}


Table~\ref{table:single} compares the average performance of the MARL policy and PP+SIPPS in replanning tasks across different numbers of iterations.
The MARL policy clearly produces fewer collisions overall, but its advantage in reducing CP is less pronounced, indicating that the MARL policy tends to cause collisions with different agents.
With re-planning task difficulty decreasing as iterations increase, the MARL policy's advantage in reducing CP diminishes. 
Given that the speed of the MARL policy is nearly seven times slower than that of PP+SIPPS, switching to PP+SIPPS becomes a necessary trade-off between time and solution quality.

Figure~\ref{fig:curve} illustrates the trend of the number of remaining CP over 2000 iterations in MAPF tasks on random-medium maps with a 65\% agent density.
As expected, in the early stage, before switching to PP+SIPPS, the number of CP for LNS2+RL decreases at a faster rate.
After switching, the gap between LNS2+RL and LNS2 continues to widen, as the high solution quality achieved by the MARL policy reduces the difficulty of subsequent replanning tasks.

\vspace{-0.2cm}
\subsection{Real world experiment}

As shown in Figure~\ref{fig:exp_start}, we implemented our algorithm on a team of 90 virtual and 10 real robots in a warehouse environment. 
Since during the planning process, our algorithm assumes the position of all agents to be perfectly known at all times, we use ground truth positions for our virtual robots and use external localization (here, the \textit{Optitrack Motion Capture System}) to obtain accurate position information for our real robots.
However, the planned path may not be executed accurately due to disturbances and control inaccuracies.
To eliminate these errors, we implement an \textit{Action Dependency Graph} (ADG), as proposed by \cite{honig2019persistent}.
An ADG introduces a precedence order for the agents occupying a cell to ensure that errors in executing the planned path do not propagate and disrupt the overall execution plan.
From our experiment, we observe that agents can reach their goals quickly without collisions, and errors are eliminated by the ADG, demonstrating the potential of using our method in the real world.
More information about our experiment is included in Appendix \ref{app_real}, and the full video is available in the supplementary material.

\vspace{-0.2cm}
\section{Conclusion}

This paper introduces LNS2+RL, a novel MAPF algorithm that combines MARL with large neighborhood search. It follows the overall framework of LNS2, beginning with a path set that may contain collisions and then iteratively replanning paths for subsets of agents until the path set becomes collision-free. 
In early iterations, LNS2+RL addresses these challenging replanning tasks with a MARL-based planner, which offers higher solution quality but at a slower pace.
As iterations progress, it adaptively switches to a fast, priority-based planner to quickly resolve remaining collisions.
Our experiments on high-difficulty tasks across various world sizes, team sizes, and map structures consistently demonstrate the superior performance of LNS2+RL compared to other SOTA MAPF algorithms.
The practicality of LNS2+RL is validated in a hybrid warehouse mockup simulation involving 10 real-world and 90 simulated robots.
However, the MARL planner we use still relies on paths computed by SIPPS, and its SoC cannot be guaranteed as optimal. 
Future work will explore how to improve this MARL-based planner.

\bibliography{my_bib}
\bibliographystyle{icml2024}


\appendix
\section{Appendix}

\subsection{Additional results}

\subsubsection{Detailed results}
The detailed results of Section~\ref{sec:map_type} can be found in Table~\ref{table:all_data}.
\begin{sidewaystable}[]
\caption{Average performance along MAPF metrics of different algorithms for instances with different world sizes, team sizes, and map structure.
For maps with sizes $10 \times 10$, $25 \times 25$, and $50 \times 50$, the time constraints for each task are 100 seconds, 600 seconds, and 5000 seconds, respectively, with test counts of 100, 50, and 20.
The success rate is the percentage of tasks fully solved within the planning time constraints (i.e., where all agents reached their goal without collisions).
The sum of cost only accounts for successful tasks.
Partial runtime data for LACAM is unavailable because some tasks in the test exceeded the maximum memory limit we set (100GB), leading to premature termination.
The remaining collision pairs metric represents the number of CP remaining in the overall solution after reaching the time constraint and is only available for LNS2+RL and LNS2.
Percentages in parentheses represent the reduced remaining CP ratio of LNS2+RL compared to LNS2.
↑ indicates that “higher is better”, and ↓ “lower is better”. ”-” represents unavailable data.
The results of the best-performing algorithms in each task are highlighted in bold.}
\label{table:all_data}
\centering
\scalebox{0.7}{
\begin{tabular}{|c|cccccccccccccccc|}
\hline
\multirow{2}{*}{Methods} & \multicolumn{16}{c|}{Random-small: 10 * 10 world size with 17.5\% static obstacle rate and 50\%, 55\%, 60\%, 65\% agents density} \\ \cline{2-17} 
 & \multicolumn{4}{c|}{Success Rate ↑} & \multicolumn{4}{c|}{Sum of Cost ↓} & \multicolumn{4}{c|}{Runtime ↓} & \multicolumn{4}{c|}{Remaining Collisions Pairs ↓} \\ \hline
LNS2+RL & \textbf{82\%} & \textbf{70\%} & 59\% & \multicolumn{1}{c|}{27\%} & 852.7 & 942.6 & 994.9 & \multicolumn{1}{c|}{1052.8} & 21.6 & 34.9 & 49.8 & \multicolumn{1}{c|}{79.9} & \textbf{\begin{tabular}[c]{@{}c@{}}0.45\\ (-25.0\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}1.06\\ (-29.8\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}2.70\\ (-27.2\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}7.44\\ (-25.7\%)\end{tabular}} \\
LNS2 & 80\% & 68\% & 50\% & \multicolumn{1}{c|}{27\%} & 831.0 & 939.4 & 972.4 & \multicolumn{1}{c|}{1064.2} & \textbf{20.8} & 33.9 & 54.1 & \multicolumn{1}{c|}{77.0} & 0.60 & 1.51 & 3.71 & 10.01 \\
EECBS & 38\% & 18\% & 3\% & \multicolumn{1}{c|}{1\%} & \textbf{825.1} & \textbf{912.7} & \textbf{929.7} & \multicolumn{1}{c|}{\textbf{1030.0}} & 63.9 & 82.8 & 97.1 & \multicolumn{1}{c|}{99.1} & - & - & - & - \\
LACAM & 75\% & \textbf{70\%} & \textbf{68\%} & \multicolumn{1}{c|}{\textbf{61\%}} & 2395.2 & 5288.3 & 5035.1 & \multicolumn{1}{c|}{306438.3} & 27.0 & \textbf{32.2} & \textbf{34.8} & \multicolumn{1}{c|}{\textbf{42.4}} & - & - & - & - \\ \hline
 & \multicolumn{16}{c|}{Random-medium: 25 * 25 world size with 17.5\% static obstacle rate and 50\%, 55\%, 60\%, 65\% agents density} \\ \hline
LNS2+RL & \textbf{72\%} & \textbf{30\%} & 2\% & \multicolumn{1}{c|}{0\%} & 16726.3 & 16727.7 & 16605.0 & \multicolumn{1}{c|}{-} & \textbf{222.1} & \textbf{486.8} & 593.5 & \multicolumn{1}{c|}{600.0} & \textbf{\begin{tabular}[c]{@{}c@{}}0.80\\ (-20.0\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}3.54\\ (-53.2\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}32.52\\ (-46.5\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}165.38\\ (-34.0\%)\end{tabular}} \\
LNS2 & 70\% & 20\% & 2\% & \multicolumn{1}{c|}{0\%} & \textbf{14157.5} & \textbf{15037.1} & \textbf{16513.0} & \multicolumn{1}{c|}{-} & 256.8 & 509.8 & \textbf{591.7} & \multicolumn{1}{c|}{600.0} & 1.00 & 7.56 & 60.82 & 250.66 \\
EECBS & 0\% & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & - & - & - & \multicolumn{1}{c|}{-} & 600.0 & 600.0 & 600.0 & \multicolumn{1}{c|}{600.0} & - & - & - & - \\
LACAM & 36\% & 26\% & \textbf{28\%} & \multicolumn{1}{c|}{\textbf{18\%}} & 35070.2 & 70508.0 & 66149.3 & \multicolumn{1}{c|}{\textbf{127656.0}} & 397.4 & - & - & \multicolumn{1}{c|}{-} & - & - & - & - \\ \hline
 & \multicolumn{16}{c|}{Random-large: 50 * 50 world size with 17.5\% static obstacle rate and 50\%, 55\%, 60\%, 65\% agents density} \\ \hline
LNS2+RL & \textbf{35\%} & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & \textbf{127514.3} & - & - & \multicolumn{1}{c|}{-} & \textbf{4308.7} & 5000.0 & 5000.0 & \multicolumn{1}{c|}{5000.0} & \textbf{\begin{tabular}[c]{@{}c@{}}2.95\\ (-42.7\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}108.30\\ (-21.2\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}968.85\\ (-35.1\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}3484.00\\ (-18.3\%)\end{tabular}} \\
LNS2 & 30\% & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & 130756.5 & - & - & \multicolumn{1}{c|}{-} & 4329.2 & 5000.0 & 5000.0 & \multicolumn{1}{c|}{5000.0} & 5.15 & 137.45 & 1477.10 & 4261.85 \\
EECBS & 0\% & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & - & - & - & \multicolumn{1}{c|}{-} & 5000.0 & 5000.0 & 5000.0 & \multicolumn{1}{c|}{5000.0} & - & - & - & - \\
LACAM & 5\% & \textbf{25\%} & 0\% & \multicolumn{1}{c|}{\textbf{5\%}} & 204820.0 & \textbf{955662.6} & - & \multicolumn{1}{c|}{\textbf{474872.0}} & - & - & - & \multicolumn{1}{c|}{-} & - & - & - & - \\ \hline
 & \multicolumn{16}{c|}{Empty: 25 * 25 world size with 0\% static obstacle rate and 72.5\%, 75\%, 77.5\%, 80\% agents density} \\ \hline
LNS2+RL & \textbf{100\%} & 84\% & 6\% & \multicolumn{1}{c|}{0\%} & 20155.2 & 20132.5 & \textbf{19278.0} & \multicolumn{1}{c|}{-} & 94.7 & 333.5 & 596.6 & \multicolumn{1}{c|}{600.0} & \textbf{\begin{tabular}[c]{@{}c@{}}0.00\\ (-0.0\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}0.26\\ (-66.7\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}4.04\\ (-1.9\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}19.08\\ (-11.0\%)\end{tabular}} \\
LNS2 & \textbf{100\%} & 68\% & 18\% & \multicolumn{1}{c|}{0\%} & \textbf{19365.8} & \textbf{19485.4} & 19390.4 & \multicolumn{1}{c|}{-} & 117.7 & 403.8 & 575.5 & \multicolumn{1}{c|}{600.0} & \textbf{0.00} & 0.78 & 4.12 & 21.44 \\
EECBS & 0\% & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & - & - & - & \multicolumn{1}{c|}{-} & 600.0 & 600.0 & 600.0 & \multicolumn{1}{c|}{600.0} & - & - & - & - \\
LACAM & \textbf{100\%} & \textbf{100\%} & \textbf{100\%} & \multicolumn{1}{c|}{\textbf{100\%}} & 24834.3 & 26929.9 & 29408.2 & \multicolumn{1}{c|}{\textbf{32298.9}} & \textbf{0.006} & \textbf{0.007} & \textbf{0.007} & \multicolumn{1}{c|}{\textbf{0.008}} & - & - & - & - \\ \hline
 & \multicolumn{16}{c|}{Maze: 25 * 25 world size with 45.76\% static obstacle rate and 20\%, 22.5\%, 25\%, 27.5\% agents density} \\ \hline
LNS2+RL & \textbf{100\%} & \textbf{78\%} & \textbf{44\%} & \multicolumn{1}{c|}{\textbf{6\%}} & 5724.3 & 6523.3 & 7307.6 & \multicolumn{1}{c|}{\textbf{8698.0}} & \textbf{16.4} & \textbf{183.6} & \textbf{416.0} & \multicolumn{1}{c|}{\textbf{568.8}} & \textbf{\begin{tabular}[c]{@{}c@{}}0.00\\ (-100\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}0.34\\ (-10.5\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}1.24\\ (-51.6\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}4.88\\ (-50.8\%)\end{tabular}} \\
LNS2 & 92\% & 74\% & 26\% & \multicolumn{1}{c|}{0\%} & \textbf{5563.3} & \textbf{6205.6} & \textbf{7210.6} & \multicolumn{1}{c|}{-} & 67.6 & 217.9 & 493.5 & \multicolumn{1}{c|}{600.0} & 0.08 & 0.38 & 2.56 & 9.92 \\
EECBS & 6\% & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & - & - & - & \multicolumn{1}{c|}{-} & 600.0 & 600.0 & 600.0 & \multicolumn{1}{c|}{600.0} & - & - & - & - \\
LACAM & 2\% & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & 41924.0 & - & - & \multicolumn{1}{c|}{-} & 588.8 & 600.0 & 600.0 & \multicolumn{1}{c|}{-} & - & - & - & - \\ \hline
 & \multicolumn{16}{c|}{Room: 25 * 25 world size with 19.52\% static obstacle rate and 40\%, 45\%, 50\%, 55\% agents density} \\ \hline
LNS2+RL & \textbf{100\%} & \textbf{98\%} & \textbf{12\%} & \multicolumn{1}{c|}{0\%} & 35497.9 & 44779.1 & 58620.2 & \multicolumn{1}{c|}{-} & \textbf{41.3} & \textbf{154.5} & \textbf{577.9} & \multicolumn{1}{c|}{600.0} & \textbf{\begin{tabular}[c]{@{}c@{}}0.00\\ (-100\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}0.02\\ (-99.9\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}12.14\\ (-91.8\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}179.08\\ (-69.3\%)\end{tabular}} \\
LNS2 & 90\% & 36\% & 2\% & \multicolumn{1}{c|}{0\%} & \textbf{26660.6} & \textbf{35221.3} & \textbf{39378.0} & \multicolumn{1}{c|}{-} & 167.1 & 467.9 & 599.2 & \multicolumn{1}{c|}{600.0} & 0.34 & 19.44 & 148.22 & 582.5 \\
EECBS & 0\% & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & - & - & - & \multicolumn{1}{c|}{-} & 600.0 & 600.0 & 600.0 & \multicolumn{1}{c|}{600.0} & - & - & - & - \\
LACAM & 0\% & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & - & - & - & \multicolumn{1}{c|}{-} & 600.0 & 600.0 & 600.0 & \multicolumn{1}{c|}{600.0} & - & - & - & - \\ \hline
 & \multicolumn{16}{c|}{Warehouse: 25 * 25 world size with 28.8\% static obstacle rate and 42.5\%, 45\%, 47.5\%, 50\% agents density} \\ \hline
LNS2+RL & \textbf{72\%} & \textbf{42\%} & \textbf{4\%} & \multicolumn{1}{c|}{0\%} & 11202.9 & 12240.2 & \textbf{12082.0} & \multicolumn{1}{c|}{-} & \textbf{296.9} & \textbf{458.9} & \textbf{595.6} & \multicolumn{1}{c|}{600.0} & \textbf{\begin{tabular}[c]{@{}c@{}}0.48\\ (-33.3\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}1.60\\ (-45.6\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}6.58\\ (-23.8\%)\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}17.22\\ (-11.5\%)\end{tabular}} \\
LNS2 & 52\% & 26\% & 0\% & \multicolumn{1}{c|}{0\%} & \textbf{11027.3} & \textbf{11892.8} & - & \multicolumn{1}{c|}{-} & 383.2 & 526.0 & 600.0 & \multicolumn{1}{c|}{600.0} & 0.72 & 2.94 & 8.64 & 19.46 \\
EECBS & 0\% & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & - & - & - & \multicolumn{1}{c|}{-} & 600.0 & 600.0 & 600.0 & \multicolumn{1}{c|}{600.0} & - & - & - & - \\
LACAM & 2\% & 0\% & 0\% & \multicolumn{1}{c|}{0\%} & 3061736.0 & - & - & \multicolumn{1}{c|}{-} & - & - & - & \multicolumn{1}{c|}{-} & - & - & - & - \\ \hline
\end{tabular}}
\end{sidewaystable}

\subsubsection{Results for different time limits and easier tasks}
\begin{figure*}[h]
\vspace{-0.4cm}
\captionsetup[subfigure]{aboveskip=-1pt} 
\centering
\begin{subfigure}[b]{0.23\textwidth}
    \includegraphics[width=\textwidth]{diff_time_random.pdf}
    \caption{Result on the random-small map with 60\% agent density and different runtime limit.}
    \label{fig:diff_random}
\end{subfigure}
\begin{subfigure}[b]{0.23\textwidth}
    \includegraphics[width=\textwidth]{diff_time_warehouse.pdf}
    \caption{Result on the warehouse map with 42.5\% agent density and different runtime limit.}
    \label{fig:diff_warehouse}
\end{subfigure}
\begin{subfigure}[b]{0.23\textwidth}
    \includegraphics[width=\textwidth]{easy_random.pdf}
    \caption{Result on the random-small map with 100 second runtime limit and different agent density.}
    \label{fig:easy_random}
\end{subfigure}
\begin{subfigure}[b]{0.23\textwidth}
    \includegraphics[width=\textwidth]{easy_warehouse.pdf}
    \caption{Result on the warehouse map with 600 second runtime limit and different agent density.}
    \label{fig:easy_warehouse}
\end{subfigure}
\includegraphics[width=0.45\textwidth]{label.pdf}
\setlength{\belowcaptionskip}{-0.5cm}
\vspace{-0.2cm}
\caption{Result with different runtime limit and lower agent density.}
\label{fig:app_map}
\end{figure*}
In this section, we use the random-small map and warehouse map as representatives of less-structure maps and high-structure maps, respectively, to evaluate the performance of algorithms under varying runtime limits and task difficulties.
Specifically, we conducted the runtime-limit experiments on the random-small map with a 60\% agent density and the warehouse map with a 42.5\% agent density.
Their results are shown in Figure~\ref{fig:diff_random} and~\ref{fig:diff_warehouse}.
These results indicate that: First, the relative performance ranking of the algorithms remains consistent across different runtime limits, as discussed in the main paper.
LNS2+RL maintains its superiority over LNS2 in both SR and CP.
Second, each algorithm has a distinct runtime limit threshold, beyond which its SR stabilizes.
Notably, the thresholds of LNS2+RL and LNS2 exceed those of LACAM and EECBS.
For instance, as depicted in Figure~\ref{fig:diff_random}, increasing the runtime limit from 20 to 60 seconds results in an 18\% increase in SR for LNS2+RL and a 4\% increase for LNS2, while the SRs for LACAM and EECBS remain unchanged.

Subsequently, we maintained a constant runtime limit while progressively reducing task difficulty (agent density).
The results, illustrated in Figures~\ref{fig:easy_random} and~\ref{fig:easy_warehouse}, indicate that: First, EECBS and LACAM perform relatively well on less challenging tasks, but the performance gaps between them and LNS2+RL and LNS2 widen as task difficulty escalates.
Second, although in simpler tasks, the less pronounced advantages of the MARL-planner over PP+SIPPS lead to an earlier meeting of the switching condition, thus reducing the proportion of MARL-planner usage.
However, LNS2+RL continues to achieve performance similar to that of LNS2.


\subsubsection{Impact of different hyperparameters}

Here, we take tasks on the random-small map with 60\% agent density as an example to show the impact of the hyperparameters used to insert the MARL planner into the overall framework.
These hyperparameters include the neighborhood size in MARL planning, the threshold $\rho$ to switch the replanning solver from MARL planner to PP+SIPPS, the time step $t_l$ to stop MARL planning, and the maximum length of the hybrid path $t_h$.
The results are shown in Table~\ref{table:neighbor}~\ref{table:threshold}~\ref{table:stop}~\ref{table:max}.
The hyperparameter values we selected outperformed others, yet the changes in hyperparameters did not exhibit a clear trend in SR. 
It's worth noting that the optimal values for hyperparameters may vary across different tasks, necessitating fine-tuning to achieve the best performance of LNS2+RL.
For instance, in handling challenging tasks, using larger neighborhood sizes and smaller switching thresholds can increase the proportion of paths computed by the MARL-planner, thereby further leveraging its advantages to improve SR.

\begin{table}[H]
\vspace{-0.5cm}
\centering
\caption{Success rate for different neighborhood sizes}
\label{table:neighbor}
\begin{tabular}{|c|c|c|c|}
\hline
Neighborhood Size & 4 & 8 & 16 \\ \hline
Success Rate & 55\% & \textbf{59\%} & 57\% \\ \hline
\end{tabular}
\end{table}
\vspace{-0.7cm}
\begin{table}[H]
\centering
\caption{Success rate for different $\rho$}
\label{table:threshold}
\begin{tabular}{|c|c|c|c|c|c|}
\hline
Switch Threshold $\rho$ & 0.2 & 0.25 & 0.3 & 0.35 & 0.4 \\ \hline
Success Rate & 58\% & 57\% & \textbf{59\%} & 50\% & 56\% \\ \hline
\end{tabular}
\end{table}
\vspace{-0.7cm}
\begin{table}[H]
\centering
\caption{Success rate for different $t_l$}
\label{table:stop}
\begin{tabular}{|c|c|c|c|}
\hline
Stop MARL planning $t_l$ & 0.8 & 1.2 & 1.6 \\ \hline
Success Rate & 55\% & \textbf{59\%} & 58\% \\ \hline
\end{tabular}
\end{table}
\vspace{-0.7cm}
\begin{table}[H]
\centering
\caption{Success rate for different $t_h$}
\label{table:max}
\begin{tabular}{|c|c|c|c|}
\hline
Maximum Length $t_h$ & 1.8 & 2.2 & 2.6 \\ \hline
Success Rate & 57\% & \textbf{59\%} & 54\% \\ \hline
\end{tabular}
\end{table}

\subsection{Map visualization}

Figure~\ref{fig:vis} illustrates examples of the maps used for testing. 
In all maps, purple cells represent static obstacles, and yellow cells represent empty areas.
The static obstacle layouts of the maze, room, and warehouse maps are fixed, and the obstacle layouts of the three random maps are randomly regenerated in each instance.
For every instance of all maps, the agents' distinct starting and goal positions are randomly chosen from empty cells within the same connected region of the map.
The tasks tested in Section~\ref{sec:map_type} are challenging, with the vacancy rates for the most difficult tasks on the seven maps only being 17.5\%, 17.5\%, 17.5\%, 20\%, 26.74\%, 25.48\%, and 21.2\% respectively.

\begin{figure*}[h]
\vspace{-0.4cm}
\captionsetup[subfigure]{aboveskip=-1pt} 
\centering
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{random_small.pdf}
    \caption{Random-small}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{random_medium.pdf}
    \caption{Random-medium}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{random_large.pdf}
    \caption{Random-large}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{empty.pdf}
    \caption{Empty}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{maze.pdf}
    \caption{Maze}
\end{subfigure}
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{room.pdf}
    \caption{Room}
\end{subfigure} 
\begin{subfigure}[b]{0.2\textwidth}
    \includegraphics[width=\textwidth]{warehouse.pdf}
    \caption{Warehouse}
\end{subfigure}
\setlength{\belowcaptionskip}{-0.5cm}
\vspace{-0.2cm}
\caption{Visualized examples of the maps used for empirical evaluation.}
\label{fig:vis}
\end{figure*}


\subsection{Hyperparameters}
\begin{table}[h]
\caption{Hyperparameters table}
\label{table:hyperparameter}
\scalebox{0.8}{
\begin{tabular}{l|l}
\hline
Hyperparameter & Value \\ \hline \hline
Number of MARL controlled agents & 8 \\ \hline
Ratio of total number of agents to world size & [(40\%, 45\%, 50\%), (50\%, 55\%, 60\%), (60\%, 65\%, 70\%)] \\ \hline
World size & (10, 25, 50) \\ \hline
Obstacle density & [(5\%, 7.5\%, 10\%), (10\%, 12.5\%, 15\%), (15\%, 17.5\%, 20\%)] \\ \hline
FOV size & 9 \\ \hline
Maximum episode length & 356 \\ \hline
$\alpha$ & (0.06, 0.05, 0.04)\\ \hline
$\beta_l, \beta_h$ & 2, 15\\ \hline
Reward shaping coefficient & 0.2 \\ \hline
$\delta, \delta_v, \delta_e$ & 10, 0.225, 0.075 \\ \hline
Spatial and temporal coefficient for trajectory prediction & 0.9, 0.1  \\ \hline
Curriculum learning switching difficulty timestep & (1e7, 2e7) \\ \hline

Learning rate & 1e-5 \\ \hline
Discount factor & 0.95 \\ \hline
Gae lamda & 0.95 \\ \hline
Clip parameter for probability ratio & 0.2 \\ \hline
Gradient clip norm & 50 \\ \hline
Entropy coefficient $\omega_e$ & 0.01 \\ \hline
Actor loss coefficient $\omega_{\pi}$ & 1 \\ \hline
Critic loss coefficient $\omega_v$ & 0.5 \\ \hline
Invalid action loss coefficient $\omega_{invalid}$ & 0.5 \\ \hline
Number of epoch & 10 \\ \hline
Number of processes & 32 \\ \hline
Mini batch size & 512 \\ \hline
Imitation learning timestep & (3e6, 2e6, 0) \\ \hline
Optimizer & AdamOptimizer \\ \hline
Training timestep for the first stage & 7e6 \\ \hline
Training timestep for the second stage & 7e6 \\ \hline
Maximum iterations & (30, 65, 100) \\ \hline

$\mu$ & 20 \\ \hline
$\rho$ &  0.3 \\ \hline
$d_l$ &  1.2 \\ \hline
$d_h$ &  2.2 \\ \hline
\end{tabular}}
\vspace{-0.2cm}
\end{table}

Table~\ref{table:hyperparameter} shows the hyperparameters used to train the MARL model and implement the LNS2+RL algorithm evaluated in Section \ref{sec:result} of our paper.
The first 29 hyperparameters are those used to train the MARL model, while the remaining hyperparameters are used to insert the MARL planner into the overall framework.
There are three different task difficulties in the curriculum learning of training stage one, so \textit{ratio of total number of agents to world size}, \textit{obstacle density}, \textit{$\alpha$}, \textit{Imitation learning timestep} and \textit{maximum iterations} contain three values/value sets, each value/value set corresponding to a task difficulty.
The hyperparameters used in the second stage of training are mostly the same as those in the first stage of training, except \textit{ratio of total number of agents to world size = (60\%, 65\%, 70\%)}, \textit{obstacle density = (15\%, 17.5\%, 20\%)},\textit{$\alpha$ = 0.04},\textit{maximum iterations = 100}

\subsection{Real world experiment}\label{app_real}

\subsubsection{Setup}
We use a warehouse environment for our experiment (Figure \ref{fig:exp_warehouse}).
We assume that each pixel has a resolution of 0.5 m per grid unit, and based on this resolution, we realize a \textit{7m x 4.5m} part of the warehouse in the real world. 
We use a total of 100 robots for our experiment, of which 90 are virtual, and 10 are real \textit{Sparklife Omnibots} robots with dimensions of approximately \textit{0.3m ×0.3m} equipped with mecanum wheels for omnidirectional movement. 
We randomly generate the start and end positions of robots in the environment, with the constraint that the start and end positions of the real robots are in the real part of the warehouse.
Since the MAPF problem assumes the accurate position of all agents for all time, we considered the ground truth positions for all virtual robots and used the \textit{OptiTrack Motion Capture System} to get accurate positions for our real robots. 
Finally, we pre-plan the complete path set using LNS2+RL before actual execution and compute the joint actions agents should take at each timestep based on this path set.

\subsubsection{Action dependency graph}
The paths generated by LNS2+RL assume perfect operation for each agent at every step in the discrete map.
However, due to the robots being imperfect and the environment being continuous, executing the planned path directly in the real world is infeasible.
\begin{wrapfigure}{r}{0.3\textwidth}
    % \vspace{-0.45cm}
    \centering
    \includegraphics[width=0.2\textwidth]{exp_warehouse.png}
    \caption{Map of the \textit{23 x 23} warehouse environment used in our experiments, where black cells represent obstacles and white cells free space.}
    \label{fig:exp_warehouse}
\end{wrapfigure}
% \vspace{-0.3cm}
For instance, a planner may output a set of actions that lets an agent (say 'A') take another agent's (say 'B) position while Agent B moves to a new cell. 
If we were to try and execute this behavior on robots directly, a number of things, like errors in localization, delays in executing motor control commands, or differences in velocities, could lead to a collision between agents.
To avoid such scenarios and make our joint set of actions executable, we follow the method proposed by \cite{honig2019persistent} and construct an \textit{Action Dependency Graph} (ADG).
The main benefit of having an ADG is that it introduces a precedence order for the agents occupying a cell. 
This means that if an agent moves faster than the remaining agents, it will still wait for agents to catch up if it occupies a cell after another agent in the originally planned path. 
This behavior ensures that the errors in executing the planned path do not propagate and hamper the execution plan.
Returning to our earlier example, using an ADG ensures that since Agent A is headed to a cell that Agent B occupies before it is in the originally planned path, it will wait for Agent B to reach a new cell before moving to occupy it.


\subsubsection{Execution}

To implement the ADG, we first need to convert each action into a task that is to be executed by a robot.
For this, we define a \textit{Task} object as follows:

\begin{verbatim}
object Task {
    taskID;   // Unique identifier for the task
    robotID;  // Identifier for the robot which does this task
    action;   // Action to be performed by the robot
    startPos; //start position of the robot before action execution
    endPos;   // Position of the robot after action execution
    time;     // Timestep when action to be executed
    status;   // Marks if the task is staged, enqueued, or done
};
\end{verbatim}

During execution, we first cycle through all agents, convert their actions into task objects, and use these tasks to construct an ADG. 
There are three statuses that an agent can have, namely:\textit{STAGED}, \textit{ENQUEUED} and \textit{DONE}.
All tasks are initialized with a \textit{STAGED} status. 
The status is changed to \textit{ENQUEUED} after all their dependent tasks in the ADG are done, meaning that this task is ready for execution. Lastly, the status is marked \textit{DONE} when the agent reaches the \textit{endPos} of the task.

\subsubsection{ROS architecture}
We have a \textit{central node} that is responsible for generating and maintaining the ADG and distributing tasks for robots when they are to be enqueued.
Furthermore, each robot has a \textit{robot node}, which is responsible for receiving a task or set of tasks from the central node, extracting a goal from it, and using a PID controller to reach the goal.
When a robot reaches its goal, it sends this information to the central node, marks this task as \textit{DONE}, and uses the ADG to enqueue additional tasks.
Figure \ref{fig:exp_ROS} portrays a snippet of the architecture for 2 robots.

\begin{figure}
    \centering
    
    \includegraphics[width=0.8\textwidth]{exp_ROS.png}
    \caption{An RQT\_Graph Snippet of our ROS architecture for 2 robots. Ovals indicate nodes, rectangles without ovals indicate topics, and rectangles with ovals indicate namespacing for multi-robots. The central\_marmot node assigns tasks to robots (here nexus0 and nexus1) based on an \textit{Action Dependency Graph}. Robots use those tasks to enqueue waypoints to the next goals. Upon reaching a goal, a robot will relay this information back to the central node.}
    \label{fig:exp_ROS}
\end{figure}

\end{document}
\begin{thebibliography}{25}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
  \providecommand{\doi}[1]{doi: #1}\else
  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi

\bibitem[Barer et~al.(2014)Barer, Sharon, Stern, and Felner]{barer2014suboptimal}
Barer, M., Sharon, G., Stern, R., and Felner, A.
\newblock Suboptimal variants of the conflict-based search algorithm for the multi-agent pathfinding problem.
\newblock In \emph{Proceedings of the International Symposium on Combinatorial Search}, volume~5, pp.\  19--27, 2014.

\bibitem[Bengio et~al.(2009)Bengio, Louradour, Collobert, and Weston]{bengio2009curriculum}
Bengio, Y., Louradour, J., Collobert, R., and Weston, J.
\newblock Curriculum learning.
\newblock In \emph{Proceedings of the 26th annual international conference on machine learning}, pp.\  41--48, 2009.

\bibitem[Boyarski et~al.(2015)Boyarski, Felner, Stern, Sharon, Betzalel, Tolpin, and Shimony]{boyarski2015icbs}
Boyarski, E., Felner, A., Stern, R., Sharon, G., Betzalel, O., Tolpin, D., and Shimony, E.
\newblock Icbs: The improved conflict-based search algorithm for multi-agent pathfinding.
\newblock In \emph{Proceedings of the International Symposium on Combinatorial Search}, volume~6, pp.\  223--225, 2015.

\bibitem[Cohen et~al.(2018)Cohen, Koenig, Kumar, Wagner, Choset, Chan, and Sturtevant]{cohen2018rapid}
Cohen, L., Koenig, S., Kumar, T.~S., Wagner, G., Choset, H., Chan, D.~M., and Sturtevant, N.~R.
\newblock Rapid randomized restarts for multi-agent path finding: Preliminary results.
\newblock In \emph{AAMAS}, pp.\  1909--1911, 2018.

\bibitem[Erdmann \& Lozano-Perez(1987)Erdmann and Lozano-Perez]{erdmann1987multiple}
Erdmann, M. and Lozano-Perez, T.
\newblock On multiple moving objects.
\newblock \emph{Algorithmica}, 2:\penalty0 477--521, 1987.

\bibitem[Goldenberg et~al.(2014)Goldenberg, Felner, Stern, Sharon, Sturtevant, Holte, and Schaeffer]{goldenberg2014enhanced}
Goldenberg, M., Felner, A., Stern, R., Sharon, G., Sturtevant, N., Holte, R.~C., and Schaeffer, J.
\newblock Enhanced partial expansion a.
\newblock \emph{Journal of Artificial Intelligence Research}, 50:\penalty0 141--187, 2014.

\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{he2016deep}
He, K., Zhang, X., Ren, S., and Sun, J.
\newblock Deep residual learning for image recognition.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pp.\  770--778, 2016.

\bibitem[H{\"o}nig et~al.(2019)H{\"o}nig, Kiesel, Tinka, Durham, and Ayanian]{honig2019persistent}
H{\"o}nig, W., Kiesel, S., Tinka, A., Durham, J.~W., and Ayanian, N.
\newblock Persistent and robust execution of mapf schedules in warehouses.
\newblock \emph{IEEE Robotics and Automation Letters}, 4\penalty0 (2):\penalty0 1125--1131, 2019.

\bibitem[Huang et~al.(2022)Huang, Li, Koenig, and Dilkina]{huang2022anytime}
Huang, T., Li, J., Koenig, S., and Dilkina, B.
\newblock Anytime multi-agent path finding via machine learning-guided large neighborhood search.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~36, pp.\  9368--9376, 2022.

\bibitem[Li et~al.(2021)Li, Ruml, and Koenig]{li2021eecbs}
Li, J., Ruml, W., and Koenig, S.
\newblock Eecbs: A bounded-suboptimal search for multi-agent path finding.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~35, pp.\  12353--12362, 2021.

\bibitem[Li et~al.(2022)Li, Chen, Harabor, Stuckey, and Koenig]{li2022mapf}
Li, J., Chen, Z., Harabor, D., Stuckey, P.~J., and Koenig, S.
\newblock Mapf-lns2: Fast repairing for multi-agent path finding via large neighborhood search.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~36, pp.\  10256--10265, 2022.

\bibitem[Luna \& Bekris(2011)Luna and Bekris]{luna2011push}
Luna, R. and Bekris, K.~E.
\newblock Push and swap: Fast cooperative path-finding with completeness guarantees.
\newblock In \emph{IJCAI}, pp.\  294--300, 2011.

\bibitem[Ma et~al.(2021)Ma, Luo, and Ma]{ma2021distributed}
Ma, Z., Luo, Y., and Ma, H.
\newblock Distributed heuristic multi-agent path finding with communication.
\newblock In \emph{2021 IEEE International Conference on Robotics and Automation (ICRA)}, pp.\  8699--8705. IEEE, 2021.

\bibitem[Ng et~al.(1999)Ng, Harada, and Russell]{ng1999policy}
Ng, A.~Y., Harada, D., and Russell, S.
\newblock Policy invariance under reward transformations: Theory and application to reward shaping.
\newblock In \emph{Icml}, volume~99, pp.\  278--287. Citeseer, 1999.

\bibitem[Okumura(2023)]{okumura2023lacam}
Okumura, K.
\newblock Lacam: Search-based algorithm for quick multi-agent pathfinding.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, volume~37, pp.\  11655--11662, 2023.

\bibitem[Phillips \& Likhachev(2011)Phillips and Likhachev]{phillips2011sipp}
Phillips, M. and Likhachev, M.
\newblock Sipp: Safe interval path planning for dynamic environments.
\newblock In \emph{2011 IEEE international conference on robotics and automation}, pp.\  5628--5635. IEEE, 2011.

\bibitem[Sartoretti et~al.(2019)Sartoretti, Kerr, Shi, Wagner, Kumar, Koenig, and Choset]{sartoretti2019primal}
Sartoretti, G., Kerr, J., Shi, Y., Wagner, G., Kumar, T.~S., Koenig, S., and Choset, H.
\newblock Primal: Pathfinding via reinforcement and imitation multi-agent learning.
\newblock \emph{IEEE Robotics and Automation Letters}, 4\penalty0 (3):\penalty0 2378--2385, 2019.

\bibitem[Schulman et~al.(2017)Schulman, Wolski, Dhariwal, Radford, and Klimov]{schulman2017proximal}
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.
\newblock Proximal policy optimization algorithms.
\newblock \emph{arXiv preprint arXiv:1707.06347}, 2017.

\bibitem[Sharon et~al.(2015)Sharon, Stern, Felner, and Sturtevant]{sharon2015conflict}
Sharon, G., Stern, R., Felner, A., and Sturtevant, N.~R.
\newblock Conflict-based search for optimal multi-agent pathfinding.
\newblock \emph{Artificial Intelligence}, 219:\penalty0 40--66, 2015.

\bibitem[Shi et~al.(2015)Shi, Chen, Wang, Yeung, Wong, and Woo]{shi2015convolutional}
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., and Woo, W.-c.
\newblock Convolutional lstm network: A machine learning approach for precipitation nowcasting.
\newblock \emph{Advances in neural information processing systems}, 28, 2015.

\bibitem[Skrynnik et~al.(2023)Skrynnik, Andreychuk, Yakovlev, and Panov]{skrynnik2023switch}
Skrynnik, A., Andreychuk, A., Yakovlev, K., and Panov, A.~I.
\newblock When to switch: Planning and learning for partially observable multi-agent pathfinding.
\newblock \emph{IEEE Transactions on Neural Networks and Learning Systems}, 2023.

\bibitem[Wagner \& Choset(2011)Wagner and Choset]{wagner2011m}
Wagner, G. and Choset, H.
\newblock M*: A complete multirobot path planning algorithm with performance bounds.
\newblock In \emph{2011 IEEE/RSJ international conference on intelligent robots and systems}, pp.\  3260--3267. IEEE, 2011.

\bibitem[Wagner \& Choset(2015)Wagner and Choset]{wagner2015subdimensional}
Wagner, G. and Choset, H.
\newblock Subdimensional expansion for multirobot path planning.
\newblock \emph{Artificial intelligence}, 219:\penalty0 1--24, 2015.

\bibitem[Wang et~al.(2020)Wang, Liu, Li, and Prorok]{wang2020mobile}
Wang, B., Liu, Z., Li, Q., and Prorok, A.
\newblock Mobile robot path planning in dynamic environments through globally guided reinforcement learning.
\newblock \emph{IEEE Robotics and Automation Letters}, 5\penalty0 (4):\penalty0 6932--6939, 2020.

\bibitem[Wang et~al.(2023)Wang, Xiang, Huang, and Sartoretti]{wang2023scrimp}
Wang, Y., Xiang, B., Huang, S., and Sartoretti, G.
\newblock Scrimp: Scalable communication for reinforcement-and imitation-learning-based multi-agent pathfinding.
\newblock In \emph{2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, pp.\  9301--9308. IEEE, 2023.

\end{thebibliography}