OSDAG: Online Scheduling for Efficient Multi-Robot Collaboration
Extracted source text · arXiv 2606.15255
\documentclass[10pt,twocolumn]{ICCAS}
% \IEEEoverridecommandlockouts % This command is only needed if
% you want to use the \thanks comman
% \overrideIEEEmargins
\usepackage{comment} % Needed to meet printer requirements.
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{multirow}
\usepackage{fancyvrb}
\usepackage{pdflscape}
\usepackage{lscape}
% \usepackage{algorithmic}
\usepackage{array}
\usepackage[caption=false,font=normalsize,labelfont=sf,textfont=sf]{subfig}
\usepackage{textcomp}
\usepackage{stfloats}
\usepackage{url}
\usepackage{verbatim}
\usepackage{graphicx}
\usepackage{cite}
\usepackage{subcaption}
\usepackage{caption}
\DeclareMathOperator*{\argmax}{argmax}
\usepackage{amssymb}
\usepackage{orcidlink}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{booktabs}
\usepackage{tikz}
\usepackage{diagbox}
\usetikzlibrary{arrows.meta, positioning, fit, backgrounds, calc, shapes.geometric, patterns, decorations.pathreplacing}
\begin{document}
\title{OSDAG: Online Scheduling for Efficient Multi-Robot Collaboration}
\author{$^\dagger$Thanh Nguyen Canh\textsuperscript{1,2}~\orcidlink{/0000-0001-6332-1002}, $^\dagger$Thang Tran Viet\textsuperscript{2}, Phuc Van Dinh\textsuperscript{2}, $^*$Xiem HoangVan\textsuperscript{2}~$\orcidlink{0000-0002-7524-6529}$, \\ and $^*$Nak Young Chong\textsuperscript{1,3}~$\orcidlink{0000-0001-5736-0769}$}
\affils{\textsuperscript{1}School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. \\ (\texttt{\{thanhnc, nakyoung\}@jaist.ac.jp}) {\small$^\dagger$Equal contribution}\\
\textsuperscript{2}University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. \\ (\texttt{\{23020773, 23020756, xiemhoang\}@vnu.edu.vn}) {\small${}^{*}$ Corresponding author}\\
\textsuperscript{3}Department of Robotics, Hanyang University, Ansan, 15588, Gyeonggi, Korea. \\}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\abstract{
Coordinating heterogeneous multi-robot systems (MRS) for complex, long-horizon tasks requires both flexible high-level reasoning and efficient low-level scheduling. Existing LLM-based approaches address the reasoning side but introduce two critical bottlenecks: (1)~repeated LLM inference during execution, which inflates latency with agent count, and (2)~offline, pre-committed scheduling, which forces robots to idle while waiting for sequentially ordered predecessors even when independent work is available. This paper presents OSDAG, a novel framework that integrates LLM-based task reasoning with Directed Acyclic Graph (DAG) representation and constraint-aware online scheduling. The LLM is invoked once to decompose a natural-language instruction into a dependency-annotated task graph, and a lightweight online scheduler then allocates ready tasks to idle agents in real time. The DAG representation encodes both precedence and resource constraints, ensuring correctness while exposing all available parallelism. Experiments across five benchmark scenarios demonstrate that OSDAG achieves $5–15 \times$ faster reasoning time compared to dialogue-based methods, reduces makespan by up to $38\%$ over sequential baselines, and maintains competitive success rates. Both simulation and real-world experiments on dual-arm manipulation tasks validate the effectiveness and practicality of the proposed approach for efficient multi-robot coordination.
The website and resources are available at \url{http://thanhnguyencanh.github.io/LLM_DAG4MultiRobot}
}
\keywords{
Multi-Robot Systems, LLMs, Task Scheduling, Directed Acyclic Graph, Online Planning, Task Allocation
}
\maketitle
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
Multi-Robot Systems (MRS) enable heterogeneous agents with distinct capabilities to collaboratively accomplish complex tasks. Compared to single-robot systems, MRS offer superior task completion time, broader workspace coverage, and flexible utilization of individual expertise, making them essential for search and rescue, warehouse automation, and industrial manufacturing~\cite{li2025large}. Traditional MRS coordination relies on manually designed control logic and planning algorithms requiring explicit behavior specification and deep domain knowledge~\cite{gerkey2004formal, giordani2010distributed}. While ensuring accuracy and predictability, these methods lack flexibility—any task or environment change necessitates costly reprogramming. Furthermore, classical approaches cannot interpret natural language, creating a gap between human intent and robot execution.
Recent advances in Large Language Models (LLMs) such as Gemini 2.5 Pro~\cite{comanici2025gemini} and GPT-4~\cite{achiam2023gpt} have opened new directions for robot coordination. Systems like SayCan~\cite{ahn2022can} and ChatGPT-Prompts\cite{wake2023chatgpt} demonstrate that natural language commands can be translated into executable robot actions with strong zero-shot generalization, enabling robots to adapt to novel tasks without reprogramming. Several studies have integrated LLMs into MRS using both centralized and decentralized paradigms. Decentralized methods like RoCo~\cite{mandi2024roco} assign each robot an LLM agent for dialogue-based coordination, but the conversation history grows linearly with the number of agents, leading to higher latency than single-query approaches. Centralized methods reduce this overhead but typically generate flat action sequences without modeling inter-task dependencies. More critically, existing frameworks—including recent works that combine LLMs with no-task-dependence execution~\cite{wang2024dart}, planning~\cite{song2023llm}, or Linear Programming~\cite{obata2024lip}—employ offline scheduling pipelines in which the entire plan is computed before any robot moves. This forces agents to execute predetermined sequences; even when Robot A finishes early, it remains idle waiting for Robot B due to rigid pre-allocated orderings, wasting valuable execution time.
Recent work further combines LLMs with structured planners to improve long-horizon reasoning and robustness. LaMMA-P~\cite{zhang2025lamma} integrates LLMs with PDDL planning, REBEL~\cite{gupte2024rebel} augments task allocation with experience-based rules, ELHPlan~\cite{ling2025elhplan} reduces token usage through intention-aware collaboration, and CoMuRoS~\cite{borate2025llm} introduces event-driven replanning for heterogeneous teams. Other methods use no-task-dependence execution~\cite{wang2024dart}, planning~\cite{song2023llm}, or optimization-based scheduling~\cite{obata2024lip}. TwoStep~\cite{bai2024twostep} decomposes goals into PDDL sub-problems solved by symbolic planners, while LiP-LLM~\cite{obata2024lip} constructs dependency graphs and formulates Linear Programming optimization for makespan minimization. Nevertheless, most existing frameworks still follow an offline pipeline in which the complete plan or schedule is computed before execution. As a result, robots may remain idle while waiting for pre-allocated task chains, even when independent subtasks are ready to execute.
To address these limitations, we propose OSDAG, a framework that integrates LLM-based reasoning with a Directed Acyclic Graph (DAG) representation and online scheduling. The LLM decomposes high-level instructions into dependency-annotated subtasks formalized as a DAG. Unlike offline methods, our online scheduler continuously monitors agent states and allocates ready tasks to idle robots in real time, enabling the immediate execution of independent subtasks in parallel. Our main contributions can be summarized as follows:
\begin{itemize}
\item A mechanism that combines LLM reasoning with DAG-based dependency modeling, where the LLM generates a directed graph capturing inter-task dependencies based on agent capabilities and environmental constraints.
\item An online scheduling algorithm that performs real-time task allocation, ensuring concurrency while minimizing idle time and adhering to dependency relations.
\item Experiments conducted in both simulation and real-world scenarios demonstrate superior performance in terms of success rate, latency, and scalability compared to baseline models.
\end{itemize}
% \section{Related Work}~\label{related}
% Large Language Models have significantly enhanced robot planning capabilities.
% ChatGPT-Prompts~\cite{wake2023chatgpt} translates natural language into structured
% action sequences while tracking environmental states.
% SayCan~\cite{rajvanshi2025sayconav} integrates LLMs with affordance models to
% prioritize physically feasible actions, achieving a 3--4$\times$ improvement over
% scripted baselines in complex scenarios.
% Code as Policies~\cite{liang2022code} generates executable Python code directly
% from free-form commands, outperforming prior methods by 30\% on zero-shot
% manipulation tasks.
% These works establish the promise of LLM-driven robotics but are designed for
% \emph{single-robot} settings, leaving inter-agent coordination and task dependency
% management unaddressed.
% Recent studies extend LLMs to Multi-Robot Systems via decentralized and centralized paradigms~\cite{li2025large}. In decentralized methods, RoCo~\cite{mandi2024roco} enables agents to exchange information and reach consensus on subtasks through dialogue. Centralized approaches like SMART-LLM~\cite{kannan2024smart} use a shared LLM for task parsing, robot selection, and assignment. However, both paradigms rely heavily on LLM inference, causing high token usage and reasoning latency---RoCo requires 5--15$\times$ more time than our method due to accumulated conversation history.
% In addition, recent works address efficiency and robustness in multi-agent planning. LaMMA-P~\cite{zhang2025lamma} combines LLMs with PDDL planning for long-horizon tasks. REBEL~\cite{gupte2024rebel} augments task allocation with experience-based rules. ELHPlan~\cite{ling2025elhplan} reduces token usage through intention-aware collaboration. CoMuRoS~\cite{borate2025llm} introduces event-driven replanning for heterogeneous teams. While these methods improve specific aspects, they still rely on offline scheduling pipelines that underutilize agent parallelism.
% To reduce LLM dependency, several works combine LLMs with structured frameworks. TwoStep~\cite{bai2024twostep} decomposes goals into PDDL sub-problems, which are solved by symbolic planners but require manual domain engineering. LiP-LLM~\cite{obata2024lip} constructs dependency graphs and formulates Linear Programming optimization for makespan minimization. However, these methods use offline scheduling—completing all planning stages before execution—forcing robots to wait for entire task chains, failing to utilize the idle time of available agents.
% Unlike prior approaches, our framework uniquely integrates LLM task decomposition with DAG-based dependency modeling and online scheduling. This enables real-time allocation, where idle agents immediately execute ready tasks, maximizing parallelism while preserving correctness.
\begin{figure*}[!ht]
\centering
\includegraphics[width=0.91\linewidth]{figures/overview.png}
\caption{System overview of OSDAG. The framework processes a high-level natural language instruction through three stages: (1)~\textit{LLM-driven Task Decomposition} grounds the instruction into agent-assigned, dependency-annotated sub-tasks using environment context and capability descriptions; (2)~\textit{Task Graph Formalization} constructs a DAG encoding precedence and resource constraints; (3)~\textit{Constraint-Aware Online Scheduling} dynamically assigns ready tasks to idle agents in real time, maximizing parallel execution while preserving task dependencies.}
\label{fig:overview}
\end{figure*}
\section{Methodology}~\label{problemformulation}
OSDAG converts a natural-language instruction into a dependency-aware task graph and executes the graph with a lightweight online scheduler. Given an instruction ${\mathcal{I}}$, the framework first grounds the task in the current multi-robot workspace, asks the LLM once to generate agent-assigned subtasks with dependencies, validates the generated structure, and then dispatches ready tasks to idle agents while preserving precedence and feasibility constraints. Fig.~\ref{fig:overview} provides an overview of the complete OSDAG pipeline.
\subsection{Problem Statement}
\label{subsec:problem_statement}
Consider a team of heterogeneous agents ${\mathcal{N}}=\{N_1,\ldots,N_N\}$ operating in a shared workspace containing objects ${\mathcal{O}}=\{o_1,\ldots,o_M\}$. Each agent $N_i$ has a reachable object set ${\mathcal{S}}(N_i) = \{o_j \in {\mathcal{O}} \mid d(N_i, o_j) \leq r_i\}$ and an executable action set ${\mathcal{A}}(N_i)$, such as \texttt{pick}, \texttt{place}, or \texttt{open}. The reachability between agent $N_i$ and object $o_j$ with their position $\mathbf{p}(N_i)$ and $\mathbf{p}(o_j)$ is determined by:
\begin{equation}
d(N_i,o_j)=\|\mathbf{p}(N_i)-\mathbf{p}(o_j)\|_2,
\end{equation}
and encoded as a binary matrix $\mathbf{R} \in \{0,1\}^{N \times M}$:
\begin{equation}
\mathbf{R}_{ij}={\mathbb{I}}[d(N_i,o_j)\leq r_i],
\end{equation}
where $r_i$ is the workspace radius of agent $N_i$ and ${\mathbb{I}}[\cdot]$ is the indicator function. Given a natural-language instruction ${\mathcal{I}}$, the objective is to generate a set of executable subtasks:
\begin{equation}
{\mathcal{T}}=\{T_1,T_2,\ldots,T_m\},
\end{equation}
assign them to feasible agents, preserve their logical dependencies, and minimize the total completion time. Each subtask is represented as:
\begin{equation}
T_i=(\mu_i,a_i,o_i,d_i,{\mathcal{D}}_i),
\end{equation}
where $\mu_i$ is the assigned agent, $a_i$ is the action, $o_i$ is the target object, $d_i$ is the destination or target state, and ${\mathcal{D}}_i \in \{id_1, \ldots, id_k\}$ denotes the prerequisite task set.
For each task $T_i$, let $s_i$, $p_i$, and $c_i=s_i+p_i$ denote its start time, execution duration, and completion time. The scheduling objective is
\begin{align}
\min \quad & C_{\max} \label{eq:objective}\\
\text{s.t.} \quad
& C_{\max} \geq c_i, && \forall i, \label{eq:makespan}\\
& s_j \geq c_i, && \forall (T_i,T_j)\in E, \label{eq:precedence}\\
& o_i \in {\mathcal{S}}(\mu_i), && \forall i, \label{eq:reachability}\\
& a_i \in {\mathcal{A}}(\mu_i), && \forall i. \label{eq:capability}
\end{align}
The constraints ensure that all precedence relations are respected and that each assigned agent can both reach the target object and execute the required action.
\begin{figure}[!t]
\centering
\includegraphics[width=\linewidth]{figures/scheme.png}
\caption{Illustration of the online scheduling process. Steps~(1)--(5) demonstrate dynamic task allocation across three robots.}
\label{fig:online_scheduling}
\end{figure}
\subsection{LLM-driven Task Decomposition}
\label{subsec:llm_decomposition}
The LLM is used as a high-level task reasoner rather than a low-level controller. Instead of repeatedly querying the LLM during execution, OSDAG invokes the LLM once to decompose the instruction into structured subtasks. The prompt is constructed as:
\begin{equation}
{\mathcal{P}}=\Phi\left({\mathcal{I}},
\{{\mathcal{S}}(N_i)\}_{i=1}^{N},
\{{\mathcal{A}}(N_i)\}_{i=1}^{N},
\boldsymbol{\sigma}\right),
\end{equation}
where $\boldsymbol{\sigma}$ denotes the current object states and workspace context. The prompt contains the user instruction, available agents, reachable objects, executable actions, and a strict output schema requiring each task to include its agent, action, object, destination, and dependencies.
The LLM generates a candidate task list:
\begin{equation}
\hat{{\mathcal{T}}}=f_{\theta}({\mathcal{P}}),
\end{equation}
where $f_{\theta}$ denotes the LLM. Each generated task is validated before execution. A task $T_i$ is accepted only if
\begin{equation}
{\mathcal{V}}(T_i)=
\big(o_i\in{\mathcal{S}}(\mu_i)\big)
\land
\big(a_i\in{\mathcal{A}}(\mu_i)\big)
\land
\big({\mathcal{D}}_i \subseteq \{1,\ldots,m\}\big).
\end{equation}
This validation removes infeasible assignments, invalid actions, and incorrect dependency references. If the generated output violates the schema or feasibility constraints, the plan is rejected and regenerated with refined constraints.
\subsection{Task Graph Formalization}
\label{subsec:task_graph}
After validation, the structured task list is converted into a Directed Acyclic Graph ${\mathcal{G}}=(V,E)$. Each task $T_i$ corresponds to a vertex $v_i\in V$, and each prerequisite relation defines a directed edge:
\begin{equation}
V=\{v_i \mid T_i\in{\mathcal{T}}\},
\quad
E=\{(v_j,v_i)\mid j\in{\mathcal{D}}_i\}.
\end{equation}
Thus, an edge $(v_j,v_i)$ indicates that task $T_j$ must be completed before task $T_i$ can start.
The graph ${\mathcal{G}}$ can also be represented by an adjacency matrix $\mathbf{A}\in\{0,1\}^{m\times m}$:
\begin{equation}
\mathbf{A}_{ji}={\mathbb{I}}[(v_j,v_i)\in E].
\end{equation}
A topological sort is applied to verify that ${\mathcal{G}}$ is acyclic. If a cycle is detected, the task graph is rejected because the dependencies are logically inconsistent. This DAG representation allows OSDAG to explicitly encode both sequential dependencies, such as \texttt{pick}$\rightarrow$\texttt{place}, and inter-agent dependencies, such as object handoff or shared-resource constraints.
Unlike flat action sequences, the DAG exposes independent task branches that can be executed in parallel. This structure is essential for reducing idle time in multi-agent execution because a task can be dispatched as soon as all its predecessors are completed, without waiting for unrelated tasks.
\begin{figure}
\centering
\resizebox{\columnwidth}{!}{%
\begin{tikzpicture}[
task/.style={minimum height=0.6cm, draw=black, thick, font=\bfseries},
robot1task/.style={task, fill=blue!30},
robot2task/.style={task, fill=orange!30},
robot3task/.style={task, fill=purple!30},
collabtask/.style={task, fill=green!30},
idle/.style={task, fill=gray!10, draw=gray!50, pattern=north east lines, pattern color=gray!30},
arrow/.style={-{Stealth[length=2mm]}, thick},
timeline/.style={},
agenttitle/.style={font=\bfseries},
]
% ============ Title ============
\node[font=\large\bfseries] at (7, 2.9) {\Large Online Scheduling (Ours)};
% ============ Time axis ============
\draw[thick, -{Stealth[length=3mm]}] (0, -1.0) -- (14.5, -1.0) node[right] {\Large Time (s)};
\foreach \x in {0, 2, 4, 6, 8, 10, 12, 14} {
\draw (\x, -1.1) -- (\x, -0.9);
\node[below] at (\x, -1.1) {\Large \x};
}
% ============ Agent Labels ============
\node[agenttitle, blue!70!black, anchor=east] at (-0.3, 1.5) {\Large Robot 1};
\node[agenttitle, orange!70!black, anchor=east] at (-0.3, 0.6) {\Large Robot 2};
\node[agenttitle, orange!70!black, anchor=east] at (-0.3, -0.3) {\Large Robot 3};
% ============ Robot 1 Timeline ============
% Task 1: pick red cube (0-3s)
\node[robot1task, minimum width=3cm, anchor=west] (r1t1) at (0, 1.5) {\Large T1: pick};
% Task 3: place red bowl (3-6s)
\node[robot1task, minimum width=3cm, anchor=west] (r1t3) at (3, 1.5) {\Large T3: place};
% Task 5: pick green cube (6-9s)
\node[robot1task, minimum width=3cm, anchor=west] (r1t5) at (6, 1.5) {\Large T5: pick};
% Idle during handoff (9-10s)
\node[collabtask, minimum width=1cm, minimum height=0.69cm, anchor=west] (r1t6a) at (9, 1.5) {\Large T6};
% Task 11: pick green cube (10-12s)
\node[robot1task, minimum width=2cm, minimum height=0.69cm, anchor=west] (r1t11) at (10, 1.5) {\Large T11};
% Task 12: pick green cube (12-14s)
\node[robot1task, minimum width=2cm, minimum height=0.69cm, anchor=west] (r1t12) at (12, 1.5) {\Large T12};
% ============ Robot 2 Timeline ============
% Task 2: pick yellow cube (0-3s)
\node[robot2task, minimum width=3cm, anchor=west] (r2t2) at (0, 0.6) {\Large T2: pick};
% Task 4: place yellow bowl (3-6s)
\node[robot2task, minimum width=3cm, anchor=west] (r2t4) at (3, 0.6) {\Large T4: place};
% Idle waiting for handoff (6-9s)
\node[idle, minimum width=3cm,minimum height=0.69cm, anchor=west] (r2idle1) at (6, 0.6) {\Large wait};
% Task 6: handoff (9-10s)
\node[collabtask, minimum width=1cm, minimum height=0.69cm, anchor=west] (r2t6b) at (9, 0.6) {\Large T6};
% Task 7: pick blue cube (10-12s)
\node[robot2task, minimum width=2cm, minimum height=0.69cm, anchor=west] (r2t7) at (10, 0.6) {\Large T7};
% Task 8: place blue bowl (12-14s)
\node[robot2task, minimum width=2cm, minimum height=0.69cm,anchor=west] (r2t8) at (12, 0.6) {\Large T8};
% Robot 3 Timeline
% Task 9: pick yellow cube (0-3s)
\node[robot3task, minimum width=3cm, anchor=west] (r3t9) at (0, -0.3) {\Large T9: pick};
% Task 10: place yellow bowl (3-6s)
\node[robot3task, minimum width=3cm, anchor=west] (r3t10) at (3, -0.3) {\Large T10: place};
\node[idle, minimum width=8cm, minimum height=0.69cm, anchor=west] (idle) at (6, -0.3) {\Large idle};
% ============ Dependency arrows ============
\draw[arrow, gray, dashed] (r1t5.south east) to[out=-45, in=135] (r2t6b.north west);
% ============ Collaboration bracket ============
\draw[decorate, decoration={brace, amplitude=5pt, raise=2pt}, thick, green!50!black]
(9, 1.9) -- (10, 1.9) node[midway, above=6pt] {\Large Handoff};
% ============ Makespan indicator ============
\draw[thick, red, dashed] (14, -1.0) -- (14, 2.1);
\node[red, font=\bfseries] at (14, 2.3) {\Large$C_{max}=14s$};
% ============ Offline Comparison (Below) ============
\node[font=\large\bfseries] at (7, -2.2) {\Large Offline Scheduling (Baseline)};
% Time axis for offline
\draw[thick, -{Stealth[length=3mm]}] (0, -5.5) -- (15.5, -5.5) node[right] {\Large Time (s)};
\foreach \x in {0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30} {
\pgfmathsetmacro{\xpos}{\x * 0.5}
\draw (\xpos, -5.6) -- (\xpos, -5.4);
\node[below] at (\xpos, -5.6) {\Large \x};
}
% Agent Labels for offline
\node[agenttitle, blue!70!black, anchor=east] at (-0.3, -3) {\Large Robot 1};
\node[agenttitle, orange!70!black, anchor=east] at (-0.3, -3.9) {\Large Robot 2};
\node[agenttitle, orange!70!black, anchor=east] at (-0.3, -4.8) {\Large Robot 3};
% ============ Robot 1 Offline Timeline ============
\node[robot1task, minimum width=1.5cm, minimum height=0.69cm,anchor=west] at (0, -3) {\Large T1};
\node[robot1task, minimum width=1.5cm, minimum height=0.69cm,anchor=west] at (1.5, -3) {\Large T3};
\node[robot1task, minimum width=1.5cm, minimum height=0.69cm,anchor=west] at (3.0, -3) {\Large T5};
\node[collabtask, minimum width=0.5cm, minimum height=0.69cm,anchor=west] at (4.5, -3) {\scriptsize T6};
\node[idle, minimum width=8cm, anchor=west] at (5.0, -3) {\Large idle ( waiting)};
\node[robot1task, minimum width=1.0cm, minimum height=0.69cm,anchor=west] at (13.0, -3) {\Large T11};
\node[robot1task, minimum width=1.0cm, minimum height=0.69cm,anchor=west] at (14.0, -3) {\Large T12};
% ============ Robot 2 Offline Timeline ============
\node[idle, minimum width=4.5cm, anchor=west] at (0, -3.9) {\Large idle (waiting)};
\node[collabtask, minimum width=0.5cm, minimum height=0.69cm,anchor=west] at (4.5, -3.9) {\scriptsize T6};
\node[robot2task, minimum width=1.5cm, minimum height=0.69cm,anchor=west] at (5, -3.9) {\Large T2};
\node[robot2task, minimum width=1.5cm, minimum height=0.69cm,anchor=west] at (6.5, -3.9) {\Large T4};
\node[robot2task, minimum width=1.0cm, minimum height=0.69cm,anchor=west] at (8.0, -3.9) {\Large T7};
\node[robot2task, minimum width=1.0cm, minimum height=0.69cm,anchor=west] at (9.0, -3.9) {\Large T8};
\node[idle, minimum width=5cm, anchor=west] at (10, -3.9) {\Large idle};
% Robot 3 offline timeline
\node[idle, minimum width=10cm,minimum height=0.69cm, anchor=west] at (0, -4.8) {\Large idle (waiting)};
\node[robot3task, minimum width=1.5cm,minimum height=0.69cm, anchor=west] at (10, -4.8) {\Large T9};
\node[robot3task, minimum width=1.5cm, minimum height=0.69cm,anchor=west] at (11.5, -4.8) {\Large T10};
\node[idle, minimum width=2cm,minimum height=0.69cm, anchor=west] at (13, -4.8) {\Large idle};
% Makespan for offline
\draw[thick, red, dashed] (15, -5.4) -- (15, -2.6);
\node[red, font=\bfseries] at (15, -2.2) {\Large $C_{max}=30s$};
% ============ Improvement annotation ============
\draw[thick, green!50!black, -{Stealth[length=3mm]}] (14.6, -0.9) -- (14.6, 1.8);
\node[green!50!black, font=\bfseries, rotate=90, anchor=south] at (15.2, 0.5) {\Large 53\% faster};
% ============ Legend ============
\begin{scope}[shift={(-1, -5.2)}]
\node[robot1task, minimum width=1cm, minimum height=0.4cm] at (0, -1.5) {};
\node[right] at (0.6, -1.5) {\Large Robot 1 Task};
\node[robot2task, minimum width=1cm, minimum height=0.4cm] at (4.6, -1.5) {};
\node[right] at (5.2, -1.5) {\Large Robot 2 Task};
\node[collabtask, minimum width=1cm, minimum height=0.4cm] at (9.2, -1.5) {};
\node[right] at (9.8, -1.5) {\Large Collaboration};
\node[idle, minimum width=1cm, minimum height=0.4cm] at (14, -1.5) {};
\node[right] at (14.6, -1.5) {\Large Idle Time};
\end{scope}
\end{tikzpicture}
}
\caption{Comparison of online scheduling (ours) and offline scheduling (baseline) on an example two-robot task graph.}
\label{fig:timeline_compare}
\end{figure}
\subsection{Constraint-Aware Online Scheduling}
\label{subsec:online_scheduling}
Given the task graph ${\mathcal{G}}=(V,E)$, OSDAG performs online scheduling during execution. The scheduler maintains the completed-task set ${\mathcal{C}}(t)$, the running-task set ${\mathcal{B}}(t)$, and the state of each agent, either \textit{Free} or \textit{Busy}. For a free agent $N_i$, the executable task set at time $t$ is defined as:
\begin{equation}
\begin{split}
{\mathcal{E}}_i(t)&=\{T_j \mid \; \mu_j=N_i,\;
T_j\notin {\mathcal{C}}(t)\cup{\mathcal{B}}(t),\\
&\; \mathrm{Pred}(T_j)\subseteq{\mathcal{C}}(t),\;
{\mathcal{R}}(N_i,T_j,t)=\mathrm{true}\},
\end{split}
\end{equation}
where $\mathrm{Pred}(T_j)$ denotes the predecessor set of $T_j$. Fig.~\ref{fig:online_scheduling} illustrates the online scheduling process. At each scheduling step, the scheduler updates the state of each robot as either \textit{Free} or \textit{Busy}, checks which tasks have completed, and immediately assigns newly executable tasks to available agents. This event-driven procedure allows independent branches of the task graph to proceed in parallel while enforcing precedence and resource constraints. The resource feasibility term is defined as:
\begin{equation}
{\mathcal{R}}(N_i,T_j,t)=
{\mathcal{R}}_{\mathrm{hold}}(N_i,T_j,t)
\land
{\mathcal{R}}_{\mathrm{reach}}(N_i,T_j),
\end{equation}
where ${\mathcal{R}}_{\mathrm{hold}}$ checks manipulation-state constraints, such as whether the agent is holding the required object, and ${\mathcal{R}}_{\mathrm{reach}}$ checks whether the target object is reachable.
When multiple tasks are executable for the same agent, the scheduler selects the earliest indexed ready task:
\begin{equation}
T_i^{*}(t)=
\operatorname*{arg\,min}_{T_j\in{\mathcal{E}}_i(t)} j.
\end{equation}
Once a task is completed, it is moved from ${\mathcal{B}}(t)$ to ${\mathcal{C}}(t)$, and the scheduler immediately re-evaluates newly available tasks. In this way, independent branches of the DAG can be executed concurrently, while all dependency and feasibility constraints are preserved. Alg.~\ref{alg:online_scheduling} summarizes the complete online scheduling procedure. Fig.~\ref{fig:timeline_compare} further compares the proposed online scheduling strategy with an offline scheduling baseline. Unlike offline scheduling, which fixes the task order before execution and may leave agents waiting for delayed dependencies, OSDAG dispatches ready tasks immediately once their prerequisites are satisfied. This reduces idle time and shortens the overall makespan.
\begin{algorithm}[!t]
\caption{Constraint-Aware Online Scheduling}
\label{alg:online_scheduling}
\begin{algorithmic}[1]
\Require Task graph ${\mathcal{G}}=(V,E)$, agents ${\mathcal{N}}$
\Ensure Makespan $C_{\max}$
\State Initialize ${\mathcal{C}}\leftarrow\emptyset$, ${\mathcal{B}}\leftarrow\emptyset$
\State Set all agents to \textit{Free}
\While{${\mathcal{C}}\neq V$}
\For{each agent $N_i\in{\mathcal{N}}$}
\If{$N_i$ is \textit{Free}}
\State Compute executable task set ${\mathcal{E}}_i(t)$
\If{${\mathcal{E}}_i(t)\neq\emptyset$}
\State $T^*\leftarrow\operatorname*{arg\,min}_{T_j\in{\mathcal{E}}_i(t)} j$
\State Assign $T^*$ to $N_i$
\State Add $T^*$ to ${\mathcal{B}}$
\EndIf
\EndIf
\EndFor
\For{each completed task $T_j$}
\State Move $T_j$ from ${\mathcal{B}}$ to ${\mathcal{C}}$
\State Set the corresponding agent to \textit{Free}
\EndFor
\State $t\leftarrow t+\Delta t$
\EndWhile
\State \Return $C_{\max}=\max_i c_i$
\end{algorithmic}
\end{algorithm}
\begin{figure*}[!t]
\centering
\begin{minipage}[b]{0.15\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/task1.png}
\caption*{(a) Task 1}
\end{minipage}
\hfill
\begin{minipage}[b]{0.145\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/task2.png}
\caption*{(b) Task 2}
\end{minipage}
\hfill
\begin{minipage}[b]{0.164\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/task3.png}
\caption*{(c) Task 3}
\end{minipage}
\hfill
\begin{minipage}[b]{0.18\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/task4.png}
\caption*{(d) Task 4}
\end{minipage}
\hfill
\begin{minipage}[b]{0.183\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/task5.png}
\caption*{(e) Task 5}
\end{minipage}
\caption{Simulation environments for the five evaluation tasks.}
\label{fig:evaluation_tasks}
\end{figure*}
\section{Experimental Results}
\label{experiment}
\subsection{Experimental Setup}
We evaluate our system in both simulation and real-world environments. Simulation experiments are conducted using the PyBullet physics engine with custom tabletop manipulation environments. We employ Gemini Flash as the LLM backbone to validate planning efficiency and reasoning performance. To comprehensively assess performance, we design five evaluation scenarios covering different task complexities and spatial configurations. \textbf{Task 1} - Place cubes into matching bowls with 2 agents; \textbf{Task 2} - Place cubes into matching bowls with 3 agents to evaluate parallel execution capability. \textbf{Task 3} - Clean table to test implicit dependency reasoning (\textit{e.g.}, opening a drawer before placing items); \textbf{Task 4} - Sort objects in order: apple $\rightarrow$ banana $\rightarrow$ cup to assess sequential dependency handling. \textbf{Task 5} - Place cubes into matching bowls to evaluate inter-agent collaboration with object handoffs.
\textbf{Evaluation Metrics}.
We evaluate performance using four complementary metrics: Planning Success Rate (SR, $\%$), Reasoning Time (RT, seconds), Makespan (MS, seconds), and Success-weighted Step Efficiency (SSE)~\cite{obata2024lip}.
\textbf{Baselines}.
We compare against three LLM-based multi-agent planning baselines: RoCo~\cite{mandi2024roco}, a dialogue-based decentralized framework; TwoStep~\cite{bai2024twostep}, a hierarchical LLM-PDDL planner; and ChatGPT-Prompts~\cite{wake2023chatgpt}, a direct flat-sequence generation method. All methods use the same task instructions and environment states.
\subsection{Simulation Results}
Table~\ref{tab:results} reports performance across all five tasks, averaged over
10 trials. OSDAG achieves substantially lower reasoning time than dialogue-based methods and reduces makespan in dependency-heavy tasks, while maintaining competitive success rates.
\begin{table*}[t]
\scriptsize
\caption{\textbf{Performance comparison across all tasks.} Best results are in \textbf{bold}. ``--'' indicates execution failure.}
\label{tab:results}
\centering
\setlength{\tabcolsep}{4pt}
\begin{tabular}{l|cccc|cccc|cccc|cccc|cccc}
\toprule
\multirow{2}{*}{\textbf{Method}}& \multicolumn{4}{c|}{\textbf{Task 1}} & \multicolumn{4}{c|}{\textbf{Task 2}} & \multicolumn{4}{c|}{\textbf{Task 3}} & \multicolumn{4}{c|}{\textbf{Task 4}} & \multicolumn{4}{c}{\textbf{Task 5}} \\
& SR & SSE & RT & MS & SR & SSE & RT & MS & SR & SSE & RT & MS & SR & SSE & RT & MS & SR & SSE & RT & MS \\
\hline
RoCo~\cite{mandi2024roco} & 100 & 1.0 & 108.5 & 19.1 & 90 & 0.88 & 206.3 & \textbf{21.9} & 0 & 0 & 134.7 & -- & 100 & 1.0 & 64.1 & 22.7 & 100 & 1.0 & 94.8 & 41.6 \\
TwoStep~\cite{bai2024twostep} & 90 & 0.9 & 15.2 & 19.1 & 70 & 0.66 & 14.3 & 22.0 & 0 & 0 & 14.8 & -- & 0 & 0 & 14.7 & -- & 90 & 1.0 & 15.0 & 41.6 \\
ChatGPT-Prompts~\cite{wake2023chatgpt} & 100 & 0.5 & \textbf{8.2} & 33.9 & 90 & 0.36 & \textbf{8.9} & 35.9 & \textbf{90} & 0.51 & \textbf{8.3} & 48.3 & 100 & 1.0 & \textbf{10.3} & 22.7 & 100 & 1.0 & \textbf{9.2} & 41.6 \\ \hline
\textbf{Ours} & \textbf{100} & \textbf{1.0} & 9.1 & \textbf{19.1} & \textbf{90} & \textbf{0.88} & 14.0 & 22.5 & 80 & \textbf{0.8} & 13.4 & \textbf{26.1} & \textbf{100} & \textbf{1.0} & 12.8 & \textbf{15.2} & \textbf{100} & \textbf{1.0} & 10.5 & \textbf{26.0} \\
\hline
\end{tabular}
\end{table*}
\textbf{Reasoning Time.}
OSDAG requires 9.1--14.0\,s per task, achieving a 5--15$\times$ speedup over
RoCo across all scenarios.
RoCo's dialogue-based coordination is the primary bottleneck: each agent appends
to a shared conversation history, so latency scales with the number of robots—
Task~2 with three agents incurs 206.3\,s for RoCo versus 14.0\,s for OSDAG.
ChatGPT-Prompts achieves the shortest RT (8--10\,s) by generating a flat sequence
in a single call, but this comes at the cost of dependency modeling: its SSE
drops to 0.36--0.51 and its makespan is 33--48\,s, substantially higher than
OSDAG's 19--26\,s across the same tasks.
OSDAG matches or approaches ChatGPT-Prompts in RT while providing full dependency
awareness, since the LLM is also invoked exactly once but is asked to produce a
richer, structured output.
\textbf{Success Rate.} Our system achieves 80\% SR on Task~3, a complex scenario involving implicit dependencies requiring LLM reasoning (e.g., opening a drawer before placing items). By embedding current object states $\boldsymbol{\sigma}$ into the prompt, OSDAG
surfaces this hidden precondition in most trials, outperforming both RoCo (0\%)
and TwoStep (0\%), which lack the object-state context to reason about it at all.
\textbf{Makespan.} Our system reduces makespan by 33\% in Task~4 (15.2 vs 22.7) and 38\% in Task~5 (26.0s vs 41.6s) compared to baselines. By fully utilizing idle time through online scheduling, agents avoid waiting for others---Robot~2 can retrieve a bowl immediately while Robot~1 is placing, eliminating idle periods prevalent in offline methods.
\subsection{Ablation Study}
To isolate each component's contribution, we conduct ablation studies with the following variants:\textbf{w/o Embedding} - Removes object reachability, action descriptions, and status embeddings from the LLM prompt; \textbf{w/o Graph}: Generates sub-tasks without dependency annotations, resulting in a flat sequence;
\textbf{w/o Online Sched.} - Replaces online scheduling with sequential per-agent execution. Table~\ref{tab:ablation} presents the ablation results across all tasks.
\begin{table*}[t]
\scriptsize
\caption{\textbf{Ablation study results.} We evaluate each component's contribution by systematic removal. Removing dependency graphs causes complete failure (SR$=$0).}
\label{tab:ablation}
\centering
\setlength{\tabcolsep}{4pt}
\begin{tabular}{l|cccc|cccc|cccc|cccc|cccc}
\toprule
\multirow{2}{*}{\textbf{Variant}} & \multicolumn{4}{c|}{\textbf{Task 1}} & \multicolumn{4}{c|}{\textbf{Task 2}} & \multicolumn{4}{c|}{\textbf{Task 3}} & \multicolumn{4}{c|}{\textbf{Task 4}} & \multicolumn{4}{c}{\textbf{Task 5}} \\
& SR & SSE & RT & MS & SR & SSE & RT & MS & SR & SSE & RT & MS & SR & SSE & RT & MS & SR & SSE & RT & MS \\
\hline
w/o Embedding & 90 & 0.9 & 16.5 & 19.1 & 70 & 0.68 & 28.3 & 22.5 & 70 & 0.7 & 18.2 & 26.1 & 100 & 1.0 & 15.7 & 15.2 & 100 & 1.0 & 12.3 & 26.0 \\
w/o Graph & 0 & -- & \textbf{8.5} & -- & 0 & -- & \textbf{9.2} & -- & 0 & -- & \textbf{8.3} & -- & 0 & -- & \textbf{10.1} & -- & 0 & -- & 10.5 & -- \\
w/o Online Schedule & 100 & 1.0 & 9.1 & 19.1 & 90 & \textbf{0.9} & 14.0 & 22.5 & 80 & 0.8 & 13.4 & 26.1 & 100 & 1.0 & 12.8 & 22.7 & 100 & 1.0 & 10.5 & 41.6 \\
\hline
\textbf{Full (Ours)} & \textbf{100} & \textbf{1.0} & 9.1 & \textbf{19.1} & \textbf{90} & 0.88 & 14.0 & \textbf{22.5} & \textbf{80} & \textbf{0.8} & 13.4 & \textbf{26.1} & \textbf{100} & \textbf{1.0} & 12.8 & \textbf{15.2} & \textbf{100} & \textbf{1.0} & \textbf{10.5} & \textbf{26.0} \\
\hline
\end{tabular}
\end{table*}
\textbf{Effect of Embedding.} Removing object and action embeddings increases reasoning time (16.5s vs 9.1s in Task~1) and reduces success rate (70\% vs 90\% in Task~2). Without grounded context, the LLM occasionally hallucinates invalid actions or incorrect agent assignments.
\textbf{Effect of Dependency Graph.} Without dependency annotations, all tasks fail (SR $= 0$) due to resource conflicts and lack of coordination. Agents attempt concurrent manipulation of the same object or violate precedence constraints, causing execution failures.
\textbf{Effect of Online Scheduling.} Disabling online scheduling increases makespan significantly despite correct plans---Task~4 rises from 15.2s to 22.7s (+49\%), and Task~5 from 26.0s to 41.6s (+60\%). This confirms that dynamic task allocation is essential for efficient parallel execution.
\subsection{Real-World Experiments}
To validate practical applicability beyond simulation, we deploy our framework in a human-robot collaboration (HRC) scenario in which the human follows text-based instructions generated by our scheduler. The robotic agent consists of a UF850 manipulator equipped with an Intel RealSense D435 RGB-D camera and a parallel gripper. The human agent receives task instructions via a text display interface, which presents the scheduled actions in real time (\textit{e.g.}, ``Human: Task 5 -- pick(green\_cube)'').
We employ YOLOv11-seg for object detection and instance segmentation, followed by depth alignment to obtain 3D object poses. The perception module outputs object categories, positions, and dimensions, which are embedded into the LLM prompt for task decomposition.
We evaluate a collaborative cube-sorting task where the robot and human work in parallel to place colored cubes into matching bowls. Fig.~\ref{fig:realworld_qualitative}(c) shows the generated task graph, and Fig.~\ref{fig:realworld_qualitative}(d) illustrates execution snapshots at key timestamps. These results demonstrate that LLM-DAG effectively coordinates heterogeneous agents with different execution interfaces, validating its applicability to real-world HRC scenarios.
\begin{figure*}
\centering
\includegraphics[width=0.9\linewidth]{figures/realworld_qualitative.png}
\caption{Qualitative results of real-world UF850 manipulation.}
\label{fig:realworld_qualitative}
\end{figure*}
\section{Conclusion}~\label{conclu}
We presented OSDAG, a novel framework for multi-robot coordination that integrates the high-level reasoning capabilities of Large Language Models with DAG-based task representation and real-time online scheduling. By embedding agent reachability, manipulation capabilities, and object states into structured prompts, the LLM generates dependency-aware subtasks that are dynamically assigned to available robots. This design strictly preserves task dependencies while enabling parallel execution of independent subtasks, achieving 5–15$\times$ faster reasoning time than dialog-based methods and up to a 38\% reduction in makespan compared to offline scheduling approaches. However, our framework relies on the LLM to generate dependency edges, and reasoning errors may increase as environmental complexity and agent count increase. The current task selection policy may not yield globally optimal schedules. Additionally, the system lacks replanning mechanisms—execution failures can cascade without dynamic recovery.
% Future work will integrate local replanning, reward-based task selection, and continuous perception updates for more robust long-horizon coordination.
%===============================================================================
\section*{Acknowledgment}
This work was supported by JST SPRING, Japan Grant Number JPMJSP2102.
\bibliographystyle{IEEEtran}
\bibliography{ref} % .bib
\newpage
\end{document}