DRAGON: A Dialogue-Based Robot for Assistive Navigation with Visual Language Grounding
Extracted source text · arXiv 2307.06924
\section{Introduction}
\label{sec:intro}
Wayfinding, defined as helping people orient themselves in an environment and guiding them from place to place, is a longstanding challenge for persons with visual impairments (PwVI)~\cite{kulyukin2004rfid,bayles2022interdisciplinary}.
A guiding robot that can verbally interact with PwVI and connect language to the world, such as finding a destination or helping the user understand the environment, has the potential to improve the quality of their lives and to also reduce the load on their caregivers~\cite{azenkot2016enabling,bhat2022confused,bayles2022interdisciplinary}.
% Such mobile robots would eliminate the need for human guidance and hence greatly increase the independence afforded to PwVI~\cite{}.
% prev works and their limitations
% To address this need, researchers have developed various assistive technologies of different forms~\cite{}.
% prob of prev work: 1. no or limited communication about semantic stuff; 2. difficulty in generalizing to new env
A large body of previous wayfinding guides tackles joint dynamics and path planning for the human-robot team~\cite{ohnbar2018personalized,nanavati2019coupled_dyads,xiao2021guidedog}.
Another line of work pairs wayfinding with communication signals such as navigation instructions~\cite{li2019ballbot,jin2021wearable} and basic environment information~\cite{wilson2007swan,guerreiro2019cabot}.
% Other wayfinding technologies attempt to recognize and convey the semantic meanings of the surrounding environment, but they usually rely on instrumentation in the environment such as bluetooth beacons, which require laborious setups and maintainance~\cite{cheraghi2017guidebeacon,sato2017navcog3,kulyukin2006robot}.
As a step further, other wayfinding technologies recognize and convey the semantic meaning of the surrounding environment such as naming the landmarks~\cite{cheraghi2017guidebeacon,sato2017navcog3,kulyukin2006robot}.
However, these methods require special environmental setups, such as multiple RFID tags and bluetooth beacons.
We aim to remove dependence on these types of special infrastructure by integrating advances in visual-language grounding into conversational wayfinding.
% Voice-controlled robots (modular)
% semantic map and navigation
% no one introduced these to wayfinding yet, why adding these to wayfinding is beneficial
% although some similar attempts have been made (CaBot, SeeWay, Landmark AI), they lack integration of various types of dialogue and handle open-vocabulary landmarks
On the other hand, technologies in vision-language navigation and voice-controlled robots have made significant progress~\cite{chang2021learning,huang2022visual,ahn2022saycan}.
These navigation agents are able to perform various tasks according to natural language commands such as ``Bring me a cup.'' with simple onboard sensors.
% This ability is usually achieved by mapping semantic landmarks in the environment and associating language commands with semantic landmarks during navigation~\cite{}.
This is usually achieved by encoding visual landmarks in a semantic map and associating language with these landmarks during navigation, which is referred to as visual-language grounding~\cite{huang2022visual,shah2022lmnav}.
However, these general-purpose frameworks assume that humans can provide step-by-step navigation instructions. Thus, they are not built for PwVI that often need help in perceiving the environment and planning paths.
% Although attempts have been made to integrate some vision or language models in wayfinding solutions~\cite{guerreiro2019cabot,saha2019closing,yang2022seeway}, how to enable free-form semantic dialogue and integrate the dialogue seamlessly into a robot navigation guide remains an open challenge.
Although attempts have been made to integrate vision or language models in wayfinding~\cite{guerreiro2019cabot,saha2019closing,yang2022seeway}, how to build a guiding robot that can intuitively exchange semantic information with users remains an open challenge.
% Computer vision and natural language processing (NLP) community has made significant process in visual-language understanding tasks such as image captioning~\cite{}, visual question answering~\cite{antol2015vqa}, and image generation from text~\cite{ramesh2022dalle2}. % todo: cite clipclap
% The vision and language models make it viable to obtain semantic information from camera images without environmental instrumentation.
% To use these models on wayfinding for PwVI, CaBot uses an object detector to notify users of nearby objects and people, yet does not allow users to give commands~\cite{guerreiro2019cabot}. Some smartphone applications have enabled two-way semantic communication~\cite{saha2019closing,yang2022seeway}. % todo: expand these two papers a bit more?
% However, how to integrate these systems into a robot platform with physical support and guidance remains an open challenge.
\begin{figure}[tb]
\centering
\vspace{5pt}
\includegraphics[scale=0.16]{Figures/open_new.png}
\caption{\modelName identifies the intents of the user through dialogue, grounds language with the environment, and guides the user to their desired goal.}
\label{fig:overview}
\vspace{-20pt}
\end{figure}
% our design and approach
% overview
% In this paper, we propose an interactive robot guiding system which is able to hold rich conversations with its users.
In this paper, we propose \modelName, a \textbf{D}ialogue-based \textbf{R}obot for \textbf{A}ssistive navigation with visual-language \textbf{G}r\textbf{o}u\textbf{n}ding.
In Fig.~\ref{fig:overview}, \modelName uses speech to communicate with the user and a physical handle for navigation guidance.
% The dialogue and navigation can be executed at the same time.
The dialogue and navigation can be executed simultaneously.
% On the hardware side, we modified a TurtleBot 2i by adding a wireless microphone for verbal communication, a camera mounted at a height to observe the environme\, and a handle for guiding.
% For dialogue,
When the user gives a speech command, Speech Recognition (SR) and Natural Language Understanding (NLU) modules first extract the user's intents and desired destinations.
% Then, several visual-language models are used to associate the language with visual observations of the environment.
% More specifically, our system finds users' desired destinations with a based CLIP model~\cite{radford2021lclip}, describes a list of upfront objects from an object detector~\cite{}, and answers questions from users using a visual question answering (VQA) model~\cite{antol2015vqa}.
% To associate language with semantic information in the environment, our system can:
The main grounding functionalities include:
(1) finding users' desired destinations with a visual-language model~\cite{radford2021lclip} and guiding them to the destinations;
(2) describing nearby objects with an object detector~\cite{zhou2022detecting}; and (3) answering questions from users using a Visual Question Answering (VQA) model~\cite{antol2015vqa}.
With dialogue and grounding, \modelName can effectively navigate users to their desired destinations and help them gain awareness of their surroundings in an intuitive manner.
% highlight the landmark recognition
To find users' intended goals on a map, we propose a novel landmark recognition module based on CLIP~\cite{radford2021lclip}.
% When we use simultaneous localization and mapping (SLAM) to create a metric map, we only need to record a single RGB image and the location of each landmark at the same time.
% The mapping process is straightforward: \modelName records a single RGB image and the location of each landmark at the same time during the creation of a metric map with Simultaneous Localization and Mapping (SLAM).
% This simultaneous recording of data avoids any additional work in our landmark recognition module.
After a straightforward mapping process, the landmark recognizer is able to select the landmark whose image best matches the user descriptions.
Our landmark recognizer is able to associate flexible and open-vocabulary commands with few constraints on user expressions.
% with the best-matching landmark image during navigation.
% Then, the corresponding goal location is passed to the path planners, allowing for easy integration of our model into the navigation stack of mobile robots.
% During navigation, our landmark recognition module chooses the landmark that has the highest similarity with the user's language commands.
% This is achieved by comparing the similarity between the description and all landmark images with a visual-language model named CLIP~\cite{}.
% Our finetuned CLIP is flexible enough to handle both specific descriptions such as ``a water sink with hand soaps'' and vague descriptions such as ``somewhere to wash my hands''.
If the description is ambiguous, our system will disambiguate user intents through additional dialogue.
Then, the corresponding goal location is passed to the path planners for navigation guidance, which allows for easy integration of our model into the navigation stack for mobile robots.
Combined with the navigation stack, the powerful and reliable landmark recognizer is essential to ensure the success and user experience of \modelName.
% To the best of our knowledge, our work is the first physical guiding robot that uses a visual-language model to retrieve image landmarks from speech.
% During navigation, after a user says verbal commands, a natural language understanding (NLU) module parses the intent and entities in the input sentence.
% To create a semantic map of the environment before navigation, we only need to record each landmark as a pair of (RGB image, robot pose) and store them on a disk.
% During navigation, after a user says verbal commands, a natural language understanding (NLU) module parses the intent and entities in the input sentence. If the command involves visual-language grounding, the system activates one of the three visual-language grounding modules: navigation to a landmark, environment description, and visual question answering (VQA).
% The environment description module can recognize objects in the camera image, and the VQA module takes the camera image and user's question as input to generate an answer. The generated sentences from all submodules are narrated to the user via text-to-speech. All grounding modules connect speech and visual observations, which are intuitive to use and improve users' awareness of the surrounding scenes.
% todo: describe experiments, user study, and results
% \todohere{Merge 3 paragraphs if no space left.}
% In this paper, we present the following contributions:
Our main contributions are as follows:
% \item We design a robot guiding system that takes advantage of visual-language models to enable seamless semantic interaction with users without vision.
(1) As an interactive navigation guide for PwVI, \modelName enables voice-based dialogue, which carries rich information and has grounding capabilities;
(2) We propose a novel landmark mapping and recognition method that can associate free-form language commands with image landmarks. Our method can be easily plugged into the navigation stack of mobile robots; and
% \item Our experiments demonstrate that the grounding modules generalize to a previously unseen real-world environment without additional training.
(3) A user study (N=5) with blindfolded participants demonstrates that \modelName is able to understand user intents through dialogue and guide them to desired destinations in an intuitive manner.
To the best of our knowledge, our work is the first to show that visual-language grounding via dialogue benefits robotic assistive navigation.
% \todohere{Delete if no space left.}
% This paper is organized as follows,
% Section \ref{sec:related} discusses relevant related works.
% We then provide an overview of our whole mobile robot system in Section \ref{sec:system-overview}.
% Our Semantic Dialogue feature is detailed in Section \ref{sec:semantic-dialogue}.
% Section \ref{sec:experiments} provides a description of our user study and Section \ref{sec:results} discusses its results.
% Finally, we provide a summary of our work and discuss avenues for future work in Section \ref{sec:conclusion}.
% Section \ref{sec:platform} review the robot platforms we are working with. Section \ref{sec:methods} details the different components of our design. Section \ref{sec:exp} describes the preliminary experiments we have conducted. Section \ref{sec:future_work} discussed our planned improvements on our design. Finally, our findings are summarised in Section \ref{sec:conclusion}.
\section{Related Works}
\label{sec:related}
\subsection{Wayfinding robots and technologies}
\label{sec:related_wayfinding}
% To automate the navigation guide for PwVI, researchers have developed technologies of different forms~\cite{saha2019closing,jin2021wearable,zhang2017indoor,guerreiro2019cabot,xiao2021guidedog}.
% These devices and robots communicate with and guide users in different ways.
\begin{table*}[hb]
\vspace{-5pt}
\centering
\caption{Benchmark for conversational wayfinding technologies. A \cmark\, means that the functionality is implemented. A \circlemark\, means partial implementation. A blank cell means the functionality is absent. (In \cite{kulyukin2006ergonomics}, the users have to enter a number sequence into a keypad to specify their destinations. \cite{saha2019closing} can only describe a fixed set of pre-mapped landmarks and can only answer two fixed questions.)}
\label{tab:benchmark}
\begin{tabular}{@{}lccccccccc@{}} % <-- Alignments: 1st column left, 2nd middle and 3rd right, with vertical lines in between
\toprule
\multirow{2}{*}{\textbf{Method}} &
\textbf{User-chosen}&
\multicolumn{2}{c}{\textbf{Speech dialogue}} &
\textbf{Enviornment} &
\multirow{2}{*}{\textbf{VQA}}&
\multirow{2}{*}{\textbf{Form}} & \textbf{Environmental}\\
\cmidrule{3-4}
& \textbf{semantic goals}& Input & Output & \textbf{description}& & & \textbf{Instrumentation}\\
\midrule
GuideBeacon~\cite{cheraghi2017guidebeacon} &\cmark&\cmark&\cmark&&&Phone application & Bluetooth beacons \\
NavCog3~\cite{sato2017navcog3}&\cmark&\cmark&\cmark&\circlemark&\circlemark& Phone application& Bluetooth beacons\\
LandmarkAI~\cite{saha2019closing} &\cmark&\cmark&\cmark &\cmark&\circlemark& Phone application& GPS\\ % can only answer 2 questions: "what does this sign read?", "is this the place I'm looking for?"
% W-ROMA~\cite{jin2021wearable} &&&\cmark&&& Wearable device + Smartphone & None\\
SeeWay~\cite{yang2022seeway} &\cmark & \cmark &\cmark&&&Phone application& WiFi \\
Robotic Shopping cart~\cite{kulyukin2006ergonomics}&\circlemark&&\cmark&&& Robot & RFID tags \\
CaBot~\cite{guerreiro2019cabot} &&&\cmark&\cmark&&Robot& Remote joystick\\
Ballbot~\cite{li2019ballbot}&\cmark&\cmark&\cmark&&&Robot & WiFi + Remote computer \\
Ours &\cmark&\cmark&\cmark&\cmark&\cmark& Robot & WiFi + Remote computer\\
\bottomrule
\end{tabular}
\end{table*}
\textbf{Navigation guidance:} To guide PwVI from point A to point B following a planned path, unactuated devices, such as smartphones and wearables, rely on haptic or audio feedback to give instructions such as going straight and turning right~\cite{jin2021wearable,yang2022seeway,cheraghi2017guidebeacon,sato2017navcog3}.
However, delays and misunderstandings might lead to inevitable deviations, which take time and effort to recover from~\cite{wilson2007swan,cheraghi2017guidebeacon}.
On the other hand, robots provide a physical holding point, which offers kinesthetic feedback to minimize deviations and reduce the mental load of users~\cite{kulyukin2006ergonomics,nanavati2019coupled_dyads,zhang2023follower}.
Such physical guidance can be combined with aforementioned verbal or haptic navigation instructions to further improve performance at the cost of a more expensive system~\cite{guerreiro2019cabot,li2019ballbot}.
To ensure both efficiency and low cost, we mount a handle on our robot to give intuitive real-time steering feedback in navigation.
% \aamir{can mention that the handle allows for different styles of "attachment", making it easier for users to hold onto it they please}
%%%%%%%%%%%%%%%%% todo: move it to literature review %%%%%%%%%%%%%%%%%%
% Although some other devices can talk to users, they often process at least one of the following problems: (1) Very limited information is provided. For example, the device emits tonal sounds to indicate nearby obstacles without identifying whether the obstacles are furniture or people~\cite{}. (2) They have difficulties holding meaningful dialogues For example, the device can speak to users but users are not able to speak back~\cite{}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\textbf{Semantic communication:}
A large part of blind navigation technologies ignores exchanging environmental information with users~\cite{li2019ballbot,nanavati2019coupled_dyads,xiao2021guidedog}.
To deal with this issue,
% , in Table~\ref{tab:benchmark},
CaBot applies object recognition to describe the user's neighborhood, yet the user cannot hold conversations with the robot or choose their destinations~\cite{guerreiro2019cabot}.
% Our work includes the same environment descriptor while providing a dialogue system and more functionalities.
% \aamir{Such a dialogue systems leads to a more comfortable(?) UI while increasing the users trust in the system.}
To enable users to choose a semantic goal (\eg\: a restroom), some works mark points of interest using bluetooth beacons~\cite{cheraghi2017guidebeacon,sato2017navcog3} or RFID tags~\cite{kulyukin2006robot,kulyukin2006ergonomics}, which requires heavy instrumentation.
As an alternative, extracting semantic information from ego-centric camera images is much cheaper and easier.
For example, SeeWay uses skybox images to represent landmarks~\cite{yang2022seeway}.
Similarly, Landmark AI offers semantic-related functionalities including describing the environment, reading road signs, and recognizing landmarks using a phone camera~\cite{saha2019closing}.
However, these phone applications are not robots and thus cannot physically guide users or provide a stable mounting point for cameras. In contrast, Table~\ref{tab:benchmark} shows that \modelName brings conversational wayfinding to the next level: A robot can simultaneously offer physical guidance and enable users to trigger a variety of functionalities through dialogue.
% The only instrumentation is WiFi which is common in indoor environments.
% heavy instrumentation (GuideBeacon, RFID),
% seeway: use visual represenations (good) but 1. mostly sim, and 2. not a robot
% To provide information about the surrounding environment, a few works utilize object recognition from camera images~\cite{saha2019closing,guerreiro2019cabot}. Our work takes this step further by not only recognizing potential points of interest or danger, but also generating sentence descriptions and answering user queries about their surroundings.
% To enable more functionalities, such as choosing destinations and providing information about the surrounding environment,
% some works design buttons or keypads to receive user input and give haptic feedback or audio feedback with simple tones~\cite{}. These communication methods are fast and relatively accurate, yet only convey limited information.
\subsection{Command following navigation}
\label{sec:related_command_follow}
% vision-language tasks and models
% In recent years, machine learning community has witnessed arising interest in visual language understanding~\cite{radford2021lclip,ramesh2022dalle2,antol2015vqa}.
% Large pre-trained models such as ImageNet~\cite{deng2009imagenet}, CLIP~\cite{radford2021lclip}, and BERT~\cite{devlin2018bert} are commonly used as backbones for these tasks due to their expressiveness and good zero-shot transfer performance.
% vision-language in robotics
% semantic mapping: Burgard paper, FILM
% exploration: RSI 2, Clip on the wheel
% As a step toward natural human-robot interaction
Tremendous efforts have been made in understanding and grounding human language instructions for various robotic tasks~\cite{chang2021learning,ahn2022saycan,huang2022visual}.
% directly learn a policy
In command following navigation, a modular pipeline usually consists of three modules:
% (1) an SR system to transcribe a speech instruction to text;
(1) an NLU system to map instructions to speaker intent;
(2) a grounding module to associate the intent with physical entities; and
(3) a SLAM and a planner to generate feasible trajectories~\cite{liu2019review,min2021film,shah2022lmnav}.
% Since each part of the modular pipeline is trained with clean data, the whole pipeline is prone to intermediate and cascading errors. To this end,
Other works attempt to learn end-to-end policies from simulated environments or datasets~\cite{anderson2018vision, chen2020soundspaces,chang2020robot,gadre2022cow}. However, due to sim-to-real gaps in perception, language, and planning, deploying these policies to the real world remains an open challenge for applications in the low data regime such as wayfinding~\cite{Zhu2020The}.
Therefore, we adopt the modular pipeline to ensure performance in the real world.
% To deal with the intermediate errors, we finetune the modules with noisy data that resembles our target domain.
% also mention topological map? https://arxiv.org/pdf/2210.05714.pdf
\subsection{Semantic landmark recognition}
% mention the difference between close & open vocalbulary models (to motivate CLIP v.s. obj detector baseline)
Understanding the semantic meanings of a scene is a vital step towards interactive navigation~\cite{huang2022visual,shah2022lmnav}.
% Some works utilize visual-language models such as CLIP~\cite{radford2021lclip} and BERT~\cite{devlin2018bert} to build semantic maps~\cite{min2021film,huang2022visual}.
% % explain
% Then, during navigation, the language commands are mapped to specific objects on the semantic map, which are passed to robot planners to reach the goal.
Some works reconstruct volumetric maps for the environment, where each grid is associated with a semantic label~\cite{mccormac2017semanticfusion,min2021film,huang2022visual}. Other works build more abstract scene graphs~\cite{hughes2022hydra,wu2021scenegraphfusion}.
However, implementing these methods on a real robot is expensive, as they require accurately calibrated depth cameras and high-performing instance segmentation models.
Another line of work collects images as landmarks to create topological graphs~\cite{savinov2018semiparametric,meng2020scaling,chaplot2020neural}. In navigation, the goal location is retrieved by computing the similarity between a goal image and all stored landmarks.
% Similarly, to reduce the cost of the system, we use RGB images as landmarks.
However, the above works only consider image goals, which are less natural than language in human-centered applications.
Inspired by Shah \etal~\cite{shah2022lmnav} and Huang \etal~\cite{huang2022visual}, we use CLIP~\cite{radford2021lclip} to associate image landmarks with users' language commands.
Compared with previous works that use closed vocabulary object detectors, which are limited to a predefined set of semantic classes \cite{mccormac2017semanticfusion,wu2021scenegraphfusion,hughes2022hydra}, our method is able to handle more flexible and open-vocabulary commands.
We use CLIP to select landmarks and keep traditional cost maps for planning, enabling easy integration of our method into the navigation stack of mobile robots.
\section{System Overview}
\label{sec:system-overview}
\begin{figure*}[!ht]
\centering
\includegraphics[scale=0.45]{Figures/system_dialog.png}
\vspace{-3pt}
% \includegraphics[width=\textwidth]{Figures/system_dialog.png}
\caption{\textbf{An overview of the system and platform of \modelName}. (a) Submodules, message passing, and user interface. (b) The robot platform.}
\label{fig:system}
\vspace{-15pt}
\end{figure*}
In this section, we describe the setup and configuration of our robot guide.
Fig.~\ref{fig:system}(a) shows an overview of our proposed system consisting of three main components: (1) The TurtleBot platform (yellow); (2) Audio communication interface (purple); (3) Dialogue and grounding modules (red). The modules communicate with each other through ROS.
\subsection{Robot platform}
% \subsubsection{Overview}
\textbf{Overview:}
\label{sec:overview-turtlebot}
% The part is shown in the green part of Fig. 2
We use the Turtlebot2i as our robot platform.
% All components displayed in green in Figure \ref{fig:system} are directly controlled by the robot platform.
% The Turtlebot2i is a lightweight robot with a mobile base that can be modified to include hardware devices and sensors as needed.
As shown in Fig.~\ref{fig:system}(b), the robot is fitted with the following sensors and equipment:
(1) An RP-Lidar A3 laser range finder is mounted on the top of the robot structure for SLAM;
(2) An Intel RealSense D435i camera is mounted on the top of a monopod facing forward for scene description and question answering;
% ~\footnote{We do not use the depth information from the D435i.};
(3) A wireless headset is used to communicate with the user. The headset is lightweight and maximally protects the users' privacy, while the absence of wires avoids tripping hazards;
(4) A T-shaped handle is attached to the top rear side of the robot as a holding point for the user's arm.
The handle allows users to choose their preferred holding configurations such as one hand or two hands.
The robot is connected to a desktop computer which provides more computation resources through WiFi.
\textbf{Planning and Navigation:}
% \subsubsection{Planning and navigation}
\label{sec:overview-planning}
% The part is shown in the green part of Fig. 2
The robot operations are managed by the standard ROS
\texttt{move\_base} navigation stack.
%for navigating itself and the human.
Before navigation, we create a 2D occupancy map of the environment using laser-based SLAM and mark the semantic landmarks at the same time (see Fig.~\ref{fig:environment} and Sec.~\ref{sec:clip} for details).
At the beginning of each trial, the goal pose is obtained from the dialogue with the user (further specified in Sec.~\ref{sec:semantic-dialogue}).
During navigation, adaptive Monte Carlo localization is used to localize the robot on the map.
We use the dynamic window approach (DWA)~\cite{fox1997dynamic} and $\text{A}^*$ as local and global planners, respectively.
The minimum translational velocity is restricted to be non-negative to prevent the robot from moving backward and colliding with the user.
The maximum velocity of the robot can be adjusted by the user (see Sec.~\ref{sec:nav_speed} for details).
% Other parameters are tuned to better guide the user in our environment as follows: (1) maximum translational velocity $v_{max}=0.3~m/s$; (2) maximum angular velocity $\omega_{max}=0.5~rad/s$; (3) the footprint as a circle that includes the robot and the potential positions of the human.% here depends on how much space left, could merge it with overview.
% \aamir{Justify the use of the headset and the T-shaped handle.}
% \aamir{I wonder if it would be useful to have another figure labeling the different portions of the turtlebot here}.
% \subsubsection{Planning and navigation}
% \label{sec:overview-planning}
% \textcolor{pink}{
% Before navigation, the robot maps the environment using simultaneous localization and mapping (SLAM) with Gmapping ROS package~\cite{gmapping} During SLAM, the robot estimates its location using its wheel odometry, LiDAR scans, and adaptative Monte Carlo localization (AMCL)~\cite{pfaff2006robust}.
% Given the extracted destination from the audio module, our program finds the corresponding goal pose and passes it to the planner.
% To plan a path for the robot-human team based on the map, we use the A* algorithm to plan a global path and the dynamic window approach (DWA) as the local path planner~\cite{fox1997dynamic}.
% To incorporate pose information into the DWA planner, we change the collision boundary of the robot so that it also includes the estimated boundary of the user from Sec.~\ref{sec:pose_est}.
% More specifically, given the estimated pose of human, we generate a boundary box on the user's reachable spaces and use a polygon to represent the box in the robot frame, which are fed into the DWA planner.
% As a result, the planned paths avoids collisions for both the user and the robot in real-time.
% }
% Given the extracted destination from the audio module, our program finds the corresponding goal pose and passes it to the planner.
% For future work, we can use data-driven methods to predict human's motion when the robot is performing a turn or a sudden stop to improve the comfort and safety of the user.
\subsection{Audio Communication Interface}
\label{sec:overview-communication}
Speech is a natural choice for human-robot communication, particularly in cases where the human has limited vision~\cite{azenkot2016enabling,bhat2022confused,bayles2022interdisciplinary}.
% To this end, we developed an audio communication system that listens to the users commands to navigate them to a requested destination and communicates the output from the Visual-language grounding modules.
To this end, as shown in purple in Fig.~\ref{fig:system}(a), we develop an audio communication interface that consists of:
(1) \textbf{Input:} When audio is captured by the \texttt{audio\_capture} package, the OpenAI Whisper speech recognition model~\cite{whisper} transcribes speech commands to text, which are passed to the NLU module.
% The SR is a ROS topic that listens to the user and publishes the transcribed text constantly in real time.
The SR module continuously transcribes the audio from the microphone and publishes the text transcriptions to a ROS topic in real time;
% In particular, we use the OpenAI Whisper STT model~\cite{whisper} to convert the speech commands from the user into text before processing it with our NLU module.
% We discuss details of our NLU in Section \ref{sec:nlu}.
(2) \textbf{Output:} We use the Google text-to-speech (TTS) service to convert the text output from the visual-language modules and navigation module to speech, which is then narrated to the user via the headset.
The TTS is another ROS topic that converts and plays the synthesized sound constantly.
% We plan on extending this module by improving from a bag-of-words model to a data-driven deep learning based model to extract the destination location from the input speech snippet.
% Such a model would be able to better extract the destination in more complex sentences based on the context of the sentence than a bag-of-words model.
\subsection{Dialogue and grounding}
\label{sec:overview-visual-language}
In the dialogue system, when SR publishes a transcribed sentence, an NLU first extracts intents and landmarks (if any) from the sentence.
Then, based on the intents, the information from the sentence is passed to different visual-language grounding modules which output language responses to TTS and/or the navigation module which changes navigation behaviors.
By doing so, the robot fulfills the user's request.
The dialogue and grounding modules are detailed in Sec.~ \ref{sec:semantic-dialogue}.
\section{Dialogue And Grounding}
\label{sec:semantic-dialogue}
The goal of \modelName is to connect the user with the environment through conversation.
In this section, we describe how our dialogue system understands user language~(Sec.~\ref{sec:nlu}), maps and localizes semantic landmarks~(Sec.~\ref{sec:clip}), provides information about the environment~(Sec.~\ref{sec:img_caption_vqa}), and adjusts the navigation preference of the user~(Sec.~\ref{sec:nav_speed}).
Our grounding system is visualized in the red parts of Fig.~\ref{fig:system}(a).
\subsection{Natural language understanding (NLU)}
\label{sec:nlu}
% input, output
The NLU takes a transcribed sentence as input and outputs user intents and entities of interest. Table~\ref{tab:intents} shows all possible user intents.
The entities are locations, objects, and object attributes which include the material and functionalities of an object.
We use Dual Intent and Entity Transformer for intent classification and entity recognition~\cite{bunk2020diet}. We train the model using a custom dataset with $1092$ sentences. % collected ourselves.
For each intent, we collect various expressions including misspelled and phonetically similar phrases, which makes our NLU robust to the nuances of human language and the errors caused by the SR.
For example, ``a think'' and ``a sink'' both refer to the kitchen sink.
We also collected expressions for multi-intents and unknown intents so that the NLU can fulfill a request containing multiple intents and ignore noise input.
For instance, ``Hello robot, can you take me to a sofa?'' will both activate the robot and set an object goal.
Once the intent and entities are extracted, the corresponding downstream module is activated and the robot informs the user via language feedback.
The NLU may pass additional input arguments to modules such as extracted entities or the whole sentence.
During navigation, the landmark recognition is triggered if the user intent is \textit{Object goal} and the NLU extracts a goal object from the input sentence.
If the user mentions additional information about the landmark, we use simple prompt engineering to make the description more specific.
For example, locations and attributes of objects can be added to the description by ``a chair in the office'' or ``a gaming chair.''
In addition, the robot uses clarification dialogue to disambiguate the desired landmark if the input description does not contain any object.
If the user only provided the location or attributes without mentioning the object name (\eg~``Take me to the kitchen''), our system provides hints to encourage the user to provide more specific descriptions (\eg~``What object are you looking for in the kitchen?'').
If there are multiple similar objects in different landmarks, our system disambiguates the user's preferred landmark (\eg~``What kind of chair are you looking for? A dining chair, an office chair, or a sofa?'').
After choosing a unique landmark, our system confirms the goal with the user (\eg~``Do you wish to go to a dining chair?'').
No further action is taken until the user affirms the goal.
With the disambiguation and confirmation dialogue, the NLU is able to precisely capture the user's desired destination, which is crucial for the whole navigation experience. %in a user friendly manner.
\begin{table}[!tb]
% \vspace{-20pt}
\begin{center}
\caption{All user intents and their descriptions.}
\label{tab:intents}
\begin{tabular}{@{}ll@{}} % <-- Alignments: 1st column left, 2nd middle and 3rd right, with vertical lines in between
\toprule
\multirow{1}{*}{\textbf{Intents}} &
\multirow{1}{*}{\textbf{Descriptions}} \\
%\cmidrule{4-5}
% \hline
%& & & RSI\,1 & Ours \\
\midrule
%No Aux+Sparse R&
Greet & Wake up the robot and begin an interaction. \\
Object goal & Go to a specific object landmark. \\&May contain entities including objects and attributes.\\
Location goal & Go to a rough goal location (kitchen, lounge, etc) \\& without mentioning a specific object.\\
& May contain location entities.\\
Affirm & Confirm the goal. \\
Deny & Deny the goal. \\
Describe & Ask for a description of the surrounding environment. \\
Ask & Ask a question about the surrounding environment. \\
Pause & Pause the current navigation.\\
Resume & Resume the current navigation.\\
Accelerate & Increment velocities, up to a limit.\\
Decelerate & Decrement velocities, down to a limit.\\
Unknown & The text does not belong to any intents above (i.e. \\ & be noise, chitchat, etc) and is ignored by the robot.\\
\bottomrule
\end{tabular}
\end{center}
\vspace{-20pt}
\end{table}
% \begin{itemize}
% \item \textbf{Greet:} The user wants to wake up the robot and the robot begins the interaction.
% \item \textbf{Object goal:} The user wants to go to a specific object landmark.
% \item \textbf{Location goal:} The user wants to go to a rough goal location (kitchen, office, etc) but does not mention a specific object landmark.
% \item \textbf{Affirm:} The user affirms the previous statement of the robot, used to confirm the goal before the robot moves.
% \item \textbf{Deny:} The user denies the previous statement of the robot, used to confirm the goal before the robot moves.
% \item \textbf{Environment description:} The user wants a description of their surrounding environment from the robot.
% \item \textbf{Ask question:} The user asks a question about their surrounding environment and the robot has to provide an answer.
% \item \textbf{Pause:} The current goal of ROS navigation stack is saved and canceled, the robot stops but can listen to other commands.
% \item \textbf{Resume:} The saved current goal is sent to the navigation stack again. The robot continues to the last destination specified by the user.
% \item \textbf{Accelerate:} The maximum translational velocity and rotation velocity increase for some fixed values, up to a limit.
% \item \textbf{Decelerate:} The maximum translational velocity and rotation velocity decrease for some fixed values, down to a limit.
% \item \textbf{Unknown:} The input text does not belong to any intents above (noise, chitchat, etc) and is ignored by the robot.
% \end{itemize}
\subsection{Landmark mapping and recognition}
\label{sec:clip}
% overview sentence
To guide the user to their object goals, we first record the images and locations of landmarks during SLAM.
Then, we use a fine-tuned CLIP model to match the user's description with goal images, whose corresponding location is sent to the navigation stack for navigation guidance.
% how to map the landmarks
The landmark mapping process is performed simultaneously with SLAM.
During SLAM, when the robot is at a landmark that might be a point of interest, we simply save the current robot pose in the map frame and an RGB image of the landmark to the disk with a single key press.
% All landmarks are stored as pairs of (pose, RGB image).
No labels or text descriptions are needed at this stage.
The resulting landmark map is shown in Fig.~\ref{fig:environment}.
% in nav, how to recognize landmarks
During navigation, this module is activated when the extracted intent is \textit{Object goal}. % in Table~\ref{tab:intents}.
After the goal is confirmed by the NLU, the CLIP model selects the landmark whose image has the highest similarity score with the descriptions of landmarks.
% how CLIP achieves this
To obtain the image-text similarity score, a text encoder and an image encoder first convert the input text and all images to vector embeddings.
Then, the text and image similarity score is computed by the cosine similarity between the pairwise text and image embeddings.
The image with the highest similarity score is selected as the goal.
Finally, the corresponding location of the chosen landmark on the map is sent to an action client, which sets the goal for the robot.
% finetuning and data for finetuning
The zero-shot performance of pre-trained CLIP models is not satisfactory in our environment due to distribution shifts.
As shown in Fig.~\ref{fig:environment}, the objects in the images are frequently cropped due to the low mounting point of our camera and the close distance between the camera and the objects.
In addition, the descriptions of landmarks from a PwVI might be vaguer than those in public datasets (\eg~``a chair'' v.s. ``a blue chair in front of a white wall'').
To this end, we fine-tune the CLIP model with a custom dataset containing $544$ image and text description pairs from our environment with a $8\times 10^{-6}$ learning rate for 35 epochs.
% Alternatively, we can use a closed-vocabulary model for grounding, such as feeding landmark images to an object detector and matching the user's desired object name with the class labels of detected instances. However, open-vocabulary models such as CLIP are able to handle free-form language and are not limited to a fixed set of classes. Thus, the input expressions are less restricted, making the grounding module easier for non-experts to use.
By using an open-vocabulary model to recognize landmarks, \modelName is able to handle free-form language and is not limited to a fixed set of object classes. Thus, the user input expressions are less restricted, making the grounding module easier for non-experts to use.
% Compared with associating the object name with images using a closed-vocabulary model, the advantages of CLIP are twofold:
% (1) CLIP is able to handle free-form language and open-vocabulary objects, whereas a closed vocabulary model has a limited set of object classes;
% (2) Fine-tuning a CLIP model only needs image and text pairs, which are relatively easy to provide.
% In contrast, fine-tuning an object detector is more expensive because one-hot labels, segmentation masks, and bounding boxes are needed.
% our advantages: 1. no need to provide language labels in mapping & can handle open-vocabulary, free-form language in navigation
% 2. can be smoothly integrated with the original SLAM
\subsection{Environment understanding modules}
\label{sec:img_caption_vqa}
To help the user gain awareness of their surroundings,
% during navigation
we use an object detector~\cite{zhou2022detecting} to describe the objects (activated if the intent is \textit{Describe}) and a VQA model~\cite{kim2021vilt} to answer the user's questions (activated if the intent is \textit{Ask}).
Both models take the current camera image as input.
The output of the object detector consists of a list of detected instances, their object classes, confidence scores, and bounding boxes.
To avoid narrating a long list and to keep the description concise, we post-process the output as follows.
We first apply non-maximum suppression and filter out the detected instances with low confidence scores.
Then, for the remaining instances, we keep the top three classes with the largest average bounding boxes, and list the object class names together with the numbers of objects (\eg~``2 chairs, 1 person, and 1 table'').
The VQA model takes the current camera image and the user's question from the SR and outputs a short answer to the question.
Similar to CLIP, we collect a dataset of $10252$ (image, question, answer) triplets to fine-tune the VQA model for $20$ epochs.
To handle free-form user expressions, the dataset contains cases where multiple questions have the same meaning but different phrasing (\eg~``Is any person in front of me?'' and ``Anyone here?'').
The processed output of the object detector and the answer from the VQA model is sent to the TTS topic, which is narrated to the user in real time.
% However,
Since both models can only take a single RGB image, our system cannot provide depth-based information or detect anything out of the camera view.
% environment description and VQA
% also briefly mention their limitations, especially list of objects
% benefit of manual VQA for blind: ~\cite{bigham2010vizwiz}
\subsection{Navigation preference customization}
\label{sec:nav_speed}
% speed up & slow down, pause & resume
To accommodate the different walking paces of users and to avoid tiring the user during navigation, the robot can change its speed (activated if the intent is \textit{Accelerate} or \textit{Decelerate}), take a pause (\textit{Pause}), and resume (\textit{Resume}).
To pause the robot, our system stores and cancels the current goal from the action client in the navigation stack.
To resume, the stored goal is sent to the action client again.
To update the speed, we change the maximum translational and rotational velocities of the DWA local planner by updating its configuration through the \texttt{dynamic\_reconfigure} package.
% Please refer to ``speed up'', ``slow down'', ``pause'', and ``resume'' in Table~\ref{tab:intents} for details.
\section{Experiments}
\label{sec:experiments}
This section describes a user study to evaluate our system.
We also describe our baseline and evaluation metrics.
\subsection{Baseline}
We compare the CLIP-based landmark recognizer with a closed-vocabulary object detector as the baseline~\cite{zhou2022detecting}~\footnote{Open-vocabulary object detectors exist. We choose a closed-vocabulary detector to represent a common closed-vocabulary grounding model.}. The vocabulary size, or the number of classes, of the detector is more than $1200$ and it is fine-tuned with the same number of data as CLIP.
In the baseline, the landmark images are passed into the object detector, which outputs the class names of detected objects.
During navigation, the baseline chooses the landmark with the highest number of objects mentioned by the user.
Since the vocabulary of object detectors is fixed,
% ~\footnote{Although the object detector we use allows for a custom vocabulary, the object detector may not be able to handle all possible user expressions},
the baseline is unable to incorporate an object's attributes or locations obtained from disambiguation.
All other modules are the same for our system and the baseline.
\subsection{User Study}
% \subsubsection{Environment and routes}
\textbf{Environment:} % and routes:}
All experiments were conducted in an everyday indoor environment in a university building.
%, a map of which is shown in Fig.~\ref{fig:environment}.
% building.
%Our environment
% The environment included (1) a kitchen area; (2) a lounge; and (3) an exit.
Three routes were created with furniture acting as obstacles. %such as furniture.
Fig.~\ref{fig:environment} provides a layout of the environment and the three routes.
% The routes are designed to have varying levels of disambiguation for the end goals \ie~some routes had destinations with goals that required specific details that were needed for the system to correctly interpret the destination (\eg~Routes 2 \& 3).
The routes are designed to have varying levels of difficulties for the system to correctly interpret the destination.
Specifically, the goal landmark A of Route 1 contains simple objects, whereas landmark B of Route 2 contains more complicated objects, and landmark C of Route 3 contains a transparent door that is hard to recognize.
% \subsubsection{Participants}
\textbf{Participants:}
% The user study was conducted with N=5 participants (mean age=26; 3 males; 2 females; all participants were university students).
The user study was conducted with N=5 participants (mean age=26; 3 males; 2 females; all participants were university students).
All participants provided informed consent under IRB \#23565, which was approved by University of Illinois Urbana Champaign on February 27th, 2023.
% All participants were college educated with at least a bachelors degree.
All participants have full (corrected) vision and are asked to wear a blindfold to simulate a visual impairment.
While our true target population are PwVI, the purpose of this pilot study is to validate the capabilities of \modelName.
Running experiments with PwVI is left for future work.
% No participant had visual impairment due to the limited availability of the target population.
% Thus, all participants were asked to wear a blindfold during all trials to simulate a visual impairment.
% Participants were given time to adjust to the blindfold during the initial trial to familiarize themselves with the system.
\begin{figure}[tb!]
\centering
\includegraphics[width=\columnwidth]{Figures/overlap_map_sharp.png}
\caption{\textbf{The map of our environment with semantic landmarks.} The images are landmarks with locations marked by red dots. The orange lines are the three routes in the user study. The red squares are the starting locations of routes.}
\label{fig:environment}
\vspace{-5pt}
\end{figure}
\begin{table}[tb!]
\begin{center}
\caption{Example expressions and their corresponding landmarks from CLIP v.s. the detector. The landmark labels are from Fig.~\ref{fig:environment}. Underlined expressions are collected from the user study.}
\label{tab:grounding}
\begin{tabular}{c l l}
\toprule
\textbf{Landmark} & \multicolumn{1}{c}{\textbf{CLIP}} & \textbf{Detector} \\
\midrule
\multirow{2}{*}{A} & \underline{door}, \underline{exit}, \underline{entrance}, gate & poster \\
& glass door, automatic door & \\
\midrule
\multirow{4}{*}{B} & \underline{sofa}, \underline{couch}, \underline{coach}, & \underline{sofa}, \\
% & blue chair, soft chair, & thermostat \\
& fabric chair,relaxing chair & thermostat \\
& thermostat, climate control& \\
\midrule
\multirow{3}{*}{C} & \underline{sink}, \underline{think},\underline{sync}, \underline{faucet} & \underline{faucet} \\
& \underline{soap}, hand wash, water pipe & bottle \\
& paper towel dispenser, bowls & dispenser\\
& kitchen countertop, drying rack & bowl\\
\bottomrule
\end{tabular}
\end{center}
\vspace{-20pt}
\end{table}
% \subsubsection{Procedure}
\textbf{Procedure:}
Participants were first familiarized with the goals of the study and requested to fill a demographic and robot technology survey.
Then, participants were provided with a test run to get familiar with the system and its intricate navigation feedback mechanism.
To begin the trial, the users were asked to command the robot to take them to a predetermined goal destination.
Participants were not constrained in either the vocabulary or the sentence structure of their speech commands.
The users were also informed that they could interact with the robot (\eg~ask for a description of their surroundings) at any point of the navigation.
After each route, we used a short questionnaire
% \footnote[2]{All materials are linked on the project webpage.}
to measure the participant's perception of the system.
% We also requested users to fill out short questionnaires after each route.
% This questionnaire was used to measure the participants perception of the system as routes got more complex and challenging~\footnotemark[1].
% The questions in the short questionnaire asked the participants to rate the following quantities on a scale of 1 to 10:
% (1) Ease of following; (2) Navigational Experience; (3) Intent Understanding; (4) Intuitiveness of the vocal communication interface; (5) Gain awareness of the environment.
A strictly structured post-survey interview was conducted after participants finished all three routes to collect their feedback with the system.
The same procedure was performed for CLIP and the detector.
The order of which method was tested first was randomized for each participant to minimize the bias introduced due to the order of testing.
The full survey can be found at \href{https://github.com/Shuijing725/DRAGON_Wayfinding/blob/main/IRB_documents/questionnaires.pdf}{this link}.
% \todohere{Write Appendix with the questions asked} Standard Likert scale responses were requested for all questions~\footnote[1]{Further details regarding questions asked are provided in the Appendix.}.
% In total, completing all routes, questionnaires, and interviews take participants a maximum of 120 minutes.
% Participants.
% blind folded participants;
% justify blind folded participants.
% Ages, demographics etc
% \textbf{Participants:}
% Besides the difference in landmark recognition, all other modules are the same for our method and the baseline.
\subsection{Metrics}
% To conduct a thorough evaluation of our system, we use both objective and subjective metrics.
\textbf{Objective Metrics:}
% For objective metrics,
We measure the accuracy of all interactions during the user study, including $312$ NLU, $30$ landmark recognition (LR) and navigation trials, $15$ environment description (EnvDes), $74$ VQA, and $15$ navigation preference adjustment (NavAdj).
The NLU is considered correct when the extracted intent and entities (if any) are both correct.
We also measure the accuracy of the NLU by taking the correctness of SR into account to analyze the effect of wrong SR.
The effect of wrong NLU outputs is ignored when evaluating its downstream modules.
The correctness of answers from VQA is based on the camera images, not on the information out of the camera view.
To compare the two LR methods, we measure the success rates of LR and the resulting navigation.
% We also provide qualitative results and example failure cases of each module.
\textbf{Subjective metrics:}
% For subjective evaluation,
For both methods, we compare the scores for categories from the short questionnaire in Table~\ref{tab:routes}.
The difference in scores for each participant was aggregated and analyzed to discount individual biases.
% From the post-survey interview, we evaluate user preferences for other modules by applying a simple Likert scale analysis and summarize their feedback for qualitative analysis.
We evaluate user preferences for the other modules through a simple Likert scale analysis on the responses from the post-survey interview.
Additionally, participants' feedback is summarized for qualitative analysis.
% We evaluate user preferences for other modules in our system by applying simple likert scale analysis on the post survey interview.
% Modules that are common between our system and the baseline were evaluated in the post survey interview with standard likert scale analysis.
% \textbf{Metrics:}
% quantitative metrics for each submodule
% objective metrics
% subjective metrics (user ratings)
% Our system's dialogue capabilities are evaluated in terms of the percent accuracy of the SR and NLU modules in for recognizing the users intent from the speech input.
% Additionally, the descriptions of the environment provided to the user are also evaluated on their accuracy.
% As described in Section \ref{sec:img_caption_vqa}, the environment descriptions contain a number of objects and their identifications.
% Therefore, the accuracy of the descriptions formulated by these detections along with the accuracy of the VQA module are also evaluated.
% % motivate why we compare the navigation and mapping
% While our system differs from the baseline primarily with the inclusion of the semantic dialogue module% detailed in Section \ref{sec:semantic-dialogue},
% , downstream modules such as the mapping and navigation modules benefit from its inclusion.
% Hence, the two systems are compared with each modules success in recognizing landmarks as well as navigating to these landmarks.
% % The performance of our system and the baseline are compared in terms of their semantic dialogue and navigation capabilities.
% % difference in routes
% The routes for the user study were designed to have varying levels of disambiguation for the end goals \ie~some routes had destinations with goals that required specific details that were needed for the system to correctly interpret the destination (\eg~Route 1 \& 3).
% Therefore, the system initiated a conversation to disambiguate the requested destination.
% Conversely, for other routes the system did not need to ask further questions about the destination (\eg~Route 2).
% Table \ref{tab:intents} provides examples of all the possible destinations that could be associated with with a landmark.
% Therefore, in order to assess the utility of our disambiguation module, we compare the users experience in using our system and the baseline.
% In particular we analyze the difference in participants scores for the two systems.
% \begin{table}[tb!]
% \begin{center}
% \caption{Example expressions and their corresponding landmarks from CLIP v.s. the detector. The landmark labels are from Fig.~\ref{fig:environment}. Underlined expressions are collected from the user study.}
% \label{tab:grounding}
% \begin{tabular}{c l l}
% \toprule
% \textbf{Landmark} & \multicolumn{1}{c}{\textbf{CLIP}} & \textbf{Detector} \\
% \midrule
% \multirow{2}{*}{A} & \underline{door}, \underline{exit}, \underline{entrance}, gate & poster \\
% & glass door, automatic door & \\
% \midrule
% \multirow{4}{*}{B} & \underline{sofa}, \underline{couch}, \underline{coach}, & \underline{sofa}, \\
% % & blue chair, soft chair, & thermostat \\
% & fabric chair,relaxing chair & thermostat \\
% & thermostat, climate control& \\
% \midrule
% \multirow{3}{*}{C} & \underline{sink}, \underline{think},\underline{sync}, \underline{faucet} & \underline{faucet} \\
% & \underline{soap}, hand wash, water pipe & bottle \\
% & paper towel dispenser, bowls & dispenser\\
% & kitchen countertop, drying rack & bowl\\
% \bottomrule
% \end{tabular}
% \end{center}
% \vspace{-20pt}
% \end{table}
\begin{table*}[b]
\vspace{-10pt}
\begin{minipage}{0.4\linewidth}
\centering
\caption{Success rates (\%) of LR and navigation (including overall success rate, and success rate if LR is correct), and the average number of dialogue rounds for a successful LR.}
\label{tab:dialogue_metrics1}
\begin{tabular}{l c c c c}
\toprule
\multirow{2}{*}{\textbf{Method}}& \multicolumn{2}{c}{\textbf{LR}} & \multicolumn{2}{c}{\textbf{Navigation}}\\
\cmidrule(lr){2-3}\cmidrule(lr){4-5}
& Overall & \# rounds & Overall & Correct LR \\
\midrule
Ours & 100 & 2.4 &100&100\\
Baseline & 46.67 & 3.71 & 46.67 & 100 \\
\bottomrule
\end{tabular}
\end{minipage}\hfill
\begin{minipage}{0.59\linewidth}
\centering
\caption{Accuracies (\%) of the SR, NLU (including overall accuracy, accuracy if SR is correct and if SR is wrong), EnvDes with fully correct and partially correct number of objects, VQA, and navigation adjustment modules.}
\label{tab:dialogue_metrics2}
\begin{tabular}{c c c c c c c c}
\toprule
\multirow{2}{*}{\textbf{SR}} & \multicolumn{3}{c}{\textbf{NLU}} & \multicolumn{2}{c}{\textbf{EnvDes}} & \multirow{2}{*}{\textbf{VQA}} & \multirow{2}{*}{\textbf{NavAdj}} \\
\cmidrule(lr){2-4}\cmidrule(lr){5-6}
& Overall & Correct SR & Wrong SR & Full & Partial & & \\
\midrule
% multiline table
\multirow{2}{*}{70.19} &
\multirow{2}{*}{85.26} &
\multirow{2}{*}{93.61} &
\multirow{2}{*}{65.59} &
\multirow{2}{*}{45.45} &
\multirow{2}{*}{75.76} &
\multirow{2}{*}{82.43} &
\multirow{2}{*}{100} \\
\\
% single line table
% 70.19 &
% 85.26 &
% 93.61 &
% 65.59 &
% 45.45 &
% 75.76 &
% 82.43 &
% 100 \\
\bottomrule
\end{tabular}
\end{minipage}
% \vspace{-20pt}
\end{table*}
\begin{figure*}[t!]
% \vspace{-5pt}
\centering
\includegraphics[width=\linewidth]{Figures/slideshow_004_door.png}
\caption{\textbf{An example navigation trial with human-robot dialogue in the user study.} In the dialogue boxes, ``H'' denotes the human and ``R'' denotes the robot. The camera view is shown in the lower right corner. }
\label{fig:real_before_after}
\vspace{-15pt}
\end{figure*}
\section{Results}
\label{sec:results}
% \aamir{Discussion of results of the user study}
In this section, we discuss the results of our user study.
A demonstration can be found in the supplementary video with results from each module and navigation trials that test the whole system.
% In the video, after receiving a verbal command from the user, the robot successfully plans a path to the user-specified goals and executes the path smoothly without collisions or severe jerks.
% \aamir{add stuff about dialogue based on the video.}
% Tables \ref{tab:dialogue_metrics1} and Table \ref{tab:dialogue_metrics2} list the quantitative metrics used to evaluate our system and the baseline.
Fig.~\ref{fig:real_before_after} also provides an example trial along with the dialogue during the user study.
% \aamir{Table comparing the baseline and our method on our quantitative metrics: time to navigation; success rate; avg dialogue interactions etc}
% The metrics discussed in Section \ref{} and detailTables \ref{tab:dialogue_metrics} and \ref{tab:mapping_metrics}.
% \aamir{Describe participant feedback about their experience}
% \aamir{Discussion on Success of Mapping and Navigation. Table III}
%\aamir{Accuracy of dialogues. Table II and Table III}
% \aamir{quantitative metrics}.
\subsection{Quantitative Evaluation}
% The NLU module achieved an overall accuracy of 85.26\% (93.61\% with correct SR and 65.59\% with incorrect SR).
% \aamir{Cases of failure occured when..}
% Environment descriptions provided by the systems were 1\% accurate with 1 of x wrong descriptions.
% \aamir{example of wrong description and some insight.}
% The answers from the VQA module were correct 1\% of the times.
% \aamir{discussion on this result.}
% Table \ref{tab:dialogue_metrics1} and Table \ref{tab:dialogue_metrics2} list the quantitative metrics comparing our system with its baseline and the performance of the different modules, respectively.
% Table \ref{tab:dialogue_metrics1} lists the quantitative metrics comparing our system with its baseline.
% \subsubsection{LR and navigation}
\textbf{LR and navigation:}
As seen in Table~\ref{tab:dialogue_metrics1}, the success rate of navigation is 100\% if LR succeeds.
This dependency indicates that the performance of LR is the key factor for navigation in the \modelName system.
% Additionally, as shown in Table~\ref{tab:dialogue_metrics1}, with a small set of finetuning data, our CLIP model with disambiguation achieves 100\% success rate in LR and navigation with fewer rounds of dialogue on average, both of which outperform the YOLO baseline.
For LR, as shown in Table~\ref{tab:dialogue_metrics1}, our CLIP model with disambiguation outperforms the detector baseline by achieving 100\% success rate in LR and navigation with fewer rounds of dialogue on average.
We attribute this result to the fact that CLIP is an open vocabulary model that can take free-form query text, which is essential for our task because the user may use different expressions to refer to the same landmark.
On the contrary, a closed vocabulary object detector can only handle a fixed set of object classes with limited expressions.
For example, in Table~\ref{tab:grounding}, although both models can handle different objects that belong to the same landmark, CLIP can associate synonyms, such as ``sofa'' and ``couch'', and wrong transcriptions, such as ``coach'', to the correct landmark.
In contrast, the closed-vocabulary detector can only handle strictly fixed expressions.
The detector misidentifies some objects such as the transparent door in Landmark C after fine-tuning.
Since our target users are usually non-experts, the baseline sometimes needs the user to rephrase multiple times to recognize the goal, which causes the user to run out of patience, and results in failure or more rounds of dialogue.
% In contrast, after fine-tuning with a small dataset, CLIP is less sensitive to rare objects.
Besides CLIP, the disambiguation dialogue also contributes to the performance. With disambiguation, additional information such as the material and functionality of objects can be merged into the query text, such as ``fabric chair'' and ``relaxing chair'' as shown in Table~\ref{tab:grounding}.
These rich descriptions are helpful in distinguishing landmarks that have the same objects with different attributes, such as the different types of chairs in Landmark A, D, and E in Fig.~\ref{fig:environment} with fewer rounds of user rephrasing.
% Our method sometimes requires more than two rounds (to extract and confirm intent) due to incorrect SR (\eg~``couch'' transcribed as ``coach'').
% Table IV (quan) + Table III (qual)
% \subsubsection{NLU}
\textbf{NLU:}
In Table \ref{tab:dialogue_metrics2}, the overall accuracy of NLU is over 15\% higher than SR, as the NLU is trained with incorrectly transcribed text and thus can work even when SR is incorrect.
However, we do notice that NLU performs better with correct SR.
The common failure cases of NLU occur when (1) The SR mistakenly breaks a sentence into two halves (\eg\: ``Is there anything?'' and ``To my right.'' are treated as two sentences); and (2) The NLU does not correctly extract intents from noisy transcriptions and chitchat, which can happen during the user study.
% As seen in Table
% shows the quantitative metrics evaluating the other modules in our system.
% Incorrect transcriptions lead to the misclassification of user intent by the NLU.
% For example, ``on it? (previous sentence) Is it (the sofa) in front of me?" is misclassified as \textit{Describe} intent while it should have been an \textit{Ask} intent due to the extra words at the beginning.
% In cases, where the SR is accurate, the NLU performs much better and achieves an accuracy improvement of 27\%.
Thus, we believe that a better SR engine would vastly benefit the performance of the whole system.
However, since \modelName will not begin navigation until the user confirms the goal in the dialogue, the wrong SR and NLU have little effect on navigation.
% \subsubsection{Other modules}
% Our system achieves a higher accuracy in LR over the baseline, while taking fewer interactions with the user to confirm their destination, as noted in Table \ref{tab:dialogue_metrics1}.
% The failure cases for the baseline in unsuccessful LR were restricted to situations where either the SR module provided incorrect transcriptions, or failures in object recognition that led to incorrect identification of objects requested by the user (\eg~paper towels and soap near the sink were misrecognized).
% The failure cases for the baseline in unsuccessful LR were restricted to situations where incorrect object recognition led to failures in identification of objects requested by the user (\eg~paper towels and soap near the sink were misrecognized).
% Table \ref{tab:grounding} provides details regarding the object keywords associated with each landmark by the CLIP model (Our system's LR) and Object detector (baseline LR).
% These failures also led to an increase in the number of interactions with the participant to correctly recognized the destination.
% Meanwhile, our system sometimes required more rounds of interaction to disambiguate the users intent due to incorrect SR (\eg~`couch' transcribed as `coach'), compared to typical two rounds of interaction to extract and confirm the intent.
% Meanwhile,
% Both systems perform similarly in terms of navigation with successful LR.
% As seen in Table \ref{tab:dialogue_metrics1}, the baseline was unsuccessful in navigation only when it failed to recognize the intended landmark.
\textbf{Other Modules:}
% The environment descriptions provided by the system depend on the object detection module.
The system's environment descriptions are sometimes inaccurate due to errors in the object detector such as: (1) detecting incorrect number of objects (\eg~3 wall sockets, when there was only 1 present); and (2) incorrect object classifications of rare or uncommon objects (\eg~a building information tablet was classified as a poster).
Although we use non-maximum suppression and confidence score threshold to reduce the errors, they are hard to entirely eliminate due to the data distribution shift and the blurry images caused by the robot motion.
Nevertheless, in Table~\ref{tab:dialogue_metrics2}, the model is able to output a list of objects with correct class names in $75.76\%$ of the cases, which might be more important to the user than a correct number of objects.
The VQA module accurately answers the user's questions in 82.43\% of the cases.
The model fails in cases where the user asks questions that the robot cannot answer based on a single RGB image.
For example, without precise depth information the VQA model only answers ``far'' or ``close'' if the question is ``How far is the person from me?''. Without a wider field of view, the model outputs objects on the front side if the question is ``What is on my right?''.
\subsection{Qualitative Evaluation}
In Table \ref{tab:routes}, participants showed an increasing preference for \modelName with CLIP over the detector in all user experience categories across all routes. %Route 1 to 3.
% Fig. \ref{fig:routes_disam}).
% \aamir{General user experience with the robot systems.}
Specifically, participants reported a 32\% improvement with a mean score difference of $1.60 \pm 0.89$ in the overall experience and a mean score difference of $1.40 \pm 0.89$ in the communication experience.
The difference increases as the goal landmark contains more complicated objects in Route 2, and objects that are difficult to detect in Route 3, where the failures in LR significantly lower the user score for the detector based system.
Particularly, participants noted that \modelName with CLIP understood their intent, asked good follow-up questions, and correctly guided them to their destination. In contrast, the closed-vocabulary detector failed at these aspects and occasionally was unable to recognize destinations even though they existed.
Participants also noted that the failures in intent understanding led to a frustrating communication experience with the detector.
% This difference indicates that a good LR plays an important role in the user experience of the whole system.
% However, users also noted that both systems failed to identify particular details in the environment (\eg~what is on the right side) and that the response time for the communication was slower than desired.
% Our system was rated higher in both its navigational and dialogue capabilities, achieving a 28\% improvement in both categories with a mean score difference of $1.40 \pm 1.15$ and $1.40 \pm 0.89$, respectively.
% % All participants reported that the the semantic dialogue and VQA capabilities of our system were more user friendly.
% All participants noted that the intent understanding of our system was superior to that of the baseline.
% In particular, a user mentioned that our system ``... was able to actually understand me, so it accurately took me to the location and correctly answer[sic] my questions." while the baseline ``... would confirm the location I wanted to go to but could not find(read understand) the right location".
% Participants found that the baseline was more mentally taxing to use, rating our system 24\% higher (mean difference of $1.2 \pm 1.3$ in scores).
% We conjecture that this significant difference is due to repeated failures in LR in Route 3, as well as occasional failures in LR in Route 2.
One user in particular mentioned that the CLIP based model ``... was able to actually understand me, so it accurately took me to the location and correctly answer~[sic] my questions." while the detector based model ``... would confirm the location I wanted to go to but could not find~[sic; participant meant understand] the right location".
However, users also mentioned potential improvements for \modelName including more detailed descriptions of the environment, a quicker response time, and warnings of potential dangers such as ``We're going through a door.''
% In general, most users mentioned that they would prefer if the robot communicated with them more and provided more descriptions about the environment.
% Some users suggested that the robot could describe ``potential danger when its navigating".
% Additionally, users mentioned that ``it would be nice if the robot kept communicating with me to tell me things like `almost there' or `We're going through a door'" and ``if it could say when it was starting to rotate".
% There was no difference found in the physical load placed on the participants by the two systems.
% \aamir{User experience with the routes and the disam.}
% \textbf{Disambiguation module:}
% Figure \ref{fig:routes_disam} shows the average of the difference in scores for each question in the short questionnaire.
% Participants rate our system consistently higher for routes where disambiguation was necessary as shown in Fig. \ref{fig:routes_disam}.
% In particular, users rated our system higher than the baseline in all user experience categories (\eg~the ease of following and intent understanding).
% Route 3 is rated particularly higher as the baseline failed to begin navigation due to a failure in LR as mentioned earlier.
For the user experience categories that are the same for both LR methods, such as the `intuitiveness of communication interface' and the ability of the system to aid in `gaining awareness of the environment', participants reported average scores of $7.07 \pm 2.17$ and $6.07 \pm 3.21$, respectively.
As evidenced by these scores, the users' opinions regarding these two categories were positive, due to the inclusion of the dialogue and grounding modules.
%These scores indicate that the users were generally satisfied with the audio communication interface and the environment descriptions provided to them.
However, participants highlighted minor inaccuracies in the environment descriptions and the slow pace of communication due to processing times and network delays as potential issues.
% \begin{figure}[tb!]
% \centering
% \includegraphics[width=\columnwidth]{Figures/routes_scores.png}
% \caption{Average difference between the scores of our system and the baseline for the questions asked in the in-between questionnaire.}
% \label{fig:routes_disam}
% %\vspace{-20pt}
% \end{figure}
\begin{table}[t]
\centering
\caption{Mean user experience scores on a scale of 1 to 10.}
\label{tab:routes}
\resizebox{\linewidth}{!}{\begin{tabular}{l cc cc cc}
\toprule
\multirow{2}{*}{\textbf{Use experience category}}& \multicolumn{2}{c}{\textbf{Route 1}} & \multicolumn{2}{c}{\textbf{Route 2}} & \multicolumn{2}{c}{\textbf{Route 3}} \\
\cmidrule(lr){2-3}\cmidrule(lr){4-5}\cmidrule(lr){6-7}
& CLIP & Detector & CLIP & Detector & CLIP & Detector \\
\midrule
Ease of following &
\textbf{8.8} &
8.6 &
\textbf{8.8} &
5.6 &
\textbf{9.2} &
1.0
\\
Navigational Experience &
\textbf{8.4} &
7.4 &
\textbf{7.6} &
4.8 &
\textbf{8.8} &
1.0
\\
Intent Understanding &
7.6 &
\textbf{8} &
\textbf{7.6} &
4.6 &
\textbf{8.4} &
3.4
\\
% \midrule
% Intuitiveness of communication &
% \multicolumn{2}{c}{7.7} &
% \multicolumn{2}{c}{7} &
% \multicolumn{2}{c}{6.5} \\
% Gain awareness of environment &
% \multicolumn{2}{c}{7.6} &
% \multicolumn{2}{c}{6} &
% \multicolumn{2}{c}{4.6} \\
\bottomrule
\end{tabular}}
\vspace{-15pt}
\end{table}
% \aamir{Separate from future work, this is about the improvements that the users gave that can be addressed in the system.}
% \vspace{-0.1in}
% \section{Conclusions, Limitations and Future Work}
\section{Conclusion and Future Work}
% \vspace{-0.05in}
\label{sec:conclusion}
In conclusion, we present \modelName, a first-of-its-kind guide robot that fulfills user intents and familiarizes the user with their surroundings through interactive dialogue.
We use CLIP to retrieve landmark destinations from commands and provide visual information through language.
The user study shows promising communication, grounding, and navigation performance of \modelName.
Our work suggests that visual-language grounding and dialogue can greatly improve human-robot interaction.
% Extending \modelName's results past our simplifying assumptions, future work should develop guiding-dialogues that alleviate the following limitations.
To extend \modelName and address its limitations, we point out the following directions for future work.
First, the current dialogue system is rule-based with fixed behaviors for each intent.
Replacing hard-coded rules with adaptive learning-based policies should generalize to more complex user behaviors and more subtasks.
Second, the environment understanding modules provide limited information. Future informative descriptions should include object relationships in images, incorporate information from the map and other sensors, and inform users about the planned path and potential dangers.
Finally, the physical interface of the platform should be redesigned to improve the user experience.
% The \modelName system demonstrates the feasibility of dialogue-based assistive navigation that future research in dialogue generation and robotics can explore further.
\modelName demonstrates the feasibility of dialogue and visual-language grounding in assistive navigation that future research in dialogue management, computer vision, and robotics can explore further.
\section*{Acknowledgements}
This work was supported by a Research Support Award from the University of Illinois Urbana-Champaign Campus Research Board.
We thank Dr. Samuel Olatunji and Dr. Peter Du for their help on the user study, Ye-Ji Mun and Zhe Huang and Neeloy Chakraborty for their feedback on the paper draft, and all participants in the user study.
% Limitations \& future work ideas: \\
% Hardware:
% \begin{enumerate}
% \item camera field of view is small, which limits the information that the robot perceives $\rightarrow$ a better camera/groups of cameras
% \item reaction time is slow $\rightarrow$ more powerful computer
% \item handle is not stable $\rightarrow$ design a better handle and a more stable mounting point
% \end{enumerate}
% Software:
% \begin{enumerate}
% \item The current dialogue system is rule-based $\rightarrow$ Use learning-based dialogue management to generalize to more complex user behaviors and more tasks.
% \item Better environment description and question answering: describe object relationships instead of listing objects, use video instead of single image as input, incorporate information from the map and other sensors, inform users about the planned path and potential dangers
% \item Take the position and movement of users into account during planning and navigation
% \item (experiment) User study with true blind people
% \end{enumerate}
\documentclass[letterpaper, 10 pt, conference]{ieeeconf} % Comment this line out
% if you need a4paper
%\documentclass[a4paper, 10pt, conference]{ieeeconf} % Use this line for a4
% paper
\IEEEoverridecommandlockouts % This command is only
% needed if you want to
% use the \thanks command
\overrideIEEEmargins
%%%% useful packages %%%%
\input{macros.tex}
\title{Designing a Wayfinding Robot for People with Visual Impairments
}
\author{Shuijing Liu*, Aamir Hasan*, Kaiwen Hong, Chun-Kai Yao, \\ Justin Lin, Weihang Liang, and Katherine Driggs-Campbell \thanks{* denotes equal contribution}
\thanks{S. Liu, A. Hasan, K. Hong, C. K. Yao, J. Lin, W. Liang, and K. Driggs-Campbell are with the Department of Electrical and Computer Engineering at the University of Illinois at Urbana-Champaign. emails: \texttt{\{sliu105, aamirh2, kaiwen2, ckyao2, jklin6, weihang2, krdc\}@illinois.edu}}
}
\begin{document}
\maketitle
\thispagestyle{empty}
\pagestyle{empty}
\begin{abstract}
People with visual impairments (PwVI) often have difficulties navigating through unfamiliar indoor environments.
However, current wayfinding tools such as white canes and guide dogs are fairly limited.
In this short paper, we present our in-progress work on the design of a wayfinding robot for PwVI.
The robot takes an audio command from the user that specifies the intended destination.
Then, the robot autonomously plans a path to navigate to the goal.
The planned path takes the real-time position of the user into account to avoid collisions for both the user and the robot and to provide a more comfortable motion path to the goal.
In addition, the robot describes the surroundings to the user periodically to prevent disorientation and potential accidents.
We demonstrate the feasibility of our design in an indoor home environment.
Finally, we analyze the limitations of our current design, as well as our insights and future plans to improve this work.
\end{abstract}
% sections
\input{Sections/1-Intro}
\vspace{-0.1in}
\section{Methodology}
\vspace{-0.05in}
\label{sec:platform}
In this section, we describe the setup and configuration of our robot guide, as shown in Fig.~\ref{fig:overview}. We also describe and briefly mention planned improvements for each of the four modules of our design: the estimation of the user's pose, the planning module which considers the estimated pose, scene description, and audio communication.
\subsection{Robot platform}
% describe the robot: camera, lidar, nuc, arm, moving base
% \textbf{Robot platform:}
We use a Stretch RE1 as our robot platform~\cite{stretch}.
The Stretch robot is a lightweight robot with a mobile base and a extendable manipulator,
An Intel RealSense D435i camera is mounted toward the user on the top of the robot body to detect the user and estimate its position and orientations.
An RP-Lidar A1 laser range finder is mounted on the top of the mobile base to perform simultaneous localization and mapping (SLAM) for navigation.
A microphone array is used for the robot to communicate with the user.
The arm serves as a holding point for the user's arm and the mobile base performs movements to guide the user in autonomous navigation.
An additional Intel RealSense D445i camera was added to aide the robot in having a full view of the environment.
% We adjust the robot speed to match the average walking pace of PwVI.
\vspace{-0.05in}
\subsection{User pose estimation}
\label{sec:pose_est}
%%%%%%%%%%%%%% shorter version %%%%%%%%%%%%%%%%
% The usual robot navigation pipeline only considers the collision avoidance of the robot but not the user, which will result in collisions or discomfort of the user.
% For this reason, we need to extract the user's position and orientation to improve the path planner.
% In the first step, we use Cross-Domain Complementary Learning~\cite{lin2020cross} to segment the user's body trunk on the camera RGB image from the background. Once the human trunk segment is identified, we use the depth values of the trunk segment on the depth image to estimate the user's y-coordinate and facing angle in the camera frame using trigonometry.
% For the x-coordinate, we first calibrate the camera parameters, which are used to approximate the offset distance of the human trunk from the center of the camera and compute the x-coordinate of the user.
% In the last step, we treat the boundary of the user as a rectangle and compute the coordinates of the vertices of the rectangle in the camera frame with a homogeneous transformation between the camera's frame and the user's frame.
The usual robot navigation pipeline only considers the collision avoidance of the robot but not the user, which will result in collisions or discomfort of the user.
For this reason, we need to extract the user's position and orientation to improve the path planner.
In the first step, we use Cross-Domain Complementary Learning~\cite{lin2020cross} to process the RGB images from the camera and segment the user's torso on the image from the background.
Once the human torso segment is identified, we use the depth values of the torso segment on the depth image to estimate the user's facing angle in the camera frame using linear regression and trigonometry.
In addition, the user's y-coordinate can be directly calculated by taking the average of the depth values inside the human body torso area.
For the x-coordinate, we first measure the ratio between the pixel-pixel distance in the image and the real-world distance through calibration.
Then, we use this ratio to approximate the offset distance of the human torso from the center of the camera and compute the x-coordinate of the user based on the camera's pose.
In the last step, we treat the boundary of the user as a rectangle and compute the coordinates of the vertices of the rectangle in the camera frame with a homogeneous transformation between the camera's frame and the user's frame.
\todohere{future work/improvements?}
\vspace{-0.05in}
\subsection{Pose-aided Navigation}
Before navigation, the robot needs to map the environment using SLAM. % with its LiDAR.
We use the Gmapping ROS package~\cite{gmapping} for SLAM, where the robot estimates its location using its wheel odometry, LiDAR scans, and adaptative Monte Carlo localization (AMCL)~\cite{pfaff2006robust}.
To plan a path for the robot-human team based on the map, we use the A* algorithm to plan a global path based on user's current position and the user-specified destination.
We use the dynamic window approach (DWA) as the local path planner~\cite{fox1997dynamic}.
During local planning, to incorporate pose information into the DWA planner, we change the collision boundary of the robot so that it also includes the estimated boundary of the user from Sec.~\ref{sec:pose_est}.
More specifically, given the estimated pose of human, we generate a boundary box on the user's reachable spaces and use a polygon to represent the box in the robot frame, which are fed into the DWA planner.
As a result, the planned paths avoids collisions for both the user and the robot in real-time.
For future work, we can use data-driven methods to predict human's motion when the robot is performing a turn or a sudden stop to improve the comfort and safety of the user.
\vspace{-0.05in}
\subsection{Scene description}
% intro: based on user study, PwVI frequently get disoriented in nav
Based on a recent user study, PwVIs frequently get disoriented during navigation, especially in large and open spaces~\cite{jeamwatthanachai2019indoor}. % Use image captioning models to inform them the landmarks -> help them remember the layout of the space & locate themselves
To help PwVIs remember the layout of the space and locate themselves, our design uses a state-of-the-art image captioning method~\cite{rennie2017self} to inform the user of its surroundings.
% inputs, realsense camera & outputs, microphone
The image captioning model takes an RGB image from the camera as the input, and outputs a sentence that describes the main objects and their relations in the image.
The model is a deep neural network trained using a dataset of images and their captions.
% Then, a text-to-speed software utters the output sentence using a microphone on the host computer~\cite{tts}. This process repeats periodically until the end of the navigation.
This process repeats periodically until the end of the navigation.
% briefly introduce the self-critical paper (1-2 sentences, don't dive into tech details!)
% how we adept it to our application
% future work:
% 1. give priority to potentially dangerous objects, such as moving people/pets and furniture that moved recently
% 2. incorporate info from other sensors (e.x. depth image) for a more complete detection
For future work, our description sentences will give higher priorities to objects that are potentially more dangerous to the user such as tripping hazards or mobile obstacles such as other people or pets.
Besides the RGB images, we plan to use other sensors such as depth images as complements to enrich the information in the descriptions.
\vspace{-0.05in}
\subsection{Audio communication}
Due to the nature of their disability, many PwVI tend to bump into objects or get lost while navigating an unfamiliar space.
A user study found that some PwVI prefer to have additional knowledge of their environment as they are navigated so as to maintain a mental model of their surroundings~\cite{megan_study}.
Additionally, the participants in the same study mentioned feeling most comfortable while verbally communicating with a robot over methods such as using phone apps or buttons.
Hence, we developed an audio communication module that takes in the users input to navigate them to a requested destination, in addition to communicating the output from the scene description module.
Our current module uses a simple speech-to-text extractor~\cite{stt} with a bag-of-words model to extract the requested destination within the testing location.
We use a text-to-speed library to ``speak'' the output sentence from the scene description module using a speaker on the robot~\cite{tts}.
We plan on extending this module by improving from a bag-of-words model to a data-driven deep learning based model to extract the destination location from the input speech snippet.
Such a model would be able to better extract the destination in more complex sentences based on the context of the sentence than a bag-of-words model.
% \subsection{Experiment setup}
% describe the guiding configuration: copy from HRI paper
% \textbf{Experiment Details:}
\section{Experiment Details}
Our experiments are conducted in a research facility that mimics a modern home.
As our design is still in development, we opted to test the robot with a research fellow in a blindfold rather than a PwVI.
In our experiments, to receive accurate real-time guidance, the user held on to the arm of the Stretch robot with a perpendicular configuration as seen in Fig. \ref{fig:overview}.
The user verbally requests the robot to guide them to a destination within the home.
The robot parses the user's request and guides them to their requested destination while monitoring their pose to avoid collisions with objects in the environment.
During navigation, the robot also describes important details about the environment such as the presence of rugs or stools in the navigation path.
% \input{Sections/4-Exp}
\input{Sections/5-Conclusion}
% references
\newpage
\clearpage
\bibliographystyle{IEEEtran}
\bibliography{BibFile}
\end{document}
\section{Experiments}
\label{sec::experiments}
LLfN is tested in simulated and physical experiments. We hypothesize that through LLfN (1) navigation performance can improve as the robot gathers more experience within a single environment, and (2) navigation in new environments can be learned while not forgetting how to navigate well in previous ones.
\subsection{Robot Platform and Implementation}
\label{subsec::implementation}
Clearpath Jackal, a four-wheeled differential-drive unmanned ground vehicle, is used for both simulated and physical experiments. The robot is equipped with a laser scanner to perceive surrounding obstacles and runs the basic Robot Operating System (ROS) \texttt{move\textunderscore base} navigation stack. While the global planner is based on Dijkstra's algorithm \cite{dijkstra1959note}, the local planner uses DWA \cite{fox1997dynamic}, a sampling-based planner which may fail to sample feasible actions if not being properly tuned for the deployment environment. We fix the global planner and improve the local DWA planner using LLfN. We include the local goal provided by the global planner as part of the state, along with LiDAR input. Based on the local DWA planner as an initial policy ($\pi_0$), LLfN learns to complement this $\pi_0$ and improve navigation performance in multiple environments. When the sampling-based DWA cannot find feasible motion, it starts recovery behavior, including rotation in place or backing up \cite{zheng2017ros}. LLfN first improves navigation by eliminating those sub-optimal behaviors and producing alternative motions for a given environment. Second, it allows the robot to adapt to new environments while still remembering previous ones.
% \peter{Have we introduced the notion of a global and local planner yet in this paper? Does global appear in the algorithm?}
The learning problem is formulated as finding a policy $\pi_\theta$ that maps from the current state $s$, which includes LiDAR input (720- and 2095-dimensional for simulated and physical experiments, respectively) and local goal ($\left(x, y\right)$, 1m away on the global path),
% \commentp{that's too informal a specification of the state space. Be precise, give the dimensionality, etc.}
to the action $a$, i.e. the linear velocity $v$ and angular velocity $\omega$.
In our implementation, the scoring function $D$ prioritizes actions proposed by DWA, if they result in a steady forward motion ($v\geq0.15\text{m/s}$, which is slightly larger than the minimal $v=0.1\text{m/s}$ to overcome friction). Importantly, all recovery behaviors from the DWA planner, including slowing down, turning in place, and moving backward, result in $v < 0.15\text{m/s}$. So with the above definition, $D$ will classify all recovery behaviors as suboptimal.
% This choice of implementation is due to the fact that DWA with default parameters is a relatively conservative algorithm, which only produces steady forward motion when it thinks the motion is absolutely safe.
Future works can investigate other implementations of $D$, or even learning it on the fly. If DWA fails to find feasible actions and would have started to execute recovery behaviors, the learned policy $\pi_\theta$ takes over. We acknowledge this implementation makes the scoring function $D$ not always accurate, e.g. it is possible that $D$ may prefer a bad action by $\pi_\theta$ when $\pi_0$ produces recovery behaviors. But in practice we observe LLfN remains effective even with such a simple $D$.
In consideration of limited onboard resources, we implement a streaming buffer $\mathcal{B}_\text{stream}$ as a regular queue to store 300 online streaming data points sequentially, and a separate memory $\mathcal{M}_\text{k}$ as a priority queue to save exemplar training data in the current environment. The number of data points within $\mathcal{B}\cup \{\mathcal{M}_k\}$ is also constrained to 300. The buffer sizes are empirically determined as the minimal value to assure efficient learning. For every sub-optimal action generated by DWA at a certain state, we compute a similarity score between this state and all states in the streaming buffer with a successful action (L2-norm of the LiDAR reading difference). We replace the data point of lowest similarity score in the training buffer with that from the streaming buffer.
% For duplicated entries, we simply update the similarity score.
% \peter{This detail also seems like it should have been introduced earlier. The actually similarity score is an implementation detail. But the way you get training data and trade off between the controller and the learned policy are aspects of the general algorithm, and should be introduced earlier.} \xuesu{see the comment above to add one more component}
The onboard data memory overhead to implement LLfN is \emph{at most} a total of 600 data points.
\subsection{Simulated Experiments}
The effect of LLfN is first studied with extensive simulated trials. The simulated Jackal has a SICK LMS111 laser scanner onboard providing 270$^{\circ}$ 720-dimensional laser scan. The three simulated navigation environments are shown in Figure \ref{fig::sim_tasks}, where the robot navigates from a fixed start to a fixed goal in the environment.
Env. 1's dense obstacles require fast response for obstacle avoidance; In Env. 2, the robot needs to keep a slow and consistent pace to drive through the narrow passage; Env. 3 requires the robot to slow down to make a sharp turn to enter the other room smoothly.
The robot is placed at its start location in each environment, and the learned policy $\pi_\theta$ to be tested is selected manually.
\begin{figure}
\centering
\subfloat[Env. 1]{\includegraphics[height=1.33cm]{figures/Task1.jpg}%
\label{fig::task1}}
\hspace{2pt}
\subfloat[Env. 2]{\includegraphics[height=1.33cm]{figures/Task2.jpg}%
\label{fig::task2}}
\hspace{2pt}
\subfloat[Env. 3]{\includegraphics[height=1.33cm]{figures/Task3.jpg}%
\label{fig::task3}}
\caption{Simulated Navigation Environments: Green segments are primarily traversed using the initial policy $\pi_0$, while red segments are mostly traversed using the learned planner $\pi_\theta$. }
\label{fig::sim_tasks}
\vspace{-10pt}
\end{figure}
% \commentp{In the figure, it looks like the robot gets teleported from one environment to the next. That's not the image I had from previous text and Figure 1 where it looks like there's some smooth transition from one environment to the next, but somehow the agent knows when it happens. Also, remember you said the agent goes from one "end" of an environment to another, but that's not what it looks like in this figure.}
We use the initial policy $\pi_0$ to execute three trials in each of the three navigation environments to collect the training data. $\pi_0$ (DWA) can eventually navigate through, given sufficient time to recover, re-sample, and re-plan.
% \commentp{This assumes that the initial policy can succeed at navigating through the whole environment, right?}
To test in-environment learning improvement, training data of each given environment is \emph{divided into} 5 segments and incrementally presented to the learner.
For example, the first training buffer for learning is constructed from the first one-fifth of the robot's experience, the second from the first two-fifths, etc. The last training buffer is from the entire experience. Note all these training buffers contain only 300 data points, who have the highest similarities to those states where the initial policy performs sub-optimally, given the presented navigation experience.
% This results in training buffers of 300\peter{I don’t understand what you’re saying here.} data points with the highest similarities to those observations where the initial policy performs sub-optimally, given the presented navigation experience.
Therefore, 6 incrementally learned navigation policies are produced, including the initial policy and the five policies learned from seeing 1/5, 2/5, ..., 5/5 of the training data from three training trials. Then the robot starts to learn the next navigation environment. For cross-environment learning, we use Sequential Training and the proposed LLfN. Sequential Training first uses the training buffer of environment 1. Starting with the final policy trained from environment 1, it sequentially trains on the training buffer of environment 2, before moving on to the training buffer of environment 3. In LLfN, while training environment 2, it only uses 150 data points from training buffer 2 with the highest similarity score, and still keeps a memory of 150 data points with the highest similarity score from training buffer 1 to assure new gradient updates won't increase the loss of environment 1. While training on environment 3, 100 data points with the highest similarity score for each environment are used, to avoid forgetting environment 1 and 2.
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{figures/sim_test.png}
\caption{Simulation Results: Each subplot (row $i$, column $j$) shows navigation performance with respect to increasing experience (x-axis), being trained using the $j$th and deployed in the $i$th environment. (Since there is no previous environment for Env. 1 and Train 1, both curves are the same.)}
\label{fig::sim_test}
\vspace{-10pt}
\end{figure}
We implement a small neural network of three hidden layers with 64 hidden neurons each, to compute linear and angular velocity based on the LiDAR input and local goal. After training, we evaluate the navigation performance in terms of traversal time in the current and previous environments. Each policy is executed three times, resulting in a total of 198 evaluation trials. We report the mean and standard deviation of the performance in Figure \ref{fig::sim_test}. If the robot is not able to reach the goal, e.g. gets stuck, a penalty time of 100s is given. Within each environment, LLfH is able to decrease traversal time with increasing learning experience. Across environments, LLfH learns new environments with increasing data while avoids catastrophic forgetting of previous ones. Sequential Training can improve navigation performance of a given environment when being presented the data from that particular environment, faster than LLfN. However, navigation performance in previous environments deteriorates with increasing experience in the current environment. The catastrophic forgetting is apparent in the diverging red line (Sequential Training) from the green line (LLfH). Note that the learned policies are used in conjunction with the initial policy in both cases.
\subsection{Physical Experiments}
LLfN is also implemented on a physical Jackal robot. In the physical experiments (Figure \ref{fig::three_tasks}), we use the same setup, while all computation is done onboard the robot using an Intel Core i5-4570TE CPU. The physical Jackal has a Velodyne LiDAR, whose 3D point cloud data is converted to 360$^{\circ}$ 2095-dimensional laser scan. We design the physical test environments such that the goal of one environment smoothly transitions to the start of the next. Therefore the robot can traverse through all three environments in one shot. For training, we manually label the current environment for the learner. During deployment, the robot uses only one $\pi_\theta$ for each traversal.
In the physical experiments, we compare LLfN with three baselines: DWA, Sequential Training, and Individual Models.
% \peter{That’s the right way to say it. But then in Table 1, it again looks like GEM is what’s being compared.}
Sequential Training and LLfN are conducted in the same way as in the simulated experiments. Individual Models are checkpoints of Sequentially Trained models up to the corresponding environment. Since these Individual Models do not suffer from forgetting, they serve as the \emph{best} models in Sequential Training. In contrast, the Sequentially Trained model is the final model after training on all 3 environments (thus ends up being much more effective on Environment 3 than Environment 1 or 2). The downside of Individual Models is that the number of models increases with the number of environments. Thanks to the constraint on data size, training each model takes less than two minutes on the robot's onboard CPU.
% \peter{The other thing that comes across to me gradually - really only in the experimental results section - is the fact that the default controller is used most of the time and the learned model is very sparsely used. Presumably that’s why you can get away with so little data. It’s a very specialized model. I still find it confusing how much data is used to train the discriminator and the control policy, and exactly how they interact. Actually, I’m now not sure that you have 2 different models. I will go back and try figure it out. But it needs to come out more clearly. Another thing that could be useful to show is where along the trajectories in the environments the default controller and the learned policy are used. How often is the policy invoked? That would have to go in supplementary material, but at least in simulation, is it possible to illustrate?}
% \begin{wraptable}{r}{0.6\textwidth}
\begin{table}[t]
\centering
\scalebox{0.85}{
\begin{tabular}{cccccc}
\toprule
& & \textbf{DWA} & \textbf{\begin{tabular}[c]{@{}c@{}}Sequential \\ Training\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}LLfN\end{tabular}} & \textbf{\begin{tabular}[c]{@{}c@{}}Individual\\ Models\end{tabular}} \\ \midrule
\multicolumn{1}{c}{\multirow{2}{*}{\textbf{Env. 1}}} & Time & 38.36$\pm$1.69 & 39.11$\pm$6.45 & \textbf{28.32}$\pm$1.85 & 30.17$\pm$\textbf{0.97}\\
\multicolumn{1}{c}{} & Rec./Col. & 1.8/0 & 1.4/0.6 & 0/0 & 0/0\\ \midrule
\multicolumn{1}{c}{\multirow{2}{*}{\textbf{Env. 2}}} & Time & 28.05$\pm$3.48 & 49.41$\pm$ 13.94 & \textbf{19.98}$\pm$1.07 & 23.41$\pm$\textbf{0.66}\\
\multicolumn{1}{c}{} & Rec./Col. & 1.4/0.2 & 1.25/0.8 & 0/0 & 0/0 \\ \midrule
\multicolumn{1}{c}{\multirow{2}{*}{\textbf{Env. 3}}} & Time & 39.80$\pm$6.39 & 21.12$\pm$1.17 & 21.80$\pm$1.51 & \textbf{21.12}$\pm$\textbf{1.17} \\
\multicolumn{1}{c}{} & Rec./Col. & 2.2/0.2 & 0/0 & 0/0 & 0/0\\ \bottomrule
\end{tabular}}
\caption{~Physical Results after Training on 3 Environments: Time (in s), Number of Recovery Behaviors (Rec.) and Collisions (Col.).}
\vspace{-5pt}
\label{tab::experiments}
\end{table}
We deploy the trained models to navigate in the three environments. For each method, the robot navigates each environment five times, resulting in a total of 60 physical trials. Table \ref{tab::experiments} reports the mean execution time for each environment with standard deviation and average number of recovery behaviors (Rec.) and collisions (Col.).
DWA exhibits the most recovery behaviors, because whenever the robot fails to sample a feasible motion, it starts recovery behavior. One collision happens in one of the environment 2 and 3 trials. Applying the model sequentially learned on environment 1, 2, and 3 causes catastrophic forgetting of the first two environments. It leads to longer execution time and higher standard deviation, with frequent recovery behaviors and collisions. LLfN can successfully avoid catastrophic forgetting: It achieves a similar time to the Individual Models approach in environment 3, while, \emph{surprisingly}, it outperforms Individual Models in environment 1 and 2 in terms of average time. One possible explanation is that the LLfN model has good backward transfer ability after gathering more diverse experience.
Utilizing extra training data and models specifically trained for each environment, the Individual Models approach is more stable (lowest standard deviation). (Video of representative trials of the four methods: \url{www.youtube.com/watch?v=ja_Rjc63xiY&t=68s}.)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%2345678901234567890123456789012345678901234567890123456789012345678901234567890
% 1 2 3 4 5 6 7 8
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb,stmaryrd,mathtools}
\usepackage{amsfonts}
%\usepackage{hyperref}
%\usepackage{algorithm}
%\usepackage{algorithmicx}
\usepackage[noend]{algpseudocode}
\usepackage{acronym}
%\usepackage{color,xcolor}
%\definecolor{orange}{rgb}{1, .36, .08}
\usepackage{verbatim}
\usepackage{booktabs}
\usepackage{siunitx}
\usepackage{graphics}
% \usepackage{graphicx}
\usepackage{graphicx,caption}
\captionsetup[figure]{font=footnotesize}
\captionsetup[table]{font=footnotesize}
% \usepackage[keeplastbox]{flushend}
\usepackage{suffix}
\usepackage{xstring}
\usepackage{xparse}
\usepackage{expl3}
\usepackage{mathrsfs}
\usepackage{tabularx}
\usepackage{makecell}
\usepackage{array}
\usepackage{hyperref}
\usepackage{cleveref}
\usepackage[ruled,linesnumbered]{algorithm2e}
\usepackage{multirow}
\usepackage{diagbox}
\usepackage{rotating}
\usepackage{bbm}
\usepackage{dsfont}
%% *** algorithm env *** %%
%\usepackage{algorithm}
%\usepackage[]{algorithm2e}
%\usepackage{algorithm}
%\usepackage{algorithmicx}
%\usepackage[noend]{algpseudocode}
%\newcommand{\eg}{e.\,g.,\ }
%\newcommand{\ie}{i.\,e.,\ }
%\newcommand{\etal}{{et~al.}}
%% *** FIGURE THINGS ***
%\usepackage{epsfig} % for postscript graphics files
% \usepackage{subfig}
\usepackage{subcaption}
\usepackage{wrapfig}
\usepackage{tikz}
\usetikzlibrary{arrows,backgrounds,calc}
\usepackage{relsize}
\usepackage{float}
\usepackage{kantlipsum}
\usepackage{lipsum}
\usepackage{stfloats}
\usepackage{siunitx}
% make citations [1]-[3] instead of [1][2][3]
\usepackage[noadjust]{cite}
\usepackage{pifont}% http://ctan.org/pkg/pifont
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\newcommand{\circlemark}{\ding{109}}
% \documentclass{powerdot}
%% some things for graphics
%\graphicspath{{./figures/}}
% for standard paper stuff
%% \usepackage[letterpaper, portrait, margin=1in]{geometry}
%% *** CITATION PACKAGES ***
%% \usepackage{cite}
%% *** ALIGNMENT PACKAGES ***
%\usepackage{array}
%% *** FONTS *** %%
%\usepackage{mathptmx} % assumes new font selection scheme installed
%\usepackage{times} % assumes new font selection scheme installed
% \usepackage{bbold}
%% *** TABLES *** %%
%\usepackage{multirow}
%% *** url *** %%
%\PassOptionsToPackage{hyphens}{url}
%\usepackage[options]{nohyperref}
%\usepackage{url}
%% correct bad hyphenation here
%\hyphenation{op-tical net-works semi-conduc-tor}
%% *** MATH THINGS *** %%
%\newcommand{\norm}[1]{\left\lVert #1 \right\rVert} % http://tex.stackexchange.com/questions/70020/how-to-write-euclidean-distance
%\newcommand{\abs}[1]{\left\lvert #1 \right\rvert} % http://tex.stackexchange.com/questions/70020/how-to-write-euclidean-distance
%%argmin
%\DeclareMathOperator*{\argmin}{argmin}
%\newcommand{\ignore}[1]{}
%%\newcommand{\comment}[1]{{\color{red} #1 }}
%\newtheorem{definition}{Definition}[section]
%\newtheorem{theorem}{Theorem}[section]
%%\newtheorem{proof}{Proof}[section]
%%introduce \powerset - hint by http://matheplanet.com/matheplanet/nuke/html/viewtopic.php?topic=136492&post_id=997377
%\DeclareFontFamily{U}{MnSymbolC}{}
%\DeclareSymbolFont{MnSyC}{U}{MnSymbolC}{m}{n}
%\DeclareFontShape{U}{MnSymbolC}{m}{n}{
% <-6> MnSymbolC5
% <6-7> MnSymbolC6
% <7-8> MnSymbolC7
% <8-9> MnSymbolC8
% <9-10> MnSymbolC9
% <10-12> MnSymbolC10
% <12-> MnSymbolC12%
%}{}
%\DeclareMathSymbol{\powerset}{\mathord}{MnSyC}{180}
%\DeclarePairedDelimiter\ceil{\lceil}{\rceil}
%\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}
%% optimization commands
%\newcommand{\M}[1]{\underset{#1}{\text{minimize}}\;}
%\newcommand{\AM}[1]{\underset{#1}{\text{argmin}}\;}
%\newcommand{\ST}{\text{subject to}}
% this package is for todo notes!
%===todonotes====
\usepackage{todonotes}
\usepackage{soul}
\definecolor{smoothgreen}{rgb}{0.7,1,0.7}
\sethlcolor{smoothgreen}
\newcommand{\todopara}[1]{\vspace{0px} %
\todo[inline, color=black!10]{\textbf{[Paragraph:]} {#1}} %
}
\newcommand{\todonote}[1]{\vspace{0px} %
\todo[inline, color=green!30]{\textbf{[Note:]} {#1}} %
}
\newcommand{\todoQ}[1]{\vspace{0px} %
\todo[inline, color=orange!50]{\textbf{[Note:]} {#1}} %
}
\newcommand{\todohere}[1]{\hl{(\textbf{TODO:} #1)}}
\newcommand{\hidetodos}{
\renewcommand{\todopara}[1]{}
\renewcommand{\todonote}[1]{}
\renewcommand{\todoQ}[1]{}
\renewcommand{\todohere}[1]{}}
\newcommand{\Lobject}{\mathcal{L}_{o}}
\newcommand{\Ltarget}{\mathcal{L}_{t}}
\newcommand{\Lsound}{\mathcal{L}_{s}}
%\hidetodos
%===end todonotes===
%%% Other people's edits %%%
%\definecolor{darkmagenta}{rgb}{0.698,0,0.698}
%\newcommand{\dorsa}[1]{\normalsize{\color{darkmagenta}(DS:\ #1)}}
%\definecolor{vg_edit_color}{rgb}{0, 0.0, 1.0}
%\newcommand{\vg}[1]{\normalsize{\color{vg_edit_color} VG:\ #1}}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% PGF preamble -- for PGF plots
\RequirePackage{luatex85}
% \documentclass[tikz]{standalone}
\usepackage{pgfplots}
\pgfplotsset{compat=newest}
\pgfplotsset{every axis legend/.append style={%
cells={anchor=west}}
}
\usepgfplotslibrary{polar}
\usetikzlibrary{arrows}
\tikzset{>=stealth'}
\definecolor{C1}{rgb}{0.0, 0.447, 0.741}
\definecolor{C1_light}{rgb}{0.0, 0.6032388663967612, 1.0}
\definecolor{C2}{rgb}{0.85, 0.325, 0.098}
\definecolor{C3}{rgb}{0.929, 0.694, 0.125}
\definecolor{C4}{rgb}{0.494, 0.184, 0.556}
\definecolor{C5}{rgb}{0.466, 0.674, 0.188}
\definecolor{C6}{rgb}{0.301, 0.745, 0.933}
\definecolor{C7}{rgb}{0.635, 0.078, 0.184}
\usepgfplotslibrary{groupplots}
\usetikzlibrary{shapes.geometric, arrows}
\tikzstyle{startstop} = [rectangle, rounded corners, minimum width=2cm, minimum height=1cm,text centered, draw=black, fill=none]
\tikzstyle{arrow} = [thick,->,>=stealth]
%% end PGF stuff
%% *** FLOAT PACKAGES ***
%%
%%\usepackage{fixltx2e}
%% fixltx2e, the successor to the earlier fix2col.sty, was written by
%% Frank Mittelbach and David Carlisle. This package corrects a few problems
%% in the LaTeX2e kernel, the most notable of which is that in current
%% LaTeX2e releases, the ordering of single and double column floats is not
%% guaranteed to be preserved. Thus, an unpatched LaTeX2e can allow a
%% single column figure to be placed prior to an earlier double column
%% figure.
%% Be aware that LaTeX2e kernels dated 2015 and later have fixltx2e.sty's
%% corrections already built into the system in which case a warning will
%% be issued if an attempt is made to load fixltx2e.sty as it is no longer
%% needed.
%% The latest version and documentation can be found at:
%% http://www.ctan.org/pkg/fixltx2e
%
%%\usepackage{stfloats}
%% stfloats.sty was written by Sigitas Tolusis. This package gives LaTeX2e
%% the ability to do double column floats at the bottom of the page as well
%% as the top. (e.g., "\begin{figure*}[!b]" is not normally possible in
%% LaTeX2e). It also provides a command:
%%\fnbelowfloat
%% to enable the placement of footnotes below bottom floats (the standard
%% LaTeX2e kernel puts them above bottom floats). This is an invasive package
%% which rewrites many portions of the LaTeX2e float routines. It may not work
%% with other packages that modify the LaTeX2e float routines. The latest
%% version and documentation can be obtained at:
%% http://www.ctan.org/pkg/stfloats
%% Do not use the stfloats baselinefloat ability as the IEEE does not allow
%% \baselineskip to stretch. Authors submitting work to the IEEE should note
%% that the IEEE rarely uses double column equations and that authors should try
%% to avoid such use. Do not be tempted to use the cuted.sty or midfloat.sty
%% packages (also by Sigitas Tolusis) as the IEEE does not format its papers in
%% such ways.
%% Do not attempt to use stfloats with fixltx2e as they are incompatible.
%% Instead, use Morten Hogholm'a dblfloatfix which combines the features
%% of both fixltx2e and stfloats:
%%
%% \usepackage{dblfloatfix}
%% The latest version can be found at:
%% http://www.ctan.org/pkg/dblfloatfix
%
%%\ifCLASSOPTIONcaptionsoff
%% \usepackage[nomarkers]{endfloat}
%% \let\MYoriglatexcaption\caption
%% \renewcommand{\caption}[2][\relax]{\MYoriglatexcaption[#2]{#2}}
%%\fi
%% endfloat.sty was written by James Darrell McCauley, Jeff Goldberg and
%% Axel Sommerfeldt. This package may be useful when used in conjunction with
%% IEEEtran.cls' captionsoff option. Some IEEE journals/societies require that
%% submissions have lists of figures/tables at the end of the paper and that
%% figures/tables without any captions are placed on a page by themselves at
%% the end of the document. If needed, the draftcls IEEEtran class option or
%% \CLASSINPUTbaselinestretch interface can be used to increase the line
%% spacing as well. Be sure and use the nomarkers option of endfloat to
%% prevent endfloat from "marking" where the figures would have been placed
%% in the text. The two hack lines of code above are a slight modification of
%% that suggested by in the endfloat docs (section 8.4.1) to ensure that
%% the full captions always appear in the list of figures/tables - even if
%% the user used the short optional argument of \caption[]{}.
%% IEEE papers do not typically make use of \caption[]'s optional argument,
%% so this should not be an issue. A similar trick can be used to disable
%% captions of packages such as subfig.sty that lack options to turn off
%% the subcaptions:
%% For subfig.sty:
%% \let\MYorigsubfloat\subfloat
%% \renewcommand{\subfloat}[2][\relax]{\MYorigsubfloat[]{#2}}
%% However, the above trick will not work if both optional arguments of
%% the \subfloat command are used. Furthermore, there needs to be a
%% description of each subfigure *somewhere* and endfloat does not add
%% subfigure captions to its list of figures. Thus, the best approach is to
%% avoid the use of subfigure captions (many IEEE journals avoid them anyway)
%% and instead reference/explain all the subfigures within the main caption.
%% The latest version of endfloat.sty and its documentation can obtained at:
%% http://www.ctan.org/pkg/endfloat
%%
%% The IEEEtran \ifCLASSOPTIONcaptionsoff conditional can also be used
%% later in the document, say, to conditionally put the References on a
%% page by themselves.
\newcommand{\aamir}[1]{\textcolor{blue}{#1}}
\newcommand{\etal}{\textit{et al}.}
\newcommand{\ie}{\textit{i}.\textit{e}.,}
\newcommand{\eg}{\textit{e}.\textit{g}.}
\newcommand{\modelName}{DRAGON\xspace}
\documentclass[letterpaper, 10 pt, conference]{ieeeconf} % Comment this line out
% if you need a4paper
%\documentclass[a4paper, 10pt, conference]{ieeeconf} % Use this line for a4
% paper
\IEEEoverridecommandlockouts % This command is only
% needed if you want to
% use the \thanks command
\overrideIEEEmargins
%%%% useful packages %%%%
\input{macros.tex}
\title{
% LEMuR: A Visual Language Enhanced Mobile Guiding Robot for People with Visual Impairments
DRAGON: A Dialogue-Based Robot for \\Assistive Navigation with Visual Language Grounding}
% Dia - Dialogue
% Vi - Visual
% La - Language
% Gu - Guidance / Guiding
% Gro - Grounded/Grounding
% Nav - Navigation
% Mob - Mobile
% Ro - Robot
% As - Assitive
% GuideDEMoN - Dialogue Enhanced Mobile Navigation Robot with Visual-Language Grounding for Guding People with Visual Impairments
% GuideDiaMoND - Dialogue Based Mobile Navigation Robot with ..
% RAVN - Robot for Visual Navigation
% GuideRAVEN: Guidance Robot for Assisting PwVI using Visual-Language Enhanced Navigation
% RAVEN: Robot for Assistance using Visual-Language Enhanced Navigation for PwVI / Robot for Assisting PwVI with Visual-Language Enhanced Navigation
\author{Shuijing Liu, Aamir Hasan, Kaiwen Hong, Runxuan Wang, Peixin Chang, Zachary Mizrachi, \\Justin Lin, D. Livingston McPherson, Wendy A. Rogers, and Katherine Driggs-Campbell
% \thanks{*,$\dagger$ denotes equal contribution}
% Need to check author order - for later
\thanks{S. Liu, A. Hasan, K. Hong, R. Wang, P. Chang, J. Lin, Z. Mizrachi, D. McPherson and K. Driggs-Campbell are with the Department of Electrical and Computer Engineering at the University of Illinois Urbana-Champaign. emails: \texttt{\{sliu105, aamirh2, kaiwen2, runxuan6,pchang17, jklin6, zdm3, dlivm, krdc\}@illinois.edu}}
\thanks{W. A. Rogers is with the Department of Applied Health Sciences at the University of Illinois Urbana-Champaign. emails: \texttt{wendyr@illinois.edu}}
}
\begin{document}
\maketitle
\thispagestyle{empty}
\pagestyle{empty}
\begin{abstract}
Persons with visual impairments (PwVI) have difficulties understanding and navigating spaces around them.
Current wayfinding technologies either focus solely on navigation or provide limited communication about the environment.
Motivated by recent advances in visual-language grounding and semantic navigation, we propose \modelName, a guiding robot powered by a dialogue system and the ability to associate the environment with natural language.
By understanding the commands from the user, \modelName is able to guide the user to the desired landmarks on the map, describe the environment, and answer questions from visual observations.
Through effective utilization of dialogue, the robot can ground the user's free-form descriptions to landmarks in the environment, and give the user semantic information through spoken language.
We conduct a user study with blindfolded participants in an everyday indoor environment.
Our results demonstrate that \modelName is able to communicate with the user smoothly, provide a good guiding experience, and connect users with their surrounding environment in an intuitive manner.
% briefly summarize main functionalities
% highlight dialogue & grounding, especially merits of CLIP
% user study process and results
% \textcolor{pink}{
% However, current wayfinding tools are fairly limited.
% In this short paper, we present our in-progress work on a wayfinding robot for PwVI.
% The robot takes an audio command from the user that specifies the intended destination.
% Then, the robot autonomously plans a path to navigate to the goal.
% We use sensors to estimate the real-time position of the user, which is fed to the planner to improve the safety and comfort of the user.
% In addition, the robot describes the surroundings to the user periodically to prevent disorientation and potential accidents.
% We demonstrate the feasibility of our design in a public indoor environment.
% Finally, we analyze the limitations of our current design, as well as our insights and future work.
% A demonstration video can be found at {\color{cyan}{\url{https://youtu.be/BS9r5bkIass}}}.
% }
\end{abstract}
% sections
\input{Sections/1-Intro}
\input{Sections/2-Related.tex}
\input{Sections/3-Overview}
\input{Sections/4-Dialog}
\input{Sections/5-Exp}
\input{Sections/6-Results}
\input{Sections/7-Conclusion}
% references
\newpage
\clearpage
\bibliographystyle{IEEEtran}
\bibliography{BibFile}
\end{document}
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@rmstyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand\BIBentrySTDinterwordspacing{\spaceskip=0pt\relax}
\providecommand\BIBentryALTinterwordstretchfactor{4}
\providecommand\BIBentryALTinterwordspacing{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
\fontdimen4\font\relax}
\providecommand\BIBforeignlanguage[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\bibitem{kulyukin2004rfid}
V.~Kulyukin, C.~Gharpure, J.~Nicholson, and S.~Pavithran, ``Rfid in
robot-assisted indoor navigation for the visually impaired,'' in
\emph{IEEE/RSJ International Conference on Intelligent Robots and Systems
(IROS)}, 2004.
\bibitem{bayles2022interdisciplinary}
M.~A. Bayles, T.~Kadylak, S.~Liu, A.~Hasan, W.~Liang, K.~Hong,
K.~Driggs-Campbell, and W.~A. Rogers, ``An interdisciplinary approach:
Potential for robotic support to address wayfinding barriers among persons
with visual impairments,'' in \emph{Human Factors and Ergonomics Society
Annual Meeting (HFES)}, 2022.
\bibitem{azenkot2016enabling}
S.~Azenkot, C.~Feng, and M.~Cakmak, ``Enabling building service robots to guide
blind people a participatory design approach,'' in \emph{ACM/IEEE
International Conference on Human-Robot Interaction}, 2016.
\bibitem{bhat2022confused}
P.~Bhat and Y.~Zhao, ```i was confused by it; it was confused by me:' exploring
the experiences of people with visual impairments around mobile service
robots,'' \emph{ACM on Human-Computer Interaction}, 2022.
\bibitem{ohnbar2018personalized}
E.~OhnBar, K.~Kitani, and C.~Asakawa, ``Personalized dynamics models for
adaptive assistive navigation systems,'' in \emph{Conference on Robot
Learning (CoRL)}, 2018.
\bibitem{nanavati2019coupled_dyads}
A.~Nanavati, X.~Z. Tan, J.~Connolly, and A.~Steinfeld, ``Follow the robot:
Modeling coupled human-robot dyads during navigation,'' in \emph{IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS)}, 2019.
\bibitem{xiao2021guidedog}
A.~Xiao, W.~Tong, L.~Yang, J.~Zeng, Z.~Li, and K.~Sreenath, ``Robotic guide
dog: Leading a human with leash-guided hybrid physical interaction,'' in
\emph{IEEE International Conference on Robotics and Automation (ICRA)}, 2021.
\bibitem{li2019ballbot}
Z.~Li and R.~Hollis, ``Toward a ballbot for physically leading people: A
human-centered approach,'' in \emph{IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS)}, 2019.
\bibitem{jin2021wearable}
L.~Jin, H.~Zhang, and C.~Ye, ``A wearable robotic device for assistive
navigation and object manipulation,'' in \emph{IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS)}, 2021.
\bibitem{wilson2007swan}
J.~Wilson, B.~N. Walker, J.~Lindsay, C.~Cambias, and F.~Dellaert, ``Swan:
System for wearable audio navigation,'' in \emph{IEEE International Symposium
on Wearable Computers}, 2007.
\bibitem{guerreiro2019cabot}
J.~Guerreiro, D.~Sato, S.~Asakawa, H.~Dong, K.~M. Kitani, and C.~Asakawa,
``Cabot: Designing and evaluating an autonomous navigation robot for blind
people,'' in \emph{International ACM SIGACCESS Conference on Computers and
Accessibility (ASSETS)}, 2019.
\bibitem{cheraghi2017guidebeacon}
S.~A. Cheraghi, V.~Namboodiri, and L.~Walker, ``Guidebeacon: Beacon-based
indoor wayfinding for the blind, visually impaired, and disoriented,'' in
\emph{IEEE International Conference on Pervasive Computing and Communications
(PerCom)}, 2017.
\bibitem{sato2017navcog3}
D.~Sato, U.~Oh, K.~Naito, H.~Takagi, K.~Kitani, and C.~Asakawa, ``Navcog3: An
evaluation of a smartphone-based blind indoor navigation assistant with
semantic features in a large-scale environment,'' in \emph{International ACM
SIGACCESS Conference on Computers and Accessibility (ASSETS)}, 2017.
\bibitem{kulyukin2006robot}
V.~Kulyukin, C.~Gharpure, J.~Nicholson, and G.~Osborne, ``Robot-assisted
wayfinding for the visually impaired in structured indoor environments,''
\emph{Autonomous robots}, 2006.
\bibitem{chang2021learning}
P.~Chang, S.~Liu, and K.~Driggs-Campbell, ``Learning visual-audio
representations for voice-controlled robots,'' in \emph{IEEE International
Conference on Robotics and Automation (ICRA)}, 2023.
\bibitem{huang2022visual}
C.~Huang, O.~Mees, A.~Zeng, and W.~Burgard, ``Visual language maps for robot
navigation,'' in \emph{IEEE International Conference on Robotics and
Automation (ICRA)}, 2023.
\bibitem{ahn2022saycan}
M.~Ahn, A.~Brohan, N.~Brown, Y.~Chebotar, O.~Cortes, B.~David, C.~Finn,
K.~Gopalakrishnan, K.~Hausman, A.~Herzog, \emph{et~al.}, ``Do as i can, not
as i say: Grounding language in robotic affordances,'' in \emph{Conference on
Robot Learning (CoRL)}, 2022.
\bibitem{shah2022lmnav}
D.~Shah, B.~Osi{\'n}ski, brian ichter, and S.~Levine, ``{LM}-nav: Robotic
navigation with large pre-trained models of language, vision, and action,''
in \emph{Conference on Robot Learning (CoRL)}, 2022.
\bibitem{saha2019closing}
M.~Saha, A.~J. Fiannaca, M.~Kneisel, E.~Cutrell, and M.~R. Morris, ``Closing
the gap: Designing for the last-few-meters wayfinding problem for people with
visual impairments,'' in \emph{International ACM SIGACCESS Conference on
Computers and Accessibility (ASSETS)}, 2019.
\bibitem{yang2022seeway}
Z.~Yang, L.~Yang, L.~Kong, A.~Wei, J.~Leaman, J.~Brooks, and B.~Li, ``Seeway:
Vision-language assistive navigation for the visually impaired,'' in
\emph{2022 IEEE International Conference on Systems, Man, and Cybernetics
(SMC)}, 2022.
\bibitem{radford2021lclip}
A.~Radford, J.~W. Kim, C.~Hallacy, A.~Ramesh, G.~Goh, S.~Agarwal, G.~Sastry,
A.~Askell, P.~Mishkin, J.~Clark, \emph{et~al.}, ``Learning transferable
visual models from natural language supervision,'' in \emph{International
Conference on Machine Learning (ICML)}, 2021.
\bibitem{zhou2022detecting}
X.~Zhou, R.~Girdhar, A.~Joulin, P.~Kr{\"a}henb{\"u}hl, and I.~Misra,
``Detecting twenty-thousand classes using image-level supervision,'' in
\emph{European Conference on Computer Vision (ECCV)}, 2022.
\bibitem{antol2015vqa}
S.~Antol, A.~Agrawal, J.~Lu, M.~Mitchell, D.~Batra, C.~L. Zitnick, and
D.~Parikh, ``Vqa: Visual question answering,'' in \emph{IEEE International
Conference on Computer Vision (ICCV)}, 2015.
\bibitem{kulyukin2006ergonomics}
V.~A. Kulyukin and C.~Gharpure, ``Ergonomics-for-one in a robotic shopping cart
for the blind,'' in \emph{ACM/IEEE International Conference on Human-Robot
Interaction}, 2006.
\bibitem{zhang2023follower}
Y.~Zhang, Z.~Li, H.~Guo, L.~Wang, Q.~Chen, W.~Jiang, M.~Fan, G.~Zhou, and
J.~Gong, ```i am the follower, also the boss': Exploring different levels of
autonomy and machine forms of guiding robots for the visually impaired,'' in
\emph{ACM Conference on Human Factors in Computing Systems (CHI)}, 2023.
\bibitem{liu2019review}
R.~Liu and X.~Zhang, ``A review of methodologies for
natural-language-facilitated human-robot cooperation,'' \emph{International
Journal of Advanced Robotic Systems}, vol.~16, no.~3, 2019.
\bibitem{min2021film}
S.~Y. Min, D.~S. Chaplot, P.~Ravikumar, Y.~Bisk, and R.~Salakhutdinov, ``Film:
Following instructions in language with modular methods,'' in
\emph{International Conference on Learning Representations (ICLR)}, 2022.
\bibitem{anderson2018vision}
P.~Anderson, Q.~Wu, D.~Teney, J.~Bruce, M.~Johnson, N.~S{\"u}nderhauf, I.~Reid,
S.~Gould, and A.~van~den Hengel, ``Vision-and-language navigation:
Interpreting visually-grounded navigation instructions in real
environments,'' in \emph{IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)}, 2018.
\bibitem{chen2020soundspaces}
C.~Chen, U.~Jain, C.~Schissler, S.~V.~A. Gari, Z.~Al-Halah, V.~K. Ithapu,
P.~Robinson, and K.~Grauman, ``Soundspaces: Audio-visual navigation in 3d
environments,'' in \emph{European Conference on Computer Vision (ECCV)},
2020.
\bibitem{chang2020robot}
P.~Chang, S.~Liu, H.~Chen, and K.~Driggs-Campbell, ``Robot sound
interpretation: Combining sight and sound in learning-based control,'' in
\emph{IEEE/RSJ International Conference on Intelligent Robots and Systems
(IROS)}, 2020.
\bibitem{gadre2022cow}
S.~Y. Gadre, M.~Wortsman, G.~Ilharco, L.~Schmidt, and S.~Song, ``Cows on
pasture: Baselines and benchmarks for language-driven zero-shot object
navigation,'' \emph{arXiv}, 2022.
\bibitem{Zhu2020The}
H.~Zhu, J.~Yu, A.~Gupta, D.~Shah, K.~Hartikainen, A.~Singh, V.~Kumar, and
S.~Levine, ``The ingredients of real world robotic reinforcement learning,''
in \emph{International Conference on Learning Representations (ICLR)}, 2020.
\bibitem{mccormac2017semanticfusion}
J.~McCormac, A.~Handa, A.~Davison, and S.~Leutenegger, ``Semanticfusion: Dense
3d semantic mapping with convolutional neural networks,'' in \emph{IEEE
International Conference on Robotics and Automation (ICRA)}, 2017.
\bibitem{hughes2022hydra}
N.~Hughes, Y.~Chang, and L.~Carlone, ``Hydra: A real-time spatial perception
system for {3D} scene graph construction and optimization,'' in
\emph{Robotics: Science and Systems}, 2022.
\bibitem{wu2021scenegraphfusion}
S.-C. Wu, J.~Wald, K.~Tateno, N.~Navab, and F.~Tombari, ``Scenegraphfusion:
Incremental 3d scene graph prediction from rgb-d sequences,'' in
\emph{IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
2021.
\bibitem{savinov2018semiparametric}
N.~Savinov, A.~Dosovitskiy, and V.~Koltun, ``Semi-parametric topological memory
for navigation,'' in \emph{International Conference on Learning
Representations (ICLR)}, 2018.
\bibitem{meng2020scaling}
X.~Meng, N.~Ratliff, Y.~Xiang, and D.~Fox, ``Scaling local control to
large-scale topological navigation,'' in \emph{IEEE International Conference
on Robotics and Automation (ICRA)}, 2020.
\bibitem{chaplot2020neural}
D.~S. Chaplot, R.~Salakhutdinov, A.~Gupta, and S.~Gupta, ``Neural topological
slam for visual navigation,'' in \emph{IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)}, 2020.
\bibitem{fox1997dynamic}
D.~Fox, W.~Burgard, and S.~Thrun, ``The dynamic window approach to collision
avoidance,'' \emph{IEEE Robotics and Automation Magazine}, vol.~4, no.~1, pp.
23--33, 1997.
\bibitem{whisper}
A.~Radford, J.~W. Kim, T.~Xu, G.~Brockman, C.~McLeavey, and I.~Sutskever,
``Robust speech recognition via large-scale weak supervision,'' 2022.
\bibitem{bunk2020diet}
T.~Bunk, D.~Varshneya, V.~Vlasov, and A.~Nichol, ``Diet: Lightweight language
understanding for dialogue systems,'' \emph{arXiv}, 2020.
\bibitem{kim2021vilt}
W.~Kim, B.~Son, and I.~Kim, ``Vilt: Vision-and-language transformer without
convolution or region supervision,'' in \emph{International Conference on
Machine Learning (ICML)}, 2021.
\end{thebibliography}