Symbolic Graph Inference for Compound Scene Understanding
Extracted source text · arXiv 2410.22626
% This is samplepaper.tex, a sample chapter demonstrating the
% LLNCS macro package for Springer Computer Science proceedings;
% Version 2.21 of 2022/01/12
%
\documentclass[runningheads]{llncs}
%
\usepackage{float} % Required for the [H] option
% \usepackage{natbib}
\usepackage[T1]{fontenc}
% T1 fonts will be used to generate the final print and online PDFs,
% so please use T1 fonts in your manuscript whenever possible.
% Other font encondings may result in incorrect characters.
%
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{xcolor}
% Used for displaying a sample figure. If possible, figure files should
% be included in EPS format.
%
% If you use the hyperref package, please uncomment the following two lines
% to display URLs in blue roman font according to Springer's eBook style:
%\usepackage{color}
%\renewcommand\UrlFont{\color{blue}\rmfamily}
%\urlstyle{rm}
%
\begin{document}
%
\title{Symbolic Graph Inference\\for Compound Scene Understanding}
%
%\titlerunning{Abbreviated paper title}
% If the paper title is too long for the running head, you can set
% an abbreviated paper title here
%
\author{FNU Aryan\inst{1} \and
Simon Stepputtis\inst{1} \and
Sarthak Bhagat\inst{1} \and
Joseph Campbell\inst{1} \and
Kwonjoon Lee\inst{2} \and
Hossein Nourkhiz Mahjoub \inst{2} \and
Katia Sycara \inst{1}}
%
\authorrunning{Aryan et al.}
% First names are abbreviated in the running head.
% If there are more than two authors, 'et al.' is used.
%
\institute{Carnegie Mellon University \and
Honda Research Institute
}
%
\maketitle % typeset the header of the contribution
%
\begin{abstract}
Scene understanding is a fundamental capability needed in many domains ranging from question-answering to robotics.
Unlike recent end-to-end approaches that must explicitly learn varying compositions of the same scene, our method reasons over their constituent objects and analyzes their arrangement to infer a scene's meaning.
We propose a novel approach that reasons over a scene's scene- and knowledge-graph, capturing spatial information while being able to utilize general domain knowledge in a joint graph search.
% Our approach formulates scene understanding as identifying the combination of its basic components.
Empirically, we demonstrate the feasibility of our method on the ADE20K dataset and compare it to current scene understanding approaches.
\keywords{Compound Scene Understanding \and Graph Search Networks}
\end{abstract}
%
%
%
\section{Introduction}
\vspace{-10pt}
Scenes understanding is crucial for numerous applications, including, but not limited to path planning for robotic agents \cite{lundeen2019autonomous} and developing assistive human companions \cite{li2023airloc}.
However, current scene understanding approaches \cite{cimpoi2015deep} often interpret them as indivisible entities, overlooking their intricate composition. However, such scenes \cite{zhou2017scene} are not merely a singular entity, but rather the sum of their parts.
For instance, consider a \textit{kitchen}: while its composition can vary, e.g., assume a \textit{kitchen} without an \textit{oven}, it is still a \textit{kitchen}, given that other descriptive components (e.g., \textit{stove, sink, fridge, ...}) are still present.
On the other hand, consider a harbor without its defining element - \textit{water} -, as shown in Fig.~\ref{qualitative}, which should not be identified as a harbor despite the presence of boats.
% This illustrates how the essence of an entity lies in its parts, rather than in its indivisible whole.
% These complex entities can be considered compound concepts comprised of various individual concepts, linked together through a knowledge graph.
One way of capturing the constituent elements of a compound scene is through a Knowledge Graph (KG), linking base elements to their compound interpretation.
For this work, we consider such complex scenes as compound concepts (e.g., \textit{kitchen}) made up of multiple primitive constituent concepts (e.g., \textit{stove} or \textit{fridge}).
% Neuro-symbolic architectures, leveraging symbolic knowledge graphs, offer valuable tools for unraveling the intricacies of these compound concepts.
However, a KG does not capture all the necessary information to reason over such scenes.
For example, in Fig.~\ref{qualitative}, the existence of a \textit{dog} and \textit{sled} could be seen as the compound concept of \textit{dogsled}, but the spatial relationships between the basic entities and their quantities play an important role.
% Understanding the specific relationships between objects, such as a dog being in front of the sled to pull it or a dog sitting on the sled, is crucial.
To this end, we propose to combine scene and knowledge graphs, harnessing the ability of Scene Graphs (SG) to account for multiple instances of the same concept as well as their spatial relationships, while knowledge graphs allow for high-level reasoning for scene understanding.
% Analyzing individual object instances and their interrelations enhances scene semantics understanding and facilitates robust scene understanding.
% However, these knowledge graphs encapsulating each constituent element are also not enough in various challenging scenarios. For example, in a dogsled scenario, it is not merely the presence of a dog and a sled that defines the compound concept, but rather the specific relationship between the dog and the sled—such as the dog being harnessed to pull the sled—that holds significance. The analysis of individual object instances and their interrelations allows for a deeper understanding of scene semantics and facilitates more robust scene understanding.
% In computational tasks such as predicting compound concepts within scenes or extracting future action candidates given some initial set of video frames, interpretability plays a pivotal role in enhancing understanding. However, non-symbolic methods like the deep learning models including ChatGPT lack interpretability, necessitating the development of approaches that prioritize this aspect. While previous methods primarily leverage knowledge graphs, the incorporation of scene graphs, capturing scene-specific details such as object instances and relationships, is imperative.
% Intuitively, the underlying method should be interpretable and should encompass crucial scene-specific details, such as the number of object instances present in the scene and the special relationships between objects. Knowledge graphs typically contain only global information about the world and may falter when more scene-specific details are required.
% Previous methodologies have integrated scene graphs and knowledge graphs separately for tasks like constructing linguistic descriptions \cite{zheng2015scene} or answering visual questions \cite{aditya2018image}. While some approaches leverage individual \cite{zareian2020bridging} \cite{chen2019knowledge}
% graphs to enhance the generation of the other. However, these approaches do not attempt to dynamically combine the graphs and reason over them jointly.
% This motivates us to propose a pipeline, as depicted in Figure \ref{approach}. Our Multi-Graph Graph Search Network merges both the scene graph and the knowledge graph, enabling joint reasoning in the combined domain.
% Such integration proves beneficial for various downstream tasks, including compound concept prediction and action anticipation.
% Previous methodologies have separately integrated scene graphs and knowledge graphs for tasks such as constructing linguistic descriptions \cite{zheng2015scene} or answering visual questions \cite{aditya2018image}. Some approaches leverage individual graphs to enhance the generation of the other \cite{zareian2020bridging} \cite{chen2019knowledge}. However, these methods do not dynamically combine the graphs and reason over them jointly.
% Both scene and knowledge graphs have been widely used in literature for tasks like Visual Question Answering \cite{aditya2018image},\cite{zheng2015scene}, Novel Concept Learning \cite{bhagat2023sample}, and Indoor Relocalization \cite{li2023airloc}, among others.
% Inspired by how humans understand a scene which includes focusing on object, their relationships and our common sense, we propose our method, depicted in Figure \ref{approach}, which achieves the best of both worlds by utilizing a dynamic combination of both graphs.
To this end, we propose a novel approach (see Fig.~\ref{approach}) that dynamically establishes a link between the spatial information in the scene graph and the domain knowledge encoded in the knowledge graph, before conducting a joint search over the combined domain.
% Our multi-graph graph search approach merges both the scene graph and the knowledge graph, enabling joint reasoning in the combined domain.
% This integration is beneficial for various downstream tasks like compound concept prediction.
% To elucidate our contributions, we introduce a graph merging module that consolidates scene and knowledge graphs, enabling seamless joint reasoning.
Additionally, we leverage techniques such as dynamic graph propagation to improve the runtime and scalability of the method. Our main contributions include:
\begin{itemize}
\vspace{-10pt}
\item We propose a novel dual-graph graph search approach that jointly reasons over scene and knowledge graphs.
\item In comparison to prior work, we utilize a dynamic exploration approach that automatically determines when enough information has been considered.
% \item Through experimental validation, demonstrating the superiority of our method over baselines like GPT4-Vision \cite{openai2024gpt4} and previous neuro-symbolic methods, including knowledge graph-only approaches.
\item In initial experiments, we demonstrate the feasibility of our approach compared to symbolic-only and neural-only approaches.
\end{itemize}
% \begin{figure}[t]
% \includegraphics[width=\textwidth]{Figs/Multi_graph GSNN Without Sction Anticipation.pdf}
% \caption{The overview of our multi-graph reasoning approach which uses both the scene and knowledge graph and jointly reasons over them} \label{approach}
% \end{figure}
\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{Figs/ICRA2.pdf}
\caption{The overview of our multi-graph reasoning approach which uses both the scene and knowledge graph and jointly reasons over them.}
\label{approach}
\vspace{-15pt}
\end{figure}
% https://docs.google.com/drawings/d/1kLncom9zq75PF0tIsZO49Rq1aTxgy0iFtXv0vdpnosY/edit?usp=sharing
% \section{Related Works}
% % [I recommend still putting this, even if it's just 3-4 sentences. Is anyone else using SG+KG reasoning? If so, for what and why?]
% Previous methodologies have integrated scene graphs and knowledge graphs separately for tasks like constructing linguistic descriptions \cite{zheng2015scene} or answering visual questions \cite{aditya2018image}. While some approaches leverage individual \cite{zareian2020bridging} \cite{chen2019knowledge}
% graphs to enhance the generation of the other. However, these approaches do not attempt to dynamically combine the graphs and reason over them jointly whice motivates us for our merged graph approach.
% In this section, we will review existing neuro-symbolic architectures, as well as established methods for scene graphs. Subsequently, we will examine existing approaches and datasets for compound concept prediction.
% \subsection{Neuro-symbolic architectures}
% Incorporating domain-specific knowledge holds significant potential, particularly in scenarios with limited data, as it can aid in recognizing and incorporating new concepts in a more sample-efficient manner. A commonly adopted approach to leverage symbolic knowledge graphs (KGs) in deep learning is through graph neural networks \cite{xie2019embedding}, \cite{lamb2020graph}. This approach offers various benefits ranging from interpretability \cite{tiddi2022knowledge} to the utilization of interconnected information. The hierarchical structure inherent in KGs has led to their utilization as priors for neuro-symbolic vision systems \cite{jiao2022graph} across diverse applications, including transfer learning and vision-language pre-training \cite{alberts2020visualsem}. \cite{chen2020knowledge} introduced a static knowledge-guided graph routing framework comprising two graph propagation frameworks, facilitating the transfer of visual and semantic features to enhance classifier training with limited samples. Marino et al. \cite{marino2016more} and Fang et al. \cite{fang2017object} utilized structured knowledge to identify underlying concepts in images. With a comprehensive graph, the structured knowledge embedded within it can be leveraged to extract critical information about previously unseen classes \cite{bhagat2023sample}. However, none of these methods incorporate any scene specific information which motivates us to use the above explained structured knowledge graph along with scene specifci knowledge.
% \subsection{Scene Graph Generation}
% Scene graph generation (SGG) has witnessed significant advancements, fueled by the growing interest in computer vision and natural language processing communities. Recent works have explored diverse perspectives, including two-stage methods \cite{zellers2018neural}, \cite{xu2017scene}, graph-based models \cite{chen2019knowledge} \cite{yang2018graph}, \cite{lin2020gps}, and the integration of attention mechanisms \cite{herzig2018mapping}, \cite{qi2019attentive}. With the emergence of Transformer architectures \cite{vaswani2017attention}, new avenues are being explored \cite{lu2021context}, \cite{dhingra2021bgt}, \cite{chen2022reltransformer}, \cite{cong2023reltr}, \cite{khandelwal2022iterative}. In the previous appraochs common relation labels like "on" and "has" dominate scene graphs, leading to bias and they overlook the spatial information and contain relations like "walking" and "chasing", etc. We want the scene graph to contain just the spatial information and not mix the knowledge. To address this, we propose a simplified approach that focuses solely on spatial features, disregarding common relations and aiming for unbiased scene graph generation.
% \subsection{Compound Concept Prediction}
% Compound concept prediction entails identifying compound concepts within an image, such as inferring a kitchen scene from the presence of objects like ovens, microwaves, and refrigerators. This task is akin to scene recognition, where the underlying scene depicted in an image is predicted. The evolution of scene recognition literature transitioned from hand-engineered features \cite{lazebnik2006beyond} to CNN-based methods \cite{xie2015hybrid}, often supplemented by orderless bag-of-features or dictionaries \cite{cimpoi2015deep}. Challenges such as inter-class similarity are addressed by hybrid approaches \cite{jiang2019deep}, which introduce techniques like multi-scale CNNs and dictionary learning layers \cite{liu2018dictionary}. Additionally, studies have also explored spatial relationships between objects \cite{li2023airloc} using RGB images and semantic segmentation \cite{lopez2020semantic}. Recently, with the emergence of Vision Transformers (ViTs) \cite{dosovitskiy2020image} and Vision Language Models (VLMs) like ChatGPT 4 Vision \cite{openai2024gpt4}, there's been a shift towards utilizing them for scene recognition and compound concept prediction tasks \cite{sakaino2023unseen}. While neurosymbolic methods \cite{bhagat2023sample} have been used previously for compound concept prediction,very few methods integrate both semantic and common-sense knowledge \cite{shao2019automatic}. This motivates us to test our Multi-graph Network on compound concept prediction.
\vspace{-15pt}
\section{Methodology}
\vspace{-10pt}
% This section elaborates the functionality of various components within our Multi-Graph Graph Search Network. We begin with the Graph Merging Module, which serves to join disparate graphs into a unified representation. Subsequently, we delve into the Merged Graph Graph Search Network, comprising propagation and importance networks. Finally, we discuss the classifier network, tailored to specific downstream tasks, thereby enhancing adaptability and performance. In addition to these core components, we elucidate supplementary techniques employed to improve our results.
% With our work, we address the problem of identifying compound concepts from the individual concepts present within them.
% Formally, we take an input image ($I$), classifying the scene name ($p$) through our proposed method. Given the input image ($I$), we employ Faster R-CNN \cite{Girshick_2015_ICCV} to obtain the object bounding boxes. Subsequently, utilizing these bounding boxes, we generate scene-specific object embeddings through a ViT-based encoder. Additionally, we derive object relations using the self-designed relation prediction network inspired from \cite{zhuang2017context}. Incorporating the obtained relation labels and scene-specific object embeddings, we construct the scene graph ($S$). We utilize a hand-crafted Knowledge Graph ($G$) as our domain knowledge source, establishing connections between various concepts and compound concepts. The nodes of the Knowledge Graph are represented uing GloVe embeddings. In the subsequent part, we elaborate Graph Merging Module, which unifies disparate graphs. Following this, we explore the Merged Graph Search Network, intended for reasoning about the merged graph.
In this work, we aim to identify compound concepts in images. Formally, we take an input image $\mathbf{I}$ and domain KG $\mathcal{K}$ to classify the set of concepts and compound concepts $\mathcal{C}$ through our proposed method $\mathcal{C} = f_\Theta(\mathbf{I}, \mathcal{K})$, where $\theta$ reflects the trainable parameters of our neural components. Our approach works in three steps: 1) creating a scene and knowledge graph from input data, 2) merging the graphs, and 3) searching over the merged graph and doing our final prediction (see Figure~\ref{approach}).
\textbf{Scene Graph Generation.}
To generate a scene graph $\mathcal{S}$, we utilize an object detector, namely Faster R-CNN~\cite{Girshick_2015_ICCV}, to detect initial concepts $\hat{\mathbf{c}}$ in the provided scene.
These objects serve as concepts in our SG, while edge-types (encoding the spatial relationships between the nodes) are predicted by a small neural network $\mathbf{A}_S = f^{\text{EDGE}}_\theta(\hat{\mathbf{c}})$ inspired by ~\cite{zhuang2017towards}.
% In the scene graph $\mathcal{S}$, objects serve as nodes, and their spatial relations serve as directional edges represented by adjacency matrix $\mathbf{A_S}$.
% We employ scene-specific object embeddings to represent each node within the graph. These 768-dimensional embeddings are generated by applying a ViT-based object encoder on object bounding boxes obtained using Faster R-CNN \cite{Girshick_2015_ICCV}.
Each node $\hat{c}_i \in \hat{\mathbf{c}}$ in the SG is represented as an embedding derived from its bounding box through ViT~\cite{DBLP:journals/corr/abs-2010-11929}.
% Additionally, we determine spatial relations between objects using a relation prediction network inspired by \cite{zhuang2017context} which infers the relationships between pairs of objects based on their positions.
% We utilize a hand-crafted knowledge graph \(G\) as a structured source of domain knowledge. We use bi-direction edges in the graph to connect various concepts to the relevant compound concepts represented by adjacency matrix $\mathbf{A_G}$. The nodes of the Knowledge Graph are represented using 100-dimensional GloVe embeddings.
% Later, we detail the Graph Merging Module, which unifies disparate graphs. Subsequently, we introduce the Merged Graph Search Network for reasoning about the merged graph.
\textbf{Graph Merging Module.}
The scene graph $\mathcal{S}$ encapsulates scene-specific information, while the knowledge graph $\mathcal{K}$ holds general domain knowledge of how individual concepts are related to compound concepts and their affordances. Our KG is hand-designed for our particular use case, inspired by the concepts present in the ADE20K dataset.
The goal of our graph-merging approach is to combine the spatial information of the SG with the domain knowledge of the KG.
% We explore different initialization schemes for scene graph nodes crucial in graph merging. The exhaustive approach activates all nodes, even redundant ones, which may introduce noise. Alternatively, random initialization selects nodes and their neighbors randomly. We enhance randomness with a two-stage approach: first, selecting a node and activating its neighbors, then repeating the process for another node not previously selected.
% Subsequently, by connecting a subset (potentially random or all) of nodes between the graphs as explained later, we formed a merged graph ($M$) representing a unified knowledge space.
% In the graph merging process, we first generate the SG as described previously and initialize the KG from the set of detected concepts $\hat{\mathbf{c}}$, forming our merged graph $\mathcal{M}$.
% % Connections are formed by dynamically linking the same SG and KG nodes.
% % % An intuitive way is to exhaustively initialize nodes, potentially including redundant ones.
% % We start graph-search from a randomly chosen node in the SG part of our merged graph $M$.
% We randomly select a node and its neighbors in the Scene Graph (SG) to initiate the graph search (see next paragraph for details on the search).
% We refer to this ``partially active graph'' over $\mathcal{M}$ as $\mathcal{M}^A$.
% Connections are formed and dynamically updated by linking shared nodes between the active SG and Knowledge Graph (KG).
In this process, we first generate the SG as described previously and initialize the KG from the set of detected concepts $\hat{\mathbf{c}}$.
We then randomly select a node and its neighbors in the SG and form connections by linking shared nodes between the SG and KG to generate our merged graph $\mathcal{M}$ containing entire SG and KG. The sub-graph consisting of connected SG and KG nodes is referred to as ``active graph'' $\mathcal{M}^A$ over $\mathcal{M}$, and it is used to initiate the graph search (see next paragraph for details on the search).
% \textcolor{red}{[Check the previous sentence. It is unclear what the difference between $M$ and $M^A$ is.]}
% Graph search starts from these initialized nodes in our merged graph $\mathcal{M}$.
After our entire graph search is completed (see method below) and SG nodes that have not been activated remain, we conduct an additional search starting from such previously non-activated nodes.
Through this approach, we ensure the entire scene has been considered, even if comprised of multiple compound scenes (e.g., having a \textit{kitchen} on the right, and a \textit{living room} on the left of an image).
% While comprehensive, this method may introduce noise into merged graph, such as including irrelevant objects like an \textit{airplane} seen through a \textit{window} when analyzing scene of a \textit{bedroom}. To this end, we propose random initialization which involves selection of nodes and their neighbors at random from the scene graph represented by $P_S$. Now, once we have the initial set of SG nodes, we dynamically create connections between the matching nodes $n_{S} \in P_S$ and $n_{G} \in G$ where $n_{G} == n_{G}$. After merging, merged graph ($M$) incorporates concatenated nodes from both graphs and adjacency matrix \( \mathbf{A_M }= f_M(\mathbf{A_G}, \mathbf{A_S}) \) where, \( f_M \) is a merge function based on graph connections.
% We start with some initial set of scene graph nodes and attach them to the knowledge graph, thereby forming the active nodes in the merged graph ($M$). We explore two primary types of scene graph initializations. Firstly, the exhaustive approach activates all nodes within the scene graph, potentially including redundant ones. While this method offers comprehensive coverage, it may also introduce noise into the merged graph. Alternatively, random initialization involves the selection of nodes and their neighbors at random from the scene graph. To enhance randomness, we implement a two-stage approach: initially selecting a node and activating its neighbors, then repeating the process for another node not previously chosen. This merging process facilitated the fusion of contextual information from both the scene and external knowledge domains, enabling a comprehensive representation of the environment. The final adjacency matrix will be: \( A_M = f(A_G, A_S) \) where, \( f \) is a merge function based on node initialization schemes and \( A_G \), \( A_S \) and \( A_M \) are adjacency matrices of knowledge, scene and merged graph respectively.
% With the merged graph established, we conducted graph search operations to explore the combined knowledge space. This approach leverages the interconnectedness of the nodes across the integrated graph, enabling efficient traversal and inference over the amalgamated information landscape.
\textbf{Merged Graph Search Network.}
% The Graph Search Neural Network (GSNN) operates over a merged graph ($M$), comprising two key components: the \textit{Propagation Network} and the \textit{Importance Network}. We draw inspiration from our prior research for these components.
% [Simon's Proposal]:
Having formed the initial connection between SG and KG, we conduct a joint graph search over the active merged graph $\mathcal{M}^A$.
Inspired by~\cite{bhagat2023sample}, we leverage a three-staged approach: 1) we utilize a \textit{propagation network}, which calculates neighborhoods of the currently considered concepts; 2) an \textit{importance network}, which decides which concepts should be expanded; and 3) a \textit{task-based classifier} that predicts final output.
In contrast to~\cite{bhagat2023sample}, we utilize a dynamically determined number of iterations between the propagation- and importance-networks:
% Note that propagation and importance networks are running iteratively over $T$ steps dynamically determined steps until no additional concepts are expanded in a particular step. We detail all three networks in following sections.
% The Propagation Network computes embeddings for each node based on its neighbors within the context of the current image ($I$), while the Importance Network determines the relevance of nodes for potential future expansion given the same image.
% The GSNN iteratively propagates and extracts pertinent information encoded in the merged graph ($M$) corresponding to the image ($I$) by alternating between these networks over multiple dynamically determined steps. Each component, along with the final classifier,
% is detailed in subsequent sections.
% \textbf{The Propagation Network} updates the states of active nodes within the active graph \( C \). Given an input \( O \in {R}^{P \times F} \), where \( P \) represents the number of active nodes and \( F \) denotes the feature size for the latent node embedding, the Propagation Network generates an output \( O' \) encoding the information of each node's neighborhood by leveraging the graph structure encoded in an adjacency matrix \( A_M \).
% Mathematically, the update equation for the Propagation Network can be expressed as:
% \[ H^{(t+1)} = f(H^{(t)}, A_M) \]
% Where \( H^{(t)} \) represents the hidden states of nodes at iteration \( t \), and \( f \) is the update function which can be formulated as:
% \[ f(H^{(t)}, A) = \sigma(AH^{(t)}W) \]
% Here, \( \sigma \) denotes the activation function and \( W \) is the learnable weight matrix for layer \( l \).
\vspace{-2pt}
\textit{Propagation Network:}
% The Propagation Network updates active node states within the active graph \( C \) and outputs \( O' \in {R}^{P \times F} \), where \( P \) represents the number of active nodes and \( F \) denotes the node feature size. It leverages adjacency matrix \( A_M \) to encode neighborhood information. The update equation is give by: \( H^{(t+1)} = \sigma(A_MH^{(t)}W) \) Where \( H^{(t)} \) is the hidden states of nodes at iteration \( t \), \( \sigma \) is the activation function, and \( W \) is the weight matrix.
The propagation network updates active graph $\mathcal{M}^A_{t}$ in each interaction $t \in T$ and outputs an updated active graph $\mathcal{M}^A_{\hat{t}}$ in which current neighbors in graph $\mathcal{M}^A_{t}$ are considered where $\hat{t}$ indicates a partial iteration $t$.
% \( \mathbf{O^t} \in {\mathbb{R}}^{P \times F} \), where \( P \) represents the number of active nodes and \( F \) denotes the node feature size and $t$ is iteration. It utilizes the adjacency matrix \( \mathbf{A_M} \), allowing nodes to exchange information and refine their representations based on connections within the graph structure.
% \textbf{Importance Network:}
% The Importance Network operates in alternation with the Propagation Network across \( T \) cycles determined dynamically, determining whether adjacent nodes to currently active ones should be activated. This step is crucial, as indiscriminate node expansion at every step could become computationally burdensome for large graphs. The importance \( v_t^n \) of each node at iteration \( t \) is calculated conditioned on the global image context \( e_I \) as follows:
% \[ v_t^n = f_I(h_t^n, e_I) \]
% \( f_I(\cdot) \) denotes a multi-layer perceptron (MLP). Nodes with importance scores above a certain threshold \( \gamma \) are activated for the subsequent propagation cycle.
\vspace{-5pt}
\textit{Importance Network:}
The importance network works alongside the propagation network in an iterative manner, determining which adjacent nodes to $\mathcal{M}^A_{\hat{t}}$ should be added to $\mathcal{M}^A_{t+1}$ in the next step. At each iteration, the importance of each adjacent node is computed based on the set of current nodes as well as the overall input image. If a node exceeds a pre-defined importance threshold $\gamma$, it is added to $\mathcal{M}^A_{t+1}$.
Through this process, intuitively, the utility of additional concepts, such as compound concepts, are explicitly evaluated and searched for, given the currently active concepts and the whole image context.
% \textbf{Final Classifier:} The third and final step involves the classification of the active nodes \( P \subseteq N \) in the input image \( I \), where \( N \) represents the set of all possible nodes. These nodes are determined from the state representations
% \(O_{final}\) of all the expanded nodes in the active graph
% \( P \), along with the global image embedding \( e_I \) . Utilizing a single fully connected layer, a probability distribution over all the nodes is predicted as :
% \( c = f_C(O_{final}, e_I) \).
% In order to facilitate interpretation by a human user, we also provide the graph of active nodes \( P \), thereby offering insights into why certain classifications may have been made.
% \vspace{-5pt}
\textit{Task-based classifier:}
% The final step involves classifying the active nodes \( P \subseteq N \), where \( N \) represents all possible nodes. These nodes are classified using the state representations \( \mathbf{O_{final}} \) of nodes in the active graph \( M_P \), along with the global image embedding \( e_I \). Using a single fully connected layer, a probability distribution over all nodes is predicted: \( c = f_C(\mathbf{O_{final}}, e_I) \).
% Additionally, we provide the graph of active nodes \( P \) for human interpretation, offering insights into classifications.
After $T$ iterations, we employ a linear classifier over $\mathcal{M}^A_T$ to determine overall concepts that should be active for a scene. Intuitively, unlike importance networks, this conducts a final filter that can remove nodes.
\textbf{Dynamic Propagation:}
% Unlike fixed-step methods, our approach optimizes runtime efficiency and ensures graph stability by dynamically setting propagation steps.
Besides the joint graph search, one of our main contributions is the dynamic selection of $T$.
% In our work, the propagation and importance network work in unison through an iterative process.
Through our dynamic approach, we halt further expansion (i.e., further iterations) if no additional nodes
have been added in the previous iteration or if no added node exceeds an importance of $\lambda$.
Through this, we allow the network to propagate and include new information as long as it is deemed important enough while simultaneously saving additional iterations by setting a threshold $\lambda$.
We tune $\lambda$ and set it to $0.75$, which significantly reduces the runtime while retraining high performance.
% Graph expansion continues until uniform node importance diminishes. When certain nodes become significantly more important, indicating potential graph stability, we introduce an expansion threshold \(\lambda\). This criterion halts propagation when the maximum importance of new nodes falls below a ratio of previously expanded nodes' importance, preventing unnecessary expansion and maintaining stability.
% \textbf{Scene Graph Initialization:} We explore different initialization schemes for scene graph nodes crucial in graph merging. The exhaustive approach activates all nodes, even redundant ones, which may introduce noise. Alternatively, random initialization selects nodes and their neighbors randomly. We enhance randomness with a two-stage approach: first, selecting a node and activating its neighbors, then repeating the process for another node not previously selected. Combining results from both stages improves classification outcomes, offering flexibility and adaptability while reducing the impact of irrelevant objects.
% \subsection{Dynamic Propagation}
% In our approach, we propose a dynamic propagation mechanism for graph expansion, which adapts the number of propagation steps based on the evolving importance of graph nodes. Traditionally, graph propagation involves setting a fixed number of steps in advance, which may result in either under-expansion or over-expansion of the graph. To address this limitation, our approach dynamically adjusts the propagation steps to optimize runtime efficiency and ensure graph stability.
% Intuitively, the graph expansion continues as long as the importance of all newly activated nodes remains relatively uniform. However, when certain nodes begin to exhibit significantly higher importance compared to others, indicating a potential stability in graph expansion, we introduce an expansion threshold \(\lambda\).
% This expansion threshold serves as a criterion to halt graph propagation. Specifically, when the maximum importance of newly expanded nodes falls below a certain ratio of the importance of previously expanded nodes, we cease further graph propagation. This adaptive strategy ensures that the graph expansion process terminates once the significance of additional nodes diminishes relative to those already incorporated, thereby preventing unnecessary expansion and maintaining graph stability.
% \subsection{Importance Masking}
% Our classifier network produces task-specific probabilities for all concepts within the graph. However, leveraging the interpretable graph structure, we identify nodes that remained inactive during graph propagation. As these inactive nodes are deemed unimportant for our task, we disregard their contributions to the final classification.
% Consequently, we adopt a masking strategy to nullify the probabilities associated with these inactive nodes, effectively setting them to zero. This selective masking not only enhances the interpretability of our results but also improves the overall performance by focusing solely on relevant graph elements.
% By incorporating this masking technique, we effectively boost the accuracy and relevance of our classifier network outputs, aligning them more closely with the task objectives and the informative structure of the graph.
% \subsection{Scene Graph Initialization Schemes}\label{subsec:Initialization}
% The initial active nodes of the scene graphs play a crucial role in graph merging. We explore various initialization schemes to determine the most effective strategy. The exhaustive initialization scheme represents the most straightforward approach, where all nodes of the scene graph are initialized, with each object instance activated once, even if multiple instances of the same object are present in the scene.
% However, it is often unnecessary to activate all scene graph nodes, and the inclusion of certain objects can potentially introduce noise into the results. For instance, in scenes where objects like airplanes are visible in the sky, they may not be pertinent to the task at hand. This motivates the exploration of alternative initialization schemes that are less exhaustive.
% We first investigate random initialization as an alternative approach. In the random initialization 1 stage, we randomly select a node from the scene graph, activate it, and subsequently activate its neighboring nodes. However, relying solely on one randomly chosen node may not always provide optimal results. Hence, we introduce the random initialization 2 stage approach. Here, we repeat the random selection process for another node that was not among the previously activated nodes. We then aggregate the classification scores obtained from both stages to generate the final results.
% This multi-stage random initialization strategy allows for greater flexibility and adaptability in selecting relevant nodes while mitigating the potential impact of irrelevant objects on the final classification outcomes.
\section{Experimental Results}
\vspace{-5pt}
% In this section, we comprehensively evaluate our approach.
% In this section, we demonstrate our methods' performance on compound concept prediction. We will first discuss about the dataset and evaluation metrics used in the experiments and outline baseline models for comparison. Through comparison, we highlight our model's strengths and showcase its efficacy with qualitative examples and conduct an ablation study to analyze key components' contributions.
In this section, we evaluate the performance of our method on compound concept prediction in scene understanding and compare it with a set of relevant baselines, both using symbolic, or neural architectures.
% We discuss the dataset, evaluation metrics, and baseline models used.
% Further, we demonstrate its efficacy qualitatively and conduct an ablation study to analyze key components' contributions.
\textbf{Dataset and Evaluation Metrics.}
% \textbf{Write bout previous datasets Places 365 why are we not using them}
The ADE20K dataset \cite{zhou2017scene} comprises high-resolution images sourced from the Places 365 and SUN datasets. It consists of ground-truth object labels (concepts) along with overall scene labels (compound concepts).
For our study, we selected scene categories containing over $100$ images. To enhance dataset clarity, we removed certain ambiguous classes and merged them with their parent class. For instance, classes like \textit{Mountain Snowy} were merged into broader category \textit{Mountain}, while \textit{Attic} was consolidated under \textit{Bedroom}. With these preprocessing steps, our experiments involved $20$ classes.
We utilize top-1 accuracy to compare the efficacy of our approach.
\textbf{Compound Concept Prediction.}
Our model's performance is evaluated against two types of baselines: object-level and image-based baselines. \textbf{Object-level} baselines do not use the whole image representation and only use the objects extracted from the image. These baselines include the KG baseline, where all detected concepts in a particular image are passed to the KG for compound concept prediction. The GPT baseline employs GPT3.5 and GPT4 \cite{openai2024gpt4}, taking active concepts as input and predicting compound concepts among all given compound concepts. Additionally, the Human baseline reflects human accuracy in classifying compound concepts given a set of active concepts. Our approach on the object level does not use Image conditioning in Importance Network.
% \begin{table}[H]
% \begin{minipage}[t]{0.5\linewidth}
% \centering
% \footnotesize
% \begin{tabular}{|l|c|c|}
% \hline
% Method (Object) & Subset & Test set \\
% \hline
% Chat GPT 3.5 & 49\% & 43.71\% \\
% ChatGPT 4 & 65\% & 61.89\% \\
% Knowledge Graph & 57\% & 65.34\% \\
% Human Baseline & 81\% & NA \\
% \textbf{Ours} & 66\% & 74.52\% \\
% \hline
% \end{tabular}
% \caption{Object based methods}
% \label{tab:perfromance_object}
% \end{minipage}%
% \begin{minipage}[t]{0.5\linewidth}
% \centering
% \footnotesize
% \begin{tabular}{|l|c|c|}
% \hline
% Method (Image) & Subset & Test set \\
% \hline
% ViT Baseline & 82\% & 85.82\% \\
% % Ours (Without SG relations) & 94\% & 96.38\% \\
% GPT-4 Vision & 96\% & NA \\
% Human Baseline & 97\% & NA \\
% \textbf{Ours} & 94\% & 96.25\% \\
% \hline
% \end{tabular}
% \caption{Image Based Methods}
% \label{tab:performance_image}
% \end{minipage}
% \end{table}
% \vspace{-\baselineskip}
\vspace{-12pt}
\begin{table}[h]
\begin{minipage}[t]{0.5\linewidth}
\centering
\footnotesize
\begin{tabular}{lcc}
\hline
Method (Object) & Subset & Test set \\
\hline
Knowledge Graph & 57\% & 65.34\% \\
GPT3.5 & 49\% & 43.71\% \\
GPT4 & 65\% & 61.89\% \\
\textbf{Ours} & 66\% & 74.52\% \\
Human Baseline & 81\% & NA \\
\hline
\end{tabular}
\caption{Object-based methods}
\label{tab:perfromance_object}
\end{minipage}%
\begin{minipage}[t]{0.5\linewidth}
\centering
\footnotesize
\begin{tabular}{lcc}
\hline
Method (Image) & Subset & Test set \\
\hline
ViT & 82\% & 85.82\% \\
% Ours (Without SG relations) & 94\% & 96.38\% \\
GPT4-Vision & 96\% & NA \\
\textbf{Ours} & 94\% & 96.25\% \\
Human Baseline & 97\% & NA \\
\hline
\end{tabular}
\caption{Image-based Methods}
\label{tab:performance_image}
\end{minipage}
\end{table}
\vspace{-\baselineskip}
\vspace{-\baselineskip}
On the other hand, \textbf{image-level} baselines involve end-to-end image input. For the VIT \cite{DBLP:journals/corr/abs-2010-11929} baseline, we finetune a pretrained transformer for compound concept classification. The human baseline represents human accuracy in classifying compound concepts given the image. We also utilize GPT4-Vision as a baseline, where it is provided with an image input and prompted to classify the compound concept among all given compound concepts. These baselines provide a comprehensive evaluation of our model's performance across different input modalities and methodologies.
In consideration of the computational cost associated with running inference on GPT4-Vision, we partition our total test set, comprising $2200$ images, into a subset of $100$ images. This allows a fair comparison with GPT4-Vision. Each compound concept is represented by $5$ images in our smaller test set.
% \begin{table}[htbp]
% \centering
% \begin{tabular}{|l|c|c|}
% \hline
% Method (Object based) & Test Subset & Complete Test dataset \\
% \hline
% Chat GPT 3.5 & 49\% & 43.71\% \\
% ChatGPT 4 & 67\% & 61.89\% \\
% Knowledge Graph only & 57\% & 65.34\% \\
% SG + KG (Ours) & 66\% & 74.52\% \\
% Human Baseline & 81\% & NA \\
% \hline
% \end{tabular}
% \caption{Performance Comparison (Object based methods)}
% \label{tab:perfromance_object}
% \end{table}
% \vspace{-\baselineskip}
% \vspace{-\baselineskip}
% \vspace{-\baselineskip}
% \begin{table}[htbp]
% \centering
% \begin{tabular}{|l|c|c|}
% \hline
% Method (Image Based) & Test Subset & Complete Test Dataset \\
% \hline
% ViT Baseline & 82\% & 85.82\% \\
% % Ours (Without SG relations) & 94\% & 96.38\% \\
% Ours & 94\% & 96.25\% \\
% GPT-4 Vision & 96\% & NA \\
% Human Baseline & 97\% & NA \\
% \hline
% \end{tabular}
% \caption{Performance Comparison (Image Based Methods)}
% \label{tab:performance_image}
% \end{table}
% \vspace{-\baselineskip}
\begin{figure}[t]
\centering
\includegraphics[width=0.9\textwidth]{Figs/Qualitative_Results.pdf}
\caption{Qualitative examples } \label{qualitative}
\vspace{-15pt}
\end{figure}
Our method without image conditioning in Tab. \ref{tab:perfromance_object} surpasses all baselines, including GPT and KG baselines. On other hand, image-conditioned method in Tab. \ref{tab:performance_image} exhibits slightly lower performance compared to GPT on the smaller test set. However, we achieve higher accuracy on the complete set and approach human-level performance. The results highlight the effectiveness of our approach in tackling compound concept classification tasks. Additionally,
% we showcase qualitative results, particularly focusing on cases where ChatGPT 4 fails while our method achieves nearly 100\% accuracy. These results further underscore the capabilities of our approach.
we showcase scenarios where GPT4-Vision fails in Fig. \ref{qualitative}, contrasting with our near-perfect accuracy. Our interpretable approach excels in complex scenarios, leveraging scene and knowledge graphs to outperform GPT4-Vision in challenging tasks.
\vspace{-12pt}
\section{Conclusion}
\vspace{-10pt}
In summary, this work presents a novel approach for compound concept predictions utilizing scene and knowledge graphs.
Through our method, we propose an effective approach to merge spatial information with general knowledge inference and demonstrate its efficacy compared to a set of state-of-the-art baselines.
In future work, we intend to expand our approach to video understanding, thus, incorporating spatio-temporal reasoning with knowledge graphs.
% We also present the ablation studies conducted for our method, as summarized in Table \ref{tab:ablation_studies}. The first row showcases results for the network without any image conditioning and scene-specific object embeddings. Further enhancements are observed in the second row, where we introduce Importance Network-based image conditioning, leading to a further improvement of 18\%. Additionally, the inclusion of object embeddings and importance masking in the third row yields improvements of 1.5\% and 2\%, respectively. These ablation studies provide valuable insights into the contributions of each component to the overall performance of our method.
% \subsection{Implementation Details}
% \subsection{Qualitative Examples}
% In this section, we provide qualitative examples in complex scenarios for compound concept prediction, comparing our method with ChatGPT 4 Vision on these instances. The examples are depicted in Figure (cite). Notably, ChatGPT 4 Vision fails in nearly all cases where our method yields accurate results, owing to its interpretability. This underscores the efficacy of our interpretable approach, which incorporates both the scene graph and knowledge graph. By leveraging the interpretive capabilities of our method, we achieve enhanced performance in challenging scenarios, demonstrating the utility of our approach in compound concept prediction tasks.
% \subsection{Ablations}
% In this section, we present the ablation studies conducted for our method, as summarized in Table \ref{tab:ablation_studies}. The first row showcases results for the network without any image conditioning and scene-specific object embeddings. Further enhancements are observed in the second row, where we introduce Importance Network-based image conditioning, leading to a further improvement of 18\%. Additionally, the inclusion of object embeddings and importance masking in the third row yields improvements of 1.5\% and 2\%, respectively. These ablation studies provide valuable insights into the contributions of each component to the overall performance of our method.
% \begin{table}[htbp]
% \centering
% \small
% \begin{tabular}{|l|c|c|}
% \hline
% \textbf{Configuration} & \textbf{Test Subset} & \textbf{Complete Test Set} \\
% \hline
% Network without image conditioning & 66\% & 74.52\% \\
% % Network with image conditioning \cite{bhagat2023sample} & \% & \%4 \\
% Importance Network-based image conditioning & 90\% & 92.56\% \\
% Object embeddings inclusion & 91\% & 94.17\% \\
% Importance masking inclusion & 94\% & 96.38\% \\
% \hline
% \end{tabular}
% \caption{Ablation Study Results}
% \label{tab:ablation_studies}
% \end{table}
% \vspace{-\baselineskip}
% \section{Conclusion}
% \subsection{A Subsection Sample}
% Please note that the first paragraph of a section or subsection is
% not indented. The first paragraph that follows a table, figure,
% equation etc. does not need an indent, either.
% Subsequent paragraphs, however, are indented.
% \subsubsection{Sample Heading (Third Level)} Only two levels of
% headings should be numbered. Lower level headings remain unnumbered;
% they are formatted as run-in headings.
% \paragraph{Sample Heading (Fourth Level)}
% The contribution should contain no more than four levels of
% headings. Table~\ref{tab1} gives a summary of all heading levels.
% \begin{table}
% \caption{Table captions should be placed above the
% tables.}\label{tab1}
% \begin{tabular}{|l|l|l|}
% \hline
% Heading level & Example & Font size and style\\
% \hline
% Title (centered) & {\Large\bfseries Lecture Notes} & 14 point, bold\\
% 1st-level heading & {\large\bfseries 1 Introduction} & 12 point, bold\\
% 2nd-level heading & {\bfseries 2.1 Printing Area} & 10 point, bold\\
% 3rd-level heading & {\bfseries Run-in Heading in Bold.} Text follows & 10 point, bold\\
% 4th-level heading & {\itshape Lowest Level Heading.} Text follows & 10 point, italic\\
% \hline
% \end{tabular}
% \end{table}
% \noindent Displayed equations are centered and set on a separate
% line.
% \begin{equation}
% x + y = z
% \end{equation}
% Please try to avoid rasterized images for line-art diagrams and
% schemas. Whenever possible, use vector graphics instead (see
% Fig.~\ref{fig1}).
% \begin{figure}
% \includegraphics[width=\textwidth]{fig1.eps}
% \caption{A figure caption is always placed below the illustration.
% Please note that short captions are centered, while long ones are
% justified by the macro package automatically.} \label{fig1}
% \end{figure}
% \begin{theorem}
% This is a sample theorem. The run-in heading is set in bold, while
% the following text appears in italics. Definitions, lemmas,
% propositions, and corollaries are styled the same way.
% \end{theorem}
% %
% % the environments 'definition', 'lemma', 'proposition', 'corollary',
% % 'remark', and 'example' are defined in the LLNCS documentclass as well.
% %
% \begin{proof}
% Proofs, examples, and remarks have the initial word in italics,
% while the following text appears in normal font.
% \end{proof}
% For citations of references, we prefer the use of square brackets
% and consecutive numbers. Citations using labels or the author/year
% convention are also acceptable. The following bibliography provides
% a sample reference list with entries for journal
% articles~\cite{ref_article1}, an LNCS chapter~\cite{ref_lncs1}, a
% book~\cite{ref_book1}, proceedings without editors~\cite{ref_proc1},
% and a homepage~\cite{ref_url1}. Multiple citations are grouped
% \cite{ref_article1,ref_lncs1,ref_book1},
% \cite{ref_article1,ref_book1,ref_proc1,ref_url1}.
% \begin{credits}
% \subsubsection{\ackname} A bold run-in heading in small font size at the end of the paper is
% used for general acknowledgments, for example: This study was funded
% by X (grant number Y).
% \subsubsection{\discintname}
% It is now necessary to declare any competing interests or to specifically
% state that the authors have no competing interests. Please place the
% statement with a bold run-in heading in small font size beneath the
% (optional) acknowledgments\footnote{If EquinOCS, our proceedings submission
% system, is used, then the disclaimer can be provided directly in the system.},
% for example: The authors have no competing interests to declare that are
% relevant to the content of this article. Or: Author A has received research
% grants from Company W. Author B has received a speaker honorarium from
% Company X and owns stock in Company Y. Author C is a member of committee Z.
% \end{credits}
% %
% ---- Bibliography ----
%
% BibTeX users should specify bibliography style 'splncs04'.
% References will then be sorted and formatted in the correct style.
%
\vspace{-10pt}
\bibliographystyle{splncs04}
\bibliography{refs}
%
% \begin{thebibliography}{8}
% \end{thebibliography}
\end{document}
\begin{thebibliography}{1}
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\urlprefix}{URL }
\providecommand{\doi}[1]{https://doi.org/#1}
\bibitem{bhagat2023sample}
Bhagat, S., Stepputtis, S., Campbell, J., Sycara, K.: Sample-efficient learning of novel visual concepts. In: Conference on Lifelong Learning Agents. pp. 637--657. PMLR (2023)
\bibitem{cimpoi2015deep}
Cimpoi, M., Maji, S., Vedaldi, A.: Deep filter banks for texture recognition and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3828--3836 (2015)
\bibitem{DBLP:journals/corr/abs-2010-11929}
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. CoRR \textbf{abs/2010.11929} (2020), \url{https://arxiv.org/abs/2010.11929}
\bibitem{Girshick_2015_ICCV}
Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (December 2015)
\bibitem{li2023airloc}
Li, B., Scherer, S., Lin, Y.J., Wang, C., et~al.: Airloc: Object-based indoor relocalization. arXiv preprint arXiv:2304.00954 (2023)
\bibitem{lundeen2019autonomous}
Lundeen, K.: Autonomous scene understanding, motion planning, and task execution for geometrically adaptive robotized construction work. Ph.D. thesis (2019)
\bibitem{openai2024gpt4}
OpenAI, Achiam, J., Adler, S., et~al.: Gpt-4 technical report (2024)
\bibitem{zhou2017scene}
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 633--641 (2017)
\bibitem{zhuang2017towards}
Zhuang, B., Liu, L., Shen, C., Reid, I.: Towards context-aware interaction recognition for visual relationship detection. In: Proceedings of the IEEE international conference on computer vision. pp. 589--598 (2017)
\end{thebibliography}