G-TAD: Sub-Graph Localization for Temporal Action Detection
Extracted source text · arXiv 1911.11462
\documentclass[10pt,twocolumn,letterpaper]{article}
\usepackage{cvpr}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
%% other packages
\usepackage{multirow}
\usepackage{tabularx}
\usepackage{booktabs}
\usepackage{amssymb}% http://ctan.org/pkg/amssymb
\usepackage{pifont}% http://ctan.org/pkg/pifont
\usepackage{setspace}
%% Used for comment
\usepackage[dvipsnames]{xcolor}
\newcommand{\AT}[1]{{\color{Blue}[#1]}}
\newcommand{\DR}[1]{{\color{PINK}[#1]}}
\newcommand{\SQ}[1]{{\color{Salmon}[#1]}}
\newcommand{\FX}[1]{{\color{ForestGreen}#1}}
\newcommand{\BG}[1]{\textbf{{\color{red}[
BG: #1
]}}}
\newcommand{\floor}[1]{\lfloor #1 \rfloor}
\newcommand{\ceil}[1]{\lceil #1 \rceil}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\newcommand{\G}{\mathcal{G}}
\newcommand{\F}{\mathcal{F}}
\newcommand{\E}{\mathcal{E}}
\newcommand{\V}{\mathcal{V}}
\renewcommand{\H}{\mathcal{H}}
\usepackage{algorithm}
\usepackage{algorithmic}
\renewcommand{\algorithmicrequire}{ \textbf{Input:}}
\renewcommand{\algorithmicensure}{ \textbf{Output:}}
% Include other packages here, before hyperref.
% If you comment hyperref and then uncomment it, you should delete
% egpaper.aux before re-running latex. (Or just hit 'q' on the first latex
% run, let it finish, and you should be clear).
\usepackage[pagebackref=true,breaklinks=true,letterpaper=true,colorlinks,bookmarks=false]{hyperref}
\cvprfinalcopy % *** Uncomment this line for the final submission
\def\cvprPaperID{576} % *** Enter the CVPR Paper ID here
\def\httilde{\mbox{\tt\raisebox{-.5ex}{\symbol{126}}}}
% Pages are numbered in submission mode, and unnumbered in camera-ready
%\ifcvprfinal\pagestyle{empty}\fi
\begin{document}
%%%%%%%%% TITLE
% \title{G-TAD: Temporal Action Detection via Sub-Graph Detection}
\title{G-TAD: Sub-Graph Localization for Temporal Action Detection}
\author{Mengmeng Xu,\quad Chen Zhao,\quad David S. Rojas,\quad Ali Thabet\quad and\quad Bernard Ghanem\\
Visual Computing Center\\
King Abdullah University of Science and Technology (KAUST), Saudi Arabia\\
{\tt\footnotesize \{Mengmeng.Xu, chen.zhao, davidsantiago.blanco, ali.thabet, bernard.ghanem\}@kaust.edu.sa}}
\maketitle
%\thispagestyle{empty}
%%%%%%%%% ABSTRACT
\begin{abstract}
%\FX{This is a full draft!}
Temporal action detection is a fundamental yet challenging task in video understanding. Video context is a critical cue to effectively detect actions, but current works mainly focus on temporal context, while neglecting semantic context as well as other important context properties. In this work, we propose a graph convolutional network (GCN) model to adaptively incorporate multi-level semantic context into video features and cast temporal action detection as a sub-graph localization problem. Specifically, we formulate video snippets as graph nodes, snippet-snippet correlations as edges, and actions associated with context as target sub-graphs. With graph convolution as the basic operation, we design a GCN block called GCNeXt, which learns the features of each node by aggregating its context and dynamically updates the edges in the graph. To localize each sub-graph, we also design a SGAlign layer to embed each sub-graph into the Euclidean space. Extensive experiments show that G-TAD is capable of finding effective video context without extra supervision and achieves state-of-the-art performance on two detection benchmarks. On ActityNet-1.3, we obtain an average mAP of $34.09\%$; on THUMOS14, we obtain $40.16\%$ in mAP@0.5, beating all the other one-stage methods.
% incorporate temporal context around actions, while neglecting semantic.
% Temporal action detection is a fundamental yet challenging task in video understanding.
% Many state-of-the-art methods predict the action probabilities using the features during the action and at some different, but related, moments in the video. However, action boundary is usually blurred, not well-defined and with a flexible uncertainty range because most actions are defined by human experience and do not have a precise temporal border.
% In this paper, we propose a novel action detection architecture that represents the video as a graph and adopt graph convolution with temporal convolution to encode each frame and their neighbour in both temporal domain and feature domain. G-TAD achieves better performance detecting action instances that have more context in the background, which can be attributed to two aspects. First, we consider the video frames, which have long-range dependency across the whole video, that can help classify the target frame. Secondly, each sub-graph is aligned into the temporal space to classify the action.
\end{abstract}
%%%%%%%%% BODY TEXT
\section{Introduction}
\input{01_intro.tex}
\section{Related Work}
\input{02_related.tex}
\section{Proposed Method}
\input{03_method.tex}
\section{Experiment}
\input{04_experiment.tex}
\section{Conclusion}
\input{05_conclusion.tex}
{\small
\bibliographystyle{ieee_fullname}
\bibliography{00_GTAD_Arxiv}
}
\onecolumn
\clearpage
\newpage
\section{Appendix}
\input{09_supp.tex}
\end{document}
Video understanding has gained much attention from both academia and industry over recent years, given the rapid growth of videos published in online platforms. Temporal action detection is one of the interesting but challenging tasks in this area.
It involves detecting the start and end frames of action instances, as well as predicting their class labels. This is onerous especially in long untrimmed videos.
% To achieve video action detection, the designed algorithm is not only about predicting the class label of possible actions, which is often called action classification task, but also aims to obtain temporal boundaries that is to figure out the starting and ending frame of a certain action instance in a long untrimmed video.
\begin{figure}[t]
\centering
\includegraphics[trim={2cm 2cm 3cm 2.5cm},width=9cm,clip]{img/GTAD_Intro.pdf}
% \small
\caption{\textbf{Graph formulation of a video.} Nodes: video snippets (a video snippet is defined as consecutive frames within a short time period). Edges: snippet-snippet correlations. Sub-graphs: actions. There are 4 types of nodes: action, start, end, and background, shown as colored dots. There are 2 types of edges: (1) temporal edges, which are pre-defined according to the snippets' temporal order; (2) semantic edges, which are learned from node features.}
\label{fig:intro}
\end{figure}
% \BG{this figure doesnt make sense; there seems to be only one red point; there is no time axis; there are several things that need to be done here; they are better communicated in person so come see me}
Video context is an important cue to effectively detect actions. Here, we refer to context as
% Video context, a group of
frames that are outside the target action but carry valuable indicative information of it.
% is an important clue to tackle this problem.
Using video context to infer potential actions is natural for human beings. In fact, empirical evidence shows that humans can reliably guess or predict the occurrence of a certain type of action by only looking at short video snippets where the action does not happen~\cite{detad, Alwassel2017ActionSS}. % \BG{reference DETAD}.
% For example, human can guess the action class and location by only watching a short snippet of the video in which the action does not happen.
% Recently, many approaches resort to the extension of the boundary of a possible temporal action (\textit{aka.} proposal) to encode the context feature to facilitate action detection.
% not clear -> what about now?
Therefore, incorporating context into temporal action detection has become an important strategy to boost detection accuracy in the recent literature \cite{ dai2017temporal, gao2017turn, chao2018rethinking, Long2019GaussianTA, TSN2016ECCV, zhao2017temporal, lin2018bsn}. Researchers have proposed various ways to take advantage of video context, such as extending temporal action boundaries by a pre-defined ratio~\cite{dai2017temporal, gao2017turn, TSN2016ECCV, zhao2017temporal, lin2018bsn}, using dilated convolution to encode context into features~\cite{chao2018rethinking}, and aggregating context features implicitly by way of a Gaussian curve~\cite{Long2019GaussianTA}. All these methods only utilize temporal context, which precedes or follows an action instance in its immediate temporal neighborhood. However, real-world videos vary dramatically in temporal extent, action content, and even editing preferences. The use of such temporal context does not fully exploit the rich merits of video context, and it may even impair detection accuracy if not properly designed for underlying videos.
So, what properties characterize desirable video context for the purpose of accurate action detection? First, context should be semantically correlated to the target action other than merely temporally located in its vicinity. Imagine the case where we manually stitch an action clip into some irrelevant frames, the abrupt scene change surrounding the action would definitely not benefit the action detection. On the other hand, snippets located at a distance from an action but containing similar semantic content might provide indicative hints for detecting the action. Second, context should be content-adaptive rather than manually pre-defined. Considering the vast variation of videos, context that helps to detect different action instances could be different in lengths and locations based on the video content. Third, context should be based on multiple semantic levels, since using only one form/level of context is unlikely to generalize well.
% Some feature engineering work adapt the action boundary frames and their temporal neighbours as the video context, by either using a fixed ratio to extend every action boundary, or creating boundary features beside the action.
% \textit{e.g.} these methods define the video context only around the temporal action.
% To encode the context feature to facilitate action detection, some feature engineering work is explored to design a extended temporal action boundary, either using a certain ratio for the extension or creating boundary feature beside the proposal, \textit{e.g.} these methods define the video context only around the temporal action. %field before recognising the temporal action.
%
% In practice, the boundary length is usually designed to be proportional to the video length or anchor length, and the ratio highly depends on human experience and dataset prior.
% Obtaining a more flexible and accurate action context can help to improve both action localization and classification.
% ref: http://openaccess.thecvf.com/content_cvpr_2017/papers/Heilbron_SCC_Semantic_Context_CVPR_2017_paper.pdf
% Context in a particular video, including scenes and objects, plays an important role for automated methods to recognize correct actions. Although
% long distance dependency
% \FX{describe the problem in graph perspective}
% In this work, we predict action classes and boundaries using both short and long temporal-distance frames.
In this paper, we endow video context with all the above properties by casting action detection as a sub-graph localization problem based on a graph convolutional network (GCN)~\cite{kipf2016semi}. We represent each video sequence as a graph, each snippet as a node, each snippet-snippet correlation as an edge, and target actions associated with context as sub-graphs, as shown in Fig.~\ref{fig:intro}. The context of a snippet is considered to be all snippets connected to it by an edge in a video graph. We define two types of edges --- temporal edges and semantic edges, each corresponding to temporal context and semantic context, respectively. Temporal edges exist between each pair of neighboring snippets, whereas semantic edges are dynamically learned from the video features at each GCN layer. Hence, multi-level context of each snippet is gradually aggregated into the features of the snippet throughout the entire GCN. The structure of each GCN block is inspired by ResNeXt~\cite{xie2017aggregated}, so we name this GCN-based feature extractor GCNeXt.
The pipeline of our proposed Graph-Temporal Action Detection method, dubbed G-TAD, is analogous to faster R-CNN~\cite{girshick2015fast, ren2015faster} in object detection. There are two critical designs in G-TAD. First, GCNeXt, which generates context-enriched features, corresponds to the backbone network, analagous to a series of CNN layers in faster R-CNN. Second, to mimic region of interest (RoI) alignment~\cite{he2017mask} in faster R-CNN, we design a sub-graph alignment (SGAlign) layer to generate a fixed-size representation for each sub-graph and embed all sub-graphs into the same Euclidean space. Finally, we apply a classifier on the features of each sub-graph to obtain detection results. We summarize our contributions as follows.
% We design a new features extractor, GCNeXt, to leverage our video graph representation. GCNeXt extracts features using both temporal and dynamic edges
% Nodes can be labelled as action, start, end, or background.
% the input data is unstructured, unordered, and permutable
% At each graph node, a GCN performs convolutions over a set of neighbors defined by an adjacency matrix, which can either be fixed (computed once for each graph) or re-computed at each GCN layer using the current node features. We leverage this idea in our new modules.
% GCNeXt uses a fixed adjacency for temporal edges to help localize sub-graphs accurately and a dynamic adjacency to discover long-range dependencies as context.
% \BG{we dont talk about the fact that the dynamic edges allow for the context to be adaptive; give the example of the same action being placed in two different contexts; with the dynamic edges the context is not the same in both cases which is advantageous; being adaptive is lacking in previous strategies to encode context for TAD}
% In this work, we utilize long temporal-distance frames to the action as its context and construct graphs from videos.
% In this work, we rephrase the temporal action detection as a sub-graph localization problem and utilizes long temporal-distance frames as the video context.
% login by my account, you missed me, this is the reason :)
%
% Fig.~\ref{fig:intro} shows an example of a video graph with all node and edge types.
%
% are predefined from the temporal order of snippets, and dynamic edges are inferred from node features. % and enable the network to convolve any correlated snippets in the video.
% The context snippet is defined as a background snippet which has dynamic dependency to an action snippet.
% Given this video representation, both action and background represent sub-graphs of the full video graph. We therefore cast action detection as a \textit{sub-graph localization} problem.
% In this way, both of action and background become sub-graphs in the video graph, and temporal action detection can be rephrased as a sub-graph localization problem. %, and we can encode local and global information to sub-graphs.
% \FX{Talking about Faster-RCNN and our work}
% deploys a series of two-dimensional convolution layers to extract features from the input images. Then a region of interest pooling or alignment~\cite{he2017mask} layer crop and resize the feature map for the following classification and regression.
%Given a collection of proposals.
% We mainly arm Faster-RCNN with two new modules to target unstructured data after casting the Fast-RCNN structure in the graph domain.
% GCNeXt blocks are proposed as an analogue of 2D CNN and designed SGAlign layer functions as regular RoI Alignment.
% Faster-RCNN is armed with two new modules to target structured data. % good!
% Firstly, we propose Graph Convolution Net cross both temporal and dynamic graphs (GCNeXt) blocks as an analogue of 2D CNN.
% Secondly, designed Sub-Graph of Interest Alignment (SGAlign) layer functions as the replacement of regular RoI Alignment.
%
% GCNs have shown significant success segmentation in point cloud \cite{Li2019CanGG}, object detection in images \cite{Yang2018GraphRF}, and human joints modelling in videos \cite{Yan2018SpatialTG}, where the input data is unstructured, unordered, and permutable \BG{refs?}.
% We leverage this idea in our GCNeXt module. We use a fixed adjacency for temporal edges, and a dynamic one for dynamic edges. Temporal edges help localize sub-graphs accurately, while dynamic ones discovers long-range dependencies as context.
% We also apply convolutions through multiple paths using a split-transform-merge strategy~\cite{xie2017aggregated}. Multi-path transformation increases the module cardinality without going deeper or wider.
% However, GCN is mostly used in classification and segmentation, or element-wise segmentation task because the ordered information will be lost.
% %
% GCNeXt module inherits the advantages of GCN: it temporarily constructs dynamic graphs based on the feature representation of each node pair. Meanwhile, it also encrypts the ordering information in the temporal graphs.
% %
% Graph convolution is applied to both temporal and dynamic graphs through multiply paths, respectively, using the split-transform-merge strategy~\cite{xie2017aggregated}. We assume the temporal graph help the network localize the sub-graph accurately, the dynamic graph discovers long-range dependencies as context, and multi-path transformation increases the module cardinality without going deeper or wider.
% comment following sentence if needed
% Moreover, the repeating GCNeXt modules aggregate a set of transformations with the same topology as a analogue of ResNeXt~\cite{xie2017aggregated} block in image processing.
% RoI pooling is a differential process that arranges the feature from 2D image domain to 1D object proposal domain.
% % As a vital operation in 2D object detection problems, Region of interest (RoI) pooling crops and resizes a local area of the feature map in the forward pass, and updates all the elements in the area in the back-propagation.
% However, this operation can only be applied on the Cartesian coordinate system to up/down-sample the grid-data based on the input/output shape ratio.
% %
% SGAlign layer embeds graph nodes to a Euclidean space based on their intrinsic and learned characteristics and interpolate their feature. Given the sub-graph area in the embedded space, we sample and resize the features in the area to generate the new presentation.
% We evaluate our proposed method on THUMOS-14 and AcitivityNet-1.3, two popular benchmarks for temporal action detection. Extensive experiments show that G-TAD is able to find global video context without extra supervision and achieves state-of-the-art results on both datasets. % under different metrics.
\noindent
\textbf{(1)} We present a novel GCN-based video model to fully exploit video context for effective temporal action detection. Using this video GCN representation, we are able to adaptively incorporate multi-level semantic context into the features of each snippet.
\noindent
\textbf{(2)} We propose G-TAD, a new sub-graph detection framework to localize actions in video graphs. G-TAD includes two main modules: GCNeXt and SGAlign. GCNeXt performs graph convolutions on video graphs, leveraging both temporal and semantic context. SGAlign re-arranges sub-graph features in an embedded space suitable for detection.
% (2) We propose our GCNeXt block to perform graph convolution cross different types of edges in order to extract the snippet features from both temporal context and correlated snippets,
% (3) We design SGAlign layer to interpolate and rescale the graph nodes in the embedding space with the aim of extracting the sub-graph features.
\noindent
\textbf{(3)} G-TAD achieves state-of-the-art performance on two popular action detection benchmarks. On ActityNet-1.3, it achieves an average mAP of $34.09\%$. On THUMOS-14, it reaches $40.16\%$ mAP@0.5, beating all contemporary one-stage methods.
% we increase record on mAP@0.7 score to $23.42\%$ from $20.8\%$ in TAL-Net~\cite{chao2018rethinking}, and the performance of mAP@0.5, $40.16\%$ beats all one-stage methods.
% To the best of our knowledge, we are the first to incorporate long range temporal dependencies in the task of action localization in video.
% \BG{i feel that we need to emphasize the point that dynamic edges allow for the context we encode to adapt to the underlying video, as compared to other detectors that have a fixed temporal definition of what context should be}
% \BG{we keep on repeating long range dependencies, but it's not necessarily about range. It's more about finding snippets in your context that help you localize and classify. These snippets can be short range in some videos and long range in others}
\subsection{Video Representation}
\begin{figure*}
\centering
\includegraphics[trim={0.5cm 3cm 1.5cm 2.5cm},width=17cm,clip]{img/GTAD_Arch.pdf}
\caption{\textbf{Overview of G-TAD architecture.} The input of G-TAD is a sequence of snippet features. We first extract features using $b=3$ GCNeXt blocks, which gradually aggregate both temporal and multi-level semantic context. Semantic context, encoded in semantic edges, is dynamically learned from features at each GCNeXt layer.
Then we feed the extracted features into the SGAlign layer, where sub-graphs defined by a set of anchors are transformed to a fixed-size representation in the Euclidean space.
%Features are arranged in a 1D vector, where each entry represents a sub-graph location.
Finally, the localization module scores and ranks the sub-graphs for detection.
% \BG{make it look like there are more than 3 GCNeXt layers. Put dot dot dot between them and define a variable in text to represent its depth. In the experiments, mention that we choose this variable to be 3}
% from both of their temporal neighbours and long-distance correlated nodes. Then the encoded feature is fed into two modules. The node classification module predicts if the input node corresponds to a start or end snippets, and the sub-graph detection module gives the confidence score that the sub-graph presents an action.
% In inference time, we fuse the score from both modules and predict the action location and class in a Bayesian way with video prior. \AT{Where is the node classification module in the graph? Also it says SubGraphAlign instead of SGAlign. Also it seems we use sub graph detection and sub graph alignment interchangeable, are they exactly the same thing? If yes then we need to clarify it in the text and figures}\FX{TODO: update caption.}
}
\label{fig:arch}
\end{figure*}
\noindent \textbf{Action Recognition}. Many CNN based methods have been proposed to address the action recognition task. Two-stream networks \cite{feichtenhofer2016convolutional,simonyan2014two,wang2015towards} use 2D CNNs to extract frame features from RGB and optical flow sequences.
% the visual feature from 2D CNNs and optical flows.
These 2D CNNs can be designed from scratch~\cite{he2016deep, simonyan2014very} or adapted from image recognition tasks~\cite{deng2009imagenet}. Other methods~\cite{tran2015learning,carreira2017quo,qiu2017learning,xu2017r} use 3D CNNs to encode spatio-temporal information from the original video. % look into the temporal dimension from the original video volume and use 3D CNNs to encode the video.
In our work, we use the pre-trained action recognition model in \cite{xiong2016cuhk,wang2016temporal} to extract video snippet features as G-TAD input, and use graph convolution as an analogue for 2D or 3D CNNs.
% %
% % Action recognition is a fundamental and important task of video understanding area.
% % With the development of deep learning, many CNN based methods have been proposed to solve the action recognition task.
% %
% There are two main frameworks for this task.
% %
% The two-stream networks \cite{feichtenhofer2016convolutional,simonyan2014two,wang2015towards} extract the visual feature from 2D CNNs and optical flows. The 2D CNN can be well-designed~\cite{he2016deep, simonyan2014very} and pretrained from image recognition datasets~\cite{deng2009imagenet}. Thus it gives a strong prior to visual appearance and can be used efficiently.
% %
% While other methods~\cite{tran2015learning,carreira2017quo,qiu2017learning,xu2017r} look into the temporal dimension from the original video volume and use 3D CNNs to encode the video. These methods are capable of learning temporal features and can have a flexible model scale by separating spatial and temporal convolutions~\cite{tran2018closer}.
% %
% Although most of the approaches regard the video as grid data, In G-TAD, we construct video as graphs where the graph node represents each snippet, while the snippet feature is extracted from pre-trained action recognition models.
\noindent \textbf{Action Detection}. The goal of temporal action detection is to predict the boundaries of action instances and their categories in untrimmed videos. Most methods~\cite{shou2016temporal,singh2016untrimmed,zhao2017temporal, zeng2019graph, chao2018rethinking, lin2018bsn} divide the task into two stages: temporal proposal generation and classification/regression of proposals. For proposal generation, they either predict action proposals using handcrafted anchors \cite{buch2017sst,caba2016fast, escorcia2016daps, gao2017turn,shou2016temporal} , or by classifying starting/ending snippets~\cite{zhao2017temporal,lin2018bsn}. Others~\cite{lin2017single,buch2017sst,lin2019bmn} tackle the problem using a single-stage model, where actions are detected directly. G-TAD is a single-stage model that scores pre-defined anchors with action confidences. We introduce a starting/ending snippet classification loss as a regularizer. % They use snippet confidence score to add/remove proposals, and score the proposal in another model branch.
% %
% The goal of temporal action detection task is to predict action instances' boundaries and categories in untrimmed videos. Most methods~\cite{shou2016temporal,singh2016untrimmed,zhao2017temporal, zeng2019graph, chao2018rethinking, lin2018bsn} take it as a two-stage problem and divide it into temporal proposal generation and classifications/regression stages, while others~\cite{lin2017single,buch2017sst,lin2019bmn} tackle the problem by a single-stage model.
% %
% For proposal generation task, most previous works \cite{buch2017sst,caba2016fast, escorcia2016daps, gao2017turn,shou2016temporal} use the anchor-based method that generates proposals with a set of pre-defined temporal boundaries.
% %
% There are also some methods \cite{zhao2017temporal,lin2018bsn} that produce proposals from classifying starting/ending snippets. They use snippet confidence score to add/remove proposals, and score the proposal in another model branch.
% However, we can still cascade a second stage on the prediction action boundaries to improve the detection performance \BG{this sentence seems out of place; do we have experiments to show that this can be done; if not, then remove it or rewrite}.
% \noindent \textbf{Video Context}. Video context has been explored in several approaches to boost action recognition performance ~\cite{li2007and, wei2016modeling, wu2007scalable, prest2012explicit, koppula2013learning, han2009selection, gupta2007objects}.
% %
% Caba \textit{et al.}~\cite{heilbron2017scc} exploited semantic information garnered from action-object and action-scene dependencies at large scale.
% Some action localization methods~\cite{TSN2016ECCV, zhao2017temporal, lin2018bsn} incorporate video frames close to the action's boundaries to encode local fixed temporal context.
% % Other works used the global context information.
% % Marszaek \textit{et al.} ~\cite{marszalek2009actions} compared the relevance of the co-occurrence between actions and scenes, and designed visual representations for retrieving small actions in movie clips.
% % Following this idea, Jain \textit{et al.} ~\cite{jain201515} conducted a large-scale study that reveals a strong co-occurrence between actions and a sparse number of objects.
% Recently, Wang \textit{et al.}~\cite{wang2019nlnn} proposed averaging all the snippet features with attention weights to gather non-local features for the video classification task.
% %
% % These methods require well-trained model on object detection or scene classification.
% In G-TAD and due its explicit usage of temporal and dynamic edges, we find long-range snippet-snippet dependencies across the video, and only aggregate features among snippets that have these dependencies.
\subsection{GCN in Videos}
\noindent \textbf{Graphs in Video Understanding}. Graphs have been widely used for data representation in various video understanding tasks, such as video feature representation~\cite{liu2019learning},
video classification~\cite{wang2018videos, chen2019graph}, and action localization~\cite{zeng2019graph}.
%
In action recognition, Liu \textit{et al.} ~\cite{liu2019learning} view a video tensor as a 3D point cloud in the spatial-temporal space.
Wang \textit{et al.} ~\cite{wang2018videos} represent a video as a space-time region graph, in which the graph nodes are defined by object region proposals. In action detection, Zeng \textit{et al.} ~\cite{zeng2019graph} consider temporal action proposals as nodes in a graph, and refine their boundaries and classification scores based on the established proposal-proposal dependencies. Differently from previous works, G-TAD takes video snippets as nodes in a graph and form dependencies between them based on both their temporal ordering and semantic similarity. %while the short-distance spatial-temporal information is extracted and preserved in the snippet feature.
\noindent
\textbf{Graph Convolutions.}
Graph Convolutional Networks (GCNs)~\cite{kipf2016semi} are widely used for non-Euclidean structures. These years have also seen their successful application in computer vision tasks due to their versatility and effectiveness, such as 3D object detection~\cite{Gkioxari2019Mesh} and point cloud segmentation ~\cite{wang2018dynamic, xie2019clouds}.
%
Meanwhile, various GCN architectures are proposed for more effective and flexible modelling. Busbridge \textit{et al.} ~\cite{velivckovic2017graph} propose graph attention networks that assign different weights to neighboring nodes based on local structures. Jiang \textit{et al.}~\cite{niepert2016learning} dynamically infer a graph, and apply graph convolution on the dynamic graph.
% \BG{rewrite; unclear what this means}.
% Simonovsky \textit{et al.} ~\cite{Simonovsky2017ecc} also generalized the convolution operator arbitrary graphs, and the filter weights are conditioned on the specific edge labels in the neighbourhood of a vertex.
Li~\textit{et al.} ~\cite{Li_2019_ICCV} propose DeepGCNs to enable GCNs to go as deep as 100 layers by using residual/dense graph connections and dilated graph convolutions. G-TAD uses a DeepGCN-like structure to apply graph convolutions on a dynamic semantic graph as well as a fixed temporal graph.
% GCN is hardly used in the detection domain because of lacking ability to encode ordered information.
% GCN is hardly used in the detection domain because of lacking ability to encode ordered information. G-TAD, the graph convolution is applied on fixed temporal graphs and dynamic graphs. We reuse DeepGCN like style to encode long-range dependency, and adopt 1D convolution to manipulate convolutions on temporal graphs. We fuse the features in every graph convolution block.
\subsection{Problem Formulation}
The input to our pipeline is a video sequence of $l_v$ frames $V=\left \{ v_n \right \}_{n=1}^{l_v}$, where $v_n$ is the $n^{\textrm{th}}$ frame .
%
Following recent video action proposal generation methods \cite{buch2017sst, escorcia2016daps, gao2017turn,lin2018bsn}, we construct our G-TAD model using feature sequences extracted from raw video frames. We uniformly sample $L=l_v/\sigma$ frames with a sampling rate $\sigma$ and refer to each sampled frame as a \textbf{snippet}.
% We down-sample the video frame by a rate of $\sigma$, and refer to each down-sampled frame as a \textbf{snippet}.
% Similar to other works, we down-sample the frame-sequence into a regular frame interval $\sigma$ \BG{rewrite; this is unclear what you mean; do you mean that you downsample by a rate of $\sigma$?}. We refer to each down-sampled frame as a \textbf{snippet}.
%
% With $L=l_v/\sigma$, the number of snippets;
Our input visual feature sequence is represented by $X=\left [ x_{l} \right ]_{l=1}^{L} \in \mathbb{R}^{C\times L}$, where $x_{l}$ is the extracted snippet feature at the ${l}$ sampled frame and $C$ is its dimension. Each video sequence has a set of $N$ annotations
% Suppose there are $N$ actions in the video, annotated as
$\Psi = \left \{ \psi _n=\left (s_{n},e_{n}, c_n \right ) \right \}_{n=1}^{N}$, where $\psi_n$ represents an action instance, and $t_{s,n}$, $t_{e,n}$, and $c_n$ are its starting time, ending time, and action class, respectively.
% instance $\psi_n$, respectively. $c_n$ is the action class of $n$-th annotation.
The temporal action detection task is to predict ${M}$ possible actions $\Phi= \left \{ \phi _m=\left (\hat{t}_{s,m},\hat{t}_{e,m}, \hat{c}_m, {p}_m \right ) \right \}_{m=1}^{M}$ from $X$. Here, $(\hat{t}_{s,m},\hat{t}_{e,m})$ represents the predicted temporal boundaries for the $m^{\textrm{th}}$ predicted action, and $(\hat{c}_m, {p}_m)$ are its predicted action class and confidence score, respectively.
%
%
% %
% We expect that elements in $\Phi$ are similar as in $\Psi$.
\subsection{G-TAD Architecture}
Our action detection framework is illustrated in Fig.~\ref{fig:arch}.
We feed snippet features $X$ into a stack of $b$ GCNeXt blocks, which is designed inspired by ResNeXt~\cite{xie2017aggregated}, to obtain context-aware features. Each GCNeXt contains two graph convolution streams. One stream operates on fixed temporal neighbors, and the other adaptively aggregates semantic context into snippet features. % \BG{change according to what we discussed} correlated snippets.
Each block follows split-transform-merge strategy with multiple convolution paths.
% the outputs of both streams are fused together with the input. % \BG{mention that this is inspired by ResNeXt}
% After feature extraction from the original video, we firstly feed the feature $X$ into couples of graph convolution blocks to obtain context-aware feature with both of their temporal neighbours and long-distance correlated snippet.
Based on a set of pre-defined temporal anchors (see Section \ref{subsec: Details}), we define a sub-graph alignment layer named SGAlign to transform the aggregated feature of each sub-graph to a feature vector. % \BG{we need to define how we set these temporal anchors in our experiments}
% and reshape the aggregated feature into a 2D map.
% Then we use the predefined temporal anchors to crop and resize the context-aware feature and reshape the output as a two-dimension feature map.
Multiple fully connected layers are used to predict the intersection over union (IoU) of every anchor and the ground truth action instance. % \AT{do you mean sGoI}
%
We provide a detailed description of both GCNeXt and SGAlign
% The details of the convolution blocks and sGoI feature are described
in Sections \ref{subsec: GCNeXt} and \ref{subsec: SGAlign}, respectively.
% \AT{can you write what sGoI stands for? I think it's the first time you use it in the paper.}
% In training, G-TAD model is randomly initialized and the regression target of subgraph localization brach is pre-computed. We also add a node classification branch after the first graph convolution blocks as a regularizer.
% % We regress the model output to the ground truth IoU map directly. In training,
% In inference, we only use the predicted subgraph scores to construct action detection result $\Phi$.
% %
% % \AT{this last part is a bit vague, where exactly do you add the branch? on the GCN output before both node classification and sub-graph detection? Also what does it mean to use the IoU map?}
% %
% The training and inference details are explained in section 3.5 and 3.6, respectively. \AT{this last paragraph is still confusing, @Frost come see me when you have time and we can fix it.}
\begin{figure}
\centering
\includegraphics[trim={0cm 0.5cm 0cm 0.5cm},width=8cm,clip] {img/GTAD_GCNeXt.pdf}
\caption{\textbf{GCNeXt block.} The input feature is processed by temporal and semantic graphs with the same cardinality. Black and purple boxes represent Edge Convolutions and 1D Convolutions, respectively. We display (input channel, output channel) in each box. Both convolution streams follow a split-transform-merge strategy and 32 paths designed to increase the diversity of transformations. The module output is the summation of both streams and the input.}
%\AT{We can probably develop this a bit more, talk about what these convolutions are, what 32 paths mean, and how the fusion is done. Don't worry about space for now, we can compress later.}}
\label{fig:GCNeXt}
\end{figure}
\subsection{GCNeXt for Context Feature Encoding} \label{subsec: GCNeXt}
Our basic graph convolution block, GCNeXt, operates on a graph representation of the video sequence. It encodes snippets using their temporal and semantic neighbors. % and long range dependencies \BG{change accordingly}.
Fig. \ref{fig:GCNeXt} illustrates the architecture of GCNeXt.
To build the video graph, we take as input $C$ dimensional feature vectors of $L$ snippets, denoted as $X \in \mathbb{R}^{C\times L}$.
% We also assume each feature represents both appearance and movement at each snippet \AT{Do we mean here that the features are computed from visual and optical flow data?}
%\AT{what's an observation? also what does M and N represented? feature size by number of snippets?}.
% We also assume the following prior knowledge:
% the snippets are uniformly sampled from the video and the feature vectors show the appearance and movement features of the snippet. % \AT{let's properly define observations so this statement makes sense}
We build a graph $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$, where $\mathcal{V}$ and $\mathcal{E}$ denote the vertex and edge set, respectively.
% We would like to build or infer the relationship between these nodes (snippets) that take the form of a graph $\mathcal{G}={\mathcal{V},\mathcal{E}}$.
% %
% Here $\mathcal{V}$ and $\mathcal{E}$ denote the vertex and edge set, respectively.
In this case, each vertex is a snippet (represented by its feature) and each edge shows a dependency between pairs of snippets.
% The number of vertices in $\mathcal{V}$ is fixed for a given video because the observation shows the vertex \AT{can you clarify what each vertex represents? is it the snippets?}.
We define two types of edges --- temporal edges $\mathcal{E}_t$ and semantic edges $\mathcal{E}_s$, so accordingly we define two graphs --- the temporal graph and the semantic graph. We describe each type of edge as well as the graph convolution process in the following.
% We consider three different types of edges in $\mathcal{E}=\mathcal{E}_t^{f} \cup\mathcal{E}_t^{b} \cup \mathcal{E}_s$.
%
%We assume there are $k$ types of nodes on the graph, which reflect the different classes of the snippets. After a certain generative process on the graph, each node presents a certain length of feature based on its own and neighbour classes.
%
%Given that the graph nodes are temporally connected based on their order, the task of temporal action detection have the same meaning of localizing a sub-graph that presenting the positive action.
% {Learning the Edge for a Graph}
%Since the observations matrix $X$ represent the appearance and movement feature of the snippet, the graph $\mathcal{G}=\{\mathcal{V}, \mathcal{E}\}$ can be inferred from the feature.
\noindent
\textbf{Temporal Edges ($\mathcal{E}_t$)}. Temporal edges encode the temporal order of the video. For each node $v_i$, there is one unique forward edge to node $v_{i+1}$, and one backward edge to node $v_{i-1}$. In this case, we have $\mathcal{E}_t=\mathcal{E}_t^{f}\cup\mathcal{E}_t^{b}$, where $\mathcal{E}_t^{f}$ and $\mathcal{E}_t^{b}$ are forward and backward temporal edge sets defined % \BG{is it snippet or frame? be consistent},
as follows:
\begin{align} \label{eq:edge_t}
\mathcal{E}_t^f &=\{(v_i,v_{i+1})|~i\in\{1,2,\dots,L-1\}\}, \\
\mathcal{E}_t^b &=\{(v_i,v_{i-1})|~i\in\{2,\dots,L-1,L\}\},
\end{align}
\noindent $L$ is the number of snippets in the video.
\noindent \textbf{Semantic Edges ($\mathcal{E}_s$)}. We define $\mathcal{E}_s$ using the notion of dynamic edge convolutions \cite{wang2018dynamic}. The goal of these edges is
% Following the idea of semantic edge convolution that widely used in the point cloud, we also use this method
to collect information from semantically correlated snippets.
%
For each node $v_{i}$ in the input graph $\mathcal{G}$, we define a set $\mathcal{E}_s$ as follows: %using Eq.~\ref{eq:edge_d}.% \AT{in Eq.~\ref{eq:edge_d}, does $i$ go to $N - 1$ or $N$?}
% we % build up a kd-tree and look for the top $K$ nearest neighbours in the feature representation.
% Mathematically, we have Eq.~\ref{eq:edge_d}.
\begin{align} %\label{eq:edge_d}
\mathcal{E}_s=\{(v_i,v_{n_i(k)})|i\in\{1,2,\dots,L\}; k\in\{1,2,\dots K\}\} \notag.
\end{align}
Here, $n_i(k)$ is the index of the $k^{\textrm{th}}$ nearest neighbour from $i$-th node, % \BG{again, is it frame, snippet or node?? be consistent} \BG{define what space these nearest neighbors are in}.
$n_i$ is constructed dynamically at every layer in the node feature
space, which enables us to find the \textit{dynamic neighbors} that intrinsically carry semantic context information. Since we recompute $n_i(k)$ at each layer, $\E_s$ is adaptively changed to represent new levels of semantic context.
% and efficiently in each forward pass. % \FX{talk about Complexity if enough space}
% \textbf{Class Edge $\mathcal{E}_c$}
% We also use relation inference to predict the class edges. A class edge connects two frames that belong to the same frame class: action or background. In other words $\mathcal{E}_c=\{(v_i,v_j|c_i=c_j;i,j=1,2,3,\dots,N)\}$. We have the ground-true class edge in the training set. However, the frame class is unknown for the test set. We deploy an edge prediction module to predict the class edge $\widetilde{\mathcal{E}_c}$ for each frame.
%\subsubsection{Cross Edge Graph Convolution}
\noindent \textbf{Graph Convolution}. %\AT{i think we probably need to start a new subsection here or bold title, it's a bit too big to follow}
A Graph Convolution transforms
% the representation of
the graph vertices represented as % by associating each vertex $v_i$ with a feature vector $x_i \in \mathbb{R}^C$ where $C$ is the feature dimension.
%
%Since we use
$X=[x_1,x_2,\dots,x_L]\in \mathbb{R}^{C \times L}$ %to represent the feature for all the $v_i\in\mathcal{V}$ in the graph $\mathcal{G}$,
through a general graph convolution operation $\mathcal{F}$ formulated as: %Eq.~\ref{eq:gc}.
%
\begin{equation} %\label{eq:gc}
\F(X,A,W)=ReLU(f_{agg}(X,A,W))\notag.
\end{equation}
%
Here, $W\in \mathbb{R}^{2C \times C}$ are trainable weights of the aggregation function $f_{agg}$, $A\in \mathbb{R}^{L \times L}$ is the adjacency matrix without self-loops (\ie edges between a node and itself), and ReLU is the rectified linear unit as activation function. We formally define the adjacency element $A_{i,j}$ as ${1}\{(i,j)\in \E\}$, where $1\{\cdot\}$ is the indicator function.
%\AT{we don't use $A$ in Eq.~\ref{eq:gc}, do we need to include it in the parameters of the equation? if not then we have to introduce it softly here, something like ``We also use the graph's adjacency defined as $A$ ... ''}
%
There are several choices for $f_{agg}$ in the literature.
% The aggregation function can be chosen in a wide range.
We use a single-layer edge convolution~\cite{DBLP:journals/corr/abs-1801-07829} as our aggregation function in Eq.~\ref{eq:agg}.
% Keep in mind that $X\in R^{C\times L}$, we can have Eq.\ref{eq:agg}.
\begin{align} \label{eq:agg}
f_{agg}(X,\E,W)&=([X^T,AX^T-X^T] W)^T \nonumber \\
% &= W^T [X;X A^T-X] \nonumber \\
% &= W_\alpha X + W_\beta (X A-X) \nonumber \\
% &= (W_\alpha-W_\beta) X + W_\beta X A \nonumber \\
&= W_0 X + W_{1} X A.
\end{align}
We use $W_\cdot\in \mathbb{R}^{C \times C}$ with different subscripts to show different trainable weights. $[\cdot,\cdot]$ represents the matrix concatenation in columns.
%
% All layers \BG{what layers? this graph convolution is defined for the input graph with features $X$; you need to rethink how you will write this for each layer of the network} are followed by a rectified linear unit (ReLU) activation.
% Also, we use the rectified linear unit (ReLU) for the activation function for all the layers.
\noindent \textbf{Residual Connection and Cardinality}. %To further increase the information of our data layers,
We require two more graph operations. First, we use the residual connection proposed in DeepGCN \cite{Li_2019_ICCV} to improve model convergence.
% the model well and achieve superior performance.
Under this setup, our graph convolution block can be formulated as: \begin{align} \label{eq:block}
\H(X,\E,W)=ReLU[&W_\alpha X + W_\beta X A_t^f + W_\gamma X A_t^b+ \nonumber \\
&W_\phi X A_s+X],
\end{align}
where $A_t^f$, $A_t^b$, and $A_s$ are adjacency matrices for $\mathcal{E}_t^f$, $\mathcal{E}_t^b$, and $\E_s$ respectively.
% \BG{there are no definitions of $X_t$ nor $X_d$ and the corresponding adjacency matrices!}
The derivation of Eq.~\ref{eq:block} is demonstrated in the \textbf{supplementary materials}, where we also prove that it can be efficiently computed by zero-padded 1D/edge convolutions.
% \AT{it is not directly clear that 1D convs can be used in eq \ref{eq:block}, instead maybe you can say that we do as other edge conv papers and implement eq \ref{eq:block} as 1D convs}
Following ResNeXt~\cite{xie2017aggregated}, GCNeXt adopts split-transform-merge strategy to explore the applicability of group convolution by changing cardinality besides going deeper or wider.
% In our implementation, we also explore the notion of cardinality used in Inception \cite{szegedy2015going} and ResNeXt~\cite{xie2017aggregated},
% we also refer to Inception\cite{szegedy2015going} and ResNeXt~\cite{xie2017aggregated} structure and explore cardinality,
% the ``next'' dimension of networks. Since input features are extracted from a very deep neural network, the split-transform-merge strategy by many paths is more efficient than going deeper or wider.
% \BG{rewrite; this strategy is from the Inception/ResNeXt literature?}.
\subsection{Sub-Graph Alignment and Localization} \label{subsec: SGAlign}
\begin{figure}
\centering
\includegraphics[trim={2.5cm 3.5cm 2.5cm 4.5cm}, width=8cm, clip]{img/GTAD_SGAlign.pdf}
\caption{\textbf{SGAlign layer}. SGAlign extracts sub-graph features using a set of anchors. In the graphs above, the colored dots represent node features, grey arrows are semantic edges, and the orange highlighted arch is the anchor. Both circles represent the same graph. SGAlign arranges node feature along the temporal/semantic graphs, and concatenates both features as output. When using the temporal graph, the order of nodes is preserved in the final representation (black lines). This is not always true for the semantic graph, since node features are represented by their feature neighbors (purple lines).
% SGAlign layer computes the value of each sampling point by linear interpolation from the nearby nodes on the graph. We use the concatenation of temporal graph nearby nodes and KNN graph nearby nodes as the SGAlign output. \AT{i think we can be more explanatory in this caption, we need to explain that both circles are the same but each side extracts features from different graphs. we should also say that SGAlign computes a certain number of points, each point by interpolating... }
}
\label{fig:ga}
\end{figure}
\noindent
\textbf{Sub-Graph of Interest Alignment (SGAlign)}. Most previous action detectors perform rescaling to extract a fixed sized \textit{proposal feature vector} for each action anchor. Given the action anchor $a=[t_s,t_e]$, they sample the video feature sequence within $a$ through linear interpolation with $\tau$ points.
%
Given our graph formulation, a \textit{sub-graph feature} is extracted instead from
% Take the advantage of the structured data,
a Sub-Graph of Interest Alignment (SGAlign) layer that aggregates the context feature in an adaptive way, and does not rely on human priors.
Fig.~\ref{fig:ga} illustrates our new graph alignment algorithm and we present its technical details next. % of the SGAlign layer.
%Graph Align layer is similar to ROIAlign in the object detection problem on the image but benefit from the graph structure.
% In Fig.~\ref{fig:ga}, the colored dots represents the graph node feature, the grey arrow the feature neighbour edge, and the black arch the anchor.
%
\begin{algorithm}[t]
\input{08_alg.tex}
\label{alg}
\end{algorithm}
Given an input of $L$ feature vectors $X\in \mathbb{R}^{C\times L}$ and an anchor $a$, we expect to sample $\tau_1$ and $\tau_2$ vectors from the temporal and semantic graphs, respectively. We repeat this process for all anchors.
% \AT{do you mean here one from the temporal graph and one from the dynamic graph? it will be a good idea to use consistent names as in other sections}
The alignment is done in four steps. \textbf{(1)} Each snippet is projected back to the temporal order given by the temporal graph. \textbf{(2)} We run an interpolation and rescaling algorithm (Alg.~\ref{alg}) to get $\tau_1$ vectors from the temporal graph and $\tau_2$ vectors from the semantic graph. \textbf{(3)} Every node's feature is replaced with the mean feature of its dynamic neighbors,
% \BG{this is the first time the term 'dynamic neighbors' is used; define this way before if you are going to use it}
and then we repeat \textbf{(1)} and \textbf{(2)} to further extract features for the semantic context.
% \BG{why do you need to repeat?}.
% \textbf{(1)} we project each snippet back to the temporal order given by the temporal graph.
% \textbf{(2)} we uniformly sample $s_1 \times \floor{\frac{t_e-t_s}{s_1}}$ vectors from features in $t_s$ to $t_e$, and we rescale them to $s_1$ vectors \AT{what do you mean rescale them? their size?}.
% \textbf{(3)} we mean-pool the neighbouring snippets defined in the dynamic graph \AT{neighboring snippets of what? the anchor location?}, sample $s_1 \times \floor{\frac{t_e-t_s}{s_1}}$ \AT{do you mean $s_1$ or $s_2$?} from the pooled vectors, and rescale the number to $s_2$ as in previous step.
\textbf{(4)} The $\tau_1$ and $\tau_2$ vectors are concatenated as the output of the SGAlign layer.
% In the sampling process, linear interpolation is used to make the sample contiguous.
In Alg. \ref{alg}, the output $Y_j$ for anchor $a_j$, is the weighted average of all the nodes in the sub-graph defined by $a_j$. In the backward pass, this weighted sum means gradients will always flow to these nodes.
% Consequently, this guarantees gradient flow to every node during the backward pass.
% Also, since the sampled vector is the weighted average of its two temporal neighbouring vectors, we are able to update all the action and context vectors in the backward pass.
% \FX{write an algorithm if possible} \AT{yes can you re-write it as an algorithm its better. then also write why we sample many vectors}
% \AT{we can reduce the size of this section, we talk too much about previous work on alignment and heuristics, can you make that part shorter? just clearly state what people currently do without going into details, then specify step by step what our graph align module does. also the relation between the edges in figure 4 and the graph align process is not clear.}
\noindent \textbf{Sub-Graph Localization}. For an anchor $(t_s, t_e)$, we calculate its Intersection-over-Union (IoU) with all ground truth actions $\psi$ in $\Psi$, and denote the maximum IoU $g_c$ as its label. We compute $g_c$ for all pairs of $(t_s, t_e)$.
% We reshape them into an $L \times L$ matrix. We call this matrix the IoU map.
% When we enumerate $t_s, t_e$ from $0$ to the video length $\sigma L$ by $\sigma$, we get $L \times L$ IoU values and reshape it to a $L$ by $L$ matrix, namely IoU map.
%
Once we get the \textit{sub-graph feature} from SGAlign layer, we use three fully connected (FC) layers to regress it to $g_c$. The last FC layer produces a two-dimension vector, where each entry shows classification $p_{cls}$ and regression $p_{reg}$ scores.
% \AT{can we explain what sub-graph localization means in plain words at the beginning, before we start explaining the technical details? also, we don't show this module anywhere in fig \ref{fig:arch}}
\subsection{Training G-TAD}
In G-TAD, the sub-graph localization is used to determine the confidence scores of anchors which are regressed for final temporal action detection. We do not need to specifically classify starting and ending nodes of actions since they are predefined by the anchors. However, we noticed that adding a node classifier during training can drastically improve the model's convergence. This classification module is ignored at test time.
% However, adding a branch to classify the start and end nodes in the training process can facilitate model convergence.
% In the training process, we add a simple node classification branch to regularize the first GCNeXt block simultaneously \BG{what does this mean?}.
\noindent \textbf{Sub-Graph Localization Loss}. Sub-graph localization predicts $(p_{cls},p_{reg})$ for each anchor position.
%\BG{this definition should be put when the terms are first used}.
With the training target being $g_c$, the sub-graph loss $L_{g}$ is defined as follows:
% \AT{what does classification mean here? what exactly are we classifying?}
%
\begin{equation}
L_{g}= \mathcal{L}_{wce}(p_{cls},1\{g_c > 0.5\}) + \lambda_1 \cdot \mathcal{L}_{2}(p_{reg},g_c),
\end{equation}
%
where
% $1\{\}$ is the indicator function \BG{you used this function before and now you define it???} and
$\mathcal{L}_{wce}$ is the weighted cross entropy loss. In our experiments, we take the tradeoff coefficient $\lambda_1=10$, since the second loss term tends to be smaller than the first. % $\mathcal{L}_{2}$ norm is usually smaller.
% \AT{i think this section is not coherent with the previous one. in the sub graph localization section we should include how the scores are computed}
%
\noindent \textbf{Node Classification Regularizer}. In the training process, we label a node as a start or end point if they are temporally close to $t_s$ or $t_e$, while all the other nodes are of the third class containing action and background nodes.
% To train Node Classification Module, we give label to each node in the graph.
%
% For a ground-truth action instance $\psi_g=( s,e )$ in annotation $\Psi$, we denote its starting and ending regions as $r_S=[ t_s-\delta, t_s+\delta ]$ and $r_E= [ t_e-\delta,t_e+\delta ]$ separately.
%
% We labelled a node as a start or end point, if it falls within $r_S$ or $r_E$ respectively. We label all other nodes as background.
% Then for each node, if it falls within the two regions, it should be labeled as start or end point. Otherwise we label it as background.
%
We add a separate branch by with 1 FC-layer after the first GCNeXt block to classify nodes to their labels $g_n$, with start/end probabilities $\boldsymbol{{p}}_n=(p_s,p_e)$.
%\BG{where do these predictions come from?? not defined before},
We add a node regularizer $L_{n}= \mathcal{L}_{wbce}(\boldsymbol{{p}}_n,g_n)$, where $\mathcal{L}_{wbce}$ stands for the weighted binary cross entropy loss.
% \begin{equation} \label{eq:r}
% L_{n}= \mathcal{L}_{wbce}(\boldsymbol{{p}}_n,g_n)
% \end{equation}
We train G-TAD in the form of a multi-task loss function, including sub-graph loss $L_{g}$, node regularizer loss $L_n$, and an $\mathcal{L}_{2}$ regularization for all the trainable parameters $\Theta$:
\begin{equation}
L = L_{g} +L_{n}+ \lambda_2 \cdot \mathcal{L}_2(\Theta).
\end{equation}
%
In our experiments, we set $\lambda_2=10^{-4}$. % and both branches are trained evenly \BG{evenly? what does that mean?}.
\subsection{Inference and Post-processing}
At inference time, G-TAD predicts classification and regression scores for each anchor $(t_s, t_e)$. From the anchors, we construct predicted actions $\Phi = \left \{ \phi _n=(s_n, e_n, c_n, p_n) \right \}_{n=1}^{J}$, where $(s_n, e_n)$ is the action boundary in the video scale, $c_n$ is the action class, and $p_n$ is the fused confidence score of this prediction and $p_{n}=p_{cls}^\alpha \cdot p_{reg}^{(1-\alpha)}$. In our experiments, we search for the optimal $\alpha$ in each setup. We apply Soft-NMS \cite{softNMS} and select the top-$M$ predictions.
% \AT{how do you choose the $M$ anchors? based on fused score?}
% \noindent
% {\bf Score fusion.}
% For each anchor $(t_s, t_e)$, we fuse its classification and regression scores
% % with its boundary probabilities
% to get the final confidence score $p_{n}$ (Edq.~\ref{eq:fused_score}). In our experiments, we set $\alpha=0.6$.
% \begin{equation} \label{eq:fused_score}
% p_{n}=p_{cls}^\alpha \cdot p_{reg}^{(1-\alpha)} %\cdot p_{s} \cdot p_{e} .
% \end{equation}
% \noindent
% {\bf Redundant anchor suppression.}
% \AT{i think this whole section can go to the supp, leave it for now and we can discuss it later}
% Around a ground truth action instance, multiple proposals have high confidence scores. Thus we need to suppress redundant anchors to obtain higher recall with fewer positives.
%Non-maximum suppression (NMS) algorithm is widely used for redundant results suppression.
%Using greedy NMS \cite{Dalal2005Histograms}, first, all proposals are sorted by their scores. Then proposal $\varphi_m$ with maximum score is used for calculating overlap IoU with all other proposals, where highly overlap proposals is removed with pre-defined threshold. Recursively applying this step to remaining proposals, we get slimmed proposals set.
% We use Soft-NMS \cite{softNMS} to suppress redundant anchors. Soft-NMS is a recently proposed non-maximum suppression (NMS) algorithm which suppresses redundant results using a score decaying function. First, all anchors are sorted by their scores. Then, the anchor with maximum score is used for calculating overlap IoU with others. Scores of highly overlapped anchors are decayed. This step is recursively applied to the remaining anchors to generate re-scored detection set.
% \BG{all throughout this section, there is no mention of our contributions and how our intuitions that inspired our design choices. There is no mention of context or adaptiveness. Please add these connections. This section is very isolated from the sections before it.}
\caption{ Interpolation and Rescaling in Alignment }
%\scriptsize
\begin{algorithmic}[1]
\REQUIRE ordered feature $X=[x_0, x_1, ..., x_L], x_l\in \mathbb{R}^C$, Anchors $=[a_0,a_1,...,a_J],a_j\in \mathbb{R}^2$, resolution $\tau$;
\STATE Init $Y\in \mathbb{R}^{J\times \tau C}$
\FOR {each anchor $a_j$ in Anchors}
\STATE $t_s$ = $a_j$[0]; $t_e$ = $a_j$[1];
\STATE $d=t_s-t_e$; $s=\floor{d/\tau}$; $T$ = $\tau s$;
\STATE $idx$ = [$t_s + k d/ T $ for $k$ in range($T$)]
\STATE $X_{in}$ = [$(\ceil{i}-i) x_{\floor{i}}+(i-\floor{i}) x_{\ceil{i}}$ for $i$ in $idx$]
\STATE $Y[j]$ = [$mean(X_{in}[k s$:$(k+1)s])$ for $k$ in range($\tau$)]
\ENDFOR
\ENSURE $Y$.
\end{algorithmic}
%% to make e into times 10 to the nth power
\providecommand{\e}[1]{\ensuremath{\times 10^{#1}}}
% 4.1
\subsection{Datasets and Metrics}
% Our experiments are based on { ActivityNet-1.3}\cite{caba2015activitynet} and { THUMOS-14} \cite{jiang2014thumos} datasets.
\noindent
{\bf ActivityNet-1.3} \cite{caba2015activitynet} is a large-scale action understanding dataset for action recognition, temporal detection, proposal generation and dense captioning tasks. It contains 19,994 temporally annotated untrimmed videos with 200 action categories, which are divided into training, validation and testing sets by 2:1:1. % Each video has an average of 1.65 action instances.
\noindent
{\bf THUMOS-14} \cite{jiang2014thumos} dataset contains 413 temporally annotated untrimmed videos with 20 action categories. We merge the 200 videos in validation to the training set and evaluate on the 213 annotated videos from the testing set. % It is more challenging to perform action detection on this dataset because (1) each video has more than 15 action instances; and (2) $71\%$ of its frames are occupied by background items.
\noindent
{\bf Detection Metric}. We take mean Average Precision (mAP) at certain IoU thresholds as the main evaluation metric.
%For each class of the action, the predicted action boundary is true-positive if it overlaps with the ground-truth action instance larger than a certain threshold.
Following the official evaluation API, the IoU thresholds are chosen
from $\{0.3, 0.4, 0.5, 0.6, 0.7\}$ and $\{0.5, 0.75, 0.95\}$ on THUMOS14 and ActivityNet-1.3, respectively. Following standard practise, we also report average mAP over 10 different IoU thresholds on ActivityNet-1.3.
% 4.2
\subsection{Implementation Details} \label{subsec: Details}
\noindent
{\bf Features and Anchors}. We use pre-extracted features for both datasets. For ActivityNet-1.3, we adopt the pre-trained two-stream network by Xiong et. al.~\cite{xiong2016cuhk}, with down-sampling ratio $\sigma = 16$.
% is set to 16.
%The spatial and temporal sub-networks adopt ResNet and BN-Inception network separately.
Each video feature sequence is rescaled to $L = 100$ snippets, using linear interpolation.
% \\
For THUMOS-14, the video features are extracted from Kinetics~\cite{zisserman2017kinetics} pretrained TSN model~\cite{TSN2016ECCV} with $\sigma = 5$.
% and down-sampling ratio$\sigma$ is set to 5.
We crop each video feature sequence with a window size $L=256$ and overlap neighbouring windows with $128$ snippets. In training, we do not use any crops void of actions.
% The cropped video which does not include any actions is not used in training.
For ActivityNet-1.3 and THUMOS-14, we enumerate all possible anchors with restriction, \eg $Anchors =\{(t_s,t_e)|~ 0< t_s<t_e<L; ~t_s,t_e \in \mathcal{N}; ~t_e-t_s < D\} $, while $L$ and $D$ are $100$ and $64$, respectively.
In SGAlign, we use $(\tau_1=32,\tau_2=4)$ for ActivityNet-1.3, and $(\tau_1=\tau_2=16)$ for THUMOS-14. % otherwise mentioned.
% When we arrange sub-graph feature, the entry at $(i,j)$ of the 2D map is the sub-graph feature from anchor $a=(i,j)$ if $a\in A$, otherwise zero vectors. % \FX{ummm weird, ask ali}
% we set the length of observation window $l_\omega$ to 256 and the maximum duration length $D$ to 64, which can cover the length of $98\%$ of action instances.
\noindent {\bf Training and Inference}. We implement and compile our framework using PyTorch 1.1, Python 3.7, and CUDA 10.0.
% Our framework employs the PyTorch-1.1 CNN modules for the GCNeXt blocks, and SGAlign layer is compiled by NVCC, CUDA-10.
% \\
We use $b=3$
% \BG{use this as the variable I refer to as the number of GCN layers in methodology section}
GCNeXt blocks and train our model end-to-end, with batch size of 16.
% in an end-to-end fashion from scratch without bells and whistles.
The learning rate is $4\e{-3}/4\e{-4}$ on ActivityNet and
% ; while we reduce the learning rate to
$6\e{-6} / 6\e{-7}$ on THUMOS14 for 5/5 epochs.
% because the dataset is smaller.
%
% In the ablation studies, we experiment with different values of $l_r$
% In ablation study, since the $l_r$ is sensitive to the module architecture, we train the model with ten different $l_r$ on each setup,
% and show the best performance on the validation set.
In inference, we take video classification scores by \cite{wang2017untrimmed} and \cite{xiong2016cuhk}, and multiply them to $p_n$ for evaluation.
% for ActivityNet-1.3, and multiply it to $p_n$ for evaluation. Since THUMOS-14 has more average instances per video, we fuse the classification confidence score of the top 2 prediction class to $p_n$ in UntrimmedNet~\cite{wang2017untrimmed}.
% At inference, we fuse G-TAD $p_n$ by top-1 video-level classification results by Xiong~\cite{xiong2016cuhk} for ActivityNet-1.3, and top-2 results in UntrimmedNet~\cite{wang2017untrimmed} for THUMOS-14.
% \BG{what does this mean? how does it tie into the method? clarify}.
For post-processing, the Soft-NMS threshold is $0.84$ to pick the top $M$ confident predictions, where M is 100 for ActivityNet and 200 for THUMOS.
% In inference, we adopt top-1 and top-2 video-level classification results by Xiong~\cite{xiong2016cuhk} and in UntrimmedNet~\cite{wang2017untrimmed} for the two datasets, respectively. In post-processing, the NMS threshold should be $0.84$ to produce top N confident prediction, where $N=100,200$, separately.
More details can be found in the \textbf{supplementary material}. To encourage reproducibility, the code and trained models will be made publicly available.
% We run the same model in the inference. On ActivityNet-1.3, we adopt top-1 video-level classification results generated by method~\cite{xiong2016cuhk} and use confidence scores from Sub-Graph Localization Module.
% On THUMOS-14, we use both top-2 video-level classification results generated
% by UntrimmedNet \cite{wang2017untrimmed} to generate classification result for each proposal.
% For ActivityNet-1.3 and THUMOS-14 datasets, we use the first 100 and 200 temporal proposals per video separately. In post-processing, the NMS threshold should be $0.84$ to reach maximum performance.
\subsection{Comparison with State-of-the-Art}
\noindent
{\bf ActivityNet-1.3}: Tab.~\ref{tab:sota_anet} compares G-TAD with state-of-the-art detectors. We report mAP at different tIoU thresholds, as well as average mAP. G-TAD reports the highest average mAP results on this large-scale and diverse dataset.
% gives a comparison between our model G-TAD and the state-of-the-art methods. The table presents three mAPs and the Average mAP. G-TAD reaches the highest mAP overall, implying that our method can localize and classify actions much more accurately than any other method.
% Our G-TAD model is compared with the state-of-the-art methods in Tab.~\ref{tab:sota_anet}. The G-TAD model
% reaches the highest mAP overall thresholds, implying that our method can localize and classify actions much more accurately than any other methods. It is worth mentioning that G-TAD gives better performance than its contemporary work, BMN and PGCN, shown in the last-second block. Noted that the Average mAP is the average over mAPs at nine IoU thresholds and the improvement is obvious.
\begin{table}[tbp]
\centering
\caption{\textbf{Action detection results on validation set of ActivityNet-1.3}, measured by mAP ($\%$) at different tIoU thresholds and the average mAP. G-TAD achieves better performance in average mAP than other methods, even the latest work of BMN and PGCN shown in the last second block.
}
\small
\begin{tabular}{p{2.1cm}p{0.62cm}<{\centering}p{0.62cm}<{\centering}p{0.62cm}<{\centering}p{0.9cm}<{\centering}}
\toprule
% & \multicolumn{4}{c}{tIoU on Validation} & Test \\
% \hline
Method & 0.5 & 0.75 & 0.95 & Average\\
\hline
Wang \textit{et al.} \cite{wang2016uts} & 43.65 & - & - & -\\
Singh \textit{et al.} \cite{singh2016untrimmed} & 34.47 & - & - & - \\
SCC \cite{heilbron2017scc} & 40.00 & 17.90 & 4.70 & 21.70 \\
% Lin \textit{et al.} \cite{lin2017temporal} & 44.39 & 29.65 & 7.09 & 29.17 \\
CDC \cite{shou2017cdc} & 45.30 & 26.00 & 0.20 & 23.80 \\
TCN \cite{dai2017temporal} & 37.49 & 23.47 & 4.47 & 23.58 \\
R-C3D \cite{xu2017r} & 26.80 & - & - & - \\
BSN \cite{lin2018bsn} & 46.45 & 29.96 & 8.02 & 30.03 \\
Chao \textit{et al.} \cite{chao2018rethinking} & 38.23 & 18.30 & 1.30 & 20.22 \\ \hline
P-GCN \cite {zeng2019graph} &48.26 &33.16 &3.27 &31.11 \\
BMN \cite{lin2019bmn} & { 50.07} & \textbf{34.78} & { 8.29} & { 33.85} \\
\hline
% baseline (ours) \\
G-TAD (ours) & \textbf{50.36} & {34.60} & \textbf{9.02} & {\bf 34.09} \\
\bottomrule
\end{tabular}
\label{tab:sota_anet}
%\normalsize
%\vspace{-0.3cm}
\end{table}
\begin{table}[!tb]
\centering
\caption{\textbf{Action detection results on testing set of THUMOS-14}, measured by mAP (\%) at different tIoU thresholds. G-TAD achieves the best performance among all one-stage methods, and the mAP@0.7 is even higher than strong two-stage TAL-Net\cite{chao2018rethinking}.
% \BG{how do we know which methods are one-stage and which ones are not?}
}
\small
\begin{tabular}{p{2.3cm}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}}
\toprule
% & \multicolumn{5}{c}{tIoU on Validation} \\
% \hline
Method & 0.3 & 0.4 & 0.5 & 0.6 & 0.7 \\ \hline
\multicolumn{6 }{c}{Two-stage Temporal Action Detection} \\ \hline
% Wang \textit{et al.} \cite{wang2014action} & 14.0 & 11.7 & 8.3 & - & -\\
% Caba \textit{et al.} \cite{caba2016fast} & - & - & 13.5 & - & -\\
% DAP \cite{escorcia2016daps} & - & - & 13.9 & - & -\\
% Oneata \textit{et al.} \cite{oneata2014lear} & 27.0 & 20.8 & 14.4 & - & -\\
% Yuan \textit{et al.} \cite{yuan2016temporal} & 33.6 & 26.1 & 18.8 & - & -\\
% S-CNN \cite{shou2016temporal} & 36.3 & 28.7 & 19.0 & - & -\\
SST \cite{buch2017sst} & - & - & 23.0 & - & -\\
CDC \cite{shou2017cdc} & 40.1 & 29.4 & 23.3 & 13.1 & 7.9\\
TURN-TAP\cite{gao2017turn} & 44.1 & 34.9 & 25.6 & - & -\\
% R-C3D \cite{xu2017r} & 44.8 & 35.6 & 28.9 & - & -\\
CBR \cite{gao2017cascaded} & 50.1 & 41.3 & 31.0 & 19.1 & 9.9\\
SSN \cite{zhao2017temporal} & 51.9 & 41.0 & 29.8 & - & -\\
BSN \cite{lin2018bsn} & 53.5 & 45.0 & 36.9 & 28.4 & 20.0 \\
TCN \cite{dai2017temporal} & - & 33.3 & 25.6 & 15.9 & 9.0\\
TAL-Net \cite{chao2018rethinking} & 53.2 & 48.5 & 42.8 & \textbf{33.8} & 20.8\\
MGG \cite{DBLP:journals/corr/abs-1811-11524} & 53.9 & 46.8 & 37.4 & 29.5 & 21.3 \\
DBG \cite{lin2019fast} & 57.8 & 49.4 & 39.8 & 30.2 & \textbf{21.7} \\
P-GCN \cite {zeng2019graph} & \textbf{63.6} & \textbf{57.8} & \textbf{49.1} & - & -\\ \hline
\multicolumn{6 }{c}{One-stage Temporal Action Detection} \\ \hline
Richard \textit{et al.} \cite{richard2016temporal} & 30.0 & 23.2 & 15.2 & - & -\\
Yeung \textit{et al.} \cite{yeung2016end} & 36.0 & 26.4 & 17.1 & - & - \\
Yuan \textit{et al.} \cite{Yuan2017} & 36.5 & 27.8 & 17.8 & - & -\\
Hou \textit{et al.} \cite{hou2017real} & 43.7 & - & 22.0 & - & -\\
SS-TAD \cite{buch2017end} & 45.7 & - & 29.2 & - & 9.6\\
BMN \cite{lin2019bmn} & \textbf{56.0} & 47.4 & 38.8 & 29.7 & 20.5 \\
% GTAN \cite{GTAN}
\hline
% baseline (ours) \\
G-TAD (ours) & {54.5} & \textbf{47.6} & \textbf{40.2} & \textbf{30.8} & \textbf{23.4} \\
\bottomrule
\end{tabular}
% \vspace{-0.3cm}
%\setlength{\belowcaptionskip}{-1cm}
\label{Tab:sota_thm}
\end{table}
\noindent
{\bf THUMOS14}: Tab.~\ref{Tab:sota_thm} compares the action localization results of G-TAD and various state-of-the-art methods on the THUMOS14 dataset. At 0.7 IoU, G-TAD reaches mAP of $23.42\%$, compared to the current best of $20.8\%$ from TAL-Net. At 0.5 IoU,
% Regarding mAP at IoU 0.5,
G-TAD outperforms all one-stage detection methods, such as SS-TAD~\cite{buch2017end} and BMN~\cite{lin2019bmn}. % \BG{how does one know which methid is one-stage and which is two-stage?}.
Comparing G-TAD with two-stage methods puts our method at an inherent disadvantage. For example,
% P-GCN and TAL-Net presents a better mAP than G-TAD, but we claim that two-stage framework have an inherent advantage on detection problem. Specifically,
P-GCN only rescores BSN proposals by mining proposal-proposal relationships and, in doing so, it increases mAP from $36.9\%$ to $49.1\%$. Our model can get such good results only from capturing more information about context.
% \BG{we only report the results here and we say nothing about the fact that by only capturing more information about context, we are able to get such good results. Motivate the good performance based on our motivations and design choices}
% Tab.~\ref{Tab:sota_thm} reports the action localization result of the various methods on THUMOS14 dataset.
% %
% Regarding mAP at IoU 0.5, G-TAD outperforms BMN, ... , all of the one-stage action detection method. In THUMOS14 dataset, we claim that two-stage methods such as TAL-Net and PGCN have inherent advantages. THUMOS-14 has more action instances per video, and each video
% contains a small portion of actions: on average, the THUMOS-14 training set has 15 instances per
% video and each video has $71\%$ background.
% %
% Indeed, P-GCN was be used to re-evaluate the BSN proposals as the second stage and mAP was increased from 36.9 to 49.1 by mining the proposal-proposal relationships. Our work can be combined with PGCN and get a similar boost to become a more robust two-stage method.
\subsection{Ablation Study}
\noindent
\textbf{GCNeXt Module}: We ablate the three main components of GCNeXt, mainly GCN on temporal edges, GCN on semantic edges, and cardinality increase.
% We proposed GCNeXt module to encode snippets using their (1) temporal neighbours and (2) long range dependencies.
% We also (3) increased the cardinality by group convolution in both graphs.
%
Tab.~\ref{tab:abl_gcnext} reports the performance obtained on ActivityNet-1.3, when each component is separately enabled/disabled. We see how each of these components contributes to the performance of the final G-TAD model. We highlight the gains from the semantic graph, showing the benefit of integrating adaptive context from semantic neighbors.
% range dependencies \BG{change accordingly; no long range}.
% The difference in first two rows shows that convolution is required to represent snippet feature. Comparing the middle three rows we can see graph convolution on $\mathcal{E}_t^\cdot$, recording the temporal order information, is more dominant than the others.
\begin{table}[tbp]
\centering
\caption{\textbf{Ablating GCNeXt Components.} We disable temporal/semantic graph convolutions and set different cardinalities for detection on ActivityNet-1.3. %Graph convolution on $\mathcal{E}_t^\cdot$ is more dominant than the others.
% \BG{change Dyna. in the table}
}
\small
\begin{tabular}{p{0.7cm}<{\centering}p{0.7cm}<{\centering}p{0.7cm}<{\centering}|p{0.65cm}<{\centering}p{0.65cm}<{\centering}p{0.65cm}<{\centering}p{0.55cm}<{\centering}p{0.65cm}<{\centering}}
% \toprule
\hline
\multicolumn{3}{c|}{ GCNeXt block} & \multicolumn{4}{c}{tIoU on Validation Set}\\
Temp. & Sem. & Card. & 0.5 & 0.75 & 0.95 & Avg. \\
\hline
\xmark & \xmark & 1 & {48.12} & {32.16} & 6.41 & {31.65} \\ \hline
\cmark & \cmark & 1 & {50.20} & \textbf{34.80} & 7.35 & {33.88} \\
\cmark & \xmark & 32& {50.13} & {34.17} & {8.70} & {33.67} \\
\xmark & \cmark & 32& {49.09} & {33.32} & 8.02 & {32.63} \\ \hline
\cmark & \cmark & 32& {\textbf{50.36}} & {{34.60}} & \textbf{9.02} & {\textbf{34.09}} \\
\hline
% \bottomrule
\end{tabular}
\label{tab:abl_gcnext}
%\normalsize
%\vspace{-0.3cm}
\end{table}
\noindent \textbf{SGAlign Module}: This layer extracts sub-graph features by densely sampling and rescaling underlying snippet features. The sampling density is defined by factor $s$ in Alg.~\ref{alg}.
% Removing dense sampling is equivalent to setting $s=1$.
% We propose SGAlign layer to extract the sub-graph feature for the following localization task.
% We densely sample and rescale the input snippet features to update all of them in the backward propagation.
% Sub-graph features from both temporal and semantic graphs are concatenated.
Tab.~\ref{tab:abl_sgoi} shows the effect of the sampling and feature concatenation from both temporal and semantic graphs on ActivityNet-1.3. While sampling densely gives us minor improvements, we obtain a larger gain by including context information from the semantic graph.
% a bigger gain comes from aggregating context information from the last GCNeXt layer.
% the feature vector from neighbouring snippets defined in $\E_d$ to include context information.
% Top three rows in Tab.~\ref{tab:abl_sgoi} shows the model performance when we disable each of them individually on ActivityNet1.3 experiments. The sampling strategy slight improve the score, while the main gain is till from context information.
% In the last row of Tab.~\ref{tab:abl_sgoi}, we sample context snippet from semantic edges in all GCNeXt blocks to perform SGAlign, and show the result is still competitive. \AT{this might be a weak ablation, why do we sample from all instead of just one? what's the intuition?}
\begin{table}[tbp]
\centering
\caption{\textbf{Ablating SGAlign Components.%Different setup in the SGAlign layer and the corresponding G-TAD performance.
}
We disable the sample-rescale process and the feature concatenation from the semnantic graph for detection on ActivityNet-1.3. The rescaling strategy leads to slight improvement, while the main gain arises from the use of context information (semantic graph). }
\small
\begin{tabular}{p{1.0cm}<{\centering}p{1.0cm}<{\centering}|p{0.65cm}<{\centering}p{0.65cm}<{\centering}p{0.65cm}<{\centering}p{0.45cm}<{\centering}p{0.65cm}<{\centering}}
% \toprule
\hline
\multicolumn{2}{p{2cm}<{\centering}|}{ SGAlign} & \multicolumn{4}{c}{tIoU on Validation Set}\\
Samp. & Concat. & 0.5 & 0.75 & 0.95 & Avg. \\
\hline
\xmark & \xmark & {49.84} & {34.58} & 8.17 & {33.78} \\ \hline
%\xmark & last layer & {50.25} & {34.94} & 7.74 & {34.10} \\
%\xmark & all layers & {50.25} & {34.94} & 7.74 & {34.10} \\
\cmark & \xmark & {49.86} & \textbf{{34.60}} & \bf{9.56} & {33.89} \\
\cmark & \cmark & \bf{50.36} & \textbf{{34.60}} & 9.02 & \bf{34.09} \\
% \cmark & all & {50.26} & \bf{34.70} & 8.52 & {33.95} \\
\hline
% \bottomrule
\end{tabular}
\label{tab:abl_sgoi}
%\normalsize
%\vspace{-0.3cm}
\end{table}
\noindent\textbf{Sensitivity to Video Length}: We report the results of the sensitivity of G-TAD to different window sizes in THUMOS-14 in Tab.~\ref{tab:abl_window}. G-TAD benefits more from larger window sizes ($L=256$ vs. $128$)
% Limited by the GPU memory, we crop each THUMOS-14 video feature by a fixed window size to keep a large batch size. The first two rows in Tab.~\ref{tab:abl_window} shows G-TAD is better when we have a larger window size $L=256$ than $128$
for large windows mean G-TAD can aggregate more context snippets from the semantic graph. Performance degrades at $L=512$, where GPU memory limited us to use a batch size of only 2.
% However, if $L$ goes to $512$, the batch size becomes two for each GPU and the performance drops.
\begin{table}[!tb]
\centering
\caption{\textbf{Effect of Video Size.} We vary the input video size (window length $L$) and see that G-TAD performance improves with larger sizes ($L=256$). Degradation occurs at $L=512$, since GPU memory limits the batch size to be significantly reduced, leading to a noticeable performance drop.}
\small
\begin{tabular}{p{1.5cm}<{\centering}|p{0.6cm}<{\centering}p{0.6cm}<{\centering}p{0.6cm}<{\centering}p{0.6cm}<{\centering}p{0.6cm}<{\centering}}
\hline
Window & \multicolumn{5}{c}{tIoU on Validation} \\
Length $L$ & 0.3 & 0.4 & 0.5 & 0.6 & 0.7 \\ \hline
128 & 51.75 & 44.90 & 38.70 & 29.03 & 21.32\\
256 & \textbf{54.50} & \textbf{47.61} & \textbf{40.16} & \textbf{30.83} & \textbf{23.42} \\
512 & 48.32 & 41.71 & 34.38 & 26.85 & 19.29 \\ \hline
\end{tabular}
\label{tab:abl_window}
\end{table}
\begin{figure}
\centering
%\vspace{5cm}
\includegraphics[trim={3cm 2cm 3cm 2cm},width=8cm,clip]{img/GTAD_Synth.pdf}
\caption{\textbf{Semantic graphs and Context}. Given two untrimmed videos (left and right), we combine action frames from one video with context frames from the second (middle). We therefore create a synthetic video with no action context. As expected, the semantic graph of the synthetic video contains no edges between action and background snippets.
% \BG{whenever showing these graphs, make sure to point out the time dimension and ordering}
% Since there is no context related to the action in the synthetic video, the semantic edges connect action to only action and connect background only to background.
}
\label{fig:qual_results}
\end{figure}
\begin{figure*}[!ht]
\centering
\vspace{-0.5cm}
\includegraphics[trim={0cm 4cm 0cm 4cm}, width=16cm,clip]{img/GTAD_Vis.pdf}
\caption{\textbf{Qualitative results.} We show qualitative detection results on ActivityNet-1.3 (top) and THUMOS-14 (bottom).}
\label{fig:vis_video_THUMOS}
\end{figure*}
\subsection{Discussion of Action Context} %\FX{More discussion based on the feature}
% \noindent
% {\bf Context Edge}
% \BG{change title accordingly}
In the ablation experiments, graph convolutions on the semantic graph improve G-TAD performance in both the GCNeXt block and in the SGAlign layer. Semantic edges connecting background to action snippets can adaptively pass the action context information to each possible action. In this section, we define 2 extra experiments to show how semantic edges encode meaningful context information.
% We create a synthetic video with no context between action and background. We show how passing this artificial video through G-TAD leaves no action-background edges int he semantic graph.
\noindent {\bf Zero-Context Video}. How zero context between action and background leads to semantic graphs with no action-background edges is visually shown by comparing semantic graphs resulting from natural videos and synthetically compiled ones.
% \noindent {\bf None-Context Video}. Without any useful context in the background to an action in a video means no \textbf{action-background semantic edges} in the semantic graph.
%We verify this by visually comparing semantic graphs resulting from natural videos and synthetically compiled ones.
In Fig.~\ref{fig:qual_results} (left and right), we present two natural videos that include actions ``wrestling" and ``playing darts", respectively. Semantic edges in their resulting graphs do exist, connecting action with background snippets, thus exemplifying the usage of context in the detection process.
%showing that graph convolution can get information from long-distance dependencies.
%
Then, we compile a synthetic video that stacks action frames from the wrestling video and background frames from the darts video, feed it to G-TAD and again visualize the semantic graph (middle). As expected, the semantic graph does not include any action-background semantic edges.
% since the only edges form either adaptively connect action snippets together or background snippets.
% \BG{mention briefly the adaptive property of context}
\begin{figure}[!ht]
\centering
\includegraphics[trim={0cm 1cm 0cm 1cm},width=7cm,clip]{img/GTAD_Corr.pdf}
\caption{\textbf{Action-Background Semantic Edge Ratio vs. Context Amount.} In the scatter plot, each purple dot corresponds to a different video graph.
%Context ratio comes from semantic edges while context amount was measured using human annotators. For easier visualization, we also plot in green the average context edge ratio for five context amount ranges.
Strong positive correlation is observed between context amount and action-background semantic edge ratio, which means we predict on average more semantic edges in the presence of large video context.
% Purple scatters arrange videos by their \textit{Context Amount} and \textbf{Context Edge Ratio} while green curve shows the average context edge ratio for five context amount ranges. This plot shows a strong positive correlation between \textit{Context Amount} and \textit{Context Edge Ratio}.
}
\label{fig:ratio}
\end{figure}
\begin{figure}[!ht]
\centering
\includegraphics[trim={1.5cm 0.1cm 1cm 0cm},width=7.5cm,clip]{img/GTAD_Evol.pdf}
\caption{\textbf{Semantic graph evolution during G-TAD training.} We visualize the semantic graphs at first, middle, and last layers during training epoch 0, 3, 6, and 9. The semantic edges at the first layer are always the same, while the semantic graphs at the middle and last layers evolve to incorporate more context.}
\label{fig:evol}
\end{figure}
\noindent{\bf Correlation to Context Amount}. We also show the correlation between context edges and context as defined by human annotators.
We define the video \textbf{context amount} as the average number of background snippets which can be used to predict the video action class.
% A video clip is consecutive frames with fixed duration.
Following DETAD~\cite{detad}, we collect context amount for all videos in ActivityNet validation set from Amazon Mechanical Turk. % \BG{rewrite the whole paragraph} \BG{in DETAD, these background clips were sampled at various times around the action and not randomly, but you dont explain that }
The scatter plot in Fig.~\ref{fig:ratio} shows the relation between \textbf{Context Amount} and
% \FX{Purple scatters in Fig.~\ref{fig:ratio} arrange videos by their \textbf{Context Amount} and
the ratio of \textbf{action-background semantic edges} over all the semantic edges.
% }
From the plot, we observe that if a video has a higher amount of context (from human annotations), it is more likely to have more action-background semantic edges in its semantic graph. We further average context edge ratios in five context amount ranges, and plot them in green. The strong positive correlation between {Context Amount} and {action-background semantic edge ratio} indicates that our G-TAD model can effectively find related context snippets in the semantic graph.
% For visualization purposes, we down-sample to 200 videos using the Fixed-size Determinantal Point Process~\cite{kulesza2011k}.
% % \noindent
% % {\bf DETAD}
% % To have a more detailed ablation study, we also use the proposed evaluation matrices in DETAD~\cite{detad}. DETAD has extra annotations to quantify the number of temporal context snippets around an action instance. Thus, we can evaluate the average mAP on videos with different constraints, to see the main improvement.
% % \begin{figure*}
% % \centering
% % \includegraphics[trim={0cm 1cm 0cm 1cm},clip,width=14cm]{img/GTAD_DETAD.pdf}
% % \caption{\textbf{DETAD analysis of our baseline model and G-TAD model.} }
% % \label{fig:detad}
% % \end{figure*}
% If a video contains more context snippets in the background, the semantic graph should tend to have a larger proportion of context edge, the connections from action snippets to background snippets.
% We verify this by study the correlation of \textit{context amount} and \textit{context edge ratio}.
% \noindent
% {\bf Context Amount} We quantify video context amount from Amazon Mechanical Turk, a popular crowdsourcing marketplace.
% Concretely, multiple Turkers are asked to classify six background clips in each ActivityNet-1.3 video, while the \textit{Context Amount} is defined as average Turkers accuracy over different Turkers and different video clips. Therefore, a higher \textit{Context Amount} means the video contains more relevant background snippets to the action.
% \noindent
% {\bf Context Edge Ratio} To normalized the action duration in the video, \textit{Context Edge Ratio}, the ratio of context edge amount over action-action edges amount in the last semantical graph, is used to present the scale of action-background connections.
% Purple scatters in Fig.~\ref{fig:ratio} locate videos by their \textit{Context Amount} and {\bf Context Edge Ratio}. We down-sample to 200 videos by the Fixed-size Determinantal Point Processes~\cite{kulesza2011k} only for visualization purpose. From the dot plot, if the video have a higher amount of context, it is more likely to have more context edges in its semantic graph.
% %Context amount is collected from amazon tuckers' responses as in DETAD~\cite{detad}. A higher value means the video contains for relevant background snippets to the action. Please find context amount details in the supplementary materials.
% % In the model inference time, we saved the semantic edges in the subgraph alignment layer. The context edge is defined as the links cross action snippets and background snippets. We take the ratio of context edge and non-context edge in the action as a character of the video.
% %
% We further average context edge ratio for five context amount ranges, and plot it by the green curves, which shows a strong positively correlation between \textit{Context Amount} and {\bf Context Edge Ratio}, meaning that our G-TAD model can learn to find related snippet in the graph convolution.
% %
% % For example, if the video have less context snippets in the background frames, the dynamic graph edges mostly connect actions to action and background to background, and vice versa.
% %
% %This correlation shows that our G-TAD model can learn to find related snippet in the graph convolution.
% %
% %model performance on different context scale video, we also calculate the correlation between graph context feature ratio and video context ratio. The video context amount is defined as the related background snippet before and after the action instance. We choose Amazon Mechanical Turk to measure the video context amount and hold the user study All videos in the validation set are labelled by three different Turkers and the annotations will be publicly available. We also measure the ratio of dynamic edges which connect action framed and background frames.
% \begin{figure}
% \centering
% \includegraphics[trim={0cm 0cm 0cm 0cm},width=6cm,clip]{img/GTAD_Corr.pdf}
% \caption{\textbf{Correlation of video context amount and graph context feature ratio.} }
% \label{fig:ratio}
% \end{figure}
% \noindent
% \textbf{Dynamic graphs in videos}
\subsection{Visualization}
We show a few qualitative detection results in Fig.~\ref{fig:vis_video_THUMOS} on both ActivityNet-1.3 and THUMOS-14. In Fig.~\ref{fig:evol}, we visualize the evolution of semantic graphs during the training process across GCNeXt layers. Specifically, we feed a video into G-TAD and visualize the semantic graphs emerging at the first, middle, and last layers at epochs 0, 3, 6, and 9 of training. The semantic graphs at the first layer are the same, since they are built on the same input features. As we progress to different layers and epochs, semantic graphs adaptively update their edges. Interestingly, we observe the presence of more context edges as training advances. This indicates that G-TAD progressively learns to incorporate multiple levels of context in the detection process.
% context edges appear more as training advances.
% \textbf{Qualitative results} Given the significant improvements, we also attempt to find out in what cases our G-TAD model improves over
% CNN methods and its weakness.
% We visualize the qualitative results of G-TAD on both ActivityNet-1.3 and THUMOS-14 in Fig.~\ref{fig:vis_video_THUMOS}.
% In the left two examples, both our G-TAD model are able to predict the action category correctly, while in the right side examples G-TAD differ slightly from the ground truth, still envelops it.
% %
% More qualitative results could be found in the supplementary material.
% \begin{figure*}
% \centering
% \includegraphics[trim={2cm 0.5cm 0cm 0.5cm},width=17cm,clip]{"img/ACTIVITYNET zumba".png}
% \includegraphics[trim={2cm 0.5cm 0cm 0.5cm},width=17cm,clip]{"img/ACTIVITYNET preparing salad".png}
% \caption{\textbf{Qualitative results in ACTIVITYNET.} }
% \label{fig:vis_video_ActivityNet}
% \end{figure*}
% \begin{figure*}
% \centering
% \includegraphics[trim={2cm 0.5cm 0cm 0.5cm},width=17cm,clip]{"img/THUMOS cliff diving".png}
% \includegraphics[trim={2cm 0.5cm 0cm 0.5cm},width=17cm,clip]{"img/THUMOS long jump".png}
% \caption{\textbf{Qualitative results in THUMOS.} }
% \label{fig:vis_video_THUMOS}
% \end{figure*}
% \noindent
% \textbf{Graph evolution in training} In Fig.~\ref{fig:evol}, we feed a video into G-TAD, and visualize the semantic graphs on first, middle and last layers in training epoch 0, 3, 6 and 9. The semantic graphs on the first layer are always the same because they are from the fixed input feature, while the semantic graphs on the middle and last layer develops a certain pattern between the action and background snippets.
In this paper, we cast the temporal action detection task as a sub-graph localization problem by formulating videos as graphs.
We take video snippets as graph nodes, snippet-snippet correlations as edges, and apply graph convolution as the basic operation.
We propose a new architecture G-TAD to localize sub-graphs. G-TAD includes GCNeXt blocks to aggregate context-enriched snippet features and an SGAlign layer to transform sub-graph features into vector representations.
G-TAD can learn enriched multi-level semantic context in an adaptive way by looking at snippet features.
Extensive experiments show that G-TAD can find global video context without extra supervision and achieve the state-of-the-art performance on both Thumos-14 and ActivityNet-1.3 under different metrics.
\subsection{Derivation and Efficient Implementation of Eq.~{\color{red} 4}}
In this section, we provide the derivation of Eq.~{\color{red} 4} in the paper (listed here in the following). We also show that Eq.~{\color{red} 4} can be efficiently implemented by zero-padded 1D/edge convolutions. \newline
\begin{align} \label{eq:block_supp}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\H(X,\E,W)=ReLU[&W_\alpha X + W_\beta X A_t^f + W_\gamma X A_t^b+ _\phi X A_s+X]. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \textrm{(4)}\nonumber
\end{align}
\noindent
\subsection{Derivation of Eq. 4}
\noindent
\textbf{a) Temporal Graph Convolution.} We first provide the derivation for temporal graph convolution.
The temporal forward edges $\mathcal{E}_t^f$ and backward edges $\mathcal{E}_t^b$ are formulated as
\begin{align}
\mathcal{E}_t^f=\{(v_i,v_{i+1})|~i\in\{1,2,\dots,L-1\}\} , \mathcal{E}_t^b=\{(v_i,v_{i-1})|~i\in\{2,\dots,L-1,L\},
%A_t^f X^T W_t^f+A_t^b X^T W_t^b
\end{align}
The corresponding adjacency matrices $A_t^f, A_t^b$ can be present by $L$ vectors, respectively, shown in Eq.~\ref{eq:tgc_supp}. We use $e_k\in \mathbb{R}^{L}$ to present the vector in which the $k$-th element is one but the others are zeros.
\begin{align} \label{eq:tgc_supp}
A_t^f = &
\begin{bmatrix}
0 & 0 & 0 & \dots & 0 & 0 \\
1 & 0 & 0 & & 0 & 0 \\
0 & 1 & 0 & & 0 & 0 \\
\vdots& & & \ddots & \vdots & \vdots \\
0 & 0 & 0 & \dots & 0 & 0 \\
0 & 0 & 0 & \dots & 1 & 0 \\
\end{bmatrix}
= [e_2,e_3,\dots, e_L, 0] \nonumber\\
A_t^b = &
\begin{bmatrix}
0 & 1 & 0 & \dots& 0 & 0 \\
0 & 0 & 1 & & 0 & 0 \\
0 & 0 & 0 & & 0 & 0 \\
\vdots& & & \ddots & \vdots & \vdots \\
0 & 0 & 0 & \dots & 0 & 1 \\
0 & 0 & 0 & \dots & 0 & 0 \\
\end{bmatrix}
= [0, e_1, e_2,\dots, e_{L-1}], \nonumber\\
\Rightarrow & A_t^f = [A_t^b]^T
%A_t^f X^T W_t^f+A_t^b X^T W_t^b
\end{align}
Given the input $X\in\mathbb{R}^{C\times L}$, after temporal graph convolution, the output, $X_t$, becomes
\begin{align} \label{eq:tgc}
X_t &= f_{agg}(X,A_t^f,W_f) + f_{agg}(X,A_t^b,W_b) =W_{f,0} X + W_{f,1} X A_t^f + W_{b,0} X + W_{b,1} X A_t^b \nonumber\\
& =(W_{f,0}+W_{b,0}) X + W_{f,1} X A_t^f + W_{b,1} X A_t^b .
%A_t^f X^T W_t^f+A_t^b X^T W_t^b
\end{align}
Here $W_*$ is the trainable weights in the neural network.
\noindent
\textbf{b) Semantic Graph Convolution.} It is straightforward to obtain Eq.~{\color{red} 4} for semantic graph convolution.
\noindent
\subsection{Efficient Implementation of Eq.~{\color{red} 4}}
In implementation of Eq.~{\color{red} 4}, we use an efficient zero-padded 1D convolution and edge convolution for temporal graph convolution and semantic graph convolution, respectively. In the following, we provide proof that our efficient implementation is equivalent to Eq.~{\color{red} 4}.
\noindent
\textbf{a) Temporal Graph Convolution.}
If a 1D convolution has kernel size 3, the weight matrix is a 3D tensor in $\mathbb{R}^{3\times C \times C}$. We denote the matrix as $W_{conv1}= [W_1,W_2,W_3],W_{1,2,3}\in \mathbb{R}^{C\times C}$.
Given the same input $X=[x_1, x_2,...,x_L]$, we pad zero on the input, $x_0=x_{L+1}=0\in \mathbb{R}^{C}$.
The output of 1D convolution can be written as $Y=[y_1,y_2,...,y_L]\in \mathbb{R}^{C\times L}$
\begin{align} \label{eq:y_k}
y_k = [W_1 x_{k-1}+W_2 x_k+W_3x_{k+1}], k=1,2,\dots,L
\end{align}
We can prove that $X_t=Y$ by multiplying $e_k$ on both sides in Eq.~\ref{eq:proof}. Please be noted that $W_*$ is the trainable weights in the neural network. We can assume $W_1=W_{b,1},W_3=W_{f,1}, W_2=W_{f,0}+W_{b,0}$
\begin{align} \label{eq:proof}
X_te_k&=W_{f,0} Xe_k + W_{f,1} X A_t^fe_k + W_{b,0} Xe_k + W_{b,1} X A_t^be_k \nonumber\\
&=(W_{f,0}+W_{b,0}) x_k + W_{f,1} X [0, e_1, e_2,\dots, e_L]^T e_k + W_{b,1} X [e_2,e_3,\dots, e_L, 0]^T e_k \nonumber\\
&=(W_{f,0}+W_{b,0}) x_k + W_{f,1} X e_{k+1} + W_{b,1} X e_{k-1} \nonumber\\
&=(W_{f,0}+W_{b,0}) x_k + W_{f,1} x_{k+1} + W_{b,1} x_{k-1} \nonumber\\
&=W_1 x_{k-1}+W_2 x_k+W_3X_{k+1}
\end{align}
\noindent
\textbf{b) Semantic Graph Convolution.}
In the semantic graph, edge convolution is directly used, so proof is done.
\subsection{Training Details}
\noindent
\textbf{Semantic Edges from Multiple Levels.}
In G-TAD, we use multiple GCNeXt blocks to adaptively incorporate multi-level semantic context into video features. After that, SGAlign layer embeds each sub-graph by concatenating aligned features from temporal and semantic graphs. However, it is not necessary to consider \textbf{only the last} GCNeXt semantic graphs to align the semantic feature. Last row in Tab.~\ref{tab:abl_sgoi_supp} present one more experiment that takes \textbf{the union of} semantic edges from all GCNeXt blocks to aggregate the semantic feature. We can find that the semantic context also helps to improve model performance under this setup.
\begin{table}[h]
\centering
\caption{\textbf{Ablating SGAlign Components.%Different setup in the SGAlign layer and the corresponding G-TAD performance.
}
We disable the sample-rescale process and the feature concatenation from the semnantic graph for detection on ActivityNet-1.3. The rescaling strategy leads to slight improvement, while the main gain arises from the use of context information (semantic graph). }
\small
\begin{tabular}{p{1.0cm}<{\centering}p{1.0cm}<{\centering}|p{0.65cm}<{\centering}p{0.65cm}<{\centering}p{0.65cm}<{\centering}p{0.45cm}<{\centering}p{0.65cm}<{\centering}}
% \toprule
\hline
\multicolumn{2}{p{2cm}<{\centering}|}{ SGAlign} & \multicolumn{4}{c}{tIoU on Validation Set}\\
Samp. & Concat. & 0.5 & 0.75 & 0.95 & Avg. \\
\hline
\xmark & \xmark & {49.84} & {34.58} & 8.17 & {33.78} \\ \hline
%\xmark & last layer & {50.25} & {34.94} & 7.74 & {34.10} \\
%\xmark & all layers & {50.25} & {34.94} & 7.74 & {34.10} \\
\cmark & \xmark & {49.86} & {{34.60}} & \bf{9.56} & {33.89} \\
\cmark & \cmark & \bf{50.36} & {{34.60}} & 9.02 & \bf{34.09} \\ \hline
\cmark & all & {50.26} & \bf{34.70} & 8.52 & {33.95} \\
\hline
% \bottomrule
\end{tabular}
\label{tab:abl_sgoi_supp}
%\normalsize
%\vspace{-0.3cm}
\end{table}
% {\small
% \bibliographystyle{ieee_fullname}
% \bibliography{egbib}
% }
\noindent
\textbf{2D Conv. for Sub-Graph Localization.}
% For all pairs of $(t_s, t_e)$, we calculate its IoU with all $\psi$ in $\Psi$, and denote the maximum IoU $g_c$ as its label.
Once we get the \textit{sub-graph feature} from SGAlign layer, instead of using three fully connected (FC) layers regress to $g_c$, we can arrange the anchors in a 2D $L\times L$ map based on the start/end time, and set zeros to the map where is no pre-designed anchors (e.g. $t_s>t_e$). In doing so, we can use 2D CNNs to regress to a $g_c$ map that arranged by the same order. We call the predicted matrix \textbf{IoU map}.
The neighbouring anchors in the 2D IoU map have similar boundary locations. Thus we can use the proposal-proposal relationship in the 2D convolutions. We set kernel size to 1, 3, and 5, and the results are shown in Tab.~\ref{tab:abl_Conv_on_IOU_map}. We do not observe any significant benefit from 2D convolutions.
\begin{table}[h]
\centering
\caption{\textbf{The model performance when we use 3 2D convolution layers to predict IoU map.}We set kernel size to 1, 3, and 5, and collect result on ActivityNet1.3. We do not observe any significant benefit from 2D convolutions.}
\small
\begin{tabular}{p{1.5cm}<{\centering}p{0.7cm}<{\centering}|p{0.65cm}<{\centering}p{0.65cm}<{\centering}p{0.65cm}<{\centering}p{0.55cm}<{\centering}p{0.65cm}<{\centering}}
% \toprule
\hline
\multicolumn{2}{p{3.2cm}|}{ Conv. on IoU map} & \multicolumn{4}{c}{mAP on Validation Set}\\
Kernel Size & Padding & 0.5 & 0.75 & 0.95 & Avg. \\
\hline
(1,1) & (0,0) & {\textbf{50.25}} & {34.66} & \textbf{9.29} & {34.08} \\
(3,3) & (1,1) & {\textbf{50.25}} & {\textbf{34.94}} & 7.74 & {\textbf{34.10}} \\
(5,5) & (2,2) & {49.88} & {34.39} & 8.96 & {33.77} \\
\hline
% \bottomrule
\end{tabular}
\label{tab:abl_Conv_on_IOU_map}
%\normalsize
%\vspace{-0.3cm}
\end{table}
\begin{thebibliography}{10}\itemsep=-1pt
\bibitem{detad}
Humam Alwassel, Fabian Caba~Heilbron, Victor Escorcia, and Bernard Ghanem.
\newblock Diagnosing error in temporal action detectors.
\newblock In {\em European Conference on Computer Vision (ECCV)}, 2018.
\bibitem{Alwassel2017ActionSS}
Humam Alwassel, Fabian~Caba Heilbron, and Bernard Ghanem.
\newblock Action search: Spotting actions in videos and its application to
temporal action localization.
\newblock In {\em European Conference on Computer Vision (ECCV)}, 2017.
\bibitem{softNMS}
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry~S. Davis.
\newblock Soft-nms – improving object detection with one line of code.
\newblock In {\em International Conference on Computer Vision (ICCV)}, 2017.
\bibitem{buch2017end}
Shyamal Buch, Victor Escorcia, Bernard Ghanem, Li Fei-Fei, and Juan~Carlos
Niebles.
\newblock End-to-end, single-stream temporal action detection in untrimmed
videos.
\newblock In {\em the British Machine Vision Conference (BMVC)}, 2017.
\bibitem{buch2017sst}
Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan
Carlos~Niebles.
\newblock Sst: Single-stream temporal action proposals.
\newblock In {\em Proceedings of the IEEE conference on Computer Vision and
Pattern Recognition (CVPR)}, 2017.
\bibitem{caba2016fast}
Fabian Caba~Heilbron, Juan Carlos~Niebles, and Bernard Ghanem.
\newblock Fast temporal activity proposals for efficient detection of human
actions in untrimmed videos.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition (CVPR)}, 2016.
\bibitem{caba2015activitynet}
Fabian Caba~Heilbron, Victor Escorcia, Bernard Ghanem, and Juan~Carlos Niebles.
\newblock Activitynet: A large-scale video benchmark for human activity
understanding.
\newblock In {\em IEEE Conference on Computer Vision and Pattern Recognition
(CVPR)}, 2015.
\bibitem{carreira2017quo}
Joao Carreira and Andrew Zisserman.
\newblock Quo vadis, action recognition? a new model and the kinetics dataset.
\newblock In {\em proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}, 2017.
\bibitem{chao2018rethinking}
Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David~A. Ross, Jia
Deng, and Rahul Sukthankar.
\newblock Rethinking the faster r-cnn architecture for temporal action
localization.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}, 2018.
\bibitem{chen2019graph}
Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and
Yannis Kalantidis.
\newblock Graph-based global reasoning networks.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}, 2019.
\bibitem{dai2017temporal}
Xiyang Dai, Bharat Singh, Guyue Zhang, Larry~S. Davis, and Yan Qiu~Chen.
\newblock Temporal context network for activity localization in videos.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision (ICCV)}, 2017.
\bibitem{deng2009imagenet}
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
\newblock Imagenet: A large-scale hierarchical image database.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition (CVPR)}, 2009.
\bibitem{escorcia2016daps}
Victor Escorcia, Fabian~Caba Heilbron, Juan~Carlos Niebles, and Bernard Ghanem.
\newblock Daps: Deep action proposals for action understanding.
\newblock In {\em Proceedings of the European Conference on Computer Vision
(ECCV)}, 2016.
\bibitem{feichtenhofer2016convolutional}
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.
\newblock Convolutional two-stream network fusion for video action recognition.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}, 2016.
\bibitem{gao2017turn}
Jiyang Gao, Zhenheng Yang, Kan Chen, Chen Sun, and Ram Nevatia.
\newblock Turn tap: Temporal unit regression network for temporal action
proposals.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision (ICCV)}, 2017.
\bibitem{gao2017cascaded}
Jiyang Gao, Zhenheng Yang, and Ram Nevatia.
\newblock Cascaded boundary regression for temporal action detection.
\newblock In {\em Proceedings of the British Machine Vision Conference (BMVC)},
2017.
\bibitem{girshick2015fast}
Ross Girshick.
\newblock Fast r-cnn.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision (ICCV)}, 2015.
\bibitem{Gkioxari2019Mesh}
Georgia Gkioxari, Jitendra Malik, and Justin Johnson.
\newblock Mesh r-cnn.
\newblock {\em arXiv preprint arXiv:1906.02739}, 2019.
\bibitem{he2017mask}
Kaiming He, Georgia Gkioxari, Piotr Doll{\'a}r, and Ross Girshick.
\newblock Mask r-cnn.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision (ICCV)}, 2017.
\bibitem{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition (ICCV)}, 2016.
\bibitem{heilbron2017scc}
Fabian~Caba Heilbron, Wayner Barrios, Victor Escorcia, and Bernard Ghanem.
\newblock Scc: Semantic context cascade for efficient action detection.
\newblock In {\em 2017 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)}, 2017.
\bibitem{hou2017real}
Rui Hou, Rahul Sukthankar, and Mubarak Shah.
\newblock Real-time temporal action localization in untrimmed videos by
sub-action discovery.
\newblock In {\em Proceedings of the British Machine Vision Conference (BMVC)},
2017.
\bibitem{jiang2014thumos}
YG Jiang, J Liu, A~Roshan Zamir, G Toderici, I Laptev, M Shah, and R
Sukthankar.
\newblock Thumos challenge: Action recognition with a large number of classes,
2014.
\bibitem{kipf2016semi}
Thomas~N Kipf and Max Welling.
\newblock Semi-supervised classification with graph convolutional networks.
\newblock {\em arXiv preprint arXiv:1609.02907}, 2016.
\bibitem{Li_2019_ICCV}
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem.
\newblock Deepgcns: Can gcns go as deep as cnns?
\newblock In {\em The IEEE International Conference on Computer Vision (ICCV)},
2019.
\bibitem{lin2019fast}
Chuming Lin, Jian Li, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie
Wang, Jilin Li, Feiyue Huang, and Rongrong Ji.
\newblock Fast learning of temporal action proposal via dense boundary
generator, 2019.
\bibitem{lin2019bmn}
Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen.
\newblock Bmn: Boundary-matching network for temporal action proposal
generation.
\newblock In {\em The IEEE International Conference on Computer Vision (ICCV)},
2019.
\bibitem{lin2017single}
Tianwei Lin, Xu Zhao, and Zheng Shou.
\newblock Single shot temporal action detection.
\newblock In {\em Proceedings of the 2017 {ACM} on Multimedia Conference, {MM}
2017, Mountain View, CA, USA}, 2017.
\bibitem{lin2018bsn}
Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.
\newblock Bsn: Boundary sensitive network for temporal action proposal
generation.
\newblock In {\em Proceedings of the European Conference on Computer Vision
(ECCV)}, 2018.
\bibitem{liu2019learning}
Xingyu Liu, Joon-Young Lee, and Hailin Jin.
\newblock Learning video representations from correspondence proposals.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}, 2019.
\bibitem{DBLP:journals/corr/abs-1811-11524}
Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih{-}Fu Chang.
\newblock Multi-granularity generator for temporal action proposal.
\newblock {\em Computing Research Repository (CoRR)}, 2018.
\bibitem{Long2019GaussianTA}
Fuchen Long, Ting Yao, Zhaofan Qiu, Xinmei Tian, Jiebo Luo, and Tao Mei.
\newblock Gaussian temporal awareness networks for action localization.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}, 2019.
\bibitem{niepert2016learning}
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov.
\newblock Learning convolutional neural networks for graphs.
\newblock In {\em Proceedings of the International conference on machine
learning (ICML)}, 2016.
\bibitem{qiu2017learning}
Zhaofan Qiu, Ting Yao, and Tao Mei.
\newblock Learning spatio-temporal representation with pseudo-3d residual
networks.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision (ICCV)}, 2017.
\bibitem{ren2015faster}
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
\newblock Faster r-cnn: Towards real-time object detection with region proposal
networks.
\newblock In {\em Advances in neural information processing systems}, 2015.
\bibitem{richard2016temporal}
Alexander Richard and Juergen Gall.
\newblock Temporal action detection using a statistical language model.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, 2016.
\bibitem{shou2017cdc}
Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu
Chang.
\newblock Cdc: Convolutional-de-convolutional networks for precise temporal
action localization in untrimmed videos.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}, 2017.
\bibitem{shou2016temporal}
Zheng Shou, Dongang Wang, and Shih-Fu Chang.
\newblock Temporal action localization in untrimmed videos via multi-stage
cnns.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}, 2016.
\bibitem{simonyan2014two}
Karen Simonyan and Andrew Zisserman.
\newblock Two-stream convolutional networks for action recognition in videos.
\newblock In {\em Advances in neural information processing systems}, 2014.
\bibitem{simonyan2014very}
Karen Simonyan and Andrew Zisserman.
\newblock Very deep convolutional networks for large-scale image recognition.
\newblock {\em arXiv preprint arXiv:1409.1556}, 2014.
\bibitem{singh2016untrimmed}
Gurkirt Singh and Fabio Cuzzolin.
\newblock Untrimmed video classification for activity detection: submission to
activitynet challenge.
\newblock {\em arXiv preprint arXiv:1607.01979}, 2016.
\bibitem{tran2015learning}
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.
\newblock Learning spatiotemporal features with 3d convolutional networks.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision (ICCV)}, 2015.
\bibitem{velivckovic2017graph}
Petar Veli{\v{c}}kovi{\'c}, Guillem Cucurull, Arantxa Casanova, Adriana Romero,
Pietro Lio, and Yoshua Bengio.
\newblock Graph attention networks.
\newblock {\em arXiv preprint arXiv:1710.10903}, 2017.
\bibitem{wang2017untrimmed}
Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van~Gool.
\newblock Untrimmednets for weakly supervised action recognition and detection.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, 2017.
\bibitem{wang2015towards}
Limin Wang, Yuanjun Xiong, Zhe Wang, and Yu Qiao.
\newblock Towards good practices for very deep two-stream convnets.
\newblock {\em arXiv preprint arXiv:1507.02159}, 2015.
\bibitem{TSN2016ECCV}
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc
{Val Gool}.
\newblock Temporal segment networks: Towards good practices for deep action
recognition.
\newblock In {\em Proceedings of the European Conference on Computer Vision
(ECCV)}, 2016.
\bibitem{wang2016temporal}
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc
Van~Gool.
\newblock Temporal segment networks: Towards good practices for deep action
recognition.
\newblock In {\em Proceedings of the European Conference on Computer Vision},
2016.
\bibitem{wang2016uts}
Ruxin Wang and Dacheng Tao.
\newblock Uts at activitynet 2016.
\newblock {\em ActivityNet Large Scale Activity Recognition Challenge}, 2016.
\bibitem{wang2018videos}
Xiaolong Wang and Abhinav Gupta.
\newblock Videos as space-time region graphs.
\newblock In {\em Proceedings of the European Conference on Computer Vision
(ECCV)}, 2018.
\bibitem{wang2018dynamic}
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael Bronstein, and Justin
Solomon.
\newblock Dynamic graph cnn for learning on point clouds.
\newblock {\em ACM Transactions on Graphics}, 2018.
\bibitem{DBLP:journals/corr/abs-1801-07829}
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay~E. Sarma, Michael~M. Bronstein, and
Justin~M. Solomon.
\newblock Dynamic graph {CNN} for learning on point clouds.
\newblock {\em Computing Research Repository (CoRR)}, 2018.
\bibitem{xie2017aggregated}
Saining Xie, Ross Girshick, Piotr Doll{\'a}r, Zhuowen Tu, and Kaiming He.
\newblock Aggregated residual transformations for deep neural networks.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition (CVPR)}, 2017.
\bibitem{xie2019clouds}
Zhuyang Xie, Junzhou Chen, and Bo Peng.
\newblock Point clouds learning with attention-based graph convolution
networks.
\newblock {\em arXiv preprint arXiv:1905.13445}, 2019.
\bibitem{xiong2016cuhk}
Yuanjun Xiong, Limin Wang, Zhe Wang, Bowen Zhang, Hang Song, Wei Li, Dahua Lin,
Yu Qiao, L. Van~Gool, and Xiaoou Tang.
\newblock Cuhk \& ethz \& siat submission to activitynet challenge 2016.
\newblock 2016.
\bibitem{xu2017r}
Huijuan Xu, Abir Das, and Kate Saenko.
\newblock R-c3d: Region convolutional 3d network for temporal activity
detection.
\newblock In {\em Proceedings of the IEEE international conference on computer
vision (ICCV)}, 2017.
\bibitem{yeung2016end}
Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei.
\newblock End-to-end learning of action detection from frame glimpses in
videos.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, 2016.
\bibitem{Yuan2017}
Zehuan Yuan, Jonathan~C. Stroud, Tong Lu, and Jia Deng.
\newblock Temporal action localization by structured maximal sums.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)}, 2017.
\bibitem{zeng2019graph}
Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang,
and Chuang Gan.
\newblock Graph convolutional networks for temporal action localization.
\newblock {\em arXiv preprint arXiv:1909.03252}, 2019.
\bibitem{zhao2017temporal}
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin.
\newblock Temporal action detection with structured segment networks.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision (ICCV)}, 2017.
\bibitem{zisserman2017kinetics}
Andrew Zisserman, Joao Carreira, Karen Simonyan, Will Kay, Brian Zhang, Chloe
Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back,
et~al.
\newblock The kinetics human action video dataset.
\newblock {\em arXiv preprint arXiv:1705.06950}, 2017.
\end{thebibliography}