Multi-Granularity Hand Action Detection
Extracted source text · arXiv 2306.10858
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
\PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_data_2023
% ready for submission
\usepackage[preprint]{neurips_data_2023}
\usepackage{diagbox}
\usepackage{bm}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{graphicx}
\usepackage{float}
\usepackage{subfigure}
\usepackage{array}
\usepackage{threeparttable}
\usepackage{amsmath,amssymb,latexsym}
\usepackage{caption}
\usepackage{overpic}
\usepackage{multirow}
\usepackage{makecell}
\usepackage{wrapfig}
\usepackage{makecell}
\usepackage{epsfig}
\usepackage{float}
\usepackage{multicol}
\usepackage{multirow}
\usepackage{amssymb}
\usepackage{wrapfig}
\makeatletter
\newcommand\figcaption{\def\@captype{figure}\caption}
\newcommand\tabcaption{\def\@captype{table}\caption}
\makeatother
\title{FHA-Kitchens: A Novel Dataset for Fine-Grained Hand Action Recognition in Kitchen Scenes}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Ting Zhe$^{1}$,
Yongqian Li$^1$,
Jing Zhang$^2$,
Yong Luo$^{1}$,\\
\textbf{Han Hu}$^3$,
\textbf{Bo Du}$^{1}$,
\textbf{Yonggang Wen}$^{4}$,
\textbf{Dacheng Tao}$^2$
\and
\vspace{-2 mm}
\\
\textsuperscript{1}Wuhan University, China,
\textsuperscript{2}The University of Sydney, Australia,\\
\textsuperscript{3}Beijing Institute of Technology, China,
\textsuperscript{4}Nanyang Technological University, Singapore\\
\vspace{-25 mm}
}
\begin{document}
\maketitle
\begin{abstract}
A typical task in the field of video understanding is hand action recognition, which has a wide range of applications. Existing works either mainly focus on full-body actions, or the defined action categories are relatively coarse-grained. In this paper, we propose \textbf{FHA-Kitchens}, a novel dataset of fine-grained hand actions in kitchen scenes. In particular, we focus on human hand interaction regions and perform deep excavation to further refine hand action information and interaction regions. Our FHA-Kitchens dataset consists of 2,377 video clips and 30,047 images collected from 8 different types of dishes, and all hand interaction regions in each image are labeled with high-quality fine-grained action classes and bounding boxes. We represent the action information in each hand interaction region as a triplet, resulting in a total of 878 action triplets. Based on the constructed dataset, we benchmark representative action recognition and detection models on the following three tracks: (1) supervised learning for hand interaction region and object detection, (2) supervised learning for fine-grained hand action recognition, and (3) intra- and inter-class domain generalization for hand interaction region detection. The experimental results offer compelling empirical evidence that highlights the challenges inherent in fine-grained hand action recognition, while also shedding light on potential avenues for future research, particularly in relation to pre-training strategy, model design, and domain generalization. The dataset will be released at \href{https://github.com/tingZ123/FHA-Kitchens}{FHA-Kitchens}.
\end{abstract}
\section{Introduction}\label{intro}
Action recognition, a prominent task within the domain of video understanding, has garnered considerable attention and possesses broad applications across various fields~\cite{zhang2020empowering}, including human-computer interaction (HCI)~\cite{49hand-object}, smart homes~\cite{55smarthome}, healthcare~\cite{48RCareWorld}, and the design and control of robot hands~\cite{54robothand}. While there has been extensive research on action recognition concerning large-scale benchmarks~\cite{3ucf101,6kinetic400} and advanced algorithms~\cite{19TSN,22slowfast,23swin}, relatively fewer studies have focused on the recognition of fine-grained hand actions. This is primarily due to the extensive diversity of hand actions, the fine-grained categorization of such actions, and the complex interaction situations, all of which pose significant challenges in terms of data collection and annotation. Nevertheless, given that a substantial portion of human actions in daily life originates from hand actions, the recognition of fine-grained hand actions assumes critical importance in both research and practical applications. Therefore, it is desirable to establish a large-scale benchmark that encompasses diverse fine-grained hand actions, as it would serve as a solid foundation for further research in this field.
\begin{figure}[!t]
\setlength{\belowcaptionskip}{-0.5cm}
\centering
\includegraphics[width=1\linewidth]{openfig.pdf}
\caption{Overview of the \textbf{FHA-Kitchens} dataset. The top shows some frames extracted from 8 dish categories. The bottom illustrates the annotation process of fine-grained actions in \emph{``fry vegetable''}.} \label{openfig}
\end{figure}
Existing studies on action recognition can be divided into two main categories based on the types of actions: full-body action and part-body action. While datasets like UCF101~\cite{3ucf101}, Kinetics 400–700~\cite{6kinetic400,7kinetics600,8kinetic700}, and AVA~\cite{10ava} provide annotations for full-body actions encompassing human pose actions and interactions with other individuals or objects, they do not sufficiently capture the fine-grained action information generated by specific body parts. Consequently, these datasets exhibit limited action diversity, high inter-class variability, and coarse-grained representations, such as ``driving'', ``drawing'', and ``sitting''. In an effort to address the need for fine-grained action recognition, datasets like FineGym~\cite{9finegym}, MPII Cooking Activities~\cite{2MPII}, and EPIC-KITCHENS~\cite{1epic} have been developed. However, these datasets possess their own limitations, including an insufficient focus on hand actions, a lack of information regarding action interactions, and incomplete localization of interaction regions. These limitations pose significant obstacles to the research endeavors aimed at tackling the inherent challenges associated with fine-grained hand action recognition tasks.
In this paper, we present a novel dataset, namely the \textbf{FHA-Kitchens} dataset, which focuses on fine-grained hand actions observed in kitchen scenes. The FHA-Kitchens dataset encompasses a total of 2,377 video clips and 30,047 images, all extracted from eight different dish types (Figure~\ref{openfig} top). Each frame within the dataset is accompanied by meticulously labeled hand interaction regions, featuring high-quality annotations of fine-grained action categories and bounding boxes. To create the FHA-Kitchens dataset, we derived data from publicly available large-scale action datasets~\cite{700_2020}, specifically targeting videos that were highly relevant to hand actions, and conducted frame extraction and cleaning processes. Subsequently, we engaged the expertise of ten voluntary annotators to meticulously label the interaction regions of each hand. Notably, we divided the hand interaction regions into three distinct sub-regions: the \textbf{left hand-object} interaction region, the \textbf{right hand-object} interaction region, and the \textbf{object-object} interaction region. For each sub-interaction region, we provided bounding box annotations. Furthermore, we categorized hand actions into three distinct types to adequately capture the actions within the sub-interaction regions. Each sub-interaction region action was annotated using a triplet format, denoted as \emph{<subject, action verb, object>}. Additionally, we took into account the ``active-passive'' relationships between object pairs and the specific contact areas involved in the interaction actions. Consequently, our annotation process encompassed a total of nine dimensions (Figure~\ref{openfig} bottom), resulting in 878 annotated action triplets for hand actions. Finally, we organized the video frames based on the action triplet classes, ultimately generating 2,377 clips that represent distinct hand action triplet classes.
The FHA-Kitchens dataset provides valuable opportunities for advancing fine-grained hand action research, owing to its extensive diversity and high-dimensional representations of fine-grained hand actions. In light of this, we propose three distinct tracks as benchmarks for assessing representative action recognition and detection models. These tracks encompass: (1) supervised learning for hand interaction region and object detection (SL-D track), (2) supervised learning for fine-grained hand action recognition (SL-AR track), and (3) intra- and inter-class domain generalization for hand interaction region detection (DG track). In the SL-D track, we investigate the representative Faster-RCNN model~\cite{24faster} with different backbone networks. In the SL-AR track, we train and evaluate several representative action recognition models (\textit{e.g.}, TSN~\cite{19TSN}, SlowFast~\cite{22slowfast}, and Video Swin~\cite{23swin}) and investigate the influence of parameter initialization. In the DG track, we investigate the generalization ability of detection models regarding intra-class generalization, where the model is trained on instances belonging to specific sub-categories within the same parent categories and subsequently tested on other sub-categories within the same parent categories, and inter-class generalization, where the model is trained on all instances encompassing a specific parent category and then tested on other different parent categories. The experimental findings furnish compelling empirical evidence that provides valuable insights into the challenges associated with fine-grained hand action recognition and shed light on potential avenues for future research, particularly in relation to pre-training strategy, model design, and domain generalization.
The main contributions of this paper are three-folds: (1) We introduce FHA-Kitchens, a novel dataset for fine-grained hand action recognition, encompassing 2,377 video clips and 30,047 images with high-quality annotations; (2) We propose to employ triplets to represent action in each hand interaction region, while also considering the ``active-passive'' relationships between interacting objects and their contact regions, resulting in 878 action triplet classes covering 131 action verbs and 384 object classes; (3) Based on FHA-Kitchens, we study several challenging yet unexplored questions in this field by benchmarking representative action recognition and detection models on the SL-D, SL-AR, and DG tracks. The obtained compelling empirical evidence highlights the challenges inherent in fine-grained hand action recognition, while also illuminating avenues for future research.
\section{Related Work}
\subsection{Action Recognition Datasets}
Pioneering action recognition datasets, such as KTH~\cite{35KTH} and Weizmann~\cite{4Weizmann}, have played a pivotal role in the advancement of this field, inspiring subsequent endeavors in constructing more challenging datasets, such as UCF101~\cite{3ucf101}, Kinetics~\cite{6kinetic400,7kinetics600,8kinetic700}, ActivityNet~\cite{5activitynet}, FineGym~\cite{9finegym}, and others~\cite{12moments,13,15,14,16}. While these datasets encompass various domains, such as sports, daily life, and work-related actions, they primarily focus on full-body actions, lacking fine-grained action information from specific body parts. Datasets like MPII Cooking Activities~\cite{2MPII} and EPIC-KITCHENS~\cite{1epic} fill this gap. They refine the action verb part and consider interaction objects, but they fail to address the localization of interaction regions or the relationships between interacting objects. Furthermore, they only represent actions generated by one of the interaction regions. However, due to the complexity and diversity of hand actions, focusing solely on simple hand interactions and interacting objects is insufficient. To mitigate this issue, our FHA-Kitchens dataset sets itself apart from existing datasets in three key aspects: (1) \emph{Action Interaction Regions}: We meticulously annotate hand interaction regions and their corresponding objects using bounding boxes; (2) \emph{Action Representation}: We categorize hand actions into three classes based on sub-interaction regions and employ a triplet to express each sub-interaction region action, thereby expanding the dimensionality to 9; and (3) \emph{Interacting Objects}: In contrast to previous datasets that solely considered the active force provider, we focus on both the active and passive relationships between interacting objects and capture their contact areas.
\begin{table*}[!htb]
\caption{Comparison of relevant datasets. AR: Action Recognition. ARL: Action Region Localization. HIA: Hand Interaction Region Detection. OD: Object Detection. ACat.: Action Category. OCat.: Object Category. Dim: Action Dimension. Box: Box Annotation of Action Region.}
\label{datasetcomparison}
\centering
\begin{threeparttable}
\scalebox{0.65}{
\begin{tabular}{cp{0.5cm}<{\centering}p{0.3cm}<{\centering}p{1.1cm}<{\centering}p{1.43cm}<{\centering}p{0.55cm}<{\centering}p{1.5cm}<{\centering}p{0.68cm}<{\centering}cp{1.2cm}<{\centering}p{1cm}<{\centering}p{2.5cm}<{\centering}}
\toprule
Dataset & Year & {Ego} & \#Clip & {Ave.Len} & {\#Frame} & \#ACat. & \#Verb & \#OCat. & Dim & Box & Task\\
\midrule
{\bf Human full-body dataset} &&&&&&&&&&& \\ \midrule
UCF101~\cite{3ucf101} & 2012 & $\times$ & 13.3K & $\sim$6\emph{s} & - & 101 & - & - & 1 & $\times$ & AR \\
ActivityNet~\cite{5activitynet} & 2015 & $\times$ & 28K & [5,10]\emph{m} & - & 203 & - & - & 1 & $\times$ & AR \\
Kinetics400~\cite{6kinetic400} & 2017 & $\times$ & 306K & 10\emph{s} & - & 400 & 359 & 318 & 2 & $\times$ & AR \\
Kinetics600~\cite{7kinetics600} & 2018 & $\times$ & 496K & 10\emph{s} & - & 600 & 550 & 502 & 2 & $\times$ & AR \\
Kinetics700~\cite{8kinetic700} & 2019 & $\times$ & 650K & 10\emph{s} & - & 700 & 644 & 591 & 2 & $\times$ & AR \\
AVA~\cite{10ava} & 2018 & $\times$ & 430 & 15\emph{m} & - & 80 & 80 & 0 & 3 & \checkmark & AR,ARL \\
AVA-kinetics~\cite{11ava-kinetics} & 2020 & $\times$ & 230K & 15\emph{m},10\emph{s} & - & 80 & 80 & 0 & 3 & \checkmark & AR,ARL \\
FineGym~\cite{9finegym} & 2020 & $\times$ & 32k:303 & 10\emph{m} & - & 530 & 530 & 0 & 3 & $\times$ & AR \\
\midrule
{\bf Hand dataset} & ~ & ~ & ~ & ~ & ~ & ~ & ~ & ~ & ~ & ~ & ~ \\
\midrule
MPII cooking~\cite{2MPII} & 2012 & $\times$ & 5,609:44 & 15\emph{m} & 881K & 65 & 65 & 0 & 1 & $\times$ & AR \\
EPIC-KITCHENS~\cite{1epic} & 2018 & \checkmark & 39.6K:432 & 3.7$\pm$5.6\emph{s} & 11.5M & 149 & 125 & 323 & 2 & $\times$ & AR,OD \\
{{\bf FHA-Kitchens}} & {{\bf 2023}} & {{\bf \checkmark}} &{ {\bf 2377:30}} & {{\bf 3\emph{m}}} & {{\bf 30,047}} & {{\bf 878}} & {{\bf 131}} & {{\bf 384}} & {{\bf 9}} & {{\bf \checkmark}} & {\bf AR,ARL,HIA,OD} \\
\bottomrule
\end{tabular} }
\end{threeparttable}
\end{table*}
\subsection{Action Detection Datasets}
Compared to action recognition datasets, there are fewer datasets available for action detection~\cite{10ava,11ava-kinetics}. This is due to the intricate and diverse nature of hand actions, as well as the relatively smaller size of interacting objects, which introduce challenges such as occlusion and truncation. The AVA dataset~\cite{10ava} focuses on action localization, providing bounding box annotations for each instance object and representing action information in three dimensions. However, this dataset primarily emphasizes full-body actions, where the action verbs are coarse-grained, and it does not account for the interacting objects involved in the actions. In our dataset, we surpass these limitations by providing precise bounding box annotations for each hand interaction region. Moreover, we refine the expression of action verbs and incorporate information about the interacting objects within each interaction region, thereby enhancing the granularity and contextual information of hand actions. A comprehensive comparison between our FHA-kitchens and existing datasets is presented in Table~\ref{datasetcomparison}.
\subsection{Action Recognition Methods}
On the one hand, existing action recognition methods can be categorized into coarse-grained~\cite{30,31} and fine-grained~\cite{26,27,28,29}. Some approaches~\cite{27,29} address fine-grained hand actions using the EPIC-KITCHENS and MPII Cooking Activities datasets, from first-person and third-person perspectives, respectively. Another method~\cite{28} focuses on refining human actions in sports scenarios using the FineGym dataset~\cite{9finegym}. However, these methods heavily rely on the specific dataset used and offer tailored solutions to the associated challenges. On the other hand, action recognition methods can be broadly summarized into two pipelines. The first pipeline employs a 2D CNN~\cite{32,33,36,40} to learn frame-level semantics and then aggregate them temporally using 1D modules. For example, TSN~\cite{19TSN} divides an action instance into multiple segments, represents it with a sparse sampling scheme, and applies average pooling to fuse predictions from each frame. TRN~\cite{34} and TSM~\cite{37tsm} replace pooling with temporal reasoning and shift modules, respectively. The second pipeline directly utilizes a 3D CNN~\cite{6kinetic400,21non-local,22slowfast,23swin,38,41} to capture spatial-temporal semantics, such as I3D~\cite{6kinetic400}, SlowFast~\cite{22slowfast}, and Video Swin Transformer~\cite{23swin}. In addition to action recognition, other video understanding tasks have also garnered research attention, including action detection and localization~\cite{25LFB,39,42,43}, action segmentation~\cite{44,45}, and action generation~\cite{46,47}.
\section{Dataset}
This section introduces the FHA-Kitchens dataset (Figure~\ref{openfig}). Specifically, we describe the data collection and annotation pipeline and present statistics regarding different aspects of the dataset.
\subsection{Data Collection and Organization}
{\bf Data Collection.}
Our dataset is derived from the large-scale action dataset Kinetics 700\_2020~\cite{700_2020}, which comprises over 700 YouTube video clips. However, as the Kinetics 700 dataset primarily focuses on person-level actions, most of the videos capture full-body actions rather than specific body parts. To narrow our focus to hand actions, we performed filtering and processing operations on the original videos, including the following three steps. (1) First, we observed that kitchen scenes often featured hand actions, with video content prominently showcasing human hand parts. Therefore, we sought out and extracted relevant videos that were set against a kitchen backdrop. (2)Then, to ensure the quality of the dataset, we selectively chose videos with higher resolutions. Specifically, 87\% of the videos were recorded at 1,280 $\times$ 720 resolution, while another 13\% had a shorter side of 480. Additionally, 67\% of the videos were captured at 30 frames per second (fps), and another 33\% were recorded at 24$\sim$25 fps. (3) Subsequently, we imposed a duration constraint on the videos, ranging from 30 seconds to 5 minutes, to exclude excessively long-duration videos. This constraint aimed to maintain a balanced distribution within the sample space. Finally, we collected a total of 30 videos, amounting to 84.22 minutes of footage, encompassing 8 distinct types of dishes.
{\bf Data Organization.}
The collected video data was reorganized and cleaned to align with our annotation criteria (Section~\ref{3.2}). First, we split the collected video data into individual frames, as our annotated units are frames. Subsequently, we conducted further cleaning of the frames by excluding those that did not depict hands or exhibited meaningless hand actions. This cleaning process took into consideration factors such as occlusion, frame quality (\textit{i.e.}, without significant blur, subtitles, and logos), meaningful hand actions, and frame continuity. As a result, we obtained a total of 30,047 high-quality candidate video frames containing diverse hand actions for our FHA-Kitchens dataset. Compared to the initial collection, 113,436 frames were discarded during the cleaning process.
\subsection{Data Annotation} \label{3.2}
To ensure high-quality annotation of hand actions for each frame, we recruited 10 voluntary annotators, whose responsibility was to annotate fine-grained action triplet classes and bounding boxes for each hand interaction region. In order to enhance annotation efficiency, we implemented a parallel annotation approach. The annotation of action triplets was carried out on the \href{https://www.mturk.com/}{Amazon Mechanical Turk} platform, while the bounding box annotation was facilitated using the \href{https://github.com/open-mmlab/labelbee-client}{LabelBee} tool. To ensure the annotation quality, three rounds of cross-checking and corrections were conducted. Specifically, the annotation content and criteria can be summarized as follows.
{\bf Bounding Box Annotation of Hand Action:} We annotated the bounding boxes for both interaction regions and interaction objects. (1) \emph{Interaction Regions}: We divided the hand's interaction region into three sub-interaction regions: left hand-object (L-O), right hand-object (R-O), and object-object (O-O) interaction regions, respectively. The L-O interaction region involves direct contact between the left hand and an object to perform an action (Figure~\ref{openfig} bottom left). Similarly, the R-O interaction region involves direct contact between the right hand and an object (Figure~\ref{openfig} bottom middle). The O-O interaction region indicates the contact between two objects (Figure~\ref{openfig} bottom right). (2) \textit{Interaction Objects}: To better understand interaction actions, we also annotated the objects within each sub-interaction region using bounding boxes. For the L-O interaction region, we annotated objects directly touched by the left hand. Similarly, for the R-O interaction region, we annotated objects directly touched by the right hand. In the O-O interaction region, we annotated objects interact with each other in the context of a specific hand action (\textit{i.e.}, \textit{utility knife} and \textit{carrot}). Additionally, we also considered the \textit{active-passive} relationship between objects, including the \textit{active force provider} (\textit{i.e.}, \textit{utility knife}) and \textit{passive force receiver} (\textit{i.e.}, \textit{carrot}), and annotate them in order in the triplet. Finally, we annotated a total of 198,839 bounding boxes over 9 types, including 49,746 hand boxes, 66,402 interaction region boxes, and 82,691 interaction object boxes. Compared to existing datasets~\cite{1epic}, we added an average of 5 additional annotation types per frame.
{\bf Hand Action Triplet Annotation:} We annotated fine-grained actions for each interaction region. Unlike existing datasets, we represented each action in a triplet format: \emph{<subject, action verb, object>}. The subject refers to the active force provider, the object refers to the passive force receiver, and the action verb describes the specific fine-grained hand action within the hand interaction region.
(1) \emph{Subject $\&$ Object}: In the left or right hand-object interaction regions, we labeled the subject as the corresponding hand and used fine-grained sub-categories names for the interacting objects. To define the object noun, we referred to the EPIC-KITCHENS~\cite{1epic} dataset. Furthermore, to enrich the description of each action, we included the contact areas of both the subject and object within the interaction region. For example, in the L-O interaction region shown in Figure~\ref{openfig} bottom left, we labeled the subject as ``\emph{hand\_left}'' and the object as ``\emph{carrot\_end}'' based on their respective contact areas within the current interaction region.
(2) \emph{Action Verb}: We used fine-grained verbs in the annotated action triplets and constructed the verb vocabulary by sourcing from EPIC-KITCHENS~\cite{1epic}, AVA~\cite{10ava}, and Kinetics 700~\cite{8kinetic700}. More details can be found in the supplement.
{\bf Object Segment Annotation:} To enrich our FHA-Kitchens, we utilized the state-of-the-art SAM model~\cite{20segment} to annotate object masks in all video frames, which can be used for action segmentation.
\begin{figure}[!t]
\centering
\begin{minipage}[!htb]{0.48\textwidth}
\vspace{-0.25cm}
\centering
\includegraphics[width=6cm]{cakefig-actionverbs.pdf}
\caption{An overview of the action verbs and their parent action categories in FHA-Kitchens.}
\label{cakefig-actionverbs}
\end{minipage}
\hspace{0.25cm}
\begin{minipage}[!htb]{0.48\textwidth}
\centering
\includegraphics[width=6.5cm,height=1.6cm]{actionverbdistribution.pdf}
\caption{The distribution of instances per action verb category (the outer ring of the circle in Figure~\ref{cakefig-actionverbs}) in the FHA-Kitchens dataset.} \label{actionverbdistribution}
\vspace{0.25cm}
\includegraphics[width=6.5cm,height=2cm]{objectnoundistribution.pdf}
\caption{The distribution of instances per object noun category from 17 super-categories in the FHA-Kitchens dataset.}
\label{objectnoundistribution}
\end{minipage}
\end{figure}
\subsection{Statistics of the FHA-Kichens Dataset}
{\bf Overview of FHA-Kitchens.} As summarized in Table~\ref{datasetcomparison}, we annotated action triplets for 30,047 frames from 2,377 clips, resulting in 878 action triplet categories, 131 action verbs and 384 interacting object nouns. Despite having fewer clips compared to the action category-rich Kinetics 700 dataset~\cite{8kinetic700}, our dataset has a larger number of action categories. Compared to the data's original annotations in Kinetics 700\_2020~\cite{700_2020}, our dataset expanded the action labels by 7 dimensions, increased the number of action categories by 52 times, and introduced 123 new action verbs. Furthermore, we provide bounding boxes for hand action regions (\textit{i.e.}, 66,402 interaction region boxes). This expansion significantly enhances the diversity of hand actions, provides valuable region-level contextual information for each action, and holds the potential for facilitating future research for a wider range of video understanding tasks. The FHA-Kitchens dataset was then randomly divided into disjoint train, validation, and test sets, with a video clip-based ratio of 7:1:2.
{\bf Annotation Statistics.}
Our annotation primarily focuses on hand interaction objects and their corresponding interaction actions, resulting in a diverse array of verbs and nouns. Following the fine-grained annotation principles~\cite{1epic}, we ensured minimal semantic overlap among action verb-noun categories, rendering them suitable for multi-category action recognition and detection. (1) Verbs: The annotated dataset comprises 131 action verbs that have been grouped into 43 parent verb categories (Figure~\ref{cakefig-actionverbs} and Figure~\ref{actionverbdistribution}). The three most prevalent parent verb categories, based on the count of sub-action verbs, are \textit{Cut}, \textit{Hold}, and \textit{Take}, representing the most frequently occurring hand actions in human interactions. Figure~\ref{actionverbdistribution} visually depicts the distribution of all verb categories within FHA-Kitchens, ensuring the presence of at least one instance for each verb category. (2) Nouns: In our annotation process, we identified a total of 384 interaction object noun categories that are associated with actions, categorized into 17 super-categories. Figure~\ref{objectnoundistribution} shows the distribution of noun categories based on their affiliations with super-categories. Notably, the super-category ``vegetables \& plants'' exhibits the highest number of sub-categories, followed by ``kitchenware'', which aligns with typical kitchen scenes.
{\bf Long-tail Property.}
The distribution of instances per action triplet category in FHA-Kitchens, as depicted in Figure~\ref{actiontripletdistribution}, depicts a long-tail property. This distribution reflects the frequency of hand interactions in real-world kitchen scenes, taking into account the varying commonness or rarity of specific hand actions. For instance, the action triplet ``<hand\_right, hold-in, utility-knife\_handle>'' consists of 9,887 instances, which is nine times more prevalent than the ``<hand\_left, hold-in, utility-knife\_handle>'' triplet. This long-tail characteristic of the distribution renders FHA-Kitchens a challenging benchmark for hand action recognition, making it suitable for investigating few-shot learning and out-of-distribution generalization in action recognition as well.
\begin{figure}[!htb]
\setlength{\belowcaptionskip}{-0.2cm}
\centering
\includegraphics[width=1\linewidth]{actiontripletdistribution.pdf}
\caption{The distribution of instances per action triplet category in the FHA-Kitchens dataset.}
\label{actiontripletdistribution}
\end{figure}
\section{Experiment} \label{4}
\subsection{Implementation Details}\label{4.1}
We benchmark several representative action recognition methods~\cite{22slowfast,23swin,19TSN} and detection methods~\cite{24faster} with different backbone networks on the proposed FHA-Kitchens dataset based on the MMAction2~\cite{51mmaction2} and MMDetection~\cite{52mmdetection} codebases. All models on the SL-D, SL-AR, and DG tracks are trained and tested using 4 NVIDIA GeForce RTX 3090 GPUs. For the SL-D and DG tracks, we employ the mean average precision (mAP)~\cite{53coco} as the primary evaluation metric, while for the SL-AR track, Top-1 accuracy and Top-5 accuracy are adopted.
\subsection{SL-D Track: Supervised Learning for Hand Interaction Region and Object Detection}
{\bf Settings.} The SL-D track aims to evaluate the performance of different detection models on hand interaction regions and objects. We employed the representative Faster-RCNN~\cite{24faster} model with different backbones. Specifically, we pre-trained the model on the MS COCO~\cite{53coco} object detection dataset and fine-tuned it on FHA-Kitchens for the tasks of hand interaction region detection and interaction object detection, respectively. We trained the model using the SGD optimizer for 100 epochs with an initial learning rate of 0.02 and a batch size of 1. The weight decay is set to 0.0001.
{\bf Results on the SL-D Track.}
The detection results are summarized in Table~\ref{sldtrack}. As can be seen, detecting interaction objects is more difficult than detecting hand interaction regions, validating the challenge posed by the diversity and fine-grained object categories in our dataset. Moreover, using a stronger backbone leads to slightly better detection results. It is noteworthy that, unlike existing action datasets, our FHA-Kitchens dataset provides abundant annotations of hand interaction regions, making it possible to investigate the model's ability to localize interaction regions and interpret hand actions in a more informative way, which is crucial for embodied intelligence research~\cite{55behavior,56embodied}. The visualization of the detection results can be found in the supplementary material.
\begin{table*}[!htb]
\begin{minipage}{\textwidth}
\begin{minipage}[t]{0.34\textwidth}
\makeatletter\def\@captype{table}
\makeatother\caption{Detection results (mAP) of hand interaction regions and objects using Faster-RCNN~\cite{24faster} with different backbones on the validation set of the SL-D track.}
\label{sldtrack}
\centering
\begin{tabular}{cc}
\toprule
\multicolumn{2}{c}{Interaction Regions} \\
\midrule
ResNet50 & ResNet101 \\
\hline
65.20 & 66.10 \\
\bottomrule
\toprule
\multicolumn{2}{c}{Interaction Objects} \\
\midrule
ResNet50 & ResNet101 \\
\hline
40.80 & 41.90 \\
\bottomrule
\end{tabular}
\end{minipage}
\hspace{0.25cm}
\begin{minipage}[t]{0.64\textwidth}
\makeatletter\def\@captype{table}
\makeatother\caption{Classification results (Top-1 and Top-5 accuracy) of fine-grained hand actions using different methods on the validation set of the SL-AR track. w/ Pre-train: using Kinetics 400 pre-trained weights. w/o Pre-train: training from scratch.}
\label{slartrack}
\footnotesize
\centering
\begin{tabular}{cccccc}
\toprule
\multirow{2}{*}{Model} & \multirow{2}{*}{Backbone} & \multicolumn{2}{c}{w/ Pre-train} & \multicolumn{2}{c}{w/o Pre-train}\\
\cmidrule(r){3-6}
\multirow{2}{*}{~} & \multirow{2}{*}{~} & Top-1 & Top-5 & Top-1 & Top-5 \\
\midrule
\multirow{2}{*}{TSN~\cite{19TSN}} & ResNet50 & 30.37 & 74.26 & 29.11 & 73.84 \\
\multirow{2}{*}{~} & ResNet101 & 30.80 & 73.42 & 30.38 & 74.26 \\
\cmidrule(r){1-6}
\multirow{2}{*}{SlowFast~\cite{22slowfast}} & ResNet50 & 33.33 & 70.46 & 27.85 & 68.35 \\
\multirow{2}{*}{~} & ResNet101 & 36.71 & 67.93 & 31.22 & 69.62\\
\cmidrule(r){1-6}
Swin~\cite{23swin} & Swin-B & 37.13 & 70.89 & 34.18 & 66.67 \\
\bottomrule
\end{tabular}
\end{minipage}
\end{minipage}
\end{table*}
\subsection{SL-AR Track: Supervised Learning for Fine-grained Hand Action Recognition}
{\bf Settings.} The SL-AR track primarily evaluates the performance of different action recognition models on fine-grained hand actions. We adopt the representative TSN~\cite{19TSN} and Slowfast~\cite{22slowfast} with the ResNet50 and ResNet101 backbones and Video Swin with the Swin-B backbone. We train these models on the FHA-Kitchens dataset using two settings: (1) {\bf Pre-training on Kinetics 400~\cite{6kinetic400}}, where we initialize the backbone with Kinetics 400 pre-trained weights and fine-tune the entire model on the FHA-Kitchens training set; and (2) {\bf Training from scratch on FHA-Kitchens}, where we randomly initialize the model weights and directly train them on FHA-Kitchens. For different models, we use the recommended optimization strategy and batch size and train them for 300 epochs.
{\bf Results on the SL-AR Track.} The results in Table~\ref{slartrack} show that the performance trends of all action recognition methods on FHA-Kitchens are similar to their performance on Kinetics 400 (Please refer to the supplementary material). However, all the models achieve much worse accuracy on our FHA-Kitchens than the coarse-grained Kinetics 400, validating the challenging nature of the fine-grained hand action recognition on FHA-Kitchens. Besides, the utilization of pre-trained weights has proven to be beneficial, resulting in improved accuracy compared to training models from scratch. This finding suggests that despite the existence of a domain gap between coarse-grained and fine-grained actions, pre-training remains an effective strategy for addressing the challenges inherent in FHA-Kitchens, which has a larger number of action categories and relatively limited training data.
Note that the main difficulty of FHA-Kitchens lies in fine-grained action recognition rather than localization. Figure~\ref{detectionvis} shows some examples of high-scoring localization but false recognition instances, demonstrating that the difficulty of recognition lies in fine-grained details.
\begin{figure}[!htb]
\centering
\vspace{-2mm}
\includegraphics[width=1\linewidth]{detectionvis.pdf}
\caption{Some examples of high-scoring localization but false recognition instances. The recognition model often struggles to discriminate fine-grained details.}
\label{detectionvis}
\vspace{-2mm}
\end{figure}
\subsection{DG Track: Intra- and Inter-class Domain Generalization for Interaction Region Detection}
\subsubsection{Intra-class Domain Generalization}
{\bf Settings.} We conducted intra-class DG experiments using the three most prevalent parent action categories, \textit{i.e.}, \textit{Cut}, \textit{Hold}, and \textit{Take}. For each parent action category, we selected the most prevalent sub-categories and adopted the cross-validation protocol, \textit{i.e.}, randomly choosing one sub-category as the test set while using all other sub-categories for training. Following the SL-D track, we selected the Faster RCNN~\cite{24faster} model with the ResNet50 backbone as the default model, which is pre-trained on the MS COCO~\cite{53coco} object detection dataset.
{\bf Results on the Intra-class DG Track.}
The results on \textit{Cut} are summarized in Table \ref{dgtrack-cut}, while the results on \textit{Hold} and \textit{Take} are shown in the supplementary material due to the page limit. The performance of all four detection models remains stable for the sub-categories seen during training but deteriorates for unseen sub-categories, as evidenced by the diagonal scores, which exhibit a minimum drop of 15 mAP. This finding suggests that there is still potential for enhancing the models' generalization abilities, \textit{e.g.}, by exploring the domain generalization or unsupervised domain adaptation techniques.
\begin{table*}[!t]
\begin{minipage}{\textwidth}
\begin{minipage}[t]{0.57\textwidth}
\makeatletter\def\@captype{table}
\makeatother\caption{Intra-class DG test results of Faster RCNN~\cite{24faster} with the ResNet50 backbone on the ``{\bf Cut}'' Setting. $\Delta_i = \frac{1}{3}\sum_{j, j\neq i}{ji} - ii, \Delta^{*}_i=\frac{1}{3}\sum_{j, j\neq i}{ij} - ii, i=0,1,2,3$.}
\label{dgtrack-cut}
\footnotesize
\centering
\resizebox{\linewidth}{!}{\begin{tabular}{c|cccc|c}
\toprule
& \multicolumn{4}{c}{Test (mAP)} & \\
\cmidrule(r){2-5}
Train & cut-slice & cut-off & cut-down & cut-dice & $\Delta^{*}$\\
\midrule
w/o cut-slice & \textbf{33.30} & 65.00 & 56.00 & 60.90 & 27.33\\
w/o cut-off & 57.10 & \textbf{48.00} & 54.80 & 62.80 & 10.23\\
w/o cut-down & 57.30 & 64.40 & \textbf{41.30} & 63.50 & 20.43\\
w/o cut-dice & 57.50 & 64.90 & 58.70 & \textbf{41.10} & 19.27 \\
\midrule
$\Delta$ & 24.00 & 16.77 & 15.20 & 21.30 & \\
\bottomrule
\end{tabular}}
\end{minipage}
\hspace{0.25cm}
\begin{minipage}[t]{0.40\textwidth}
\makeatletter\def\@captype{table}
\makeatother\caption{Inter-class DG test results. $\Delta_i=ii-\frac{1}{2}\sum_{j,j\neq i}{ji}, \Delta^{*}_i=ii-\frac{1}{2}\sum_{j, j\neq i}{ij}, i=0,1,2$.}
\label{dgtrack-hold}
% \footnotesize
\centering
\resizebox{0.9\linewidth}{!}{\begin{tabular}{c|ccc|c}
\toprule
& \multicolumn{3}{c}{Test (mAP)} & \\
\cmidrule(r){2-4}
Train & Cut & Hold & Take & $\Delta^{*}$\\
\midrule
Cut & \textbf{37.40} & 29.50 & 29.20 & 8.05\\
Hold & 48.70 & \textbf{52.30} & 41.80 & 7.05\\
Take & 14.00 & 13.20 & \textbf{41.20} & 27.60\\
\midrule
$\Delta$ & 6.05 & 30.95 & 5.70\\
\bottomrule
\end{tabular}}
\end{minipage}
\end{minipage}
\vspace{-4mm}
\end{table*}
\subsubsection{Inter-class Domain Generalization}
{\bf Settings.} We chose the three most prevalent parent action categories \textit{Cut}, \textit{Hold}, and \textit{Take}, and adopted the cross-validation protocol, \textit{i.e.}, randomly choosing one parent category for training and using the other parent categories for testing. Other settings follow those in the intra-class DG track.
{\bf Results on the Inter-class DG Track.} The results are listed in Table \ref{dgtrack-hold}. Similar to the results in the intra-class DG track, the detection models perform well on the seen categories while deteriorating on the unseen categories. Nevertheless, it is interesting to find that the performance gap ($\bigtriangleup_0=6.05$ and $\bigtriangleup^{*}_0=8.05$) between \textit{Cut} and others are smaller than those in the intra-class DG track, implying that there is likely a large intra-class variance, and the detection model is prone to overfitting the seen categories, particularly when the volume of training data is smaller (there are 7,463 training frames in \textit{Hold} while only 1,680 in \textit{Take}).
\section{Discussion} \label{diss}
Through the SL-D, SL-AR, and DG tracks experiments, we have made a preliminary investigation of some unexplored research questions regarding fine-grained action detection and recognition. The obtained compelling empirical evidence not only highlights the inherent challenges associated with fine-grained hand action analysis but also reveals promising avenues for future research, demonstrating the value of the proposed FHA-Kitchens dataset. Firstly, there is promising potential for further enhancing performance by delving deeper into the fine-grained categories, leveraging insightful ideas from the fields of fine-grained image analysis~\cite{57fg_image} and action recognition~\cite{27}. Secondly, considering the long-tail distribution of data in FHA-Kitchens, exploring strategies to balance the representation of both head and tail categories in the context of hand action detection and recognition warrants further investigation. Thirdly, while this paper focuses solely on closed-set detection and recognition tasks, exploring open-set settings holds both research and practical significance, which is supported by our FHA-Kitchens dataset. Lastly, the availability of mask annotations for all instances of hand interaction regions enables the study of action segmentation tasks, which can provide pixel-level interpretations of interaction objects.
\section{Conclusion}
In this paper, we build a novel dataset of fine-grained hand actions in kitchen scenes, i.e., FHA-Kitchens. The dataset offers a rich diversity in terms of viewpoints, occlusions, granularity, and action categories, providing new possibilities for fine-grained action research. We benchmark representative action recognition and detection methods on FHA-Kitchens and obtain compelling empirical evidence to understand the representation ability of different models, the impact of pre-training, the benefit of using diverse fine-grained hand actions for training, as well as intra- and inter-class domain generalization. We anticipate that FHA-Kitchens would pave the way for future research in this field.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \section*{References}
% \nocite{*} % to test all bib entrys
% \bibliographystyle{unsrt}
% \bibliography{arxiv}
\begin{thebibliography}{10}
\bibitem{55smarthome}
Mussab Alaa, Aws~Alaa Zaidan, Bilal~Bahaa Zaidan, Mohammed Talal, and Miss
Laiha~Mat Kiah.
\newblock A review of smart home applications based on internet of things.
\newblock {\em Journal of Network and Computer Applications}, 97:48--65, 2017.
\bibitem{4Weizmann}
Moshe Blank, Lena Gorelick, Eli Shechtman, Michal Irani, and Ronen Basri.
\newblock Actions as space-time shapes.
\newblock In {\em Tenth IEEE International Conference on Computer Vision Volume
1}, volume~2, pages 1395--1402. IEEE, 2005.
\bibitem{7kinetics600}
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew
Zisserman.
\newblock A short note about kinetics-600.
\newblock {\em arXiv preprint arXiv:1808.01340}, 2018.
\bibitem{8kinetic700}
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman.
\newblock A short note on the kinetics-700 human action dataset.
\newblock {\em arXiv preprint arXiv:1907.06987}, 2019.
\bibitem{6kinetic400}
Joao Carreira and Andrew Zisserman.
\newblock Quo vadis, action recognition? a new model and the kinetics dataset.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 6299--6308, 2017.
\bibitem{52mmdetection}
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu~Xiong, Xiaoxiao Li,
Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng,
Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu,
Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen~Change Loy, and
Dahua Lin.
\newblock {MMDetection}: Open mmlab detection toolbox and benchmark.
\newblock {\em arXiv preprint arXiv:1906.07155}, 2019.
\bibitem{51mmaction2}
MMAction2 Contributors.
\newblock Openmmlab's next generation video understanding toolbox and
benchmark.
\newblock \url{https://github.com/open-mmlab/mmaction2}, 2020.
\bibitem{30}
Navneet Dalal and Bill Triggs.
\newblock Histograms of oriented gradients for human detection.
\newblock In {\em 2005 IEEE Computer Society Conference on Computer Vision and
Pattern Recognition}, volume~1, pages 886--893. Ieee, 2005.
\bibitem{31}
Navneet Dalal, Bill Triggs, and Cordelia Schmid.
\newblock Human detection using oriented histograms of flow and appearance.
\newblock In {\em Computer Vision--European Conference on Computer Vision 2006:
9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006.
Proceedings, Part II 9}, pages 428--441. Springer, 2006.
\bibitem{1epic}
Dima Damen, Hazel Doughty, Giovanni~Maria Farinella, Sanja Fidler, Antonino
Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett,
Will Price, et~al.
\newblock Scaling egocentric vision: The epic-kitchens dataset.
\newblock In {\em Proceedings of the European Conference on Computer Vision},
pages 720--736, 2018.
\bibitem{41}
Ali Diba, Mohsen Fayyaz, Vivek Sharma, Amir~Hossein Karami, Mohammad~Mahdi
Arzani, Rahman Yousefzadeh, and Luc Van~Gool.
\newblock Temporal 3d convnets: New architecture and transfer learning for
video classification.
\newblock {\em arXiv preprint arXiv:1711.08200}, 2017.
\bibitem{45}
Li~Ding and Chenliang Xu.
\newblock Weakly-supervised action segmentation with iterative soft boundary
assignment.
\newblock In {\em Proceedings of the IEEE conference on Computer Vision and
Pattern Recognition}, pages 6508--6516, 2018.
\bibitem{36}
Jeffrey Donahue, Lisa Anne~Hendricks, Sergio Guadarrama, Marcus Rohrbach,
Subhashini Venugopalan, Kate Saenko, and Trevor Darrell.
\newblock Long-term recurrent convolutional networks for visual recognition and
description.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 2625--2634, 2015.
\bibitem{22slowfast}
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.
\newblock Slowfast networks for video recognition.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 6202--6211, 2019.
\bibitem{40}
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.
\newblock Convolutional two-stream network fusion for video action recognition.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 1933--1941, 2016.
\bibitem{39}
Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman.
\newblock Video action transformer network.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 244--253, 2019.
\bibitem{10ava}
Chunhui Gu, Chen Sun, David~A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing
Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul
Sukthankar, et~al.
\newblock Ava: A video dataset of spatio-temporally localized atomic visual
actions.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 6047--6056, 2018.
\bibitem{56embodied}
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li~Fei-Fei.
\newblock Embodied intelligence via learning and evolution.
\newblock {\em Nature communications}, 12(1):5721, 2021.
\bibitem{5activitynet}
Fabian~Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan~Carlos Niebles.
\newblock Activitynet: A large-scale video benchmark for human activity
understanding.
\newblock In {\em 2015 IEEE Conference on Computer Vision and Pattern
Recognition}, pages 961--970. IEEE, 2015.
\bibitem{13}
Fabian~Caba Heilbron, Joon-Young Lee, Hailin Jin, and Bernard Ghanem.
\newblock What do i annotate next? an empirical study of active learning for
action localization.
\newblock In {\em Proceedings of the European Conference on Computer Vision},
pages 199--216, 2018.
\bibitem{28}
James Hong, Matthew Fisher, Micha{\"e}l Gharbi, and Kayvon Fatahalian.
\newblock Video pose distillation for few-shot, fine-grained sports action
recognition.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 9254--9263, 2021.
\bibitem{49hand-object}
Hezhen Hu, Weilun Wang, Wengang Zhou, and Houqiang Li.
\newblock Hand-object interaction image generation.
\newblock {\em arXiv preprint arXiv:2211.15663}, 2022.
\bibitem{15}
Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael~J
Black.
\newblock Towards understanding action recognition.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision}, pages 3192--3199, 2013.
\bibitem{20segment}
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura
Gustafson, Tete Xiao, Spencer Whitehead, Alexander~C Berg, Wan-Yen Lo, et~al.
\newblock Segment anything.
\newblock {\em arXiv preprint arXiv:2304.02643}, 2023.
\bibitem{44}
Colin Lea, Michael~D Flynn, Rene Vidal, Austin Reiter, and Gregory~D Hager.
\newblock Temporal convolutional networks for action segmentation and
detection.
\newblock In {\em proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 156--165, 2017.
\bibitem{11ava-kinetics}
Ang Li, Meghana Thotakuri, David~A Ross, Jo{\~a}o Carreira, Alexander
Vostrikov, and Andrew Zisserman.
\newblock The ava-kinetics localized human actions video dataset.
\newblock {\em arXiv preprint arXiv:2005.00214}, 2020.
\bibitem{55behavior}
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto
Mart{\'\i}n-Mart{\'\i}n, Chen Wang, Gabrael Levine, Michael Lingelbach,
Jiankai Sun, et~al.
\newblock Behavior-1k: A benchmark for embodied ai with 1,000 everyday
activities and realistic simulation.
\newblock In {\em Conference on Robot Learning}, pages 80--93. Proceedings of
Machine Learning Research, 2023.
\bibitem{46}
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang.
\newblock Flow-grounded spatial-temporal video prediction from still images.
\newblock In {\em Proceedings of the European Conference on Computer Vision},
pages 600--615, 2018.
\bibitem{37tsm}
Ji~Lin, Chuang Gan, and Song Han.
\newblock Tsm: Temporal shift module for efficient video understanding.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 7083--7093, 2019.
\bibitem{53coco}
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Doll{\'a}r, and C~Lawrence Zitnick.
\newblock Microsoft coco: Common objects in context.
\newblock In {\em Computer Vision--European Conference on Computer Vision 2014:
13th European Conference, Zurich, Switzerland, September 6-12, 2014,
Proceedings, Part V 13}, pages 740--755. Springer, 2014.
\bibitem{29}
Fang Liu, Liang Zhao, Xiaochun Cheng, Qin Dai, Xiangbin Shi, and Jianzhong
Qiao.
\newblock Fine-grained action recognition by motion saliency and mid-level
patches.
\newblock {\em Applied Sciences}, 10(8):2811, 2020.
\bibitem{23swin}
Ze~Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.
\newblock Video swin transformer.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 3202--3211, 2022.
\bibitem{12moments}
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah~Adel
Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et~al.
\newblock Moments in time dataset: one million videos for event understanding.
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},
42(2):502--508, 2019.
\bibitem{27}
Jonathan Munro and Dima Damen.
\newblock Multi-modal domain adaptation for fine-grained action recognition.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 122--132, 2020.
\bibitem{26}
Bingbing Ni, Xiaokang Yang, and Shenghua Gao.
\newblock Progressively parsing interactional objects for fine grained action
detection.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 1020--1028, 2016.
\bibitem{54robothand}
G.~Palli, S.~Pirozzi, C.~Natale, G.~De~Maria, and C.~Melchiorri.
\newblock Mechatronic design of innovative robot hands: Integration and control
issues.
\newblock In {\em 2013 IEEE/ASME International Conference on Advanced
Intelligent Mechatronics}, pages 1755--1760, 2013.
\bibitem{24faster}
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
\newblock Faster r-cnn: Towards real-time object detection with region proposal
networks.
\newblock {\em Advances in Neural Information Processing Systems}, 28, 2015.
\bibitem{2MPII}
Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele.
\newblock A database for fine grained activity detection of cooking activities.
\newblock In {\em 2012 IEEE Conference on Computer Vision and Pattern
Recognition}, pages 1194--1201. IEEE, 2012.
\bibitem{35KTH}
Christian Schuldt, Ivan Laptev, and Barbara Caputo.
\newblock Recognizing human actions: a local svm approach.
\newblock In {\em Proceedings of the 17th International Conference on Pattern
Recognition, 2004. ICPR 2004.}, volume~3, pages 32--36. IEEE, 2004.
\bibitem{9finegym}
Dian Shao, Yue Zhao, Bo~Dai, and Dahua Lin.
\newblock Finegym: A hierarchical video dataset for fine-grained action
understanding.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 2616--2625, 2020.
\bibitem{14}
Gunnar~A Sigurdsson, G{\"u}l Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev,
and Abhinav Gupta.
\newblock Hollywood in homes: Crowdsourcing data collection for activity
understanding.
\newblock In {\em Computer Vision--European Conference on Computer Vision 2016:
14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016,
Proceedings, Part I 14}, pages 510--526. Springer, 2016.
\bibitem{32}
Karen Simonyan and Andrew Zisserman.
\newblock Two-stream convolutional networks for action recognition in videos.
\newblock {\em Advances in Neural Information Processing Systems}, 27, 2014.
\bibitem{700_2020}
Lucas Smaira, Jo{\~a}o Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew
Zisserman.
\newblock A short note on the kinetics-700-2020 human action dataset.
\newblock {\em arXiv preprint arXiv:2010.10864}, 2020.
\bibitem{3ucf101}
Khurram Soomro, Amir~Roshan Zamir, and Mubarak Shah.
\newblock Ucf101: A dataset of 101 human actions classes from videos in the
wild.
\newblock {\em arXiv preprint arXiv:1212.0402}, 2012.
\bibitem{47}
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Rahul Sukthankar, Kevin Murphy,
and Cordelia Schmid.
\newblock Relational action forecasting.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 273--283, 2019.
\bibitem{38}
Du~Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar
Paluri.
\newblock A closer look at spatiotemporal convolutions for action recognition.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 6450--6459, 2018.
\bibitem{19TSN}
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu~Qiao, Dahua Lin, Xiaoou Tang, and Luc
Van~Gool.
\newblock Temporal segment networks: Towards good practices for deep action
recognition.
\newblock In {\em European Conference on Computer Vision}, pages 20--36.
Springer, 2016.
\bibitem{33}
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu~Qiao, Dahua Lin, Xiaoou Tang, and Luc
Van~Gool.
\newblock Temporal segment networks for action recognition in videos.
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},
41(11):2740--2755, 2018.
\bibitem{21non-local}
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.
\newblock Non-local neural networks.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition}, pages 7794--7803, 2018.
\bibitem{57fg_image}
Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac~Aodha, Jianxin Wu, Yuxin Peng, Jinhui
Tang, Jian Yang, and Serge Belongie.
\newblock Fine-grained image analysis with deep learning: A survey.
\newblock {\em IEEE Transactions on Pattern Analysis and Machine Intelligence},
44(12):8927--8948, 2021.
\bibitem{25LFB}
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp
Krahenbuhl, and Ross Girshick.
\newblock Long-term feature banks for detailed video understanding.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 284--293, 2019.
\bibitem{42}
Huijuan Xu, Abir Das, and Kate Saenko.
\newblock R-c3d: Region convolutional 3d network for temporal activity
detection.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision}, pages 5783--5792, 2017.
\bibitem{48RCareWorld}
Ruolin Ye, Wenqiang Xu, Haoyuan Fu, Rajat~Kumar Jenamani, Vy~Nguyen, Cewu Lu,
Katherine Dimitropoulou, and Tapomayukh Bhattacharjee.
\newblock Rcare world: A human-centric simulation world for caregiving robots.
\newblock In {\em 2022 IEEE/RSJ International Conference on Intelligent Robots
and Systems}, pages 33--40. IEEE, 2022.
\bibitem{zhang2020empowering}
Jing Zhang and Dacheng Tao.
\newblock Empowering things with intelligence: a survey of the progress,
challenges, and opportunities in artificial intelligence of things.
\newblock {\em IEEE Internet of Things Journal}, 8(10):7789--7817, 2020.
\bibitem{16}
Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan.
\newblock Hacs: Human action clips and segments dataset for recognition and
temporal localization.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 8668--8678, 2019.
\bibitem{43}
Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin.
\newblock Temporal action detection with structured segment networks.
\newblock In {\em Proceedings of the IEEE International Conference on Computer
Vision}, pages 2914--2923, 2017.
\bibitem{34}
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba.
\newblock Temporal relational reasoning in videos.
\newblock In {\em Proceedings of the European Conference on Computer Vision},
pages 803--818, 2018.
\end{thebibliography}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\maketitle
\appendix
\setcounter{figure}{0}
\renewcommand{\thefigure}{S\arabic{figure}}
\setcounter{table}{0}
\renewcommand{\thetable}{S\arabic{table}}
\section{Appendix}
\subsection{More quantitative and qualitative results}
Table~\ref{arfullresults} presents the performance of different action recognition methods on the Kinetics 400~\cite{6kinetic400} dataset and the proposed FHA-Kitchens dataset, with and without pre-trained models. From the experimental results, it can be observed that the performance trends of all action recognition methods on FHA-Kitchens are similar to their performance on Kinetics 400~\cite{6kinetic400}, while the models perform much better on the coarse-grained actions of Kinetics 400. For the best-performing Swin~\cite{23swin} model, the top-1 accuracy on Kinetics 400 surpasses the top-1 accuracy on FHA-Kitchens by 43.44\%, which validates the challenges of fine-grained hand action recognition on FHA-Kitchens. Besides, the utilization of pre-trained weights has proven to be beneficial, resulting in improved accuracy compared to training models from scratch. This finding suggests that despite the existence of a domain gap between coarse-grained and fine-grained actions, pre-training remains an effective strategy for addressing the challenges inherent in FHA-Kitchens, which have a larger number of action categories and relatively limited training data.
\begin{table}[htbp]
\centering
\caption{Classification results (Top-1 and Top-5 accuracy) of fine-grained hand actions using different methods on the validation set of the SL-AR track. w/ Pre-train (IN1k): using ImageNet pre-trained weights. w/o Pre-train: Training from scratch (the Kinetics 400 dataset results from mmaction2~\cite{51mmaction2}). w/ Pre-train (K400): using Kinetics 400 pre-trained weights.}
\begin{tabular}{ccccccc}
\toprule
\multirow{2}{*}{Dataset} & \multirow{2}{*}{Model} & \multirow{2}{*}{Backbone} & \multicolumn{2}{c}{w/ Pre-train (IN1k)} & \multicolumn{2}{c}{w/o Pre-train}\\
\cmidrule(r){4-7}
\multirow{2}{*}{~} & \multirow{2}{*}{~} & \multirow{2}{*}{~} & Top-1 & Top-5 & Top-1 & Top-5 \\
\midrule
\multirow{5}{*}{Kinetics 400~\cite{6kinetic400}} & \multirow{2}{*} {TSN~\cite{19TSN}} & ResNet50 & 72.83 & 90.65 & - & - \\
\multirow{5}{*}{~} & \multirow{2}{*}{~} & ResNet101 & 75.89 & 92.07 & - & - \\
\cmidrule(r){2-7}
\multirow{5}{*}{~} & \multirow{2}{*}{SlowFast~\cite{22slowfast}} & ResNet50 & - & - & 76.65 & 92.86 \\
\multirow{5}{*}{~} & \multirow{2}{*}{~} & ResNet101 & - & - & 78.65 & 93.88\\
\cmidrule(r){2-7}
\multirow{5}{*}{~} & Swin~\cite{23swin} & Swin-B & 80.57 & 94.49 & - & - \\
\bottomrule
\toprule
\multirow{2}{*}{Dataset} & \multirow{2}{*}{Model} & \multirow{2}{*}{Backbone} & \multicolumn{2}{c}{w/ Pre-train (K400)} & \multicolumn{2}{c}{w/o Pre-train}\\
\cmidrule(r){4-7}
\multirow{2}{*}{~} & \multirow{2}{*}{~} & \multirow{2}{*}{~} & Top-1 & Top-5 & Top-1 & Top-5 \\
\midrule
\multirow{5}{*}{FHA-Kitchens} & \multirow{2}{*}{TSN~\cite{19TSN}} & ResNet50 & 30.37 & 74.26 & 29.11 & 73.84 \\
\multirow{5}{*}{~} & \multirow{2}{*}{~} & ResNet101 & 30.80 & 73.42 & 30.38 & 74.26 \\
\cmidrule(r){2-7}
\multirow{5}{*}{~} & \multirow{2}{*}{SlowFast~\cite{22slowfast}} & ResNet50 & 33.33 & 70.46 & 27.85 & 68.35 \\
\multirow{5}{*}{~} & \multirow{2}{*}{~} & ResNet101 & 36.71 & 67.93 & 31.22 & 69.62\\
\cmidrule(r){2-7}
\multirow{5}{*}{~} & Swin~\cite{23swin} & Swin-B & 37.13 & 70.89 & 34.18 & 66.67 \\
\bottomrule
\end{tabular}
\label{arfullresults}
\end{table}
\begin{table}[htbp]
\centering
\caption{Intra-class DG test results of Faster RCNN~\cite{24faster} with the ResNet50 backbone on the ``{\bf Hold}'' Setting. $\Delta_i = \frac{1}{2}\sum_{j, j\neq i}{ji} - ii, \Delta^{*}_i=\frac{1}{2}\sum_{j, j\neq i}{ij} - ii, i=0,1,2$.}
\label{dgtrack-hold1}
% \footnotesize
\centering
\begin{tabular}{c|ccc|c}
\toprule
& \multicolumn{3}{c}{Test (mAP)} & \\
\cmidrule(r){2-4}
Train & hold-up & hold-in & hold-around & $\Delta^{*}$\\
\midrule
w/o hold-up & \textbf{44.00} & 53.50 & 71.70 & 18.60 \\
w/o hold-in & 44.30 & \textbf{7.30} & 69.10 & 49.40\\
w/o hold-around & 52.30 & 52.80 & \textbf{47.80} & 4.75 \\
\midrule
$\Delta$ & 4.30 & 45.85 & 22.60 & \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[!ht]
\centering
\caption{Intra-class DG test results of Faster RCNN~\cite{24faster} with the ResNet50 backbone on the ``{\bf Take}'' Setting. $\Delta_i = \frac{1}{2}\sum_{j, j\neq i}{ji} - ii, \Delta^{*}_i=\frac{1}{2}\sum_{j, j\neq i}{ij} - ii, i=0,1,2$.}
\label{dgtrack-take1}
% \footnotesize
\centering
\begin{tabular}{c|ccc|c}
\toprule
& \multicolumn{3}{c}{Test (mAP)} & \\
\cmidrule(r){2-4}
Train & pick-up & grab & catch & $\Delta^{*}$\\
\midrule
w/o pick-up & \textbf{0.40} & 47.10 & 46.60 & 46.45\\
w/o grab & 19.00 & \textbf{4.50} & 39.00 & 24.50\\
w/o catch & 19.10 & 46.00 & \textbf{15.60} & 16.95 \\
\midrule
$\Delta$ & 18.65 & 42.05 & 27.20 & \\
\bottomrule
\end{tabular}
\end{table}
Here, we present the test results of two additional parent category actions (see in Table~\ref{dgtrack-hold1}, \ref{dgtrack-take1}) within the Intra-class DG track (Section~4.4.1 of the paper). It is evident from the results that the performance on sub-categories seen during training surpasses that of unseen sub-categories. This discrepancy is supported by the diagonal scores, which reveal a significant decline of up to 46 mAP. These findings align with the observations in Section~4.4.1 of the paper, emphasizing the existence of large intra-class variance and underscoring the need for more efforts to improve the generalization abilities of the models.
The visual results of the SL-D and SL-AR track experiments are presented in Figure~\ref{visdet3}, Figure~\ref{visdet2}, and Figure~\ref{visdet1}. We showcase the visualizations for interaction region detection, interaction object detection, and action recognition, focusing on various interaction scenarios that vary in the complexity of hand interaction regions. In the recognition results, we provide fine-grained action verbs corresponding to the three hand interaction regions, denoted as \emph{<L-O action verb, R-O action verb, O-O action verb>}. Figure~\ref{visdet3} shows some challenging cases of hand interactions, providing compelling evidence of the good prediction performance of detection and recognition models, \textit{i.e.}, the Faster-RCNN~\cite{24faster} with a ResNet50 backbone for detection and a pre-trained TSN~\cite{19TSN} model with a ResNet50 backbone for action recognition. Moreover, Figure~\ref{visdet2} and Figure~\ref{visdet1} also demonstrate accurate detection and recognition results for some common interaction cases.
\subsection{More statistics of the FHA-Kitchens Dataset}
In this part, we re-arrange some figures in the paper to make them more readable and provide more statistics of the FHA-Kitchens dataset. Our annotation primarily focuses on hand interaction objects and their corresponding interaction actions, resulting in a diverse array of verbs and nouns. (1) Verbs: The annotated dataset comprises 131 action verbs that have been grouped into 43 parent verb categories (Figure~\ref{cakefig-actionverbs1} and Figure~\ref{actionverbdistribution1}). The three most prevalent parent verb categories, based on the count of sub-action verbs, are \textit{Cut}, \textit{Hold}, and \textit{Take}, representing the most frequently occurring hand actions in human interactions. Figure~\ref{actionverbdistribution1} visually depicts the distribution of all verb categories within FHA-Kitchens, ensuring the presence of at least one instance for each verb category. Specifically, the mapping between action verb IDs and their corresponding category names can be seen in Table~\ref{verbid}. (2) Nouns: In our annotation process, we identified a total of 384 interaction object noun categories that are associated with actions, categorized into 17 super-categories. Figure~\ref{objectnoundistribution1} shows the distribution of noun categories based on their affiliations with super-categories. Notably, the super-category ``vegetables \& plants'' exhibits the highest number of sub-categories, followed by ``kitchenware'', which aligns with typical kitchen scenes. Specifically, the mapping between interaction object noun IDs and their corresponding category names can be seen in Table~\ref{nounid1}, Table~\ref{nounid2}, and Table~\ref{nounid3}.
\begin{figure}[!t]
\centering
\includegraphics[width=0.8\linewidth]{cakefig-actionverbs.pdf}
\caption{An overview of the action verbs and their parent action categories in FHA-Kitchens.}
\label{cakefig-actionverbs1}
\end{figure}
\begin{figure}[!t]
\centering
\includegraphics[width=1\linewidth]{actionverbdistribution.pdf}
\caption{The distribution of instances per action verb category (the outer ring of the circle in Figure~\ref{cakefig-actionverbs1}) in the FHA-Kitchens dataset.} \label{actionverbdistribution1}
\end{figure}
The distribution of instances per action triplet category in FHA-Kitchens, as depicted in Figure~\ref{actiontripletdistribution1}, depicts a long-tail property. This distribution reflects the frequency of hand interactions in real-world kitchen scenes, taking into account the varying commonness or rarity of specific hand actions. For instance, the action triplet ``<hand\_right, hold-in, utility-knife\_handle>'' consists of 9,887 instances, which is nine times more prevalent than the ``<hand\_left, hold-in, utility-knife\_handle>'' triplet. This long-tail characteristic of the distribution renders FHA-Kitchens a challenging benchmark for hand action recognition, making it suitable for investigating few-shot learning and out-of-distribution generalization in action recognition as well.
\begin{figure}[!t]
\centering
\includegraphics[width=1\linewidth]{actiontripletdistribution.pdf}
\caption{The distribution of instances per action triplet category in the FHA-Kitchens dataset.}
\label{actiontripletdistribution1}
\end{figure}
We performed a comprehensive statistical analysis on the bounding boxes of the three hand interaction regions and the corresponding interaction objects. Specifically, we focused on two aspects: the box area and the aspect ratio. Detailed results can be found in Figure~\ref{area} and Figure~\ref{ratio}. Figure~\ref{area} shows the considerable range of sizes covered by our bounding boxes, with many interaction objects exhibiting small and challenging sizes for accurate detection. Moreover, in Figure~\ref{ratio}, the aspect ratios of the bounding boxes exhibit notable variation. The aspect ratios of the three regions tend to concentrate within the range of [0.5,2], which can be attributed to the typical composition of interaction regions involving two interacting objects. Consequently, the bounding box encompasses the combined region of both objects. For instance, the R-O interaction region frequently involves the interaction between the ``right hand'' and ``utility knife''. In such cases, the aspect ratio of the bounding box is observed to be 2:1, as depicted in Figure~\ref{annvis}. These findings highlight the significant challenges of the detection task in our dataset.
\begin{figure}[!htb]
\centering
\includegraphics[width=13.5cm]{area.pdf}
\caption{The distributions of bounding box areas of interaction objects (left) and interaction regions (right) in the FHA-Kitchens dataset.}
\label{area}
\end{figure}
\begin{figure}[htbp]
\centering
\includegraphics[width=13.5cm]{ratio.pdf}
\caption{The distributions of bounding box aspect ratios of interaction objects (left) and interaction regions (right) in the FHA-Kitchens dataset.}
\label{ratio}
\end{figure}
\newpage
\subsection{Motivation}
\noindent \textbf{1. For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.}
\textbf{A1:} FHA-Kitchens is created to facilitate research in the field of fine-grained hand action recognition. It is important to study several challenging questions in the context of more training data from diverse fine-grained hand actions, such as: 1) How do different representative action recognition models perform on fine-grained hand action tasks? 2) How do state-of-the-art detection models perform on the refined hand interaction regions and interaction objects? 3) How about the impact of pre-training, e.g., on the full-body actions dataset~\cite{6kinetic400}, in the context of the large-scale dataset with diverse fine-grained hand actions? and 4) How do the intra-class and inter-class generalization capabilities of models trained with specific fine-grained hand actions or parent hand actions perform? However, existing action datasets primarily focus on full-body human actions and lack emphasis on hand actions, with limited granularity in their treatment. Therefore, it is impossible to study these questions using existing datasets. In contrast, FHA-Kitchens primarily focuses on hand actions and refines hand interaction regions into three sub-interaction regions. Each sub-interaction region action was annotated using a triplet format, denoted as <subject, action verb, object>. Overall, the action information is expanded to nine dimensions, significantly enhancing the granularity of actions and providing valuable resources for researchers to study these questions effectively.
FHA-Kitchens aims to provide a better, more comprehensive, and finer-grained benchmark for hand action recognition. While there are existing datasets available for hand action recognition, they only cover limited hand action information and lack an in-depth understanding of hand actions. With its diverse and fine-grained hand action information, the FHA-Kitchens dataset enables a better evaluation of performance for fine-grained hand action recognition methods.
\noindent \textbf{2. Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?}
\textbf{A2:} Our dataset is created by the authors as well as some volunteer undergraduate students from Wuhan University, including Ting Zhe, Yongqian Li, Chengli Chen, Xin Ding, Jiali Li, Tengzheng Li, RunFu Guo, Heng Gao, Xing Zhao, Weisheng Chen, Chaoyu Mai, Yipan Wei.
\noindent \textbf{3. Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.}
\textbf{A3:} This work is supported in part by the National Natural Science Foundation of China under the grant of program No.62276195, the Special Fund of Hubei Luojia Laboratory under Grant 220100014, and the Fundamental Research Funds for the Central Universities under the grant of program No.2042023kf1033.
\subsection{Composition}
\noindent \textbf{1. What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances(e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.}
\textbf{A1:} FHA-Kitchens consists of video frames, including 878 fine-grained hand interaction action triplets. It primarily focuses on fine-grained actions generated within hand interaction regions, such as \emph{cut-slice} and \emph{hold-in}. For each frame, we provide bounding boxes for three hand sub-interaction regions (\textit{i.e.}, left hand-object (L-O), right hand-object (R-O), and object-object (O-O) interaction regions) and the interaction objects. Each sub-interaction region action was annotated using a triplet format, denoted as \emph{<subject, action verb, object>}. Additionally, we provide segmentation masks related to hands and interaction objects.
\noindent \textbf{2. How many instances are there in total (of each type, if appropriate)?}
\textbf{A2:} The FHA-Kitchens contains 30,047 frames from 2,377 video clips, with each frame annotated for three fine-grained hand interaction regions, resulting in a total of 878 fine-grained action triplets. Among them, there are 597 frames where no hand interaction action occurs, represented as {L-O\_triplet:<none>, R-O\_triplet:<none>, O-O\_triplet: <none>}.
\noindent \textbf{3. Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).}
\textbf{A3:} FHA-Kitchens is a real-world sample of human hands part in the kitchen scenes, including information about their actions. The data is sourced from an existing large-scale full-body action dataset~\cite{700_2020}, from which we selected videos featuring hand interaction actions. We extracted a total of 30 videos, amounting to 84.22 minutes of footage, encompassing 8 distinct types of dishes. Due to the diversity of real-world human hand actions, it's impossible to cover all types of actions. The FHA-Kitchens dataset focuses primarily on fine-grained tasks related to hand actions. To address the granularity issue, we improved the action information in the existing dataset. Compared to the data's original annotations in Kinetics-700\_2020~\cite{700_2020}, our dataset expanded the action labels by 7 dimensions, increased the number of action categories by 52 times, and introduced 123 new action verbs. We provide a finer-grained set of hand action instances than ever before, facilitating further research in fine-grained hand action recognition.
\noindent \textbf{4. What data does each instance consist of? “Raw” data (e.g., unprocessed text or images)or features? In either case, please provide a description.}
\textbf{A4:} Each instance consists of at most 9 kinds of bonding boxes (\textit{i.e.}, three hand sub-interaction regions and interaction objects within interaction region) and sub-interaction region corresponding triplet descriptions (i.e., \emph{<subject, action verb, object>}). Additionally, we took into account the ``active-passive'' relationships between object pairs and the specific contact areas involved in the interaction actions. Consequently, our annotation process encompassed a total of nine dimensions, resulting in 878 annotated action triplets for hand actions. The annotation details are listed in Figure\ref{annlist}, and the corresponding visualizations are shown in Figure\ref{annvis}.
\begin{figure}[htbp]
\centering
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[width=6.8cm]{annvis.pdf}
\caption{Visualization of bounding box annotations for the example of ``\emph{fry vegetables}''.}
\label{annvis}
\end{minipage}
\hspace{0.15cm}
\begin{minipage}[t]{0.48\textwidth}
\centering
\includegraphics[width=6.8cm]{annlist.pdf}
\caption{Descriptive list of action triplets and bounding box annotations.}
\label{annlist}
\end{minipage}
\end{figure}
\noindent \textbf{5. Is there a label or target associated with each instance? If so, please provide a description.}
\textbf{A5:} Yes. Due to our parallel annotation process, we generated annotation files in different styles. However, we consolidated all the bounding box and triplet annotation information into a single CSV file. In the merged CSV file, each instance is annotated with labels following the style of the Kinetics~\cite{6kinetic400,7kinetics600,8kinetic700,700_2020} and AVA~\cite{10ava} datasets, which include video\_name, video\_id, clip\_id, clip\_name, image\_id, timestamp, L-O\_triplet, L-O\_action\_verb\_id, L-O\_action\_class, L-O\_action\_bbox, left\_hand\_bbox, O1\_class, O1\_bbox , R-O\_triplet, R-O\_action\_id, R-O\_action\_class, R-O\_action\_bbox, right\_hand\_bbox, O2\_class, O2\_bbox, O-O\_triplet, O-O\_action\_id, O-O\_action\_class, O-O\_action\_bbox, subject\_class, subject\_bbox, object\_class, object\_bbox, action\_verb\_triplet, action\_verb\_triplet\_id.
\noindent \textbf{6. Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.}
\textbf{A6:} Yes. Some instances may not have all nine types of bonding boxes and their corresponding action triplets and segmentation annotation because of severe occlusion, truncation, blur, or small scale. We just annotated ``None'' in our annotation file to represent this situation.
\noindent \textbf{7. Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit.}
\textbf{A7:} Yes. We provide different styles of annotations files, in COCO-style, the annotations are connected by image id and category id, you can easily access them by \href{https://github.com/jin-s13/xtcocoapi} {COCO APIs}. In CSV-style, one line represents the annotations of one frame and can be processed by the pandas library easily.
\noindent \textbf{8. Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.}
\textbf{A8:} Yes. We randomly split the dataset into the disjoint train, validation, and test sets following the ratio of 7:1:2.
\noindent \textbf{9. Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.}
\textbf{A9:} Although we conducted three rounds of cross-checking and corrections, there may still be some errors in the annotations, \textit{e.g.}, inappropriate bounding box annotations or small drifts of the bounding box locations, incorrectly written verbs or nouns, insufficient granularity in verb or noun descriptions, inappropriate formatting of triplets, etc. However, we have made every effort to minimize such occurrences.
To analyze the quality of annotations, we randomly selected 500 frames and conducted manual evaluations for correctness. The results are reported in Table~\ref{error_rate_table}. These error rates are comparable to recently published datasets~\cite{1epic}.
\begin{table}[htbp]
\centering
\caption{Error rate in FHA-Kitchens. I-O: Interaction Objects, I-R: Interaction Regions.}
\begin{tabular}{cccccc}
\hline
&\multicolumn{1}{l}{\textbf{Frames}} & \multicolumn{1}{l}{\textbf{I-O Boxes}} & \multicolumn{1}{l}{\textbf{I-R Boxes}} & \multicolumn{1}{l}{\textbf{Verb}} & \multicolumn{1}{l}{\textbf{Noun}} \\
\hline
\textbf{Total Number} & 500 & 3,006 & 1,503 & 1,503 & 2,006 \\
\textbf{Error Rate (\%)} & - & 4.9 & 2.5 & 2.2 & 5.3 \\
\hline
\end{tabular}
\label{error_rate_table}
\end{table}
\noindent \textbf{10. Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.}
\textbf{A10:} Our dataset was derived from a large-scale publicly available dataset, namely \href{https://www.deepmind.com/open-source/kinetics} {Kinetics-700\_2020}~\cite{700_2020}, which is publicly available for download from their website. The Kinetics dataset follows the Creative Commons Attribution 4.0 International License. We would like to express our gratitude to the authors for their significant contributions to the research community.
\noindent \textbf{11. Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctorpatient confidentiality, data that includes the content of individuals non-public communications)? If so, please provide a description.}
\textbf{A11:} No.
\noindent \textbf{12. Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.}
\textbf{A12:} No.
\subsection{Collection Process}
\noindent \textbf{1. How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.}
\textbf{A1:} The FHA-Kitchens dataset follows the annotation styles of Kinetics-700\_2020~\cite{700_2020} and AVA~\cite{10ava} datasets, integrating all the annotation information (bounding box and triplet) into a single CSV file. Each row in the file represents the annotations for a single frame, and these annotations are visually accessible within the labels.
\noindent \textbf{2. What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?}
\textbf{A2:} The data in FHA-Kitchens come from dataset publicly available datasets described above, which can be directly downloaded from their websites.
\noindent \textbf{3. If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?}
\textbf{A3:} Currently, we focus exclusively on hand interaction actions in kitchen scenes, thus primarily extracting data that includes hand interaction actions in kitchen scenes.
\noindent \textbf{4. Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?}
\textbf{A4:} The first two authors collected this dataset. The annotation compensation is based on the prevailing market rates.
\noindent \textbf{5. Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.}
\textbf{A5}: It took about 1 week to collect the data and about 6 weeks to complete organization and annotation, as each participant labeled the bonding boxes and action triplets about four hours per workday. And the segmentation masks are generated by the Segment-Anything Model~\cite{20segment} guided by the bonding boxes, and corrected by human annotators for about one week.
\subsection{Preprocessing/cleaning/labeling}
\noindent \textbf{1. Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.}
\textbf{A1:} Yes. Since we focus on hand actions, we performed filtering and processing operations on the original videos, including the following three steps. (1) First, we observed that kitchen scenes often featured hand actions, with video content prominently showcasing human hand parts. Therefore, we sought out and extracted relevant videos that were set against a kitchen backdrop. (2)Then, to ensure the quality of the dataset, we selectively chose videos with higher resolutions. Specifically, 87\% of the videos were recorded at 1,280 $\times$ 720 resolution, while another 13\% had a shorter side of 480. Additionally, 67\% of the videos were captured at 30 frames per second (fps), and another 33\% were recorded at 24$\sim$25 fps. (3) Subsequently, we imposed a duration constraint on the videos, ranging from 30 seconds to 5 minutes, to exclude excessively long-duration videos. This constraint aimed to maintain a balanced distribution within the sample space. Finally, we collected a total of 30 videos, amounting to 84.22 minutes of footage, encompassing 8 distinct types of dishes.
The collected video data was reorganized and cleaned to align with our annotation criteria. First, we split the collected video data into individual frames, as our annotated units are frames. Subsequently, we conducted further cleaning of the frames by excluding those that did not depict hands or exhibited meaningless hand actions. This cleaning process took into consideration factors such as occlusion, frame quality (\textit{i.e.}, without significant blur, subtitles, and logos), meaningful hand actions, and frame continuity. As a result, we obtained a total of 30,047 high-quality candidate video frames containing diverse hand actions for our FHA-Kitchens dataset. Compared to the initial collection, 113,436 frames were discarded during the cleaning process.
We recruited 10 voluntary annotators, whose responsibility was to annotate fine-grained action triplet classes and bounding boxes for each hand interaction region. In order to enhance annotation efficiency, we implemented a parallel annotation pipeline. The annotation of action triplets was carried out on the Amazon Mechanical Turk platform, while the bounding box annotation was facilitated using the LabelBee tool. To ensure the annotation quality, three rounds of cross-checking and corrections were conducted.
\noindent \textbf{2. Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data.}
\textbf{A2:} No.
\noindent \textbf{3. Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.}
\textbf{A3:} The annotation of action triplets was carried out on the \href{https://www.mturk.com/}{Amazon Mechanical Turk} platform, while the bounding box annotation was facilitated using the \href{https://github.com/open-mmlab/labelbee-client}{LabelBee} tool.
\subsection{Uses}
\noindent \textbf{1. Has the dataset been used for any tasks already? If so, please provide a description.}
\textbf{A1:} No.
\noindent \textbf{2. Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.}
\textbf{A2:} N/A.
\noindent \textbf{3. What (other) tasks could the dataset be used for?}
\textbf{A3:} FHA-Kitchens can be used for the research of fine-grained hand action recognition and hand interaction region and object detection. Besides, it can also be used for specific machine learning topics such as domain generalization and action segmentation. Please see the Discussion part of the paper.
\noindent \textbf{4. Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?}
\textbf{A4:} No.
\noindent \textbf{5. Are there tasks for which the dataset should not be used? If so, please provide a description.}
\textbf{A5:} No.
\subsection{Distribution}
\noindent \textbf{1. Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.}
\textbf{A1:} Yes. The dataset will be made publicly available to the research community.
\noindent \textbf{2. How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?}
\textbf{A2:} It will be publicly available on the project website at \href{https://github.com/tingZ123/FHA-Kitchens}{GitHub}.
\noindent \textbf{3. When will the dataset be distributed?}
\textbf{A3:} The dataset will be distributed once the paper is accepted after peer review.
\noindent \textbf{4. Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.}
\textbf{A4:} It will be distributed under the MIT license.
\noindent \textbf{5. Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.}
\textbf{A5:} No.
\noindent \textbf{6. Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.}
\textbf{A6:} No.
\subsection{Maintenance}
\noindent \textbf{1. Who will be supporting/hosting/maintaining the dataset?}
\textbf{A1:} The authors.
\noindent \textbf{2. How can the owner/curator/manager of the dataset be contacted (e.g., email address)?}
\textbf{A2:} They can be contacted via email available on the \href{https://github.com/tingZ123/FHA-Kitchens} {project website}.
\noindent \textbf{3. Is there an erratum? If so, please provide a link or other access point.}
\textbf{A3:} No.
\noindent \textbf{4. Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?}
\textbf{A4:} No. We have carefully double-checked the annotations to reduce the labeling errors. There may be very few labeling errors, which can be treated as noise.
\noindent \textbf{5. Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to users.}
\textbf{A5:} N/A.
\noindent \textbf{6. If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description.}
\textbf{A6:} N/A.
\begin{figure}[!htb]
% \setlength{\belowcaptionskip}{-0.5cm}
\centering
\includegraphics[width=1\linewidth]{visdet3.pdf}
\caption{Some visual examples of three-hand interaction regions in our FHA-Kitchens. I-R: Interaction Region, I-O: Interaction Object, I-A: Interaction Region Action Verb}
\label{visdet3}
\end{figure}
\begin{figure}[!htb]
\centering
\includegraphics[width=1\linewidth]{visdet2.pdf}
\caption{Some visual examples of two-hand interaction regions in our FHA-Kitchens. I-R: Interaction Region, I-O: Interaction Object, I-A: Interaction Region Action Verb}
\label{visdet2}
\end{figure}
\begin{figure}[!htb]
\centering
\includegraphics[width=1\linewidth]{visdet1.pdf}
\caption{Some visual examples of one-hand interaction region in our FHA-Kitchens. I-R: Interaction Region, I-O: Interaction Object, I-A: Interaction Region Action Verb}
\label{visdet1}
\end{figure}
\begin{table}[!ht]
\centering
\caption{Vocabulary of fine-grained hand action verbs.}
\label{verbid}
\footnotesize
\centering
\resizebox{0.65\linewidth}{!}{\begin{tabular}{lll|lll}
\toprule
ID & Verb & \#Instance & ID & Verb & \#Instance \\
\hline
0 & hold-around & 1,593 & 66 & contain & 144\\
1 & hold-at & 788 & 67 & roll-on & 82 \\
2 & fill-with & 265 & 68 & stick-to & 50 \\
3 & pinch-on & 1,115 & 69 & touch-to & 12 \\
4 & rub-around & 45 & 70 & smooth-out & 144 \\
5 & hold-in & 20,520 & 71 & sprinkle-on & 203 \\
6 & touch-on & 42 & 72 & squeeze-around & 189 \\
7 & hold-with & 341 & 73 & press-down & 675 \\
8 & press-on & 9,369 & 74 & cut-up & 100 \\
9 & cut-out & 889 & 75 & shovel-up & 123 \\
10 & fix-on & 2,037 & 76 & grab-out & 1 \\
11 & peel-off & 1,413 & 77 & close & 51 \\
12 & slice-along & 1,306 & 78 & rotate & 51 \\
13 & grab & 2,531 & 79 & open & 72 \\
14 & cut-half & 230 & 80 & open-down & 27 \\
15 & take-up & 134 & 81 & ratate-around & 1 \\
16 & pinch & 609 & 82 & hold-down & 41 \\
17 & catch & 446 & 83 & cut-dice & 443 \\
18 & put-down & 1,406 & 84 & dig-seeds & 124 \\
19 & roll-up & 1,296 & 85 & chop & 346 \\
20 & fix & 293 & 86 & push-forward & 8 \\
21 & scrub-inside & 53 & 87 & cut-halves & 22 \\
22 & lay-down & 111 & 88 & peel & 87 \\
23 & hold-onto & 453 & 89 & push-ahead & 2 \\
24 & pick-up & 526 & 90 & screw-on & 5 \\
25 & cut-slice & 2,808 & 91 & sprinkle-into & 16 \\
26 & take-out & 178 & 92 & scoop-up & 85 \\
27 & turn-off & 22 & 93 & hold-along & 129 \\
28 & cut-down & 1,040 & 94 & scrape-on & 331 \\
29 & cut-off & 820 & 95 & stick-with & 13 \\
30 & grab-up & 115 & 96 & cut-in & 275 \\
31 & put-up & 56 & 97 & rub-on & 11 \\
32 & break-apart & 221 & 98 & put-on & 2 \\
33 & touch & 483 & 99 & push-off & 10 \\
34 & cut-into-halves & 136 & 100 & place-on & 17 \\
35 & bring-up & 140 & 101 & cut & 15 \\
36 & pour-out & 832 & 102 & dip-in & 9 \\
37 & pour-into & 395 & 103 & stretch-out & 31 \\
38 & pour & 154 & 104 & flip & 8 \\
39 & scrape & 70 & 105 & set-aside & 22 \\
40 & rotate-around & 161 & 106 & julienne & 165 \\
41 & screw-down & 19 & 107 & unroll & 270 \\
42 & remove-out & 69 & 108 & adjust & 46 \\
43 & hold-up & 664 & 109 & place-down & 262 \\
44 & scoop-out & 76 & 110 & pile & 49 \\
45 & open-up & 21 & 111 & pull & 137 \\
46 & hold & 946 & 112 & attach-to & 18 \\
47 & hold-on & 187 & 113 & grab-in & 15 \\
48 & squeeze & 334 & 114 & knock-on & 29 \\
49 & squeeze-out & 254 & 115 & press-against & 13 \\
50 & mix-together & 187 & 116 & stir-in & 114 \\
51 & spread-on & 342 & 117 & pull-up & 47 \\
52 & twist-off & 20 & 118 & point-at & 25 \\
53 & wrap-around & 204 & 119 & pull-out & 33 \\
54 & break-off & 417 & 120 & scrape-down & 6 \\
55 & grab-at & 621 & 121 & grab-onto & 558 \\
56 & grab-on & 384 & 122 & hold-into & 88 \\
57 & cut-through & 25 & 123 & hold-over & 67 \\
58 & chop-dice & 63 & 124 & stir-into & 49 \\
59 & sprinkle & 272 & 125 & press-onto & 1 \\
60 & insert-into & 106 & 126 & roll & 68 \\
61 & put-in & 30 & 127 & roll-out & 33 \\
62 & dig-out & 23 & 128 & dip & 51 \\
63 & cut-chunks & 91 & 129 & brush-onto & 54 \\
64 & churn & 369 & 130 & flip-over & 83 \\
65 & knead & 298 & & & \\
\bottomrule
\end{tabular}}
\end{table}
\begin{table}[!ht]
\centering
\caption{Vocabulary of fine-grained interaction object nouns.}
\label{nounid1}
\centering
\resizebox{1.05\linewidth}{!}{
\begin{tabular}{c|lll|c|lll}
\toprule
Super category & ID & Noun & \#Instance & Super category & ID & Noun & \#Instance \\
\hline
\multirow{65}{*}{Vegetables\&Plants} & 0 & basil-end & 201 & \multirow{47}{*}{Fruits} & 65 & apple-all & 22 \\
\multirow{65}{*}{~} & 1 & beet-end & 143 & \multirow{47}{*}{~} & 66 & apple-end & 34 \\
\multirow{65}{*}{~} & 2 & beet-head & 136 & \multirow{47}{*}{~} & 67 & apple-head & 253 \\
\multirow{65}{*}{~} & 3 & beet-middle & 36 & \multirow{47}{*}{~} & 68 & apple-middle & 301 \\
\multirow{65}{*}{~} & 4 & bell-pepper-all & 2 & \multirow{47}{*}{~} & 69 & avocado-end & 156 \\
\multirow{65}{*}{~} & 5 & bell-pepper-end & 210 & \multirow{47}{*}{~} & 70 & avocado-head & 12 \\
\multirow{65}{*}{~} & 6 & bell-pepper-head & 67 & \multirow{47}{*}{~} & 71 & avocado-left & 34 \\
\multirow{65}{*}{~} & 7 & bell-pepper-middle & 139 & \multirow{47}{*}{~} & 72 & avocado-middle & 174 \\
\multirow{65}{*}{~} & 8 & broccoli-head & 38 & \multirow{47}{*}{~} & 73 & avocado-right & 34 \\
\multirow{65}{*}{~} & 9 & carrot-end & 1,625 & \multirow{47}{*}{~} & 74 & block-watermelon-edge & 5 \\
\multirow{65}{*}{~} & 10 & carrot-head & 205 & \multirow{47}{*}{~} & 75 & green-melon-all & 4,074 \\
\multirow{65}{*}{~} & 11 & carrot-middle & 638 & \multirow{47}{*}{~} & 76 & green-melon-end & 68 \\
\multirow{65}{*}{~} & 12 & chopped-vegetables-surface & 38 & \multirow{47}{*}{~} & 77 & green-melon-middle & 68 \\
\multirow{65}{*}{~} & 13 & courgette-end & 921 & \multirow{47}{*}{~} & 78 & half-apple-head & 139 \\
\multirow{65}{*}{~} & 14 & courgette-middle & 39 & \multirow{47}{*}{~} & 79 & half-apple-middle & 47 \\
\multirow{65}{*}{~} & 15 & cucumber-end & 288 & \multirow{47}{*}{~} & 80 & half-pineapple-head & 6 \\
\multirow{65}{*}{~} & 16 & cucumber-middle & 165 & \multirow{47}{*}{~} & 81 & half-pineapple-middle & 99 \\
\multirow{65}{*}{~} & 17 & cucumber-strip-all & 14 & \multirow{47}{*}{~} & 82 & half-tomato-end & 1 \\
\multirow{65}{*}{~} & 18 & cucumber-strip-end & 72 & \multirow{47}{*}{~} & 83 & half-tomato-middle & 1 \\
\multirow{65}{*}{~} & 19 & cucumber-strip-middle & 22 & \multirow{47}{*}{~} & 84 & half-watermelon-edge & 79 \\
\multirow{65}{*}{~} & 20 & garlic-middle & 240 & \multirow{47}{*}{~} & 85 & half-watermelon-end & 17 \\
\multirow{65}{*}{~} & 21 & garlic-end & 164 & \multirow{47}{*}{~} & 86 & half-watermelon-head & 301 \\
\multirow{65}{*}{~} & 22 & garlic-head & 46 & \multirow{47}{*}{~} & 87 & half-watermelon-middle & 232 \\
\multirow{65}{*}{~} & 23 & ginger-end & 248 & \multirow{47}{*}{~} & 88 & lemon-end & 156 \\
\multirow{65}{*}{~} & 24 & ginger-head & 169 & \multirow{47}{*}{~} & 89 & lemon-middle & 108 \\
\multirow{65}{*}{~} & 25 & ginger-middle & 70 & \multirow{47}{*}{~} & 90 & melon-skin-all & 119 \\
\multirow{65}{*}{~} & 26 & green-beans-end & 937 & \multirow{47}{*}{~} & 91 & melon-skin-end & 576 \\
\multirow{65}{*}{~} & 27 & green-pepper-dice & 1 & \multirow{47}{*}{~} & 92 & melon-pulp-all & 28 \\
\multirow{65}{*}{~} & 28 & green-pepper-end & 710 & \multirow{47}{*}{~} & 93 & melon-pulp-end & 163 \\
\multirow{65}{*}{~} & 29 & green-pepper-head & 142 & \multirow{47}{*}{~} & 94 & melon-pulp-middle & 49 \\
\multirow{65}{*}{~} & 30 & green-pepper-middle & 505 & \multirow{47}{*}{~} & 95 & melon-slice-end & 200 \\
\multirow{65}{*}{~} & 31 & half-bell-pepper-end & 116 & \multirow{47}{*}{~} & 96 & orange-all & 22 \\
\multirow{65}{*}{~} & 32 & half-bell-pepper-middle & 110 & \multirow{47}{*}{~} & 97 & orange-end & 24 \\
\multirow{65}{*}{~} & 33 & half-onion-all & 23 & \multirow{47}{*}{~} & 98 & orange-head & 209 \\
\multirow{65}{*}{~} & 34 & half-onion-head & 11 & \multirow{47}{*}{~} & 99 & orange-middle & 547 \\
\multirow{65}{*}{~} & 35 & half-onion-middle & 11 & \multirow{47}{*}{~} & 100 & peelless-orange-middle & 93 \\
\multirow{65}{*}{~} & 36 & mushroom-middle & 15 & \multirow{47}{*}{~} & 101 & piece-orange-edge & 276 \\
\multirow{65}{*}{~} & 37 & nori-all & 506 & \multirow{47}{*}{~} & 102 & pineapple-all & 26 \\
\multirow{65}{*}{~} & 38 & nori-end & 262 & \multirow{47}{*}{~} & 103 & pineapple-end & 476 \\
\multirow{65}{*}{~} & 39 & onion-end & 78 & \multirow{47}{*}{~} & 104 & pineapple-head & 959 \\
\multirow{65}{*}{~} & 40 & onion-head & 28 & \multirow{47}{*}{~} & 105 & pineapple-middle & 1,346 \\
\multirow{65}{*}{~} & 41 & onion-middle & 49 & \multirow{47}{*}{~} & 106 & slice-pineapple-end & 25 \\
\multirow{65}{*}{~} & 42 & pepper-seeds-all & 4 & \multirow{47}{*}{~} & 107 & slice-pineapple-middle & 63 \\
\multirow{65}{*}{~} & 43 & piece-onion-middle & 38 & \multirow{47}{*}{~} & 108 & watermelon-edge & 29 \\
\multirow{65}{*}{~} & 44 & piece-tomato-end & 41 & \multirow{47}{*}{~} & 109 & watermelon-end & 966 \\
\multirow{65}{*}{~} & 45 & purple-cabbage-end & 77 & \multirow{47}{*}{~} & 110 & watermelon-head & 155 \\
\multirow{65}{*}{~} & 46 & purple-cabbage-head & 70 & \multirow{47}{*}{~} & 111 & watermelon-middle & 631 \\
\cline{5-8}
\multirow{65}{*}{~} & 47 & purple-cabbage-middle & 23 & \multirow{12}{*}{Dairy\&Eggs} & 112 & boiled-egg-end & 226 \\
\multirow{65}{*}{~} & 48 & red-pepper-all & 21 & \multirow{12}{*}{~} & 113 & boiled-egg-head & 22 \\
\multirow{65}{*}{~} & 49 & red-pepper-head & 25 & \multirow{12}{*}{~} & 114 & boiled-egg-middle & 14 \\
\multirow{65}{*}{~} & 50 & red-pepper-middle & 18 & \multirow{12}{*}{~} & 115 & boiled-egg-shell & 88 \\
\multirow{65}{*}{~} & 51 & small-tomato-head & 9 & \multirow{12}{*}{~} & 116 & egg-all & 248 \\
\multirow{65}{*}{~} & 52 & small-tomato-middle & 27 & \multirow{12}{*}{~} & 117 & egg-head & 1 \\
\multirow{65}{*}{~} & 53 & spinach-end & 4 & \multirow{12}{*}{~} & 118 & egg-middle & 26 \\
\multirow{65}{*}{~} & 54 & spinach-head & 35 & \multirow{12}{*}{~} & 119 & egg-liquid-all & 10 \\
\multirow{65}{*}{~} & 55 & spinach-middle & 30 & \multirow{12}{*}{~} & 120 & egg-shell-all & 86 \\
\multirow{65}{*}{~} & 56 & spring-garlic-all & 18 & \multirow{12}{*}{~} & 121 & egg-shell-edge & 34 \\
\multirow{65}{*}{~} & 57 & spring-garlic-end & 53 & \multirow{12}{*}{~} & 122 & milk-all & 594 \\
\multirow{65}{*}{~} & 58 & spring-garlic-head & 23 & \multirow{12}{*}{~} & 123 & yolk-all & 112 \\
\cline{5-8}
\multirow{65}{*}{~} & 59 & spring-garlic-middle & 52 & \multirow{6}{*}{Meat\&Fish} & 124 & chicken-dice & 8 \\
\multirow{65}{*}{~} & 60 & sun-flower-seeds & 104 & \multirow{6}{*}{~} & 125 & raw-chicken-dice & 69 \\
\multirow{65}{*}{~} & 61 & tomato-cube & 22 & \multirow{6}{*}{~} & 126 & crab-shred & 328 \\
\multirow{65}{*}{~} & 62 & tomato-end & 280 & \multirow{6}{*}{~} & 127 & meat-end & 515 \\
\multirow{65}{*}{~} & 63 & tomato-middle & 17 & \multirow{6}{*}{~} & 128 & meat-head & 573 \\
\multirow{65}{*}{~} & 64 & tomato-sliced-middle & 109 & \multirow{6}{*}{~} & 129 & meat-middle & 142 \\
% \bottomrule
\hline
\end{tabular}}
\end{table}
\begin{table}[!ht]
\centering
\caption{Vocabulary of fine-grained interaction object nouns.}
\label{nounid2}
% \footnotesize
\centering
\resizebox{1.05\linewidth}{!}{ \begin{tabular}{c|lll|c|lll}
\toprule
Super category & ID & Noun & \#Instance & Super category & ID & Noun & \#Instance \\
\hline
\multirow{9}{*}{Meat\&Fish} & 130 & meat-piece-end & 442 & \multirow{49}{*}{Containers} & 194 & bottle-body & 52 \\
\multirow{9}{*}{~} & 131 & meat-slice-all & 1 & \multirow{49}{*}{~} & 195 & box-lid-bottom & 21 \\
\multirow{9}{*}{~} & 132 & meat-slice-end & 202 & \multirow{49}{*}{~} & 196 & bottle-cap & 8 \\
\multirow{9}{*}{~} & 133 & piece-pepperoni-all & 87 & \multirow{49}{*}{~} & 197 & bottle-cap-all & 106 \\
\multirow{9}{*}{~} & 134 & piece-pepperoni-end & 158 & \multirow{49}{*}{~} & 198 & bottle-cap-bottom & 19 \\
\multirow{9}{*}{~} & 135 & salmon-piece-all & 36 & \multirow{49}{*}{~} & 199 & can-cover-edg & 2 \\
\multirow{9}{*}{~} & 136 & salmon-piece-end & 399 & \multirow{49}{*}{~} & 200 & can-cover-edge & 106 \\
\multirow{9}{*}{~} & 137 & salmon-piece-middle & 14 & \multirow{49}{*}{~} & 201 & ceramic-cup-all & 68 \\
\multirow{9}{*}{~} & 138 & salmon-slice-end & 44 & \multirow{49}{*}{~} & 202 & ceramic-cup-body & 671 \\
\cline{1-4}
\multirow{13}{*}{Spices\&Sauces} & 139 & butter-all & 45 & \multirow{49}{*}{~} & 203 & ceramic-cup-handle & 12 \\
\multirow{13}{*}{~} & 140 & crumbles-cheese-all & 116 & \multirow{49}{*}{~} & 204 & ceramic-lid-all & 25 \\
\multirow{13}{*}{~} & 141 & cheese-all & 27 & \multirow{49}{*}{~} & 205 & ceramic-lid-edge & 85 \\
\multirow{13}{*}{~} & 142 & green-butter-all & 85 & \multirow{49}{*}{~} & 206 & ceramic-teapot-handle & 132 \\
\multirow{13}{*}{~} & 143 & mozzarella-all & 84 & \multirow{49}{*}{~} & 207 & ceramic-teacup-body & 69 \\
\multirow{13}{*}{~} & 144 & mozzarella-end & 12 & \multirow{49}{*}{~} & 208 & ceramic-teacup-edge & 138 \\
\multirow{13}{*}{~} & 145 & pizza-sauce-end & 193 & \multirow{49}{*}{~} & 209 & cup-edge & 79 \\
\multirow{13}{*}{~} & 146 & powder-all & 29 & \multirow{49}{*}{~} & 210 & glass-bottle-edge & 40 \\
\multirow{13}{*}{~} & 147 & sauce-all & 397 & \multirow{49}{*}{~} & 211 & glass-bottle-top & 59 \\
\multirow{13}{*}{~} & 148 & slice-cheese-end & 44 & \multirow{49}{*}{~} & 212 & glass-cup-body & 1 \\
\multirow{13}{*}{~} & 149 & tomato-sauce-all & 151 & \multirow{49}{*}{~} & 213 & glass-cup-edge & 217 \\
\multirow{13}{*}{~} & 150 & tomato-sauce-edge & 15 & \multirow{49}{*}{~} & 214 & glass-cup-handle & 51 \\
\multirow{13}{*}{~} & 151 & sauce-mixed & 70 & \multirow{49}{*}{~} & 215 & glass-goblet-stem & 333 \\
\cline{1-4}
\multirow{5}{*}{Liquids} & 152 & can-opener & 108 & \multirow{49}{*}{~} & 216 & glastic-bottle-edge & 4 \\
\multirow{5}{*}{~} & 153 & green-mixture-all & 248 & \multirow{49}{*}{~} & 217 & glastic-bottle-top & 4 \\
\multirow{5}{*}{~} & 154 & jam-all & 23 & \multirow{49}{*}{~} & 218 & grass-bottle-top & 30 \\
\multirow{5}{*}{~} & 155 & oil-all & 60 & \multirow{49}{*}{~} & 219 & iron-basin-body & 29 \\
\multirow{5}{*}{~} & 156 & olive-oil-all & 51 & \multirow{49}{*}{~} & 220 & iron-basin-edge & 145 \\
\cline{1-4}
\multirow{3}{*}{Baked\&Baking} & 157 & baking-paper-edge & 99 & \multirow{49}{*}{~} & 221 & iron-basin-middle & 115 \\
\multirow{3}{*}{~} & 158 & baking-paper-top & 25 & \multirow{49}{*}{~} & 222 & iron-cup-body & 1,005 \\
\multirow{3}{*}{~} & 159 & baking-plate-edge & 54 & \multirow{49}{*}{~} & 223 & iron-cup-handle & 325 \\
\cline{1-4}
\multirow{15}{*}{Cooked Food} & 160 & piece-pizza-end & 125 & \multirow{49}{*}{~} & 224 & iron-dipper-handle & 14 \\
\multirow{15}{*}{~} & 161 & pizza-all & 63 & \multirow{49}{*}{~} & 225 & plastic-basin-edge & 33 \\
\multirow{15}{*}{~} & 162 & pizza-end & 27 & \multirow{49}{*}{~} & 226 & plastic-bottle-bottom & 21 \\
\multirow{15}{*}{~} & 163 & pizza-middle & 29 & \multirow{49}{*}{~} & 227 & plastic-bottle-edge & 352 \\
\multirow{15}{*}{~} & 164 & sandwich-edge & 32 & \multirow{49}{*}{~} & 228 & plastic-bottle-top & 78 \\
\multirow{15}{*}{~} & 165 & sandwich-end & 297 & \multirow{49}{*}{~} & 229 & plastic-cup-body & 19 \\
\multirow{15}{*}{~} & 166 & sandwich-head & 141 & \multirow{49}{*}{~} & 230 & sauce-container-end & 4 \\
\multirow{15}{*}{~} & 167 & sandwich-middle & 123 & \multirow{49}{*}{~} & 231 & small-cup-edge & 195 \\
\multirow{15}{*}{~} & 168 & sandwich-side & 85 & \multirow{49}{*}{~} & 232 & small-plastic-bottle-edge & 154 \\
\multirow{15}{*}{~} & 169 & sandwich-top & 27 & \multirow{49}{*}{~} & 233 & small-plastic-bottle-end & 120 \\
\multirow{15}{*}{~} & 170 & sandwich-all & 23 & \multirow{49}{*}{~} & 234 & small-plastic-bottle-top & 167 \\
\multirow{15}{*}{~} & 171 & sushi-roll-all & 151 & \multirow{49}{*}{~} & 235 & teapot-lid-edge & 126 \\
\multirow{15}{*}{~} & 172 & sushi-roll-end & 1,659 & \multirow{49}{*}{~} & 236 & teapot-lid-handle & 267 \\
\multirow{15}{*}{~} & 173 & sushi-roll-head & 233 & \multirow{49}{*}{~} & 237 & wine-bottle-bottom & 75 \\
\multirow{15}{*}{~} & 174 & sushi-roll-middle & 622 & \multirow{49}{*}{~} & 238 & yogurt-box-bottom & 63
\\
\cline{1-4}
\multirow{15}{*}{Packaging} & 175 & bamboo-mat-edge & 284 & \multirow{49}{*}{~} & 239 & yogurt-box-edge & 62 \\
\multirow{15}{*}{~} & 176 & bamboo-mat-end & 528 & \multirow{49}{*}{~} & 240 & yogurt-box-handle & 2 \\
\multirow{15}{*}{~} & 177 & bamboo-mat-head & 8 & \multirow{49}{*}{~} & 241 & yogurt-box-top & 14 \\
\multirow{15}{*}{~} & 178 & bamboo-mat-middle & 244 & \multirow{49}{*}{~} & 242 & sauce-cup-all & 9 \\
% \cmidrule(r){5-8}
\cline{5-8}
\multirow{15}{*}{~} & 179 & mozzarella-bag-end & 71 & \multirow{14}{*}{Cutlery} & 243 & bowl-bottom & 69 \\
\multirow{15}{*}{~} & 180 & mozzarella-bag-middle & 24 & \multirow{14}{*}{~} & 244 & bowl-edge & 198 \\
\multirow{15}{*}{~} & 181 & onion-bag-end & 86 & \multirow{14}{*}{~} & 245 & glass-bowl-all & 23 \\
\multirow{15}{*}{~} & 182 & onion-bag-middle & 30 & \multirow{14}{*}{~} & 246 & glass-bowl-body & 23 \\
\multirow{15}{*}{~} & 183 & pepperoni-bag-end & 60 & \multirow{14}{*}{~} & 247 & glass-bowl-bottom & 91 \\
\multirow{15}{*}{~} & 184 & pepperoni-bag-middle & 16 & \multirow{14}{*}{~} & 248 & glass-bowl-edge & 415 \\
\multirow{15}{*}{~} & 185 & piping-bag-all & 251 & \multirow{14}{*}{~} & 249 & glass-bowl-handle & 8 \\
\multirow{15}{*}{~} & 186 & pizza-box-edge & 140 & \multirow{14}{*}{~} & 250 & grass-bowl-edge & 13 \\
\multirow{15}{*}{~} & 187 & tea-leaves-bag-body & 38 & \multirow{14}{*}{~} & 251 & green-bowl-edge & 29 \\
\multirow{15}{*}{~} & 188 & tea-leaves-bag-bottom & 42 & \multirow{14}{*}{~} & 252 & small-bowl-edge & 16 \\
\multirow{15}{*}{~} & 189 & tea-leaves-bag-top & 38 & \multirow{14}{*}{~} & 253 & steel-bowl-edge & 153 \\
\cline{1-4}
\multirow{4}{*}{Containers} & 190 & black-bottle-top & 42 & \multirow{14}{*}{~} & 254 & steel-bowl-top & 13 \\
\multirow{4}{*}{~} & 191 & bottle-all & 1 & \multirow{14}{*}{~} & 255 & metal-bowl-edge & 470 \\
\multirow{4}{*}{~} & 192 & bottle-edge & 19 & \multirow{14}{*}{~} & 256 & plastic-bowl-all & 46 \\
\multirow{4}{*}{~} & 193 & bottle-top & 30 & & & & \\
\bottomrule
\end{tabular}}
\end{table}
\begin{table}[!ht]
\centering
\caption{Vocabulary of fine-grained interaction object nouns.}
\label{nounid3}
% \footnotesize
\centering
\resizebox{1.05\linewidth}{!}{
\begin{tabular}{c|lll|c|lll}
\toprule
Super category & ID & Noun & \#Instance & Super category & ID & Noun & \#Instance \\
\hline
\multirow{24}{*}{Cutlery} & 257 & plastic-bowl-body & 10 & \multirow{16}{*}{Kitchenware} & 321 & shovel-body & 49 \\
\multirow{24}{*}{~} & 258 & plastic-bowl-edge & 31 & \multirow{16}{*}{~} & 322 & shovel-handle & 49 \\
\multirow{24}{*}{~} & 259 & porcelain-bowl-edge & 39 & \multirow{16}{*}{~} & 323 & sieve-spoon-body & 13 \\
\multirow{24}{*}{~} & 260 & porcelain-bowl-middle & 2 & \multirow{16}{*}{~} & 324 & sieve-spoon-handle & 12 \\
\multirow{24}{*}{~} & 261 & porcelain-bowl-top & 17 & \multirow{16}{*}{~} & 325 & small-iron-pot-handle & 90 \\
\multirow{24}{*}{~} & 262 & fork-handle & 9 & \multirow{16}{*}{~} & 326 & small-knife-body & 663 \\
\multirow{24}{*}{~} & 263 & iron-spoon-body & 353 & \multirow{16}{*}{~} & 327 & small-knife-handle & 706 \\
\multirow{24}{*}{~} & 264 & iron-spoon-handle & 395 & \multirow{16}{*}{~} & 328 & tea-strainer-body & 108 \\
\multirow{24}{*}{~} & 265 & plastic-scoop-body & 48 & \multirow{16}{*}{~} & 329 & tea-strainer-edge & 4 \\
\multirow{24}{*}{~} & 266 & plastic-scoop-handle & 94 & \multirow{16}{*}{~} & 330 & tea-strainer-handle & 224 \\
\multirow{24}{*}{~} & 267 & plastic-spoon-body & 22 & \multirow{16}{*}{~} & 331 & turnplate-corner & 15 \\
\multirow{24}{*}{~} & 268 & plastic-spoon-handle & 22 & \multirow{16}{*}{~} & 332 & turnplate-edge & 9 \\
\multirow{24}{*}{~} & 269 & spoon-body & 692 & \multirow{16}{*}{~} & 333 & utility-knife-body & 10,419 \\
\multirow{24}{*}{~} & 270 & spoon-handle & 1,011 & \multirow{16}{*}{~} & 334 & utility-knife-handle & 11,157 \\
\multirow{24}{*}{~} & 271 & tablespoon-body & 327 & \multirow{16}{*}{~} & 335 & carrot-peeler-body & 255 \\
\multirow{24}{*}{~} & 272 & tablespoon-handle & 332 & \multirow{16}{*}{~} & 336 & carrot-peeler-handle & 518 \\
\cline{5-8}
\multirow{24}{*}{~} & 273 & teaspoon-body & 51 & \multirow{3}{*}{Appliances} & 337 & grater-body & 70 \\
\multirow{24}{*}{~} & 274 & teaspoon-handle & 191 & \multirow{3}{*}{~} & 338 & grater-handle & 70 \\
\multirow{24}{*}{~} & 275 & wooden-spoon-body & 116 & \multirow{3}{*}{~} & 339 & oven-door-handle & 22 \\
\cline{5-8}
\multirow{24}{*}{~} & 276 & wooden-spoon-handle & 152 & \multirow{20}{*}{Rice\&Flour} & 340 & ball-dough-end & 27 \\
\multirow{24}{*}{~} & 277 & wooden-spatula-body & 25 & \multirow{20}{*}{~} & 341 & ball-dough-head & 24 \\
\multirow{24}{*}{~} & 278 & wooden-spatula-handle & 28 & \multirow{20}{*}{~} & 342 & ball-dough-middle & 173 \\
\multirow{24}{*}{~} & 279 & table-knife-handle & 34 & \multirow{20}{*}{~} & 343 & dough-all & 413 \\
\multirow{24}{*}{~} & 280 & tableware-handle & 11 & \multirow{20}{*}{~} & 344 & dough-flour-all & 117 \\
\cline{1-4}
\multirow{40}{*}{Kitchenware} & 281 & beater-end & 25 & \multirow{20}{*}{~} & 345 & dough-flour-middle & 35 \\
\multirow{40}{*}{~} & 282 & plate-edge & 335 & \multirow{20}{*}{~} & 346 & flat-dough-all & 42 \\
\multirow{40}{*}{~} & 283 & plate-end & 20 & \multirow{20}{*}{~} & 347 & flat-dough-edge & 206 \\
\multirow{40}{*}{~} & 284 & brush-body & 105 & \multirow{20}{*}{~} & 348 & flat-dough-end & 963 \\
\multirow{40}{*}{~} & 285 & brush-handle & 195 & \multirow{20}{*}{~} & 349 & flat-dough-middle & 13 \\
\multirow{40}{*}{~} & 286 & can-opener-edge & 83 & \multirow{20}{*}{~} & 350 & flour-all & 35 \\
\multirow{40}{*}{~} & 287 & can-opener-end & 25 & \multirow{20}{*}{~} & 351 & green-dough-all & 123 \\
\multirow{40}{*}{~} & 288 & ceramic-plate-all & 91 & \multirow{20}{*}{~} & 352 & little-dough-al & 1 \\
\multirow{40}{*}{~} & 289 & ceramic-plate-edge & 208 & \multirow{20}{*}{~} & 353 & little-dough-all & 386 \\
\multirow{40}{*}{~} & 290 & chopping-board-edge & 2 & \multirow{20}{*}{~} & 354 & oval-dough-end & 170 \\
\multirow{40}{*}{~} & 291 & cooking-spoon-body & 77 & \multirow{20}{*}{~} & 355 & rice-all & 585 \\
\multirow{40}{*}{~} & 292 & cooking-spoon-handle & 382 & \multirow{20}{*}{~} & 356 & slice-bread-end & 92 \\
\multirow{40}{*}{~} & 293 & food-mixer-handle & 372 & \multirow{20}{*}{~} & 357 & bread-end & 510 \\
\multirow{40}{*}{~} & 294 & food-mixer & 369 & \multirow{20}{*}{~} & 358 & bread-head & 199 \\
\multirow{40}{*}{~} & 295 & pizza-cutter-body & 29 & \multirow{20}{*}{~} & 359 & bread-middle & 258 \\
\cline{5-8}
\multirow{40}{*}{~} & 296 & pizza-cutter-handle & 17 & \multirow{17}{*}{Dessert} & 360 & chocolate-cake-edge & 242 \\
\multirow{40}{*}{~} & 297 & food-plate-center & 75 & \multirow{17}{*}{~} & 361 & chocolate-cake-top & 111 \\
\multirow{40}{*}{~} & 298 & food-plate-edge & 36 & \multirow{17}{*}{~} & 362 & chocolate-all & 21 \\
\multirow{40}{*}{~} & 299 & handle-all & 108 & \multirow{17}{*}{~} & 363 & candies-all & 159 \\
\multirow{40}{*}{~} & 300 & iron-plate-edge & 76 & \multirow{17}{*}{~} & 364 & candy-all & 4 \\
\multirow{40}{*}{~} & 301 & kitchen-knife-body & 552 & \multirow{17}{*}{~} & 365 & candy-top & 51 \\
\multirow{40}{*}{~} & 302 & kitchen-knife-handle & 1,215 & \multirow{17}{*}{~} & 366 & chocolate-bar-middle & 87 \\
\multirow{40}{*}{~} & 303 & knife-body & 26 & \multirow{17}{*}{~} & 367 & chocolate-chips-all & 49 \\
\multirow{40}{*}{~} & 304 & knife-handle & 27 & \multirow{17}{*}{~} & 368 & chocolate-cream-all & 189 \\
\multirow{40}{*}{~} & 305 & jam-knife-body & 50 & \multirow{17}{*}{~} & 369 & chocolate-cream-top & 261 \\
\multirow{40}{*}{~} & 306 & jam-knife-handle & 53 & \multirow{17}{*}{~} & 370 & chocolate-donut-all & 35 \\
\multirow{40}{*}{~} & 307 & metal-plate-edge & 47 & \multirow{17}{*}{~} & 371 & cookie-all & 41 \\
\multirow{40}{*}{~} & 308 & metal-spatula-body & 71 & \multirow{17}{*}{~} & 372 & cookie-end & 57 \\
\multirow{40}{*}{~} & 309 & metal-spatula-handle & 123 & \multirow{17}{*}{~} & 373 & cookie-head & 1 \\
\multirow{40}{*}{~} & 310 & pizza-spatula-body & 70 & \multirow{17}{*}{~} & 374 & cookies-all & 49 \\
\multirow{40}{*}{~} & 311 & pizza-spatula-handle & 99 & \multirow{17}{*}{~} & 375 & cookie-top & 9 \\
\multirow{40}{*}{~} & 312 & pizza-tray-edge & 122 & \multirow{17}{*}{~} & 376 & cream-all & 139 \\
\cline{5-8}
\multirow{40}{*}{~} & 313 & plastic-spatula-body & 328 & \multirow{3}{*}{Drink} & 377 & tea-all & 14 \\
\multirow{40}{*}{~} & 314 & plastic-spatula-handle & 531 & \multirow{3}{*}{~} & 378 & tea-leaves-all & 219 \\
\multirow{40}{*}{~} & 315 & rolling-pin-body & 198 & \multirow{3}{*}{~} & 379 & whisk-head-top & 9 \\
% \cmidrule(r){5-8}
\cline{5-8}
\multirow{40}{*}{~} & 316 & rolling-pin-handle & 174 & \multirow{4}{*}{Uncategorised} & 380 & hand-left & 23,305 \\
\multirow{40}{*}{~} & 317 & rolling-pin-middle & 82 & \multirow{4}{*}{~} & 381 & hand-right & 26,441 \\
\multirow{40}{*}{~} & 318 & rolling-pin-miidle & 5 & \multirow{4}{*}{~} & 382 & towel-all & 82 \\
\multirow{40}{*}{~} & 319 & serrated-knife-body & 38 & \multirow{4}{*}{~} & 383 & towel-edge & 38 \\
\multirow{40}{*}{~} & 320 & serrated-knife-handle & 43 & & & & \\
\bottomrule
\end{tabular}}
\end{table}
\begin{figure}[!t]
\centering
\rotatebox{270}{\includegraphics[width=1.5\linewidth]{objectnoundistribution.pdf}}
\caption{The distribution of instances per object noun category from 17 super-categories in the FHA-Kitchens dataset.}
\label{objectnoundistribution1}
\end{figure}
\end{document}