GraphQ IR: Unifying the Semantic Parsing of Graph Query Languages with One Intermediate Representation
Extracted source text · arXiv 2205.12078
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% Remove the "review" option to generate the final version.
% For the final version, omit the review option
\usepackage[]{acl}
%\usepackage{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% User-specified imported packages
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{multirow}
\usepackage{lipsum}
\usepackage{indentfirst}
\usepackage{amsmath}
\usepackage{dsfont}
\usepackage{relsize}
\usepackage{svg}
\usepackage{amsthm}
\usepackage{xspace}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{listings}
\usepackage{threeparttable}
\usepackage{array}
\usepackage{float}
\usepackage{ragged2e}
\usepackage{xcolor}
\newcommand{\red}[1]{\textcolor{red}{#1}}
\newcommand{\add}[1]{\textcolor{black}{#1}}
\newcommand{\tool}{{GraphQ IR}\xspace}
\newcommand{\toolx}{{GraphQ IR*}\xspace}
\newcommand{\tools}{{GraphQ IR's}\xspace}
\newcommand{\eg}{\textit{e.g.},\xspace}
\newcommand{\ie}{\textit{i.e.},\xspace}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
% \title{Unifying the Semantic Parsing of Graph Query Language with One Intermediate Representation}
\title{GraphQ IR: Unifying Semantic Parsing of Graph Query Languages \\with Intermediate Representation}
% Author information can be set in various styles:
% For several authors from the same institution:
\author{
\textbf{Lunyiu Nie}$^{1}$, \textbf{Shulin Cao}$^{1}$, \textbf{Jiaxin Shi}$^{2}$, \textbf{Tian Qi}$^{2}$, \textbf{Lei Hou}$^{1}$, \textbf{Juanzi Li}$^{1}$, \textbf{Jidong Zhai}$^{1}$ \\
$^1$ Department of Computer Science and Technology, Tsinghua University \\
$^2$ Huawei Cloud Computing Technologies Co., Ltd. \\
\texttt{\{nlx20, caosl19\}@mails.tsinghua.edu.cn},\hspace{.5em}\texttt{shijx12@gmail.com}, \\
\texttt{tian.qi1@huawei.com},\hspace{.5em}\texttt{\{houlei,lijuanzi,zhaijidong\}@tsinghua.edu.cn}
}
% if the names do not fit well on one line use
% Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \And ... \And
% Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \AND
% Author 2 \\ Address line \\ ... \\ Address line \And
% Author 3 \\ Address line \\ ... \\ Address line}
% \author{First Author \\
% Affiliation / Address line 1 \\
% Affiliation / Address line 2 \\
% Affiliation / Address line 3 \\
% \texttt{email@domain} \\\And
% Second Author \\
% Affiliation / Address line 1 \\
% Affiliation / Address line 2 \\
% Affiliation / Address line 3 \\
% \texttt{email@domain} \\}
\begin{document}
\maketitle
\begin{abstract}
Subject to the semantic gap \add{lying} between natural and formal language, neural semantic parsing is typically bottlenecked by the paucity and imbalance of data. In this paper, we propose a unified intermediate representation (IR) for graph query languages, namely \tool. With the IR's natural-language-like representation that bridges the semantic gap and \add{its formally defined syntax that maintains the graph structure}, neural semantic parser can more effectively convert user queries into our \tool, which can be later automatically compiled into different downstream graph query languages. Extensive experiments show that our approach can consistently achieve state-of-the-art performance on benchmarks \textsc{Kqa Pro}, \textsc{Overnight} and \textsc{MetaQA}. Evaluations under compositional generalization and few-shot learning settings also validate the promising generalization ability of \tool with at most 11\% accuracy improvement.
\end{abstract}
\input{sections/1_intro}
\input{sections/2_principles}
\input{sections/3_implementation}
\input{sections/4_exp}
\input{sections/5_discussion}
\input{sections/6_related}
\input{sections/7_conclu}
% Entries for the entire Anthology, followed by custom entries
\bibliography{ref}
\bibliographystyle{acl_natbib}
\clearpage
\input{sections/appendix}
\end{document}
\section{Introduction}
By mapping natural language utterances to logical forms, the task of semantic parsing has been widely explored in various applications like querying database \cite{zhong2017seq2sql, yu2018spider} or knowledge base \cite{zhang2018variational, talmor2018web}, virtual assistant conversation \cite{campagna2019genie, fischer2021diy} and general-purpose code generation \cite{ling2016latent, yin2017syntactic, nan2020hisyn}.
Early attempts in this field usually rely on compositional grammar such as CCG and DCS \cite{zettlemoyer2005learning, kwiatkowksi2010inducing, liang2013learning}. Recently, most works formulate semantic parsing as a Seq2Seq problem and adopt encoder-decoder neural networks that only require parallel data of natural language utterance and corresponding logical form for supervision \cite{zhong2017seq2sql, yu2020grappa, damonte2021one}.
\add{However, these neural approaches still suffer from two major challenges}: (a) \textbf{Semantic gap.} As shown in Figure \ref{fig:examples}, in graph query languages (\eg SPARQL, Cypher, and newly emerged KoPL, etc.), graph nodes and edges constitute the key semantics of the logical forms \cite{perez2009semantics}, which are very different from the expression of natural language queries. Such discrepancy significantly hinders the learning of neural semantic parser, especially in the era of pretrained language models \cite{lewis2020bart}. (b) \textbf{Imbalance of data.} Due to the intensive labor and language-specific expertise required in annotation \cite{li2020context}, in spite of the various datasets prepared in SPARQL \cite{talmor2018web, dubey2019lc, keysers2019measuring}, very few works target the semantic parsing of other graph query languages, such as Cypher and Gremlin, that are commonly used in industries \cite{seifer2019empirical}. Moreover, datasets of different languages are also isolated since no existing tools can support the conversion \cite{a2022sparql}. Such imbalance and isolation of data has impeded the semantic parsing of low-resource languages in both academia and industry.
% However, these neural approaches still suffer from two major challenges: (a) \textbf{Semantic gap.} As shown in Figure \ref{fig:examples}, the key semantics of graph query languages (\eg SPARQL, Cypher, etc.) is mainly constituted by the graph nodes and edges~\cite{perez2009semantics}, which is very different from the expression of natural language queries. Such discrepancy significantly hinders the learning of neural semantic parser, especially in the era of pretrained language models~\cite{lewis2020bart}.
% (b) \textbf{Imbalance of data.} Various works annotate datasets for SPARQL \cite{talmor2018web, dubey2019lc, keysers2019measuring}, whereas little exist for other graph query languages like Cypher and Gremlin, that are widely used in industries \cite{seifer2019empirical}. The isolation of different datasets and data paucity of low-resourced languages hinders both academia and industry.
% To mitigate the gap and overcome the bottleneck, many works adopt alternate forms of supervision that are cheaper to acquire, such as the schema of database \cite{hwang2019comprehensive}, execution results of the logical forms \cite{clarke2010driving, wang2018robust, wang2021learning}, and auxiliary grammars for constraining the decoding space \cite{shin2021constrained, baranowski2021grammar}. More recently, the canonical utterances generated in data preparation procedure has attracted increasing attention of the community, because of two advantages: 1) canonical utterances can be automatically generated from pre-defined grammar rules at low costs \cite{berant2014semantic, yu2020grappa}; and 2) canonical utterances have representation similar to natural language, as demonstrated in Figure \ref{fig:examples}, thus can be utilized as a middleware for bridging natural and formal language \cite{su2017cross}.
To overcome the above challenges, many works adopt complementary forms of supervision, such as the schema of database \cite{hwang2019comprehensive}, execution results of the logical forms \cite{clarke2010driving, wang2018robust, wang2021learning}, constrained decoding algorithms \cite{shin2021constrained, marion2021structured} and canonical utterances \cite{berant2014semantic, su2017cross, yu2020grappa}. Despite effective, these approaches either incur training inefficiency or performance loss \cite{cao2019semantic}. Besides, all these methods are tightly coupled to a specific dataset or logical form, thus cannot be easily adapted to other tasks or languages \cite{kamath2018survey}.
\begin{figure}[t]
\centering
\resizebox{\linewidth}{!}{
\includegraphics{figures/examples.pdf}
}
\caption{An example of a property graph extracted from Wikidata \cite{vrandevcic2014wikidata}. We present a relevant user query with its corresponding logical forms in different languages and \tool. }
\label{fig:examples}
\end{figure}
In this paper, we propose a unified intermediate representation for graph query languages, namely \tool, to resolve these issues from a novel perspective. Instead of directly mapping the input utterance to the target logical form, we first translate natural language into \tool, then compile it into the target graph query language (\eg SPARQL, KoPL, $\lambda$-DCS, Cypher, etc.). The designs of \tool weigh up the semantics of both natural and formal language by: \add{(a) producing the IR sequences with word order rules consistent to English \cite{tomlin2014basic}, such that the compositional semantics of IR can be more easily aligned with the natural language utterance so as to close the semantic gap; and (b) capturing the fundamental graph structures like nodes, edges and properties, such that the IR can be losslessly compiled into any downstream graph query languages so as to unify the datasets of different languages for alleviating the data bottleneck. }
Therefore, \add{language-specific grammar features that originally pose a huge obstacle to semantic parsing are now explicitly handled by the compiler, and the neural semantic parser can concentrate on the comprehension of user queries.} Additionally, with our source-to-source compiler that supports bi-directional translation between \tool and various graph query languages, our work also provides a novel toolkit that can be utilized as a transpiler for unifying different datasets or as a natural language user interface for querying graph databases.
%Instead of directly mapping the input utterance to the target logical form, we train the neural semantic parser to translate natural language into an IR whose semantic representations can be easily aligned with the user utterance. The generated IR sequence that preserves necessary syntactic structures can be thereafter automatically compiled into any graph query languages (\eg SPARQL, KoPL, $\lambda$-DCS, Cypher, etc.). With the \tool as a bridge, separate datasets that are originally prepared in different graph query languages can be now linked together as an integrated solution for alleviating the data bottleneck.
To validate the effectiveness of \tool, we conducted extensive experiments on benchmarks \textsc{Kqa Pro}, \textsc{Overnight} and \textsc{MetaQA}. Results show that our approach can consistently outperforms the previous works by a significant margin. Especially under the compositional generalization and few-shot learning settings, our approach with \tool \add{can demonstrate at most 11\% increase on accuracy} over the baselines with strong generalization abilities. \add{Supplementary analysis further shows that \tool is easy to debug with 89\% of the errors can be fixed with simple corrections.}
The main contributions of our work include:
\begin{itemize}
\itemsep0em
\item We propose \tool for unifying the semantic parsing of graph query languages, and present the IR design principles that are critical to the success of semantic parsing;
\item Experimental results show that our approach can achieve state-of-the-art performance across benchmarks and strong robustness even under the compositional generalization and few-shot learning settings.
\item Our implemented compiler can be also utilized as a translator among different graph query languages. We will release our code and toolkit for the uses of the community.
\end{itemize}
% The main contributions of our work include: (a) We propose the idea of unifying the semantic parsing among different logical forms, and present the design principles for intermediate representation that are critical in the success of semantic parsing; (b) We applied those principles in designing a unified intermediate representation for graph query languages. Experiment results have validated that our \tool can achieve state-of-the-art performance across benchmarks and strong generalization even under compostional generalization and few-shot settings. (c) Additionally as a unified source-to-source compiler for graph query languages, our work can be also utilized by future researchers for achieving lossless translation among different logical forms.
\input{sections/tab_grammar}
\section{\tool}
\add{We propose \tool as a novel intermediate representation that aims to bridge the semantic gap between natural and formal language as well as unify different languages to break the data bottleneck.
% However, how to improve the alignment between IR and user query, and ensure the deterministic transformation from IR to downstream languages?
In this section, we define the property graph, formalize the \tool, and summarize the key principles in designing \tool.}
\subsection{Definition}
\label{sec:grammar}
As the top of Figure \ref{fig:examples} demonstrates, a property graph normally includes \textbf{Entity} (the graph nodes, \eg Stanley Kubrick), \textbf{Attribute} (the node properties, \eg date of birth), \textbf{Concept} (the node types, \eg film), \textbf{Relationship} (the graph edges, \eg spouse) and \textbf{Qualifier} (the edge properties, \eg start time).
Accordingly, \tool is built on top of the above key structures as terminal nodes meantime having its productions consistent with the compositional semantics of natural language. We formally define the context-free grammar of \tool by a quadruple $G=(V,T,S,P)$ composed of non-terminal symbols $V$, terminal symbols $T$, the start symbol $S$ and production rules $P$. We specify the productions of \tool in extended Backus–Naur form \cite{parr2013definitive} and describe a subset of \tools grammar in Table \ref{tab:grammar}. To exemplify, in Figure \ref{fig:examples}, the production of \tools \texttt{EntitySet} is equivalent to a relative clause involving the operations over graph elements \texttt{Concept}, \texttt{Relation} and \texttt{Entity}.
\subsection{Principles}
We summarize several principles in designing \tool, concluded as: presenting the semantics close to the natural language while preserving the structure identical to the formal language.
\subsubsection{Diminishing structural discrepancy}
To facilitate the training of neural semantic parser, the target IR sequence should share similar structures in correspondence to the input utterance.
To achieve this, first, the structure of IR should match how users typically raise queries. Therefore, we simplify the triple-based structure in graph query languages into a more natural subject-verb-object syntactic construction \cite{tomlin2014basic}. Take Figure \ref{fig:examples}'s task setting as an example, the two triples \texttt{(?e instance\_of ?c)} and \texttt{(?c name "film")} as the entity concept constraint in SPARQL are simplified to the sentence subject ``\texttt{<C>} \textit{film} \texttt{</C>}'' in \tool. Multi-hop relationship and attribute queries are also formulated as relative clauses following the English expression thus can be comfortably generated by a language-model-based neural semantic parser.
Secondly, IR should also leave out the variables (\eg \texttt{?e}, \texttt{?c} in SPARQL) and operators (\eg \texttt{MATCH}, \texttt{DISTINCT}, \texttt{WHERE}, \texttt{RETURN}, etc.) in graph query languages that cannot be easily aligned to natural language. Alternatively, human-readable operators are adopted in \tool for better comprehension by the language models.
\begin{figure*}[ht!]
\centering
\includegraphics[width=\textwidth]{figures/main.pdf}
\caption{Overall implementation of our proposed framework. The user queries are first converted to \tool sequences and subsequently translated into the downstream graph query languages by the compiler. }
\label{fig:main}
\end{figure*}
\subsubsection{Eliminating semantic redundancy and ambiguity}
In formal languages it is quite common that multiple parallel implementations may achieve the same functionalities. However, such redundancy and ambiguity in semantics may pose challenges to the neural semantic parser.
For instance, in $\lambda$-DCS, there co-exists at least three implementations for constraining the entities' concept, respectively through
\noindent
\textbf{\textit{EventNP}}: \verb|(call @getProperty (NP)| \verb| (call @reverse (RelNP)))|;
\noindent
\textbf{\textit{TypeNP}}: \texttt{(call @getProperty (call @singleton (TypeNP)) (string !type))}; and
\noindent
\textbf{\textit{DomainNP}}: \verb|(call @getProperty ((CP)| \verb|(call @domain (RelNP)))(RelNP))|.
\vspace{0.15cm}
% another example of Qualifier in SPARQL
When designing an IR, such redundant and ambiguous semantics should be clarified into more definitive and orthogonal representation \cite{campagna2019genie}. Thus in \tools, we unify all such unnecessary distinctions and prune redundant structure in the logical forms to distill the core semantics. In the previous example, \tool instead only requires a simple concept constraint ``\texttt{<C>} \textit{concept} \texttt{</C>}'' as the noun modifier. This not only makes the language more regular and hence simpler for both the users and semantic parser, but also facilitates the next-step compilation from IR to the downstream formal languages.
\subsubsection{Inducing compilation-aiding syntax}
Apart from the aforementioned designs to improve the alignment with natural language, the grammar of IR also needs to maintain the key structures in graph queries so as to enable the subsequent automatic compilation.
% strong and static typing
Specifically, IR should keep track of the data types of graph structural elements. We design \tool to be strong-typing by explicitly stating the type of terminal nodes with respective special tokens, \eg \texttt{<E>} for \texttt{Entity}, \texttt{<R>} for \texttt{Relation}, \texttt{<A>} for \texttt{Attribute}, etc. Values of different types are also differentiated in \tool with our pre-defined or user custom indicators, \eg \texttt{string}, \texttt{number}, \texttt{date}, \texttt{time}, etc.
% hierarchical scoping
Furthermore, IR should also preserve the hierarchical dependencies that are critical for multi-hop queries. We induce \texttt{<ES>} as a scoping token in \tool to explicitly indicate the underlying dependencies among the clauses produced by an \texttt{EntitySet}, as shown in Table \ref{tab:grammar}. Such scoping tokens in \tool can facilitate the compiler to recover the hierarchical structure and finally transform the IR sequences into one of the graph query languages deterministically.
\begin{table*}[t!]
\centering
\small
\resizebox{\textwidth}{!}{
\begin{tabular}{p{0.175\textwidth}>{\raggedright}p{0.55\textwidth}>{\raggedright\arraybackslash}p{0.3\textwidth}}
\toprule
\textbf{Non-terminal} & \textbf{Productions} & \textbf{Description} \\
\midrule
$S$ &$\rightarrow$ \texttt{EntityQuery} | \texttt{AttributeQuery} | \texttt{RelationQuery} | \texttt{QualifierQuery} | \texttt{ValueQuery} | \texttt{CountQuery} & Start symbol of a query sequence \\
\hline
\texttt{EntityQuery} &$\rightarrow$ \textit{what is} \texttt{EntitySet} & Query that returns certain entities \\
\hline
\texttt{RelationQuery} &$\rightarrow$ \textit{what is the relation from} \texttt{EntitySet} \textit{to} \texttt{EntitySet} & Query that returns the relationship between given entities \\
\hline
\texttt{CountQuery} &$\rightarrow$ \textit{how many} \texttt{EntitySet} & Query that returns the quantity of certain entities \\
\hline
\texttt{EntitySet} &$\rightarrow$ \texttt{<ES> EntitySet LOP EntitySet </ES>} | \texttt{<ES>} \texttt{EntitySet Constraint} \texttt{</ES>} | \texttt{Concept} | \texttt{Entity} & A non-empty set of entities that match certain conditions \\
\hline
\texttt{Constraint} &$\rightarrow$ \textit{whose} \texttt{Attribute COP Value} | \textit{that} \texttt{Relation DIR} \textit{to} \texttt{EntitySet} | \textit{that} \texttt{Relation DIR} \textit{to} \texttt{COP} \texttt{Value} \texttt{EntitySet} & Clauses that constrain the entities condition \\
\hline
\texttt{Entity} &$\rightarrow$ \texttt{<E>} \textit{entity} \texttt{</E>} & Entity node of a specified name \\
\hline
\texttt{Concept} &$\rightarrow$ \texttt{<C>} \textit{concept} \texttt{</C>} & Set of entities of a specified type\\
\hline
\texttt{Attribute} &$\rightarrow$ \texttt{<A>} \textit{attribute} \texttt{</A>} & Specified property of an entity \\
\hline
\texttt{Relation} &$\rightarrow$ \texttt{<R>} \textit{relation} \texttt{</R>} & Relationship between entities \\
\hline
\texttt{Value} &$\rightarrow$ \texttt{VTYPE <V>} \textit{value} \texttt{</V>} | \texttt{Attribute} \textit{of} \texttt{Entity} & Specified value of certain constraints \\
\hline
\texttt{LOP} &$\rightarrow$ \textit{and} | \textit{or} | \textit{not} & Logical operators that perform set union, intersection or exclusion \\
\hline
\texttt{COP} &$\rightarrow$ \textit{is} | \textit{is not} | \textit{larger than} | \textit{smaller than} | \textit{at least} | \textit{at most} & Comparison operators that take value as operand \\
\hline
\texttt{DIR} &$\rightarrow$ \textit{forward} | \textit{backward} & Relationship direction constraint \\
\hline
\texttt{VTYPE} &$\rightarrow$ \textit{string} | \textit{number} | \textit{year} | \textit{date} | \textit{time} & Value type constraint \\
\bottomrule
\end{tabular}
}
\caption{A subset of \tool grammar rules. We separate alternative productions at the same level using ``|'' and mark the terminal nodes in italics. A more complete set of grammar is presented in Appendix Table \ref{tab:full_grammar}. }
\label{tab:grammar}
\end{table*}
\section{Implementation}
\label{sec:imple}
% We depict the whole picture of our proposed framework in Figure \ref{fig:main}. The neural semantic parser first translates the natural language question into \tool, which is thereafter fed into the compiler and parsed into an abstract syntax tree for downstream graph query language code generation.
We depict the whole picture of our proposed framework in Figure \ref{fig:main}. The neural semantic parser first translates the input natural language utterance into \tool. Thereafter, the \tool sequence is fed into the compiler and parsed into an abstract syntax tree for downstream graph query language code generation.
\subsection{Neural Semantic Parser}
\label{sec:parser}
To verify the above principles in practice, we formulate the conversion from natural language to our \tool as a Seq2Seq task and adopt an encoder-decoder framework for implementing the neural semantic parser.
As shown in the left part of Figure \ref{fig:main}, the encoder module of the semantic parser first maps the input natural language utterance $X$ to a high dimensional feature space with non-linear transformations for capturing the dependencies among the input tokens. The decoder module subsequently then interprets the hidden representations and generates the IR sequence by factorizing the probability distribution:
\begin{equation}
p(\textit{IR}) = \prod_{i=1}^{n} p(y_i|X,y_1,...,y_{i-1}),
\end{equation}
where $y_i$ is the $i$-th token of IR sequence with in total $n$ tokens. Specifically, we implement this encoder-decoder network with BART, which has a standard Transformer architecture that get pretrained on over 160GB of English text corpus \cite{lewis2020bart}. Therefore, BART is proficient in comprehending the diverse user utterances and generating the \tool that has natural-language-like semantic representations.
Please note that the implementation in this part is orthogonal to our \tool, thus can be also substituted by other semantic parsing models.
\subsection{Compiler}
The implementation of compiler comprises a front-end module that generates abstract syntax trees from \tool and a back-end module that transforms the tree structure into the target code.
Following the grammar rules formally defined in Section \ref{sec:grammar}, the compiler front-end first performs lexical analysis and syntax analysis on the IR sequence generated in Section \ref{sec:parser}. The LL(*) parser that performs the leftmost derivation of a sentence with tokens look-ahead \cite{parr2011ll} will automatically parse the sequence of \tool into an abstract syntax tree that contains the dependencies and hierarchical structure. The compiler back-end will then traverse the abstract syntax tree and restructure the elements and dependencies into one of the downstream graph query languages (\ie SPARQL, KoPL, $\lambda$-DCS, Cypher, etc) following our pre-defined transformation rules. To illustrate this process, we present 2 examples of generating $\lambda$-DCS and SPARQL queries respectively in Appendix Figure \ref{fig:ast_1} and Figure \ref{fig:ast_2}.
% As an essential part of our work, this source-to-source compiler that can achieve bidirectional translation between \tool and graph query languages will be open source for the use of other researchers.
\section{Experiments}
This section evaluates the effectiveness of \tool on several benchmarks under different task settings.
\begin{table*}[ht!]
\centering
\small
\resizebox{\textwidth}{!}{
\begin{tabular}{lcccccccc}
\toprule
& \textbf{Multi-hop} & \textbf{Qualifier} & \textbf{Comparison} & \textbf{Logical} & \textbf{Count} & \textbf{Verify} & \textbf{Zero-shot} & \textbf{Overall} \\
\midrule
\textbf{Baselines} & & \\
RGCN \cite{schlichtkrull2018rgcn} & 34.00 & 27.61 & 30.03 & 35.85 & 41.91 & 65.88 & - & 35.07\\
BART+KoPL \cite{shi2020kqa} & 85.57 & 80.57 & 93.83 & 86.33 & 84.20 & 84.94 & 86.29 & 87.17\\
BART+SPARQL \cite{shi2020kqa} & 88.49 & 83.09 & 96.12 & 88.67 & 85.78 & 92.33 & 87.88 & 89.68 \\
CFQ IR \cite{herzig2021unlocking} & 87.51 & 81.32 & 95.70 & 90.33 & 86.23 & 92.20 & 87.12 & 88.96 \\
\hline
\textbf{Our Approach} & & \\
\tool & \textbf{90.38} & \textbf{84.90} & \textbf{97.15} & \textbf{92.64} & \textbf{89.39} & \textbf{94.20} & \textbf{94.20} & \textbf{91.70}\\
% \toolx & & \\
\bottomrule
\end{tabular}
}
\caption{Experimental results on \textsc{Kqa Pro} dataset. Data are categorized into \textsc{Multi-hop} queries with multi-hop inference, \textsc{Qualifier} knowledge queries, \textsc{Comparison} between several entities, \textsc{Logical} union or intersection, \textsc{Count} queries for the quantity of entities, \textsc{Verify} queries with a boolean answer, and \textsc{Zero-shot} queries whose answer is not seen in the training set.}
\label{tab:kqapro}
\end{table*}
\begin{table*}[ht!]
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{p{0.375\textwidth}>{\centering}p{0.0725\textwidth}>{\centering}p{0.0725\textwidth}>{\centering}p{0.0725\textwidth}>{\centering}p{0.0725\textwidth}>{\centering}p{0.0725\textwidth}>{\centering}p{0.0725\textwidth}>{\centering}p{0.0725\textwidth}>{\centering}p{0.0725\textwidth}>{\centering\arraybackslash}p{0.075\textwidth}}
\toprule
& \textbf{Bas.} & \textbf{Blo.} & \textbf{Cal.} & \textbf{Hou.} & \textbf{Pub.} & \textbf{Rec.} & \textbf{Res.} & \textbf{Soc.} & \textbf{Overall} \\
\midrule
\textbf{Baselines} \\
SPO \cite{wang2015building} & 46.3 & 41.9 & 74.4 & 54.0 & 59.0 & 70.8 & 75.9 & 48.2 & 58.8 \\
Seq2Action \cite{chen2018sequence} & 88.2 & 61.4 & 81.5 & 74.1 & 80.7 & 82.9 & 80.7 & 82.1 & 79.0 \\
DUAL \cite{cao2019semantic} & 84.9 & 61.2 & 78.6 & 67.2 & 78.3 & 80.6 & 78.9 & 81.3 & 76.4 \\
2-stage DUAL* \cite{cao2020unsupervised} & 87.2 & \textbf{65.7} & 80.4 & 75.7 & 80.1 & 86.1 & 82.8 & 82.7 & 80.1 \\
\hline
\textbf{Our Approach} \\
\tool & 88.2 & 64.7 & 78.6 & 72.0 & 77.6 & 83.3 & 84.9 & 81.6 & 79.5 \\
\toolx & \textbf{88.2} & 65.4 & \textbf{81.6} & \textbf{81.5} & \textbf{82.6} & \textbf{92.9} & \textbf{89.8} & \textbf{84.1} & \textbf{82.1} \\
\bottomrule
\end{tabular}
}
\caption{Experimental results on \textsc{Overnight} dataset. Methods with asterisk (*) means that external data of other domains is utilized to enhance the performance.}
\label{tab:overnight}
\end{table*}
\subsection{Datasets}
For evaluation, we choose SPARQL, KoPL, $\lambda$-DCS, and Cypher as the target graph query languages, and take \textsc{Kqa Pro}, \textsc{Overnight}, \textsc{MetaQA} as the corresponding datasets.
\paragraph{KQA Pro} \textsc{Kqa Pro}~\cite{shi2020kqa} is a large-scale dataset for complex question answering over Wikidata knowledge base~\cite{vrandevcic2014wikidata}. It is so far the largest KBQA corpus that contains 117,790 natural language questions along with the corresponded SPARQL and KoPL logical forms, covering diverse question types, \eg multi-hop inference, logical union and intersection, etc.
%It was constructed by first synthesizing canonical questions, KoPLs and SPARQLs with templates, and then rewriting canonical questions into natural language via crowdsourcing.
In our experiment, it is divided into 94,376 train, 11,797 dev and 11,797 test cases.
\paragraph{Overnight} \textsc{Overnight}~\cite{wang2015building} is a semantic parsing dataset with 13,682 examples across 8 different domains extracted from Freebase \cite{bollacker2008freebase}. Each domain has natural language questions and pairwise $\lambda$-DCS queries executable on SEMPRE \cite{berant2013semantic}. It exhibits diverse linguistic phenomena and semantic structures across domains, e.g., temporal knowledge in \textsc{Calendar} domain and spatial knowledge in \textsc{Blocks} domain. We use the same train/dev/test splits as in the previous work~\cite{wang2015building}.
\paragraph{MetaQA} \textsc{MetaQA}~\cite{zhang2018variational} is a large-scale dataset containing more than 400k multi-hop question-answer pairs over MovieQA knowledge base \cite{tapaswi2016movieqa}. Since previous works have achieved $\sim$100\% accuracy on its SPARQL annotation \cite{huang2021unseen, shi2021transfernet}, we reconstruct \textsc{MetaQA} into Cypher as a few-shot learning benchmark. To the best of our knowledge, this is also the first Cypher dataset in the research field of graph query language parsing.
% consists of question-answer
% pairs in the domain of movies
\subsection{Metric}
We adopt \textit{execution accuracy} as our metric \add{based on whether the generated logical forms can return correct answers.} For queries with multiple answers, we require the execution results to exactly match all the ground truth answers.
\subsection{Configurations}
For the neural semantic parser, we used the BART$_{base}$ model \cite{lewis2020bart} released by Facebook on HuggingFace\footnote{https://huggingface.co/facebook/bart-base}. 12 special tokens (\eg \texttt{<ES>}) were added to the tokenizer vocabulary as the structure indicators for \tool. We used the AdamW optimizer \cite{loshchilov2018fixing} with the learning rate set to $3e^{-5}$ and weight decay set to $1e^{-5}$ following the default settings.
% and searched the learning rate in $\{1e^{-4}, 3e^{-5}, 1e^{-5}\}$, and the weight decay rate in $\{1e^{-4}, 1e^{-5}, 1e^{-6}\}$. According to the performance on validation set, we finally
For the compiler, we used ANTLR \cite{parr2013definitive} version 4.9.2 for analyzing our specified grammar rules and building up the corresponding lexer and parser toolkit. For evaluation, we used Virtuoso\footnote{https://github.com/openlink/virtuoso-opensource} 7.20, SEMPRE\footnote{https://github.com/percyliang/sempre} 2.4, Neo4j\footnote{https://github.com/neo4j/neo4j} 4.4 and KoPL 0.3\footnote{https://pypi.org/project/KoPL/} as the back-ends respectively for executing the SPARQL, $\lambda$-DCS, Cypher and KoPL queries.
Our whole experiments were performed on a single machine with 8 NVIDIA Tesla V100 (32GB memory) GPUs on CUDA 11.
\subsection{Results}
As Table \ref{tab:kqapro} illustrates, our proposed approach with \tool consistently outperforms the previous approaches and has achieved the new state-of-the-art results on \textsc{Kqa Pro} dataset. In particular, \tool exhibits even larger improvements over the baselines under the more complex \textsc{Multi-hop}, \textsc{Qualifier} and \textsc{Zero-shot} task settings. We attribute this to its natural-language-like semantic representations that can be well accepted by the pretrained language model and its formal-language-like syntactic structure that can be flexibly combined or decomposed to achieve better generalization.
As for the benchmark \textsc{Overnight}, our approaches with \tool also significantly surpass the baselines, as shown in Table \ref{tab:overnight}. Due to the distinct vocabulary and grammar rules, previous works usually train separate parsers for each task domain \cite{wang2015building, chen2018sequence}. However, as can be observed from the superior performance of \toolx that get trained on the aggregated data of all eight domains, with an extra layer of IR for unification, domain-specific grammar rules are now consolidated into one universal representation and the training of one domain can thereby benefit from the others' data.
To further validate our proposed IR's robustness, we also examine \tool under compositional generalization and few-shot learning task settings.
\begin{table}[t]
\centering
\small
\resizebox{\columnwidth}{!}{
\begin{tabular}{lccccc}
\toprule
& \textbf{Overall} &\textbf{Qualifier} & \textbf{Comparison} &\textbf{Logical} \\
\midrule
BART & 50.58 & 21.55 & 87.66 & 50.60 \\
CFQ IR & 50.70 & 25.33 & 93.77 & 50.73 \\
\tool & \textbf{54.91} & \textbf{40.46} & \textbf{95.19} & \textbf{54.90} \\
\bottomrule
\end{tabular}
}
\caption{Experimental results on \textsc{Kqa Pro} compositional generalization data split.}
\label{tab:cg}
\end{table}
\begin{table}[t]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{p{0.35\columnwidth}>{\centering}p{0.25\columnwidth}>{\centering}p{0.25\columnwidth}>{\centering\arraybackslash}p{0.25\columnwidth}}
\toprule
& \textbf{1-shot} & \textbf{3-shot} & \textbf{5-shot}\\
\midrule
BART & 73.93 & 91.99 & 94.37 \\
\tool & 72.05 & 93.73 & 95.16 \\
\toolx & \textbf{84.91} & \textbf{95.31} & \textbf{96.13} \\
\bottomrule
\end{tabular}
}
\caption{Few-shot learning results on \textsc{MetaQA} dataset. \toolx refers to our model that has formerly trained on \textsc{Kqa Pro} dataset.}
\label{tab:metaqa}
\end{table}
\begin{table*}[ht!]
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{p{0.15\textwidth}>{\centering}p{0.05\textwidth}>{\centering}p{0.075\textwidth}>{\raggedright\arraybackslash}p{1.35\textwidth}}
\toprule
\textbf{Error Type} & \textbf{\#} & \textbf{\# OSC} & \textbf{Example} \\
\midrule
\multirow{4}[25]{0.15\textwidth}{Inaccurate data annotation} & \multirow{4}[25]{*}{28} & \multirow{4}[25]{*}{-} & \textbf{User utterance:} Out of newscasts that last 110 minutes, which is the shortest? \\
\cline{4-4}
&&& \textbf{Gold SPARQL:} \texttt{SELECT ?e WHERE \{ ?e instance\_of ?c . ?c name "newscast" . ?e duration ?pv\_1 . ?pv\_1 unit "minute" . \red{?pv\_1 value ?v\_1 . FILTER ( ?v\_1 != "110"\string^\string^xsd:double )} . ?e duration ?pv . ?pv value ?v \} ORDER BY ?v LIMIT 1} \\
\cline{4-4}
&&& \textbf{Generated IR:} \texttt{which one has the smallest <A> duration </A> among <ES> <C> newscast </C> whose <A> duration </A> \red{is number <V> 110 minute </V>} </ES>} \\
\cline{4-4}
&&& \textbf{Compiled SPARQL:} \texttt{SELECT ?e WHERE \{ ?e instance\_of ?c . ?c name "newscast" . ?e duration ?pv\_1 . ?pv\_1 unit "minute" . \red{?pv\_1 value "110"\string^\string^xsd:double} . ?e duration ?pv . ?pv value ?v \} ORDER BY ?v LIMIT 1} \\
\hline
\multirow{4}[15]{0.15\textwidth}{Ambiguous query expression} & \multirow{4}[15]{*}{27} & \multirow{4}[15]{*}{27} & \textbf{User utterance:} How is the kid's movie The Spiderwick Chronicles related to John Sayles? \\
\cline{4-4}
&&& \textbf{Gold SPARQL:} \texttt{SELECT DISTINCT ?p WHERE \{ ?e\_1 name "The Spiderwick Chronicles" . ?e\_1 genre ?e\_3 . \red{?e\_3 name "children's film"} . ?e\_2 name "John Sayles" . ?e\_1 ?p ?e\_2 \}} \\
\cline{4-4}
&&& \textbf{Generated IR:} \texttt{what is the relation from <ES> <E> The Spiderwick Chronicles </E> (<ES> ones that <R> genre </R> backward to \red{<E> kid film </E>} </ES>) </ES> to <E> John Sayles </E>} \\
\cline{4-4}
&&& \textbf{Compiled SPARQL:} \texttt{SELECT DISTINCT ?p WHERE \{ ?e\_1 name "The Spiderwick Chronicles" . ?e\_1 genre ?e\_3 . \red{?e\_3 name "kid film"} . ?e\_2 name "John Sayles" . ?e\_1 ?p ?e\_2 \}} \\
\hline
\multirow{4}[15]{0.15\textwidth}{Unspecified graph structure} & \multirow{4}[15]{*}{13} & \multirow{4}[15]{*}{9} & \textbf{User utterance:} When did Tashkent become the capital of Uzbekistan? \\
\cline{4-4}
&&& \textbf{Gold SPARQL:} \texttt{SELECT DISTINCT ?qpv WHERE \{ \red{?e\_1 name "Tashkent" . ?e\_2 name "Uzbekistan" . ?e\_1 capital\_of ?e\_2} . [ fact\_h ?e\_1 ; fact\_r capital\_of ; fact\_t> ?e\_2 ] start\_time ?qpv \}} \\
\cline{4-4}
&&& \textbf{Generated IR:} \texttt{what is the qualifier <Q> start time </Q> of \red{<E> Uzbekistan </E> that <R> capital </R> to <E> Tashkent </E>}} \\
\cline{4-4}
&&& \textbf{Compiled SPARQL:} \texttt{SELECT DISTINCT ?qpv WHERE \{ \red{?e\_1 name "Uzbekistan" . ?e\_2 name "Tashkent" . ?e\_1 capital ?e\_2} . [ fact\_h ?e\_1 ; fact\_r capital ; fact\_t ?e\_2 ] start\_time ?qpv \}} \\
\hline
\multirow{4}[25]{0.15\textwidth}{Nonequivalent semantics} & \multirow{4}[25]{*}{32} & \multirow{4}[25]{*}{25} & \textbf{User utterance:} When did Joseph L. Mankiewicz graduate from Columbia University? \\
\cline{4-4}
&&& \textbf{Gold SPARQL:} \texttt{SELECT DISTINCT ?qpv WHERE \{ ?e\_1 name "Joseph L. Mankiewicz" . ?e\_2 name "Columbia University" . ?e\_1 educated\_at ?e\_2 . \red{[ fact\_h ?e\_1 ; fact\_r educated\_at ; fact\_t ?e\_2 ] end\_time ?qpv} \}} \\
\cline{4-4}
&&& \textbf{Generated IR:} \texttt{what is the qualifier \red{<Q> start time </Q>} of <E> Joseph L. Mankiewicz </E> that <R> educated at </R> to <E> Columbia University </E>} \\
\cline{4-4}
&&& \textbf{Compiled SPARQL:} \texttt{SELECT DISTINCT ?qpv WHERE \{ ?e\_1 name "Joseph L. Mankiewicz" . ?e\_2 name "Columbia University" . ?e\_1 educated\_at ?e\_2 . \red{[ fact\_h ?e\_1 ; fact\_r educated\_at ; fact\_t ?e\_2 ] start\_time ?qpv} \}} \\
\bottomrule
\end{tabular}
}
\caption{The analysis of 4 error types based on the failure cases as occurred in benchmark \textsc{Kqa Pro}'s test data. ``\# OSC'' refers to the number of errors that can be fixed with one step correction on the IR's structure. }
\label{tab:error}
\end{table*}
% may explain how we split the CG train/test dataset
\paragraph{Compositional Generalization} Current neural semantic parsers often fail in reaching good compositional generalization, \ie the capability of generalizing from the known components to produce novel combinations \cite{pasupat2015compositional, keysers2019measuring, furrer2020compositional}. To measure our IR's compositional generalization ability, we create a new \textsc{Kqa Pro} data split based on the logical form length and test the parsers to generate long queries based on the short query components seen in the training data.
% To measure the robustness of our IR, we test the parsers to generate long queries based on the short query components seen in the training data using the compositional generalization split of \textsc{Kqa Pro} \cite{shi2020kqa}.
The results are listed in Table \ref{tab:cg}. Comparing with the plain-BART baseline and the CFQ IR \cite{herzig2021unlocking} that is specially designed for improving the compositional generalization on SPARQL, \tool achieves the best performance in overall data as well as in complex task settings, which can be again credited to our IR designs that simplify the redundant semantics and preserve the key structural features.
\paragraph{Few-shot learning}
In practice, it is important for a parser to remain robust in a novel task domain lack of data annotations. Therefore, we reconstruct the \textsc{MetaQA} dataset into Cypher, a graph query language commonly used in the industries but rarely studied in previous semantic parsing research works \cite{seifer2019empirical}, and assess our models under the few-shot learning setting. We adjust the data to ensure only 1, 3 and 5 samples of each question type appear in the training set respectively under the 1-shot, 3-shot and 5-shot settings. The results in Table \ref{tab:metaqa} indicate that our IR can generalize well with very few labelled data. Moreover, the \toolx model that has in advance trained on \textsc{Kqa Pro} data demonstrates the most outstanding performance, especially under the hardest 1-shot setting. This further affirms the necessities of unification. With \tool, the annotations originally prepared for other graph query languages (\eg SPARQL) can be now easily transferred to facilitate a novel task domain lack of data.
\section{Error Analysis}
% To investigate \tools potentials and bottleneck, we further look into the failures of our approach when incorrect logical forms are generated. Out of the total 979 errors in benchmark \textsc{Kqa Pro}'s test set, we randomly sampled 100 cases for our in-depth analysis and categorized them into 4 types as shown in Table \ref{tab:error}. According to our analysis, 89\% of the sampled errors can be simply fixed by the revision of data annotation or one step correction on the \tool element.
% \paragraph{Inaccurate data annotation} 28 errors are caused by inconsistent or misinterpreted information in data annotation, such as in Table \ref{tab:error}'s example, \textit{last 110 minutes} in user utterance is contradictory to the \texttt{( ?v\_1 != "110"\string^\string^xsd:double )} in gold SPARQL. Therefore, we attribute this type of errors to the dataset rather than the failure of our approach.
% %Crowdsource workers may make careless mistakes during the dataset annotation, thus the natural language utterances may contain inconsistent or misinterpreted information that contradicts to the corresponding gold logical form. In Table \ref{tab:error}'s example, when user asks for newscasts that \textit{lasts} 110 minutes, the gold SPARQL in dataset however filters the newscasts to have duration \textit{not equal to} 110 minutes. Since our generated IR and SPARQL do carry correct semantics as correspond to the input utterance, we attribute this type of errors to the dataset rather than the failure of our approach.
% \paragraph{Ambiguous query expression} Due to the ambiguity in natural language \cite{roth1998learning, resnik1999semantic}, the semantics of the user utterance may represent in more than one way (\eg \textit{kid film} or \textit{children movie}), whereas the identifier of nodes and edges is unique in a knowledge base (\ie \texttt{children's film} in Table \ref{tab:error}'s example). All 27 errors of this type can be fixed with one step of correction in entity name, relation name, etc.
% %The ambiguity in natural language, as have been widely studied by the community \cite{roth1998learning, resnik1999semantic}, is the root cause of another category of errors. The semantics of the user utterance may be expressed and comprehended in more than one way (\eg \textit{kid film} or \textit{children movie}), while in contrast the nodes and edges in knowledge base are represented with unique identifiers (\ie \textit{children's film} in Table \ref{tab:error}'s example). All 27 errors of this type can be fixed with one step of correction in entity name, relation name, etc.
% \paragraph{Unspecified graph structure} Logical forms of different structures (\eg \texttt{(A capital B)} and \texttt{(B capital\_of A)}) can convey the same semantics in a directed cycle graph, but some of them may be invalid due to the absence of corresponded structure in a knowledge base. 9 out of 13 errors can be fixed in one step by modifying the query structure.
% %Logical forms of different structures can convey the same semantics due to the directed cycle properties of the graph knowledge base. For instance, although \textit{Uzbekistan's capital is Tashkent} and \textit{Tashkent is the capital of Uzbekistan} shares the equivalent semantics, the former statement can be executed successfully while the latter is invalid since the edge of opposite direction is absent in the knowledge base. 9 out of 13 errors can be fixed in one step by changing the relationship direction in queries.
% \paragraph{Nonequivalent semantics} The rest 32 errors all include incorrect query element (\eg string and numerical values) or structure (\eg edges and properties) that conveys nonequivalent semantics, such as misinterpreting \textit{graduate} to \texttt{start\_time} in Table \ref{tab:error}'s case. Yet majority of these error cases (25 out of 32) still need only one step correction on \tool element or SPARQL triple.
% %This type of errors can be attributed to the neural semantic parser's failure in comprehending the natural language representations, such as misinterpreting \textit{graduate} to \texttt{start\_time} rather than \texttt{end\_time} in Table \ref{tab:error}'s case. Yet majority of these error cases (25 out of 32) still need only one step correction on \tool element or SPARQL triple.
To investigate \tools potentials and bottleneck, we look into the failures of our approach when incorrect logical forms are generated.
Out of the total 979 errors in \textsc{Kqa Pro}'s test set, we randomly sampled 100 cases and categorized them into 4 types as shown in Table \ref{tab:error}:
(1) \textbf{Inaccurate data annotation} (28\%). The information in user utterance (\eg \textit{110 minutes}) is inconsistent with the gold SPARQL (\eg \texttt{v\_1 != "110"}) as annotated in the original dataset. We attribute this type of errors to the dataset rather than the failure of our approach.
%Crowdsource workers may make careless mistakes during the dataset annotation, thus the natural language utterances may contain inconsistent or misinterpreted information that contradicts to the corresponding gold logical form. In Table \ref{tab:error}'s example, when user asks for newscasts that \textit{lasts} 110 minutes, the gold SPARQL in dataset however filters the newscasts to have duration \textit{not equal to} 110 minutes. Since our generated IR and SPARQL do carry correct semantics as correspond to the input utterance, we attribute this type of errors to the dataset rather than the failure of our approach.
(2) \textbf{Ambiguous query expression} (27\%). The semantics of the user utterance may present in more than one way (\eg \textit{kid film} or \textit{children's film}) due to the ambiguity in natural language, whereas the schema of knowledge base is pre-defined (\eg only \texttt{children's film} is considered a valid entity).
% All 27 errors of this type can be fixed with one step of correction in entity name, relation name, etc.
%The ambiguity in natural language, as have been widely studied by the community \cite{roth1998learning, resnik1999semantic}, is the root cause of another category of errors. The semantics of the user utterance may be expressed and comprehended in more than one way (\eg \textit{kid film} or \textit{children movie}), while in contrast the nodes and edges in knowledge base are represented with unique identifiers (\ie \textit{children's film} in Table \ref{tab:error}'s example). All 27 errors of this type can be fixed with one step of correction in entity name, relation name, etc.
(3) \textbf{Unspecified graph structure} (13\%). Logical forms of different structures (\eg \texttt{(Uzbekistan capital Tashkent)} and \texttt{(Tashkent capital\_of Uzbekistan)}) can convey the same semantics in a directed cycle graph, but some of them contain structures that are absent in a knowledge base.
% 9 out of 13 errors can be fixed in one step by modifying the query structure.
%Logical forms of different structures can convey the same semantics due to the directed cycle properties of the graph knowledge base. For instance, although \textit{Uzbekistan's capital is Tashkent} and \textit{Tashkent is the capital of Uzbekistan} shares the equivalent semantics, the former statement can be executed successfully while the latter is invalid since the edge of opposite direction is absent in the knowledge base. 9 out of 13 errors can be fixed in one step by changing the relationship direction in queries.
(4) \textbf{Nonequivalent semantics} (32\%). The output includes incorrect query element (\eg string and numerical values) or structure (\eg edges and properties) that conveys nonequivalent semantics, such as misinterpreting \textit{graduate} to \texttt{start\_time}.
% Yet majority of these error cases (25 out of 32) still need only one step correction on \tool element or SPARQL triple.
%This type of errors can be attributed to the neural semantic parser's failure in comprehending the natural language representations, such as misinterpreting \textit{graduate} to \texttt{start\_time} rather than \texttt{end\_time} in Table \ref{tab:error}'s case. Yet majority of these error cases (25 out of 32) still need only one step correction on \tool element or SPARQL triple.
Overall, 89\% of the sampled errors can be simply fixed by the revision of data annotation or one step correction on the \tool element, demonstrating that our proposed approach with \tool can generate high quality logical forms and is easy to debug.
% The detailed OSC information of each type is listed in Table~\ref{tab:error}.
\section{Related Work}
\subsection{Semantic Parsing}
Semantic parsing aims to translate natural language utterances into executable logical forms, such as CCG~\cite{zettlemoyer2005learning}, $\lambda$-DCS~\cite{liang2013lambda,pasupat2015compositional,wang2015building}, SQL~\cite{zhong2017seq2sql, yu2020grappa}, AMR~\cite{banarescu2013abstract}, SPARQL~\cite{sun2020sparqa} and KoPL~\cite{shi2020kqa,Cao2021ProgramTF}.
Most recent works take semantic parsing as a Seq2Seq translation task via encoder-decoder framework, which is challenging due to the semantic and structural gaps between natural utterances and logical forms. To overcome such issues, current semantic parsers usually (1) rely on a large amount of labeled data~\cite{shi2020kqa}; or (2) leverage external resources for mini the structual mismatch, \eg injecting grammar rules during decoding~\cite{Wu2021FromPT,shin2021constrained}; or (3) employ synthetic data to diminish the semantic mismatch~\cite{xu2020autoqa, Wu2021FromPT}.
%In this work, we resolve the gaps from another view, by designing a \tool for graph query languages.
Compared with previous works, our proposed \tool allows semantic parser adapting to different downstream formal query languages without extra efforts and demonstrates promising performance under the compositional generalization and few-shot settings.
\subsection{Intermediate Representation}
Intermediate representations (IR) are usually generated for the internal use of compilers and represent the code structure of input programs \cite{aho2007compilers}. Good IR designs with informative and distinctive mid-level features can provide huge benefits for optimization, translation and downstream code generation \cite{lattner2004llvm}, especially in areas like deep learning \cite{chen2018tvm, cyphers2018intel} and heterogeneous computing \cite{lattner2020mlir}.
Recently, IR has also become common in many semantic parsing works that include an auxiliary representation in-between natural language and logical form. Most of them take a top-down approach and adopt IR similar to natural language \cite{su2017cross, herzig2019don, shin2021constrained}, whereas another category of works instead construct IR based on the key structure of target logical forms in a bottom-up manner \cite{wolfson2020break, marion2021structured}. For example, \citeauthor{herzig2021unlocking} designed CFQ IR that rewrites SPARQL by grouping the triples of same elements \citeyearpar{herzig2021unlocking}.
Although these works partially mitigate the mismatch between natural and formal language, they either neglected the structural information requisite for downstream compilation or failed in removing the formal representations that are unnatural to the language models. In contrast, \tool can benefit from both approaches with its natural-language-like semantic representation and formal-language-like syntactic definition. The unification also saves the huge costs as incurred when preparing separate IR and compiler for each graph query languages in previous works.
% \subsection{Knowledge Base Question Answering}
% For natural language question answering over knowledge bases, current methods can be mainly grouped into two categories: 1) semantic parsing methods, as described above; and 2) information retrieval methods~\cite{miller2016key,saxena2020improving,schlichtkrull2018rgcn,zhou2018interpretable,qiu2020stepwise} that construct a question-specific graph extracted from the knowledge base and rank all the entities in the extracted graph based on their relevance to the question. Compared with information retrieval methods, semantic parsing based methods can provide better interpretability by generating expressive logic forms.
\section{Conclusion and Future Work}
This paper proposes a novel intermediate representation, namely \tool, for bridging the structural and semantic gap between natural language and graph query languages. Evaluation results show that our approach using \tool consistently surpasses the baselines on multiple benchmarks covering different formal languages, \ie SPARQL, KoPL, $\lambda$-DCS and Cypher. Moreover, \tool also demonstrates superior generalization ability and robustness under the compositional generalization and few-shot learning settings.
As an early step towards the unification of semantic parsing, our work opens up a number of future directions. For example, many code optimization techniques (\eg common subexpression elimination) can be incorporated into IR for extra performance boosting. By bringing in multiple levels of IR, our framework may be also extended to support relational database query languages like SQL. Moreover, since current design of \tool still requires non-trivial manual efforts, the automation of such procedure, \eg in prompt-like manners, is definitely worth future exploration.
\appendix
% \newpage
% \section*{Appendix}
% In this section, we will show the more complete set of \tool grammar rules and the detailed examples of the abstract syntax tree for graph query languages.
% \label{sec:appendix}
% \section{\tool Grammar Rules}
\begin{table*}[htbp]
% \scriptsize
\small
\centering
\resizebox{\textwidth}{!}{
\begin{tabular}{p{0.3\textwidth}>{\raggedright\arraybackslash}p{0.9\textwidth}}
\toprule
\textbf{Non-terminal} & \textbf{Productions} \\
\midrule
$S$ &$\rightarrow$ \texttt{EntityQuery} | \texttt{AttributeQuery} | \texttt{RelationQuery} | \texttt{QualifierQuery} | \texttt{CountQuery} | \texttt{VerifyQuery} | \texttt{ValueQuery} \\
\hline
\texttt{EntityQuery} &$\rightarrow$ \textit{what is} \texttt{EntitySet} \\
\hline
\texttt{AttributeQuery} &$\rightarrow$ \textit{what is the attribute} \texttt{Attribute} \textit{of} \texttt{EntitySet} \\
\hline
\texttt{RelationQuery} &$\rightarrow$ \textit{what is the relation from} \texttt{EntitySet} \textit{to} \texttt{EntitySet} \\
\hline
\texttt{QualifierQuery} &$\rightarrow$ \textit{what is the qualifier} \texttt{Qualifier} \textit{of} \texttt{EntitySet Constraint} \\
\hline
\texttt{CountQuery} &$\rightarrow$ \textit{how many} \texttt{EntitySet} \\
\hline
\texttt{VerifyQuery} &$\rightarrow$ \textit{whether} \texttt{EntitySet Constraint} \\
\hline
\texttt{ValueQuery} &$\rightarrow$ \textit{what is} \texttt{Value} \\
\hline
\texttt{EntitySet} &$\rightarrow$ \texttt{<ES> EntitySet LOP EntitySet </ES>} | \texttt{<ES>} \texttt{EntitySet Constraint} \texttt{</ES>} | \texttt{<ES> Concept EntitySet </ES>} | \texttt{Concept} | \texttt{Entity} \\
\hline
\texttt{Constraint} &$\rightarrow$ \texttt{AttributeConstraint} \texttt{QualifierConstraint}? | \texttt{RelationConstraint} \texttt{QualifierConstraint}? \\
\hline
\texttt{AttributeConstraint} &$\rightarrow$ \textit{whose} \texttt{Attribute COP Value} | \textit{that have} \texttt{SOP} \texttt{Attribute} \\
\hline
\texttt{RelationConstraint} &$\rightarrow$ \textit{that} \texttt{Relation DIR} \textit{to} (\texttt{COP Value})? \texttt{EntitySet} | \textit{that} \texttt{Relation DIR} \textit{to} \texttt{SOP} \texttt{EntitySet} \\
\hline
\texttt{QualifierConstraint} &$\rightarrow$ \texttt{Qualifier} \texttt{COP Value}\\
\hline
\texttt{Entity} &$\rightarrow$ \texttt{<E>} \textit{entity} \texttt{</E>} \\
\hline
\texttt{Concept} &$\rightarrow$ \texttt{<C>} \textit{concept} \texttt{</C>} \\
\hline
\texttt{Attribute} &$\rightarrow$ \texttt{<A>} \textit{attribute} \texttt{</A>}\\
\hline
\texttt{Relation} &$\rightarrow$ \texttt{<R>} \textit{relation} \texttt{</R>} \\
\hline
\texttt{Value} &$\rightarrow$ \texttt{VTYPE <V>} \texttt{Value LOP Value} | \texttt{VOP} \textit{of} \texttt{Value} | \texttt{Attribute} \textit{of} \texttt{Entity} | \texttt{VTYPE <V>} \textit{value} \texttt{</V>} \\
\hline
\texttt{LOP} &$\rightarrow$ \textit{and} | \textit{or} | \textit{not} \\
\hline
\texttt{VOP} &$\rightarrow$ \textit{sum} | \textit{average} | \textit{maximum} | \textit{minimum} \\
\hline
\texttt{COP} &$\rightarrow$ \textit{is} | \textit{is not} | \textit{larger than} | \textit{smaller than} | \textit{at least} | \textit{at most} \\
\hline
\texttt{SOP} &$\rightarrow$ \textit{largest} | \textit{smallest} \\
\hline
\texttt{DIR} &$\rightarrow$ \textit{forward} | \textit{backward} \\
\hline
\texttt{VTYPE} &$\rightarrow$ \textit{string} | \textit{number} | \textit{year} | \textit{date} | \textit{time} \\
\bottomrule
\end{tabular}
}
\caption{A more complete set of \tool grammar rules that covers the common graph query patterns. Italic words refer to the terminal symbols. For simplicity, here we omit the special production rules for handling the corner cases.}
\label{tab:full_grammar}
\end{table*}
\begin{figure*}[t]
\centering
\resizebox{.93\linewidth}{!}{
\includegraphics{figures/ast_2.pdf}
}
\caption{A user query in \textsc{Overnight}. As aforementioned in Section \ref{sec:imple}, the neural semantic parser first translate the input utterance into the \tool. The front-end of compiler then parses the \tool sequence into an abstract syntax tree, which is subsequently transformed into the corresponding $\lambda$-DCS sequence by the compiler back-end. }
\label{fig:ast_1}
\end{figure*}
\begin{figure*}[t]
\centering
\resizebox{.93\linewidth}{!}{
\includegraphics{figures/ast_1.pdf}
}
\caption{A user query in \textsc{KQA Pro}. Similarly, the compiler parses the generated \tool sequence into an abstract syntax tree, then transform its tree structure into the corresponding SPARQL sequence. }
\label{fig:ast_2}
\end{figure*}
\begin{thebibliography}{62}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Agrawal et~al.(2022)Agrawal, Singhal, and Mutharaju}]{a2022sparql}
Lakshya~A Agrawal, Nikunj Singhal, and Raghava Mutharaju. 2022.
\newblock A sparql to cypher transpiler: Proposal and initial results.
\newblock In \emph{5th Joint International Conference on Data Science \&
Management of Data (9th ACM IKDD CODS and 27th COMAD)}, pages 312--313.
\bibitem[{Aho et~al.(2007)Aho, Lam, Sethi, and Ullman}]{aho2007compilers}
Alfred~V Aho, Monica~S Lam, Ravi Sethi, and Jeffrey~D Ullman. 2007.
\newblock \emph{Compilers: principles, techniques, \& tools}.
\newblock Pearson Education India.
\bibitem[{Banarescu et~al.(2013)Banarescu, Bonial, Cai, Georgescu, Griffitt,
Hermjakob, Knight, Koehn, Palmer, and Schneider}]{banarescu2013abstract}
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf
Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider.
2013.
\newblock Abstract meaning representation for sembanking.
\newblock In \emph{Proceedings of the 7th linguistic annotation workshop and
interoperability with discourse}, pages 178--186.
\bibitem[{Berant et~al.(2013)Berant, Chou, Frostig, and
Liang}]{berant2013semantic}
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.
\newblock Semantic parsing on freebase from question-answer pairs.
\newblock In \emph{Proceedings of the 2013 conference on empirical methods in
natural language processing}, pages 1533--1544.
\bibitem[{Berant and Liang(2014)}]{berant2014semantic}
Jonathan Berant and Percy Liang. 2014.
\newblock Semantic parsing via paraphrasing.
\newblock In \emph{Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 1415--1425.
\bibitem[{Bollacker et~al.(2008)Bollacker, Evans, Paritosh, Sturge, and
Taylor}]{bollacker2008freebase}
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.
2008.
\newblock Freebase: a collaboratively created graph database for structuring
human knowledge.
\newblock In \emph{Proceedings of the 2008 ACM SIGMOD international conference
on Management of data}, pages 1247--1250.
\bibitem[{Campagna et~al.(2019)Campagna, Xu, Moradshahi, Socher, and
Lam}]{campagna2019genie}
Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica~S
Lam. 2019.
\newblock Genie: A generator of natural language semantic parsers for virtual
assistant commands.
\newblock In \emph{Proceedings of the 40th ACM SIGPLAN Conference on
Programming Language Design and Implementation}, pages 394--410.
\bibitem[{Cao et~al.(2019)Cao, Zhu, Liu, Li, and Yu}]{cao2019semantic}
Ruisheng Cao, Su~Zhu, Chen Liu, Jieyu Li, and Kai Yu. 2019.
\newblock Semantic parsing with dual learning.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics}, pages 51--64.
\bibitem[{Cao et~al.(2020{\natexlab{a}})Cao, Zhu, Yang, Liu, Ma, Zhao, Chen,
and Yu}]{cao2020unsupervised}
Ruisheng Cao, Su~Zhu, Chenyu Yang, Chen Liu, Rao Ma, Yanbin Zhao, Lu~Chen, and
Kai Yu. 2020{\natexlab{a}}.
\newblock Unsupervised dual paraphrasing for two-stage semantic parsing.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 6806--6817.
\bibitem[{Cao et~al.(2020{\natexlab{b}})Cao, Shi, Pan, Nie, Xiang, Hou, Li, He,
and Zhang}]{shi2020kqa}
Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou,
Juanzi Li, Bin He, and Hanwang Zhang. 2020{\natexlab{b}}.
\newblock Kqa pro: A dataset with explicit compositional programs for complex
question answering over knowledge base.
\newblock \emph{arXiv preprint arXiv:2007.03875}.
\bibitem[{Cao et~al.(2021)Cao, Shi, Yao, Lv, Yu, Hou, Li, Liu, and
Xiao}]{Cao2021ProgramTF}
Shulin Cao, Jiaxin Shi, Zijun Yao, Xin Lv, Jifan Yu, Lei Hou, Juan-Zi Li,
Zhiyuan Liu, and Jinghui Xiao. 2021.
\newblock Program transfer for answering complex questions over knowledge
bases.
\bibitem[{Chen et~al.(2018{\natexlab{a}})Chen, Sun, and Han}]{chen2018sequence}
Bo~Chen, Le~Sun, and Xianpei Han. 2018{\natexlab{a}}.
\newblock Sequence-to-action: End-to-end semantic graph generation for semantic
parsing.
\newblock \emph{arXiv preprint arXiv:1809.00773}.
\bibitem[{Chen et~al.(2018{\natexlab{b}})Chen, Moreau, Jiang, Zheng, Yan, Shen,
Cowan, Wang, Hu, Ceze et~al.}]{chen2018tvm}
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen
Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et~al.
2018{\natexlab{b}}.
\newblock $\{$TVM$\}$: An automated $\{$End-to-End$\}$ optimizing compiler for
deep learning.
\newblock In \emph{13th USENIX Symposium on Operating Systems Design and
Implementation (OSDI 18)}, pages 578--594.
\bibitem[{Clarke et~al.(2010)Clarke, Goldwasser, Chang, and
Roth}]{clarke2010driving}
James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010.
\newblock Driving semantic parsing from the world’s response.
\newblock In \emph{Proceedings of the fourteenth conference on computational
natural language learning}, pages 18--27.
\bibitem[{Cyphers et~al.(2018)Cyphers, Bansal, Bhiwandiwalla, Bobba, Brookhart,
Chakraborty, Constable, Convey, Cook, Kanawi et~al.}]{cyphers2018intel}
Scott Cyphers, Arjun~K Bansal, Anahita Bhiwandiwalla, Jayaram Bobba, Matthew
Brookhart, Avijit Chakraborty, Will Constable, Christian Convey, Leona Cook,
Omar Kanawi, et~al. 2018.
\newblock Intel ngraph: An intermediate representation, compiler, and executor
for deep learning.
\newblock \emph{arXiv preprint arXiv:1801.08058}.
\bibitem[{Damonte and Monti(2021)}]{damonte2021one}
Marco Damonte and Emilio Monti. 2021.
\newblock One semantic parser to parse them all: Sequence to sequence
multi-task learning on semantic parsing datasets.
\newblock \emph{arXiv preprint arXiv:2106.04476}.
\bibitem[{Dubey et~al.(2019)Dubey, Banerjee, Abdelkawi, and
Lehmann}]{dubey2019lc}
Mohnish Dubey, Debayan Banerjee, Abdelrahman Abdelkawi, and Jens Lehmann. 2019.
\newblock Lc-quad 2.0: A large dataset for complex question answering over
wikidata and dbpedia.
\newblock In \emph{International semantic web conference}, pages 69--78.
Springer.
\bibitem[{Fischer et~al.(2021)Fischer, Campagna, Choi, and
Lam}]{fischer2021diy}
Michael~H Fischer, Giovanni Campagna, Euirim Choi, and Monica~S Lam. 2021.
\newblock Diy assistant: a multi-modal end-user programmable virtual assistant.
\newblock In \emph{Proceedings of the 42nd ACM SIGPLAN International Conference
on Programming Language Design and Implementation}, pages 312--327.
\bibitem[{Furrer et~al.(2020)Furrer, van Zee, Scales, and
Sch{\"a}rli}]{furrer2020compositional}
Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Sch{\"a}rli. 2020.
\newblock Compositional generalization in semantic parsing: Pre-training vs.
specialized architectures.
\newblock \emph{arXiv preprint arXiv:2007.08970}.
\bibitem[{Herzig and Berant(2019)}]{herzig2019don}
Jonathan Herzig and Jonathan Berant. 2019.
\newblock Don’t paraphrase, detect! rapid and effective data collection for
semantic parsing.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP)}, pages 3810--3820.
\bibitem[{Herzig et~al.(2021)Herzig, Shaw, Chang, Guu, Pasupat, and
Zhang}]{herzig2021unlocking}
Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and
Yuan Zhang. 2021.
\newblock Unlocking compositional generalization in pre-trained models using
intermediate representations.
\newblock \emph{arXiv preprint arXiv:2104.07478}.
\bibitem[{Huang et~al.(2021)Huang, Kim, and Zou}]{huang2021unseen}
Xin Huang, Jung-jae Kim, and Bowei Zou. 2021.
\newblock Unseen entity handling in complex question answering over knowledge
base via language generation.
\newblock In \emph{Findings of the Association for Computational Linguistics:
EMNLP 2021}, pages 547--557.
\bibitem[{Hwang et~al.(2019)Hwang, Yim, Park, and Seo}]{hwang2019comprehensive}
Wonseok Hwang, Jinyeong Yim, Seunghyun Park, and Minjoon Seo. 2019.
\newblock A comprehensive exploration on wikisql with table-aware word
contextualization.
\newblock \emph{arXiv preprint arXiv:1902.01069}.
\bibitem[{Kamath and Das(2018)}]{kamath2018survey}
Aishwarya Kamath and Rajarshi Das. 2018.
\newblock A survey on semantic parsing.
\newblock In \emph{Automated Knowledge Base Construction (AKBC)}.
\bibitem[{Keysers et~al.(2019)Keysers, Sch{\"a}rli, Scales, Buisman, Furrer,
Kashubin, Momchev, Sinopalnikov, Stafiniak, Tihon
et~al.}]{keysers2019measuring}
Daniel Keysers, Nathanael Sch{\"a}rli, Nathan Scales, Hylke Buisman, Daniel
Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz
Stafiniak, Tibor Tihon, et~al. 2019.
\newblock Measuring compositional generalization: A comprehensive method on
realistic data.
\newblock \emph{arXiv preprint arXiv:1912.09713}.
\bibitem[{Kwiatkowksi et~al.(2010)Kwiatkowksi, Zettlemoyer, Goldwater, and
Steedman}]{kwiatkowksi2010inducing}
Tom Kwiatkowksi, Luke Zettlemoyer, Sharon Goldwater, and Mark Steedman. 2010.
\newblock Inducing probabilistic ccg grammars from logical form with
higher-order unification.
\newblock In \emph{Proceedings of the 2010 conference on empirical methods in
natural language processing}, pages 1223--1233.
\bibitem[{Lattner and Adve(2004)}]{lattner2004llvm}
Chris Lattner and Vikram Adve. 2004.
\newblock Llvm: A compilation framework for lifelong program analysis \&
transformation.
\newblock In \emph{International Symposium on Code Generation and Optimization,
2004. CGO 2004.}, pages 75--86. IEEE.
\bibitem[{Lattner et~al.(2020)Lattner, Amini, Bondhugula, Cohen, Davis,
Pienaar, Riddle, Shpeisman, Vasilache, and Zinenko}]{lattner2020mlir}
Chris Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques
Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasilache, and Oleksandr
Zinenko. 2020.
\newblock Mlir: A compiler infrastructure for the end of moore's law.
\newblock \emph{arXiv preprint arXiv:2002.11054}.
\bibitem[{Lewis et~al.(2020)Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy,
Stoyanov, and Zettlemoyer}]{lewis2020bart}
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.
\newblock Bart: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 7871--7880.
\bibitem[{Li et~al.(2020)Li, Qu, and Haffari}]{li2020context}
Zhuang Li, Lizhen Qu, and Reza Haffari. 2020.
\newblock Context dependent semantic parsing: a survey.
\newblock In \emph{International Conference on Computational Linguistics 2020},
pages 2509--2521. Association for Computational Linguistics (ACL).
\bibitem[{Liang(2013)}]{liang2013lambda}
Percy Liang. 2013.
\newblock Lambda dependency-based compositional semantics.
\newblock \emph{arXiv preprint arXiv:1309.4408}.
\bibitem[{Liang et~al.(2013)Liang, Jordan, and Klein}]{liang2013learning}
Percy Liang, Michael~I Jordan, and Dan Klein. 2013.
\newblock Learning dependency-based compositional semantics.
\newblock \emph{Computational Linguistics}, 39(2):389--446.
\bibitem[{Ling et~al.(2016)Ling, Blunsom, Grefenstette, Hermann,
Ko{\v{c}}isk{\`y}, Wang, and Senior}]{ling2016latent}
Wang Ling, Phil Blunsom, Edward Grefenstette, Karl~Moritz Hermann,
Tom{\'a}{\v{s}} Ko{\v{c}}isk{\`y}, Fumin Wang, and Andrew Senior. 2016.
\newblock Latent predictor networks for code generation.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 599--609.
\bibitem[{Loshchilov and Hutter(2018)}]{loshchilov2018fixing}
Ilya Loshchilov and Frank Hutter. 2018.
\newblock Fixing weight decay regularization in adam.
\bibitem[{Marion et~al.(2021)Marion, Nowak, and
Piccinno}]{marion2021structured}
Pierre Marion, Pawel Nowak, and Francesco Piccinno. 2021.
\newblock Structured context and high-coverage grammar for conversational
question answering over knowledge graphs.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 8813--8829.
\bibitem[{Nan et~al.(2020)Nan, Guan, and Shen}]{nan2020hisyn}
Zifan Nan, Hui Guan, and Xipeng Shen. 2020.
\newblock Hisyn: human learning-inspired natural language programming.
\newblock In \emph{Proceedings of the 28th ACM Joint Meeting on European
Software Engineering Conference and Symposium on the Foundations of Software
Engineering}, pages 75--86.
\bibitem[{Parr(2013)}]{parr2013definitive}
Terence Parr. 2013.
\newblock \emph{The definitive ANTLR 4 reference}.
\newblock Pragmatic Bookshelf.
\bibitem[{Parr and Fisher(2011)}]{parr2011ll}
Terence Parr and Kathleen Fisher. 2011.
\newblock Ll (*) the foundation of the antlr parser generator.
\newblock \emph{ACM Sigplan Notices}, 46(6):425--436.
\bibitem[{Pasupat and Liang(2015)}]{pasupat2015compositional}
Panupong Pasupat and Percy Liang. 2015.
\newblock Compositional semantic parsing on semi-structured tables.
\newblock \emph{arXiv preprint arXiv:1508.00305}.
\bibitem[{P{\'e}rez et~al.(2009)P{\'e}rez, Arenas, and
Gutierrez}]{perez2009semantics}
Jorge P{\'e}rez, Marcelo Arenas, and Claudio Gutierrez. 2009.
\newblock Semantics and complexity of sparql.
\newblock \emph{ACM Transactions on Database Systems (TODS)}, 34(3):1--45.
\bibitem[{Schlichtkrull et~al.(2018)Schlichtkrull, Kipf, Bloem, Berg, Titov,
and Welling}]{schlichtkrull2018rgcn}
Michael Schlichtkrull, Thomas~N Kipf, Peter Bloem, Rianne van~den Berg, Ivan
Titov, and Max Welling. 2018.
\newblock Modeling relational data with graph convolutional networks.
\newblock In \emph{European semantic web conference}, pages 593--607. Springer.
\bibitem[{Seifer et~al.(2019)Seifer, H{\"a}rtel, Leinberger, L{\"a}mmel, and
Staab}]{seifer2019empirical}
Philipp Seifer, Johannes H{\"a}rtel, Martin Leinberger, Ralf L{\"a}mmel, and
Steffen Staab. 2019.
\newblock Empirical study on the usage of graph query languages in open source
java projects.
\newblock In \emph{Proceedings of the 12th ACM SIGPLAN international conference
on software language engineering}, pages 152--166.
\bibitem[{Shi et~al.(2021)Shi, Cao, Hou, Li, and Zhang}]{shi2021transfernet}
Jiaxin Shi, Shulin Cao, Lei Hou, Juanzi Li, and Hanwang Zhang. 2021.
\newblock Transfernet: An effective and transparent framework for multi-hop
question answering over relation graph.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 4149--4158.
\bibitem[{Shin et~al.(2021)Shin, Lin, Thomson, Chen~Jr, Roy, Platanios, Pauls,
Klein, Eisner, and Van~Durme}]{shin2021constrained}
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen~Jr, Subhro Roy,
Emmanouil~Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and
Benjamin Van~Durme. 2021.
\newblock Constrained language models yield few-shot semantic parsers.
\newblock In \emph{Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing}, pages 7699--7715.
\bibitem[{Su and Yan(2017)}]{su2017cross}
Yu~Su and Xifeng Yan. 2017.
\newblock Cross-domain semantic parsing via paraphrasing.
\newblock In \emph{Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing}, pages 1235--1246.
\bibitem[{Sun et~al.(2020)Sun, Zhang, Cheng, and Qu}]{sun2020sparqa}
Yawei Sun, Lingling Zhang, Gong Cheng, and Yuzhong Qu. 2020.
\newblock Sparqa: skeleton-based semantic parsing for complex questions over
knowledge bases.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~34, pages 8952--8959.
\bibitem[{Talmor and Berant(2018)}]{talmor2018web}
Alon Talmor and Jonathan Berant. 2018.
\newblock The web as a knowledge-base for answering complex questions.
\newblock In \emph{Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long Papers)}, pages 641--651.
\bibitem[{Tapaswi et~al.(2016)Tapaswi, Zhu, Stiefelhagen, Torralba, Urtasun,
and Fidler}]{tapaswi2016movieqa}
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel
Urtasun, and Sanja Fidler. 2016.
\newblock Movieqa: Understanding stories in movies through question-answering.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 4631--4640.
\bibitem[{Tomlin(2014)}]{tomlin2014basic}
Russell~S Tomlin. 2014.
\newblock \emph{Basic Word Order (RLE Linguistics B: Grammar): Functional
Principles}.
\newblock Routledge.
\bibitem[{Vrande{\v{c}}i{\'c} and Kr{\"o}tzsch(2014)}]{vrandevcic2014wikidata}
Denny Vrande{\v{c}}i{\'c} and Markus Kr{\"o}tzsch. 2014.
\newblock Wikidata: a free collaborative knowledgebase.
\newblock \emph{Communications of the ACM}, 57(10):78--85.
\bibitem[{Wang et~al.(2021)Wang, Lapata, and Titov}]{wang2021learning}
Bailin Wang, Mirella Lapata, and Ivan Titov. 2021.
\newblock Learning from executions for semantic parsing.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 2747--2759.
\bibitem[{Wang et~al.(2018)Wang, Tatwawadi, Brockschmidt, Huang, Mao, Polozov,
and Singh}]{wang2018robust}
Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi~Mao,
Oleksandr Polozov, and Rishabh Singh. 2018.
\newblock Robust text-to-sql generation with execution-guided decoding.
\newblock \emph{arXiv preprint arXiv:1807.03100}.
\bibitem[{Wang et~al.(2015)Wang, Berant, and Liang}]{wang2015building}
Yushi Wang, Jonathan Berant, and Percy Liang. 2015.
\newblock Building a semantic parser overnight.
\newblock In \emph{Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 1332--1342.
\bibitem[{Wolfson et~al.(2020)Wolfson, Geva, Gupta, Gardner, Goldberg, Deutch,
and Berant}]{wolfson2020break}
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel
Deutch, and Jonathan Berant. 2020.
\newblock Break it down: A question understanding benchmark.
\newblock \emph{Transactions of the Association for Computational Linguistics},
8:183--198.
\bibitem[{Wu et~al.(2021)Wu, Chen, Xin, Han, Sun, Zhang, Chen, Yang, and
Cai}]{Wu2021FromPT}
Shan Wu, Bo~Chen, Chunlei Xin, Xianpei Han, Le~Sun, Weipeng Zhang, Jiansong
Chen, Fan Yang, and Xunliang Cai. 2021.
\newblock From paraphrasing to semantic parsing: Unsupervised semantic parsing
via synchronous semantic decoding.
\newblock In \emph{Proceedings of the 59th Annual Meeting of the Association
for Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 5110--5121.
\bibitem[{Xu et~al.(2020)Xu, Semnani, Campagna, and Lam}]{xu2020autoqa}
Silei Xu, Sina Semnani, Giovanni Campagna, and Monica Lam. 2020.
\newblock Autoqa: From databases to qa semantic parsers with only synthetic
training data.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 422--434.
\bibitem[{Yin and Neubig(2017)}]{yin2017syntactic}
Pengcheng Yin and Graham Neubig. 2017.
\newblock A syntactic neural model for general-purpose code generation.
\newblock In \emph{Proceedings of the 55th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 440--450.
\bibitem[{Yu et~al.(2020)Yu, Wu, Lin, Tan, Yang, Radev, Xiong
et~al.}]{yu2020grappa}
Tao Yu, Chien-Sheng Wu, Xi~Victoria Lin, Yi~Chern Tan, Xinyi Yang, Dragomir
Radev, Caiming Xiong, et~al. 2020.
\newblock Grappa: Grammar-augmented pre-training for table semantic parsing.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Yu et~al.(2018)Yu, Zhang, Yang, Yasunaga, Wang, Li, Ma, Li, Yao,
Roman et~al.}]{yu2018spider}
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James
Ma, Irene Li, Qingning Yao, Shanelle Roman, et~al. 2018.
\newblock Spider: A large-scale human-labeled dataset for complex and
cross-domain semantic parsing and text-to-sql task.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing}, pages 3911--3921.
\bibitem[{Zettlemoyer and Collins(2005)}]{zettlemoyer2005learning}
Luke~S Zettlemoyer and Michael Collins. 2005.
\newblock Learning to map sentences to logical form: structured classification
with probabilistic categorial grammars.
\newblock In \emph{Proceedings of the Twenty-First Conference on Uncertainty in
Artificial Intelligence}, pages 658--666.
\bibitem[{Zhang et~al.(2018)Zhang, Dai, Kozareva, Smola, and
Song}]{zhang2018variational}
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander~J Smola, and Le~Song.
2018.
\newblock Variational reasoning for question answering with knowledge graph.
\newblock In \emph{Thirty-second AAAI conference on artificial intelligence}.
\bibitem[{Zhong et~al.(2017)Zhong, Xiong, and Socher}]{zhong2017seq2sql}
Victor Zhong, Caiming Xiong, and Richard Socher. 2017.
\newblock Seq2sql: Generating structured queries from natural language using
reinforcement learning.
\newblock \emph{arXiv preprint arXiv:1709.00103}.
\end{thebibliography}