DEER: Descriptive Knowledge Graph for Explaining Entity Relationships
Extracted source text ยท arXiv 2205.10479
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% Remove the "review" option to generate the final version.
\usepackage{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\usepackage{balance}
\usepackage{url}
\usepackage{graphicx}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{textcomp}
\usepackage{xcolor}
\usepackage{tikz}
\usetikzlibrary{topaths,calc}
\usepackage[ruled]{algorithm2e}
\usepackage{subfigure}
\usepackage{booktabs}
\usepackage{balance}
\usepackage{colortbl}
\usepackage{enumitem}
\usepackage{multirow}
% \usepackage{ulem}
\usepackage[normalem]{ulem}
% \hypersetup{draft}
\usepackage{etoolbox}
% \usepackage[section]{placeins}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\revisejie}[1]{\textcolor{blue}{#1}}
\newcommand{\jx}[1]{\textcolor{red}{JX: #1}}
\title{DKG: A Descriptive Knowledge Graph for Explaining Relationships between Entities}
\author{Jie Huang$^{*,1}$ $\quad$ Kerui Zhu$^{*,1}$ $\quad$ Kevin Chen-Chuan Chang$^{1}$ \\ \textbf{Jinjun Xiong$^{2}$ $\quad$ Wen-mei Hwu$^{1,3}$} \\
$^1$University of Illinois at Urbana-Champaign, USA \\
$^2$University at Buffalo, USA \\
$^3$NVIDIA, USA \\
\texttt{\{jeffhj, keruiz2, kcchang, w-hwu\}@illinois.edu} \\
\texttt{jinjun@buffalo.edu}
}
\begin{document}
\maketitle
\begin{abstract}
We propose \textit{Descriptive Knowledge Graph (DKG)} -- an open and interpretable form of modeling relationships between entities. In DKGs, relationships between entities are represented by relation descriptions.
For instance, the relationship between entities of \textit{machine learning} and \textit{algorithm} can be described as ``\emph{Machine learning} explores the study and construction of \emph{algorithms} that can learn from and make predictions on data.''
To construct DKGs, we propose a self-supervised learning method to extract relation descriptions with the analysis of dependency patterns and a transformer-based relation description synthesizing model to generate relation descriptions. Experiments demonstrate that our system can extract and generate high-quality relation descriptions for explaining entity relationships.
% \footnote{Code and data are in the supplementary materials and will be released as open-source after the review process.}
\end{abstract}
\newenvironment{starfootnotes}
{\par\edef\savedfootnotenumber{\number\value{footnote}}
\renewcommand{\thefootnote}{*}
\setcounter{footnote}{0}}
{\par\setcounter{footnote}{\savedfootnotenumber}}
\begin{starfootnotes}
\footnotetext{Asterisk indicates equal contribution.}
\end{starfootnotes}
\section{Introduction}
\begin{figure*}[tp!]
\centerline{\includegraphics[width=0.78\linewidth]{DKG.pdf}}
\vspace{-1mm}
\caption{Descriptive Knowledge Graph. Here we show \textit{machine learning} and several of its related entities, with corresponding relation descriptions extracted by our system in the edges.}
\vspace{-4mm}
\label{fig:graph}
\end{figure*}
\begin{figure}[tp!]
\centerline{\includegraphics[width=0.91\linewidth]{PKG.pdf}}
\vspace{-1mm}
\caption{Prescriptive Knowledge Graph (Wikidata), where $'$ denotes the inversed direction and ? means the relation is not present in the graph.}
\vspace{-5mm}
\label{fig:pkg}
\end{figure}
Relationships exist widely between entities. For example, a person may be related to another person or an institution, and a scientific concept can be connected to another concept.
At the same time, relationships between entities can be subtle or complex, e.g., the relationship between \textit{machine learning} and \textit{algorithm}.
To model relationships between entities, researchers usually construct knowledge graphs (KGs) \cite{ji2021survey,hogan2021knowledge}, where nodes are entities, e.g., \textit{machine learning}, and edges are relations, e.g., \textit{subclass of} (Figure \ref{fig:pkg}).
However, KGs usually require a pre-specified set of relation types, and the covered relation types are usually coarse-grained and simple.
% This indicates two potential limitations of KGs.
This indicates existing KGs lack two desired features.
The first is \textit{\textbf{openness}}: for entities with a relationship not covered by the type set, KGs cannot handle their relationship directly.
Besides, in many cases, the relationship between entities is complex or idiosyncratic that it cannot be simply categorized to a relation type (usually denoted by a short phrase). For instance, for related entities \textit{machine learning} and \textit{algorithm},
% whose relationship is a bit implicit,
Wikidata \cite{vrandevcic2014wikidata}\footnote{More precisely, Wikidata is a knowledge base. Refer to \cite{ji2021survey} to see the minor difference between knowledge graph and knowledge base.} does not include a relation for them, and it is also not easy to come up with a relation type to describe their relationship.
The second feature is about \textit{\textbf{interpretability}}.
With the relational facts in KGs, humans may still have difficulty in understanding entity relationships. For instance, from fact ``(data mining, \textit{facet of}, database)'' in Wikidata, humans may guess \textit{data mining} and \textit{database} are related fields, but they cannot understand how exactly they are related.
Although techniques like knowledge graph reasoning \cite{lao2011random,xiong2017deeppath,chen2018variational} or open relation extraction \cite{etzioni2008open} can represent more complex relationships to some extent, they do not fundamentally solve the limitations \cite{huang2022open}. For instance, neither a multi-hop reasoning path in KGs nor a triple extracted by open relation extraction, e.g., (data mining methods, \textit{to be integrate within}, the framework of traditional database systems), is easy to interpret.
Based on the above analysis, we propose a new form of modeling relationships between entities: \textbf{Descriptive Knowledge Graph (DKG)}. We define a DKG as a graph, where nodes are entities and edges are descriptive statements of entity relationships (refer to Figure \ref{fig:graph} for an example). DKGs have great potential to help users understand entity relationships more easily and intuitively by providing relation descriptions for any two related entities (including those not present in traditional KGs) and facilitate downstream tasks on entities and entity relationships such as definition modeling \cite{noraset2017definition}, relation extraction \cite{bach2007review}, and knowledge graph completion \cite{lin2015learning}.
Compared to traditional KGs, what we refer to as \textit{Prescriptive Knowledge Graphs (PKGs)}\footnote{We use the term ``prescriptive'' (normative and structural) to contrast with ``descriptive'' (open and interpretable).} later, DKGs are 1) \textbf{\textit{more open}}: a DKG does not require a pre-specified set of relation types like a PKG. In principle, all entity relationships, either explicit or implicit, can be represented by DKG, as long as they can be connected in a sentence -- which is not possible for PKGs. For instance, for \textit{machine learning} and \textit{algorithm}, the relationship between them can be represented by ``\emph{Machine learning} explores the study and construction of \emph{algorithms} that can learn from and make predictions on data.'';
2) \textbf{\textit{more interpretable}}: the relationships between entities are represented by interpretable sentence relation descriptions, instead of a simple short phrase like ``facet of''.
It should be noted that we do not emphasize that DKGs are superior to PKGs. In fact, they can complement each other: DKGs provide a more open and interpretable (i.e., descriptive) way to understand entity relationships, while PKGs help understand relationships from a more normative and structural (i.e., prescriptive) perspective.
The key to building a DKG is to acquire high-quality relation descriptions.
However, writing or collecting relation descriptions manually requires enormous human efforts and expertise (in our human evaluation in Section \ref{sec:exp_construction}, it takes $\sim$3 minutes to evaluate whether a sentence is a good relation description).
Considering this, we propose a novel two-step approach to construct a DKG with Wikipedia, where no manual annotation is required.
Specifically, we first \textbf{\textit{extract}} relation descriptions from corpus in a self-supervised manner, where a scoring function is introduced to measure the \textit{explicitness}, i.e., how explicit is the relationship represented by the sentence, and \textit{significance}, i.e., how significant is the relationship represented, with the analysis of dependency patterns.
Second, a transformer-based relation description synthesizing model is introduced to \textbf{\textit{generate}} relation descriptions for interesting entity pairs whose relation descriptions are not extracted in the first step.
% \revisejie{[TODO: argue the power, e.g., generating relation descriptions for interesting entity pairs dynamically?]}
Both quantitative and qualitative experiments, along with ablation study, demonstrate the effectiveness of our proposed methods. We also conduct case study and error analysis and suggest several promising directions for future work.
\section{Related Work}
There are several previous attempts on acquiring entity relation descriptions. For instance, \citet{voskarides2015learning} study a learning to rank problem of ranking relation descriptions by training a Random Forest classifier with manually annotated data. Subsequently, \citet{huang2017learning} build a pairwise ranking model based on convolutional neural networks by leveraging query-title pairs derived from clickthrough data of a Web search engine, and \citet{voskarides2017generating} attempt to generate descriptions for relationship instances in KGs by filling created sentence templates with appropriate entities. However, all these methods are not ``open''. First, they rely and demand heavily on features of entities and relations. Second, these models only deal with entities with several pre-specified relation types, e.g., 9 in \cite{voskarides2015learning} and 10 in \cite{voskarides2017generating}, and only explicit relation types, e.g., \textit{isMemberOfMusicGroup}, are covered. Besides, these works do not systematically define what constitutes a good relational description.
The work most relevant to ours is \textit{Open Relation Modeling} \cite{huang2022open}, which aims to generate relation descriptions for entity pairs. To achieve this, the authors propose to fine-tune BART \cite{lewis2020bart} to reproduce definitions of entities.
Compared to their problem, i.e., text generation, the focus of this paper is on graph construction.
Besides, their relation descriptions are limited to definitional sentences, which assumes that one entity appears in the other's definition; however, the assumption is not true for many related entities. In addition, their methodology does not incorporate sufficient knowledge about entities and relations for generation.
There are also some less relevant works that can be related. For example, \citet{lin2020commongen,liu2021kg} study \textit{CommonGen}, which aims to generate coherent sentences containing the given common concepts. \citet{dognin2020dualtkb,agarwal2021knowledge} study the data-to-text generation \cite{kukich1983design}, which aims to convert facts in KGs into natural language. None of these works meet the requirements on openness and interpretability.
\section{Descriptive Knowledge Graph}
\label{sec:DKG}
A \textbf{Descriptive Knowledge Graph (DKG)} is a graph that represents relationships between entities with sentence descriptions. Formally, we define a DKG as $\mathcal{G} = \{ \mathcal{E}, \mathcal{R} \}$, where $\mathcal{E}$ is the set of entities and $\mathcal{R}$ is the set of relation description facts. A relation description fact is denoted as a triple $(x,s,y)$, where $x \in \mathcal{E}$ and $y \in \mathcal{E}$. $s$ is a sentence describing the relationship between $x$ and $y$ (Figure \ref{fig:graph}).
% An example of DKG is shown in Figure \ref{fig:graph}.
\begin{table}[tp!]
% \small
% \footnotesize
\scriptsize
% \setlength\tabcolsep{1.5pt}
\setlength\tabcolsep{1.5pt}
\begin{tabular}{c|p{0.92\linewidth}}
\toprule
\# & \multicolumn{1}{c}{\textbf{Sentence}} \\
\midrule[1pt]
$s_1$ & \emph{Machine learning} explores the study and construction of \emph{algorithms} that can learn from and make predictions on data.\\
\hline
$s_2$ & \emph{Machine learning} is employed in a range of computing tasks where designing and programming explicit, rule-based \emph{algorithms} is infeasible. \\
\hline
$s_3$ & In \emph{machine learning} and optimization, many \emph{algorithms} are adaptive or have adaptive variants, which usually means that the algorithm parameters are automatically adjusted according to statistics about the optimisation thus far. \\
\bottomrule
\end{tabular}
\vspace{-1mm}
\caption{Example sentences containing both \textit{machine learning} and \textit{algorithm}.}
\label{table:example}
\vspace{-5mm}
\end{table}
To build a DKG, the first step is to collect entities and identify related entity pairs, which can be simply achieved by utilizing existing resources, e.g., Wikipedia, and entity relevance analysis, e.g., cosine similarity of entity embeddings in Wikipedia2vec \cite{yamada2020wikipedia2vec}.
And then, we need to acquire high-quality relation descriptions for entity pairs.
% The key to obtaining a high-quality DKG is to obtain high-quality relation descriptions.
Taking entity pair (\textit{machine learning}, \textit{algorithm}) as an example, a relation description of them can be
$s_1$ in Table \ref{table:example}.
From the perspective of human understanding,
we identify three requirements for a good relation description:
\begin{itemize}[noitemsep,nolistsep,leftmargin=*]
\item \textbf{Explicitness}: The relationship of the target entities is described explicitly. E.g., in $s_1$, ``\textit{machine learning explores the study and construction of algorithms}'' describes the relationship explicitly; while in $s_2$, the relationship between \textit{machine learning} and \textit{algorithm} is expressed implicitly so that the relationship is difficult to reason.
\item \textbf{Significance}: The relationship of the target entities is the point of the sentence. In $s_1$, all the tokens in the sentence are associated with the relationship between \textit{machine learning} and \textit{algorithm}; while in $s_3$, although the description is explicit, ``\textit{many ... far}'' only characterizes \textit{algorithm}, but not the target entity relationship.
\item \textbf{Correctness}: The relationship between target entities is described correctly.
\end{itemize}
There are other requirements to ensure a good relation description, e.g., the sentence is coherent, grammatical, of reasonable length.
Compared to the above ones, these requirements are general requirements for any sentence, but not specific to our problem;
therefore, we put less emphasis on them.
To acquire relation descriptions that satisfy the above requirements, we propose a novel two-step approach: first extracting relation descriptions from a corpus in a self-supervised manner (Section \ref{sec:extraction}), and then generating relation descriptions for interesting entity pairs whose relation descriptions are not extracted in the previous step (Section \ref{sec:generation}).
% \section{DKG Construction}
\section{Relation Description Extraction}
\label{sec:extraction}
In this section, we introduce a self-supervised learning method for extracting entity relation descriptions from Wikipedia according to the requirements discussed in Section \ref{sec:DKG}.
\subsection{Preprocessing and Filtering}
\label{sec:preprocessing}
The goal of preprocessing and filtering is to collect entities and map entity pairs to candidate relation descriptions.
To ensure correctness, we use Wikipedia as the source corpus, which is a high-quality corpus covering a wide range of domains.
Because this process mainly relies on heuristic rules and existing tools, to save space, we refer the readers to Appendix \ref{app:preprocessing} for the details.
\begin{figure*}[tp!]
\centerline{\includegraphics[width=\linewidth]{dep_s1-2.pdf}}
\vspace{-1mm}
\caption{Dependency tree of $s_1$.}
\vspace{-4mm}
\label{fig:dep_s1}
\end{figure*}
\subsection{Scoring}
In this section, we design a scoring function to measure the quality of relation descriptions.
Since we use Wikipedia as the source corpus, the \textit{correctness} of the extracted sentences can be largely guaranteed;
thus, we focus on measuring \textit{explicitness} and \textit{significance} of candidate relation descriptions.
\subsubsection{Shortest Dependency Path as Relation}
Inspired by \citet{wu2010open}, we use the shortest dependency path to represent the relation pattern between the target entities in a sentence.
For instance, Figure \ref{fig:dep_s1} shows the dependency tree of $s_1$ processed by spaCy\footnote{\url{https://spacy.io}}.
% For instance, Figure \ref{fig:dep_s1} shows the dependency tree of $s_1$.
The shortest path between \textit{machine learning} and \textit{algorithm} is:
``learning $\overleftarrow{nsubj}$ explores $\overrightarrow{dobj}$ study $\overrightarrow{prep\vphantom{b}}$ of $\overrightarrow{pobj}$ algorithms''.
Following their notation, we call such a path a \textit{corePath}.
To represent the relation pattern, we collect dependencies in the path and append ``i\_'' to the dependencies with an inversed direction. E.g., the relation pattern for the above path is [$i\_nsubj, dobj, prep, pobj$].
We remove dependencies that do not affect human understanding. Specifically, we drop the \textit{conj} and \textit{appos} dependencies and replace two consecutive \textit{prep} with one.
Besides \textit{corePath}, we also collect the shortest paths between the \textit{corePath} and the tokens outside the \textit{corePath} to represent the relationships between entity relationships and tokens.
For instance, in Figure \ref{fig:dep_s1}, \textit{construction} is a token outside the \textit{corePath} between \textit{machine learning} and \textit{algorithm}. The shortest path between it and the \textit{corePath} is:
``study $\overrightarrow{conj}$ construction''. We call this kind of path as \textit{subPath}. Similar to \textit{corePath}, we generate the relation pattern from \textit{subPath} and drop the \textit{conj}, \textit{appos} and \textit{compound} dependencies.
\subsubsection{Explicitness}
\label{sec:explicitness}
Given two entities and a candidate relation description $s$, we measure the explicitness by calculating the normalized logarithmic frequency of the relation pattern of the \textit{corePath}:
\begin{equation}
\label{eq:E}
ExpScore(s) = \frac{\log(f_p+1)}{\log(f_{max}+1)},
\end{equation}
where $f_{max}$ is the frequency of the most frequent \textit{corePath} relation pattern and $f_p$ is the frequency of the relation pattern in the present \textit{corePath}. The intuition here is that humans tend to use explicit structure to explain relations. Thus, we assume that a relation description is more explicit if its relation pattern is more frequent. Intuitively, if a relation pattern is unpopular, it is likely that this pattern is either too complicated or contains some rarely used dependencies. Both of these cases may increase the difficulty in reasoning.
Similar to \cite{wu2010open},
we only consider patterns that start with \textit{nsubj} or \textit{nsubjpass}, indicating that one of the target entities is the subject of the sentence. This restriction helps increase the explicitness of the selected relation description sentences because if one entity is the subject, the sentence is likely to contain a ``argument-predicate-argument'' structure connecting the target entities.
\subsubsection{Significance}
We measure the significance as the proportion of information that is relevant to the entity relationship in a sentence.
To measure the relevance of each token in the sentence to the entity relationship, we divide tokens into three categories: 1) \textit{core token} if the token is in the \textit{corePath}; 2) \textit{modifying token} if the token is in a \textit{subPath} that is connected to the \textit{corePath} through a modifying dependency; and 3) \textit{irrelevant token} for the rest tokens.
The intuition here is that a sub-dependency tree connected to the \textit{corePath} with a modifying dependency is supposed to modify the relationship. We predefined a set of modifying dependencies in Table \ref{table:modifying_dependencies}.
We calculate a score for each token in the sentence based on its category and dependency analysis. Then, the significance score is the average of all the token's scores.
Formally, for a candidate relation description $s$, the significance score is calculated as
\begin{equation}
\label{eq:S}
SigScore(s) = \frac{\sum_{t \in s}{w(t)}}{|s|},
\end{equation}
where
\begin{equation}
w(t) =
\begin{cases}
1 & \text{if $t \in ct$} \\
% \frac{max(\log(f'_{p_t}) - \log(f'_{min}),0)}{\log(f'_{max}) - \log(f'_{min})} & \text{if $t \in mp$}, \\
\frac{\log(f'_{p_t}+1)}{\log(f'_{max}+1)} & \text{if $t \in mt$} \\
0 & \text{otherwise}
\end{cases},
\end{equation}
where $ct$ is the set of \textit{core tokens} and $mt$ is the set of \textit{modifying tokens}.
$f'_{p_t}$ is the frequency of the \textit{subPath} relation pattern from the \textit{corePath} to the present token $t$ and $f'_{max}$ is the frequency of the most frequent \textit{subPath} relation pattern. The intuition is: with higher relation pattern frequency, the modifying token is more explicitly related to the entity relationship, and thus, should have a higher score. This also comes with another useful characteristic: the score will decrease token by token as we move along the \textit{subPath} because the frequency of a \textit{subPath} relation pattern cannot be greater than the frequency of its parent. With this characteristic, we can penalize the long modifying \textit{subPath} as it will distract the focus from the entity relationship and is less explicitly related to the relationship.
\subsubsection{Relation Descriptive Score}
To calculate the explicitness and significance, we need to build a database of relation patterns for both \textit{corePath} and \textit{subPath}. We construct both databases with the candidate relation descriptions and corresponding entity pairs collected from Section \ref{sec:preprocessing} with spaCy.
We also require the two target entities in the sentence are highly related. Intuitively, if two entities are more related, the sentences containing them are more likely to be relation descriptions; therefore, the extracted \textit{corePath} relation patterns are more likely to indicate entity relationships.
We measure the relevance of two entities by calculating the cosine similarity of the entity embeddings in Wikipedia2Vec. We filter out entity pairs (and the associated sentences) with a relevance score $< 0.5$.
With the databases of relation patterns, we can calculate the explicitness and significance scores for a candidate relation description. The final score, named \textbf{\textit{Relation Descriptive Score (RDScore)}}, is computed as the harmonic mean:
\begin{equation}
\label{eq:RDScore}
RDScore(s) = 2 \cdot \frac{ExpScore(s) \cdot SigScore(s)}{ExpScore(s) + SigScore(s)}.
\end{equation}
For each entity pair,
we calculate \textit{RDScore} for all the candidate relation descriptions and select the candidate with the highest score as the final relation description.
To build an initial DKG, we keep edges with an entity relevance score $\geq 0.5$\footnote{Since there is no boundary that delineates whether two entities are related, we consider the relevance threshold as a hyperparameter.}
and with a relation description whose \textit{RDScore} $\geq 0.6$\footnote{This threshold is also a hyperparameter to balance the density of the graph and the quality of relation descriptions.}. We refer to this graph as \textbf{Wiki-DKG$_0$}.
\section{Relation Description Generation}
\label{sec:generation}
\begin{figure*}[tp!]
\centerline{\includegraphics[width=0.95\linewidth]{RelationSyn.pdf}}
\vspace{-1mm}
\caption{The framework of \textit{RelationSyn}. Given entity pair $(x, y)$ whose relation description is not present in the initial DKG, we first retrieve several reasoning paths from the graph. And then, we encode (local synthesize) each reasoning path into a latent vector and concatenate all the latent vectors. Finally, we decode (global synthesize) the vector to produce relation description $s'$ for $x$ and $y$.}
\vspace{-4mm}
\label{fig:RelationSyn}
\end{figure*}
In the previous section, we extract relation descriptions for entity pairs in a self-supervised manner and build an initial DKG with Wikipedia automatically.
However, for some related entity pairs, there may not exist a sentence that contains both entities; and although such a sentence exists, it may not be extracted by the system.
To solve this problem, in this section, we introduce \textit{Relation Description Generation} -- generating relation descriptions for interesting entity pairs.
% \subsection{Model}
\label{sec:relationsyn}
We form relation description generation as a conditional text generation task: given two entities, generating a sentence describing the relationship between them with the initial DKG. Formally, we apply the knowledge-enhanced sequence-to-sequence formulation \cite{yu2020survey}: given an entity pair $(x,y)$ and an initial DKG $\mathcal{G}_0$, the probability of the output relation description $s$ is computed auto-regressively:
\begin{equation}
P(s|x, y, \mathcal{G}_0) = \prod_{i=1}^{m} P(s_i|s_{0:i-1}, x, y, \mathcal{G}_0),
\end{equation}
where $m$ is the length of $s$, $s_i$ is the $i$th token of $s$, and $s_0$ is a special start token.
To incorporate the DKG for generation, we propose \textit{\textbf{Relation Description Synthesizing (RelationSyn)}}. RelationSyn consists of two processes: first retrieving relevant relation descriptions (reasoning paths) from the graph and then synthesizing them into a final relation description (Figure \ref{fig:RelationSyn}).
\subsection{Retrieval}
\label{sec:retrieval}
To generate a relation description, the model needs knowledge about the target entities and their relationship. To provide knowledge,
we retrieve reasoning paths of the target entities from the graph.
In DKG, we define a reasoning path $q$ as a path connecting the target entities, which is called $k$-hop if it is connected by $k$ edges.
For instance, in Figure \ref{fig:RelationSyn}, there are two $2$-hop reasoning paths between $x$ and $y$: $(x,s_{11},e_{11},s_{12},y)$ and $(x,s_{21},e_{21},s_{22},y)$, and two $3$-hop reasoning paths: $(x,s_{21},e_{21},s_{23},e_{32},s_{33},y)$ and $(x,s_{31},e_{31},s_{32},e_{32},s_{33},y)$ in the DKG.
To measure the quality of reasoning paths, we define \textit{PathScore} as the harmonic mean of \textit{RDScore} of relation descriptions in the path:
\begin{equation}
PathScore(q) = \frac{|\mathcal{S}_q|}{\sum_{s \in \mathcal{S}_q}{\frac{1}{RDScore(s)}}},
\end{equation}
where $\mathcal{S}_q$ is the set of relation descriptions in $q$, and $|\mathcal{S}_q|=k$.
Reasoning paths are helpful for relation description generation. For instance,
% to generate a relation description for \textit{deep learning} and \textit{computer science},
from reasoning path $(\text{deep learning},$ $s_1',$ $\text{machine learning}, s_2',$ $\text{computer science})$ (refer to Figure \ref{fig:graph} for $s_1'$ and $s_2'$), we can infer the relationship between \textit{deep learning} and \textit{computer science}: \textit{deep learning} is a subset of \textit{ML}, while \textit{ML} is a subfield of \textit{CS}; therefore, \textit{deep learning} is a subfield of \textit{CS}. Besides, from the sentence description, we can acquire rich entity information about \textit{deep learning} and \textit{computer science}, e.g., \textit{deep learning} focuses heavily on the use of artificial intelligence.
However, not all reasoning paths are equally useful.
Longer reasoning paths are usually more difficult to reason, while paths with higher \textit{PathScore} usually contain more explicit and significant relation descriptions.
Therefore, when retrieving reasoning paths for an entity pair, we first sort the paths by their length (shorter first) and then by their \textit{PathScore} (higher first).
\subsection{Synthesizing}
According to Section \ref{sec:retrieval}, we may retrieve multiple reasoning paths for an entity pair whose relation description is missed in the initial DKG.
In this section, we focus on synthesizing relation descriptions in the retrieved reasoning paths into a final relation description of the target entities based on T5 \cite{raffel2020exploring} and Fusion-in-Decoder \cite{izacard2021leveraging}.
We first convert each reasoning path to a sequence using the following encoding scheme: e.g., $(x,s_{31},e_{31},s_{32},e_{32},s_{33},y)$ $\to$ ``entity1: $x$ entity2: $y$ path: $x$; $e_{31}$; $e_{32}$; $y$ sentence1: $s_{31}$ sentence2: $s_{32}$ sentence3: $s_{33}$''. And then, we encode the sequence with the encoder of T5. In this way, the relation descriptions in each reasoning path are synthesized into a latent vector, named ``\textbf{local synthesizing}''.
After local synthesizing, we concatenate the latent vectors of all the retrieved reasoning paths to form a global latent vector. The decoder of T5 performs attention over the global latent vector and produces the final relation description. We name this process as ``\textbf{global synthesizing}''.
Combining retrieval and synthesizing, given two entities, we first retrieve $m$ reasoning paths connecting the target entities according to their length and \textit{PathScore}, and then synthesize them to produce the target relation description. We refer to this model as \textbf{RelationSyn-$m$}.
\section{Evaluation}
In this section, we verify the proposed methods for building DKGs by conducting experiments on relation description extraction and generation.
% \subsection{DKG Construction}
\subsection{Relation Description Extraction}
\label{sec:exp_construction}
We first present the statistics of the initial DKG built with Wikipedia in Table \ref{table:Wiki-DKG}.
To evaluate the quality of relation descriptions in the graph, we randomly sample 100 entity pairs from the graph\footnote{More specifically, for better comparison with generation later, we sample 100 entity pairs from the test set in Table \ref{table:data_completion}.} and ask three human annotators to assign a graded value (1-5) for each relation description according to Table \ref{table:rating_scale}.
Since previous works on relation description extraction are supervised and only limited to several explicit relation types, e.g., 9 in \cite{voskarides2015learning}, it is impractical and meaningless to compare with them. For instance, the relationship of (\textit{Arthur
Samuel}, \textit{Machine Learning}) is not available or even not considered by all the previous methods. Therefore, we verify the effectiveness of our model by comparing different variants of the model:
\begin{itemize}[noitemsep,nolistsep,leftmargin=*]
\item \textbf{Random}: A sentence with the target entities is randomly selected as the relation description.
\item \textbf{ExpScore}: The sentence with the highest \textit{explicitness} is selected according to Eq. \eqref{eq:E}.
\item \textbf{SigScore}: The sentence with the highest \textit{significance} is selected according to Eq. \eqref{eq:S}.
\item \textbf{RDScore}: The sentence with the highest \textit{RDScore} is selected according to Eq. \eqref{eq:RDScore}.
\end{itemize}
\begin{table}[tp!]
\small
\begin{center}
\begin{tabular}{r|r|c}
\toprule
\textbf{\# nodes} & \textbf{\# edges} & \textbf{average sentence length} \\
\midrule
1,294,502 & 2,503,695 & 19.6\\
\bottomrule
\end{tabular}
\end{center}
\vspace{-1mm}
\caption{The statistics of Wiki-DKG$_0$.}
\vspace{-1mm}
\label{table:Wiki-DKG}
\end{table}
\begin{table}[tp!]
\small
\begin{center}
\begin{tabular}{l|c}
\toprule
& \textbf{Rating (1-5)} \\
\midrule
Random & 2.84 \\
\hline
ExpScore & 3.80 \\
SigScore & 3.92 \\
\hline
RDScore & \textbf{4.20} \\
\bottomrule
\end{tabular}
\vspace{-1mm}
\caption{Qualitative results of extraction.}
\vspace{-4mm}
\label{table:human_evaluation_extracion}
\end{center}
\end{table}
Table \ref{table:human_evaluation_extracion} shows the human evaluation results for relation description extraction.
From the results, we observe that both our explicitness and significance measurements are important to ensure a good relation description.
In addition, \textit{RDScore} achieves an average rating of 4.20, which means that most of the selected sentences are high-quality relation descriptions, further indicating that the quality of the constructed initial Wiki-DKG is high.
In Appendix \ref{app:ablation}, we also verify our method indirectly by using relation descriptions extracted with different variants as input for generation.
\subsection{Relation Description Generation}
\label{sec:exp_completion}
In this section, we perform experiments on relation description generation.
\subsubsection{Experimental Setup}
{\flushleft \textbf{Data construction}.}
% We form the relation description generation task as follows:
We build the dataset for relation description generation as follows:
for an entity pair with a relation description in Wiki-DKG$_0$, we hide the relation description and consider it as the target for generation. The goal is to recover/generate the target relation description with the rest of the graph\footnote{To increase the difficulty of the task, we assume these two entities do not co-occur in the corpus, i.e., we do not utilize any sentence containing both the target entities for generation.}.
For instance, in Figure \ref{fig:RelationSyn}, we hide the edge (relation description $s$) between $x$ and $y$ and use the remaining reasoning paths to recover $s$.
We train and test on entity pairs with $\geq5$ reasoning paths connecting them. The statistics of the data are reported in Table \ref{table:data_completion}.
\begin{table}[tp!]
\small
\begin{center}
\begin{tabular}{c|r|r|r}
\toprule
& \textbf{train} & \textbf{valid} & \textbf{test} \\
\midrule
size & 672,930 & 14,019 & 14,020 \\
\bottomrule
\end{tabular}
\end{center}
\vspace{-1mm}
\caption{The statistics of data for generation.}
\vspace{-4mm}
\label{table:data_completion}
\end{table}
\begin{table*}[tp!]
\begin{center}
\small
% \footnotesize
% \scriptsize
% \setlength\tabcolsep{1.5pt} % default value: 6pt
\begin{tabular}{l|c|c|c|c}
\toprule
& \textbf{BLEU} & \textbf{ROUGE} & \textbf{METEOR} & \textbf{BERTScore} \\
% & \textbf{BL} & \textbf{R-L} & \textbf{MT} & \textbf{BS} \\
\midrule
RealtionBART-Vanilla \cite{huang2022open} & 20.36 & 40.95 & 20.97 & 82.27 \\
RealtionBART-MP + PS \cite{huang2022open} & 20.97 & 41.22 & 21.25 & 82.25 \\
\hline
\textbf{RelationSyn-$0$} & 21.50 & 40.35 & 21.08 & 82.68 \\
\hline
\textbf{RelationSyn-$1$} & 23.61 & 42.16 & 22.28 & 83.45 \\
\textbf{RelationSyn-$3$} & 24.12 & 43.00 & 22.67 & 83.77 \\
\textbf{RelationSyn-$5$} & \textbf{24.66} & \textbf{43.25} & \textbf{22.93} & \textbf{83.79} \\
\bottomrule
\end{tabular}
\vspace{-1mm}
\caption{Quantitative results of relation description generation.}
\vspace{-4mm}
\label{table:quan_completion}
\end{center}
\end{table*}
{\flushleft \textbf{Models}.} The task of relation description generation is relevant to \textit{Open Relation Modeling} \cite{huang2022open} -- a recent work aimed at generating sentences capturing general relations between entities conditioned on entity pairs. To the best of our knowledge, no other existing work can generate relation descriptions for any two related entities (since open relation modeling has only just been introduced). Therefore, we mainly compare the models proposed in \cite{huang2022open}
with several variants of our model:
\begin{itemize}[noitemsep,nolistsep,leftmargin=*]
\item \textbf{RelationBART (Vanilla)}: The vanilla model proposed in \cite{huang2022open} for generating entity relation descriptions, where BART \cite{lewis2020bart} is fine-tuned on a training data whose inputs are entity pairs and outputs are corresponding relation descriptions.
\item \textbf{RelationBART-MP + PS}: The best model proposed in \cite{huang2022open}, which incorporates Wikidata by selecting the most interpretable and informative reasoning path in the KG automatically for helping generate relation descriptions.
\item \textbf{RelationSyn-$0$}: A reduced variant of our model, where the encoding scheme of the input is only ``entity1: $x$ entity2: $y$'', i.e., no reasoning path and relation description is input to the encoder.
\item \textbf{RelationSyn-$m$}: The proposed relation description synthesizing model (Section \ref{sec:relationsyn}), where $m$ is the maximum number of retrieved reasoning paths for an entity pair.
\end{itemize}
{\flushleft \textbf{Metrics}.} We perform both quantitative and qualitative evaluation. Following \cite{huang2022open}, we apply several automatic metrics for, including BLEU \cite{papineni2002bleu}, ROUGE-L \cite{lin2004rouge}, METEOR \cite{banerjee2005meteor}, and BERTScore \cite{zhang2019bertscore}.
Among them, BLEU, ROUGE, and METEOR focus on measuring surface similarities between the generated relation descriptions and the target relation descriptions, and BERTScore is based on the similarities of contextual token embeddings.
We also ask three human annotators to evaluate the output relation descriptions with the same rating scale in Table \ref{table:rating_scale}.\footnote{Details about implementation are in Appendix \ref{app:implementation}.}
\begin{table}[tp!]
\small
\begin{center}
\begin{tabular}{l|c}
\toprule
& \textbf{Rating (1-5)} \\
\midrule
\textit{Random} & 2.84 \\
\textit{RDScore (Oracle)} & 4.20 \\
\hline
RealtionBART-MP + PS & 3.12 \\
\textbf{RelationSyn-$0$} & 3.10 \\
\textbf{RelationSyn-$1$} & 3.36 \\
\textbf{RelationSyn-$5$} & \textbf{3.48} \\
\bottomrule
\end{tabular}
\vspace{-1mm}
\caption{Qualitative results of generation.}
\vspace{-5mm}
\label{table:qual_completion}
\end{center}
\end{table}
\subsubsection{Quantitative Evaluation}
Table \ref{table:quan_completion} reports the results of relation description generation with the automatic metrics. We observe that our best model \textit{RelationSyn-$5$} outperforms the state-of-the-art model for open relation modeling significantly. We also observe that \textit{RelationSyn-$1$} performs better than \textit{RelationSyn-$0$}, which means that reasoning paths in DKG are helpful for relation description generation. In addition, as the number of reasoning paths, i.e., $m$, increases, the performance of RelationSyn-$m$ improves. This demonstrates that the proposed model can synthesize multiple relation descriptions in different reasoning paths into a final relation description.
\subsubsection{Qualitative Evaluation}
We also conduct qualitative experiments to measure the quality of generated relation descriptions. For a better comparison with extraction, we sample the same 100 entity pairs from the test set as in Section \ref{sec:exp_construction}.
% Table \ref{table:qual_completion} summarizes the results.
From the results in Table \ref{table:qual_completion}, we observe that the quality of generated relation descriptions is higher than that of random sentences containing the target entities.
The best model, \textit{RelationSyn-$5$}, achieves a rating of 3.48, which means the model can generate reasonable relation descriptions. However, the performance is still much worse than \textit{Oracle}, i.e., relation descriptions extracted by our best extraction model (\textit{RDScore}).
This indicates that generating high-quality relation descriptions is still a challenging task.
\subsection{Ablation Study, Case Study and Error Analysis}
\label{sec:ablation}
We provide ablation study in Appendix \ref{app:ablation} and case study and error analysis in Appendix \ref{sec:examples}.
\section{Conclusion and Discussion}
In this paper, we propose DKG -- an open and interpretable form of modeling relationships between entities. To avoid tremendous human efforts, we design a novel self-supervised learning approach to extract relation descriptions from Wikipedia.
To provide relation descriptions for related entity pairs whose relation descriptions are not extracted in the previous step, we study relation description generation by synthesizing relation descriptions in the retrieved reasoning paths.
Experiments on both extraction and generation demonstrate that our system can produce high-quality relation descriptions for building DKGs.
There are several promising directions for future work. First, in this paper, DKGs are modeled as undirected graphs, it is interesting to explore the direction of the edge and the diversity of relation descriptions. Second, we build a DKG with Wikipedia, future work may explore scaling up the graph and constructing domain-specific DKGs with other corpora. Third, DKGs can be utilized as knowledge sources to facilitate downstream applications and related tasks such as relation extraction \cite{bach2007review} and knowledge graph completion \cite{lin2015learning}.
% Entries for the entire Anthology, followed by custom entries
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}
\clearpage
% \newpage
\appendix
\begin{table}[tp!]
% \small
\scriptsize
\begin{center}
\begin{tabular}{l|c}
\toprule
\textbf{Dependency label} & \textbf{Description} \\
\midrule
acl & clausal modifier of noun (adjectival clause) \\
\hline
advcl & adverbial clause modifier \\
\hline
advmod & adverbial modifier \\
\hline
amod & adjectival modifier \\
\hline
det & determiner \\
\hline
mark & marker \\
\hline
meta & meta modifier \\
\hline
neg & negation modifier \\
\hline
nn & noun compound modifier \\
\hline
nmod & modifier of nominal \\
\hline
npmod & noun phrase as adverbial modifier \\
\hline
nummod & numeric modifier \\
\hline
poss & possession modifier \\
\hline
prep & prepositional modifier \\
\hline
quantmod & modifier of quantifier \\
\hline
relcl & relative clause modifier \\
\hline
appos & appositional modifier \\
\hline
aux & auxiliary \\
\hline
auxpass & auxiliary (passive) \\
\hline
compound & compound \\
\hline
cop & copula \\
\hline
ccomp & clausal complement \\
\hline
xcomp & open clausal complement \\
\hline
expl & expletive \\
\hline
punct & punctuation \\
\hline
nsubj & nominal subject \\
\hline
csubj & clausal subject \\
\hline
csubjpass & clausal subject (passive) \\
\hline
dobj & direct object \\
\hline
iobj & indirect object \\
\hline
obj & object \\
\hline
pobj & object of preposition \\
\bottomrule
\end{tabular}
\caption{Manually collected modifying dependencies in spaCy.}
\vspace{-4mm}
\label{table:modifying_dependencies}
\end{center}
\end{table}
\section{Preprocessing and Filtering}
\label{app:preprocessing}
We introduce our preprocessing to the raw Wikipedia dump\footnote{\url{https://dumps.wikimedia.org} (version: enwiki/20210320)}.
For each article, we extract the plain text by WikiExtractor\footnote{\url{https://github.com/attardi/wikiextractor}}.
We split the Wikipedia articles into sentences with the NLTK library\footnote{\url{https://www.nltk.org}} and map entity pairs to candidate relation descriptions with the following steps:
{\flushleft {\textbf{Entity collection}}}. We collect Wikipedia page titles (surface form) as our entities. To acquire knowledge and utilize the pre-trained entity embeddings in Wikipedia2Vec \cite{yamada2020wikipedia2vec} in the later steps, we only keep entities that can be recognized by Wikipedia2Vec.
{\flushleft {\textbf{Local mention-entity mapping}}}. Wikipedia2Vec uses hyperlinks to collect a global mention-entity dictionary to map the entity mention to the referent entities, like mapping ``apple'' to ``Apple Inc'' or ``Apple (food)''.
In this work, we follow a similar approach to build the mapping.
To maintain high accuracy and low ambiguity, we craft the entity mention from the entity by removing the content wrapped by parenthesis and the content after the first comma. For example, a mention-entity pair could be (``Champaign'', ``Champaign, Illinois'') or (``Python'', ``Python (programming language)'').
Unlike Wikipedia2Vec, we create a local dictionary for each Wikipedia page.
When processing a page, we dynamically update the dictionary with mention-entity pairs collected from the hyperlinks, and extract the entity occurrence with the updating dictionary in one pass. This can reduce the ambiguity when two entities with the same entity mention co-occur on one page and also avoid collecting trivial entity occurrence on the page.
{\flushleft {\textbf{Hyperlink mapping correction}}}.
Using hyperlinks to collect entities will lead to errors under some conditions:
1) The original link is redirected to a new page, where the title does not match with the entity in the link; 2) The entity in the link is lower-cased and thus, does not match with any title. Under the first condition, we just skip this entity because we require that the entity mention must appear in the sentence to prove its occurrence. Under the second situation, if there is only one page title matching with the entity under the case-insensitive setting, we correct the entity to this page title. Otherwise, if there is more than one match, we use the entity embeddings in Wikipedia2Vec to measure the cosine similarity between each matched title and the title of the current page and correct the entity with the most relevant one.
\begin{table}[tp!]
\small
\setlength\tabcolsep{1.5pt} % default value: 6pt
% \begin{tabular}{c|p{0.9\linewidth}}
\begin{tabular}{c|p{0.85\linewidth}}
\toprule
\textbf{Rating} & \multicolumn{1}{c}{\textbf{Criterion}} \\
\midrule[0.75pt]
5 & The relation description is explicit, significant, and correct, with which users can understand the relationship correctly and easily. \\
\hline
4 & The relation description is a bit less explicit (reasoning is a bit indirect or description is a bit unclear), less significant (containing a little irrelevant content), and less correct (containing minor errors that do not affect the understanding). \\
\hline
3 & The relation description is fairly explicit, significant, and correct, while users can still understand the relationship. \\
\hline
2 & The relation description is not explicit (reasoning is difficult or description is unclear), significant (containing much irrelevant content), or correct (containing major errors that affect the understanding), while users can still infer the relationship to some extent. \\
\hline
1 & The relation description is completely wrong or does not show any relationship between the two entities. \\
\bottomrule
\end{tabular}
\caption{Annotation guidelines excerpt.}
\vspace{-4mm}
\label{table:rating_scale}
\end{table}
\begin{table*}[tp!]
\begin{center}
\small
% \footnotesize
% \scriptsize
% \setlength\tabcolsep{1.5pt} % default value: 6pt
\begin{tabular}{l|c|c|c|c}
\toprule
& \textbf{BLEU} & \textbf{ROUGE} & \textbf{METEOR} & \textbf{BERTScore} \\
% & \textbf{BL} & \textbf{R-L} & \textbf{MT} & \textbf{BS} \\
\midrule
\textbf{RelationSyn-$0$} & 21.50 & 40.35 & 21.08 & 82.68 \\
\hline
\textbf{RelationSyn-$1$ (w/o sentence)} & 22.69 & 41.26 & 21.69 & 83.03 \\
\textbf{RelationSyn-$1$ (w/ Random)} & 23.04 & 41.93 & 22.05 & 83.30 \\
\textbf{RelationSyn-$1$ (w/ ExpScore)} & 23.32 & 41.95 & 22.13 & 83.29 \\
\textbf{RelationSyn-$1$ (w/ SigScore)} & 23.31 & 42.14 & 22.15 & 83.39 \\
\hline
\textbf{RelationSyn-$1$} & \textbf{23.61} & \textbf{42.16} & \textbf{22.28} & \textbf{83.45} \\
\bottomrule
\end{tabular}
\vspace{-1mm}
\caption{Ablation study for relation description generation.}
\vspace{-4mm}
\label{table:ablation_study}
\end{center}
\end{table*}
{\flushleft {\textbf{Filtering}}}.
\label{sec:filtering}
Sometimes the entity mention extracted from the sentence may be part of a bigger noun phrase, which is not an entity mention. For example, suppose we recognize ``algorithm'' and ``graph'' as entity mentions in the sentence ``The breadth-first-search algorithm is a way to explore the vertexes of a graph layer by layer.'' However, this is not a good relation description between``algorithm'' and ``graph'' because the subject is ``breadth-first-search algorithm'' rather than ``algorithm''. Therefore, it is necessary to determine the completed noun phrase for each entity mention. With the dependency tree of the sentence, we recursively find all the child tokens and the ancestor tokens that are connected to the entity mention with a \textit{compound} dependency. To avoid any confusion, we simply reject the entity occurrence if its completed noun phrase and entity mention are different.
Besides, to ensure that the length of relation descriptions is reasonable, we only remain the sentences with the number of tokens $\in [5, 50]$. We also only remain sentences whose shortest dependency path pattern between two target entities starts with \textit{nsubj} or \textit{nsubjpass} (more details are in Section \ref{sec:explicitness}).
\section{Implementation Details}
\label{app:implementation}
We train and evaluate all the baselines and variants on the same train/valid/test split. For \textit{RelationBART (Vanilla)} and \textit{RelationBART-MP + PS}, we apply the official implementation\footnote{\url{https://github.com/jeffhj/open-relation-modeling}} and adopt the default hyperparameters. The training converges in 50 epochs. For our models, we modify the implementation of Fusion-in-Decoder\footnote{\url{https://github.com/facebookresearch/FiD}} and initialize the model with the T5-base configuration. All the baseline models for \textit{RelationSyn} are trained under the same batch size of 8 with a learning rate of $0.0001$ and evaluated on the
validation set every 5000 steps. The training is considered converged and terminated with no better performance on the validation set in 20 evaluations. The training of all models converges in 20 epochs. The training time is about one week on a single NVIDIA A40 GPU.
\section{Ablation Study}
\label{app:ablation}
To further validate the proposed methods, we conduct an ablation study with the following variants:
\begin{itemize}[noitemsep,nolistsep,leftmargin=*]
\item \textbf{RelationSyn-$1$ (w/o sentence)}: a variant of \textit{RelationSyn-$1$}, where the encoding scheme of the input is only ``entity1: $x$ entity2: $y$ path: $x$; $e_{31}$; $e_{32}$; $y$'', i.e., no relation description is input to the encoder.
\item \textbf{RelationSyn-$1$ (w/ Random)}: The input relation descriptions to \textit{RelationSyn} are extracted by \textit{Random} (i.e., a random sentence containing the target entities), instead of \textit{RDScore}.
\item \textbf{RelationSyn-$1$ (w/ ExpScore)}: The input relation descriptions to \textit{RelationSyn} are extracted by \textit{ExpScore}.
\item \textbf{RelationSyn-$1$ (w/ SigScore)}: The input relation descriptions to \textit{RelationSyn} are extracted by \textit{SigScore}.
\end{itemize}
From the results in Table \ref{table:ablation_study}, we observe that \textit{RelationSyn-$1$} achieves better results than all the reduced variants. However, the difference is not quite significant. We think this is because, although relation descriptions extracted by \textit{Random}, \textit{ExpScore} and \textit{SigScore} are worse than those extracted by \textit{RDScore}, they can still provide reasonable knowledge for generating relation descriptions for the target entities.
In addition, the results are also consistent with the results in Table \ref{table:qual_completion}. This indirectly verifies that the combination of \textit{explicitness} and \textit{significance}, i.e., \textit{RDScore}, can extract better relation descriptions, since better relation descriptions of related entities are expected to be synthesized into better relation descriptions of the target entities.
\section{Case Study and Error Analysis}
\label{sec:examples}
\begin{table*}[tp!]
% \small
% \footnotesize
\scriptsize
% \setlength\tabcolsep{1.5pt}
\setlength\tabcolsep{1.5pt}
\begin{tabular}{p{0.11\linewidth}|p{0.14\linewidth}|p{0.14\linewidth}|p{0.14\linewidth}|p{0.14\linewidth}|p{0.14\linewidth}|p{0.14\linewidth}}
\toprule
& \textbf{ExpScore} & \textbf{SigScore} & \textbf{RDScore} & \textbf{RelationSyn-$0$} & \textbf{RelationSyn-$1$} & \textbf{RelationSyn-$5$} \\
\midrule[1pt]
(Schizophrenia, Hallucination) & During the early 20th century, auditory hallucinations were second to visual hallucinations in frequency, but they are now the most common manifestation of schizophrenia, although rates vary between cultures and regions. & Auditory verbal hallucinations attributed to an external source, rather than internal, are considered the defining factor for the diagnoses of schizophrenia. & A hallucination of a single individual person of one or more talking voices is particularly associated with psychotic disorders such as schizophrenia, and hold special significance in diagnosing these conditions. & The hallucinations of the nocturnal synapse are associated with schizophrenia. & Unlike most other forms of hallucination, schizophrenia does not cause a symptom change. & Schizophrenia is characterized by delusions, hallucinations, and other negative symptoms. \\
\hline
(Moth, Momphidae) & The Momphidae, or mompha moths, is a family of moths with some 115 described species. & Mompha terminella is a moth in the family Momphidae found in Europe and North America. & The Momphidae, or mompha moths, is a family of moths with some 115 described species. & The Momphidae are a family of moths in the superfamily Gelechioidea. & The Momphidae are a family of moths in the superfamily Gelechioidea. & The Momphidae are a family of moths in the superfamily Gelechioidea. \\
\hline
(Singapore, Indonesia) & China, Singapore, Malaysia, Australia, and Japan are the top five sources of visitors to Indonesia. & In 2007, Indonesia enacted a ban against exporting sand specifically to Singapore. & Indonesia is among the top source of foreign visitors to Singapore. & Singapore participated in the 2018 Asian Games in Jakarta and Palembang, Indonesia from 18 August to 2 September 2018. & Singapore is the second largest trading partner of Indonesia. & Singapore is the second largest trading partner of Indonesia in terms of foreign trade volume. \\
\hline
(Chota Nagpur Plateau, Giridih district) & Giridih district is a part of the Chota Nagpur plateau, with rocky soil and extensive forests. & Giridih district is a part of the Chota Nagpur plateau, with rocky soil and extensive forests. & Giridih district is a part of the Chota Nagpur plateau, with rocky soil and extensive forests. & Giridih district is situated on the Chota Nagpur Plateau. & Giridih district is located on the Chota Nagpur Plateau. & Giridih district is spread over a part of the Chota Nagpur Plateau. \\
\hline
(Sun, Hipparchus) & Theon of Smyrna wrote that according to Hipparchus, the Sun is 1,880 times the size of the Earth, and the Earth twenty-seven times the size of the Moon; apparently this refers to volumes, not diameters. & Hipparchus already had developed a way to calculate the longitude of the Sun at any moment. & Both Aristarchus and Hipparchus drastically underestimated the distance of the Sun from the Earth. & Hipparchus, in his "Summa Thermodynamics", identifies the Sun with the Sun, a planet with a radius of. & Hipparchus, in his "History of the Sun", referred to the Sun as "the Sun of the sand". & In 127 BC, Hipparchus measured the apparent diameters of the Sun and Moon with his "diopter". \\
\bottomrule
\end{tabular}
\caption{Sample of relation descriptions produced by \textit{ExpScore}, \textit{SigScore}, \textit{RDScore}, \textit{RelationSyn-$0$}, \textit{RelationSyn-$1$}, and \textit{RelationSyn-$5$}.}
\vspace{-4mm}
\label{table:case}
\end{table*}
In Table \ref{table:case}, we show some sample outputs in the test set of relation description generation of three extraction models: \textit{ExpScore}, \textit{SigScore}, \textit{RDScore}, and three generation models: \textit{RelationSyn-$0$}, \textit{RelationSyn-$1$}, \textit{RelationSyn-$5$}.
From the results, we observe that if we only consider the explicitness of the sentence, the selected relation description may contain a lot of stuff that is irrelevant to the entity relationship, e.g., (\textit{Schizophrenia}, \textit{Hallucination}). And if we only consider the significance, the relationship between entities may be described implicitly; thus the relationship is difficult to reason out, e.g., (\textit{Singapore}, \textit{Indonesia}). And the combination of them, i.e., \textit{RDScore}, yields better relation descriptions.
For generation, sometimes, the models generate similar relation descriptions, e.g., (\textit{Moth}, \textit{Momphidae}) and \textit{(Chota Nagpur Plateau, Giridih district)}; while in other cases the relation descriptions generated by the models are quite different. From the results, we notice that \textit{RelationSyn-$0$} suffers severely from \emph{hallucinations}, i.e., generating irrelevant or contradicted facts. For instance, the relation descriptions generated for (\textit{Singapore}, \textit{Indonesia}) and (\textit{Sun}, \textit{Hipparchus}) are not correct or relevant. By incorporating relation descriptions in the reasoning paths as knowledge, hallucination is alleviated to some extent, leading to better performance of \textit{RelationSyn-$1$} and \textit{RelationSyn-$5$}.
From the human evaluation results, we also find that the correctness of relation descriptions extracted by \textit{RDScore} is largely guaranteed. However, sometimes, the extracted sentences are still a bit implicit or not significant. In contrast to this, the relation descriptions generated by \textit{RelationSyn} are usually explicit and significant (the average \textit{RDScore} of the relation descriptions generated by \textit{RelationSyn-$5$} is 0.83, compared to 0.80 of \textit{Oracle}), but contain major or minor errors. We think this is because most of the relation descriptions extracted by \textit{RDScore} are explicit and significant, and the generation model can mimic the dominant style of relation descriptions in the training set.
However, it is still challenging to generate fully correct relation descriptions by synthesizing existing relation descriptions.
We also examine the five entity pairs in Table \ref{table:case} in Wikidata. Among them, only the relation between \textit{Singapore} and \textit{Indonesia} is present in Wikidata. This further confirms that DKGs can model a wider range of entity relationships.
\end{document}
\begin{thebibliography}{29}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Agarwal et~al.(2021)Agarwal, Ge, Shakeri, and
Al-Rfou}]{agarwal2021knowledge}
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021.
\newblock Knowledge graph based synthetic corpus generation for
knowledge-enhanced language model pre-training.
\newblock In \emph{Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies}, pages 3554--3565.
\bibitem[{Bach and Badaskar(2007)}]{bach2007review}
Nguyen Bach and Sameer Badaskar. 2007.
\newblock A review of relation extraction.
\newblock \emph{Literature review for Language and Statistics II}, 2:1--15.
\bibitem[{Banerjee and Lavie(2005)}]{banerjee2005meteor}
Satanjeev Banerjee and Alon Lavie. 2005.
\newblock Meteor: An automatic metric for mt evaluation with improved
correlation with human judgments.
\newblock In \emph{Proceedings of the acl workshop on intrinsic and extrinsic
evaluation measures for machine translation and/or summarization}, pages
65--72.
\bibitem[{Chen et~al.(2018)Chen, Xiong, Yan, and Wang}]{chen2018variational}
Wenhu Chen, Wenhan Xiong, Xifeng Yan, and William~Yang Wang. 2018.
\newblock Variational knowledge graph reasoning.
\newblock In \emph{Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long Papers)}, pages 1823--1832.
\bibitem[{Dognin et~al.(2020)Dognin, Melnyk, Padhi, dos Santos, and
Das}]{dognin2020dualtkb}
Pierre Dognin, Igor Melnyk, Inkit Padhi, Cicero dos Santos, and Payel Das.
2020.
\newblock Dualtkb: A dual learning bridge between text and knowledge base.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 8605--8616.
\bibitem[{Etzioni et~al.(2008)Etzioni, Banko, Soderland, and
Weld}]{etzioni2008open}
Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel~S Weld. 2008.
\newblock Open information extraction from the web.
\newblock \emph{Communications of the ACM}, 51(12):68--74.
\bibitem[{Hogan et~al.(2021)Hogan, Blomqvist, Cochez, d'Amato, Melo, Gutierrez,
Kirrane, Gayo, Navigli, Neumaier et~al.}]{hogan2021knowledge}
Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d'Amato, Gerard~de Melo,
Claudio Gutierrez, Sabrina Kirrane, Jos{\'e} Emilio~Labra Gayo, Roberto
Navigli, Sebastian Neumaier, et~al. 2021.
\newblock Knowledge graphs.
\newblock \emph{Synthesis Lectures on Data, Semantics, and Knowledge},
12(2):1--257.
\bibitem[{Huang et~al.(2022)Huang, Chang, Xiong, and Hwu}]{huang2022open}
Jie Huang, Kevin Chen-Chuan Chang, Jinjun Xiong, and Wen-mei Hwu. 2022.
\newblock Open relation modeling: Learning to define relations between
entities.
\newblock In \emph{Findings of the Association for Computational Linguistics:
ACL 2022}.
\bibitem[{Huang et~al.(2017)Huang, Zhang, Zhao, Ding, and
Wang}]{huang2017learning}
Jizhou Huang, Wei Zhang, Shiqi Zhao, Shiqiang Ding, and Haifeng Wang. 2017.
\newblock Learning to explain entity relationships by pairwise ranking with
convolutional neural networks.
\newblock In \emph{Proceedings of the 26th International Joint Conference on
Artificial Intelligence}, pages 4018--4025.
\bibitem[{Izacard and Grave(2021)}]{izacard2021leveraging}
Gautier Izacard and Edouard Grave. 2021.
\newblock Leveraging passage retrieval with generative models for open domain
question answering.
\newblock In \emph{EACL 2021-16th Conference of the European Chapter of the
Association for Computational Linguistics}, pages 874--880. Association for
Computational Linguistics.
\bibitem[{Ji et~al.(2021)Ji, Pan, Cambria, Marttinen, and
Philip}]{ji2021survey}
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S~Yu Philip. 2021.
\newblock A survey on knowledge graphs: Representation, acquisition, and
applications.
\newblock \emph{IEEE Transactions on Neural Networks and Learning Systems}.
\bibitem[{Kukich(1983)}]{kukich1983design}
Karen Kukich. 1983.
\newblock Design of a knowledge-based report generator.
\newblock In \emph{21st Annual Meeting of the Association for Computational
Linguistics}, pages 145--150.
\bibitem[{Lao et~al.(2011)Lao, Mitchell, and Cohen}]{lao2011random}
Ni~Lao, Tom Mitchell, and William Cohen. 2011.
\newblock Random walk inference and learning in a large scale knowledge base.
\newblock In \emph{Proceedings of the 2011 conference on empirical methods in
natural language processing}, pages 529--539.
\bibitem[{Lewis et~al.(2020)Lewis, Liu, Goyal, Ghazvininejad, Mohamed, Levy,
Stoyanov, and Zettlemoyer}]{lewis2020bart}
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed,
Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.
\newblock Bart: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 7871--7880.
\bibitem[{Lin et~al.(2020)Lin, Zhou, Shen, Zhou, Bhagavatula, Choi, and
Ren}]{lin2020commongen}
Bill~Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula,
Yejin Choi, and Xiang Ren. 2020.
\newblock Commongen: A constrained text generation challenge for generative
commonsense reasoning.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: Findings}, pages 1823--1840.
\bibitem[{Lin(2004)}]{lin2004rouge}
Chin-Yew Lin. 2004.
\newblock Rouge: A package for automatic evaluation of summaries.
\newblock In \emph{Text summarization branches out}, pages 74--81.
\bibitem[{Lin et~al.(2015)Lin, Liu, Sun, Liu, and Zhu}]{lin2015learning}
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015.
\newblock Learning entity and relation embeddings for knowledge graph
completion.
\newblock In \emph{Twenty-ninth AAAI conference on artificial intelligence}.
\bibitem[{Liu et~al.(2021)Liu, Wan, He, Peng, and Philip}]{liu2021kg}
Ye~Liu, Yao Wan, Lifang He, Hao Peng, and S~Yu Philip. 2021.
\newblock Kg-bart: Knowledge graph-augmented bart for generative commonsense
reasoning.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~35, pages 6418--6425.
\bibitem[{Noraset et~al.(2017)Noraset, Liang, Birnbaum, and
Downey}]{noraset2017definition}
Thanapon Noraset, Chen Liang, Larry Birnbaum, and Doug Downey. 2017.
\newblock Definition modeling: Learning to define word embeddings in natural
language.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial
Intelligence}, volume~31.
\bibitem[{Papineni et~al.(2002)Papineni, Roukos, Ward, and
Zhu}]{papineni2002bleu}
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.
\newblock Bleu: a method for automatic evaluation of machine translation.
\newblock In \emph{Proceedings of the 40th annual meeting of the Association
for Computational Linguistics}, pages 311--318.
\bibitem[{Raffel et~al.(2020)Raffel, Shazeer, Roberts, Lee, Narang, Matena,
Zhou, Li, and Liu}]{raffel2020exploring}
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter~J Liu. 2020.
\newblock Exploring the limits of transfer learning with a unified text-to-text
transformer.
\newblock \emph{Journal of Machine Learning Research}, 21:1--67.
\bibitem[{Voskarides et~al.(2017)Voskarides, Meij, and
Rijke}]{voskarides2017generating}
Nikos Voskarides, Edgar Meij, and Maarten~de Rijke. 2017.
\newblock Generating descriptions of entity relationships.
\newblock In \emph{European Conference on Information Retrieval}, pages
317--330. Springer.
\bibitem[{Voskarides et~al.(2015)Voskarides, Meij, Tsagkias, De~Rijke, and
Weerkamp}]{voskarides2015learning}
Nikos Voskarides, Edgar Meij, Manos Tsagkias, Maarten De~Rijke, and Wouter
Weerkamp. 2015.
\newblock Learning to explain entity relationships in knowledge graphs.
\newblock In \emph{Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers)}, pages 564--574.
\bibitem[{Vrande{\v{c}}i{\'c} and Kr{\"o}tzsch(2014)}]{vrandevcic2014wikidata}
Denny Vrande{\v{c}}i{\'c} and Markus Kr{\"o}tzsch. 2014.
\newblock Wikidata: a free collaborative knowledgebase.
\newblock \emph{Communications of the ACM}, 57(10):78--85.
\bibitem[{Wu and Weld(2010)}]{wu2010open}
Fei Wu and Daniel~S Weld. 2010.
\newblock Open information extraction using wikipedia.
\newblock In \emph{Proceedings of the 48th annual meeting of the association
for computational linguistics}, pages 118--127.
\bibitem[{Xiong et~al.(2017)Xiong, Hoang, and Wang}]{xiong2017deeppath}
Wenhan Xiong, Thien Hoang, and William~Yang Wang. 2017.
\newblock Deeppath: A reinforcement learning method for knowledge graph
reasoning.
\newblock In \emph{Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing}, pages 564--573.
\bibitem[{Yamada et~al.(2020)Yamada, Asai, Sakuma, Shindo, Takeda, Takefuji,
and Matsumoto}]{yamada2020wikipedia2vec}
Ikuya Yamada, Akari Asai, Jin Sakuma, Hiroyuki Shindo, Hideaki Takeda,
Yoshiyasu Takefuji, and Yuji Matsumoto. 2020.
\newblock Wikipedia2vec: An efficient toolkit for learning and visualizing the
embeddings of words and entities from wikipedia.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations}, pages 23--30.
\bibitem[{Yu et~al.(2020)Yu, Zhu, Li, Hu, Wang, Ji, and Jiang}]{yu2020survey}
Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and
Meng Jiang. 2020.
\newblock A survey of knowledge-enhanced text generation.
\newblock \emph{arXiv preprint arXiv:2010.04389}.
\bibitem[{Zhang et~al.(2019)Zhang, Kishore, Wu, Weinberger, and
Artzi}]{zhang2019bertscore}
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian~Q Weinberger, and Yoav Artzi.
2019.
\newblock Bertscore: Evaluating text generation with bert.
\newblock In \emph{International Conference on Learning Representations}.
\end{thebibliography}