Knowledge Graphs are not Created Equal: Exploring the Properties and Structure of Real KGs
Extracted source text · arXiv 2311.06414
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2022
% ready for submission
%\usepackage[final]{neurips_2023}
\usepackage[final]{neurips_glfrontiers_2023}
\usepackage{natbib}
\setcitestyle{round}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2022}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2022}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2022}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor,colortbl} % colors
\usepackage{rotating}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{tikz}
%\usepackage{caption}
%\usepackage{subcaption}
\usepackage[lofdepth,lotdepth]{subfig}
\usepackage{amsmath}
\bibliographystyle {abbrvnat}
\usepackage[inline]{enumitem}
% uncomment this to check for poorly filled hboxes and vboxes
%\usepackage{showframe}
\usepackage{color}
\definecolor{orange}{rgb}{1,0.5,0}
\newcommand{\js}[1]{{\color{red}{[#1]}}} % COLOR NOTE (don't remove)
\newcommand{\nt}[1]{{\color{blue}{[#1]}}} % COLOR NOTE (don't remove)
\newcommand{\ntt}[1]{{[#1]}} % COLOR NOTE (don't remove)
\newcommand{\estevam}[1]{{\color{cyan}{[#1]}}}
%math macros
\newcommand{\m}[1]{$#1$}
\newcommand{\inm}{\ensuremath{\!\in\!}}
% table macros -rotating headers
\newcommand{\rotateheader}[1]{\begin{turn}{80}#1\end{turn}}
%\title{Exploring the Properties of Real Knowledge Graph from Different Scientific Domains}
%\title{Comparative Analysis of \textit{Real} Knowledge Graphs across Application Domains} % in Different Domains}
\title{Knowledge Graphs are not Created Equal: Exploring the Properties and Structure of Real KGs}
%\title{Not All Knowledge Graphs are Created Equal: Exploring the Properties and Structure of Real Knowledge Graphs}
%\title{The Properties and Structure of Real Knowledge Graphs}
%\title{Knowledge Graphs are not Created Equal: Exploring the Properties of Real Knowledge Graphs}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Nedelina Teneva \thanks{Corresponding author} \\
Megagon Labs\\
Mountain View, CA \\
\texttt{nedteneva@gmail.com} \\
% examples of more authors
\And
Estevam Hruschka \\
Megagon Labs \\
Mountain View, CA \\
\texttt{estevam@megagon.ai} \\
}
\begin{document}
\maketitle
\setcounter{footnote}{0} %reset footnote count
\begin{abstract}
Despite the recent popularity of knowledge graph (KG) related tasks and benchmarks such as KG embeddings, link prediction, entity alignment and evaluation of the reasoning abilities of pretrained language models as KGs,
%Despite multiple efforts in model development and benchmarks,
the structure and properties of real KGs are not well studied.
In this paper, we perform a large scale comparative study of \m{29} real KG datasets from diverse domains
such as the natural sciences, medicine, and NLP to analyze
their properties and structural patterns. Based on our findings, we make several recommendations regarding KG-based model development and evaluation. We believe that the rich structural information contained in KGs can benefit the development of better KG models across fields and we hope this study will contribute to breaking the existing data silos between different areas of research (e.g., ML, NLP, AI for sciences).
\end{abstract}
\vspace{-6pt}
\section{Introduction}\vspace{-6pt}
Recent years have been marked by an increased use of multimodal and structured datasets in the form of knowledge graphs (KGs) to enhance applications in diverse scientific and technical disciplines such as natural language processing (NLP), natural sciences and medicine, manufacturing and process automation to name a few \citep{KGphysics,kg-opportunities}. The wide applicability of KGs is not surprising: they are scalable data objects that store factual (i.e., with high degree of certainty) information in the form of triples and allow for encoding both topological %(KG nodes/entities are connected by directed graph edges)
and semantic information. %(edges can have semantic meaning) .
%For example medical KGs may store facts such as `(enzyme) A catalyzes (reaction) R` and a language KG may contain the fact `light is`.
The growing interest in using KG across various domains has led to a surge in new KG dataset releases:
for example, \m{43\%} the $37$ datasets available in the PyKEEN v1.10 \footnote{Permalink: https://ezproxy.library.und.edu/login?url=https://github.com/pykeen/pykeen} KG embedding library \citep{ali2021pykeen} have been published since 2020.
Many other libraries -- e.g., OGB \citep{ogb,org-largescale}, LibKGE \citep{ruffinelli2020you, libkge} and PyTorch Geometric \citep{pyg} -- consolidate multiple KG models in a central repository or provide tools for task-specific benchmarking.
%This observation implies an increasing practical role KG datasets play in machine learning research, both for model development and model benchmarking. Additionally, multiple libraries provide tools for benchmarking of KG tasks by consolidating otherwise fragmented model implementations -- examples include OGB \citep{ogb,org-largescale}, LibKGE \citep{ruffinelli2020you, libkge} and PyG \citep{pyg}.
Lastly, several recent studies focus on scalable benchmarking of KG related tasks:
%, including KG link prediction and Entity Alignment (EA),
%, which shares similarities with record linkage in statistics \nt{ref}.
%Finally, various works are dedicated to scalable benchmarking of KG-related tasks such as KG link prediction and Entity Alignment (EA), which is closely related to the problem of record linkage in statistics \nt{Ref}.
for example \cite{bringingiglight} compare the performance of \m{21} KG link prediction models, \cite{sun2020benchmarking} evaluate \m{12} embedding-based entity alignment (EA) approaches on dedicated benchmark datasets, while \cite{alkhamissi2022review} propose a KG-based framework for assessing
the performance of pretrained language models (PLMs) with the goal of achieving parity between PLMs and KGs.
%quantify the reasoning abilities of pretrained language models (PLMs) by evaluating them as KGs.
%These observations suggests that KG datasets and are increasingly important in machine learning research, serving both for model development and model benchmarking.
%\estevam{the three main elements of this paragraph (KG dataset release, the several libraries, and the studies focusing on scalable benchmark) are not very well connected. Consider rephrasing.}
Even with the abundance of KG datasets, benchmarking tools, and extensive large-scale model comparisons across various KG tasks,
%and their use in numerous tasks such as KG entity alignment, KG link prediction, LLM-KG fusion and LM-as-KB evaluation benchmarks \cite{refs}, as well as the increased availability of libraries supporting benchmarks on these tasks,
to the best of our knowledge, there are no studies that address a much more fundamental question, namely: \textit{what properties and structure do real KGs have and how do they compare to each other in terms of these properties?}
We argue that a systematic approach of analyzing KG properties (the goal of this paper)
has the potential to inform algorithmic and dataset development across disciplines and empower the next generation of KG-based applications in NLP, biomedicine and other disciplines.
\textbf{Our Contribution}. In order to begin addressing the above question,
we analyze the structure of KGs in terms of their network statistics, topology and relation types.
%we approach KGs as topological objects rather than objects containing semantic information which is the case in some language based KGs-based approaches \nt{fix sent}.
Our large scale comparative study is based on \m{29} real KG datasets from diverse domains such as biology, medicine, and NLP. Towards our goal, we: (i) measure various KG properties (e.g., KG density, degree distribution); (2) analyze the KG structure in terms of the relational types and the KG topology; and (3) describe common/distinct structural patterns we observe in KG datasets derived from fundamentally different underlying domains.
We also highlight potential issues that may arise
%as one may encounter
when using off-the-shelf KG techniques (e.g. embeddings, link prediction models)
without accounting for the domain-specific KG structural properties.
Based on our findings, we make several recommendations for future model development and evaluation.%new evaluation strategies which take into account the fine-grained KG structure when benchmarking KG models.
\textbf{Scope}. Our primary goal is to analyze KG datasets and their properties along different dimensions, rather than benchmark task-specific models on said datasets. While evaluating how KG properties affect downstream algorithms such as link prediction or EA is a natural extension of this work, we intentionally leave this analysis for a follow up study in order to focus the analysis on the KG properties themselves.
%(see details in Sec. \ref{conclusion}).
%We hope that our dataset-focused findings will inform future KG model development and benchmarking and encourage wider ML model evaluating on datasets with richer underlying structure.
%\nt{clarify scope is not to benchmark specific algorithms/tasks. follow up work is needed: exhaustive comparison of methods against a large collection of datasets is beyond the scope, and relegated for future work given the enormity of the task.}
\input{datasetstable}
\vspace{-6pt}
\section{Application of Knowledge Graphs in Different Domains}\label{sec: kgapplication}
\vspace{-6pt}
\textbf{Notation}. For a given set of entities \m{E} and a set of relations \m{R}, a knowledge graph
\m{\mathcal{K}\!\!\subseteq \!\! K \!=\! E \!\times\! R \!\times\! E} is a directed multi-relational graph that contains triples of the form \m{(h,r,t)\inm \mathcal{K}} in which \m{h,t\inm E} represent the head and tail entities and \m{r\inm R} is the relation between them.
KG embedding models (e.g., TransE \citep{transe}, DistMult \citep{distmult}) learn latent vector representations of the entities \m{e\inm E} and relations \m{r\inm R} that best preserve the KG's structural properties.
%Besides for link prediction and entity , they have been used for tasks such as entity disambiguation, and clustering as well as for downstream tasks such as question answering, recommendation systems, and relation extraction.
\textbf{Natural Language Processing}.
In NLP PLMs have gained immense popularity in recent years due to their impressive ability to process and generate human-like text. PLMs, such as GPT-4 \citep{gpt4}, Llama \citep{llama} or Alpaca \citep{alpaca}, are able to generate answers to complex user queries on a variety of technical topics. % such as law and medicine \nt{RF that compars them}.
However, these models are known to suffer from a lack of grounding of their outputs (in factual, common sense and domain specific knowledge) and from having difficulties in properly dealing with the meaning of inter-related concepts \citep{carta2023grounding}.
Based on these facts, many approaches have been proposed for strengthening PLMs with KGs.
\cite {yang2023chatgpt} categorize these approaches in three main groups:
(i) before-training enhancement,
(ii) during-training enhancement, and
(iii) post-training enhancement. Other recent surveys on knowledge-enhanced PLMs
\citep{wei2021knowledge,zamini2022review,hu2023survey} also propose approaches which
incorporate factuality, common sense, physical and domain specific knowledge to mitigate the weaknesses of PLMs,
but none of those approaches consider the properties of the KGs at hand.
%\nt{We, in this paper, argue that not all KGs should be treated equally, thus, not all approaches that use KGs to enhance PLMs should be used with all types of KGs.}
%Feel free to revert this comment. However, their performance is still far from satisfactory in many real world scenarios -- a popular hypothesis is that PLMs exhibit this behavior because they do not explicitly model the relational commonsense knowledge about everyday concepts, which is crucial to many downstream tasks that need common sense to understand or generate \citep{zhou2020pre}. To address this issues, external knowledge from outside sources such as a knowledge base or a knowledge graph is incorporated into the text generation system. Several approaches have been proposed for supplementing pre-trained language models with the rich, factual knowledge contained in knowledge bases:
%\begin{enumerate*}[label={(\roman*)}]
% \item by injecting PLMs wtih entity representations/embeddings computed from external KG sources (e.g., \citep{todo}); or
% \item augmenting them with knowledge information by a performing a KG-related task such as entity or relation category classification (e.g., \citep{jaket}).
%\end{enumerate*}
%see Sec 2 in https://arxiv.org/pdf/2010.00796.pdf
%\citep {zamini2022review} provide of survey of methods for enhancing text generation with external knowledge bases and graphs.
%However, LLMs are known to "hallucinate," generating plausible-sounding yet factually incorrect information. One of the ways to address this issue is by enhancing their reasoning abilities, often by leveraging external sources of factual data sources such as Knowledge Graphs (KGs) and databases, helping LLMs to ground their responses in verified data and reduce hallucinations.
Spurred by the LAMA benchmark \citep{lama}, which analyzed the amount of relational knowledge already present in a wide range of PLMs (without fine-tuning), many works also use KGs for fine-grained evaluation of different aspects of PLMs such as their ability to recover factual knowledge or their consistency \citep{heinzerling-inui-2021-language}.
For a survey of recent methods for interpreting and evaluating "LMs as KBs", see \citep{alkhamissi2022review}.
% website: https://bkhmsi.github.io/lms-as-kbs/
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Figure %%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure} [t]
\centering
\begin{minipage}{0.32\textwidth}
\includegraphics[width=\textwidth]{figures/relation-vs-entities.png}
\end{minipage}
\begin{minipage}{0.32\textwidth}
\includegraphics[width=\textwidth]{figures/ent-vs-triples.png}
\end{minipage}
\begin{minipage}{0.32\textwidth}
\includegraphics[width=\textwidth]{figures/rel-vs-triples.png}
\end{minipage}
\caption{Relations between (left) number of entities vs. number of relations; (center) number of entities vs. number of triples; (right) number of relations vs. number of triples in different KGs. Biomedical, semantic web and societal datasets are colored in resp. olive, red, and blue. }
%All axes are \m{log}-scale.}
\label{fig: relationships}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Figures %%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[t]
\centering
\begin{minipage}{0.32\textwidth}
\includegraphics[width=\textwidth]{degree-plot-top20-recolored.png}
\end{minipage}
\begin{minipage}{0.32\textwidth}
\includegraphics[width=\textwidth]{triples_per_relation_recolored.png}
\end{minipage}
\begin{minipage}{0.32\textwidth}
\includegraphics[width=\textwidth]{metapaths-counts.png}
\end{minipage}
\caption{(left) Degree distribution; (center) number of triples per KG relation, \m{log} scale; (right) metapath (\texttt{mp}) length distribution on the \m{y}-axis, \m{log} scale.}
\label{fig: multiples}
\end{figure}
\textbf{Natural Sciences and Medicine}. KGs are used in various biomedical applications \citep{nicholson2020constructing,zitnikmultimodal} and have recently found use in precision medicine \citep{KGprecisionmedicine}. In biology and medicine KGs typically describe the relationships between biomedical entities such as diseases, drugs, phenotypes, or regulatory pathways. They are a convenient tool for aggregating knowledge fragmented across publications, repositories, ontologies and databses \citep{KGprecisionmedicine}. KG embeddings and link prediction find application in pharmaceutical applications (e.g. discovery and drug repurposing), clinical applications (e.g., disease diagnosis and treatment) and genomics (e.g., the study of phenotyping) \citep{morselli2021network,KGprecisionmedicine,wang2023scientific}.
Other natural science disciplines such as physics \citep{KGphysics} and geology \citep{KGgeology} make use of multimodal data such as scientific literature and other natural language datasets to construct domain-specific KGs.
\textbf{Other domains}. Many other domains such as cybersecuirty, finance, education, factory monitoring and process automation, geopolitical events and combating human trafficking benefit from KGs and apply KG tasks such as EA and link prediction -- \cite {li2020domain} and \cite{KGphysics} provide a review of domain-specific KGs and their downstream use in these areas.
\textbf {Knowledge Graph Datasets Analyzed}. %\label{sec: datasets}
Among all the datasets in the various domains described above, we used a set of \m{29} KGs listed in Tab. \ref{tbl: datasetstable} together with their summary statistics. We categorize the KGs into three distinct groups:
\vspace{-6pt}
\begin{enumerate} [label=(\alph*), left= 1pt]
\item \m{9} \textbf{biomedical KGs} which store facts related to biology and medicine such as relationships between genes, proteins or cellular pathways. Datasets in this group are typically derived from high quality public databases such as DrugBank \citep{drugbank} and PubChem \citep{kim2016pubchem} -- for construction details see \citep{pharmkg}.
\item \m{17} \textbf{semantic web KGs} which incorporate knowledge extracted using the tools of the semantic web or analogous mechanism such as RDF \citep{fensel2005spinning}. While the underlying KG ontologies in these datasets may be manually created by experts, the underlying \m{(h,r,t)} triples are typically extracted by parsing outsourced textual content such as Wikipedia or its structured counterpart Wikidata
\footnote {https://www.wikidata.org/wiki/Wikidata:Database\_download}.
Many datasets in this category are derived from each other or share common sources -- for example ConceptNet \citep{conceptnet} is based, in part, on DBpedia \citep{dbpedia}, while CSKG \citep{cskg} makes use of ConceptNet and Wikidata.%, and WordNet,
\item \m{3} \textbf{societal KGs} are a set of manually curated datasets that contain factual information about different domains such as geography and international relations -- e.g., the Nations dataset encodes relationships between countries such as "military alliance" and "independence [sic. from]". We note that these KGs are conceptually similar to the ones in the biomedical domain in the sense that they are based on relationships between objects, rathern than semantics.
% Countries \footnote{https://github.com/mledoze/countries}
% GeoJSON \footnote{https://geojson.org}- encoding a variety of geographic data
\end{enumerate}
\vspace{-6pt}
We use the datasets through the PyKEEN v\m{1.10} software package
(see details in the Appx.) % \ref{sec: appx-data}).
\vspace{-6pt}
\section{The Properties and Structure of Knowledge Graphs}
\vspace{-6pt}
%While these properties affect the performance of downstream KG tasks and algorithms, the underlying KG graph structure and its effect on these tasks and algorithms has not been studied in a cohesive manner. A notable exception is the work of \citep{toutanova} who identified the inverse relation problem which leads to information leakage between the train and test splits of the Freebase (FB15) and Wordnet (WN18) datasets and released new dataset versions called FB15k-237 and WN18RR, respectively.
The structural characteristics of KGs play an important role in the applicability and the performance of various tasks such as KG embeddings, link prediction and reasoning. For example, KG properties such as relation type (e.g., inverse, symmetric), cardinality and statistics affect the KG connectivity patterns which get encoded in the KG node and relation embeddings and used in downstream models.
The effect releation types have was demonstrated by \cite {toutanova} who first described the "inverse relation problem" in KG link prediction: essentially, an information leakage between the train and test splits due to the presence of \textit{inverse} relations in the training dataset splits. They identified the issue in the FB15k dataset and released its updated version FB15k-237, both commonly used in NLP benchmarks.
%This was demonstrated by \cite {toutanova} who identified the `inverse relation` problem in the FB15 dataset\nt{ref}, which leads to an information leakage between the train and test splits due to the presence of inverse relations in the training dataset splits.
Moreover, the inference abilities of KG embedding algorithms vary by relation type and carinality
For example, TransE cannot model symmetric and one-to-many relations well due to its scoring function \m{f (r,h,t) = -|| h\! +\!r\!-\!t||}. Similarly, the distance functions for DistMult and ComplEx cannot model composite relations. \cite{cao2022} provide a detailed review on this topic.
In the machine learning (ML) and NLP KG-related literature semantic web datasets are prevalent.
At the same time the performance of KG embeddings, EA models and link prediction (both embedding-based and Graph Neural Network (GNN)-based \citep{cucala2021explainable} approaches) is often demonstrated only on a
small set of semantic web datasets such as FB15k, WN18 and Wikidata5M.
On the other hand, in the biomedical domain it has been widely accepted that semantic web datasets %such as FB15k and WN18
do not reflect the domain specific properties of biomedical KGs due to a variety of factors \citep{pharmkg,openbiolink}.
% such as 1) as described by \cite {pharmkg,openbiolink},
One such factor are the interaction effects in KGs: biomedical KGs have been found to be sparse, incomplete and containing richly structured ontological hierarchies with large interaction networks instead of capturing knowledge networks (e.g., FB15k) or hierarchical taxonomies (e.g., WN18)\citep {pharmkg,openbiolink}. %In addition, they are usually sparse and incomplete \citep{pharmkg}.
%Another factor is the relationship between the process used to construct the knowledge graph (whether it is automated or manual) and the size of biomedical knowledge.
Another distinguishing characteristic of biomedical KGs is the nature of the entities stored in them -- for example automatically created datasets such as OpenBiolink \citep{openbiolink} contain large number of meta data entities and trivial biomedical entities which can interfere with KG embeddings and link prediction models as reported by \cite {pharmkg}.
%Due to the size of knowledge in the biomedical domain, their creation mechanisms affect the structural properties.
%For example, some of datasets are manually curated from high quality underlying data sources (e.g., PharmGKB), however they often contain only simple connections (triples) between entity pairs, leading to loss of essential biologically relevant mechanisms and associations \cite{pharmkg}.
%On the other hand, automatically created datasets such as OpenBiolink contain large number of meta data entities and trivial biomedical entities which can interfere with KGE embeddings and link prediction algorithms as reported by \cite {pharmkg}.
\textbf{KG vs. Graph Structure}. %KGs dataset properties and structure remain relatively underexplored as topological objects, partially because
KGs remain relatively unexplored as topological objects, partially because
KG datasets % -- aside from a few specialized KGs like WN18 and Yago \nt{check} --
were not readily available in libraries or benchmarks, such as PyKEEN or PyTorch Geometric until recent years. In contrast, as network analysis gained momentum in the 1990s, thanks to the growing availability of web, social and other real networks, the topology and numerical characteristics of directed and undirected graphs (the homogeneous counterparts of KGs) have been extensively studied.
%Graphs are also studied in graph theory, network analysis \citep{newman2003structure}, complex systems, numerical linear algebra \nt{refs}.
Multiple libraries and benchmarks have been established for the analysis of graph/network properties -- examples include the general purpose network and graph mining library SNAP \citep{snap} and the SuiteSparse matrix collection \citep{suitesparse-collection,suitesparse-web} which systematizes graphs together with their numerical characteristics.
% http://sparse.tamu.edu/about
% https://snap.stanford.edu
\textbf{Graph Structure in Existing Benchmarks}.
%\textbf{KG link prediction}.
Many KG benchmarks \citep{bringingiglight,kgxboard} focus on standardizing model training, hyper parameter tuning or task-specific model evaluation. However, only a few benchmarks provide support for evaluation of KG tasks with respect to the underlying KG structural properties.
%One example is the KGxBoard framework \citep {kgxboard} which provides a %explainable and interactive
%leaderboard for the evaluation of KG link prediction models with respect to relation cardinality and symmetry
One example is the KGxBoard framework for KG link prediction evaluation \citep {kgxboard}.
KG link prediction performance is often measured using metrics (e.g., precision) averaged over a held-out set, however, as noted by \cite{kgxboard}, single-score summary metrics cannot reveal exactly what the model has learned or failed to learn.
To remedy this, the KGxBoard framework implements a fine-grained performance reporting per relation type.
While in principle the KGxBoard framework offers support for multiple relation types,
the authors did not perform
a systematic evaluation of link prediction performance per relation type/property.
In another benchmark, \cite {bringingiglight} evaluate \m{21} KG link prediction models with respect to four relational patterns on \m{4} datasets (mix of semantic web and societal), % (WN18RR, FB15k-237, Kinships and YAGO3-10).
however, they do not analyze the link prediction performance in the context of the KG relational distributions and properties.
Lastly, \citep{sun2020benchmarking,eareeval} benchmark KG heterogenity and its effect on EA performance.
%In the related area of Graph Neural Networks (GNN) which has also seen an elevated interest and gained applicability in recent years, the question of how to guide a better dataset selection in GNN model benchmarking has been explored. Notably, \citep{liu2022taxonomy} perform data-driven analysis to devise a taxonomy of GNN datasets and study how these dataset characteristics are leveraged by GNNs.
%
%\textbf{PLLs}. On a related note, recent work on Large Language Models (LLMs) have ...
\input{comparisontable}
\vspace{-6pt}
\section{Methodology and Results}\label{sec: results}
\vspace{-6pt}
%\estevam{I'm missing a description of the methodology. It might be interesting to add a short description of the methodology before jumping into the results and analysis. We might even keep it all in this section, but it might also be interesting to add ``Methodology'' (or something similar to it) to the section title.}
\textbf{Methodology}.
We perform a series of data analysis steps to measure various KG properties and structural dimensions across all datasets.
Below we provide details on each KG dimension we considered, describe the experiments we conducted along each dimension and summarize empirical observations and findings. Unless otherwise noted, we used the datasets in their entirety (incl. train, test, and validation splits).
%The code from our experiments will be made available in the final version.
%Due to space limitations, some of the experimental details are included in the Appx. %\ref{sec: appx}.
%Due to space constraints and the large number of datasets considered, some of our analysis is in the Appx. \ref{sec: appx}.
%Due to space constraints and the large number of datasets analyzed, only a sample of our analysis is shown, all remaining figures are in Appx. \ref{sec: appx}. \nt{remove?}
\textbf{Entities, relations and triples}. In Fig. \ref{fig: relationships} we show the relations between the number of entities, relations and triples in each dataset.
When plotting the number of entities vs. number of relations in Fig. \ref{fig: relationships} (left), we notice that on average semantic web KGs have more diversity in terms of the entity/relation count, while most biomedical KGs cluster at similar entity/relation count. This implies that regardless of the fact that many of the semantic web datasets are derivatives of each other, as mentioned in Sec. \ref{sec: kgapplication}, they exhibit some diversity along these two dimensions.
Notably, Fig. \ref{fig: relationships} (center) -- plotting the number of entities vs. number of triples -- shows the biomedical KGs cluster together with the exception of the PharmKG8K and UMLS datasetas. In the same panel (top right corner) we also observe that the largest KG datasets in our study are predominantly semantic web ones.
Finally, Fig. \ref{fig: relationships} (right) shows that biomedical datasets exhibit less diversity in terms of the relation vs. triples count, in comparison the semantic web datasets.
\textbf{Average degree and degree distribution}
KG entities are connected to each other by directed edges (corresponding to the relations), hence an entity can have outgoing edges (or an out-degree) when it is the head entity in a triple and incoming edges (or an in-degree) when it is the tail entity in a triple.
We sum the in-degrees and out-degrees to obtain the entity degrees analogously to the way node degrees are computed in undirected graphs.
A smaller average degree indicates that the KG is sparser.
Fig. \ref{fig: multiples} (left) shows the degree distribution (over all entitie).
The average degree for each KG is also shown in Tab. \ref{tbl: datasetstable}. From this table we notice that the semantic datasets have some of the lowest average degrees across all datasets (e.g., ConceptNet with an average degree of \m{2}.
Among the semantic datasets, FB15k has the singularly highest average degree,
while its derivative FB15k-237 has a value that aligns with the rest of the semantic datasets.
On the other hand, the average degrees of biomedical datasets are split into two groups: \m{5} of the biomedical datasets have average degrees in the low \m{100}'s, while PharKG, Globi, OpenBioLink, and BioKG
have average degrees that are closer to the values of the semantic datasets.
The two societal datasets Kinship and Nations demonstrate a significantly higher average degree than the rest of the KGs
(see the Appx.) %\ref{sec: app-degrees}).
In Fig. \ref{fig: multiples} (left) several datasets show a distinct degree distribution in the lower degrees.
ConceptNet, CSKG have the highest number of low-degree nodes, while the degree distributions of Wikidata5M and OGBWikiKG2 are not as smooth, with oscillations at degrees \m{3} and \m{5}, respectively.
%Nations and Kinship datasets exhibit much higher average degrees (the smallest node degrees are \m{146} and \m{192}, respectively) in comparison to all the other KGs analyzed (see Appx. \ref{sec: app-degrees} for details).
\textbf{KG Density}.
The KG density is computed as the ratio \m{|E|/|R|^2} and like in homegeneous graphs higher density implies more sparsity
(see Tab. \ref{tbl: datasetstable}, last column). The degree trends we described in the previous paragraph can also be traced along this dimension.
Related to the KG density, we also quantify the KG connectivity by plotting the triples per KG relation in Fig. \ref{fig: multiples} (center).
The thick tails in the plot show that AristoV4, followed by FB15k, have a high number of relations with a low number triples per relation.
Several other semantic datasets follow the same pattern, while \textit{none} of the biomedical datasets do, with the exception of DRKG.
\textbf{Relation cardinality}: Relation cardinality describes the numerical relationship between the possible head and tail entities of a relation
(i.e., how many entities a relation can have as a tail or head). The possible types are:
(i) one-to-one (1-1), (ii) one-to-many (1-M), (iii) many-to-many (M-M), (iv) many-to-one (M-1) \citep{kgxboard}.
%duration/PlaceOfBirth
For illustration, the relation \texttt{surrounds} (from the UMLS dataset) is 1-1,
%the relation \texttt{Duration} (from the Nations dataset) is M-1 since many events can have the same duration,
%while the relation \texttt{PartOf} (from the WN18 dataset) is 1-M since multiple tail entities can complete this relation.
while the relation \texttt{GeneActivationGene} (from the OpenBioLink dataset) is M-M because various genes can activate multiple other genes.
Fig. \ref{fig: relcardinality} plots the relation cardinality distribution for each dataset considered.
We observe several different dataset profiles:
\textbf{(i) 1-1 dominance}: In \m{15} of the \m{28} datasets %in Fig. \ref{fig: relcardinality}
the leading relation type is 1-1 including many semantic datasets.
Overall, the number of 1-1 relations is more pronounced in the semantic datasets than in the biomedical datasets.
\textbf{(ii) M-M dominance}: In \m{11} of the \m{28} datasets, the leading relation type is M-M.
Biological datasets (e.g. BioKG, Hetionet, OpenBioLink, PharmKG8k) are dominated by M-M relations
with the exception of PharmKG and Globi which exhibit a distinct profile. %which look similar to some semantic datasets such as OpenEA.
All societal datasets also fall in this category.
\textbf{(iii)} \textbf{mixed cardinalities}: We observe that some of the most frequently used semantic datasets (FB15k, FB15k-237, Yago310)
have a significant number of all \m{4} cardinalities unlike the rest of the analyzed datasets.
\textbf{(iv) mixed profile with a 1-1 or M-M skew}:
All biomedical datasets tend to be skewed towards M-M relations, except PharmKG and Globi.
On the other hand, all semantic datasets tent to be skewed towards 1-1 relations, with the notable exception of Yago310.
CoDex* datasets are unique since they have predominantly M-1 relations.
%Biological datasets (e.g. BioKG, Hetionet, OpenBioLink, PharmKG8k) are dominated by M-M relations with the exception of PharmKG and Globi which look similar to some semantic datasets such as OpenEA.
\input{cardinalities}
\textbf{Relational patterns}. We considered four relation types previously described in the KG literature \citep {bringingiglight, toutanova}. A relation \m{r \inm R} is:
\begin{enumerate}[left=1pt, label={(\roman*)}]
\item \textbf{asymetric} if \m{(h,r,t) \inm T \!\! \implies\!\! (t,r,h) \! \notin\! T}
\item \textbf{symmetric} if \m{(h,r,t) \inm T \!\! \implies\!\!(t,r,h) \inm T}
\item \textbf{inverse} to \m{r_{inv}\inm R} if \m{(h,r,t) \inm T \!\! \implies\!\! (t,r_{inv},h) \inm T}.
If there exists a \m{r^\prime \inm R}, s.t. \m{r^\prime \!\not=\! r} and \m{r^\prime} is inverse of \m{r},
then \m{r} is an inverse relation.
\item \textbf{composite} of two relations \m{r_1,r_2 \inm R} if
\m{(x,r_1,y)\inm T\!\wedge\! (y,r_2,z)\inm T \!\!\implies\!\! (z, r, z)\inm T}.
\end{enumerate}
Fig. \ref{fig: reltypes} plots the relation pattern distribution for each dataset considered. Notably, some semantic datasets such as
DB100K, OpenEA and DBpedia50 have a small amount of inverse relations which may affect benchmarking on these datasets in light of the "inverse relation problem" discussed above. Across all datasets we observe dominance of anti-symmetric relations, with the exception of the societal datasets and
some of the most frequently used benchmarking datasets in NLP, such as FB15 and WN18RR, which show presence of all \m{4} relation types.
Some semantic and societal datasets have composite relations, while none of the biomedical do.
\textbf{Metapaths}:
KG metapaths are widely used in the biomedical literature for assessing the connectivity of KGs and deriving insights about the
clinical or biological relevance of interactions such as gene-gene or drug effects \citep{su2020network,fu2016predicting, hetionet,zhang2020predicting}.
A metapath is defined as a sequence of relations separated by edge types (metanodes). For example,
a metapath of length \m{\ell} is of the form
%\m{(e_1, r_1, e_2), (e2,r_2, e_3), \dots, (e_{\ell-1}, r_{\ell-1}, e_{\ell})},
$ e_1 \xrightarrow[]{r_1} e_2 \xrightarrow[]{r_2} \dots \xrightarrow[]{r_{\ell-1}} e_{\ell} $
where each of \m{e_1,e_2,\text{ and} e_{\ell}} belongs to a specific metanode. %and \m{\{r_1,r_2,\dots r_{\ell-1} \}} belong to the same metaedge.
For example, in the Hetionet dataset\citep{hetionet} the metanodes \texttt{Compound}, \texttt{Gene} and \texttt{Disease} form the metapath
$ \texttt{Compound} \xrightarrow[]{\texttt{binds}} \texttt{Gene} \xrightarrow[]{\texttt{associates}} \texttt{Disease}$
of length \m{2}.
%Additional details on metapath construction and the role they play in the interpretability of KG-based models in biomedicine can be found in \citep{su2020network,fu2016predicting, hetionet,zhang2020predicting}.
The number of metapaths of a given length provides a way for quantifying the level of relational composition
without having explicit composite relations encoded in the KG.
Interestingly, \cite{cohen2023crawling} test the reasoning abilities of PLMs by a prompting strategy that forces them to survey
entity neighborhoods;
%as one of the assess reasoning abilities of PLMs in the context of LM-as-KBs,
although the authors do not put their work in the context of KG metapaths.
%Compound–binds–Gene–associates–Disease
%DRUG_BINDING_GENE
%Note that the term metanode overloaded -- in practice each disease instance is represented in the KG by a disease entity connected by to a disease entity.
In practice, metapaths of length of greater than \m{4} are considered too long to make a significant contribution in link prediction task \citep{hetionet, fu2016predicting}.
Fig. \ref{fig: multiples} (right) compares the metapath length distribution over paths of length \m{2,3} and \m{4} for all KGs (see Appx. for additional details).
From the figure we see that the biomedical KGs (with the exception of Globi) contain a significantly higher number of metapaths than the other KG types.
However, the semantic dataset FB15k and the societal KGs Kinship and Nations exhibit a profile similar to the one of the biomedical datasets.
\input{reltypesfig}
%\input{cardinalities}
\textbf{Findings}.
Tab. \ref{tbl: comparisontable} summarizes our observations above in a color coded visual to complement the analysis.
Below we highlight the key takeaways from our empirical analysis and based on them, make several recommendations:
%their implications for downstream KG model development, evaluation and benchmkarking in different domains. \nt{recommendations?}
\vspace{-6pt}
\begin{itemize}[left= 1pt]
\item Given the dataset diversity observed in this study, we conclude that not all KGs are created equal.
This has implications for model evaluation in ML, NLP and other domains in which KGs are used.
Thus, we recommend that researchers consider a broader set of datasets beyond the ones derived from the semantic web
for their KG model development and evaluation.
Fine-grained model evaluation -- for example, as a function of relation type, cardinality, KG density or degree distribution -- has the potential to further drive the development of new KG-based models or inform model selection given specific KG properties.
%On the other hand, practitioners need to be aware that even if an embedding algorithm performs well on semantic datasets it may still be a weak model for modeling e.g., biomedical graphs due to the structural and property-related differences.
%\item The observed differences in relation/entity statistics (KG density, degree distribution),
%relational patterns and cardinality across KGs lead us to conclude that not all KGs are created equal.
%This has implications for model evaluation in both ML, NLP and other domains in which KGs are used.
%Evaluation of newly introduced models as well as existing benchmarks
%need to consider a variety of datasets with diverse properties along the dimensions described above.
%Currently, most KG link prediction, entity alignment and LM-as-KBs benchmarks in ML and NLP typically consider a handful of semantic datasets (e.g., FB15k(237), WN18). On the other hand, biomedical KGs have properties which distinguish them from their semantic web counter parts, which implies that even if an embedding algorithm performs well on semantic datasets it may still be a poor model for modeling biomedical graphs.
%the benchmarking of othe broader use of KGs across domains -- given the proliferation of methods across domains
%it is worth investigating whether structural patterns or whether KGs in some domains have domain-specific properties that may influence the performance and evaluation of KG algorithms.
\item Inverse relations are present in some datasets, including some released after the "inverse relation" leakage problem was reported by \cite{toutanova}. Given the implications of this problem in downstream applications,
we recommend KG libraries and benchmarks consider adding tools for handling/removing inverse relations in order to bring
visibility to the leakage problems. %issue to practicioners Since it is we recommend that benchmarks and explicitly remove
%This observation has implications for entity alignment ... \nt{Refs below}.
\item As mentioned in Sec. \ref{sec: kgapplication}, many recent knowledge-enhanced models have been proposed.
Based on our findings, we caution that the success of KG-enhanced PLMs may vary depending on the underlying KG structure.
While additional analysis is needed in this direction, given the varied structure of real KGs , development of domain-specific PLMs may benefit differently from different types of KGs.
%While additional analysis is needed in this direction,
%%\item Composite relations - different branches and reasoning abilities
%intuitively, since KGs differ in terms of their properties and structure,
%not all approaches that use KGs to enhance PLMs may work well with all types of KGs.
\item The overall negligible amount of composite relations in many datasets (including biological, semantic, and societal) is one
interesting observation that merits further analysis. Composite relations lead to triangles in KGs and intuitively imply
diminished reasoning pathways in KGs, which may have an effect on LMs-as-KGs benchmarks and evaluations.
%may be indicative of the diminished capacity of these datasets in tasks requiring long reasoning chains.
Additionally, the distinct presence of composite relations in FB15k237 (one of the most frequently used datasets in NLP research) may lead to flaws in KG-based NLP model evaluation, unless performance on a variety of other datasets is also considered.
% + Heterogeneity in EA: sampling is important \citet{sun2020benchmarking}
%\item We believe that the rich structural information contained in KGs can benefit the development of better KG models across fields (e.g., ML, NLP, AI for sciences). Hence, we hope that studies like this will contribute to breaking the existing data silos between different areas of research.
\item \cite{openbiolink} hypothesize that the size of biomedical KGs tends to be large, which calls into double whether model results from smaller datasets are informative. We argue that beyond size, practitioners should consider the role KG properties and structural patterns
during the design, hypotheses testing and model development.
\item We hypothesize that analyzing the properties and structure of existing KGs can also benefit the future design of more robust KG datasets which incorporate diversity along different dimensions, such as the ones explored in this paper.
\end{itemize}
% Help enable explainability in biomedical domain: Such systems in medicine are often related to a decision-support task, where data may be incomplete, uncertain, ambiguous, or missing. Moreover, such data have a high complexity and heterogeneity, being expressed as often interrelated and intertwined data in various formats such as structured, semi-structured, or unstructured alphanumeric data, movies, images, sounds, waveform signals, and so on. From Zitnik: https://www.sciencedirect.com/science/article/pii/S0933365722001750
% Kinship/Nations + semantic are oftern used. Two different groups with different properties...
%KEEP: This has implications for the broader use of KGs across domains -- given the proliferation of methods across domains it is worth investigating whether structural patterns or whether KGs in some domains have domain-specific properties that may influence the performance and evaluation of KG algorithms.
% In what ways to KGs across domains differ or resemble each other? or are there structural patterns that influence the structure. It remains to be shown
% whether
\vspace{-6pt}
\section{Conclusions and Future Work}\label{conclusion}
In summary, we analyzed \m{29} real KG datasets from several different domains and evaluated their properties and structural
characteristics along various dimensions. Based on our findings we make several recommendations for KG-based model development and evaluation.
Our study has implications for the broader use of KGs across domains: given the proliferation of models (in KG link prediction, EA, LM-as-KG evaluation) across domains (NLP, natural sciences, medicine and other disciples), it is worth investigating whether (and how) structural patterns, as well as their inter-domain variability across KGs, may correlate or influence KG model performance. Given the scope and scale of such an investigation, we leave it for a follow up study and encourage others -- within the NLP, ML and the ML for Sciences communities -- to further explore this topic.
% \textbf{++++++++ NOTES}
% Transductive/Inductive datasets ...%
% Most of the KG link prediction and entity alignment methods are only benchmarked on a handful of dasaset such as FB and WN in the machine learning and NLP community. On the other hand, real KGs in fields such as biomedicine and \nt{TK} have properties which distinguishes them from their semantic web counter parts, which means that if an embedding algorithm performs well on FB/WN it may still be a poor model for modeling biomedical graphs.%
% Additionally it is plausible that semantic KGs are distinct from KGs derived from physical processes (genes, proteins, etc.). Even in NLP, there is a distinction between common sense KG and an ontological KG. Another distinction if instance-level vs Concept KG graph. %
% \nt{check if graph analysis can help in KG construction}.
%We also adresss the question of KGs similar across domains? or are there structural patterns that influence the structure
%can inform future directions and resolve algorithmic bottlenecks
% We approach KGs as topological objecst whose structure may tell us ...
% whose properties inform downstream applications and can provide evidence can inform algorithmic development
%Methodology have crossed domains (e.g. PLMs) but does the underlying KG structure look similar in different domains. For example, do graphs in NLP have the same structure as those in medicine? In real networks we know that there are properties that govern graphs independent of the way they are constructed, however KG construction is noisy, could be highly manual or based on automated tools. Unlike in social networks which are based on physical connectivity (friends on Facebook, URLs linking to each other), modern day KGs are semi-automatically constructed, with human in the loop \nt{refs}.
% Biforcation between fields wherein each We argue that a systematic approach of analyzing KG graph properties (goal of this paper)
% had the potential to inform KG algorithmic development (across all domains) and empower the next generation of KG applications in NLP, biomedical and other domains.
%We note that more generally graph structure and its effect on downstream graph algorithms (e.g. low rank approximations of graph and kernel matrices, sparse graph dictionary learning and sparse PCA, etc.) \nt{Refs} have been thoroughly studied in the graph theory, network analysis, complex systems and matrix factorizations research communities \nt{Refs}. While knowledge graphs draw on several of these research areas (e.g. graph-based algorithms; matrix and tensor factorizations; and database optimization -- see Section \ref{literature}), their interplay has not been explored on real KGs.
% \textbf{What may explain the different structure?}
% Even in NLP, there is a distinction between common sense KG and an ontological KG. Another distinction if instance-level vs Concept KG graph.
%
% - generic knowledge graphs and domain-specific knowledge graphs: % https://dl.acm.org/doi/fullHtml/10.1145/3522586
% - Common Sense vs Ontological KG
% - Instance level vs Concept
% http://www.vldb.org/pvldb/vol13/p2326-sun.pdf and
% TODO cite https://dlab.epfl.ch/people/west/pub/Leone-Huber-Arora-GarciaDuran-West_VLDB-22.pdf --- GraphVite [87], PyTorch-BigGraph [37], and DGL-KE [83]
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\bibliography{kg-bib}
\newpage
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\appendix
\section{Appendix}\label{sec: appx}
\subsection{Datasets used} \label{sec: appx-data}
We used all the datasets in the PyKEEN library as described in the paper with the
exception of several datasets (e.g., WK3l15k, WK3l120k, CN3l, and CKG) whose underlying files are no longer available for download at the URLs the library points to.
\subsection{Degree plots}\label{sec: app-degrees}
The Nations and Kinship datasets were not included in Fig. \ref{fig: multiples} due to the high number of high degree nodes in them which leads to plot scaling issues of the remaining \m{26} datasets. The Nation's \m{14} entities have degrees in the range \m{146-514}; the Kinship's entities have degrees in the \m{192-206} range.
For similar reasons we exclude the highest-degree entity (\texttt{men}) of the ConceptNet dataset in the plot in Fig. \ref{fig: multiples}.
\subsection{Relation types, cardinalities, and metapaths} \label{sec: appx-card}
Relationship type determination, i.e. whether a relation is (anti)-symmetric, inverse, composite, is based on association rule mining. The relation classifications are based on checking whether the corresponding rules hold with sufficient support and confidence -- we calculated the support using a confidence of \m{95\%}. We used the reference implementation in available in PyKEEN \cite{ali2021pykeen}. Note that a relation can be of several different types.
Relation cardinality is computed similarly to the relation type.
Metapath lengths are approximated by sampling (uniformly at random) an entity \m{e} from each KG and counting all the paths of length \m{2, 3} and \m{4} originating from \m{e}. Each KG was sampled \m{3} times, so the metapath numbers reported in Fig. \ref{fig: multiples} (right) are averaged over
\m{3} independent entity samples, for each KG.
\subsection{Additional plots for Fig. \ref{fig: reltypes}, and Fig. \ref{fig: relcardinality}} \label{sec: appx-moreplots}
\begin{figure}[h]
\centering
\begin{minipage}{0.32\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/CoDExMedium_rel_cardinality.png}
\end{minipage}
\begin{minipage}{0.32\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/CoDExMedium-relation-types.png}
\end{minipage}
\caption{CoDExMedium dataset: supplemental panels for Fig. \ref{fig: reltypes}, and Fig. \ref{fig: relcardinality}.}
\label{fig: codex}
\end{figure}
% empty, skipping: Kinships
% degree count
% 3 192 1
% 5 199 1
% 4 200 1
% 2 204 6
% 1 205 13
% 0 206 82
% empty, skipping: Nations
% degree count
% 0 146 2
% 6 215 1
% 3 232 1
% 7 243 1
% 2 249 1
% 1 260 1
% 4 284 1
% 9 287 1
% 5 302 1
% 8 313 1
% 12 331 1
% 10 462 1
% 11 514 1
%
%\textbf{Tensor Factorizations}.
%The applicability of tensor-based methods comes from the observation that any knowledge graph can be abstracted to a 3D tensor in which each fiber represents the while the third fiber represents the . Rescal
%Tucker factorization for KG embeddings.
%tensor-based methods have recently been proposed where binary relations are represented as square adjacency matrices that are stacked as frontal slices in a three-way adjacency tensor.
%\citep{rescal,krompass2014large}
%\textbf{Graph-based Algorithms}.
%\textbf{Databases and Parameter Server}: \href{https://dl.acm.org/doi/pdf/10.1145/3514221.3517860}{https://dl.acm.org/doi/pdf/10.1145/3514221.3517860}
%\textbf{Extraction graphs?}
%\section {Embedding Algorithms}
%\textbf{Embedding Algorithm Classes}. There are several different classes of KG embedding algorithms -- namely, algebraic, geometric, and analytical -- best summarized by \citet {cao2022}.
\end{document}
%TODO include references to datasets
\begin{table}[t]
\caption{Datasets that were analyzed in this study. \#E, \#R, and \#T denote the number of entities, relations and triples, respectively. \m{deg} denotes the average degree of all the KG entities. \m{d} denotes the KG density, shown in \m{\log} scale: a lower column value implies a denser KG.}
\label{tbl: datasetstable}
\centering
\begin{tabular}{llrrrlrr}
\toprule
%\multicolumn{2}{c}{Part} \\
\cmidrule(r){1-2}
\# & \textbf{dataset} & \textbf{\# E} & \textbf{\# R} & \textbf{\# T} & \textbf{category} & \textbf{\m{deg}} & \textbf{\m{-\log(d)}} \\
\midrule
1 & AristoV4 & \m{ 42,016 } & \m{ 1,593 } & \m{ 279,425 } & biomed & \m{ 7 } & \m{ 3.80 } \\
2 & BioKG & \m{ 105,524 } & \m{ 17 } & \m{ 2,067,997 } & biomed & \m{ 20 } & \m{ 3.73 } \\
3 & CoDExLarge & \m{ 77,951 } & \m{ 69 } & \m{ 612,437 } & semantic & \m{ 8 } & \m{ 4.00 } \\
4 & CoDExMedium & \m{ 17,050 } & \m{ 51 } & \m{ 206,205 } & semantic & \m{ 12 } & \m{ 3.15 } \\
5 & CoDExSmall & \m{ 2,034 } & \m{ 42 } & \m{ 36,543 } & semantic & \m{ 18 } & \m{ 2.05 } \\
6 & ConceptNet & \m{ 28,370,083 } & \m{ 50 } & \m{ 34,074,917 } & semantic & \m{ 1 } & \m{ 7.37 } \\
7 & Countries & \m{ 271 } & \m{ 2 } & \m{ 1,158 } & society & \m{ 4 } & \m{ 1.80 } \\
8 & CSKG & \m{ 2,087,833 } & \m{ 58 } & \m{ 4,598,728 } & semantic & \m{ 3 } & \m{ 5.98 } \\
9 & DB100K & \m{ 99,604 } & \m{ 470 } & \m{ 697,479 } & semantic & \m{ 7 } & \m{ 4.15 } \\
10 & DBpedia50 & \m{ 24,624 } & \m{ 351 } & \m{ 34,421 } & semantic & \m{ 1 } & \m{ 4.25 } \\
11 & DRKG & \m{ 97,238 } & \m{ 107 } & \m{ 5,874,257 } & biomed & \m{ 60 } & \m{ 3.21 } \\
12 & FB15k & \m{ 14,951 } & \m{ 1,345 } & \m{ 592,213 } & semantic & \m{ 40 } & \m{ 2.58 } \\
13 & FB15k-237 & \m{ 14,505 } & \m{ 237 } & \m{ 310,079 } & semantic & \m{ 21 } & \m{ 2.83 } \\
14 & Globi & \m{ 404,207 } & \m{ 39 } & \m{ 1,966,385 } & biomed & \m{ 5 } & \m{ 4.92 } \\
15 & Hetionet & \m{ 45,158 } & \m{ 24 } & \m{ 2,250,197 } & biomed & \m{ 50 } & \m{ 2.96 } \\
16 & Kinships & \m{ 104 } & \m{ 25 } & \m{ 10,686 } & society & \m{ 103 } & \m{ 0.01 } \\
17 & Nations & \m{ 14 } & \m{ 55 } & \m{ 1,992 } & society & \m{ 143 } & \m{ -1.01 } \\
18 & OGBWikiKG2 & \m{ 2,500,604 } & \m{ 535 } & \m{ 17,137,181 } & semantic & \m{ 7 } & \m{ 5.56 } \\
19 & OpenBioLink & \m{ 180,992 } & \m{ 28 } & \m{ 4,563,407 } & biomed & \m{ 25 } & \m{ 3.86 } \\
20 & OpenEA & \m{ 15,000 } & \m{ 248 } & \m{ 38,265 } & semantic & \m{ 3 } & \m{ 3.77 } \\
21 & PharmKG & \m{ 188,296 } & \m{ 39 } & \m{ 1,093,236 } & biomed & \m{ 6 } & \m{ 4.51 } \\
22 & PharmKG8k & \m{ 7,247 } & \m{ 28 } & \m{ 485,787 } & biomed & \m{ 67 } & \m{ 2.03 } \\
23 & PrimeKG & \m{ 129,375 } & \m{ 30 } & \m{ 8,100,498 } & biomed & \m{ 63 } & \m{ 3.32 } \\
24 & UMLS & \m{ 135 } & \m{ 46 } & \m{ 6,529 } & biomed & \m{ 48 } & \m{ 0.45 } \\
25 & WD50K & \m{ 40,107 } & \m{ 473 } & \m{ 232,344 } & semantic & \m{ 6 } & \m{ 3.84 } \\
26 & Wikidata5M & \m{ 4,594,149 } & \m{ 822 } & \m{ 20,624,239 } & semantic & \m{ 4 } & \m{ 6.01 } \\
27 & WN18 & \m{ 40,943 } & \m{ 18 } & \m{ 151,442 } & semantic & \m{ 4 } & \m{ 4.04 } \\
28 & WN18RR & \m{ 40,559 } & \m{ 11 } & \m{ 92,583 } & semantic & \m{ 2 } & \m{ 4.25 } \\
29 & YAGO3-10 & \m{ 123,143 } & \m{ 37 } & \m{ 1,089,000 } & semantic & \m{ 9 } & \m{ 4.14 } \\
% removed this one (low quality version of OpenBioLink. Nations Literal not used
\bottomrule
\end{tabular}
\end{table}
\begin{table}[t]
\caption{Summary of findings from Sec. \ref{sec: results}: size, density (triples per relation), relation cardinaility, relation pattern and number of metapaths. Colors blue, red, gray denote low, medium, mixed levels of each characteristic for the specified KG domain (each row).
}
\label{tbl: comparisontable}
\centering
\begin{tabular}{lrrrrrrrrrrrrrr}
\toprule
%\multicolumn{2}{c}{Part} \\
%\cmidrule(r){1-2}
KG domain & \rotateheader{size} & \rotateheader{dens.} & \rotateheader{M-M} & \rotateheader{M-1} & \rotateheader{1-M} & \rotateheader{1-1} & \rotateheader{antiS} & \rotateheader{S} & \rotateheader{Inv} & \rotateheader{Comp} & \rotateheader{\#paths} \\
\midrule
semantic & \cellcolor{red} & \cellcolor{blue} & \cellcolor{blue} & \cellcolor{gray} &\cellcolor{gray} & \cellcolor{red} & \cellcolor{red} & \cellcolor{gray} & \cellcolor{gray} & \cellcolor{gray} & \cellcolor{blue} \\
biomedical & \cellcolor{gray} & \cellcolor{red} & \cellcolor{red} & \cellcolor{red} & \cellcolor{blue}& \cellcolor{blue} & \cellcolor{red} &\cellcolor{gray} & \cellcolor{blue} & \cellcolor{blue} & \cellcolor{red} \\
societal & \cellcolor{blue} & \cellcolor{blue} & \cellcolor{red} & \cellcolor{blue} & \cellcolor{blue} & \cellcolor{blue} & \cellcolor{red} & \cellcolor{red} & \cellcolor{blue} & \cellcolor{red} & \cellcolor{red} \\
\bottomrule
\end{tabular}
\end{table}
% need to regenrate images with tight layout
\begin{figure}[t]
\centering
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/AristoV4_rel_cardinality.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/BioKG_rel_cardinality.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/CoDExLarge_rel_cardinality.png}
\end{minipage}
%\begin{minipage}{0.24\textwidth}
% \includegraphics[width=\textwidth]{figures/relcardinality/CoDExMedium_rel_cardinality.png }
%\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/CoDExSmall_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/ConceptNet_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/Countries_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/CSKG_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/DB100K_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/DBpedia50_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/DRKG_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/FB15k_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/FB15k237_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/Globi_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/Hetionet_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/Kinships_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/Nations_rel_cardinality.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/OGBWikiKG2_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/OpenBioLink_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/OpenEA_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/PharmKG_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/PharmKG8k_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/PrimeKG_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/UMLS_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/WD50KT_rel_cardinality.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/Wikidata5M_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/WN18_rel_cardinality.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/WN18RR_rel_cardinality.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/relcardinality/YAGO310_rel_cardinality.png}
\end{minipage}
\caption{Distribution of relation cardinalities in different KGs. Each bar is marked with the \# number of relations of the specified type.
Due to space limits, the CoDExMedium dataset is shown in the Appx.}
\label{fig: relcardinality}
\end{figure}
%\vspace{-24pt}
% need to regenrate images with tight layout
\begin{figure}[t]
\centering
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/AristoV4-relation-types.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/BioKG-relation-types.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/CoDExLarge-relation-types.png}
\end{minipage}
%\begin{minipage}{0.24\textwidth}
% \includegraphics[width=\textwidth]{figures/reltypes/CoDExMedium-relation-types.png }
%\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/CoDExSmall-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/ConceptNet-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/Countries-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/CSKG-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/DB100K-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/DBpedia50-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/DRKG-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/FB15k-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/FB15k237-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/Globi-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/Hetionet-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/Kinships-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/Nations-relation-types.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/OGBWikiKG2-relation-types.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/OpenBioLink-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/OpenEA-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/PharmKG-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/PharmKG8k-relation-types.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/PrimeKG-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/UMLS-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/WD50KT-relation-types.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/Wikidata5M-relation-types.png }
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/WN18-relation-types.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/WN18RR-relation-types.png}
\end{minipage}
\begin{minipage}{0.24\textwidth}
\includegraphics[width=\textwidth]{figures/reltypes/YAGO310-relation-types.png}
\end{minipage}
\caption{Proportion of (anti-)symmetric, inverse and composite relations in different KGs. Each bar is marked with the \# of relations of the specified type with the \m{y}-axis showing the corresponding triples (as a \% of all triples). Due to space limits, the CoDExMedium dataset is shown in the Appx.}
\label{fig: reltypes}
\end{figure}
\begin{thebibliography}{51}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Ali et~al.(2021{\natexlab{a}})Ali, Berrendorf, Hoyt, Vermue, Galkin, Sharifzadeh, Fischer, Tresp, and Lehmann]{bringingiglight}
M.~Ali, M.~Berrendorf, C.~T. Hoyt, L.~Vermue, M.~Galkin, S.~Sharifzadeh, A.~Fischer, V.~Tresp, and J.~Lehmann.
\newblock Bringing light into the dark: A large-scale evaluation of knowledge graph embedding models under a unified framework.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine Intelligence}, 44\penalty0 (12):\penalty0 8825--8845, 2021{\natexlab{a}}.
\bibitem[Ali et~al.(2021{\natexlab{b}})Ali, Berrendorf, Hoyt, Vermue, Sharifzadeh, Tresp, and Lehmann]{ali2021pykeen}
M.~Ali, M.~Berrendorf, C.~T. Hoyt, L.~Vermue, S.~Sharifzadeh, V.~Tresp, and J.~Lehmann.
\newblock {PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings}.
\newblock \emph{Journal of Machine Learning Research}, 22\penalty0 (82):\penalty0 1--6, 2021{\natexlab{b}}.
\newblock URL \url{http://jmlr.org/papers/v22/20-825.html}.
\bibitem[AlKhamissi et~al.(2022)AlKhamissi, Li, Celikyilmaz, Diab, and Ghazvininejad]{alkhamissi2022review}
B.~AlKhamissi, M.~Li, A.~Celikyilmaz, M.~Diab, and M.~Ghazvininejad.
\newblock A review on language models as knowledge bases.
\newblock \emph{arXiv preprint arXiv:2204.06031}, 2022.
\bibitem[Auer et~al.(2007)Auer, Bizer, Kobilarov, Lehmann, Cyganiak, and Ives]{dbpedia}
S.~Auer, C.~Bizer, G.~Kobilarov, J.~Lehmann, R.~Cyganiak, and Z.~Ives.
\newblock Dbpedia: A nucleus for a web of open data.
\newblock In \emph{international semantic web conference}, pages 722--735. Springer, 2007.
\bibitem[Breit et~al.(2020)Breit, Ott, Agibetov, and Samwald]{openbiolink}
A.~Breit, S.~Ott, A.~Agibetov, and M.~Samwald.
\newblock Openbiolink: a benchmarking framework for large-scale biomedical link prediction.
\newblock \emph{Bioinformatics}, 36\penalty0 (13):\penalty0 4097--4098, 2020.
\bibitem[Broscheit et~al.(2020)Broscheit, Ruffinelli, Kochsiek, Betz, and Gemulla]{libkge}
S.~Broscheit, D.~Ruffinelli, A.~Kochsiek, P.~Betz, and R.~Gemulla.
\newblock {L}ib{KGE} - {A} knowledge graph embedding library for reproducible research.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations}, pages 165--174, 2020.
\newblock URL \url{https://www.aclweb.org/anthology/2020.emnlp-demos.22}.
\bibitem[Cao et~al.(2022)Cao, Fang, Meng, and Liang]{cao2022}
J.~Cao, J.~Fang, Z.~Meng, and S.~Liang.
\newblock Knowledge graph embedding: A survey from the perspective of representation spaces.
\newblock \emph{arXiv preprint arXiv:2211.03536}, 2022.
\bibitem[Carta et~al.(2023)Carta, Romac, Wolf, Lamprier, Sigaud, and Oudeyer]{carta2023grounding}
T.~Carta, C.~Romac, T.~Wolf, S.~Lamprier, O.~Sigaud, and P.-Y. Oudeyer.
\newblock Grounding large language models in interactive environments with online reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2302.02662}, 2023.
\bibitem[Chandak et~al.(2023)Chandak, Huang, and Zitnik]{KGprecisionmedicine}
P.~Chandak, K.~Huang, and M.~Zitnik.
\newblock Building a knowledge graph to enable precision medicine.
\newblock \emph{Scientific Data}, 10\penalty0 (1):\penalty0 67, 2023.
\bibitem[Cohen et~al.(2023)Cohen, Geva, Berant, and Globerson]{cohen2023crawling}
R.~Cohen, M.~Geva, J.~Berant, and A.~Globerson.
\newblock Crawling the internal knowledge-base of language models.
\newblock \emph{arXiv preprint arXiv:2301.12810}, 2023.
\bibitem[Cucala et~al.(2021)Cucala, Grau, Kostylev, and Motik]{cucala2021explainable}
D.~J.~T. Cucala, B.~C. Grau, E.~V. Kostylev, and B.~Motik.
\newblock Explainable gnn-based models over knowledge graphs.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\bibitem[Davis and Hu(2011)]{suitesparse-collection}
T.~A. Davis and Y.~Hu.
\newblock The university of florida sparse matrix collection.
\newblock \emph{ACM Transactions on Mathematical Software (TOMS)}, 38\penalty0 (1):\penalty0 1--25, 2011.
\bibitem[Ektefaie et~al.(2023)Ektefaie, Dasoulas, Noori, Farhat, and Zitnik]{zitnikmultimodal}
Y.~Ektefaie, G.~Dasoulas, A.~Noori, M.~Farhat, and M.~Zitnik.
\newblock Multimodal learning with graphs.
\newblock \emph{Nature Machine Intelligence}, 5\penalty0 (4):\penalty0 340--350, 2023.
\bibitem[Fensel(2005)]{fensel2005spinning}
D.~Fensel.
\newblock \emph{Spinning the Semantic Web: bringing the World Wide Web to its full potential}.
\newblock MIT press, 2005.
\bibitem[Fey and Lenssen(2019)]{pyg}
M.~Fey and J.~E. Lenssen.
\newblock Fast graph representation learning with {PyTorch Geometric}.
\newblock In \emph{ICLR Workshop on Representation Learning on Graphs and Manifolds}, 2019.
\bibitem[Fu et~al.(2016)Fu, Ding, Seal, Chen, Sun, and Bolton]{fu2016predicting}
G.~Fu, Y.~Ding, A.~Seal, B.~Chen, Y.~Sun, and E.~Bolton.
\newblock Predicting drug target interactions using meta-path-based semantic network analysis.
\newblock \emph{BMC bioinformatics}, 17\penalty0 (1):\penalty0 1--10, 2016.
\bibitem[Heinzerling and Inui(2021)]{heinzerling-inui-2021-language}
B.~Heinzerling and K.~Inui.
\newblock Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries.
\newblock In \emph{Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume}, pages 1772--1791, Online, Apr. 2021. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2021.eacl-main.153}.
\newblock URL \url{https://aclanthology.org/2021.eacl-main.153}.
\bibitem[Himmelstein et~al.(2017)Himmelstein, Lizee, Hessler, Brueggeman, Chen, Hadley, Green, Khankhanian, and Baranzini]{hetionet}
D.~S. Himmelstein, A.~Lizee, C.~Hessler, L.~Brueggeman, S.~L. Chen, D.~Hadley, A.~Green, P.~Khankhanian, and S.~E. Baranzini.
\newblock Systematic integration of biomedical knowledge prioritizes drugs for repurposing.
\newblock \emph{Elife}, 6:\penalty0 e26726, 2017.
\bibitem[Hu et~al.(2023)Hu, Liu, Zhao, Hou, Nie, and Li]{hu2023survey}
L.~Hu, Z.~Liu, Z.~Zhao, L.~Hou, L.~Nie, and J.~Li.
\newblock A survey of knowledge enhanced pre-trained language models.
\newblock \emph{IEEE Transactions on Knowledge and Data Engineering}, 2023.
\bibitem[Hu et~al.(2020)Hu, Fey, Zitnik, Dong, Ren, Liu, Catasta, and Leskovec]{ogb}
W.~Hu, M.~Fey, M.~Zitnik, Y.~Dong, H.~Ren, B.~Liu, M.~Catasta, and J.~Leskovec.
\newblock Open graph benchmark: Datasets for machine learning on graphs.
\newblock \emph{arXiv preprint arXiv:2005.00687}, 2020.
\bibitem[Hu et~al.(2021)Hu, Fey, Ren, Nakata, Dong, and Leskovec]{org-largescale}
W.~Hu, M.~Fey, H.~Ren, M.~Nakata, Y.~Dong, and J.~Leskovec.
\newblock Ogb-lsc: A large-scale challenge for machine learning on graphs.
\newblock \emph{arXiv preprint arXiv:2103.09430}, 2021.
\bibitem[Ilievski et~al.(2021)Ilievski, Szekely, and Zhang]{cskg}
F.~Ilievski, P.~Szekely, and B.~Zhang.
\newblock Cskg: The commonsense knowledge graph.
\newblock In \emph{The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6--10, 2021, Proceedings 18}, pages 680--696. Springer, 2021.
\bibitem[Kim et~al.(2016)Kim, Thiessen, Bolton, Chen, Fu, Gindulyte, Han, He, He, Shoemaker, et~al.]{kim2016pubchem}
S.~Kim, P.~A. Thiessen, E.~E. Bolton, J.~Chen, G.~Fu, A.~Gindulyte, L.~Han, J.~He, S.~He, B.~A. Shoemaker, et~al.
\newblock Pubchem substance and compound databases.
\newblock \emph{Nucleic acids research}, 44\penalty0 (D1):\penalty0 D1202--D1213, 2016.
\bibitem[Kolodziej et~al.(2019)Kolodziej, Aznaveh, Bullock, David, Davis, Henderson, Hu, and Sandstrom]{suitesparse-web}
S.~P. Kolodziej, M.~Aznaveh, M.~Bullock, J.~David, T.~A. Davis, M.~Henderson, Y.~Hu, and R.~Sandstrom.
\newblock The suitesparse matrix collection website interface.
\newblock \emph{Journal of Open Source Software}, 4\penalty0 (35):\penalty0 1244, 2019.
\bibitem[Leone et~al.(2022)Leone, Huber, Arora, Garc{\'\i}a-Dur{\'a}n, and West]{eareeval}
M.~Leone, S.~Huber, A.~Arora, A.~Garc{\'\i}a-Dur{\'a}n, and R.~West.
\newblock A critical re-evaluation of neural methods for entity alignment.
\newblock \emph{Proceedings of the VLDB Endowment}, 15\penalty0 (8):\penalty0 1712--1725, 2022.
\bibitem[Leskovec and Sosi{\v{c}}(2016)]{snap}
J.~Leskovec and R.~Sosi{\v{c}}.
\newblock Snap: A general-purpose network analysis and graph-mining library.
\newblock \emph{ACM Transactions on Intelligent Systems and Technology (TIST)}, 8\penalty0 (1):\penalty0 1, 2016.
\bibitem[Li et~al.(2020)Li, Zakhozhyi, Zhu, and Salazar]{li2020domain}
Y.~Li, V.~Zakhozhyi, D.~Zhu, and L.~J. Salazar.
\newblock Domain specific knowledge graphs as a service to the public: Powering social-impact funding in the us.
\newblock In \emph{Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining}, pages 2793--2801, 2020.
\bibitem[Morselli~Gysi et~al.(2021)Morselli~Gysi, Do~Valle, Zitnik, Ameli, Gan, Varol, Ghiassian, Patten, Davey, Loscalzo, et~al.]{morselli2021network}
D.~Morselli~Gysi, {\'I}.~Do~Valle, M.~Zitnik, A.~Ameli, X.~Gan, O.~Varol, S.~D. Ghiassian, J.~Patten, R.~A. Davey, J.~Loscalzo, et~al.
\newblock Network medicine framework for identifying drug-repurposing opportunities for covid-19.
\newblock \emph{Proceedings of the National Academy of Sciences}, 118\penalty0 (19):\penalty0 e2025581118, 2021.
\bibitem[Nicholson and Greene(2020)]{nicholson2020constructing}
D.~N. Nicholson and C.~S. Greene.
\newblock Constructing knowledge graphs and their biomedical applications.
\newblock \emph{Computational and structural biotechnology journal}, 18:\penalty0 1414--1428, 2020.
\bibitem[OpenAI(2023)]{gpt4}
OpenAI.
\newblock Gpt-4 technical report, 2023.
\bibitem[Peng et~al.(2023)Peng, Xia, Naseriparsa, and Osborne]{kg-opportunities}
C.~Peng, F.~Xia, M.~Naseriparsa, and F.~Osborne.
\newblock Knowledge graphs: Opportunities and challenges.
\newblock \emph{Artificial Intelligence Review}, pages 1--32, 2023.
\bibitem[Petroni et~al.(2019)Petroni, Rockt{\"a}schel, Lewis, Bakhtin, Wu, Miller, and Riedel]{lama}
F.~Petroni, T.~Rockt{\"a}schel, P.~Lewis, A.~Bakhtin, Y.~Wu, A.~H. Miller, and S.~Riedel.
\newblock Language models as knowledge bases?
\newblock \emph{arXiv preprint arXiv:1909.01066}, 2019.
\bibitem[Ruffinelli et~al.(2020)Ruffinelli, Broscheit, and Gemulla]{ruffinelli2020you}
D.~Ruffinelli, S.~Broscheit, and R.~Gemulla.
\newblock You {CAN} teach an old dog new tricks! on training knowledge graph embeddings.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\newblock URL \url{https://openreview.net/forum?id=BkxSmlBFvr}.
\bibitem[Speer et~al.(2017)Speer, Chin, and Havasi]{conceptnet}
R.~Speer, J.~Chin, and C.~Havasi.
\newblock Conceptnet 5.5: An open multilingual graph of general knowledge.
\newblock In \emph{Proceedings of the AAAI conference on artificial intelligence}, volume~31, 2017.
\bibitem[Su et~al.(2020)Su, Tong, Zhu, Cui, and Wang]{su2020network}
C.~Su, J.~Tong, Y.~Zhu, P.~Cui, and F.~Wang.
\newblock Network embedding in biomedical data science.
\newblock \emph{Briefings in bioinformatics}, 21\penalty0 (1):\penalty0 182--197, 2020.
\bibitem[Sun et~al.(2020)Sun, Zhang, Hu, Wang, Chen, Akrami, and Li]{sun2020benchmarking}
Z.~Sun, Q.~Zhang, W.~Hu, C.~Wang, M.~Chen, F.~Akrami, and C.~Li.
\newblock A benchmarking study of embedding-based entity alignment for knowledge graphs.
\newblock \emph{arXiv preprint arXiv:2003.07743}, 2020.
\bibitem[Taori et~al.(2023)Taori, Gulrajani, Zhang, Dubois, Li, Guestrin, Liang, and Hashimoto]{alpaca}
R.~Taori, I.~Gulrajani, T.~Zhang, Y.~Dubois, X.~Li, C.~Guestrin, P.~Liang, and T.~B. Hashimoto.
\newblock Stanford alpaca: An instruction-following llama model.
\newblock \url{https://github.com/tatsu-lab/stanford_alpaca}, 2023.
\bibitem[Toutanova and Chen(2015)]{toutanova}
K.~Toutanova and D.~Chen.
\newblock Observed versus latent features for knowledge base and text inference.
\newblock In \emph{Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality}, pages 57--66, Beijing, China, July 2015. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/W15-4007}.
\newblock URL \url{https://aclanthology.org/W15-4007}.
\bibitem[Touvron et~al.(2023)Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozi{\`e}re, Goyal, Hambro, Azhar, et~al.]{llama}
H.~Touvron, T.~Lavril, G.~Izacard, X.~Martinet, M.-A. Lachaux, T.~Lacroix, B.~Rozi{\`e}re, N.~Goyal, E.~Hambro, F.~Azhar, et~al.
\newblock Llama: Open and efficient foundation language models.
\newblock \emph{arXiv preprint arXiv:2302.13971}, 2023.
\bibitem[Wang et~al.(2023)Wang, Fu, Du, Gao, Huang, Liu, Chandak, Liu, Van~Katwyk, Deac, et~al.]{wang2023scientific}
H.~Wang, T.~Fu, Y.~Du, W.~Gao, K.~Huang, Z.~Liu, P.~Chandak, S.~Liu, P.~Van~Katwyk, A.~Deac, et~al.
\newblock Scientific discovery in the age of artificial intelligence.
\newblock \emph{Nature}, 620\penalty0 (7972):\penalty0 47--60, 2023.
\bibitem[Wang et~al.(2014)Wang, Zhang, Feng, and Chen]{transe}
Z.~Wang, J.~Zhang, J.~Feng, and Z.~Chen.
\newblock Knowledge graph embedding by translating on hyperplanes.
\newblock In \emph{Proceedings of the AAAI conference on artificial intelligence}, volume~28, 2014.
\bibitem[Wei et~al.(2021)Wei, Wang, Zhang, Bhatia, and Arnold]{wei2021knowledge}
X.~Wei, S.~Wang, D.~Zhang, P.~Bhatia, and A.~Arnold.
\newblock Knowledge enhanced pretrained language models: A compreshensive survey.
\newblock \emph{arXiv preprint arXiv:2110.08455}, 2021.
\bibitem[Widjaja et~al.(2022)Widjaja, Gashteovski, Ben~Rim, Liu, Malon, Ruffinelli, Lawrence, and Neubig]{kgxboard}
H.~Widjaja, K.~Gashteovski, W.~Ben~Rim, P.~Liu, C.~Malon, D.~Ruffinelli, C.~Lawrence, and G.~Neubig.
\newblock {KG}x{B}oard: Explainable and interactive leaderboard for evaluation of knowledge graph completion models.
\newblock In \emph{Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations}, pages 338--350, Abu Dhabi, UAE, Dec. 2022. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/2022.emnlp-demos.34}.
\newblock URL \url{https://aclanthology.org/2022.emnlp-demos.34}.
\bibitem[Wishart et~al.(2018)Wishart, Feunang, Guo, Lo, Marcu, Grant, Sajed, Johnson, Li, Sayeeda, et~al.]{drugbank}
D.~S. Wishart, Y.~D. Feunang, A.~C. Guo, E.~J. Lo, A.~Marcu, J.~R. Grant, T.~Sajed, D.~Johnson, C.~Li, Z.~Sayeeda, et~al.
\newblock Drugbank 5.0: a major update to the drugbank database for 2018.
\newblock \emph{Nucleic acids research}, 46\penalty0 (D1):\penalty0 D1074--D1082, 2018.
\bibitem[Yang et~al.(2014)Yang, Yih, He, Gao, and Deng]{distmult}
B.~Yang, W.-t. Yih, X.~He, J.~Gao, and L.~Deng.
\newblock Embedding entities and relations for learning and inference in knowledge bases.
\newblock \emph{arXiv preprint arXiv:1412.6575}, 2014.
\bibitem[Yang et~al.(2023)Yang, Chen, Li, Ding, and Wu]{yang2023chatgpt}
L.~Yang, H.~Chen, Z.~Li, X.~Ding, and X.~Wu.
\newblock Chatgpt is not enough: Enhancing large language models with knowledge graphs for fact-aware language modeling.
\newblock \emph{arXiv preprint arXiv:2306.11489}, 2023.
\bibitem[Zamini et~al.(2022)Zamini, Reza, and Rabiei]{zamini2022review}
M.~Zamini, H.~Reza, and M.~Rabiei.
\newblock A review of knowledge graph completion.
\newblock \emph{Information}, 13\penalty0 (8):\penalty0 396, 2022.
\bibitem[Zhang et~al.(2020)Zhang, Liu, Li, Zhu, Liang, and An]{zhang2020predicting}
L.~Zhang, B.~Liu, Z.~Li, X.~Zhu, Z.~Liang, and J.~An.
\newblock Predicting mirna-disease associations by multiple meta-paths fusion graph embedding model.
\newblock \emph{BMC bioinformatics}, 21\penalty0 (1):\penalty0 1--19, 2020.
\bibitem[Zheng et~al.(2021)Zheng, Rao, Song, Zhang, Xiao, Fang, Yang, and Niu]{pharmkg}
S.~Zheng, J.~Rao, Y.~Song, J.~Zhang, X.~Xiao, E.~F. Fang, Y.~Yang, and Z.~Niu.
\newblock Pharmkg: a dedicated knowledge graph benchmark for bomedical data mining.
\newblock \emph{Briefings in bioinformatics}, 22\penalty0 (4):\penalty0 bbaa344, 2021.
\bibitem[Zhu et~al.(2017)Zhu, Zhou, Xu, Liu, Tan, et~al.]{KGgeology}
Y.~Zhu, W.~Zhou, Y.~Xu, J.~Liu, Y.~Tan, et~al.
\newblock Intelligent learning for knowledge graph towards geological data.
\newblock \emph{Scientific Programming}, 2017, 2017.
\bibitem[Zou(2020)]{KGphysics}
X.~Zou.
\newblock A survey on application of knowledge graph.
\newblock In \emph{Journal of Physics: Conference Series}, volume 1487, page 012016. IOP Publishing, 2020.
\end{thebibliography}