Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities
Extracted source text · arXiv 2505.20099
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[preprint]{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}
%Custom
\usepackage[edges]{forest}
\usepackage{booktabs}
\usepackage{enumitem}
\usepackage{bbding}
\usepackage{array}
\usepackage{float}
\usepackage{placeins}
% for space compress
\usepackage[skip=3pt]{caption}
\setlength{\partopsep}{0pt}
\setlength{\topsep}{0pt}
\setlength{\itemsep}{0pt}
\setlength{\parskip}{0pt}
\setlength{\parsep}{0pt}
\setlist{nolistsep}
\usepackage[compact]{titlesec}
\setlength{\textfloatsep}{2pt}
\setlength{\floatsep}{2pt}
\setlength{\intextsep}{2pt}
\setlength{\abovecaptionskip}{1pt}
\setlength{\belowcaptionskip}{1pt}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\newcommand{\eat}[1]{}
\newcommand{\spara}[1]{\smallskip\noindent{\bf #1}}
\title{Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities}
%\thanks{\llap{}\:\:\:Corresponding authors.}; \footnotemark[1]
\author{
\textbf{Chuangtao Ma\textsuperscript{1}},
\textbf{Yongrui Chen\textsuperscript{2}},
\textbf{Tianxing Wu\textsuperscript{2}},
\textbf{Arijit Khan\textsuperscript{1}},
\textbf{Haofen Wang\textsuperscript{3}}
\\
\textsuperscript{1}Aalborg University, Denmark,
\textsuperscript{2}Southeast University, China,
\textsuperscript{3}Tongji University, China
\\
\small{
\texttt{\{chuma, arijitk\}@cs.aau.dk, \{yongruichen, tianxingwu\}@seu.edu.cn, haofenwang@tongji.edu.cn}}
}
\begin{document}
\maketitle
\begin{abstract}
Large language models (LLMs) have demonstrated remarkable performance on question-answering (QA) tasks because of their superior capabilities in natural language understanding and generation. However, LLM-based QA struggles with complex QA tasks due to poor reasoning capacity, outdated knowledge, and hallucinations.
Several recent works synthesize LLMs and knowledge graphs (KGs) for QA to address the above challenges.
In this survey, we propose a new structured taxonomy that categorizes the methodology of synthesizing LLMs and KGs for QA according to the categories of QA and the KG's role when integrating with LLMs.
We systematically survey state-of-the-art advances in synthesizing LLMs and KGs for QA and compare and analyze these approaches in terms of strength, limitations, and KG requirements.
We then align the approaches with QA and discuss how these approaches address the main challenges of different complex QA.
Finally, we summarize the advancements, evaluation metrics, and benchmark datasets and highlight open challenges and opportunities.
\end{abstract}
\section{Introduction}
Question answering (QA) plays a fundamental role in artificial intelligence, natural language processing, information retrieval, and data management areas since it has a wide range of applications such as text generation, chatbots, dialog generation, web search, entity linking, natural language query, fact-checking, etc.
The pre-trained language models (PLMs) and recent LLMs have shown superior performance in several QA tasks such as KBQA (Knowledge bases QA), KGQA (Knowledge graph QA), CDQA (Closed domain QA), etc.
However, PLM and LLM-based methods are incapable of handling complex QA scenarios due to the following limitations. {\bf(1)} {\em Limited complex reasoning capability:} LLMs encapsulate very limited reasoning capability since they have been pre-trained with the tasks of predicting the next word in a text sequence. {\bf(2)} {\em Lack of up-to-date and domain-specific knowledge:} LLMs are incapable of generating accurate and up-to-date responses for domain-specific QA because they have been pre-trained on the data and knowledge with a cutoff date. {\bf(3)} {\em Tendency to generate hallucinated content:} LLMs usually generate hallucinated content due to the lack of factual verification and logical consistency checking.
\vspace{-1mm}
\spara{Challenges.} Retrieval augmented generation (RAG)~\citep{mao2021generation} is proposed for open-domain QA by retrieving the relevant contexts from large documents, and several techniques, such as graph neural networks (GNNs)~\citep{li2025graph}, have been investigated to enhance the retrieval coverage from passages.
Although RAG-based QA can generate better responses in comparison to NoRAG-based QA, it still has a limited reasoning capacity and understanding of user interactions during complex QA.
Complex QA usually involves knowledge interactions and fusion among data across modalities and sources, and an excellent understanding of complex queries and user interactions, whereas the RAG-based QA suffers a lot from the following technical challenges. {\bf(1)} {\em Knowledge conflicts:} Conflicts occur due to the fusion of inconsistent and overlapping knowledge between LLMs and external sources in RAG-based QA that may further tend to generate inconsistent answers. {\bf(2)} {\em Poor relevance and quality of retrieved context:} The accuracy of the generated answers in RAG-based QA largely depends on the relevance and quality of the retrieved context, where irrelevant context leads to incorrect results. {\bf(3)} {\em Lack of iterative and multi-hop reasoning:} RAG-based QA struggles to generate accurate and explainable answers for questions requiring global and summarized contexts due to a lack of iterative.
The emergence of synthesizing LLMs+KGs provides a unique opportunity to address the above challenges and limitations of LLMs for knowledge-intensive tasks, e.g., complex QA~\citep{ma2025unifying}.
The graph retrieval augmented generation (GraphRAG)~\citep{zhang2025survey, peng2024graph, han2024retrieval} and knowledge graph retrieval augmented generation (KG-RAG) ~\citep{sanmartin2024kg, yang2024kg} have demonstrated the strengths of unifying LLMs with KGs for complex QA.
These GraphRAG and KG-RAG based QA approaches introduce several modules, such as knowledge integration and fusion, reasoning guidelines, and knowledge validation and refinement, to mitigate the above challenges.
\spara{Motivation.} In recent years, a rapidly increasing number of works on synthesizing LLMs and KGs for QA have been conducted to achieve complex QA under open-domain and long-context settings.
This survey aims to address the aforementioned limitations by outlining the recent progress in integrating LLMs with KGs for complex QA, summarizing technical advances, and identifying open challenges and future research opportunities.
Our survey differs from the existing surveys (\textbf{details in Appendix \S \ref{subsec:appendix-related-survey}}), it provides a comprehensive overview of the recent advancements of QA from the perspective of the roles of KGs when synthesizing LLMs and KGs for complex QA.
\spara{Taxonmy.} We categorize the methodology of synthesizing LLMs and KGs and complex QA from different perspectives, and a structured taxonomy (\textbf{details in Appendix \S ~\ref{sec:appendix-taxonomy}}) is given in Figure~\ref{fig:taxonomy}.
Notably, these categories, divided from different perspectives, are non-exclusive, where a work might be classified into multiple categories from different perspectives.
The online resources of this survey can be accessed via Github.\footnote{\url{https://github.com/machuangtao/LLM-KG4QA/}}
\begin{figure*}[ht]
\centering
\tiny
\begin{forest}
for tree={
forked edges,
grow'=0,
draw,
rounded corners,
node options={align=center},
calign=edge midpoint,
font=\scriptsize,
},
[Large Language Models Meet Knowledge Graphs for QA, rotate=90, text width=6cm, fill=black!10
[Complex QA §\ref{sec:complexqa}, text width=2.0cm, for tree={fill=cyan!20}
[Multi-doc QA §\ref{subsec:multi-doc}, text width=2.5cm
[
\textbf{KGP}~\cite{wang2024knowledge};
\textbf{CuriousLLM}~\citep{yang2025curiousllm};
\textbf{VisDom}~\citep{suri2024visdom},
text width=10.0cm, node options={align=left}
]
]
[Multi-modal QA §\ref{subsec:multi-modal}, text width=2.5cm
[
%\textbf{LaKo}~\citep{chen2022lako};
\textbf{MMJG}~\citep{wang2022knowledge};
\textbf{RAMQA}~\citep{bai2025ramqa};
\textbf{KVQA}~\citep{dong2024modality},
%\textbf{ReasVQA}~\citep{liang2025reasvqa},
text width=10.0cm, node options={align=left}
]
]
[Muti-hop QA §\ref{subsec:multi-hop}, text width=2.5cm
[
\textbf{GraphLLM}~\citep{qiao2024graphllm};
%\textbf{PokeMQA}~\citep{gu2024pokemqa};
\textbf{HOLMES}~\citep{panda2024holmes};
%\textbf{LLM-KGMQA}~\citep{wang2024llm};
\textbf{GMeLLo}~\citep{chen2024llm},
%\textbf{SR-RAG}~\citep{saleh2024sg},
text width=10.0cm, node options={align=left}
]
]
[Muti-run and Conversational QA §\ref{subsec:multirun-conversational}, text width=2.5cm
[
\textbf{CoRnNetA}~\citep{liu2024conversational};
\textbf{LLMs-Graph-Reasoning}~\citep{chen2024llm};
\textbf{SELF-multi-RAG}~\citep{roy2024learning},
% \textbf{ConvKGYarn}~\citep{pradeep2024convkgyarn};
%\textbf{Chatty-Gen}~\citep{omar2025dialogue},
text width=10.0cm, node options={align=left}
]
]
[Explainable QA §\ref{subsec:x-qa}, text width=2.5cm
[
\textbf{RoHT}~\citep{zhang2023reasoning};
\textbf{EXPLAIGNN}~\citep{christmann2023explainable};
\textbf{RID}~\citep{feng2025retrieval},
text width=10.0cm, node options={align=left}
]
]
[Temporal QA §\ref{subsec:tem-qa}, text width=2.5cm
[
\textbf{TimeR$^4$}~\citep{qian2024timer4};
\textbf{GenTKGQA}~\citep{gao2024two};
\textbf{KG-IRAG}~\citep{yang2025kg},
text width=10.0cm, node options={align=left}
]
]
]
[KGs as Background Knowledge §\ref{sec:kgasbk}, text width=2.0cm, for tree={fill=red!20}
[Knowledge Integration and Fusion §\ref{subsec:kgasb1}, text width=2.5cm
[
\textbf{InfuserKI}~\citep{wang2024infuserki};
\textbf{KEFF}~\citep{zhao2025improving};
\textbf{KnowLA}~\citep{luo2024knowla};
\textbf{KG-Adapter}~\citep{tian2024kg};
\textbf{GAIL}~\citep{zhang2024gail},
text width=10.0cm, node options={align=left}
]
]
[Retrieval Augmented Generation §\ref{subsec:kgasb3}, text width=2.5cm
[
\textbf{PLRTQA}~\citep{alawwad2024enhancing};
\textbf{GNN-Ret}~\citep{li2025graph};
\textbf{GRAG}~\citep{hu2024grag};
\textbf{LEGO-GraphRAG}~\citep{cao2024lego};
\textbf{KG-RAG}~\citep{xu2024retrieval};
\textbf{KG$^2$RAG}~\citep{zhu2025knowledge},
text width=10.0cm, node options={align=left}
]
]
]
[KGs as Reasoning Guidelines §\ref{sec:kgasrg}, text width=2.0cm, for tree={fill=yellow!20}
[Offline KG Guidelines §\ref{subsec:kgasrg1}, text width=2.5cm
[
\textbf{SR}~\citep{zhang2022subgraph};
\textbf{InfuserKI}~\citep{wang2024infuserki};
\textbf{KnowLA}~\citep{luo2024knowla};
\textbf{KG-Adapter}~\citep{tian2024kg};
\textbf{GAIL}~\citep{zhang2024gail};
\textbf{KELDaR}~\citep{li2024framework},
text width=10.0cm, node options={align=left}
]
]
[Online KG Guidelines §\ref{subsec:kgasrg2}, text width=2.5cm
[
\textbf{Oreo}~\citep{hu2022empowering};
\textbf{KBIGER}~\citep{du2022knowledge};
\textbf{LLM-ARK}~\citep{huang2023llm};
\textbf{ToG}~\citep{sun2023think};
\textbf{ToG-2}~\citep{ma2025think};
\textbf{KG-CoT}~\citep{zhao2024kg},
text width=10.0cm, node options={align=left}
]
]
[Agent-based KG Guidelines §\ref{subsec:kgasrg3}, text width=2.5cm
[
\textbf{KG-Agent}~\citep{jiang2024kg};
\textbf{ODA}~\citep{sun2024oda};
\textbf{GREASELM}~\citep{zhang2022greaselm},
text width=10.0cm, node options={align=left}
]
]
]
[KGs as Refiners and Validators §\ref{sec:kgasrv}, text width=2.0cm, for tree={fill=orange!20}
[KG-Driven Filtering and Validation §\ref{subsec:kgasrv1}, text width=2.5cm
[
\textbf{ACT-Selection}~\citep{salnikov2023answer};
\textbf{Q-KGR}~\citep{zhang2024question};
\textbf{KG-Rank}~\citep{yang2024kg};
\textbf{KGR}~\citep{guan2024mitigating},
text width=10.0cm, node options={align=left}
]
]
[KG-Augmented Output Refinement §\ref{subsec:kgasrv2}, text width=2.5cm
[
\textbf{EFSUM}~\citep{ko2024evidence};
\textbf{InteractiveKBQA}~\citep{xiong2024interactive};
\textbf{LPKG}~\citep{wang2024learning},
text width=10.0cm, node options={align=left}
]
]
]
[Advancements §\ref{sec:advance}, text width=2.0cm, for tree={fill=blue!20}
[Hybrid Method §\ref{subsec:hybridm}, text width=2.5cm, for tree={fill=blue!20}
[
\textbf{KG-RAG}~\citep{sanmartin2024kg};
\textbf{LongRAG}~\citep{zhao2024longrag};
\textbf{KG-Rank}~\citep{yang2024kg};
\textbf{FRAG}~\citep{zhao2024frag};
\textbf{KGQA}~\citep{ji2024retrieval},
text width=10.0cm, node options={align=left}
]
]
[Optimization §\ref{subsec:optimization}, text width=2.5cm, for tree={fill=blue!20}
[
\textbf{PG-RAG}~\citep{liang2024empowering};
\textbf{KGP}~\citep{wang2024knowledge};
\textbf{SPOKE KG-RAG}~\citep{soman2024biomedical},
% \textbf{GoR}~\citep{zhang2024graph};
% \textbf{KG-Retriever}~\citep{chen2024kg};
% \textbf{LMQL}~\citep{beurer2023prompting};
% \textbf{StraGo}~\citep{wu2024strago};
% \textbf{KS-LLM}~\citep{zheng2024ks};
% \textbf{QUASAR}~\citep{christmann2024rag};
% \textbf{BriefContext}~\citep{zhang2024mapreduce};
% \textbf{KGLens}~\citep{zheng2024kglens};
% \textbf{Coke}~\citep{dong2024cost};
% \textbf{CGPE}~\citep{tao2024cluegoptimizing},
text width=10.0cm, node options={align=left}
]
]
]
]
\end{forest}
\vspace{0.1mm}
\caption{A Structured Taxonomy of Synthesizing LLMs and KGs for QA. }
\label{fig:taxonomy}
\vspace{-3mm}
\end{figure*}
\spara{Contributions.} Considering the popularity and mainstream adoption of both LLMs and KGs, and their wide applications in QA, our survey is timely.
The contributions of this work are summarized below.
\textbf{(1)} We introduce the structured taxonomy that categorizes state-of-the-art (SOTA) works on synthesizing LLMs+KGs for QA. \textbf{(2)} We review the SOTA works on synthesizing LLMs and KGs for QA in various categories and discuss the recent advanced topics in this field. \textbf{(3)} We create the alignments between various approaches in synthesizing LLMs and KGs and complex QA and highlight how these approaches address the specific challenges of different complex QA; \textbf{(4)} We discuss the current challenges and research opportunities on synthesizing LLMs and KGs for QA.
\section{Complex QA} \label{sec:complexqa}
The methodology in synthesizing LLMs and KGs for QA has been exploited as follows.
\subsection{Multi-document QA} \label{subsec:multi-doc}
Multi-document QA refers to the QA over contexts from multiple documents, while efficiently and effectively retrieving the relevant knowledge from multiple contexts is the main technical challenge.
To reduce the retrieval latency and improve the quality of the retrieved context for multi-document QA, KGP~\citep{wang2024knowledge} introduces an LLM-based graph traversal agent for retrieving relevant knowledge from KG.
Similarly, CuriousLLM~\citep{yang2025curiousllm} integrates a knowledge graph prompting, reasoning-infused LLM agent, and graph traversal agent to augment LLMs for multi-document QA.
VisDom~\citep{suri2024visdom} introduces a novel multimodal RAG for multi-document question answering by integrating and fusing the multi-modal knowledge and leveraging the (Chain-of-thought) CoT-based reasoning.
\vspace{-1mm}
\subsection{Multi-modal QA} \label{subsec:multi-modal}
Multi-modal QA refers to the QA over multi-modal data, and visual QA (VQA) is one of the typical multi-modal QA.
%LaKo~\citep{chen2022lako} proposes a knowledge-driven VQA method that retrieves the factual triples from large KG and then effectively fuses with the language models based on late knowledge-to-text injection for text generation.
To retrieve the most relevant knowledge from the external KG for enhancing VQA, MMJG~\citep{wang2022knowledge} introduces an adaptive knowledge selection to jointly select knowledge from visual and text knowledge based on the knowledge-aware attention and multi-modal guidance.
%Multi-modal QA usually involves knowledge retrieval from multi-modal documents, such as text, images, video, etc.
To effectively retrieve the evidence from multi-modal data, RAMQA~\citep{bai2025ramqa} enhances multi-modal retrieval-augmented QA by integrating learning-to-rank with training of generative models via multi-task learning.
%To facilitate cross-modal reasoning,
KVQA~\citep{dong2024modality} integrates LLMs with multimodal knowledge by using a two-stage prompting and a pseudo-siamese graph medium fusion to balance intra-modal and inter-modal reasoning.
%To address the challenges of cross-modal reasoning, KVQA~\citep{dong2024modality} introduces an enhanced KBQA by integrating LLMs and multimodal knowledge through a two-stage prompting strategy and a tailored pseudo-siamese graph medium fusion (PS-GMF) for balancing intra-modal processing and inter-modal exchange.
%ReasVQA~\citep{liang2025reasvqa} enhances the reasoning capabilities of VideoQA models by integrating reasoning generation from Multimodal LLMs and reasoning refinement based on multi-task learning.
\subsection{Multi-hop QA} \label{subsec:multi-hop}
Multi-hop QA differs from simple QA, it usually involves multi-step reasoning to generate the final answers.
The basic idea is to decompose the multi-hop questions into multiple single-hop questions, then generate the answers for each single-hop question, and finally integrate them~\citep{linders2025knowledge}.
For instance, GraphLLM~\citep{qiao2024graphllm} leverages LLMs to decompose the multi-hop question into several simple sub-questions and retrieve the sub-graphs via GNNs and LLMs to generate the answers for sub-questions based on graph reasoning.
%PokeMQA~\citep{gu2024pokemqa} enhances multi-hop question answering by decoupling and decompiling multi-hop questions into simple questions and then integrating with LLMs for reliable reasoning based on knowledge editing.
%It differs from the above work that
HOLME~\citep{panda2024holmes} utilizes a context-aware retrieved and pruned hyper-relational KG that is constructed based on the entity-document graph to enhance LLMs for generating the answers of multi-hop QA.
%Similarly, LLM-KGMQA~\citep{wang2024llm} directly addresses the multi-hop QA in the medical domain by introducing LLMs agent-based entity-link, subgraph construction, and knowledge fusion and pruning for multi-hop knowledge paths reasoning and generation.
To enable accurate fact retrieval and reasoning of LLMs for multi-hop QA, GMeLLo~\citep{chen2024llm} effectively integrates the explicit knowledge of KGs with the linguistic knowledge of LLMs by introducing the fact triple extraction, relation chain extraction, and query and answer generation.
%SR-RAG~\citep{saleh2024sg} enhances the capability of LLMs for multi-hop QA by introducing the subgraph RAG that retrieves the subgraph from KG based on Cypher query and feeds it as a context to LLM for generating the final answers.
\subsection{Multi-run and Conversational QA} \label{subsec:multirun-conversational}
The challenge of multi-run and conversational QA lies in how to make the language model (LM) easily understand the questions and intermediate interactions.
To make user interactions easily understood by machines, CoRnNetA~\citep{liu2024conversational} introduces LLM-based question reformulations, reinforcement learning agents, and soft reward mechanisms to improve the interpretation of multi-turn interactions with KGs.
The conversational QA involves several multi-run QA to refine and get accurate answers through multiple rounds of interactions.
The knowledge aggregation module and graph reasoning are introduced for joint reasoning between the graph and LLMs~\citep{jain2024integrating} to address the challenges of understanding the question and context for conversational QA.
%Thereby, the knowledge aggregation module and graph reasoning are introduced to address the challenges of understanding the question and context for conversational QA, in which the relevant evidence is retrieved, ranked, and aggregated from multiple knowledge sources for joint reasoning between graph and LLMs~\citep{jain2024integrating}.
To improve the contextual understanding and the answer quality for conversational QA, SELF-multi-RAG~\citep{roy2024learning} leverages LLMs to retrieve from the summarized conversational history and reuse the retrieved knowledge for augmentation.
%To facilitate the dataset generation for conversational QA, ConvKGYarn~\citep{pradeep2024convkgyarn} aims to ensure the quality and contextual relevance of the conversational QA dataset by introducing a configurable framework that leverages knowledge graphs and LLMs to efficiently generate scalable conversational QA datasets.
%Chatty-Gen~\citep{omar2025dialogue} automates the generation of high-quality dialogue benchmarks from knowledge graphs based on multi-stage RAG.
\subsection{Explainable QA} \label{subsec:x-qa}
Explainable QA (XQA) aims to provide explanations for the generated answers based on the reasoning over the factual KGs.
To effectively integrate the multiple sources of knowledge for XQA, RoHT~\citep{zhang2023reasoning} introduces a two-stage XQA method that implements the probabilistic reasoning based on the constructed Hierarchical Question Decomposition Tree (HQDT) from the aggregated knowledge.
To trace the provenance and improve the explainability of the answers, EXPLAIGNN~\citep{christmann2023explainable} constructs a heterogeneous graph from retrieved KB knowledge and user explanations and generates explanatory evidence based on GNN with question-level attention.
%EXPLAIGNN~\citep{christmann2023explainable} constructs a heterogeneous graph based on the retrieved knowledge from KB and then integrates it with the user explanations, where GNN and question-level attention were employed for providing explanatory evidence of the answers.
RID~\citep{feng2025retrieval} directly integrates the unsupervised retrieval with LLMs\eat{the decoding of LLMs by dynamically adjusting decoding granularity and direction for adaptively generating the explainable answers} based on reinforcement learning-driven knowledge distillation.
\subsection{Tempral QA} \label{subsec:tem-qa}
The challenges of temporal QA lie in fully understanding the implicit time constraints and effectively incorporating them with temporal knowledge for temporal reasoning.
To improve the accuracy of LLMs in answering temporal questions, TimeR$^4$~\citep{qian2024timer4} introduces a Retrieve-Retriev-Rerank pipeline to augment the temporal reasoning of LLMs\eat{with temporal knowledge} by temporal knowledge-based fine-tuning.
%In comparison to static temporal knowledge, dynamic structured knowledge with temporal information enables incremental updating and dynamic integration of the temporal knowledge with LLMs.
Similarly, GenTKGQA~\citep{gao2024two} introduces a temporal GNN and virtual knowledge indicators to capture\eat{the correlation of neighboring nodes with} temporal knowledge embeddings, and further dynamically integrates retrieved subgraphs into LLMs for temporal reasoning.
To facilitate reasoning of LLMs with\eat{knowledge retrieval from} KGs, KG-IRAG~\citep{yang2025kg} enables LLMs to incrementally retrieve knowledge and evaluate its sufficiency for augmenting the capabilities of LLMs in answering time-sensitive and event-based queries involving temporal dependencies.
\section{Approaches and Alignments} \label{sec:approaches-alignment}
The strengths and limitations of the approach in synthesizing LLMs and KGs for QA are summarized (\textbf{details in Table~\ref{tab:taxonomy-comparsion}}), and the research progress on the alignment of these approaches with the complex QA are discussed (\textbf{details in Table~\ref{tab:alignment-approaches}}).
\subsection{KGs as Background Knowledge} \label{sec:kgasbk}
KGs usually play the role of background knowledge when synthesizing LLMs for complex QA, where knowledge fusion and RAG are the main technical paradigms (\textbf{comparison in Table~\ref{tab:summary-comparsion-kg-as-bgk}}).
\subsubsection{Knowledge Integration and Fusion} \label{subsec:kgasb1}
Knowledge integration and fusion aim to enhance language models (LMs) by integrating unknown knowledge into LMs for \eat{knowledge-intensive tasks, such as} QA.
\begin{figure}[ht]
\centering
\includegraphics[width=1.0 \linewidth]{fig/fig1.pdf}
\caption{Knowledge Integration and Fusion.}
\label{fig:1}
\end{figure}
As shown in Figure~\ref{fig:1}, the KGs and text are aligned via local subgraph extraction and entity linking, and then fed into the cross-model encoder to bidirectionally fuse text and KG to jointly train the language models for complex QA tasks~\citep{yasunaga2022deep,zhang2022greaselm}.
%In these methods, the local subgraph is extracted from external KGs and then aligned with the text to form the text and KG pair, by entity linking, that is then fed into the cross-model encoder to bidirectionally fuse text and KG to learn the joint reasoning over text and KG.
To address knowledge forgetting and noisy knowledge during knowledge integration, InfuserKI~\citep{wang2024infuserki} and KEFF~\citep{zhao2025improving} introduce the adaptive selection and knowledge enhancement filter, respectively, which selects the new knowledge and integrates it with LLMs.
Fine-tuning LLMs with text and knowledge graphs can improve their performance of LLMs on specified tasks.
%KnowLA~\citep{luo2024knowla} and
For instance, KG-Adapter~\citep{tian2024kg} improves parameter-efficient fine-tuning of LLMs by introducing a knowledge adaptation layer to LLMs.
GAIL~\citep{zhang2024gail} fine-tunes LLMs for lightweight KGQA models based on retrieved SPARQL-question pairs from KGs.
\subsubsection{Retrieval Augmented Generation} \label{subsec:kgasb3}
RAG serves as a retrieval and augmentation mechanism, as shown in Figure~\ref{fig:3}. It first retrieves relevant knowledge from the \eat{textual knowledge organized in the form of} text chunks based on vector-similarity retrieval, and then augments the LLMs by integrating the retrieved context with LLMs~\citep{alawwad2024enhancing, li2025graph}.
\begin{figure}[ht]
\centering
\includegraphics[width=1.0 \linewidth]{fig/fig3.pdf}
\caption{Retrieval Augmented Generation.}
\label{fig:3}
\end{figure}
However, the mainstream RAG methods retrieve the relevant knowledge from the textual chunks, which ignores the structured information and inter-relations of these textual chunks.
To mitigate this limitation, Graph RAG~\citep{hu2024grag} and KG-RAG~\citep{sanmartin2024kg, xu2024retrieval, linders2025knowledge} are proposed.
Instead of retrieving the knowledge from textual chunks, Graph RAG directly retrieves the relevant knowledge from graph data.
%Then it integrates the retrieved and pruned subgraphs with the query by aggregating and aligning the graph embeddings with text vectors based on GNNs.
GRAG~\citep{hu2024grag} retrieves the top-$k$ relevant subgraphs from the textual graph \eat{that is indexed based on PLMs}and then integrates the retrieved subgraphs with the query by aggregating and aligning the graph embeddings with text vectors based on GNNs.
LEGO-GraphRAG~\citep{cao2024lego} decomposes the retrieval process into subgraph-extraction, path-filtering, and path-refinement modules, thereby improving the reasoning capabilities of LLMs with retrieved knowledge. %to generate accurate answers for RAG-based QA.
KG$^2$RAG~\citep{zhu2025knowledge} retrieves the relevant subgraph from KG \eat{by leveraging BFS search over KGs}and expanding the textual chunks with the retrieved KG\eat{, and then incorporates the expanded chunks with the prompt} for augmenting the generation.
\subsubsection{Aligning with Complex QA}
In the approaches of synthesizing LLMs and KGs for QA where KG plays the role of background knowledge, the retrieved relevant knowledge from the factual knowledge graph can reconcile knowledge conflicts from the multiple documents for multiple-doc QA~\citep{wang2024knowledge}.
The cross-modal reasoning \eat{over the retrieved knowledge} can facilitate the cross-modal interaction and alignment for multi-modal QA~\citep{suri2024visdom}.
Additionally, the question decomposition of multi-hop QA can be augmented by fusing the knowledge from LLMs and KGs, which further facilitates iterative reasoning for generating the accurate final question~\citep{saleh2024sg, cao2024lego}.
The RAG~\citep{roy2024learning} and KG-RAG~\citep{sanmartin2024kg} can also improve the capabilities of LLMs in understanding the user's interactions for generating accurate answers for multi-run and conversational QA.
However, the key technical challenge behind this methodology is \textit{how to retrieve the relevant knowledge from large-scale KGs and then effectively fuse with LLMs without inducing knowledge conflicts?}
\subsection{KGs as Reasoning Guidelines} \label{sec:kgasrg}
%To enhance the reasoning capability of LLMs and further improve the accuracy and context relevancy of generated answers.
KGs can provide reasoning guidelines for LLMs to access precise knowledge from \eat{and logical connections between} factual evidence based on reasoning.
Recent methods (\textbf{comparison in Table~\ref{tab:summary-comparsion-kg-as-rg}}) for integrating the reasoning of KG and LLMs can be classified into three categories.
\subsubsection{Offline KG Guidelines} \label{subsec:kgasrg1}
In offline KG guidelines, KG supplies potential paths or subgraphs before the reasoning process by the LLM, from which the LLM selects the most relevant path for reasoning.
For instance, SR~\citep{zhang2022subgraph} trains a subgraph retriever that operates independently from the downstream reasoning process, while Keqing~\citep{wang2023keqing} decomposes complex questions using predefined templates, retrieves candidate entities and triples from a KG.
EtD~\citep{liu2024explore} initially uses a GNN to identify promising candidates and extracts relevant fine-grained knowledge pertinent to the questions, and then creates a knowledge-enhanced multiple-choice prompt to guide LLM for generating the final answer.
Recent studies have been exploring the application of novel formats of guidelines.
GCR~\citep{luo2024graph} first converts a KG into a KG-Trie and then develops a graph-constrained decoding and a lightweight LLM to generate multiple reasoning paths and candidate answers.
KELDaR~\citep{li2024framework} introduces the question decomposition and atomic retrieval modules to extract implicit information\eat{of reasoning as guidelines for atomic retrieval on KG} and retrieves the relevant subgraphs from KG to augment LLM for QA.
\subsubsection{Online KG Guidelines} \label{subsec:kgasrg2}
This paradigm emphasizes that the guidance of the KG directly involves the LLM's reasoning process, rather than merely relying on existing static knowledge. %In each reasoning step, the LLM needs to retrieve from the KG first, after which the next decision is made based on the retrieved information.
For example, Oreo~\citep{hu2022empowering} uses a contextualized random walk across a KG and conducts a single step of reasoning through the specific layers.
KBIGER~\citep{du2022knowledge} considers the ($k$-1)-th reasoning graph to construct the $k$-th instruction for reasoning and corrects erroneous predictions of intermediate entities.
LLM-ARK~\citep{huang2023llm} approaches reasoning tasks as sequential decision-making processes and employs Proximal Policy Optimization (PPO) for optimization.
ToG~\citep{sun2023think} allows LLMs to iteratively perform beam search over KGs, by which the most promising reasoning paths and the most likely reasoning outcomes are generated.
In contrast, ToG-2~\citep{ma2025think} utilizes entities as intermediaries to guide LLMs toward precise answers based on iterative retrieval between documents and KGs.
KG-CoT~\citep{zhao2024kg} leverages external KGs to generate reasoning paths for joint reasoning of LLMs and KGs to enhance the reasoning capabilities of LLMs for QA.
\subsubsection{Agent-based KG Guidelines} \label{subsec:kgasrg3}
KGs can also be integrated into the reasoning process of LLMs as a component within an Agent system, as shown in Figure~\ref{fig:4}.
\begin{figure}[ht]
\centering
\includegraphics[width=1.0 \linewidth]{fig/fig4.pdf}
\caption{KGs as Reasoning Guidelines.}
\label{fig:4}
\end{figure}
This integration allows the Agent to leverage structured knowledge for augmenting the decision-making and problem-solving capabilities of LLMs.
KG-Agent~\citep{jiang2024kg} is a multifunctional toolbox integrating LLMs with a KG-based executor and a knowledge memory system that autonomously selects tools and updates the memory to enhance the reasoning of LLMs over KGs.
ODA~\citep{sun2024oda} incorporates KG reasoning capabilities through a global observation approach, which improves reasoning abilities by employing a cyclical paradigm of observation, action, and reflection.
GREASELM~\citep{zhang2022greaselm} effectively integrates encoded representations from LMs and GNNs by introducing several modality interaction layers to seamlessly blend structured knowledge with language contexts.
\subsubsection{Aligning with Complex QA}
The approaches incorporating the KGs with LLMs can enable multi-hop and iterative reasoning over the factual graph of KGs and further augment the reasoning capability of LLMs for complex QA.
The challenges of knowledge retrieval, knowledge conflicts, interactions across modalities, and complex question decomposition in multi-document QA, multi-modal QA, and multi-hop QA can be easily mitigated based on joint reasoning over the factual KGs and LLMs~\citep{suri2024visdom,liang2025reasvqa,qiao2024graphllm}.
Moreover, the joint reasoning over the factual KGs and LLMs provides the logical inference chains and anchor for LLMs to generate explainable answers with clear evidence from factual KGs~\citep{zhao2024kg\eat{, christmann2023explainable}}.
Although the joint reasoning over the factual KGs and LLMs can address the challenges of complex QA well, joint KG–LLM reasoning remains inefficient because large-scale graph traversal is computationally intensive and time-consuming.
Moreover, the reasoning capabilities of KGs mainly depend on the completeness and knowledge coverage of KGs, where the incomplete, inconsistent, and outdated knowledge from KGs might induce noise or conflicts. \eat{,especially for intricate reasoning that requires integrating knowledge from multiple sources.}
The main challenge lies in \textit{how to improve the reasoning efficiency over the large-scale graph and reasoning capabilities under incomplete KG?}
\subsection{KGs as Refiners and Validators} \label{sec:kgasrv}
The factual evidence from KGs enables LLMs to refine and verify \eat{the correctness and relevancy of}the intermediate answers, as shown in Figure~\ref{fig:5}.
In these methods (\textbf{comparison in Table~\ref{tab:summary-comparsion-kg-as-rav}}), KGs act as refiner and validator.
\begin{figure}[ht]
\centering
\includegraphics[width=1.0 \linewidth]{fig/fig5.pdf}
\caption{KGs as Refiners and Validators.}
\label{fig:5}
\end{figure}
\subsubsection{KG-Driven Filtering and Validation} \label{subsec:kgasrv1}
KGs enhance the accuracy and reliability of LLM outputs by filtering and validating candidate answers with structured and verified information.
For instance, ACT-Selection~\citep{salnikov2023answer} filters and re-ranks answer candidates based on their types extracted from Wikidata.
Q-KGR~\citep{zhang2024question} improves the reasoning capabilities of LLMs by filtering out the irrelevant knowledge based on the ranking of the relevance score between the question and knowledge.
KGs can improve the factual accuracy of generated answers, as demonstrated by KG-Rank~\citep{yang2024kg}, which integrates medical KGs with re-ranking techniques to enhance the credibility of generated responses.
Moreover, KGR~\citep{guan2024mitigating} autonomously extracts and validates factual statements,\eat{in model outputs} significantly boosting performance on factual QA.
\subsubsection{KG-Augmented Output Refinement} \label{subsec:kgasrv2}
Integrating KGs with LLMs is essential to refining the outputs of LLMs for greater clarity and accuracy. EFSUM~\citep{ko2024evidence} employs LLM as a fact summarizer to generate relevant summaries from KGs, thereby improving performance in zero-shot QA. InteractiveKBQA~\citep{xiong2024interactive} enables iterative knowledge interactions, allowing LLMs to generate logical forms and refine outputs based on user feedback. LPKG~\citep{wang2024learning} fine-tunes LLMs with KG-derived planning data to enhance the planning capabilities of LLMs for sophisticated reasoning in complex QA.
\subsubsection{Aligning with Complex QA}
The approaches that leverage the retrieved factual evidence from KGs for refinement and validation are designed to augment the capability of LLMs in understanding user interactions and verifying the intermediate reasoning for multi-hop QA~\citep{chen2024llm} and multi-run and conversational QA~\citep{xiong2024interactive}, respectively.
However, the common case is that the knowledge from KGs is not available or not enough to verify the intermediate results, since the factual knowledge from KGs might be incomplete~\citep{zhou2025evaluating}.
In particular, the knowledge conflicts between the intermediate answer and KG facts might induce irrelevant results due to the poorly verified intermediate results.
Meanwhile, the refinement and validation of results largely depend on the correctness, timeliness, and completeness of factual knowledge in KGs.
Thereby, the main challenges behind this methodology are \textit{how to handle the knowledge conflicts between the intermediate answer and KG facts and incrementally update the KGs to ensure factual knowledge in KGs is up-to-date and correct?}
\eat{the response time for complex QA is mainly limited by the verification and refinement latency since the verification and refinement are based on facts and reasoning paths retrieved by graph traversal and reasoning over the large-scale KG, which is a time-consuming process.}
\subsection{Advancements} \label{sec:advance}
The advanced methods can be categorized into two subcategories: hybrid methods and optimization.
\subsubsection{Hybrid Method} \label{subsec:hybridm}
In addition to acting as a single role, the approaches (\textbf{comparison in Table~\ref{tab:summary-comparsion-hybrid-approah}}) where KGs serve multiple roles when synthesizing with LLMs for QA have been investigated.
For instance, KG-RAG~\citep{sanmartin2024kg} introduces the Chain-of-Explorations (CoE) to rank and obtain the top-$k$ nodes or relationships based on vector similarity, by which the most relevant knowledge is selected and then fed into LLMs for the final answer.
LongRAG~\citep{zhao2024longrag} retrieves the top-$k$ most relevant chunks based on a hybrid retriever and analyzes their relevance with the query by introducing a CoT-guided filter.
Furthermore, the KG-augmented prompting is introduced to LLMs for augmenting the generation of the final answer.
In KG-Rank~\citep{yang2024kg}, multiple ranking methods\eat{-- similarity ranking, answer expansion ranking, and maximal marginal relevance (MMR)}are introduced to refine the retrieved triples for augmenting the reasoning with the most relevant knowledge.
FRAG~\citep{zhao2024frag} introduces reasoning-aware and flexible-retrieval modules to retrieve reasoning paths from KGs, thereby guiding and augmenting LLM for efficient reasoning and answer generation.
KGQA~\citep{ji2024retrieval} combines the CoT-based prompting with graph retrieval to improve the retrieval quality and the multi-hop reasoning capability of LLMs for QA.
\subsubsection{Optimization} \label{subsec:optimization}
To mitigate the low efficiency and high computing costs of existing methods in unifying LLMs and KGs for complex QA, several optimization techniques (\textbf{extension in Appendix \S~\ref{sec:appendix-extended-optimization}}) are proposed to improve the efficiency of synthesizing LLMs and KGs.
{\bf(1)} \textit{Index-based optimization.} It aims to accelerate the process of learning embeddings and vector storage for given questions and knowledge context.
For instance, PG-RAG~\citep{liang2024empowering} proposes dynamic and adaptable knowledge retrieval indexes based on LLMs that can effectively handle complex queries and improve the overall performance of RAG systems in QA tasks.
{\bf(2)} \textit{Prompt-based optimization.} It mainly enhances the quality of the prompts based on prompt engineering.\eat{the combination of textual prompting and scripting, retrieved knowledge, and insights from the historical prompts.}
For example, KGP~\citep{wang2024knowledge} proposes a KG prompting approach to enhance the prompt for LLMs and optimize the knowledge retrieval\eat{, i.e., reduce retrieval latency and improve retrieval quality,}by introducing the KG construction module and LLM-based graph traversal agent.
%The rank-based optimization strategy selects the relevant knowledge or context from KGs to augment LLMs for QA based on the ranking of the relevance score.
{\bf(3)} \textit{Cost-based optimization.} It aims to minimize computation costs by reducing the number of calls to LLMs and accelerating knowledge retrieval.
In particular, SPOKE KG-RAG~\citep{soman2024biomedical} proposes a token-based optimized KG-RAG framework that integrates explicit and implicit knowledge from KG with LLM to enhance LLMs for cost-effective QA.
\subsubsection{Aligning with Complex QA}
The hybrid method that involves the multiple roles of KG when synthesizing LLMs and KGs can address the limitations and challenges of complex QA well.
The combination of knowledge fusion, RAG, CoT-based reasoning, and ranking-based refinement is capable of accelerating the complex question decomposition~\citep{wang2023keqing} for multi-hop QA, enhancing the context understanding for multi-run and conversation QA~\citep{roy2024learning}, facilitating the interactions across modalities for multi-modal QA~\citep{dong2024modality}, and improving the explainability of the generated answers~\citep{christmann2023explainable}.
%The low efficiency of retrieval and reasoning, as well as the high computing costs of hybrid methods, would be mitigated by the above optimizations.
However, the remaining challenge is \textit{how to achieve efficient vector indexing and search over large-scale KGs and make a balance between the cost and performance?}
\section{Evaluation and Application} \label{sec:evaluation-application}
We summarize the metrics \& dataset and showcase the related applications and demos.
\subsection{Evaluation and Benchmark Dataset}
The metrics and benchmark datasets are summarized and compared (\textbf{details in Appendix \S \ref{subsec:appendix-evaluation}}).
\spara{Metrics.} The metrics for evaluating the approach in synthesizing LLMs with KGs for QA are summarized: {\bf(1)} \textit{Answer Quality (AnsQ)}; {\bf(2)}\textit{Retrieval Quality (RetQ)}; {\bf(3)} \textit{Reasoning Quality (ReaQ)}.
\spara{Benchmark Dataset.} The recent benchmark datasets are summarized and compared (\textbf{details in Table~\ref{tab:benchmark-dataset-comparison}}) with the focus on \textit{Answer Quality}, \textit{Retrieval Quality}, and \textit{Reasoning Quality}.
\subsection{Applications}
We showcase the industrial and scientific applications and demos (\textbf{details in Appendix \S \ref{subsec:appendix-application}}) in synthesizing LLMs with KGs for QA.
\color{black}
\section{Open Challenges and Opportunities} \label{sec:open-challenges-opporitunities}
We summarize the challenges by highlighting the opportunities and discussing the future direction.
\spara{Scaling to both Effectiveness and Efficiency.}
LLM+KG systems retrieve the facts \emph{and} perform multi‐hop reasoning under tight latency and memory budgets. Three bottlenecks are emerging:
\textbf{(1) Structure‐aware retrieval}: Vanilla dense or sparse retrieval treats a KG as an unordered triples, thereby discarding topological cues that are vital for pruning the search space~\citep{tian2025systematic}. Hierarchical graph partitioning, dynamic neighbourhood expansion, and learned path‐prior proposal networks are promising ways to expose structure to the retriever while keeping the index sub‐linear.
\textbf{(2) Amortized reasoning}: Current prompting pipelines repeatedly query the KG for every beam or CoT step. Caching subgraphs, reusing intermediate embeddings, and exploiting incremental‐compute friendly hardware\eat{(e.g.\ graph‐specific GPUs)} can mitigate the quadratic blow‐up of iterative reasoning.
\textbf{(3) Lightweight answer validation}: Most guardrails rely on large LLMs, where\eat{Compact graph‐aware critics,} probabilistic logic programs, or bloom filter sketches could provide on‐device verification with \emph{O(1)} additional parameters. An opportunity is to design the retriever and validator that uncertainty estimates from the former guide selective execution of the latter.
\spara{Knowledge Alignment and Dynamic Integration.}
Once a KG snapshot is injected into an LLM, it starts to become outdated. Just like real‐world KGs usually involve adding new entities, deleting relations, and resolving contradictions. Future work should:
\textbf{(1) Quantify alignment}: We lack metrics that score not only semantic overlap but also \emph{structural} compatibility between parametric knowledge in the LLM and symbolic knowledge in the KG. Contrastive probing with synthetic counterfactuals or topology‐aware alignment losses may fill this gap.
\textbf{(2) Facilitate real‐time updates}: Parameter‐efficient tuning (e.g.\ LoRA modules keyed by graph deltas) and retrieval‐time patching (streaming KGs with temporal indices) are early steps toward \emph{stream‐time} knowledge alignment.
\textbf{(3) Detect and resolve conflicts}: Bayesian trust networks, source‐aware knowledge distillation, and multi‐agent debate protocols can estimate and reconcile confidence scores across modalities and sources. Incorporating these into the decoding objective is an open challenge with high pay‐off.
\spara{Explainable and Fairness-Aware QA.}
The scale of LLMs poses challenges to explainability and fairness in QA. While integrating KGs offers a path toward interpretable reasoning, it also introduces computational challenges and fairness concerns. Future work may consider the following directions:
\textbf{(1) Reasoning over subgraphs}: Retrieving subgraphs from large-scale KGs is computationally expensive and often results in overly complex or incomprehensible explanations. Structure-aware retrieval and reranking methods should be employed to identify subgraphs consistent with the gold paths. Furthermore, CoT-based prompting can be used to guide LLMs in generating explicit reasoning steps grounded in the retrieved subgraphs.
\textbf{(2) Fairness-aware knowledge retrieval}:
LLMs can capture social biases from training data, but KGs may contain incomplete or biased knowledge.
As a result, the fairness concerns remain in RAG~\citep{wu2024does}.
Incorporating fairness-aware techniques into KG retrieval (e.g., reranking based on bias detection) and integrating them with counterfactual prompting can mitigate bias.
\textbf{(3) Multi-turn QA}: Single-turn QA restricts the exploration of diverse perspectives and limits the exploration of reasoning processes. Developing multi-turn QA with retrieval strategies that can dynamically detect and adjust for bias and improve further the explainability and fairness through multi-turn interactions.
\section{Conclusion}
This survey has systematically examined the synthesis of LLMs and KGs in QA, presenting a novel taxonomy that categorizes methodologies based on QA types and the role of KGs.
Our analysis highlights the strengths and limitations of current approaches, emphasizing the potential opportunities of leveraging the KGs to augment LLMs to overcome challenges such as limited reasoning capabilities, outdated knowledge, and hallucinations in complex QA scenarios.
Despite significant advancements, several remaining challenges include efficient reasoning, knowledge alignment, and explainable QA.
Future research should focus on developing adaptive frameworks that dynamically integrate up-to-date knowledge with LLMs, as well as establishing efficient methods for scaling reasoning, explainability, and fairness.
\section*{Limitations}
This survey covers the taxonomy and summary of the recent advancements in synthesizing LLMs and KGs for QA and the challenges of complex QA, however, we are aware that this survey may miss some newly released works due to the rapid expansion of works on this topic.
Moreover, the survey mainly highlights the alignments between the recent methodologies of incorporating LLMs and KGs for QA and the challenges of the various complex QA tasks, while these taxonomies from different perspectives are non-exclusive, and the overlap between the two taxonomies may arise.
Furthermore, this survey underemphasizes the quantitative and experimental evaluation of different methodologies due to the various implementation details, the diversity of the benchmark datasets, and non-standardized evaluation metrics.
\bibliography{custom}
\clearpage
\appendix
\setlength{\abovecaptionskip}{8pt}
\setlength{\belowcaptionskip}{0pt}
\section{Related Survey and Paper Selection}
\subsection{Related Survey} \label{subsec:appendix-related-survey}
Previous surveys have drawn the roadmap of unifying LLMs+KGs~\citep{pan2024unifying}, have discussed the opportunities and challenges~\citep{pan2023large} of leveraging LLMs for knowledge extraction, ontologies and KGs constructions, have summarized the integration and synthesis paradigms of LLMs and KGs~\citep{kau2024combining, ibrahim2024survey}, and gave an overview of the knowledge injection methods between LLMs and domain-specific knowledge~\citep{song2025injecting}.
Additionally, the existing surveys give an overview of multilingual knowledge graph question answering ~\citep{perevalov2024multilingual}, review temporal knowledge graph QA \citep{su2024temporal}, complex QA~\citep{daull2023complex}, and discuss search engines, KGs and LLMs from the perspective of user information seeking for QA \citep{hogan2025large}.
The comparative analysis of these surveys across LLMs, KGs, LLMs+KGs, GraphRAG, and QA is given in Table~\ref{tab:survey-comparsion}.
\FloatBarrier
\begin{table*}
\footnotesize
\centering
\begin{tabular}{lccccc}
\toprule
\textbf{Survey} & \textbf{KGs} & \textbf{LLMs} & \textbf{LLMs+KGs} & \textbf{Graph RAG} & \textbf{QA} \\
\midrule
Unifying LLMs and KGs: Roadmap~\citep{pan2024unifying} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \XSolid \\
LLMs+KGs: Opportunities and Challenges~\citep{pan2023large} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \XSolid \\
Combing KGs and LLMs~\citep{kau2024combining} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \XSolid \\
mQAKG~\citep{perevalov2024multilingual} & \CheckmarkBold & \XSolid & \XSolid & \XSolid & \CheckmarkBold \\
TKGQA~\citep{su2024temporal} & \CheckmarkBold & \XSolid & \XSolid & \XSolid & \CheckmarkBold \\
LLMs, KGs, and SEs~\citep{hogan2025large} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \XSolid & \CheckmarkBold \\
Complex QA~\citep{daull2023complex} & \CheckmarkBold & \CheckmarkBold & \XSolid & \XSolid & \CheckmarkBold \\
Knowledge Injection~\citep{song2025injecting} & \CheckmarkBold & \CheckmarkBold & \XSolid & \XSolid & \CheckmarkBold \\
\midrule
{\rm[This work]} & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold \\
\bottomrule
\end{tabular}
\caption{Comparsion of Existing Survey Across LLMs, KGs, LLMs+KGs, GraphRAG, and QA. mKGQA: multi-lingual Question Answering for Knowledge Graphs, TKGQA: Temporal Knowledge Graph QA. \CheckmarkBold~Covered or Dicussed, \XSolid~Not Covered or Dicussed.}
\label{tab:survey-comparsion}
\end{table*}
We summarize that the previous surveys on synthesizing LLMs and KGs for QA show limitations in terms of the scope and tasks. {\bf(1)} The scope of the surveys\eat{~\citep{pan2024unifying, pan2023large, kau2024combining, ibrahim2024survey}} is dedicated for general knowledge-intensive tasks, such as knowledge extraction, entity linking, KG construction and completion, and text generation, etc; {\bf(2)} the QA task in the surveys\eat{~\citep{perevalov2024multilingual, su2024temporal}} is limited to the close-domain QA over the knowledge graph; {\bf(3)} the survey\eat{~\citep{hogan2025large}} on integrating LLMs, KGs, and search engines for answering user's questions is from the perspective of user-centric information needs.
\subsection{Paper Selection}
We first retrieve research papers since 2021 by using Google Scholar and PaSa\footnote{\url{https://pasa-agent.ai/}} using the phrases ``knowledge graph and language model for question answering" and ``KG and LLM for QA", and extend the search scope of the benchmark dataset paper to 2019, and then screen and select them based on their relevancy and publication venue quality.
\section{Taxonomy} \label{sec:appendix-taxonomy}
%
\subsection{Complex QA}
We divide the complex QA into the following categories according to the technical challenges.
\begin{itemize}[noitemsep]
\item \textbf{Multi-document QA.} Multi-document QA (Multi-doc QA) retrieves and synthesizes relevant information from various sources to provide a comprehensive answer.
\item \textbf{Multi-modal QA.} It refers to the QA over data and knowledge involving multiple modalities such as text, audio, images, video, etc.
\item \textbf{Muti-hop QA.} It usually involves decomposing the complex question and generating the final answers based on multi-step and iterative reasoning over a factual KG.
\item \textbf{Multi-run and conversational QA.} It involves user interactions to determine the final answer since some complex questions can not be answered in a single turn.
\item \textbf{Explainable QA.} XQA (Explainable QA) not only gives the answers to the question but also provides explanations for the given answers.
\item \textbf{Temporal QA.} It refers to the questions with temporal intent over the temporal KGs that have entities, relations, and associated temporal conditions.
\end{itemize}
\subsection{Approaches of Synthesizing LLMs and KGs for QA}
We categorize the methodology of synthesizing LLMs and KGs for QA based on the role of KGs.
\begin{itemize}[noitemsep]
\item \textbf{KG as Background Knowledge.} When KGs are used as background knowledge to enhance LLMs for QA, the questions are parsed to identify the relevant subgraphs from KGs, then they are integrated with LLMs based on knowledge integration and GraphRAG.
\item \textbf{KGs as Reasoning Guidelines.} KGs can serve as reasoning guidelines for LLMs for QA tasks by providing structured real-world facts. This factual knowledge and their reliable reasoning paths can improve the explainability of the generated answers.
\item \textbf{KGs as Refiners and Validators.} KGs can be treated as refiners and validators for LLMs in QA tasks, where LLMs can verify initial answers with factual knowledge and filter out inaccurate responses by integrating KG to ensure that the final responses are precise.
\item \textbf{Hybrid Methods.} The hybrid methods involve a KG's multiple roles, namely, background knowledge, reasoning guidelines, refiner and validator, when synthesizing LLMs and KGs for QA.
\end{itemize}
We summarize the strengths and limitations of each method of synthesizing LLMs and KGs for QA in Table~\ref{tab:taxonomy-comparsion} according to our previous taxonomy.
\FloatBarrier
\begin{table*}
\centering
\footnotesize
\resizebox{\linewidth}{!}{
\begin{tabular}{llll}
\toprule
\textbf{Approach} & \textbf{Strength} & \textbf{Limitation} & \textbf{KG Requirement} \\
\midrule
KG as Background Knowledge & Broad Coverage & Static Knowledge & High Domain Coverage \\
KG as Reasoning Guidelines & Multi-hop Capabilities & Computational Overhead & Rich Relational Paths \\
KG as Refiners and Validator & Hallucination Reduction & Validation Latency & High Accuracy \& Recency \\
Hybrid Approach & Limitation Mitigation & High Computing Cost & Dynamic Adaptation \\
\bottomrule
\end{tabular}}
\caption{Comparison of Approaches with Different Roles of KGs.}
\label{tab:taxonomy-comparsion}
\end{table*}
\section{Summary and Alignment} \label{sec:appendix-summary-and-alignment}
\subsection{Summary Tables of Approaches}
The detailed summarization and comparison of the different categories of approaches in terms of main techniques, language model (LM), knowledge graph (KG), dataset, QA types, and evaluation metrics are given in the following tables.
\spara{KG as Background Knowledge.} Table~\ref{tab:summary-comparsion-kg-as-bgk} gives a summary and comparison of approaches in the category of KG as background knowledge.
\begin{table*}[ht]
\centering
\footnotesize
\resizebox{\linewidth}{!}{
\begin{tabular}{p{4cm}p{3cm}p{3cm}p{2.5cm}p{2cm}p{2cm}p{2.5cm}}
\toprule
\textbf{Methods} & \textbf{Techniques} & \textbf{LM(s)} & \textbf{KG(s)} & \textbf{Dataset(s)} &\textbf{QA Type} & \textbf{Metric(s)} \\
\midrule
\textbf{InfuserKI}~\citep{wang2024infuserki} & Knowledge-based Fine Tuning & Llama-2-7B & UMLS, Movie KG (MetaQA) & PubMedQA, MetaQA-1HopQA & KGQA & NR, RR, F1 \\
\textbf{KnowLA}~\citep{luo2024knowla} & Knowledgeable Adaptation & Llama2-7B, Alpaca2 & WordNet, ConceptNet, Wikidata & CSQA, SIQA, BBH, WQSP, TriviaQA & MCQA, CBQA, TruthfulQA & Acc, CE Score, BLEU, ROUGE \\
\textbf{KG-Adapter}~\citep{tian2024kg} & Parameter Efficient Fine Tuning \& Joint Reasoning & Llama-2-7B-base, Zephyr-7B-alpha &ConceptNet, Freebase & OBQA, CSQA, WQSP, CWQ & KGQA, MCQA, OBQA, CWQ & Acc, Hits@1 \\
\textbf{GAIL}~\citep{zhang2024gail} & GAIL Fine-tuning & Llama-2-7B, BERTa & Freebase & WQSP, CWQ, GrailQA & KGQA & EM, F1, Hits@1 \\
\textbf{GRAG}~\citep{hu2024grag} & Textual Graph RAG & Llma-2-7B & WebQSP, ExplaGraphs & GraphQA, WQSP & KGQA & F1, Hits@1, Acc \\
\textbf{LEGO-GraphRAG}~\citep{cao2024lego} & Modular Graph RAG & Qwen2-72B, Sentence Transformer & Freebase & WQSP, CWQ, GrailQA & KBQA, CWQ & R, F1, Hits@1 \\
\textbf{KG$^2$RAG}~\citep{zhu2025knowledge} & Graph-guided Chunks Expansion & Llama-3-8B & Dataset Inherent KGs & HotpotQA & Multi-hop QA & F1, P, R \\
\textbf{KG-RAG}~\citep{xu2024retrieval} & Vector-based Subgraph Retrieval & GPT-4 & Self-constructed KGs & Curated Dataset & KGQA & MRR, Recall@K, NDCG@K, BLEU, ROUGE, METEOR \\
\bottomrule
\end{tabular}}
\caption{Summary and Comparison of Methods in the Category of KG as Background Knowledge. NR: Newly-learned Rate, RR: Remembering Rate, CE Score: Cross Entropy Score, EM: Exact Match, MRR: Mean Reciprocal Rank, BLEU: Bilingual Evaluation Understudy, ROUGE: Recall-Oriented Understudy, NDCG: Normalized Discounted Cumulative Gain, METEOR: Metric for Evaluation of Translation with Explicit Ordering.}
\label{tab:summary-comparsion-kg-as-bgk}
\end{table*}
\spara{KG as Reasoning Guidelines.} Table~\ref{tab:summary-comparsion-kg-as-rg} gives a summary and comparison of approaches in the category of KG as reasoning guidelines.
\begin{table*}[ht]
\centering
\footnotesize
\resizebox{\linewidth}{!}{
\begin{tabular}{p{4cm}p{3cm}p{3cm}p{2.5cm}p{2cm}p{2cm}p{2.5cm}}
\toprule
\textbf{Methods} & \textbf{Techniques} & \textbf{LM(s)} & \textbf{KG(s)} & \textbf{Dataset(s)} &\textbf{QA Type} & \textbf{Metric(s)} \\
\midrule
\textbf{SR}~\citep{zhang2022subgraph} & Trainable Subgraph Retriever, Fine-tuning & RoBERTa-base & Dataset Inherent KGs & WQSP, CWQ & KBQA & Hits@1, F1 \\
\textbf{KELDaR}~\citep{li2024framework} & Question Decomposition Tree, Atomic KG Retrieval & GPT-3.5-Turbo, GPT-4-Turbo & Dataset Inherent KGs & WQSP, CWQ & KGQA, Multi-hop QA & EM \\
\textbf{Oreo}~\citep{hu2022empowering} & Knowledge Interaction and Injection, KG Random Walk & RoBERTA-base, T5-base & Dataset Inherent KGs, Wikidata & NQ, WQ, WQSP, TriviaQA, CWQ, HotpotQA & CBQA, OBQA, Multi-hop QA & Acc \\
\textbf{KBIGER}~\citep{du2022knowledge} & Interative Instruction Reasoning & LSTM based Pre-trained Model & Dataset Inherent KGs, Freebase & WQSP, CWQ, GrailQA & Multi-hop KBQA & Hits@1, F1 \\
\textbf{ToG}~\citep{sun2023think} & Beam-search base Retrieval, LLM Agent & GPPT-3.5-Turbo, GPT-4, Llama-2-70B-Chat Sentence Transformer & Dataset Inherent KGs, Freebase, Wikidata & CWQ, WQSP, GrailQA, QALD10-en, WQ & KBQA, Open-Domain QA & Hits@1 \\
\textbf{ToG-2}~\citep{ma2025think} & Hibrid RAG, Knowledge-guided Context Retrieval & GPT-3.5-Turbo, GPT-4o, Llama-3-8B, Qwen2-7B & Wikipedia, Wikidata & WQSP, QALD10-en, AdvHotpotQA, HotpotQA, ToG-FinQA & Multi-hop KBQA \& Document QA, Domain-specific QA & Acc, EM, R, F1 \\
\textbf{KG-CoT}~\citep{zhao2024kg} & CoT-based KG and LLM Joint Reasoning & GPT-4, GPT-3.5-Turbo, Llama-7B, Llama-13B & Dataset Inherent KGs, Freebase & WQSP, CWQ, SQ, WQ & KBQA, Multi-hop QA & Acc, Hit@K \\
\textbf{KG-Agent}~\citep{jiang2024kg} & KG-Agent based Instruction Tuning & Dvinci-003, GPT-4, Llama-2-7B & NQ-Wiki, TQ-Wiki, WQ-Freebas & WQSP, CWQ, GrailQA & KGQA, ODQA & Hits@1, F1 \\
\textbf{ODA}~\citep{sun2024oda} & ODA-based Knowledge Graph Retrieval & GPT-4, GPT-3.5 & Wikidata & QALD10-en & KBQA & Hits@1, Acc \\
\textbf{GREASELM}~\citep{zhang2022greaselm} & Mint-based KG and LM Cross-modal Fusion and Pretraing & RoBERTA-Large, AristoRoBERTA, SapBERT, PubmedBERT & ConcepNet, UMLS, DrugBank & CQA, OBQA, MedQAt & Multiple-choice QA & Acc \\
\bottomrule
\end{tabular}}
\caption{Summary and Comparison of Methods in the Category of KG as Reasoning Guidelines. EM: Exact Match, Acc: Accuracy, R: Recall.}
\label{tab:summary-comparsion-kg-as-rg}
\end{table*}
\spara{KG as Refiners and Validators.} Table~\ref{tab:summary-comparsion-kg-as-rav} gives a summary and comparison of approaches in the category KG as refiners and validators.
\begin{table*}[ht]
\centering
\footnotesize
\resizebox{\linewidth}{!}{
\begin{tabular}{p{4cm}p{3cm}p{3cm}p{2.5cm}p{2cm}p{2cm}p{2.5cm}}
\toprule
\textbf{Methods} & \textbf{Techniques} & \textbf{LM(s)} & \textbf{KG(s)} & \textbf{Dataset(s)} &\textbf{QA Type} & \textbf{Metric(s)} \\
\midrule
\textbf{ACT-Selection}~\citep{salnikov2023answer} & ACT-based Answer Selection and Ranking & T5-Large-SSM & Wikidata & SQ, RuBQ, Mintaka & KGQA, CBQA, Multi-lingual KGQA & Hit@1 \\
\textbf{Q-KGR}~\citep{zhang2024question} & Question-guided KG Re-scoring, FNN-based Knowledge Injection & FLAN-T5-XL, RoBERTa-Large, Llama-2-7B & ConceptNet & OBQA, PIQA & KGQA & Acc \\
\textbf{KG-Rank}~\citep{yang2024kg} & Similarity and MMR based Ranking & GPT-4, Llama-2-7B, Llama-2-13B & UMLS, DBpedia & LiveQA, ExpertQA-Bio, ExpertQA-Med, MedQA & Domain-specific QA & ROUGE-L, BERTScore, MoverScore, BLEURT \\
\textbf{KGR}~\citep{guan2024mitigating} & Refine-then-Retrieve, Knowledge Truthfulness Verfication & GPT-4, Llama-2-7B, Vanilla Llama-2-7B, Transformer & CKG, PrimeKG & MedQuAD & Domain-specific QA & Truthfulness Score \\
\textbf{EFSUM}~\citep{ko2024evidence} & KG Fact Summarization, KG Helpfulness and Faithfulness Filter & GPT-3.5-Turbo, Flan-T5-XL, Llma-2-7B-Chat & Dataset Inherent KGs (Freebase, Wikidata) & WQSP, Mintaka & KGQA, Multi-hop QA & Acc \\
\textbf{InteractiveKBQA}~\citep{xiong2024interactive} & Multi-turn Interaction for Observation and Thinking & GPT-4-Turbo, Mistral-7B, Llama-2-13B & Freebase, Wikidata, Movie KG & WQSP, CWQ, KQA Pro, MetaQA & KBQA, Domain-specific QA & F1, Hits@1, EM, Acc \\
\textbf{LPKG}~\citep{wang2024learning} & Planning LLM Tuning, Inference, and Execution & GPT-3.5-Turbo, CodeQwen1.5-7B-Chat, Llama-3-8B-Instruct & Dataset Inherent KGs (Wikidata), Wikidata15K & HotpotQA, 2WikiMQA, Bamboogle, MuSiQue, CLQA-Wiki & KGQA, Multi-hop QA & EM, P, R \\
\bottomrule
\end{tabular}}
\caption{Summary and Comparison of Methods in the Category of KG as Refiners and Validators. Acc: Accuracy, EM: Exact Match, BLEURT: Bilingual Evaluation Understudy with Representations from Transformers.}
\label{tab:summary-comparsion-kg-as-rav}
\end{table*}
\spara{Hybrid Approach.} Table~\ref{tab:summary-comparsion-hybrid-approah} gives a summary and comparison of hybrid approaches.
\begin{table*}[ht]
\centering
\footnotesize
\resizebox{\linewidth}{!}{
\begin{tabular}{p{4cm}p{3cm}p{3cm}p{2.5cm}p{2cm}p{2cm}p{2.5cm}}
\toprule
\textbf{Methods} & \textbf{Techniques} & \textbf{LM(s)} & \textbf{KG(s)} & \textbf{Dataset(s)} &\textbf{QA Type} & \textbf{Metric(s)} \\
\midrule
\textbf{LongRAG}~\citep{zhao2024longrag} & Domain-specific Fine-Tuning for RAG \& CoT-guided Filter & ChatGLM3-6B, Qwen1.5-7B, Vicuna-v1.5-7B, Llama-3-8B, GPT-3.5-Turbo, GLM-4 & Wikidata & HotpotQA, 2WikiMQA, MusiQue & KBQA, Multi-hop QA & F1 \\
\textbf{SimGRAG}~\citep{cai2024simgrag} & Instruction Fine-Tuning for RAG with Filter & Llama-3-8B-Instruct, Gemma-2-27B-it & Wikipedia, PubMed & PubMedQA, BioASQm MedQA, MedMCQA, LiveQA, MedicationQA & Domain-specific QA, Multi-choice QA & Acc, Rouge-L, MAUVE, EM, F1 \\
\textbf{KGQA}~\citep{ji2024retrieval} & KG-related Instruction Tuning with CoT Reasoning & Llama-2-7B-Chat, BGE-1.5-en-base & Dataset Inherent KGs & WQAP, CWQ & KGQA & Hits@1, F1 \\
\textbf{KG-IRAG}~\citep{yang2025kg} & Incremental Retrieval and Iterative Reasoning & Llama-3-8B-Instruct, GPT-3.5-Turbo, GPT-4o-mini, GPT-4o & Self-constructed KGs & TFNSW & Temporal QA & EM, F1, HR, HAL \\
\textbf{PIP-KAG}~\citep{huang2025pip} & Parameteric Pruning for KAG & Llma-3-8B-Instruct & Dataset Inherent KGs & CoConflictQA & KGQA, Multi-hop QA & EM, ConR, MR \\
\textbf{RAG-KG-IL}~\citep{yu2025rag} & Agent-based Incremental Learning and Knowledge Dynamic Update & GPT-4o & Self-constructed KGs & Curated Dataset from NHS& Domain-specific QA & AM, HVC \\
\textbf{CoT-RAG}~\citep{li2025cot} & KG-driven CoT Generation and Knowledge-aware RAG with Pseudo-program KGs & ERNIE-Speed-128K, GPT-4o-mini & Self-curated Pseudo-Program KGs & HotpotQA, CSQA, SIQA & KGQA, Multi-hop QA & RA, Robustness \\
\bottomrule
\end{tabular}}
\caption{Summary and Comparison of Methods in the Category of Hybrid Approach. EM: Exact Match, HR: Hit Rate, HAL: Hallucination, ConR: Context Recall, MR: Memorization Ratio, AM: Accuracy Matching, HVC: Human-verified Completeness, RA: Reasoning Accuracy.}
\label{tab:summary-comparsion-hybrid-approah}
\end{table*}
\subsection{Alignment of Approaches to Complex QA}
The alignment of the existing approaches in synthesizing LLMs and KGs to diverse complex QA is presented in Table~\ref{tab:alignment-approaches}.
\begin{table*}[ht]
\footnotesize
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{lcccccc}
\toprule
\textbf{Approach} & \textbf{Multi-doc QA} & \textbf{Multi-modal QA} & \textbf{Multi-hop QA} & \textbf{Multi-run QA} & \textbf{XQA} & \textbf{Temporal QA} \\
\midrule
KG as Background Knowledge & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & $\triangle$ & \CheckmarkBold \\
KG as Reasoning Guidelines & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \XSolid & \CheckmarkBold & \CheckmarkBold \\
KG as Refiners and Validator & $\triangle$ & \XSolid & $\triangle$ & \CheckmarkBold & \XSolid & $\triangle$ \\
Hybrid Methods & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold \\
\bottomrule
\end{tabular}}
\caption{Research Progress on Alignment of Existing Approaches of synthesizing LLMs and KGs with Complex QA. \CheckmarkBold~Fully investigated. $\triangle$~Partially investigated. \XSolid~Not yet investigated.}
\label{tab:alignment-approaches}
\end{table*}
\section{Extended Optimization} \label{sec:appendix-extended-optimization}
%The main issues behind these complex QA are the low efficiency and high computing costs, since the explainable QA usually involves large-scale graph reasoning, and massive multi-modal data from heterogeneous sources need to be processed to support visual and conversational QA.
The extended optimization techniques are summarized and discussed as follows.
\spara{Index-based Optimization.} It is a time-consuming and complex task to create indexes from long-range facts and retrieve the relevant knowledge from large-scale graphs.
To address this issue, GoR~\citep{zhang2024graph} leverages GNN and BERT score-based objectives to optimize node embeddings during graph indexing.
KG-Retriever~\citep{chen2024kg} leverages a hierarchical index graph to enhance knowledge correlations and improve information retrieval for efficient knowledge indexing.
NodeRAG~\citep{xu2025noderag} integrates the heterogeneous graphs and fine-grained retrieval with RAG by optimizing the indexing of graph structures.
DRO~\cite{shi2025direct} proposes a direct retrieval-agumentation method to directly estimate the distribution of document permutations from the select model for jointly learning the selection model and generative LM model.
\spara{Prompt-based Optimization.} To facilitate the deep fusion between the retrieved knowledge from KGs with the internal knowledge of LLMs, several prompt-based optimization approaches are proposed.
For instance, StraGo~\citep{wu2024strago} enhances the quality and stability of the prompts based on the insights and strategic guidance learned from the historical prompts by using in-context learning.
Meanwhile, several ranking strategies have been exploited to retrieve the top relevant knowledge from diverse knowledge bases, and the most relevant contexts are further fed to LLMs together with the prompts.
For example, KG-Rank~\citep{yang2024kg} leverages re-ranking techniques based on the score measuring relevance and redundancy to rank the top-$k$ triples from KGs and then combine them with the prompt to generate the answers for QA.
Similarly, KS-LLM~\citep{zheng2024ks} introduces the evidence sentence selection module to retrieve the most relevant evidence sentences based on the ranking of the Euclidean distance between the triples and each evidence sentence.
Instead of direct ranking of context-based relevance, BriefContext~\citep{zhang2024mapreduce} introduces the preflight check to predict the relevance ranking between the user query and the retrieved documents, which is divided into multiple chunks for multiple RAG subtasks based on the map-reduce strategy.
To leverage the multi-source knowledge for RAG-based QA, QUASAR~\citep{christmann2024rag} enhances RAG-based QA by effectively integrating unstructured text, structured tables, and knowledge graphs for evidence retrieval and re-ranking and filtering the relevant evidence from the retrieved evidence.
\spara{Cost-based Optimization.} To implement efficient and effective knowledge probing of LLMs, GLens~\citep{zheng2024kglens} initially leverages the Thompson sampling strategy to measure the alignment between KGs and LLMs for addressing the knowledge blind spots of LLMs, and then designs a graph-guided question generator to convert KGs to text and designs a sampling strategy on the parameterized KG structure for accelerating KG traversal.
Similarly, Coke~\citep{dong2024cost} minimizes the calls of LLMs for KGQA by introducing a cluster-level Thompson sampling to formulate the accuracy expectation and an optimized context-aware policy to distinguish the expert model based on question semantics.
LMQL~\citep{beurer2023prompting} minimizes the call of LLMs by generating an efficient inference procedure based on the LMP (Language Model Programming) constraints and control flow.
It differs from the above work that CGPE~\citep{tao2024cluegoptimizing} optimizes the knowledge retrieval based on clue-guided path exploration and information matching from knowledge bases to enhance the capabilities of LLMs for unfamiliar questions and reduce the costs of LLMs.
To summarize, even if several optimizations and ranking strategies have been recently investigated to reduce the costs of graph retrieval, graph reasoning, and the length of the context of LLMs, however, the relevant subgraphs extraction, graph reasoning, and vector-based retrieval remain a computationally costly task.
\section{Evaluations and Applications}
\label{sec:appendix-evaluation-applications}
The details of evaluation metrics, benchmark datasets, industrial and scientific applications, and demos of synthesizing LLMs and KGs for QA are summarized and compared as follows.
\subsection{Evaluations} \label{subsec:appendix-evaluation}
\spara{Metrics.} The evaluation metrics in synthesizing LLMs with KGs for QA are: {\bf(1)} \textit{the metrics measuring the answer quality}, BERTScore~\citep{peng2024graph}, answer relevance (AR), hallucination (HAL)~\cite{yang2025kg}, accuracy matching, human-verficed completeness~\citep{yu2025rag}; {\bf(2)}
\textit{the metrics measuring the retrieval quality of RAG}, context relevance \citep{es2024ragas}, faithfulness score (FS)~\citep{yang2024kg}, precision, context recall~\citep{yu2024evaluation, huang2025pip}, mean reciprocal rank (MRR)~ \citep{xu2024retrieval}, normalized discounted cumulative gain (NDCG)~\citep{xu2024retrieval}; {\bf(3)} \textit{the metrics measuring the reasoning quality for multi-hop QA}, Hop-Acc~\citep{gu2024pokemqa}, reasoning accuracy (RA)~\citep{li2025cot}.
\spara{Benchmark Dataset.} The recent benchmark datasets are summarized and compared in Table~\ref{tab:benchmark-dataset-comparison} with the focus on the following evaluations.
{\bf(1)} \textit{Answer Quality (AnsQ)}: the correctness of the generated answer with supported evidence and retrieved context in comparison to the ground-truth answer; {\bf(2)} \textit{Retrieval Quality (RetQ)}: the relevance of the provided or retrieved context in comparison to the human-validated relevant context; {\bf(3)} \textit{Reasoning Quality (ReaQ)}: the correctness of generated reasoning chains and intermediate steps that explain how the final answer is derived.
\begin{table*}[ht]
\centering
\footnotesize
\resizebox{\linewidth}{!}{
\begin{tabular}{p{5cm}llllp{5cm}}
\toprule
\textbf{Dataset} & \textbf{Category} & \textbf{AnsQ} & \textbf{RetQ} & \textbf{ReaQ} & \textbf{Brief Description} \\
\midrule
\href{https://www.microsoft.com/en-us/download/details.aspx?id=52763}{WebQSP}~\citep{yih2016value} & KBQA & \CheckmarkBold & \CheckmarkBold & $\triangle$ & Contains SPARQL queries for knowledge-based QA. \\
\href{https://zenodo.org/records/7655130}{BioASQ-QA}~\citep{krithara2023bioasq} & KBQA & \CheckmarkBold & \CheckmarkBold & \XSolid & Includes exact and ideal answers of question over multi-doc summarization. \\
\href{https://github.com/aannonymouuss/CAQA-Benchmark}{CAQA}~\citep{hu2024benchmarking} & KBQA & $\triangle$ & \CheckmarkBold & $\triangle$ & Evaluates complex reasoning and attribution. \\
\href{https://github.com/D3Mlab/cr-lt-kgqa}{CR-LT KGQA}~\citep{guo2024cr} & KGQA & \CheckmarkBold & $\triangle$ & \CheckmarkBold & Long-tail entities and commonsense reasoning. \\
\href{https://exaqt.mpi-inf.mpg.de/}{EXAQT}~\citep{jia2021complex} & KGQA & \CheckmarkBold & \CheckmarkBold & $\triangle$ & Supports temporal QA with multiple entities, predicates, and conditions.\\
\href{https://github.com/jonathanherzig/commonsenseqa}{CommonsenseQA}~\citep{talmor2019commonsenseqa} & Multi-choice QA & \CheckmarkBold & $\triangle$ & \XSolid & Contains questions with the correct answer and four distractor answers. \\
\href{https://github.com/jind11/MedQA}{MedQA}~\citep{jin2021disease} & Multi-choice QA & \CheckmarkBold & $\triangle$ & $\triangle$ & Medical multi-choice QA dataset with multilingual medical examination text \\
\href{https://github.com/probe2/multi-hop/}{MINTQA}~\citep{he2024mintqa} & Multi-hop QA &\CheckmarkBold & \CheckmarkBold & \CheckmarkBold & Supports LLMs evaluation on new and tail knowledge. \\
\href{https://github.com/zhudotexe/fanoutqa}{FanOutQA}~\citep{zhu2024fanoutqa} & Multi-Hop QA & $\triangle$ & $\triangle$ & \CheckmarkBold & Contains multi-hop question over multi-document.\\
\href{https://github.com/yale-nlp/M3SciQA}{M3SciQA}~\citep{li2024m3sciqa} & Multi-modal QA & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & QA over the context across multiple document. \\
\href{https://github.com/linyongnan/OMG-QA}{OMG-QA}~\citep{nan2024omg} & Multi-modal QA &\CheckmarkBold &\CheckmarkBold & $\triangle$ & Evaluates the retrieval and reasoning of QA across modalities. \\
\href{https://scienceqa.github.io/}{ScienceQA}~\citep{lu2022learn} & Multi-modal QA & \CheckmarkBold & \XSolid& \CheckmarkBold & Supports multi-choice questions across disciplines in science topics.\\
\href{https://github.com/temptabqa/temptabqa}{TempTabQA}~\citep{gupta2023temptabqa} & Temporal QA & \CheckmarkBold & $\triangle$ &\CheckmarkBold & Supports temporal reasoning over semi-structured tables.\\
\href{https://github.com/jannatmeem95/PAT-Questions}{PATQA}~\citep{meem2024pat} & Temporal QA & \CheckmarkBold & $\triangle$ & \CheckmarkBold & Present-anchored temporal QA \\
\href{https://github.com/weiyifan1023/MenatQA}{MenatQA}~\citep{wei2023menatqa} & Temporal QA & $\triangle$ & $\triangle$ & \CheckmarkBold & Evaluates the temporal reasoning capability of LLMs \\
\href{https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/Loong}{Loong}~\citep{wang2024leave} & Long-context QA & \CheckmarkBold & \XSolid & $\triangle$ & QA over multi-doc with a relevant document for the final answer. \\
\href{https://github.com/mikejqzhang/SituatedQA}{SituatedQA}~\citep{zhang2021situatedqa} & Open-retrieval QA & \CheckmarkBold & \CheckmarkBold & \XSolid & Includes correct question with temporal or geographical context.\\
\href{https://github.com/datadotworld/cwd-benchmark-data}{ChatData}~\citep{sequeda2024benchmark} & LLM-KG QA & \CheckmarkBold & \CheckmarkBold &\XSolid & LLMs+KGs QA over the enterprise SQL database. \\
\href{https://github.com/AKSW/LLM-KG-Bench}{LLM-KG-Bench}~\citep{meyer2023developing} & LLM-KG QA & \CheckmarkBold & \XSolid & \XSolid & LLMs in knowledge graph engineering. \\
\href{https://github.com/chen-zichen/XplainLLM_dataset}{XplainLLM}~\citep{chen2024xplain} & LLM-KG QA & \CheckmarkBold & $\triangle$ & \CheckmarkBold & Focuses on QA explainability and reasoning \\
\href{https://anonymous.4open.science/r/OKGQA-CBB0}{OKGQA}~\citep{sui2024can} & LLM-KG QA & \CheckmarkBold & \CheckmarkBold & \CheckmarkBold & Evaluates LLMs+KGs for open-ended QA. \\
\href{https://github.com/HKUDS/MiniRAG/tree/main/dataset/LiHua-World}{LiHua-World}~\citep{fan2025minirag} & LLM-KG QA &\CheckmarkBold &\CheckmarkBold &\CheckmarkBold & Evaluates capability of LLMs on multi-hop QA in the scenario of RAG. \\
\href{https://stark.stanford.edu/}{STaRK}~\citep{wu2024stark} & LLM-KG QA & \CheckmarkBold &\CheckmarkBold & \CheckmarkBold & Evaluates the performance of LLMs-driven RAG for QA.\\
\href{https://github.com/OpenBMB/PIP-KAG}{CoConflictQA}~\citep{huang2025pip} & LLM-KG QA &\CheckmarkBold &$\triangle$ &\CheckmarkBold & Evaluates contextual faithfulness for QA in the scenario of KAG. \\
\href{https://huggingface.co/datasets/Askio/mmrag_benchmark}{mmRAG}~\citep{xu2025mmrag} & LLM-KG QA &\CheckmarkBold & \CheckmarkBold &\CheckmarkBold & Evaluates multi-modal RAG including QA dataset across text, tables, and KGs. \\
\bottomrule
\end{tabular}}
\caption{Comparison of LLMs+KGs for QA Benchmark Dataset. \CheckmarkBold~The dataset was primarily designed to support this evaluation. $\triangle$~The dataset can be adapted for this evaluation, but this evaluation is not their main focus. \XSolid~The dataset does not support this evaluation.}
\label{tab:benchmark-dataset-comparison}
\end{table*}
\subsection{Applications} \label{subsec:appendix-application}
We showcase the applications and demos in synthesizing LLMs with KGs for QA.
\spara{KAG}~\citep{liang2024kag} (by Antgroup)\footnote{\url{https://github.com/OpenSPG/KAG}} is a domain-knowledge augmented generation framework that leverages KGs and vector retrieval to bi-directionally enhance LLMs for knowledge-intensive tasks, such as QA.
\spara{PIKE-RAG}~\citep{wang2025pike} (by Microsoft)\footnote{\url{https://github.com/microsoft/PIKE-RAG}} is a specialized knowledge and rationale augmented generation system with a focus on extracting, understanding, and applying domain-specific knowledge to guide LLMs toward accurate responses.
\spara{GraphRAG-QA} (by NebulaGraph)\footnote{\url{https://github.com/wey-gu/demo-kg-build}} is an industrial demo of GraphRAG integrating several query engines for augmenting QA, NLP2Cypher-based KG query engine~\cite{wu2022nebula}, vector RAG query engine, and Graph vector RAG query engine.
\spara{MedRAG}~\citep{zhao2025medrag} (by Nanyang Technological University et al.)\footnote{\url{https://github.com/SNOWTEAM2023/MedRAG}} is a KG-elicited reasoning enhanced RAG-based healthcare copilot that generates diagnoses and treatment recommendations based on the input patient manifestations.
\spara{Fact Finder}~\cite{steinigen2024fact} (by Fraunhofer IAIS and Bayer)\footnote{\url{https://github.com/chrschy/fact-finder}} is a QA system that augments LLMs with the query-based KG retrieval from medical KG to improve the correctness and completeness of the answers.
\spara{AprèsCoT}~\citep{hirdel2025aprescot} (by University of Waterloo)\footnote{\url{http://lg-research-2.uwaterloo.ca:8050/aprescot}} is a visualization tool for understanding and explaining the answers generated by LLMs with CoT-based prompting and visualized path exploration and inference over KGs.
\end{document}
\begin{thebibliography}{126}
\providecommand{\natexlab}[1]{#1}
\bibitem[{Alawwad et~al.(2025)Alawwad, Alhothali, Naseem, Alkhathlan, and Jamal}]{alawwad2024enhancing}
Hessa~Abdulrahman Alawwad, Areej Alhothali, Usman Naseem, Ali Alkhathlan, and Amani Jamal. 2025.
\newblock Enhancing textbook question answering task with large language models and retrieval augmented generation.
\newblock \emph{Pattern Recognit.}, 162:111332.
\bibitem[{Bai et~al.(2025)Bai, Grant, and Wang}]{bai2025ramqa}
Yang Bai, Christan~Earl Grant, and Daisy~Zhe Wang. 2025.
\newblock {RAMQA}: A unified framework for retrieval-augmented multi-modal question answering.
\newblock \emph{arXiv:2501.13297}.
\bibitem[{Beurer-Kellner et~al.(2023)Beurer-Kellner, Fischer, and Vechev}]{beurer2023prompting}
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2023.
\newblock Prompting is programming: A query language for large language models.
\newblock \emph{Proc. ACM Program. Lang.}, 7:1946--1969.
\bibitem[{Cai et~al.(2024)Cai, Guo, Pei, Bian, and Zheng}]{cai2024simgrag}
Yuzheng Cai, Zhenyue Guo, Yiwen Pei, Wanrui Bian, and Weiguo Zheng. 2024.
\newblock {SimGRAG}: Leveraging similar subgraphs for knowledge graphs driven retrieval-augmented generation.
\newblock \emph{arXiv preprint arXiv:2412.15272}.
\bibitem[{Cao et~al.(2024)Cao, Gao, Li, Xie, and Zhou}]{cao2024lego}
Yukun Cao, Zengyi Gao, Zhiyang Li, Xike Xie, and S~Kevin Zhou. 2024.
\newblock {LEGO-GraphRAG}: Modularizing graph-based retrieval-augmented generation for design space exploration.
\newblock \emph{arXiv:2411.05844}.
\bibitem[{Chen et~al.(2024{\natexlab{a}})Chen, Jiang, Qin, Rawal, Tan, Choi, Xiong, and Ai}]{chen2024llm}
Ruirui Chen, Weifeng Jiang, Chengwei Qin, Ishaan Rawal, Cheston Tan, Dongkyu Choi, Bo~Xiong, and Bo~Ai. 2024{\natexlab{a}}.
\newblock {LLM}-based multi-hop question answering with knowledge graph integration in evolving environments.
\newblock In \emph{EMNLP}, pages 14438--14451.
\bibitem[{Chen et~al.(2024{\natexlab{b}})Chen, Bai, Su, Luan, Liu, and Shi}]{chen2024kg}
Weijie Chen, Ting Bai, Jinbo Su, Jian Luan, Wei Liu, and Chuan Shi. 2024{\natexlab{b}}.
\newblock {KG-Retriever}: Efficient knowledge indexing for retrieval-augmented large language models.
\newblock \emph{arXiv:2412.05547}.
\bibitem[{Chen et~al.(2024{\natexlab{c}})Chen, Chen, Singh, and Sra}]{chen2024xplain}
Zichen Chen, Jianda Chen, Ambuj Singh, and Misha Sra. 2024{\natexlab{c}}.
\newblock Xplain{LLM}: A knowledge-augmented dataset for reliable grounded explanations in {LLMs}.
\newblock In \emph{EMNLP}, pages 7578--7596.
\bibitem[{Christmann et~al.(2023)Christmann, Roy, and Weikum}]{christmann2023explainable}
Philipp Christmann, Rishiraj~Saha Roy, and Gerhard Weikum. 2023.
\newblock Explainable conversational question answering over heterogeneous sources via iterative graph neural networks.
\newblock In \emph{SIGIR}, pages 643--653.
\bibitem[{Christmann and Weikum(2024)}]{christmann2024rag}
Philipp Christmann and Gerhard Weikum. 2024.
\newblock {RAG}-based question answering over heterogeneous data and text.
\newblock \emph{arXiv:2412.07420}.
\bibitem[{Daull et~al.(2023)Daull, Bellot, Bruno, Martin, and Murisasco}]{daull2023complex}
Xavier Daull, Patrice Bellot, Emmanuel Bruno, Vincent Martin, and Elisabeth Murisasco. 2023.
\newblock Complex {QA} and language models hybrid architectures, survey.
\newblock \emph{arXiv:2302.09051}.
\bibitem[{Dong et~al.(2024{\natexlab{a}})Dong, Zhang, Zhou, Chen, Zha, and Huang}]{dong2024cost}
Junnan Dong, Qinggang Zhang, Chuang Zhou, Hao Chen, Daochen Zha, and Xiao Huang. 2024{\natexlab{a}}.
\newblock Cost-efficient knowledge-based question answering with large language models.
\newblock In \emph{NeurIPS}.
\bibitem[{Dong et~al.(2024{\natexlab{b}})Dong, Zhang, Zhou, Zha, Zheng, and Huang}]{dong2024modality}
Junnan Dong, Qinggang Zhang, Huachi Zhou, Daochen Zha, Pai Zheng, and Xiao Huang. 2024{\natexlab{b}}.
\newblock Modality-aware integration with large language models for knowledge-based visual question answering.
\newblock In \emph{ACL}, pages 2417--2429.
\bibitem[{Du et~al.(2022)Du, Huang, Zhang, and Zhao}]{du2022knowledge}
Haowei Du, Quzhe Huang, Chen Zhang, and Dongyan Zhao. 2022.
\newblock Knowledge-enhanced iterative instruction generation and reasoning for knowledge base question answering.
\newblock In \emph{NLPCC}, pages 431--444.
\bibitem[{Es et~al.(2024)Es, James, Anke, and Schockaert}]{es2024ragas}
Shahul Es, Jithin James, Luis~Espinosa Anke, and Steven Schockaert. 2024.
\newblock {RAGAs}: Automated evaluation of retrieval augmented generation.
\newblock In \emph{EACL}, pages 150--158.
\bibitem[{Fan et~al.(2025)Fan, Wang, Ren, and Huang}]{fan2025minirag}
Tianyu Fan, Jingyuan Wang, Xubin Ren, and Chao Huang. 2025.
\newblock {MiniRAG}: Towards extremely simple retrieval-augmented generation.
\newblock \emph{arXiv:2501.06713}.
\bibitem[{Feng et~al.(2025)Feng, Wang, Qiu, and Liu}]{feng2025retrieval}
Jianzhou Feng, Qin Wang, Huaxiao Qiu, and Lirong Liu. 2025.
\newblock Retrieval in decoder benefits generative models for explainable complex question answering.
\newblock \emph{Neural Networks}, 181:106833.
\bibitem[{Gao et~al.(2024)Gao, Qiao, Kan, Wen, He, and Li}]{gao2024two}
Yifu Gao, Linbo Qiao, Zhigang Kan, Zhihua Wen, Yongquan He, and Dongsheng Li. 2024.
\newblock Two-stage generative question answering on temporal knowledge graph using large language models.
\newblock In \emph{ACL Findings}, pages 6719--6734.
\bibitem[{Gu et~al.(2024)Gu, Zhou, Han, Liu, Wang, and Wang}]{gu2024pokemqa}
Hengrui Gu, Kaixiong Zhou, Xiaotian Han, Ninghao Liu, Ruobing Wang, and Xin Wang. 2024.
\newblock {PokeMQA}: Programmable knowledge editing for multi-hop question answering.
\newblock In \emph{ACL}, pages 8069--8083.
\bibitem[{Guan et~al.(2024)Guan, Liu, Lin, Lu, He, Han, and Sun}]{guan2024mitigating}
Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, and Le~Sun. 2024.
\newblock Mitigating large language model hallucinations via autonomous knowledge graph-based retrofitting.
\newblock In \emph{AAAI}, pages 18126--18134.
\bibitem[{Guo et~al.(2024)Guo, Toroghi, and Sanner}]{guo2024cr}
Willis Guo, Armin Toroghi, and Scott Sanner. 2024.
\newblock {CR-LT-KGQA}: A knowledge graph question answering dataset requiring commonsense reasoning and long-tail knowledge.
\newblock \emph{arXiv:2403.01395}.
\bibitem[{Gupta et~al.(2023)Gupta, Kandoi, Vora, Zhang, He, Reinanda, and Srikumar}]{gupta2023temptabqa}
Vivek Gupta, Pranshu Kandoi, Mahek Vora, Shuo Zhang, Yujie He, Ridho Reinanda, and Vivek Srikumar. 2023.
\newblock {T}emp{T}ab{QA}: Temporal question answering for semi-structured tables.
\newblock In \emph{EMNLP}, pages 2431--2453.
\bibitem[{Han et~al.(2024)Han, Wang, Shomer, Guo, Ding, Lei, Halappanavar, Rossi, Mukherjee, Tang et~al.}]{han2024retrieval}
Haoyu Han, Yu~Wang, Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halappanavar, Ryan~A Rossi, Subhabrata Mukherjee, Xianfeng Tang, and 1 others. 2024.
\newblock Retrieval-augmented generation with graphs ({GraphRAG}).
\newblock \emph{arXiv:2501.00309}.
\bibitem[{He et~al.(2024)He, Hu, Long, Chen, and Pan}]{he2024mintqa}
Jie He, Nan Hu, Wanqiu Long, Jiaoyan Chen, and Jeff~Z Pan. 2024.
\newblock {MINTQA}: A multi-hop question answering benchmark for evaluating {LLMs} on new and tail knowledge.
\newblock \emph{arXiv:2412.17032}.
\bibitem[{Hogan et~al.(2025)Hogan, Luna~Dong, Vrande{\v{c}}i{\'c}, and Weikum}]{hogan2025large}
Aidan Hogan, Xin Luna~Dong, Denny Vrande{\v{c}}i{\'c}, and Gerhard Weikum. 2025.
\newblock Large language models, knowledge graphs and search engines: A crossroads for answering users' questions.
\newblock \emph{arXiv:2501.06699}.
\bibitem[{Hu et~al.(2024{\natexlab{a}})Hu, Chen, Wu, Qi, Bi, Wu, and Pan}]{hu2024benchmarking}
Nan Hu, Jiaoyan Chen, Yike Wu, Guilin Qi, Sheng Bi, Tongtong Wu, and Jeff~Z Pan. 2024{\natexlab{a}}.
\newblock Benchmarking large language models in complex question answering attribution using knowledge graphs.
\newblock \emph{arXiv:2401.14640}.
\bibitem[{Hu et~al.(2024{\natexlab{b}})Hu, Lei, Zhang, Pan, Ling, and Zhao}]{hu2024grag}
Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo~Pan, Chen Ling, and Liang Zhao. 2024{\natexlab{b}}.
\newblock {GRAG}: Graph retrieval-augmented generation.
\newblock \emph{arXiv:2405.16506}.
\bibitem[{Hu et~al.(2022)Hu, Xu, Yu, Wang, Yang, Zhu, Chang, and Sun}]{hu2022empowering}
Ziniu Hu, Yichong Xu, Wenhao Yu, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Kai-Wei Chang, and Yizhou Sun. 2022.
\newblock Empowering language models with knowledge graph reasoning for question answering.
\newblock In \emph{EMNLP}, pages 9562--9581.
\bibitem[{Huang et~al.(2025)Huang, Liu, Yan, Yi, Chen, Liu, Sun, Xiao, Yu, and Xiong}]{huang2025pip}
Pengcheng Huang, Zhenghao Liu, Yukun Yan, Xiaoyuan Yi, Hao Chen, Zhiyuan Liu, Maosong Sun, Tong Xiao, Ge~Yu, and Chenyan Xiong. 2025.
\newblock {PIP-KAG}: Mitigating knowledge conflicts in knowledge-augmented generation via parametric pruning.
\newblock \emph{arXiv preprint arXiv:2502.15543}.
\bibitem[{Huang(2023)}]{huang2023llm}
Yuxuan Huang. 2023.
\newblock Evaluating and enhancing large language models for conversational reasoning on knowledge graphs.
\newblock \emph{arXiv:2312.11282}.
\bibitem[{Ibrahim et~al.(2024)Ibrahim, Aboulela, Ibrahim, and Kashef}]{ibrahim2024survey}
Nourhan Ibrahim, Samar Aboulela, Ahmed Ibrahim, and Rasha Kashef. 2024.
\newblock A survey on augmenting knowledge graphs {(KGs)} with large language models {(LLMs)}: models, evaluation metrics, benchmarks, and challenges.
\newblock \emph{Discov. Artif. Intell.}, 4(1):76.
\bibitem[{Jain and Lapata(2024)}]{jain2024integrating}
Parag Jain and Mirella Lapata. 2024.
\newblock Integrating large language models with graph-based reasoning for conversational question answering.
\newblock \emph{arXiv:2407.09506}.
\bibitem[{Ji et~al.(2024)Ji, Wu, Li, Chen, Zhong, Jia, and Zhang}]{ji2024retrieval}
Yixin Ji, Kaixin Wu, Juntao Li, Wei Chen, Mingjie Zhong, Xu~Jia, and Min Zhang. 2024.
\newblock Retrieval and reasoning on {KGs}: Integrate knowledge graphs into large language models for complex question answering.
\newblock In \emph{EMNLP}, pages 7598--7610.
\bibitem[{Jia et~al.(2021)Jia, Pramanik, Saha~Roy, and Weikum}]{jia2021complex}
Zhen Jia, Soumajit Pramanik, Rishiraj Saha~Roy, and Gerhard Weikum. 2021.
\newblock Complex temporal question answering on knowledge graphs.
\newblock In \emph{CIKM}, pages 792--802.
\bibitem[{Jiang et~al.(2024)Jiang, Zhou, Zhao, Song, Zhu, Zhu, and Wen}]{jiang2024kg}
Jinhao Jiang, Kun Zhou, Wayne~Xin Zhao, Yang Song, Chen Zhu, Hengshu Zhu, and Ji-Rong Wen. 2024.
\newblock {Kg-Agent}: An efficient autonomous agent framework for complex reasoning over knowledge graph.
\newblock \emph{arXiv:2402.11163}.
\bibitem[{Jin et~al.(2021)Jin, Pan, Oufattole, Weng, Fang, and Szolovits}]{jin2021disease}
Di~Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021.
\newblock What disease does this patient have? a large-scale open domain question answering dataset from medical exams.
\newblock \emph{Appl. Sci.}, 11(14):6421.
\bibitem[{Kau et~al.(2024)Kau, He, Nambissan, Astudillo, Yin, and Aryani}]{kau2024combining}
Amanda Kau, Xuzeng He, Aishwarya Nambissan, Aland Astudillo, Hui Yin, and Amir Aryani. 2024.
\newblock Combining knowledge graphs and large language models.
\newblock \emph{arXiv:2407.06564}.
\bibitem[{Ko et~al.(2024)Ko, Cho, Chae, Yeo, and Lee}]{ko2024evidence}
Sungho Ko, Hyunjin Cho, Hyungjoo Chae, Jinyoung Yeo, and Dongha Lee. 2024.
\newblock Evidence-focused fact summarization for knowledge-augmented zero-shot question answering.
\newblock In \emph{EMNLP}, pages 10636--10651.
\bibitem[{Krithara et~al.(2023)Krithara, Nentidis, Bougiatiotis, and Paliouras}]{krithara2023bioasq}
Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023.
\newblock {BioASQ-QA}: A manually curated corpus for biomedical question answering.
\newblock \emph{Scientific Data}, 10(1):170.
\bibitem[{Li et~al.(2024{\natexlab{a}})Li, Shangguan, Zhao, Li, Liu, and Cohan}]{li2024m3sciqa}
Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, and Arman Cohan. 2024{\natexlab{a}}.
\newblock {M3SciQA}: A multi-modal multi-document scientific {QA} benchmark for evaluating foundation models.
\newblock In \emph{EMNLP}, pages 15419--15446.
\bibitem[{Li et~al.(2025{\natexlab{a}})Li, Fang, Shi, Khan, Wang, Feng, Wang, Zhang, and Cui}]{li2025cot}
Feiyang Li, Peng Fang, Zhan Shi, Arijit Khan, Fang Wang, Dan Feng, Weihao Wang, Xin Zhang, and Yongjian Cui. 2025{\natexlab{a}}.
\newblock {CoT-RAG}: Integrating chain of thought and retrieval-augmented generation to enhance reasoning in large language models.
\newblock \emph{arXiv preprint arXiv:2504.13534}.
\bibitem[{Li et~al.(2024{\natexlab{b}})Li, Song, Zhou, Tian, Wang, Yang, and Zhang}]{li2024framework}
Yading Li, Dandan Song, Changzhi Zhou, Yuhang Tian, Hao Wang, Ziyi Yang, and Shuhao Zhang. 2024{\natexlab{b}}.
\newblock A framework of knowledge graph-enhanced large language model based on question decomposition and atomic retrieval.
\newblock In \emph{EMNLP Findings}, pages 11472--11485.
\bibitem[{Li et~al.(2025{\natexlab{b}})Li, Guo, Shao, Song, Bian, Zhang, and Wang}]{li2025graph}
Zijian Li, Qingyan Guo, Jiawei Shao, Lei Song, Jiang Bian, Jun Zhang, and Rui Wang. 2025{\natexlab{b}}.
\newblock Graph neural network enhanced retrieval for question answering of large language models.
\newblock In \emph{NAACL}, pages 6612--6633.
\bibitem[{Liang et~al.(2025)Liang, Meng, Zhang, Wang, Wei, and Zhao}]{liang2025reasvqa}
Jianxin Liang, Xiaojun Meng, Huishuai Zhang, Yueqian Wang, Jiansheng Wei, and Dongyan Zhao. 2025.
\newblock {ReasVQA}: Advancing {VideoQA} with imperfect reasoning process.
\newblock \emph{arXiv:2501.13536}.
\bibitem[{Liang et~al.(2024{\natexlab{a}})Liang, Sun, Gui, Zhu, Jiang, Zhong, Qu, Zhao, Bo, Yang et~al.}]{liang2024kag}
Lei Liang, Mengshu Sun, Zhengke Gui, Zhongshu Zhu, Zhouyu Jiang, Ling Zhong, Yuan Qu, Peilong Zhao, Zhongpu Bo, Jin Yang, and 1 others. 2024{\natexlab{a}}.
\newblock {KAG}: Boosting {LLMs} in professional domains via knowledge augmented generation.
\newblock \emph{arXiv preprint arXiv:2409.13731}.
\bibitem[{Liang et~al.(2024{\natexlab{b}})Liang, Niu, Zhang, Song, Wang, Yang, Xiong, Tang, and Xi}]{liang2024empowering}
Xun Liang, Simin Niu, Sensen Zhang, Shichao Song, Hanyu Wang, Jiawei Yang, Feiyu Xiong, Bo~Tang, and Chenyang Xi. 2024{\natexlab{b}}.
\newblock Empowering large language models to set up a knowledge retrieval indexer via self-learning.
\newblock \emph{arXiv:2405.16933}.
\bibitem[{Linders and Tomczak(2025)}]{linders2025knowledge}
Jasper Linders and Jakub~M Tomczak. 2025.
\newblock Knowledge graph-extended retrieval augmented generation for question answering.
\newblock \emph{arXiv preprint arXiv:2504.08893}.
\bibitem[{Liu et~al.(2024{\natexlab{a}})Liu, Zhang, Li, and Yao}]{liu2024explore}
Guangyi Liu, Yongqi Zhang, Yong Li, and Quanming Yao. 2024{\natexlab{a}}.
\newblock Explore then determine: A {GNN-LLM} synergy framework for reasoning over knowledge graph.
\newblock \emph{arXiv:2406.01145}.
\bibitem[{Liu et~al.(2024{\natexlab{b}})Liu, Hill, Du, Wang, and Tong}]{liu2024conversational}
Lihui Liu, Blaine Hill, Boxin Du, Fei Wang, and Hanghang Tong. 2024{\natexlab{b}}.
\newblock Conversational question answering with language models generated reformulations over knowledge graph.
\newblock In \emph{ACL}, pages 839--850.
\bibitem[{Lu et~al.(2022)Lu, Mishra, Xia, Qiu, Chang, Zhu, Tafjord, Clark, and Kalyan}]{lu2022learn}
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022.
\newblock Learn to explain: Multimodal reasoning via thought chains for science question answering.
\newblock \emph{NeurIPS}, 35:2507--2521.
\bibitem[{Luo et~al.(2024{\natexlab{a}})Luo, Zhao, Gong, Haffari, and Pan}]{luo2024graph}
Linhao Luo, Zicheng Zhao, Chen Gong, Gholamreza Haffari, and Shirui Pan. 2024{\natexlab{a}}.
\newblock Graph-constrained reasoning: Faithful reasoning on knowledge graphs with large language models.
\newblock \emph{arXiv:2410.13080}.
\bibitem[{Luo et~al.(2024{\natexlab{b}})Luo, Sun, Zhao, Zhao, and Hu}]{luo2024knowla}
Xindi Luo, Zequn Sun, Jing Zhao, Zhe Zhao, and Wei Hu. 2024{\natexlab{b}}.
\newblock {KnowLA}: Enhancing parameter-efficient finetuning with knowledgeable adaptation.
\newblock In \emph{NAACL}, pages 7146--7159.
\bibitem[{Ma et~al.(2025{\natexlab{a}})Ma, Chen, Wu, Khan, and Wang}]{ma2025unifying}
Chuangtao Ma, Yongrui Chen, Tianxing Wu, Arijit Khan, and Haofen Wang. 2025{\natexlab{a}}.
\newblock Unifying large language models and knowledge graphs for question answering: Recent advances and opportunities.
\newblock In \emph{EDBT}, pages 1174--1177.
\bibitem[{Ma et~al.(2025{\natexlab{b}})Ma, Xu, Jiang, Li, Qu, and Guo}]{ma2025think}
Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, and Jian Guo. 2025{\natexlab{b}}.
\newblock {Think-on-Graph 2.0}: Deep and interpretable large language model reasoning with knowledge graph-guided retrieval.
\newblock In \emph{ICLR}.
\bibitem[{Mao et~al.(2021)Mao, He, Liu, Shen, Gao, Han, and Chen}]{mao2021generation}
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021.
\newblock Generation-augmented retrieval for open-domain question answering.
\newblock In \emph{ACL|IJCNLP}, pages 4089--4100.
\bibitem[{Meem et~al.(2024)Meem, Rashid, Dong, and Hristidis}]{meem2024pat}
Jannat~Ara Meem, Muhammad~Shihab Rashid, Yue Dong, and Vagelis Hristidis. 2024.
\newblock {PAT}-questions: A self-updating benchmark for present-anchored temporal question-answering.
\newblock In \emph{ACL}, pages 13129--13148.
\bibitem[{Meyer et~al.(2023)Meyer, Frey, Junghanns, Brei, Bulert, Gr{\"u}nder-Fahrer, and Martin}]{meyer2023developing}
Lars-Peter Meyer, Johannes Frey, Kurt Junghanns, Felix Brei, Kirill Bulert, Sabine Gr{\"u}nder-Fahrer, and Michael Martin. 2023.
\newblock Developing a scalable benchmark for assessing large language models in knowledge graph engineering.
\newblock In \emph{SEMANTICS}.
\bibitem[{Nan et~al.(2024)Nan, Fang, Rasteh, Lahabi, Zou, Zhao, and Cohan}]{nan2024omg}
Linyong Nan, Weining Fang, Aylin Rasteh, Pouya Lahabi, Weijin Zou, Yilun Zhao, and Arman Cohan. 2024.
\newblock {OMG-QA}: Building open-domain multi-modal generative question answering systems.
\newblock In \emph{EMNLP}, pages 1001--1015.
\bibitem[{Pan et~al.(2023)Pan, Razniewski, Kalo, Singhania, Chen, Dietze, Jabeen, Omeliyanenko, Zhang, Lissandrini, Biswas, de~Melo, Bonifati, Vakaj, Dragoni, and Graux}]{pan2023large}
Jeff~Z. Pan, Simon Razniewski, Jan-Christoph Kalo, Sneha Singhania, Jiaoyan Chen, Stefan Dietze, Hajira Jabeen, Janna Omeliyanenko, Wen Zhang, Matteo Lissandrini, Russa Biswas, Gerard de~Melo, Angela Bonifati, Edlira Vakaj, Mauro Dragoni, and Damien Graux. 2023.
\newblock Large language models and knowledge graphs: Opportunities and challenges.
\newblock \emph{Trans. Graph Data Knowl.}, 1(1):1--38.
\bibitem[{Pan et~al.(2024)Pan, Luo, Wang, Chen, Wang, and Wu}]{pan2024unifying}
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2024.
\newblock Unifying large language models and knowledge graphs: A roadmap.
\newblock \emph{IEEE Trans. Knowl. Data Eng.}, 36(7):3580--3599.
\bibitem[{Panda et~al.(2024)Panda, Agarwal, Devaguptapu, Kaul, and A~P}]{panda2024holmes}
Pranoy Panda, Ankush Agarwal, Chaitanya Devaguptapu, Manohar Kaul, and Prathosh A~P. 2024.
\newblock {HOLMES}: Hyper-relational knowledge graphs for multi-hop question answering using {LLMs}.
\newblock In \emph{ACL}, pages 13263--13282.
\bibitem[{Peng et~al.(2024)Peng, Zhu, Liu, Bo, Shi, Hong, Zhang, and Tang}]{peng2024graph}
Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2024.
\newblock Graph retrieval-augmented generation: A survey.
\newblock \emph{arXiv:2408.08921}.
\bibitem[{Perevalov et~al.(2024)Perevalov, Both, and Ngonga~Ngomo}]{perevalov2024multilingual}
Aleksandr Perevalov, Andreas Both, and Axel-Cyrille Ngonga~Ngomo. 2024.
\newblock Multilingual question answering systems for knowledge graphs--a survey.
\newblock \emph{Semantic Web}, 15(5):2089--2124.
\bibitem[{Qian et~al.(2024)Qian, Zhang, Zhao, Zhou, Sui, Zhang, and Song}]{qian2024timer4}
Xinying Qian, Ying Zhang, Yu~Zhao, Baohang Zhou, Xuhui Sui, Li~Zhang, and Kehui Song. 2024.
\newblock {T}ime{R}$^4$: Time-aware retrieval-augmented large language models for temporal knowledge graph question answering.
\newblock In \emph{EMNLP}, pages 6942--6952.
\bibitem[{Qiao et~al.(2024)Qiao, Li, Huang, Wang, Liang, Lin, and Guo}]{qiao2024graphllm}
Zijian Qiao, Nan Li, Chenxi Huang, Gangliang Wang, Shenglin Liang, Hui Lin, and Qinglang Guo. 2024.
\newblock {GraphLLM}: A general framework for multi-hop question answering over knowledge graphs using large language models.
\newblock In \emph{NLPCC}, pages 136--148. Springer.
\bibitem[{Roy et~al.(2024)Roy, Ribeiro, Blloshmi, and Small}]{roy2024learning}
Nirmal Roy, Leonardo Ribeiro, Rexhina Blloshmi, and Kevin Small. 2024.
\newblock Learning when to retrieve, what to rewrite, and how to respond in conversational {QA}.
\newblock In \emph{EMNLP}, pages 10604--10625.
\bibitem[{Saleh et~al.(2024)Saleh, T{\"u}r, and Saygin}]{saleh2024sg}
Ahmmad~OM Saleh, G{\"o}khan T{\"u}r, and Yucel Saygin. 2024.
\newblock {SG-RAG}: Multi-hop question answering with large language models through knowledge graphs.
\newblock In \emph{ICNLSP}, pages 439--448.
\bibitem[{Salnikov et~al.(2023)Salnikov, Lysyuk, Braslavski, Razzhigaev, Malykh, and Panchenko}]{salnikov2023answer}
Mikhail Salnikov, Maria Lysyuk, Pavel Braslavski, Anton Razzhigaev, Valentin~A Malykh, and Alexander Panchenko. 2023.
\newblock Answer candidate type selection: Text-to-text language model for closed book question answering meets knowledge graphs.
\newblock In \emph{KONVENS}, pages 155--164.
\bibitem[{Sanmartin(2024)}]{sanmartin2024kg}
Diego Sanmartin. 2024.
\newblock {KG-RAG}: Bridging the gap between knowledge and creativity.
\newblock \emph{arXiv:2405.12035}.
\bibitem[{Sequeda et~al.(2024)Sequeda, Allemang, and Jacob}]{sequeda2024benchmark}
Juan Sequeda, Dean Allemang, and Bryon Jacob. 2024.
\newblock A benchmark to understand the role of knowledge graphs on large language model's accuracy for question answering on enterprise {SQL} databases.
\newblock In \emph{GRADES-NDA@SIGMOD/PODS}, pages 1--12.
\bibitem[{Shi et~al.(2025)Shi, Yan, Sun, Feng, Ren, Ma, Wang, Yin, de~Rijke, and Ren}]{shi2025direct}
Zhengliang Shi, Lingyong Yan, Weiwei Sun, Yue Feng, Pengjie Ren, Xinyu Ma, Shuaiqiang Wang, Dawei Yin, Maarten de~Rijke, and Zhaochun Ren. 2025.
\newblock Direct retrieval-augmented optimization: Synergizing knowledge selection and language models.
\newblock \emph{arXiv preprint arXiv:2505.03075}.
\bibitem[{Shirdel et~al.(2025)Shirdel, Rorseth, Godfrey, Golab, Srivastava, and Szlichta}]{hirdel2025aprescot}
Moein Shirdel, Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, and Jarek Szlichta. 2025.
\newblock {Apr{\`e}sCoT}: Explaining {LLM} answers with knowledge graphs and chain of thought.
\newblock In \emph{EDBT}, pages 1142--1145.
\bibitem[{Soman et~al.(2024)Soman, Rose, Morris, Akbas, Smith, Peetoom, Villouta-Reyes, Cerono, Shi, Rizk-Jackson et~al.}]{soman2024biomedical}
Karthik Soman, Peter~W Rose, John~H Morris, Rabia~E Akbas, Brett Smith, Braian Peetoom, Catalina Villouta-Reyes, Gabriel Cerono, Yongmei Shi, Angela Rizk-Jackson, and 1 others. 2024.
\newblock Biomedical knowledge graph-optimized prompt generation for large language models.
\newblock \emph{Bioinformatics}, 40(9):btae560.
\bibitem[{Song et~al.(2025)Song, Yan, Liu, Fang, Li, Yan, and Chen}]{song2025injecting}
Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, and Xiuying Chen. 2025.
\newblock Injecting domain-specific knowledge into large language models: a comprehensive survey.
\newblock \emph{arXiv preprint arXiv:2502.10708}.
\bibitem[{Steinigen et~al.(2024)Steinigen, Teucher, Ruland, Rudat, Flores-Herr, Fischer, Milosevic, Schymura, and Ziletti}]{steinigen2024fact}
Daniel Steinigen, Roman Teucher, Timm~Heine Ruland, Max Rudat, Nicolas Flores-Herr, Peter Fischer, Nikola Milosevic, Christopher Schymura, and Angelo Ziletti. 2024.
\newblock {Fact Finder}--enhancing domain expertise of large language models by incorporating knowledge graphs.
\newblock \emph{arXiv preprint arXiv:2408.03010}.
\bibitem[{Su et~al.(2024)Su, Li, Chen, Bai, Jin, and Guo}]{su2024temporal}
Miao Su, ZiXuan Li, Zhuo Chen, Long Bai, Xiaolong Jin, and Jiafeng Guo. 2024.
\newblock Temporal knowledge graph question answering: A survey.
\newblock \emph{arXiv:2406.14191}.
\bibitem[{Sui and Hooi(2024)}]{sui2024can}
Yuan Sui and Bryan Hooi. 2024.
\newblock Can knowledge graphs make large language models more trustworthy? {An} empirical study over open-ended question answering.
\newblock \emph{arXiv:2410.08085}.
\bibitem[{Sun et~al.(2024{\natexlab{a}})Sun, Xu, Tang, Wang, Lin, Gong, Ni, Shum, and Guo}]{sun2023think}
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni, Heung-Yeung Shum, and Jian Guo. 2024{\natexlab{a}}.
\newblock {Think-on-Graph}: Deep and responsible reasoning of large language model with knowledge graph.
\newblock In \emph{ICLR}.
\bibitem[{Sun et~al.(2024{\natexlab{b}})Sun, Tao, Li, and Arakawa}]{sun2024oda}
Lei Sun, Zhengwei Tao, Youdi Li, and Hiroshi Arakawa. 2024{\natexlab{b}}.
\newblock {ODA}: Observation-driven agent for integrating {LLMs} and knowledge graphs.
\newblock In \emph{ACL}, pages 7417--7431.
\bibitem[{Suri et~al.(2024)Suri, Mathur, Dernoncourt, Goswami, Rossi, and Manocha}]{suri2024visdom}
Manan Suri, Puneet Mathur, Franck Dernoncourt, Kanika Goswami, Ryan~A Rossi, and Dinesh Manocha. 2024.
\newblock {VisDoM}: Multi-document {QA} with visually rich elements using multimodal retrieval-augmented generation.
\newblock \emph{arXiv:2412.10704}.
\bibitem[{Talmor et~al.(2019)Talmor, Herzig, Lourie, and Berant}]{talmor2019commonsenseqa}
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.
\newblock {CommonsenseQA}: A question answering challenge targeting commonsense knowledge.
\newblock In \emph{NAACL}, pages 4149--4158.
\bibitem[{Tao et~al.(2024)Tao, Huang, Huang, and Jiang}]{tao2024cluegoptimizing}
Dehao Tao, Feng Huang, Yongfeng Huang, and Minghu Jiang. 2024.
\newblock {Clue-Guided Path Exploration}: Optimizing knowledge graph retrieval with large language models to address the information black box challenge.
\newblock \emph{arXiv:2401.13444}.
\bibitem[{tau Yih et~al.(2016)tau Yih, Richardson, Meek, Chang, and Suh}]{yih2016value}
Wen tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016.
\newblock The value of semantic parse labeling for knowledge base question answering.
\newblock In \emph{ACL}, pages 201--206.
\bibitem[{Tian et~al.(2024)Tian, Luo, Xu, Yuan, Jiang, Wei, and Wang}]{tian2024kg}
Shiyu Tian, Yangyang Luo, Tianze Xu, Caixia Yuan, Huixing Jiang, Chen Wei, and Xiaojie Wang. 2024.
\newblock {KG-Adapter}: Enabling knowledge graph integration in large language models through parameter-efficient fine-tuning.
\newblock In \emph{ACL}, pages 3813--3828.
\bibitem[{Tian et~al.(2025)Tian, Xing, Li, Luo, Yuan, Chen, Jiang, and Wang}]{tian2025systematic}
Shiyu Tian, Shuyue Xing, Xingrui Li, Yangyang Luo, Caixia Yuan, Wei Chen, Huixing Jiang, and Xiaojie Wang. 2025.
\newblock A systematic exploration of knowledge graph alignment with large language models in retrieval augmented generation.
\newblock In \emph{AAAI}, pages 25291--25299.
\bibitem[{Wang et~al.(2023)Wang, Xu, Peng, Zhang, Chen, Wang, Feng, and An}]{wang2023keqing}
Chaojie Wang, Yishi Xu, Zhong Peng, Chenxi Zhang, Bo~Chen, Xinrun Wang, Lei Feng, and Bo~An. 2023.
\newblock keqing: knowledge-based question answering is a nature chain-of-thought mentor of {LLM}.
\newblock \emph{arXiv:2401.00426}.
\bibitem[{Wang et~al.(2024{\natexlab{a}})Wang, Bao, Wang, Yu, Liu, Cheng, and Chen}]{wang2024infuserki}
Fali Wang, Runxue Bao, Suhang Wang, Wenchao Yu, Yanchi Liu, Wei Cheng, and Haifeng Chen. 2024{\natexlab{a}}.
\newblock Infuserki: Enhancing large language models with knowledge graphs via infuser-guided knowledge integration.
\newblock In \emph{EMNLP}, pages 3675--3688.
\bibitem[{Wang et~al.(2022)Wang, Zhang, Du, Wang, and Zhang}]{wang2022knowledge}
Jianfeng Wang, Anda Zhang, Huifang Du, Haofen Wang, and Wenqiang Zhang. 2022.
\newblock Knowledge-enhanced visual question answering with multi-modal joint guidance.
\newblock In \emph{IJCKG}, pages 115--120.
\bibitem[{Wang et~al.(2025)Wang, Fu, Wang, Song, and Bian}]{wang2025pike}
Jinyu Wang, Jingjing Fu, Rui Wang, Lei Song, and Jiang Bian. 2025.
\newblock {PIKE-RAG: sPecIalized KnowledgE and Rationale Augmented Generation}.
\newblock \emph{arXiv preprint arXiv:2501.11551}.
\bibitem[{Wang et~al.(2024{\natexlab{b}})Wang, Chen, Hu, Yang, Liu, Shen, Wei, Zhang, Gu, Zhou, Pan, Zhang, and Chen}]{wang2024learning}
Junjie Wang, Mingyang Chen, Binbin Hu, Dan Yang, Ziqi Liu, Yue Shen, Peng Wei, Zhiqiang Zhang, Jinjie Gu, Jun Zhou, Jeff~Z. Pan, Wen Zhang, and Huajun Chen. 2024{\natexlab{b}}.
\newblock Learning to plan for retrieval-augmented large language models from knowledge graphs.
\newblock In \emph{EMNLP}, pages 7813--7835.
\bibitem[{Wang et~al.(2024{\natexlab{c}})Wang, Chen, Cheng, Liao, Zhang, Wu, Yu, Xu, Zhang, Luo, Li, Yang, Huang, and Li}]{wang2024leave}
Minzheng Wang, Longze Chen, Fu~Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024{\natexlab{c}}.
\newblock Leave no document behind: Benchmarking long-context {LLMs} with extended multi-doc {QA}.
\newblock In \emph{EMNLP}, pages 5627--5646.
\bibitem[{Wang et~al.(2024{\natexlab{d}})Wang, Lipka, Rossi, Siu, Zhang, and Derr}]{wang2024knowledge}
Yu~Wang, Nedim Lipka, Ryan~A. Rossi, Alexa Siu, Ruiyi Zhang, and Tyler Derr. 2024{\natexlab{d}}.
\newblock Knowledge graph prompting for multi-document question answering.
\newblock In \emph{AAAI}, volume~38, pages 19206--19214.
\bibitem[{Wei et~al.(2023)Wei, Su, Ma, Yu, Lei, Zhang, Zhao, and Liu}]{wei2023menatqa}
Yifan Wei, Yisong Su, Huanhuan Ma, Xiaoyan Yu, Fangyu Lei, Yuanzhe Zhang, Jun Zhao, and Kang Liu. 2023.
\newblock {MenatQA}: A new dataset for testing the temporal comprehension and reasoning abilities of large language models.
\newblock In \emph{EMNLP}, pages 1434--1447.
\bibitem[{Wu et~al.(2022)Wu, Yi, Yu, Liu, and Wang}]{wu2022nebula}
Min Wu, Xinglu Yi, Hui Yu, Yu~Liu, and Yujue Wang. 2022.
\newblock {Nebula} graph: An open source distributed graph database.
\newblock \emph{arXiv preprint arXiv:2206.07278}.
\bibitem[{Wu et~al.(2024{\natexlab{a}})Wu, Zhao, Yasunaga, Huang, Cao, Huang, Ioannidis, Subbian, Zou, and Leskovec}]{wu2024stark}
Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis Ioannidis, Karthik Subbian, James~Y Zou, and Jure Leskovec. 2024{\natexlab{a}}.
\newblock Stark: Benchmarking {LLM} retrieval on textual and relational knowledge bases.
\newblock \emph{NeurIPS}, 37:127129--127153.
\bibitem[{Wu et~al.(2024{\natexlab{b}})Wu, Li, Wu, Tao, and Fang}]{wu2024does}
Xuyang Wu, Shuowei Li, Hsin-Tai Wu, Zhiqiang Tao, and Yi~Fang. 2024{\natexlab{b}}.
\newblock Does {RAG} introduce unfairness in {LLMs}? evaluating fairness in retrieval-augmented generation systems.
\newblock \emph{arXiv preprint arXiv:2409.19804}.
\bibitem[{Wu et~al.(2024{\natexlab{c}})Wu, Gao, Zhu, Zhou, Sun, Yang, Lou, Ding, and Yang}]{wu2024strago}
Yurong Wu, Yan Gao, Bin Zhu, Zineng Zhou, Xiaodi Sun, Sheng Yang, Jian-Guang Lou, Zhiming Ding, and Linjun Yang. 2024{\natexlab{c}}.
\newblock {StraGo}: Harnessing strategic guidance for prompt optimization.
\newblock In \emph{EMNLP}, pages 10043--10061.
\bibitem[{Xiong et~al.(2024)Xiong, Bao, and Zhao}]{xiong2024interactive}
Guanming Xiong, Junwei Bao, and Wen Zhao. 2024.
\newblock {Interactive-KBQA}: Multi-turn interactions for knowledge base question answering with large language models.
\newblock In \emph{ACL}, pages 10561--10582.
\bibitem[{Xu et~al.(2025{\natexlab{a}})Xu, Chen, Feng, and Gong}]{xu2025mmrag}
Chuan Xu, Qiaosheng Chen, Yutong Feng, and Cheng Gong. 2025{\natexlab{a}}.
\newblock {mmRAG}: A modular benchmark for retrieval-augmented generation over text, tables, and knowledge graphs.
\newblock \emph{arXiv preprint arXiv:2505.11180}.
\bibitem[{Xu et~al.(2025{\natexlab{b}})Xu, Zheng, Li, Chen, Liu, Chen, and Sun}]{xu2025noderag}
Tianyang Xu, Haojie Zheng, Chengze Li, Haoxiang Chen, Yixin Liu, Ruoxi Chen, and Lichao Sun. 2025{\natexlab{b}}.
\newblock {NodeRAG}: Structuring graph-based rag with heterogeneous nodes.
\newblock \emph{arXiv preprint arXiv:2504.11544}.
\bibitem[{Xu et~al.(2024)Xu, Cruz, Guevara, Wang, Deshpande, Wang, and Li}]{xu2024retrieval}
Zhentao Xu, Mark~Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. 2024.
\newblock Retrieval-augmented generation with knowledge graphs for customer service question answering.
\newblock In \emph{SIGIR}, pages 2905--2909.
\bibitem[{Yang et~al.(2024)Yang, Liu, Marrese-Taylor, Zeng, Ke, Li, Cheng, Chen, Caverlee, Matsuo, and Li}]{yang2024kg}
Rui Yang, Haoran Liu, Edison Marrese-Taylor, Qingcheng Zeng, Yuhe Ke, Wanxin Li, Lechao Cheng, Qingyu Chen, James Caverlee, Yutaka Matsuo, and Irene Li. 2024.
\newblock {KG-Rank}: Enhancing large language models for medical {QA} with knowledge graphs and ranking techniques.
\newblock In \emph{BioNLP Workshop@ACL}, pages 155--166.
\bibitem[{Yang et~al.(2025)Yang, Xue, Razzak, Hacid, and Salim}]{yang2025kg}
Ruiyi Yang, Hao Xue, Imran Razzak, Hakim Hacid, and Flora~D Salim. 2025.
\newblock {KG-IRAG}: A knowledge graph-based iterative retrieval-augmented generation framework for temporal reasoning.
\newblock \emph{arXiv preprint arXiv:2503.14234}.
\bibitem[{Yang and Zhu(2025)}]{yang2025curiousllm}
Zukang Yang and Zixuan Zhu. 2025.
\newblock {CuriousLLM}: Elevating multi-document {QA} with reasoning-infused knowledge graph prompting.
\newblock In \emph{NAACL}, pages 274--286.
\bibitem[{Yasunaga et~al.(2022)Yasunaga, Bosselut, Ren, Zhang, Manning, Liang, and Leskovec}]{yasunaga2022deep}
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher~D Manning, Percy~S Liang, and Jure Leskovec. 2022.
\newblock Deep bidirectional language-knowledge graph pretraining.
\newblock In \emph{NeurIPS}, pages 37309--37323.
\bibitem[{Yu et~al.(2024)Yu, Gan, Zhang, Tong, Liu, and Liu}]{yu2024evaluation}
Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi~Liu, and Zhaofeng Liu. 2024.
\newblock Evaluation of retrieval-augmented generation: A survey.
\newblock In \emph{CCF Conference on Big Data}, pages 102--120.
\bibitem[{Yu and McQuade(2025)}]{yu2025rag}
Hong~Qing Yu and Frank McQuade. 2025.
\newblock {RAG-KG-IL}: A multi-agent hybrid framework for reducing hallucinations and enhancing llm reasoning through rag and incremental knowledge graph learning integration.
\newblock \emph{arXiv preprint arXiv:2503.13514}.
\bibitem[{Zhang et~al.(2024{\natexlab{a}})Zhang, Xu, Jin, Chen, Fang, Liu, Rousseau, Xu, Lu, Weng, and Peng}]{zhang2024mapreduce}
Gongbo Zhang, Zihan Xu, Qiao Jin, Fangyi Chen, Yilu Fang, Yi~Liu, Justin~F Rousseau, Ziyang Xu, Zhiyong Lu, Chunhua Weng, and Yifan Peng. 2024{\natexlab{a}}.
\newblock A mapreduce approach to effectively utilize long context information in retrieval augmented language models.
\newblock \emph{arXiv:2412.15271}.
\bibitem[{Zhang et~al.(2024{\natexlab{b}})Zhang, Feng, and You}]{zhang2024graph}
Haozhen Zhang, Tao Feng, and Jiaxuan You. 2024{\natexlab{b}}.
\newblock Graph of records: Boosting retrieval augmented generation for long-context summarization with graphs.
\newblock \emph{arXiv:2410.11001}.
\bibitem[{Zhang et~al.(2023)Zhang, Cao, Zhang, Lv, Li, Hou, Shi, and Tian}]{zhang2023reasoning}
Jiajie Zhang, Shulin Cao, Tingjian Zhang, Xin Lv, Juanzi Li, Lei Hou, Jiaxin Shi, and Qi~Tian. 2023.
\newblock Reasoning over hierarchical question decomposition tree for explainable question answering.
\newblock In \emph{ACL}, pages 14556--14570.
\bibitem[{Zhang et~al.(2022)Zhang, Zhang, Yu, Tang, Tang, Li, and Chen}]{zhang2022subgraph}
Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022.
\newblock Subgraph retrieval enhanced model for multi-hop knowledge base question answering.
\newblock In \emph{ACL}.
\bibitem[{Zhang and Choi(2021)}]{zhang2021situatedqa}
Michael Zhang and Eunsol Choi. 2021.
\newblock {SituatedQA}: Incorporating extra-linguistic contexts into {QA}.
\newblock In \emph{EMNLP}, pages 7371--7387.
\bibitem[{Zhang et~al.(2025)Zhang, Chen, Bei, Yuan, Zhou, Hong, Dong, Chen, Chang, and Huang}]{zhang2025survey}
Qinggang Zhang, Shengyuan Chen, Yuanchen Bei, Zheng Yuan, Huachi Zhou, Zijin Hong, Junnan Dong, Hao Chen, Yi~Chang, and Xiao Huang. 2025.
\newblock A survey of graph retrieval-augmented generation for customized large language models.
\newblock \emph{arXiv:2501.13958}.
\bibitem[{Zhang et~al.(2021)Zhang, Bosselut, Yasunaga, Ren, Liang, Manning, and Leskovec}]{zhang2022greaselm}
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher~D Manning, and Jure Leskovec. 2021.
\newblock {GreaseLM}: Graph reasoning enhanced language models for question answering.
\newblock In \emph{ICLR}.
\bibitem[{Zhang et~al.(2024{\natexlab{c}})Zhang, Chen, Bai, Kang, Guo, and Zhang}]{zhang2024question}
Yu~Zhang, Kehai Chen, Xuefeng Bai, Zhao Kang, Quanjiang Guo, and Min Zhang. 2024{\natexlab{c}}.
\newblock Question-guided knowledge graph re-scoring and injection for knowledge graph question answering.
\newblock In \emph{EMNLP}, pages 8972--8985.
\bibitem[{Zhang et~al.(2024{\natexlab{d}})Zhang, Wen, and Zhao}]{zhang2024gail}
Zhiqiang Zhang, Liqiang Wen, and Wen Zhao. 2024{\natexlab{d}}.
\newblock A {GAIL} fine-tuned {LLM} enhanced framework for low-resource knowledge graph question answering.
\newblock In \emph{CIKM}, pages 3300--3309.
\bibitem[{Zhao(2024)}]{zhao2024frag}
Dongfang Zhao. 2024.
\newblock {FRAG}: Toward federated vector database management for collaborative and secure retrieval-augmented generation.
\newblock \emph{arXiv:2410.13272}.
\bibitem[{Zhao et~al.(2025{\natexlab{a}})Zhao, Song, Xie, Zhang, Yang, and Li}]{zhao2025improving}
Qi~Zhao, Qi~Song, Tian Xie, Haiyue Zhang, Hongyu Yang, and Xiangyang Li. 2025{\natexlab{a}}.
\newblock Improving pre-trained language models with knowledge enhancement and filtering framework.
\newblock In \emph{NAACL Findings}, pages 3860--3871.
\bibitem[{Zhao et~al.(2024{\natexlab{a}})Zhao, Wang, Cen, Zha, Tan, Dong, and Tang}]{zhao2024longrag}
Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, and Jie Tang. 2024{\natexlab{a}}.
\newblock {LongRAG}: A dual-perspective retrieval-augmented generation paradigm for long-context question answering.
\newblock In \emph{EMNLP}, pages 22600--22632.
\bibitem[{Zhao et~al.(2024{\natexlab{b}})Zhao, Zhao, Wang, Wang, and Xu}]{zhao2024kg}
Ruilin Zhao, Feng Zhao, Long Wang, Xianzhi Wang, and Guandong Xu. 2024{\natexlab{b}}.
\newblock {KG-CoT}: {Chain-of-thought} prompting of large language models over knowledge graphs for knowledge-aware question answering.
\newblock In \emph{IJCAI}, pages 6642--6650.
\bibitem[{Zhao et~al.(2025{\natexlab{b}})Zhao, Liu, Yang, and Miao}]{zhao2025medrag}
Xuejiao Zhao, Siyan Liu, Su-Yin Yang, and Chunyan Miao. 2025{\natexlab{b}}.
\newblock {MedRAG}: Enhancing retrieval-augmented generation with knowledge graph-elicited reasoning for healthcare copilot.
\newblock In \emph{Proceedings of the ACM on Web Conference 2025}, pages 4442--4457.
\bibitem[{Zheng et~al.(2024{\natexlab{a}})Zheng, Bai, Zhang, Su, Niu, and Jaitly}]{zheng2024kglens}
Shangshang Zheng, He~Bai, Yizhe Zhang, Yi~Su, Xiaochuan Niu, and Navdeep Jaitly. 2024{\natexlab{a}}.
\newblock {KGLens}: Towards efficient and effective knowledge probing of large language models with knowledge graphs.
\newblock \emph{arXiv:2312.11539}.
\bibitem[{Zheng et~al.(2024{\natexlab{b}})Zheng, Che, Wu, Zhang, Nie, Liu, and Tao}]{zheng2024ks}
Xinxin Zheng, Feihu Che, Jinyang Wu, Shuai Zhang, Shuai Nie, Kang Liu, and Jianhua Tao. 2024{\natexlab{b}}.
\newblock {KS-LLM}: Knowledge selection of large language models with evidence document for question answering.
\newblock \emph{arXiv:2404.15660}.
\bibitem[{Zhou et~al.(2025)Zhou, Zhu, He, Chen, Kharlamov, and Staab}]{zhou2025evaluating}
Dongzhuoran Zhou, Yuqicheng Zhu, Yuan He, Jiaoyan Chen, Evgeny Kharlamov, and Steffen Staab. 2025.
\newblock Evaluating knowledge graph based retrieval augmented generation methods under knowledge incompleteness.
\newblock \emph{arXiv preprint arXiv:2504.05163}.
\bibitem[{Zhu et~al.(2024)Zhu, Hwang, Dugan, and Callison-Burch}]{zhu2024fanoutqa}
Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. 2024.
\newblock {FanOutQA}: A multi-hop, multi-document question answering benchmark for large language models.
\newblock In \emph{ACL}, pages 18--37.
\bibitem[{Zhu et~al.(2025)Zhu, Xie, Liu, Li, and Hu}]{zhu2025knowledge}
Xiangrong Zhu, Yuexiang Xie, Yi~Liu, Yaliang Li, and Wei Hu. 2025.
\newblock Knowledge graph-guided retrieval augmented generation.
\newblock \emph{arXiv preprint arXiv:2502.06864}.
\end{thebibliography}