Connect the Dots: Knowledge Graph-Guided Crawler Attack on Retrieval-Augmented Generation Systems
Extracted source text · arXiv 2601.15678
%% bare_conf_compsoc.tex
%% V1.4b
%% 2015/08/26
%% by Michael Shell
%% See:
%% http://www.michaelshell.org/
%% for current contact information.
%%
%% This is a skeleton file demonstrating the use of IEEEtran.cls
%% (requires IEEEtran.cls version 1.8b or later) with an IEEE Computer
%% Society conference paper.
%%
%% Support sites:
%% http://www.michaelshell.org/tex/ieeetran/
%% http://www.ctan.org/pkg/ieeetran
%% and
%% http://www.ieee.org/
%%*************************************************************************
%% Legal Notice:
%% This code is offered as-is without any warranty either expressed or
%% implied; without even the implied warranty of MERCHANTABILITY or
%% FITNESS FOR A PARTICULAR PURPOSE!
%% User assumes all risk.
%% In no event shall the IEEE or any contributor to this code be liable for
%% any damages or losses, including, but not limited to, incidental,
%% consequential, or any other damages, resulting from the use or misuse
%% of any information contained here.
%%
%% All comments are the opinions of their respective authors and are not
%% necessarily endorsed by the IEEE.
%%
%% This work is distributed under the LaTeX Project Public License (LPPL)
%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used,
%% distributed and modified. A copy of the LPPL, version 1.3, is included
%% in the base LaTeX documentation of all distributions of LaTeX released
%% 2003/12/01 or later.
%% Retain all contribution notices and credits.
%% ** Modified files should be clearly indicated as such, including **
%% ** renaming them and changing author support contact information. **
%%*************************************************************************
% *** Authors should verify (and, if needed, correct) their LaTeX system ***
% *** with the testflow diagnostic prior to trusting their LaTeX platform ***
% *** with production work. The IEEE's font choices and paper sizes can ***
% *** trigger bugs that do not appear when using other class files. *** ***
% The testflow support page is at:
% http://www.michaelshell.org/tex/testflow/
\documentclass[conference,compsoc]{IEEEtran}
\input{packages}
% Some/most Computer Society conferences require the compsoc mode option,
% but others may want the standard conference format.
%
% If IEEEtran.cls has not been installed into the LaTeX system files,
% manually specify the path to it like:
% \documentclass[conference,compsoc]{../sty/IEEEtran}
% Some very useful LaTeX packages include:
% (uncomment the ones you want to load)
% *** MISC UTILITY PACKAGES ***
%
%\usepackage{ifpdf}
% Heiko Oberdiek's ifpdf.sty is very useful if you need conditional
% compilation based on whether the output is pdf or dvi.
% usage:
% \ifpdf
% % pdf code
% \else
% % dvi code
% \fi
% The latest version of ifpdf.sty can be obtained from:
% http://www.ctan.org/pkg/ifpdf
% Also, note that IEEEtran.cls V1.7 and later provides a builtin
% \ifCLASSINFOpdf conditional that works the same way.
% When switching from latex to pdflatex and vice-versa, the compiler may
% have to be run twice to clear warning/error messages.
% *** CITATION PACKAGES ***
%
\ifCLASSOPTIONcompsoc
% IEEE Computer Society needs nocompress option
% requires cite.sty v4.0 or later (November 2003)
\usepackage[nocompress]{cite}
\else
% normal IEEE
\usepackage{cite}
\fi
% cite.sty was written by Donald Arseneau
% V1.6 and later of IEEEtran pre-defines the format of the cite.sty package
% \cite{} output to follow that of the IEEE. Loading the cite package will
% result in citation numbers being automatically sorted and properly
% "compressed/ranged". e.g., [1], [9], [2], [7], [5], [6] without using
% cite.sty will become [1], [2], [5]--[7], [9] using cite.sty. cite.sty's
% \cite will automatically add leading space, if needed. Use cite.sty's
% noadjust option (cite.sty V3.8 and later) if you want to turn this off
% such as if a citation ever needs to be enclosed in parenthesis.
% cite.sty is already installed on most LaTeX systems. Be sure and use
% version 5.0 (2009-03-20) and later if using hyperref.sty.
% The latest version can be obtained at:
% http://www.ctan.org/pkg/cite
% The documentation is contained in the cite.sty file itself.
%
% Note that some packages require special options to format as the Computer
% Society requires. In particular, Computer Society papers do not use
% compressed citation ranges as is done in typical IEEE papers
% (e.g., [1]-[4]). Instead, they list every citation separately in order
% (e.g., [1], [2], [3], [4]). To get the latter we need to load the cite
% package with the nocompress option which is supported by cite.sty v4.0
% and later.
% *** GRAPHICS RELATED PACKAGES ***
%
\ifCLASSINFOpdf
% \usepackage[pdftex]{graphicx}
% declare the path(s) where your graphic files are
% \graphicspath{{../pdf/}{../jpeg/}}
% and their extensions so you won't have to specify these with
% every instance of \includegraphics
% \DeclareGraphicsExtensions{.pdf,.jpeg,.png}
\else
% or other class option (dvipsone, dvipdf, if not using dvips). graphicx
% will default to the driver specified in the system graphics.cfg if no
% driver is specified.
% \usepackage[dvips]{graphicx}
% declare the path(s) where your graphic files are
% \graphicspath{{../eps/}}
% and their extensions so you won't have to specify these with
% every instance of \includegraphics
% \DeclareGraphicsExtensions{.eps}
\fi
% graphicx was written by David Carlisle and Sebastian Rahtz. It is
% required if you want graphics, photos, etc. graphicx.sty is already
% installed on most LaTeX systems. The latest version and documentation
% can be obtained at:
% http://www.ctan.org/pkg/graphicx
% Another good source of documentation is "Using Imported Graphics in
% LaTeX2e" by Keith Reckdahl which can be found at:
% http://www.ctan.org/pkg/epslatex
%
% latex, and pdflatex in dvi mode, support graphics in encapsulated
% postscript (.eps) format. pdflatex in pdf mode supports graphics
% in .pdf, .jpeg, .png and .mps (metapost) formats. Users should ensure
% that all non-photo figures use a vector format (.eps, .pdf, .mps) and
% not a bitmapped formats (.jpeg, .png). The IEEE frowns on bitmapped formats
% which can result in "jaggedy"/blurry rendering of lines and letters as
% well as large increases in file sizes.
%
% You can find documentation about the pdfTeX application at:
% http://www.tug.org/applications/pdftex
% *** MATH PACKAGES ***
%
%\usepackage{amsmath}
% A popular package from the American Mathematical Society that provides
% many useful and powerful commands for dealing with mathematics.
%
% Note that the amsmath package sets \interdisplaylinepenalty to 10000
% thus preventing page breaks from occurring within multiline equations. Use:
%\interdisplaylinepenalty=2500
% after loading amsmath to restore such page breaks as IEEEtran.cls normally
% does. amsmath.sty is already installed on most LaTeX systems. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/pkg/amsmath
% *** SPECIALIZED LIST PACKAGES ***
%
%\usepackage{algorithmic}
% algorithmic.sty was written by Peter Williams and Rogerio Brito.
% This package provides an algorithmic environment fo describing algorithms.
% You can use the algorithmic environment in-text or within a figure
% environment to provide for a floating algorithm. Do NOT use the algorithm
% floating environment provided by algorithm.sty (by the same authors) or
% algorithm2e.sty (by Christophe Fiorio) as the IEEE does not use dedicated
% algorithm float types and packages that provide these will not provide
% correct IEEE style captions. The latest version and documentation of
% algorithmic.sty can be obtained at:
% http://www.ctan.org/pkg/algorithms
% Also of interest may be the (relatively newer and more customizable)
% algorithmicx.sty package by Szasz Janos:
% http://www.ctan.org/pkg/algorithmicx
% *** ALIGNMENT PACKAGES ***
%
%\usepackage{array}
% Frank Mittelbach's and David Carlisle's array.sty patches and improves
% the standard LaTeX2e array and tabular environments to provide better
% appearance and additional user controls. As the default LaTeX2e table
% generation code is lacking to the point of almost being broken with
% respect to the quality of the end results, all users are strongly
% advised to use an enhanced (at the very least that provided by array.sty)
% set of table tools. array.sty is already installed on most systems. The
% latest version and documentation can be obtained at:
% http://www.ctan.org/pkg/array
% IEEEtran contains the IEEEeqnarray family of commands that can be used to
% generate multiline equations as well as matrices, tables, etc., of high
% quality.
% *** SUBFIGURE PACKAGES ***
%\ifCLASSOPTIONcompsoc
% \usepackage[caption=false,font=footnotesize,labelfont=sf,textfont=sf]{subfig}
%\else
% \usepackage[caption=false,font=footnotesize]{subfig}
%\fi
% subfig.sty, written by Steven Douglas Cochran, is the modern replacement
% for subfigure.sty, the latter of which is no longer maintained and is
% incompatible with some LaTeX packages including fixltx2e. However,
% subfig.sty requires and automatically loads Axel Sommerfeldt's caption.sty
% which will override IEEEtran.cls' handling of captions and this will result
% in non-IEEE style figure/table captions. To prevent this problem, be sure
% and invoke subfig.sty's "caption=false" package option (available since
% subfig.sty version 1.3, 2005/06/28) as this is will preserve IEEEtran.cls
% handling of captions.
% Note that the Computer Society format requires a sans serif font rather
% than the serif font used in traditional IEEE formatting and thus the need
% to invoke different subfig.sty package options depending on whether
% compsoc mode has been enabled.
%
% The latest version and documentation of subfig.sty can be obtained at:
% http://www.ctan.org/pkg/subfig
% *** FLOAT PACKAGES ***
%
%\usepackage{fixltx2e}
% fixltx2e, the successor to the earlier fix2col.sty, was written by
% Frank Mittelbach and David Carlisle. This package corrects a few problems
% in the LaTeX2e kernel, the most notable of which is that in current
% LaTeX2e releases, the ordering of single and double column floats is not
% guaranteed to be preserved. Thus, an unpatched LaTeX2e can allow a
% single column figure to be placed prior to an earlier double column
% figure.
% Be aware that LaTeX2e kernels dated 2015 and later have fixltx2e.sty's
% corrections already built into the system in which case a warning will
% be issued if an attempt is made to load fixltx2e.sty as it is no longer
% needed.
% The latest version and documentation can be found at:
% http://www.ctan.org/pkg/fixltx2e
%\usepackage{stfloats}
% stfloats.sty was written by Sigitas Tolusis. This package gives LaTeX2e
% the ability to do double column floats at the bottom of the page as well
% as the top. (e.g., "\begin{figure*}[!b]" is not normally possible in
% LaTeX2e). It also provides a command:
%\fnbelowfloat
% to enable the placement of footnotes below bottom floats (the standard
% LaTeX2e kernel puts them above bottom floats). This is an invasive package
% which rewrites many portions of the LaTeX2e float routines. It may not work
% with other packages that modify the LaTeX2e float routines. The latest
% version and documentation can be obtained at:
% http://www.ctan.org/pkg/stfloats
% Do not use the stfloats baselinefloat ability as the IEEE does not allow
% \baselineskip to stretch. Authors submitting work to the IEEE should note
% that the IEEE rarely uses double column equations and that authors should try
% to avoid such use. Do not be tempted to use the cuted.sty or midfloat.sty
% packages (also by Sigitas Tolusis) as the IEEE does not format its papers in
% such ways.
% Do not attempt to use stfloats with fixltx2e as they are incompatible.
% Instead, use Morten Hogholm'a dblfloatfix which combines the features
% of both fixltx2e and stfloats:
%
% \usepackage{dblfloatfix}
% The latest version can be found at:
% http://www.ctan.org/pkg/dblfloatfix
% *** PDF, URL AND HYPERLINK PACKAGES ***
%
%\usepackage{url}
% url.sty was written by Donald Arseneau. It provides better support for
% handling and breaking URLs. url.sty is already installed on most LaTeX
% systems. The latest version and documentation can be obtained at:
% http://www.ctan.org/pkg/url
% Basically, \url{my_url_here}.
% *** Do not adjust lengths that control margins, column widths, etc. ***
% *** Do not use packages that alter fonts (such as pslatex). ***
% There should be no need to do such things with IEEEtran.cls V1.6 and later.
% (Unless specifically asked to do so by the journal or conference you plan
% to submit to, of course. )
% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}
\begin{document}
%
% paper title
% Titles are generally capitalized except for words such as a, an, and, as,
% at, but, by, for, in, nor, of, on, or, the, to and up, which are usually
% not capitalized unless they are the first or last word of the title.
% Linebreaks \\ can be used within to get better formatting as desired.
% Do not put math or special symbols in the title.
%\title{Crawling in Silence: Knowledge Graph–Powered Crawler Attack on Retrieval-Augmented Generation Systems}
\title{Connect the Dots: Knowledge Graph–Guided Crawler Attack on Retrieval-Augmented Generation Systems}
% author names and affiliations
% use a multiple column layout for up to three different
% affiliations
% \author{\IEEEauthorblockN{Mengyu Yao}
% \IEEEauthorblockA{School of Electrical and\\Computer Engineering\\
% Georgia Institute of Technology\\
% Atlanta, Georgia 30332--0250\\
% Email: http://www.michaelshell.org/contact.html}
% \and
% \IEEEauthorblockN{Ziqi Zhang}
% \IEEEauthorblockA{Twentieth Century Fox\\
% Springfield, USA\\
% Email: homer@thesimpsons.com}
% \and
% \IEEEauthorblockN{James Kirk\\ and Montgomery Scott}
% \IEEEauthorblockA{Starfleet Academy\\
% San Francisco, California 96678-2391\\
% Telephone: (800) 555--1212\\
% Fax: (888) 555--1212}}
% conference papers do not typically use \thanks and this command
% is locked out in conference mode. If really needed, such as for
% the acknowledgment of grants, issue a \IEEEoverridecommandlockouts
% after \documentclass
% for over three affiliations, or if they all won't fit within the width
% of the page (and note that there is less available width in this regard for
% compsoc conferences compared to traditional conferences), use this
% alternative format:
%
\author{\IEEEauthorblockN{Mengyu Yao\IEEEauthorrefmark{1},
Ziqi Zhang\IEEEauthorrefmark{2},
Ning Luo\IEEEauthorrefmark{2},
Shaofei Li\IEEEauthorrefmark{1},
Yifeng Cai\IEEEauthorrefmark{1},
Xiangqun Chen\IEEEauthorrefmark{1},
Yao Guo\IEEEauthorrefmark{1} and
Ding Li\IEEEauthorrefmark{1}}
\IEEEauthorblockA{\IEEEauthorrefmark{1}Key Laboratory of High-Confidence Software Technologies (MOE), School of Computer Science, Peking University}
\IEEEauthorblockA{\IEEEauthorrefmark{2}Department of Computer Science, University of Illinois Urbana-Champaign}}
% use for special paper notices
%\IEEEspecialpapernotice{(Invited Paper)}
% make the title area
\maketitle
% As a general rule, do not put math, special symbols or citations
% in the abstract
\begin{abstract}
Retrieval-augmented generation (RAG) systems integrate document retrieval with large language models and have been widely adopted. However, in privacy-related scenarios (e.g., medical diagnosis), RAG introduces a new privacy risk: adversaries can issue carefully crafted queries to exfiltrate sensitive content from the underlying corpus gradually. Although recent studies have demonstrated multi-turn extraction attacks, they rely on heuristics and fail to perform long-term extraction planning. As a result, they typically stagnate after a few rounds and exfiltrate only a small fraction of the private corpus.
%
% Recent studies have demonstrated multi-turn extraction attacks, but existing methods are short-sighted as they only rely on local heuristics and fail to plan beyond the immediate context. As a result, they typically stagnate after a few rounds and exfiltrate only a small fraction of the private corpus.
To address these limitations, we formulate the RAG extraction attack as an adaptive stochastic coverage problem (ASCP). In ASCP, each query is treated as a probabilistic action that aims to maximize conditional marginal gain (CMG), enabling principled long-term planning under uncertainty. However, integrating ASCP with practical RAG attack faces three key challenges: unobservable CMG, intractability in the action space, and feasibility constraints. To overcome these challenges, we maintain a global attacker-side state to guide the attack. Building on this idea, we introduce \toolname, which builds a knowledge graph to represent revealed information, uses this global state to estimate CMG, and plans queries in semantic space that target unretrieved regions. It also generates benign-looking prompts that elicit the remaining content while preserving stealth.
% To bridge ASCP and practice, our core idea is to maintain a global attacker-side state and to use it to drive planning and realization. We introduce \toolname, which builds a knowledge graph to represent revealed information, uses this global state to estimate CMG and to plan queries in semantic space that target unretrieved regions, and generates benign appearing prompts that elicit the remaining content while preserving stealth.
%
In comprehensive experiments across diverse RAG architectures and datasets, our proposed method, \toolname, consistently outperforms all baselines. It achieves up to 84.4\% corpus coverage within a fixed query budget and deliver an average improvement of 20.7\% over the top-performing baseline. It also maintains high semantic fidelity and strong content reconstruction accuracy with low attack cost. Crucially, \toolname proves its robustness by maintaining effectiveness against advanced RAG systems employing query rewriting and multi-query retrieval strategies. Our work reveals significant security gaps and highlights the pressing need for stronger safeguards for RAG.
\end{abstract}
% \begin{abstract}
% Retrieval-augmented generation (RAG) systems integrate document retrieval with large language models and have been widely adopted. However, in privacy-related scenarios (e.g., medical diagnosis), RAG introduces a new privacy risk: adversaries can issue carefully crafted queries to exfiltrate sensitive content from the underlying corpus gradually. Although recent studies have demonstrated multi-turn extraction attacks, they rely on heuristics and fail to perform long-term extraction planning. As a result, they typically stagnate after a few rounds and exfiltrate only a small fraction of the private corpus.
% To address these limitations, we formulate the RAG extraction attack as an adaptive stochastic coverage problem (ASCP). In ASCP, each query is treated as a probabilistic action that aims to maximize conditional marginal gain (CMG), enabling principled long-term planning under uncertainty. However, integrating ASCP with practical RAG attack faces three key challenges: unobservable CMG, intractability in the action space, and feasibility constraints. To overcome these challenges, we maintain a global attacker-side state to guide the attack. Building on this idea, we introduce RAGCrawler, which builds a knowledge graph to represent revealed information, uses this global state to estimate CMG, and plans queries in semantic space that target unretrieved regions. It also generates benign-looking prompts that elicit the remaining content while preserving stealth.
% In experiments across multiple RAG architectures and datasets, RAGCrawler achieved up to 91.70\% corpus coverage (compared to 0–39\% for baselines) and an average of 74.18\% (versus 1.38\% and 19.14\% for baselines). It also maintains high semantic fidelity and strong content reconstruction accuracy with low attack cost. Furthermore, RAGCrawler remains effective against advanced RAG variants that employ query rewriting and multi-query retrieval. Our work highlights the urgent need for stronger safeguards for RAG.
% \end{abstract}
% no keywords
% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
\IEEEpeerreviewmaketitle
\input{sections/intro_new}
%\input{sections/motivation}
\input{sections/motivating_example}
\input{sections/background}
\input{sections/threat_model}
\input{sections/method}
\input{sections/evaluation}
\input{sections/discussion}
\input{sections/conclusion}
% An example of a floating figure using the graphicx package.
% Note that \label must occur AFTER (or within) \caption.
% For figures, \caption should occur after the \includegraphics.
% Note that IEEEtran v1.7 and later has special internal code that
% is designed to preserve the operation of \label within \caption
% even when the captionsoff option is in effect. However, because
% of issues like this, it may be the safest practice to put all your
% \label just after \caption rather than within \caption{}.
%
% Reminder: the "draftcls" or "draftclsnofoot", not "draft", class
% option should be used if it is desired that the figures are to be
% displayed while in draft mode.
%
%\begin{figure}[!t]
%\centering
%\includegraphics[width=2.5in]{myfigure}
% where an .eps filename suffix will be assumed under latex,
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{Simulation results for the network.}
%\label{fig_sim}
%\end{figure}
% Note that the IEEE typically puts floats only at the top, even when this
% results in a large percentage of a column being occupied by floats.
% An example of a double column floating figure using two subfigures.
% (The subfig.sty package must be loaded for this to work.)
% The subfigure \label commands are set within each subfloat command,
% and the \label for the overall figure must come after \caption.
% \hfil is used as a separator to get equal spacing.
% Watch out that the combined width of all the subfigures on a
% line do not exceed the text width or a line break will occur.
%
%\begin{figure*}[!t]
%\centering
%\subfloat[Case I]{\includegraphics[width=2.5in]{box}%
%\label{fig_first_case}}
%\hfil
%\subfloat[Case II]{\includegraphics[width=2.5in]{box}%
%\label{fig_second_case}}
%\caption{Simulation results for the network.}
%\label{fig_sim}
%\end{figure*}
%
% Note that often IEEE papers with subfigures do not employ subfigure
% captions (using the optional argument to \subfloat[]), but instead will
% reference/describe all of them (a), (b), etc., within the main caption.
% Be aware that for subfig.sty to generate the (a), (b), etc., subfigure
% labels, the optional argument to \subfloat must be present. If a
% subcaption is not desired, just leave its contents blank,
% e.g., \subfloat[].
% An example of a floating table. Note that, for IEEE style tables, the
% \caption command should come BEFORE the table and, given that table
% captions serve much like titles, are usually capitalized except for words
% such as a, an, and, as, at, but, by, for, in, nor, of, on, or, the, to
% and up, which are usually not capitalized unless they are the first or
% last word of the caption. Table text will default to \footnotesize as
% the IEEE normally uses this smaller font for tables.
% The \label must come after \caption as always.
%
%\begin{table}[!t]
%% increase table row spacing, adjust to taste
%\renewcommand{\arraystretch}{1.3}
% if using array.sty, it might be a good idea to tweak the value of
% \extrarowheight as needed to properly center the text within the cells
%\caption{An Example of a Table}
%\label{table_example}
%\centering
%% Some packages, such as MDW tools, offer better commands for making tables
%% than the plain LaTeX2e tabular which is used here.
%\begin{tabular}{|c||c|}
%\hline
%One & Two\\
%\hline
%Three & Four\\
%\hline
%\end{tabular}
%\end{table}
% Note that the IEEE does not put floats in the very first column
% - or typically anywhere on the first page for that matter. Also,
% in-text middle ("here") positioning is typically not used, but it
% is allowed and encouraged for Computer Society conferences (but
% not Computer Society journals). Most IEEE journals/conferences use
% top floats exclusively.
% Note that, LaTeX2e, unlike IEEE journals/conferences, places
% footnotes above bottom floats. This can be corrected via the
% \fnbelowfloat command of the stfloats package.
% conference papers do not normally have an appendix
% use section* for acknowledgment
% \ifCLASSOPTIONcompsoc
% % The Computer Society usually uses the plural form
% \section*{Acknowledgments}
% \else
% % regular IEEE prefers the singular form
% \section*{Acknowledgment}
% \fi
% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
%\IEEEtriggeratref{8}
% The "triggered" command can be changed if desired:
%\IEEEtriggercmd{\enlargethispage{-5in}}
% references section
% can use a bibliography generated by BibTeX as a .bbl file
% BibTeX documentation can be easily obtained at:
% http://mirror.ctan.org/biblio/bibtex/contrib/doc/
% The IEEEtran BibTeX style support page is at:
% http://www.michaelshell.org/tex/ieeetran/bibtex/
%\bibliographystyle{IEEEtran}
% argument is your BibTeX string definitions and bibliography database(s)
%\bibliography{IEEEabrv,../bib/paper}
%
% <OR> manually copy in the resultant .bbl file
% set second argument of \begin to the number of references
% (used to reserve space for the reference number labels box)
\bibliographystyle{IEEEtran}
\bibliography{IEEEabrv,reference}
\input{sections/appendix}
%\input{sections/ethics}
% that's all folks
\end{document}
%\usepackage{cite}
\let\Bbbk\relax
\usepackage{amsmath,amssymb,amsfonts}
% \usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
%\usepackage{tcolorbox}
\usepackage[all]{nowidow}
% \usepackage[linesnumbered,lined,vlined,ruled,commentsnumbered,noend]{algorithm2e}
\usepackage{listings}
\usepackage{url}
\usepackage{makecell}
\usepackage{hyperref}
\usepackage{adjustbox}
\usepackage{xspace}
\usepackage{multirow}
\usepackage{subfloat}
\usepackage{balance}
\usepackage{tikz}
\usepackage{ctable}
\usepackage{listings}
\usepackage{color}
\usepackage[nobiblatex]{xurl}
\usepackage{mathrsfs}
\usepackage[normalem]{ulem}
\useunder{\uline}{\ul}{}
\usepackage{algpseudocode}
\usepackage{caption}
%\usepackage{cleveref}
\usepackage{acronym}
\usepackage{fancybox}
\usepackage{verbatim}
\usepackage[normalem]{ulem}
\usepackage{float}
\usepackage{newfloat}
\usepackage{mdframed}
\let\labelindent\relax
\usepackage{booktabs}
% \usepackage[ruled,linesnumbered]{algorithm2e}
\usepackage{subfigure}
\usepackage[flushleft]{threeparttable}
\usepackage{booktabs}
\usepackage{tabularray}
\usepackage[utf8]{inputenc}
\usepackage[skins]{tcolorbox}
\usepackage{amsthm}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}
\usepackage{enumitem}
\usepackage{pifont}
\usepackage{tcolorbox}
\usepackage{subcaption}
\usepackage{multirow}
\usepackage[normalem]{ulem}
%
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage[table]{xcolor}
\usepackage{xspace}
\usepackage{ulem}
\definecolor{comp1}{RGB}{212,235,255}
\definecolor{comp2}{RGB}{255,245,200}
\definecolor{comp3}{RGB}{255,228,196}
\newcommand{\CYF}[1]{\textcolor{purple}{(CYF: #1)}}
\newcommand{\zzq}[1]{\textcolor{red}{(ZZQ: #1)}}
\newcommand{\ziqi}[1]{\textcolor{red}{(ZZQ: #1)}}
\newcommand{\TODO}[1]{\textcolor{orange}{(TODO: #1)}}
\newcommand{\revise}[1]{\textcolor{blue}{#1}}
\newcommand{\ymy}[1]{\textcolor{orange}{(YMY: #1)}}
\newcommand{\ning}[1]{\textcolor{red}{NL: #1}}
\usepackage{amsthm}
\newtheorem{theorem}{Theorem}
\definecolor{RiseGreen}{RGB}{200,230,200} % 浅绿色文字背景
\definecolor{FallGray}{RGB}{230,230,230} % 浅灰色文字背景
\newcommand{\dingli}[1]{\textcolor{red}{(DL: #1)}}
\DeclareMathOperator*{\argmax}{arg\,max} % in your preamble
\DeclareMathOperator*{\argmin}{arg\,min} % in your preamble
% 定义命令：\rise{数值}, \fall{数值}
\setlength{\fboxsep}{1.5pt}
% \newcommand{\rise}[1]{{$\uparrow$#1}}
% \newcommand{\fall}[1]{\colorbox{FallGray}{$\downarrow$#1}}
\newcommand{\rise}[1]{#1}
\newcommand{\fall}[1]{-{#1}}
% 保留小数
\usepackage{siunitx}
\sisetup{
round-mode = places,
round-precision = 3,
detect-weight = true,
detect-family = true
}
%\newcommand{\toolname}{\emph{RAGCrawler}\xspace}
\usepackage{graphicx}
% 定义一个命令，行内小图标
\newcommand{\icon}[1]{%
\raisebox{-0.2em}{\includegraphics[height=1em]{#1}}%
}
\newcommand{\toolname}{\textsc{RAGCrawler}\xspace}
\newtheorem{definition}{Definition}
% \newcommand{\longtoolname}{\textsc{\underline{C}\underline{R}oss-modal-\underline{E}nhanced \underline{A}IP attack toward \underline{M}RS}s}
\section{Introduction}
\vspace{-2mm}
% RAG’s popularity and benefits (setting the context)
Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm that integrates information retrieval with large language models (LLMs), allowing systems to dynamically access and utilize external knowledge bases for tasks such as question answering, summarization, and decision support~\cite{karpukhin2020dense,lewis2020retrieval}.
By providing timely, domain-specific evidence beyond a model’s parametric memory, RAG has seen rapid adoption across settings ranging from enterprise knowledge management~\cite{xu2024generative} to healthcare~\cite{al2023transforming}, and customer service~\cite{loukas2023making}.
% The leakage threat in RAG systems and its severe consequences (motivating the problem)
RAG's retrieval capabilities introduce significant privacy risks, as many of the data sources it accesses or stores contain confidential information. Recent studies demonstrate that sensitive data can leak from RAG by eliciting verbatim passages~\cite{zeng2024good,qifollow}, exposing personal identifiers from clinical notes or confidential clauses from internal documents. Such leaks can lead to non-compliance with GDPR and HIPAA~\cite{voigt2017eu,annas2003hipaa}, resulting in legal, financial, and reputational harm~\cite{lasso2024ragsecurity}. Abusing the retrieval capabilities, adversaries can exploit the retriever as a high-precision oracle to exfiltrate private corpus content through carefully crafted prompts.
Understanding of the threat posed by extraction attacks on RAG remains limited, as existing studies often overlook adversaries' global planning capabilities when crafting the prompts.
In known attacks, prompts often build on recent responses, expanding on the retrieved context to uncover progressively more information. For example, an attack might extract a keyword from a recent answer and query it further~\cite{wang2025silent}, or instruct the LLM to elaborate on a just-mentioned fact~\cite{jiang2024rag}. These heuristic methods lack global planning and will often drifts off topic (as shown in the left panel Fig.~\ref{fig:example}(a)) or re-touches previously seen content (as shown in the middle panel Fig.~\ref{fig:example}(a)), producing low-value responses and stalling early, leaving a substantial portion of the corpus remains unexplored as illustrated in Fig~\ref{fig:example}(b). %\ning{add reference}
Most importantly, while the empirical results demonstrate the feasibility of extracting content from RAG, the theoretical grounding and provable guarantees remain missing.
% \sout{These heuristic methods yield initial leakage, but once the immediately related content is exhausted, the attack loses direction\ning{what that means?}. As a result, it stalls long before the corpus is fully extracted, even though plenty of unexplored information remains to be exfiltrated.} \ymy{While these heuristic methods can uncover some content initially, they lack the ability to identify unexplored regions once the local context is exhausted. As a result, the attack often re-touches previously seen content or drifts off topic, producing low-value responses and stalling early, even though a substantial portion of the corpus remains unexplored.} Empirically, as illustrated in Fig.~\ref{fig:teaser}, prior attacks plateau early due to losing attacking directions: RAGThief~\cite{jiang2024rag} stops after only 7 queries and IKEA~\cite{wang2025silent} around 100, whereas \toolname continues to exfiltrate new content.
% RAG's retrieval capabilities introduce significant privacy risks, as many of the data sources it accesses or stores contain confidential information. Recent studies demonstrate that sensitive data can leak from RAG by eliciting verbatim passages~\cite{zeng2024good,qifollow}, and reveal personal identifiers from clinical notes or confidential clauses from internal documents, triggering non-compliance with GDPR and HIPAA~\cite{voigt2017eu,annas2003hipaa} and causing legal, financial, and reputational harm~\cite{lasso2024ragsecurity}.
% An adversary can the retriever as a high-precision oracle to exfiltrate private corpus content via carefully crafted prompts. These prompts usually expand on recent content returned in the last few answers. For example, an attack might take a keyword from a recent answer and query it further~\cite{wang2025silent}, or ask the LLM to elaborate on a just-mentioned fact~\cite{jiang2024rag}.
% \ning{confirm the following}
% By distributing the extraction across numerous queries, the adversary can gradually reconstruct a high-coverage surrogate corpus that retains much of the victim’s utility\ning{what that means?}, thereby undermining trade secrets and intellectual property rights~\cite{lemley2011surprising,pooley2015myth}}\ning{what that means?}.
% To achieve this, an sdversary usually
% expanding on the recent content returned in the last few answers. For example, an attack might take a keyword from a recent answer and query it further~\cite{wang2025silent}, or ask the LLM to elaborate on a just-mentioned fact~\cite{jiang2024rag}.
% They can even perform multi-turn extraction attacks~\cite{cohen2024unleashing,di2024pirates,wang2025silent,jiang2024rag}. By amortizing extraction\ning{what that means?} across numerous queries, an adversary could in principle assemble a high-coverage surrogate corpus that preserves much of the victim’s utility, undermining trade secrets and intellectual property rights\cite{lemley2011surprising,pooley2015myth}.
% Why existing attack methods are insufficient – they lack a notion of overall coverage and thus stall (setting up the gap)
% \begin{figure}[t]
% \centering
% \includegraphics[width=0.9\linewidth]{figs/teaser-2.pdf}
% \caption{Coverage Rate \textit{v.s.} Query Number ($\sim$120) on the TREC-COVID (BGE retriever and Llama 3.1 Instruct-8B generator). Prior attacks stall early (RAGThief~\cite{jiang2024rag} at step 7, IKEA~\cite{wang2025silent} at 104), while RAGCrawler (Ours) steadily increases coverage throughout the query budget.\ning{change: font font size, line label}}
% \label{fig:teaser}
% \vspace{-1.5em}
% \end{figure}
% Despite growing awareness of this threat, existing extraction methods are fundamentally limited by a shortsighted design.
% All recent attacks share a common drawback: they lack a notion of \textbf{global planning}. Instead, they plan myopically, typically by expanding on the recent content returned in the last few answers. For example, an attack might take a keyword from a recent answer and query it further~\cite{wang2025silent}, or ask the LLM to elaborate on a just-mentioned fact~\cite{jiang2024rag}. These shallow heuristic methods can yield initial leakage, but once the immediately related content is exhausted, the attack loses the direction. As a result, it stalls long before the corpus is fully extracted, even though plenty of unexplore information remains to be exfiltrated. Empirically, as illustrated in Fig.~\ref{fig:teaser}, prior attacks plateau early due to losing attacking directions: RAGThief~\cite{jiang2024rag} stops after only 7 queries and IKEA~\cite{wang2025silent} around 100, whereas \toolname continues to exfiltrate new content.
% Core insight and approach – formalizing the attack as a coverage problem and the potential of an adaptive greedy solution, but noting practical challenges.
%The core reason existing methods fail to plan globally lies in RAG’s black-box constraints. Attackers operate through a limited interface: they can only submit queries and receive natural-language answers, with no direct visibility into which documents were retrieved. The returned answers are often paraphrased or partial, so the attacker cannot be certain exactly what part of the corpus was exposed. In other words, it is difficult to estimate what knowledge has already been revealed versus what remains hidden. This lack of feedback makes global coverage planning infeasible for naive approaches. The attacker might want to ``cover'' the entire corpus, but without a way to track progress, they’re essentially crawling in the dark.
% To address this, we introduce a new perspective: corpus-level extraction in RAG can be formalized as an instance of the \textbf{Adaptive Stochastic Coverage Problem (ASCP)}~\cite{nemhauser1978analysis,wolsey1982analysis,golovin2011adaptive}. In this formulation, each query corresponds to an action that probabilistically reveals a hidden subset of the corpus, and the attacker’s objective is to maximize the expected coverage under a fixed query budget. As the underlying coverage function is adaptively monotone and submodular, a greedy policy that selects the query with the highest expected marginal gain at each step admits strong approximation guarantees. This formulation enables a new class of attacks that plan queries based on global coverage estimates.
% To address the limitations of existing methods, our key insight is that global planning can be achieved by formalizing corpus-level extraction in RAG as an instance of the \textbf{Adaptive Stochastic Coverage Problem (ASCP)}~\cite{nemhauser1978analysis,wolsey1982analysis,golovin2011adaptive}. In this formulation, each query can be viewed as an action that probabilistically reveals a subset of hidden documents, and the attacker’s goal is to maximize the expected coverage of the corpus given a limited number on queries. The ASCP is a well-studied class of problems in submodular optimization: if the “coverage function” (fraction of corpus revealed) is adaptively monotone (any new query can only increase coverage) and adaptively submodular (each additional query yields diminishing returns), then a simple greedy strategy that always queries for the largest expected new gain is theoretically near-optimal by achieving a $(1 - 1/e)$ approximation to the best possible plan. This formulation indicates that an attack on RAG, which globally plans queries to maximize expected conditional marginal gain, can approach the theoretical limit of what can be extracted.
In this work, we provide theoretical guarantees for data extraction and systematical approach to analysis the extraction capability of practical adversaries. Our key insight is that these extraction attacks can be formalized as an instance of the \textbf{Adaptive Stochastic Coverage Problem (ASCP)}~\cite{nemhauser1978analysis,wolsey1982analysis,golovin2011adaptive}, where different attacks can be viewed as various strategies for solving the underlying coverage objective.
ASCP is well studied in submodular optimization and the near-optimal strategy exists: if the coverage function, representing the fraction of the corpus revealed, is both adaptively monotone and adaptively submodular, then a greedy policy that, at each step, selects the query with the largest {conditional expected marginal gain (CMG)} is a near-optimal strategy.
In our formulation, each query is treated as an action that stochastically reveals a subset of hidden documents, and the objective is to maximize the expected corpus coverage given a limited query budget. Leveraging results from ASCP, we show that an attack on RAG, which globally plans queries to maximize CMG, can approach the theoretical upper bound on extracting RAG's content.
% denoted $\Delta(q\mid\psi)$, achieves a near-optimal $(1 - 1/e)$ approximation to the optimal strategy.
% In this work, we propose to cast corpus-level extraction in RAG as an instance of the \textbf{Adaptive Stochastic Coverage Problem (ASCP)}~\cite{nemhauser1978analysis,wolsey1982analysis,golovin2011adaptive}. In this view, each query is an action that stochastically reveals a subset of hidden documents, and the attacker aims to maximize expected corpus coverage under a limited number of queries. ASCP is well studied in submodular optimization: if the coverage function (fraction of corpus revealed) is adaptively monotone and adaptively submodular, then a greedy policy that, at each step, selects the query with the largest \emph{conditional expected marginal gain (CMG)}, denoted $\Delta(q\mid\psi)$, achieves a near-optimal $(1-1/e)$ approximation to the best plan. This suggests that an attack on RAG which globally plans queries by maximizing CMG can approach the theoretical limit of extractable content.
% How this paper addresses those challenges – introduce RAGCrawler’s three key ideas (knowledge graph state, semantic-space planning, stealthy query generation) that align with the challenges. This effectively “previews” the design at a high level.
Translating theoretical understanding into real-world attacks is challenging because of the following gaps that arise when instantiating the conditional margin gain (CMG), action space, and budget constraints of ASCP for RAG attacks:
\ding{182}~\textbf{CMG of the attacker’s action is unobservable:} the attacker cannot directly measure the coverage increment produced by a query and subsequent retrievals, since they lack a global view of the document corpus. Consequently, the true CMG is \emph{unobservable} and cannot be computed exactly.
\ding{183}~\textbf{Attacker action spaces are intractable:} the query space $Q$ is effectively unbounded (any natural-language string). Exhaustive search for the CMG-maximizing query is infeasible absent additional structure or strong priors.
\ding{184}~\textbf{Defense of real-word RAG poses its own feasibility constraints:} queries in real-world RAG must remain natural and innocuous to avoid detection or refusal; high-gain intents must be expressed in policy-compliant surface forms that blend into ordinary usage. In addition, as real-world RAG systems often employ advanced query processing techniques such as query rewriting and multi-query retrieval, it becomes even more critical to articulate the intent explicitly to prevent the original meaning from being altered during these processes.%\ning{real-world rag arch}
% Translating the theoretical understanding into real-world attacks is challenging due to the following gaps when instantitating the conditional margin gain (CMG), action space, and budgets for RAG attacks:
% \ding{182}~\textbf{CMG of attacker's action is unobservable:} the attacker cannot directly quantify the proper coverage increments after querying and retrivals, as the attacker does not have a global view of all documents. As a result, the true CMG is \emph{unobservable} and cannot be computed directly.
% \ding{183}~\textbf{Attacker' action spaces are intractable:} the query space $Q$ is essentially unbounded (any natural-language string). Exhaustively searching for the CMG-maximizing query is infeasible without additional structure.
% \ding{184}~\textbf{Feasibility constraints:} queries must remain natural and innocuous to avoid alarms or refusals; high-gain intents must be realized in policy-compliant surface forms that blend into normal usage.
We propose a novel attack framework, \toolname, to address these challenges. \toolname maintains a structured attacker-side knowledge graph (KG) to give the attacker an overview of acquired information. With this global view, our attack executes a three-stage loop.
%
First, KG construction utilizes existing answers and a coverage proxy to map graph growth to an estimate of coverage marginal gain, making it observable.
%
Second, strategy scheduling replaces the intractable search with a two-stage selection process, driven by a score that fuses historical gain with structural exploration.
%
Third, the query generation produces top-ranked anchors into natural, policy-compliant queries with history-aware deduplication.
These three steps together act as an adaptive greedy coverage maximization strategy for \toolname, which guarantees that \toolname achieves near-optimal coverage of the RAG content.
%Via this workflow, we make CMG measurable without observing retrieval sets and keep high-gain intents natural and compliant. while approximating adaptive greedy coverage maximization\ning{I can not parse the last sentence.}.
% , narrow an intractable action space to structured anchors,
% At its core is an efficient procedure for maintaining a structured attacker-side knowledge graph that gives the attacker a bird’s-eye view of information acquired so far. With this global view, our attack, essentially implementing the greedy strategy for ASCP, executes a three-stage loop that turns latent coverage into measurable progress under realistic constraints and overcomes the above challenges:
% \begin{enumerate}[leftmargin=10pt]
% \item Knowledge graph (KG) construction: each answer is parsed into entities and relations to update the attacker-side knowledge graph. We then use a coverage proxy to map graph growth to an estimate of coverage marginal gain, CMG, making CMG observable;
% \item Stratey scheduling: we replace the unbounded natural-language space with a compact semantic action space of anchors (entity--relation pairs) derived from the KG; a balanced estimator that blends optimistic upper-confidence-bound exploration with a gap-aware graph prior assigns each anchor a CMG score and constrains the action space;
% \item Query generation: the top-ranked anchor is rendered into a natural, policy-compliant query with history-aware de-duplication; refusals and empty returns penalize the payoff and steer planning away from unproductive regions, and answers are parsed back into nodes and edges to update KG, thereby closing the loop.
% \end{enumerate}
% We propose a coverage-guided attack framework, \toolname to address these challenges. At the heart of our framework is the idea of efficiently \textbf{maintaining a structured attacker-side knowledge graph} that gives the attacker an bird-view on information learned so far. % ...
% With this global view, the attacks, implementing the greedy strategy for ASCP, runs a three-stage loop that turns latent coverage into measurable progress under realistic constraints.
% In \textbf{State}, each answer is parsed into entities and typed relations to update the attacker-side knowledge graph $S_t$; a coverage proxy on $S_t$ maps graph growth to an estimate of coverage marginal gain, CMG, making CMG observable. In \textbf{Plan}, we replace the unbounded natural-language space with a compact semantic action space of anchors derived from $S_t$ such as entity--relation pairs; a balanced estimator that blends optimistic upper-confidence-bound exploration with a gap-aware graph prior assigns each anchor a CMG score and constrains the action space. In \textbf{Realize}, the top-ranked anchor is rendered into a natural, policy-compliant query with relation-aware templates and history-aware de-duplication; refusals and empty returns penalize the payoff and steer planning away from unproductive regions, and answers are parsed back into nodes and edges to form $S_{t+1}$, closing the loop. This loop makes CMG measurable without observing retrieval sets, narrows an intractable action space to structured anchors, and keeps high-gain intents natural and compliant while approximating adaptive greedy coverage maximization.
%Concretely, for challenge \ding{182} we introduce a knowledge-graph-based coverage model that serves as a proxy for the hidden coverage function. This global state tracks entities and relationships extracted from answers, giving the attacker a partial map of the corpus and highlighting which regions remain unexplored. For challenge \ding{183}, we develop a semantic planner that operates over the structured state. Instead of generating queries blindly in the huge space of sentences, it chooses query targets in the form of graph elements, including entities or topics that look promising, and uses an optimistic heuristic to pick the next query that could yield high new coverage. For challenge \ding{184}, we design a stealth-aware query generator that takes the planner’s target and realizes it as a fluent, context-appropriate question. This generator uses templates and phrasing strategies to ensure the clarity and coherence of queries. Together, these insights enable adaptive global planning that steadily expands corpus exposure while remaining under the radar.
To evaluate the effectiveness of \toolname, we conduct a comprehensive evaluation across four benchmarks, four RAG generators (with safeguard variants), and two RAG retrievers. Results show that \toolname consistently outperforms baselines in coverage rate, semantic fidelity, and reconstruction fidelity. \toolname achieves an average coverage rate of 66.80\%, which is a 20.70\% improvement over the best baseline. Besides, the extracted content is of high quality, enabling the construction of a substitute RAG system that achieves a 0.6992 semantic similarity to the original RAG's responses. \toolname maintains effective against generators with safeguards (reaching an average coverage rate of 55.30\%) and potential defense mechanisms (including query rewriting and multi-query retrieval).
% Code is available at \url{https://anonymous.4open.science/r/RAGCrawler}.
% To validate the efficacy of our proposed attack, we implement \toolname and conduct experiments across four benchmarks and multiple system configurations. Our evaluation includes four RAG generators (with safeguard variants) and two RAG retrievers. The results show that \toolname consistently outperforms previous methods in key metrics like coverage rate, semantic fidelity, and reconstruction fidelity across diverse corpora and RAG setups.
% Specifically, \toolname achieves an average coverage rate of 66.80\%, which is a 20.70\% improvement over the best baseline, alongside an average semantic fidelity of 0.605. This performance underscores the attack's ability to achieve both broad and deep extraction of the knowledge base. Furthermore, the extracted content is of high quality, enabling the construction of a substitute RAG system that achieves a 0.6992 semantic similarity to the original RAG's responses for the same queries. Notably, \toolname maintains strong effectiveness even against RAG generators employing safeguards, such as GPT models, registering an average success rate of 55.3\%. We also demonstrate that advanced RAG defense mechanisms, including query rewriting and multi-query retrieval, fail to mitigate our attack. These findings highlight the inherent limitations of static, query-level defense mechanisms and underscore the urgent need for adaptive, behavior-aware security protocols that can analyze query provenance and user interaction patterns to identify coordinated extraction attempts.
%We implement the attack and show the effectiveness of \toolname through extensive experiments on four benchmarks and several system configurations (with four RAG generators and two RAG retrievers). Across different private corpora and various RAG setups, achieved by varying the retrieval models and LLMs, \toolname consistently outperforms prior attacks in coverage rate, semantic fidelity, and reconstruction fidelity. On average, it achieves a 66.80\% coverage rate, which is 20.70\% higher than the strongest baseline. It also reaches an average semantic fidelity of 0.605,
In summary, our contributions are:
\begin{itemize}[topsep=0pt, leftmargin=1em]
\item We formalize data extraction attack in RAG as RAG Crawling, an adaptive stochastic coverage problem, and provide a rigorous theoretical foundation for the attack. %.\ning{providng theoretical foundation on XXX}
\item We design \toolname, a novel coverage-guided crawler that offers the provable near-optimal attack strategy under real-world constraints.
\item We conduct extensive evaluations across diverse settings to demonstrate the effectiveness of \toolname, as well as against modern RAG defenses.
\end{itemize}
% \begin{itemize}[topsep=0pt, leftmargin=1em]
% \item We formalize corpus-level attack in RAG (RAG Crawling) as an adaptive stochastic coverage problem and ground the attack objective in expected marginal gain with approximation guarantees, thereby providing a rigorous theoretical foundation for this attack class. %.\ning{providng theoretical foundation on XXX}
% \item We design and implement \toolname, a novel coverage-guided crawler that operationalizes our theoretical framework. \toolname maintains a knowledge graph-based global state, plans in semantic space, and realizes natural and compliant queries, approximating a theoretical optimal solution under real-world constriants.
% \item We conduct extensive evaluations across four diverse benchmarks, utilizing four different LLM generators and two distinct retrievers. \toolname show consistent gains gains in corpus coverage rate, as well as in the semantic and reconstruction fidelity of the exfiltrated data.
% \item We evaluate our attack against modern RAG defenses, including LLM generators with built-in guardrails (e.g., GPT) and advanced measures like query rewriting and multi-query retrieval. We show that these contemporary defenses are largely ineffective, thereby exposing critical vulnerabilities in real-world RAG deployments.
% \end{itemize}
% In summary, our main contributions are as follows:
% * **First Formalization of RAG Crawling:** We are the first to provide a rigorous theoretical foundation for corpus-level attacks on RAG systems, a threat we term "RAG Crawling." We formalize this attack as an adaptive stochastic coverage problem and establish a principled objective function grounded in maximizing expected marginal gain, complete with theoretical approximation guarantees.
% * **Novel Coverage-Guided Attack Framework:** We design and implement **\toolname**, a novel coverage-guided crawler that operationalizes our theoretical framework. To overcome the limitations of a black-box setting, \toolname maintains a global state of the extracted knowledge using a dynamic knowledge graph, plans its attack path in the semantic space to maximize information gain, and generates queries that are both natural and compliant, effectively mimicking human behavior to evade simple detection.
% * **Comprehensive and Superior Empirical Results:** We conduct extensive evaluations across four diverse benchmarks, utilizing four different LLM generators and two distinct retrievers. The results unequivocally demonstrate that **\toolname** consistently and significantly outperforms prior art, achieving substantial gains in corpus coverage rate, as well as in the semantic and reconstruction fidelity of the exfiltrated data.
% * **Demonstration of Critical System Vulnerabilities:** We stress-test our attack against modern RAG defenses, including LLM generators with built-in guardrails (e.g., GPT-4) and advanced security measures like query rewriting and multi-query retrieval. We show that these contemporary defenses are largely ineffective, thereby exposing critical vulnerabilities in real-world RAG deployments and highlighting the urgent need for a new class of adaptive, behavior-aware security mechanisms.
\begin{figure*}[h]
\centering
\includegraphics[width=0.8\linewidth]{figs/motivating_example.pdf}
\vspace{-0.5em}
\caption{\textbf{A motivating example.} (a) A data extraction scenario on a private medical corpus. The continuation strategy follows the recent answer’s context, which can drift away from the corpus content; the keyword strategy reuses extracted keywords to form new queries, often yielding redundant information from the same semantic region; our proposed strategy maintains a global graph, detects unrevealed facts and targets them with new queries. (b) Knowledge coverage vs. query number for each strategy on the example corpus. (c) Distribution of retrieved facts in the corpus’ semantic space. Each point is a fact or document, with colors indicating queries from different strategies.}
\label{fig:example}
\vspace{-1.5em}
\end{figure*}
\section{A Motivating Example}
\label{sec:motivating-example}
Consider an attacker targeting a private medical corpus, as shown in Fig.~\ref{fig:example}(a). The corpus contains de-identified records (represented by \icon{figs/emoji/man.png}, \icon{figs/emoji/woman.png}, and \icon{figs/emoji/baby.png}). The attacker engages the RAG system in a dialogue and gradually coaxes out sensitive medical facts about each individual via carefully crafted follow-up queries. Previous results already show that even after removing direct personal identifiers, inference attacks can still re-identify individuals and reveal sensitive attributes when sufficient information is obtained~\cite{narayanan2008robust,sweeney2000simple,cai2025can}. This scenario (though synthetic for illustration) reflects a real threat model: by asking seemingly innocuous questions in sequence, an attacker can piece together confidential information about different people from the underlying private database. As illustrated in Fig.~\ref{fig:concrete-graph}, gaps in knowledge by identifying insufficiently covered areas can be detected by a knowledge graph and then used to generate targeted queries to fill those gaps, delving deeper into the sensitive medical facts related to each individual.%\ning{making a concrete KG }
\begin{figure}[b]
\centering
\vspace{-1.8em}
\includegraphics[width=0.9\linewidth]{figs/concrete-graph.pdf}
\vspace{-0.5em}
\caption{\textbf{Illustration of gap detection by the Global Strategy (Ours).} The strategy identifies from the graph that the patient \icon{figs/emoji/baby.png} has a missing \emph{Personal Medical History}, and guides the next exploration step to complete the knowledge.}
\label{fig:concrete-graph}
\end{figure}
Existing work can be categorized into two types: continuation expansion and keyword expansion.
%
In the first strategy (e.g., RAGThief~\cite{jiang2024rag}), the new query is built on the context of existing answers. The problem of this strategy is that the query can easily drift away from the actual corpus content. For example, in Fig.~\ref{fig:example}(a), the last answer reveals \icon{figs/emoji/baby.png}’s family medical history and mentions the baby’s grandfather \icon{figs/emoji/oldman.png}. A continuation-based attacker will ask about the grandfather next, but since \icon{figs/emoji/oldman.png} is not in the corpus, this line of questioning yields no useful information.
%
A keyword-based strategy (e.g., IKEA~\cite{wang2025silent}) pivots each new query around key terms gleaned from previous answers, but the problem is that it tends to stay within the same semantic neighborhood of the corpus. For example, in Fig.~\ref{fig:example}(a), keyword-based attack repeatedly ask about \icon{figs/emoji/baby.png}'s disease and does not explore new topics.
% \ning{making explaination of other work short}
% Existing multi-turn extraction attacks on RAG systems predominantly follow one of two query expansion strategies: continuation expansion or keyword expansion.
% In a continuation-based strategy (e.g., RAG-Thief~\cite{jiang2024rag}), each new query builds directly upon the context of the existing answers. The attacker essentially ``continues the conversation'', using the model’s most recent output as a springboard for the next query. This approach leverages the corpus’s coherence to dig deeper into a topic. However, a continuation strategy can drift away from the actual corpus content. If the last answer veers into a tangential detail or speculation, a naive continuation will follow that thread, which potentially leads the attack off-course. For example, in Fig.~\ref{fig:example}(a) the last answer reveals \icon{figs/emoji/baby.png}’s family medical history and mentions the baby’s grandfather \icon{figs/emoji/oldman.png}. A continuation-based attacker will ask about that grandfather next, but since \icon{figs/emoji/oldman.png} is not actually in the corpus, this line of questioning yields no useful information.
% A keyword-based strategy (e.g., IKEA~\cite{wang2025silent}) pivots each new query around key terms gleaned from previous answers. After each response, the attacker identifies important ``anchor'' keywords and formulates the next question to center on those terms. This keyword expansion technique keeps queries topically relevant to the information uncovered so far. In practice, however, this strategy tends to stay within the same semantic neighborhood of the corpus. Many questions will repeatedly hit documents related to the facts already extracted.
% These attacks are essentially myopic optimization strategies for ASCP that operate without a global view of extraction progress. To overcome these limitations, we build and continually update a structured global model of the knowledge extracted up to each point in the attack. For instance, a keyword-based attacker will repeatedly issue queries about one particular patient’s symptoms (say focusing on the baby \icon{figs/emoji/baby.png}), with a small chance of asking unexplored topics. The attack achieves high local recall for that one area, but yields diminishing returns as it revisits semantically similar documents without expanding to new topics.
Our work uses \textbf{a growing knowledge graph} to represent discovered entities and relations. A node represents an entity (e.g. a patient, a symptom), and edges capture relations or co-occurrences between entities (for example, patient $\rightarrow$ symptom if a patient is noted to have a certain symptom). As the attacker obtains new answers, the graph is expanded with any newly revealed entities or connections. By maintaining this global knowledge state, the attacker can detect gaps in what has been revealed so far. As illustrated in Fig.~\ref{fig:concrete-graph}, the knowledge graph dynamically reveals incomplete regions of information. For instance, the graph shows that the patient \icon{figs/emoji/baby.png} has no extracted \emph{Personal Medical History}, while the other patients \icon{figs/emoji/man.png} and \icon{figs/emoji/woman.png} already have their medical histories identified. Then the next query will be directed to the uncovered data of \icon{figs/emoji/baby.png}’s unobserved medical history, and expanding overall coverage.
% By detecting this missing connection, the Global Strategy identifies a potential knowledge gap and directs the next query toward uncovering \icon{figs/emoji/baby.png}’s unobserved medical history, thereby expanding the overall knowledge coverage.
% For instance, in Fig.~\ref{fig:concrete-graph} the knowledge graph might show that one patient \icon{figs/emoji/baby.png} has no personal medical history extracted yet, while the other patients \icon{figs/emoji/man.png} and \icon{figs/emoji/woman.png} each have their medical history already revealed. Recognizing this gap, the next query can be explicitly directed to uncover \icon{figs/emoji/baby.png}’s missing medical history.
% This example attacks can be extended to steal general RAG (detailed in Sec~\ref{sec:problem-formulation}), and the resulting attack actually can achieve near-optimal coverage by maintaining an evolving knowledge graph to automatically detect missing entities or links and estimate their potential information gain, thereby selecting the most valuable subsequent query. The near-optimality is proved using ASCP results and also demonstrated in our evaluation results.
%\ning{add how to generalize the example to systematical approach.}
% This global, coverage-aware approach leads to significantly more efficient knowledge extraction from the corpus.
Fig.~\ref{fig:example}(b) shows the knowledge coverage rate v.s. the number of queries for our attack and two baseline strategies. Guided by the global knowledge graph, our attack (green line) achieves a much steeper increase in coverage compared to the continuation-based (orange curve; focusing on unrelated queries) and keyword-based (blue curve; focusing on repetitive queries) strategies.
% The global strategy (green curve) achieves a much steeper increase in coverage compared to the continuation-based (orange curve) and keyword-based (blue curve) strategies. Early on, all methods retrieve some easy ``low-hanging'' facts from the database. However, the continuation attacker soon plateaus as it chases increasingly off-track details, and the keyword attacker yields slowing gains due to its repetitive focus on one area. In contrast, the global method rapidly uncovers diverse facts across the corpus, maintaining momentum by intelligently targeting new gaps identified in the global graph. After a moderate number of queries, the global strategy has extracted substantially more of the knowledge base than either baseline, demonstrating its superior effectiveness in this attack scenario. See Sec.~\ref{sec:evaluation} for more details.
Fig.~\ref{fig:example}(c) provides a qualitative view of how each strategy explores the semantic space of the corpus. In this visualization, each point represents a distinct fact or document. The color represents the strategy that retrieves the document. The black box represents the space of the private corpus. We can observe that, guided by the global knowledge graph, our attack spread widely across the private space and covers many different clusters of information. The continuation-based queries (orange points) follow a narrow path through the space, gradually drifting away from the space center. The keyword-based queries (blue points) form a few dense clusters around specific anchor topics. %More details are illustrated in Sec~\ref{sec:problem-formulation}.
% , and points are colored according to which strategy’s queries retrieved them. The continuation-based queries (orange points) follow a narrow path through the space, gradually drifting away from the central content of the corpus as the conversation veers off-topic. The keyword-based queries (blue points) form a few dense clusters around certain anchor topics, reflecting that this strategy keeps returning to the same well of information. In stark contrast, the global strategy’s queries (green points) are spread widely across the space, covering many different clusters of information. Such comprehensive coverage underscores the advantage of maintaining a global view: the attack agent systematically learns what it doesn’t know yet (by tracking uncovered versus covered knowledge) and actively directs new queries to those unseen regions. In fact, the optimality of the attack we proposed is theoretically guaranteed. The details are illustrated in Sec~\ref{sec:problem-formulation}.
% This motivating example highlights how a global planning approach can substantially improve multi-turn RAG extraction attacks. By maintaining a global knowledge graph and being aware of coverage, the attacker is guided to efficiently map out an entire private knowledge base, where naive strategies would either falter by going off on tangents or linger in loops around already-known information. This serves as the driving force behind our work.
\section{Background}
In this section, we introduce the background, including Retrieval-Augmented Generation Systems (Sec.~\ref{sec:bg-rag}), Adaptive Stochastic Coverage Problem (Sec.~\ref{sec:bg-ac}) and Knowledge Graph (Sec.~\ref{sec:bg-kg}).
\subsection{Retrieval-Augmented Generation Systems}
\label{sec:bg-rag}
\vspace{-2mm}
A Retrieval-Augmented Generation (RAG) system $\mathcal{S}$ typically comprises a generator $\mathcal{G}$ (an LLM), a retriever $\mathcal{R}$, and a document collection $\mathcal{D}$. Given a user query $q$, the retriever $\mathcal{R}$ begins by producing an embedding for $q$ and based on some similarity function (typically cosine similarity), fetching the k most relevant documents:
\begin{equation}\label{equation:rag-retrieve}
\mathcal{D}_k(q) = \arg\text{ top-}k_{d \in \mathcal{D}} \, \text{sim}(q, d).
\end{equation}
where $sim(\cdot)$ represents the similarity function, and $\arg\text{ top-k}$ selects the top-k documents with the highest similarity scores. The generator $\mathcal{G}$ then generates an output based on the contextual information from the retrieved documentsThe generator then produces an answer conditioned on the query and retrieved context \cite{lewis2020retrieval}:
\begin{equation}\label{equation:rag-gen}
a = \mathcal{G}\big(\text{wrapper}(q, \mathcal{D}_k(q))\big),
\end{equation}
where $\text{wrapper}(\cdot,\cdot)$ denotes the system prompt that interleaves $q$ with the retrieved evidence.
In practice, deployed RAG pipelines often incorporate guardrails or enhancements. The most commonly used are \emph{query rewriting} and \emph{multi-query retriever}.
\noindent{\bf Query rewriting}~\cite{lin2020conversational,ma2023query,mo2023convgqr,wang2025maferw} transforms the original query to improve retrievability, resolve ambiguity, repair typos, or strip unsafe instructions. The downstream retrieval and generation processes then operate on the rewrited query.
% \begin{equation}
% \hat{q} = \text{rewrite}(q),
% \label{eq:rewrite}
% \end{equation}
% after which Eqs.~\eqref{equation:rag-retrieve}--\eqref{equation:rag-gen} apply to $\hat{q}$.
\noindent{\bf Multi-query retrieval}~\cite{multiquery,li2024dmqr} generates multiple paraphrases of the original query and performs independent retrieval for each. The retrieved results are then aggregated into a candidate pool. Instead of concatenating all retrieved documents, RAG systems typically re-rank them to select the most relevant ones, with \emph{Reciprocal Rank Fusion (RRF)}~\cite{cormack2009reciprocal} being a widely adopted method. Multi-query retrieval is not a dedicated defense, but its altered retrieval dynamics may affect the attack surface.
% \begin{equation}\label{eq:mq}
% \begin{aligned}
% \mathcal{Q}_r &= \{ q^{(i)} \}_{i=1}^{m} = \text{paraphrase}_m(\hat{q}),\\
% \mathcal{D}_k^{(i)} &= \arg\text{ top-}k_{d \in \mathcal{D}} \, \text{sim}\big(q^{(i)}, d\big), \quad i=1,\dots,m.
% \end{aligned}
% \end{equation}
% Let $\mathcal{C}=\bigcup_{i=1}^{m}\mathcal{D}_k^{(i)}$ denote the de-duplicated pool, and let $r_i(d)$ be the rank of $d$ in list $i$. A commonly adopted practice is to re-rank the aggregated candidates rather than directly concatenating all retrieved documents as context. Among various fusion methods, Reciprocal Rank Fusion (RRF) \cite{cormack2009reciprocal} is widely used, which assigns
% \begin{equation}
% \text{RRF}(d) = \sum_{i=1}^{m} \frac{1}{c + r_i(d)}, \quad c>0.
% \label{eq:rrf}
% \end{equation}
% from which the top-$k$ items are selected to form
% \begin{equation}
% \mathcal{D}_k = \arg\text{ top-}k_{d \in \mathcal{C}} \, \text{RRF}(d),
% \end{equation}
% after which generation proceeds as in Eq.~\ref{equation:rag-gen}.
% %This multi-query+fusion strategy improves recall and reduces brittleness to any single formulation at modest additional cost.
% \ning{do we use these formulae?}
\subsection{Adaptive Stochastic Coverage Problem}
\label{sec:bg-ac}
\vspace{-2mm}
The Adaptive Stochastic Coverage Problem (ASCP) considers scenarios in which each action reveals information in a stochastic manner, and subsequent decisions depend on the observations made so far. In this setting, each action corresponds to selecting a set whose information coverage is uncertain until it is executed. The coverage outcome of the same item can vary depending on the results of previously chosen actions. The objective is to maximize the overall information coverage within limited budgets, such as a limited number of actions.
An ASCP sepcification can formally specified using item universe $\mathcal{U}$, action space \(\mathcal{Q}\), stochastic query outcome \(O(q)\), cost and budget \(c(q), B\), and coverage function \(f(S)\)%\ning{explains without RAG context}.
% \begin{itemize}
% \item {Item universe} (\(\mathcal{U}\)):
% \item {Action space} (\(\mathcal{Q}\)):
% \item {Stochastic query outcome} (\(O(q)\)):
% \item {Cost and budget} (\(c(q), B\)):
% \item {Coverage function} (\(f(S)\)):
% \end{itemize}
\begin{itemize}[leftmargin=10pt]
\item \textbf{Item universe ($\mathcal{U}$)}: The item universe $\mathcal{U}$ represents the set of all items that are available for coverage. These items are the targets for the actions, and the goal is to maximize coverage across this universe.
\item \textbf{Action space ($\mathcal{Q}$)}: The action space $\mathcal{Q}$ consists of all possible actions that can be taken. Each action $q$ corresponds to covering a set of items, but the outcome of this action is uncertain until it is executed.
\item \textbf{Stochastic query outcome ($O(q)$)}: The stochastic query outcome $O(q)$ defines the uncertain result of executing action $q$. It captures the probabilistic nature of the outcome, reflecting how the coverage outcome can vary depending on prior actions.
\item \textbf{Cost and budget ($c(q), B$)}: The cost $c(q)$ represents the resources required to perform action $q$, and $B$ is the total available budget. The objective is to maximize coverage while staying within the budget constraint.
\item \textbf{Coverage function ($f(S)$)}: The coverage function $f(S)$ quantifies the total information coverage achieved by a set of actions $S$. It reflects the degree to which the selected actions have covered items from the item universe $\mathcal{U}$.
\end{itemize}
%The algorithm for ACSP can follow a greedy strategy: at each step, it estimates the expected benefit of every possible action based on the current observations, selects the action with the highest expected gain, observes its actual outcome, and updates the accumulated knowledge accordingly. This process repeats until a stopping condition is reached.
\begin{theorem}[Approximation guarantee of adaptive greedy~\cite{nemhauser1978analysis,wolsey1982analysis,krause2007near,golovin2011adaptive,khuller1999budgeted}]
\label{theorem:adaptive-greedy}
Let $f$ be the coverage function in an instance of the ASCP. Assume that $f$ is adaptively monotone and adaptively submodular, meaning that taking more actions never reduces expected coverage and that each additional action yields diminishing returns.
Let $\pi_{\mathrm{greedy}}$ be the adaptive greedy policy that, at each step, selects the action with the most significant conditional expected marginal gain, and let $\pi^{\star}$ be an optimal adaptive policy under the same number of actions.
Then, the adaptive greedy policy $\pi_{\mathrm{greedy}}$ achieves the approximately optimal expected coverage\footnote{$(1-1/e)\!\approx\!0.63$, i.e., greedy attains at least this fraction of the optimal expected coverage.} under a fixed action budget.
\end{theorem}
% The \emph{Adaptive Stochastic Coverage Problem (ASCP)} considers sequential information acquisition under uncertainty: a decision-maker selects items one by one, observes each item's coverage, and adapts subsequent choices to maximize coverage under resource limits.
% \ning{rewrite the setting, and the greedy algorithms, add figures}
% Let $\mathcal{U}$ be a universe of aspects and $\mathcal{E}$ a set of items. Each item $e\in\mathcal{E}$ has a random state drawn from a prior $\Phi$ and contributes to a coverage function $f(S,\Phi)$ for a selected set $S\subseteq\mathcal{E}$. A history-dependent policy $\pi$ maps observed outcomes to the next action and induces a random set $S^\pi$ with cost $\sum_{e\in S^\pi} c(e)$. Two canonical formulations are the budgeted adaptive maximum-coverage objective
% \begin{equation}
% \max_{\pi}\ \mathbb{E}\!\left[f(S^\pi,\Phi)\right]
% \quad\text{s.t.}\quad
% \mathbb{E}\!\left[\textstyle\sum_{e\in S^\pi} c(e)\right]\le B,
% \label{eq:adaptive-max}
% \end{equation}
% and its dual, the minimum expected cost to reach target coverage $\tau$:
% \begin{equation}
% \min_{\pi}\ \mathbb{E}\!\left[\textstyle\sum_{e\in S^\pi} c(e)\right]
% \quad\text{s.t.}\quad
% \mathbb{E}\!\left[f(S^\pi,\Phi)\right]\ge \tau.
% \label{eq:adaptive-min}
% \end{equation}
% Write $\psi$ for a partial realization (items selected so far with their observed states) and $A$ for its domain. The \emph{conditional expected marginal gain} of $e$ given $\psi$ is
% \[
% \Delta(e\,|\,\psi)\ :=\ \mathbb{E}\!\left[f(A\!\cup\!\{e\},\Phi)-f(A,\Phi)\ \middle|\ \Psi=\psi\right].
% \]
% Function $f$ is \emph{adaptively monotone} if $\Delta(e\,|\,\psi)\ge 0$ for all $e,\psi$, and \emph{adaptively submodular} if conditional marginal gains diminish with more observations, i.e., for any $\psi\subseteq\psi'$ (on the same ground set), $\Delta(e\,|\,\psi)\ge \Delta(e\,|\,\psi')$. Under adaptive monotonicity and adaptive submodularity with modular costs, the \emph{adaptive greedy} rule that at each step selects
% \[
% e^\star \in \arg\max_{e\notin A}\ \frac{\Delta(e\,|\,\psi)}{c(e)}
% \]
% admits tight constant-factor guarantees: a $(1-1/e)$ approximation for \eqref{eq:adaptive-max} and an $O(\log \tau)$ approximation for \eqref{eq:adaptive-min}, generalizing classical (non-adaptive) submodular coverage results to the sequential setting~\cite{nemhauser1978analysis,wolsey1982analysis,krause2007near,golovin2011adaptive,khuller1999budgeted}.
% Applications of this problem include sensor and experiment placement as well as active information gathering under uncertainty~\cite{krause2007near}; budgeted maximum coverage, including document and test selection~\cite{khuller1999budgeted}; and, more broadly, adaptive submodular policies that use feedback to achieve cost-effective coverage~\cite{golovin2011adaptive}.
%We reduce data extraction attack in RAG, i.e.,\emph{RAG Crawling} to the \emph{ASCP} to guide our solution (see Sec.~\ref{sec:problem-formulation}).
We reduce the data extraction attack against RAG systems, which we term {RAG Crawling} to the ASCP to guide our solution (see Sec.~\ref{sec:problem-formulation}).
% (Optional references: Nemhauser et al. 1978; Wolsey 1982; Khuller et al. 1999; Krause & Guestrin 2007; Golovin & Krause 2011.)
\subsection{Knowledge Graph}
\label{sec:bg-kg}
\vspace{-2mm}
Knowledge graph (KG) is a widely-used knowledge representation technology to organize huge amounts of scattered data into structured knowledge. Formally, a KG is a directed, labeled graph $\mathcal{G}=(\mathcal{E},\mathcal{L},\mathcal{R})$, where $\mathcal{E}$ is the set of entities, $\mathcal{R}$ is the set of relation types (edge labels), and $\mathcal{L}\subseteq \mathcal{E}\times\mathcal{R}\times\mathcal{E}$ is the set of labeled, directed edges (facts). Each edge $l\in\mathcal{L}$ is a triple $(h,r,t)$ that connects head $h\in\mathcal{E}$ to tail $t\in\mathcal{E}$ via relation $r\in\mathcal{R}$. For example, $(\text{Python},\ \texttt{instanceOf},\ \text{Programming Language})$.% and $(\text{Paris},\ \texttt{capitalOf},\ \text{France})$.
Compared with unstructured text, KGs provide schema-aware, canonicalized facts with explicit connectivity and constraints~\cite{ji2021survey}. We leverage the knowledge graph to construct an attacker-side state that makes coverage estimable, constrains exploration to a compact semantic action space, and records provenance for robust, auditable updates.
%\ning{where we us the knowledge graph: one sentence.}
% \subsection{Traditional Web Crawling}
% \label{sec:crawling}
% \ning{why do we talk about webCrawling}
% Traditional web crawling is the process of systematically discovering and refreshing content on the open Web under resource and politeness constraints. Formally, the Web can be modeled as a directed graph $G=(V,E)$ with pages $V$ and hyperlinks $E$. A crawler maintains a visited set $U_t\subseteq V$ and a frontier $F_t\subseteq V\setminus U_t$. At step $t$, it selects a URL $u_t\in F_t$, fetches it, and updates
% \begin{equation}
% U_{t+1}=U_t\cup\{u_t\},
% \end{equation}
% \begin{equation}
% F_{t+1}=(F_t\setminus\{u_t\})\ \cup\ \big(\mathrm{Out}(u_t)\setminus (U_t\cup F_t)\big).
% \end{equation}
% Coverage after $t$ steps is naturally measured as $C(t)=|U_t|/|V|$~\cite{olston2010web}.
% Because web pages evolve, crawlers also solve a \emph{revisit scheduling} problem. If page $u$ changes according to a Poisson process with rate $\lambda_u$, then with revisit period $T$ the expected fraction of time the local copy remains fresh is
% \begin{equation}
% \mathrm{Freshness}_u(T)=\frac{1}{T}\int_{0}^{T} e^{-\lambda_u t}\,dt=\frac{1-e^{-\lambda_u T}}{\lambda_u T}.
% \end{equation}
% Such models underlie algorithms for optimal recrawl intervals~\cite{cho2003effective}.
% Our problem (i.e., crawling in RAG systems) shares with web crawling the objective of \emph{budgeted coverage growth}: each step must select the next probe to maximize expected coverage under rate limits. However, crucial differences prevent direct transfer of crawler algorithms. Classical crawlers exploit an explicit hyperlink frontier $\mathrm{Out}(u)$, observe page identities, and estimate change rates $\lambda_u$ for freshness scheduling. In contrast, in RAG crawling the knowledge base is \emph{latent}: queries return paraphrased, noisy answers without explicit document IDs or links. The action space is not a finite set of URLs but an effectively unbounded set of natural-language queries. Moreover, in a black-box RAG, the conditional marginal gain of a probe is unobservable and nonstationary. Therefore, frontier-based BFS, link-based ranking, and revisit policies in traditional crawling cannot be applied in this setting.
% While Web crawling supports search engines by indexing \emph{public} pages, RAG crawling targets \emph{private} or \emph{proprietary} corpora. A successful attacker can reconstruct proprietary knowledge bases by issuing systematic queries, which poses risks far beyond traditional crawling: intellectual property theft, privacy and compliance violations, and persistent leakage of sensitive domain facts once reconstructed~\cite{cohen2024unleashing,jiang2024rag,wang2025silent}.
%\subsection{Knowledge Graph}
% \section{Threat Model and Problem Formulation}
\section{Threat Model}
\label{subsec:threat-model}
\noindent\textbf{Scenario.}
As illustrated in Fig.~\ref{fig:threat-model}, we consider a production RAG service accessed through a black box interface such as an API or chat. Internally, a retriever selects passages or documents from a private corpus $\mathcal{D}$ and a generator composes an answer from the query and the retrieved evidence~\cite{karpukhin2020dense,lewis2020retrieval}. In deployment the generator may rewrite, summarize, or refuse according to guardrails and access policies~\cite{multiquery,beck2025ragqr}. The service does not reveal retrieval scores or document identifiers. The attacker has no insider access and interacts as a normal user. The attacker aims to extract as much of $\mathcal{D}$ as possible while remaining unnoticed by steering retrieval to cover new items across turns.
Prior work has identified two broad classes of such extraction attacks in RAG systems: \emph{Explicit} attacks append overt adversarial instructions to elicit verbatim disclosure of retrieved text~\cite{cohen2024unleashing,jiang2024rag,di2024pirates,zeng2024good}.
\emph{Implicit} attacks use benign-looking queries and aggregate factual content that the system legitimately returns across turns, reconstructing private material via paraphrases and fragments~\cite{wang2025silent}.
Our focus is the implicit regime, which better matches real usage and evades straightforward refusal rules.
\begin{figure}[tp]
\centering
%\vspace{-1em}
\includegraphics[width=0.9\linewidth]{figs/threat_model.pdf}
\caption{\textbf{Threat model of RAG Crawling.}
An external attacker interacts with a production RAG system only via the public interface.
For each query, the retriever selects a set of documents from the private corpus and the LLM generates an answer conditioned on this evidence.
Across turns, the attacker aims to stealthily expand the union of leaked retrieved items under a query budget.}
\label{fig:threat-model}
\vspace{-1.6em}
\end{figure}
% \begin{itemize}[leftmargin=8pt]
% \item \textbf{Explicit extraction attacks.}
% These attacks involve overtly malicious or jailbreak-style prompts. The adversary appends adversarial instructions to queries in an attempt to force verbatim disclosure of retrieved text. A typical explicit attack query takes the form \[
% \text{query} \;+\; \{\text{adversarial command}\},
% \], aiming to trick the generator into outputting raw passages of retrieval set $\mathcal{D}$~\cite{cohen2024unleashing,jiang2024rag,di2024pirates,zeng2024good}.
% \item \textbf{Implicit extraction attacks.}
% In this regime the adversary issues well-formed, seemingly benign queries without explicit jailbreak tokens. The attacker relies on retrieving and aggregating factual fragments returned by the RAG system across multiple non-adversarial turns to reconstruct documents~\cite{wang2025silent}. Such attacks exploit the fact that ordinary, policy-compliant answers can nevertheless reveal information that, when aggregated, reconstructs sensitive content.
% \end{itemize}
% In this paper, we focus on the implicit extraction threat model as the primary concern. Explicit prompt-based attacks, though powerful in principle, can often be trivially mitigated by standard large language model (LLM) safeguards (e.g. refusal policies and content filters) in production systems~\cite{wang2025silent}. By contrast, implicit attacks operate under the guise of normal usage and thus evade straightforward detection or rule-based refusals. Focusing on implicit extraction is therefore both practical and pressing.
\noindent\textbf{Adversary's Goal.}
The attacker aims to exfiltrate the private knowledge base $\mathcal{D}$ as completely as possible while avoiding detection.
Leakage may appear as verbatim text, paraphrases, summaries, or factual statements that reflect retrieved evidence.
%
For each round $t$, the adversary submits a query $q_t$. In response, the RAG system internally consults a hidden set of documents $\mathcal{D}_k(q_t) \subseteq \mathcal{D}$ to generate its answer.
The adversary's goal is to maximize the number of unique documents retrieved over a sequence of $B$ queries, thereby learning about the system's underlying knowledge base. This budget $B$ models real-world constraints such as API costs and the need to evade detection. This objective is formally expressed as the following:
% Information can be revealed whenever new items are retrieved even if document IDs are never shown.
% A natural measure of progress is the breadth of retrieval under a budget of $B$ queries. This budget models real-world constraints such as API costs and the need to evade detection, forcing the attacker to optimize their query strategy.
% Accordingly, we adopt a \emph{coverage}-based formulation of the adversarial objective\cite{jiang2024rag,wang2025silent,di2024pirates}\ning{I can not parse this}:
\begin{equation}\label{eq:coverage}
\max_{\{q_1, \cdots, q_B\}}\frac{\left| \bigcup_{t=1}^{B} \mathcal{D}_k(q_t) \right|}{|\mathcal{D}|}
\end{equation}
%Higher coverage means that more of $\mathcal{D}$ has supported the answers and thus may have been revealed regardless of wording.
Stealth remains a parallel objective: each $q_t$ should look natural and innocuous to avoid detection or refusal.
% The adversary’s primary aim in the explicit setting is to provoke verbatim reproduction of content from $\mathcal{D}_k$. In the implicit setting, the model’s outputs need not be literal copies, but they may nevertheless disclose information about $\mathcal{D}_k$ via factual statements or paraphrases. In both cases, success is measured by the extent to which the private corpus $\mathcal{D}$ has been “touched” or exposed through retrieval-conditioned responses.
% Accordingly, we adopt a \emph{coverage}-based formulation of the adversarial objective that applies to both explicit and implicit extraction attacks~\cite{jiang2024rag,wang2025silent,di2024pirates}:
% \begin{equation}\label{eq:coverage}
% \mathrm{Coverage} \;=\; \frac{\bigl| \bigcup_{t\in [1, B]} \mathcal{D}_k^t \bigr|}{|\mathcal{D}|}
% \end{equation}
% where $\mathcal{D}_k^{t}$ denotes the subset of the private corpus retrieved in round $t$, and $B$ is the total query budget.
% Beyond maximizing coverage, a secondary but equally important adversarial objective is \emph{stealth}: the attacker must avoid detection or triggering defenses. To this end, each query $q_t$ should appear natural and innocuous so that the attack blends into normal usage patterns.
\noindent\textbf{Adversary’s Capabilities.}
We assume a black box setting.
The attacker submits adaptive natural language queries $q_t$ and observes only final answers $a_t$.
They cannot see retrieval scores, document identifiers, or intermediate states, and they cannot read internal indices or modify retriever weights.
They do not poison the target corpus and do not use privileged operator tools.
Following prior work~\cite{cohen2024unleashing,jiang2024rag,wang2025silent}, the attacker knows only the topic of the target knowledge base, such as a keyword or short phrase that can be inferred from public materials or light probing.
Under these constraints, leverage comes solely from crafting query sequences and interpreting observed answers.
% We consider a black-box setting in which the adversary has no access to the internal parameters, architecture, or private knowledge base of the victim RAG system. The attacker can submit adaptive natural-language queries $q_t$ and observe only the final textual answers $a_t$, not retrieval scores, document identifiers, or intermediate states. The adversary cannot read internal indices, modify retriever weights, or directly inspect the exact retrieval sets $\mathcal{D}_k^t$ unless such information is inadvertently leaked in a response. We also exclude attacks that poison the target corpus or rely on privileged operator interfaces.
% %
% We assume that the adversary knows the \emph{topic} of the target RAG knowledge base, which may take the form of a keyword or short phrase. This assumption is reasonable in practice ~\cite{cohen2024unleashing,jiang2024rag,wang2025silent} because products often disclose high-level domains in public materials such as documentation or marketing pages. Dataset descriptions, URLs, or taxonomy labels may also reveal thematic cues, and preliminary probing without privileged access usually suffices to infer a coarse topical scope. Under these constraints, the adversary’s leverage comes solely from crafting query sequences and interpreting observed answers.
\section{Reducing RAG Crawling to ASCP}
\label{sec:problem-formulation}
We now explain how we model the data extraction in RAG, RAG Crawling, as {ASCP} (introduced in Sec.~\ref {sec:bg-ac}). This formal mapping clarifies how we can potentially apply the greedy coverage-maximizing approach in our setting:
\begin{itemize}[leftmargin=10pt]
\item \textbf{Item universe ($\mathcal{U}$)}: We take $\mathcal{U}$ to be the set of atomic documents or passages in the private corpus $\mathcal{D}$. Essentially, the ``items'' to be covered are the documents in $\mathcal{D}$.
\item \textbf{Action space ($\mathcal{Q}$)}: We equate the set of possible actions $\mathcal{Q}$ with the set of all valid queries the
attacker could ask might pose. In theory $\mathcal{Q}$ is infinite (any string could be a query), but we will address how to search it later. For now, conceptually, each $q \in\mathcal{Q}$ is a possible question the attacker can pose to the target system.
\item \textbf{Stochastic query outcome ($\Phi$)}: When the attacker issues a particular query $q$, the RAG system will retrieve some set of documents $\mathcal{D}_k(q) \subset \mathcal{D}$ (Sec.~\ref{sec:bg-rag}) and then produce an answer. We model this retrieval result as a random subset drawn from a distribution $\Phi(\cdot \mid q)$. Essentially, $\Phi$ encapsulates our uncertainty about what part of $\mathcal{D}$ will be exposed by query $q$.
\item \textbf{Cost and budget}: We assume each query has a unit cost (e.g., each question counts as 1 against the budget $B$). This is a reasonable simplification: perhaps some queries could be considered ``more costly'' if they are longer or trigger more computation, but in our threat model the primary cost is just the opportunity cost of using up one of the limited queries we can send. Thus, a budget $B$ simply means the attacker is limited to $B$ queries in total.
\item \textbf{Coverage function}: We define $f(S, \Phi)$ as the fraction of unique documents in $\mathcal{D}$ that have been retrieved at least once by the set of queries $S$. Since the attacker cannot directly observe $\mathcal{D}_k(q)$, this is a latent coverage measure.
\end{itemize}
Under this reduction, the attacker is effectively an adaptive agent who, at each step $t = 1, 2, \dots, B$, selects a query $q_t \in \mathcal{Q}$ based on the interaction history so far, aiming to maximize the overall expected coverage $f(S, \Phi)$ by the end of $B$ queries.
We now define the notion of adaptive marginal gain under the RAG Crawling model and show that it satisfies the core assumptions required for approximate greedy optimization in {ASCP}.
\begin{definition}[Conditional Marginal Gain in RAG Crawling]
Let $S = \{q_1, \dots, q_t\}$ denote the set of queries issued so far, and let $\psi$ denote the observable interaction history, that is, the sequence of issued queries and their corresponding observable system responses such as generated answers. Let $\Phi$ represent the underlying stochastic retrieval process that maps queries to latent sets of retrieved documents. The {conditional marginal gain} of issuing a new query $q \in \mathcal{Q}$ given history $\psi$ is defined as
\begin{equation}
\Delta(q \mid \psi) := \mathbb{E}_{\Phi} \left[
f(S \cup \{q\}, \Phi) - f(S, \Phi)
\;\middle|\; \psi
\right],
\label{eq:delta-query}
\end{equation}
%where $f(S, \Phi)$ is the coverage function measuring the fraction of unique documents in $\mathcal{D}$ retrieved at least once by the set of queries $S$ under retrieval process $\Phi$.
\end{definition}
\begin{theorem}[Adaptive Monotonicity and Submodularity in RAG Crawling]
\label{theorem:monotonicity-submodularity}
The RAG Crawling problem instance satisfies adaptive monotonicity and {adaptive submodularity}.
\end{theorem}
The proof can be found in Appendix~\ref{appendix:proof}.
Building on Theorem~\ref{theorem:adaptive-greedy} and Theorem~\ref{theorem:monotonicity-submodularity}, we have the following:
\begin{theorem}[Near-optimality guarantee of \toolname]
\label{theorem:adaptive-greedy-rag}
The adaptive greedy policy for RAG Crawling, which at each step $t$ selects a query
\begin{equation}
q_t \in \arg\max_{q \in \mathcal{Q}} \Delta(q \mid \psi_{t-1}),
\label{eq:greedy-policy}
\end{equation}
achieves an approximately optimal expected coverage.
\end{theorem}
%\ning{write this as a theorem, formally saying that our scheme is approximated optimal}
While this theorem provides a principled optimization target, implementing an adaptive greedy strategy in practice is far from trivial. Several challenges arise when translating the theoretical ASCP framework into a practical attack against real-world RAG systems:
\begin{itemize}[leftmargin=12pt, topsep=4pt]
\item[\ding{182}] \textbf{The CMG is unobservable.} The attacker cannot directly observe the true coverage gain of a query, as the retrieved document sets remain hidden and inaccessible.
\item[\ding{183}] \textbf{The attacker's action space is intractable.} The action space $\mathcal{Q}$ includes all valid natural language strings, making it effectively infinite. Without additional structure or priors, exhaustive search for high-gain queries becomes computationally infeasible.
\item[\ding{184}] \textbf{Real-world RAG systems's defense.} Queries must appear natural and comply with safety policies to avoid refusal, rejection, or detection by safety filters.
\end{itemize}
%The optimization objective aligns with Eq.~\ref{eq:adaptive-max} .
% \ning{write the following as two theorems. }
% \noindent\textbf{Conditional Expected Marginal Gain.} Following this formulation, we inherit the definitions of conditional expected marginal gain.
% Let $\psi = \{(q_1, a_1), \dots, (q_t, a_t)\}$ denote the interaction history so far. The conditional expected marginal gain of issuing query $q$ is:
% \begin{equation}
% \Delta(q \mid \psi) := \mathbb{E}_\Phi \left[
% f(S^\pi \cup \{q\}, \Phi) - f(S^\pi, \Phi)
% \ \middle|\ \Psi = \psi
% \right],
% \label{eq:delta-query}
% \end{equation}
% where $S^\pi$ is the query set so far under policy $\pi$.
% \noindent\textbf{Properties.}
% This problem instance satisfies the two standard properties of \emph{ASCP}:
% \begin{itemize}[leftmargin=10pt]
% \item {Adaptive Monotonicity.} For any history $\psi$ and query $q$, $\Delta(q \mid \psi) \ge 0$: a new query cannot reduce coverage.
% \item {Adaptive Submodularity.} For any $\psi \subseteq \psi'$, we have $\Delta(q \mid \psi) \ge \Delta(q \mid \psi')$: marginal gains diminish as more of the corpus is already covered.
% \end{itemize}
% These properties guarantee that an \emph{adaptive greedy} policy that selects
% \begin{equation}
% q_t \in \arg\max_{q \in \mathcal{Q}} \Delta(q \mid \psi_{t-1}) \label{eq:greedy-policy}
% \end{equation}
% achieves a $(1 - 1/e)$ approximation to the optimal policy under budget $B$~\cite{golovin2011adaptive}.
% Therefore, if we had a way to compute or estimate $\Delta(q \mid \psi)$ for candidate queries, a greedy strategy that always picks the query with the largest estimated marginal gain would be a natural approach. This is the ideal that RAGCrawler strives for: approximate the adaptive greedy policy for coverage maximization despite not observing $\mathcal{D}_k(q)$ directly.
% In summary, by framing the problem as an ASCP, we gain a principled goal (maximize expected coverage)
% and a candidate strategy (adaptive greedy selection). The next section discusses why existing attacks fail in these terms and how our key insight will make this strategy attainable.
% \noindent\textbf{Practical Challenges.}
% Two obstacles prevent direct application of greedy optimization:
% \begin{itemize}[leftmargin=12pt]
% \item \textbf{Latent feedback.} The retrieval sets $\mathcal{D}_k^t$ are not observable. Model outputs $r_t$ are paraphrased, filtered, or summarized, and thus cannot directly reveal $\Delta(q \mid \psi)$.
% \item \textbf{Infinite query space.} The action space $\mathcal{Q}$ comprises all natural-language queries. Maximizing over $\mathcal{Q}$ is computationally infeasible.
% \end{itemize}
% We address these challenges in subsequent sections by developing approximate estimators and selection strategies.
% The problem of crawling in RAG can be reduced to an \textit{Adaptive Coverage} problem. We adopt the RAG pipeline from Sec.~\ref{sec:bg-rag} (Eq.~\ref{equation:rag-gen}) and the coverage-based adversarial goal from Sec.~\ref{subsec:threat-model} (Eq.~\ref{equation:coverage}).
% Let the \emph{action space} be the set of all admissible natural-language queries
% $\mathcal{Q}$.
% At round $t$, the attacker chooses $q_t\in\mathcal{Q}$, the system responds with $r_t$ via Eq.~\ref{equation:rag-gen}, and the retriever (latently) selects a set $\mathcal{D}_k^t$.
% We write the \emph{touched set} as the latent retrieval set itself:
% \[
% \mathcal{T}_t \;=\; \mathcal{D}_k^t \subseteq \mathcal{D} \quad \text{(latent, not exposed to the caller)}.
% \]
% Let the cumulative (latent) state be
% \begin{equation}
% \label{eq:lat-union}
% \mathcal{U}_t \;=\; \bigcup_{i=1}^{t} \mathcal{T}_i, \qquad \mathcal{U}_0=\varnothing,
% \end{equation}
% and define the normalized utility (coverage)
% \begin{equation}
% \label{eq:coverage-utility}
% f(\mathcal{U}_B)\;=\;\frac{|\mathcal{U}_B|}{|\mathcal{D}|}.
% \end{equation}
% By construction, $f(\mathcal{U}_B)$ coincides with the coverage-based adversarial objective in Eq~\ref{equation:coverage}.
% \noindent\textbf{Properties.}
% For history $H_{t-1}=\{(q_iry_i)\}_{i=1}^{t-1}$ (implying $\mathcal{U}_{t-1}$) and a candidate query $q$, define the conditional marginal gain as the expected increase in latent coverage:
% \begin{equation}
% \label{eq:cmg}
% \Delta(q\mid H_{t-1})
% \;=\;
% \mathbb{E}\!\left[
% \frac{\big|\big(\mathcal{U}_{t-1}\cup \mathcal{T}(q)\big)\setminus \mathcal{U}_{t-1}\big|}{|\mathcal{D}|}
% \ \middle|\ H_{t-1}
% \right],
% \end{equation}
% where $\mathcal{T}(q)$ denotes the latent retrieval set that would be selected if issuing $q$ at the next round (again not observable to the attacker).
% This problem instance satisfies two standard adaptive-coverage properties:
% \begin{itemize}[leftmargin=8pt]
% \item\textbf{Adaptive Monotonicity.}
% For any history $H_{t-1}$ and candidate query $q$, $\Delta(q \mid H_{t-1}) \ge 0$.
% Intuitively, an additional query cannot reduce the union of already retrieved documents—at worst it retrieves only previously seen items (zero gain), at best it adds new ones.
% \item\textbf{Adaptive Submodularity.}
% For any $H\subseteq H'$ implying $\mathcal{U}\subseteq \mathcal{U}'$, we have $\Delta(q\mid H)\ge \Delta(q\mid H')$.
% As more of the corpus has already been retrieved, the chance that the same query uncovers \emph{new} items diminishes (diminishing returns).
% \end{itemize}
% Therefore, the expected coverage $\mathbb{E}[f(\mathcal{U}_B)]$ is adaptive-monotone and adaptive-submodular. By~\cite{nemhauser1978analysis}, the greedy policy
% \(
% q_t \in \arg\max_{q\in\mathcal{Q}} \Delta(q\mid H_{t-1})
% \)
% achieves a $(1-1/e)$ approximation \footnote{$\mathrm{e}$ is Euler's number, the base of the natural logarithm, $\mathrm{e}\approx 2.71828$.} to the optimal policy under a cardinality budget $B$.
% \noindent\textbf{Practical obstacles.}
% However, plain greedy is not directly applicable. In practice due to two deployment-dominant challenges:
% \begin{enumerate}[leftmargin=12pt]
% \item \textbf{Unobservable marginal gains.} $\Delta(q\mid H)$ is not directly observable: the pipeline (Sec.~\ref{sec:rag}) does not expose the latent retrieval sets $\mathcal{D}_k^t$ or their distributions, and answers may be paraphrased, summarized, or filtered, making naive estimators brittle.
% \item \textbf{Unbounded action space.} The query space $\mathcal{Q}$ is effectively infinite (all natural-language strings). Therefore, exact greedy over $\mathcal{Q}$ is computationally infeasible even if $\Delta(\cdot\mid\cdot)$ were known.
% \end{enumerate}
%\noindent We address these obstacles in this paper.
% \begin{figure*}[t]
% \centering
% \includegraphics[width=0.8\linewidth]{figs/overview.pdf}
% \caption{\textbf{Workflow of \toolname.}
% \textbf{Top:} In the \emph{Seed Phase}, $m$ diversified seeds $\mathcal{Q}_0$ are sent to the \emph{Victim RAG}; the \emph{KG-Constructor} converts responses $\mathcal{A}_0$ into the initial attacker-side surrogate state $S_0$.
% At round $t$, \toolname maintains $S_t$, issues $q_t$, receives $a_t$, and updates the state.
% \textbf{Bottom:} (1) \emph{KG-Constructor}: maintains an attacker-side KG state from RAG responses and reveals uncovered regions.
% (2) \emph{Strategy Scheduler}: ranks entities with a balanced score and selects the anchor $(e_t^{*}, r_t^{*})$.
% (3) \emph{Query Generator}: converts the chosen anchor into a stealthy natural-language query.}
% \label{fig:workflow}
% \vspace{-1.5em}
% \end{figure*}
% \section{Design}
% \label{sec:design}
% To address these challenges, our core ides is to maintain an \textbf{evolving knowledge graph}, as outlined in Sec.~\ref{sec:motivating-example}.
In next section, we explain how \toolname, a black-box crawling framework designed, reconstructs the latent knowledge base of a RAG system under a fixed query budget.
Grounded in the Adaptive Stochastic Coverage Problem (ASCP) formulation,
\toolname approximates the theoretical greedy coverage-maximization strategy by maintaining an evolving knowledge-graph state,
identifying high-gain anchors in the semantic space of this state, and realizing them through stealthy queries.
\subsection{Overview}
\label{sec:overview}
\vspace{-1mm}
In this section, we explain the operational workflow of \toolname, illustrated in Fig.~\ref{fig:overview}.
The attacker begins by issuing an initial query~$q_0$ to the victim RAG system and receiving the corresponding response~$a_0$.
Upon receiving a query, the RAG retrieves a hidden set of documents from its private corpus and synthesizes an answer based on the retrieved evidence.
Although the retrieved content is not directly observable, the generated response inevitably reveals partial fragments of the underlying corpus.
The \textbf{KG-Constructor} module parses~$a_0$ into structured triples and integrates them into the initial attacker-side KG~$\mathcal{G}_0$, which serves as an observable approximation of the latent corpus coverage.
Based on this state, the \textbf{Strategy Scheduler} estimates the CMG over candidate entity–relation anchors in~$\mathcal{G}_0$, selects the most promising anchor~$Anc_0$, and forwards it to the \textbf{Query Generator}.
The \textbf{Query Generator} realizes this anchor as a fluent and policy-compliant query~$q_1$, which is then sent to the RAG system to obtain the next response~$a_1$.
The new answer~$a_1$ is subsequently processed by the \textbf{KG-Constructor} to update the knowledge graph to~$\mathcal{G}_1$.
Generalizing this process, at iteration~$t$ the attacker issues~$q_t$, receives~$a_t$, and updates the state~$\mathcal{G}_t$ through the same sequence of modules, forming a closed adaptive loop of \emph{state estimation}, \emph{strategic planning}, and \emph{query realization}.
This adaptive loop systematically addresses the three challenges of the ASCP framework. Next, we elaborate on the specific mechanisms employed for each challenge.
\begin{figure}
\centering
\includegraphics[width=0.6\linewidth]{figs/overview.pdf}
\caption{\textbf{The workflow of \toolname.} At each step, (1) The KG-Constructor processes the latest system response to update the knowledge graph. (2) The Strategy Scheduler analyzes this graph to select a strategic anchor. (3) The Query Generator then uses this anchor to formulate the next query, completing the loop.}
\label{fig:overview}
\vspace{-1.8em}
\end{figure}
% This adaptive loop directly addresses the three challenges identified in the ASCP framework.
% The \textbf{KG-Constructor} transforms unobservable retrieval feedback into a structured proxy (KG), providing a semantic substrate on which the CMG can be estimated.
% Building upon this graph-induced semantic space, the \textbf{Strategy Scheduler} performs CMG estimation and structured action planning, effectively constraining the otherwise intractable query space.
% Finally, the \textbf{Query Generator} ensures feasibility and stealth by realizing the selected anchors as fluent and compliant natural-language queries.
% the \textbf{KG-Constructor} renders latent feedback observable and enables CMG estimation;
% the \textbf{Strategy Scheduler} constrains the unbounded query space by planning in a compact semantic action space;
% and the \textbf{Query Generator} ensures feasibility and stealth by realizing anchors as fluent, compliant surface forms.
% Detailed designs for each module are presented in the following subsections.
% Figure~\ref{fig:overview} illustrates the \toolname pipeline, which implements a greedy adaptive strategy under ASCP. The system begins with a seed phase (issuing one or more initial queries to form $\mathcal{G}_0$) and then repeats a loop of \emph{state--plan--realize}. In each iteration, the KG-Constructor updates the attacker’s knowledge graph $\mathcal{G}_t$; the Strategy Scheduler selects a high-gain entity–relation anchor based on $\mathcal{G}_t$; and the Query Generator issues a natural-language query for that anchor. The only interface between our \toolname and private corpus is the query and answer issued by the RAG system. Each query can reveal some documents in the RAG xxx. By maintaining $\mathcal{G}_t$ as a global view of discovered content, \toolname turns the latent coverage objective into a measurable proxy.
% \noindent \textbf{KG-Constructor.} Addresses the challenge of \textit{latent feedback} by constructing a visible approximation of the latent coverage function $f(S^\pi, \Phi)$.
% Based on the insight of \emph{knowledge-graph-based coverage modeling},
% it maintains a normalized attacker-side coverage proxy ${\mathcal{G}}_t$ as a schema-guided knowledge graph.
% The module incrementally extracts entity–relation triples from responses and merges aliases into canonical forms,
% providing the evolving state required for CMG estimation.
% \noindent \textbf{Strategy Scheduler.}
% Implements the core \emph{adaptive greedy} selection rule of ASCP by estimating conditional marginal gains (CMG) for candidate (entity, relation) pairs.
% It addresses the challenge of an \emph{unbounded query space} by following the insight of \emph{planning in semantic space}.
% Each candidate is scored with a CMG proxy that combines historical payoff with graph-structural priors, and the top-ranked anchor $(e_t^*, r_t^*)$ is selected for the next probe by the pricinple of \emph{greedy policy}.
% \noindent \textbf{Query Generator.}
% Addresses the \textit{stealth and usability} challenge by realizing each selected anchor into a fluent, natural-language query.
% Following the insight of \emph{stealth-aware realization}, it generates surface queries that are semantically aligned with the selected intent
% while avoiding refusals or detection through relation-aware templates, meta-generation, and history-aware de-duplication.
% The overall workflow is illustrated in Fig.~\ref{fig:overview}. At each step xxx. \toolname begins with a \emph{Seed Phase}, where $n$ diversified queries $\mathcal{Q}_0$ are issued to the \emph{Victim RAG}, and responses $\mathcal{R}_0$ are converted into the initial attacker-side state ${\mathcal{S}}_0$. After seeding, the system enters a closed-loop \textbf{state–plan–realize} cycle\ning{rewrite and refer to the figures}, where each step is implemented by one of three coordinated modules:
\subsection{KG-Constructor}
\label{sec:design-kg}
\vspace{-1mm}
This module addresses the first challenge in the {ASCP} formulation: \emph{the CMG of the attacker’s action is unobservable}.
Since the attacker cannot directly observe which documents have been retrieved by the RAG system, the true coverage increment of each query remains hidden.
The goal of this module is therefore to construct and maintain an evolving KG~$\mathcal{G}_t$ that approximates the latent coverage function~$f(S^\pi, \Phi)$, making CMG estimable from surface-level answers. The workflow is shown in Fig.~\ref{fig:kg-constructor}.
\noindent{\bf Why existing approaches do not work.} Existing KG construction methods are ill-suited for our dynamic setting. Traditional techniques such as OpenIE and schema-constrained extraction~\cite{yates2007textrunner, stanovsky2018supervised}, rely on fixed relation inventories and curated ontologies, assuming stable data access and consistent syntax. Recent LLM-based approaches~\cite{papaluca2024zero, bian2025llm} show promise for adaptive graph construction but conflicts with the need for efficient incremental updates that do not require full access to the underlying data.
Furthermore, graph-augmented RAG frameworks such as GraphRAG~\cite{edge2024local} and LightRAG~\cite{guo2024lightrag} use LLMs to extract triples from documents within the corpus and leverage the graph to enhance retrieval. These frameworks primarily focus on optimizing retrieval quality by merging exactly identical entities and relations. While this improves retrieval efficiency, it neglects the need for compact and efficient coverage tracking across multiple rounds of dynamic query responses.
\noindent{\bf Our methods.} We develop a KG-Constructor that emphasizes efficient incremental updates.
Instead of requiring access to the entire dataset or repeatedly rebuilding the global graph, we maintain the KG in a lightweight manner by constructing a subgraph at each step and merging
it into the existing graph. As illustrated in Fig.~\ref{fig:kg-constructor}, the workflow consists of three main modules:
% 1) Topic-Specific Prior,
% 2) Iterative Extraction and Reflection, and
% {3) Incremental Graph Update and Semantic Merging}.
%The procedures are detailed as follows.
\noindent\textbf{1) Topic-Specific Prior.}
We begin by establishing a topic-specific prior that defines the contextual focus of the constructed KG.
This prior acts as a soft constraint on the LLM’s extraction process, guiding it toward topic-relevant entities and relations while filtering out off-topic or noisy content.
By narrowing the semantic search space, it enables more targeted and consistent knowledge acquisition.
%
Concretely, we prompt the LLM to infer a compact set of {entity categories} and {relation types} that characterize the target domain.
For example, in a medical corpus, entity categories may include \textit{disease}, \textit{symptom}, and \textit{treatment}, while relation types may include \textit{has\_symptom}, \textit{treated\_by}, and \textit{caused\_by}.
During extraction, the LLM is instructed to prioritize these schema elements when forming structured triples, ensuring that all generated relations remain within the topical boundary.
%
This bounded, schema-guided design offers two key advantages.
First, it stabilizes relation typing and prevents the introduction of spurious or semantically inconsistent edges, maintaining structural coherence across incremental updates.
Second, it keeps the prompt context compact and computationally efficient: the LLM does not need to reference all existing nodes and relations in the graph, but only the abstract schema of entity and relation types as contextual constraints.
This design greatly reduces prompting overhead, enabling scalable LLM calls while preserving high topical precision in the extracted knowledge.
\begin{figure}[t]
\centering
\includegraphics[width=0.7\linewidth]{figs/h_kg_constructor.pdf}
\caption{\textbf{The workflow of KG-Constructor.} Guided by a Topic-Specific Prior, an Iterative Extraction and Reflection process generates a knowledge subgraph ($\Delta\mathcal{G}_t$), which is then refined through Incremental Graph Update and Semantic Merging to produce the final graph state, $\mathcal{G}_t$.}
\label{fig:kg-constructor}
\vspace{-1.5em}
\end{figure}
\noindent\textbf{2) Iterative Extraction and Reflection.} This component uses the attacker-side LLM to process the new answer. To reduce missing edges caused by conservative extraction, we
adopt a multi-round extraction–reflection procedure. As shown in Fig.~\ref{fig:kg-constructor}, each answer is first processed by an ``extraction'' pass, and then reprocessed through a ``reflection'' pass that revisits the same content with awareness of previously omitted facts. This reflection loop enables the LLM to infer implicitly expressed relations and produce additional
triples, forming a more comprehensive knowledge subgraph $\Delta\mathcal{G}_t$.
\noindent\textbf{3) Incremental Graph Update and Semantic Merging.}
After obtaining the new knowledge subgraph~$\Delta\mathcal{G}_t$, it is incrementally integrated with the existing graph~$\mathcal{G}_{t-1}$ to form the updated graph~$\mathcal{G}_{t}$.
This integration is implemented as a lightweight merging operation without rebuilding graph from scratch.
To ensure that the graph structure reflects genuine semantic expansion rather than redundant surface-level variations, we perform a semantic merging stage offline.
All entity mentions and relation phrases are encoded into a shared embedding space using a pre-trained encoder.
Node or edge pairs whose cosine similarity exceeds a threshold~$\tau$ are automatically identified and merged.
This joint reconciliation of entities and relations consolidates semantically equivalent concepts, normalizes synonymy across updates, and maintains a compact yet expressive knowledge graph. This provides a stable basis for CMG estimation and planing space for {Strategy Scheduler} (Sec.~\ref{sec:design-sched}).
% We develop a KG construction that emphasizes efficient incremental updates. Instead of requiring access to the entire dataset or repeatedly rebuilding the global graph, we maintain the KG in a lightweight manner by constructing a subgraph at each step and lightly merging it into the existing graph. By focusing on local, incremental changes based on newly received answers, and tracking incremental coverage gain, our method enables an effective proxy for CMG estimation.
% The workflow starts with a topic-specific prior, an attacker-side
% LLM performs iterative extraction and reflection loops to obtain new knowledge $\Delta\mathcal{G}_t$,
% which is then combined with the existing graph $\mathcal{G}_{t-1}$ to produce an updated graph$\mathcal{G}_{t}$. A final embedding-based semantic merging step consolidates redundant or
% synonymous nodes and edges. In the following sections, we describe the key components of our method in detail, including the 1) {Topic-Specific Prior}; 2) {Attacker-Side LLM for Extraction and Reflection} ; and 3){Incremental Graph Update. The xxx is in Fig.~\ref{fig:kg-constructor}.
% \noindent\textbf{Topic-Specific Prior.} We begin by establishing a prior contextual focus for the graph. This topic-specific
% prior guides the LLM’s extraction towards relevant facts and relations. By biasing the process to a chosen topic, we
% reduce off-topic noise and narrow the search space. Specifically, we leverage the LLM to infer some topic-specific entity categories to constrain extraction and provide the LLM with the current \emph{relation schema}. This is feasible because the number of relation types for a given topic is typically small, such as \textit{symptom}, \textit{treatment}, and \textit{comorbidity} for a medical corpus.
% \noindent\textbf{Preprocessing.}
% We first xxx LLM to the topic and let it specify topic-specific entity categories to constrain extraction and provide the LLM with the current \emph{relation schema}.
% This is feasible because the number of relation types for a given topic is typically small, such as \textit{symptom}, \textit{treatment}, and \textit{comorbidity} for a medical corpus.
% The LLM extractor is prompted to prioritize selecting relations from this schema when identifying structured triples.
% This bounded design stabilizes relation typing and prevents the introduction of noisy or spurious relations, while keeping the prompt context compact enough for efficient LLM calls.
% \noindent\textbf{Postprocessing.}
% After extraction, we perform semantic merging of entities and relations offline.
% Each entity mention and relation phrase is encoded into an embedding space using a pre-trained encoder, and pairs whose cosine similarity exceeds a threshold are merged.
% This joint entity–relation reconciliation ensures that coverage statistics reflect genuine semantic gain rather than surface-form variation or redundant mentions.
% To reduce missing edges caused by conservative extraction, we adopt a multi-round extraction–reflection procedure.
% Each answer is reprocessed through a secondary “reflection” prompt that revisits the same content with awareness of previously omitted facts.
% This reflection step enables the LLM to infer implicitly expressed relations and produce additional triples that improve graph completeness.
% Through these design, the KG-Constructor Module constructs a normalized, semantically coherent, and incrementally extensible state $\mathcal{G}_t$, enabling reliable measurement of coverage and providing a stable foundation for the Strategy Scheduler described in Sec.~\ref{sec:design-sched}.
% Traditional methods like OpenIE and schema-constrained extraction~\cite{yates2007textrunner, stanovsky2018supervised} rely on stable data access and fixed relation inventories, making them ineffective in dynamic, hidden data environments.
% Recent LLM-based approaches~\cite{papaluca2024zerofewshot, mihindukulasooriya2025prompt, bai2025kggeneration}, while promising, assume document-level visibility and focus on static corpora, making global graph reconstruction inefficient in real-time, fragmented environments.
% Graph-augmented RAG frameworks such as GraphRAG~\cite{edge2024local} and LightRAG~\cite{guo2024lightrag} prioritize retrieval optimization by merging identical entities and relations, but this approach sacrifices coverage efficiency and can lead to redundancy, distorting CMG estimation.
% Most existing KG construction techniques are ill-suited to this incremental, black-box setting.
% Traditional pipelines such as OpenIE and schema-constrained extraction~\cite{yates2007textrunner,stanovsky2018supervised} depend on fixed relation inventories and curated ontologies, assuming stable syntax and consistent data access.
% These methods are designed for environments where data can be revisited or re-queried, and each new fact can be reliably updated within the context of a stable knowledge base.
% In contrast, our attack operates in a black-box RAG environment, where the attacker only observes generated answers on the fly, and does not have direct access to the underlying retrieval sets or source documents.
% While incremental KG methods can track coverage in static or continuously available data streams, they do not handle the case where the corpus is effectively hidden from the attacker and only stochastic, fragmentary information is accessible.
% Recent LLM-based approaches~\cite{papaluca2024zerofewshot,mihindukulasooriya2025prompt,bai2025kggeneration} have shown promise for adaptive graph construction, but they typically assume document-level visibility and focus on graph construction for known or static corpora.
% These methods typically rebuild global graphs or update them based on explicit document or data access. However, in our black-box setting, relying on full graph reconstruction is inefficient and unsuitable for tracking incremental coverage gains across multiple rounds of dynamic, fragmentary answers.
% Furthermore, these methods often prioritize semantic completeness and high-precision graph construction, which may be incompatible with the need for lightweight, efficient updates under strict query budgets.
% Graph-augmented RAG frameworks such as GraphRAG~\cite{edge2024local} and LightRAG~\cite{guo2024lightrag} also employ LLMs for triple extraction from retrieved documents.
% However, these methods focus on optimizing retrieval quality by merging exactly identical entities and relations, neglecting the compactness and consistency needed for coverage tracking across multiple turns.
% In our case, global consistency and exhaustive matching would introduce redundancy and alias drift, distorting CMG estimation and wasting queries.
% Therefore, our solution involves lightweight, incremental updates that emphasize coverage efficiency rather than semantic perfection.
% Traditional methods such as OpenIE and schema-constrained extraction~\cite{yates2007textrunner,stanovsky2018supervised} assume fixed relation inventories and curated ontologies, which collapse when answers vary in style, completeness, and granularity.
% Recent research has explored large language models for open-ended knowledge graph extraction~\cite{papaluca2024zerofewshot,mihindukulasooriya2025prompt,bai2025kggeneration}.
% Although these approaches increase flexibility compared to traditional pipelines, they mostly focus on single-pass graph construction from a known corpus.
% Few explicitly consider the problem of incrementally updating a graph state over sequential interactions.
% In contrast, our attack setting involves a black-box RAG interface with multi-round queries and evolving evidence; without incremental updating, methods risk redundancy, alias-drift, and biased CMG estimation.
% Inspired by recent graph-augmented RAG systems such as GraphRAG~\cite{edge2024local} and LightRAG~\cite{guo2024lightrag}, we leverage large language models to extract structured triples from each textual answer. Nevertheless, direct adoption of these methods is insufficient.
% Although they are designed for RAG documents, they mainly focus on merging \emph{exactly identical} entities and relations because their objective is to enhance retrieval quality rather than to maintain a globally coherent or compact knowledge structure.
% In RAG generation, it is sufficient that a query matches one mention of the correct entity, and redundant nodes or fragmented relations have little effect on retrieval accuracy.
% In our crawler, the compactness of the graph is essential for tracking coverage progress.
% Each redundant or aliased node is misinterpreted as a new discovery, wasting queries and distorting the estimation of marginal gain.
% Our attack therefore requires recognizing \emph{semantically equivalent} expressions and unifying aliases, abbreviations, and paraphrases such as “heart failure” and “cardiac insufficiency” to avoid redundant exploration and to improve crawl efficiency.
% A straightforward way to address this problem is to instruct the LLM directly reason over the current graph by placing the entire structure in its context, prompting it to generate new entities and relations conditioned on the existing nodes.
% However, this is impractical in practice.
% Even a moderate crawl involving about 1000 documents can produce on the order of $10^3$ entities, which far exceeds the context capacity of current language models.
% Including such a graph in every prompt would not only surpass token limits but also cause severe inefficiency and noise accumulation.
% To balance semantic fidelity and computational feasibility, we therefore decouple local extraction from global normalization.
% Each answer is first processed locally by the LLM to extract triples, while lightweight pre-processing and post-processing modules handle schema guidance, alias merging, and relation unification, as described below.
% Traditional knowledge graph construction techniques~\cite{hogan2021knowledge}, such as OpenIE-based pipelines and schema-constrained extraction~\cite{yates2007textrunner,stanovsky2018supervised}, assume stable syntax and fixed relation inventories.
% They rely on predefined linguistic patterns and curated ontologies, assumptions that hold in homogeneous or domain-specific corpora but fail in our setting, where answers vary widely in phrasing and completeness.
%Building such a state is challenging because RAG responses are often paraphrased, incomplete, and noisy.
% Entities may appear under multiple aliases, and relations are expressed implicitly or inconsistently across turns.
% A knowledge graph representation~\cite{hogan2021knowledge} provides a principled way to organize this evolving information, capturing entities and their typed relations while supporting incremental updates as new answers arrive.
% This module aims to convert free-text answer $a_t$ into a structured representation that serves as the attacker-side coverage state.
% This is difficult because answers are often paraphrased, incomplete, and noisy, making it non-trivial to identify consistent entities and relations across rounds.
% A suitable representation for this evolving state is a \emph{knowledge graph}~\cite{hogan2021knowledge}, which naturally encodes entities and their relations while supporting incremental updates.
\subsection{Strategy Scheduler}
\label{sec:design-sched}
\vspace{-1mm}
\begin{figure}
\centering
\includegraphics[width=0.6\linewidth]{figs/strategy_scheduler-2.pdf}
\caption{\textbf{The workflow of Strategy Scheduler.} The scheduler selects an action from the graph $\mathcal{G}_t$ via a two-stage process. First, it samples an entity ($e_t^*$) based on a score that estimates CMG by balancing two key metrics. Second, it selects a relation ($r_t^*$) by identifying the entity's largest local information deficit. A score cache with lazy updates ensures the efficiency of this process.}
\label{fig:strategy-scheduler}
\vspace{-1.5em}
\end{figure}
This module address the second challenge of the {ASCP} framework: \emph{attacker action spaces are intractable}. Operating on the knowledge graph from the KG-Constructor, its task is to select the optimal entity-relation pair (e.g., (\icon{figs/emoji/baby}, \textit{Family Medical History})) to guide the next query. The primary difficulty is that the space of potential strategic actions is combinatorially vast, rendering a brute-force search for the optimal pair computationally infeasible. Our key insight is an asymmetry between entities and relations: entities are the primary drivers of coverage expansion, while relations refine the exploration direction. Exploiting this asymmetry, we reframe the problem into a hierarchical two-stage decision: first select an entity, then pick which of its unexplored relations to probe. The workflow is shown in Fig.~\ref{fig:strategy-scheduler}.
% The Strategy Scheduler takes the attacker-side KG produced by {KG-Constructor} as input and selects the optimal entity–relation pair (e.g., (\icon{figs/emoji/baby}, \textit{Family Medical History})) for the next query. The core difficulty is the vast, unstructured action space, which makes direct joint search over pairs infeasible.
% The Strategy Scheduler takes the attacker-side knowledge graph from KG-Constructor as input and selects the optimal entity-relation pair (e.g., (\icon{figs/emoji/baby}, Family Medical History)) for the next query. The core challenge is that the action space is vast and unstructured, rendering a direct search for this pair infeasible. Our key insight is the fundamental asymmetry in the roles of entities and relations: entities are the primary drivers of coverage expansion, while relations serve to refine the direction of exploration. This insight allows us to reframe the intractable problem of jointly searching for the optimal pair. Instead, we decompose the search into a more manageable, two-stage hierarchical process: first selecting an entity, and then deciding which of its unexplored relations to pursue.
%The strategy scheduler takes the knowledge graph from the KG constructor and selects the optimal entity-relation pair (e.g., (\icon{figs/emoji/baby}, family medical history)) to explore further. Directly searching for the optimal is not XXXX, as the search space is large and unstructured. Our key insight is that the entities are the driving force behind the increasing coverage, while the relation can be considered a form of fine-graining. We therefore design a strategy scheduler that partitions the search into two steps: first deciding on an optimal entity to explore, and then deciding which unexplored relations of the entity to explore.
% We leverage a strategy scheduler module to address the second challenge of the {ASCP} framework: {attacker action spaces are intractable}. Since the action space of all natural language queries is effectively infinite, a principled strategy is required to make this problem tractable.
% To achieve this, the scheduler performs CMG estimation and structured action planning based on the graph produced by the KG-Constructor, effectively constraining the otherwise intractable query space.
\noindent{\bf Entities Selection. }To identify the entity that maximizes expected coverage, \toolname fist estimate the CMG of each entity in $\mathcal{G}_t$ using two complementary metrics: \emph{empirical payoff} and \emph{graph prior}. The \emph{empirical payoff} captures historical information gain and reflects how informative an entity has been in past queries. The \emph{graph prior} exploits the topology of the knowledge graph to identify entities located in structurally under-explored regions.
By jointly capturing exploitation through past utility and exploration through structural potential, these metrics provide a more complete CMG estimation. Their values are combined via a dynamically weighted sum to produce a final score for each entity, and the anchor entity $e_t^*$ is selected using robust Top-K softmax sampling.
\noindent{\it Empirical payoff.}
We define the empirical payoff of an entity $e$ via an upper-confidence style score:
\begin{equation}
\mathrm{EmpiricalPayoff}(e) = \bar g_e + c \sqrt{\frac{\log N}{n_e + 1}},
\end{equation}
where $\bar g_e$ is the empirical information gain of $e$, $n_e$ is the number of times $e$ has been selected as the anchor, $N$ is the total number of anchor selections, and $c > 0$ controls the strength of the confidence bonus.
The first term $\bar g_e$ estimates the contribution of $e$ to past knowledge-graph expansion. It is calculated as the normalized average increase in newly discovered entities and relations within a sliding window $t_w$:
\begin{equation}
\bar g_e = \frac{1}{n_e}\sum_{i \in \mathcal{I}(e)}
\Big(
\underbrace{\Delta \mathcal{E}_i / \max(\Delta \mathcal{E}_{t_w})}_{\mathrm{normalized new entities}}
+
\underbrace{\Delta \mathcal{R}_i / \max(\Delta \mathcal{R}_{t_w})}_{\mathrm{normalized new relations}}
\Big),
\end{equation}
where $\mathcal{I}(e)$ denotes the set of rounds where $e$ served as the anchor, and $\Delta\mathcal{E}_i$ and $\Delta\mathcal{R}_i$ represent the newly discovered entities and relations in round $i$.
The second component is an exploration bonus, rooted in the ``optimism in the face of uncertainty'' principle from multi-armed bandit literature~\cite{kaufmann2012bayesian,auer2002finite}. This term adds a confidence-based reward that encourages the agent to visit less-frequently selected entities, thereby mitigating the risk of converging to a local optimum.
\noindent{\it Graph prior.}
{While the \emph{empirical payoff} provides a self-contained mechanism for balancing exploitation and statistical exploration, it remains blind to the underlying topology of the knowledge graph. To overcome this limitation, we introduce a complementary, structure-aware exploration term: the \emph{graph prior}. This prior injects topological intelligence into the strategy, directing it toward graph regions with high discovery potential. It is composed of two distinct terms:}
\begin{equation}
\mathrm{GraphPrior}(e) = \mathrm{DegreeScore}(e) + \mathrm{AdjScore}(e).
\end{equation}
{The first term, $\mathrm{DegreeScore}$, promotes exploration in less-dense regions. It penalizes highly connected entities, operating on the intuition that these ``hub'' nodes are more likely to be information-saturated:}
\begin{equation}
\mathrm{DegreeScore}(e) = 1 - \frac{\deg(e)}{\max_{u\in \mathcal{E}}\deg(u)+\epsilon}.
\end{equation}
{The second term, $\mathrm{AdjScore}$, directly measures an entity's relational deficit. This deficit quantifies how common a specific relation type is among an entity's peers, given that the entity itself lacks that relation. A high $\mathrm{AdjScore}$ thus signals a significant discovery opportunity, prioritizing entities that are most likely to form a new, expected type of connection:}
\begin{equation}
\mathrm{AdjScore}(e) = \max_{r\in \mathcal{R}} \mathrm{Deficit}(e,r),
\end{equation}
where the deficit for a specific relation $r$ is defined as:
\begin{equation}
\resizebox{\linewidth}{!}{$ \mathrm{Deficit}(e,r) = \begin{cases}
0, & \text{if } r \in \mathrm{EdgeType}(e), \\[6pt]
\displaystyle
\frac{1}{|\mathcal{E}_{e.\mathrm{type}}|} \sum_{u\in \mathcal{E}_{e.\mathrm{type}}} \#\mathrm{Edge}(u,r), & \text{if } r \notin \mathrm{EdgeType}(e). \end{cases} $}
\end{equation}
Here, $\mathrm{EdgeType}(e)$ is the set of relation types already connected to $e$, and $\mathcal{E}_{e.\mathrm{type}}$ is the set of all entities sharing the same semantic type as $e$.
\noindent{\it Entity scoring and sampling.}
We integrate \emph{empirical payoff} and \emph{graph prior} into a final composite score. Critically, their relative importance may change over the course of an attack. We therefore combine them using time-varying weights to enable dynamic control of the exploration-exploitation balance:
\begin{equation}
\mathrm{Score}(e) \;=\; \alpha_t \cdot \mathrm{EmpricalPayoff}(e) \;+\; \beta_t \cdot \mathrm{GraphPrior}(e),
\end{equation}
where $\alpha_t$ gradually increases with time step $t$ to favor high-confidence anchors in later stages, while $\beta_t$ remains positive to maintain structural awareness.
Finally, to mitigate the risk of prematurely converging on a single, seemingly optimal path due to noisy gain estimates, we avoid a deterministic argmax selection. Instead, we sample the anchor entity $e_t^*$ from a Top-$K$ softmax distribution over the candidate scores, promoting strategic diversity and enhancing robustness.
%To enhance robustness against noisy gain estimates, we do not simply select the highest-scoring entity. Instead, we sample an anchor entity $e_t^*$ from a Top-$K$ softmax distribution over candidates scores.
\noindent\textbf{Relation Selection.}
Given the sampled anchor $e_t^*$, we choose a relation by probing its largest local deficit:
\begin{equation}
r^*(e_t^*) \;=\; \arg\max_{r\in \mathcal{R}\setminus \mathrm{EdgeType}(e_t^*)}\ \mathrm{Deficit}(e_t^*,r).
\end{equation}
To focus on meaningfully large gaps, we compare this maximum to the global distribution of current deficits,
\begin{equation}
\mathcal{DS}_t \;=\; \big\{\, \mathrm{Deficit}(e,r)\;:\; e\in \mathcal{E}_t,\ r\in \mathcal{R}\setminus \mathrm{EdgeType}(e)\,\big\},
\end{equation}
and set $r_t^*=r^*(e_t^*)$ only if it exceeds the $90$th percentile of $\mathcal{DS}_t$; otherwise, we set $r_t^*=\varnothing$ to indicate that no single relation is a sufficiently promising target.
% If the deficit for $r^*(e_t^*)$ exceeds the 90th-percentile of $\mathcal{DS}_t$, $r_t^* = r^*(e_t^*)$. Otherwise, $r_t^* = \varnothing$, indicating that no single relation is a sufficiently promising target.
Ultimately, the scheduler outputs the pair $(e_t^*, r_t^*)$, which provides structured guidance to Query Generator (Sec.~\ref{sec:design-qgen}).
\noindent\textbf{Optimization.}
%
To ensure scalability, the scheduler employs a lightweight caching mechanism. Previously computed $\mathrm{Score}(e)$ values are stored in a score cache. Following an update to the knowledge graph $\mathcal{G}_t$, a lazy-update strategy is triggered: only the scores of entities directly affected by the changes are invalidated and recomputed. The scores for all unaffected entities are served directly from the cache.
%
This approach decouples the computational cost of the scheduling step from the total size of the graph, ensuring that the decision loop remains efficient even as the attacker's graph expands significantly.
% To maintain scalability, the scheduler maintains a lightweight \emph{score cache} storing previously computed $\mathrm{Score}(e)$ values.
% When new entities appear, only those affected by updates are marked as \emph{dirty} and recomputed, while others retain cached scores.
% This lazy-update mechanism prevents the computational cost from increasing with the size of ${G}_t$, keeping the scheduling loop efficient as the graph expands.
% {We define the empirical payoff for an entity $e$, denoted as $\bar g_e$, as the average information gain observed from past selections of $e$ as an anchor. This gain is measured by the normalized growth in nodes ($\Delta \mathcal{E}$) and relations ($\Delta \mathcal{R}$) within a sliding window $t_w$:}
% \ning{direct formulae and explanation.}
% \begin{equation}
% \bar g_e = \frac{1}{n_e}\sum_{i \in \mathcal{I}(e)}
% \Big(
% \underbrace{\Delta \mathcal{E}_i / \max(\Delta \mathcal{E}_{t_w})}_{\mathrm{normalized new entities}}
% +
% \underbrace{\Delta \mathcal{R}_i / \max(\Delta \mathcal{R}_{t_w})}_{\mathrm{normalized new relations}}
% \Big),
% \end{equation}
% {where $n_e$ is the visit count for entity $e$, $\mathcal{I}(e)$ is the set of rounds anchored at $e$, and $\Delta \mathcal{E}_i, \Delta \mathcal{R}_i$ are the node and edge increments in round $i$.}
% {However, relying solely on this average payoff would lead to a purely greedy strategy, potentially overlooking less-visited but highly promising entities. To address this and encourage strategic exploration, we augment the empirical payoff with a confidence bonus based on the Upper Confidence Bound (UCB) principle~\cite{kaufmann2012bayesian,auer2002finite}. This results in the final score for this component:}
% \revise{where $N$ is the total number of anchor selections so far, and $c > 0$ is a hyperparameter controlling the level of optimism.}
% To ensure the selected entity yields the optimal coverage, \toolname uses \emph{empirical payoff} and \emph{graph prior} to quantify the CMG estimation of each entity.
% Empirical payoff is designed to exploit entities that have historically yielded high information gain. and This prior injects topological intelligence into the strategy, directing it toward graph regions with high discovery potential.\emph{Empirical payoff} is an exploitation-focused metric that quantifies the historical information gain from selecting the entity.
% Graph prior is an exploration-focused metric that leverages graph topology to identify entities in under-probed regions. Combining these two xxx can ..
% Empirical Payoffs and graph prior are combined via a dynamically weighted sum to produce a final score for each entity. The anchor entity $e_t^*$ is then selected using robust Top-K softmax sampling.
% \ning{why these two metrics and why we combine them this way is good. }
% To exploit proven strategies, we calculate an empirical payoff for each candidate entity $e$. This score is based on the average information gain observed when $e$ was previously chosen as an anchor. The gain for each round aggregates normalized node and edge growth within a sliding window $t_w$:
% \[
% \bar g_e = \frac{1}{n_e}\sum_{i \in \mathcal{I}(e)}
% \Big(
% \underbrace{\Delta \mathcal{E}_i / \max(\Delta \mathcal{E}_{t_w})}_{\mathrm{normalized new entities}}
% +
% \underbrace{\Delta \mathcal{R}_i / \max(\Delta \mathcal{R}_{t_w})}_{\mathrm{normalized new relations}}
% \Big),
% \]
% where $n_e$ is the visit count for entity $e$, $\mathcal{I}(e)$ is the set of rounds anchored at $e$, and $\Delta \mathcal{E}_i, \Delta \mathcal{R}_i$ are the node and edge increments in round $i$. Following the UCB principle~\cite{kaufmann2012bayesian,auer2002finite}, we add a confidence bonus to encourage optimistic exploration of less-visited but potentially high-reward candidates:
% \begin{equation}
% \mathrm{UCB}(e) = \bar g_e + c \sqrt{\frac{\log N}{n_e + 1}},
% \end{equation}
% where $N$ is the total number of anchor selections so far, and $c > 0$ is a hyperparameter that controls level of optimism.
% While standard exploration-exploitation strategies (UCB) encourage visiting less-frequent actions, they typically lack awareness of the underlying structure connecting different entities. To overcome this limitation, we introduce a Graph Prior that leverages the topology of the constructed graph ($\mathcal{G}_t$) to guide exploration more intelligently. This prior directs the search toward regions of the graph that are most likely to yield new information by combining two signals:
% \begin{itemize}[leftmargin=10pt]
% \item\revise{Empirical Payoff: An exploitation-focused metric that quantifies the historical information gain from selecting the entity.}
% \item \revise{Graph Prior:} An exploration-focused metric that leverages graph topology to identify entities in under-probed regions.
% % \item {Empirical Payoff (UCB Reward):\ning{what is UCB?}} An exploitation-focused metric that quantifies the historical information gain from selecting the entity.
% % \item {Graph Prior (Structural Awareness):} An exploration-focused metric that leverages graph topology to identify entities in under-probed regions.
% \end{itemize}
% Finally, the candidate relation $r_t^*$ is determined by analyzing the local relational deficits of the selected $e_t^*$. The workflow is illustrated in Fig.~\ref{fig:strategy-scheduler}.
% \ning{rewrite the following.}
% We develop a decoupled two-stage \ning{why two stage?} strategy for \toolname to handle the prohibitively large action space.
% The action space of all possible (entity, relation) pairs is vast and grows combinatorially as the graph expands.
% Directly scoring every pair would be computationally prohibitive.
% we first select an entity and then select a relation.
% compute a composite score for each candidate entity $e \in \mathcal{G}_t$ to approximate its CMG.
% Our scoring function augments a traditional exploitation metric with a novel, structure-aware exploration prior. This design allows the agent to balance immediate information gain with the long-term strategic value of probing structurally promising regions of the graph. The score is composed of two components:
%, with details of each design as follows.
% First, we identify a promising anchor entity, which serves as a stable proxy for an information-rich or under-explored region. This is done by computing a score for each candidate entity that balances exploitation (based on historical information gain) and exploration (based on structural potential). After selecting an entity $e_t^*$, we perform a fine-grained decision by identifying the most promising candidate missing relation $r_t^*$ in local neighborhood of $e^*_t$. This two-stage approach allows the scheduler to navigate the exploration-exploitation trade-off in a structured and scalable manner when estimating CMG.
% This module addresses the challenge the second challenge in the \emph{ASCP} and estimate CMG based on the graph produced by KG-Constructor. Since the action space of all natural language queries is effectively infinite, a principled strategy is required to make this problem tractable. To achieve this, the scheduler performs structured action planning based on the graph produced by the KG-Constructor, effectively constraining the otherwise intractable query space.
% This design follows the intuition of UCB-style bandit algorithms for balancing exploration and exploitation and aligns with adaptive coverage principles in submodular planning \cite{golovin2011adaptive}. Finally, the scheduler outputs the selected anchor via weighted sampling.
% Empirical Payoff (UCB Reward).
% For each candidate entity $e$ in $\mathcal{S}_t$, we maintain an empirical payoff $\bar{p}_e$ and a visit count $n_e$. $\bar{p}_e$ measures the average information gain achieved when $e$ was previously selected as the anchor. The gain per-round aggregates node and edge growth within a sliding window $t_w$:
% Graph Prior (Structural Awareness).
% To avoid myopically reusing already saturated neighborhoods, we add a structural potential term $\mathrm{Structure}(e)$, which favors under-covered regions. This potential combines two complementary signals:
% This module selects the next anchor to probe. Specifically, given the current graph state $\mathcal{G}_t$ and the remaining budget $B-t$, the scheduler determines which entity and relation to probe next to maximize expected coverage growth. Since the CMG $\Delta(q \mid \psi_{t-1})$ cannot be directly observed in a black-box pipeline, we compute an approximate score instead. This score balances two core metrics:
% \noindent\textbf{Empirical payoff (UCB reward).}
% For each candidate entity $e$ in $\mathcal{S}_t$, we maintain an empirical payoff $\bar p_e$ and a visit count $n_e$. The payoff measures the average information gain achieved when $e$ has previously been selected as the anchor. Each per-round gain aggregates node and edge growth within a sliding window $t_w$:
% \[
% \bar p_e = \frac{1}{n_e}\sum_{i \in \mathcal{I}(e)}
% \Big(
% \underbrace{\Delta \mathcal{E}_i / \max(\Delta \mathcal{E}_{t_w})}_{\mathrm{normalized new entities}}
% +
% \underbrace{\Delta \mathcal{R}_i / \max(\Delta \mathcal{R}_{t_w})}_{\mathrm{normalized new relations}}
% \Big),
% \]
% where $\mathcal{I}(e)$ is the set of rounds anchored at $e$, and $\Delta \mathcal{E}_i, \Delta \mathcal{R}_i$ are the node and edge increments recorded by the KG module in round $i$.
% The optimistic estimate then adds a confidence bonus:
% \begin{equation}
% \mathrm{UCB}(e) = \bar p_e + c \sqrt{\frac{\log N}{n_e + 1}},
% \end{equation}
% where $N$ is the total number of anchor selections so far and $c>0$ controls optimism.
% \noindent\textbf{Graph prior (gap awareness).}
% To avoid myopic reuse of already saturated neighborhoods, we add a structural potential that prefers under-covered regions.
% This potential combines two complementary signals: one based on local degree, and another based on relational deficits.
% \begin{equation}
% \mathrm{Structure}(e) = \mathrm{DegreeScore}(e) + \mathrm{AdjScore}(e),
% \end{equation}
% The degree term penalizes highly connected nodes that have already contributed many edges to the graph, encouraging exploration of sparser regions:
% \begin{equation}
% \mathrm{DegreeScore}(e) = 1 - \frac{\deg(e)}{\max_{h\in \mathcal{E}}\deg(h)+\epsilon}.
% \end{equation}
% The adjacency term measures how many relation types typically co-occur for entities of the same type but are still missing around $e$. A higher deficit suggests that querying this entity may yield new relations:
% \begin{equation}
% \mathrm{AdjScore}(e) = \max_{r\in \mathcal{R}} \mathrm{Deficit}(e,r),
% \end{equation}
% where the deficit of a specific relation $r$ is computed as
% \begin{equation}
% \resizebox{\linewidth}{!}{$ \mathrm{Deficit}(e,r) = \begin{cases}
% 0, & r \in \mathrm{EdgeType}(e), \\[6pt]
% \displaystyle
% \frac{1}{|E_{e.\mathrm{type}}|} \sum_{u\in E_{e.\mathrm{type}}} \#\mathrm{Edge}(e,r), & r \notin \mathrm{EdgeType}(e). \end{cases} $}
% \end{equation}
% Here, $\mathrm{EdgeType}(e)$ is the set of relation types already observed for $e$, and $\mathcal{E}_{e.\mathrm{type}}$ is the set of entities with the same semantic type.
% Combining the degree term and the deficit term provides a balanced structural prior that guides the scheduler toward informative but underexplored regions of the graph.
% \noindent\textbf{Balanced score and sampling output.}
% We blend the two signals with time-varying weights:
% \begin{equation}
% \mathrm{Score}(e) \;=\; \alpha_t \cdot \mathrm{UCB}(e) \;+\; \beta_t \cdot \mathrm{Structure}(e),
% \end{equation}
% where $\alpha_t$ increases with $t$ to favor high-confidence anchors while $\beta_t{>}0$ preserves gap awareness.
% Rather than a hard argmax, we sample anchors from a Top-$K$ softmax over $\mathrm{Score}(e)$ to improve robustness under noisy gain estimates.
% For the selected entity $e_t^*$, the scheduler also decides a \emph{candidate missing relation}
% $e_t^*\in R\cup\{\varnothing\}$ by inspecting relation deficits around $e_t^*$. Concretely, let
% \[
% r^*(e_t^*) \;=\; \arg\max_{r\in \mathcal{R}\setminus \mathrm{EdgeType}(e_t^*)}\ \mathrm{Deficit}(e_t^*,r),
% \]
% and define the round-$t$ reference distribution
% \[
% \mathcal{DS}_t \;=\; \big\{\, \mathrm{Deficit}(e,r)\;:\; e\in \mathcal{E}_t,\ r\in \mathcal{R}\setminus \mathrm{EdgeType}(e)\,\big\}.
% \]
% If $\mathrm{Deficit}\!\left(e_t^*, r^*(e_t^*)\right)$ exceeds the $90$th percentile of $\mathcal{DS}_t$
% (equivalently, if $\mathrm{AdjScore}(e_t^*)$ is above the $90$th percentile among candidates at round $t$),
% we set $r_t^*=r^*(v_t^*)$, and otherwise we set $r_t^*=\varnothing$.
% Thus the scheduler outputs a pair $(e_t^*, r_t^*)$, which directly conditions the generator in Sec.~\ref{sec:design-qgen}.
\subsection{Query Generator}
\label{sec:design-qgen}
\vspace{-1mm}
\begin{figure}
\centering
\includegraphics[width=0.75\linewidth]{figs/h_query_generator.pdf}
\caption{\textbf{The workflow of Query Generator.} It receives an anchor pair $(e_t^*, r_t^*)$ and selects a generation strategy. A candidate query $q_{t+1}$ is produced and checked against historical queries. If too similar, the generator requests a new anchor from the scheduler and resamples.}
\label{fig:query-generator}
\vspace{-1.5em}
\end{figure}
This module addresses the third challenge in the {ASCP} formulation: \emph{Real-world RAG systems impose practical constraints}. We cannot directly send a structured anchor $(e_t^*, r_t^*)$ to the system, and relying solely on simple templates is not a viable solution, as it can easily be flagged by query provenance mechanisms. Therefore, the core task of this module is to translate the abstract, strategic anchor pair from the {Strategy Scheduler} into a concrete, executable, and natural-sounding query $q_{t+1}$. This process must address two primary challenges: 1) generating a query that is contextually relevant to the anchor's strategic intent, and 2) ensuring the query remains novel to avoid redundancy and maximize the utility of the limited query budget.
To achieve this, the generator involves three key steps: 1) adaptive query formulation, 2) history-aware de-duplication, and 3) a feedback mechanism for system-level learning. The workflow is shown in Fig.~\ref{fig:query-generator}.
% To achieve this, the generator employs a conditional strategy. Based on the specificity of the anchor, it selects either a targeted, Relation-Driven approach or a more exploratory, Neighborhood-based approach. Crucially, every generated query undergoes a History-Aware De-duplication check. If a candidate query is too similar to previously asked questions, the generator requests a new anchor from the scheduler and retries, creating a robust resampling loop. This ensures that each query sent to the black-box system is both purposeful and unique. The workflow is illustrated in Fig.~\ref{fig:query-generator} and details are as follows.
\noindent\textbf{1) Adaptive Query Formulation.}
The generator's first step is to intelligently select a formulation strategy based on the specificity of the anchor $(e_t^*, r_t^*)$ provided by the scheduler. This dual-mode approach allows the system to flexibly switch between precision and discovery:
\begin{itemize}[leftmargin=10pt]
\item \textbf{Relation-Driven Probing}. When the {Strategy Scheduler} identifies a plausible relation deficit $r_t^*$ for an entity $e_t^*$ with high confidence, the generator activates this targeted strategy. It instantiates a relation-aware template conditioned on the $(e_t^*, r_t^*)$ pair and realizes it via an LLM into a fluent, natural language query $q_{t+1}$ designed for precise fact-probing.
\item \textbf{Neighborhood-based Generation.} In the absence of a sufficiently salient relation, the generator shifts to an exploratory mode. It analyzes the local neighborhood of $e_t^*$ within the current knowledge graph $\mathcal{G}_t$. By leveraging the KG's schema and the generative capabilities of LLM, it hypothesizes plausible missing relations and formulates one or more exploratory query variants. %This enables principled knowledge discovery even without a specific target.
\end{itemize}
% \noindent\textbf{Relation-Driven Generation (when $r_t^*\neq\varnothing$).}
% When the \textbf{Strategy Scheduler} identifies a plausible relation deficit $r_t^*$ for an entity $e_t^*$ with high confidence (i.e., the deficit passes the percentile test in Sec.~\ref{sec:design-sched}), this targeted strategy is activated. The generator instantiates a relation-aware template conditioned on the $(e_t^*, r_t^*)$ pair and realizes it via the LLM into a fluent, natural language query $q_{t+1}$ designed for precise fact-probing.
% \noindent\textbf{Neighborhood-based Generation (when $r_t^*=\varnothing$).}
% In the absence of a sufficiently salient relation, the generator shifts to an exploratory mode. It analyzes the local neighborhood of $e_t^*$ within the current knowledge graph $\mathcal{G}_t$. By leveraging the KG's schema, it utilize xxx capability of LLM ,hypothesizes plausible missing relations and generates one or more exploratory query variants, enabling principled knowledge discovery even without a specific target.
\noindent\textbf{2) History-aware De-duplication}
To maximize the information yield of each query, we introduce a robust quality control gate powered by a history-aware resampling loop. Before being dispatched, each candidate query $q$ is compared against the historical query log $\mathcal{Q}_{\mathrm{hist}}$. If its similarity $\mathrm{sim}(q,\mathcal{Q}_{\mathrm{hist}})\!\ge\!\tau$, the generator triggers a resample operation, drawing a new anchor from the {Strategy Scheduler}’s Top-$K$ distribution and re-attempting the formulation, up to $M$ trials. This mechanism is critical for avoiding diminishing returns. Furthermore, this loop provides an elegant convergence heuristic: if all $M$ trials yield duplicate queries, we declare that the system has reached global convergence.
\noindent\textbf{3) Penalties and Feedback Loop.}
Beyond producing a valid query, the generator's final role is to enable system-level learning. Critically, queries result in refusals, empty responses or identified as duplicates incur strategy-specific penalties. This penalty signal is fed back to the scheduler as a reduced payoff, directly influencing its subsequent {Empirical Payoff} score calculations and anchor selections. This feedback mechanism, which connects the outcome of the generator's action back to the high-level strategy, allows the system to learn from failed interactions and dynamically adapt its strategy over time.
%The generator is not merely a producer but a crucial component in a closed-loop learning system. An issued query $q_{t+1}$ yields an answer $a_{t+1}$, which the KG-Constructor processes to update the state to $\mathcal{G}_{t+1}$. Critically, refusals or empty responses incur strategy-specific penalties (targeted vs. fallback). This penalty is fed back to the scheduler as a reduced payoff, directly influencing subsequent \emph{Empirical Payoff} score calculations and anchor selections in \textbf{Strategy Scheduler}. This feedback mechanism allows the system to learn from failed interactions and dynamically adapt its strategy over time.
% This module turns the scheduler’s anchor into a stealth-compliant query while avoiding redundancy.
% Given $(e_t^*, r_t^*)$ from Sec.~\ref{sec:design-sched}, the generator chooses between a targeted query (if $r_t^*\neq\varnothing$) and a Meta-QGen fallback (if $r_t^*=\varnothing$), and performs history-aware de-duplication with early exit.
% \noindent\textbf{Relation-Driven Generation (when $r_t^*\neq\varnothing$).}
% When the state $\mathcal{S}_t$ indicates that relation $e_t^*$ is plausibly missing around $e_t^*$ (i.e., the selected deficit passes the percentile test in Sec.~\ref{sec:design-sched}), we instantiate a relation-aware template conditioned on $(e_t^*, r_t^*)$ and realize it into a fluent surface form to obtain $q_{t+1}$.
% \noindent\textbf{Neighborhood-based Generation (when $r_t^*=\varnothing$).}
% If no relation is sufficiently salient, the generator conditions on the local neighborhood of $e_t^*$ in $\mathcal{S}_t$ to hypothesize a plausible missing relation from the schema and realize one or more exploratory variants.
% \noindent\textbf{History de-duplication and convergence.}
% Each candidate realization is compared with \emph{History Queries}; if $\mathrm{sim}(q,\mathcal{Q}_{\mathrm{hist}})\!\ge\!\tau_{\mathrm{dup}}$, we \emph{resample} from the scheduler’s Top-$K$ distribution (up to $M$ trials).
% If all $M$ trials are near-duplicates, we declare \emph{global convergence} and terminate under the fixed budget. %\ymy{revise: in the experiment, we found}
% \noindent\textbf{Penalties and feedback.}
% Issuing $q_{t+1}$ yields $a_{t+1}$ that the KG-Constructor converts into $\Delta \mathcal{S}_{t+1}$ to update $\mathcal{S}_{t+1}$.
% Refusals or empty content incur penalties (targeted vs.\ fallback) that are fed back as reduced payoff, influencing subsequent UCB rewards and anchor selection.
\section{Evaluation}
\label{sec:evaluation}
We conduct comprehensive experiments to answer the following research questions:
%\begin{tcolorbox}[size=small]
\begin{itemize}[noitemsep, leftmargin=8pt]
\item \textbf{RQ1:}
How effective is \toolname\ compared with existing methods?
\item \textbf{RQ2:}
How does the the retriever in victim RAG affect coverage performance?
\item \textbf{RQ3:}
How does the internal LLM agent within attack method influence performance?
\item \textbf{RQ4:}
How robust is the attack against RAG variants employing query rewriting or multi-query retrieval?
\item \textbf{RQ5:}
How do the hyperparameters and each technique affect effectiveness of \toolname?
% \item \textbf{RQ6:}
% What are the costs associated with the attack?
\end{itemize}
%\end{tcolorbox}
\subsection{Evaluation Setup}
\vspace{-1mm}
\noindent\textbf{Dataset.}
We evaluate on four datasets spanning scientific and medical domains with varying styles and scales. Three datasets, {TREC-COVID}, {SciDocs}, and {NFCorpus}, are drawn from the BEIR benchmark~\cite{thakur2021beir}, containing about 171.3K, 25.6K, and 5.4K documents, respectively. We further include {Healthcare-Magic-100k}~\cite{healthcaremagic} (abbreviated as {Healthcare}), with roughly 100K patient–provider Q\&A samples.
{TREC-COVID}, {SciDocs}, {NFCorpus}, and {Healthcare} focus on biomedical literature, scientific papers, consumer health, and clinical dialogues, respectively. For efficiency and fair comparison, we randomly select 1,000 de-duplicated documents from each corpus as the victim RAG collection~\cite{jiang2024rag,di2024pirates}. We confirm that the sampled subsets preserve similar semantic distributions to the full corpora using Energy Distance~\cite{szekely2013energy} and the Classifier Two-Sample Test (C2ST)~\cite{lopez2016revisiting}, {with details in Appendix~\ref{appendix:sample-validation}.}
% {TREC-COVID} covers biomedical research literature, {SCIDOCS} general scientific papers, and {NFCorpus} consumer health information, while {Healthcare} focuses on conversational clinical dialogues. Together, they span retrieval scenarios from formal expert text to layman and interactive medical content. For efficiency and fair comparison, we randomly select 1,000 de-duplicated documents from each corpus as the victim RAG collection~\cite{jiang2024rag,di2024pirates}. We confirm that the sampled subsets preserve similar semantic distributions to the full corpora using Energy Distance~\cite{szekely2013energy} and the Classifier Two-Sample Test (C2ST)~\cite{lopez2016revisiting}, {with details in Appendix~\ref{appendix:sample-validation}.}
\noindent\textbf{Generator and Retriever.}
We employ two retrievers in our evaluations: BGE~\cite{zhang2023retrieve} and GTE~\cite{li2023towards}. For generators, we evaluate four large language models: {Llama~3.1 Instruct-8B} (denoted as Llama-3-8B)~\cite{dubey2024llama}, {Command-R-7B}~\cite{command-r7b}, {Microsoft-Phi-4}~\cite{abdin2024phi}, and {GPT-4o-mini}~\cite{gpt4o-mini}. These generators represent diverse families of reasoning and instruction-tuned models, covering both open-source and proprietary systems and varying parameter scales. The inclusion of
GPT-4o-mini, with its built-in safety guardrails~\cite{gpt-guardrail},
demonstrates our attack's robustness against industry defenses.
%These generators collectively represent diverse families of reasoning and instruction-tuned models, covering both open-source and proprietary systems as well as varying parameter scales.
% Such diversity ensures the robustness and generality of our evaluation across heterogeneous RAG configurations.
\noindent\textbf{Attacker-side LLM.}
We adopt two attacker-side LLMs to simulate adversarial query generation. In the main experiments, we use {Doubao-1.5-lite-32K}~\cite{doubao} due to its high speed, cost efficiency, and independence from the RAG’s generator family. This aligns with the black-box assumption.
We also evaluate a smaller open-source alternative, {Qwen-2.5-7B-Instruct}~\cite{qwen2.5-7b}, to examine transferability between different model families and sizes.
\noindent\textbf{RAG Setting.}
We cover three RAG configurations: {vanilla RAG}, RAG with {query rewriting}, and RAG employing {multi-query retrieval with RRF re-ranking} (default: 3 queries per request).
We set the retrieval number to \(k = 10\) and analyze its impact in Sec.~\ref{sec:ablation}.
The retrieved documents are provided to the generator as contextual input through a structured system prompt.
Further details on the system prompts are provided in Appendix~\ref{appendix:prompts}.
\noindent\textbf{Baselines.}
We compare \toolname\ against two state-of-the-art black-box RAG extraction attacks: {RAGThief}~\cite{jiang2024rag} and {IKEA}~\cite{wang2025silent}. RAGThief represents continuation-based attack, and IKEA represents keyword-based attack.
To ensure a fair comparison, all attacks employ the same attacker-side LLM. For all experiments, we use the default hyperparameters reported in the original papers. Each attack is allowed to issue at most 1,000 queries to the victim RAG, ensuring consistent query budgets across methods.
% RAGThief expands previous dissimilar responses using an LLM to generate new queries and appends an adversarial instruction that prompts the RAG generator to reproduce retrieved content verbatim (corresponding to \emph{Explicit attack}). In contrast, IKEA constructs a set of anchor words as keywords and iteratively expands them from each response to form new queries without using adversarial instructions (\emph{Implicit attack}).
%
%\noindent\textbf{Implementation and Hyperparameters.}
\noindent\textbf{Metrics.}
We evaluate extraction along two axes \emph{coverage} and \emph{content quality}:
\textbf{(i) Coverage Rate.}
This metric quantifies how much of the victim RAG’s private corpus has been exposed through retrieval, corresponding to the adversary’s goal in Eq.~\ref{eq:coverage}. Following prior work~\cite{jiang2024rag,wang2025silent}, this metric is only calculated on queries for which the victim RAG produces a \emph{non-refusal} response.
%
\textbf{(ii) Semantic Fidelity.}
To quantify the semantic overlap between the private corpus $\mathcal{D}$ and the extracted snippets $\mathcal{A}$, we first measure the similarity for each document $d \in \mathcal{D}$ to its best-aligned snippet using a fixed encoder $E_{\mathrm{eval}}(\cdot)$:
\begin{equation}
s(d)=\max_{e\in\mathcal{E}} \ \cos\bigl(E_{\mathrm{eval}}(d),\,E_{\mathrm{eval}}(e)\bigr).
\end{equation}
The average of $s(d)$ over the entire corpus represents the overall semantic fidelity of the extracted text.
%
To assess practical utility, we also measure \textbf{Reconstruction Fidelity}. We construct a surrogate RAG from the extracted knowledge and evaluate it using the official query sets corresponding to the documents in TREC-COVID, SciDocs, and NFCorpus corpora, totaling 1,860, 2,181, and 1,991 queries, respectively. The Healthcare dataset is excluded due to the absence of official query annotations. We report: (a)Success Rate (fraction of non-refusal responses), (b) Answer Similarity (embedding cosine similarity between surrogate and victim answers); and (c) ROUGE-L score.
% To quantify the practical downstream utility of the extracted content, we assess its fidelity in a knowledge reconstruction task. For this, we construct a surrogate RAG system using the corpus extracted by an attack method. We then evaluate this surrogate system's performance on the official query sets provided with the TREC-COVID, SciDocs, and NFCorpus datasets. These benchmark queries are designed to test retrieval and generation over the respective corpora, thus providing a standardized basis for evaluation.
% We measure performance using three metrics: (a) Success Rate, defined as the fraction of queries that elicit a non-refusal response from the surrogate RAG; (b) Answer Similarity, calculated as the cosine similarity between the embeddings of the surrogate's and the victim's answers; and (c) ROUGE-L, which measures the longest common subsequence between the generated answers. Together, these metrics move beyond simple content volume to evaluate the functional and economic substitutability of the extracted knowledge base, offering a comprehensive measure of the attack's real-world impact.
\subsection{Effectiveness (RQ1)}
\label{sec:effectiveness}
\vspace{-2mm}
%\input{tables/coverage_result}
We evaluate the effectiveness of \toolname\ through three key metrics: corpus coverage rate, semantic fidelity and reconstruction fidelity. Across all evaluations, \toolname\ demonstrates a significant leap in performance over existing baseline attacks.
\input{tables/coverage-with-sim}
\begin{figure*}[t]
\centering
\includegraphics[width=0.9\linewidth]{figs/all-coverage-2.pdf}
\caption{Coverage Rate vs. Query Number (1,000 Budget) across datasets and generators (BGE Retriever). \toolname\ steadily increases coverage and consistently surpasses both RAGThief and IKEA. Comprehensive results are in Appendix~\ref{appendix:all-coverage}.}
\label{fig:main-coverage}
\vspace{-1.5em}
\end{figure*}
\noindent\textbf{Coverage Rate.}
\toolname\ achieves consistently superior extraction efficiency and breadth. As shown in the coverage rate curves in Fig.~\ref{fig:main-coverage} and Fig.~\ref{fig:all-coverage}, \toolname\ steadily increases the corpus coverage throughout the attack, reaching a final average rate of 66.8\% under a 1,000-query budget (Tab.~\ref{tab:all-coverage}). This high efficiency is particularly pronounced in challenging, broad-domain corpora such as {SciDocs} and {TREC-COVID}. The success of \toolname\ is attributable to its global knowledge graph and UCB-based planning algorithm. This design enables it to dynamically and strategically traverse the entire semantic space of the corpus, ensuring it continuously targets and extracts from underexplored regions. Furthermore, \toolname\ remains highly effective against RAG systems with built-in guardrails (e.g., those using {GPT-4o-mini}). Its use of stealthy, policy-compliant prompts allows it to circumvent defenses that easily detect and block the explicit jailbreak attempts used by other methods.
In contrast, baseline methods exhibit significant limitations. The strongest baseline, IKEA, plateaus early and achieves an average coverage of only 46.1\% (a 20.7 percentage point deficit compared to \toolname). RAGThief is even less effective, with an average coverage of a mere 9.6\%. These heuristic-based methods tend to repeatedly query semantically similar regions or drift off-topic, failing to achieve comprehensive corpus exploration. RAGThief, in particular, fails completely against guarded models GPT-4o-mini, as its direct jailbreak prompts are consistently refused.
\noindent\textbf{Semantic Fidelity.}
Beyond extracting a larger volume of content, \toolname\ also ensures the extracted information is of higher quality. \toolname\ achieves the highest Semantic Fidelity (SF) score across all dataset-generator pairs, with a cross-dataset average of 0.605 (Tab.~\ref{tab:all-coverage}). This indicates that the content it extracts is highly faithful to the source documents. This high fidelity is a direct result of our design of the Query Generator, which maintains contextual coherence and ensures that the extracted text chunks are semantically sound and accurate. By comparison, IKEA and RAGThief score lower, with average SF scores of 0.559 and 0.417, respectively. Their less-targeted query strategies may yield fragmented or contextually distorted content, diminishing the quality of the extracted knowledge.
\input{tables/reconstruction}
\noindent\textbf{Reconstruction Fidelity.}
To assess the practical utility of the extracted knowledge, we measure its effectiveness in a downstream task: building a surrogate RAG system. A surrogate RAG constructed from the knowledge base extracted by \toolname\ demonstrates markedly superior performance. As reported in Tab.~\ref{tab:reconstruction}, it achieves answer success rates ranging from 38.1\% to 52.6\% on official query sets. Moreover, it obtains the highest scores in embedding similarity (up to 0.699) and ROUGE-L (up to 0.240), confirming the high quality and functional value of the recovered knowledge. This operational effectiveness stems from the comprehensive and semantically integral knowledge base that \toolname\ extracts. In contrast, surrogate systems built from the knowledge extracted by IKEA and RAGThief perform poorly. This highlights that their extracted content is not only less complete but also less useful for reconstructing the original system's capabilities.
\noindent\textbf{Takeaway.}
\toolname\ outperforms prior methods across all datasets and generators, achieving an average coverage rate of 66.8\%, compared to 46.1\% for IKEA and 9.6\% for RAGThief. It also obtains a superior average semantic fidelity of 0.605. Furthermore, surrogate RAG systems built from its extractions yield the highest answer success rates, confirming that \toolname\ constitutes a more potent and realistic threat to RAG systems.
%\toolname\ outperforms prior methods across all datasets and generators, achieving an average coverage rate of 66.8\% and a semantic fidelity of 0.605. Surrogate RAG systems built from its extractions also obtain the highest answer success rates. These findings indicate that \toolname\ constitutes a more potent and realistic threat to RAG systems.
% We first evaluate the overall effectiveness of \toolname\ compared to the baseline attacks RAGThief and IKEA. Across four benchmark datasets and multiple RAG generators, \toolname\ consistently achieves the highest corpus coverage under identical query budgets.
% Tab.~\ref{tab:all-coverage} shows the coverage rate and semantic fidelity under a 1,000-query budget for each method across dataset–generator pairs. Fig.~\ref{fig:main-coverage} and \ref{fig:all-coverage} illustrates the coverage rate curves.
% In all settings, \toolname increases coverage steadily and consistently reaches a higher final value than both IKEA and RAGThief, demonstrating superior extraction efficiency. On average, \toolname achieves a final coverage rate of 66.8\%, significantly outpacing IKEA’s 46.1\% and RAGThief’s 9.6\%, representing a 20.7\% improvement over the strongest baseline.
% On {Healthcare}, where conversational variability are prevalent, IKEA performs relatively well due to its random keyword regeneration mechanism. This enables it to explore more previously unvisited regions of the corpus. However, \toolname still maintains a significant lead, achieving 80.7\% coverage compared to IKEA’s 68.7\% under the Llama-3-8B configuration.
% The advantage of \toolname becomes particularly evident on more challenging corpora such as {SciDocs} and {TREC-COVID}, which feature broad-domain or heterogeneous documents. In these settings, heuristic methods like RAGThief and IKEA tend to plateau early, either by repeatedly querying similar regions or by drifting off-topic. In contrast, \toolname’s global knowledge graph and UCB-based planning allow it to dynamically traverse the semantic space and target underexplored regions, ensuring continuous extraction progress.
% In generators equipped with guardrails (e.g., {GPT-4o-mini}), \toolname remains robust, whereas RAGThief fails completely due to its reliance on explicit jailbreak prompts, which trigger policy-based refusals such as: ``I’m sorry, but I can’t provide verbatim text chunks from my knowledge base.'' In contrast, \toolname adopts stealthy yet compliant surface forms that circumvent such restrictions, achieving substantial coverage even under tightly constrained models.
% \input{tables/coverage-with-sim}
% In addition to extracting a larger amount of content, \toolname also demonstrates superior semantic quality. Table~\ref{tab:all-coverage} presents the final Coverage Rate (CR) and Semantic Fidelity (SF). Across all datasets, \toolname achieves the highest SF scores. Averaged over all datasets, \toolname reaches an SF of 0.605, compared to 0.559 for IKEA and 0.417 for RAGThief. These results indicate that \toolname not only extracts more documents but also maintains higher semantic integrity, ensuring that the retrieved content remains both accurate and faithful to the source corpus.
% To evaluate the practical utility of the higher-quality content extracted, we further assess reconstruction fidelity by constructing surrogate RAG systems using the knowledge extracted by each attack method. Table~\ref{tab:reconstruction} reports the results on TREC-COVID, SciDocs, and NFCorpus, all of which provide official query sets for evaluation. Specifically, we include all queries associated with documents in the corpus, totaling 1,860, 2,181, and 1,991 queries, respectively. The Healthcare dataset is excluded due to the absence of official query annotations.
% Surrogate systems built on the knowledge extracted by \toolname consistently achieve the best performance across all metrics, with answer success rates ranging from 38.1\% to 52.6\%, significantly exceeding those of IKEA and RAGThief. Moreover, the embedding similarity and ROUGE-L scores further confirm the high quality of \toolname’s extractions, reaching up to 0.699 and 0.240, respectively. These findings demonstrate that the knowledge recovered by \toolname is not only semantically faithful but also operationally effective, supporting realistic downstream tasks such as knowledge reconstruction.
% \noindent\textbf{Takeaway.}
% \toolname outperforms prior methods across all datasets and generators, achieving an average coverage rate of 66.8\% and a semantic fidelity of 0.605. Surrogate RAG systems built from its extractions also obtain the highest answer success rates. These findings indicate that \toolname constitutes a more potent and realistic threat to RAG systems.%, as it can extract not only a larger volume of content but also more accurate and functionally useful private knowledge.
\subsection{Retriever Sensitivity (RQ2)}
\label{sec:retriever}
\vspace{-2mm}
To evaluate the robustness of \toolname\ against different retrieval architectures, we evaluated the attack methods on victim RAG systems with GTE~\cite{li2023towards} as retriever. %
\toolname\ demonstrates robustness and adaptability, maintaining its superior performance.
As shown in Table~\ref{tab:gte-coverage} for Llama-3-8B generator, \toolname\ achieves an average coverage rate (CR) of 76.6\% and a semantic fidelity (SF) of 0.596. Notably, its coverage improves compared to its performance with the BGE retriever (69.0\%, from Table~\ref{tab:all-coverage}), showcasing its ability to effectively capitalize on the document set surfaced by any underlying retriever. %This robustness stems from its core design: \toolname\ operates on the semantic space revealed by the retriever's outputs, using its knowledge graph and UCB-based planner to systematically explore the provided content, making its strategy largely independent of the specific retrieval algorithm.
\input{tables/gte-coverage}
In contrast, while the baseline methods also see a performance uplift with the GTE retriever (IKEA's average coverage increases to 53.9\% and RAGThief's to 28.2\%), they still remain far less effective. This indicates that while a different retriever may surface ``easier'' content for all methods, the heuristic-driven approaches of the baselines are fundamentally less capable of exploiting this opportunity to the same degree as \toolname's strategic exploration.
\noindent\textbf{Takeaway.}
\toolname’s effectiveness is not contingent on a specific retriever. When the victim RAG employs a GTE retriever, \toolname\ achieves an average coverage of 76.6\%, substantially outperforming both IKEA (53.9\%) and RAGThief (28.2\%). Its strategic, graph-based exploration makes it a more versatile and potent threat across diverse RAG system implementations.
% To assess how different retriever backbones affect the performance of \toolname and competing methods, we replace the default bi-encoder retriever (BGE) with GTE~\cite{li2023towards}, while keeping LLaMA-3-8B as the fixed generator. This setup allows us to evaluate how the choice of retriever influences the performance of each attack method.
% Table~\ref{tab:gte-coverage} reports the Coverage Rate (CR) and Semantic Fidelity (SF) for all methods using GTE as the retriever. The results show that \toolname continues to outperform all baselines even with a different retriever, achieving the highest coverage and fidelity across all datasets.
% On average, \toolname achieves a coverage rate (CR) of 76.6\%, which is substantially higher than IKEA’s 53.9\% and RAGThief’s 28.2\%, and also exceeds its own average CR with BGE (69.0\%, as reported in Table~\ref{tab:all-coverage}). Similarly, its semantic fidelity (SF) reaches 0.596, outperforming RAGThief (0.484) and IKEA (0.561).
% These results demonstrate that \toolname’s performance remains robust across different retrieval backends, highlighting its ability to effectively exploit whatever documents the retriever provides, regardless of the underlying retrieval architecture.
% \noindent\textbf{Takeaway.}
% \toolname’s superiority is largely independent of the retriever used in the victim RAG system. When the system employs GTE as its retriever, \toolname achieves an average coverage rate of 76.6\%, compared to 53.9\% for IKEA and 28.2\% for RAGThief on Llama-3-8B. %It consistently extracts more and higher-fidelity content across different retrieval architectures.
% \begin{figure*}[t]
% \centering
% \includegraphics[width=0.85\linewidth]{figs/qwen.pdf}
% \caption{Coverage Rate vs. Query Number (1,000 Query Budget) for different attacks with GTE Retriever. \toolname consistently}
% \label{fig:gte-coverage}
% \end{figure*}
\subsection{Agent Sensitivity (RQ3)}
\label{sec:agent}
\vspace{-1mm}
To investigate the influence of the attacker's LLM agent on attack performance, we evaluated the effectiveness with a smaller open-source model, Qwen-2.5-7B-Instruct.
\toolname\ demonstrates high effectiveness irrespective of the agent's scale. Even when powered by the smaller Qwen-2.5-7B-Instruct agent, \toolname\ continues to achieve the highest performance across all metrics, as shown in Table~\ref{tab:qwen-coverage}. It reaches an average coverage of 68.9\% and a semantic fidelity of 0.596, results that are comparable to, and in some cases even exceed, its performance with the larger agent (e.g., 54.2\% coverage on TREC-COVID with Qwen vs. 49.4\% with Doubao). This resilience stems from our framework's design, which decouples high-level strategic planning from low-level content generation. The knowledge graph and strategy scheduler dictate the overall attack trajectory, requiring the LLM agent merely to execute localized and well-defined sub-tasks. %As a result, the core effectiveness of our method is not critically dependent on the agent's advanced reasoning capabilities.
\input{tables/qwen-coverage}
In comparison, the baseline methods remain significantly less effective. With the Qwen-2.5-7B-Instruct agent, IKEA achieves an average coverage of 51.3\%, while RAGThief reaches 34.6\%. Although RAGThief sees a slight performance improvement (likely because the constrained divergent ability of smaller model make it less prone to the off-topic drift), it still lags far behind \toolname. %This shows that while the choice of agent can marginally influence performance, the strategic framework of \toolname\ provides a more fundamental and decisive advantage.
\noindent\textbf{Takeaway.}
The effectiveness of \toolname\ is largely independent of the LLM agent. When using a smaller model such as Qwen-2.5-7B-Instruct, it achieves an average coverage of 68.9\%, decisively outperforming baselines. This highlights that the primary strength of our attack lies in its strategic planning, not the raw capability of agent model.
% To investigate how the choice of the attacker’s LLM agent influences \toolname’s performance, we replace the previously used Doubao-1.5-lite-32K with a smaller open-source model, Qwen-2.5-7B, and re-run the attacks. The results in Table~\ref{tab:qwen-coverage} show that \toolname continues to achieve the highest coverage and semantic fidelity, even when using a smaller agent.% The coverage rate curves are presented in Fig.~\ref{fig:qwen-coverage}.
% Overall, Qwen-2.5-7B delivers performance comparable to the larger Doubao-1.5-lite-32K when used as the attacker for \toolname, with only minor variations in coverage. For example, on TREC-COVID, Qwen-2.5-7B achieves a coverage rate of 54.2\%, outperforming Doubao-1.5-lite-32K at 49.4\%. On other datasets, there is a modest decline in performance with Qwen-2.5-7B, yet \toolname still maintains a clear lead over all baselines. On average, \toolname achieves 68.9\% coverage, compared to 34.6\% for RAGThief and 51.3\% for IKEA.
% Interestingly, RAGThief shows slightly improved results with Qwen-2.5-7B, possibly because smaller models tend to remain more focused and less prone to off-topic reasoning, which can enhance exploration in certain cases. Nevertheless, \toolname consistently outperforms all alternatives, demonstrating that its effectiveness is largely independent of the attacker’s LLM capacity.
% \noindent\textbf{Takeway.}
% \toolname’s effectiveness is largely independent of the attacker’s LLM capacity. Even when using a smaller agent such as Qwen-2.5-7B, it maintains the highest coverage (average 68.9\% across datasets on Llama-3-8B) and semantic fidelity (0.596).
\subsection{Defense Robustness (RQ4)}
\label{sec:defense}
\vspace{-1mm}
To further explore potential defenses, we extend our evaluation beyond the test against GPT-4o-mini's built-in guardrails (Sec.~\ref{sec:effectiveness}) to two practical mechanisms widely adopted by RAG: query rewriting and multi-query retrieval. %These defenses are designed to enhance RAG system robustness by sanitizing or expanding user inputs. Our findings indicate that not only are these defenses ineffective against the attacks, but they can, in some cases, be subverted to amplify data extraction.
\input{tables/query-rewriting}
\noindent\textbf{Query Rewriting.}
In this configuration, the victim RAG system employs an LLM to rewrite incoming queries, aiming to clarify user intent and neutralize potential adversarial patterns before retrieval and generation. \toolname\ exhibits exceptional resilience against this defense, maintaining superior performance metrics. As detailed in Table~\ref{tab:query-rewriting-coverage}, \toolname\ achieves an average coverage rate of 74.1\% and semantic fidelity of 0.591. Although intended as a safeguard, the query rewriting process is paradoxically exploited by our method to enhance extraction. By explicitly refining the query’s semantic intent, the rewriter enables the retriever to surface a more relevant and diverse set of documents than the original input would yield. \toolname's adaptive planner capitalizes on this context to accelerate corpus exploration; for instance, on the NFCorpus dataset, coverage increases from 73.8\% to 85.4\% when this defense is active.
In contrast, baseline methods fail to effectively exploit this dynamic. While RAGThief sees its average coverage improve to 57.9\% and IKEA reaches 46.7\%, both remain significantly behind \toolname. RAGThief benefits incidentally, as the rewriting step strips away its obvious adversarial suffixes; however, lacking a strategic framework to systematically capitalize on the enhanced retrieval results, it cannot match \toolname's efficiency, underscoring the fundamental limitations of heuristic-based approaches.
\input{tables/multi-query}
\noindent\textbf{Multi-query Retrieval.}
We next evaluate against multi-query retrieval, which expands each query into several variants whose retrieved results are re-ranked and fused, a process that can influence attack surface. Once again, \toolname\ excels, achieving the highest average coverage of 69.5\% and a semantic fidelity of 0.593 (Table~\ref{tab:multi-query}). This mechanism provides the attacker with a more diverse set of retrieved documents from different semantic clusters. The global graph in \toolname\ is uniquely positioned to exploit this; it integrates this diverse information to build a more comprehensive map of the corpus, thereby generating more effective and far-reaching follow-up queries.
The baseline methods, however, are not equipped to fully leverage this enriched context. IKEA and RAGThief attain coverage rates of 46.1\% and 40.8\%, respectively. Their localized strategies are unable to fully synthesize the information from the multiple retrieval results.%, demonstrating that simply broadening the retrieval scope does not inherently thwart a sophisticated, globally-aware attacker.
\noindent\textbf{Takeaway.}
\toolname\ remains highly effective against practical RAG defenses. %Against a query rewriting defense, it achieves 74.1\% coverage, substantially outperforming RAGThief (57.9\%) and IKEA (46.7\%). Similarly, it maintains its superiority against multi-query retrieval, achieving 69.5\% coverage versus 46.1\% for IKEA and 40.8\% for RAGThief.
The results demonstrate that common defenses designed to sanitize inputs or strenghthen retrieval can be subverted by a strategic attacker and may even amplify the extraction threat, highlighting the need for more advanced safeguards.
% To assess the threat posed by \toolname\ in more realistic scenarios, we evaluate its performance against two widely adopted defense mechanisms: query rewriting and multi-query retrieval. Our findings indicate that not only are these defenses ineffective against a strategic attack, but they can, in some cases, be subverted to amplify knowledge extraction.
% To further approximate real-world conditions and explore potential defenses, beyond the GPT-4o-mini test in Sec.~\ref{sec:effectiveness}, where the generator includes built-in guardrails that refuse unsafe queries, we evaluate \toolname against stronger RAG variants equipped with practical defense mechanisms. Specifically, we consider two strategies widely adopted in deployed RAG systems, namely query rewriting and multi-query retrieval as introduced in Section~\ref{sec:bg-rag}. Both mechanisms are designed to enhance robustness and mitigate potentially harmful or ambiguous inputs. However, they fail to effectively reduce the attack surface and can even amplify extraction in certain cases.
% \noindent\textbf{Query Rewriting.} In this setting, the victim RAG system automatically reformulates user queries before retrieval to sanitize and clarify intent. As reported in Table~\ref{tab:query-rewriting-coverage}, \toolname maintains the best overall performance across all datasets, achieving the highest coverage and semantic fidelity. On average, it reaches 74.1\% coverage, compared to 57.9\% for RAGThief and 46.7\% for IKEA, while its average semantic fidelity remains the highest at 0.591 (RAGThief 0.537, IKEA 0.549).
% Interestingly, coverage occasionally improves under this defense. For example, on NFCorpus, \toolname attains 85.4\% coverage with query rewriting, higher than 73.8\% without it. The reformulated queries often make intent more explicit, allowing the retriever to access additional relevant documents. Although RAGThief also benefits slightly, likely because the rewriting step removes its adversarial suffixes and increases embedding diversity, \toolname still outperforms all baselines due to its adaptive planning and semantic steering.
% \noindent\textbf{Multi-query Retrieval.}
% We next evaluate a multi-query retrieval mechanism where each user query is rewritten into several variants, triggering multiple retrievals whose results are merged through rank fusion. Table~\ref{tab:multi-query} shows that \toolname again achieves the highest coverage and semantic fidelity across all benchmarks.
% On average, \toolname achieves the best performance with a coverage rate of 69.5\% and a semantic fidelity of 0.593. RAGThief also improves under this setting, reaching 40.8\% coverage and 0.511 fidelity, while IKEA attains 46.1\% coverage and 0.557 fidelity.
% %
% Notably, multi-query retrieval also slightly increases performance in some cases. On {SciDocs}, the coverage rate rises from 66.1\% to 69.7\%, and on {NFCorpus}, from 79.7\% to 84.9\%, indicating that broader retrieval improves the diversity of the evidence retrieved and provides richer context for subsequent query generation. This happens because issuing multiple related subqueries allows the retriever to access different yet complementary document clusters, enabling \toolname’s global planner to integrate more semantically diverse information and generate more effective follow-up queries.
% Overall, both query rewriting and multi-query retrieval fail to mitigate the attack. Instead, they often expand the retrieval space or clarify the intent of malicious queries, inadvertently aiding the attacker. The global semantic planning of \toolname enables it to exploit these conditions, maintaining strong extraction capability even under reinforced RAG configurations.
% These findings underscore the limitations of static query-level defenses and highlight the need for adaptive, behavior-aware countermeasures. Future defenses should analyze query provenance and session-level user behavior to detect coordinated extraction patterns, rather than treating each query in isolation.
% \noindent\textbf{Takeways.}
% Both \emph{query rewriting} and \emph{multi-query retrieval} fail to mitigate the attack. These results highlight the limitations of static, query-level defenses and point to the need for adaptive, behavior-aware mechanisms that analyze query provenance and user interactions to detect coordinated extraction attempts.
% To xxx real-world, apart from the GPT-4o-mini test in xxx, we evaluate \toolname against stronger RAG variants that might be deployed to thwart extraction. In particular, we consider two mechanisms commonly used in practice: query rewriting and multi-query retrieval as introduced in Sec~\ref{sec:bg-rag}. These are intended to make the system more robust or secure. Surprisingly, our experiments reveal that such measures do not stop the attack, in fact, they can inadvertently boost the coverage of all attacks in some cases. The intuition is that rewriting tends to make queries more concrete or explicit, and multi-query aggregation broadens the retrieval scope; both can lead to more content being fetched for the attacker.
% %这里需要在table中加delta值吗（相比没有employ this mechanism的）
% \noindent\textbf{Query Rewriting.} We enabled an automated query reformulation step on the victim RAG (simulating a defense that rephrases potentially disallowed queries). Tab.~\ref{tab:query-rewriting-coverage} shows the final coverage and fidelity under this defense. \toolname maintains the best performance across all datasets, still extracting the most content with the highest semantic fidelity. In fact, none of the methods are significantly hampered by the paraphrasing; the coverage of \toolname remains very high (often 70–85\% range) and in some cases even increases relative to the no-defense scenario. For example, on NFCorpus, \toolname achieves 85.4\% coverage with query rewriting – higher than its 73.8\% under the normal setting – while RAGThief jumps from ~38.6\% to 66.4\%. The rewritten queries evidently preserve the attack intent but use clearer or more acceptable wording, which can retrieve additional documents. SF，average 0.608 to 0.591, RAGTheif 0.537, IKEA 0.549. Crucially, \toolname’s careful planning still gives it an edge: it outperforms RAGThief and IKEA on every dataset despite the defense. Average 74.1\% compared to 57.9\% RAGTheif, 46.7\% IKEA. Its semantic fidelity also remains highest (e.g. 0.591 vs 0.537 for IKEA on TREC-COVID), showing that even rewritten queries can be steered by our attacker to stay on-target. RAGTheif比之前的提升更多，因为query rewriting被去掉了jailbreak的attack，让它的query被embedding时可以在空间上更多样（因为之前都加一样的后缀adversarial command，会对embdding有影响）
% \noindent\textbf{Multi-Query Retrieval Defense.} We next tested a scenario where the RAG system issues multiple queries per user query (we used 3 parallel queries with results merged via rank fusion) to dilute any single distorted prompt. Again, \toolname exhibits strong resilience. Tab.~\ref{tab:multi-query} summarizes the outcomes: \toolname still achieves the highest coverage and fidelity in this multi-query setting, outperforming the baselines on all benchmarks. As with query rewriting, the enhanced retrieval breadth actually helps reveal more content overall. For instance, on TREC-COVID, RAGCrawler attains 47.4% coverage under multi-query retrieval, compared to 32.6% for RAGThief and 18.9% for IKEA. The absolute coverage for RAGCRAWLER is a bit lower than in vanilla RAG (since some parallel queries may be redundant), but its advantage over the baselines persists or even widens. We observe similar patterns on other datasets (e.g. RAGCRAWLER 76.1% vs IKEA 60.5% on Healthcare). Qualitatively, multi-query retrieval tends to return a more diverse set of relevant documents, which RAGCRAWLER can incorporate into its global knowledge graph and then formulate follow-up queries to dig even deeper. The fact that it continues to lead in SF as well (e.g. 0.587 vs 0.580 for the next-best on Healthcare) indicates it isn’t sacrificing relevance despite the expanded results.
% In summary, neither defensive strategy is effective at stopping RAGCRAWLER. If anything, they highlight a counter-intuitive vulnerability: making queries “safer” or retrieval broader can feed more material to an adaptive attacker. RAGCRAWLER’s high-level semantic planning is robust to such transformations, allowing it to retain state-of-the-art extraction capability.
% Answer to RQ4: common deployed mechanism for defense or better performance: query rewriting and multi-query aggregation fail. It continues to extract the most content with the best fidelity in these scenarios, sometimes even benefitting from the defense mechanisms. This underscores the difficulty of thwarting such adaptive attacks with superficial query processing alone. %因为它们只对单点的query进行处理，没有adaptive的防御意识
\begin{figure}[b]
\centering
\includegraphics[width=0.85\linewidth]{figs/score-cache-2.pdf}
\caption{Score Computations with vs. without Caching. Caching reduces the number of similarity score updates by $\sim$2.67$\times$ on average, greatly improving efficiency.}
\label{fig:score-cache}
\vspace{-1.5em}
\end{figure}
\subsection{Ablation Study (RQ5)}
\label{sec:ablation}
\vspace{-1mm}
%We analyze the hyperparameter choice
%We analyze the design choice in \toolname.
% Working in concert, the KG-Constructor, Strategy Scheduler, and Query Generator approximate the adaptive greedy policy used for ASCP, making their synergy central to \toolname’s effectiveness. None of these modules can be removed without causing \toolname\ to degenerate into baseline-like behavior: without the knowledge graph, the system loses coverage state to estimate on; without the scheduler, it cannot prioritize unexplored areas and cannot ; without the query generator, planning cannot be executed and the system falls back to static queries. Together, these modules enable \toolname’s global semantic planning and are indispensable for achieving high coverage and semantic fidelity.
% \toolname's effectiveness relies on the coordinated interaction between the KG-Constructor, Strategy Scheduler, and Query Generator. These modules jointly approximate the adaptive greedy policy of ASCP: the knowledge graph encodes global semantic state, the scheduler selects high-CMG anchors, and the generator renders them into adaptive, compliant queries. Ablating any single module breaks this loop, i.e.,disabling long-term planning, exploration prioritization, or query realization, reducing \toolname to a baseline-like local explorer. Thus, all three components are essential and tightly coupled.
We first study the influence of score cache and victim retrieval depth. More ablations on modules and hyperparameter choices are provided in Appendix~\ref{appendix:hyper-parameter}.
\begin{figure}[b]
\centering
\vspace{-1.5em}
\includegraphics[width=0.75\linewidth]{figs/retrieval-depth-2.pdf}
\caption{Coverage Rate vs. Query Number for different victim depth $k$ on TREC-COVID, Llama-3-8B, BGE.}
\label{fig:retrieve_num}
\end{figure}
\noindent\textbf{Score Cache.}
The Strategy Scheduler module includes a score cache to avoid redundant computations of the CMG for similar states. We measured the impact of this optimization on computational overhead. As shown in Fig.~\ref{fig:score-cache}, across the evaluated datasets the attacker would need to update over 1.67 million scoring operations without caching, versus only about 0.63 million with our cache (a 62\% reduction in workload). In practice, this translates to a faster attack runtime and lower cost, enabling the attacker to scale to larger corpora or tighter time budgets. The benefit of caching grows with the number of queries, since later attack rounds re-use many of the same entity-relation state components.
% Average over all datasets, we need to update 1,675,138 scores without the score cache while only 628,092 with score cache (2.67$\times$ reduction).
% We also analyze the impact of key hyperparameters in our method. One is the exploration factor $c$ in the UCB formula used by the strategy scheduler (which controls how much the planner favors less-explored anchors). Another is the similarity threshold $\tau$ in the query generator for deciding when to stop refining an anchor into a query. We vary these values and measure final coverage on a development set (TREC-COVID). Results are shown in
\noindent\textbf{Victim Retrieval Depth.}
We also analyzed sensitivity of \toolname to the victim RAG's retrieval depth (k), the number of documents retrieved per query. As illustrated in Fig.~\ref{fig:retrieve_num} for the TREC-COVID dataset, a larger $k$ allows the attacker to acquire knowledge more rapidly; for instance, coverage within 1,000 queries increases from approximately 28\% at k=5 to over 60\% at k=20. However, this effect is subject to diminishing returns. As $k$ grows, the additional documents retrieved are increasingly likely to have semantic overlap with previously seen content, providing less novel information for the planner to exploit. %This demonstrates that while the attack benefits from more retrieved context, its core efficiency is not solely dependent on this parameter.
% Answer to RQ5?
% \subsection{Cost Analysis (RQ6)}
% \input{tables/costs}
% \toolname\ is exceptionally cost-effective. A full attack consumes 6-10 million LLM tokens, translating to a direct monetary cost of just \$0.33 to \$0.53 per dataset using a commercial API (Table~\ref{tab:cost}). As demonstrated in Sec.~\ref{sec:agent}, this cost can be reduced to \textbf{virtually zero} by using freely available open-source models without a significant performance penalty.
% %
% This minimal attacker expenditure creates a stark economic asymmetry when compared to the high cost of developing the victim RAG, where acquiring a specialized corpus alone can cost tens of thousands of dollars~\cite{adasci2024ragcost}. Consequently, \toolname\ poses a dual threat: it enables low-cost service piracy by allowing an adversary to reconstruct the knowledge base (Sec.\ref{sec:effectiveness}) and exposes the victim to severe legal and privacy liabilities under regulations like HIPAA and GDPR if the exfiltrated data is sensitive\cite{annas2003hipaa,voigt2017eu}.
% In this section, we estimate the cost of executing \toolname under realistic conditions. As reported in Table~\ref{tab:cost}, the attacker processes approximately 6--10 million tokens per dataset. Based on Doubao-1.5-lite-32K’s API pricing (\$0.042 per million input tokens and \$0.084 per million output tokens), the total cost per attack is remarkably low, i.e., roughly \$0.33 to \$0.53 per dataset. Moreover, as shown in Sec.~\ref{sec:agent}, substituting a freely available open-source model such as Qwen-2.5-7B does not meaningfully degrade the performance of \toolname, enabling the attack to be executed at \textbf{virtually zero cost}, aside from any fees for the RAG service and GPU hosting.
% In contrast, manually constructing a RAG corpus is resource-intensive. Acquiring a specialized domain dataset alone can cost tens of thousands of dollars~\cite{adasci2024ragcost}, not to mention the ongoing expenses associated with cleaning, structuring, embedding, and maintaining the index. Once \toolname has extracted the target corpus, the stolen content can be reused to construct a fully functional surrogate RAG system (see Sec.~\ref{sec:effectiveness}), eliminating the need for continuous access to paid services and increasing the risk of unauthorized replication or pirated knowledge bases. If the exfiltrated corpus contains sensitive material (such as clinical conversations, legal documents, or customer records), the attack may also result in serious legal liability under privacy regulations like HIPAA and GDPR~\cite{annas2003hipaa,voigt2017eu}.
%Finally, we assess the cost of carrying out \toolname in a real-world scenario. As shown in Table \ref{tab:cost}, each dataset involved the processing of roughly 6–10 million tokens by the attacker’s components. Using the Doubao-1.5-lite-32k API (priced at approximately \$0.042 per million input tokens and \$0.084 per million output tokens), the entire attack incurs only about \$0.33–\$0.53 per dataset. In other words, \toolname can exfiltrate a private corpus for well under \$1 in API costs. Moreover, as shown in Section \ref{sec:agent}, using a free open-source small model (Qwen-2.5-7B) in place of the API does not degrade RAGCrawler’s performance. This means the attacker could carry out \toolname at virtually \textbf{zero cost}, further highlighting the practical threat. %This low expenditure is significantly cheaper than the cost of legitimately constructing a RAG knowledge corpus or paying for a managed RAG service building an in-house RAG system can require on the order of tens of thousands of dollars upfront and thousands more per month in upkeep
% Finally, we evaluate the cost of running \toolname, in terms of tokens processed and monetary expense, to assess the practicality of the attack. Tab~\ref{tab:cost} breaks down the token usage and estimated API cost for each dataset (using the Doubao-1.5 LLM pricing). The majority of tokens are consumed in the Knowledge Graph Construction (KG-C) stage, where the attacker parses each retrieved answer to update its KG state. This involves feeding the answer text into the LLM to extract entities/relations, which for 1000 queries amounts to about 5.5–9 million input tokens (and ~0.7–1.3M output tokens) depending on the dataset. In contrast, the Query Generator (QG) stage uses relatively few tokens, on the order of only 0.3–0.8M input and a few tens of thousands of output tokens total – since each query is short and the prompts are brief. Summing these, the overall token consumption per attack is quite modest for a modern LLM: roughly 6–10 million input tokens and under 1.5 million output tokens. At DOubao API rates (\$0.042 per 1M input tokens, \$0.084 per 1M output), this translates to a total cost well under \$1 for the entire multi-turn attack on a given corpus. Empirically, our estimates range from \$0.33 (TREC-COVID) to \$0.53 (Healthcare) to exfiltrate each corpus – an almost trivial expense. Even factoring in additional overhead or slightly larger query counts, the attack remains highly affordable. The query budget we assumed (1000 queries) is sufficient to extract 70–80\% of the corpus, as shown above, meaning an attacker could obtain the vast majority of sensitive data for only a few dollars across multiple runs.
% Answer to RQ6: ...
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
%\renewcommand{\arraystretch}{1.1}
\begin{table}[t]
%\small
\caption{Coverage Rate (CR) and Semantic Fidelity (SF) of attacks across datasets and generators (BGE Retriever). Best results are in \textbf{bold}.
\toolname consistently achieves the highest CR and SF, outperforming baselines across all settings.}
\label{tab:all-coverage}
\centering
% \vspace{-0.5em}
\begin{adjustbox}{max width = 0.8\columnwidth}
\begin{tabular}{@{\hspace{2pt}}c@{\hspace{2pt}}|@{\hspace{2pt}}c@{\hspace{2pt}}|@{\hspace{2pt}}c@{\hspace{2pt}}|ccc}
\toprule
\textbf{Dataset} & \textbf{Generator} & \textbf{Metric} & {RAGTheif} & {IKEA} & \textbf{RAGCrawler} \\ \midrule
\multirow{8}{*}{\rotatebox{90}{TREC-COVID}} & \multirow{2}{*}{Llama-3-8B} & CR & 0.131 & {0.161} & \textbf{0.494} \\
& & SF & 0.447 & 0.495 & \textbf{0.591} \\ \cmidrule{2-6}
& \multirow{2}{*}{Command-R} & CR & 0.154 & {0.173} & \textbf{0.544} \\
& & SF & 0.444 & 0.525 & \textbf{0.614} \\ \cmidrule{2-6}
& \multirow{2}{*}{Microsoft-Phi-4} & CR & 0.121 & {0.197} & \textbf{0.474} \\
& & SF & 0.468 & 0.547 & \textbf{0.619} \\ \cmidrule{2-6}
& \multirow{2}{*}{GPT-4o-mini} & CR & 0.000 & {0.197} & \textbf{0.465} \\
& & SF & - & {0.543} & \textbf{0.572} \\ \midrule
\multirow{8}{*}{\rotatebox{90}{Scidocs}} & \multirow{2}{*}{Llama-3-8B} & CR & 0.053 & {0.513} & \textbf{0.661} \\
& & SF & 0.264 & {0.495} & \textbf{0.523} \\ \cmidrule{2-6}
& \multirow{2}{*}{Command-R} & CR & 0.093 & {0.522} & \textbf{0.717} \\
& & SF & 0.295 & {0.514} & \textbf{0.561} \\ \cmidrule{2-6}
& \multirow{2}{*}{Microsoft-Phi-4} & CR & 0.065 & {0.545} & \textbf{0.711} \\
& & SF & 0.324 & {0.534} & \textbf{0.563} \\ \cmidrule{2-6}
& \multirow{2}{*}{GPT-4o-mini} & CR & 0.000 & {0.421} & \textbf{0.516} \\
& & SF & - & 0.475 & \textbf{0.492} \\ \midrule
\multirow{8}{*}{\rotatebox{90}{NFCorpus}} & \multirow{2}{*}{Llama-3-8B} & CR & 0.061 & 0.503 & \textbf{0.797} \\
& & SF & 0.451 & 0.644 & \textbf{0.698} \\ \cmidrule{2-6}
& \multirow{2}{*}{Command-R} & CR & 0.169 & 0.517 & \textbf{0.844} \\
& & SF & 0.467 & 0.656 & \textbf{0.705} \\ \cmidrule{2-6}
& \multirow{2}{*}{Microsoft-Phi-4} & CR & 0.113 & 0.566 & \textbf{0.813} \\
& & SF & 0.487 & 0.663 & \textbf{0.717} \\ \cmidrule{2-6}
& \multirow{2}{*}{GPT-4o-mini} & CR & 0.000 & 0.493 & \textbf{0.631} \\
& & SF & - & 0.631 & \textbf{0.653} \\ \midrule
\multirow{8}{*}{\rotatebox{90}{HealthCare}} & \multirow{2}{*}{Llama-3-8B} & CR & 0.361 & 0.687 & \textbf{0.807} \\
& & SF & 0.536 & 0.588 & \textbf{0.618} \\ \cmidrule{2-6}
& \multirow{2}{*}{Command-R} & CR & 0.170 & 0.592 & \textbf{0.766} \\
& & SF & 0.383 & 0.578 & \textbf{0.582} \\ \cmidrule{2-6}
& \multirow{2}{*}{Microsoft-Phi-4} & CR & 0.052 & 0.588 & \textbf{0.654} \\
& & SF & 0.434 & 0.558 & \textbf{0.599} \\ \cmidrule{2-6}
& \multirow{2}{*}{GPT-4o-mini} & CR & 0.000 & 0.693 & \textbf{0.799} \\
& & SF & - & 0.490 & \textbf{0.577} \\ \bottomrule
\end{tabular}
\end{adjustbox}
\vspace{-1.8em}
\end{table}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
\begin{table}[t]
\caption{Reconstruction Fidelity of Surrogate RAG Systems using extracted knowledge (Llama-3-8B Generator, BGE Retriever). Best results are in \textbf{bold}.
Surrogates built from extracted knowledge of \toolname achieve the highest performance in all metrics.}
\label{tab:reconstruction}
%\vspace{-0.5em}
\centering
\begin{adjustbox}{max width = 0.8\columnwidth}
\begin{tabular}{c|c|c@{\hspace{3pt}}c@{\hspace{3pt}}c}
\toprule
\textbf{Dataset} & \textbf{Method} & \textbf{SuccessRate} & \textbf{Similarity} & \textbf{Rouge-L} \\ \midrule
\multirow{3}{*}{TREC-COVID} & RAGTheif & 0.1129 & 0.4920 & 0.1547 \\
& IKEA & 0.2247 & 0.4779 & 0.1730 \\
& \textbf{RAGCrawler} & \textbf{0.3839} & \textbf{0.6098} & \textbf{0.2408} \\ \midrule
\multirow{3}{*}{SciDocs} & RAGTheif & 0.0486 & 0.4275 & 0.1131 \\
& IKEA & 0.0179 & 0.4829 & 0.1277 \\
& \textbf{RAGCrawler} & \textbf{0.3810} & \textbf{0.5900} & \textbf{0.2285} \\ \midrule
\multirow{3}{*}{NFCorpus} & RAGTheif & 0.0447 & 0.5156 & 0.1063 \\
& IKEA & 0.4215 & 0.6064 & 0.2013 \\
& \textbf{RAGCrawler} & \textbf{0.5259} & \textbf{0.6992} & \textbf{0.2334} \\ \bottomrule
\end{tabular}
\end{adjustbox}
\vspace{-1.5em}
\end{table}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
\begin{table}[t]
\caption{Coverage rate (CR) and semantic fidelity (SF) of attacks on RAG using GTE as retriever (Llama-3-8B generator). \toolname remains dominant.}
\label{tab:gte-coverage}
%\vspace{-0.5em}
\centering
\begin{adjustbox}{max width = 0.8\columnwidth}
\begin{tabular}{c|c|ccc}
\toprule
\textbf{Dataset} & \textbf{Metric} & RAGTheif & IKEA & \textbf{RAGCrawler} \\ \midrule
\multirow{2}{*}{TREC-COVID} & CR & 0.191 & 0.391 & \textbf{0.765} \\
& SF & 0.453 & 0.565 & \textbf{0.610} \\ \midrule
\multirow{2}{*}{SciDocs} & CR & 0.162 & 0.368 & \textbf{0.833} \\
& SF & 0.345 & 0.495 & \textbf{0.539} \\ \midrule
\multirow{2}{*}{NFCorpus} & CR & 0.386 & 0.622 & \textbf{0.738} \\
& SF & 0.589 & 0.648 & \textbf{0.671} \\ \midrule
\multirow{2}{*}{Healthcare} & CR & 0.388 & 0.671 & \textbf{0.728} \\
& SF & 0.548 & 0.576 & \textbf{0.567} \\ \bottomrule
\end{tabular}
\end{adjustbox}
\vspace{-1.5em}
\end{table}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
\begin{table}[t]
\caption{Coverage rate (CR) and semantic fidelity (SF) of each attacks when using Qwen-2.5-7B-Instruct as attacker’s agent (victim RAG: Llama-3-8B generator, BGE retriever). Best results are in \textbf{bold}. \toolname continues to attain the highest coverage and fidelity under a smaller model.}
\label{tab:qwen-coverage}
\centering
%\vspace{-0.5em}
\begin{adjustbox}{max width = 0.8\columnwidth}
\begin{tabular}{c|c|ccc}
\toprule
\textbf{Dataset} & \textbf{Metric} & RAGTheif & IKEA & \textbf{RAGCrawler} \\ \midrule
\multirow{2}{*}{TREC-COVID} & CR & 0.271 & 0.183 & \textbf{0.542} \\
& SF & 0.453 & 0.499 & \textbf{0.610} \\ \midrule
\multirow{2}{*}{SciDocs} & CR & 0.314 & 0.559 & \textbf{0.675} \\
& SF & 0.345 & 0.495 & \textbf{0.539} \\ \midrule
\multirow{2}{*}{NFCorpus} & CR & 0.386 & 0.622 & \textbf{0.738} \\
& SF & 0.589 & 0.648 & \textbf{0.671} \\ \midrule
\multirow{2}{*}{Healthcare} & CR & 0.414 & 0.687 & \textbf{0.799} \\
& SF & 0.548 & 0.576 & \textbf{0.567} \\ \bottomrule
\end{tabular}
\end{adjustbox}
\vspace{-1.5em}
\end{table}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
\begin{table}[b]
\vspace{-1.5em}
\caption{Coverage rate (CR) and semantic fidelity (SF) of attacks under \emph{query rewriting} (victim RAG: Llama-3-8B generator, BGE retriever). Best results are in \textbf{bold}. \toolname still achieves the highest coverage and fidelity in this setting, indicating resilience to \emph{query rewriting}.}
\label{tab:query-rewriting-coverage}
\centering
%\vspace{-0.5em}
\begin{adjustbox}{max width = 0.8\columnwidth}
\begin{tabular}{c|c|ccc}
\toprule
\textbf{Dataset} & \textbf{Metric} & RAGTheif & IKEA & \textbf{RAGCrawler} \\ \midrule
\multirow{2}{*}{TREC-COVID} & CR & 0.381 & 0.241 & \textbf{0.601} \\
& SF & 0.519 & 0.537 & \textbf{0.591} \\ \midrule
\multirow{2}{*}{SciDocs} & CR & 0.561 & 0.542 & \textbf{0.743} \\
& SF & 0.442 & 0.479 & \textbf{0.507} \\ \midrule
\multirow{2}{*}{NFCorpus} & CR & 0.664 & 0.489 & \textbf{0.854} \\
& SF & 0.633 & 0.618 & \textbf{0.687} \\ \midrule
\multirow{2}{*}{Healthcare} & CR & 0.709 & 0.595 & \textbf{0.767} \\
& SF & 0.556 & 0.564 & \textbf{0.577} \\ \bottomrule
\end{tabular}
\end{adjustbox}
\end{table}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
\begin{table}[t]
\caption{Coverage rate (CR) and semantic fidelity (SF) of attacks under \emph{multi-query retrieval} (victim RAG: Llama-3-8B generator, BGE retriever). Best results are in \textbf{bold}. \toolname still achieves the highest coverage and fidelity, indicating resilience to \emph{multi-query retrieval}.}
\label{tab:multi-query}
\centering
%\vspace{-0.5em}
\begin{adjustbox}{max width = 0.8\columnwidth}
\begin{tabular}{c|c|ccc}
\toprule
\textbf{Dataset} & \textbf{Metric} & RAGTheif & IKEA & \textbf{RAGCrawler} \\ \midrule
\multirow{2}{*}{TREC-COVID} & CR & 0.326 & 0.189 & \textbf{0.474} \\
& SF & 0.525 & 0.523 & \textbf{0.581} \\ \midrule
\multirow{2}{*}{SciDocs} & CR & 0.352 & 0.508 & \textbf{0.697} \\
& SF & 0.364 & 0.494 & \textbf{0.514} \\ \midrule
\multirow{2}{*}{NFCorpus} & CR & 0.392 & 0.540 & \textbf{0.849} \\
& SF & 0.588 & 0.631 & \textbf{0.692} \\ \midrule
\multirow{2}{*}{Healthcare} & CR & 0.562 & 0.605 & \textbf{0.761} \\
& SF & 0.569 & 0.580 & \textbf{0.587} \\ \bottomrule
\end{tabular}
\end{adjustbox}
\vspace{-1.5em}
\end{table}
\section{Discussion and Related Work}
\noindent\textbf{Cost Analysis.}
\toolname\ is highly cost-effective, with a monetary cost of only \$0.33–\$0.53 per dataset (Tab.~\ref{tab:cost}) and is reducible to \textbf{near-zero} using open-source models (Sec.~\ref{sec:agent}). This minimal expenditure creates a stark economic asymmetry against the high development costs of victim RAG systems~\cite{adasci2024ragcost}. This asymmetry poses a dual threat: enabling low-cost service piracy by reconstructing the knowledge base (Tab.~\ref{tab:reconstruction}) and exposing victims to severe legal and privacy liabilities under regulations like HIPAA and GDPR if sensitive data is exfiltrated~\cite{annas2003hipaa,voigt2017eu}.
% \toolname\ is exceptionally cost-effective with a direct monetary cost of just \$0.33 to \$0.53 per dataset
% using a commercial API (Table~\ref{tab:cost}). As demonstrated in Sec.~\ref{sec:agent}, this cost can be reduced to \textbf{virtually zero} by using freely available open-source models without a significant performance penalty.
% %
% This minimal attacker expenditure creates a stark economic asymmetry when compared to the high cost of developing the victim RAG, which can cost tens of thousands of dollars~\cite{adasci2024ragcost}. Consequently, \toolname\ poses a dual threat: it enables low-cost service piracy by allowing an adversary to reconstruct the knowledge base (Tab.\ref{tab:reconstruction}) and exposes the victim to severe legal and privacy liabilities under regulations like HIPAA and GDPR if the exfiltrated data is sensitive\cite{annas2003hipaa,voigt2017eu}
\noindent\textbf{Defenses.} %Our attack's use of individually benign queries makes effective defense challenging without harming system performance. We found contemporary defenses like built-in guardrails and query rewriting largely ineffective, as they analyze queries in isolation. Crucially, the malicious pattern is not evident in any single interaction but emerges only from the sequence as a whole. This exposes a critical vulnerability in static security models and underscores the need for dynamic, behavior-aware defenses.
%
Our attack highlights the inherent limitations of defenses that rely on analyzing queries in isolation. Its use of individually benign queries makes it difficult to counter with conventional tools like guardrails or query rewriting without significant performance penalties. The core challenge is that the malicious pattern is an emergent property of the entire interaction sequence, not an attribute of any single query. This exposes a critical blind spot in static security models and argues for a pivot towards dynamic, behavior-aware defenses.
%
Based on this analysis, we believe query provenance analysis~\cite{li2025query,han2020unicorn,li2023nodlink,milajerdi2019holmes} holds significant promise. Because this technique is designed to track relationships across entire sequences, it is theoretically capable of identifying the progressive, goal-oriented patterns of an extraction attack. Verifying its practical effectiveness would be a valuable next step for the research community.
%
%To this end, we propose query provenance analysis~\cite{li2025query,han2020unicorn,li2023nodlink,milajerdi2019holmes} as a promising direction. By analyzing entire query sequences, this approach may be well-suited to detect the progressive, goal-oriented patterns characteristic of an extraction attack. Exploring the feasibility and performance trade-offs of such a stateful defense mechanism presents a critical and compelling avenue for future research. %While a full implementation and comprehensive evaluation of this approach are beyond the scope of this work, they represent a critical and important avenue for future research.
\noindent\textbf{Security Risks in RAG.} This paper concentrates on Data Extraction Attacks against RAG systems, where an adversary queries the system to steal or reconstruct its private knowledge base~\cite{qifollow, jiang2024rag, wang2025silent, di2024pirates, cohen2024unleashing}.
%
RAG systems are also susceptible to other security risks~\cite{lasso2024ragsecurity,ni2025towards2}. Membership Inference Attacks (MIA) enable an adversary to determine if a specific document is present in the private corpus, thereby exposing the existence of sensitive records~\cite{liu2025mask, naseh2025riddle, anderson2025my, feng2025ragleak}. Furthermore, in Corpus Poisoning attacks, an adversary injects malicious data into the RAG knowledge base~\cite{zou2025poisonedrag, chaudhari2024phantom, cho2024typos, shafran2024machine}. Retrieval of this corrupted data can manipulate the system's output, allowing the attacker to propagate misinformation or harmful content.
% \noindent\textbf{Web Crawling.} Traditional web crawling involves systematically discovering and updating content on the open web, adhering to resource limitations and politeness constraints~\cite{olston2010web, cho2003effective}. While web crawling primarily supports search engines by indexing {public} webpages, RAG Crawling, as explored in this work, focuses on {private} or {proprietary} corpora.
%
% This failure of static, per-query defenses points to the need for adaptive, behavior-aware mechanisms. The sequence of queries in our attack may exhibit a clear pattern of topical progression. Therefore, we suggest a promising future direction is to investigate query provenance analysis~\cite{li2025query}. By analyzing the entire sequence of user interactions to identify suspicious patterns, such systems could detect coordinated extraction attempts where isolated query analysis fails.
% \noindent\textbf{Security Risks in RAG.} In this paper, we focus on one critical vulnerability in RAG systems: Data Extraction Attacks~\cite{qifollow, jiang2024rag, wang2025silent, di2024pirates, cohen2024unleashing}. This threat is analogous to model stealing in conventional machine learning, where the goal is to replicate or steal a proprietary model. In the context of RAG systems, a corpus extraction attack targets the knowledge base, which is often the most valuable asset. The adversary systematically queries the system to steal or reconstruct the private database, capturing sensitive intellectual property or data that defines the RAG application's capabilities, which may include confidential business strategies, proprietary research, or personal and medical information.
% RAG, as a hybrid technology, introduces other significant security risks. One prominent threat is the Membership Inference Attack (MIA)~\cite{liu2025mask, naseh2025riddle, anderson2025my, feng2025ragleak}, where attackers query a RAG system to determine if a specific document exists within its private corpus. This can expose existence of sensitive information, such as confidential strategic plans or clinical records.
% Another risk is Corpus Poisoning, similar to traditional data poisoning. In this attack, an adversary injects false or misleading data into the RAG system's external knowledge base~\cite{zou2025poisonedrag, chaudhari2024phantom, cho2024typos, shafran2024machine}. When the system retrieves this corrupted data to generate a response, the output can be manipulated, allowing the attacker to spread misinformation or induce harmful content.
\section{Conclusion}
In this work, we formalized the critical privacy risk of data extraction from RAG systems as an adaptive stochastic coverage problem. We introduced \toolname, a novel attack framework that builds a knowledge graph to track revealed information. Our comprehensive experiments demonstrated that \toolname significantly outperforms all baselines, achieving up to 84.4\% corpus coverage within a fixed query budget. Furthermore, it shows remarkable robustness against advanced RAG that employs query rewriting and multi-query retrieval. Our findings expose a fundamental vulnerability in current RAG architectures, underscoring the urgent need for robust safeguards to protect private knowledge bases.
\section{Appendix}
\subsection{Sample Distribution Validation}
\label{appendix:sample-validation}
To ensure that our 1,000-document samples are representative of the full corpora, we compare their semantic distributions using Energy Distance~\cite{szekely2013energy} and the Classifier Two-Sample Test (C2ST)~\cite{lopez2016revisiting}. %These two metrics assess whether two samples come from the same distribution without strong parametric assumptions, even in high-dimensional settings.
Energy Distance quantifies the difference between two distributions using Euclidean distances, and is well-suited for high-dimensional or non-Gaussian data. A smaller energy distance with a large permutation test $p$-value suggests no statistically significant difference between the sampled and full sets. As shown in Table~\ref{tab:sample-dist}, all four corpora yield low distances (around 0.03) and high $p$-values ($> 0.5$), indicating strong alignment between samples and full sets.
C2ST (Classifier Two-Sample Test) reframes distribution comparison as a binary classification task . If a classifier cannot distinguish between samples and the full corpus (AUC $\approx 0.5$), the two distributions are considered similar. In Table~\ref{tab:sample-dist}, all AUC scores fall within the 0.46–0.51 range, reinforcing that no meaningful distinction can be learned between the subsets and their corresponding full corpora.
\input{tables/sample-validation}
\subsection{Proof of Theorems}
\label{appendix:proof}
\noindent\textbf{Proof of Theorem 2: Adaptive Monotonicity and Submodularity in RAG Crawling}
\begin{proof}
\textbf{(Adaptive Monotonicity)}
The coverage function $f(S, \Phi)$ is monotone with respect to the query set $S$. Issuing an additional query $q$ can only increase or preserve the number of unique documents retrieved. That is, for any realization of $\Phi$,
\[
f(S \cup \{q\}, \Phi) \ge f(S, \Phi).
\]
Taking the expectation conditioned on $\psi$ preserves this inequality, so $\Delta(q \mid \psi) \ge 0$.
\textbf{(Adaptive Submodularity)}
Suppose $\psi'$ extends $\psi$, meaning that the attacker has issued more queries and observed more information. Let $S$ and $S'$ denote the corresponding sets of issued queries, where $S \subseteq S'$. Since more of the corpus has potentially been covered under $S'$, the additional gain from issuing a new query $q$ can only decrease. Formally, for any realization of $\Phi$, we have
\[
f(S \cup \{q\}, \Phi) - f(S, \Phi) \ge f(S' \cup \{q\}, \Phi) - f(S', \Phi).
\]
Taking expectation conditioned on $\psi$ and $\psi'$ respectively, we obtain
\[
\Delta(q \mid \psi) \ge \Delta(q \mid \psi').
\]
\end{proof}
\subsection{Additional Ablations}
\label{appendix:hyper-parameter}
\noindent\textbf{Modules.} The effectiveness of \toolname\ stems from the synergistic integration of its three architectural modules.
Without the KG-Constructor, the system loses its global state memory, becoming a stateless, reactive loop unable to distinguish explored regions, a behavior characteristic of IKEA. Without the Strategy Scheduler, the system cannot perform UCB-based prioritization; a fallback to random anchor selection would mimic IKEA’s keyword-based strategy. Finally, without the Query Generator, the scheduler's strategic plan cannot be translated into executable queries, rendering the planning moot.
\begin{figure}[t]
\centering
\includegraphics[width=0.85\linewidth]{figs/parameter_choice.pdf}
\vspace{-0.5em}
\caption{Hyperparameter Choices of \toolname.}
\label{fig:hyperparameter}
\vspace{-1.5em}
\end{figure}
\noindent\textbf{Hyperparameter Choices.}
We examine two key hyperparameters in \toolname: the UCB exploration coefficient $c$ used by the Strategy Scheduler for query selection, and the similarity threshold $\tau$ used by the Query Generator to filter high-similarity queries (thereby controlling early stopping).
%
As shown in Fig.~\ref{fig:hyperparameter}, a moderate UCB coefficient of $c = 0.5$ delivers the best performance by balancing exploitation and exploration. Increasing or decreasing $c$ from this value noticeably degrades performance, underscoring that $c = 0.5$ is near-optimal for a proper exploration–exploitation trade-off. Similarly, the threshold $\tau$ exhibits a clear sweet spot. Setting $\tau$ too low often causes premature termination. On the other hand, an overly high threshold produces queries that are not sufficiently distinct from one another, which limits exploration efficiency. Our chosen $\tau=0.8$ navigates between these extremes.
\subsection{Coverage Rate Curves}
\label{appendix:all-coverage}
For completeness, we provide all Coverage Rate curves across datasets and settings for all experiments in Sec~\ref{sec:effectiveness}, Sec.~\ref{sec:retriever} and Sec.~\ref{sec:agent} in Fig.~\ref{fig:all-coverage}, Fig.~\ref{fig:gte-coverage} and Fig.~\ref{fig:qwen-coverage}.
\begin{figure*}
\centering
\includegraphics[width=0.85\linewidth]{figs/main-coverage.pdf}
\caption{Coverage Rate vs. Query Number (1,000 Budget) across four datasets and four generators.}
\label{fig:all-coverage}
\vspace{-1em}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=0.85\linewidth]{figs/gte.pdf}
\caption{Coverage Rate vs. Query Number (1,000 Budget) across four datasets (GTE Retriever, Llama-3-8B generator).}
\label{fig:gte-coverage}
\end{figure*}
\begin{figure*}
\centering
\includegraphics[width=0.85\linewidth]{figs/qwen.pdf}
\caption{Coverage Rate vs. Query Number (1,000 Budget) across four datasets with Qwen-2.5-7B-Instruct as attacker LLM (BGE Retriever, Llama-3-8B generator).}
\label{fig:qwen-coverage}
%\vspace{-1em}
\end{figure*}
\subsection{Prompts for Experimental Stages}
\label{appendix:prompts}
We document the exact prompts used in RAG settings. More prompts in our method can be found in our code.
\begin{tcolorbox}[colframe=green!50!black, colback=green!5!white, coltitle=white, title=RAG System Prompt]
You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, say that you don't know.
\textbf{Context:} [context]
\textbf{Query:}:[query]
\textbf{Answer:}
\end{tcolorbox}
\begin{tcolorbox}[colframe=blue!50!black, colback=blue!5!white, coltitle=white, title=Query Rewriting Prompt in RAG System]
You are an helpful assistant. Your task is to rewrite the given user question to clarify the user's intent and remove any adversarial intent. Provide directly output the rewritten question without any other text.
\textbf{Original Question:} [query]
\textbf{Answer:}
\end{tcolorbox}
\begin{tcolorbox}[colframe=orange!50!black, colback=orange!5!white, coltitle=white, title=Multi-query Prompt in RAG System]
You are an helpful assistant. Your task is to generate 3 different versions of the given user question to retrieve relevant documents from a vector database. By generating multiple perspectives on the user question, your goal is to help the user overcome some of the limitations of the distance-based similarity search. Provide directly output these alternative questions separated by newlines without any other text.
\textbf{Original Question:} [query]
\textbf{Answer:}
\end{tcolorbox}
\subsection{Cost of the attack}
We estimate the cost of executing \toolname under realistic conditions. As reported in Table~\ref{tab:cost}, the attacker processes approximately 6--10 million tokens per dataset. Based on Doubao-1.5-lite-32K’s API pricing (\$0.042 per million input tokens and \$0.084 per million output tokens), the total cost per attack is remarkably low, i.e., roughly \$0.33 to \$0.53 per dataset. Moreover, as shown in Sec.~\ref{sec:agent}, substituting a freely available open-source model such as Qwen-2.5-7B does not degrade the performance of \toolname, enabling the attack to be executed at \textbf{virtually zero cost}, aside from any fees for the RAG service and GPU hosting.
\input{tables/costs}
\subsection{Additional Experiments Results}
\begin{figure}[t]
\centering
\includegraphics[width=0.85\linewidth]{figs/additional_curve.pdf}
\caption{Coverage Rate vs. Query Number (5,000 Budget) on NFCorpus and Healthcare with GPT-4o-mini as attacker LLM (BGE Retriever, Llama-3-8B generator).}
\label{fig:additional_curve}
%\vspace{-1.5em}
\end{figure}
To further investigate the impact of scale, a complementary study was conducted with an expanded configuration of 2,000 documents and a 5,000-Budget. The coverage rate curves for the NFCorpus and Healthcare datasets from this study are depicted in Fig.~\ref{fig:additional_curve}. %More additional experiments results are included in our repository \url{https://anonymous.4open.science/r/RAGCrawler}.
\begin{table}[t]
\centering
\caption{Semantic distribution similarity between each full corpus and its 1,000-document sampled subset, measured by Energy Distance (with permutation test $p$-value) and the Classifier Two-Sample Test (C2ST, using AUC). A small Energy Distance with a high $p$-value and an AUC near 0.5 indicate distributional similarity.}
\label{tab:sample-dist}
%\vspace{-0.5em}
\begin{adjustbox}{max width = 0.8\columnwidth}
\begin{tabular}{lcc}
\toprule
\textbf{Corpus} & \textbf{Energy Distance ($value \,/\, p$)} & \textbf{C2ST (AUC)} \\
\midrule
TREC-COVID & $0.0349 \,/\, 0.5249$ & $0.4993$ \\
SciDocs & $0.0329 \,/\, 0.5781$ & $0.4938$ \\
NFCorpus & $0.0281 \,/\, 0.6412$ & $0.5053$ \\
Healthcare & $0.0276 \,/\, 0.5947$ & $0.4679$ \\
\bottomrule
\vspace{-3.5em}
\end{tabular}
\end{adjustbox}
\end{table}
\begin{table}[b]
\vspace{-1.5em}
\caption{Token usage and estimated cost per corpus for KG-Constructor (KG-C) and Query Generator (QG) using Doubao-1.5-lite-32k. “InTok” and “OutTok” denote input and output token counts. Each attack costs under \$1.}
\label{tab:cost}
\centering
%\vspace{-0.5em}
\begin{adjustbox}{max width = 0.8\columnwidth}
\begin{tabular}{l|c@{\hspace{4pt}}c@{\hspace{4pt}}c@{\hspace{4pt}}c}
\toprule
\textbf{Metric} & \textbf{TREC-COVID} & \textbf{SciDocs} & \textbf{NFCorpus} & \textbf{Healthcare} \\
\midrule
KG-C InTok (M) & 5.561 & 6.209 & 5.997 & 9.094 \\
KG-C OutTok (M) & 0.724 & 1.277 & 1.076 & 1.373 \\
QG InTok (M) & 0.847 & 0.596 & 0.366 & 0.783 \\
QG OutTok (M) & 0.042 & 0.027 & 0.017 & 0.035 \\
\midrule
\textbf{Total Cost (\$)} & \textbf{0.333} & \textbf{0.395} & \textbf{0.359} & \textbf{0.533} \\
\bottomrule
\end{tabular}
\end{adjustbox}
\end{table}