AI-Powered Assistant for Long-Term Access to RHIC Knowledge
Extracted source text · arXiv 2509.09688
%%%%%%%%%%%%%%%%%%%%%%%%%% example_onecolumn.tex %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% This is example_onecolumn.tex, an example file for use with the SIAM LaTeX2E
% Proceedings macros. It is designed to provide single-column output. Please
% take the time to read the following comments, as they document how to use the
% SIAM Proceedings macro files. This file can be composed and printed out for
% use as sample output.
%
% Please submit suggestions for changes to: tex@siam.org.
%
% This file is to be used as an example for style only. It should not be read
% for content.
%%%%%%%%%%%%%%% PLEASE NOTE THE FOLLOWING STYLE RESTRICTIONS %%%%%%%%%%%%%%%
%% 1. There are no new tags. Existing LaTeX tags have been formatted to match
%% the SIAM Proceedings style.
%%
%% 2. Do not change the margins or page size! Do not change from the default
%% text font or the default text size of 10pt!
%%
%% 3. We recommend that you use BibTeX and siamplain.bst to list your references.
%% You can use \cite in the text to mark your reference citations and
%% \bibitem in the listing of references at the end of your chapter. See
%% the examples in the following file. If you do use BibTeX, please supply
%% your bib or bbl file with the manuscript file.
%%
%% 4. This macro is set up for two levels of headings (\section and
%% \subsection). The macro will automatically number the headings for you.
%%
%% 5. No running heads are to be used.
%%
%% 6. Theorems, Lemmas, Definitions, Equations, etc. are to be double numbered,
%% indicating the section and the occurrence of that element within that
%% section. (For example, the first theorem in the second section would be
%% numbered 2.1. The macro will automatically do the numbering for you.
%%
%% 7. Figures and Tables must be single-numbered. Use existing LaTeX tags for
%% these elements. Numbering will be done automatically.
%%
%% 8. Page numbering is not included in this macro since pagination
%% will be set by the program committee.
%%
\documentclass[twoside,leqno]{article}
% Comment out the line below if using A4 paper size
\usepackage[letterpaper]{geometry}
\usepackage{siamproceedings}
\usepackage{wrapfig}
\usepackage[T1]{fontenc}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{epstopdf}
\usepackage{enumitem}
\usepackage{color}
\usepackage{algorithmic}
\usepackage[normalem]{ulem}
\ifpdf
\DeclareGraphicsExtensions{.eps,.pdf,.png,.jpg}
\else
\DeclareGraphicsExtensions{.eps}
\fi
% Add a serial/Oxford comma by default.
\newcommand{\creflastconjunction}{, and~}
\newcommand{\atif}[1]{{\color{red}{#1}}}
% Used for creating new theorem and remark environments
\newsiamremark{remark}{Remark}
\newsiamremark{hypothesis}{Hypothesis}
\crefname{hypothesis}{Hypothesis}{Hypotheses}
\newsiamthm{claim}{Claim}
%%Currently, the algorithm title font matches the figure and table title
%%fonts. To make the algorithm title font appear as small caps, uncomment
%%the following code:
%\makeatletter
%\renewcommand{\ALG@name}{\sc Algorithm}
%\makeatother
\usepackage{amsopn}
\DeclareMathOperator{\diag}{diag}
\begin{document}
\title{\Large AI-Powered Assistant for Long-Term Access to RHIC Knowledge}
\author{
Mohammad Atif \thanks{Computational Science Department, Brookhaven National Lab, Upton, NY (USA) \textsuperscript{\P}fmohammad@bnl.gov} \textsuperscript{\P}
\and Vincent Garonne\thanks{Physics Department, Brookhaven National Laboratory, Upton, NY (USA) \textsuperscript{\S}jeromel@bnl.gov}
\and Eric Lancon\textsuperscript{\dag}
\and Jerome Lauret\textsuperscript{\dag \S}
\and Alexandr Prozorov\thanks{FNSPE, Czech Technical University, Prague, Czech Republic}
\and Michal Vranovsky\textsuperscript{\ddag}
}
% (\email{fmohammad@bnl.gov}).
% (\email{vincent.garonne@bnl.gov}).
% (\email{elancon@bnl.gov}).
% (\email{jeromel@bnl.gov}).
% ( \email{alexandr.prozorov@cvut.cz}).
% (\email{mvranovsk@ssh.sdcc.bnl.gov}).
\date{}
\maketitle
% Copyright Statement
% When submitting your final paper to a SIAM proceedings, it is requested that you include
% the appropriate copyright in the footer of the paper. The copyright added should be
% consistent with the copyright selected on the copyright form submitted with the paper.
% Please note that "20XX" should be changed to the year of the meeting.
% Default Copyright Statement
\fancyfoot[R]{\scriptsize{Copyright \textcopyright\ 20XX by SIAM\\
Unauthorized reproduction of this article is prohibited}}
% Depending on which copyright you agree to when you sign the copyright form, the copyright
% can be changed to one of the following after commenting out the default copyright statement
% above.
%\fancyfoot[R]{\scriptsize{Copyright \textcopyright\ 20XX\\
%Copyright for this paper is retained by authors}}
%\fancyfoot[R]{\scriptsize{Copyright \textcopyright\ 20XX\\
%Copyright retained by principal author's organization}}
%\pagenumbering{arabic}
%\setcounter{page}{1}%Leave this line commented out.
\begin{abstract}
As the Relativistic Heavy Ion Collider (RHIC) at Brookhaven National Laboratory concludes 25 years of operation, preserving not only its vast data holdings ($\sim$1 ExaByte) but also the embedded scientific knowledge becomes a critical priority.
The RHIC Data and Analysis Preservation Plan (DAPP) introduces an AI-powered assistant system that provides natural language access to documentation, workflows, and software, with the aim of supporting reproducibility, education, and future discovery.
Built upon Large Language Models using Retrieval-Augmented Generation and the Model Context Protocol, this assistant indexes structured and unstructured content from RHIC experiments and enables domain-adapted interaction.
We report on the deployment, computational performance, ongoing multi-experiment integration, and architectural features designed for a sustainable and explainable long-term AI access.
Our experience illustrates how modern AI/ML tools can transform the usability and discoverability of scientific legacy data.
\end{abstract}
\section{Introduction.}
The Relativistic Heavy Ion Collider (RHIC) has operated since the year 2000, generating over 600 publications, and will end with close to an exabyte of data toward its end of life. RHIC has been shaping the nuclear physics community through its flagship experiments: STAR, PHENIX, and sPHENIX \cite{abdallah2022search,abdulhamid2024estimate,adam2020beam,adam2020bulk,STAR_2005gfr,ichimiya2009status,roland2021sphenix,tannenbaum2019highlights}. With final data collection concluding in 2025, the RHIC Data and Analysis Preservation Plan (DAPP) was launched to ensure that this irreplaceable legacy remains accessible and reusable for scientific and educational purposes. DAPP aims at preserving not only the raw data and published results, but the vast repository of tacit knowledge—experimental know-how, analysis techniques, detector-specific insights, and institutional memory—that typically resides only in the minds of researchers and scattered across internal documentation. This challenge is particularly acute given the decades-long timescales of the experiments, where today's graduate students may need to access and understand methodologies developed twenty years ago.
Among DAPP's key innovations is an AI-powered assistant that enables users to query preserved knowledge via natural language, dramatically lowering the barrier to reuse and understanding of complex datasets and workflows.
Unlike web search or public-facing Large Language Models like ChatGPT, Gemini, which rely on publicly available data, our AI assistant can tap into a wealth of specific, highly relevant, trusted, and curated knowledge tailored to our experiments, collaborations, and internal research processes.
This paper presents a custom web scraper followed by architecture and initial deployment of the AI assistant within STAR, and its future as a generalizable framework for sustainable scientific data access.
% 3.3 Integration with Preservation Infrastructure
% The assistant is part of a broader ecosystem including:
% OpenData portal: For curated, public-access datasets.
% InvenioRDM repository: For publications, metadata, and DOIs.
% REANA workflow engine: For executing preserved analyses in containers.
% Once our home-grown ChatBot is integrated with in-house, proprietary information, it will become an indispensable tool for our scientific community.
% a scientific ChatBot designed to outperform general AI by providing accurate, relevant, and deep contextual answers from trusted scientific sources.
% A home-grown chatbot offers personalized, secure access using proprietary data.
% Our chatbot should provide scientifically or technically "weighted" answers, offering more in-depth information than typical publicly accessible tools designed for non-experts.
\subsection*{Challenges}
\label{sec:motivation}
Building such a system requires addressing several critical technical and operational demands beyond conventional chatbots. The assistant must ingest and organize many PB of heterogeneous data while fusing semantic search with containerized execution environments, enabling users to retrieve archived results and reproduce analyses without reviving legacy software stacks.
Strict collaboration agreements necessitate fine-grained role- and group-aware access controls that respect data-security tiers (public, collaboration-restricted, and controlled) while honoring publication embargoes. To maintain credibility, the system continuously refreshes its knowledge base with real-time literature and facility updates, validated against technically demanding insider-level questions that routinely challenge open web tools.
Usability requirements drive customizable user profiles that adapt explanation depth, visualization style, and API integration for novices, seasoned analysts, and detector experts alike, with engagement metrics informing continuous improvement.
This paper presents the architecture and initial deployment of this AI assistant using STAR data as the first implementation, establishing a template for future experiments like the Electron-Ion Collider (EIC) where knowledge preservation could become an integral part of experimental design from its outset.
% An in-house conversational AI for a large high-energy-physics facility must satisfy far more demanding motivations and confront sharper challenges than generic web chatbots.
% Its primary goal is to preserve and democratize decades of highly specialized knowledge while accelerating day-to-day scientific workflows.
% This requires ingesting and organizing PB of heterogeneous RHIC data, legacy code, and expert “tacit” know-how that is disappearing as senior collaborators retire.
% The assistant must therefore fuse semantic search with containerized execution environments so that users can retrieve archived results and reproduce analyses without reviving software stacks.
% Because experiments operate under strict collaboration agreements and data-security tiers (public, collaboration-restricted, and controlled), the interface must incorporate fine-grained role- and group-aware access controls, delivering answers that respect both privacy and publication embargoes.
% To remain credible, the system also has to refresh its internal models with real-time external literature and facility status, and to expose an evaluation suite of technically tricky, insider-level questions that routinely stump open web tools—thereby quantifying the added value of domain conditioning.
% Finally, usability drives the need for customizable profiles that adapt the depth of explanation, visualization style, and API hooks to novices, seasoned analysts, or detector experts alike, while engagement metrics and iterative search logs feed continuous improvement.
% Together, these requirements frame a chatbot whose success hinges on balancing scientific accuracy, security, and personalized utility within an ever-evolving experimental landscape.
\section{Recursive Multi-Format Web-Content Extraction Framework}
\begin{wrapfigure}{r}{0.43\textwidth}
\begin{center}
\includegraphics[width=0.45\textwidth]{figures/scraper.pdf}
\end{center}
\caption{Recursive Multi-Format Web-Content Extraction Framework}
\label{fig:scraper}
\end{wrapfigure}
% **** We are missing a short intro speaking of the diverse elements?
% == Try 2
As foundational groundwork for building the AI assistant's knowledge base, we developed a comprehensive web content indexing system capable of systematically harvesting and processing diverse digital archives containing decades of experimental documentation, analysis notes, and institutional knowledge scattered across collaboration websites and document repositories. The bespoke multi-format content harvesting framework creates analysis-ready textual corpora from target web domains with deliberate focus on respectful rate-limiting and high-fidelity data extraction from academic and archival research environments (see Figure \ref{fig:scraper}).
The framework's architecture uses an iterative crawling mechanism starting from seed URLs, employing requests and BeautifulSoup \cite{BeautifulSoup} libraries to parse HTML pages and discover hyperlinks. A crucial "dynamic filter" prevents following external links, curating highly relevant, domain-specific datasets, while configurable blacklists exclude irrelevant content such as calendars and login pages.%—essential for maintaining signal-to-noise ratio.
Beyond standard HTML parsing, the system identifies linked documents by file extensions and invokes specialized parsers valuable for institutional repositories with mixed formats. For PDFs, it leverages the advanced marker library's machine learning models to convert visual structure into clean Markdown, significantly improving over traditional tools by preserving layout and semantic meaning \cite{marker_pdf}. The framework handles modern and legacy Microsoft Office documents (.docx, .doc, .pptx, .ppt) and PostScript files (.ps, .eps) through intelligent use of command-line utilities like LibreOffice and Ghostscript for on-the-fly conversions. Upon extraction, text undergoes rigorous cleaning and receives comprehensive metadata headers containing source URL, page title, and timestamp, ensuring clear data provenance. The framework reports detailed statistics on redirects, filtered links, and file extension frequencies, providing diagnostics that facilitate reproducibility studies and inform scraping optimization, thus creating a high-fidelity foundation for AI knowledge base construction.
% === I resshaped, please check
% As foundational groundwork for building the AI assistant's knowledge base, we first developed a comprehensive web content indexing system capable of systematically harvesting and processing the diverse digital archives that contain decades of experimental documentation, analysis notes, and institutional knowledge scattered across collaboration websites and document repositories. The developed bespoke multi-format content harvesting framework for creating comprehensive, analysis-ready textual corpora from target web domains reflects a deliberate focus on respectful rate-limiting to avoid overwhelming servers and high-fidelity data extraction from static and document-heavy websites characteristic of academic and archival research environments (see Figure \ref{fig:scraper}).
% The framework's architecture is built around an iterative crawling mechanism that starts from a single seed URL and uses the requests and BeautifulSoup \cite{BeautifulSoup} libraries to parse HTML pages and discover new hyperlinks. To ensure the crawler remains focused on the target domain, we implement a crucial "dynamic filter" that prevents it from following external links, thereby curating a highly relevant, domain-specific dataset. Further refinement is achieved through a configurable blacklist of URL patterns, allowing the exclusion of irrelevant content such as calendars, login pages, and version control repositories—essential for maintaining signal-to-noise ratio in institutional websites.
% Beyond standard HTML parsing, the system identifies linked documents by their file extensions and invokes specialized parsers, providing remarkable breadth valuable when mining institutional repositories where mixed formats are common. For PDF documents, it leverages the advanced marker library, which employs machine learning models to convert the visual structure of a PDF into clean, readable Markdown—a significant improvement over traditional PDF-to-text tools as it better preserves the document's intended layout and semantic meaning. The framework demonstrates exceptional resourcefulness in handling both modern and legacy Microsoft Office documents (.docx, .doc, .pptx, .ppt) and PostScript files (.ps, .eps), formats frequently encountered in decades of experimental data archives. It intelligently uses external command-line utilities like LibreOffice and Ghostscript to perform on-the-fly conversions into modern, parsable formats, ensuring that valuable information locked within these legacy files is not overlooked.
% Upon extraction, the text undergoes a rigorous cleaning process and is prepended with a comprehensive metadata header containing the source URL, page title, and timestamp, ensuring clear data provenance throughout the knowledge base. Finally, the framework reports detailed statistics on redirects, dynamically filtered links, and file extension frequencies, providing useful diagnostics that facilitate reproducibility studies and enable informed decision-making about further scraping optimization, creating a respectful, high-fidelity foundation for subsequent AI knowledge base construction.
% We develop a bespoke multi-format data acquisition framework for creating a comprehensive, analysis-ready textual corpus from a target web domain (see Figure \ref{fig:scraper}).
% Its primary function is to systematically crawl a website, extract content
% from not only HTML pages but also a variety of linked document formats
%and store the resulting metadata and sanitized text in a structured, clean YAML format.
%Its design choices reflect a deliberate focus on respectful rate-limiting to avoid overwhelming servers and high-fidelity data extraction from static and document-heavy websites, making it an exemplary tool for academic and archival research.
%The framework's architecture is built around an iterative crawling mechanism. Starting from a single seed URL, it uses the requests and BeautifulSoup libraries to parse HTML pages and discover new hyperlinks.
% It maintains a dictionary of all found links to track their visitation status, systematically exploring the website in a breadth-first-like manner until no new, unscraped links remain.
%To ensure the crawler remains focused on the target domain, we implement a crucial "dynamic filter" that prevents it from following external links, thereby curating a highly relevant, domain-specific dataset.
%Further refinement is achieved through a configurable blacklist of URL patterns, allowing the exclusion of irrelevant content such as calendars, login pages, and version control repositories.
%Beyond standard HTML parsing, the linked documents are identified by their file extensions and invoke specialized parsers. This breadth is valuable when mining institutional repositories where mixed formats are common. For PDF documents, it leverages the advanced marker library, which employs machine learning models to convert the visual structure of a PDF into clean, readable Markdown. This is a significant improvement over traditional PDF-to-text tools, as it better preserves the document's intended layout and semantic meaning. Furthermore, the script demonstrates remarkable resourcefulness in handling both modern and legacy Microsoft Office documents (.docx, .doc, .pptx, .ppt) and PostScript files (.ps, .eps). As legacy formats are often encountered in decades of experimental data, it intelligently uses external command-line utilities like LibreOffice and Ghostscript to perform on-the-fly conversions into modern, parsable formats. This robust, multi-pronged extraction strategy ensures that valuable information locked within these files is not overlooked. Upon extraction, the text undergoes a rigorous cleaning process, which is prepended with a metadata header containing the source URL, page title, and a timestamp, ensuring clear data provenance.
% Finally, the framework reports statistics on redirects, dynamically filtered links, and extension frequencies which provide useful diagnostics and facilitate reproducibility studies. This enables informed decision-making about further scraping.
% Extensible Multi-format scraping
% Parses HTML, PDF, Word (DOC/DOCX), PowerPoint (PPT/PPTX), and EPS/PS files in one pass with on-the-fly document conversion
% Cleanses formats and boilerplate yielding high-quality plain text
% Captures metadata (URL, page title, timestamp) and content into yaml
% Recursive crawl with smart filters
% Traverses a site depth-first while skipping unwanted paths and file types
% Dynamic filtering keeps it inside the seed domain.
% Respectful rate-limiting: configurable SLEEP delay prevents overwhelming servers.
% Extension-aware analytics and report:
% Tracks processed, skipped, redirected, and dynamically-filtered link
% Reports grouped stats for quick checks and decision-making about further scraping
\section{AI Assistant: Architecture and Deployment}
\subsection{Retrieval-Augmented Generation (RAG) and Model Context Protocol (MCP)}
The harvested content feeds into a Retrieval-Augmented Generation (RAG) architecture \cite{lewis2020retrieval} that embeds thousands of RHIC documents—technical notes, conference slides, code snippets, and software documentation—into a searchable ChromaDB vector database \cite{chromadb}. When users pose questions in natural language, the system retrieves semantically similar passages to inform responses with domain-specific context.
A critical architectural innovation is the Model Context Protocol (MCP) wrapper \cite{anthropic2024mcp,hou2025mcp}, which exposes each logical step of the assistant's reasoning chain—retrieval, summarization, inference, and evaluation—as composable "contexts" that can be independently configured and monitored. This protocol implementation via FastAPI-MCP decorates every `/ask` request with a lightweight JSON header specifying the ordered stack of models, maximum context budget per stage, and security tier inherited from the caller's session credentials.
At runtime, the MCP dispatcher constructs an orchestration graph capable of concurrent queries to multiple backends (inference engines or LLM APIs).
This is followed by attach relevant documents from ChromaDB and finally pipelining results to decoder LLMs for response generation.
% The serializable nature of these contexts enables identical graph replay for offline regression testing or HPC batch processing without application code modification.
This separation of execution logic ("what" to compute via the context graph) from deployment target ("where" to run on local GPU, cloud API, or containerized microservice) provides flexibility to incorporate emerging models while guaranteeing reproducibility, provenance tracking, and accountability across the entire workflow.
% ---> do we need this?? kinda already implied in previous text --> The assistant is deployed on STAR documentation and integrated with internal BNL infrastructure.
% At the core of the assistant is a RAG architecture \cite{lewis2020retrieval} that embeds thousands of RHIC documents -- technical notes, conference slides, code snippets, software documentation -- into a searchable ChromaDB vector database. When a user asks a question in natural language, the assistant retrieves semantically similar data to inform the responses.
% An important cross-cutting layer in the architecture is the MCP wrapper \cite{anthropic2024mcp,hou2025mcp}, which exposes each logical step of the assistant’s reasoning chain -- retrieval, summarization, inference, and evaluation -- as first-class, composable “contexts”.
% This protocol is implemented via FastAPI-MCP which decorates every `/ask` request with a lightweight JSON header that specifies the ordered stack of models, the maximum context budget per stage, and the security tier inherited from the caller’s session.
% At run-time the MCP dispatcher constructs an orchestration graph that can (i) fork a query to multiple back-ends (e.g., inference engines, or LLM APIs), (ii) splice in RAG nodes that attach ChromaDB passages, and (iii) cascade the result to a decoder LLM for response generation.
% Because contexts are serializable, the same graph can be replayed offline for regression tests or batched on an HPC queue without modifying application code.
% This separation of "what" to do (the context graph) from "where" to run it (local GPU, cloud API, or containerized micro-service) gives the assistant the flexibility to incorporate emerging models or domain-specific tools while guaranteeing reproducibility, provenance-tracking, and accountability across the entire workflow.
% The assistant is deployed on STAR documentation and integrated with internal BNL infrastructure.
% \subsection{Model Context Protocol (MCP)}
% MCP provides a standardized abstraction for switching between LLM backends, tracking provenance of responses, and ensuring experiment-specific context can be enforced.
% This makes the architecture modular and future-proof against LLM evolution.
% \section{Deployment}
% \subsection{Prototype}
% Users can now ask:
% “How do I apply track quality cuts in STAR 2014 data?”
% And receive structured responses including parameter values, code examples, and links to documentation.
\subsection{Performance Comparison of Inference Engines}
% \begin{figure*}
% \centering
% \includegraphics[trim={0.5cm 0.35cm 1.2cm 1cm},clip,width=0.295\linewidth]{figures/throughput_A6000.png}
% \includegraphics[trim={4cm 0.35cm 1.2cm 1cm},clip,width=0.23\linewidth]{figures/throughput_A100.png}
% \includegraphics[trim={4cm 0.35cm 1.8cm 1cm},clip,width=0.345\linewidth]{figures/throughput_H100.png}
% \caption{Throughput (tokens/sec) observed on various inference engines across one, two, and four GPUs.}
% \label{fig:throughput}
% \end{figure*}
% \begin{figure*}
% \centering
% \includegraphics[trim={0.5cm 0.35cm 1.2cm 1cm},clip,width=0.295\linewidth]{figures/utilization_A6000.png}
% \includegraphics[trim={4cm 0.35cm 1.2cm 1cm},clip,width=0.23\linewidth]{figures/utilization_A100.png}
% \includegraphics[trim={4cm 0.35cm 1.8cm 1cm},clip,width=0.345\linewidth]{figures/utilization_H100.png}
% \caption{Percentage utilization of GPUs on various inference engines across one, two, and four GPUs.}
% \label{fig:utilization}
% \end{figure*}
\begin{figure*}
\centering
\includegraphics[trim={0.5cm 2.35cm 1.2cm 1cm},clip,width=0.295\linewidth]{figures/throughput_A6000.png}
\includegraphics[trim={4cm 2.35cm 1.2cm 1cm},clip,width=0.23\linewidth]{figures/throughput_A100.png}
\includegraphics[trim={4cm 2.35cm 1.8cm 1cm},clip,width=0.345\linewidth]{figures/throughput_H100.png}
\includegraphics[trim={0.5cm 0.35cm 1.2cm 2cm},clip,width=0.295\linewidth]{figures/utilization_A6000.png}
\includegraphics[trim={4cm 0.35cm 1.2cm 2cm},clip,width=0.23\linewidth]{figures/utilization_A100.png}
\includegraphics[trim={4cm 0.35cm 1.8cm 2cm},clip,width=0.345\linewidth]{figures/utilization_H100.png}
\caption{Throughput (tokens/sec) (top row) and percentage utilization (bottom row) of GPUs on various inference engines across one, two, and four GPUs.}
\label{fig:thruput_util}
\end{figure*}
Recently, several specialized LLM inference engines have been developed with different design goals for service-oriented infrastructures. Ref. \cite{park2025survey} surveyed 25 open-source and commercial inference engines for their optimization and performance characteristics.
% They demonstrated that token-level throughput and time to respond hinge on an engine’s support for KV-cache reuse, continuous or hybrid batching, kernel fusion and low-bit quantization-optimizations.
The goal for a successful inference engine is to ameliorate GPU under-utilization while reducing per-request overheads.
Thus, a fair comparison must consider token throughput, GPU utilization, and the available hardware, because the very same model can exhibit markedly different behavior across engines and deployments.
The remainder of this section therefore compares three representative engines (vLLM, LlamaCpp, Ollama) to illuminate the practical trade-offs among speed, scalability and cost that shape modern chatbot services.
The analysis of inference engine performance across different GPU architectures in Figure \ref{fig:thruput_util} reveals distinct trends in throughput and resource utilization, heavily influenced by both the hardware generation and the underlying design philosophy of the software. As expected, throughput scales directly with the generational advancements of the GPU hardware; for any given engine and configuration, the newer H100 consistently outperforms the A100, which in turn surpasses the older A6000 architecture. However, the scaling observed on increasing the number of GPUs highlights fundamental differences in the engines' design philosophies. vLLM, for instance, demonstrates a clear design focus on maximizing throughput via parallelization, as its tokens-per-second rate consistently increases with the addition of GPUs while maintaining high average utilization, typically above 80\%. In stark contrast, LlamaCpp, while often exhibiting the highest single-GPU throughput, shows a dramatic decrease in resource utilization as more GPUs are added, dropping from ~90\% on a single GPU to as low as ~21\% on four. This suggests its multi-GPU strategy may prioritize memory distribution for larger models over computational parallelism, leading to under-utilization of compute resources on additional GPUs. Ollama's performance scaling is generally less efficient than the other two, with both throughput and utilization declining as the GPU count increases. In summary, while newer hardware like the H100 provides the highest raw performance ceiling, achieving optimal multi-GPU throughput is critically dependent on the inference engine's architecture, with vLLM's parallel-first approach showing superior scaling over LlamaCpp's apparent memory-centric distribution model in these tests.
\section{Benchmarking, Validation, and Metrics of Success}
% Validation Framework: Incorporates expert feedback and usage metrics to refine responses and filter hallucinations.
% Technology Watch: Annual reviews ensure compatibility with evolving formats, AI methods, and hardware platforms.
% FAIR Alignment: All outputs are designed to be Findable, Accessible, Interoperable, and Reusable.
It is important to define success metrics for a specialized scientific ChatBot as many generic LLMs (ChatGPT, Gemini, Claude, ...) exist and available to end-user, challenging the merits of an in-house tool. Our approach is tailored to the scientific community it is focused on our ability to deliver accurate, contextually deep answers from trusted scientific sources, surpassing general AI tools. Key criteria include precision, seamless integration with scientific knowledge databases, enhanced time efficiency, and comprehensive search capabilities across diverse scientific domains. In addition, we will provide secure access to proprietary data, while addressing the challenge of keeping it updated with real-time external information through strategies like automated data synchronization and hybrid search approaches (where both real-time and local database are consulted and the results merged).
% \sout{In our approach, model responses were systematically evaluated against expert-validated reference answers using a multi-faceted assessment approach. Domain specialists with many years of experience established ground truth answers for each benchmark question, providing the basis for substantial inter-rater reliability. Significance of performance differences between RAG-enhanced and baseline models was observed. Our initial (manual, expert vetted) evaluation revealed that models incorporating RAG knowledge consistently outperformed baseline variants, with embedding-space similarity analysis identified as a promising direction for future evaluation enhancement.}
In this work, we qualitatively evaluate LLM responses against expert-validated reference answers.
Domain specialists with several years of experience have established ground truth answers for each benchmark question like "How are space charge effects considered in the STAR experiment?" and "What is the TOF resolution in STAR experiment?"
We compare the answers to the above question generated by Llama3.3-70B, Mistral-Large-2411, both augmented with the
RAG framework accessing local documents, and a commercial version of ChatGPT o3.
The evaluation revealed distinct performance characteristics.
For an initial query all three models provided relevant and accurate information, with stylistic variations.
The Llama3.3 model was noted for its concise and detailed narrative, whereas, Mistral was similarly precise but slightly more verbose. In contrast, ChatGPT o3 adopted a more pedagogical, verbose style that, while comprehensive, included extraneous details.% that might be extraneous for an initial expert query.
A key aspect of the comparison was the ability to cite sources. While the RAG-based models inherently provide references from their local knowledge base, it was observed that ChatGPT o3 also cited its sources, which included public archives like arXiv, academic networks such as ResearchGate, and even publicly accessible files on institutional websites.
The primary differentiator and principal value of the local RAG-based approach emerged from its ability to incorporate private, unpublished information while providing excellent answers at minimal operational costs. Specifically, Llama3.3 and Mistral models successfully and quickly retrieved context from internal collaboration mailing lists, which are inaccessible to public-facing commercial LLMs. This access to informal scientific discourse—including technical discussions, troubleshooting exchanges, and expert clarifications from mailing lists and conversations—may explain why the RAG-based responses appeared more aligned with the practical, conversational tone that scientists expect when consulting colleagues. Therefore, we conclude a locally deployed RAG system holds a distinct and critical advantage for queries that depend on proprietary or non-public data. We further speculate that in cases where information is not yet published, a RAG-based approach leveraging internal documents is not just valuable but essential for comprehensive and accurate scientific inquiry. The development of a formal, multi-dimensional evaluation benchmark to systematically quantify these observations is currently in progress.
% On the more specific query related to Time-of-Flight (TOF), a significant performance divergence was observed. ChatGPT o3 markedly outperformed the RAG-based models, delivering a more thorough and historically nuanced response that included year-by-year variations. This result was notably different from previous informal tests, suggesting rapid evolution in the capabilities of commercial models.
\section{Conclusion and future work}
% --- Jerome's replacement, expanding with some of our discussed plans
We presented a domain-specific AI assistant that effectively preserves and serves nuclear physics knowledge through RAG architecture enhanced with MCP orchestration. The system's key advantage lies in accessing proprietary data sources—including internal mailing lists and unpublished materials—unavailable to commercial LLMs. Initial evaluation confirms RAG-enhanced models deliver contextually appropriate responses aligned with scientific expectations, demonstrating essential value for knowledge preservation in collaborative research environments.
Our evaluation methodology established expert-validated reference answers with substantial inter-rater reliability, revealing significant differences between RAG-enhanced and baseline models.
% This manual, expert-vetted approach demonstrated consistent RAG superiority, with embedding-space similarity analysis identified as a promising evaluation enhancement.
Future work will deploy independent LLM judges using structured prompts and explicit evaluation criteria to assess outputs across factual accuracy, completeness, relevance, and clarity dimensions.
Our immediate plans focus on extending the AI assistant to all RHIC experiments, implementing a comprehensive role-aware access control fabric that cleanly separates public, collaboration-restricted, and controlled artifacts. Security implementations will address the sensitive nature of proprietary experimental data through enterprise-grade encryption (AES-256 at rest, TLS 1.3 in transit), institutional SSO integration, and comprehensive audit logging. The role-aware access control system maps user authentication tokens to document classification hierarchies, enabling permissions that respect collaboration agreements.
% while maintaining data sovereignty requirements essential for international research partnerships.
Custom ingestion pipelines will incorporate community-adopted binary formats alongside heterogeneous media, capturing tacit expertise before it is lost to staff turnover.
The web scraping framework requires enhancement for scalability, fault tolerance, and authentication support for protected institutional resources, with asynchronous I/O improving throughput while maintaining respectful access patterns.
% Persistent frontier management through SQLite or Redis will enable distributed crawling and fault recovery, complemented by a plugin registry for extensible format support.
To maintain scientific currency, we plan an adaptive synchronization layer combining automated harvesting of open-access publications via APIs and RSS feeds, on-demand trusted web searches for queries beyond the local corpus, and human-in-the-loop dashboards enabling domain specialists to flag and annotate emerging results. The framework's domain-agnostic modularity positions it as a knowledge-stewardship engine for the forthcoming Electron-Ion Collider era and beyond,
% with iterative search logging and benchmark-guided optimization driving continuous improvement toward
thus leading to a sustainable, self-evolving assistant that integrates reproducibility, education, and discovery at scale.
% In the future we plan to extend the AI assistant across the PHENIX and sPHENIX experiments, employing a role-aware and access-control fabric that cleanly separates public, collaboration-restricted, and controlled artifacts.
% Custom ingestion pipelines—capable of parsing both structured formats such as ROOT tuples and heterogeneous media such as PDFs or archived web pages—will extend the knowledge graph, thereby capturing tacit expertise before it is lost to retirements and staff turnover.
% Further development of the web scraping framework requires addressing scalability, fault tolerance, and compliance.
% Asynchronous I/O will raise throughput while remaining polite.
% Persisting the frontier in SQLite or Redis will enable checkpointing, distributed crawling, and faithful recovery in case of faults.
% We also plan to develop plugin registry to simplify future format support and extension.
% For benchmarking, we plan to have an independent large language model served as an automated judge, employing structured prompts with explicit evaluation criteria to assess model outputs across multiple dimensions including factual accuracy, completeness, relevance, and clarity.
% To keep the system scientifically current, we plan to develop an adaptive synchronization layer that combines (i) automated harvesting of open-access publications and data via APIs and RSS feeds, (ii) on-demand “trusted web” searches when queries fall outside the local corpus, and (iii) a human-in-the-loop dashboard through which domain specialists can flag, rate, and annotate emerging results for permanent inclusion.
% Precision, retrieval latency, integration depth with experiment databases, and end-to-end time savings will serve as primary operational metrics.
% Conceptually, the framework is domain-agnostic; its modular retrieval and security primitives can be transplanted to other large-scale scientific collaborations. We envisage the platform as a ready-made knowledge-stewardship engine for the forthcoming Electron–Ion Collider era. Beyond high-energy physics, iterative search logging and benchmark-guided prompt tuning will drive continuous improvement, ultimately realizing a sustainable, self-evolving assistant that fuses reproducibility, education, and discovery at scale.
% \section*{Acknowledgments.}
% We would like to acknowledge the assistance of volunteers in putting
% together these updated SIAM Proceedings macros and example files.
% SIAM recommends using BibTeX
\bibliographystyle{siamplain}
\bibliography{references}
\end{document}
\begin{thebibliography}{10}
\bibitem{marker_pdf}
\url{https://github.com/datalab-to/marker}.
\bibitem{abdallah2022search}
{\sc M.~S. Abdallah, B.~Aboona, J.~Adam, L.~Adamczyk, J.~Adams, J.~Adkins,
G.~Agakishiev, I.~Aggarwal, M.~Aggarwal, Z.~Ahammed, et~al.}, {\em Search for
the chiral magnetic effect with isobar collisions at s {NN}= 200 {G}e{V} by
the {STAR} {C}ollaboration at the {BNL} {R}elativistic {H}eavy {I}on
{C}ollider}, Physical Review C, 105 (2022), p.~014901.
\bibitem{abdulhamid2024estimate}
{\sc M.~I. Abdulhamid, B.~Aboona, J.~Adam, J.~R. Adams, G.~Agakishiev,
I.~Aggarwal, M.~M. Aggarwal, Z.~Ahammed, A.~Aitbaev, I.~Alekseev, et~al.},
{\em Estimate of background baseline and upper limit on the chiral magnetic
effect in isobar collisions at $\sqrt {sNN} = 14.5$ {G}e{V} at the {BNL}
{R}elativistic {H}eavy {I}on {C}ollider}, Physical Review C, 110 (2024),
p.~014905.
\bibitem{adam2020beam}
{\sc J.~Adam, L.~Adamczyk, J.~Adams, J.~Adkins, G.~Agakishiev, M.~Aggarwal,
Z.~Ahammed, I.~Alekseev, D.~Anderson, R.~Aoyama, et~al.}, {\em Beam energy
dependence of net-$\lambda$ fluctuations measured by the {STAR} experiment at
the {BNL} {R}elativistic {H}eavy {I}on {C}ollider}, Physical Review C, 102
(2020), p.~024903.
\bibitem{adam2020bulk}
{\sc J.~Adam, L.~Adamczyk, J.~Adams, J.~Adkins, G.~Agakishiev, M.~Aggarwal,
Z.~Ahammed, I.~Alekseev, D.~Anderson, A.~Aparin, et~al.}, {\em Bulk
properties of the system formed in au+ au collisions at $\sqrt {sNN} = 14.5$
{G}e{V} at the bnl star detector}, Physical Review C, 101 (2020), p.~024905.
\bibitem{STAR_2005gfr}
{\sc J.~Adams et~al.}, {\em {Experimental and theoretical challenges in the
search for the quark gluon plasma: The {STAR} Collaboration's critical
assessment of the evidence from {RHIC} collisions}}, Nucl. Phys. A, 757
(2005), pp.~102--183, \url{https://doi.org/10.1016/j.nuclphysa.2005.03.085},
\url{https://arxiv.org/abs/nucl-ex/0501009}.
\bibitem{anthropic2024mcp}
{\sc {Anthropic}}, {\em Introducing the model context protocol}.
\newblock \url{https://www.anthropic.com/news/model-context-protocol}, Nov.
2024.
\newblock Accessed: 2025-07-31.
\bibitem{chromadb}
{\sc ChromaDB}, {\em Chroma{DB}}.
\newblock \url{https://www.trychroma.com/}, 2025.
\bibitem{hou2025mcp}
{\sc X.~Hou, Y.~Zhao, S.~Wang, and H.~Wang}, {\em Model context protocol
({MCP}): Landscape, security threats, and future research directions}, arXiv
preprint arXiv:2503.23278, (2025).
\bibitem{ichimiya2009status}
{\sc R.~Ichimiya, N.~Apadula, Y.~Akiba, E.~Atomssa, S.~Chollet, O.~Drapier,
H.~En'yo, K.~Fujiwara, F.~Gastaldi, R.~G. De~Cassagnac, et~al.}, {\em Status
and overview of development of the silicon pixel detector for the {PHENIX}
experiment at the {BNL} {RHIC}}, Journal of Instrumentation, 4 (2009),
p.~P05001.
\bibitem{lewis2020retrieval}
{\sc P.~Lewis, E.~Perez, A.~Piktus, F.~Petroni, V.~Karpukhin, N.~Goyal,
H.~K{\"u}ttler, M.~Lewis, W.-t. Yih, T.~Rockt{\"a}schel, et~al.}, {\em
Retrieval-augmented generation for knowledge-intensive {NLP} tasks}, Advances
in neural information processing systems, 33 (2020), pp.~9459--9474.
\bibitem{park2025survey}
{\sc S.~Park, S.~Jeon, C.~Lee, S.~Jeon, B.-S. Kim, and J.~Lee}, {\em A survey
on inference engines for large language models: Perspectives on optimization
and efficiency}, arXiv preprint arXiv:2505.01658, (2025).
\bibitem{BeautifulSoup}
{\sc {Richardson, L.}}, {\em Beautiful soup documentation}.
\newblock \url{https://www.crummy.com/software/BeautifulSoup/bs4/doc/},
2007-2023.
\bibitem{roland2021sphenix}
{\sc G.~Roland}, {\em The s{PHENIX} experiment at {RHIC}}, (2021).
\bibitem{tannenbaum2019highlights}
{\sc M.~Tannenbaum}, {\em Highlights from {BNL} and {RHIC} 2016}, New Physics
Frontiers In The Lhc-2 Era, The-Proceedings Of The 54th Course Of The
International School Of Subnuclear Physics, 54 (2019), p.~395.
\end{thebibliography}