SimplyRetrieve: A Private and Lightweight Retrieval-Centric Generative AI Tool
Extracted source text · arXiv 2308.03983
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% Remove the "review" option to generate the final version.
\usepackage{EMNLP2023}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out.
% However, it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
\usepackage{graphicx}
\usepackage{tabularx}
\usepackage{makecell}
\usepackage{xcolor}
\def\toolname{SimplyRetrieve}
\def\githubname{RCGAI}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\title{\toolname: A Private and Lightweight Retrieval-Centric Generative AI Tool}
% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
% Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
% Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \And ... \And
% Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \AND
% Author 2 \\ Address line \\ ... \\ Address line \And
% Author 3 \\ Address line \\ ... \\ Address line}
\author{Youyang Ng,
Daisuke Miyashita,
Yasuto Hoshi,
Yasuhiro Morioka, \\
{\bf Osamu Torii},
{\bf Tomoya Kodama},
{\bf Jun Deguchi} \\
Kioxia Corporation, Japan \\
\texttt{youyang.ng@kioxia.com} \\}
\begin{document}
\maketitle
\begin{abstract}
Large Language Model (LLM) based Generative AI systems have seen significant progress in recent years. Integrating a knowledge retrieval architecture allows for seamless integration of private data into publicly available Generative AI systems using pre-trained LLM without requiring additional model fine-tuning. Moreover, Retrieval-Centric Generation (RCG) approach, a promising future research direction that explicitly separates roles of LLMs and retrievers in context interpretation and knowledge memorization, potentially leads to more efficient implementation. \emph{\toolname{}} is an open-source tool with the goal of providing a localized, lightweight, and user-friendly interface to these sophisticated advancements to the machine learning community. \toolname{} features a GUI and API based RCG platform, assisted by a Private Knowledge Base Constructor and a Retrieval Tuning Module. By leveraging these capabilities, users can explore the potential of RCG for improving generative AI performance while maintaining privacy standards. The tool is available at \url{https://github.com/\githubname/\toolname} with an MIT license.
\end{abstract}
\section{Introduction}
\label{sec:introduction}
Generative-based Natural Language Processing (NLP) has witnessed significant progress \citep{GPT3} in recent years. With the introduction of Transformer \citep{Transformer} architecture, the possibility of developing high-accuracy language models that can perform tasks such as text generation, text summarization and language translation has become a reality. These models \citep{GPT3,PALM}, when scaled up to billions of parameters \citep{emergent}, have shown remarkable improvements in text generation tasks such as zero-shot inference, popularized the term \emph{Generative AI}. Instead of model fine-tuning, careful design of prompts has proven effective in adapting these models to specific domains for various tasks \citep{GPT3}. This has given rise to the field of prompt-engineering. Additionally, Chain-of-Thought \citep{CoT,kojima2022large} decomposes a complex task assigned into manageable steps, thereby expanding the capabilities of generative-based language models even further.
\begin{figure}[t]
\centering
\includegraphics[width=0.8\columnwidth]{fig/fig_retrievalcentric.png}
\caption{Retrieval-Centric Generation (RCG) approach presents an innovative concept that leverages the mutually beneficial interaction between LLMs and retrievers for more efficient context interpretation and knowledge memorization. Increased clarity in role-separation between context interpretation and knowledge memorization can potentially boost the performance of generative AI systems.}
\label{Fig.1:retrievalcentric}
\end{figure}
Training large language models (LLMs) requires immense computational resources, often involving thousands of high-end GPUs. Fine-tuning these models can also be challenging. Although prompt-engineering helped to reduce the need for fine-tuning, there was still noticeable instruction misalignment when interacting with a human user. To address this issue, techniques such as reinforcement learning from human feedback (RLHF) \citep{RLHF} have been explored to align the behavior of LLMs with human values \citep{InstructHF,ChatGPT}. Additionally, QLoRA \citep{qlora}, combining low-rank adaptation technique \citep{hu2022lora} and quantization technique, has made it possible to fine-tune these models on individual developer's hardware, making them more accessible to a wider range of users. Despite these advances, there are still limitations to the capacity of LLMs, and they do not inherently recognize information that was not present during training and fine-tuning. Memorization of factual knowledge in the long tail is also a challenge \citep{mallen-etal-2023-trust}.
Most recently, there has been growing interest in integrating external knowledge sources into LLMs for generating text \citep{RETRO, REALM, RAG}. Similar approaches have also been proposed in solving computer vision tasks \citep{knnclip,Iscen_2023_CVPR}. Retrieval-Augmented Generation (RAG) \citep{RAG} architecture is an approach that enhances the capabilities of LLMs by incorporating external data sources using a sparse or dense retriever \citep{karpukhin-etal-2020-dense}, enabling the use of privately owned data without requiring retraining or fine-tuning the LLM \citep{LangChain}. However, developing retrieval-augmented LLM-based generative models is still in its early stages. Our proposed tool can help facilitate these developments.
Additionally, we introduce a new architectural concept called \emph{Retrieval-Centric Generation} (RCG), which builds upon the Retrieval-Augmented Generation approach by emphasizing the crucial role of the LLM in interpreting context and entrusting knowledge memorization to the retriever component, putting greater importance on retriever, as depicted in Figure \ref{Fig.1:retrievalcentric}. By separating context interpretation from knowledge memorization, this approach has the potential to reduce the scale \citep{carlini2023quantifying} of the LLM required for generative tasks, leading to more efficient and interpretable results. Moreover, this approach may help mitigate hallucinations \citep{maynez-etal-2020-faithfulness} by limiting the scope of the LLM's generation. Once we define RCG as above, we can re-define RAG that enables more permissible usage of LLM's inherent knowledge, whereas RCG prioritizes clear demarcations between context interpretation and knowledge memorization.
\toolname{} is an open-source tool aimed at providing a localized, lightweight, and user-friendly interface to Retrieval-Centric Generation approach to the machine learning community. This tool encompasses a GUI and API based RCG platform, assisted by a Private Knowledge Base Constructors and a Retrieval Tuning Module. \toolname{} is designed to be simple and accessible to the community, as well as end-users. Our retrieval-centric platform incorporates multiple selectable knowledge bases featuring Mixtures-of-Knowledge-Bases (MoKB) mode and Explicit Prompt-Weighting (EPW) of retrieved knowledge base. By designing \toolname{} with these features, we enable the machine learning community to explore and develop with a lightweight, private data interface to LLM-based generative AI systems, with a focus on retrieval-centric generation. Potential developments that can be explored using this tool include: (1) examining the effectiveness of retrieval-centric generation in developing safer, more interpretable, and responsible AI systems; (2) optimizing the efficiency of separating context interpretation and knowledge memorization within retrieval-centric generation approach; and (3) improving prompt-engineering techniques for retrieval-centric generation. \toolname{} is available at \url{https://github.com/\githubname/\toolname}.
Our contributions can be summarized as follows:
\begin{itemize}
\item We propose \toolname{}, an innovative and user-friendly tool that leverages GUI and API platform to facilitate a Retrieval-Centric Generation approach. This platform is further strengthened by two key components: Private Knowledge Base Constructor and Retrieval Tuning Module.
\item We open sourced our tool to the machine learning community and identify potential development directions of Retrieval-Centric Generation.
\end{itemize}
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{fig/fig_tooloverview.png}
\caption{\toolname{} is an open-source tool that provides a localized, lightweight, and user-friendly interface to the Retrieval-Centric Generation approach for the machine learning community. This tool features a GUI and API based RCG platform, assisted by a Private Knowledge Base Constructor and a Retrieval Tuning Module.}
\label{Fig.2:tooloverview}
\end{figure*}
\section{Related Works}
\label{sec:relatedworks}
The emergence of Retrieval-Augmented Generation architecture has spurred the development of numerous open-source tools. The ChatGPT Retrieval Plugin\footnote{\url{https://github.com/openai/chatgpt-retrieval-plugin}}, for instance, integrates the ability to retrieve and enhance personal or organizational documents into the widely used ChatGPT model \citep{ChatGPT}. Similarly, fastRAG \citep{fastRAG} provides a streamlined platform for constructing efficient retrieval-augmented generation pipelines. Additionally, LangChain \citep{LangChain} offers a comprehensive generative chat AI library featuring agents, data augmentation, and memory capabilities. Finally, Haystack \citep{haystack} presents an all-encompassing NLP framework supporting question answering, answer generation, semantic document search, and retrieval-augmentation. Both LangChain and Haystack employ agent-based pipelining techniques and can process complex queries. However, this complexity may hinder the explainability of LLMs, making it challenging to interpret their performance in retrieval-augmented settings.
On the other hand, our work offers a lightweight and transparent approach to implementing sophisticated retrieval-centric, as well as retrieval-augmented architecture, while maintaining a strong emphasis on response interpretability and wider accessibility to the community. Unlike previous works such as PrivateGPT \citep{privategpt}, which provides a privacy-preserving chat AI tool but lacks customization options and analytical capabilities, our tool offers a comprehensive set of features for tailoring and analyzing retrieval-centric generation.
Furthermore, to the best of our knowledge, we are the first to introduce RCG concept and show initial experiments of it using our tool.
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{fig/fig_guioverall.png}
\caption{The GUI design of \toolname{} features four primary tabs. The Chat tab serves as the central query and response interface with retrieval-centric tuning panel. The Prompt tab provides an intuitive editor for modifying, updating, and saving prompts used by the AI. The Config tab enables users to customize various tool settings and save their preferences. Finally, the Analysis tab offers a comprehensive analytics platform for analyzing and logging data related to \toolname{}'s performance and usage.}
\label{Fig.3:guioverview}
\end{figure*}
\section{Tool Design}
\label{sec:tooldesign}
\toolname{} is designed to deploy RCG pipeline: construct knowledge base, tune architecture, make predictions. In this paper, we focus on describing the core specifications of the tool. For details about the setup procedures, refer to the repository of \url{https://github.com/\githubname/\toolname}.
\subsection{GUI and API based Retrieval-Centric Generation Platform}
As shown in Figure \ref{Fig.2:tooloverview}, there are two dense models in our tool: an LLM and an Approximate Nearest Neighbor Search (ANNS) based Knowledge Retriever. The LLM can be any one of the off-the-shelf open-source LLM models available in Hugging Face \citep{huggingface}, ranging from 1B to more than 100B-scale in parameters such as \citet{llama,touvron2023llama}. The Knowledge Retriever employs a dense retriever that is compatible with various embedding models available in Hugging Face. Additionally, our tool allows integration of multiple knowledge bases simultaneously, enabling user-selectable knowledge bases depending on the specific use case.
In terms of the GUI, we have designed a simple yet intuitive layout using Gradio \citep{abid2019gradio}, which provides a familiar streaming chatbot interface with user control for managing the running modes of the retriever, engineering prompts, and configuring the tool. As depicted in Figure \ref{Fig.3:guioverview}, our GUI features a comprehensive \emph{retrieval-centric tuning panel} for functions including \emph{manual knowledge base selection} from multiple sources and \emph{Mixture-of-Knowledge-Base} modes. Moreover, we employ \emph{Explicit Prompt-Weighting} of retrieval to adjust the level of influence exerted by the retriever. To ensure seamless integration, we also developed a comprehensive API access function using the Gradio Client Interface, and we allow multi-user concurrent access to both UIs, leveraging Gradio's queue functionality to manage requests efficiently.
The retrieval-centric tuning panel enables lightweight and simplistic access to RCG. By using the manual knowledge base selection mode, users can construct and import multiple private knowledge bases simultaneously into this tool. The ability to select the most relevant knowledge base for each task allows users to maintain control over the selection process while avoiding any unexpected outcomes. Our MoKB mode enables automatic selection of the most suitable knowledge base based on the similarity between the query and knowledge base functional descriptions. We use semantic cosine similarity of embedding space to calculate these scores, providing an efficient and lightweight approach to knowledge base auto-selection. By updating the functional descriptions in the configuration file, users can further enhance the accuracy of the selection algorithm.
Additionally, our Explicit Prompt-Weighting feature allows manual adjustment of the degree of influence of retrievers on the language model, enabling customized control over the balance between retriever and LLM. Through prompt-engineering or token weight adjustment, users can adapt the tool to their specific needs, ensuring optimal performance. \toolname{} has incorporated Explicit Prompt-Weighting through prompt-engineering, where the weightage can be adjusted to fine-tune the percentage of knowledge tokens to be used in the prompt out of retrieved tokens. However, we have not implemented token weight adjustment in this study and leave it for future work.
\subsection{Private Knowledge Base Constructor}
Our Retrieval-Centric Generation Platform is assisted by a Private Knowledge Base Constructor that creates a local and personalized knowledge base using the user's documents. This constructor employs a scalable documents loader that can handle large volumes of documents by chunking and streaming the loading, splitting and knowledge base creation processes, allowing for efficient document processing. The constructor supports various document formats such as PDF, TXT, DOC, DOCX, PPT, PPTX, HTML, MD, CSV, among others, and can be easily expanded by editing configuration file. Additionally, the length of passages in the documents splitting function is easily configurable to meet specific requirements.
After generating the sources for the knowledge base, we use a dense encoder to convert the text into numerical embeddings that can be used for semantic search and retrieve. To accommodate large-scale knowledge bases, we utilize ANNS for efficient semantic retrieval. By default, our tool employs the Hierarchical Navigable Small Worlds (HNSW) \citep{10.1109/TPAMI.2018.2889473} algorithm, but we also provide support for flat indexing and the IVFPQ-HNSW method, which combines inverted file indexing with product quantization and HNSW course quantizers. The Index Constructor function automatically creates the required index files for semantic searching. We implement our indexing function by using Faiss library \citep{johnson2019billion}.
\subsection{Retrieval Tuning Module}
The Retrieval Tuning Module of our tool includes three key functionalities: prompt-engineering, tool configuration, and analysis and data logging. The prompt-engineering functionality allows users to easily edit, update, and save retrieval-related prompts using a user-friendly Prompt Tab within our GUI. Available prompts are \emph{AI Prefix, Retriever Prefix, Retriever Suffix, Model Prefix and Model Suffix}. The configuration functionality enables users to modify and save all configurable settings via the Config Tab within our GUI. Finally, the analysis and data logging functionality collects and displays retrieval-related analysis data, including retrieved knowledge base, query, response, sentence-level and token-level similarity scores, in the Analysis Tab of our GUI. Similarity scores are calculated based on both semantic cosine similarity of sentence-to-sentence embeddings and all-token-to-token embeddings. This approach allows us to capture both local and global similarities between sentences, leading to more accurate assessments of their comparability. Additionally, users can save all logged data to a log file for further analysis. GUI designs are depicted in Figure \ref{Fig:guiprompt}, \ref{Fig:guiconfig} and \ref{Fig:guianalysis} of Appendix \ref{sec:applications}. To deploy an end-user mode, users can simply disable the update functions in the Retrieval Tuning Module through command-line options.
\section{Evaluations}
\label{sec:evaluations}
In this section, we perform several qualitative evaluations to demonstrate the usability and behavior of our tool. We construct our knowledge base using the most recent information available on the website of an organization\footnote{\url{https://www.kioxia.com/en-jp/top.html}}. We utilize the models publicly available on Hugging Face, \emph{Wizard-Vicuna-13B}\footnote{\url{https://huggingface.co/ehartford/Wizard-Vicuna-13B-Uncensored}} \citep{xu2023wizardlm,vicuna2023} as the LLM and \emph{Multilingual-E5-base}\footnote{\url{https://huggingface.co/intfloat/multilingual-e5-base}} \citep{wang2022text} as the encoder for our evaluations, unless specified otherwise. We load both models into a single Nvidia A100 GPU in 8-bit INT8 mode for lower memory usage and higher inference speed. We set temperature of LLM to 0. We utilize HNSW for indexing of knowledge bases and set the number of passages retrieved to 5.
\subsection{Qualitative Evaluation}
\label{sec:evaluations-qualitative}
We compare the results of three approaches: Retrieval-Centric Generation (RCG), Retrieval-Augmented Generation (RAG), and Retrieval-OFF Generation (ROG). Note that in this work, we define RAG as allowing more permissible integration of LLM's inherent and externally retrieved knowledge, whereas RCG prioritizes clear demarcations between context interpretation and knowledge memorization. Investigating advanced methods in extracting RCG behavior is a promising research topic. In this work, we conduct simple experiments using prompt-engineering technique to reveal the potential of RCG approach. Specifically, for RCG, we employ a retrieval suffix prompt that reads ``\emph{answer the following question with the provided knowledge.}'' For RAG, we use a less constraining prompt that reads ``\emph{answer the following question. You may use the provided knowledge.}'' Finally, for ROG, which is the baseline in our evaluation, we disable the retriever altogether. By using the prompt-engineering function in \toolname{}, we can easily perform these experiments. Queries are human annotated. In Table \ref{table:prompt-engineering} and \ref{table:retrieval-centric}, blue-colored texts indicates factually correct responses while red-colored texts indicates hallucinations.
\begin{table}[hbtp]
\small
\centering
Query: where is Kioxia's factory?
\begin{tabular}{p{0.1\columnwidth}p{0.8\columnwidth}}
\hline
Approach & Response \\
\hline \hline
RCG & \textcolor{blue}{Kioxia has factories located in both Yokkaichi city, Mie prefecture and Kitakami city, Iwate prefecture in Japan.} \\
\hline
RAG & \textcolor{red}{Kioxia has factories located in various regions across the globe including Japan, Thailand, Malaysia, China, and the United States. However,} \textcolor{blue}{its main production site is located in Yokkaichi City, Mie Prefecture, Japan.} \\
\hline
ROG & \textcolor{red}{Our main production site for HDD and SSD products is located in Yamagata, Japan. We also have manufacturing sites in Thailand and the Philippines that produce optical disc media such as Blu-ray Discs and DVDs.} \\
\hline
\end{tabular}
\caption{Qualitative comparison between Retrieval-Centric, Retrieval-Augmented and Retrieval-OFF approaches using prompt-engineering in \toolname{}.}
\label{table:prompt-engineering}
\end{table}
Based on the results in Table \ref{table:prompt-engineering}, we observed that RCG successfully answered our query accurately, whereas RAG attempted to integrate the retrieved knowledge base with LLM's own knowledge of the specific organization, resulting in partially erroneous information. Meanwhile, ROG responded to our query with LLM's preexisting mistaken knowledge, generating falsehoods. Our findings suggest that crafting clear and direct prompts, such as ``\emph{answer the given question using the provided knowledge}'', can encourage retrieval-centric behavior from the LLM, although further refined prompt-engineering may be necessary for optimal performance.
As shown in Table \ref{table:retrieval-centric}, RCG effectively addressed the query and produced accurate results. When combining RCG with a 50\% Explicit Prompt-Weighting (RCG-EPW), where only the first 50\% of retrieved knowledge base are injected into the prompt, the model generated partial responses but still maintained factual accuracy. In contrast, the pure LLM without RC enabled (ROG) provided hallucinatory responses, highlighting the importance of incorporating retrieval-centric techniques in language models for improved performance.
\begin{table}[hbtp]
\small
\centering
Query: what did Kioxia do in Image Classification
\begin{tabular}{p{0.2\columnwidth}p{0.7\columnwidth}}
\hline
Approach & Response \\
\hline \hline
RCG & \textcolor{blue}{Kioxia developed a high capacity storage device which could store large amounts of image data required by AI algorithms. This allowed them to deploy memory centric AI systems capable of handling multiple classification tasks simultaneously without compromising performance or accuracy.} \\
\hline
RCG-EPW & \textcolor{blue}{Deployment of Memory-centric AI with high capacity storage} \\
\hline
ROG & \textcolor{blue}{The company developed a new image classification algorithm} \textcolor{red}{that achieved top-1 accuracy of 82.7\% on the ImageNet validation set, which is an improvement over previous state-of-the-art results by more than one percentage point. This was made possible through advancements in hardware and software technologies such as Tensor Core GPUs and our own deep learning framework, ONNX-MXNET.} \\
\hline
\end{tabular}
\caption{Effects of Retrieval-Centric Generation in \toolname{}, based on the knowledge base about an organization.}
\label{table:retrieval-centric}
\end{table}
\subsection{Accuracy \& Speed Evaluations}
\label{sec:evaluations-accuracy-speed}
In addition to evaluating the effectiveness of RCG using human annotations, we also conduct an internal evaluation of our tool's performance using a self-generated dataset. To create this dataset, we pass relevant passages through the language model Llama-2-13B-chat \citep{touvron2023llama} to generate 10 query and label pairs. For details on how we generated this dataset, refer to Appendix \ref{sec:evaluation-data}. We employ Rouge-L score \citep{lin-2004-rouge} as our performance metric. We perform this evaluation by using the API function of \toolname{}. Our results in Table \ref{table:response-accuracy-speed} show that RCG significantly improves the Rouge-L score compared to the baseline approach of ROG, while also slightly more competitive than RAG. Moreover, despite the fact that RCG processes longer prompts than ROG due to the addition of knowledge tokens, we observe a decrease in processing time owing to the increased precision and brevity of the generated responses. Specifically, number of response tokens generated in RCG are in average 36\% less than those generated in ROG. This efficient performance may facilitate broader adoption within the community, as users can expect quicker response generation without sacrificing accuracy.
\begin{table}[h]
\small
\centering
\begin{tabular}{|c|c|c|}
\hline
Approach & Rouge-L Score & time/query(s) \\
\hline \hline
ROG & 0.186 & 17.22 \\
\hline
RAG & 0.359 & 18.41 \\
\hline
RCG & 0.413 & 11.67 \\
\hline
\end{tabular}
\caption{Response accuracy \& speed evaluation of \toolname{}.}
\label{table:response-accuracy-speed}
\end{table}
Finally, our findings suggest that even a modestly sized LLM of 13B parameters can demonstrate satisfactory performance in RCG approach towards never-seen-before factual knowledge without any model fine-tuning, potentially facilitates the deployment of Generative AI systems in real-world scenarios. See Appendix \ref{sec:applications} for further discussions and \ref{sec:ablation-study} for ablation studies.
\section{Conclusion}
\label{sec:conclusion}
We introduced \toolname{}, an open-source tool that aims to provide a localizable, lightweight, and user-friendly GUI and API platform for a Retrieval-Centric Generation approach based on LLMs. Our tool enables developers and end-users to easily interact and develop with a privacy-preserving and locally implemented LLM-based RCG system, which we believe will contribute to the democratization of these technologies within the machine learning community. Increased clarity in role-separation between context interpretation and knowledge memorization can potentially boost the performance and interpretability of generative AI systems, facilitating deployments.
\section*{Limitations}
\label{sec:limitations}
It is important to note that this tool does not provide a foolproof solution for ensuring a completely safe and responsible response from generative AI models, even within a retrieval-centric approach. The development of safer, interpretable, and responsible AI systems remains an active area of research and ongoing effort.
Generated texts from this tool may exhibit variations, even when only slightly modifying prompts or queries, due to the next token prediction behavior of current-generation LLMs. This means users may need to carefully fine-tune both the prompts and queries to obtain optimal responses.
% \section*{Ethics Statement}
% Discussion about Ethics Statement
\bibliography{custom}
\bibliographystyle{acl_natbib}
\appendix
\section{Appendix}
\label{sec:appendix}
\subsection{GUI Design of Retrieval Tuning Module}
\label{sec:gui-retrieval}
Figure \ref{Fig:guiprompt} shows the GUI design of prompt-engineering interface.
Figure \ref{Fig:guiconfig} shows the GUI design of tool configuration interface.
Figure \ref{Fig:guianalysis} shows the GUI design of analysis and data logging interface.
\subsection{Applications}
\label{sec:applications}
\toolname{} has vast potential for various practical applications. For instance, it can serve as the foundation for building private, personalized, and lightweight generative AI systems. Sensitive and personal information can be securely stored and processed within the retrieval-centric platform. This approach enables organizations to develop interpretable and locally tailored generative AI systems for critical infrastructure. Additionally, the use of a relatively smaller language model as a contextual interpreter in this approach facilitates seamless integration into edge computing environments. The decreasing costs of data storage devices also make it feasible to establish large-scale knowledge bases. Furthermore, \toolname{} paves the way for the development of LLM-based personalized AI assistants. Lastly, an in-depth exploration of LLM-based retrieval-centric generation using \toolname{} may offer valuable insights and opportunities for future research.
\subsection{Prompt Catalogs}
\label{sec:prompt-catalogs}
Table \ref{table:prompt-evaluation} shows the prompts used in the evaluation results of Section \ref{sec:evaluations} while Table \ref{table:prompt-catalogs} shows sample prompts that may exhibit retrieval-centric behaviors. Prompts are passed to LLM in the following format: \emph{AI Prefix + Retriever Prefix + Retrieved Knowledge Base + Retriever Suffix + Model Prefix + Query + Model Suffix}.
\subsection{Evaluation Data}
\label{sec:evaluation-data}
Table \ref{table:evaluation-data} presents the data used for evaluating the performance of our proposed tool in Section \ref{sec:evaluations-accuracy-speed}. We employed the Llama-2-13B-chat model \citep{touvron2023llama} with a customized prompt (\emph{"relevant information." Please create a query and answer from the paragraph above}) to generate query and label pairs automatically from relevant information on the website of an organization.
\subsection{Ablation Study}
\label{sec:ablation-study}
As shown in Table \ref{table:ablation-study}, our ablation study reveals that adjusting Explicit Prompt-Weighting in \toolname{} leads to significant improvements in Rouge-L scores. Interestingly, increasing the weightage to 50\% yields the highest improvement, beyond which the performance remains relatively stable. This suggests that the top 50\% of retrieved knowledge bases are crucial for achieving high accuracy. However, it is important to note that these findings may not generalize to all datasets or knowledge bases, and further investigation may be necessary to determine optimal weightages for specific use cases. In comparing the response times for each query across different settings, we observe that the response times remain relatively consistent for all cases of RCG, while they increase significantly in the baseline (ROG) setting. Despite the fact that RCG processes longer prompts than the baseline, we observe a decrease in processing time owing to the increased precision and brevity of the generated responses.
\onecolumn
\begin{table*}[hbtp]
\small
\centering
\begin{tabular}{|c|c|c|}
\hline
Approach & Rouge-L & time/query(s) \\
\hline \hline
ROG & 0.186 & 17.22 \\
\hline
RCG-EPW-10 & 0.275 & 12.72 \\
\hline
RCG-EPW-20 & 0.313 & 13.00 \\
\hline
RCG-EPW-30 & 0.403 & 13.06 \\
\hline
RCG-EPW-40 & 0.354 & 11.98 \\
\hline
RCG-EPW-50 & 0.414 & 12.46 \\
\hline
RCG-EPW-60 & 0.331 & 11.36 \\
\hline
RCG-EPW-70 & 0.392 & 13.56 \\
\hline
RCG-EPW-80 & 0.306 & 16.32 \\
\hline
RCG-EPW-90 & 0.378 & 13.13 \\
\hline
RCG & 0.413 & 11.67 \\
\hline
\end{tabular}
\caption{Ablation study of Explicit Prompt-Weighting in \toolname{}.}
\label{table:ablation-study}
\end{table*}
\begin{table*}[hbtp]
\small
\centering
\begin{tabularx}{\textwidth}{|l|l|l|l|l|}
\hline
AI Prefix & Retriever Prefix & Retriever Suffix & Model Prefix & Model Suffix \\
\hline \hline
& " & \makecell[Xt]{" \\ answer the following question with the provided knowledge. \\ \vphantom{ }} & & \makecell[Xt]{ \\ AI:} \\
\hline
\end{tabularx}
\caption{Prompts used in the evaluation results of Section \ref{sec:evaluations}.}
\label{table:prompt-evaluation}
\end{table*}
\begin{table*}[hbtp]
\small
\centering
\begin{tabularx}{\textwidth}{|l|l|l|l|l|}
\hline
AI Prefix & Retriever Prefix & Retriever Suffix & Model Prefix & Model Suffix \\
\hline \hline
\makecell[Xt]{you are a Retrieval-Centric AI. Knowledge below are provided. \\ \vphantom{ }} & " & \makecell[Xt]{" \\ only use the provided knowledge to answer the following question. \\ \vphantom{ }} & & \makecell[Xt]{ \\ Response:} \\
\hline
& " & \makecell[Xt]{" \\ answer the following question with the provided knowledge. \\ \vphantom{ }} & & \makecell[Xt]{ \\ AI:} \\
\hline
& " & \makecell[Xt]{" \\ only use the provided knowledge to answer the following question. \\ \vphantom{ }} & & \makecell[Xt]{ \\ AI:} \\
\hline
\makecell[Xt]{you are a Retrieval-Centric AI. Knowledge below are provided. \\ \vphantom{ }} & " & \makecell[Xt]{" \\ only use the provided knowledge to answer the following question. \\ \vphantom{ }} & & \makecell[Xt]{ \\ AI:} \\
\hline
\end{tabularx}
\caption{Sample Prompts Catalog of Retrieval-Centric Generation in \toolname{}.}
\label{table:prompt-catalogs}
\end{table*}
\begin{figure*}[hbtp]
\centering
\includegraphics[width=\textwidth]{fig/fig_guiprompt.png}
\caption{The Prompt-Engineering interface of \toolname{}. The Tab is for editing, updating and saving of model-related and retrieval-related prompts. Available prompts are \emph{AI Prefix, Retriever Prefix, Retriever Suffix, Model Prefix and Model Suffix}.}
\label{Fig:guiprompt}
\end{figure*}
\begin{figure*}[hbtp]
\centering
\includegraphics[width=\textwidth]{fig/fig_guiconfig.png}
\caption{The Tool Configuration interface of \toolname{}. The Tab is for modifying, updating and saving all configurable settings.}
\label{Fig:guiconfig}
\end{figure*}
\begin{figure*}[hbtp]
\centering
\includegraphics[width=\textwidth]{fig/fig_guianalysis.png}
\caption{The Analysis and Data Logging interface of \toolname{}. The Tab is for analysis and logging of retrieved knowledge base, query, response, retrieval sentence level and tokens level similarity scores. Users can save all the logged data for further analysis.}
\label{Fig:guianalysis}
\end{figure*}
\begin{table*}[hbtp]
\small
\centering
\begin{tabularx}{\textwidth}{p{0.2\textwidth}p{0.7\textwidth}}
\hline
Query & Label \\
\hline \hline
What is KIOXIA known for in terms of innovation? & Based on the text, KIOXIA is known for inventing NAND flash memory in 1987 and advancing the information society on a global scale through its R\&D efforts. The company has also been recognized for presenting its research and development achievements through various channels, including major academic societies and research conferences. \\
\hline
What is the purpose of establishing KIOXIA Iwate Corporation? & The purpose of establishing KIOXIA Iwate Corporation is to meet the growing demand for flash memory by leveraging advanced manufacturing processes that utilize AI. \\
\hline
When did Kioxia start operating its new fabrication facility (Fab7)? & According to the text, Kioxia started operating its new fabrication facility (Fab7) in the fall of 2022. \\
\hline
What are some ways in which the Yokkaichi Plant contributes to society? & The Yokkaichi Plant contributes to society by manufacturing high-quality flash memory and SSDs that support the digitalization of various industries, as well as through its research and development (R\&D) activities. Additionally, the plant prioritizes legal compliance, environmental protection, and social initiatives, making it a responsible corporate citizen. \\
\hline
What is KIOXIA's approach to corporate social responsibility? & According to the passage, KIOXIA's approach to corporate social responsibility involves making their Yokkaichi Plant a factory that is admired and trusted by everyone in the community. They aim to contribute to local development and meet the expectations of local communities by being a responsible member of society. \\
\hline
What kind of environmental initiatives has KIOXIA taken at its Yokkaichi Plant? & At KIOXIA's Yokkaichi Plant, all employees, including resident suppliers, actively participate in various initiatives to contribute to society by improving the environment. These include measures to mitigate the environmental burden of the company's semiconductor manufacturing facilities. \\
\hline
What was the total revenue for the Kioxia group in fiscal year 21? & According to the passage, the total revenue for the Kioxia group in fiscal year 21 was 1,526.5 billion yen. \\
\hline
What initiatives has the Yokkaichi Plant taken to protect the environment? & The Yokkaichi Plant has taken various initiatives such as reducing greenhouse gas emissions, conserving water resources, and implementing recycling programs to protect the environment. These efforts aim to not only reduce the plant's ecological footprint but also raise awareness among local communities about the importance of environmental conservation. \\
\hline
What is KIOXIA's mission statement? & According to the passage, KIOXIA's mission statement is to uplift the world with 'memory', which means they aim to provide high-quality memory and SSD products that offer fast speeds, large capacities, and reliable performance. Additionally, the company seeks to achieve this goal through collaboration and knowledge sharing across different departments and organizations. \\
\hline
What is the purpose of collecting over two billion data items daily? & To maintain high-quality products by analyzing the vast amounts of data generated during the manufacturing process using advanced technologies like deep learning and AI. \\
\hline
\end{tabularx}
\caption{Dataset used in the evaluation results of Section \ref{sec:evaluations-accuracy-speed}.}
\label{table:evaluation-data}
\end{table*}
\twocolumn
\end{document}
\begin{thebibliography}{34}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\bibitem[{Abid et~al.(2019)Abid, Abdalla, Abid, Khan, Alfozan, and
Zou}]{abid2019gradio}
Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and
James Zou. 2019.
\newblock \href {http://arxiv.org/abs/1906.02569} {Gradio: Hassle-free sharing
and testing of ml models in the wild}.
\newblock \emph{arXiv preprint arXiv:1906.02569}.
\bibitem[{Borgeaud et~al.(2022)Borgeaud, Mensch, Hoffmann, Cai, Rutherford,
Millican, Van Den~Driessche, Lespiau, Damoc, Clark, De~Las~Casas, Guy,
Menick, Ring, Hennigan, Huang, Maggiore, Jones, Cassirer, Brock, Paganini,
Irving, Vinyals, Osindero, Simonyan, Rae, Elsen, and Sifre}]{RETRO}
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza
Rutherford, Katie Millican, George~Bm Van Den~Driessche, Jean-Baptiste
Lespiau, Bogdan Damoc, Aidan Clark, Diego De~Las~Casas, Aurelia Guy, Jacob
Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones,
Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals,
Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre.
2022.
\newblock \href {https://proceedings.mlr.press/v162/borgeaud22a.html}
{Improving language models by retrieving from trillions of tokens}.
\newblock In \emph{Proceedings of the 39th International Conference on Machine
Learning}, volume 162 of \emph{Proceedings of Machine Learning Research},
pages 2206--2240. PMLR.
\bibitem[{Brown et~al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal,
Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan,
Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess,
Clark, Berner, McCandlish, Radford, Sutskever, and Amodei}]{GPT3}
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared~D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon
Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris
Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever,
and Dario Amodei. 2020.
\newblock \href
{https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf}
{Language models are few-shot learners}.
\newblock In \emph{Advances in Neural Information Processing Systems},
volume~33, pages 1877--1901. Curran Associates, Inc.
\bibitem[{Carlini et~al.(2023)Carlini, Ippolito, Jagielski, Lee, Tramer, and
Zhang}]{carlini2023quantifying}
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian
Tramer, and Chiyuan Zhang. 2023.
\newblock \href {https://openreview.net/forum?id=TatRHT_1cK} {Quantifying
memorization across neural language models}.
\newblock In \emph{The Eleventh International Conference on Learning
Representations}.
\bibitem[{Chase(2022)}]{LangChain}
Harrison Chase. 2022.
\newblock \href {https://github.com/hwchase17/langchain} {{LangChain}}.
\bibitem[{Chiang et~al.(2023)Chiang, Li, Lin, Sheng, Wu, Zhang, Zheng, Zhuang,
Zhuang, Gonzalez, Stoica, and Xing}]{vicuna2023}
Wei-Lin Chiang, Zhuohan Li, Zi~Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin
Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph~E. Gonzalez, Ion Stoica, and
Eric~P. Xing. 2023.
\newblock \href {https://lmsys.org/blog/2023-03-30-vicuna/} {Vicuna: An
open-source chatbot impressing gpt-4 with 90\%* chatgpt quality}.
\bibitem[{Chowdhery et~al.(2022)Chowdhery, Narang, Devlin, Bosma, Mishra,
Roberts, Barham, Chung, Sutton, Gehrmann, Schuh, Shi, Tsvyashchenko, Maynez,
Rao, Barnes, Tay, Shazeer, Prabhakaran, Reif, Du, Hutchinson, Pope, Bradbury,
Austin, Isard, Gur-Ari, Yin, Duke, Levskaya, Ghemawat, Dev, Michalewski,
Garcia, Misra, Robinson, Fedus, Zhou, Ippolito, Luan, Lim, Zoph, Spiridonov,
Sepassi, Dohan, Agrawal, Omernick, Dai, Pillai, Pellat, Lewkowycz, Moreira,
Child, Polozov, Lee, Zhou, Wang, Saeta, Diaz, Firat, Catasta, Wei,
Meier-Hellstern, Eck, Dean, Petrov, and Fiedel}]{PALM}
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung~Won Chung, Charles Sutton, Sebastian
Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez,
Abhishek Rao, Parker Barnes, Yi~Tay, Noam Shazeer, Vinodkumar Prabhakaran,
Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob
Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm
Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia,
Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David
Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David
Dohan, Shivani Agrawal, Mark Omernick, Andrew~M. Dai,
Thanumalayan~Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica
Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi
Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei,
Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.
2022.
\newblock \href {http://arxiv.org/abs/2204.02311} {Palm: Scaling language
modeling with pathways}.
\newblock \emph{arXiv preprint arXiv:2204.02311}.
\bibitem[{Christiano et~al.(2017)Christiano, Leike, Brown, Martic, Legg, and
Amodei}]{RLHF}
Paul~F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario
Amodei. 2017.
\newblock \href
{https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf}
{Deep reinforcement learning from human preferences}.
\newblock In \emph{Advances in Neural Information Processing Systems},
volume~30. Curran Associates, Inc.
\bibitem[{Dettmers et~al.(2023)Dettmers, Pagnoni, Holtzman, and
Zettlemoyer}]{qlora}
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.
\newblock \href {http://arxiv.org/abs/2305.14314} {Qlora: Efficient finetuning
of quantized llms}.
\newblock \emph{arXiv preprint arXiv:2305.14314}.
\bibitem[{Guu et~al.(2020)Guu, Lee, Tung, Pasupat, and Chang}]{REALM}
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.
\newblock \href {https://proceedings.mlr.press/v119/guu20a.html} {Realm:
Retrieval-augmented language model pre-training}.
\newblock In \emph{Proceedings of the 37th International Conference on Machine
Learning}, ICML'20. JMLR.org.
\bibitem[{Hu et~al.(2022)Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and
Chen}]{hu2022lora}
Edward~J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu~Wang, and Weizhu Chen. 2022.
\newblock \href {https://openreview.net/forum?id=nZeVKeeFYf9} {Lo{RA}: Low-rank
adaptation of large language models}.
\newblock In \emph{International Conference on Learning Representations}.
\bibitem[{Iscen et~al.(2023)Iscen, Fathi, and Schmid}]{Iscen_2023_CVPR}
Ahmet Iscen, Alireza Fathi, and Cordelia Schmid. 2023.
\newblock \href
{https://openaccess.thecvf.com/content/CVPR2023/html/Iscen_Improving_Image_Recognition_by_Retrieving_From_Web-Scale_Image-Text_Data_CVPR_2023_paper.html}
{Improving image recognition by retrieving from web-scale image-text data}.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)}, pages 19295--19304.
\bibitem[{Izsak et~al.(2023)Izsak, Berchansky, Fleischer, and
Laperdon}]{fastRAG}
Peter Izsak, Moshe Berchansky, Daniel Fleischer, and Ronen Laperdon. 2023.
\newblock \href {https://github.com/IntelLabs/fastrag} {{fastRAG: Efficient
Retrieval Augmentation and Generation Framework}}.
\bibitem[{Johnson et~al.(2019)Johnson, Douze, and
J{\'e}gou}]{johnson2019billion}
Jeff Johnson, Matthijs Douze, and Herv{\'e} J{\'e}gou. 2019.
\newblock \href {https://doi.org/10.1109/TBDATA.2019.2921572} {Billion-scale
similarity search with {GPUs}}.
\newblock \emph{IEEE Transactions on Big Data}, 7(3):535--547.
\bibitem[{Karpukhin et~al.(2020)Karpukhin, Oguz, Min, Lewis, Wu, Edunov, Chen,
and Yih}]{karpukhin-etal-2020-dense}
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey
Edunov, Danqi Chen, and Wen-tau Yih. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.emnlp-main.550} {Dense
passage retrieval for open-domain question answering}.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP)}, pages 6769--6781, Online. Association
for Computational Linguistics.
\bibitem[{Kojima et~al.(2022)Kojima, Gu, Reid, Matsuo, and
Iwasawa}]{kojima2022large}
Takeshi Kojima, Shixiang~Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke
Iwasawa. 2022.
\newblock \href {https://openreview.net/forum?id=e2TBb5y0yFf} {Large language
models are zero-shot reasoners}.
\newblock In \emph{Advances in Neural Information Processing Systems}.
\bibitem[{Lewis et~al.(2020)Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal,
K\"{u}ttler, Lewis, Yih, Rockt\"{a}schel, Riedel, and Kiela}]{RAG}
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir
Karpukhin, Naman Goyal, Heinrich K\"{u}ttler, Mike Lewis, Wen-tau Yih, Tim
Rockt\"{a}schel, Sebastian Riedel, and Douwe Kiela. 2020.
\newblock \href
{https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf}
{Retrieval-augmented generation for knowledge-intensive nlp tasks}.
\newblock In \emph{Advances in Neural Information Processing Systems},
volume~33, pages 9459--9474. Curran Associates, Inc.
\bibitem[{Lin(2004)}]{lin-2004-rouge}
Chin-Yew Lin. 2004.
\newblock \href {https://aclanthology.org/W04-1013} {{ROUGE}: A package for
automatic evaluation of summaries}.
\newblock In \emph{Text Summarization Branches Out}, pages 74--81, Barcelona,
Spain. Association for Computational Linguistics.
\bibitem[{Malkov and Yashunin(2020)}]{10.1109/TPAMI.2018.2889473}
Yu~A. Malkov and D.~A. Yashunin. 2020.
\newblock \href {https://doi.org/10.1109/TPAMI.2018.2889473} {Efficient and
robust approximate nearest neighbor search using hierarchical navigable small
world graphs}.
\newblock \emph{IEEE Trans. Pattern Anal. Mach. Intell.}, 42(4):824–836.
\bibitem[{Mallen et~al.(2023)Mallen, Asai, Zhong, Das, Khashabi, and
Hajishirzi}]{mallen-etal-2023-trust}
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and
Hannaneh Hajishirzi. 2023.
\newblock \href {https://aclanthology.org/2023.acl-long.546} {When not to trust
language models: Investigating effectiveness of parametric and non-parametric
memories}.
\newblock In \emph{Proceedings of the 61st Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)}, pages 9802--9822,
Toronto, Canada. Association for Computational Linguistics.
\bibitem[{Maynez et~al.(2020)Maynez, Narayan, Bohnet, and
McDonald}]{maynez-etal-2020-faithfulness}
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.acl-main.173} {On
faithfulness and factuality in abstractive summarization}.
\newblock In \emph{Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics}, pages 1906--1919, Online. Association for
Computational Linguistics.
\bibitem[{Nakata et~al.(2022)Nakata, Ng, Miyashita, Maki, Lin, and
Deguchi}]{knnclip}
Kengo Nakata, Youyang Ng, Daisuke Miyashita, Asuka Maki, Yu-Chieh Lin, and Jun
Deguchi. 2022.
\newblock \href {https://doi.org/10.1007/978-3-031-19836-6_26} {Revisiting a
knn-based image classification system with high-capacity storage}.
\newblock In \emph{Computer Vision -- ECCV 2022}, pages 457--474, Cham.
Springer Nature Switzerland.
\bibitem[{OpenAI(2023)}]{ChatGPT}
OpenAI. 2023.
\newblock Chatgpt.
\newblock \url{https://openai.com/blog/chatgpt}.
\bibitem[{Ouyang et~al.(2022)Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin,
Zhang, Agarwal, Slama, Ray, Schulman, Hilton, Kelton, Miller, Simens, Askell,
Welinder, Christiano, Leike, and Lowe}]{InstructHF}
Long Ouyang, Jeffrey Wu, Xu~Jiang, Diogo Almeida, Carroll Wainwright, Pamela
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John
Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda
Askell, Peter Welinder, Paul~F Christiano, Jan Leike, and Ryan Lowe. 2022.
\newblock \href
{https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf}
{Training language models to follow instructions with human feedback}.
\newblock In \emph{Advances in Neural Information Processing Systems},
volume~35, pages 27730--27744. Curran Associates, Inc.
\bibitem[{Pietsch et~al.(2019)Pietsch, Möller, Kostic, Risch, Pippi,
Jobanputra, Zanzottera, Cerza, Blagojevic, Stadelmann, Soni, and
Lee}]{haystack}
Malte Pietsch, Timo Möller, Bogdan Kostic, Julian Risch, Massimiliano Pippi,
Mayank Jobanputra, Sara Zanzottera, Silvano Cerza, Vladimir Blagojevic,
Thomas Stadelmann, Tanay Soni, and Sebastian Lee. 2019.
\newblock \href {https://github.com/deepset-ai/haystack} {{Haystack: the
end-to-end NLP framework for pragmatic builders}}.
\bibitem[{PrivateGPT()}]{privategpt}
PrivateGPT.
\newblock \href {https://github.com/imartinez/privateGPT} {{PrivateGPT}}.
\newblock Accessed: 2023-07-04.
\bibitem[{Touvron et~al.(2023{\natexlab{a}})Touvron, Lavril, Izacard, Martinet,
Lachaux, Lacroix, Rozière, Goyal, Hambro, Azhar, Rodriguez, Joulin, Grave,
and Lample}]{llama}
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro,
Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume
Lample. 2023{\natexlab{a}}.
\newblock \href {http://arxiv.org/abs/2302.13971} {Llama: Open and efficient
foundation language models}.
\newblock \emph{arXiv preprint arXiv:2302.13971}.
\bibitem[{Touvron et~al.(2023{\natexlab{b}})Touvron, Martin, Stone, Albert,
Almahairi, Babaei, Bashlykov, Batra, Bhargava, Bhosale, Bikel, Blecher,
Ferrer, Chen, Cucurull, Esiobu, Fernandes, Fu, Fu, Fuller, Gao, Goswami,
Goyal, Hartshorn, Hosseini, Hou, Inan, Kardas, Kerkez, Khabsa, Kloumann,
Korenev, Koura, Lachaux, Lavril, Lee, Liskovich, Lu, Mao, Martinet, Mihaylov,
Mishra, Molybog, Nie, Poulton, Reizenstein, Rungta, Saladi, Schelten, Silva,
Smith, Subramanian, Tan, Tang, Taylor, Williams, Kuan, Xu, Yan, Zarov, Zhang,
Fan, Kambadur, Narang, Rodriguez, Stojnic, Edunov, and
Scialom}]{touvron2023llama}
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
Dan Bikel, Lukas Blecher, Cristian~Canton Ferrer, Moya Chen, Guillem
Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,
Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar
Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa,
Isabel Kloumann, Artem Korenev, Punit~Singh Koura, Marie-Anne Lachaux,
Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier
Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew
Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan
Silva, Eric~Michael Smith, Ranjan Subramanian, Xiaoqing~Ellen Tan, Binh Tang,
Ross Taylor, Adina Williams, Jian~Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan
Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien
Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.
2023{\natexlab{b}}.
\newblock \href {http://arxiv.org/abs/2307.09288} {Llama 2: Open foundation and
fine-tuned chat models}.
\newblock \emph{arXiv preprint arXiv:2307.09288}.
\bibitem[{Vaswani et~al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and Polosukhin}]{Transformer}
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan~N Gomez, \L~ukasz Kaiser, and Illia Polosukhin. 2017.
\newblock \href
{https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf}
{Attention is all you need}.
\newblock In \emph{Advances in Neural Information Processing Systems},
volume~30. Curran Associates, Inc.
\bibitem[{Wang et~al.(2022)Wang, Yang, Huang, Jiao, Yang, Jiang, Majumder, and
Wei}]{wang2022text}
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang,
Rangan Majumder, and Furu Wei. 2022.
\newblock \href {http://arxiv.org/abs/2212.03533} {Text embeddings by
weakly-supervised contrastive pre-training}.
\newblock \emph{arXiv preprint arXiv:2212.03533}.
\bibitem[{Wei et~al.(2022{\natexlab{a}})Wei, Tay, Bommasani, Raffel, Zoph,
Borgeaud, Yogatama, Bosma, Zhou, Metzler, Chi, Hashimoto, Vinyals, Liang,
Dean, and Fedus}]{emergent}
Jason Wei, Yi~Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian
Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed~H.
Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William
Fedus. 2022{\natexlab{a}}.
\newblock \href {https://openreview.net/forum?id=yzkSU5zdwD} {Emergent
abilities of large language models}.
\newblock \emph{Transactions on Machine Learning Research}.
\newblock Survey Certification.
\bibitem[{Wei et~al.(2022{\natexlab{b}})Wei, Wang, Schuurmans, Bosma, ichter,
Xia, Chi, Le, and Zhou}]{CoT}
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia,
Ed~Chi, Quoc~V Le, and Denny Zhou. 2022{\natexlab{b}}.
\newblock \href
{https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf}
{Chain-of-thought prompting elicits reasoning in large language models}.
\newblock In \emph{Advances in Neural Information Processing Systems},
volume~35, pages 24824--24837. Curran Associates, Inc.
\bibitem[{Wolf et~al.(2020)Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac,
Rault, Louf, Funtowicz, Davison, Shleifer, von Platen, Ma, Jernite, Plu, Xu,
Le~Scao, Gugger, Drame, Lhoest, and Rush}]{huggingface}
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe
Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien
Plu, Canwen Xu, Teven Le~Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest,
and Alexander Rush. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.emnlp-demos.6} {Transformers:
State-of-the-art natural language processing}.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations}, pages 38--45, Online.
Association for Computational Linguistics.
\bibitem[{Xu et~al.(2023)Xu, Sun, Zheng, Geng, Zhao, Feng, Tao, and
Jiang}]{xu2023wizardlm}
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu~Zhao, Jiazhan Feng, Chongyang
Tao, and Daxin Jiang. 2023.
\newblock \href {http://arxiv.org/abs/2304.12244} {Wizardlm: Empowering large
language models to follow complex instructions}.
\newblock \emph{arXiv preprint arXiv:2304.12244}.
\end{thebibliography}