Natural Context Drift Undermines the Natural Language Understanding of Large Language Models
Extracted source text · arXiv 2509.01093
\documentclass[11pt]{article}
\pdfminorversion=5
\pdfobjcompresslevel=0
% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[preprint]{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% Own package includes
\usepackage{hyperref}
\usepackage{url}
\usepackage{tabularx}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{soul}
\usepackage{graphicx}
\usepackage{xcolor,colortbl}
\definecolor{lightgray}{gray}{0.9}
\usepackage{bbding}
\usepackage{pifont}
\usepackage{diagbox}
\usepackage{longtable}
\usepackage{colortbl}
\usepackage{rotating}
\usepackage{wrapfig}
\usepackage[normalem]{ulem}
\usepackage[most]{tcolorbox}
\usepackage{subcaption}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\title{Natural Context Drift Undermines the Natural\\ Language Understanding of Large Language Models}
% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
% Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
% Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \And ... \And
% Author n \\ Address line \\ ... \\ Address line}
% To start a separate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\ ... \\ Address line
% \AND
% Author 2 \\ Address line \\ ... \\ Address line \And
% Author 3 \\ Address line \\ ... \\ Address line}
\author{Yulong Wu\textsuperscript{1}, Viktor Schlegel\textsuperscript{1, 2} and Riza Batista-Navarro\textsuperscript{1} \\
\textsuperscript{1} Department of Computer Science, University of Manchester, United Kingdom \\
\textsuperscript{2} Imperial College London, Imperial Global Singapore \\
\texttt{\{yulong.wu, riza.batista\}@manchester.ac.uk} \\
\texttt{v.schlegel@imperial.ac.uk}\\}
%\author{
% \textbf{First Author\textsuperscript{1}},
% \textbf{Second Author\textsuperscript{1,2}},
% \textbf{Third T. Author\textsuperscript{1}},
% \textbf{Fourth Author\textsuperscript{1}},
%\\
% \textbf{Fifth Author\textsuperscript{1,2}},
% \textbf{Sixth Author\textsuperscript{1}},
% \textbf{Seventh Author\textsuperscript{1}},
% \textbf{Eighth Author \textsuperscript{1,2,3,4}},
%\\
% \textbf{Ninth Author\textsuperscript{1}},
% \textbf{Tenth Author\textsuperscript{1}},
% \textbf{Eleventh E. Author\textsuperscript{1,2,3,4,5}},
% \textbf{Twelfth Author\textsuperscript{1}},
%\\
% \textbf{Thirteenth Author\textsuperscript{3}},
% \textbf{Fourteenth F. Author\textsuperscript{2,4}},
% \textbf{Fifteenth Author\textsuperscript{1}},
% \textbf{Sixteenth Author\textsuperscript{1}},
%\\
% \textbf{Seventeenth S. Author\textsuperscript{4,5}},
% \textbf{Eighteenth Author\textsuperscript{3,4}},
% \textbf{Nineteenth N. Author\textsuperscript{2,5}},
% \textbf{Twentieth Author\textsuperscript{1}}
%\\
%\\
% \textsuperscript{1}Affiliation 1,
% \textsuperscript{2}Affiliation 2,
% \textsuperscript{3}Affiliation 3,
% \textsuperscript{4}Affiliation 4,
% \textsuperscript{5}Affiliation 5
%\\
% \small{
% \textbf{Correspondence:} \href{mailto:email@domain}{email@domain}
% }
%}
\begin{document}
\maketitle
\begin{abstract}
How does the natural evolution of context paragraphs affect question answering in generative Large Language Models (LLMs)? To investigate this, we propose a framework for curating naturally evolved, human-edited variants of reading passages from contemporary QA benchmarks and for analyzing LLM performance across a range of semantic similarity scores, which quantify how closely each variant aligns with content seen during pretraining. Using this framework, we evaluate six QA datasets and eight LLMs with publicly available training data. Our experiments reveal that LLM performance declines as reading passages naturally diverge from the versions encountered during pretraining—even when the question and all necessary information remains present at inference time. For instance, average model accuracy on \textsc{BoolQ} drops by over $30$\% from the highest to lowest similarity bins, with slopes exceeding $70$ across several LLMs. These findings suggest that natural text evolution poses a significant challenge to the language understanding capabilities of LLMs.
\end{abstract}
\section{Introduction}
\label{sec:Introduction}
Large Language Models (LLMs), pre-trained on massive web-scale corpora, have proven effective at Question Answering (QA) over text passages~\citep{openai2024gpt4technicalreport, deepseekai2025deepseekv3technicalreport, deepseekai2025deepseekr1incentivizingreasoningcapability, yang2025qwen3technicalreport, olmo20252olmo2furious}, a task that has long been established as a testbed for evaluating natural language understanding~\citep{chen2018neural}. Nonetheless, concerns remain regarding their genuine reading comprehension abilities and generalization, as revealed by research efforts on robustness evaluation~\citep{wu-etal-2023-machine, levy-etal-2023-guiding}, benchmark contamination impact analysis~\citep{palavalli-etal-2024-taxonomy, li-etal-2024-open-source}, and others.
Differentiating from previous work, this paper offers a new perspective on understanding the limitations of generative LLMs by asking: what happens when reading paragraphs continue to evolve and diverge from their appearance during pretraining? This scenario is common in real-world applications, where test data naturally changes over time due to ongoing human edits, content updates, or shifts in context, (e.g., Wikipedia articles~\citep{yang-etal-2017-identifying-semantic}), and therefore requires genuine language understanding from LLMs. To the best of our knowledge, however, no prior work has systematically investigated this phenomenon in QA.
To address this gap, we propose a framework to analyse how LLMs performance changes as the reading paragraph semantically diverges from the content of its source in the model’s training corpora. Among various examples of evolving text corpora, we focus on Wikipedia, as it serves as a primary source for reading passages in widely used QA benchmarks~\citep{wang2022modernquestionansweringdatasets}, is commonly included in LLM training~\citep{zhao2025surveylargelanguagemodels}, and most importantly, the evolution of text is clearly documented via revision histories. This enables us to curate human-edited variants of passages that reflect natural text evolution over time. Our approach adopts a gradual perspective by computing a continuous semantic similarity score at the paragraph level and correlating it with LLM's QA accuracy.
Within the developed framework, we empirically evaluate six QA benchmark datasets and eight LLMs with fully open-source training data. Our study finds that, across models with different training corpora and architectural configurations, as context paragraphs naturally evolve and become semantically distant from the Wikipedia content sharing the same article title seen during pretraining, the reading comprehension performance of LLMs generally deteriorates. In contrast, human annotators are less affected by such semantic drift and maintain relatively stable accuracy regardless of passage similarity, suggesting that the observed performance drop is specific to LLMs and not due to deficiencies in the edited passages themselves.
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{latex/methodology/methodology.drawio.pdf}
\caption{An overview of the analysis framework. Module~\textit{Naturally evolved human-edited reading paragraphs extraction} is adapted from \citep{wu2025payattentionrealworld} with minor modifications. APC: Answers Preserving Checking.}
\label{fig:methodology}
\end{figure*}
\section{Methodology}
\label{sec:Methodology}
In our framework (Figure~\ref{fig:methodology}), we extract revision histories of paragraphs from QA benchmarks, order them by semantic similarity to the version that appears in an LLM's training corpus, and correlate the LLMs' answer accuracy on those passages to the similarity thus obtained. The framework consists of two modules, described in detail below.
\paragraph{Naturally evolved human-edited reading paragraphs extraction.} To obtain edited versions of original reading paragraphs from contemporary QA benchmark datasets that genuinely reflect real-world scenarios, we adopt the natural perturbation pipeline proposed by~\citet{wu2025payattentionrealworld}, with two slight modifications: 1) we remove the constraint of retaining only candidate passage pairs where both paragraphs exceed 500 characters, allowing broader dataset applicability and preservation of diverse editing patterns; and 2) for the matched
original passages with multiple occurrences, we retain all edited versions for each (see passage \textit{OP\textsubscript{2}} in Figure~\ref{fig:methodology} as an example) to support subsequent correlation analysis. Appendix~\ref{sec:Answer Preservation Check and Data Statistics} provides details on Answers Preserving Checking and data statistics.
\paragraph{Semantic similarity-LLM average accuracy correlation analysis.} For each naturally evolved, human-edited reading paragraph and its associated question, we generate predictions using an LLM and label them as 1 (correct) or 0 (incorrect), based on a selected evaluation metric. We also collect predictions using the question alone to test whether the LLM already possesses parametric knowledge of the answer. Instances in which the LLM answers correctly without access to the passage are excluded, as they may call into question the paragraph’s contribution to the answer \citep{glockner2025neoqaevidencebasedquestionanswering}~\footnote{Appendix~\ref{sec:Percentage of Instances where LLMs Succeed on Context-Free Question Answering} shows that, within each generated dataset in Module~\textit{Naturally evolved human-edited reading paragraphs extraction}, the percentage of instances in which \texttt{OLMo} LLMs succeed on context-free QA, and are thus filtered out.}. Meanwhile, we extract English Wikipedia content from the LLM’s training corpora that shares the same article title as the edited passage and compute the semantic similarity between them. The maximum similarity score is used as a proxy for how closely the passage resembles the training data. We then group the similarity scores into ten bins, compute the average LLM accuracy within each bin, and plot accuracy trends from highest to lowest similarity. Uncertainty for the result in each bin is estimated using the Wilson score interval with 95\% confidence ($z$ = 1.96) \citep{Wilson01061927}. Finally, to validate the observed trend, we assess human performance across the same bins.
\section{Experiments Setup}
\label{sec:Experiments Setup}
\begin{figure*}[htb!]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/instruction-tuned/olmo_results/squadv1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 1.1}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/instruction-tuned/olmo_results/squadv2_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 2.0}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/instruction-tuned/olmo_results/droberta_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{D(RoBERTa)}}
\end{subfigure}
\vspace{0.5cm}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/instruction-tuned/olmo_results/boolq_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{BoolQ}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/instruction-tuned/olmo_results/wikiwhy_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{WikiWhy}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/instruction-tuned/olmo_results/hotpotqa-v1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{HotpotQA}}
\end{subfigure}
\caption{Average accuracy of the instruction-finetuned \texttt{OLMo} LLMs across ten semantic similarity bins.}
\label{fig:olmo_results}
\end{figure*}
\begin{figure*}[htb!]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/instruction-tuned/other_results/squadv2_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 2.0}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/instruction-tuned/other_results/wikiwhy_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{WikiWhy}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/instruction-tuned/other_results/hotpotqa-v1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{HotpotQA}}
\end{subfigure}
\caption{Average accuracy of other instruction-finetuned LLMs across ten semantic similarity bins.}
\label{fig:other_results}
\end{figure*}
Broadly, we address the following question: \textit{How well do LLMs perform on QA as the reading paragraphs naturally evolve from the versions present in their training corpus?} To this end, we select QA benchmarks that feature context paragraphs from Wikipedia, whose edit histories allow us to trace naturally evolved versions of each paragraph, and evaluate LLMs with an open-source training corpus, as detailed below.
\textbf{Datasets:} We use the development set of six English QA datasets spanning extractive, yes/no, abstractive, cause-effect reasoning and multi-hop reasoning challenge: \textsc{SQuAD 1.1} \citep{rajpurkar-etal-2016-squad}, \textsc{SQuAD 2.0} \citep{rajpurkar-etal-2018-know}, \textsc{AdversarialQA - D(RoBERTa)} \citep{bartolo-etal-2020-beat}, \textsc{BoolQ} \citep{clark-etal-2019-boolq}, \textsc{WikiWhy} \citep{ho2023wikiwhy} (version 1.2) and \textsc{HotpotQA} \citep{yang-etal-2018-hotpotqa} (in the ``distractor'' setting). For \textsc{HotpotQA}, which includes paragraphs from multiple Wikipedia articles within a single context, we retain only the edits applied to the gold passages, i.e., those containing supporting facts that determine the answer, and disregard other distractors.
\textbf{LLMs:} We evaluate eight transparent instruction-tuned LLMs across six model families, all of which include Wikipedia as part of their publicly available training data: \texttt{\textbf{OLMo}} (\texttt{OLMo-7B-0724-Instruct-hf}) \citep{groeneveld-etal-2024-olmo}, \texttt{\textbf{OLMo 2}} (\texttt{OLMo-2-1124-7B-Instruct} and \texttt{OLMo-2-1124-13B-Instruct}) \citep{olmo20252olmo2furious}, \texttt{\textbf{OLMoE}} (\texttt{OLMoE-1B-7B-0125-Instruct}) \citep{muennighoff2025olmoeopenmixtureofexpertslanguage}, LLM360's \texttt{\textbf{AmberChat}} \citep{liu2024llm}, \texttt{\textbf{TinyLlama}} (\texttt{TinyLlama-1.1B-Chat-v1.0}) \citep{zhang2024tinyllamaopensourcesmalllanguage}, Databricks' \texttt{\textbf{Dolly}} (\texttt{dolly-v2-7b} and \texttt{dolly-v2-12b}) \citep{DatabricksBlog2023DollyV2}. To isolate the potential impact of instruction tuning, we also conduct experiments on the base (non-aligned) versions of the same models. All LLMs are tested in a zero-shot setting, and the inference prompts are provided in Appendix~\ref{sec:qa_prompts}. Model experimentation is carried out using the HuggingFace's~\textit{Transformers} library \citep{wolf-etal-2020-transformers}, the vLLM \citep{10.1145/3600006.3613165}, and two 80GB NVIDIA A100 GPUs. LLM prediction correctness is determined using Inclusion Match (IM), which considers a prediction correct if it includes any ground truth answer \citep{levy-etal-2023-guiding, bhuiya-etal-2024-seemingly}. For \textsc{WikiWhy}, due to its free-form format, correctness is based on semantic similarity (using the~\texttt{all-MiniLM-L6-v2} model) between the LLM’s response and the ground truth answer; predictions scoring above $0.6$ are considered correct.
\textbf{Semantic Similarity Measure:} We measure semantic-level textual similarity between the edited Wikipedia paragraph and the versions found in the Wikipedia subset of each LLM’s training corpus--sourced from \textsc{Dolma v1.7}\footnote{\url{https://huggingface.co/datasets/allenai/dolma}} \citep{soldaini-etal-2024-dolma} for \texttt{OLMo}, \textsc{RedPajama v1}\footnote{\url{https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T}} \citep{together2023redpajama} for \texttt{AmberChat}, \textsc{SlimPajama}\footnote{\url{https://huggingface.co/datasets/cerebras/SlimPajama-627B}} \citep{cerebras2023slimpajama} for \texttt{TinyLlama} and \textsc{Pile}\footnote{As \textsc{Pile} is no longer officially hosted or distributed, we use a publicly accessible replication of its original Wikipedia component: \url{https://github.com/noanabeshima/wikipedia-downloader}.} \citep{gao2020pile800gbdatasetdiverse, biderman2022datasheetpile} for \texttt{Dolly}, using a Sentence Transformers model \texttt{all-MiniLM-L6-v2} \citep{reimers-gurevych-2019-sentence}. In addition, we employ three alternative embedding models for the similarity measure to enhance the generalizability of our study: \texttt{sentence-t5-base} \citep{ni-etal-2022-sentence}, \texttt{all-mpnet-base-v2} and \texttt{bge-small-en-v1.5} \citep{10.1145/3626772.3657878}. In measuring semantic similarity for~\textsc{HotpotQA}, we consider only the paragraphs in the edited context whose corresponding Wikipedia article titles have matching content in the Wikipedia subset of an LLM's training corpus.
\section{Results and Discussion}
\label{sec:Results and Discussion}
\textit{\textbf{As a general trend, the QA performance of instruction-finetuned \texttt{OLMo} LLMs deteriorates as the reading paragraph semantically deviates from the training corpus.}} This is clearly visualized in Figure~\ref{fig:olmo_results}, where, across the evaluated benchmark datasets, model average accuracy generally declines as the reading paragraphs evolve and exhibit lower semantic similarity to the versions found in the Wikipedia subset of the training corpus. For example, on \textsc{BoolQ}, the accuracy of \texttt{OLMo-2-1124-7B-Instruct} declines sharply from 71.1\% in the highest similarity bin 0.9-1.0 to 12.5\% in the lowest 0.1-0.2. Comparable drops are also observed, for instance, for \texttt{OLMo-7B-0724-Instruct} on \textsc{SQuAD 2.0} and \textsc{WikiWhy}, and for \texttt{OLMo-2-1124-13B-Instruct} on \textsc{D(RoBERTa)} and \textsc{HotpotQA}, illustrating the broader impact of natural context drift in diverse QA challenges. To further substantiate the observed trend, we perform a slope analysis as shown in Appendix~\ref{sec:Slope Analysis of Accuracy vs. Semantic Similarity} Figure~\ref{fig:slope}, where linear regressions are fitted to each model’s accuracy trajectory across semantic similarity bins. Aggregated across all tasks and models, the mean slope is 65.78 ± 41.55, with a mean Pearson correlation of 0.684 ± 0.291, highlighting a consistent and statistically grounded relationship between semantic divergence and performance degradation. Further, this downward trend is not exclusive to the \texttt{OLMo} family. As shown in Figure~\ref{fig:other_results}, \textit{\textbf{a comparable decline in average performance across decreasing similarity ranges is likewise observed in other instruction-finetuned LLMs, including \texttt{AmberChat}, \texttt{TinyLlama-1.1B-Chat-v1.0} and \texttt{dolly-v2-7b/12b}, despite differences in model size, architecture, training corpora and procedure.}} To further investigate the generalisability of our findings, we conduct an ablation study using alternative embedding models for semantic similarity measurement (Figure~\ref{fig:olmo_results_alternative_metrics} in Appendix~\ref{sec:Consequences of Context Drift Persist Across Other Embedding Models}), evaluating the base pre-trained versions of the instruction-finetuned LLMs (Figure~\ref{fig:olmo_results_Base} and Figure~\ref{fig:other_results_Base} in Appendix~\ref{sec:Context Drift Effects Hold Without Instruction Tuning}) and prompting LLMs to perform chain-of-thought (CoT) reasoning \citep{10.5555/3600270.3602070} when generating responses (Figure~\ref{fig:olmo_results_CoT} and Figure~\ref{fig:other_results_CoT} in Appendix~\ref{sec:Chain-of-Thought Does Not Alleviate Drift-Induced Degradation}). \textit{\textbf{Overall, the observed effects of context drift persist under these alternative settings, supporting the robustness and generalisability of our conclusions.}}
We find from Figure~\ref{fig:olmo_results}, Figure~\ref{fig:other_results}, and Figure~\ref{fig:slope} that \textit{\textbf{the impact of natural context drift is more pronounced in benchmarks that emphasize surface-form alignment, whereas tasks requiring deeper reasoning exhibit greater resilience.}} The downward trend is less pronounced for datasets requiring advanced reasoning, such as \textsc{WikiWhy} and \textsc{HotpotQA}. This may be because natural text evolution introduces diverse linguistic cues and contextual variations that activate broader reasoning mechanisms in LLMs, partially offsetting the impact of semantic drift. In contrast, surface-level QA tasks such as \textsc{SQuAD} rely more heavily on lexical or structural cues \citep{schlegel-etal-2020-framework, wu-etal-2021-understanding} and therefore appear more sensitive to the text evolution. Dataset-level statistics support this contrast: \textsc{BoolQ} and \textsc{SQuAD 2.0} exhibit quite steep average slopes (67.23 and 83.50, respectively) and strong Pearson correlations between average accuracy and semantic similarity (e.g., \textsc{BoolQ}'s average correlation = 0.867 ± 0.064). Meanwhile, \textsc{HotpotQA} demonstrates a much shallower 23.94 average slope and lower correlation, indicating greater robustness to textual edits.
\textit{\textbf{Unlike LLMs, human performance in reading comprehension is not influenced by deviations in measured semantic similarity.}} To determine whether the observed decline in LLMs’ accuracy is due to reduced semantic similarity or the possibility that the edited reading paragraphs became degraded and unanswerable, we evaluate human performance across semantic similarity bins within the four investigated datasets. For each dataset, we randomly sample an equal number of edited QA instances from each bin, assign two annotators to label the answers, and involve a third annotator to resolve any disagreements. Details of the human annotation protocol are provided in Appendix~\ref{sec:Human Annotation Details}. As shown in Figure~\ref{fig:humanQAaccuracy}, across all QA benchmarks, human performance does not consistently decline with decreasing semantic similarity, supporting the conclusion that the degradation in LLM accuracy stems from semantic drift rather than a loss of question answerability.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{latex/experimental_results/humanQAaccuracy.pdf}
\caption{Accuracy of human annotators on QA tasks across semantic similarity bins.}
\label{fig:humanQAaccuracy}
\end{figure}
\textit{\textbf{There may be little concern regarding the impact of paragraph leakage from other data sources beyond Wikipedia.}} A natural question arises as to whether, given the vast scale of LLMs' training corpora, the edited versions of the reading paragraphs might also appear in sources beyond the Wikipedia subset we focus on, potentially affecting the findings. Therefore, using the infini-gram engine~\citep{liu2024infinigram}, we aim to estimate as accurately as possible the percentage of edited reading paragraphs that appear verbatim in the complete training corpus of the evaluated LLMs across the six examined QA benchmarks, as shown in Table~\ref{tab:other_sources}. Infini-gram enables efficient querying over the whole training data of \texttt{OLMo-2-1124-13B-Instruct} (whose results are also used as a proxy for \texttt{OLMo-2-1124-7B-Instruct}, given the same training data shared) and \texttt{OLMoE-1B-7B-0125-Instruct}. For the remaining LLMs, we are limited to querying their pretraining corpora (which already account for a significant portion of the overall training data): \textsc{Dolma 1.7} for \texttt{OLMo-7B-0724-Instruct-hf}, \textsc{RedPajama v1} for \texttt{AmberChat} and \texttt{TinyLlama-1.1B-Chat-v1.0} (used as a proxy), \textsc{Pile} for \texttt{dolly-v2-7b} and \texttt{dolly-v2-12b}. As shown in Table~\ref{tab:other_sources}, verbatim inclusion of the edited paragraphs in the LLMs’ training corpora remains negligible across the board. With additional consideration that our analysis does not include all the edited reading paragraphs, we believe that the impact of these paragraphs appearing in other data sources may not be significant. Finally, we emphasise that our methodology focuses on measuring the semantic similarity between the edited reading paragraphs and the content from the same source, i.e., Wikipedia. Therefore, extending the analysis to other potential sources falls outside the scope of this paper, and we leave this for future investigation.
\begin{table}[htb!]
\centering
\resizebox{\linewidth}{!}{
\begin{tabular}{lcccccc}
\toprule
&\textsc{SQuAD 1.1}&\textsc{SQuAD 2.0}&\textsc{D(RoBERTa)}&\textsc{BoolQ}&\textsc{WikiWhy}&\textsc{HotpotQA}\\
\midrule
\texttt{OLMo-7B-0724-Instruct-hf}&1.59&3.94&1.33&4.30&1.52&-\\
\texttt{OLMo-2-1124-7B/13B-Instruct}&4.85&8.53&4.79&2.75&1.14&-\\
\texttt{OLMoE-1B-7B-0125-Instruct}&4.77&8.69&4.79&2.75&1.14&-\\
\texttt{AmberChat}\&\texttt{TinyLlama-1.1B-Chat-v1.0}&1.04&2.40&2.13&4.47&1.14&-\\
\texttt{dolly-v2-7b/12b}&0.18&1.07&-&0.40&0.76&-\\
\bottomrule
\end{tabular}}
\caption{Percentage (\%) of edited reading paragraphs that appear verbatim in the training corpora of the evaluated LLMs.}
\label{tab:other_sources}
\end{table}
\section{Conclusion}
\label{sec:Conclusion}
We introduce a novel methodology for examining how real-world natural context evolution affects the language understanding of LLMs when it deviates semantically from their training data. Leveraging Wikipedia revision histories, we curate naturally human-edited variants of benchmark reading passages, compute their semantic similarity to versions present in the models' training corpus, and correlate these similarity scores with model performance. Our empirical findings show that, while natural text evolution has little to no effect on human QA performance, LLMs generally exhibit a consistent downward trend in accuracy as semantic similarity decreases. We hope this study contributes to the growing body of research focused on understanding and addressing the limitations of LLMs in real-world, evolving textual contexts.
\section*{Limitations}
Our work has several limitations. First, we focus exclusively on the QA task; extending these findings to other downstream tasks remains an open avenue for future research. Second, we evaluate only transparent LLMs with fully open-access training corpora. While this constraint is necessary for our methodology, some of these models still lag behind state-of-the-art proprietary LLMs in performance. Applying our framework to such proprietary models would be a valuable extension, although it would require access to their training data, which is currently unavailable.
% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{latex/anthology,latex/custom}
\appendix
\section{Answer Preservation Check and Data Statistics}
\label{sec:Answer Preservation Check and Data Statistics}
For answer preservation checking, in the extractive QA setting (including the extractive question set for \textsc{HotpotQA}), we ensure that at least one (or all) of the ground truth answers can still be found in the edited passage. For \textsc{BoolQ}, we manually inspect the generated edited test set and remove instances where the edited passage contains fewer than 56 characters. We also check \textsc{WikiWhy}, but no filtering is applied. Table~\ref{tab:statistics} presents the statistics of the extracted data for each QA dataset.
\begin{table}[htb!]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{lccccc}
\toprule
\textbf{Dataset} & \textbf{Titles} & \textbf{Passages} & \textbf{Edited} & \textbf{Avg. per} & \textbf{Questions} \\
& \textbf{(ext./total)} & & \textbf{Passages} & \textbf{Passage} & \\
\midrule
\textsc{SQuAD 1.1} \citep{rajpurkar-etal-2016-squad} & 47/48 & 825 & 1,531 & 8.85 & 3,920 \\
\midrule
\textsc{SQuAD 2.0} \citep{rajpurkar-etal-2018-know} & 47/48 & 466 & 914 & 17.82 & 4,281 \\
\midrule
\textsc{D(RoBERTa)} \citep{bartolo-etal-2020-beat} & 47/48 & 135 & 249 & 5.56 & 376 \\
\midrule
\textsc{BoolQ} \citep{clark-etal-2019-boolq} & 2,488/2,651 & 957 & 2,559 & 3.16 & 1,064 \\
\midrule
\textsc{WikiWhy} \citep{ho2023wikiwhy} & 833/873 & 193 & 251 & 1.36 & 193 \\
\midrule
\textsc{HotpotQA} \citep{yang-etal-2018-hotpotqa} & 10,971/13,783 & 2,948 & 7,350 & 2.79 & 2,970 \\
\bottomrule
\end{tabular}}
\caption{Summary of QA benchmarks with paragraph evolution histories extracted from Wikipedia edits.}
\label{tab:statistics}
\end{table}
\section{Percentage of Instances where LLMs Succeed on Context-Free Question Answering}
\label{sec:Percentage of Instances where LLMs Succeed on Context-Free Question Answering}
\begin{table}[htb!]
\centering
\resizebox{\columnwidth}{!}{
\begin{tabular}{lcccc}
\toprule
\diagbox{\textbf{Dataset}}{\textbf{LLM}} & \texttt{OLMo-7B-0724-Instruct-hf} & \texttt{OLMo-2-1124-7B-Instruct} & \texttt{OLMo-2-1124-13B-Instruct} & \texttt{OLMoE-1B-7B-0125-Instruct} \\
\midrule
\textsc{SQuAD 1.1} & $2.63$ & $5.41$ & $5.81$ & $2.42$ \\
\textsc{SQuAD 2.0} & $53.55$ & $52.07$ & $56.48$ & $55.43$ \\
\textsc{D(RoBERTa)} & $2.13$ & $12.52$ & $8.66$ & $6.26$ \\
\textsc{BoolQ} & $36.35$ & $48.69$ & $61.40$ & $45.85$ \\
\textsc{WikiWhy} & $0.00$ & $3.42$ & $0.00$ & $2.66$ \\
\textsc{HotpotQA} & $2.77$ & $9.30$ & $7.89$ & $8.21$ \\
\bottomrule
\end{tabular}}
\caption{Percentage of instances with questions that are correctly answered by an LLM without access to the context paragraph.}
\label{tab:memorisation_percentage}
\end{table}
\section{Inference Prompts for QA Tasks}
\label{sec:qa_prompts}
This appendix provides the complete prompt templates used for zero-shot inference across all QA datasets evaluated in this study.
\begin{center}
\textbf{\textsc{SQuAD 1.1}} \& \textbf{\textsc{SQuAD 2.0}} \& \textbf{\textsc{D(RoBERTa)}}
\end{center}
\begin{quote}
\textit{Use the provided article delimited by triple quotes to answer question. Provide only the shortest continuous span from the context without any additional explanation. If the question is unanswerable, return ``unanswerable''.}
\texttt{"""{passage}"""}
\textit{Question:} \texttt{question}
\end{quote}
\textit{Rationale:} Explicitly requests the shortest continuous span to encourage precise extraction for extractive reading comprehension tasks.
\begin{quote}
\textit{\textbf{parametric knowledge testing:} Provide an answer to the given question. If the question is unanswerable, return ``unanswerable''. Do not provide any explanation.}
\textit{Question:} \texttt{question}
\end{quote}
\begin{center}
\textbf{\textsc{BoolQ}}
\end{center}
\begin{quote}
\textit{Use the provided article delimited by triple quotes to answer question. Return only TRUE or FALSE. If the question is unanswerable, return ``unanswerable''. Do not provide any explanation.}
\texttt{"""{passage}"""}
\textit{Question:} \texttt{question}
\end{quote}
\textit{Rationale:} Constrains output format to TRUE/FALSE responses for binary classification tasks.
\begin{quote}
\textit{\textbf{parametric knowledge testing:} Provide an answer to the given question. Return only TRUE or FALSE. If the question is unanswerable, return ``unanswerable''. Do not provide any explanation.}
\textit{Question:} \texttt{question}
\end{quote}
\begin{center}
\textbf{\textsc{WikiWhy}} \& \textbf{\textsc{HotpotQA}}
\end{center}
\begin{quote}
\textit{Use the provided article delimited by triple quotes to answer question. If the question is unanswerable, return ``unanswerable''. Do not provide any explanation.}
\texttt{"""{passage}"""}
\textit{Question:} \texttt{question}
\end{quote}
\textit{Rationale:} Allows for more flexible answer generation while maintaining the unanswerable option for cause-effect reasoning and multi-hop reasoning tasks.
\begin{quote}
\textit{\textbf{parametric knowledge testing:} Provide an answer to the given question. If the question is unanswerable, return ``unanswerable''. Do not provide any explanation.}
\textit{Question:} \texttt{question}
\end{quote}
\section{Slope Analysis of Accuracy vs. Semantic Similarity}
\label{sec:Slope Analysis of Accuracy vs. Semantic Similarity}
\begin{figure*}[htbp]
\centering
\begin{subfigure}[b]{0.45\textwidth}
\centering
\includegraphics[width=\textwidth]{latex/experimental_results/quantitative_measure/squadv1_summary.pdf}
\caption{\textsc{SQuAD 1.1}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\textwidth}
\centering
\includegraphics[width=\textwidth]{latex/experimental_results/quantitative_measure/squadv2_summary.pdf}
\caption{\textsc{SQuAD 2.0}}
\end{subfigure}
\vspace{1em}
\begin{subfigure}[b]{0.45\textwidth}
\centering
\includegraphics[width=\textwidth]{latex/experimental_results/quantitative_measure/droberta_summary.pdf}
\caption{\textsc{D(RoBERTa)}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.45\textwidth}
\centering
\includegraphics[width=\textwidth]{latex/experimental_results/quantitative_measure/boolq_summary.pdf}
\caption{\textsc{BoolQ}}
\end{subfigure}
\vspace{1em}
\begin{subfigure}[b]{0.45\textwidth}
\centering
\includegraphics[width=\textwidth]{latex/experimental_results/quantitative_measure/hotpotqa-v1_summary.pdf}
\caption{\textsc{HotpotQA}}
\end{subfigure}
\caption{Average accuracy of instruction-finetuned \texttt{OLMo} LLMs across semantic similarity bins. Each point represents the mean accuracy within a similarity bin, and the dashed lines are linear regression fits summarizing the accuracy trend for each model. Slope values indicate how rapidly model accuracy changes with semantic divergence.}
\label{fig:slope}
\end{figure*}
\section{Consequences of Context Drift Persist Across Other Embedding Models}
\label{sec:Consequences of Context Drift Persist Across Other Embedding Models}
\begin{figure*}[htb!]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/sentence-t5/olmo_results/squadv1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 1.1}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/sentence-t5/olmo_results/boolq_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{BoolQ}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/sentence-t5/olmo_results/wikiwhy_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{WikiWhy}}
\end{subfigure}
\vspace{0.5cm}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/all-mpnet-base-v2/olmo_results/squadv1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 1.1}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/all-mpnet-base-v2/olmo_results/boolq_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{BoolQ}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/all-mpnet-base-v2/olmo_results/wikiwhy_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{WikiWhy}}
\end{subfigure}
\vspace{0.5cm}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/bge/olmo_results/squadv1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 1.1}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/bge/olmo_results/boolq_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{BoolQ}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/bge/olmo_results/wikiwhy_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{WikiWhy}}
\end{subfigure}
\caption{Average accuracy of the instruction-finetuned \texttt{OLMo} LLMs across ten semantic similarity bins. The first, second, and third rows show the results obtained using the embedding models \texttt{sentence-t5-base}, \texttt{all-mpnet-base-v2}, and \texttt{bge-small-en-v1.5}, respectively.}
\label{fig:olmo_results_alternative_metrics}
\end{figure*}
\section{Context Drift Effects Hold Without Instruction Tuning}
\label{sec:Context Drift Effects Hold Without Instruction Tuning}
Figures~\ref{fig:olmo_results_Base} and~\ref{fig:other_results_Base} show that non-aligned base LLMs are also susceptible to the harmful effects of natural context drift.
\begin{figure*}[htb!]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/pre-trained/olmo_results/squadv1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 1.1}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/pre-trained/olmo_results/squadv2_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 2.0}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/pre-trained/olmo_results/droberta_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{D(RoBERTa)}}
\end{subfigure}
\vspace{0.5cm}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/pre-trained/olmo_results/boolq_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{BoolQ}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/pre-trained/olmo_results/wikiwhy_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{WikiWhy}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/pre-trained/olmo_results/hotpotqa-v1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{HotpotQA}}
\end{subfigure}
\caption{Average accuracy of the Base \texttt{OLMo} LLMs across ten semantic similarity bins.}
\label{fig:olmo_results_Base}
\end{figure*}
\begin{figure*}[htb!]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/pre-trained/other_results/squadv2_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 2.0}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/pre-trained/other_results/wikiwhy_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{WikiWhy}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/pre-trained/other_results/hotpotqa-v1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{HotpotQA}}
\end{subfigure}
\caption{Average accuracy of other Base LLMs across ten semantic similarity bins.}
\label{fig:other_results_Base}
\end{figure*}
\section{Chain-of-Thought Does Not Alleviate Drift-Induced Degradation}
\label{sec:Chain-of-Thought Does Not Alleviate Drift-Induced Degradation}
\begin{figure*}[htb!]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/cot_instruction-tuned/olmo_results/squadv1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 1.1}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/cot_instruction-tuned/olmo_results/squadv2_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 2.0}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/cot_instruction-tuned/olmo_results/droberta_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{D(RoBERTa)}}
\end{subfigure}
\vspace{0.5cm}
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/cot_instruction-tuned/olmo_results/boolq_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{BoolQ}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/cot_instruction-tuned/olmo_results/wikiwhy_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{WikiWhy}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/cot_instruction-tuned/olmo_results/hotpotqa-v1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{HotpotQA}}
\end{subfigure}
\caption{Average accuracy of the instruction-finetuned \texttt{OLMo} LLMs across ten semantic similarity bins, with the prompt encouraging elaborated answers and CoT reasoning.}
\label{fig:olmo_results_CoT}
\end{figure*}
\begin{figure*}[htb!]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/cot_instruction-tuned/other_results/squadv2_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{SQuAD 2.0}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/cot_instruction-tuned/other_results/wikiwhy_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{WikiWhy}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{latex/experimental_results/cot_instruction-tuned/other_results/hotpotqa-v1_Semantic_similarity_accuracy_plot.pdf}
\caption{\textsc{HotpotQA}}
\end{subfigure}
\caption{Average accuracy of other instruction-finetuned LLMs across ten semantic similarity bins, with the prompt encouraging elaborated answers and CoT reasoning.}
\label{fig:other_results_CoT}
\end{figure*}
\section{Human Annotation Details}
\label{sec:Human Annotation Details}
We adopt the same instructions provided to human annotators in \citep{wu2025payattentionrealworld} and recruit doctoral students from the university as annotators. All participants possess strong English reading comprehension skills. Before beginning the main annotation task, annotators are asked to label a small set of instances and resolve any disagreements through discussion until consensus is reached. During annotation, they are provided only with the edited reading paragraph and the corresponding question, without being informed that the paragraph has been modified, in order to minimize potential bias.
\end{document}
\begin{thebibliography}{42}
\providecommand{\natexlab}[1]{#1}
\bibitem[{Bartolo et~al.(2020)Bartolo, Roberts, Welbl, Riedel, and Stenetorp}]{bartolo-etal-2020-beat}
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020.
\newblock \href {https://doi.org/10.1162/tacl_a_00338} {Beat the {AI}: Investigating adversarial human annotation for reading comprehension}.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 8:662--678.
\bibitem[{Bhuiya et~al.(2024)Bhuiya, Schlegel, and Winkler}]{bhuiya-etal-2024-seemingly}
Neeladri Bhuiya, Viktor Schlegel, and Stefan Winkler. 2024.
\newblock \href {https://doi.org/10.18653/v1/2024.emnlp-main.147} {Seemingly plausible distractors in multi-hop reasoning: Are large language models attentive readers?}
\newblock In \emph{Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, pages 2514--2528, Miami, Florida, USA. Association for Computational Linguistics.
\bibitem[{Biderman et~al.(2022)Biderman, Bicheno, and Gao}]{biderman2022datasheetpile}
Stella Biderman, Kieran Bicheno, and Leo Gao. 2022.
\newblock \href {https://arxiv.org/abs/2201.07311} {Datasheet for the pile}.
\newblock \emph{Preprint}, arXiv:2201.07311.
\bibitem[{Chen(2018)}]{chen2018neural}
Danqi Chen. 2018.
\newblock \emph{Neural Reading Comprehension and Beyond}.
\newblock Ph.D. thesis, Stanford University.
\bibitem[{Clark et~al.(2019)Clark, Lee, Chang, Kwiatkowski, Collins, and Toutanova}]{clark-etal-2019-boolq}
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.
\newblock \href {https://doi.org/10.18653/v1/N19-1300} {{B}ool{Q}: Exploring the surprising difficulty of natural yes/no questions}.
\newblock In \emph{Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)}, pages 2924--2936, Minneapolis, Minnesota. Association for Computational Linguistics.
\bibitem[{Computer(2023)}]{together2023redpajama}
Together Computer. 2023.
\newblock \href {https://github.com/togethercomputer/RedPajama-Data} {Redpajama: An open source recipe to reproduce llama training dataset}.
\bibitem[{Conover et~al.(2023)Conover, Hayes, Mathur, Xie, Wan, Shah, Ghodsi, Wendell, Zaharia, and Xin}]{DatabricksBlog2023DollyV2}
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.
\newblock \href {https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm} {Free dolly: Introducing the world's first truly open instruction-tuned llm}.
\bibitem[{DeepSeek-AI et~al.(2025{\natexlab{a}})DeepSeek-AI, Guo, Yang, Zhang, Song, Zhang, Xu, Zhu, Ma, Wang, Bi, Zhang, Yu, Wu, Wu, Gou, Shao, Li, Gao, Liu, Xue, Wang, Wu, Feng, Lu, Zhao, Deng, Zhang, Ruan, Dai, Chen, Ji, Li, Lin, Dai, Luo, Hao, Chen, Li, Zhang, Bao, Xu, Wang, Ding, Xin, Gao, Qu, Li, Guo, Li, Wang, Chen, Yuan, Qiu, Li, Cai, Ni, Liang, Chen, Dong, Hu, Gao, Guan, Huang, Yu, Wang, Zhang, Zhao, Wang, Zhang, Xu, Xia, Zhang, Zhang, Tang, Li, Wang, Li, Tian, Huang, Zhang, Wang, Chen, Du, Ge, Zhang, Pan, Wang, Chen, Jin, Chen, Lu, Zhou, Chen, Ye, Wang, Yu, Zhou, Pan, Li, Zhou, Wu, Ye, Yun, Pei, Sun, Wang, Zeng, Zhao, Liu, Liang, Gao, Yu, Zhang, Xiao, An, Liu, Wang, Chen, Nie, Cheng, Liu, Xie, Liu, Yang, Li, Su, Lin, Li, Jin, Shen, Chen, Sun, Wang, Song, Zhou, Wang, Shan, Li, Wang, Wei, Zhang, Xu, Li, Zhao, Sun, Wang, Yu, Zhang, Shi, Xiong, He, Piao, Wang, Tan, Ma, Liu, Guo, Ou, Wang, Gong, Zou, He, Xiong, Luo, You, Liu, Zhou, Zhu, Xu, Huang, Li, Zheng, Zhu, Ma, Tang, Zha, Yan, Ren, Ren, Sha, Fu,
Xu, Xie, Zhang, Hao, Ma, Yan, Wu, Gu, Zhu, Liu, Li, Xie, Song, Pan, Huang, Xu, Zhang, and Zhang}]{deepseekai2025deepseekr1incentivizingreasoningcapability}
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu~Wu, Z.~F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025{\natexlab{a}}.
\newblock \href {https://arxiv.org/abs/2501.12948} {Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning}.
\newblock \emph{Preprint}, arXiv:2501.12948.
\bibitem[{DeepSeek-AI et~al.(2025{\natexlab{b}})DeepSeek-AI, Liu, Feng, Xue, Wang, Wu, Lu, Zhao, Deng, Zhang, Ruan, Dai, Guo, Yang, Chen, Ji, Li, Lin, Dai, Luo, Hao, Chen, Li, Zhang, Bao, Xu, Wang, Zhang, Ding, Xin, Gao, Li, Qu, Cai, Liang, Guo, Ni, Li, Wang, Chen, Chen, Yuan, Qiu, Li, Song, Dong, Hu, Gao, Guan, Huang, Yu, Wang, Zhang, Xu, Xia, Zhao, Wang, Zhang, Li, Wang, Zhang, Zhang, Tang, Li, Tian, Huang, Wang, Zhang, Wang, Zhu, Chen, Du, Chen, Jin, Ge, Zhang, Pan, Wang, Xu, Zhang, Chen, Li, Lu, Zhou, Chen, Wu, Ye, Ye, Ma, Wang, Zhou, Yu, Zhou, Pan, Wang, Yun, Pei, Sun, Xiao, Zeng, Zhao, An, Liu, Liang, Gao, Yu, Zhang, Li, Jin, Wang, Bi, Liu, Wang, Shen, Chen, Zhang, Chen, Nie, Sun, Wang, Cheng, Liu, Xie, Liu, Yu, Song, Shan, Zhou, Yang, Li, Su, Lin, Li, Wang, Wei, Zhu, Zhang, Xu, Xu, Huang, Li, Zhao, Sun, Li, Wang, Yu, Zheng, Zhang, Shi, Xiong, He, Tang, Piao, Wang, Tan, Ma, Liu, Guo, Wu, Ou, Zhu, Wang, Gong, Zou, He, Zha, Xiong, Ma, Yan, Luo, You, Liu, Zhou, Wu, Ren, Ren, Sha, Fu, Xu, Huang, Zhang, Xie,
Zhang, Hao, Gou, Ma, Yan, Shao, Xu, Wu, Zhang, Li, Gu, Zhu, Liu, Li, Xie, Song, Gao, and Pan}]{deepseekai2025deepseekv3technicalreport}
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025{\natexlab{b}}.
\newblock \href {https://arxiv.org/abs/2412.19437} {Deepseek-v3 technical report}.
\newblock \emph{Preprint}, arXiv:2412.19437.
\bibitem[{Gao et~al.(2020)Gao, Biderman, Black, Golding, Hoppe, Foster, Phang, He, Thite, Nabeshima, Presser, and Leahy}]{gao2020pile800gbdatasetdiverse}
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.
\newblock \href {https://arxiv.org/abs/2101.00027} {The pile: An 800gb dataset of diverse text for language modeling}.
\newblock \emph{Preprint}, arXiv:2101.00027.
\bibitem[{Glockner et~al.(2025)Glockner, Jiang, Ribeiro, Gurevych, and Dreyer}]{glockner2025neoqaevidencebasedquestionanswering}
Max Glockner, Xiang Jiang, Leonardo F.~R. Ribeiro, Iryna Gurevych, and Markus Dreyer. 2025.
\newblock \href {https://arxiv.org/abs/2505.05949} {Neoqa: Evidence-based question answering with generated news events}.
\newblock \emph{Preprint}, arXiv:2505.05949.
\bibitem[{Groeneveld et~al.(2024)Groeneveld, Beltagy, Walsh, Bhagia, Kinney, Tafjord, Jha, Ivison, Magnusson, Wang, Arora, Atkinson, Authur, Chandu, Cohan, Dumas, Elazar, Gu, Hessel, Khot, Merrill, Morrison, Muennighoff, Naik, Nam, Peters, Pyatkin, Ravichander, Schwenk, Shah, Smith, Strubell, Subramani, Wortsman, Dasigi, Lambert, Richardson, Zettlemoyer, Dodge, Lo, Soldaini, Smith, and Hajishirzi}]{groeneveld-etal-2024-olmo}
Dirk Groeneveld, Iz~Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 24 others. 2024.
\newblock \href {https://doi.org/10.18653/v1/2024.acl-long.841} {{OLM}o: Accelerating the science of language models}.
\newblock In \emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pages 15789--15809, Bangkok, Thailand. Association for Computational Linguistics.
\bibitem[{Ho et~al.(2023)Ho, Sharma, Chang, Saxon, Levy, Lu, and Wang}]{ho2023wikiwhy}
Matthew Ho, Aditya Sharma, Justin Chang, Michael Saxon, Sharon Levy, Yujie Lu, and William~Yang Wang. 2023.
\newblock \href {https://openreview.net/forum?id=vaxnu-Utr4l} {Wikiwhy: Answering and explaining cause-and-effect questions}.
\newblock In \emph{The Eleventh International Conference on Learning Representations}.
\bibitem[{Kwon et~al.(2023)Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, and Stoica}]{10.1145/3600006.3613165}
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody~Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023.
\newblock \href {https://doi.org/10.1145/3600006.3613165} {Efficient memory management for large language model serving with pagedattention}.
\newblock In \emph{Proceedings of the 29th Symposium on Operating Systems Principles}, SOSP '23, page 611–626, New York, NY, USA. Association for Computing Machinery.
\bibitem[{Levy et~al.(2023)Levy, Ravfogel, and Goldberg}]{levy-etal-2023-guiding}
Mosh Levy, Shauli Ravfogel, and Yoav Goldberg. 2023.
\newblock \href {https://doi.org/10.18653/v1/2023.findings-emnlp.569} {Guiding {LLM} to fool itself: Automatically manipulating machine reading comprehension shortcut triggers}.
\newblock In \emph{Findings of the Association for Computational Linguistics: EMNLP 2023}, pages 8495--8505, Singapore. Association for Computational Linguistics.
\bibitem[{Li et~al.(2024)Li, Guo, Guerin, and Lin}]{li-etal-2024-open-source}
Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. 2024.
\newblock \href {https://doi.org/10.18653/v1/2024.findings-emnlp.30} {An open-source data contamination report for large language models}.
\newblock In \emph{Findings of the Association for Computational Linguistics: EMNLP 2024}, pages 528--541, Miami, Florida, USA. Association for Computational Linguistics.
\bibitem[{Liu et~al.(2024{\natexlab{a}})Liu, Min, Zettlemoyer, Choi, and Hajishirzi}]{liu2024infinigram}
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. 2024{\natexlab{a}}.
\newblock \href {https://openreview.net/forum?id=u2vAyMeLMm} {Infini-gram: Scaling unbounded n-gram language models to a trillion tokens}.
\newblock In \emph{First Conference on Language Modeling}.
\bibitem[{Liu et~al.(2024{\natexlab{b}})Liu, Qiao, Neiswanger, Wang, Tan, Tao, Li, Wang, Sun, Pangarkar, Fan, Gu, Miller, Zhuang, He, Li, Koto, Tang, Ranjan, Shen, Iriondo, Mu, Hu, Schulze, Nakov, Baldwin, and Xing}]{liu2024llm}
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi~Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, and 8 others. 2024{\natexlab{b}}.
\newblock \href {https://openreview.net/forum?id=QdWhj0QZFw} {{LLM}360: Towards fully transparent open-source {LLM}s}.
\newblock In \emph{First Conference on Language Modeling}.
\bibitem[{Muennighoff et~al.(2025)Muennighoff, Soldaini, Groeneveld, Lo, Morrison, Min, Shi, Walsh, Tafjord, Lambert, Gu, Arora, Bhagia, Schwenk, Wadden, Wettig, Hui, Dettmers, Kiela, Farhadi, Smith, Koh, Singh, and Hajishirzi}]{muennighoff2025olmoeopenmixtureofexpertslanguage}
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, and 5 others. 2025.
\newblock \href {https://arxiv.org/abs/2409.02060} {Olmoe: Open mixture-of-experts language models}.
\newblock \emph{Preprint}, arXiv:2409.02060.
\bibitem[{Ni et~al.(2022)Ni, Hernandez~Abrego, Constant, Ma, Hall, Cer, and Yang}]{ni-etal-2022-sentence}
Jianmo Ni, Gustavo Hernandez~Abrego, Noah Constant, Ji~Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022.
\newblock \href {https://doi.org/10.18653/v1/2022.findings-acl.146} {Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models}.
\newblock In \emph{Findings of the Association for Computational Linguistics: ACL 2022}, pages 1864--1874, Dublin, Ireland. Association for Computational Linguistics.
\bibitem[{OLMo et~al.(2025)OLMo, Walsh, Soldaini, Groeneveld, Lo, Arora, Bhagia, Gu, Huang, Jordan, Lambert, Schwenk, Tafjord, Anderson, Atkinson, Brahman, Clark, Dasigi, Dziri, Guerquin, Ivison, Koh, Liu, Malik, Merrill, Miranda, Morrison, Murray, Nam, Pyatkin, Rangapur, Schmitz, Skjonsberg, Wadden, Wilhelm, Wilson, Zettlemoyer, Farhadi, Smith, and Hajishirzi}]{olmo20252olmo2furious}
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others. 2025.
\newblock \href {https://arxiv.org/abs/2501.00656} {2 olmo 2 furious}.
\newblock \emph{Preprint}, arXiv:2501.00656.
\bibitem[{OpenAI et~al.(2024)OpenAI, Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman, Almeida, Altenschmidt, Altman, Anadkat, Avila, Babuschkin, Balaji, Balcom, Baltescu, Bao, Bavarian, Belgum, Bello, Berdine, Bernadett-Shapiro, Berner, Bogdonoff, Boiko, Boyd, Brakman, Brockman, Brooks, Brundage, Button, Cai, Campbell, Cann, Carey, Carlson, Carmichael, Chan, Chang, Chantzis, Chen, Chen, Chen, Chen, Chen, Chess, Cho, Chu, Chung, Cummings, Currier, Dai, Decareaux, Degry, Deutsch, Deville, Dhar, Dohan, Dowling, Dunning, Ecoffet, Eleti, Eloundou, Farhi, Fedus, Felix, Fishman, Forte, Fulford, Gao, Georges, Gibson, Goel, Gogineni, Goh, Gontijo-Lopes, Gordon, Grafstein, Gray, Greene, Gross, Gu, Guo, Hallacy, Han, Harris, He, Heaton, Heidecke, Hesse, Hickey, Hickey, Hoeschele, Houghton, Hsu, Hu, Hu, Huizinga, Jain, Jain, Jang, Jiang, Jiang, Jin, Jin, Jomoto, Jonn, Jun, Kaftan, Łukasz Kaiser, Kamali, Kanitscheider, Keskar, Khan, Kilpatrick, Kim, Kim, Kim, Kirchner, Kiros, Knight, Kokotajlo, Łukasz Kondraciuk,
Kondrich, Konstantinidis, Kosic, Krueger, Kuo, Lampe, Lan, Lee, Leike, Leung, Levy, Li, Lim, Lin, Lin, Litwin, Lopez, Lowe, Lue, Makanju, Malfacini, Manning, Markov, Markovski, Martin, Mayer, Mayne, McGrew, McKinney, McLeavey, McMillan, McNeil, Medina, Mehta, Menick, Metz, Mishchenko, Mishkin, Monaco, Morikawa, Mossing, Mu, Murati, Murk, Mély, Nair, Nakano, Nayak, Neelakantan, Ngo, Noh, Ouyang, O'Keefe, Pachocki, Paino, Palermo, Pantuliano, Parascandolo, Parish, Parparita, Passos, Pavlov, Peng, Perelman, de~Avila Belbute~Peres, Petrov, de~Oliveira~Pinto, Michael, Pokorny, Pokrass, Pong, Powell, Power, Power, Proehl, Puri, Radford, Rae, Ramesh, Raymond, Real, Rimbach, Ross, Rotsted, Roussez, Ryder, Saltarelli, Sanders, Santurkar, Sastry, Schmidt, Schnurr, Schulman, Selsam, Sheppard, Sherbakov, Shieh, Shoker, Shyam, Sidor, Sigler, Simens, Sitkin, Slama, Sohl, Sokolowsky, Song, Staudacher, Such, Summers, Sutskever, Tang, Tezak, Thompson, Tillet, Tootoonchian, Tseng, Tuggle, Turley, Tworek, Uribe, Vallone,
Vijayvergiya, Voss, Wainwright, Wang, Wang, Wang, Ward, Wei, Weinmann, Welihinda, Welinder, Weng, Weng, Wiethoff, Willner, Winter, Wolrich, Wong, Workman, Wu, Wu, Wu, Xiao, Xu, Yoo, Yu, Yuan, Zaremba, Zellers, Zhang, Zhang, Zhao, Zheng, Zhuang, Zhuk, and Zoph}]{openai2024gpt4technicalreport}
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia~Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024.
\newblock \href {https://arxiv.org/abs/2303.08774} {Gpt-4 technical report}.
\newblock \emph{Preprint}, arXiv:2303.08774.
\bibitem[{Palavalli et~al.(2024)Palavalli, Bertsch, and Gormley}]{palavalli-etal-2024-taxonomy}
Medha Palavalli, Amanda Bertsch, and Matthew Gormley. 2024.
\newblock \href {https://doi.org/10.18653/v1/2024.conda-1.3} {A taxonomy for data contamination in large language models}.
\newblock In \emph{Proceedings of the 1st Workshop on Data Contamination (CONDA)}, pages 22--40, Bangkok, Thailand. Association for Computational Linguistics.
\bibitem[{Rajpurkar et~al.(2018)Rajpurkar, Jia, and Liang}]{rajpurkar-etal-2018-know}
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.
\newblock \href {https://doi.org/10.18653/v1/P18-2124} {Know what you don{'}t know: Unanswerable questions for {SQ}u{AD}}.
\newblock In \emph{Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, pages 784--789, Melbourne, Australia. Association for Computational Linguistics.
\bibitem[{Rajpurkar et~al.(2016)Rajpurkar, Zhang, Lopyrev, and Liang}]{rajpurkar-etal-2016-squad}
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.
\newblock \href {https://doi.org/10.18653/v1/D16-1264} {{SQ}u{AD}: 100,000+ questions for machine comprehension of text}.
\newblock In \emph{Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing}, pages 2383--2392, Austin, Texas. Association for Computational Linguistics.
\bibitem[{Reimers and Gurevych(2019)}]{reimers-gurevych-2019-sentence}
Nils Reimers and Iryna Gurevych. 2019.
\newblock \href {https://doi.org/10.18653/v1/D19-1410} {Sentence-{BERT}: Sentence embeddings using {S}iamese {BERT}-networks}.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)}, pages 3982--3992, Hong Kong, China. Association for Computational Linguistics.
\bibitem[{Schlegel et~al.(2020)Schlegel, Valentino, Freitas, Nenadic, and Batista-Navarro}]{schlegel-etal-2020-framework}
Viktor Schlegel, Marco Valentino, Andre Freitas, Goran Nenadic, and Riza Batista-Navarro. 2020.
\newblock \href {https://aclanthology.org/2020.lrec-1.660/} {A framework for evaluation of machine reading comprehension gold standards}.
\newblock In \emph{Proceedings of the Twelfth Language Resources and Evaluation Conference}, pages 5359--5369, Marseille, France. European Language Resources Association.
\bibitem[{Soboleva et~al.(2023)Soboleva, Al-Khateeb, Myers, Steeves, Hestness, and Dey}]{cerebras2023slimpajama}
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob~R Steeves, Joel Hestness, and Nolan Dey. 2023.
\newblock \href {https://huggingface.co/datasets/cerebras/SlimPajama-627B} {{SlimPajama: A 627B token cleaned and deduplicated version of RedPajama}}.
\newblock \url{https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama}.
\bibitem[{Soldaini et~al.(2024)Soldaini, Kinney, Bhagia, Schwenk, Atkinson, Authur, Bogin, Chandu, Dumas, Elazar, Hofmann, Jha, Kumar, Lucy, Lyu, Lambert, Magnusson, Morrison, Muennighoff, Naik, Nam, Peters, Ravichander, Richardson, Shen, Strubell, Subramani, Tafjord, Walsh, Zettlemoyer, Smith, Hajishirzi, Beltagy, Groeneveld, Dodge, and Lo}]{soldaini-etal-2024-dolma}
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li~Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, and 17 others. 2024.
\newblock \href {https://doi.org/10.18653/v1/2024.acl-long.840} {Dolma: an open corpus of three trillion tokens for language model pretraining research}.
\newblock In \emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pages 15725--15788, Bangkok, Thailand. Association for Computational Linguistics.
\bibitem[{Wang(2022)}]{wang2022modernquestionansweringdatasets}
Zhen Wang. 2022.
\newblock \href {https://arxiv.org/abs/2206.15030} {Modern question answering datasets and benchmarks: A survey}.
\newblock \emph{Preprint}, arXiv:2206.15030.
\bibitem[{Wei et~al.(2022)Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le, and Zhou}]{10.5555/3600270.3602070}
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed~H. Chi, Quoc~V. Le, and Denny Zhou. 2022.
\newblock Chain-of-thought prompting elicits reasoning in large language models.
\newblock In \emph{Proceedings of the 36th International Conference on Neural Information Processing Systems}, NIPS '22, Red Hook, NY, USA. Curran Associates Inc.
\bibitem[{Wilson(1927)}]{Wilson01061927}
Edwin~B. Wilson. 1927.
\newblock \href {https://doi.org/10.1080/01621459.1927.10502953} {Probable inference, the law of succession, and statistical inference}.
\newblock \emph{Journal of the American Statistical Association}, 22(158):209--212.
\bibitem[{Wolf et~al.(2020)Wolf, Debut, Sanh, Chaumond, Delangue, Moi, Cistac, Rault, Louf, Funtowicz, Davison, Shleifer, von Platen, Ma, Jernite, Plu, Xu, Le~Scao, Gugger, Drame, Lhoest, and Rush}]{wolf-etal-2020-transformers}
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le~Scao, Sylvain Gugger, and 3 others. 2020.
\newblock \href {https://doi.org/10.18653/v1/2020.emnlp-demos.6} {Transformers: State-of-the-art natural language processing}.
\newblock In \emph{Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations}, pages 38--45, Online. Association for Computational Linguistics.
\bibitem[{Wu et~al.(2021)Wu, Schlegel, and Batista-Navarro}]{wu-etal-2021-understanding}
Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.eacl-main.311} {Is the understanding of explicit discourse relations required in machine reading comprehension?}
\newblock In \emph{Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume}, pages 3565--3579, Online. Association for Computational Linguistics.
\bibitem[{Wu et~al.(2023)Wu, Schlegel, and Batista-Navarro}]{wu-etal-2023-machine}
Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2023.
\newblock \href {https://doi.org/10.18653/v1/2023.ijcnlp-short.21} {Are machine reading comprehension systems robust to context paraphrasing?}
\newblock In \emph{Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)}, pages 184--196, Nusa Dua, Bali. Association for Computational Linguistics.
\bibitem[{Wu et~al.(2025)Wu, Schlegel, and Batista-Navarro}]{wu2025payattentionrealworld}
Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro. 2025.
\newblock \href {https://arxiv.org/abs/2502.16523} {Pay attention to real world perturbations! natural robustness evaluation in machine reading comprehension}.
\newblock \emph{Preprint}, arXiv:2502.16523.
\bibitem[{Xiao et~al.(2024)Xiao, Liu, Zhang, Muennighoff, Lian, and Nie}]{10.1145/3626772.3657878}
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024.
\newblock \href {https://doi.org/10.1145/3626772.3657878} {C-pack: Packed resources for general chinese embeddings}.
\newblock In \emph{Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval}, SIGIR '24, page 641–649, New York, NY, USA. Association for Computing Machinery.
\bibitem[{Yang et~al.(2025)Yang, Li, Yang, Zhang, Hui, Zheng, Yu, Gao, Huang, Lv, Zheng, Liu, Zhou, Huang, Hu, Ge, Wei, Lin, Tang, Yang, Tu, Zhang, Yang, Yang, Zhou, Zhou, Lin, Dang, Bao, Yang, Yu, Deng, Li, Xue, Li, Zhang, Wang, Zhu, Men, Gao, Liu, Luo, Li, Tang, Yin, Ren, Wang, Zhang, Ren, Fan, Su, Zhang, Zhang, Wan, Liu, Wang, Cui, Zhang, Zhou, and Qiu}]{yang2025qwen3technicalreport}
An~Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo~Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025.
\newblock \href {https://arxiv.org/abs/2505.09388} {Qwen3 technical report}.
\newblock \emph{Preprint}, arXiv:2505.09388.
\bibitem[{Yang et~al.(2017)Yang, Halfaker, Kraut, and Hovy}]{yang-etal-2017-identifying-semantic}
Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2017.
\newblock \href {https://doi.org/10.18653/v1/D17-1213} {Identifying semantic edit intentions from revisions in {W}ikipedia}.
\newblock In \emph{Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing}, pages 2000--2010, Copenhagen, Denmark. Association for Computational Linguistics.
\bibitem[{Yang et~al.(2018)Yang, Qi, Zhang, Bengio, Cohen, Salakhutdinov, and Manning}]{yang-etal-2018-hotpotqa}
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher~D. Manning. 2018.
\newblock \href {https://doi.org/10.18653/v1/D18-1259} {{H}otpot{QA}: A dataset for diverse, explainable multi-hop question answering}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing}, pages 2369--2380, Brussels, Belgium. Association for Computational Linguistics.
\bibitem[{Zhang et~al.(2024)Zhang, Zeng, Wang, and Lu}]{zhang2024tinyllamaopensourcesmalllanguage}
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024.
\newblock \href {https://arxiv.org/abs/2401.02385} {Tinyllama: An open-source small language model}.
\newblock \emph{Preprint}, arXiv:2401.02385.
\bibitem[{Zhao et~al.(2025)Zhao, Zhou, Li, Tang, Wang, Hou, Min, Zhang, Zhang, Dong, Du, Yang, Chen, Chen, Jiang, Ren, Li, Tang, Liu, Liu, Nie, and Wen}]{zhao2025surveylargelanguagemodels}
Wayne~Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 3 others. 2025.
\newblock \href {https://arxiv.org/abs/2303.18223} {A survey of large language models}.
\newblock \emph{Preprint}, arXiv:2303.18223.
\end{thebibliography}