Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties
Extracted source text · arXiv 2505.20875
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
\PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
\usepackage[preprint]{neurips_2025}
\usepackage{xspace}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
% \usepackage{xcolor} % colors
% \usepackage{pgfplotstable}
% \usepackage{pgf}
% \usepackage{colortbl}
% \usepackage{tabularx}
% \usepackage{graphicx}
% \usepackage[table]{xcolor}
\usepackage{subcaption}
\usepackage{titletoc}
\usepackage{enumitem}
\usepackage{booktabs}
\usepackage{adjustbox}
% \usepackage{makecell}
% \usepackage{arydshln}
% \usepackage{amssymb}
% \usepackage{pifont}
% \usepackage{kotex}
% \usepackage{amsmath}
% \usepackage{algorithm}
\usepackage{wrapfig}
\usepackage{multirow}
% \usepackage{algcompatible}
% \usepackage{algorithmic}
% \usepackage{algorithm2e}
% \usepackage{float}
% \usepackage{algorithmicx}
% \usepackage{algcompatible}
% \usepackage{algpseudocode}
\usepackage[defaultcolor=magenta]{changes}
% \usepackage[outline]{contour}
% \usepackage{tabularx}
\newcommand{\eg}{\emph{e}.\emph{g}., }
\newcommand{\ie}{\emph{i}.\emph{e}., }
\newcommand{\fname}{Trans-EnV\xspace}
\newcommand{\jyedit}[1]{\textcolor{purple}{[JY: #1]}}
\newcommand{\shedit}[1]{\textcolor{orange}{[SH: #1]}}
\newcommand{\jeedit}[1]{\textcolor{blue}{[JE: #1]}}
\newcommand{\mycomment}[1]{}
\title{\fname: A Framework for Evaluating the Linguistic Robustness of LLMs Against \\ English Varieties}
% \title{Trans-Env: A Framework for Evaluating the Robustness of LLMs Against English Varieties}
% \title{Trans-Env: A Framework for Evaluating the Linguistic Robustness of LLMs Against \\ English Varieties}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{Jiyoung Lee$^{\dagger}$\thanks{Equal Contribution.} , Seungho Kim$^{\dagger*}$, Jieun Han$^{\dagger}$, Jun-Min Lee$^{\dagger}$\\
\textbf{Kitaek Kim$^{\ddagger}$, Alice Oh$^{\dagger}$, Edward Choi$^{\dagger}$} \\
$^{\dagger}$KAIST, \qquad $^{\ddagger}$Seoul National University \\
\texttt{$^{\dagger}$\{jiyounglee0523, shokim, jieun\_han, ljm565, edwardchoi\}@kaist.ac.kr} \\
\texttt{$^{\dagger}$alice.oh@kaist.edu} \qquad
\texttt{$^{\ddagger}$kitaek@snu.ac.kr}
}
% \author{%
% David S.~Hippocampus\thanks{Use footnote for providing further information
% about author (webpage, alternative address)---\emph{not} for acknowledging
% funding agencies.} \\
% Department of Computer Science\\
% Cranberry-Lemon University\\
% Pittsburgh, PA 15213 \\
% \texttt{hippo@cs.cranberry-lemon.edu} \\
% examples of more authors
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \AND
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
\begin{document}
\maketitle
\begin{abstract}
Large Language Models (LLMs) are predominantly evaluated on Standard American English (SAE), often overlooking the diversity of global English varieties.
This narrow focus may raise fairness concerns as degraded performance on non-standard varieties can lead to unequal benefits for users worldwide.
Therefore, it is critical to extensively evaluate the linguistic robustness of LLMs on multiple non-standard English varieties.
We introduce Trans-EnV, a framework that automatically transforms SAE datasets into multiple English varieties to evaluate the linguistic robustness.
Our framework combines (1) linguistics expert knowledge to curate variety-specific features and transformation guidelines from linguistic literature and corpora, and (2) LLM-based transformations to ensure both linguistic validity and scalability.
Using Trans-EnV, we transform six benchmark datasets into 38 English varieties and evaluate seven state-of-the-art LLMs.
Our results reveal significant performance disparities, with accuracy decreasing by up to 46.3\% on non-standard varieties.
These findings highlight the importance of comprehensive linguistic robustness evaluation across diverse English varieties.
Each construction of Trans-EnV was validated through rigorous statistical testing and consultation with a researcher in the field of second language acquisition, ensuring its linguistic validity.
Our code and datasets are publicly available.\footnote{Code: \url{https://github.com/jiyounglee-0523/TransEnV}
Dataset:\url{https://huggingface.co/collections/jiyounglee0523/transenv-681eadb3c0c8cf363b363fb1}}
\end{abstract}
\section{Introduction}
Large Language Models (LLMs) \citep{achiam2023gpt, team2024gemini, guo2025deepseek} have shown impressive performance, even surpassing humans on several tasks \citep{luo2025large, jones2024ai, gor2024great}.
However, most evaluation benchmarks are written in Standard American English (SAE), overlooking the rich diversity of English varieties.
% However, there exist many \textit{English varieties}, further refer to as \textit{variety} for simplicity, which include non-standard forms such as regional dialects or English written by non-native speakers \citep{chambers1998dialectology}.
English is spoken in a wide range of \textit{varieties}, including regional dialects and forms used by non-native speakers \citep{chambers1998dialectology}.
% Performance differences in LLMs by varieties may provide unequal benefit to global users.
% Prior works have shown that LLMs are weak at English varieties \citep{zhou2020challenges, blasi2021systematic}.
% Such performance gap may provide unequal benefit to global users.
This narrow linguistic focus may raise fairness concerns, as LLMs tend to underperform on non-standard varieties \citep{zhou2020challenges, blasi2021systematic}, potentially leading to unequal benefits for global users.
Therefore, assessing LLM performance in multiple English varieties is essential to ensure its fairness.
% and inclusivity in language technology.
While several datasets have been introduced to evaluate the robustness of LLMs to varieties \citep{blodgett-etal-2018-twitter, faisal2021sd, ruder2023xtreme, khanuja2020gluecos}, they remain limited in size, variety coverage, and task diversity, making them inadequate for comprehensive evaluation.
To rigorously assess the linguistic robustness of LLMs, it is necessary to evaluate on existing benchmark datasets across a diverse range of English varieties.
% To rigorously assess LLM robustness, it is essential to test models on widely used benchmark datasets that are transformed into multiple varieties along with SAE.
This, in turn, requires a framework capable of automatically converting SAE benchmarks into any desired target variety.
Although prior studies have proposed such transformations \citep{lin2024one, reusens2024native, gupta2024aavenue, ziems2022value, ziems2022multi}, these approaches often suffer from scalability challenges, lack of expert knowledge, or failure to adequately capture linguistic diversity.
To this end, we introduce \textbf{\fname}, a framework that automatically transforms SAE datasets into a desired target variety.
We focus on two widely studied types of variety: regional dialect and English as a Second Language (ESL) English \citep{jenkins2015repositioning, bieswanger2008varieties, salih2021future}.
Regional dialects, henceforth called as \textit{dialects}, refer to geographically localized varieties of English, such as Scottish or Irish English.
ESL English refers to language produced by speakers whose first language is not English.
We begin by collecting linguistic features from expert-curated resources and large-scale ESL corpora to ensure rigorous and accurate information.
Then, for each feature, we create transformation guidelines that specify the steps to apply the feature to an SAE sentence.
Then we use an LLM to transform SAE sentences by following the guidelines.
An overview of \fname is provided in Figure~\ref{fig:overview}.
LLMs are used both for the guideline generation and the sentence transformation, making \fname both labor-efficient and scalable across datasets.
To ensure linguistic validity, we consulted a researcher in the field of second language acquisition throughout the entire development, aligning our methodology with established linguistic theory and practice.
% We use LLM in generating transformation guideline and transformation, our framework is labor-effieict and scalable in any datasets.
% Also, to ensure the linguistic validity of our framework, we consulted applied linguistic at every step of development to align our framework with established linguistic theory and practice.
% Our framework is grounded on expert-curated linguistic resources and corpora of ESL learner data, which provides rigorous and accurate linguistic features for varieties.
% Then when transform, we leverage the LLM power of generating diverse and natural sentence variations.
% This allows for high-quality rewriting that reflects both grammatical correctness and contextual appropriateness across varieties.
% All components of the framework are grounded in literature from linguistic that are organized from XX publication or extracted from XX corpora.
% Throughout the development process, we consulted linguistic experts to ensure our methodology aligns with accepted linguistic theory and practice.
We translate six widely used benchmark datasets into 38 varieties consisting of 18 dialects and 20 ESL English varieties.
We evaluate seven state-of-the-art LLMs on the transformed benchmarks and observe that model performance generally degrades across most varieties, with particularly pronounced drops in ESL English (12.5\% and 46.3\% performance drop at maximum for each dialect and ESL English).
In addition, we find that linguistic robustness is notably weaker on tasks that require reasoning.
Models specialized in reasoning tend to be more robust than others, suggesting that strong reasoning capabilities may contribute to improved robustness.
Our contributions are summarized as follows:
\begin{itemize}[leftmargin=3.5mm, itemsep=1mm, parsep=0pt]
\item We introduce \fname, a framework that automatically transforms SAE-written datasets into specified target English varieties. By leveraging an LLM for transformation, our framework is both labor-efficient and scalable across multiple datasets.
\item \fname is grounded in expert-curated linguistic resources, and its construction was validated through rigorous statistical testing and consultation with a researcher in the field of second language acquisition, ensuring its linguistic validity.
\item We conduct extensive experiments by transforming six widely used benchmark datasets into 38 varieties and evaluating seven LLMs. The results show that LLMs exhibit notable weaknesses in handling non-standard English varieties, particularly in ESL English.
\end{itemize}
\section{Related Work}
\paragraph{English Variety Disparity in LLMs.}
% English Variety Disparity가 llm에서 중요한 이유
English variety refers to various forms of English used across different regions, communities, or learner groups, including both Standard American English and non-standard forms.
The non-standard forms include regional dialects (\eg Scottish English) and ESL learner usages (\eg Arabic English) \citep{migdadi2020global, hamid2013second}.
% Recent works \citep{joshi2025natural, srirag2024evaluating, deas2023evaluation, won2025advancing, liang2023holisticevaluationlanguagemodels} highlight the LLM performance gap between SAE and other varieties of English.
Despite such diversity in English, LLMs do not perform well particularly in non-standard English in the form of dialects \citep{joshi2025natural, srirag2024evaluating, deas2023evaluation, won2025advancing, liang2023holisticevaluationlanguagemodels} and ESL learners' usages \citep{mizumoto2024identifying, liang2023gpt}.
For instance, in non-SAE settings, LLMs underperform in tasks such as language generation and understanding \citep{deas2023evaluation, gupta2024aavenue, srirag2024evaluating, mizumoto2024identifying}, reasoning capability \citep{lin2024one, zhou2025disparities} and instruction following \citep{fleisig2024linguistic}.
In addition, LLMs respond in a more stereotyping, demeaning, unnatural, and condescending manner to under-represented varieties \citep{fleisig2024linguistic, hofmann2024ai, liang2023gpt}.
\citet{zhou2020challenges} showed that LLMs have shown persistent biases against non-SAE particularly in tasks involving toxicity.
% \paragraph{Prior Methods on English Variety Dataset Construction.}
\paragraph{English Variety Dataset.}
Previous work on transforming datasets for varieties typically uses one of three methods: (i) manual curation \citep{lin2024one, reusens2024native}, (ii) LLM generation \citep{gupta2024aavenue}, and (iii) rule-based transformation \citep{ziems2022value, ziems2022multi}.
The first approach involves humans manually creating the entire dataset.
While this method can produce high-quality results, recruiting qualified human annotators is resource-intensive, and scaling to multiple English varieties and datasets is challenging.
%This approach is restricted to generating small dataset and covers less varieties.
% For instance, \citet{lin2024one} handles only one dialect, employing 13 human workers to construct a dataset of 1.2K instances.
% Similarly, \citet{reusens2024native} targeted non-native English, by employing an average of 2.4 human workers per different L1 background.
The second approach relies entirely on LLMs to generate varieties.
While this method is scalable and convenient, several studies have highlighted the limitations of LLMs in accurately reproducing under-represented varieties of English\citep{mizumoto2024identifying, uchida2025generative, barayan2024analysing, farajidizaji2023possible}, underscoring the risk of relying solely on LLMs.
The third approach employs deterministic, rule-based transformations to transform SAE sentences into targeted varieties.
However, this method demands substantial human effort to manually craft transformation rules for each linguistic feature.
Moreover, it falls short in capturing the full spectrum of linguistic variation, including lexical choices and pragmatic nuances.
As a result, these transformations tend to be context-specific and challenging to generalize, limiting their applicability across different domains and language varieties \citep{srirag2024besstie, gupta2024aavenue}.
In contrast, our approach integrates expert-curated resources with the linguistic capabilities of LLMs to construct a robust framework that captures diverse and accurate language expressions across varieties, while ensuring both linguistic validity and scalability.
\section{\fname: A Framework for Transforming SAE into Varieties}\
Constructing \fname consists of three main steps: (i) data collection, (ii) generation of transformation guidelines, and (iii) transforming SAE sentences into the target English variety.
In the data collection phase, we compile a set of varieties $\mathcal{V} = \{v_1, \ldots, v_n\}$, where $n$ denotes the total number of varieties.
Each variety is associated with a set of linguistic features, which we refer to as \textit{features} for brevity.
Let $\mathcal{L} = \{l_1, \ldots, l_m\}$ denote the complete set of unique features across all varieties where $m$ is the total number of features, and let $\mathcal{L}_{v_i} = \{l_1^{(i)}, \ldots, l_k^{(i)}\} \subset \mathcal{L}$ represent the subset of features specific to variety $v_i$, where $k$ is the number of associated features.
During the transformation guideline generation phase, we construct a guideline $g_j$ for each feature $l_j$ specifying the operations required to apply $l_j$ to a given sentence.
The set of all guidelines is denoted by $\mathcal{G} = \{g_1, \ldots, g_m\}$, with $\mathcal{G}_{v_i} = \{g_1^{(i)}, \ldots, g_k^{(i)}\} \subset \mathcal{G}$ denoting the subset of guidelines corresponding to $v_i$, following the notation above.
In the final transformation stage, we convert SAE sentences into the target variety $v_i$ by sequentially applying each feature in $\mathcal{L}_{v_i}$ following the corresponding guidelines in $\mathcal{G}_{v_i}$.
% \jyedit{dialect와 달리 ESL은 여러가지 영향을 받기 때문에 더 fine-grained되도록 만들었다.}
% Since ESL English is influenced by both the learner's English proficiency and their native language (L1) \citep{luk2009acquisition, murakami2016l1, sung2024influence}, our framework incorporates both factors.
% To further account for the vocabulary limitations commonly observed among English learners, we include an additional step at the beginning of the transformation, which replaces advanced vocabulary with simpler synonyms or phrases that more closely align with the target proficiency level.
% An overview of \fname is illustrated in Figure~\ref{fig:overview}.
Since ESL English is influenced by both the learner's proficiency and native language (L1) \citep{luk2009acquisition, murakami2016l1, sung2024influence}, our framework considers both factors.
To address the vocabulary limitations common among English learners, we add an initial step that simplifies advanced words into more accessible synonyms or phrases.
Figure~\ref{fig:overview} provides an overview of \fname.
% We provide detailed explanations of how $\mathcal{V}$ is selected, how $\mathcal{L}$ is compiled, and how $\mathcal{G}$ is constructed in the following sections.
% Note that \fname is grounded in expert linguistic knowledge and follows a rigorous development process.
% \jyedit{우리 framework가 rigorous 하다는 것을 추가 해야 하나. LLM을 사용했다는 것도 여기에 작성해야 하나,semantic judge가 있다는 것도 추가}
% \subsection{Task Formulation}
% We first collect a set of varieties defined as $\mathcal{V} = \{v_1, \ldots, v_n\}$, where $n$ is the total number of varieties and collect associated linguistic features, further refer to as \textit{features} for simplicity.
% Let $\mathcal{L} = \{l_1, \ldots, l_m\}$ be the complete set of all unique linguistic features from $\mathcal{V}$ and $\mathcal{L}_{v_i} = \{l_1^{(i)}, \ldots, l_k^{(i)}\}\subset \mathcal{L}$ be the subset of features that characterizes $v_i$.
% We then construct a set of transformation guidelines $\mathcal{G} = \{g_1, \ldots, g_m\}$, where each guideline $g_j$ defines a series of transformation steps for applying the feature $l_j$ to a given sentence.
% Here, similar to above we will denote $\mathcal{G}_{v_i} = \{g_1^{(i)}, \ldots, g_k^{(i)}\}\subset \mathcal{G}$ as the associated guideline to $v_i$.
% \quad
% We define $\mathcal{V} = \{v_1, \ldots, v_n\}$ as the set of unique varieties, including both dialects and ESL English, and $\mathcal{L} = \{l_1, \ldots, l_m\}$ as the complete set of unique linguistic features.
% % observed across all varieties.
% For simplicity, we refer to linguistic features as \textit{features} hereafter.
% Here, $n$ represents the total number of varieties, and $m$ is the total number of unique features.
% We then construct a set of transformation guidelines $\mathcal{G} = \{g_1, \ldots, g_m\}$, where each guideline $g_j$ defines a series of transformation steps for applying the feature $l_j$ to a given sentence.
% For each variety $v_i$, there is an associated subset of features $\mathcal{L}_{v_i} = \{l_1^{(i)}, \ldots, l_k^{(i)}\}\subset \mathcal{L}$ that characterizes $v_i$ and the corresponding subset of guidelines $\mathcal{G}_{v_i} = \{g_1^{(i)}, \ldots, g_k^{(i)}\} \subset \mathcal{G} $ , where $k$ is the number of features specific to $v_i$.
% Dialect features modify sentence structure in ways that reflect nonstandard dialects of English, whereas ESL features often introduce grammatical errors that result in ungrammatical or broken English.
% % Generally, dialect features are adding features, but ESL features are grammar errors that break the original sentences into broken English. \jyedit{설명 잘 하기}
% In addition to linguistic features, English learners are also limited by their vocabulary. To account for this, we include an additional step where advanced vocabulary is replaced with synonyms or phrases that better match the target level of English proficiency.
% Since vocabulary is another critical step to consider in ESL English, we have additional before step that replaces hard vocabularies into easier ones to the targeted variety.
% Simulating ESL speaker English is harder than dialect since it has to consider vocabulary choice, english proficiency, and its mother language, as these factors all influence the ESL speaker english.
% Here, we use standard Common European Framework of Reference for Languages (CEFR) \citep{council2001common} as indicator of english flunecy.
% Although original CEFR levels are categorized in six, we use only three higher cateogires (A, B, C).
% First we replace hard vocabularies in the sentence into adequate vocabularies in the targeted CEFR level.
% Th
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figures/Main_Figure.pdf}
\caption{\label{fig:overview}Overview of \fname{}.
(a) \textit{Data Collection} and \textit{Transformation Guideline Generation}: We gather English varieties and their associated linguistic features from linguistic literature and large-scale corpora. For each feature, we construct a transformation guideline that defines the procedure for applying the feature.
(b) \textit{Transformation into Target Variety}: Given a SAE sentence $s$ and a target variety $v_i$, LLM transforms $s$ by sequentially applying the features of $v_i$ by following guidelines.
}
\end{figure}
\subsection{Data Collection}
\paragraph{Dialect.}
We utilize the Electronic World Atlas of Varieties of English\footnote{\url{https://ewave-atlas.org/}} (eWAVE) \citep{ewave}, a comprehensive database that documents 235 linguistic features across 77 varieties of English.
This dataset was compiled by 84 professional linguists and is grounded in 175 peer-reviewed publications.
Each variety in eWAVE is annotated for every feature using a four-level scale indicating the degree of presence.
% : (i) pervasive or obligatory, (ii) neither pervasive nor extremely rare, (iii) exists but is extremely rare, and (iv) attested absence.
Among the 77 varieties, some are English-based creoles and pidgins, which, despite sharing vocabulary and structural elements with English, have diverged significantly and are considered distinct languages \citep{vicente2007english, bakker2024pidgins}.
To systematically distinguish English dialects from these non-English varieties, we apply K-Nearest Neighbors (KNN) clustering on the 77 varieties each represented by 235 linguistic features, treating them as input embeddings.
We then select clusters that contain widely recognized English dialects, such as Australian English and Scottish English.
% From this, we use the dialects from the the group composed of standard or widely recognized English dialects such as Australian English or Scottish English.
This process yields 18 dialects.
% The linguistic researcher verified that this subset is appropriate for our research scope, with the remaining varieties considered outside the intended focus.
The appropriateness of this subset for our research scope was verified by the specialist in second language acquisition, with the remaining varieties considered outside the intended focus.
% As a result, we chose 18 dialects for our study.
% The 18 dialects are confirmed by the applied linguistic.
For each dialect $v_i$, we define its features $\mathcal{L}_{v_i}$ as those annotated with the highest level of presence in eWAVE.
Additional details on eWAVE, the clustering procedure, and the selected dialects are provided in Appendix~\ref{app:dataset_construction}.
\paragraph{ESL English-Proficiency.}
We adopt the Common European Framework of Reference for Languages (CEFR) \citep{council2001common} as our indicator of English proficiency.
CEFR is a widely used standard that defines six proficiency levels (A1–C2).
For our purposes, we focus on the three higher categories—A (Basic), B (Intermediate), and C (Proficient)—as finer-grained distinctions often lead to overfitting in AI applications \citep{gaillat2022predicting, law2024application}.
Given that level C closely approximates native English proficiency, which LLMs are generally capable of, we focus on levels A and B.
We collect features from the English Grammar Profile (EGP) \citep{o2017english}, which catalogs 1,222 features mapped to CEFR levels by systematically analyzing a large corpus containing millions of texts written by ESL learners from diverse L1 backgrounds and proficiency levels \citep{profile2012english}.
Each feature is presented in the form of a "can-do" descriptor (\eg \textit{can use adjective phrases to modify nouns} at B2), which reflects the general grammatical abilities expected at each CEFR level.
To simulate a target CEFR level, we exclude "can-do" features associated with higher levels--- for example, to transform a sentence into level A, we identify and remove all B-level and C-level "can-do" features present in that sentence.
Thus, features defining each CEFR level are derived from those of the higher levels as removal targets.
% Thus, the transformation guideline for level $v_i$ includes features $\mathcal{L}_{v_i}$ drawn from levels above $v_i$.
% For vocabulary, we use the Oxford 5000 word list \citep{moore2020oxford}, developed by Oxford University Press, which maps each word to a specific CEFR level.
% The list is widely used in both NLP \citep{imperial-tayyar-madabushi-2024-specialex, he-yiu-2022-controllable} and language education \citep{dirham2023review}.
\paragraph{ESL English-L1.}
Existing linguistic studies on the influence of L1 typically examine fewer than six morphemes per language, resulting in a limited set of L1-specific features across languages.
To address this gap, we empirically derive L1-specific features through controlled experiments, following established methodologies in linguistics \citep{jarvis2000methodological, murakami2016l1}.
To disentangle L1-specific features from those of general second-language acquisition, features should be extracted from essays by learners within the same proficiency level, ensuring that observed features only reflect L1 influence \citep{jarvis2000methodological}.
Therefore, we need learner essays annotated with both L1 and CEFR level to conduct the controlled analysis.
We use three open-source ESL learner corpora: CLC-FCE \citep{yannakoudakis2011new}, ICLE \citep{granger1993international}, and EFCamDat \citep{geertzen2013automatic}, all of which contain English essays written by learners, annotated with their native language.
Among these, only EFCamDat includes CEFR proficiency annotations.
We use GPT-4o mini\footnote{Model version: \texttt{gpt-4o-mini-2024-07-18}. The model achieves 77.3\% accuracy on CEFR-SP \citep{arase2022cefr} for three-level CEFR classification.}
to predict CEFR levels as pseudo-proficiency indicators for CLC-FCE and ICLE.
From the corpora, we select 10 L1s with sufficient data: Arabic, Chinese-Mandarin, French, German, Italian, Japanese, Portuguese, Russian, Spanish, and Turkish.
We apply an automatic grammar checker\footnote{\url{https://pypi.org/project/language-tool-python/}} to each sentence to identify grammatical features, which are then grouped into higher-level categories.
For each L1 and CEFR level, we compute feature frequencies and conduct statistical t-tests to identify features significantly associated with specific L1s and CEFR level ($p < 0.05$).
On average, 10 distinct features were identified for each L1 at each CEFR level.
We confirmed that the extracted features align with prior linguistic findings \citep{murakami2016l1}, and their validity was verified by native speakers of Spanish and French as well as a specialist in second language acquisition.
For ESL English, $\mathcal{L}_{v_i}$ combines feature from the target CEFR level the corresponding L1.
For example, for variety defined by CEFR level A and Arabic as the L1, the feature set includes both CEFR level A features and Arabic-specific features at CEFR level A.
We verified that features from CEFR and L1s do not conflict.
Full experimental details and summaries of extracted features are provided in Appendix~\ref{app:features_of_native_language}.
\subsection{Transformation Guideline Generation}
As LLMs often fail to apply features to SAE sentences when prompted with feature names alone, it is essential to provide explicit, well-defined transformation guidelines and enforce step-by-step execution.
Therefore, we generate a transformation guideline $g_j$ for each feature $l_j$, which outlines a detailed, step-by-step procedure for applying $l_j$ to a given sentence.
Each guideline consists of two steps: \textit{Qualification} and \textit{Application}.
The \textit{Qualification} step determines whether the feature is applicable to the sentence.
For instance, for the feature \textit{`She/her used for inanimate references'}, this step verifies the presence of an inanimate referent and a pronoun that refers to it in the sentence.
The \textit{Application} step provides detailed instructions to implement the transformation, \eg identifying the inanimate referent and replacing its corresponding pronouns with \textit{she} or \textit{her}.
We use GPT-4\footnote{Model version: \texttt{gpt-4-0613}} to generate these guidelines via one-shot prompting.
All generated guidelines were reviewed by the researcher in the second language acquisition and were deemed appropriate for use.
Further details on generation configuration, prompts used, and examples are provided in Appendix~\ref{app:guideline_generation}.
\subsection{Transforming into English Varieties}
\label{sec:transform}
% \input{algorithm/transform_process}
Given an SAE sentence $s$ and a target variety $v_i$, we transform $s$ using the associated guideline set $\mathcal{G}_{v_i}$.
We utilize a feature transformer model $T$, which applies each guideline to $s$, and a semantic checker model $S$, which verifies whether the transformed sentence preserves the original meaning.
Both $T$ and $S$ can be any AI model capable of interpreting and executing the provided guidelines.
For ESL English varieties, vocabulary replacement is a crucial step due to the limited lexical range of English learners.
To ensure that the vocabulary aligns with the target CEFR levels, we compile a vocabulary-to-CEFR mapping from the Oxford 5000\footnote{\url{https://www.oxfordlearnersdictionaries.com/wordlists/oxford3000-5000}} and supplementary word lists,\footnote{\url{https://www.oxfordlearnersdictionaries.com/topic/}} resulting in 23,411 labeled words.
For words that are not covered by the list, we use GPT-4o to provide their CEFR levels.
% However, this did not fully cover the vocabulary across our six datasets, leaving 14,731 unique words without CEFR labels.
% To address this, we used GPT-4o to estimate CEFR levels for the remaining words.
Simply replacing all higher level words with target level words is not an optimal transformation strategy.
Analyzing the CEFR-labeled English text dataset \footnote{\url{https://www.kaggle.com/datasets/amontgomerie/cefr-levelled-english-texts/data}} showed that texts labeled as CEFR A and B levels contained a small proportion of higher-level words, up to 14.3\% and 9.6\% at the 90th percentile, respectively.
Based on this observation, we allow up to 15\% of higher-level vocabulary in transformed texts to reflect realistic ESL proficiency.
We replace high-level words using $T$ as the first step of transformation.
In cases where the vocabulary could not be sufficiently simplified (\eg{} complex questions from the MMLU professional law), we exclude those samples from the final dataset, as they were considered too difficult for ESL learners at the target level.
Table \ref{tab:vocabulary_ratio} in Appendix~\ref{app:vocab_replace} presents the final dataset sizes and the ratio of successful transformations.
Further details and examples of vocabulary transformation are provided in Appendix~\ref{app:vocab_replace}.
% For ESL English, transformation starts with replacing hard vocabularies with easier ones. \jyedit{승호님, 여기에 vocab 치환하는 거 적어주시면 됩니다.} If there are any vocabs that are not included in Oxford 5000, we used gpt-4o to generate psuedo CEFR vocab level\footnote{It showed decent performance.}.
% We believe that CEFR A level speaker may know some higher level vocabs.
% therefore, we allowed certain percentage of higher level words.
% As we analysised CEFR Levelled English Texts\footnote{\url{https://www.kaggle.com/datasets/amontgomerie/cefr-levelled-english-texts/data}}, approximately CEFR A level sentences have 0.15\% of CEFR vocab B or C words and CEFR B level sentences have 8\% of CEFR level C words.
% Therefore, when replacing vocabs, we allowed 15\% of higher level vocabs.
% We use $T$ to replace the vocabs by prompting.
% For those samples where the vocabularies cannot be replaced to the eaiser one, such as those questions from professional law or medical, we removed those samples from the CEFR level dataset as those questions cannot be asked by the targeted CEFR level people.
% As we proceed this process, the final dataest size and the 변환율 of each dataset is shown in Table \jyedit{XX}.
Next, for both dialect and ESL English, we randomly shuffle features in $\mathcal{L}_{v_i}$ and apply them sequentially.
When applying a feature $l_j^{(i)}$, $T$ determines whether the \textit{Qualification} condition specified in $g_j^{(i)}$ is satisfied.
If the condition is met, $T$ performs the transformation following the \textit{Application} step, producing a transformed sentence.
The transformed sentence is then passed to the next feature $l_{(j+1)}^{(i)}$.
If the condition is not satisfied, the feature is skipped.
After each transformation, $S$ verifies whether the original and transformed sentences preserve the same meaning.
Only transformations that pass this semantic check are retained and used in the subsequent steps.
Full prompts used in the transformation process and examples of the transformation procedure are provided in Appendix~\ref{app:prompts}.
\subsection{Analysis of \fname}
\label{sec:analysis}
% \jyedit{하나의 섹션으로 따로 둬야 하나 고민}
We applied \fname to six benchmark QA datasets, three knowledge-based datasets: MMLU \citep{hendrycks2020measuring}, ARC \citep{clark2018think}, TruthfulQA \citep{lin2021truthfulqa}, and three reasoning-based datasets: GSM8K \citep{cobbe2021training}, HellaSwag \citep{zellers2019hellaswag}, WinoGrande \citep{sakaguchi2021winogrande}.
We used Gemma-2-27B-Instruct \citep{team2024gemma} as the feature transformer model $T$, and LLaMA-3.3-70B-Instruct \citep{grattafiori2024llama} as the semantic checker model $S$.
Each dataset is transformed into total of 38 varieties---18 dialects and 20 ESL English.
\paragraph{Transformation Coverage and Intensity.}
Table~\ref{tab:transformation_ratio} reports the average number of features applied per sample and the overall proportion of transformed samples across datasets.
On average, around two features were applied per sample.
Given that most samples are relatively short, consisting of one or two sentences, this level of transformation is considered reasonable.
In most cases, over 80\% of the samples were modified and ESL English samples exhibited a higher rate of transformation than dialect.
This may be attributed to ESL English features being more closely tied to everyday usage than those of dialects.
We found that untransformed samples are significantly short or simple in structure, such as \textit{``Let p = (1, 2, 5, 4)(2, 3) in $S_5$. Find the index of <p> in $S_5$.''} or \textit{``What is `coring'?''}, which left little room for transformation.
We provide examples of transformed sentences in Table~\ref{tab:transformation_sample}.
Detailed statistics for each variety within each dataset are provided in Appendix~\ref{app:transformation_ratio}.
\input{tables/transformation_analysis}
\input{tables/example_transformation_sample}
% \paragraph{Comparison with Baseline.}
% - 우리 framework랑 기존 많이 사용하는 basleine인 LLM을 사용해서 변환하는 것과 rule-based trnasformation인 Multi-value를 비교한다.
\paragraph{Human Evaluation.}
\begin{wraptable}{R}{0.5\textwidth}
\centering
\vspace{-0.2cm}
\input{tables/transformer_model}
\vspace{-0.2cm}
\end{wraptable}
We evaluate the effectiveness of our framework using six different models as Feature Transformer ($T$): LLaMA-3.1-8B-Instruct \citep{dubey2024llama3herd}, Gemma-2-27B-Instruct \citep{team2024gemma}, Gemma-3-27B-Instruct \citep{gemma_2025}, Qwen2.5-32B-Instruct \citep{qwen2.5}, GPT-4 \citep{openai2023gpt4}, and GPT-4.1-mini \citep{openai2025gpt41mini}.\footnote{Model versions: \texttt{gpt-4-turbo-2024-04-09}, \texttt{gpt-4.1-mini-2025-04-14}}
For human evaluation, we recruited three annotators with academic-level English proficiency who were fully informed about the linguistic features used in our study.
Each model generated 25 transformed outputs, resulting in 150 samples per annotator.
Outputs were evaluated on two criteria: (Q1) whether the model correctly followed the Qualification and Application steps specified in the guidelines, and (Q2) whether the transformed sentence preserved the original meaning.
Table~\ref{tab:human_eval} presents the results.
A sample is considered valid if it received majority approval from the annotators.
All models, except LLaMA-3.1-8B-Instruct, achieved over 90\% validity, indicating that our framework is broadly compatible with high-capacity LLMs.
We also evaluated the semantic checker model $S$ using the outputs from LLaMA-3.3-70B-Instruct \citep{grattafiori2024llama}.
Specifically, we randomly sampled 200 outputs that $S$ rejected and 200 that it accepted.
Human annotators then labeled whether each output preserved the original semantics.
Using these human annotations as gold labels, $S$ achieved a precision of 83.6\%, recall of 97.0\%, and F1 score of 89.8\%.
These results indicate that $S$ performs reliably in distinguishing meaning-preserving transformations.
Additional details on the human evaluation procedure are provided in Appendix~\ref{app:human_evaluation}.
% We also evaluated semantic checker $S$ for Llama-3.3-70B-Instruct.
% We randomly sampled 200 outputs where the $S$ reject and 200 samples which $S$ accepted.
% Then, for the output, human annotators annotate whether the semantics are preserved or not.
% As a result, treating the human label as gold label, we got precision of 83.6\%, recall of 0.97\%, and F1 of 89.8\%.
% From this results, we have checked that the $S$ works properly.
% \paragraph{Transformation Validity.}
% gemma-2랑 llama-70b를 사용하는게 맞는 것인지
% llama-70b는 f1 90\% 나옴
\input{tables/main_results_dialects}
\input{tables/main_results_esl}
\section{Experiments}
% \subsection{Experiment Setting}
% \paragraph{Datasets.}
% dataets: three Knowledge QA dataset (MMLU, ARC, TruthfulQA) and three reasoning QA datasets (GSM8K, HellaSwag, WinoGrande), QA 데이터셋만 다룬다.
% zero-shot으로 진행
% 실제 prompt, generational config는 appendix에 넣을 예정
% In this study, we transform six benchmark QA datasets—MMLU, ARC, TruthfulQA, GSM8K, HellaSwag, and WinoGrande—into 38 English varieties using the Trans-EnV framework.
% The transformation is performed in a zero-shot setting, without providing any in-context examples of the target varieties.
% Among them, MMLU, ARC, and TruthfulQA are categorized as knowledge-based QA datasets, as they require responses grounded in external factual knowledge.
% In contrast, GSM8K, HellaSwag, and WinoGrande are classified as reasoning-based QA datasets, which primarily test logical reasoning and contextual understanding.
% MMLU evaluates domain knowledge and comprehension with questions spanning 57 diverse subject areas.
% ARC assesses scientific reasoning and common science knowledge at the elementary and middle school levels.
% TruthfulQA measures the factuality and reliability of model outputs in response to questions prone to misinformation.
% GSM8K tests arithmetic and mathematical reasoning skills using grade-school level word problems.
% HellaSwag evaluates commonsense reasoning via next-sentence selection in everyday scenarios.
% WinoGrande assesses coreference resolution and contextual reasoning through pronoun disambiguation tasks.
% 6개 모델
% reasoning pair가 있음 (deepseek llama <-> llama, gpt-4o <-> o4, gemini-2.0 <-> gemini 2.5)
% 각 모델 버전 footnote에 적어두기
We evaluated the transformed datasets on seven state-of-the-art models: Qwen2.5-72B-Instruct \citep{qwen2.5}, DeepSeek-R1-Distill-Llama-70B \citep{deepseekai2025deepseekr1incentivizingreasoningcapability}, LLaMA-3.3-70B-Instruct \citep{grattafiori2024llama}, Gemini 2.0 Flash \citep{google_gemini2flash_2025}, Gemini 2.5 Pro \citep{google_gemini25pro_2025}, GPT-4o-mini \citep{openai_gpt4omini_2024}, and o4-mini \citep{openai_o4mini_2025}.\footnote{Model versions: \texttt{gemini-2.5-pro-exp-03-25}, \texttt{gpt-4o-mini-2024-07-18}, \texttt{o4-mini-2025-04-16}}
% This evaluation provides a comprehensive analysis of how varieties influence model performances across six QA benchmarks.
% Performance is measured using standard metrics specific to each dataset.
% , allowing for direct comparison between varieties and models.
\subsection{Experiment Results}
Tables~\ref{tab:dialect_main} and \ref{tab:esl_main} report model performances on dialect and ESL English varieties, respectively.
% Due to the space constraints, we report results from four datasets and include the full results in Appendix~\ref{app:full_results}.
"Orig." denotes the original SAE dataset, and "Mean" represents the average performance across all dialects or varieties of a CEFR level, excluding the SAE dataset.
Values in bold are those that performed better than the original.
Cell color reflects performance deviation across varieties: blue indicates higher performance, and orange indicates lower performance.
Overall, models tend to perform worse on non-SAE varieties, with maximum performance drops of 12.5\% for dialects and 46.3\% for ESL English.
Models with strong reasoning capabilities, such as DeepSeek-R1-70B or o4-mini, exhibit greater robustness, suggesting that reasoning ability may support robustness in varieties.
Also, performance degradation is more pronounced in reasoning-based QA (4.7\% for dialects and 22.6\% for ESL English) compared to knowledge-based QA datasets (1.3\% for dialects and 10.9\% for ESL English) on average, implying that reasoning tasks are more sensitive to linguistic variations.
Additionally, the overall performance drop is bigger for ESL English than for dialects.
In 4 out of the 6 datasets, CEFR A varieties yield lower mean scores than those of CEFR B, indicating that simpler or a higher deviance SAE pose greater challenges for LLMs.
Certain English varieties exhibit consistent relative performance patterns across datasets and models.
Among dialects, Newfoundland English (NFE), Welsh English (WeE), and Irish English (IrE) tend to yield lower scores, whereas Australian English (AuE), Southeast English (SE-Eng), and Southwest English (SW-Eng) show relatively stronger performance.
For ESL varieties, Arabic (ar) and Turkish (tr) underperform, while French (fr) and Italian (it) achieve higher scores.
We attribute these trends partly to data availability.
Dialects with lower performance tend to have fewer native speakers (on the order of millions), while better-performing dialects are spoken in regions where English is an official language.
Similarly, Arabic and Turkish are under-represented in two multilingual pretraining corpora (1.66\% and 1.93\% in mC4 \citep{xue2021mt5massivelymultilingualpretrained}; 2.0\% and 1.7\% in OSCAR \citep{OrtizSuarezSagotRomary2019}, respectively) while French and Italian are better represented (2.89\% and 2.43\% in mC4; 8.9\% and 5.3\% in OSCAR, respectively).
These observations suggest that the amount of pretraining data available for each variety is a critical factor influencing the LLM robustness across varieties.
\subsection{Correlation between Linguistic Distance with Performance Degradation}
\label{sec:experiment_analysis}
\begin{figure}[tbp]
\centering
\includegraphics[width=0.9\linewidth]{figures/distance.pdf}
\caption{Correlation between linguistic distance and model performance degradation.}
\label{fig:distance}
\end{figure}
% We investigate the relationship between the degree of performance degradation and the linguistic distance from SAE.
% For dialects, we embed each variety using the 77 linguistic features provided by the eWAVE database and compute the distance between each dialect embedding and that of colloquial American English.
% We then plot the model performance drop against these distances.
% The resulting scatter plot reveals a clear trend: varieties that are linguistically further from SAE tend to induce larger performance drops in LLMs.
We investigate the relationship between LLM performance and the linguistic distance from SAE.
We represent each dialect variety as a 235-dimensional vector using the features from the eWAVE database and encoding its prevalence as values.
% , where feature prevalence is numerically encoded (e.g., \textit{{“pervasive or obligatory”: 1.0, “neither pervasive nor extremely rare”: 0.6}}).
We apply singular value decomposition to reduce dimensionality while preserving over 90\% of the variance, and compute the Euclidean distance between each dialect and the SAE reference.\footnote{We used Colloquial American English as SAE reference.}
Figure~\ref{fig:distance} (a) plots the linguistic distance against performance degradation in ARC and WinoGrande, revealing a positive correlation.
This result indicates that models perform worse on dialects that are linguistically farther from SAE.
A similar pattern was observed in ESL English.
The Defense Language Institute Foreign Language Center (DLIFLC) categorizes non-English languages into four levels of difficulty for native English speakers.
Category 1 (easiest) includes French, Italian, Portuguese, and Spanish; Category 2 includes German; Category 3 includes Russian and Turkish; and Category 4 (hardest) includes Arabic, Chinese-Mandarin, and Japanese.
Figure~\ref{fig:distance} (b) presents performance degradation by L1 category using box plots in GSM8K and ARC.
The results show that, although there is variance within each category, ESL English derived from categories 1 and 2 yield smaller performance drops, while those from categories 3 and 4 lead to more significant degradation.
%Both results \replaced{suggest}{indicate} that LLMs are \replaced{generally fine-tuned}{overfitting} to SAE English.
These findings indicate that LLMs are strongly biased toward SAE, and that their robustness declines as the linguistic properties of the target variety diverge from it.
Plots for all datasets are presented in Appendix~\ref{app:full_figure}.
% \subsection{Performance Drop in WinoGrande.}
\section{Conclusion}
In this paper, we introduce \fname, a framework that automatically transforms SAE datasets into a wide range of target English varieties.
\fname is grounded in expert-curated linguistic resources, validated through rigorous experimentation, and developed with the guidance of a researcher specializing in second language acquisition.
By leveraging an LLM for transformations, our framework is scalable across various datasets and varieties.
When applied to benchmark datasets, \fname achieves high transformation coverage, over 80\%, and human evaluations confirmed its linguistic validity.
We transform six benchmark datasets into 38 variates, and experimental results with seven state-of-the-art LLMs reveal significant performance degradation on non-standard varieties, underscoring the importance of evaluating linguistic robustness across diverse forms of English.
\section{Limitations \& Future Work}
% While Trans-EnV offers a scalable and linguistically grounded framework for transforming SAE benchmarks into diverse English varieties, several limitations remain.
This work focuses primarily on English varieties.
Although \fname{} is designed to be extensible to other languages, such extensions would require language-specific resources and transformation guidelines, which we leave for future work.
Our evaluation is currently limited to QA tasks as an initial step toward assessing the linguistic robustness. Extending to other tasks, such as summarization or dialogue, remains an important direction for future research.
% While Trans-EnV provides a scalable and linguistically grounded framework for transforming SAE benchmarks into diverse English varieties, several limitations remain. First, our work focuses only on varieties of English.
% While the proposed framework is theoretically extensible to languages other than English, this would require language-specific linguistic resources and transformation guidelines.
% We leave this multilingual extension as an important direction for future work.
% Second, our method relies LLM-as-a-judge to validate the transformed output.
% To mitigate potential concerns regarding the reliability of this approach, we conducted a human evaluation, which showed strong agreement with LLM-based assessments.
% While we did not observe any major inconsistencies during our human evaluation, the use of LLMs as evaluators may still introduce subtle biases, and we believe this warrants further investigation in future work.
% Lastly, our current evaluation focuses exclusively on QA tasks.
% While the proposed framework is inherently generalizable and could be extended to other open-ended tasks---its effectiveness in these domains has not yet been empirically validated.
% Additionally, the current framework is designed specifically for text-based data, which limits its applicability to multimodal tasks involving auditory or prosodic signals.
% In particular, spoken language data, which often carries meaning through intonation, stress, or rhythm, poses additional challenges that are not addressed in the current text-only setting.
% Extending the framework to accommodate speech-based inputs and evaluating its robustness in such settings remains a promising direction for future work.
\paragraph{Broader Impacts.}
This work aims to improve the robustness of LLMs for users who speak non-standard varieties of English.
By enabling evaluation and transformation across diverse English varieties, \fname{} supports the broader goals of global accessibility and social responsibility in language technologies.
Nonetheless, the absence of professional linguistic resources may lead to suboptimal or invalid transformations.
To mitigate this risk, we encourage researchers to incorporate expert-curated linguistic datasets when adapting or extending \fname{}.
% Our work contributes in developing robust LLMs, particularly for users who speak non-standard varieties of English.
% \fname{} aligns with the broader goal of promoting global accessibility and social responsibility in language technologies.
% However, the absence of professional linguistic resources during construction may result in suboptimal transformations.
% To ensure the validity of \fname{} in downstream applications, we recommend that researchers incorporate expert-curated linguistic datasets.
% To ensure the quality and reliability of the framework, incorporating expert-curated datasets is essential.
% Our work contributes to the robust LLMs, particularly for users who speak non-standard English.
% Trans-EnV supports the broader goal of making language models more globally accessible and socially responsible.
% However, omitting professional linguistic resources during the construction of \fname{} may result in suboptimal transformations.
% Thus, incorporating expert-curated datasets is essential to ensure the quality and reliability of our framework.
% However, the framework also presents potential risks. If used without linguistic or cultural sensitivity, transformed data may inadvertently misrepresent dialects or learner varieties, reinforcing stereotypes or deficit narratives.
% These risks highlight the importance of grounding future multilingual extensions in high-quality linguistic resources and collaborating with linguistic experts, to ensure that transformations are both accurate and culturally respectful.
% \begin{ack}
% Use unnumbered first level headings for the acknowledgments. All acknowledgments
% go at the end of the paper before the list of references. Moreover, you are required to declare
% funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work).
% More information about this disclosure can be found at: \url{https://neurips.cc/Conferences/2025/PaperInformation/FundingDisclosure}.
% Do {\bf not} include this section in the anonymized submission, only in the final paper. You can use the \texttt{ack} environment provided in the style file to automatically hide this section in the anonymized submission.
% \end{ack}
\newpage
% \section*{References}
% References follow the acknowledgments in the camera-ready paper. Use unnumbered first-level heading for
% the references. Any choice of citation style is acceptable as long as you are
% consistent. It is permissible to reduce the font size to \verb+small+ (9 point)
% when listing the references.
% Note that the Reference section does not count towards the page limit.
% \medskip
% {
% \small
% [1] Alexander, J.A.\ \& Mozer, M.C.\ (1995) Template-based algorithms for
% connectionist rule extraction. In G.\ Tesauro, D.S.\ Touretzky and T.K.\ Leen
% (eds.), {\it Advances in Neural Information Processing Systems 7},
% pp.\ 609--616. Cambridge, MA: MIT Press.
% [2] Bower, J.M.\ \& Beeman, D.\ (1995) {\it The Book of GENESIS: Exploring
% Realistic Neural Models with the GEneral NEural SImulation System.} New York:
% TELOS/Springer--Verlag.
% [3] Hasselmo, M.E., Schnell, E.\ \& Barkai, E.\ (1995) Dynamics of learning and
% recall at excitatory recurrent synapses and cholinergic modulation in rat
% hippocampal region CA3. {\it Journal of Neuroscience} {\bf 15}(7):5249-5262.
% }
{
\small
\bibliographystyle{plainnat}
\bibliography{references}
}
% \input{tex/Checklist}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\appendix
\input{tex/Appendix.tex}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}
\begin{table}[t]
\caption{\label{tab:transformation_ratio} Average number of features applied per sample and proportion of transformed samples}
\begin{center}
\resizebox{0.9\linewidth}{!}{%
\begin{tabular}{lcccccc}
\toprule
& MMLU & ARC & TruthfulQA & GSM8K & HellaSwag & WinoGrande \\ \midrule
Dialect & 1.81 / 71.4\% & 1.67 / 69.7\% & 1.44 / 63.0\% & 1.61 / 64.2\% & 2.52 / 83.4\% & 3.22 / 92.1\% \\
% \hline
ESL English & 2.41 / 92.7\% & 2.48 / 94.5\% & 2.03 / 88.4\% & 2.63 / 95.1\% & 2.64 / 94.9\% & 2.76 / 97.3\% \\ \cmidrule(lr){1-7}
Total & 2.12 / 82.6\% & 2.09 / 82.7\% & 1.75 / 76.4\% & 2.15 / 80.5\% & 2.58 / 89.4\% & 2.98 / 94.8\% \\
\bottomrule
\end{tabular}
}
\end{center}
\end{table}
\begin{table}[t]
\caption{\label{tab:transformation_sample} Transformation examples by \fname with sequential application of two features.
}
\begin{center}
\resizebox{\linewidth}{!}{%
\begin{tabular}{ccl}
\toprule
\multirow{6}{*}{Example 1} &
\textbf{SAE} & There are 66 fish in the fish tank. One-third of the fish have red stripes \ldots fish have red stripes and blue stripes?
\\ \cmidrule(r){2-3}&
\textbf{Feat. 1} & Regularization of plural formation: extension of -s to StE irregular plurals
\\&
\textbf{Transf. 1} & There are 66 \underline{fishs} in the fish tank. One-third of the \underline{fishs} have red stripes \ldots \underline{fishs} have red stripes and blue stripes?
\\ \cmidrule(r){2-3}&
\textbf{Feat. 2} & Existential / presentational there’s/there is/there was with plural subjects
\\&
\textbf{Transf. 2} & \underline{There's} 66 fishs in the fish tank. One-third of the fishs have red stripes \ldots fishs have red stripes and blue stripes?
\\ \midrule\multirow{6}{*}{Example 2} &
\textbf{SAE} &Joe has twice as many cars as Robert. He sells 20\% these ones and gives away twice as many cars as the number \ldots
\\ \cmidrule(r){2-3}&
\textbf{Feat. 1} & Usage of a singular noun when a plural form is required
\\&
\textbf{Transf. 1} & Joe has twice as many \underline{car} as Robert. He sells 20\% \underline{this one} and gives away twice as many \underline{car} as the number \ldots
\\ \cmidrule(r){2-3}&
\textbf{Feat. 2} & Omission of a preposition
\\&
\textbf{Transf. 2} & Joe has twice \underline{\hspace{0.2cm}} many car Robert. He sells 20\% this one and gives away twice \underline{\hspace{0.2cm}} many car \underline{\hspace{0.2cm}} the number \ldots
% SAE: Joe has twice as many cars as Robert. He sells 20\% these ones and gives away twice as many cars as the number \ldots \\
% \begin{tabular}[c]{@{}l@{}} \textbf{feature 1}: Usage of a singular noun when a plural form is required\\ \quad Joe has twice as many \underline{car} as Robert. He sells 20\% \underline{this one} and gives away twice as many \underline{car} as the number \ldots \end{tabular} \\
% \begin{tabular}[c]{@{}l@{}} \textbf{feature 2}: Omission of a preposition\\ \quad Joe has twice \underline{\hspace{0.2cm}} many car Robert. He sells 20\% this one and gives away twice \underline{\hspace{0.2cm}} many \underline{\hspace{0.2cm}} the number \ldots \end{tabular} \\
% final: Joe has twice \underline{\hspace{0.2cm}} many \underline{car} Robert. He sells 20\% \underline{this one} and gives away twice \underline{\hspace{0.2cm}} many \underline{car} \underline{\hspace{0.2cm}} the number \ldots
\\ \bottomrule
\end{tabular}
}
\end{center}
\end{table}
% \begin{table}[h]
\begin{center}
\caption{\label{tab:human_eval} Human evaluation on transformed sentences from six different Feature Transformers.}
\resizebox{0.45\textwidth}{!}{%
\begin{tabular}{lccc}
\toprule
Feature Transformer & Q1 & Q2 & Final \\
\midrule
LLaMA-3.1-8B & 25 & 14 & 14 / 25 (56\%) \\
Gemma-2-27B & 25 & 23 & 23 / 25 (92\%) \\
Gemma-3-27B & 25 & 24 & 24 / 25 (95\%) \\
Qwen2.5-32B & 24 & 24 & 23 / 25 (92\%) \\
GPT-4 & 25 & 24 & 24 / 25 (96\%) \\
GPT-4.1-mini & 25 & 24 & 24 / 25 (96\%) \\ \midrule
Total & 149 & 133 & 132 / 150 (88\%) \\
\bottomrule
\end{tabular}
}
\end{center}
\begin{table}[t]
\centering
\caption{Benchmark results of seven models on dialect varieties.}
\vspace{2mm}
\includegraphics[width=0.85\linewidth]{tables/dialects_full.pdf}
% \includegraphics[width=\linewidth]{tables/dialects_4.pdf}
\label{tab:dialect_main}
\end{table}
\begin{table}[t]
\centering
\caption{Benchmark results of seven models on ESL English varieties.}
\vspace{2mm}
\includegraphics[width=0.85\linewidth]{tables/esl_full.pdf}
% \includegraphics[width=\linewidth]{tables/esl_4.pdf}
\label{tab:esl_main}
\end{table}
\clearpage
\startcontents[supplementary]
\printcontents[supplementary]{l}{1}{\section*{Appendix}}
\clearpage
\section{Datasheet for Datasets}
\label{sec:datasheet_for_datasets}
The following section is answers to questions listed in datasheets for datasets.
\input{tex/Datasheets}
\section{Experiment Setting}
\subsection{Computer Resources}
\label{app:computer_resource}
Experiments were conducted using four NVIDIA RTX A6000 GPUs and two NVIDIA A100-SXM4-80GB GPUs.
Our implementation is built on vLLM (v0.5.5), PyTorch (v2.4.0), Hugging Face Transformers (v4.47.0), and Datasets (v3.1.0).
On average, each dataset required approximately 10 hours for transformation.
% 1. A6000, A100 GPU 사용
% 2. vllm, pytorch, python, huggingface, dataets 버전
% 3. 각 실험마다 필요한 amount of compute, total compute.
\section{Dataset Construction}
\label{app:dataset_construction}
\subsection{English Dialects}
\label{app:remove_crelo_pidigns}
\subsubsection{Electronic World Atlas of Varieties of English (eWAVE)}
The Electronic World Atlas of Varieties of English (eWAVE) \citep{ewave} is a curated database documenting 235 linguistic features across 77 English varieties. Developed by 84 professional linguists and grounded in 175 peer-reviewed sources, eWAVE provides a structured taxonomy of features spanning 12 grammatical categories: Pronouns, Noun Phrase, Tense and Aspect, Modal Verbs, Verb Morphology, Negation, Agreement, Relativization, Complementation, Adverbial Subordination, Adverbs and Prepositions, and Discourse and Word Order. Each feature is accompanied by illustrative examples. Varieties are annotated with six levels of feature prevalence: (i) feature is pervasive or obligatory, (ii) feature is neither pervasive nor extremely rare, (iii) feature exists, but is extremely rare, (iv) attested absence of feature, (v) feature is not applicable (given the structural make-up of the variety/P/C), and (vi) no information on feature is available.
\subsubsection{Dialect Selection}
We first mapped the presence strength of each feature per dialect to one of four discrete levels.
\begin{itemize}[leftmargin=3.5mm, itemsep=1mm, parsep=0pt]
\item feature is pervasive or obligatory: 1.0
\item feature is neither pervasive nor extremely rare: 0.5
\item feature exists, but is extremely rare: 0.25
\item attested absence of feature, feature is not applicable, no information on feature is available: 0
\end{itemize}
We then applied Singular Value Decomposition (SVD) for dimensionality reduction, retaining 90\% of the variance.
Using the reduced feature representations for each dialect, we performed K-Nearest Neighbors (KNN) clustering with the number of clusters set to 5.
The choice of 5 clusters was informed by both the Elbow Method and Silhouette Scores, which indicated that 5 was the most optimal number of clusters.
Then we selected clusters with famous English dialects such as African American Vernacular English and Welsh English.
The final 18 dialects and their abbreviations are as follows: African American Vernacular English (AAVE), Irish English (IrE), Australian English (AuE), Bahamian English (BahE), East Anglian English (EAngE), Appalachian English (AppE), English dialects in the Southeast of England (SE-Eng), Australian Vernacular English (AuE-V), English dialects in the North of England (NE-Eng), English dialects in the Southwest of England (SW-Eng), Manx English (Manx), New Zealand English (NZE), Newfoundland English (NfE), Ozark English (OzE), Scottish English (ScE), Southeast American enclave dialects (SE-AmE), Tristan da Cunha English (TdCE), Welsh English (WaE).
% \input{tables/selected_dialects}
\subsection{ESL English-L1}
\label{app:features_of_native_language}
\subsubsection{Number of Samples in Compiled Dataset.}
Table~\ref{tab:l1_sample_num} shows the number of samples per L1 and per CEFR level collected from three learner corpora: CLC-FCE \citep{yannakoudakis2011new}, ICLE \citep{granger1993international}, and EFCamDat \citep{geertzen2013automatic}.
\input{tables/l1_samples_num}
\subsubsection{CEFR Pseudo-Label Generation}
The CLC-FCE and ICLE datasets do not include annotated CEFR levels.
To address this, we employed \texttt{gpt-4o-mini-2024-07-18} to generate pseudo-CEFR labels.
The prompt used for label generation is provided in Table~\ref{tab:psuedo_cefr_label}.
\input{tables/psuedo_cefr_prompt}
\subsubsection{Outputs from the Automatic Grammar Checker}
The outputs from the automatic grammar checker are overly specific, identifying narrow error types such as ``I told her (to) break a leg'' or ``this render (renders) the \ldots''.
To enable more effective analysis, we consolidated similar low-level errors into broader categories.
For instance, the category ``Omission of a Preposition'' includes examples like ``I told her (to) break a leg'' and ``It would be great (to) write a story.''
The category ``Mismatch between Article and Noun'' captures cases such as ``I like to use a pens and paper,'' ``I have received a 150 likes,'' and ``The cat is an animals.''
In total, we define 42 higher-level categories:
``Confusion between effects and affects'',
``Double negation'',
``Gerund complement after psych/perception verb'',
``Inappropriate formulaic closing'',
``Incorrect existential agreement with plural noun'',
``Incorrect passive voice usage'',
``Incorrect pluralization after `either of' '',
``Incorrect use of `if' instead of `whether' '',
``Incorrect use of gerund after `advise' '',
``Incorrect verb usage with auxiliary'',
``Mismatch between article and noun'',
``Mismatch between noun and adjective'',
``Mismatch between subject and verb'',
``Missing complementizer `to' after `allow'' ',
``Missing determiner after quantifier'',
``Misusage of irregular past tense verbs'',
``Misuse of `have' and `having' '',
``Non-standard negation with `let’s' '',
``Omission of a preposition'',
``Omission of a verb'',
``Omission of object pronoun'',
``Omission of required articles'',
``Omission of subject'',
``Plural noun required after quantifier phrase'',
``Redundant discourse marker usage'',
``Redundant modal construction'',
``Redundant phrase repetition'',
``Redundant verb in question form'',
``Singular form in fixed polite expression'',
``Usage of `couple times' instead of `a couple of times' '',
``Usage of a plural noun when a singular form is required'',
``Usage of a plural noun where a singular is required after `is there any' '',
``Usage of a singular noun when a plural form is required'',
``Usage of an adjective where an adverb is required'',
``Usage of an auxiliary verb when unnecessary'',
``Usage of an incorrect past participle form'',
``Usage of first-person subject with `according to' '',
``Usage of passive voice when active voice is required' '',
``Usage of plural auxiliary `do' with singular subject `anyone' '',
``Use of `much' with countable noun'',
``Use of continuous aspect with stative verbs'',
``Use of plural noun with each/every.''
\subsubsection{L1-Specific Features}
The following are the extracted features categorized by L1.
\begin{itemize}[leftmargin=3.5mm, itemsep=1mm, parsep=0pt]
\item Arabic: Usage of a plural noun where a singular is required after `is there any', Incorrect passive voice usage, Usage of `couple times' instead of `a couple of times', Omission of a preposition, Mismatch between article and noun, Omission of a verb, Usage of a singular noun when a plural form is required, Omission of subject, Missing determiner after quantifier, Mismatch between article and noun
\item Chinese-Mandarin: Usage of plural auxiliary `do' with singular subject `anyone', Inappropriate formulaic closing, Mismatch between subject and verb, Singular form in fixed polite expression, Omission of subject, Usage of an incorrect past participle form, Mismatch between article and noun, Incorrect existential agreement with plural noun, Usage of passive voice when active voice is required
\item French: Non-standard negation with `let’s', Usage of `couple times' instead of `a couple of times', Redundant verb in question form, Misuse of `have' and `having', Usage of a plural noun where a singular is required after `is there any', Use of plural noun with each/every, Gerund complement after psych/perception verb, Omission of a preposition, Omission of a verb, Usage of first-person subject with `according to'
\item German: Incorrect passive voice usage, Usage of `couple times' instead of `a couple of times', Misuse of `have' and `having', Gerund complement after psych/perception verb, Omission of a preposition, Incorrect verb usage with auxiliary, Misusage of irregular past tense verbs, Use of `much' with countable noun, Usage of an adjective where an adverb is required, Incorrect use of gerund after `advise'
\item Italian: Incorrect use of `if' instead of `whether', Usage of `couple times' instead of `a couple of times', Usage of a plural noun where a singular is required after `is there any', Redundant discourse marker usage, Incorrect pluralization after `either of', Gerund complement after psych/perception verb, Use of plural noun with each/every, Usage of a singular noun when a plural form is required, Omission of a verb, Misusage between `not' and `never'
\item Japanese: Use of continuous aspect with stative verbs, Mismatch between noun and adjective, Redundant modal construction, Usage of a singular noun when a plural form is required, Omission of a preposition, Gerund complement after psych/perception verb, Missing determiner after quantifier, Plural noun required after quantifier phrase, Omission of required articles, Omission of object pronoun
\item Portuguese: Omission of a preposition, Omission of subject, Gerund complement after psych/perception verb, Usage of an auxiliary verb when unnecessary, Usage of a singular noun when a plural form is required, Missing complementizer `to' after `allow', Singular form in fixed polite expression, Redundant phrase repetition, Double negation, Incorrect existential agreement with plural noun
\item Russian: Redundant verb in question form, Mismatch between article and noun, Misusage of preposition, Mismatch between subject and verb, Omission of a verb, Omission of subject, Missing complementizer `to' after `allow', Omission of a preposition, Redundant verb, Redundant preposition
\item Spanish: Non-standard negation with 'let’s', Incorrect pluralization after `either of', Mismatch between article and noun, Omission of subject, Omission of a preposition, Incorrect verb usage with auxiliary, Usage of a singular noun when a plural form is required, Missing Determiner after Quantifier, Redundant verb, Misusage of article in uncountable noun
\item Turkish: Confusion between effects and affects, Usage of first-person subject with `according to', Usage of a singular noun when a plural form is required, Omission of a preposition, Missing complementizer `to' after `allow', Omission of subject, Usage of a plural noun when a singular form is required, Missing determiner after quantifier, Mismatch between article and noun, Redundant adverb
\end{itemize}
\input{tables/guideline_prompt}
\input{tables/transformation_guideline_example}
\subsection{Transformation Guideline Generation}
\label{app:guideline_generation}
% \subsubsection{Generation Setting}
We use \texttt{gpt-4-0613} to generate transformation guidelines via one-shot prompting, with a temperature of 0.8 and top-$p$ sampling set to 0.95.
The model is provided with the name of the linguistic feature, a brief description, and representative examples.
It is then instructed to (1) describe the linguistic characteristics of the feature, and (2) outline a step-by-step transformation procedure consisting of two phases: \textit{Qualification}, which checks whether the feature applies to a given sentence, and \textit{Application}, which modifies the sentence accordingly.
We emphasize that the transformation process should focus strictly on lexical rules, avoiding subjective elements such as emotional or cultural interpretation, metaphor, or judgments of significance.
The full prompt used for generating transformation guidelines is shown in Table~\ref{tab:guideline_prompt}, and examples of the resulting guidelines are presented in Table~\ref{tab:examples_transformation_guidelines}.
% We use \texttt{gpt-4-0613} to generate transformation guidelines, with a temperature of 0.8 and top-$p$ sampling set to 0.95.
% The model is provided with the name of the linguistic feature, a brief description, and representative examples.
% Then the model is asked to generate its linguistic characteristics of the given linguistic feautre then outline detailed steps to transform a given sentence to reflect characteristic, breaking the process into two main phases: Qualification and Application.
% We highlight that the questions should be strictly related to lexical rules and should not ask for any decisions asking
% for significance or emotions.
% Also the questions should strictly be related to lexicon and not include context, culture, or metaphor where answers might differ by people such as determining significant role or emotional attachment.
% Guideline generation is performed via one-shot prompting.
% The full prompt is shown in Table~\ref{tab:guideline_prompt}.
% Table~\ref{tab:examples_transformation_guidelines} shows examples of transformation guidelines.
\subsection{Transforming into English Varieties}
\subsubsection{Transformation of Vocabulary into Target CEFR Levels}
\label{app:vocab_replace}
\begin{figure}[hbp]
\centering
\includegraphics[width=\linewidth]{figures/vocab_histogram.pdf}
\caption{Histograms of distributions of higher-level word usage in CEFR A and CEFR B texts.}
\label{fig:vocab_histogram}
\end{figure}
To ensure that the transformed outputs for ESL English varieties reflect realistic proficiency levels, we incorporated a vocabulary substitution step guided by CEFR-level annotations.
To acknowledge that ESL learners often know a small fraction of advanced words even at lower proficiency levels, we first analyze a CEFR-labeled English text dataset to find out the ratio of higher-level words used by lower CEFR proficiency level learners, as mentioned in Section \ref{sec:transform}.
Figure \ref{fig:vocab_histogram} presents the distribution of higher-level vocabulary in the dataset (\eg, B or C level words in A level texts).
Notably, for both target levels A and B, at least 90\% of the samples contain no more than 15\% of vocabulary from higher CEFR levels than the designated target level.
This empirical finding motivated our decision to allow up to 15\% of higher-level vocabulary in transformed outputs.
This threshold balances fidelity to learner-level constraints with linguistic realism, acknowledging that ESL learners often know a small fraction of advanced words even at lower proficiency levels.
\input{tables/vocab_label_prompt}
\input{tables/vocab_transformation_prompt}
Table \ref{tab:vocab_label_prompt} shows the prompt used for finding pseudo-labels for words without a CEFR label in the Oxford vocabulary lists, and Table \ref{tab:vocab_transformation_prompt} presents the prompt used for transforming higher-level vocabulary in a sentence to a target level.
The value of \texttt{min\_transform\_words} is set to 15\% of the total word count in \texttt{question\_text} and serves as the threshold for permitted higher-level words.
Table \ref{tab:vocabulary_ratio} presents the transformation success rates by CEFR level and dataset, showing how often our pipeline was able to produce outputs that met CEFR-level vocabulary constraints while preserving semantic equivalence.
\input{tables/vocabulary_ratio}
% model, prompt, gpt-4o psuedo label generation prompt, CEFR text result (15\% 정당성 받은거),
\subsection{Prompts used for Transformation}
\label{app:prompts}
Table~\ref{tab:transformation_prompt} presents the one-shot prompt used to transform a Standard American English (SAE) sentence $s$ into a target variety using the feature transformation model $T$. Each transformation is guided by a feature-specific guideline and example. The model is instructed to follow the guideline strictly, preserving the structure and core meaning of the original sentence while disregarding grammatical correctness.
To ensure semantic fidelity, we employ a semantic checker model $S$ using a zero-shot prompt, as shown in Table~\ref{tab:semantic_check_prompt}. The verification process emphasizes the preservation of key content elements such as keywords, numerical information, and core propositions, while ignoring minor grammatical deviations, including incorrect or missing prepositions and redundancy.
\input{tables/transformation_prompt}
\input{tables/semantic_check_prompt}
\subsection{Transformation Ratio}
\label{app:transformation_ratio}
Tables~\ref{dialect_transform_full} and~\ref{l1_transform_full} report the average number of features applied per sample and the overall proportion of transformed samples for dialect and ESL English, respectively, as discussed in Section~\ref{sec:analysis}. Consistent with the results presented in the main paper, ESL English exhibits a higher transformation rate and a greater average number of features applied per sample compared to dialects.
\input{tables/dialect_transform_full}
\input{tables/l1_transform_full}
\subsection{Human Evaluation}
\label{app:human_evaluation}
\begin{figure}[h!]
\centering
\fbox{\includegraphics[width=0.9\linewidth]{figures/human_eval_2.pdf}}
\caption{Interface used for human evaluation.}
\label{fig:human_eval}
\end{figure}
Human annotators were shown one sample at a time, with a total of 150 samples randomly shuffled, 25 from each model.
For each sample, annotators answered two binary (yes/no) questions: (Q1) whether the model correctly followed the Qualification and Application steps specified in the transformation guideline, and (Q2) whether the transformed sentence preserved the original meaning.
The interface presented to annotators is shown in Figure~\ref{fig:human_eval}.
A sample was considered valid if it received majority approval from the annotators.
\section{Experiments}
\subsection{Experiment Setting}
% \label{app:full_results}
We evaluated the transformed datasets on seven state-of-the-art models: Qwen2.5-72B-Instruct \citep{qwen2.5}, DeepSeek-R1-Distill-Llama-70B \citep{deepseekai2025deepseekr1incentivizingreasoningcapability}, LLaMA-3.3-70B-Instruct \citep{grattafiori2024llama}, Gemini 2.0 Flash \citep{google_gemini2flash_2025}, Gemini 2.5 Pro \citep{google_gemini25pro_2025}, GPT-4o-mini \citep{openai_gpt4omini_2024}, and o4-mini \citep{openai_o4mini_2025}.\footnote{Model versions: \texttt{gemini-2.5-pro-exp-03-25}, \texttt{gpt-4o-mini-2024-07-18}, \texttt{o4-mini-2025-04-16}}
We set the maximum number of generated tokens to 2048 and conducted all experiments in a zero-shot setting.
The system prompt used was: ``Do not reason for too long. If the question is a multiple choice question, answer with the option letter. If none of the given options match, you may guess or say `none of the above.' Start your final sentence with `The answer is '.''
To extract the model’s prediction, we parsed the output beginning from the phrase ``The answer is'', using the subsequent text as the final answer.
\subsection{Full Experiment Analysis}
\label{app:full_figure}
Figures~\ref{fig:distance_full} and~\ref{fig:esl_distance_full} present the full analysis results across all datasets, corresponding to the analysis in Section~\ref{sec:experiment_analysis}.
Consistent with the findings in the main paper, we observe a positive correlation between linguistic distance from Standard American English (SAE) and performance degradation, although the strength of this relationship varies across datasets.
In ESL English, despite some deviations, performance drop generally increases with the difficulty level of the English variety.
\begin{figure}[hbp]
\centering
\includegraphics[width=\linewidth]{figures/distance_full.pdf}
\caption{Correlation between linguistic distance and model performance degradation.}
\label{fig:distance_full}
\end{figure}
\begin{figure}[hbp]
\centering
\includegraphics[width=\linewidth]{figures/esl_distance_full.pdf}
\caption{Boxplot by difficulty category and model performance degradation.}
\label{fig:esl_distance_full}
\end{figure}
\subsection{Motivation}
\begin{itemize}
\item Question:
\textbf{For what purpose was the dataset created?}
Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.
\item[] Answer:
To evaluate the linguistic robustness of language models across diverse English varieties by transforming Standard American English (SAE) datasets.
\item Question:
\textbf{Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?}
\item[] Answer:
The authors of this paper.
\item Question:
\textbf{Who funded the creation of the dataset?}
If there is an associated grant, please provide the name of the grantor and the grant name and number.
\item[] Answer:
This work was supported by Institute for Information \& communications Technology Planning \& Evaluation(IITP) grant funded by the Korea government(MSIT) (RS-2019-II190075, Artificial Intelligence Graduate School Program(KAIST)).
\end{itemize}
\subsection{Composition}
\begin{itemize}
\item Question:
\textbf{What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?}
Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
\item[] Answer:
QA datasets (sentences) transformed into various English varieties.
\item Question:
\textbf{How many instances are there in total (of each type, if appropriate)?}
\item[] Answer: There are about 952K instances in total.
\item Question:
\textbf{Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?}
If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)?
If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
\item[] Answer: The dataset contains all instances from the existing benchmark datasets.
\item Question:
\textbf{What data does each instance consist of?}
“Raw” data (e.g., unprocessed text or images) or features?
In either case, please provide a description.
\item[] Answer:
Each instance consists of the transformed text, answer choices, and label.
\item Question:
\textbf{Is there a label or target associated with each instance?}
If so, please provide a description.
\item[] Answer:
Yes, each label comes from the original QA datasets.
\item Question:
\textbf{Is any information missing from individual instances?}
If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.
\item[] Answer:
No, there is no information missing from individual instances.
\item Question:
\textbf{Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?}
If so, please describe how these relationships are made explicit.
\item[] Answer:
No.
\item Question:
\textbf{Are there recommended data splits (e.g., training, development/validation, testing)?}
If so, please provide a description of these splits, explaining the rationale behind them.
\item[] Answer:
This dataset is for testing only.
\item Question:
\textbf{Are there any errors, sources of noise, or redundancies in the dataset?}
If so, please provide a description.
\item[] Answer:
No, we have verified that there are no errors in the datasets.
\item Question: \textbf{Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?}
If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a dataset consumer?
Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
\item[] Answer:
Our dataset is self-contained.
\item Question:
\textbf{Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor–patient confidentiality, data that includes the content of individuals’ non-public communications)?}
If so, please provide a description.
\item[] Answer:
No.
\item Question:
\textbf{Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?}
If so, please describe why.
\item[] Answer:
No.
\end{itemize}
% \subsection{Collection Process}
% \begin{itemize}
% \item Question:
% \textbf{How was the data associated with each instance acquired?}
% Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)?
% If the data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.
% \item[] Answer:
% We utilized publicly available open-source datasets.
% \item Question:
% \textbf{What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)?}
% How were these mechanisms or procedures validated?
% \item[] Answer:
% We utilized publicly available open-source datasets.
% \item Question:
% \textbf{If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? }
% \item[] Answer:
% We used the complete datasets.
% \item Question:
% \textbf{Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?}
% \item[] Answer:
% \item Question:
% \textbf{Over what timeframe was the data collected?}
% Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)?
% If not, please describe the timeframe in which the data associated with the instances was created.
% \item[] Answer:
% \item Question:
% \textbf{Were any ethical review processes conducted (e.g., by an institutional review board)?}
% If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.
% \item[] Answer:
% No.
% \end{itemize}
\subsection{Preprocessing/cleaning/labeling}
\begin{itemize}
\item Question:
\textbf{Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?}
If so, please provide a description. If not, you may skip the remaining questions in this section.
\item[] Answer:
\item Question:
\textbf{Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)?}
If so, please provide a link or other access point to the “raw” data.
\item[] Answer:
No.
\item Question:
\textbf{Is the software that was used to preprocess/clean/label the data available?}
If so, please provide a link or other access point.
\item[] Answer:
\begin{itemize}
\item Google Sheets: \texttt{https://docs.google.com/spreadsheets/}
\item Python: \texttt{https://www.python.org/}
\end{itemize}
\end{itemize}
\subsection{Uses}
\begin{itemize}
\item Question:
\textbf{Has the dataset been used for any tasks already?}
If so, please provide a description.
\item[] Answer:
No.
\item Question:
\textbf{Is there a repository that links to any or all papers or systems that use the dataset?}
If so, please provide a link or other access point.
\item[] Answer:
No.
\item Question:
\textbf{What (other) tasks could the dataset be used for?}
\item[] Answer:
N/A
\item Question:
\textbf{Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?}
For example, is there anything that a dataset consumer might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other risks or harms (e.g., legal risks, financial harms)?
If so, please provide a description. Is there anything a dataset consumer could do to mitigate
these risks or harms?
\item[] Answer:
N/A
\item Question:
\textbf{Are there tasks for which the dataset should not be used?}
If so, please provide a description.
\item[] Answer:
N/A
\end{itemize}
\subsection{Distribution}
\begin{itemize}
\item Question:
\textbf{Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?}
If so, please provide a description.
\item[] Answer:
Yes, the dataset will be made publicly accessible through Hugging Face.
\item Question:
\textbf{How will the dataset will be distributed (e.g., tarball on website, API, GitHub)?}
Does the dataset have a digital object identifier (DOI)?
\item[] Answer:
The datasets will be distributed on Hugging Face with public access.
\item Question:
\textbf{When will the dataset be distributed?}
\item[] Answer:
The dataset is publicly available on Hugging Face since May 12, 2025.
\item Question:
\textbf{Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?}
If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
\item[] Answer:
The datasets are distributed under the CC BY-SA 4.0 license.
\item Question:
\textbf{Have any third parties imposed IP-based or other restrictions on the data associated with the instances?}
If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.
\item[] Answer:
No.
\item Question:
\textbf{Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?}
If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.
\item[] Answer:
No.
\end{itemize}
\subsection{Maintenance}
\begin{itemize}
\item Question:
\textbf{Who will be supporting/hosting/maintaining the dataset?}
\item[] Answer:
The dataset is hosted on Hugging Face.
\item Question:
\textbf{How can the owner/curator/manager of the dataset be contacted (e.g., email address)?}
\item[] Answer: Contact the authors of this paper via email.
\item Question:
\textbf{Is there an erratum?}
If so, please provide a link or other access point.
\item[] Answer:
No.
\item Question:
\textbf{Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?}
If so, please describe how often, by whom, and how updates will be communicated to dataset consumers (e.g., mailing list, GitHub)?
\item[] Answer:
The datasets will be updated if necessary.
\item Question:
\textbf{If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)?}
If so, please describe these limits and explain how they will be enforced.
\item[] Answer:
The dataset does not relate with people.
\item Question:
\textbf{Will older versions of the dataset continue to be supported/hosted/maintained?}
If so, please describe how. If not, please describe how its obsolescence will be communicated to dataset consumers.
\item[] Answer:
Yes.
\item Question:
\textbf{If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?}
If so, please provide a description. Will these contributions be validated/verified?
If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to dataset consumers?
If so, please provide a description.
\item[] Answer:
No, our datasets are freely available for others to use.
\end{itemize}
\begin{table}[h]
\caption{Number of samples collected from CLC-FCE, ICLE, and EFCamDat.}
\label{tab:l1_sample_num}
\centering
% \resizebox{\textwidth}{!}{%
\begin{tabular}{lcccccccc}
\toprule
& \multicolumn{2}{c}{CLC-FCE} & \multicolumn{2}{c}{ICLE} & \multicolumn{2}{c}{EFCamDat} & \multicolumn{2}{c}{Total} \\ \cmidrule[0.5pt](lr){2-3} \cmidrule[0.5pt](lr){4-5} \cmidrule[0.5pt](lr){6-7} \cmidrule[0.5pt](lr){8-9}
& A & B & A & B & A & B & A & B \\ \midrule
Arabic & 0 & 0 & 0 & 0 & 24,155 & 4,857 & 24,155 & 4,857 \\
Chinese-Mandarin & 9 & 107 & 1 & 45 & 106,654 & 22,289 & 106,664 & 22,441 \\
French & 2 & 245 & 0 & 0 & 22,244 & 9,646 & 22,246 & 9,891 \\
German & 2 & 120 & 3 & 42 & 25,040 & 14,501 & 25,045 & 14,663 \\
Italian & 2 & 121 & 1 & 8 & 22,787 & 11,672 & 22,790 & 11,801 \\
Japanese & 6 & 134 & 10 & 171 & 11,653 & 5,081 & 11,669 & 5,386 \\
Portuguese & 1 & 114 & 1 & 43 & 248,200 & 61,751 & 248,202 & 61,908 \\
Russian & 10 & 134 & 0 & 12 & 35,081 & 13,287 & 35,091 & 13,433 \\
Spanish & 16 & 351 & 6 & 47 & 52,786 & 11,456 & 52,808 & 11,854 \\
Turkish & 8 & 126 & 0 & 61 & 7,899 & 2,237 & 7,907 & 2,424 \\
\bottomrule
\end{tabular}
% }
\end{table}
\begin{table}[h]
\centering
\caption{\label{tab:psuedo_cefr_label} Prompt used for pseudo CEFR label generation.}
\resizebox{0.9\textwidth}{!}{%
\begin{tabular}{l}
\toprule
\begin{tabular}[c]{@{}l@{}}\textbf{System:} \\ You are a linguistic expert.\\ \\ \textbf{User:} \\ Classify the given sentence among three CEFR levels (A, B, C). Respond only CEFR level. \\ Sentence: \texttt{\{sentence\}}\end{tabular} \\ \bottomrule
\end{tabular}
}
\end{table}
\begin{table}[t]
\centering
\caption{\label{tab:guideline_prompt} Prompt for generating transformation guidelines}
\resizebox{\textwidth}{!}{%
\begin{tabular}{p{\textwidth}}
\toprule
\textbf{System:} \\ You are a linguistic expert. I am a student trying to understand the given linguistic feature and transform a sentence reflecting the feature. As an expert, first explain the linguistic characteristics of the given linguistic feature. Then, outline detailed steps to transform a given sentence to reflect the characteristic, breaking the process into two main phases: Qualification and Application. Qualification refers to steps that identify if the linguistic feature is applicable to the given sentence in yes/no question format where answers to all questions should be `yes' if feature dialect is applicable.
Here, questions in qualification should be strictly related to lexical rules and should not ask for any decisions asking for significance or emotions.
The questions should also not include questions about context, culture, or metaphors.
Application refers to the action items that a model should take in order to reflect the given linguistic feature.
All questions and action items should strictly be related to lexicon.
All questions and action items should not include context, culture, or metaphor where answers might differ by people such as determining significant role or emotional attachment.\\ \\
\textbf{User:} \\
Linguistic Feature: She/her used for inanimate referents. \\
Description: Pronouns, pronoun exchange, nominal gender. \\
Examples: The boat I had, was a seventy-two foot boat., She was built in Joneses-Slip in nineteen-fifty-five by my father., She's a nice bike., I left the boat there as she was., This is another pot and saucer., A bit dusty! You see that one isn't exactly glazed proper, burnt proper is she?, She's a twenty point five metre sloop again designed by Kel Steinman an Australian, So the Bransfield is the BAS ship then? Yeah she was the sort of kingpin until they brought the James Clark Ross \\ \break
\textbf{Assistant:}\\
Linguistic Characteristic: She/Her Used for Inanimate Referents
\\
The phenomenon of using `she' or `her' for inanimate objects is primarily influenced by cultural associations, personification, or grammatical gender in certain languages. In English, it is less common but can occur in specific contexts, such as referring to ships, cars, or countries, often reflecting affection, personification, or historical conventions. This use can evoke an emotional connection or imply a particular viewpoint about the inanimate referent. \\ \break
Steps to Transform a Sentence: To transform a given sentence to reflect the use of `she/her' for inanimate referents, we can break the process down into two main phases: Qualification and Application.
Qualification
1. Is there an inanimate referent in the sentence?
2. Does the original sentence contain a pronoun that can be replaced with `she' or `her'?
If the answers to all relevant questions are "Yes," then the linguistic feature is applicable.
Application
1. Identify the inanimate referent in the sentence that will be modified.
2. Replace any pronouns referring to the inanimate referent with "she" or "her."
\\ \break
\textbf{User:} \\
Linguistic Feature: \texttt{\{linguistic feature\}}
\\ Description: \texttt{\{description of linguistic feature\}} \\
Examples:
\texttt{\{examples of linguistic feature\}} \\
\bottomrule
\end{tabular}
}
\end{table}
% \begin{table}[ht]
% \caption{Examples of Transformation Guidelines.}
% \label{tab:transformation_guidelines}
% \begin{subtable}{\linewidth}
% \begin{tcolorbox}[
% %colback=orangebg,
% colframe=black,
% fontupper=\footnotesize
% ]
% \textbf{Feature: Myself/meself instead of I in coordinate subjects}
% \bigbreak
% \textbf{Qualification}:
% \begin{enumerate}[leftmargin=3.5mm, itemsep=1mm, parsep=0pt]
% \item Is there a coordinate subject in the sentence? A coordinate subject is formed when two subjects are joined by a conjunction like `and' or `or'.
% \item Does the coordinate subject include `I'?
% \end{enumerate}
% \bigbreak
% If the answers to all relevant questions are `Yes', then the linguistic feature is applicable.
% \bigbreak
% \textbf{Application}:
% \begin{enumerate}[leftmargin=3.5mm, itemsep=1mm, parsep=0pt]
% \item Identify the coordinate subject in the sentence that includes `I'.
% \item Replace `I' with `myself' in the coordinate subject.
% \end{enumerate}
% \end{tcolorbox}
% % \caption{Aspect}
% \end{subtable}
% \begin{subtable}{\linewidth}
% \begin{tcolorbox}[
% % colback=bluebg,
% colframe=black,
% fontupper=\footnotesize
% ]
% Feature: Omission of Required Articles
% \bigbreak
% Qualification:
% \begin{enumerate}[leftmargin=3.5mm, itemsep=1mm, parsep=0pt]
% \item Does the sentence contain a noun that requires an article (`a', `an', or `the') for grammatical correctness or clarity?
% \item Is the noun countable and in singular form, or does it refer to something specific that needs `the'?
% \end{enumerate}
% \bigbreak
% If the answers to all relevant questions are `Yes', then the linguistic feature is applicable.
% \bigbreak
% Application:
% \begin{enumerate}[leftmargin=3.5mm, itemsep=1mm, parsep=0pt]
% \item Identify the noun(s) that require an article for grammatical correctness.
% \item Remove the article (`a', `an', or `the') preceding the noun or leave the noun without any article.
% \end{enumerate}
% \end{tcolorbox}
% % \caption{Opinion}
% \end{subtable}
% \end{table}
\begin{table}[h]
\centering
\caption{\label{tab:examples_transformation_guidelines} Examples of transformation guidelines.}
\resizebox{0.9\linewidth}{!}{%
\begin{tabular}{p{\textwidth}}
\toprule
\textbf{Feature: Myself/meself instead of I in coordinate subjects}
\\ \break
\textbf{Qualification}: \\
% \begin{enumerate} \\
1. Is there a coordinate subject in the sentence? A coordinate subject is formed when two subjects are joined by a conjunction like `and' or `or'. \\
2. Does the coordinate subject include `I'?
% \end{enumerate}
\\
\break
If the answers to all relevant questions are `Yes', then the linguistic feature is applicable.
\\ \break
\textbf{Application}: \\
% \begin{enumerate}
1. Identify the coordinate subject in the sentence that includes `I'. \\
2. Replace `I' with `myself' in the coordinate subject.
% \end{enumerate}
\\ \midrule
\textbf{Feature: Omission of Required Articles}
\\ \break
\textbf{Qualification:} \\
1. Does the sentence contain a noun that requires an article (`a', `an', or `the') for grammatical correctness or clarity? \\
2. Is the noun countable and in singular form, or does it refer to something specific that needs `the'? \\
\break
If the answers to all relevant questions are `Yes', then the linguistic feature is applicable.
\\ \break
\textbf{Application:} \\
1. Identify the noun(s) that require an article for grammatical correctness. \\
2. Remove the article (`a', `an', or `the') preceding the noun or leave the noun without any article.\\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[h]
\centering
\caption{\label{tab:vocab_label_prompt} Vocabulary pseudo-label prompt.}
\resizebox{0.9\textwidth}{!}{%
\begin{tabular}{p{\textwidth}}
\toprule
\textbf{System:}
You are an expert in classifying vocabulary into CEFR levels. Given a single word, classify it into its appropriate CEFR level when used with its most common definition. If it is a proper noun, answer with A1. Answer only with one of the following: A1, A2, B1, B2, C1, C2.
\\
\textbf{User:}
\texttt{\{word\}}
\\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[h]
\centering
\caption{\label{tab:vocab_transformation_prompt} Vocabulary transformation prompt.}
\resizebox{0.9\textwidth}{!}{%
\begin{tabular}{p{\textwidth}}
\toprule
\textbf{System:}
You are an expert in transforming vocabulary of higher CEFR levels to level \texttt{\{target\_level\}}. You are given higher level words that appear in the question: \texttt{\{words\_to\_transform\}}. Please replace at least \texttt{\{min\_transform\_words\}} words with synonyms in level \texttt{\{target\_level\}}.
\\
\textbf{User:}
\texttt{\{question\_text\}}
\\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[h]
\caption{\label{tab:vocabulary_ratio} Number and ratio of valid vocabulary transformations by dataset.}
\begin{center}
\resizebox{0.6\textwidth}{!}{%
\begin{tabular}{ccccc}
\toprule
Dataset & Size & Target CEFR & Valid Transf. & Transf. Ratio
\\ \midrule
\multirow{2}{*}[-3pt]{MMLU} & \multirow{2}{*}[-3pt]{$14042$} & A & $7246$ & 51.6\%
\\ \cmidrule(r){3-5}
& & B & $11970$ & 85.2\%
\\ \cmidrule(){1-5}
\multirow{2}{*}[-3pt]{GSM8K} & \multirow{2}{*}[-3pt]{$1319$} & A & $1219$ & 92.4\%
\\ \cmidrule(r){3-5}
& & B & $1315$ & 99.7\%
\\ \cmidrule(){1-5}
\multirow{2}{*}[-3pt]{ARC} & \multirow{2}{*}[-3pt]{$1172$} & A & $774$ & 66.0\%
\\ \cmidrule(r){3-5}
& & B & $1132$ & 96.6\%
\\ \cmidrule(){1-5}
\multirow{2}{*}[-3pt]{HellaSwag} & \multirow{2}{*}[-3pt]{$10042$} & A & $7593$ & 75.6\%
\\ \cmidrule(r){3-5}
& & B & $9903$ & 98.6\%
\\ \cmidrule(){1-5}
\multirow{2}{*}[-3pt]{TruthfulQA} & \multirow{2}{*}[-3pt]{$817$} & A & $623$ & 76.3\%
\\ \cmidrule(r){3-5}
& & B & $781$ & 95.6\%
\\ \cmidrule(){1-5}
\multirow{2}{*}[-3pt]{WinoGrande} & \multirow{2}{*}[-3pt]{$1267$} & A & $945$ & 74.6\%
\\ \cmidrule(lr){3-5}
& & B & $1247$ & 98.4\%
\\ \bottomrule
\end{tabular}
}
\end{center}
\end{table}
\begin{table}[t]
\centering
\caption{\label{tab:transformation_prompt} Prompt for transforming into varieties.}
\resizebox{\textwidth}{!}{%
\begin{tabular}{p{\textwidth}}
\toprule
\textbf{System:} Your task is to rephrase the given sentence by following the guideline.
\{transformation guideline\}
\\
1. **Qualification**: \\
- Answer the qualification questions for the linguistic feature with either "yes" or "no." \\
- Answer the questions in a very strict manner. \\
- Proceed to the next step only if **all** answers are "yes." \\
- Otherwise, stop in qualification phase with generating `**Transformed Sentence:** (No change)'. \\
\\
2. **Application**: \\
- Make only the **necessary changes** to apply the linguistic feature, ensuring no loss of information. \\
- Provide the final transformed sentence, adhering strictly to the format and structure of the given example. \\
\\
\#\#\# Mandatory \\
- Proceed to Application only if all answers to the qualification questions are `yes'. \\
- Preserve the structure of the original sentence as much as possible with no information loss. \\
- Follow the guideline, not considering standard English grammar. \\
- Final sentence should start with `**Transformed Sentence:**' either with sentence of (No change).
\\
\textbf{User:}
**Original Sentence**: \texttt{\{example sentence\}} \\ \\
\textbf{Assistant:}
\texttt{\{example output\}} \\ \\
\textbf{User:}
**Original Sentence**: \texttt{\{SAE written sentence\}}
\\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[t]
\centering
\caption{\label{tab:semantic_check_prompt} Prompt for semantic check.}
\resizebox{\textwidth}{!}{%
\begin{tabular}{p{\textwidth}}
\toprule
\textbf{User:} Determine whether the meaning of Sentence 1 is significantly altered or lost in Sentence 2. \\ \\
\#\#\# Consideration \\
- All keywords from Sentence 1 should be in Sentence 2. \\
- All numbers in Sentence 1 should match with Sentence 2. \\
- Focus on core information only. \\
- Ignore grammar; it is not a factor for consideration. \\
- Missing or incorrect prepositions should not be considered. \\
- Ignore repetition of phrases. Repetition is not a factor for consideration. \\
- Base your decision solely on whether essential information is missing. \\ \\
Respond with either `yes' or `no' only.\\ \\
Sentence 1: \texttt{\{SAE written sentence\}} \\
Sentence 2: \texttt{\{transformed sentence\}} \\ \\
Answer:
\\
\bottomrule
\end{tabular}
}
\end{table}
\begin{table}[]
\centering
\caption{\label{dialect_transform_full} Average number of features applied per sample and proportion of transformed samples in dialect.}
\resizebox{\textwidth}{!}{%
\begin{tabular}{lcccccc}
\toprule
& MMLU & ARC & TruthfulQA & GSM8K & Hellaswag & WinoGrande \\ \midrule
AAVE & 1.12 / 61.8\% & 1.17 / 65.1\% & 0.76 / 45.0\% & 0.80 / 54.7\% & 2.01 / 87.4\% & 2.06 / 88.4\% \\
AppE & 1.53 / 70.6\% & 1.14 / 64.9\% & 1.08 / 60.5\% & 1.11 / 63.4\% & 2.26 / 88.9\% & 2.70 / 96.5\% \\
AuE & 0.80 / 65.3\% & 0.76 / 64.8\% & 0.49 / 41.5\% & 0.40 / 33.0\% & 0.91 / 66.5\% & 1.60 / 96.8\% \\
AusVE & 0.95 / 57.5\% & 0.76 / 50.7\% & 0.78 / 57.8\% & 1.05 / 70.5\% & 1.53 / 82.9\% & 1.63 / 91.9\% \\
BahE & 2.63 / 70.5\% & 1.94 / 53.7\% & 1.76 / 63.4\% & 2.91 / 76.6\% & 3.20 / 83.9\% & 6.22 / 99.5\% \\
EAngE & 3.54 / 87.7\% & 3.08 / 86.1\% & 2.87 / 90.0\% & 3.75 / 90.2\% & 4.58 / 95.9\% & 5.94 / 99.8\% \\
IrE & 2.67 / 91.0\% & 2.92 / 95.0\% & 2.49 / 87.8\% & 1.80 / 78.8\% & 4.82 / 98.9\% & 4.53 / 100.0\% \\
Manx & 1.86 / 86.8\% & 1.64 / 86.9\% & 1.57 / 80.7\% & 0.84 / 60.5\% & 2.57 / 95.8\% & 3.22 / 98.3\% \\
NE-Eng & 0.70 / 59.6\% & 0.77 / 70.5\% & 0.43 / 38.8\% & 0.58 / 54.7\% & 1.43 / 89.9\% & 1.05 / 77.0\% \\
NZE & 2.07 / 84.7\% & 2.12 / 88.2\% & 1.48 / 70.3\% & 2.15 / 85.8\% & 3.10 / 97.3\% & 3.48 / 99.4\% \\
NfE & 4.17 / 95.4\% & 3.98 / 96.4\% & 3.31 / 92.5\% & 4.3 / 96.7\% & 5.55 / 98.9\% & 7.63 / 99.9\% \\
OzE & 2.50 / 86.6\% & 2.73 / 91.9\% & 2.17 / 85.8\% & 2.75 / 89.8\% & 3.59 / 96.6\% & 4.07 / 99.2\% \\
SE-AmE & 2.50 / 79.9\% & 2.19 / 70.8\% & 2.03 / 79.1\% & 2.98 / 84.9\% & 3.65 / 91.4\% & 4.72 / 99.6\% \\
SE-Eng & 0.20 / 17.4\% & 0.14 / 13.2\% & 0.07 / 6.6\% & 0.22 / 19.7\% & 0.26 / 22.9\% & 0.30 / 25.7\% \\
SW-Eng & 0.90 / 66.3\% & 0.77 / 62.9\% & 0.55 / 43.6\% & 0.33 / 30.0\% & 0.84 / 64.4\% & 1.67 / 96.9\% \\
ScE & 1.15 / 69.8\% & 1.06 / 67.9\% & 1.06 / 63.5\% & 0.76 / 51.2\% & 1.20 / 70.3\% & 2.05 / 97.8\% \\
TdCE & 0.94 / 44.9\% & 0.69 / 35.9\% & 0.47 / 31.1\% & 1.12 / 54.7\% & 2.17 / 85.8\% & 2.10 / 92.9\% \\
WeE & 2.27 / 90.1\% & 2.11 / 89.9\% & 2.51 / 97.1\% & 1.08 / 61.0\% & 1.81 / 83.5\% & 3.01 / 98.7\% \\ \bottomrule
\end{tabular}
}
\end{table}
\begin{table}[]
\centering
\caption{\label{l1_transform_full} Average number of features applied per sample and proportion of transformed samples in ESL English.}
\resizebox{\textwidth}{!}{%
\begin{tabular}{llcccccc}
\toprule
& \textbf{} & MMLU & ARC & TruthfulQA & GSM8K & Hellaswag & WinoGrande \\ \midrule
\multirow{10}{*}{A} & ar & 2.65 / 96.6\% & 2.77 / 98.8\% & 2.05 / 92.6\% & 3.06 / 99.5\% & 2.88 / 98.2\% & 2.87 / 99.8\% \\
& de & 2.17 / 93.4\% & 2.30 / 94.8\% & 1.92 / 91.8\% & 2.15 / 94.9\% & 2.88 / 96.0\% & 2.98 / 99.7\% \\
& es & 3.15 / 97.1\% & 3.50 / 99.7\% & 2.74 / 97.1\% & 3.53 / 99.3\% & 3.55 / 98.3\% & 3.63 / 99.6\% \\
& fr & 1.00 / 84.6\% & 0.99 / 86.6\% & 0.83 / 74.8\% & 1.15 / 87.2\% & 1.16 / 87.6\% & 1.11 / 92.8\% \\
& it & 1.03 / 80.8\% & 1.05 / 87.5\% & 0.75 / 68.1\% & 1.20 / 87.3\% & 1.33 / 89.6\% & 1.19 / 87.7\% \\
& ja & 3.21 / 96.5\% & 3.41 / 98.8\% & 2.54 / 94.2\% & 3.20 / 98.1\% & 3.93 / 98.1\% & 3.83 / 100.0\% \\
& pt & 2.92 / 98.1\% & 3.07 / 99.5\% & 2.89 / 99.4\% & 3.30 / 99.8\% & 3.27 / 98.3\% & 3.36 / 99.9\% \\
& ru & 3.02 / 97.5\% & 3.28 / 99.7\% & 2.85 / 99.0\% & 3.53 / 99.5\% & 3.33 / 98.6\% & 3.56 / 99.9\% \\
& tr & 2.94 / 96.9\% & 3.08 / 97.9\% & 2.34 / 92.9\% & 3.29 / 98.0\% & 3.18 / 97.6\% & 3.22 / 99.9\% \\
& zh & 1.63 / 88.0\% & 1.67 / 90.6\% & 1.23 / 83.6\% & 2.02 / 93.6\% & 1.77 / 89.6\% & 1.84 / 93.2\% \\ \midrule
\multirow{10}{*}{B} & ar & 2.83 / 96.4\% & 2.82 / 98.2\% & 2.09 / 91.5\% & 3.15 / 98.5\% & 2.84 / 98.8\% & 2.89 / 99.2\% \\
& de & 2.09 / 92.3\% & 2.01 / 91.9\% & 1.95 / 91.8\% & 1.96 / 91.5\% & 2.54 / 94.7\% & 2.98 / 99.7\% \\
& es & 3.27 / 97.4\% & 3.43 / 98.9\% & 2.89 / 97.4\% & 3.59 / 98.9\% & 3.30 / 98.8\% & 3.51 / 99.9\% \\
& fr & 0.97 / 83.0\% & 0.91 / 82.3\% & 0.79 / 70.2\% & 1.05 / 84.2\% & 1.01 / 77.8\% & 1.11 / 89.2\% \\
& it & 0.93 / 73.4\% & 0.87 / 73.6\% & 0.65 / 56.2\% & 1.07 / 79.4\% & 1.30 / 87.8\% & 1.21 / 88.5\% \\
& ja & 3.31 / 96.6\% & 3.16 / 98.0\% & 2.53 / 92.4\% & 3.06 / 97.9\% & 3.52 / 98.3\% & 3.82 / 99.8\% \\
& pt & 2.95 / 98.2\% & 3.05 / 99.0\% & 2.91 / 99.0\% & 3.25 / 99.5\% & 2.94 / 97.2\% & 3.29 / 99.8\% \\
& ru & 3.15 / 97.5\% & 3.33 / 99.6\% & 3.01 / 98.6\% & 3.59 / 99.3\% & 3.02 / 97.7\% & 3.52 / 99.9\% \\
& tr & 3.06 / 96.2\% & 2.96 / 96.8\% & 2.24 / 88.2\% & 3.26 / 98.5\% & 3.00 / 98.3\% & 3.19 / 99.9\% \\
& zh & 1.83 / 93.3\% & 1.94 / 97.3\% & 1.44 / 88.3\% & 2.2 / 97.8\% & 1.97 / 96.1\% & 2.04 / 96.6\% \\
\bottomrule
\end{tabular}
}
\end{table}
\begin{thebibliography}{73}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Achiam et~al.(2023)Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman, Almeida, Altenschmidt, Altman, Anadkat, et~al.]{achiam2023gpt}
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia~Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et~al.
\newblock Gpt-4 technical report.
\newblock \emph{arXiv preprint arXiv:2303.08774}, 2023.
\bibitem[Arase et~al.(2022)Arase, Uchida, and Kajiwara]{arase2022cefr}
Yuki Arase, Satoru Uchida, and Tomoyuki Kajiwara.
\newblock Cefr-based sentence difficulty annotation and assessment.
\newblock \emph{arXiv preprint arXiv:2210.11766}, 2022.
\bibitem[Bakker(2024)]{bakker2024pidgins}
Peter Bakker.
\newblock Pidgins and creoles with germanic lexifier languages.
\newblock In \emph{Oxford Research Encyclopedia of Linguistics}. 2024.
\bibitem[Barayan et~al.(2024)Barayan, Camacho-Collados, and Alva-Manchego]{barayan2024analysing}
Abdullah Barayan, Jose Camacho-Collados, and Fernando Alva-Manchego.
\newblock Analysing zero-shot readability-controlled sentence simplification.
\newblock \emph{arXiv preprint arXiv:2409.20246}, 2024.
\bibitem[Bieswanger(2008)]{bieswanger2008varieties}
Markus Bieswanger.
\newblock Varieties of english in current english language teaching.
\newblock \emph{Stellenbosch Papers in Linguistics}, 38\penalty0 (1):\penalty0 27--47, 2008.
\bibitem[Blasi et~al.(2021)Blasi, Anastasopoulos, and Neubig]{blasi2021systematic}
Damian Blasi, Antonios Anastasopoulos, and Graham Neubig.
\newblock Systematic inequalities in language technology performance across the world's languages.
\newblock \emph{arXiv preprint arXiv:2110.06733}, 2021.
\bibitem[Blodgett et~al.(2018)Blodgett, Wei, and O{'}Connor]{blodgett-etal-2018-twitter}
Su~Lin Blodgett, Johnny Wei, and Brendan O{'}Connor.
\newblock {T}witter {U}niversal {D}ependency parsing for {A}frican-{A}merican and mainstream {A}merican {E}nglish.
\newblock In Iryna Gurevych and Yusuke Miyao, editors, \emph{Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pages 1415--1425, Melbourne, Australia, July 2018. Association for Computational Linguistics.
\newblock \doi{10.18653/v1/P18-1131}.
\newblock URL \url{https://aclanthology.org/P18-1131/}.
\bibitem[Chambers and Trudgill(1998)]{chambers1998dialectology}
Jack~K Chambers and Peter Trudgill.
\newblock \emph{Dialectology}.
\newblock Cambridge University Press, 1998.
\bibitem[Clark et~al.(2018)Clark, Cowhey, Etzioni, Khot, Sabharwal, Schoenick, and Tafjord]{clark2018think}
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
\newblock Think you have solved question answering? try arc, the ai2 reasoning challenge.
\newblock \emph{arXiv preprint arXiv:1803.05457}, 2018.
\bibitem[Cobbe et~al.(2021)Cobbe, Kosaraju, Bavarian, Chen, Jun, Kaiser, Plappert, Tworek, Hilton, Nakano, et~al.]{cobbe2021training}
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et~al.
\newblock Training verifiers to solve math word problems.
\newblock \emph{arXiv preprint arXiv:2110.14168}, 2021.
\bibitem[Deas et~al.(2023)Deas, Grieser, Kleiner, Patton, Turcan, and McKeown]{deas2023evaluation}
Nicholas Deas, Jessi Grieser, Shana Kleiner, Desmond Patton, Elsbeth Turcan, and Kathleen McKeown.
\newblock Evaluation of african american language bias in natural language generation.
\newblock \emph{arXiv preprint arXiv:2305.14291}, 2023.
\bibitem[DeepSeek-AI(2025)]{deepseekai2025deepseekr1incentivizingreasoningcapability}
DeepSeek-AI.
\newblock Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
\newblock URL \url{https://arxiv.org/abs/2501.12948}.
\bibitem[Dubey et~al.(2024)]{dubey2024llama3herd}
Abhimanyu Dubey et~al.
\newblock The llama 3 herd of models, 2024.
\newblock arXiv preprint arXiv:2407.21783.
\bibitem[Faisal et~al.(2021)Faisal, Keshava, Anastasopoulos, et~al.]{faisal2021sd}
Fahim Faisal, Sharlina Keshava, Antonios Anastasopoulos, et~al.
\newblock Sd-qa: Spoken dialectal question answering for the real world.
\newblock \emph{arXiv preprint arXiv:2109.12072}, 2021.
\bibitem[Farajidizaji et~al.(2023)Farajidizaji, Raina, and Gales]{farajidizaji2023possible}
Asma Farajidizaji, Vatsal Raina, and Mark Gales.
\newblock Is it possible to modify text to a target readability level? an initial investigation using zero-shot large language models.
\newblock \emph{arXiv preprint arXiv:2309.12551}, 2023.
\bibitem[Fleisig et~al.(2024)Fleisig, Smith, Bossi, Rustagi, Yin, and Klein]{fleisig2024linguistic}
Eve Fleisig, Genevieve Smith, Madeline Bossi, Ishita Rustagi, Xavier Yin, and Dan Klein.
\newblock Linguistic bias in chatgpt: Language models reinforce dialect discrimination.
\newblock \emph{arXiv preprint arXiv:2406.08818}, 2024.
\bibitem[Gaillat et~al.(2022)Gaillat, Simpkin, Ballier, Stearns, Sousa, Bouy{\'e}, and Zarrouk]{gaillat2022predicting}
Thomas Gaillat, Andrew Simpkin, Nicolas Ballier, Bernardo Stearns, Annanda Sousa, Manon Bouy{\'e}, and Manel Zarrouk.
\newblock Predicting cefr levels in learners of english: The use of microsystem criterial features in a machine learning approach.
\newblock \emph{ReCALL}, 34\penalty0 (2):\penalty0 130--146, 2022.
\bibitem[Geertzen et~al.(2013)Geertzen, Alexopoulou, Korhonen, et~al.]{geertzen2013automatic}
Jeroen Geertzen, Theodora Alexopoulou, Anna Korhonen, et~al.
\newblock Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database (efcamdat).
\newblock In \emph{Proceedings of the 31st Second Language Research Forum. Somerville, MA: Cascadilla Proceedings Project}, pages 240--254, 2013.
\bibitem[Google(2025{\natexlab{a}})]{google_gemini25pro_2025}
Google.
\newblock Gemini 2.5 pro.
\newblock \url{https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-pro}, 2025{\natexlab{a}}.
\bibitem[Google(2025{\natexlab{b}})]{google_gemini2flash_2025}
Google.
\newblock Gemini 2.0 flash.
\newblock \url{https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-0-flash}, 2025{\natexlab{b}}.
\bibitem[Gor et~al.(2024)Gor, Daum{\'e}~III, Zhou, and Boyd-Graber]{gor2024great}
Maharshi Gor, Hal Daum{\'e}~III, Tianyi Zhou, and Jordan Boyd-Graber.
\newblock Do great minds think alike? investigating human-ai complementarity in question answering with caimira.
\newblock \emph{arXiv preprint arXiv:2410.06524}, 2024.
\bibitem[Granger(1993)]{granger1993international}
Sylviane Granger.
\newblock The international corpus of learner english.
\newblock \emph{English language corpora: Design, analysis and exploitation}, pages 57--71, 1993.
\bibitem[Grattafiori et~al.(2024)Grattafiori, Dubey, Jauhri, Pandey, Kadian, Al-Dahle, Letman, Mathur, Schelten, Vaughan, et~al.]{grattafiori2024llama}
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et~al.
\newblock The llama 3 herd of models.
\newblock \emph{arXiv preprint arXiv:2407.21783}, 2024.
\bibitem[Guo et~al.(2025)Guo, Yang, Zhang, Song, Zhang, Xu, Zhu, Ma, Wang, Bi, et~al.]{guo2025deepseek}
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et~al.
\newblock Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2501.12948}, 2025.
\bibitem[Gupta et~al.(2024)Gupta, Meng, Yurtseven, O'Brien, and Zhu]{gupta2024aavenue}
Abhay Gupta, Philip Meng, Ece Yurtseven, Sean O'Brien, and Kevin Zhu.
\newblock Aavenue: Detecting llm biases on nlu tasks in aave via a novel benchmark.
\newblock \emph{arXiv preprint arXiv:2408.14845}, 2024.
\bibitem[Hamid and Baldauf~Jr(2013)]{hamid2013second}
M~Obaidul Hamid and Richard~B Baldauf~Jr.
\newblock Second language errors and features of world englishes.
\newblock \emph{World Englishes}, 32\penalty0 (4):\penalty0 476--494, 2013.
\bibitem[Hendrycks et~al.(2020)Hendrycks, Burns, Basart, Zou, Mazeika, Song, and Steinhardt]{hendrycks2020measuring}
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
\newblock Measuring massive multitask language understanding.
\newblock \emph{arXiv preprint arXiv:2009.03300}, 2020.
\bibitem[Hofmann et~al.(2024)Hofmann, Kalluri, Jurafsky, and King]{hofmann2024ai}
Valentin Hofmann, Pratyusha~Ria Kalluri, Dan Jurafsky, and Sharese King.
\newblock Ai generates covertly racist decisions about people based on their dialect.
\newblock \emph{Nature}, 633\penalty0 (8028):\penalty0 147--154, 2024.
\bibitem[Jarvis(2000)]{jarvis2000methodological}
Scott Jarvis.
\newblock Methodological rigor in the study of transfer: Identifying l1 influence in them interlanguage lexicon.
\newblock \emph{Language learning}, 50\penalty0 (2):\penalty0 245--309, 2000.
\bibitem[Jenkins(2015)]{jenkins2015repositioning}
Jennifer Jenkins.
\newblock Repositioning english and multilingualism in english as a lingua franca.
\newblock \emph{Englishes in Practice}, 2\penalty0 (3):\penalty0 49--85, 2015.
\bibitem[Jones(2024)]{jones2024ai}
Nicola Jones.
\newblock Ai now beats humans at basic tasks—new benchmarks are needed, says major report.
\newblock \emph{Nature}, 628\penalty0 (8009):\penalty0 700--701, 2024.
\bibitem[Joshi et~al.(2025)Joshi, Dabre, Kanojia, Li, Zhan, Haffari, and Dippold]{joshi2025natural}
Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold.
\newblock Natural language processing for dialects of a language: A survey.
\newblock \emph{ACM Computing Surveys}, 57\penalty0 (6):\penalty0 1--37, 2025.
\bibitem[Khanuja et~al.(2020)Khanuja, Dandapat, Srinivasan, Sitaram, and Choudhury]{khanuja2020gluecos}
Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury.
\newblock Gluecos: An evaluation benchmark for code-switched nlp.
\newblock \emph{arXiv preprint arXiv:2004.12376}, 2020.
\bibitem[Kortmann et~al.(2020)Kortmann, Lunkenheimer, and Ehret]{ewave}
Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors.
\newblock \emph{eWAVE}.
\newblock 2020.
\newblock URL \url{https://ewave-atlas.org/}.
\bibitem[Law(2024)]{law2024application}
Locky Law.
\newblock Application of generative artificial intelligence (genai) in language teaching and learning: A scoping literature review.
\newblock \emph{Computers and Education Open}, page 100174, 2024.
\bibitem[Liang et~al.(2023{\natexlab{a}})Liang, Bommasani, Lee, Tsipras, Soylu, Yasunaga, Zhang, Narayanan, Wu, Kumar, Newman, Yuan, Yan, Zhang, Cosgrove, Manning, Ré, Acosta-Navas, Hudson, Zelikman, Durmus, Ladhak, Rong, Ren, Yao, Wang, Santhanam, Orr, Zheng, Yuksekgonul, Suzgun, Kim, Guha, Chatterji, Khattab, Henderson, Huang, Chi, Xie, Santurkar, Ganguli, Hashimoto, Icard, Zhang, Chaudhary, Wang, Li, Mai, Zhang, and Koreeda]{liang2023holisticevaluationlanguagemodels}
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce~Zhang, Christian Cosgrove, Christopher~D. Manning, Christopher Ré, Diana Acosta-Navas, Drew~A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang~Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda.
\newblock Holistic evaluation of language models, 2023{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2211.09110}.
\bibitem[Liang et~al.(2023{\natexlab{b}})Liang, Yuksekgonul, Mao, Wu, and Zou]{liang2023gpt}
Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou.
\newblock Gpt detectors are biased against non-native english writers.
\newblock \emph{Patterns}, 4\penalty0 (7), 2023{\natexlab{b}}.
\bibitem[Lin et~al.(2024)Lin, Mao, La~Malfa, Hofmann, de~Wynter, Wang, Chen, Wooldridge, Pierrehumbert, and Wei]{lin2024one}
Fangru Lin, Shaoguang Mao, Emanuele La~Malfa, Valentin Hofmann, Adrian de~Wynter, Xun Wang, Si-Qing Chen, Michael Wooldridge, Janet~B Pierrehumbert, and Furu Wei.
\newblock One language, many gaps: Evaluating dialect fairness and robustness of large language models in reasoning tasks.
\newblock \emph{arXiv preprint arXiv:2410.11005}, 2024.
\bibitem[Lin et~al.(2021)Lin, Hilton, and Evans]{lin2021truthfulqa}
Stephanie Lin, Jacob Hilton, and Owain Evans.
\newblock Truthfulqa: Measuring how models mimic human falsehoods.
\newblock \emph{arXiv preprint arXiv:2109.07958}, 2021.
\bibitem[Luk and Shirai(2009)]{luk2009acquisition}
Zoe Pei-sui Luk and Yasuhiro Shirai.
\newblock Is the acquisition order of grammatical morphemes impervious to l1 knowledge? evidence from the acquisition of plural-s, articles, and possessive’s.
\newblock \emph{Language Learning}, 59\penalty0 (4):\penalty0 721--754, 2009.
\bibitem[Luo et~al.(2025)Luo, Rechardt, Sun, Nejad, Y{\'a}{\~n}ez, Yilmaz, Lee, Cohen, Borghesani, Pashkov, et~al.]{luo2025large}
Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin~K Nejad, Felipe Y{\'a}{\~n}ez, Bati Yilmaz, Kangjoo Lee, Alexandra~O Cohen, Valentina Borghesani, Anton Pashkov, et~al.
\newblock Large language models surpass human experts in predicting neuroscience results.
\newblock \emph{Nature human behaviour}, 9\penalty0 (2):\penalty0 305--315, 2025.
\bibitem[Migdadi et~al.(2020)Migdadi, Yunus, and Al~Garni]{migdadi2020global}
Hamzah~Faleh Migdadi, Kamariah Yunus, and AF~Al~Garni.
\newblock A global view towards understanding of standard and non-standard varieties of english.
\newblock \emph{International journal of academic research in business and social sciences}, 10\penalty0 (2):\penalty0 103--115, 2020.
\bibitem[Mizumoto et~al.(2024)Mizumoto, Yasuda, and Tamura]{mizumoto2024identifying}
Atsushi Mizumoto, Sachiko Yasuda, and Yu~Tamura.
\newblock Identifying chatgpt-generated texts in efl students’ writing: Through comparative analysis of linguistic fingerprints.
\newblock \emph{Applied Corpus Linguistics}, 4\penalty0 (3):\penalty0 100106, 2024.
\bibitem[Murakami and Alexopoulou(2016)]{murakami2016l1}
Akira Murakami and Theodora Alexopoulou.
\newblock L1 influence on the acquisition order of english grammatical morphemes: A learner corpus study.
\newblock \emph{Studies in Second Language Acquisition}, 38\penalty0 (3):\penalty0 365--401, 2016.
\bibitem[of~Europe. Council for Cultural Co-operation. Education Committee. Modern Languages~Division(2001)]{council2001common}
Council of~Europe. Council for Cultural Co-operation. Education Committee. Modern Languages~Division.
\newblock \emph{Common European framework of reference for languages: Learning, teaching, assessment}.
\newblock Cambridge University Press, 2001.
\bibitem[OpenAI(2023)]{openai2023gpt4}
OpenAI.
\newblock Gpt-4 technical report.
\newblock \emph{arXiv preprint arXiv:2303.08774}, 2023.
\newblock URL \url{https://arxiv.org/abs/2303.08774}.
\bibitem[OpenAI(2024)]{openai_gpt4omini_2024}
OpenAI.
\newblock Gpt-4o mini.
\newblock \url{https://platform.openai.com/docs/models/gpt-4o-mini}, 2024.
\bibitem[OpenAI(2025{\natexlab{a}})]{openai2025gpt41mini}
OpenAI.
\newblock Gpt-4.1 mini model card, 2025{\natexlab{a}}.
\newblock URL \url{https://platform.openai.com/docs/models/gpt-4.1-mini}.
\newblock Accessed: 2025-05-16.
\bibitem[OpenAI(2025{\natexlab{b}})]{openai_o4mini_2025}
OpenAI.
\newblock Openai o4-mini.
\newblock \url{https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf}, 2025{\natexlab{b}}.
\bibitem[{Ortiz Su{\'a}rez} et~al.(2019){Ortiz Su{\'a}rez}, Sagot, and Romary]{OrtizSuarezSagotRomary2019}
Pedro~Javier {Ortiz Su{\'a}rez}, Beno{\^i}t Sagot, and Laurent Romary.
\newblock Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures.
\newblock Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 -- 16, Mannheim, 2019. Leibniz-Institut f{\"u}r Deutsche Sprache.
\newblock \doi{10.14618/ids-pub-9021}.
\newblock URL \url{http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215}.
\bibitem[O’Keeffe and Mark(2017)]{o2017english}
Anne O’Keeffe and Geraldine Mark.
\newblock The english grammar profile of learner competence: Methodology and key findings.
\newblock \emph{International Journal of Corpus Linguistics}, 22\penalty0 (4):\penalty0 457--489, 2017.
\bibitem[Profile(2012)]{profile2012english}
English Profile.
\newblock What is english profile?
\newblock \emph{Website: http://www. englishprofile. org/. Accessed: January}, 2012.
\bibitem[Reusens et~al.(2024)Reusens, Borchert, De~Weerdt, and Baesens]{reusens2024native}
Manon Reusens, Philipp Borchert, Jochen De~Weerdt, and Bart Baesens.
\newblock Native design bias: Studying the impact of english nativeness on language model performance.
\newblock \emph{arXiv preprint arXiv:2406.17385}, 2024.
\bibitem[Ruder et~al.(2023)Ruder, Clark, Gutkin, Kale, Ma, Nicosia, Rijhwani, Riley, Sarr, Wang, et~al.]{ruder2023xtreme}
Sebastian Ruder, Jonathan~H Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel~A Sarr, Xinyi Wang, et~al.
\newblock Xtreme-up: A user-centric scarce-data benchmark for under-represented languages.
\newblock \emph{arXiv preprint arXiv:2305.11938}, 2023.
\bibitem[Sakaguchi et~al.(2021)Sakaguchi, Bras, Bhagavatula, and Choi]{sakaguchi2021winogrande}
Keisuke Sakaguchi, Ronan~Le Bras, Chandra Bhagavatula, and Yejin Choi.
\newblock Winogrande: An adversarial winograd schema challenge at scale.
\newblock \emph{Communications of the ACM}, 64\penalty0 (9):\penalty0 99--106, 2021.
\bibitem[Salih(2021)]{salih2021future}
Abdelrahman~Abdalla Salih.
\newblock The future of english and its varieties: An applied linguistic perspective.
\newblock \emph{English Language Teaching}, 14\penalty0 (4):\penalty0 16--24, 2021.
\bibitem[Srirag et~al.(2024{\natexlab{a}})Srirag, Joshi, Painter, and Kanojia]{srirag2024besstie}
Dipankar Srirag, Aditya Joshi, Jordan Painter, and Diptesh Kanojia.
\newblock Besstie: A benchmark for sentiment and sarcasm classification for varieties of english.
\newblock \emph{arXiv preprint arXiv:2412.04726}, 2024{\natexlab{a}}.
\bibitem[Srirag et~al.(2024{\natexlab{b}})Srirag, Sahoo, and Joshi]{srirag2024evaluating}
Dipankar Srirag, Nihar~Ranjan Sahoo, and Aditya Joshi.
\newblock Evaluating dialect robustness of language models via conversation understanding.
\newblock \emph{arXiv preprint arXiv:2405.05688}, 2024{\natexlab{b}}.
\bibitem[Sung et~al.(2024)Sung, Kim, and Nam]{sung2024influence}
Min-Chang Sung, Kitaek Kim, and Bora Nam.
\newblock Influence of topic-prominent l1s on the use of l2 english copula be: a corpus-based study.
\newblock \emph{International Review of Applied Linguistics in Language Teaching}, \penalty0 (0), 2024.
\bibitem[Team et~al.(2024{\natexlab{a}})Team, Georgiev, Lei, Burnell, Bai, Gulati, Tanzer, Vincent, Pan, Wang, et~al.]{team2024gemini}
Gemini Team, Petko Georgiev, Ving~Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et~al.
\newblock Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
\newblock \emph{arXiv preprint arXiv:2403.05530}, 2024{\natexlab{a}}.
\bibitem[Team(2025)]{gemma_2025}
Gemma Team.
\newblock Gemma 3.
\newblock 2025.
\newblock URL \url{https://arxiv.org/abs/2503.19786}.
\bibitem[Team et~al.(2024{\natexlab{b}})Team, Riviere, Pathak, Sessa, Hardin, Bhupatiraju, Hussenot, Mesnard, Shahriari, Ram{\'e}, et~al.]{team2024gemma}
Gemma Team, Morgane Riviere, Shreya Pathak, Pier~Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L{\'e}onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram{\'e}, et~al.
\newblock Gemma 2: Improving open language models at a practical size.
\newblock \emph{arXiv preprint arXiv:2408.00118}, 2024{\natexlab{b}}.
\bibitem[Team(2024)]{qwen2.5}
Qwen Team.
\newblock Qwen2.5: A party of foundation models, September 2024.
\newblock URL \url{https://qwenlm.github.io/blog/qwen2.5/}.
\bibitem[Uchida(2025)]{uchida2025generative}
Satoru Uchida.
\newblock Generative ai and cefr levels: Evaluating the accuracy of text generation with chatgpt-4o through textual features.
\newblock \emph{Vocabulary Learning and Instruction}, 14\penalty0 (1):\penalty0 2078--2078, 2025.
\bibitem[Vicente(2007)]{vicente2007english}
Veruska~Salvador Vicente.
\newblock English-based pidgins and creoles: from social to cognitive hypotheses of acquisition.
\newblock \emph{Revista Virtual de Estudos da Linguagem}, 5\penalty0 (9):\penalty0 1--30, 2007.
\bibitem[Won et~al.(2025)Won, Shin, Kim, and Yoo]{won2025advancing}
Dong-Ok Won, Yu~Kyoung Shin, Ho-Jung Kim, and Isaiah~WonHo Yoo.
\newblock Advancing language assessment with gpt: Is it nonnative-language friendly?
\newblock \emph{Language Assessment Quarterly}, pages 1--28, 2025.
\bibitem[Xue et~al.(2021)Xue, Constant, Roberts, Kale, Al-Rfou, Siddhant, Barua, and Raffel]{xue2021mt5massivelymultilingualpretrained}
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.
\newblock mt5: A massively multilingual pre-trained text-to-text transformer, 2021.
\newblock URL \url{https://arxiv.org/abs/2010.11934}.
\bibitem[Yannakoudakis et~al.(2011)Yannakoudakis, Briscoe, and Medlock]{yannakoudakis2011new}
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock.
\newblock A new dataset and method for automatically grading esol texts.
\newblock In \emph{Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies}, pages 180--189, 2011.
\bibitem[Zellers et~al.(2019)Zellers, Holtzman, Bisk, Farhadi, and Choi]{zellers2019hellaswag}
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.
\newblock Hellaswag: Can a machine really finish your sentence?
\newblock \emph{arXiv preprint arXiv:1905.07830}, 2019.
\bibitem[Zhou et~al.(2025)Zhou, Wan, Gabriel, Li, Gates, Sap, and Hartvigsen]{zhou2025disparities}
Runtao Zhou, Guangya Wan, Saadia Gabriel, Sheng Li, Alexander~J Gates, Maarten Sap, and Thomas Hartvigsen.
\newblock Disparities in llm reasoning accuracy and explanations: A case study on african american english.
\newblock \emph{arXiv preprint arXiv:2503.04099}, 2025.
\bibitem[Zhou(2020)]{zhou2020challenges}
Xuhui Zhou.
\newblock \emph{Challenges in automated debiasing for toxic language detection}.
\newblock University of Washington, 2020.
\bibitem[Ziems et~al.(2022{\natexlab{a}})Ziems, Chen, Harris, Anderson, and Yang]{ziems2022value}
Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang.
\newblock Value: Understanding dialect disparity in nlu.
\newblock \emph{arXiv preprint arXiv:2204.03031}, 2022{\natexlab{a}}.
\bibitem[Ziems et~al.(2022{\natexlab{b}})Ziems, Held, Yang, Dhamala, Gupta, and Yang]{ziems2022multi}
Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang.
\newblock Multi-value: A framework for cross-dialectal english nlp.
\newblock \emph{arXiv preprint arXiv:2212.08011}, 2022{\natexlab{b}}.
\end{thebibliography}