NativQA Framework: Enabling LLMs and VLMs with Native, Local, and Everyday Knowledge
Extracted source text · arXiv 2504.05995
% This must be in the first 5 lines to tell arXiv to use pdfLaTeX, which is strongly recommended.
\pdfoutput=1
% In particular, the hyperref package requires pdfLaTeX in order to break URLs across lines.
\documentclass[11pt]{article}
% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[preprint]{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
\usepackage{booktabs}
\usepackage{soul}
\usepackage{color}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{enumitem,kantlipsum}
\usepackage[normalem]{ulem}
%\usepackage{tabularx}
\usepackage{color,colortbl}
\usepackage{todonotes}
\usepackage{soul}
\usepackage{threeparttable, makecell}
\usepackage{makecell}
% \usepackage[table]{xcolor}
\usepackage{xcolor}
\usepackage{tcolorbox}
\usepackage{comment}
\usepackage{array}
\usepackage{lscape}
\usepackage{multirow}
\usepackage{graphicx}
\usepackage[normalem]{ulem}
\useunder{\uline}{\ul}{}
\usepackage{array}
\usepackage{multirow}
\usepackage{geometry}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{amsmath}
\usepackage{subcaption}
\usepackage{fancybox}
\usepackage{tikz}
% \usepackage[finalizecache,cachedir=.]{minted}
% \usemintedstyle{borland}
\usepackage{listings}
% \lstset{basicstyle=\ttfamily} % <<< This line added
\usepackage{color}
\usepackage{pifont}% http://ctan.org/pkg/pifont
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
\usepackage{enumitem}
\setlist{nosep,topsep=-\parskip}
\usepackage{soul}
\usepackage{xcolor}
\definecolor{dkgreen}{rgb}{0,0.6,0}
\usepackage{arabtex}
\usepackage{utf8}
\setcode{utf8}
\lstset{frame=tb,
language=Java,
aboveskip=3mm,
belowskip=3mm,
showstringspaces=false,
columns=flexible,
basicstyle={\small\ttfamily},
numbers=none,
numberstyle=\tiny\color{gray},
keywordstyle=\color{dkgreen},
commentstyle=\color{dkgreen},
% stringstyle=\color{black},
breaklines=true,
breakatwhitespace=true,
tabsize=3,
escapeinside={`}{`},
breakindent=0pt,
otherkeywords={these}
}
\renewcommand{\UrlFont}{\ttfamily\small}
% \lstset{
% basicstyle=\ttfamily\small,
% breaklines=true,
% aboveskip=4mm,
% belowskip=4mm,
% captionpos=b, % Position the caption at the bottom
% %frame=single, % Adds a frame around the listing
% columns=fullflexible
% }
%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}
\definecolor{lightblue}{rgb}{.50,.95,1}
\definecolor{tri}{rgb}{.25,.88,.82}
\definecolor{lilac}{rgb}{0.85,0.64,0.85}
\newcommand\fa[1]{\hlc[lightblue]{{\bf FA}: #1}}
\newcommand\mh[1]{\hlc[lilac]{{\bf MH}: #1}}
\newcommand{\nqa}{\textbf{\emph{NativQA}}}
\newcommand{\nqaf}{\textbf{\emph{NativQA framework}}}
\newcommand{\mnqa}{Multi\emph{NativQA}}
\newcommand{\code}[1]{\texttt{#1}}
\usepackage{amssymb}
\usepackage{hyperref}
% \usepackage{paralist}
\usepackage{verbatim}
\usepackage{setspace}
\renewcommand{\UrlFont}{\ttfamily\small}
\usepackage{colortbl}
\usepackage{booktabs}
\lstset{
basicstyle=\ttfamily,
breaklines=true,
breakindent=0pt,
aboveskip=4mm,
belowskip=4mm,
captionpos=b, % Position the caption at the bottom
frame=single, % Adds a frame around the listing
columns=fullflexible
}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\title{NativQA Framework: \\Enabling LLMs with Native, Local, and Everyday Knowledge}
\author{
Firoj Alam,\textsuperscript{$^1$}
Md Arid Hasan,\textsuperscript{$^2$}
Sahinur Rahman Laskar,\textsuperscript{$^3$}\\
{\bf
Mucahid Kutlu,\textsuperscript{$^4$}
Kareem Darwish,\textsuperscript{$^1$}
Shammur Absar Chowdhury\textsuperscript{$^1$}
} \\
\textsuperscript{$^1$}Qatar Computing Research Institute, Qatar, \\
\textsuperscript{$^2$}University of New Brunswick, Canada,
\textsuperscript{$^3$}UPES, India,
\textsuperscript{$^4$}Qatar University, Qatar \\
fialam@hbku.edu.qa, arid.hasan@unb.ca \\
\url{https://gitlab.com/nativqa/nativqa-framework}
}
\begin{document}
\maketitle
\begin{abstract}
The rapid advancement of large language models (LLMs) has raised concerns about cultural bias, fairness, and their applicability in diverse linguistic and underrepresented regional contexts. To enhance and benchmark the capabilities of LLMs, there is a need to develop large-scale resources focused on multilingual, local, and cultural contexts. In this study, we propose \nqaf{}, that can seamlessly construct large-scale, culturally and regionally aligned QA datasets in native languages. The framework utilizes user-defined seed queries and leverages search engines to collect location-specific, everyday information. It has been evaluated across 39 locations in 24 countries and in 7 languages, ranging from extremely low-resource to high-resource languages, resulting in over 300K Question Answer (QA) pairs. The developed resources can be used for LLM benchmarking and further fine-tuning. The framework has been made publicly available for the community.
% with license: CC BY-NC-SA 4.0.
\end{list}
\end{abstract}
% \textbf{Content needed for the demo paper}
% \begin{itemize}[noitemsep,topsep=0pt,leftmargin=*,labelsep=.5em]
% \item What problem does the proposed system address?
% \item Why is the system important and what is its impact?
% \item What is the novelty in the approach/technology on which this system is based?
% \item Who is the target audience?
% \item How does the system work?
% \item How does it compare with existing systems?
% \item How is the system licensed?
% \item How was the system evaluated? Were user studies/human evaluation experiments conducted?
% \item Submissions can contain up to 6 pages
% \item Appendices are allowed but are limited to a maximum of 2 pages
% \item most 2.5 minutes screencast video demonstrating the system
% \item A link to a live demo website
% \item Reviewing will be single-blind, so authors do not need to conceal their identity.
% \end{itemize}
\input{sections/intro}
\input{sections/nativqa_framework}
\input{sections/experiments}
\input{sections/features_usecases}
\input{sections/related_work}
\input{sections/conclusions}
\section{Limitations}
\label{sec:limitations}
The proposed \nqaf{} enables the development of datasets with cultural and native information for everyday knowledge. The \nqaf{} currently uses human-in-the-loop processes in different phases such as seed query creation and manual revision of QA pairs. Though it also supports LLMs for seed query generation and QA validation, however, it is important to have a human-in-the-loop setup to verify the LLMs outputs. The framework does not include a built-in annotation platform for manual QA verification. However, any annotation platform (e.g., Pybossa\footnote{\url{https://github.com/Scifabric/pybossa}}) can be adapted for this purpose.
\section*{Ethics Statement}
\label{sec:ethics}
The proposed \nqaf{} does not involve collecting any personally identifiable information. Therefore, we do not foresee any issues that may lead to potential risks.
\bibliography{bib/bibliography}
% \clearpage
% \newpage
\setcounter{table}{0}
\setcounter{figure}{0}
\renewcommand{\thetable}{A\arabic{table}}
\renewcommand{\thefigure}{A\arabic{figure}}
\section*{Appendix}
\label{sec:appendix}
\appendix
\input{sections/appendices}
\end{document}
\section{Introduction}
\label{sec:introduction}
\begin{figure}[!tbh]
\centering
\begin{subfigure}{0.45\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/language_city_donut_reversed.png}
\caption{Language and region coverage.}
\end{subfigure}
\hfill
\begin{subfigure}{0.45\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/topics.png}
\caption{Topics.}
\end{subfigure}
\vspace{-0.2cm}
\caption{The \nqa~\textbf{framework} has been used to collect QA pairs across 39 locations in 24 countries, covering 7 languages. (a) The plot shows city-level data distribution grouped by language—inner circles denote languages, outer wedges represent cities (with country names), sized by sample count. (b) Topics evaluated with the corresponding datasets are also shown.
}
\label{fig:nativqa_dataset_plot}
\vspace{-0.4cm}
\end{figure}
Large Language Models (LLMs) have exhibited cultural bias toward certain cultures, largely due to the predominance of training data in high-resource languages and the limited digital representation of low-resource languages \cite{li2024culturellm,pawar2024survey,durmus2024towards}. Given that most LLMs are trained primarily on English or translated data, which often fails to capture how people naturally speak, ask questions, and express their needs in their native languages and dialects. Since the emergence of LLMs, there has been growing concern about how to develop and benchmark models that perform equitably across diverse languages and cultures~\cite{NEURIPS2024_77f089cd,myung2024blend,kim-etal-2024-click,pawar2024survey}. In addition to cultural representation,\footnote{Cultural ability includes understanding the diversity of cultural elements across different cultures \cite{pawar2024survey}. In the context of cultural QA, it can be defined as question answering that reflects the types of questions likely to be asked and answered by people of a particular culture, either in their native or second language \cite{arora2024calmqa}.} there is also concern about whether LLMs can effectively respond to everyday, long-form, and complex queries that require paragraph-length answers \cite{arora2024calmqa,myung2024blend,hasan2024nativqa}. These concerns have driven efforts to develop multilingual~\cite{ustun2024aya,touvron2023llama2,team2024gemma} and language-specific~\cite{fanar2024,nahin2025titullms,nguyen-etal-2024-seallms} closed and open-source LLMs.
To benchmark and fine-tune models with region- and culture-specific knowledge, there have been efforts to develop language- and culture-specific resources, along with benchmarked and fine-tuned models \cite{chiu2024culturalbench,shi-etal-2024-culturebank,li2024culturellm}. Methodologies involved in these efforts include collecting QA pairs from community question-answering forums \cite{shi-etal-2024-culturebank}, hiring native speakers to write questions and answers \cite{chiu2024culturalbench}, translating content from English into local languages~\cite{myung2024blend}, and using LLMs to generate QA pairs \cite{putri2024can}.
% For benchmarking purposes, the QA formats typically include multiple-choice questions, short-answer questions~\cite{myung2024blend}, long-answer questions~\cite{arora2024calmqa}, and questions with multiple correct answers~\cite{chiu2024culturalbench,hasan2024nativqa}.
A common finding across studies is that while most LLMs perform reasonably well on high-resource languages, they tend to struggle with mid- and low-resource languages~\cite{myung2024blend}. Several studies have demonstrated that fine-tuning with culture-specific knowledge can improve model performance, even in smaller or quantized models~\cite{li2024culturellm,NEURIPS2024_77f089cd,hasan2024nativqa}. These findings suggest that achieving inclusivity in LLMs requires datasets that reflect diverse regions, cultures, and content generated by native users.
One of the major challenges in creating linguistically and culturally rich datasets is the significant manual effort involved. As LLMs continue to advance, the need for scalable methods to build these resources becomes more critical. However, there is currently a lack of tools designed to support the efficient development of such datasets.
% of location- and language-specific resources, which limits the broader community's ability to create culturally relevant datasets.
% As LLMs evolve, there is a growing need for scalable tools to build culturally and geographically relevant datasets. However, current resources often lack support for location- and language-specific development, limiting broader inclusivity and applicability.
% To address this gap, we propose \nqa{}~\textbf{\textit{framework}}--a system designed to help communities seamlessly develop local and culture-specific QA datasets, thereby enhancing LLMs with cultural knowledge and enabling effective benchmarking.
To bridge this gap, we propose \nqaf{}--a system that enables communities to easily build culture-specific QA datasets for enriching and benchmarking LLMs.
% The \nqaf{} takes a seed query (e.g., \textcolor{blue}{\textit{``visit Baladna Farm in Qatar''}}) and a location specifier (e.g., ``qa'') as inputs, and generates a list of QA pairs as output, as illustrated in Figure~\ref{fig:nativqa_pipeline}.
We evaluated \nqaf{} for 39 locations, from 24 countries covering 7 languages and 18 topics, which resulted over 300K data points as shown in Figure \ref{fig:nativqa_dataset_plot}. To summarize, the contributions of our study are as follows.
% \noindent Our contribution in this study is as follows:
\begin{itemize}[noitemsep,topsep=0pt,leftmargin=*,labelsep=.5em]
\item We propose \nqaf{}, for creating culture- and region-specific QA datasets that promote inclusivity in LLMs by enabling community-driven development of diverse, culturally aligned benchmarks.
\item The \nqaf{} offers several key features:
\textit{(i)} seamless integration of both user- and LLM-generated queries to collect QA pairs;
\textit{(ii)} location-agnostic -- users can collect data from any region;
\textit{(iii)} support for multiple search engines;
\textit{(iv)} image search capability;
\textit{(v)} automated removal of duplicate QA pairs;
\textit{(vi)} domain reliability checks; and
\textit{(vii)} an efficient caching mechanism to reduce redundant API calls, leading to cost savings and mitigating time-out issues.
\item We conducted an extensive evaluation to assess its feasibility for broader adoption and demonstrated that it can be effectively used across different locations, cultures, and languages.
\item We have made the \nqaf{} publicly available for the community.
\end{itemize}
\section{\nqaf{}}
\label{sec:nativqa_framework}
In Figure \ref{fig:nativqa_pipeline}, we present the \nqaf{} consisting of three inter-connected modules such as \textit{(i)} Query Collection, \textit{(ii)} QA Collection, and \textit{(iii)} QA Validation. The top part of the figure shows the entire process while the bottom part shows an example from query collection to QA validation. In the below sections we discuss them in details.
\begin{figure*}[!htb]
\centering
\includegraphics[scale=0.2]{figures/nativqa_framework.png}
\vspace{-0.2cm}
\caption{\nqaf{}, demonstrating the entire pipeline, from query collection to the final dataset development process.}
\label{fig:nativqa_pipeline}
\vspace{-0.4cm}
\end{figure*}
\subsection{\code{Query Collection (QC) Module}}
The objective of this module is to collect open-ended queries focused on various \textit{predetermined topics} derived from common knowledge in everyday communication. The set of topics should be established prior to beginning the query collection process, as this facilitates the gathering of topic-specific queries. As shown in Figure~\ref{fig:nativqa_dataset_plot}, the framework has been evaluated using a predetermined set of 18 topics that are culture- or region-dependent (e.g., Events, Literature, etc.). However, the topics can be anything and can be task specific. The query collection process can be carried out using three different approaches:
\begin{itemize}[noitemsep,topsep=0pt,leftmargin=*,labelsep=.5em]
\item \textbf{Manual query collection:} Involves asking annotators to write topic-specific queries they might pose when seeking everyday information.
\item \textbf{Template-based generation:} Defines templates for different topics with a placeholder for the location. For example, \textit{``main cultural festivals in [LOCATION]''}, where LOCATION can be a city, country, or region name.
\item \textbf{LLM-based generation:} Uses LLMs to generate queries.
\end{itemize}
% \textit{(i)} asking annotators to write domain/topic-specific queries that they might ask when seeking everyday information, \textit{(ii)} using template based approach, and \textit{(iii)} using LLMs to generate such queries.
The manual query collection process requires recruiting location- or region-specific annotators. Moreover, like any other annotation task, it is time-consuming.
% and requires monitoring for both progress and quality.
For example, in \citet{hasan2024nativqa}, country-specific annotators were hired to manually write topic specific queries.
% based on a predefined list of 18 topics mentioned earlier.
In contrast, the latter approaches are more cost- and time-effective.
The template-based approach involves defining location-agnostic templates, which can be seamlessly adapted for different locations. The LLM-based approach requires prompt engineering to generate appropriate queries. We use the term \textit{seed query} ($Q_0$) to refer to the queries generated by all approaches. These queries are then used in later steps by the QA collection module to gather QA pairs and additional queries.
\noindent \textbf{Filtering:} Given that queries can have exact and/or near duplicates, it is important to remove such duplicates as part of the filtering process. The \textit{filtering} module handles this by using exact matching for exact duplicates and a similarity-based approach for near duplicates. These steps results in the \textit{\textbf{final set of queries}}.
\subsection{\code{QA Collection (QAC) Module}}
The next step is to collect QA pairs using a search engine, e.g., Google. Most search engines support features like ``People also ask'' or ``Related queries,'' which list questions asked by real users that are potentially relevant to the initial seed query, as shown in Figure~\ref{fig:google_example}. Moreover, these questions are accompanied by answers extracted by the search engine, along with links to the sources of the answers.
% \begin{figure}[]
% \centering
% \includegraphics[width=0.7\columnwidth]{figures/search_example.png}
% \caption{QA list in response to a query obtained from Google.}
% \vspace{-0.3cm}
% \label{fig:google_example}
% \vspace{-0.4cm}
% \end{figure}
The \code{QA collection module} implements Algorithm~\ref{alg:collect_qa_pairs} (outlined in Appendix). It takes the seed queries $Q_0$ and the number of iterations $N_{\text{iter}}$ as input. For each iteration $i \in N_{\text{iter}}$, it collects QA pairs $P_{QA}^i$ and related queries $S^i_{\text{rel}}$ for each query $q \in Q$, then passes them to the \code{filtering module} and updates the current query set $Q$. This process is repeated for all iterations to obtain the \code{final QA set}, $S_{QA}$, for the enriched queries $Q$. The idea of the \code{filtering module} is to remove duplicates that may arise in different iterations.
The \code{QA collection module} leverages search engine APIs to collect QA pairs, specifying \textit{location} and \textit{language} parameters in the API call. These parameters define the search criteria. Location parameter is especially important, as the focus is on collecting QA pairs from specific regions. The module also stores relevant metadata (e.g., answer source URL) for later use.
% \begin{algorithm}[]
% \footnotesize
% \caption{\footnotesize Collecting QA pairs using seed queries $\varrho_0$. $P_{QA}^i$: QA pair, $S\varrho_{rel}^i$: related queries. ExtractQA(*) and ExtractRelatedQueries (*) are functions that return questions, $Q$-answers, $A$ pairs with attribution $L$, and related queries, respectively, which are obtained from the search engine for a given query, $q$. DeDuplication (*) removes any duplicate entries from the set to ensure uniqueness.}
% \label{alg:collect_qa_pairs}
% \begin{algorithmic}[1]
% \State \textbf{Input:}
% \State \hspace{\algorithmicindent} Seed queries: \( \varrho_0 = \{\hat{\varrho_1}, \hat{\varrho_2}, \ldots, \hat{\varrho_m}\} \)
% \State \hspace{\algorithmicindent} Number of iterations: \( N_{iter} \)
% \State \textbf{Output:}
% \State \hspace{\algorithmicindent} Set of QA pairs: \( S_{QA} \)
% \State \hspace{\algorithmicindent} Set of enriched queries: \( S\varrho \)
% \State $S_{QA} \gets \emptyset$
% \State $S\varrho \gets \varrho_0$
% \For {$i$ from 1 to $N_{iter}$}
% \State $P_{QA}^i \gets \emptyset$
% \State $S\varrho_{rel}^i \gets \emptyset$
% \For {$q \in S\varrho$}
% \State $(Q^q, A^q, L^q) \gets \text{ExtractQA}(q)$
% \State $P_{QA}^i \gets P_{QA}^i \cup \{(q', a', l') \mid q' \in Q^q, a' \in A^q, l' \in L^q\}$
% \State $S\varrho_{rel}^i \gets S\varrho_{rel}^i \cup \text{ExtractRelatedQueries}(q)$
% \EndFor
% \State $P_{QA}^i \gets \text{DeDuplication}(P_{QA}^i)$
% \State $S_{QA} \gets S_{QA} \cup P_{QA}^i$
% \State $S\varrho \gets S\varrho \cup S\varrho_{rel}^i$
% \EndFor
% \State \textbf{return} $S_{QA}, S\varrho$
% \end{algorithmic}
% \vspace{-0.1cm}
% \end{algorithm}
\subsection{\code{QA Validation (QAV) Module}}
The next step of the framework is to validate the QA pairs. It is important for several reasons: \textit{(i)} the retrieved related queries might be less relevant, \textit{(ii)} answers might be incomplete, \textit{(iii)} the source of the answer can be less reliable. For the \code{QAV} the framework integrates the below two approaches.
\noindent \paragraph{1. Domain Reliability Checking (DRC).} The answers collected by \nqaf{} include a link to the source web page from which the answer was extracted. Therefore, the aim of the DRC is to retain QA pairs based on the reliable web domain where the answer appears. We hypothesize that answers from pages hosted on reliable domains are more likely to be trustworthy and can be directly included in the dataset. The framework currently supports two options for DRC:
\begin{itemize}[noitemsep,topsep=0pt,leftmargin=*,labelsep=.5em]
\item Based on the domains of the QA pairs listed in $S_{QA}$, annotators manually check and label them as \textit{very reliable}, \textit{partially reliable}, \textit{not sure}, or \textit{completely unreliable}. For this task, the annotation guidelines proposed in \citet{hasan2024nativqa} can be adopted.
\item In the current implementation, \nqaf{} provides $2,080$ source URLs obtained from the \textit{MultiNativQA} dataset~\cite{hasan2024nativqa}, where annotators identified them as very reliable.
These domain URLs used to verify the reliability of the answers.
\end{itemize}
This approach is more practical and scalable for developing large-scale QA pairs, as it reduces the manual effort required to obtain hand-curated answers. However, due to the semi-supervised nature of the approach, it has certain limitations. Some domains are authentic (e.g., Google, Facebook); however, the content they host might not be reliable due to the questionable user-generated content.
\noindent \paragraph{QA Annotation (QAA).}
To increase more reliability of the curated QA pairs, different steps can be taken. Currently \nqaf{} supports two different approaches:
\begin{itemize}[noitemsep,topsep=0pt,leftmargin=*,labelsep=.5em]
\item \textbf{Manual annotation:} This approach involves manually reviewing and editing the answers. It includes three types of annotations.
\textit{(i)} \textit{Question validation:} Human annotators assess the quality of the questions by assigning one of the following labels: good or bad. A ``good question'' is defined as a fact-seeking question that can be answered with an entity or an explanation. In contrast, a ``bad question'' is either ambiguous, incomprehensible, based on a clear false presupposition, opinion-seeking, or does not seek factual information.
\textit{(ii)} \textit{Answer editing:} If a given answer does not fully address the question, or includes extra or incorrect information, the annotator must edit the answer using content from the source web page to ensure completeness and accuracy. Annotators can only use the provided source web pages to maintain the scope and the reliability of the answers.
\textit{(iii)} \textit{Location relevance:} This task involves determining whether the question is relevant to the specified [LOCATION].
\item \textbf{LLM-based annotation:} While manual annotation by humans is more reliable, it is also time-consuming and costly. To reduce such effort, \nqaf{} supports the use of LLM-based approaches as models like GPT-4 have demonstrated effectiveness in improving task-specific model performance~\cite{li-etal-2023-synthetic,li2024data}.
In the LLM-based approach, a concise version of the annotation guidelines \cite{hasan2024nativqa} and task definitions is adapted into the prompt to check the question, edit the answer, and identify location relevance.
\end{itemize}
\subsection{Executing the Framework}
\label{sec:framework:running}
Once the seed queries are curated, running \nqaf{} is straightforward -- executed with a single command and customizable parameters.
It reads the \textit{seed queries} from the specified input file (\code{--input\_file <$file$>}) and, for each query, calls the search API with additional parameters such as \code{--country\_code <$c$>}, \code{--location <$l$>}, and the number of iterations $n$ (\code{--limit <$n$>}). An example command is provided below. The \nqaf{} is equipped with other scripts (e.g., domain reliability checking), which are detailed as a part of the framework's documentation.
\begin{lstlisting}[language=bash]
$python -m nativqa --engine google --search_type text --input_file <data/test_query.csv> --country_code qa --location <"Doha, Qatar"> --env <envs/api_key.env> --n_iter n
\end{lstlisting}
\section{Evaluation of \nqaf{}}
\label{sec:experiments}
\noindent\textbf{Evaluated in multiple studies.}
The \nqaf{} used in \cite{hasan2024nativqa} to collect data from 9 locations in 7 languages, where seed queries and QA validation performed manually. This effort resulted in approximately 64K annotated QA pairs.
\citet{mousi-etal-2025-aradice} used the framework for Arabic cultural and dialectal QA data collection to benchmark the cultural and dialectal capabilities of LLMs.
\noindent\textbf{Evaluation through additional data collection.} We further evaluated the framework for additional 30 locations by creating seed query templates. We developed Arabic and English seed query templates from the seed queries that were used to collect the \textit{MultiNativQA dataset}~\cite{hasan2024nativqa}. We selected seed queries that contain mentions of locations in the \textit{MultiNativQA}, which were then used with the \textit{[LOCATION]} template (e.g., \textcolor{blue}{\textit{women's wedding attire in [LOCATION]}}). The \textit{[LOCATION]} template was subsequently replaced with other locations to create seed queries for different regions. We manually validated the seed queries generated from the template and removed those that were specific to a particular location (e.g., \textcolor{blue}{\textit{1971 liberation war in Bangladesh/[LOCATION]}}). For Arabic, we used only the seed queries from the template that were employed to collect QA data for Qatar, while the English queries were adopted from the manual seed queries for both Bangladesh and New York.
In Appendix~\ref{sec:app_additional_dataset}, we discuss the details of the data distribution (over 300K) that have been collected using the \nqaf{}.
% The reported distribution includes datasets from \textit{MultiNativQA} as well as additional datasets we collected to evaluate the efficacy of the framework.
% \begin{figure}[t]
% \centering
% \includegraphics[scale=0.8]{figures/full_city_lang_dist.pdf}
% \caption{Distribution of the dataset collected using \nqa{} framework across different languages, and locations.}
% \vspace{-0.4cm}
% \label{fig:nativqa_dataset_plot}
% \vspace{-0.4cm}
% \end{figure}
% \noindent \textbf{Capitalizing LLMs.} In the \nqaf{}, we used LLMs in two stages: \textit{(i)} query generation and \textit{(ii)} QA validation.
% \noindent \textbf{Query generation.} For query generation, we designed prompts specifying the location and topic, and used LLMs to generate queries. To assess the quality of the generated queries, we manually reviewed a random sample and found that they were reasonable and reflected the specified location and topic. Notably, most of them matched human-generated queries at the surface level. Generating a reasonable set of queries requires some prompt engineering. We will release the scripts with prompts used as part of the framework.
% % (as authors of the paper)
% % \todo{I think we should express it differently. So we were the reviewers of our work and we found it reasonable. It will be good if we can provide some reason why we found it reasonable.}
% % Generating a reasonable set of queries requires some prompt engineering. We will release the prompt used as part of the framework.
% % \todo{"will release" or "release"?} -- will release
% \noindent \textbf{QA validation.} For QA validation, we manually evaluated the output of LLMs on 500 QA pairs collected from Egypt. We only selected due to the availability of annotators from Egypt. The goal was to assess whether human annotators preferred the LLM-edited answers or the ones retrieved from a search engine. We provided clear instructions to the annotators to select the better answer based on accuracy, clarity and readability, as well as completeness and relevance. Annotators were given three options: \textit{(i)} \textit{Answer 1} (our original answer), \textit{(ii)} \textit{LLM-edited answer}, and \textit{(iii)} \textit{neither}. We recruited two annotators who are native Arabic speakers residing in Egypt. They were compensated at a standard rate managed by a third-party company. We computed inter-annotator agreement using observed agreement, Gwet's AC1 \cite{gwet2008computing}, and Cohen’s Kappa. The observed agreement was 0.842, indicating strong overall alignment. Gwet's AC1 was 0.806, while Cohen’s Kappa was relatively low ($\kappa$ = 0.147), reflecting the known limitation of Kappa in skewed distributions—since annotators selected the \textit{LLM-edited answer} in the majority of cases, Cohen's method underestimates agreement when one class dominates. The high observed agreement suggests that using LLMs to edit answers may be a promising direction.
% For both query generation and QA validation, we used GPT-4o (version: 2024-11-20). However, any models can be used for such tasks.
\noindent \textbf{Capitalizing LLMs.} In the \nqaf{}, we used LLMs in two stages: \textit{(i)} query generation and \textit{(ii)} LLM-based QA annotation. The use of LLMs for such tasks involves prompting them to produce the desired outcomes. We will release the prompts and scripts as part of the framework.
\noindent \textbf{LLM-based Query generation.} For query generation, we designed prompts specifying the location and topic, and used LLMs to generate queries. To assess the quality of the generated queries, \textcolor{black}{we manually reviewed a random sample and found that they were acceptable to human annotators (as they matched human-generated queries) and reflected the specified location and topic.}
% Notably, most of them matched human-generated queries at the surface level.
% Generating a reasonable set of queries requires some prompt engineering.
% We will release the scripts with prompts used as part of the framework.
% (as authors of the paper)
% \todo{I think we should express it differently. So we were the reviewers of our work and we found it reasonable. It will be good if we can provide some reason why we found it reasonable.}
% Generating a reasonable set of queries requires some prompt engineering. We will release the prompt used as part of the framework.
% \todo{"will release" or "release"?} -- will release
\noindent \textbf{LLM-based QA annotation.} For LLM-based QA annotation, a concise version of the annotation guidelines and task definitions is adapted to the prompt to validate the question, edit the answer, and identify the location relevance.
% \footnote{We will release the scripts with prompts as part of the framework.}
To manually assess the quality of the annotated QA pairs, we choose two different approaches: \textit{(i)} QA validation by selecting the best answer,
% for LLM-based QA annotation
and \textit{(ii)} QA annotation for LLM-based QA annotation.
\begin{itemize}[noitemsep,topsep=0pt,leftmargin=*,labelsep=.5em]
\item \textbf{Manual QA validation - selecting best answer.} For QA validation for LLM-based QA annotation, we manually evaluated the output of LLMs on 500 QA pairs collected from Egypt. We selected this set due to the availability of annotators from Egypt. The goal was to assess whether human annotators preferred the LLM-edited answers or the ones retrieved from a search engine. We provided clear instructions to the annotators to select the better answer based on accuracy, clarity and readability, as well as completeness and relevance (see in Appendix \ref{app:qa_validation}). Annotators were given three options: \textit{(i)} \textit{Answer 1} (original answer), \textit{(ii)} \textit{LLM-edited answer}, and \textit{(iii)} \textit{neither}. We recruited two annotators who are native Arabic speakers residing in Egypt. They were compensated at a standard rate managed by a third-party company. We computed inter-annotator agreement using observed agreement and Gwet's AC1 \cite{gwet2008computing}, and Cohen’s Kappa. The observed agreement and Gwet's AC1 were 0.842 and 0.806, respectively, indicating strong overall alignment.
% and Gwet's AC1 was 0.806, while Cohen’s Kappa was relatively low ($\kappa$ = 0.147), reflecting the known limitation of Kappa in skewed distributions—since annotators selected the \textit{LLM-edited answer} in the majority of cases, Cohen's method underestimates agreement when one class dominates.
The high observed agreement suggests that using LLMs to edit answers may be a promising direction.
\item \textbf{Manual QA validation - answer scoring.}
% QA annotation for LLM-based QA editing.}
%\todo[inline]{Discuss about new annotation}
%and the quality of the LLM-generated answers for QA pairs.
% For editing the answers, if a given answer does not fully address the question or includes additional or incorrect information, the annotators must edit the answer based on the content of the source web page. This step ensures the completeness and accuracy of the answer. We follow the same annotation guideline described by \citet{hasan2024nativqa}.
This approach involves manually checking the quality of LLM-edited answers and scoring them. We used a 5-point Likert scale for various evaluation metrics, including \textit{clarity}, \textit{faithfulness}, \textit{informativeness}, and \textit{plausibility}. These metrics are selected from various related studies on the evaluation of natural language explanations \cite{huang-etal-2024-chatgpt, zavolokina2024think}. We provide detailed annotation guidelines for human evaluators in Appendix \ref{app:verifying_quality}. Due to the availability of native human annotators, the evaluation was carried out for two locations, such as Sudan and Yemen, on $\sim$350 examples from each test set. Each LLM-edited answer was annotated by three and two human evaluators for Sudan and Yemen, respectively.
%\todo[inline]{need to compute agreement index score for Yemen and Sudan}
We presented the average Likert-scale scores for all the evaluation metrics in Table \ref{tab:annotator_likert_score}.
% For each QA pair, we first calculate the average across annotators and then across all QA pairs.
Furthermore, we measured annotation agreement on ordinal scales using the agreement index $r^*_{wg(j)}$ \cite{james1984estimating}, which evaluates how the observed rating variance compares to the maximum variance expected under total disagreement. The values over 0.85 for Sudan and 0.86 for Yemen indicate a strong agreement~\cite{o2017overview}, as shown in Table \ref{tab:annotator_likert_score}.
In addition, we analyze the annotations and find that 100\% %316
answers were not edited by annotators for the Sudan data.
% any of the three annotators, while the remaining 34 answers were not edited by two annotators for the Sudan data.
For Yemen, the percentage of not edited answer was 92\%.
% 233 and 89 answers were not edited by two and one annotators, respectively, while the remaining 28 answers were edited by both annotators.
Both human annotation results and analysis strongly suggest that LLMs are highly capable of QA annotation in the NativQA framework.
\end{itemize}
\begin{table}[!ht]
\centering
\setlength{\tabcolsep}{1pt}
\scalebox{0.62}{
\begin{tabular}{lrrrr}
\toprule
\textbf{Location} & \textbf{Clarity} & \textbf{Faithfulness} & \textbf{Informativeness} & \textbf{Plausibility} \\ \midrule
\multicolumn{5}{c}{\textbf{Average Likert score}} \\ \midrule
Khartoum, Sudan & 4.41 & 4.42 & 4.26 & 4.29 \\ \midrule
Sanaa, Yemen & 4.05 & 4.05 & 3.98 & 4.06 \\\midrule
\multicolumn{5}{c}{\textbf{Agreement index} $r^*_{wg(j)}$} \\ \midrule
Khartoum, Sudan & 0.85 & 0.85 & 0.85 & 0.85 \\ \midrule
Sanaa, Yemen & 0.88 & 0.86 & 0.89 & 0.88 \\
% Tunis, Tunisia & 2.04 & 2.02 & 2.01 & 2.02 \\
\bottomrule
\end{tabular}
}
\vspace{-0.3cm}
\caption{\textit{Average Likert scale} and \textit{agreement} scores for each human evaluation metric for LLM-edited QA.}
\label{tab:annotator_likert_score}
\vspace{-0.3cm}
\end{table}
For both query generation and LLM-based QA annotation, we used GPT-4o (version: 2024-11-20). However, any models can be used for such tasks.
% \begin{table}[!ht]
% \centering
% \setlength{\tabcolsep}{1pt}
% \scalebox{0.70}{
% \begin{tabular}{lrrrr}
% \toprule
% \textbf{Location} & \textbf{Clarity} & \textbf{Faithfulness} & \textbf{Informativeness} & \textbf{Plausibility} \\ \midrule
% Khartoum, Sudan & 0.85 & 0.85 & 0.85 & 0.85 \\ \midrule
% Sanaa, Yemen & 0.88 & 0.86 & 0.89 & 0.88 \\
% \bottomrule
% \end{tabular}
% }
% \caption{Annotation agreement for each evaluation metric for Sudan and Yemen.}
% \label{tab:annotator_likert_score_agreement}
% %\vspace{-0.1cm}
% \end{table}
\section{Features of \nqaf{}}
\label{sec:features}
\nqaf{} features a generic framework that is agnostic to both location and language. It enables a scalable regional and language data collection process while offering simplicity in implementation and flexibility in customization.
\noindent\textbf{Modularity.}
The \nqaf{}, as shown in Figure \ref{fig:nativqa_pipeline}, follows loosely coupled design principles, effectively separating the seed query collection, QA curation, and QA validation components. Their interactions ensure a modular and flexible architecture.
\noindent\textbf{Generality.}
The framework is designed to offer generality, allowing effortless customization of location, language, preferred search engines, and modalities. It supports both text- and image-based search functionalities. Data, along with metadata, are stored in JSON and JSONL formats. Overall, the framework provides a high level of generality and can be readily applied and adapted to a wide range of use cases and research scenarios.
\noindent\textbf{Caching.}
One of the significant challenges when accessing APIs is managing timeouts and other failure issues. This often requires re-running experiments involving API calls, which not only demands additional effort but also increases costs. To address this problem, we have developed a caching mechanism that allows users to bypass API calls for queries that have already been successfully processed. Specifically, all intermediate outputs are saved when processing a data sample. During a re-run, samples with cached model responses do not access the API.
% but instead load the response from the cache.
% \subsection{Language-agnostic}
% The \nqaf{} is language-agnostic.
\noindent\textbf{Open-source.}
We made \nqaf{} accessible to the community by releasing it as open-source. This will also enable the continued development of the framework within the community.
\section{Related Work}
\label{sec:related work}
% In this section, we mainly discuss existing works in two subsections: cultural knowledge in LLMs and different approaches to dataset development.
% \todo[inline]{@Sahinur? Please organize the text based on the sub-sections}
\paragraph{Local and Cultural Knowledge in LLMs.}
LLMs excel on standard NLP benchmarks~\cite{achiam2023gpt,touvron2023llama2,bubeck2023sparks,bang2023multitask,ahuja2023mega,hendy2023good,abdelali-etal-2024-larabench}, yet often struggle with tasks requiring culturally grounded knowledge, especially in low-resource languages and dialects~\cite{pawar2024survey}. This gap highlights the need for more inclusive models. Recent studies demonstrate the importance of enabling LLMs to understand and generate content reflective of diverse linguistic forms, social norms, and local experiences~\cite{NEURIPS2024_77f089cd,li2024culturellm,shi-etal-2024-culturebank}. A culturally inclusive model should be capable of understanding topics like healthcare, education, or cuisine within local contexts. However, current LLMs frequently fall short in capturing region-specific expressions and indigenous knowledge~\cite{myung2024blend,chiu2024culturalbench}.
% LLMs have consistently demonstrated strong performance across a wide range of NLP tasks, with numerous efforts evaluating their capabilities on standard benchmarks~\cite{achiam2023gpt,touvron2023llama2,bubeck2023sparks,bang2023multitask,ahuja2023mega,hendy2023good,abdelali-etal-2024-larabench}. However, despite their scale and generalization abilities, LLMs often underperform on tasks that require culturally or locally grounded knowledge, particularly in low-resource languages and dialectal varieties~\cite{pawar2024survey}. This limitation has sparked critical questions around the need for more culturally inclusive models. Recent work emphasizes the importance of enhancing LLMs to understand, reason about, and generate content grounded in everyday experiences, linguistic forms, and societal norms of diverse user communities~\cite{NEURIPS2024_77f089cd,li2024culturellm,shi-etal-2024-culturebank}. A culturally inclusive model should be capable of interpreting how topics such as healthcare, education, or cuisine are framed in local contexts across different languages and dialects. Yet, current models often fall short in medium- to low-resource settings due to limited exposure to region-specific expressions, indigenous knowledge, and local conversations ~\cite{myung2024blend,chiu2024culturalbench}.
\paragraph{Cultural, Information-Seeking, and Multilingual Resources.}
Focusing on cultural aspects, there have been efforts to develop datasets that benchmark and fine-tune LLMs for diverse languages and regions~\cite{arora2024calmqa,shi-etal-2024-culturebank,NEURIPS2024_77f089cd}. In addition to cultural knowledge, there have also been initiatives to build resources that reflect everyday information-seeking behavior~\cite{myung2024blend}.
Efforts targeting everyday information-seeking queries include the development of mono- and multilingual resources by collecting data from diverse sources such as Wikipedia~\cite{yang-etal-2018-hotpotqa, kwiatkowski-etal-2019-natural}, Google Search QA~\cite{khashabi-etal-2021-gooaq-open}, Reddit forums~\cite{fan-etal-2019-eli5}, and question–answer pairs written directly by native speakers~\cite{clark-etal-2020-tydi}. Other approaches include combining native and machine-translated content~\cite{arora2024calmqa}, and generating QA using LLMs~\cite{putri2024can,NEURIPS2024_77f089cd}.
% longpre-etal-2021-mkqa,asai-etal-2021-xor
% However, cultural knowledge is also an essential domain for improving LLMs' abilities to reason and greatly influencing their performance in a variety of downstream applications.
% Very limited research has been explored that aligns cultural knowledge in LLM. For example, BLEnD \cite{myung2024blend,shi-etal-2024-culturebank}, CulturePark \cite{DBLP:conf/nips/LiTYW0024}, CulturalBench \cite{DBLP:journals/corr/abs-2410-02677} for LLM that focuses on cultural aspects of multiple languages.
% Notable studies \cite{yang-etal-2018-hotpotqa, kwiatkowski-etal-2019-natural, khashabi-etal-2021-gooaq-open,ekram-etal-2022-banglarqa,wang2023helpsteer} are limited to a single language and Wikipedia online source for evaluating LLMs’ cultural sensitivities. The reason behind the limited research on culturally knowledge-specific LLM evaluation is the lack of a suitable culturally knowledge-specific multilingual-QA dataset and the lack of a suitable framework / approaches for the development of such kind of multilingual-QA cultural-specific dataset.
% \subsection{Approaches/Methods for Dataset Development}
\paragraph{Motivation for \nqaf{}.}
Prior work has advanced the inclusion of native language questions and culturally specific content. Datasets like TyDi QA~\cite{clark-etal-2020-tydi}, BLEnD~\cite{myung2024blend}, and CaLMQA~\cite{arora2024calmqa} broaden cultural coverage, while search engine-based datasets such as NQ~\cite{kwiatkowski-etal-2019-natural}, MS MARCO~\cite{nguyen2016ms}, and GooQA~\cite{khashabi-etal-2021-gooaq-open} offer scalability and real-world relevance, albeit primarily in dominant languages. Despite these advances, a unified framework addressing multilinguality, location specificity, and scalability remains missing. \nqaf{} bridges this gap by enabling large-scale data collection from real-world search queries with culturally grounded, native phrasing in underrepresented languages.
% This makes it a uniquely valuable tool for developing data to train and evaluate LLMs in diverse, localized contexts—something existing datasets only partially achieve.
% The above studies make significant contributions toward incorporating native language questions and local cultural content. Early efforts such as TyDi QA~\cite{clark-etal-2020-tydi}, BLEnD~\cite{myung2024blend}, and CaLMQA~\cite{arora2024calmqa} expanded the coverage of culturally specific knowledge. Search engine-based datasets (e.g., NQ~\cite{kwiatkowski-etal-2019-natural}, MS MARCO~\cite{nguyen2016ms}, and GooQA~\cite{khashabi-etal-2021-gooaq-open}) demonstrated scalability and real-world relevance, although they often focus on a single dominant language. Despite this progress, the field still lacks a unified framework that jointly addresses multilinguality, location specificity, and scalability. \nqaf{} sets itself apart by combining scalability with cultural specificity: it leverages real-world search queries and results to collect data at scale, while explicitly capturing native phrasing and culturally grounded contexts in underrepresented languages. This blend of scale and cultural relevance makes the \textit{NativQA framework} a uniquely valuable tool for data collection that supports the training and evaluation of large language models (LLMs). As LLM development progresses, frameworks like \textit{NativQA} fill a critical gap by providing native QA data essential for truly localized AI -- an aspect only partially addressed by prior datasets.
\section{Conclusions and Future Work}
\label{sec:conclusions}
In this study, we propose the \nqaf{}, which a scalable, language-independent framework to build culturally and regionally aligned QA datasets in native languages. It utilizes search engines capabilities to collect QA based on seed queries that native users ask. Based on the seed queries and target search locale one can collect a large number of QA pairs relatively quickly. The framework has already been used in published studies. We additionally evaluated its utility for another 30 locations from different countries and languages. Its functionality for multi-modalities can enable further research in this area and caching mechanism will help in reducing the search engines API cost. In future we aim to extend \nqaf{} for other modalities such as audio/video and evaluate for more under-representative locations.
\section{QA from Search Engines}
In Figure \ref{fig:google_example}, we present a screenshot of a Google search results page for the query \textit{``visit Baladna Farm in Qatar.''} The ``People also ask'' section contains related questions, which we used to collect question-answer (QA) pairs. These questions were subsequently used as queries to collect additional QA pairs.
% Can we visit Baladna Farm in Qatar?
% Is Baladna public or private?
% Who is the owner of Baladna Farm?
% What is the biggest farm in Qatar?
\begin{figure}[]
\centering
\includegraphics[width=0.95\columnwidth]{figures/search_example.png}
\caption{QA list in response to a query obtained from Google.}
\vspace{-0.1cm}
\label{fig:google_example}
\vspace{-0.2cm}
\end{figure}
\section{Algorithm for QA Collection}
\label{sec:app_algorithm}
\noindent\textbf{Algorithm~\ref{alg:collect_qa_pairs}: Collecting QA Pairs from Search Queries.}
\noindent
For QA collection we report Algorithm~\ref{alg:collect_qa_pairs}. It outlines a bootstrapped approach for collecting question-answer (QA) pairs using an initial set of seed queries $Q_0$. The process is iterative, aiming to expand the query set and extract more QA pairs at each step. The algorithm takes as input a predefined set of seed queries $Q_0$ and the number of iterations $N_{\text{iter}}$.
At each iteration, for every query in the current set \( S\varrho \), we retrieve QA pairs and related queries using the functions \texttt{ExtractQA(*)} and \texttt{ExtractRelatedQueries(*)}, respectively. \texttt{ExtractQA(*)} returns a set of question-answer pairs $P_{QA}^i$ with corresponding attribution labels \( L \), while \texttt{ExtractRelatedQueries(*)} retrieves additional queries that are semantically or topically related to the input query.
Duplicate QA pairs are removed through the \texttt{DeDuplication(*)} function to ensure the uniqueness of the collected data. The new QA pairs are added to the global set $S_{QA}$, and the related queries are incorporated into the query pool \( S\varrho \) for use in subsequent iterations.
This approach enables iterative enrichment of both the query pool and the QA dataset, facilitating broader coverage of topics while maintaining data quality through deduplication.
\begin{algorithm}[]
\footnotesize
\caption{\footnotesize Collecting QA pairs using seed queries $\varrho_0$. $P_{QA}^i$: QA pair, $S\varrho_{rel}^i$: related queries. ExtractQA(*) and ExtractRelatedQueries (*) are functions that return questions, $Q$-answers, $A$ pairs with attribution $L$, and related queries, respectively, which are obtained from the search engine for a given query, $q$. DeDuplication (*) removes any duplicate entries from the set to ensure uniqueness.}
\label{alg:collect_qa_pairs}
\begin{algorithmic}[1]
\State \textbf{Input:}
\State \hspace{\algorithmicindent} Seed queries: \( \varrho_0 = \{\hat{\varrho_1}, \hat{\varrho_2}, \ldots, \hat{\varrho_m}\} \)
\State \hspace{\algorithmicindent} Number of iterations: \( N_{iter} \)
\State \textbf{Output:}
\State \hspace{\algorithmicindent} Set of QA pairs: \( S_{QA} \)
\State \hspace{\algorithmicindent} Set of enriched queries: \( S\varrho \)
\State $S_{QA} \gets \emptyset$
\State $S\varrho \gets \varrho_0$
\For {$i$ from 1 to $N_{iter}$}
\State $P_{QA}^i \gets \emptyset$
\State $S\varrho_{rel}^i \gets \emptyset$
\For {$q \in S\varrho$}
\State $(Q^q, A^q, L^q) \gets \text{ExtractQA}(q)$
\State $P_{QA}^i \gets P_{QA}^i \cup \{(q', a', l') \mid q' \in Q^q, a' \in A^q, l' \in L^q\}$
\State $S\varrho_{rel}^i \gets S\varrho_{rel}^i \cup \text{ExtractRelatedQueries}(q)$
\EndFor
\State $P_{QA}^i \gets \text{DeDuplication}(P_{QA}^i)$
\State $S_{QA} \gets S_{QA} \cup P_{QA}^i$
\State $S\varrho \gets S\varrho \cup S\varrho_{rel}^i$
\EndFor
\State \textbf{return} $S_{QA}, S\varrho$
\end{algorithmic}
\vspace{-0.1cm}
\end{algorithm}
\section{Distribution of Additional Dataset}
\label{sec:app_additional_dataset}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
% Please add the following required packages to your document preamble:
% \usepackage{multirow}
\begin{table*}[]
\centering
\begin{tabular}{clrrrr}
\toprule
\textbf{Language} & \multicolumn{1}{c}{\textbf{Location}} & \multicolumn{1}{c}{\textbf{Train}} & \multicolumn{1}{c}{\textbf{Dev}} & \multicolumn{1}{c}{\textbf{Test}} & \multicolumn{1}{c}{\textbf{Total}} \\ \toprule
\multirow{18}{*}{Arabic} & Abu Dhabi, UAE & 4,896 & 692 & 1,406 & \textbf{6,994} \\
& Algiers, Algeria & 15,198 & 2,149 & 4,364 & \textbf{21,711} \\
& Baghdad, Iraq & 17,910 & 2,533 & 5,143 & \textbf{25,586} \\
& Beirut, Lebanon & 17,748 & 2,510 & 5,096 & \textbf{25,354} \\
& Cairo, Egypt & 8,631 & 1,221 & 2,478 & \textbf{12,330} \\
& Damascus, Syria & 7,902 & 1,117 & 2,269 & \textbf{11,288} \\
& Doha, Qatar & 3,649 & 492 & 988 & \textbf{5,129} \\
& Gaza, Palestinian & 3,975 & 562 & 1,142 & \textbf{5,679} \\
& Khartoum, Sudan & 3303 & 467 & 948 & \textbf{4,718} \\
& Kuwait City, Kuwait & 21,676 & 3,066 & 6,224 & \textbf{30,966} \\
& Manama, Bahrain & 17,599 & 2,489 & 5,053 & \textbf{25,141} \\
& Muscat, Oman & 3,479 & 492 & 999 & \textbf{4,970} \\
& Nouakchott, Mauritania & 13,520 & 1,912 & 3,882 & \textbf{19,314} \\
& Rabat, Morocco & 17,341 & 2,453 & 4,979 & \textbf{24,773} \\
& Riyadh, Saudi Arabia & 4,150 & 587 & 1,191 & \textbf{5,928} \\
& Sanaa, Yemen & 3,373 & 477 & 968 & \textbf{4,818} \\
& Tripoli, Libya & 3,300 & 467 & 947 & \textbf{4,714} \\
& Tunis, Tunisia & 10,352 & 1,464 & 2,972 & \textbf{14,788} \\ \midrule
Assamese & Assam, India & 1,131 & 157 & 545 & \textbf{1,833} \\ \midrule
\multirow{2}{*}{Bangla} & Dhaka, Bangladesh & 7,018 & 953 & 1,521 & \textbf{9,492} \\
& Kolkata, India & 6,891 & 930 & 2,146 & \textbf{9,967} \\ \midrule
\multirow{17}{*}{English} & California, USA & 1,065 & 151 & 306 & \textbf{1,522} \\
& Dhaka, Bangladesh & 4,761 & 656 & 1,113 & \textbf{6,530} \\
& Doha, Qatar & 8,212 & 1,164 & 2,322 & \textbf{11,698} \\
& Florida, USA & 777 & 110 & 223 & \textbf{1,110} \\
& Georgia, USA & 816 & 115 & 234 & \textbf{1,165} \\
& Hawaii, USA & 788 & 112 & 226 & \textbf{1,126} \\
& Illinois, USA & 877 & 124 & 252 & \textbf{1,253} \\
& Massachusetts, USA & 842 & 119 & 242 & \textbf{1,203} \\
& Michigan, USA & 796 & 113 & 228 & \textbf{1,137} \\
& New York, USA & 4,518 & 623 & 1,313 & \textbf{6,454} \\
& North Carolina, USA & 809 & 115 & 232 & \textbf{1,156} \\
& Ohio, USA & 862 & 122 & 247 & \textbf{1,231} \\
& Ontario, Canada & 743 & 105 & 213 & \textbf{1,061} \\
& Pennsylvania, USA & 786 & 111 & 226 & \textbf{1,123} \\
& Quebec, Canada & 720 & 102 & 207 & \textbf{1,029} \\
& Texas, USA & 847 & 120 & 243 & \textbf{1,210} \\
& Washington, USA & 874 & 124 & 251 & \textbf{1,249} \\ \midrule
Hindi & Delhi, India & 9,288 & 1,286 & 2,745 & \textbf{13,319} \\ \midrule
Nepali & Kathmandu, Nepal & -- & -- & 561 & \textbf{561} \\ \midrule
Turkish & Istanbul, Turkey & 3,527 & 483 & 1,218 & \textbf{5,228} \\ \midrule
\textbf{Total} & & \textbf{234,950} & \textbf{33,045} & \textbf{67,863} & \textbf{335,858} \\ \bottomrule
\end{tabular}
\caption{Statistics of the data collected using NativQA Framework. – Only testing split due to limited dataset size.}
\label{tab:data_split}
\end{table*}
In Figure \ref{fig:nativqa_dataset_stat_plot}, we show the distribution of the dataset across major cities and languages. The QA dataset covers seven languages—Arabic, English, Hindi, Bangla, Turkish, Assamese, and Nepali—capturing a rich linguistic landscape spanning 39 global cities. Arabic QA pairs come from a variety of MENA countries, including Egypt, Iraq, Lebanon, and Qatar. English content is drawn from both Western countries (e.g., the USA and Canada) and multilingual regions such as Qatar and Bangladesh. South Asian languages like Hindi, Bangla, Assamese, and Nepali are represented through key urban centers including Delhi, Dhaka, Assam, Kolkata, and Kathmandu.
% This broad linguistic and geographic coverage ensures high cultural diversity, and enables robust evaluation of cross-lingual generalization in real-world, location-aware scenarios.
Moreover, we present the details of the collected data using \nqaf{} in Table \ref{tab:data_split}. We
% followed the same approach described by \citet{hasan2024nativqa} to
split the data into training (70\%), development (10\%), and test (20\%) sets.
% The dataset includes cities from Arabic-speaking countries (e.g., Cairo, Baghdad, Doha), South Asian regions (e.g., Delhi for Hindi, Dhaka for Bangla), and English-dominant areas in North America (e.g., California, Ontario).
\begin{figure}[!htb]
\centering
\includegraphics[scale=0.8]{figures/full_city_lang_dist.pdf}
\caption{Distribution of the dataset collected using \nqa{} framework across different languages, and locations.}
\vspace{-0.4cm}
\label{fig:nativqa_dataset_stat_plot}
\vspace{-0.4cm}
\end{figure}
\section{Annotation Guideline} % for Selecting the Best Answer
\label{sec:app_annotation_guideline}
% \subsection{Manual QA validation for LLM-based QA annotation}
\subsection{Manual QA validation - Selecting Best Answer}
\label{app:qa_validation}
% \subsubsection{Objective}
The purpose of this task is to evaluate two possible answers for a given question and select the one that is more accurate, clear, and well-structured.
Each question will have two answer options, and the task is to choose the better answer based on the evaluation criteria discussed below.
\subsubsection{Annotation Task}
For each question, there are two answer options. The task is to carefully compare them and select the one that best meets the below criteria:
\begin{enumerate}[label=\arabic*.]
\item \textbf{Accuracy.}
The answer must be factually correct and free from misinformation. It should properly address the question and not introduce unrelated details.
\item \textbf{Clarity \& Readability.}
The answer should be grammatically correct, well-structured, and easy to read. It should avoid ambiguity, repetition, or fragmented sentences. The information should be presented in a logical and coherent way.
\item \textbf{Completeness \& Relevance.}
The answer should provide enough relevant information to fully respond to the question. It should avoid unnecessary details that do not contribute to answering the question. If an answer is too short or vague, it may be less useful than a more complete response.
\end{enumerate}
\subsubsection*{Example}
\textbf{Question:}
\begin{RLtext}\footnotesize
ما الفرق بين برج القاهرة وبرج إيفل؟
\end{RLtext}
\textbf{Option 1:} \\
\begin{RLtext}\footnotesize
برج القاهرة بيتكلم في 100 7 وتمانين متر. اطول من برج ايفل برج ايفل فضل اطول برج في العالم. لمدة 40 سنة طوله كام 300 متر ده 300 5 وتمانين متر.
\end{RLtext}
\textbf{Option 2:} \\
\begin{RLtext}\footnotesize
برج القاهرة يبلغ ارتفاعه 187 مترًا، بينما برج إيفل يبلغ ارتفاعه 330 مترًا. برج إيفل كان أطول برج في العالم لمدة 40 عامًا.
\end{RLtext}
\noindent
\textbf{Evaluation:} \\
Option 2 is the better choice because:
\begin{itemize}
\item It provides accurate numerical information about both towers.
\item It is clear and well-structured, while Option 1 contains fragmented and unclear phrasing.
\item It avoids redundant details and presents the comparison concisely.
\end{itemize}
\textbf{Final Selection: Option 2}
\subsubsection{Annotation Instructions}
\begin{itemize}
\item The question and both answer options should be read carefully.
\item Both answers should be evaluated based on the following three main criteria:
\begin{itemize}
\item Accuracy (Is the information correct?)
\item Clarity (Is it well-written and easy to understand?)
\item Completeness \& Relevance (Is the question fully and appropriately answered?)
\end{itemize}
\item The better answer should be selected based on these criteria.
\item If the answers are of equal quality, the one that is more precise and easier to understand should be chosen.
\item If neither answer is acceptable (e.g., both contain major errors), ``neither'' should be selected, and a comment explaining the reasoning should be provided.
\end{itemize}
\subsubsection{Special Cases and Edge Cases}
\begin{itemize}
\item If one answer is factually correct but lacks clarity, while the other is well-written but factually incorrect, the correct answer should be preferred and revised for clarity if necessary.
\item If both answers are equally valid in terms of content, the answer that is clearer and better structured should be selected.
% \item If neither answer is coherent or factually acceptable, the ``Neither'' option should be selected, accompanied by an explanatory comment.
\end{itemize}
% \subsubsection{Special Cases \& Edge Cases}
% \begin{itemize}
% % \item[$\blacktriangleright$] If both answers contain incorrect information → Select ``Neither'' and provide feedback.
% \item[$\blacktriangleright$] If one answer is correct but unclear, and the other is incorrect but well-written → Prefer the correct answer, and edit.
% \item[$\blacktriangleright$] If both answers are equally valid → Select the one that is clearer and more structured.
% \item[$\blacktriangleright$] If neither answer makes sense → Select ``Neither'' and provide feedback.
% \end{itemize}
% \subsection{Verifying the Quality of LLM-edited Answers}
\subsection{Manual QA validation - Answer Scoring.}
\label{app:verifying_quality}
% \subsubsection{Objective}
% The purpose of this task is to verify the LLM-edited answers based on the source URLs and to edit the answers where applicable. Each question will include the LLM-edited answer, along with the source URL. The annotator's task is to verify that the answer is accurate, clear, and well-structured based on the source URL and evaluate the LLM-edited answer according to the evaluation criteria provided below.
The goal of this task is to assess and, if necessary, revise LLM-edited answers using the information provided in the associated source URLs. Each question includes an LLM-generated answer and its corresponding source. Annotators are expected to verify the quality of the answer based on the source content, and evaluate it according to the criteria outlined below.
\begin{figure*}[!htb]
\centering
\includegraphics[scale=0.5]{figures/annotation_platform_llm.png}
\caption{An example of the annotation interface for QA annotation for LLM-based QA annotation.}
%\vspace{-0.4cm}
\label{fig:annotation_interface_llm}
%\vspace{-0.4cm}
\end{figure*}
\subsubsection{Annotation Task}
% We follow the same annotation guideline described by \citet{hasan2024nativqa} to edit the LLM-edited answers.
To assess the quality of LLM-edited answers, we used four different metrics described below. For each metric, assign a score from 1 (very poor) to 5 (excellent), following the descriptions below.
\noindent \textbf{Informativeness.} Measures the extent to which the assessed text (answer) provides relevant, meaningful, complete (covers all important points), correct (factually accurate), and useful information in response to the given prompt or context. Highly informative content (answer) directly addresses key points; low-informative content (answer) is vague, incomplete, incorrect, or misses critical details.
\begin{itemize}
\item 1 = Not informative: Lacks relevant or meaningful content; does not address the question/context; contains major errors or omissions.
\item 2 = Slightly informative: Provides minimal information; many important details are missing, unclear, or inaccurate.
\item 3 = Moderately informative: Some relevant details are present but lack depth, accuracy, or completeness.
\item 4 = Informative: Well-detailed; addresses most key points clearly, correctly, and thoroughly.
\item 5 = Very informative: Exceptionally detailed, insightful, accurate, and complete; fully addresses all aspects of the prompt/context.
\end{itemize}
\noindent \textbf{Clarity.} Assesses how clearly the assessed text (answer) conveys its meaning. Clear text is well-structured, concise (free from unnecessary information), and easy to understand, free from ambiguity, awkward phrasing, or grammatical errors.
\begin{itemize}
\item 1 = Very unclear: Difficult or impossible to understand; contains major grammatical or structural problems; excessively wordy or redundant.
\item 2 = Somewhat unclear: Some meaning is discernible, but significant ambiguity, awkwardness, or unnecessary verbosity remains.
\item 3 = Neutral: Generally understandable but may require effort or contain minor issues, including some unnecessary length.
\item 4 = Clear: Easy to read and understand with minimal ambiguity; well-structured; mostly concise.
\item 5 = Very clear: Highly readable, precise, concise, and effortlessly understandable.
\end{itemize}
\noindent \textbf{Plausibility.} Refers to whether the answer makes sense and is reasonable in the given context. A plausible response is coherent, logical, correct (factually accurate), and consistent with the input (e.g., question, original text, source image/audio).
\begin{itemize}
\item 1 = Not plausible at all: Response is illogical, unrelated, factually incorrect, or clearly incorrect.
\item 2 = Weakly plausible: Some connection, but contains major logical flaws, inaccuracies, or inconsistencies.
\item 3 = Moderately plausible: Partially logical, but some elements are incomplete, questionable, or partially incorrect.
\item 4 = Plausible: Reasonable, makes sense, factually correct, and is mostly consistent with the context.
\item 5 = Highly plausible: Fully logical, coherent, accurate, and strongly aligned with the context.
\end{itemize}
\noindent \textbf{Faithfulness.} Measures how accurately the assessed text reflects or stays true to the source information or reasoning. Faithful responses avoid adding unrelated details, omitting crucial information (completeness), or misrepresenting facts (correctness and honesty).
\begin{itemize}
\item 1 = Not faithful at all: Largely or entirely unrelated to the source/context; misrepresents information; misses key points or adds incorrect information.
\item 2 = Weakly faithful: Some relevant aspects, but much is misleading, incomplete, or missing.
\item 3 = Moderately faithful: Captures some correct elements, but contains errors, omissions, or partial misrepresentation.
\item 4 = Faithful: Accurately represents the main points and intent of the source/context, without major errors or omissions.
\item 5 = Highly faithful: Fully preserves the original meaning/reasoning, covers all important aspects, and is free from errors or misleading information.
\end{itemize}
\subsection{Annotation Instructions}
\begin{itemize}
\item \textbf{Read carefully:} Always review both the context and the text to be assessed as instructed in the platform.
\item \textbf{Apply each metric independently:} Judge informativeness, clarity, plausibility, and faithfulness separately for each item.
\item \textbf{Be objective:} Follow the criteria closely; avoid personal biases or preferences.
\item \textbf{Consider the context:} Assess whether the text meets the needs of the specific task (e.g., Does the answer address the question?).
\item \textbf{Use full scale:} Do not hesitate to assign low or high scores where appropriate. A score of 1 means very poor; 5 means excellent.
\end{itemize}
\subsection{Annotation Interface}
Figure~\ref{fig:annotation_interface_llm} present the annotation interface used by annotators to evaluate the edited answers. The source article, shown on the right side of the interface, is used to verify the accuracy and completeness of the answer.
\section{Release}
\label{apndix:release}
The \nqaf{} is released under the CC BY-NC-SA 4.0 -- Creative Commons Attribution 4.0 International License: \url{https://creativecommons.org/licenses/by-nc-sa/4.0/}.
\begin{thebibliography}{37}
\providecommand{\natexlab}[1]{#1}
\bibitem[{Abdelali et~al.(2024)Abdelali, Mubarak, Chowdhury, Hasanain, Mousi, Boughorbel, Abdaljalil, El~Kheir, Izham, Dalvi, Hawasly, Nazar, Elshahawy, Ali, Durrani, Milic-Frayling, and Alam}]{abdelali-etal-2024-larabench}
Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El~Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024.
\newblock \href {https://aclanthology.org/2024.eacl-long.30/} {{LA}ra{B}ench: Benchmarking {A}rabic {AI} with large language models}.
\newblock In \emph{Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)}, pages 487--520, St. Julian{'}s, Malta. Association for Computational Linguistics.
\bibitem[{Achiam et~al.(2023)Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman, Almeida, Altenschmidt, Altman, Anadkat et~al.}]{achiam2023gpt}
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia~Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et~al. 2023.
\newblock Gpt-4 technical report.
\newblock \emph{arXiv preprint arXiv:2303.08774}.
\bibitem[{Ahuja et~al.(2023)Ahuja, Diddee, Hada, Ochieng, Ramesh, Jain, Nambi, Ganu, Segal, Ahmed, Bali, and Sitaram}]{ahuja2023mega}
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023.
\newblock \href {https://doi.org/10.18653/v1/2023.emnlp-main.258} {{MEGA}: Multilingual evaluation of generative {AI}}.
\newblock In \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, pages 4232--4267, Singapore. Association for Computational Linguistics.
\bibitem[{Arora et~al.(2024)Arora, Karpinska, Chen, Bhattacharjee, Iyyer, and Choi}]{arora2024calmqa}
Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, and Eunsol Choi. 2024.
\newblock {CaLMQA}: Exploring culturally specific long-form question answering across 23 languages.
\newblock \emph{arXiv preprint arXiv:2406.17761}.
\bibitem[{Bang et~al.(2023)Bang, Cahyawijaya, Lee, Dai, Su, Wilie, Lovenia, Ji, Yu, Chung, V.~Do, Xu, and Fung}]{bang2023multitask}
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V.~Do, Yan Xu, and Pascale Fung. 2023.
\newblock A multitask, multilingual, multimodal evaluation of {C}hat{GPT} on reasoning, hallucination, and interactivity.
\newblock In \emph{Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)}, pages 675–--718, Indonesia. Association for Computational Linguistics.
\bibitem[{Bubeck et~al.(2023)Bubeck, Chandrasekaran, Eldan, Gehrke, Horvitz, Kamar, Lee, Lee, Li, Lundberg, Nori, Palangi, Ribeiro, and Zhang}]{bubeck2023sparks}
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin~Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco~Tulio Ribeiro, and Yi~Zhang. 2023.
\newblock \href {https://arxiv.org/abs/2303.12712} {Sparks of artificial general intelligence: Early experiments with {GPT}-4}.
\newblock Technical report, Microsoft Research.
\bibitem[{Chiu et~al.(2024)Chiu, Jiang, Lin, Park, Li, Ravi, Bhatia, Antoniak, Tsvetkov, Shwartz et~al.}]{chiu2024culturalbench}
Yu~Ying Chiu, Liwei Jiang, Bill~Yuchen Lin, Chan~Young Park, Shuyue~Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, et~al. 2024.
\newblock {CulturalBench}: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of llms.
\newblock \emph{arXiv preprint arXiv:2410.02677}.
\bibitem[{Clark et~al.(2020)Clark, Choi, Collins, Garrette, Kwiatkowski, Nikolaev, and Palomaki}]{clark-etal-2020-tydi}
Jonathan~H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.
\newblock \href {https://doi.org/10.1162/tacl_a_00317} {{T}y{D}i {QA}: A benchmark for information-seeking question answering in typologically diverse languages}.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 8:454--470.
\bibitem[{Durmus et~al.(2024)Durmus, Nguyen, Liao, Schiefer, Askell, Bakhtin, Chen, Hatfield-Dodds, Hernandez, Joseph, Lovitt, McCandlish, Sikder, Tamkin, Thamkul, Kaplan, Clark, and Ganguli}]{durmus2024towards}
Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024.
\newblock Towards measuring the representation of subjective global opinions in language models.
\newblock In \emph{First Conference on Language Modeling}.
\bibitem[{Fan et~al.(2019)Fan, Jernite, Perez, Grangier, Weston, and Auli}]{fan-etal-2019-eli5}
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019.
\newblock \href {https://doi.org/10.18653/v1/P19-1346} {{ELI}5: Long form question answering}.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, pages 3558--3567, Florence, Italy. Association for Computational Linguistics.
\bibitem[{Gwet(2008)}]{gwet2008computing}
Kilem~Li Gwet. 2008.
\newblock Computing inter-rater reliability and its variance in the presence of high agreement.
\newblock \emph{British Journal of Mathematical and Statistical Psychology}, 61(1):29--48.
\bibitem[{Hasan et~al.(2024)Hasan, Hasanain, Ahmad, Laskar, Upadhyay, Sukhadia, Kutlu, Chowdhury, and Alam}]{hasan2024nativqa}
Md~Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur~Rahman Laskar, Sunaya Upadhyay, Vrunda~N Sukhadia, Mucahid Kutlu, Shammur~Absar Chowdhury, and Firoj Alam. 2024.
\newblock \href {https://arxiv.org/abs/2407.09823} {{NativQA}: Multilingual culturally-aligned natural query for llms}.
\bibitem[{Hendy et~al.(2023)Hendy, Abdelrehim, Sharaf, Raunak, Gabr, Matsushita, Kim, Afify, and Awadalla}]{hendy2023good}
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young~Jin Kim, Mohamed Afify, and Hany~Hassan Awadalla. 2023.
\newblock How good are {GPT} models at machine translation? a comprehensive evaluation.
\newblock \emph{arXiv preprint arXiv:2302.09210}.
\bibitem[{Huang et~al.(2024)Huang, Kwak, Park, and An}]{huang-etal-2024-chatgpt}
Fan Huang, Haewoon Kwak, Kunwoo Park, and Jisun An. 2024.
\newblock \href {https://aclanthology.org/2024.lrec-main.277/} {{C}hat{GPT} rates natural language explanation quality like humans: But on which scales?}
\newblock In \emph{Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)}, pages 3111--3132, Torino, Italia. ELRA and ICCL.
\bibitem[{James et~al.(1984)James, Demaree, and Wolf}]{james1984estimating}
Lawrence~R James, Robert~G Demaree, and Gerrit Wolf. 1984.
\newblock Estimating within-group interrater reliability with and without response bias.
\newblock \emph{Journal of applied psychology}, 69(1):85.
\bibitem[{Khashabi et~al.(2021)Khashabi, Ng, Khot, Sabharwal, Hajishirzi, and Callison-Burch}]{khashabi-etal-2021-gooaq-open}
Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. 2021.
\newblock \href {https://doi.org/10.18653/v1/2021.findings-emnlp.38} {{G}oo{AQ}: Open question answering with diverse answer types}.
\newblock In \emph{Findings of the Association for Computational Linguistics: EMNLP 2021}, pages 421--433, Punta Cana, Dominican Republic. Association for Computational Linguistics.
\bibitem[{Kim et~al.(2024)Kim, Suk, Oh, Yoo, Thorne, and Oh}]{kim-etal-2024-click}
Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. 2024.
\newblock \href {https://aclanthology.org/2024.lrec-main.296/} {{CLI}c{K}: A benchmark dataset of cultural and linguistic intelligence in {K}orean}.
\newblock In \emph{Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)}, pages 3335--3346, Torino, Italia. ELRA and ICCL.
\bibitem[{Kwiatkowski et~al.(2019)Kwiatkowski, Palomaki, Redfield, Collins, Parikh, Alberti, Epstein, Polosukhin, Devlin, Lee, Toutanova, Jones, Kelcey, Chang, Dai, Uszkoreit, Le, and Petrov}]{kwiatkowski-etal-2019-natural}
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew~M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.
\newblock \href {https://doi.org/10.1162/tacl_a_00276} {Natural questions: A benchmark for question answering research}.
\newblock \emph{Transactions of the Association for Computational Linguistics}, 7:452--466.
\bibitem[{Li et~al.(2024{\natexlab{a}})Li, Chen, Wang, Sitaram, and Xie}]{li2024culturellm}
Cheng Li, Mengzhuo Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024{\natexlab{a}}.
\newblock {CultureLLM}:: Incorporating cultural differences into large language models.
\newblock \emph{Advances in Neural Information Processing Systems}, 37:84799--84838.
\bibitem[{Li et~al.(2024{\natexlab{b}})Li, Teney, Yang, Wen, Xie, and Wang}]{NEURIPS2024_77f089cd}
Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024{\natexlab{b}}.
\newblock {CulturePark}: Boosting cross-cultural understanding in large language models.
\newblock In \emph{Advances in Neural Information Processing Systems}, volume~37, pages 65183--65216.
\bibitem[{Li et~al.(2024{\natexlab{c}})Li, Bonatti, Abdali, Wagle, and Koishida}]{li2024data}
Yinheng Li, Rogerio Bonatti, Sara Abdali, Justin Wagle, and Kazuhito Koishida. 2024{\natexlab{c}}.
\newblock Data generation using large language models for text classification: An empirical case study.
\newblock \emph{arXiv preprint arXiv:2407.12813}.
\bibitem[{Li et~al.(2023)Li, Zhu, Lu, and Yin}]{li-etal-2023-synthetic}
Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. 2023.
\newblock \href {https://doi.org/10.18653/v1/2023.emnlp-main.647} {Synthetic data generation with large language models for text classification: Potential and limitations}.
\newblock In \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, pages 10443--10461, Singapore. Association for Computational Linguistics.
\bibitem[{Mousi et~al.(2025)Mousi, Durrani, Ahmad, Hasan, Hasanain, Kabbani, Dalvi, Chowdhury, and Alam}]{mousi-etal-2025-aradice}
Basel Mousi, Nadir Durrani, Fatema Ahmad, Md~Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur~Absar Chowdhury, and Firoj Alam. 2025.
\newblock \href {https://aclanthology.org/2025.coling-main.283/} {{A}ra{D}i{CE}: Benchmarks for dialectal and cultural capabilities in {LLM}s}.
\newblock In \emph{Proceedings of the 31st International Conference on Computational Linguistics}, pages 4186--4218, Abu Dhabi, UAE. Association for Computational Linguistics.
\bibitem[{Myung et~al.(2024)Myung, Lee, Zhou, Jin, Putri, Antypas, Borkakoty, Kim, Perez-Almendros, Ayele et~al.}]{myung2024blend}
Junho Myung, Nayeon Lee, Yi~Zhou, Jiho Jin, Rifki~Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew~Ali Ayele, et~al. 2024.
\newblock {BLEnD}: A benchmark for llms on everyday knowledge in diverse cultures and languages.
\newblock In \emph{Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS)}, Vancouver, Canada.
\bibitem[{Nahin et~al.(2025)Nahin, Nandi, Sarker, Muhtaseem, Kowsher, Shill, Ibrahim, Menon, Muntasir, and Alam}]{nahin2025titullms}
Shahriar~Kabir Nahin, Rabindra~Nath Nandi, Sagor Sarker, Quazi~Sarwar Muhtaseem, Md~Kowsher, Apu~Chandraw Shill, Md~Ibrahim, Mehadi~Hasan Menon, Tareq~Al Muntasir, and Firoj Alam. 2025.
\newblock {TituLLMs}: A family of bangla llms with comprehensive benchmarking.
\newblock \emph{arXiv preprint arXiv:2502.11187}.
\bibitem[{Nguyen et~al.(2016)Nguyen, Rosenberg, Song, Gao, Tiwary, Majumder, and Deng}]{nguyen2016ms}
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li~Deng. 2016.
\newblock {MS MARCO}: A human generated machine reading comprehension dataset.
\newblock \emph{choice}, 2640:660.
\bibitem[{Nguyen et~al.(2024)Nguyen, Zhang, Li, Aljunied, Hu, Shen, Chia, Li, Wang, Tan, Cheng, Chen, Deng, Yang, Liu, Zhang, and Bing}]{nguyen-etal-2024-seallms}
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew~Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024.
\newblock \href {https://doi.org/10.18653/v1/2024.acl-demos.28} {{S}ea{LLM}s - large language models for {S}outheast {A}sia}.
\newblock In \emph{Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)}, pages 294--304, Bangkok, Thailand. Association for Computational Linguistics.
\bibitem[{O'Neill(2017)}]{o2017overview}
Thomas~A O'Neill. 2017.
\newblock An overview of interrater agreement on likert scales for researchers and practitioners.
\newblock \emph{Frontiers in psychology}, 8:777.
\bibitem[{Pawar et~al.(2024)Pawar, Park, Jin, Arora, Myung, Yadav, Haznitrama, Song, Oh, and Augenstein}]{pawar2024survey}
Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz~Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2024.
\newblock Survey of cultural awareness in language models: Text and beyond.
\newblock \emph{arXiv preprint arXiv:2411.00860}.
\bibitem[{Putri et~al.(2024)Putri, Haznitrama, Adhista, and Oh}]{putri2024can}
Rifki~Afina Putri, Faiz~Ghifari Haznitrama, Dea Adhista, and Alice Oh. 2024.
\newblock Can llm generate culturally relevant commonsense qa data? case study in indonesian and sundanese.
\newblock \emph{arXiv preprint arXiv:2402.17302}.
\bibitem[{Shi et~al.(2024)Shi, Li, Zhang, Ziems, Yu, Horesh, Paula, and Yang}]{shi-etal-2024-culturebank}
Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Sunny Yu, Raya Horesh, Rog{\'e}rio Abreu~De Paula, and Diyi Yang. 2024.
\newblock \href {https://doi.org/10.18653/v1/2024.findings-emnlp.288} {{C}ulture{B}ank: An online community-driven knowledge base towards culturally aware language technologies}.
\newblock In \emph{Findings of the Association for Computational Linguistics: EMNLP 2024}, pages 4996--5025, Miami, Florida, USA. Association for Computational Linguistics.
\bibitem[{Team et~al.(2025)Team, Abbas, Ahmad, Alam, Altinisik, Asgari, Boshmaf, Boughorbel, Chawla, Chowdhury, Dalvi, Darwish, Durrani, Elfeky, Elmagarmid, Eltabakh, Fatehkia, Fragkopoulos, Hasanain, Hawasly, Husaini, Jung, Lucas, Magdy, Messaoud, Mohamed, Mohiuddin, Mousi, Mubarak, Musleh, Naeem, Ouzzani, Popovic, Sadeghi, Sencar, Shinoy, Sinan, Zhang, Ali, Kheir, Ma, and Ruan}]{fanar2024}
Fanar Team, Ummar Abbas, Mohammad~Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus'ab Husaini, Soon-Gyo Jung, Ji~Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev~Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine~El Kheir, Xiaosong Ma, and Chaoyi Ruan. 2025.
\newblock \href {https://arxiv.org/abs/2501.13944} {Fanar: An arabic-centric multimodal generative ai platform}.
\bibitem[{Team et~al.(2024)Team, Mesnard, Hardin, Dadashi, Bhupatiraju, Pathak, Sifre, Rivi{\`e}re, Kale, Love et~al.}]{team2024gemma}
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi{\`e}re, Mihir~Sanjay Kale, Juliette Love, et~al. 2024.
\newblock Gemma: Open models based on gemini research and technology.
\newblock \emph{arXiv preprint arXiv:2403.08295}.
\bibitem[{Touvron et~al.(2023)Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, Batra, Bhargava, Bhosale et~al.}]{touvron2023llama2}
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et~al. 2023.
\newblock Llama 2: Open foundation and fine-tuned chat models.
\newblock \emph{arXiv preprint arXiv:2307.09288}.
\bibitem[{{\"U}st{\"u}n et~al.(2024){\"U}st{\"u}n, Aryabumi, Yong, Ko, D'souza, Onilude, Bhandari, Singh, Ooi, Kayid et~al.}]{ustun2024aya}
Ahmet {\"U}st{\"u}n, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et~al. 2024.
\newblock Aya model: An instruction finetuned open-access multilingual language model.
\newblock \emph{arXiv preprint arXiv:2402.07827}.
\bibitem[{Yang et~al.(2018)Yang, Qi, Zhang, Bengio, Cohen, Salakhutdinov, and Manning}]{yang-etal-2018-hotpotqa}
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher~D. Manning. 2018.
\newblock \href {https://doi.org/10.18653/v1/D18-1259} {{H}otpot{QA}: A dataset for diverse, explainable multi-hop question answering}.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing}, pages 2369--2380, Brussels, Belgium. Association for Computational Linguistics.
\bibitem[{Zavolokina et~al.(2024)Zavolokina, Sprenkamp, Katashinskaya, Jones, and Schwabe}]{zavolokina2024think}
Liudmila Zavolokina, Kilian Sprenkamp, Zoya Katashinskaya, Daniel~Gordon Jones, and Gerhard Schwabe. 2024.
\newblock Think fast, think slow, think critical: designing an automated propaganda detection tool.
\newblock In \emph{Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems}, pages 1--24.
\end{thebibliography}