AutoSpecNER: A Fine-Grained Named Entity Recognition Dataset for Vehicle Specification Extraction
Extracted source text · arXiv 2606.24387
\documentclass[11pt]{article}
\usepackage[preprint]{acl} % use [final] or [preprint] to show authors
\usepackage{times}
\usepackage{latexsym}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{microtype}
\usepackage{inconsolata}
\usepackage{graphicx}
\usepackage{tabularx}
\usepackage{booktabs}
\usepackage{soul}
\usepackage{siunitx}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage[british]{babel}
\usepackage{enumitem}
\usepackage[normalem]{ulem}
% acl.sty already loads hyperref, so do not load it again.
\hypersetup{
colorlinks=true,
linkcolor=blue,
filecolor=magenta,
urlcolor=blue,
citecolor=blue,
}
% If the title/author block is too tall, uncomment and adjust:
% \setlength\titlebox{6cm}
\title{AutoSpecNER: A Fine-Grained Named Entity Recognition Dataset for Vehicle Specification Extraction}
% \title{AutoSpecNER: A Dataset and Benchmark for Named Entity Recognition of Vehicle Specifications}
% \title{AutoSpecNER: A Fine-Grained Named Entity Recognition Dataset and Benchmark for Vehicle Specification Extraction from Automotive Advertisements}
\author{
\textbf{Jordan Lee}\textnormal{\textsuperscript{1,2,*},}
\textbf{Filippos Ventirozos}\textnormal{\textsuperscript{1,2},}
\textbf{Abdirahman Abdullahm}\textnormal{\textsuperscript{1},}
\textbf{Ioanna Nteka}\textnormal{\textsuperscript{2},} \\
\textbf{Peter Appleby}\textnormal{\textsuperscript{2}},
\textbf{Matthew Shardlow}\textnormal{\textsuperscript{1}} \\
\textsuperscript{1}Department of Computing and Mathematics, Manchester Metropolitan University, UK \\
\textsuperscript{2}Autotrader Research Group, Autotrader UK \\
\texttt{\{f.ventirozos,m.shardlow\}@mmu.ac.uk} \\
{\small \textsuperscript{*}Work conducted during an internship at Autotrader UK.}
}
\begin{document}
\maketitle
\begin{abstract}
Vehicle advertisements contain rich specification information, but automotive NER resources remain limited. We introduce \textbf{AutoSpecNER}, an expert-annotated dataset for fine-grained entity recognition in vehicle listings. The dataset includes 659 advertisements from a popular car-selling website, with over 10,000 entities annotated across 15 categories, including \texttt{MODEL}, \texttt{ENGINE\_SPEC}, and \texttt{BATTERY\_CAPACITY}. Annotation quality was validated through inter-annotator agreement, achieving an average score of 91.5\%. We benchmark rule-based extraction, fine-tuned transformer encoders, and large language models. DeBERTa achieves the best performance with a 90\% micro-F1 score, outperforming the rule-based baseline (43\%) and the strongest large language model (77.8\%).
\end{abstract}
% Vehicle advertisements contain rich specification information, but automotive NER resources remain limited. We introduce Auto-SpecNER, an expert-annotated dataset for fine-grained entity recognition in vehicle listings. The dataset includes 659 advertisements from a popular car-selling website, with over 10,000 entities annotated across 15 categories, including MODEL, ENGINE_SPEC, and BATTERY_CAPACITY. Annotation quality was validated through inter-annotator agreement, achieving an average score of 91.5%. We benchmark rule-based extraction, fine-tuned transformer encoders, and large language models. DeBERTa achieves the best performance with a 90% micro-F1 score, outperforming the rule-based baseline (43%) and the strongest large language model (77.8%).
\section{Introduction}
The automotive industry generates vast amounts of unstructured text through online vehicle advertisements. These advertisements contain valuable specification information embedded in free-form descriptions, but extracting this structured data manually is impractical at scale. This challenge is compounded by the recent introduction of AI-generated advertisement content, which can contain hallucinations---factually incorrect information that may mislead consumers.
Named entity recognition (NER) offers a solution by automatically identifying and extracting specific information spans from text. While NER has been successfully applied to various domains including biomedical texts \citep{Majid:2024}, news articles \citep{Sang2003}, and social media \citep{Derczynski2017}, the automotive advertisement domain presents unique challenges:
\begin{itemize}
\item \textbf{Domain-specific terminology}: Technical specifications like ``2.0L TDI'', ``DSG transmission'', or ``Santorini [black colour]'' require specialised understanding.
\item \textbf{Fine-grained distinctions}: Differentiating between similar concepts (e.g., exterior vs. interior colour, battery capacity vs. range).
\item \textbf{Multi-word entities}: Complex specifications often span multiple tokens (e.g., "18 minutes with 350kW charger").
\item \textbf{Mixed content sources}: Advertisements include both user-generated content with typos and informal language, and AI-generated content with potential hallucinations.
\end{itemize}
% Previous work by \citet{ventirozos2024shifting} demonstrated NER's applicability to vehicle advertisements but focused on coarse-grained transactional categories (sales options, historic events, vehicle condition). We extend this work by developing a fine-grained annotation schema specifically for vehicle identifiable specifications, enabling applications such as automated specification tables, cross-validation between text and structured data, and hallucination detection in AI-generated content.
% Our contributions include: (1) a custom 15-label annotation schema for automotive specifications validated through inter-annotator agreement, (2) a comprehensive dataset of 659 annotated advertisements with 11,117 entity annotations from both human and AI sources, (3) systematic evaluation comparing rules-based, encoder, and LLM approaches with detailed performance analysis, and (4) identification of hallucinations in AI-generated advertisements demonstrating practical applications.
% Online vehicle advertisements are a primary source of information for consumers. While these listings often include structured tables of specifications, the free-text description section contains additional, nuanced, and sometimes conflicting details. Manually parsing these descriptions is inefficient and does not scale. Named Entity Recognition (NER) offers a robust solution for automatically extracting structured data from this unstructured text.
This paper presents two main contributions. First, we introduce Automotive Specification NER (\textbf{AutoSpecNER}), a new, publicly available dataset\footnote{Available at \href{https://github.com/FilipposVentirozos/AutoSpecNER}{github.com/FilipposVentirozos/AutoSpecNER}.} for fine-grained NER tailored to the automotive domain. The dataset contains 659 vehicle advertisements annotated with 15 entity types critical for vehicle identification and comparison.
% Unlike existing NER resources which focus on general domains like news (CoNLL-2003, \citet{Sang2003}) or noisy social media text (WNUT, \citet{Derczynski2017}), AutoSpecNER provides a specialised, high-quality resource for a specific commercial sector.
Second, we provide a comprehensive benchmark evaluation on AutoSpecNER. We compare the performance of three diverse approaches: a rules-based system, transformer-based encoder models, and large language models (LLMs) prompted using few-shot and self-verification techniques \citep{wang2023gpt}. Our analysis confirms that while NER is a viable technique for this task, the choice of model has a profound impact on performance, with fine-tuned encoders demonstrating superior capabilities.
\section{Related Work}
\subsection{Domain-Specific NER and Fine-Grained Entity Recognition}
Standard NER benchmarks like CoNLL-2003 \citep{Sang2003} focus on coarse-grained entities (person, location, organisation) inadequate for technical domains. Fine-grained entity recognition \citep{ling2012fine} requires distinguishing between closely related entity types—a challenge particularly acute in automotive contexts where hierarchical relationships exist between entities (e.g., ``2024 Ford F-150 Limited'' contains YEAR, MAKE, MODEL, and TRIM entities).
Recent work has explored product and attribute extraction \citep{putthividhya2011bootstrapped,chen2023does}, demonstrating unique challenges in technical NER: domain-specific abbreviations, overlapping entity boundaries, and hierarchical entity relationships. Fine-grained annotation schemas have proven essential for capturing technical specifications in industrial domains \citep{bikaun2024maintie}, yet automotive-specific resources remain limited.
\subsection{Automotive NER Research}
The automotive domain has received minimal attention in NER research. \citet{hu2024named} addressed NER for automotive accessories in Chinese, while recent work by \citet{ventirozos2024shifting} introduces the Auto-AdvER approach for English vehicle advertisements to understand the condition, historic claims and sales options offered. Recent datasets like FindVehicle \citep{guan2024findvehicle} target vehicle retrieval with entity types including vehicle colour, brand, model, and location, but lack the granularity needed for technical specification extraction.
\citet{park2023admit} introduced ADMit, combining adversarial training and multi-task learning for automotive NER for a Korean and English. Their work addresses domain adaptation between general and automotive-specific terminology but focuses on FAQ systems rather than technical specifications. Our work differs by targeting fine-grained vehicle identifiables critical for specification verification and hallucination detection.
\subsection{Neural Approaches and Domain Adaptation}
Transformer-based models have become standard for NER, with BERT \citep{devlin-etal-2019-bert} and its variants achieving strong baseline performance. Domain-specific pre-training significantly improves technical entity recognition, as demonstrated in manufacturing domains where morphological patterns guide entity recognition \citep{li2024corpus}.
Few-shot NER methods combining knowledge graphs and contrastive learning show promise for low-resource domains \citep{zhang2024kcl}, addressing the scarcity of labelled automotive data. While LLMs demonstrate competitive performance through prompting \citep{wang2023gpt}, recent studies \citep{naguib-etal-2024-shot} show that smaller, specialised models often outperform LLMs in low-resource technical domains, making them more practical for deployment in automotive applications.
\section{The AutoSpecNER Dataset}
\subsection{Data Collection and Composition}
We were provided 659 vehicle advertisements from one of the UK's biggest online vehicle-selling websites. The dataset comprises two distinct sources:
% which hosts everyday around half-million cars for selling
\textbf{User-generated advertisements (350)}:
The dataset was a sample of an equal distribution of ads written by individual sellers (stratified across different UK counties) and dealerships (stratified across the largest dealerships in the UK). The advertisements contain natural language variation, informal descriptions, typographical errors, and inconsistent formatting.
Example: \textit{``\uwave{lovley} ford focus 1.8 \uwave{diesal}, 2015 plate, full mot till next yr, grey \uwave{metalic} paint''}.
\textbf{AI-generated advertisements (309)}: Created\footnote{These were generated by the team of the aforementioned UK website company. These AI generated descriptions were shown to the users and the users decided whether they want to keep them or edit/delete them.} using Google Gemini\footnote{gemini-2.0-flash-001} and Meta's LLaMA3\footnote{llama-3.1-8b-instruct} by providing in the prompt the vehicle specifications. These are grammatically correct and well-structured but may contain hallucinations. For example in this advertisement for an Audi RSQ8, where the generated advert incorrectly refers to the vehicle as an Audi RS6:
\begin{quote}
``The Audi \uwave{RS6} is a high-performance car that boasts a powerful 4.0-litre V8 engine. This petrol engine is paired with an automatic transmission...''
\end{quote}
Other hallucinations exist which are more subtle than this such as many adverts have additional information inserted not present in the specifications, but could be true. An example of this is in the following advert in which a Volkswagen Polo Match is referred to as a Volkswagen Polo EVO Match:
\begin{quote}
``With only 21,029 miles on the clock, this 2021 Volkswagen Polo \uwave{EVO} Match is manufacturer approved...''
\end{quote}
This dual-source approach enables investigation of how NER models handle both human errors (typos, informality) and AI errors (hallucinations, specification confusion). Crucially, each advertisement is accompanied by structured meta-data (a fact table) that lists the vehicle's actual specifications. This is a vital characteristic, as it allows for the verification of extracted entities and forms the basis for potential error/hallucination detection systems.
% It was intended for there to be an even split between user and AI-generated advertisements however, in order to increase the support for the labels 'Recharge Time', 'Battery Capacity', and 'Battery Range', additional advertisements were annotated. These adverts were found in the user dataset using the 'standard\_vehicle\_fuel\_type' in the specifications provided, however in the AI-generated adverts this was not available so this was done by searching for mentions of 'Electric' in the 'additional\_info' of the specification which only returned 9 electric vehicles whereas hundreds of electric vehicles were found in the user dataset of which 50 were annotated.
\subsection{Corpus Annotation and Inter-Annotator Agreement}
\label{sec:annotation}
To ensure the reliability and interpretability of our proposed 15-label schema, we undertook a multi-stage annotation process. The process was iterative, involving an initial schema refinement phase to produce clear guidelines\footnote{The annotation guidelines can be found at \href{https://github.com/FilipposVentirozos/AutoSpecNER}{github.com/FilipposVentirozos/AutoSpecNER}.}, followed by a final validation phase to confirm their efficacy.
% \paragraph{Schema Refinement and Guideline Development}
Our initial phase involved two annotators independently labelling a pilot set of 50 advertisements, balanced between user- and AI-generated content. While this yielded a promising micro F1-score of 0.81, a qualitative analysis of the disagreements revealed systematic ambiguities. This analysis was crucial for developing a set of explicit annotation principles.
The key principles that emerged from this refinement process were:
\begin{itemize}[leftmargin=10pt]
\item \textbf{Prioritise Specificity over General Description:} Labels are reserved for specific, quantifiable details, not general descriptive statements. For instance, the phrase ``low CO2 emissions'', while describing an engine's characteristic, is not annotated as \textit{ENGINE\_SPEC} because it lacks a specific value.
\item \textbf{Maintain Entity Separation for Adjacent, Distinct Items:} When multiple entities of the same type appear adjacently but refer to distinct details, they must be annotated as separate entities. For example, in the text ``...TDCI 1.6L ECOnetic...'', each component (``TDCI'', ``1.6L'', ``ECOnetic'') is labelled as a separate \textit{ENGINE\_SPEC} entity.
\item \textbf{Disambiguate Overlapping Concepts:} Clear distinctions were established to prevent confusion. The \textit{MAKE} label, for example, is restricted to the primary vehicle manufacturer; in `...a Ford Focus with a Mercedes...engine`, only ``Ford'' is labelled as \textit{MAKE}. Similarly, composite names like ``e-SKYACTIV G Centre-Line'' are split into ``e-SKYACTIV G'' (\textit{ENGINE\_SPEC}) and ``Centre-Line'' (\textit{TRIM}).
\item \textbf{Enforce Strict Contextual Boundaries:} Annotators were instructed to capture the full, self-contained descriptive phrase. For \textit{BATTERY\_RANGE}, the entire phrase ``maximum range of 280 miles when new'' is captured, preserving the qualifying context.
\item \textbf{Exclude Speculative and Ancillary Information:} The guidelines were refined to ensure that only the direct attribute of the advertised vehicle is annotated. In cases where an advert mentions other available models (e.g., ``The Golf is also available as a Station Wagon (Estate)''), only the body type of the primary advertised vehicle (e.g., ``Hatchback'') is to be labelled.
\item \textbf{Annotate the Text, Not the Structured Specification:}
Annotations should reflect the entity span as it appears in the advertisement, even when it differs from the accompanying structured specification data or appears factually incorrect. For example, if the advert describes the trim as ``220d Luxury'', annotators label the span in the text rather than normalising it to the official specification value, ``Luxury''. This ensures that models learn to extract the claims made in the free-text advert, which can later be compared against structured specifications to detect inconsistencies or hallucinations.
\end{itemize}
\subsection{Annotation Schema Overview}
The final schema comprises 15 entity labels, organised into four groups and defined with representative examples in Table~\ref{tab:schema}.
\begin{table}[t]
\centering
\footnotesize
\setlength{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.15}
\begin{tabularx}{\linewidth}{@{}l>{\raggedright\arraybackslash}X>{\raggedright\arraybackslash}X@{}}
\toprule
\textbf{Label} & \textbf{Definition} & \textbf{Example} \\
\midrule
\multicolumn{3}{@{}l}{\textit{Vehicle identity}}\\
\texttt{MAKE} & Vehicle manufacturer & ``CUPRA'' \\
\texttt{MODEL} & Specific model name & ``Focus'' \\
\texttt{TRIM} & Trim level / variant of a model & ``220d Luxury'' \\
\texttt{YEAR} & Registration year & ``2021'' \\
\addlinespace
\multicolumn{3}{@{}l}{\textit{Powertrain}}\\
\texttt{ENGINE\_SPEC} & Engine detail: volume, cylinders, or technology code & ``TDCI'' \\
\texttt{FUEL\_TYPE} & Fuel powering the vehicle & ``petrol'' \\
\texttt{TRANSMISSION} & Gear system type & ``automatic'' \\
\addlinespace
\multicolumn{3}{@{}l}{\textit{Body \& appearance}}\\
\texttt{BODY\_TYPE} & Shape/style of the chassis & ``Panel Van'' \\
\texttt{EXTERIOR\_COLOUR} & Exterior paint colour, incl. descriptors & ``Black sapphire metallic'' \\
\texttt{INTERIOR\_COLOUR} & Colour/material of interior features & ``black leather seats'' \\
\texttt{NO\_SEATS} & Total seating capacity & ``5 seats'' \\
\texttt{BOOT\_SIZE} & Boot storage capacity (litres) & ``6100 litres'' \\
\addlinespace
\multicolumn{3}{@{}l}{\textit{Electric vehicle}}\\
\texttt{BATTERY\_CAPACITY}& Battery capacity (kWh) & ``68 kWh'' \\
\texttt{BATTERY\_RANGE} & Travel distance, incl. qualifiers & ``max range of 280 miles when new'' \\
\texttt{RECHARGE\_TIME} & Recharge time, incl. charger/level context & ``empty to 80\% in as little as 53 minutes'' \\
\bottomrule
\end{tabularx}
\caption{The 15-label \textbf{AutoSpecNER} annotation schema, grouped by category with definitions and representative examples.}
\label{tab:schema}
\end{table}
\subsubsection{Final Validation and Agreement}
Following the guideline refinement, a final inter-annotator agreement study was conducted. This involved three annotators (the authors of this paper) of mixed nationalities with English as either their first or second professional language and varying levels of prior exposure to NLP annotation tasks, who labelled a new, larger random sample of 100 advertisements (50 user-generated and 50 AI-generated). Using the finalised guidelines, this validation achieved a strict-matching average micro F1-score\footnote{In NER in most cases Kappa is equivalent to F1, hence, we just measure F1 \citep{richie-etal-2022-inter,hripcsak2005agreement}} of \textbf{91.5\%} across all three annotators (standard deviation: \textbf{3.2\%}). Precision consistently exceeded recall by approximately 2 percentage points, indicating strong agreement on positive entity identification while annotators exhibited greater conservatism when entity boundaries or labels were ambiguous.
\subsection{Dataset Statistics}
The dataset contains a total of 11,117 labelled entities overall. Entity frequencies approximate a power law distribution, from MODEL (2,426 instances) to NO\_SEATS (10 instances). This reflects natural occurrence patterns - every advertisement mentions the model, but seat count is specified only when notable. One can look into Figure~\ref{fig:label_proportion} to view the label distributions for the user and the AI advertisement generated text respectively. In total, the dataset contains 44 different brands, 237 unique models and the vehicles range from older (first registered in 2006) to newest (this year 2025).
\begin{figure}[h]
\centering
\includegraphics[width=0.99\linewidth]{figures/LabelProportionFinal.png}
\caption{Proportion of labels from each source within the dataset. In dark blue are the the text advertisements generated by either Gemini or Llama and in green are the ones written by users.}
\label{fig:label_proportion}
\end{figure}
% \textbf{Source Differences}: AI-generated advertisements average 17.9 entities per text versus 14.2 for user-generated, with AI content showing higher consistency in entity formatting but containing factual errors in 12\% of advertisements.
% \textbf{Entity Complexity}: Average entity length varies from 1.2 tokens (YEAR) to 4.7 tokens (RECHARGE\_TIME), with battery-related entities showing the highest variance in expression patterns.
\section{Experiments}
To prepare the annotated corpus for training, we performed two key steps: pre-processing the span-level annotations into a token-level format and partitioning the corpus into training (\textasciitilde70\%), validation (\textasciitilde15\%), and testing (\textasciitilde15\%) sets, the distribution of the entities can be found in Table~\ref{tab:dataset_splits}.
% To make the data compatible with token-based NER models, the character-level span annotations were converted into the standard IOB2 (Inside, Outside, Beginning) format. This was achieved by tokenizing each text using the model-specific tokenizer and then aligning the original gold-standard spans to the resulting tokens to assign the appropriate B-, I-, or O- tags.
\begin{table}[h!]
\centering
\scalebox{0.9}{
\sisetup{group-separator={,}} % Use comma for thousands separator
\begin{tabular}{@{}l S[table-format=4.0] S[table-format=3.0] S[table-format=3.0] S[table-format=5.0]@{}}
\toprule
\textbf{Label} & {\textbf{Train}} & {\textbf{Validation}} & {\textbf{Test}} \\
\midrule
ENGINE\_SPEC & 1694 & 283 & 468 \\
MODEL & 1630 & 347 & 449 \\
TRIM & 1399 & 285 & 349 \\
MAKE & 838 & 172 & 219 \\
FUEL\_TYPE & 483 & 70 & 120 \\
YEAR & 353 & 75 & 111 \\
EXTERIOR\_COLOUR & 297 & 58 & 125 \\
RECHARGE\_TIME & 260 & 78 & 10 \\
INTERIOR\_COLOUR & 178 & 16 & 107 \\
BODY\_TYPE & 116 & 32 & 33 \\
TRANSMISSION & 114 & 9 & 16 \\
BATTERY\_CAPACITY & 86 & 44 & 32 \\
BATTERY\_RANGE & 71 & 41 & 3 \\
BOOT\_SIZE & 16 & 8 & 12 \\
NO\_SEATS & 4 & 0 & 6 \\
\midrule
\textbf{Overall} & \textbf{7539} & \textbf{1518} & \textbf{2060} \\
\bottomrule
\end{tabular}
}
\caption{Distribution of entity counts across the final training, validation, and testing sets.}
\label{tab:dataset_splits}
\end{table}
% % \subsection{Evaluation}
% We evaluate all models---encoders, LLMs, and rules-based approaches---using character-level IOB2 tagging to ensure fair comparison across different tokenisation schemes and the text generated by the decoder-type language models. We compute the metrics using the \texttt{seqeval} library~\cite{seqeval}.
The three different ML approaches operate on different units: sub-word tokens for the encoders, whitespace tokens for the rule-based matcher, and LLM-generated free-form text. We therefore evaluate every system on a single, tokeniser-independent character-level IOB2 representation. Specifically, for each advert, the gold character-offset spans are expanded into a per-character IOB2 sequence in which every character carries exactly one tag. Within an entity span, the first character is tagged \texttt{B-} and each subsequent character—including whitespace within a multi-word entity—is tagged \texttt{I-}; every character that falls outside any entity span is tagged \texttt{O} (``Outside''). Each system's predictions are mapped onto the same per-character IOB2 representation, so all systems are scored on identical units; we then reconstruct entities and report entity-level precision, recall, and micro-F1 with \texttt{seqeval}~\cite{seqeval}.
\subsection{Models}
We evaluated three classes of models on the AutoSpecNER dataset to establish a robust performance benchmark.
\subsubsection{Rules-Based Approach}
To establish a performance baseline and explore a non-AI alternative, we developed a hybrid rules-based system for entity extraction. This approach was designed to be computationally inexpensive at inference time and serves as a benchmark against which our neural models are compared. The system combines two core methodologies: taxonomy-based matching and regular expressions.
\paragraph{Methodology}
Our system employs a two-pronged strategy, applying different techniques based on the nature of the target entity.
\paragraph{Taxonomy-Based Matching}
For eight of the core vehicle attributes (\texttt{MAKE}, \texttt{MODEL}, \texttt{TRIM}, \texttt{FUEL\_TYPE}, \texttt{BODY\_TYPE}, \texttt{TRANSMISSION}, \texttt{INTERIOR\_COLOUR}, and \texttt{EXTERIOR\_COLOUR}), we leveraged an external taxonomy provided by the company that provided the data. This taxonomy acts as a gazetteer of known values for each entity. The system iterates through the advertisement text, comparing text spans to the values in the gazetteer. We evaluated several matching strategies:
\begin{itemize}
\item \textbf{Strict Matching:} Requires an exact, case-insensitive string match.
\item \textbf{Partial Matching:} To account for minor variations and typographical errors, we implemented partial matching using a normalised Levenshtein distance function. Predictions were made only if the similarity score exceeded a pre-determined threshold.
\item \textbf{Heuristic Filtering:} To improve precision, we experimented with two disambiguation heuristics: (1) a \textit{most-common-value} filter, which retains only the most frequently matched entity for labels expected to appear once (e.g., \texttt{MAKE}); and (2) \textit{hierarchical filtering}, which uses known make-model-trim relationships from the taxonomy to prune invalid predictions (e.g., removing a predicted trim that does not belong to the predicted model).
\end{itemize}
\paragraph{Regular Expressions}
For entities with highly predictable syntactic patterns that were not present in the taxonomy, we employed regular expressions. This approach was used for three specific labels: \texttt{BATTERY\_CAPACITY}, \texttt{NUMBER\_OF\_SEATS}, and \texttt{YEAR}. The patterns, detailed in Table~\ref{tab:regex_patterns}, were designed to be robust enough to capture common formulations of these attributes.
\begin{table*}[h!]
\centering
\begin{tabular}{@{}lll@{}}
\toprule
\textbf{Label} & \textbf{Pattern} & \textbf{Description} \\
\midrule
BATTERY\_CAPACITY & \texttt{\textbackslash b(\textbackslash w+)\textbackslash s+kWh\textbackslash b} & Matches numeric values followed by "kWh". \\
NO\_SEATS & \texttt{(\textbackslash d+)\textbackslash s*\textbackslash w*seat\textbackslash w*} & Matches seat counts (e.g., "5 seats"). \\
YEAR & \texttt{\textbackslash b(?:19{[}0-9{]}...)} & Matches four-digit years from 1900-2030. \\
\bottomrule
\end{tabular}
\caption{Regular expression patterns used for specific entity extraction.}
\label{tab:regex_patterns}
\end{table*}
% \paragraph{Algorithmic Workflow}
% The overall workflow of the rules-based system is summarised in Algorithm~\ref{alg:rules_workflow}. The process iterates through each advertisement, applying the taxonomy and regex methods sequentially to build a final set of predicted entities.
% \begin{algorithm*}[h!]
% \caption{Rules-Based Entity Extraction Workflow}
% \label{alg:rules_workflow}
% \begin{algorithmic}[1]
% \STATE \textbf{Input:} Advertisement text $T$, Taxonomy $Tax$, Regex Patterns $Reg$
% \STATE \textbf{Input (Config):} $UseHierarchicalFilter$, $UsePartialMatch$, $UseMostCommonFilter$
% \STATE \textbf{Output:} Set of final extracted entities $E_{final}$
% \STATE $E_{candidate} \gets \emptyset$
% \STATE
% \COMMENT{Part 1: Taxonomy Matching}
% \FOR{each token $t$ in $T$}
% \FOR{each label $L$ in $Tax$}
% \STATE $V_L \gets$ Get values for $L$ from $Tax$
% \IF{$UseHierarchicalFilter$}
% \STATE $V_L \gets$ Filter $V_L$ based on prior predictions in $E_{candidate}$
% \ENDIF
% \IF{ExactMatch($t$, $V_L$)}
% \STATE Add prediction $(t, L)$ to $E_{candidate}$
% \ELSIF{$UsePartialMatch$ \AND PartialMatch($t$, $V_L$) > threshold}
% \STATE Add prediction $(t, L)$ to $E_{candidate}$
% \ENDIF
% \ENDFOR
% \ENDFOR
% \STATE
% \COMMENT{Part 2: Regex Matching}
% \FOR{each label $L$ in $Reg$}
% \STATE $M \gets$ Find all matches of pattern for $L$ in $T$
% \STATE $E_{candidate} \gets E_{candidate} \cup M$
% \ENDFOR
% \STATE
% \COMMENT{Part 3: Final Filtering}
% \IF{$UseMostCommonFilter$}
% \STATE $E_{final} \gets$ Apply most-common-value filter to $E_{candidate}$
% \ELSE
% \STATE $E_{final} \gets E_{candidate}$
% \ENDIF
% \STATE \textbf{return} $E_{final}$
% \end{algorithmic}
% \end{algorithm*}
% Our baseline employs deterministic pattern matching using AutoTrader's vehicle taxonomy (covering 147 makes, 3,842 models, 15,234 trim variants) and regular expressions for structured entities.
% \textbf{Exact Matching}: Direct string comparison against taxonomy entries. Fast (0.07s/advert) but brittle to variations.
% \textbf{Fuzzy Matching}: Normalized Levenshtein distance with configurable threshold (empirically optimized at 0.85). The distance calculation:
% $$d_{norm}(s_1, s_2) = 1 - \frac{d_{lev}(s_1, s_2)}{\max(|s_1|, |s_2|)}$$
% This handles typos ("diesal" → "diesel") but increases complexity to O(n²m) where n is text length and m is taxonomy size.
% \textbf{Regular Expressions}: Pattern-based extraction for structured entities:
% - YEAR: \verb|\b(19[0-9]{2}|20[0-2][0-9])\b|
% - BATTERY\_CAPACITY: \verb|\b(\d+)\s*kWh\b|
% - NO\_SEATS: \verb|(\d+)\s*-?\s*seat|
% \textbf{Hierarchical Filtering}: Validates entity relationships (trim must match model, model must match make) using taxonomy structure, reducing false positives by 31\%.
\subsubsection{Transformer Encoders} We fine-tuned four widely-used encoder models: BERT-base-cased \citep{devlin-etal-2019-bert}, RoBERTa-base \citep{Liu2019Roberta}, ModernBERT-base \citep {warner-etal-2025-smarter} and DeBERTa-v3-base \citep{he2023deberta}. To perform the training and inference, we utilised the HuggingFace library \citep{wolf2020huggingfacestransformersstateoftheartnatural} for token classification.
We perform grid search over learning rates $\{5\times10^{-5}, 3\times10^{-5}\}$, batch sizes $\{4, 8, 16\}$, and weight decay values $\{0.0, 0.01, 0.1\}$ to identify optimal hyperparameters for each encoder architecture. We train all models for a maximum of 50 epochs with early stopping patience of 3 epochs, monitoring the weighted F1 score on the validation set. The best configuration across all models uses a learning rate $2\times10^{-5}$, batch size 8, weight decay 0.01, with a warmup of 100 steps and a maximum sequence length of 512 tokens used in all experiments.
\subsubsection{Large Language Models}
To provide a contemporary comparison to encoder-based models, we evaluated four open sourced LLMs for the NER task: Qwen3~\citep{qwen3} Mistral-7B \citep{Mistral}, LLaMA-3-8B \citep{llama3}, Gemma-3~\citep{gemmateam2025gemma3technicalreport}. Additionally, we evaluated two closed-sourced LLMs: Gemini-2.5~\citep{geminiteam2025geminifamilyhighlycapable} and GPT-5~\citep{openai2025gpt5}. These models were selected as representative of smaller and resource-efficient architectures. All the specific versions with their respective parameter sizes are listed in Table~\ref{tab:overall_performance}.
\paragraph{Prompt Engineering}
We adopted the GPT-NER methodology \citep{wang2023gpt}, implementing a single-turn per-label prompting strategy, where each advertisement was processed 15 times, once for each entity type. The refined prompting approach incorporated three key components:
\begin{itemize}
\item \textbf{Task-specific prompts}: Each prompt focused on a single entity type with a concise task description and label definition
\item \textbf{Few-shot examples}: 2 few-shot in-context learning examples were used.
\item \textbf{Simplified output format}: The output would copy verbatim the example that we want labelled and the entities of interested would be marked with boundary delimiters (@@...||) rather than full NER formatting.
\end{itemize}
\paragraph{Self-Verification}
Following \citep{wang2023gpt}, a self-verification step was implemented to mitigate the high false positive rate characteristic of LLM-based NER. This secondary prompt presented the model's initial predictions for validation, significantly improving precision whilst marginally reducing recall. Preliminary experiments, found in Appendix~\ref{sec:app:prelim_verification}, demonstrated the efficacy of including this step.
Complete prompt templates are provided in Appendix~\ref{sec:appendix:llm-prompts}.
\section{Results}
\label{sec:results}
\subsection{Rules-based Approach}
\label{ssec:rules_based}
The rules-based approach utilised a combination of taxonomy-based string matching and regular expressions. Performance was highly dependent on the complexity of the entity and the matching method employed. Methods involving partial matching via Fuzzy and Levenshtein algorithms, while more accurate for variable text, were significantly slower at inference time compared to exact matching.
Table~\ref{tab:label_performance} summarises the best F1-scores achieved for each label. As shown, simpler, more regular entities like \texttt{YEAR}, \texttt{MAKE}, and \texttt{MODEL} achieved strong performance (0.77--0.79 F1). In contrast, more complex and nuanced labels such as \texttt{INTERIOR\_COLOUR} and \texttt{EXTERIOR\_COLOUR} performed poorly, with F1-scores below 0.03.
An analysis of precision and recall scores reported in Appendix~\ref{app:rules_pr} reveal that regex-based methods, where applicable (e.g., for \texttt{YEAR}), achieved high recall at the cost of precision, whereas taxonomy-based matching often resulted in higher precision but lower recall, as it failed to capture variations not present in the pre-defined lists. Overall, when evaluated across all present labels, the best-performing configuration of the rules-based approach achieved a micro-averaged F1-score of 0.43.
\subsection{Encoder-based Models}
\label{ssec:encoders}
% The models' performance was evaluated using macro (unweighted) and weighted averages. The macro average treats each label equally, providing insight into performance across all classes, whereas the weighted average accounts for label imbalance by considering the support for each class.
We evaluated four transformer-based encoders: BERT, RoBERTa, ModernBERT and DeBERTa. Table~\ref{tab:overall_performance} summarises the character-level performance of all evaluated models. Fine-tuned encoder models achieve higher performance, with micro-F1 scores ranging from 82.9\% to 90.1\%, compared to LLMs which achieve only 77.8\% to 41.7\% micro-F1.
\begin{table}[t]
\centering
\small
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Model} & \textbf{Type} & \textbf{Micro-F1\textdownarrow
} \\
\midrule
microsoft/deberta-v3-base & Encoder & 0.901 \\
FacebookAI/roberta-base & Encoder & 0.895 \\
bert-base-cased & Encoder & 0.873 \\
answerdotai/ModernBERT-base & Encoder & 0.829 \\
\midrule
Gemini-2.5-Flash-Lite & LLM & 0.778 \\
GPT-5-Nano & LLM & 0.775 \\
Gemma-3-27B-IT & LLM & 0.753 \\
Qwen3-30B-A3B-Instruct & LLM & 0.735 \\
Llama-3.1-8B-Instruct & LLM & 0.682 \\
Mistral-7B-Instruct-v0.2 & LLM & 0.417 \\
% Yi-34B-Chat & LLM & 0.246 \\
\bottomrule
\end{tabular}
\caption{Overall character-level performance comparison. Encoders consistently outperform LLMs with 2-shot in-context learning. The micro-F1 scores are shown in descending order.}
\label{tab:overall_performance}
\end{table}
The DeBERTa model demonstrated the best overall performance, achieving an F1 score of 90.1\%. After resampling the data splits and repeating the experiments three times, the standard deviation was 2.3\%, indicating relatively stable performance across runs. This strong performance may be attributable to DeBERTa’s disentangled attention mechanism, which more effectively captures positional information, a feature that is particularly relevant for the consistently structured AI-generated adverts.
Table~\ref{tab:label_performance} shows per-label F1 scores for all the family models. The encoders perform best on numeric entities such as \textbf{YEAR} and \textbf{BATTERY\_CAPACITY}, standardised attributes like \textbf{FUEL\_TYPE} and \textbf{TRANSMISSION}, and vehicle identifiers including \textbf{MAKE} and \textbf{MODEL}. In contrast, they struggle with more subjective attributes such as \textbf{INTERIOR\_COLOUR} and \textbf{EXTERIOR\_COLOUR}, likely due to the wide lexical variation in colour descriptions (e.g., ``midnight blue'', ``pearl white'', ``metallic silver'').
Our analysis of the training data size indicated that the dataset was sufficient for the task, as model performance (measured by evaluation loss) began to plateau after approximately 300--350 training samples. Further details and the corresponding graph are available in Appendix~\ref{app:training_size}. The one exception is the \texttt{NO\_SEATS} which was scarcely reported in the text descriptions.
The decoder models, which are much larger in parameter size---many times more than 100 times larger---than their encoder counterparts excelled in various labels, especially in those with less support, comparatively to the encoders.
\begin{table*}[t]
\centering
\small
\begin{tabular}{@{}lcccccc@{}}
\toprule
\textbf{Entity Type} & \textbf{Best Encoder} & \textbf{Encoder F1} & \textbf{Best LLM} & \textbf{LLM F1} & \textbf{Rules F1} \\
\midrule
MAKE & RoBERTa & \textbf{0.938} & GPT-5-Nano & 0.825 & 0.773 \\
MODEL & RoBERTa & \textbf{0.950} & Gemini-2.5-Flash-Lite & 0.878 & 0.789 \\
TRIM & DeBERTa-v3 & 0.870 & Gemini-2.5-Flash-Lite & \textbf{0.715} & 0.482 \\
YEAR & DeBERTa-v3 & \textbf{0.982} & Gemini/Gemma & 0.948 & 0.773 \\
BODY\_TYPE & DeBERTa-v3 & 0.721 & GPT-5-Nano & \textbf{0.769} & 0.531 \\
FUEL\_TYPE & BERT-cased & \textbf{0.957} & Gemini-2.5-Flash-Lite & 0.773 & 0.574 \\
TRANSMISSION & DeBERTa-v3 & \textbf{0.938} & GPT-5-Nano & 0.476 & 0.255 \\
ENGINE\_SPEC & DeBERTa-v3 & \textbf{0.933} & Gemini-2.5-Flash-Lite & 0.907 & 0.279 \\
EXTERIOR\_COLOUR & DeBERTa-v3 & 0.725 & GPT-5-Nano & \textbf{0.763} & 0.024 \\
INTERIOR\_COLOUR & RoBERTa & \textbf{0.623} & Gemini/GPT & 0.270 & 0.007 \\
NO\_SEATS & N/A & 0.000 & Gemma-3-27B-IT & \textbf{0.750} & 0.258 \\
BOOT\_SIZE & DeBERTa-v3 & 0.909 & Gemini-2.5-Flash-Lite & \textbf{1.000 }& --- \\
BATTERY\_CAPACITY & DeBERTa-v3 & \textbf{1.000 }& Gemini/GPT & 0.714 & 0.462 \\
BATTERY\_RANGE & DeBERTa-v3/RoBERTa & 0.800 & Gemini-2.5-Flash-Lite & \textbf{1.000} & --- \\
RECHARGE\_TIME & RoBERTa & \textbf{1.000} & N/A & 0.000 & --- \\
\bottomrule
\end{tabular}
\caption{Per-label F1 scores comparing encoder, LLM, and rules-based performance. In bold are the top performants for each label. Some model names are shortened for illustration purposes but are the same version with the ones presented in Table~\ref{tab:overall_performance}.}
\label{tab:label_performance}
\end{table*}
% As expected, the self-verification step consistently increased precision while reducing recall. The performance increase from self-verification was most significant in lower-support labels with lower initial performance metrics. For instance, with LLaMA3, the F1-score for \texttt{BATTERY\_CAPACITY} increased from 0.291 to 0.552 and for \texttt{BOOT\_SIZE} from 0.112 to 0.213.
% Most strikingly, \textbf{LLMs achieve exactly 0.0\% F1 on 13 out of 15 entity types}. The only two entity types where LLMs show any extraction capability are MAKE (65.0\%--68.5\% precision, 65.0\%--96.4\% recall) and MODEL (77.6\%--92.5\% precision, 18.4\%--88.4\% recall). However, even for these two labels, LLM performance remains substantially below encoder models (93.4\% and 94.9\% F1 respectively). Analysis of \texttt{gpt-5-nano} predictions across 88 test examples containing 886 total entity mentions reveals that LLMs extract only 150 entities (16.9\% of ground truth), with MAKE accounting for 137 entities (91.3\% of all extractions) and MODEL for 13 entities (8.7\%), while all other types yield 0 extractions. This pattern is consistent across all tested LLMs: they predominantly extract MAKE and MODEL while completely ignoring other entity types, even when those types appear in highly regular patterns. Notably, LLMs fail to extract even trivial, unambiguous patterns such as ``2023'' and ``2024'' (YEAR, missed 15$\times$ and 12$\times$ respectively), ``Petrol'' and ``Electric'' (FUEL\_TYPE, missed 18$\times$ and 14$\times$), ``Automatic'' and ``Manual'' (TRANSMISSION, missed 22$\times$ and 8$\times$), ``2.0L'' (ENGINE\_SPEC, missed 9$\times$), and ``SUV'' (BODY\_TYPE, missed 11$\times$).
% We identify three primary factors contributing to these LLM failures. First, our 2-shot learning approach may not be indicative enough for the LLMs to perform inference. Second, vehicle specifications involve domain-specific terminology (``kWh'', ``bhp'', ``L'') and complex entity boundaries (e.g., ``2.0L Turbocharged Inline-4'' as a single ENGINE\_SPEC entity), for which pre-trained LLMs lack adequate exposure during training, unlike general entities such as persons or organisations. Third, analysis of the training data shows that MAKE and MODEL account for 47.3\% of all entity mentions, while types like RECHARGE\_TIME and NO\_SEATS comprise less than 2\% each, creating a long-tail distribution where demonstration examples likely over-represent common types and provide insufficient signal for rare entity types.
Gemini-2.5 scored the highest amongst the LLMs but still second to the encoders. Its predictions across the test set reveal heterogeneous performance across entity types: strong (F1 $>$ 75\%) on YEAR, \texttt{MAKE}, and \texttt{MODEL}; moderate (F1 50--75\%) on \texttt{ENGINE\_SPEC}, \texttt{EXTERIOR\_COLOUR}, \texttt{TRIM}, \texttt{BATTERY\_CAPACITY}, and \texttt{BODY\_TYPE}; and poor (F1 $<$ 50\%) on \texttt{INTERIOR\_COLOUR}, \texttt{TRANSMISSION}, and notably \texttt{RECHARGE\_TIME} (0\% F1 despite 10 test instances).
LLMs exhibit three primary failure modes. First, \textit{boundary detection errors}: extracting ``300 miles'' instead of ``approximately 300 miles'' \texttt{BATTERY\_RANGE}), or ``408'' instead of ``408 horsepower'' (\texttt{ENGINE\_SPEC}). Second, \textit{variant normalisation failures}: achieving high recall on simple \texttt{FUEL\_TYPE} values (``petrol'', ``diesel'', ``electric'') while missing hybrid variants (``Petrol Hybrid'', ``Petrol Plug-in Hybrid'' , ``Diesel Hybrid'').
We identify three primary factors contributing to these LLM failures. First, our 2-shot learning approach may not be indicative enough for the LLMs to perform inference. Second, vehicle specifications involve domain-specific terminology (``kWh'', ``bhp'', ``T-GDI'', ``BiTurbo'') and complex entity boundaries (e.g., ``2.0L Turbocharged Inline-4'' as \texttt{ENGINE\_SPEC} entities), for which pre-trained LLMs lack adequate exposure during training, unlike general entities such as persons or organisations. Third, the severe lexical variation within certain entity types (e.g., \texttt{INTERIOR\_COLOUR}: ``Bengal red Nappa leather'', ``Light Oyster and Ebony'', ``Full Black Valcona Leather'') exceeds what can be captured in 2-shot demonstrations, while more standardised types (\texttt{YEAR}, \texttt{MAKE}) benefit from prior knowledge in the pre-training corpus.
Albeit, the LLMs proved resourceful, achieving the highest F1 score amongst the other approaches, when the support was quite low in the labels \texttt{NO\_SEATS}, \texttt{BATTERY\_RANGE} and \texttt{BOOT\_SIZE}.
\subsection{Computational Resources}
\label{sec:setup}
All encoder-based transformer models were fine-tuned on an Apple M4 MacBook. Open-source LLM inference was performed using vLLM~\citep{kwon2023efficient} on NVIDIA RTX PRO 6000 (96GB VRAM) or NVIDIA H200 SXM (141GB VRAM) GPUs, depending on model size. Closed-source models were accessed through their respective provider APIs. To ensure deterministic outputs, we set the temperature to 0.0 where supported and used greedy decoding for all generations. Additional hardware and runtime details are provided in Appendix~\ref{app:hardware}.
% All encoder-based transformer models were fine-tuned on an Apple M4 MacBook Pro with 36GB unified memory. Inference for encoder models completed between seconds to minute for the entire test set. For LLM experiments, we distinguished between open-source and closed-source models. Smaller open-source LLMs (parameters $<$10B) performed inference using vLLM~\citep{kwon2023efficient} on an NVIDIA RTX PRO 6000 GPU with 96GB VRAM, while larger models utilized an NVIDIA H200 SXM GPU with 141GB VRAM memory. LLM inference, including the entity extraction and verification steps, required approximately 1.5 hours for each LLM to process all test samples (that is for 109 [n. samples] $\times$ 15 [number of entities] $\times$ 2 [verification test] number of queries). All closed-source LLMs (i.e. GPT, Gemini) were accessed through their respective provider APIs. All LLMs used the OpenAI-compatible API format. To ensure deterministic outputs, we set temperature to 0.0 where supported by the API and employed greedy decoding for all LLM generations.
% \subsection{Potential Applications and Impact}
% \label{sec:impact}
% The high accuracy of the fine-tuned encoder models enables several valuable downstream applications.
% \begin{itemize}
% \item \textbf{Social Media Monitoring:} While this work focused on advertisements, the models could be evaluated for extracting vehicle attributes from unstructured social media posts to monitor consumer trends, popular features, and brand perception.
% \item \textbf{Hallucination and Error Detection:} The extracted entities can be programmatically compared against structured vehicle specification data. This creates a framework for automatically detecting factual errors in user-generated adverts or hallucinations in AI-generated content, enhancing platform integrity.
% \end{itemize}
\section{Discussion}
The current investigation has implications for deploying NER systems in production automotive applications. When the goal is extracting entity types and populating structured vehicle databases, fine-tuned encoder models---despite their training requirement---remain the best solution overall, with DeBERTa leading when the supporting examples are sufficient, followed by the LLMs and last the rule-based approaches.
Looking beyond immediate deployment constraints and efficiency considerations, several avenues could improve LLM performance on this task. Increasing the few-shot to more examples per entity type could improve prediction accuracy. Similarly, retrieval-augmented prompting could dynamically retrieve relevant examples that could boost the score as well. Finally, models further pre-trained or post-trained on automotive corpora may better handle specialised terminology and entity patterns.
The high accuracy of the fine-tuned encoder models enables several valuable downstream applications. While this work focused on advertisements, the models could be evaluated for tracking vehicles from unstructured social media posts to monitor consumer trends, popular features, and brand perception. Additionally, the process of extracting entities can be used to detect hallucinations in AI-generated content by verifying whether a model that generated the content included additional or different specifications and whether it adhered to the structured vehicle specification data given, thereby enhancing platform integrity. It was notable how on average we could spot an around 20\% increase in F1 score when detecting entities in the AI generated text alone. The fine-tuned encoders could prove resourceful at tracking these mentions from AI generated text, raising flags when necessary.
\section{Conclusions}
\label{sec:conclusions}
In this paper, we addressed the high-impact challenge of extracting fine-grained vehicle specifications from the text descriptions of car advertisements. Our dataset and annotation schema \textbf{AutoSpecNER} comprises 659 advertisements---written by real users and AI-generated proprietary data acquired from the UK's biggest online vehicle-selling website---labelled with 15 specification types ranging from \texttt{MAKE} and \texttt{MODEL} to \texttt{RECHARGE\_TIME}, with the schema validated at a commendable 91.5\% inter-annotator F1 score. Benchmarking rule-based, fine-tuned encoder, and few-shot decoder approaches, we found the fine-tuned encoders exceptionally well-suited to this task, with DeBERTa achieving a top micro F1-score of 0.901, followed by the decoders (led by Gemini-2.5) and last the rule-based approaches; the LLMs proved particularly resourceful for low-support labels scarcely reported in the text. Finally, this dataset can support market trend analysis and, crucially, the detection of factual hallucinations in AI-generated content.
% \newpage
\section{Limitations}
\label{sec:limitations}
The dataset was drawn exclusively from a proprietary database maintained by a single UK-based company. Although substantial, it is unlikely to capture the full diversity of linguistic usage, document structures, and formatting conventions observed across different countries, platforms, and automotive retailers. In addition, the corpus is UK-specific and monolingual (English), preventing assessment of cross-lingual performance. Improving generalisability would require a more heterogeneous, multi-source, and multilingual dataset.
\section{Ethical Considerations}
The development of the \textbf{AutoSpecNER} dataset and its associated models has been guided by key ethical considerations. The dataset was constructed from publicly accessible advertisements available on a major commercial platform. A manual review of the user-generated content was conducted, which confirmed the absence of Personally Identifiable Information (PII) in the source data. While the data's public availability and freedom from PII mitigate privacy concerns, its origin introduces a risk of socio-economic bias. The model may exhibit performance disparities across content from different seller types, and if the system is more accurate on professionally formatted advertisements than on informal text from private sellers, its application could inadvertently create an unfair marketplace.
Consideration must also be given to the potential for misuse and downstream harms. The same technology that extracts specifications can be repurposed for malicious ends. Models trained on this data could be exploited to generate convincing but fraudulent vehicle advertisements, either for individual scams or at scale to manipulate market perceptions of a vehicle's value and availability. In a deployed application, over-reliance on the system's output could lead to automation bias, where users and platform administrators place undue trust in the automated extractions. This could cause model errors to be systematically propagated, leading consumers to make purchasing decisions based on incorrect information.
Finally, we address the environmental impact of this research. Our experiments involved training and evaluating multiple large-scale models, which consumed significant computational resources and energy. However, our findings make a positive contribution in this area by demonstrating that smaller, fine-tuned encoder models significantly outperform their much larger and more resource-intensive LLM counterparts for this task. This result advocates for a more sustainable approach to deploying NLP in production environments, showing that for specialised industrial tasks, more energy-efficient models can also be the most effective.
% The use of this technology warrants consideration of several ethical issues.
% \begin{itemize}
% \item \textbf{Algorithmic Bias:} There is a risk that the model may exhibit performance disparities across different types of content. For example, if it is more accurate on professionally formatted advertisements from large dealerships than on those from private sellers, its application could inadvertently disadvantage smaller sellers. Similarly, biases related to specific vehicle types or brands could arise if they are represented differently in the training data.
% \end{itemize}
% \section{Bibliographical References}
% \label{sec:reference}
% \renewcommand{\refname}{} % Add this line to remove the automatic title
% \bibliographystyle{lrec2026-natbib}
\bibliography{main_bib,lr_bib}
% \bibliographystyle{lrec2026-natbib}
% \bibliographystylelanguageresource{main_bib,lr_bib}
% % =============================================================================
% % APPENDIX
% % =============================================================================
\appendix
\section{Preliminary Self-Verification Experiments}
\label{sec:app:prelim_verification}
We conducted preliminary experiments to assess the effect of the LLM self-verification step. Verification improved overall F1 for both Mistral-7B and LLaMA3-8B, mainly by increasing precision while slightly reducing recall. The effect was strongest for Mistral, whose weighted F1 increased from 0.375 to 0.409. For LLaMA3, the gain was smaller, with weighted F1 increasing from 0.473 to 0.481. However, verification produced larger gains for some lower-support labels. For instance, with LLaMA3, the F1-score for \texttt{BATTERY\_CAPACITY} increased from 0.291 to 0.552 and for \texttt{BOOT\_SIZE} from 0.112 to 0.213.
\begin{table}[h!]
\centering
\small
\begin{tabular}{lcc}
\toprule
\textbf{Model} & \textbf{Initial F1} & \textbf{Verified F1} \\
\midrule
Mistral-7B, macro & 0.184 & 0.211 \\
Mistral-7B, weighted & 0.375 & 0.409 \\
LLaMA3-8B, macro & 0.204 & 0.209 \\
LLaMA3-8B, weighted & 0.473 & 0.481 \\
\bottomrule
\end{tabular}
\caption{Effect of self-verification in preliminary LLM experiments.}
\label{tab:prelim_verification}
\end{table}
\section{LLM Prompts}
\label{sec:appendix:llm-prompts}
We use a per-entity extraction strategy following \citet{wang2023gpt}. Each advertisement is processed once per entity type using the prompts below.
\subsection*{Extraction Prompt}
\noindent\textbf{System:}
\begin{quote}
\texttt{You are a NER Model. Your task is to label \{entity\_type\} entities in the given sentence.}
\end{quote}
\noindent\textbf{User:}
\begin{quote}
\texttt{Below are some examples.}\\[0.5em]
\texttt{\{few\_shot\_examples\}Input: \{text\}}\\
\texttt{Output:}
\end{quote}
\noindent where each few-shot example follows the template:
\begin{quote}
\texttt{Input: \{input\_text\}}\\
\texttt{Output: \{output\_text\}}
\end{quote}
\subsection*{Self-Verification Prompt}
\noindent\textbf{System:}
\begin{quote}
\texttt{You are a verification expert. Your task is to verify if an extracted entity is correct.}
\end{quote}
\noindent\textbf{User:}
\begin{quote}
\texttt{Original text:}\\
\texttt{\{text\}}\\[0.5em]
\texttt{Entity type: \{entity\_type\}}\\
\texttt{Extracted entity: \{entity\}}\\[0.5em]
\texttt{Is this extraction correct and accurate? Consider:}\\
\texttt{1. Is the entity actually present in the text?}\\
\texttt{2. Is the entity type classification correct?}\\
\texttt{3. Are the boundaries accurate (no extra or missing characters)?}\\[0.5em]
\texttt{Answer only with a ``yes'' if all statements are true, otherwise say ``no''. Do not say anything more.}\\[0.5em]
\texttt{Verification result:}
\end{quote}
\section{Rules-based Precision and Recall}
\label{app:rules_pr}
Table~\ref{tab:rules_precision_recall} show the per-label precision and recall scores for the best-performing rules-based methods.
\begin{table}[h!]
\centering
\caption{Precision and Recall Scores for Best Rules-Based Methods.}
\label{tab:rules_precision_recall}
\begin{tabular}{lcc}
\toprule
\textbf{Label} & \textbf{Precision} & \textbf{Recall} \\
\midrule
battery & 1.000 & 0.300 \\
model & 0.880 & 0.775 \\
body\_type & 0.824 & 0.500 \\
make & 0.666 & 0.949 \\
year & 0.631 & 0.997 \\
trim & 0.575 & 0.584 \\
fuel\_type & 0.479 & 0.717 \\
engine\_spec & 0.295 & 0.289 \\
transmission & 0.197 & 0.655 \\
number\_of\_seats & 0.154 & 0.800 \\
exterior\_colour & 0.015 & 0.134 \\
interior\_colour & 0.004 & 0.064 \\
\bottomrule
\end{tabular}
\end{table}
\section{Encoder Training Set Size Analysis}
\label{app:training_size}
Figure~\ref{fig:loss_vs_samples} shows the evaluation loss of the encoder models relative to the number of training samples. The plateauing of the loss curve suggests that the dataset size of 659 adverts was sufficient for the models to learn the primary patterns in the data.
\begin{figure*}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/LossvsTrainingSetSize.png} % You need to have this image file
\caption{Evaluation loss vs. number of training samples for encoder models.}
\label{fig:loss_vs_samples}
\end{figure*}
% \section{LLM Few-Shot Performance}
% \label{app:llm_fewshot}
% Figure~\ref{fig:fewshot_perf} shows the performance of LLaMA3 and Mistral on a 50-advert sample with a varying number of few-shot examples in the prompt.
% \begin{figure*}[h!]
% \centering
% \includegraphics[width=0.9\textwidth]{figures/performance vs fewshot llm.png} % You need to have this image file
% \caption{Performance metrics for Mistral and LLaMA3 using a varying number of few-shot examples.}
% \label{fig:fewshot_perf}
% \end{figure*}
% \section{LLM Per-Label Performance}
% \label{app:llm_perf}
% Tables~\ref{tab:llama3-perf} and \ref{tab:mistral-perf} detail the per-label performance for LLaMA3 and Mistral, respectively, showing both initial and self-verified results.
% \begin{table*}[h!]
% \centering
% \caption{Per-label precision, recall, and F1 for initial and verified predictions using LLaMA3.}
% \label{tab:llama3-perf}
% \resizebox{\textwidth}{!}{%
% \begin{tabular}{lcccccc}
% \toprule
% & \multicolumn{3}{c}{\textbf{All (Initial)}} & \multicolumn{3}{c}{\textbf{Verified}} \\
% \cmidrule(r){2-4} \cmidrule(r){5-7}
% \textbf{Label} & \textbf{Precision} & \textbf{Recall} & \textbf{F1} & \textbf{Precision} & \textbf{Recall} & \textbf{F1} \\
% \midrule
% MAKE & 0.669 & 0.828 & 0.740 & 0.685 & 0.826 & 0.749 \\
% MODEL & 0.686 & 0.646 & 0.665 & 0.706 & 0.642 & 0.672 \\
% TRIM & 0.528 & 0.549 & 0.538 & 0.576 & 0.536 & 0.555 \\
% FUEL\_TYPE & 0.393 & 0.614 & 0.479 & 0.500 & 0.599 & 0.545 \\
% ENGINE\_SPEC & 0.370 & 0.405 & 0.387 & 0.431 & 0.400 & 0.415 \\
% EXTERIOR\_COLOUR & 0.255 & 0.573 & 0.353 & 0.493 & 0.540 & 0.515 \\
% YEAR & 0.365 & 0.903 & 0.520 & 0.410 & 0.889 & 0.561 \\
% BODY\_TYPE & 0.205 & 0.524 & 0.294 & 0.255 & 0.456 & 0.328 \\
% INTERIOR\_COLOUR & 0.088 & 0.221 & 0.126 & 0.192 & 0.221 & 0.205 \\
% TRANSMISSION & 0.259 & 0.223 & 0.240 & 0.284 & 0.223 & 0.250 \\
% BATTERY\_CAPACITY & 0.211 & 0.471 & 0.291 & 0.667 & 0.471 & 0.552 \\
% BATTERY\_RANGE & 0.022 & 0.235 & 0.040 & 0.033 & 0.235 & 0.058 \\
% BOOT\_SIZE & 0.060 & 0.889 & 0.112 & 0.121 & 0.889 & 0.213 \\
% \bottomrule
% \end{tabular}}
% \end{table*}
% \begin{table*}[h!]
% \centering
% \caption{Per-label precision, recall, and F1 for initial and verified predictions using Mistral.}
% \label{tab:mistral-perf}
% \resizebox{\textwidth}{!}{%
% \begin{tabular}{lcccccc}
% \toprule
% & \multicolumn{3}{c}{\textbf{Mistral (Initial)}} & \multicolumn{3}{c}{\textbf{Mistral (Verified)}} \\
% \cmidrule(r){2-4} \cmidrule(r){5-7}
% \textbf{Label} & \textbf{P} & \textbf{R} & \textbf{F1} & \textbf{P} & \textbf{R} & \textbf{F1} \\
% \midrule
% MAKE & 0.687 & 0.572 & 0.624 & 0.742 & 0.572 & 0.646 \\
% MODEL & 0.679 & 0.386 & 0.493 & 0.724 & 0.381 & 0.499 \\
% YEAR & 0.236 & 0.763 & 0.360 & 0.364 & 0.763 & 0.493 \\
% TRIM & 0.381 & 0.283 & 0.325 & 0.407 & 0.279 & 0.331 \\
% TRANSMISSION & 0.195 & 0.340 & 0.248 & 0.436 & 0.329 & 0.375 \\
% ENGINE\_SPEC & 0.030 & 0.048 & 0.037 & 0.033 & 0.044 & 0.038 \\
% FUEL\_TYPE & 0.283 & 0.540 & 0.372 & 0.352 & 0.533 & 0.424 \\
% BODY\_TYPE & 0.089 & 0.504 & 0.152 & 0.107 & 0.485 & 0.175 \\
% BOOT\_SIZE & 0.019 & 0.888 & 0.037 & 0.097 & 0.777 & 0.172 \\
% BATTERY\_RANGE & 0.004 & 0.117 & 0.008 & 0.036 & 0.117 & 0.055 \\
% \bottomrule
% \end{tabular}}
% \end{table*}
\section{Hardware and Runtime Details}
\label{app:hardware}
All encoder-based transformer models were fine-tuned on an Apple M4 MacBook Pro equipped with 36GB unified memory. Inference for the encoder-based models required between a few seconds and approximately one minute to process the complete test set.
For open-source LLMs, models with fewer than 10 billion parameters were executed using vLLM~\citep{kwon2023efficient} on an NVIDIA RTX PRO 6000 GPU with 96GB VRAM. Larger models were executed on an NVIDIA H200 SXM GPU with 141GB VRAM.
LLM inference included both the entity extraction and entity verification stages. Processing the full evaluation set required approximately 1.5 hours per model. This corresponded to approximately $109 \times 15 \times 2 = 3,270$ model queries, where 109 is the number of test samples, 15 is the number of target entities, and 2 corresponds to the verification stage.
Closed-source models (e.g., GPT and Gemini) were accessed through their respective provider APIs. All LLMs were interfaced using an OpenAI-compatible API format. To ensure deterministic outputs, we set the temperature to 0.0 where supported by the API and used greedy decoding for all generations.
\end{document}