What Is The Performance Ceiling of My Classifier? Utilizing Category-Wise Influence Functions for Pareto Frontier Analysis
Extracted source text · arXiv 2510.03950
\documentclass{article}
\usepackage{arxiv}
% \usepackage{newtxtext,newtxmath}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{lipsum} % Can be removed after putting your text content
\usepackage{graphicx}
\usepackage{natbib}
\usepackage{doi}
%%%%%%%%%%%%%
\input{math_commands.tex}
\usepackage{hyperref}
\usepackage{url}
%\usepackage[ruled,vlined]{algorithm2e}
\usepackage{colortbl}
\usepackage{xcolor}
\usepackage[dvipsnames]{xcolor}
\usepackage{amsmath}
\usepackage{wrapfig}
\usepackage{siunitx}
\newcommand{\td}{{\bf\color{red} FIXME~}}
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{dsfont}
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\usepackage[textsize=tiny]{todonotes}
\usepackage{enumitem}
\newcommand{\anshuman}[1]{{\color{red}{\textbf{AC: }#1}}}
% \usepackage{cleveref} % smart cross-referencing
%%%%%%%%%%%%%
\title{"\textit{What Is The Performance Ceiling of My Classifier}?"\\ Utilizing Category-Wise Influence Functions for Pareto Frontier Analysis}
% Here you can change the date presented in the paper title
%\date{September 9, 1985}
% Or remove it
\date{}
%%%%%%%%
% Multiple affiliations variant of author block
\usepackage{authblk}
\renewcommand\Authfont{\bfseries}
\setlength{\affilsep}{0em}
% box is needed for correct spacing with authblk
%\newbox{\orcid}\sbox{\orcid}{\includegraphics[scale=0.06]{orcid.pdf}}
\author[1]{%
%\href{https://orcid.org/0000-0001-7979-1009}%{\usebox{\orcid}\hspace{1mm}
Shahriar Kabir Nahin%}%
}
\author[2]{%
% \href{https://orcid.org/0009-0006-1325-9507}{\usebox{\orcid}\hspace{1mm}
Wenxiao Xiao%
}
\author[2]{%
%\href{https://orcid.org/0000-0003-0118-3147}{\usebox{\orcid}\hspace{1mm}
Joshua Liu%
}
\author[1]{%
%\href{https://orcid.org/0000-0001-9376-2896}{\usebox{\orcid}\hspace{1mm}
Anshuman Chhabra%
}
\author[2]{%
%\href{https://orcid.org/0000-0003-0118-3147}{\usebox{\orcid}\hspace{1mm}
Hongfu Liu%
}
\affil[1]{Bellini College of AI, Cybersecurity, and Computing, University of South Florida\thanks{Email: \texttt{\{shahriarkabir, anshumanc\}@usf.edu}}}
\affil[2]{Michtom School of Computer Science, Brandeis University\thanks{Email: \texttt{\{wenxiaoxiao, joshliu, hongfuliu\}@brandeis.edu}}}
% Uncomment to override the `A preprint' in the header
%\renewcommand{\headeright}{Technical Report}
%\renewcommand{\undertitle}{Technical Report}
\renewcommand{\shorttitle}{"\textit{What Is The Performance Ceiling of My Classifier}?"}
%%% Add PDF metadata to help others organize their library
%%% Once the PDF is generated, you can check the metadata with
%%% $ pdfinfo template.pdf
% \hypersetup{
% pdftitle={A template for the arxiv style},
% pdfsubject={q-bio.NC, q-bio.QM},
% pdfauthor={David S.~Hippocampus, Elias D.~Striatum},
% pdfkeywords={First keyword, Second keyword, More},
% }
\begin{document}
\maketitle
\begin{abstract}
\looseness-1 Data-centric learning seeks to improve model performance from the perspective of data quality, and has been drawing increasing attention in the machine learning community. Among its key tools, influence functions provide a powerful framework to quantify the impact of individual training samples on model predictions, enabling practitioners to identify detrimental samples and retrain models on a cleaner dataset for improved performance.
However, most existing work focuses on the question ``what data benefits the learning model?” In this paper, we take a step further and investigate a more fundamental question: ``what is the performance ceiling of the learning model?” Unlike prior studies that primarily measure improvement through overall accuracy, we emphasize category-wise accuracy and aim for Pareto improvements, ensuring that every class benefits, rather than allowing tradeoffs where some classes improve at the expense of others. To address this challenge, we propose category-wise influence functions and introduce an influence vector that quantifies the impact of each training sample across all categories. Leveraging these influence vectors, we develop a principled criterion to determine whether a model can still be improved, and further design a linear programming–based sample reweighting framework to achieve Pareto performance improvements. Through extensive experiments on synthetic datasets, vision, and text benchmarks, we demonstrate the effectiveness of our approach in estimating and achieving a model's performance improvement across multiple categories of interest.
\end{abstract}
\section{Introduction}\label{introduction}
Data-centric learning has recently emerged as a central research topic in the machine learning community \citep{feldman2020neural, chhabra2024what, richardson2023add}, shifting attention from purely algorithmic model design towards improving the quality of training data. Unlike conventional preprocessing techniques such as normalization or outlier removal, that operate independently of the learning algorithm, data-centric learning is tightly coupled with the downstream model. Its primary goal is to assess whether each training sample is beneficial or detrimental with respect to a specific learning objective, thereby guiding principled data curation and optimization.
Sample influence estimation is a fundamental task in data-centric learning. A straightforward approach for estimating influence involves training the model twice-- once with and once without a target sample, and then comparing the change in model performance on a validation set. However, this approach is computationally expensive and often infeasible for large-scale datasets/models. To address this challenge, the seminal work of \citet{koh2017understanding} introduced \textit{influence functions}, a technique from robust statistics \citep{hampel1974influence, cook1982residuals} that enables approximate influence estimation without retraining. This approach allows efficient estimation of sample influence, inspiring a series of follow-up studies and variants on its applications on deep models~\citep{schioppa2022scaling,chhabra2025outlier,kwon2023datainf,wang2024data}.
Broadly speaking, sample influence estimation seeks to answer the question: “\textit{which training samples help the model, and which ones harm it}~\citep{chhabra2024what}?” In practice, samples identified as detrimental are removed, and the model is retrained on the refined dataset, often resulting in measurable performance gains. This naturally raises deeper questions: \textit{can the model’s performance be improved even further by iteratively repeating this process? And if so, what is the ultimate performance ceiling of the learning model?}
In this paper, we focus on the multi-class classification setting: given a training set, a validation set, and a classification model, we aim to answer two key questions from the data perspective: (1) has the classifier already reached its maximum potential performance, and (2) if not, how can its performance be further improved? Importantly, the notion of “improvement” in our work does not simply refer to an increase in overall accuracy. Instead, we adopt a \textit{Pareto improvement} perspective: we seek performance gains where every class benefits, avoiding tradeoff scenarios in which some classes improve at the expense of others. We summarize our contributions as follows:\vspace{-1mm}
\begin{itemize}[wide=10pt, leftmargin=*, nosep]
\item We tackle a fundamental yet largely overlooked question: “\textit{What is the performance ceiling of a classifier?}” Specifically, we determine whether a given classifier has already reached its maximum potential performance and, if not, how it can be further improved.\vspace{1mm}
\item We introduce category-wise\footnote{We use the terms \textit{category} and \textit{class} interchangeably throughout the paper.} influence functions to assess the model’s Pareto frontier on each category and systematically analyze its performance ceiling. Leveraging these influence scores, we further propose a linear programming–based sample reweighting framework to achieve Pareto performance improvements across classes of interest.\vspace{1mm}
\item We validate our category-wise influence functions on synthetic and benchmark datasets and present detailed case studies showing how to determine whether a classifier has reached its performance ceiling using our linear programming–based sample reweighting framework.
\end{itemize}
\section{Related Work}
\looseness-1 \textbf{Sample Influence Estimation.} Influence functions comprise a set of methods from robust statistics \citep{hampel1974influence, cook1982residuals} that can be used to approximately estimate influence without requiring retraining, i.e., they can help create a conceptual link that traces model performance to samples in the training set. For gradient-based models trained using empirical risk minimization, the seminal work by \citet{koh2017understanding} utilizes a Taylor-series approximation and LiSSA optimization \citep{agarwal2017second} to compute sample influences and relies on the Hessian matrix. Follow-up works such as Representer Point \citep{yeh2018representer} and Hydra \citep{chen2021hydra} improve influence estimation performance for deep learning models. More recently, efficient influence estimation methods such as DataInf \citep{kwon2023datainf}, Arnoldi iteration \citep{schioppa2022scaling}, and Kronecker-factored approximation curvature \citep{grosse2023studying} have been proposed which can even be employed for larger models. Some other approaches directly utilize the gradient space to measure influence \citep{pruthi2020estimating, charpiat2019input}, while others use some ensemble methods~\citep{bae2024training,kim2024gex, dai2023training}. Recent work has also found that \textit{self-influence} only on the training set can be a useful measure for detecting sample influence \citep{bejan2023make, thakkar2023self}. Influence functions have been widely used in the community for a number of data-centric applications \citep{feldman2020neural, chhabra2024what, richardson2023add}, but the focus of these works has predominantly been on using the overall accuracy of the model as a proxy for performance measurement. This contrasts with the main motivation in our paper, where we seek to study how different samples in the training set influence different categories/classes of the data. Such class-wise tradeoff analysis based on the Pareto frontier is of paramount importance in multiple applications, such as category-aware domain adaptation \citep{wenxiao_category} and fair classification \citep{fair_pareto, martinez2020minimax}.
\textbf{Pareto frontier Analysis.}
Pareto frontier analysis is widely employed in many domains, where multiple objectives need to be optimized simultaneously, necessitating solutions that can effectively measure the tradeoffs between each of the objectives. For instance, \cite{NEURIPS2023_960573a3} analyzed Pareto tradeoffs across resources for model training including data, model architectures, and computation. \cite{NEURIPS2019_685bfde0} introduced Pareto multi-task learning, where multiple Pareto-optimal solutions are generated efficiently to help practitioners choose the best one according to their tradeoff preferences. Other work, such as the Iterated Pareto Referent Optimization method proposed by \cite{10.5555/3709347.3743813} for multi-objective reinforcement learning, decomposes the search for optimal paths into a sequence of single objectives. Similarly, \cite{NEURIPS2023_32285dd1} handled distributional pareto-optimal policies in reinforcement learning under uncertainty. Pareto-optimal tradeoffs have also been studied in other problem domains, such as neural architecture search \citep{JMLR:v20:18-598}. As is evident, none of these works investigate category-wise tradeoffs during training and analysis of the Pareto-front for assessing the performance ceiling of a given classifier, unlike our work.
%Pareto frontier is one of the core concepts of multi-objective optimization. When multiple objectives are optimized together, there can be multiple optimal solutions for each of the objectives, but improvement in one objective may not be possible without making another worse. These solutions form the Pareto frontier.
%Specific to ensuring robustness in classification tasks, \cite{NEURIPS2020_61d77652} attempted to mitigate the accuracy-robustness tradeoff
\textbf{Other Data-Centric Learning.} Many works in data-centric learning study research questions beyond Pareto frontier analysis and category-wise influence estimation. \textit{Datamodels} \citep{ilyas2022datamodels} estimate training sample contributions as well, but only for one test sample at a time. Other approaches such as \citep{jain2022efficient, paul2021deep, killamsetty2021retrieve} aim to accelerate deep learning training time via subset/coreset selection. Data pruning, augmentation, and relabeling approaches \citep{yang2022dataset, tan2024data, kong2021resolving, chhabra2022fair, richardson2023add} and model pruning approaches \citep{lyu2023deeper} based on influence analysis have also been proposed. Another related area of research is \textit{active learning} \citep{cohn1996active}, which seeks to iteratively identify optimal samples to annotate given a large unlabeled training data pool \citep{isal, nguyen2022measure, wei2015submodularity}.
\section{Proposed Approach}
\subsection{Preliminaries}
Let $T$$=$$\{z_i\}_{i=1}^n$ be a training set, where $z_i$ $=$$(x_i, y_i)$ includes the input space sample features $x_i$ and output space label $y_i$. A classifier trained using empirical risk minimization on the empirical loss $\ell$ can be written as: $\Hat{\theta}$$=$$\argmin_{\theta \in \Theta} \frac{1}{n} \sum_{i=1}^n \ell(z_i; \theta)$. Influence functions~\citep{koh2017understanding} constitute methods from robust statistics~\citep{hampel1974influence,cook1982residuals,martin1986influence} that can help measure the effect of changing an infinitesimal weight of training samples on the model utility/performance. Downweighting a training sample $z_j$ by a very small fraction $\epsilon$ leads to a model parameter: $\Hat{\theta}(z_j; -\epsilon) = \argmin_{\theta \in \Theta} \frac{1}{n} (\sum_{i=1}^n \ell(z_i; \theta)$$-$$\epsilon \ell(z_j; \theta))$. By evaluating the limit as $\epsilon$ approaches 1, we can estimate the \textit{influence score} associated with the removal of $z_j$ from the training set in terms of loss on the validation $V$ set, without undertaking any computationally expensive leave-one-out re-training as:
\begin{equation}
\label{eq:influence}
\begin{aligned}
\mathcal{I}^{\Hat{\theta}}(z_j, V) = \sum\nolimits_{z \in V}\nabla_{\Hat{\theta}}\ell(z; \hat{\theta})^{\top}\mathbf{H}^{-1}_{\Hat{\theta}}\nabla_{\hat{\theta}}\ell(z_j; \Hat{\theta}).
\end{aligned}
\end{equation}
where $\nabla_{\Hat{\theta}}\ell(z_j; \Hat{\theta})$ is the gradient of sample $z_j$ to model parameters, and $\mathbf{H}_{\Hat{\theta}}$$=$$\sum_{i=1}^n \nabla_{\Hat{\theta}}^2 \ell(z_i; \Hat{\theta})$ denotes the Hessian. Higher values of $\mathcal{I}^{\Hat{\theta}}(z_j,V)$ indicate a more positively influential sample (i.e., one that decreases the overall classification loss) and conversely, lower values correspond to a more negatively influential sample. Also note that while influence functions have demonstrated their benefits in deep learning non-convex models, such as on BERT~\citep{han2020explaining}, ResNets~\citep{isal, yang2022dataset}, and CNNs~\citep{koh2017understanding, schioppa2022scaling}, there is ongoing research that studies their suitability to these models~\citep{bae2022if, basu2020influence, epifano2023revisiting, schioppa2024theoretical}. While this research question regarding applicability is not the focus of our paper we employ influence function formulations that have been shown to work well for deep models \citep{grosse2023studying, kwon2023datainf}.
\subsection{Research Question}
%previous work only talks about overall accuracy
% we argue that by sacrificing some categories performance can be improved -- having some weights downweight and upweight
% influence score vs influence vector
% 3.4 talk about pareto frontier -> all samples >0 beneficial samples to all categories
% 3.4 pareto frontier is the vector mixed with different values
% three types: one is all beneficial, one is all detrimental, one is mixed
% model can deliver multiple results and unlike previous works not just deliver one result -- not deterministic -- all dependent on the data -- whatever your goal is -- user can decide
% https://proceedings.mlr.press/v235/xiao24b.html
While measuring the influence of training samples on the predictive performance of a model can serve as a powerful tool for numerous data-centric learning applications, prior work \citep{koh2017understanding, chhabra2024what, han2020explaining, kwon2023datainf, schioppa2024theoretical, isal, yang2022dataset, chhabra2025outlier} has undertaken this analysis using only the \textit{overall accuracy} of the model as an indicator for performance. However, this is a restrictive scenario, and it can be beneficial for users/developers to consider a category-wise analysis for fine-grained understanding model's performance.
\looseness-1The basis for category-wise analysis stems from the fact that different training samples can lead the model to learn different predictive patterns, and hence, they can impact their own class and other classes in varying ways. Following this rationale, it should be possible to ascertain how different samples in the training set impact different categories of the dataset and therefore, utilize this information as a model developer to make relevant tradeoffs that are desirable for the given application at hand. These tradeoffs appear in numerous learning problems-- such as \textit{fair classification}~\citep{fair_pareto, martinez2020minimax}, where performance of multiple different categories need to be maximized jointly and \textit{category-aware active domain adaptation}~\citep{wenxiao_category}, where performance impacts across different categories/classes need to be individually identified, among several others. In contrast to past work which focuses solely on overall class accuracy, this forms the main motivation for our research focus in this work, where we seek to study how performance can be improved for certain classes by potentially sacrificing performance of others. The space of solutions that describe these different tradeoffs between categories is also known as the \textit{Pareto frontier} \citep{lotov2008visualizing}.
In this paper, we consider whether a given classifier has already reached its maximum potential performance, i.e., Pareto frontier, if not, how it can be further improved. We will now discuss our proposed methods for obtaining the Pareto frontier (and thus, the classifier's performance ceiling).
\subsection{Category-wise Influence Vector Estimation}
\looseness-1 We discuss our proposed approach for obtaining the Pareto frontier for the classifier across different categories. Analytically, the Pareto frontier for a training sample can be described using an \textit{influence vector} ${P}(z) \in \mathbb{R}^K$ of length $K$ (given $K$ classes/categories) where each cell ${P}^k(z)$ for $k \in [K]$ is the impact to the category $k$ when the sample $z$ is removed from the training set. We now develop category-wise influence scores for obtaining this Pareto frontier vector.
\looseness-1Let the subset of samples of set $S$ that belong to class $k$ be denoted as $S^k$. We can then measure the influence score for training sample $z$ as ${P}^k(z) = \mathcal{I}^{\Hat{\theta}}(z, S^k)$ for all $k \in [K]$ using Eq. (\ref{eq:influence}). It is important to note that the category-wise influence vector ${P}(z)$ is a useful solution aimed at answering the research question (\textit{what is the classifier's performance ceiling?}) we had formulated above. More specifically, the influence vector allows users/developers to easily gauge the classifier's performance ceiling-- if all values of ${P}(z) > 0$, the sample $z$ is \textit{beneficial} to all categories and if all values of ${P}(z) \leq 0$, the sample $z$ is \textit{detrimental} to all categories. The third case is when ${P}(z)$ takes on \textit{mixed} values that are both positive and negative.
% , thus outlining the Pareto frontier. This Pareto frontier vector can be utilized by developers to downweight/upweight samples from different categories according to the needs of their particular application, thereby improving performance as desired. Moreover, via the influence vector, the model can now be used to service the needs of a given application, and unlike past work, can be tuned as needed using the training data to fit any specific application requirements. Model performance is no longer deterministically tied to maximizing overall performance and can be decided by the users/developers assessing class-wise tradeoffs themselves.
Without loss of generalizability, consider the case scenario with two categories: $\mathcal{C}_1$ and $\mathcal{C}_2$. Given a model, we can calculate the influence vector for all training samples, and visualize the influence vector in Figure~\ref{fig:tradeoff}. Training samples located in the joint negative region are expected to hinder performance in both categories; thus, their removal could lead to simultaneous improvements in both aspects. Conversely, samples in the joint positive region can be further leveraged or retrained to enhance performance. These observations imply that the presence of samples in either the joint negative or positive regions indicates room for Pareto improvement, suggesting that the current model has not yet reached the Pareto frontier. However, an important question arises: \textit{if no samples exist in the joint negative or positive regions, does this imply that the Pareto frontier has been achieved?} This question motivates further investigation into the geometric conditions and theoretical guarantees for achieving the Pareto frontier.
\begin{wrapfigure}[12]{r}{2.3in}\vspace{-5mm}
\includegraphics[width=1\linewidth]{figures/tradeoff.png}\vspace{-8mm}
\caption{Influence space for 2 categories.} \label{fig:tradeoff}
\end{wrapfigure}
The answer to the above question is no, indicating that the current model can still be improved. At the individual sample level, if all samples may be located within tradeoff regions, involving or removing any sample will lead the tradeoff effect. However, when samples are considered collectively as a set, combining certain samples can yield a new sample that falls into the joint positive region. For example, as shown by the red arrows $z_1$ and $z_2$ in Figure~\ref{fig:tradeoff}, these two training samples lie in different tradeoff regions. Yet, when combined, the new sample becomes jointly beneficial to both categories. This observation suggests that a reweighting strategy \textit{can} be employed to modify the training set for Pareto improvements. Unlike approaches that simply decide whether to include or exclude individual samples, reweighting considers combinations of samples, providing a more general and flexible mechanism for optimization. Based on this insight, we can establish a new condition for achieving the Pareto frontier: all samples are close to the line $y=-x$ (the dashed green line). In the following, we will provide a reweighting framework based on the influence vector to achieve Pareto improvement.
% \textbf{Remark.} Note that we have an influence vector $P(z)$ detailing how each training sample impacts all other categories. However, in practice, and for visualization purposes, it might be more useful to aggregate this information across training samples from each class category, to obtain a Pareto frontier matrix $\mathcal{P} \in \mathbb{R}^{K\times K}$ that can describe how training samples from each category collectively affect different classes/categories. To do so, we can simply add influence vectors for samples from each category $i \in [K]$ as: $\mathcal{P}_i = \sum_{z \in T \cap y = i} {P}(z)$. Thus, $\mathcal{P}_i \in \mathbb{R}^K$ describes how samples from category $i$ affect performance across the $K$ different categories. Note that the diagonal entries of $\mathcal{P}$ detail the extent to which samples affect their own category.
\begin{algorithm}[t]
\caption{\textsc{Pareto-LP-GA}}
\label{alg:influence-ga}
\begin{algorithmic}[1]
\Statex \textbf{Input:} Training set $T$, model parameters $\Hat{\theta}^e$ for epoch $e$, influence vector $P(z), \forall z \in T$, target set $\mathcal{C}_{\text{target}}$, GA iterations $G$
\Statex \textbf{Output}: Optimized per-sample weights ${w^*}$, optimized class-wise performance thresholds ${\alpha^*}$
\State \textbf{initialize} population $\alpha^0 = \{\alpha^0_k\}_{k\in [K]}$ randomly
\For{$g \in [G]$}
\For{candidate threshold set $\alpha^g$ in current population}
\State \textbf{solve} the following LP to obtain $w$:
\begin{align*}
\max_{w} \quad & \sum\nolimits_{k \in \mathcal{C}_{\text{target}}} \sum\nolimits_{z_i\in T} w_i P^k(z_i) \\
\text{subject to} \quad & \sum\nolimits_{z_i\in T} w_i P^k(z_i) \geq \alpha^g_k \sum\nolimits_{z_i \in T} P^k(z_i), \quad \forall k \in [K] \nonumber
\end{align*}
\State $\Hat{\theta}^{e+1} \gets \textsc{TrainOneEpoch}(\Hat{\theta}^e, T, w)$%\Comment{Train for one epoch}
\State \textbf{compute} relative change in performance $\Delta_k^{e+1}$ from $\Hat{\theta}^e$ to $\Hat{\theta}^{e+1}$ for $k \in [K]$
\State \textbf{compute} fitness $F(\alpha^g)$ for current candidate threshold set as follows:
\begin{align*}
F(\alpha^g) = \frac{1}{|\mathcal{C}_{\text{target}}|} \sum_{k \in \mathcal{C}_{\text{target}}} \mathds{1}_{[\Delta_k^{e+1} \leq 0]} (-\infty) + \frac{1}{|\mathcal{C} \setminus \mathcal{C}_{\text{target}}|} \sum_{k \notin \mathcal{C}_{\text{target}}} \mathds{1}_{[\Delta_k^{e+1} < 0]} (\Delta_k^{e+1})
\end{align*}
\EndFor
\State \textbf{apply} selection, crossover, and mutation operations on population
\State \textbf{store} $\alpha^*$ and $w^*$ that maximizes fitness so far
\EndFor
\State \Return ${w^*}, {\alpha^*}$
\end{algorithmic}
\end{algorithm}
\subsection{\textsc{Pareto-LP-GA}: Improving Pareto Performance Using Influence Vectors}
We now aim to utilize our category-aware influence vector $P(z), \forall z\in T$ to improve the performance of a given classifier. More specifically, we utilize the category-wise influence vector to obtain per-sample weights for training losses that improve performance on a target subset of categories while controlling degradation on the remaining categories, via \textit{linear programming} (LP) \citep{dantzig2002linear}.
Furthermore, while influence scores are useful estimators for tuning category-wise performance, they cannot be utilized to obtain class-wise performance thresholds (equivalently, \textit{slack variables} in the linear program). Hence, we utilize a \textit{genetic algorithm} (GA) \citep{forrest1996genetic} to help identify these class-specific slack variables and ensure that the weighted LP obtains highly optimized solutions.
We thus propose our approach \textsc{Pareto-LP-GA}, which is an influence vector-guided linear programming approach for training sample weight optimization combined with a GA search.
We apply this weighted model in the context of two different settings: (a) \textit{Direct Improvement (DI)}: this refers to improving specific categories in a particular epoch as desired by the model developer where target categories are selected by the developer based on current per-class accuracy observations; and (b) \textit{Course Correction (CC)}: the developer while training the model observes accuracy drops in a certain epoch, and decides to modify the training trajectory for the detrimental epoch identified. A detrimental epoch means one after which the accuracy of some classes decreases significantly, indicating potential Pareto-optimal class/category tradeoffs.
The \textsc{Pareto-LP-GA} procedure is provided in Algorithm \ref{alg:influence-ga}. Denoting the full set of $K$ classes as $\mathcal{C}$, let $\mathcal{C}_{\text{target}}$ be the target subset of categories we wish to improve the performance for, while ensuring minimal performance degradation in other classes $\mathcal{C} \setminus \mathcal{C}_{\text{target}}$. Algorithm \ref{alg:influence-ga} takes in as input the training set $T$, the model parameters trained until a certain epoch $e$, our category-aware influence vector $P(z)$, the target classes $\mathcal{C}_{\text{target}}$ to improve performance for, the total iterations $G$. First, we initialize the population variable that controls for class-wise performance threshold along the Pareto frontier randomly, denoted as $\alpha_k, \forall k \in [K]$. Then, for each iteration of the genetic algorithm (GA), we solve a linear program (Line 4) that seeks to optimize for the per-sample weights by ensuring estimated performance via the category-aware influence vector is maximized on target classes while ensuring class-wise performance is above each category/class threshold ($\alpha_k$). Subsequently, we train the model for the next epoch ($e+1$) by applying the current optimized weight set (Line 5). We then measure relative change in performance ($\Delta_k^{e+1}$) between epochs $e \rightarrow e+1$ for \textit{Direct Improvement} (for \textit{Course Correction} this performance change $\Delta_k^{e+1}$ is instead calculated between the original epoch $e+1$ and the newly weighted epoch $e+1$; the rest of the procedure remains identical). Now, while we have optimized the weight set, we still need to obtain the optimal class-wise thresholds via the GA search. Hence, we formulate the fitness function (Line 7) such that if performance for $\mathcal{C}_{\text{target}}$ decreases at all (i.e., $\Delta_k^{e+1} \leq 0$) the fitness is set to a large-magnitude negative value (denoted as $-\infty$). Moreover, for non-target classes $\mathcal{C} \setminus \mathcal{C}_{\text{target}}$, if performance decreases, the fitness score reflects the degree of degradation. Thus, the GA $\alpha$ search also optimizes for performance improvement along desired target classes while ensuring minimal performance reduction across non-target classes. Post this step, we apply the standard GA operations (selection, crossover, mutation, etc.) on the population. Eventually, the algorithm return the optimized weight set $w^* = [w_1^*, \ldots, w_n^*]$ for the training set, that will be applied to the loss computation during next epoch training.
% Using our influence score ${P}^k(z)$ which represents the influence of training sample $z$ on category $k \in [K]$, and $\alpha^k$ be a class-wise threshold on performance, we can write the linear programming problem for obtaining a weight set $w$ as follows:
% \begin{align}
% \max_{w} \quad & \sum_{k \in \mathcal{C}_{\text{target}}} \sum_{i=1}^n w_i P^k(z_i) \label{eq:LP-equation}\\
% \text{subject to} \quad & \sum_{i=1}^n w_i P^k(z_i) \geq \alpha_k \sum_{i=1}^n P^k(z_i), \quad \forall k \in [K] \nonumber
% \end{align}
% Clearly, the choices for $\alpha_k$ crucially determine and control the performance impact of the obtained weight set on other classes. Thus, we utilize and apply a Genetic Algorithm (GA) \cite{forrest1996genetic} to obtain $\alpha_k$ such that performance for classes in $\mathcal{C}_{\text{target}}$ is increased while controlling the degradation/tradeoff of the remaining classes $\mathcal{C} \setminus\mathcal{C}_{\text{target}}$. Moreover, let $\Delta^{k}_e$ be the relative change in accuracy for class $k$ after we apply our weight set from the LP aboveto an epoch $e$, and $\tau$ be some tolerance parameter, then the fitness function of the GA can be written as:
% \begin{equation}
% F(\alpha) = \frac{1}{|[K] \setminus \mathcal{C}_{\text{target}}|} \sum_{k \notin \mathcal{C}_{\text{target}}} \mathbf{1}(\Delta^k_e + \tau < 0) \cdot \Delta^k_e + \frac{1}{|\mathcal{C}_{\text{target}}|} \sum_{k \in \mathcal{C}_{\text{target}}} \mathbf{1}(\Delta^e_k \leq 0) \cdot (-\infty),
% \label{eq:fitness-function-1}
% \end{equation}
% If $\tilde \alpha$, $\tilde w$ and $\tilde \theta$ denote the optimal solutions provided by the GA then the overall \textsc{Pareto-LP-GA} method can be summarized as algorithm \ref{alg:influence-ga}.
% \begin{algorithm}[t]
% \caption{\textsc{Pareto-LP-GA}}
% \label{alg:influence-ga}
% \KwIn{Training dataset $T$; model parameters $\theta$; influence scores $P(z)$; target set $\mathcal{C}_{\text{target}}$; iterations $G$}
% \KwOut{Optimized ${\alpha^*}$; optimized per-sample weights ${w^*}$}
% % Compute baseline accuracies $a^e_c$ for all $c \in [K]$
% Initialize population $\alpha$ randomly\;
% \For{$g \gets 1$ \KwTo $G$}{
% \ForEach{candidate threshold set $\alpha = \{\alpha_k\}_{k\in K}$ in current population}{
% Solve the following LP to obtain $w$:\;
% \begin{align}
% \max_{w} \quad & \sum_{k \in \mathcal{C}_{\text{target}}} \sum_{i=1}^n w_i P^k(z_i) \label{eq:LP-equation}\\
% \text{subject to} \quad & \sum_{i=1}^n w_i P^k(z_i) \geq \alpha_k \sum_{i=1}^n P^k(z_i), \quad \forall k \in [K] \nonumber
% \end{align}
% $\theta \gets \textsc{TrainOneEpoch}(\theta, T, w)$\;
% Compute $\Delta^k$ for each $k \in [K]$\ with trained $\theta$\;
% Compute fitness $F(\alpha)$ using Equation~\eqref{eq:fitness-function-1}\;
% }
% Apply selection, crossover, and mutation\;
% Preserve elites and inject random candidates\;
% }
% \Return{$\tilde{\alpha}, \tilde{w}, \tilde{\theta}$}\;
% \end{algorithm}
% As can be seen in subfigures \textbf{B} and \textbf{C} for the linearly separable dataset, a number of mislabeled samples possess negative influence for both classes/categories and removing those should improve overall performance. Moreover, as is evident from the training set distribution, a majority of clean samples have positive influence for one class but negative influence for the other class, indicating the Pareto frontier. Similarly, subfigures \textbf{E} and \textbf{F} show that the influence scores for samples in the non-linearly separable dataset are positively influential for one class but negatively influential for the other, indicating Pareto-optimality. Moreover, the samples with the highest influence magnitudes are localized to around the decision boundary and are misclassified from each class owing to the linearity of the model.
\begin{figure*}[t]
\centering
\includegraphics[width=0.89\linewidth]{figures/toy_example2-new.png}\vspace{-4mm}
\caption{Validation of our category-wise influence function methods for analyzing the Pareto frontier on two synthetic binary classification datasets with logistic regression. Subfigures \textbf{A-C} showcase results on a synthetic dataset that is linearly separable and contains noisy detrimental training samples, where performance can improve by mislabeled sample removal. Subfigures \textbf{D-F} detail results for our method on a non-linearly separable dataset without any noisy samples, where performance improvements cannot be made for either class without sacrificing performance for the other. Subfigures \textbf{A} and \textbf{D} showcase the distribution of training samples for each of the two datasets with blue and orange denoting the ground-truth class labels. Subfigures \textbf{B} and \textbf{E} showcase the category-wise influence score distribution for both datasets. Further, subfigures \textbf{C} and \textbf{F} map the influence values to the training samples using color intensity in accordance with class colors to denote the influence magnitudes, where the original class color means positive and red color means negative. }
\label{fig:toy}\vspace{-4mm}
\end{figure*}
\section{Synthetic Data Verification}
\looseness-1 We now analyze the efficacy of our criterion on the model's performance ceiling with two synthetic binary classification datasets using logistic regression, as shown in Figure \ref{fig:toy}. The top row subfigures \textbf{A-C} denote a linearly separable dataset which consists of noisy samples that are mislabeled. The dataset consists of 300 blue class samples and 300 orange class samples,\footnote{The descriptions of data and models can be seen in Appendix~\ref{appendix:models_datasets}.} generated using a circular uniform distribution. Noises were added to the training set by choosing random points from each group, 50 from blue and 20 from orange, and then flipping their label. Clearly, removing these noisy samples should improve performance and our category-wise influence functions should reflect their detrimental influence for both classes/categories. The large majority of non-noisy samples should positively influence one of the classes and negatively influence the other class. Hence, removing these samples should sacrifice performance for one of the classes and should be reflected in the category-wise influence distribution. As can be observed in subfigures \textbf{B} and \textbf{C} which showcase the category-wise influences and the samples corresponding to those influence values, respectively, this is indeed the case. Essentially, the mislabeled noisy samples from both categories possess negative influence for both categories, and hence, removing those should improve performance as expected. Additionally, a large number of non-noisy samples are positively/negatively influential for the blue/orange class (and vice-versa), validating our criterion further.
\looseness-1The bottom row consisting of subfigures \textbf{D-F} denotes a dataset that is non-linearly separable where performance improvements for both categories/classes cannot be jointly made. Here, the dataset consists of 350 samples for the blue category and 350 samples for the orange category, generated using a circular uniform distribution. For the orange samples, the radius of the distribution was changed depending on the angle from the center. In the ideal scenario, the Pareto frontier obtained via our category-wise influence functions should indicate that samples are positively influential for one class and negatively influential for the other class. Moreover, the samples with the maximum influence magnitude are those that appear around the decision boundary and ones that will inadvertently be misclassified due to the linear logistic regression classifier. In subfigures \textbf{E} and \textbf{F}, we can see that this holds true with the Pareto frontier visualized. The influence vectors of all training samples form a roughly straight line. As we demonstrate through these results, category-wise influence vectors can reveal the Pareto frontier for the two datasets accurately, and help users/developers make tradeoffs as required depending on the needs of their given application.
\begin{figure*}[t]
\centering
\includegraphics[width=0.92\linewidth]{figures/cifar_benef-2.png}
\includegraphics[width=0.92\linewidth]{figures/cifar_detri-2.png}\vspace{-2mm}
\caption{Real-world data experiments on \textit{CIFAR10}~\citep{krizhevsky2009learning} image dataset.}
\label{fig:cifar}\vspace{-4mm}
\end{figure*}
\section{Real-world Data Experiments}
In this section, we present our experimental results on real-world datasets in two parts: a validation of the effectiveness of category-wise influence functions and an evaluation of our \textsc{Pareto-LP-GA} for Pareto performance improvement.
\subsection{Category-wise Influence Functions}
\looseness-1 We evaluate the category-wise influence using four widely adopted benchmark datasets with 4-10 categories: two vision datasets (\textit{CIFAR-10}~\citep{krizhevsky2009learning} and \textit{STL-10}~\citep{pmlr-v15-coates11a}) and two text datasets (namely, \textit{Emotion}~\citep{saravia2018carer} and \textit{AG$\_$News}~\citep{zhang2015character}). For calculating category-wise sample influence vectors, we employ EKFAC~\citep{grosse2023studying} due to its fast implementation on deep models.
First, we evaluate whether category-wise influence serves as an effective indicator for measuring performance changes. To achieve this, we select the top 10\% of beneficial and detrimental samples for each category, remove these samples from the training set, and retrain the model to observe the performance change. Figures~\ref{fig:cifar} and~\ref{fig:emotion} illustrate the performance changes for \textit{CIFAR-10} and \textit{Emotion}. Similar phenomena are observed for \textit{STL-10} and \textit{AG$\_$News}, and these results are deferred to Appendix~\ref{app:real}. In these figures, Subfigures \textbf{A} and \textbf{D} illustrate the cumulative influence values of the removed training samples for beneficial and detrimental samples, respectively. Subfigures \textbf{B} and \textbf{E} display the corresponding performance changes after retraining without these samples. The results are presented as heat maps, with diagonal blocks representing the targeted categories, highlighted using bold boundaries and distinct textures.
The phenomena observed in the diagonal blocks reveal clear and consistent patterns: removing beneficial samples with positive influence on a target category leads to a performance drop in the corresponding category, while removing detrimental samples with negative influence results in a performance increase. This confirms the effectiveness of category-wise influence in predicting performance changes for both beneficial samples and detrimental samples.
Beyond the diagonal blocks, the patterns are more mixed, as the selected samples based on one category may have varying impacts on other categories. To further analyze these non-diagonal patterns, we plot scatter diagrams of the cumulative influence for each category against their performance changes in Subfigures \textbf{C} and \textbf{F}. Subfigure \textbf{C} is derived from Subfigures \textbf{A} and \textbf{B}, while Subfigure \textbf{F} is based on Subfigures \textbf{D} and \textbf{E}. The Spearman correlation coefficient exceeding 0.8 indicates a strong relationship between sample influence and performance change, not only within the target category but also across categories. This demonstrates that category-wise influence functions can serve as an effective indicator for inferring performance changes. Moreover, they can be utilized to evaluate whether a classifier has reached its performance ceiling across different categories, providing valuable insights for targeted model improvement. We also present the numbers of beneficial and detrimental samples for each classes in Appendix~\ref{app:real}, where the beneficial samples mainly come from their category but detrimental samples come from other categories.
\begin{figure*}[t]
\centering
\includegraphics[width=0.92\linewidth]{figures/emo_benef-2.png}
\includegraphics[width=0.92\linewidth]{figures/emo_detri-2.png}\vspace{-2mm}
\caption{Real-world data experiments on \textit{Emotion}~\citep{saravia2018carer} text dataset.}
\label{fig:emotion}\vspace{-4mm}
\end{figure*}
\subsection{Performance Ceiling Check and Improvement}
%In this part, we provide two case studies on checking the classifier's performance ceiling.
%\anshuman{Work in Progress}
% first to describe the experimental setup and protocol, dataset used
% why cifar? mention that accuracy across classes is already very high for other datasets -- give examples and show why there is not a lot of room for pareto optimal improvement
% describe the two settings: (a) directly improve epoch as desired by model developer and (b) go back and "course correct" to change the training trajectory when a detrimental epoch is discovered
% mention why we are choosing the target classes that we used -- mention this based on class performance %
% then talk about results in the context of the tables
\looseness-1We evaluate our \textsc{Pareto-LP-GA} method using the \textit{CIFAR10} dataset and ResNet model to showcase performance improvements for both \textit{Direct Improvement} (\textit{DI}) and \textit{Course Correction} (\textit{CC}) settings. The reason we chose \textit{CIFAR10} for experiments was because during training multiple epochs exhibit major Pareto tradeoffs across categories, while ensuring their is room for potential improvement. In contrast, for our text datasets (\textit{Emotion} and \textit{AG\_News}), the NLP models achieved accuracies exceeding 90\% across all classes within the first epoch, leading to little room for Pareto frontier improvement. Similarly, \textit{STL-10} consists of \textit{cleaner} images than \textit{CIFAR10}, generally leading to better performance.
\begin{table}[t]
\vspace{2.5mm}
\centering
\caption{Comparison of category-wise accuracies for performance improvement in the \textit{Direct Improvement} (left) and \textit{Course Correction} (right) settings. Target classes are highlighted in blue. As can be observed, performance in target categories increases significantly while non-target classes see minimal reductions (or potential gains).}
\par\vspace{-2mm}
\label{tab:DI-CC-comparison}
\begin{minipage}[t]{0.44\textwidth}
\centering
% \textbf{(a) Direct÷ Improvement}
\label{tab:DI-11}
\resizebox{\textwidth}{!}{%
\begin{tabular}{c*{2}{S[table-format=1.3]}c}
\hline
\textbf{Category} & \textbf{Epoch-10} & \textbf{Epoch-11 (DI)} & \textbf{Change (\%)} \\ \hline
\rowcolor{blue!8}
0 & 0.699 & 0.811 & \textbf{\textcolor{ForestGreen}{+16.02}} \\
1 & 0.888 & 0.881 & -0.78 \\
\rowcolor{blue!8}
2 & 0.667 & 0.743 & \textbf{\textcolor{ForestGreen}{+11.39}} \\
3 & 0.647 & 0.632 & -2.31 \\
4 & 0.729 & 0.720 & -1.2 \\
5 & 0.755 & 0.755 & \textcolor{ForestGreen}{+0.00} \\
6 & 0.802 & 0.845 & \textcolor{ForestGreen}{+5.73} \\
7 & 0.849 & 0.848 & -0.11 \\
8 & 0.948 & 0.920 & -2.90 \\
9 & 0.817 & 0.818 & \textcolor{ForestGreen}{+0.12} \\ \hline
\end{tabular}}
\end{minipage}
\hfill
\begin{minipage}[t]{0.54\textwidth}
\centering
% \textbf{(b) Course Correction}
\label{tab:CC-16}
\resizebox{\textwidth}{!}{%
\begin{tabular}{c*{2}{S[table-format=1.3]}cc}
\hline
\textbf{Category} & \textbf{Epoch-15} & \textbf{Epoch-16} & \textbf{Epoch-16 (CC)} & \textbf{Change (\%)} \\ \hline
0 & 0.876 & 0.876 & 0.889 & \textcolor{ForestGreen}{+1.48} \\
1 & 0.866 & 0.868 & 0.870 & \textcolor{ForestGreen}{+0.23} \\
2 & 0.651 & 0.741 & 0.736 & -0.67 \\
3 & 0.582 & 0.678 & 0.677 & -0.14 \\
4 & 0.729 & 0.783 & 0.785 & \textcolor{ForestGreen}{+0.25} \\
\rowcolor{blue!8}
5 & 0.821 & 0.785 & 0.798 & \textbf{\textcolor{ForestGreen}{+1.65}} \\
6 & 0.859 & 0.859 & 0.855 & -0.46 \\
\rowcolor{blue!8}
7 & 0.885 & 0.837 & 0.848 & \textbf{\textcolor{ForestGreen}{+1.31}} \\
8 & 0.909 & 0.929 & 0.917 & -1.29 \\
\rowcolor{blue!8}
9 & 0.917 & 0.864 & 0.888 & \textbf{\textcolor{ForestGreen}{+2.77}} \\ \hline
\end{tabular}}
\end{minipage}\vspace{-2mm}
\end{table}
% {0: 0.898, 1: 0.867, 2: 0.732, 3: 0.665, 4: 0.776, 5: 0.797, 6: 0.85, 7: 0.849, 8: 0.918, 9: 0.895}
% {0: 0.889, 1: 0.87, 2: 0.736, 3: 0.677, 4: 0.785, 5: 0.798, 6: 0.855, 7: 0.848, 8: 0.917, 9: 0.888}
Before demonstrating performance improvements, we first examine whether the influence vectors of the training samples approximately lie on a hyperplane. To this end, we apply Principal Component Analysis \citep{wold1987principal} and compute the explained variance ratio of the first principal component. Across all targeted cases, this ratio consistently exceeds 0.2, indicating that the influence vectors do not fit a hyperplane and suggesting room for Pareto improvement.
We present results for \textit{DI} and \textit{CC} in Table~\ref{tab:DI-CC-comparison}.
For \textit{DI} in \textit{CIFAR10}, we identified two categories (0 and 2) with relatively lower accuracy after observing performance at \textit{Epoch 10} (0.699 for class-0, and 0.667 for class-2). These two classes will constitute our target categories. Then, we employ \textsc{Pareto-LP-GA} to obtain a weight set and apply weighted training to achieve \textit{Epoch 11}. The results demonstrate that \textsc{Pareto-LP-GA} significantly enhances performance on the target categories, achieving improvements of 16.02\% and 11.39\% for classes 0 and 2, respectively. Notably, several non-target categories also experienced performance gains, such as category 6, which improved by 5.73\%. However, for the non-target categories, performance degradation remains very minimal, showcasing the performance ceiling of the classifier via the Pareto improvement.
For \textit{CC}, we identified \textit{Epoch 16} as detrimental for classes 5, 7, and 9, where accuracies declined substantially (from \textit{Epoch 15}). That is, between Epochs 15 and 16, performance for class 5 drops from 0.821 $\rightarrow$ 0.785; for class 7 drops from 0.885 $\rightarrow$ 0.837; and for class 9, drops from 0.917 $\rightarrow$ 0.864. Thus, these classes constitute our target categories for \textit{CC}. We obtain the weight set using \textsc{Pareto-LP-GA} and after applying weighted training, we seek to obtain a new Epoch 16 that optimizes for the Pareto-optimal class tradeoffs as desired. This can be observed in our results -- accuracy improves by 1.65\%, 1.31\%, and 2.77\% in target categories 5, 7, and 9, respectively, reversing the original trends and bringing slight improvements. Additionally, class-0, class-1, and class-4 experienced 1.48\%, 0.23\%, and 0.25\% performance enhancement, respectively, and there were only modest accuracy reductions in other categories ($\leq 1.29\%$). Overall, our proposed \textsc{Pareto-LP-GA} method demonstrates its ability to enhance training performance in specific target categories with minor accuracy tradeoffs in non-target categories.
\section{Conclusion}
In this paper, we extended the conventional influence function to a category-wise influence function and introduced the concept of an influence vector, which quantifies the impact of each training sample across all categories. Building on this formulation, we analyzed whether a classifier has already reached its maximum potential performance: formally, the Pareto frontier across categories, and design a linear programming–based sample reweighting framework for Pareto improvements. We validated the correctness of our performance-ceiling criterion through experiments on synthetic datasets, and further demonstrated the effectiveness of the proposed category-wise influence function on real-world datasets. Finally, we presented case studies that showcase how our sample reweighting approach can lead to tangible Pareto improvements across multiple categories.
% \newpage
% \section{Reproducibility Statement}\nonumber
% We provide our code and implementation in a GitHub repository: \url{https://anonymous.4open.science/r/Classifier-Performance-Ceiling-C7FB/}. All the experiments were run multiple times to ensure reproducibility. Additionally, any parameters required for reproducibility (e.g., seeds) are provided in the repo and in Appendix \ref{appendix:models_datasets}. The experiments were conducted on a Linux server with 8x NVIDIA A6000 GPUs.
% \section{Ethics Statement}\nonumber
% In this paper, we study category-aware influence functions for classification models and propose methods that can utilize them for Pareto-optimal performance improvements across classes. Our work is beneficial for the community as it enables developers to navigate the performance ceiling of the classifier depending on the needs of their given application. Our methods are broadly applicable to any classification application domains (e.g., text and vision), further underscoring the positive impact of our methods.
\bibliography{references}
\bibliographystyle{unsrtnat}
\newpage
\appendix
\section*{Appendix}
\section{Detailed Information on Datasets and Model Training}\label{appendix:models_datasets}
We describe dataset details, model training, and other information used in the main paper, below.
\subsection{Datasets} \label{appendix:datasets}
We introduce the synthetic datasets and real-world datasets used in the experiments.
\subsubsection{Synthetic Datasets}
The datasets in Figure~\ref{fig:toy}\textbf{A} is synthetic dataset that is linearly separable and contains noisy detrimental training samples. It consists of 300 blue
class samples and 300 orange class samples, generated using a circular uniform distribution. Noise was added to the training set by choosing random points from each group, 50 from blue and 20 from orange, and then flipping their labels.
Similarly, the datasets in Figure~\ref{fig:toy}\textbf{D} is is non-linearly separable where performance improvements for both categories cannot be joint made. This datasets consists of 350 samples for the blue category and 350 samples for the orange category, generated using a circular uniform distribution. For the orange samples, the radius of the distribution was changed depending on the angle from the center. There is no flip-labeled samples in this dataset.
\subsubsection{Real-world Datasets}
We use four widely adopted real-world benchmark datasets with 4-10 categories: two image datasets (\textit{CIFAR-10}~\citep{krizhevsky2009learning} and \textit{STL-10}~\citep{pmlr-v15-coates11a}) and two text datasets (\textit{Emotion}~\citep{saravia2018carer} and \textit{AG$\_$News}~\citep{zhang2015character}).
\subsection{Models} \label{appendix:models}
We train \textit{bert-base-cased}~\citep{DBLP:journals/corr/abs-1810-04805} for the \textit{Emotion} and \textit{AG\_News} NLP datasets, and \textit{ResNet-9}~\citep{he2016deep} for the \textit{CIFAR10} and \textit{STL-10} vision datasets in our experiments.
\subsection{Parameter Details} \label{appendix:params}
%\td
Table \ref{tab:hyperparameters} summarizes the hyperparameters used in all of our experiments.
% \begin{table}[t]
% \centering
% \caption{Hyperparameters for each experiment.}\vspace{1mm}
% \resizebox{0.4\textwidth}{!}{%
% \begin{tabular}{cl}
% \hline
% \textbf{Experiment} & \textbf{Hyperparameters} \\ \hline
% \textit{CIFAR10} &
% train\_batch\_size: 512 \\
% & eval\_batch\_size: 1024 \\
% & learning\_rate: 0.4 \\
% & weight\_decay: 0.001 \\
% & num\_train\_epochs: 25 \\
% %& seed: 1004 \\
% \hline
% \textit{STL-10} &
% train\_batch\_size: 512 \\
% & eval\_batch\_size: 1024 \\
% & learning\_rate: 0.4 \\
% & weight\_decay: 0.001 \\
% & num\_train\_epochs: 30 \\
% %& seed: 1004 \\
% \hline
% \textit{Emotion} &
% train\_batch\_size: 512 \\
% & eval\_batch\_size: 1024 \\
% & learning\_rate: 0.4 \\
% & weight\_decay: 0.001 \\
% & num\_train\_epochs: 25 \\
% %& seed: 1004 \\
% \hline
% \textit{AG\_News} &
% train\_batch\_size: 512 \\
% & eval\_batch\_size: 1024 \\
% & learning\_rate: 0.4 \\
% & weight\_decay: 0.001 \\
% & num\_train\_epochs: 25 \\
% %& seed: 1004 \\
% \hline
% \textsc{Pareto-LP-GA} &
% train\_batch\_size: 512 \\
% & eval\_batch\_size: 1024 \\
% & learning\_rate: 0.0001 \\
% & weight\_decay: 0.0001 \\
% & num\_train\_epochs: 20 \\
% %& seed: 1004 \\
% & GA iterations: 20 \\
% & Population\_size: 24 \\
% & crossover\_rate: 1.0 \\
% & mutation\_rate: 0.25 \\
% & mutation\_strength: 0.25 \\
% & num\_elites: 6 \\
% & num\_mutated\_elites: 6 \\
% & num\_randoms: 6 \\
% & num\_crossover\_children: 6 \\
% \hline
% \end{tabular}
% \label{tab:hyperparameters}}
% \end{table}
\begin{table}[h]
\centering
\caption{Hyperparameters for each dataset and algorithm.}\vspace{1mm}
\resizebox{0.7\textwidth}{!}{%
\begin{tabular}{cl|cl}
\hline
\textbf{Dataset} & \textbf{Hyperparameters} & \textbf{Algorithm} & \textbf{Hyperparameters} \\ \hline
\textit{CIFAR10} &
train\_batch\_size: 512 & \textsc{Pareto-LP-GA} & train\_batch\_size: 512 \\
& eval\_batch\_size: 1024 & & eval\_batch\_size: 1024 \\
& learning\_rate: 0.4 & & learning\_rate: 0.0001 \\
& weight\_decay: 0.001 & & weight\_decay: 0.0001 \\
& num\_train\_epochs: 25 & & num\_train\_epochs: 20 \\
%& seed: 1004 \\
\cline{1-2}
\textit{STL-10} & train\_batch\_size: 512 & & GA iterations: 20 \\
& eval\_batch\_size: 1024 & & Population\_size: 24 \\
& learning\_rate: 0.4 & & crossover\_rate: 1.0 \\
& weight\_decay: 0.001 & & mutation\_rate: 0.25 \\
& num\_train\_epochs: 30 & & mutation\_strength: 0.25 \\
%& seed: 1004 \\
\cline{1-2}
\textit{Emotion} &
train\_batch\_size: 512 & & num\_elites: 6 \\
& eval\_batch\_size: 1024 & & num\_mutated\_elites: 6 \\
& learning\_rate: 0.4 & & num\_randoms: 6 \\
& weight\_decay: 0.001 & & num\_crossover\_children: 6 \\
& num\_train\_epochs: 25 \\
%& seed: 1004 \\
\cline{1-2}
\textit{AG\_News} &
train\_batch\_size: 512 \\
& eval\_batch\_size: 1024 \\
& learning\_rate: 0.4 \\
& weight\_decay: 0.001 \\
& num\_train\_epochs: 25 \\
%& seed: 1004 \\
\hline
\end{tabular}
\label{tab:hyperparameters}}
\end{table}
\section{Additional Experiments}
\subsection{Additional Experiments on Synthetic Data}
In the experiment shown in Figure~\ref{fig:toy}, Subfigures \textbf{ABC}, the technique of identifying the noisy training points using their influence scores was discussed. Specifically, the removal of points which had negative influence scores with respect to both groups. In this experiment, we continue this path through to completion on the same dataset situation. We present the results in Figure~\ref{fig:toy2}. The dataset and model used are identical to the same from Subfigures \textbf{ABC} in Figure~\ref{fig:toy}. The following method was applied: Create a linear regression model on the training set, calculate the category-aware influence scores of each training point, remove any training points with negative scores, and retrain the model on the new training set. It is shown that with each iteration of this technique, the model moves closer to the performance frontier.
\begin{figure*}[t]
\centering
\includegraphics[width=0.92\linewidth]{figures/appendixa-new.png}\vspace{-4mm}
\caption{Experiment demonstrating the use of the category-wise influence function for dataset augmentation. The dataset used in this demonstration is identical to that in the top row of Figure 1. Each row displays the state of the dataset, the changes that our method will make, and the result, of the dataset trimming procedure discussed in Appendix A. Subfigures A, D, and G display the state of the training dataset. The color of each point indicates its label. The linear decision boundary is drawn, and its accuracy across both classes is shown in the legend. Subfigures B, E, and H show the category-wise influence score of each training point. Training data points in green are indicated by the score to be detrimental to model performance on both classes. These points will be removed by the improvement procedure. Subfigures C, F, and H show the training dataset after removal. The resulting decision boundary and its accuracy is also indicated. Note how the noise furthest from the decision boundary is removed first, since these have the largest effect on the decision boundary. Additionally, through each iteration, the accuracy of the linear model is improved when using the trimmed dataset. After three iterations, all noise has been removed and the model is at the performance ceiling.}
\label{fig:toy2}
\end{figure*}
\subsection{Additional Experiments on Real-world Data}\label{app:real}
Figures~\ref{fig:stl10} and~\ref{fig:agnews} illustrate the performance changes for \textit{STL-10} and \textit{AG$\_$News} by removing beneficial and detrimental samples according to influence vectors. They provide similar phenomena with the results on \textit{CIFAR-10} and \textit{Emotion}, which we illustrate in the main paper.
Figure~\ref{fig:number_benef_detri} present the numbers of beneficial and detrimental samples for
each classes on \textit{CIFAR-10} dataset, where the beneficial samples mainly come from their category but detrimental samples come from other categories.
\begin{figure*}[t]
\centering
\includegraphics[width=0.92\linewidth]{figures/stl10_benef-2.png}
\includegraphics[width=0.92\linewidth]{figures/stl10_detri-2.png}\vspace{-4mm}
\caption{Real-world data experiments on \textit{STL-10}~\citep{pmlr-v15-coates11a} image dataset.}
\label{fig:stl10}
\end{figure*}
\begin{figure*}[t]
\centering
\includegraphics[width=0.92\linewidth]{figures/news_benef-2.png}
\includegraphics[width=0.92\linewidth]{figures/news_detri-2.png}\vspace{-4mm}
\caption{Real-world data experiments on \textit{AG\_News}~\citep{zhang2015character} text dataset.}
\label{fig:agnews}
\end{figure*}
\begin{figure*}[t]
\centering
\includegraphics[width=0.92\linewidth]{figures/number_benef_detri.pdf}
\caption{The numbers of beneficial and detrimental samples for each classes on \textit{CIFAR10}~\citep{krizhevsky2009learning} image dataset.}
\label{fig:number_benef_detri}
\end{figure*}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{58}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Feldman and Zhang(2020)]{feldman2020neural}
Vitaly Feldman and Chiyuan Zhang.
\newblock What neural networks memorize and why: Discovering the long tail via influence estimation.
\newblock \emph{Advances in Neural Information Processing Systems}, 2020.
\bibitem[Chhabra et~al.(2024)Chhabra, Li, Mohapatra, and Liu]{chhabra2024what}
Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu.
\newblock ``what data benefits my classifier?" enhancing model performance and interpretability through influence-based data selection.
\newblock In \emph{International Conference on Learning Representations}, 2024.
\bibitem[Richardson et~al.(2023)Richardson, Sattigeri, Wei, Ramamurthy, Varshney, Dhurandhar, and Gilbert]{richardson2023add}
Brianna Richardson, Prasanna Sattigeri, Dennis Wei, Karthikeyan~Natesan Ramamurthy, Kush Varshney, Amit Dhurandhar, and Juan~E Gilbert.
\newblock Add-remove-or-relabel: Practitioner-friendly bias mitigation via influential fairness.
\newblock In \emph{ACM Conference on Fairness, Accountability, and Transparency}, 2023.
\bibitem[Koh and Liang(2017)]{koh2017understanding}
Pang~Wei Koh and Percy Liang.
\newblock Understanding black-box predictions via influence functions.
\newblock In \emph{International Conference on Machine Learning}, 2017.
\bibitem[Hampel(1974)]{hampel1974influence}
Frank~R Hampel.
\newblock The influence curve and its role in robust estimation.
\newblock \emph{Journal of the American Statistical Association}, 1974.
\bibitem[Cook and Weisberg(1982)]{cook1982residuals}
R~Dennis Cook and Sanford Weisberg.
\newblock \emph{Residuals and influence in regression}.
\newblock New York: Chapman and Hall, 1982.
\bibitem[Schioppa et~al.(2022)Schioppa, Zablotskaia, Vilar, and Sokolov]{schioppa2022scaling}
Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov.
\newblock Scaling up influence functions.
\newblock In \emph{AAAI Conference on Artificial Intelligence}, 2022.
\bibitem[Chhabra et~al.(2025)Chhabra, Li, Chen, Mohapatra, and Liu]{chhabra2025outlier}
Anshuman Chhabra, Bo~Li, Jian Chen, Prasant Mohapatra, and Hongfu Liu.
\newblock Outlier gradient analysis: Efficiently identifying detrimental training samples for deep learning models.
\newblock In \emph{International Conference on Machine Learning}, 2025.
\bibitem[Kwon et~al.(2024)Kwon, Wu, Wu, and Zou]{kwon2023datainf}
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou.
\newblock Data{I}nf: Efficiently {E}stimating {D}ata {I}nfluence in {L}o{RA}-tuned {LLM}s and {D}iffusion {M}odels.
\newblock In \emph{International Conference on Learning Representations}, 2024.
\bibitem[Wang et~al.(2025)Wang, Mittal, Song, and Jia]{wang2024data}
Jiachen~T Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia.
\newblock Data shapley in one training run.
\newblock In \emph{International Conference on Learning Representations}, 2025.
\bibitem[Agarwal et~al.(2017)Agarwal, Bullins, and Hazan]{agarwal2017second}
Naman Agarwal, Brian Bullins, and Elad Hazan.
\newblock Second-order stochastic optimization for machine learning in linear time.
\newblock \emph{The Journal of Machine Learning Research}, 2017.
\bibitem[Yeh et~al.(2018)Yeh, Kim, Yen, and Ravikumar]{yeh2018representer}
Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep~K Ravikumar.
\newblock Representer point selection for explaining deep neural networks.
\newblock \emph{Advances in Neural Information Processing Systems}, 2018.
\bibitem[Chen et~al.(2021)Chen, Li, Yu, Wu, and Miao]{chen2021hydra}
Yuanyuan Chen, Boyang Li, Han Yu, Pengcheng Wu, and Chunyan Miao.
\newblock Hydra: Hypergradient data relevance analysis for interpreting deep neural networks.
\newblock In \emph{AAAI Conference on Artificial Intelligence}, 2021.
\bibitem[Grosse et~al.(2023)Grosse, Bae, Anil, Elhage, Tamkin, Tajdini, Steiner, Li, Durmus, Perez, et~al.]{grosse2023studying}
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et~al.
\newblock Studying large language model generalization with influence functions.
\newblock \emph{arXiv preprint arXiv:2308.03296}, 2023.
\bibitem[Pruthi et~al.(2020)Pruthi, Liu, Kale, and Sundararajan]{pruthi2020estimating}
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan.
\newblock Estimating training data influence by tracing gradient descent.
\newblock \emph{Advances in Neural Information Processing Systems}, 2020.
\bibitem[Charpiat et~al.(2019)Charpiat, Girard, Felardos, and Tarabalka]{charpiat2019input}
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka.
\newblock Input similarity from the neural network perspective.
\newblock \emph{Advances in Neural Information Processing Systems}, 2019.
\bibitem[Bae et~al.(2024)Bae, Lin, Lorraine, and Grosse]{bae2024training}
Juhan Bae, Wu~Lin, Jonathan Lorraine, and Roger Grosse.
\newblock Training data attribution via approximate unrolled differentation.
\newblock \emph{arXiv preprint arXiv:2405.12186}, 2024.
\bibitem[Kim et~al.(2024)Kim, Kim, and Yang]{kim2024gex}
SungYub Kim, Kyungsu Kim, and Eunho Yang.
\newblock Gex: A flexible method for approximating influence via geometric ensemble.
\newblock \emph{Advances in Neural Information Processing Systems}, 2024.
\bibitem[Dai and Gifford(2023)]{dai2023training}
Zheng Dai and David~K Gifford.
\newblock Training data attribution for diffusion models.
\newblock \emph{arXiv preprint arXiv:2306.02174}, 2023.
\bibitem[Bejan et~al.(2023)Bejan, Sokolov, and Filippova]{bejan2023make}
Irina Bejan, Artem Sokolov, and Katja Filippova.
\newblock Make every example count: On the stability and utility of self-influence for learning from noisy nlp datasets.
\newblock In \emph{Conference on Empirical Methods in Natural Language Processing}, 2023.
\bibitem[Thakkar et~al.(2023)Thakkar, Bolukbasi, Ganapathy, Vashishth, Chandar, and Talukdar]{thakkar2023self}
Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar.
\newblock Self-influence guided data reweighting for language model pre-training.
\newblock In \emph{Conference on Empirical Methods in Natural Language Processing}, 2023.
\bibitem[Xiao et~al.(2024)Xiao, Gu, and Liu]{wenxiao_category}
Wenxiao Xiao, Jiuxiang Gu, and Hongfu Liu.
\newblock Category-aware active domain adaptation.
\newblock In \emph{International Conference on Machine Learning}, 2024.
\bibitem[Lees et~al.(2019)Lees, Balashankar, Welty, and Subramanian]{fair_pareto}
Alyssa~Whitlock Lees, Ananth Balashankar, Chris Welty, and Lakshminarayanan Subramanian.
\newblock What is fair? exploring pareto-efficiency for fairness constraint classifiers.
\newblock In \emph{arxiv}, 2019.
\bibitem[Martinez et~al.(2020)Martinez, Bertran, and Sapiro]{martinez2020minimax}
Natalia Martinez, Martin Bertran, and Guillermo Sapiro.
\newblock Minimax pareto fairness: A multi objective perspective.
\newblock In \emph{International Conference on Machine Learning}, 2020.
\bibitem[Edelman et~al.(2023)Edelman, Goel, Kakade, Malach, and Zhang]{NEURIPS2023_960573a3}
Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang.
\newblock Pareto frontiers in deep feature learning: Data, compute, width, and luck.
\newblock \emph{Advances in Neural Information Processing Systems}, 2023.
\bibitem[Lin et~al.(2019)Lin, Zhen, Li, Zhang, and Kwong]{NEURIPS2019_685bfde0}
Xi~Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong.
\newblock Pareto multi-task learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 2019.
\bibitem[R\"{o}pke et~al.(2025)R\"{o}pke, Reymond, Mannion, Roijers, Now\'{e}, and R\u{a}dulescu]{10.5555/3709347.3743813}
Willem R\"{o}pke, Mathieu Reymond, Patrick Mannion, Diederik~M Roijers, Ann Now\'{e}, and Roxana R\u{a}dulescu.
\newblock Divide and conquer: Provably unveiling the pareto front with multi-objective reinforcement learning.
\newblock In \emph{International Conference on Autonomous Agents and Multiagent Systems}, 2025.
\bibitem[Cai et~al.(2023)Cai, Zhang, Zhao, Bian, Sugiyama, and Llorens]{NEURIPS2023_32285dd1}
Xin-Qiang Cai, Pushi Zhang, Li~Zhao, Jiang Bian, Masashi Sugiyama, and Ashley Llorens.
\newblock Distributional pareto-optimal multi-objective reinforcement learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 2023.
\bibitem[Elsken et~al.(2019)Elsken, Metzen, and Hutter]{JMLR:v20:18-598}
Thomas Elsken, Jan~Hendrik Metzen, and Frank Hutter.
\newblock Neural architecture search: A survey.
\newblock \emph{Journal of Machine Learning Research}, 20\penalty0 (55):\penalty0 1--21, 2019.
\bibitem[Ilyas et~al.(2022)Ilyas, Park, Engstrom, Leclerc, and Madry]{ilyas2022datamodels}
Andrew Ilyas, Sung~Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry.
\newblock Datamodels: Predicting predictions from training data.
\newblock \emph{arXiv preprint arXiv:2202.00622}, 2022.
\bibitem[Jain et~al.(2023)Jain, Nandy, Aggarwal, Tendulkar, Iyer, and De]{jain2022efficient}
Eeshaan Jain, Tushar Nandy, Gaurav Aggarwal, Ashish~V. Tendulkar, Rishabh~K Iyer, and Abir De.
\newblock Efficient {D}ata {S}ubset {S}election to {G}eneralize {T}raining {A}cross {M}odels: {T}ransductive and {I}nductive {N}etworks.
\newblock \emph{Advances in Neural Information Processing Systems}, 2023.
\bibitem[Paul et~al.(2021)Paul, Ganguli, and Dziugaite]{paul2021deep}
Mansheej Paul, Surya Ganguli, and Gintare~Karolina Dziugaite.
\newblock Deep learning on a data diet: Finding important examples early in training.
\newblock \emph{Advances in Neural Information Processing Systems}, 2021.
\bibitem[Killamsetty et~al.(2021)Killamsetty, Zhao, Chen, and Iyer]{killamsetty2021retrieve}
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer.
\newblock Retrieve: Coreset selection for efficient and robust semi-supervised learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 2021.
\bibitem[Yang et~al.(2022)Yang, Xie, Peng, Xu, Sun, and Li]{yang2022dataset}
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li.
\newblock Dataset pruning: Reducing training data by examining generalization influence.
\newblock In \emph{International Conference on Learning Representations}, 2022.
\bibitem[Tan et~al.(2024)Tan, Wu, Du, Chen, Wang, Wang, and Qi]{tan2024data}
Haoru Tan, Sitong Wu, Fei Du, Yukang Chen, Zhibin Wang, Fan Wang, and Xiaojuan Qi.
\newblock Data pruning via moving-one-sample-out.
\newblock \emph{Advances in Neural Information Processing Systems}, 2024.
\bibitem[Kong et~al.(2021)Kong, Shen, and Huang]{kong2021resolving}
Shuming Kong, Yanyan Shen, and Linpeng Huang.
\newblock Resolving training biases via influence-based data relabeling.
\newblock In \emph{International Conference on Learning Representations}, 2021.
\bibitem[Chhabra et~al.(2022)Chhabra, Singla, and Mohapatra]{chhabra2022fair}
Anshuman Chhabra, Adish Singla, and Prasant Mohapatra.
\newblock Fair clustering using antidote data.
\newblock In \emph{Algorithmic Fairness through the Lens of Causality and Robustness Workshop}, 2022.
\bibitem[Lyu et~al.(2023)Lyu, Jang, Ryu, and Yang]{lyu2023deeper}
Hyeonsu Lyu, Jonggyu Jang, Sehyun Ryu, and Hyun~Jong Yang.
\newblock Deeper understanding of black-box predictions via generalized influence functions.
\newblock \emph{arXiv preprint arXiv:2312.05586}, 2023.
\bibitem[Cohn et~al.(1996)Cohn, Ghahramani, and Jordan]{cohn1996active}
David~A Cohn, Zoubin Ghahramani, and Michael~I Jordan.
\newblock Active learning with statistical models.
\newblock \emph{Journal of Artificial Intelligence Research}, 1996.
\bibitem[Liu et~al.(2021)Liu, Ding, Zhong, Li, Dai, and He]{isal}
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He.
\newblock Influence selection for active learning.
\newblock In \emph{IEEE/CVF International Conference on Computer Vision}, 2021.
\bibitem[Nguyen et~al.(2022)Nguyen, Shaker, and H{\"u}llermeier]{nguyen2022measure}
Vu-Linh Nguyen, Mohammad~Hossein Shaker, and Eyke H{\"u}llermeier.
\newblock How to measure uncertainty in uncertainty sampling for active learning.
\newblock \emph{Machine Learning}, 2022.
\bibitem[Wei et~al.(2015)Wei, Iyer, and Bilmes]{wei2015submodularity}
Kai Wei, Rishabh Iyer, and Jeff Bilmes.
\newblock Submodularity in data subset selection and active learning.
\newblock In \emph{International Conference on Machine Learning}, 2015.
\bibitem[Martin and Yohai(1986)]{martin1986influence}
R~Douglas Martin and Victor~J Yohai.
\newblock Influence functionals for time series.
\newblock \emph{The Annals of Statistics}, 1986.
\bibitem[Han et~al.(2020)Han, Wallace, and Tsvetkov]{han2020explaining}
Xiaochuang Han, Byron~C Wallace, and Yulia Tsvetkov.
\newblock Explaining black box predictions and unveiling data artifacts through influence functions.
\newblock In \emph{Annual Meeting of the Association for Computational Linguistics}, 2020.
\bibitem[Bae et~al.(2022)Bae, Ng, Lo, Ghassemi, and Grosse]{bae2022if}
Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger~B Grosse.
\newblock If influence functions are the answer, then what is the question?
\newblock \emph{Advances in Neural Information Processing Systems}, 2022.
\bibitem[Basu et~al.(2020)Basu, Pope, and Feizi]{basu2020influence}
Samyadeep Basu, Phil Pope, and Soheil Feizi.
\newblock Influence {F}unctions in {D}eep {L}earning {A}re {F}ragile.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\bibitem[Epifano et~al.(2023)Epifano, Ramachandran, Masino, and Rasool]{epifano2023revisiting}
Jacob~R Epifano, Ravi~P Ramachandran, Aaron~J Masino, and Ghulam Rasool.
\newblock Revisiting the fragility of influence functions.
\newblock \emph{Neural Networks}, 162:\penalty0 581--588, 2023.
\bibitem[Schioppa et~al.(2024)Schioppa, Filippova, Titov, and Zablotskaia]{schioppa2024theoretical}
Andrea Schioppa, Katja Filippova, Ivan Titov, and Polina Zablotskaia.
\newblock Theoretical and practical perspectives on what influence functions do.
\newblock \emph{Advances in Neural Information Processing Systems}, 2024.
\bibitem[Lotov and Miettinen(2008)]{lotov2008visualizing}
Alexander~V Lotov and Kaisa Miettinen.
\newblock Visualizing the pareto frontier.
\newblock In \emph{Multiobjective Optimization: Interactive and Evolutionary Approaches}. 2008.
\bibitem[Dantzig(2002)]{dantzig2002linear}
George~B Dantzig.
\newblock Linear programming.
\newblock \emph{Operations research}, 50\penalty0 (1):\penalty0 42--47, 2002.
\bibitem[Forrest(1996)]{forrest1996genetic}
Stephanie Forrest.
\newblock Genetic algorithms.
\newblock \emph{ACM Computing Surveys}, 28\penalty0 (1):\penalty0 77--80, 1996.
\bibitem[Krizhevsky et~al.(2009)Krizhevsky, Hinton, et~al.]{krizhevsky2009learning}
Alex Krizhevsky, Geoffrey Hinton, et~al.
\newblock Learning multiple layers of features from tiny images.
\newblock \emph{University of Toronto}, 2009.
\bibitem[Coates et~al.(2011)Coates, Ng, and Lee]{pmlr-v15-coates11a}
Adam Coates, Andrew Ng, and Honglak Lee.
\newblock An analysis of single-layer networks in unsupervised feature learning.
\newblock In \emph{International Conference on Artificial Intelligence and Statistics}, 2011.
\bibitem[Saravia et~al.(2018)Saravia, Liu, Huang, Wu, and Chen]{saravia2018carer}
Elvis Saravia, Hsien-Chi~Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen.
\newblock Carer: Contextualized affect representations for emotion recognition.
\newblock In \emph{Conference on Empirical Methods in Natural Language Processing}, 2018.
\bibitem[Zhang et~al.(2015)Zhang, Zhao, and LeCun]{zhang2015character}
Xiang Zhang, Junbo Zhao, and Yann LeCun.
\newblock Character-level convolutional networks for text classification.
\newblock 2015.
\bibitem[Wold et~al.(1987)Wold, Esbensen, and Geladi]{wold1987principal}
Svante Wold, Kim Esbensen, and Paul Geladi.
\newblock Principal component analysis.
\newblock \emph{Chemometrics and intelligent laboratory systems}, 2\penalty0 (1-3):\penalty0 37--52, 1987.
\bibitem[Devlin et~al.(2018)Devlin, Chang, Lee, and Toutanova]{DBLP:journals/corr/abs-1810-04805}
Jacob Devlin, Ming{-}Wei Chang, Kenton Lee, and Kristina Toutanova.
\newblock {BERT:} pre-training of deep bidirectional transformers for language understanding.
\newblock \emph{CoRR}, abs/1810.04805, 2018.
\bibitem[He et~al.(2016)He, Zhang, Ren, and Sun]{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In \emph{IEEE/CVF Conference on Computer Vision and Pattern Recognition}, 2016.
\end{thebibliography}