An Algorithmic Framework for Fairness Elicitation
Extracted source text ยท arXiv 1905.10660
\newif\ifdraft \draftfalse
\newif\iffull \fulltrue
% If you want to change these flags just for yourself, do it by adding
% commands to a files called texdirectives.tex ...
\makeatletter \@input{tex.flags} \makeatother
\documentclass[11pt]{article}
% Switch between display or inline math depending on mode
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
\usepackage{fullpage}
\usepackage{amsmath, amssymb, amsthm}
\usepackage{mathtools}
\usepackage{color}
\usepackage{kpfonts}
\usepackage{xcolor}
\usepackage{multicol}
\usepackage{bbm}
\usepackage{hyperref}
\usepackage{mleftright}
\usepackage{authblk}
\definecolor{DarkGreen}{rgb}{0.1,0.5,0.1}
\definecolor{DarkRed}{rgb}{0.5,0.1,0.1}
\definecolor{DarkBlue}{rgb}{0.1,0.1,0.5}
\usepackage[]{hyperref}
\hypersetup{
unicode=false, % non-Latin characters in Acrobat's bookmarks
pdftoolbar=true, % show Acrobat toolbar?
pdfmenubar=true, % show Acrobat menu?
pdffitwindow=false, % page fit to window when opened
pdftitle={}, % title
pdfauthor={}
pdfsubject={}, % subject of the document
pdfnewwindow=true, % links in new window
pdfkeywords={keywords}, % list of keywords
colorlinks=true, % false: boxed links; true: colored links
linkcolor=DarkRed, % color of internal links
citecolor=DarkGreen, % color of links to bibliography
filecolor=DarkRed, % color of file links
urlcolor=DarkBlue, % color of external links
}
\usepackage{xspace}
\usepackage{cleveref}
\usepackage{thmtools, thm-restate}
\usepackage{natbib}
% \usepackage[]{color-edits}
\usepackage[suppress]{color-edits}
\addauthor{sw}{blue}
\newcommand{\sw}[1]{\ifdraft \textcolor{blue}{[Steven: #1]}\fi}
\newcommand{\cj}[1]{\ifdraft \textcolor{red}{[Chris: #1]}\fi}
\newcommand{\ar}[1]{\ifdraft \textcolor{purple}{[Aaron: #1]}\fi}
\newcommand{\sn}[1]{\ifdraft \textcolor{green}{[Seth: #1]}\fi}
\newcommand{\ls}[1]{\ifdraft \textcolor{magenta}{[Logan: #1]}\fi}
\newcommand{\mk}[1]{\ifdraft \textcolor{orange}{[Michael: #1]}\fi}
%Macros
\newcommand\ZZ{\mathbb{Z}}
\newcommand\NN{\mathbb{N}}
\newcommand\RR{\mathbb{R}}
\newcommand\cA{\mathcal{A}}
\newcommand\cB{\mathcal{B}}
\newcommand\cE{\mathcal{E}}
\newcommand\cF{\mathcal{F}}
\newcommand\cK{\mathcal{K}}
\newcommand\cM{\mathcal{M}}
\newcommand\cP{\mathcal{P}}
\newcommand\cQ{\mathcal{Q}}
\newcommand\cR{\mathbb{R}}
\newcommand\cX{\mathcal{X}}
\newcommand\cD{\mathcal{D}}
\newcommand\cU{\mathcal{U}}
\newcommand\cN{\mathcal{N}}
\newcommand\cS{\mathcal{S}}
\newcommand\cV{\mathcal{V}}
\newcommand\cI{\mathcal{I}}
\newcommand{\E}{\mathop{\mathbb{E}}}
\newcommand\bmid{\mathrel{\Big|}}
\newcommand{\norm}[1]{\left\lVert#1\right\rVert}
\newcommand{\Tau}{\mathcal{T}}
\newcommand{\Lagr}{\mathcal{L}}
\DeclareMathOperator{\poly}{poly}
\DeclareMathOperator{\polylog}{polylog}
\newcommand{\ex}[1]{\mathbb{E}\left[#1\right]}
\DeclareMathOperator*{\Expectation}{\mathbb{E}}
\newcommand{\Ex}[2]{\Expectation_{#1}\left[#2\right]}
\DeclareMathOperator*{\Probability}{Pr}
\newcommand{\prob}[1]{\Pr\left[#1\right]}
\newcommand{\Prob}[2]{\Probability_{#1}\left[#2\right]}
\newcommand{\ind}{\mathbbm{1}}
\newcommand{\proj}{\textit{proj}}
\newcommand{\err}{\textit{err}}
\newcommand{\eps}{\varepsilon}
\def\epsilon{\varepsilon}
\DeclareMathOperator{\OPT}{OPT}
\DeclareMathOperator*{\argmin}{\mathrm{argmin}}
\DeclareMathOperator*{\argmax}{\mathrm{argmax}}
\newcommand{\INDSTATE}[1][1]{\STATE\hspace{#1\algorithmicindent}}
\newcommand*\samethanks[1][\value{footnote}]{\footnotemark[#1]}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{fact}[theorem]{Fact}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{prop}[theorem]{Proposition}
\newtheorem{assumpt}[theorem]{Assumption}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
% CJ: What is this? I've just commented this out....
%\widowpenalty=10000
%\clubpenalty=10000
\title{Eliciting and Enforcing\\ Subjective Individual Fairness}
\author{Christopher Jung\thanks{University of Pennsylvania}, Michael Kearns\samethanks, Seth Neel\samethanks, \\ Aaron Roth\samethanks, Logan Stapleton\thanks{University Of Minnesota}, Zhiwei Steven Wu\samethanks}
\begin{document}
\maketitle
\input{abstract}
\input{intro}
\input{preliminary}
\input{algorithm}
\input{generalization}
\input{experiments}
\paragraph*{Acknowledgements}
AR is supported in part by NSF grants AF-1763307, CNS-1253345, and an
Amazon Research Award. ZSW is supported in part by a Google Faculty
Research Award, a J.P. Morgan Faculty Award, and a Facebook Research
Award. Part of this work was completed while ZSW was visiting the
Simons Institute for the Theory of Computing at UC Berkeley.
\vfill
\thispagestyle{empty}
\setcounter{page}{0}
\pagebreak
\bibliographystyle{plainnat}
\bibliography{./refs}
\appendix
\input{appendix}
\end{document}
%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% End:
\begin{abstract}
We revisit the notion of individual fairness first proposed by \citet{awareness}, which asks that
``similar individuals should be treated similarly''. A primary difficulty with this definition is that
it assumes a completely specified fairness metric for the task at hand.
In contrast, we consider a framework for {\em fairness elicitation\/}, in which fairness is indirectly specified
only via a sample of pairs of individuals who should be treated (approximately) equally on the task. We make no assumption that these pairs are consistent with any metric.
We provide a provably convergent oracle-efficient algorithm for minimizing error subject to the
fairness constraints, and prove generalization theorems for both accuracy and fairness.
Since the constrained
pairs could be elicited either from a panel of judges, or from particular individuals, our framework
provides a means for algorithmically enforcing subjective
notions of fairness. We report on preliminary findings of a behavioral study of subjective fairness using
human-subject fairness constraints elicited on the COMPAS criminal recidivism dataset.
\end{abstract}
\section{Introduction}
\emph{Individual Fairness} for algorithmic decision making was originally formulated as the compelling idea that ``similar individuals should be treated similarly'' by \citet{awareness}. In its original formulation, ``similarity'' was determined by a \emph{task-specific metric} on individuals, which would be provided to the algorithm designer. Since then, the formulation of this task-specific fairness metric has been the primary obstacle that has stood in the way of adoption and further development of this conception of individual fairness. This is for two important reasons:
\begin{enumerate}
\item First, although people might have strong intuitions about what kinds of decisions are \emph{unfair}, it is difficult for them to distill these intuitions into a concisely defined quantitative measure.
\item Second, different people disagree on what constitutes ``fairness''. There is no reason to suspect that even if particular individuals were able to distill their intuitive notions of fairness into some quantitative measure, that those measures would be consistent with one another, or even internally consistent.
\end{enumerate}
In this work, we propose a practical but rigorous approach aimed at circumventing this difficulty, while staying close to the original idea that ``similar individuals should be treated similarly''. We are motivated by the following idea: Even if people cannot distill their conception of fairness as a quantitative metric, they can still be asked to express their opinion about whether particular pairs of individuals should be treated similarly or not.
Thus, one could choose a panel of ``judges'', or even a particular person,
and elicit opinions from them about whether certain pairs of decisions were fair or not.
There is no reason to suspect that these pairwise opinions will be consistent in any sense,
or that they will form a metric. Nevertheless, once such
a set of pairwise fairness constraints has been elicited,
and once a data distribution and hypothesis class are fixed, there is a well-defined
learning problem: minimize classification error subject to the constraint that the
violation of the specified pairs
is held below some fixed threshold. By varying this threshold, we can in principle define a Pareto frontier of classifiers,
optimally trading off error with the elicited conception of individual fairness --- without ever having to
commit to a restricted class of fairness notions. We would like to find the classifiers that realize this Pareto frontier. In this paper, we solve the computational, statistical, and conceptual issues necessary to do this, and
demonstrate the effectiveness of our approach via a behavioral study.
\subsection{Results}
\paragraph{Our Model} We model individuals as having features in $\mathcal{X}$ and binary labels, drawn from some distribution $\mathcal{P}$.
A committee of \emph{judges}\footnote{Though we develop our formalism as a committee of judges, note that
it permits the special case of a single subjective judge, which we make use of
in our behavioral study.}
$u \in \mathcal{U}$ has preferences that certain individuals should be treated the same way by a classifier --- i.e. that the probability that they are given a positive label should be the same. We represent these preferences abstractly as a set of pairs $C_u \subseteq \mathcal{X}\times \mathcal{X}$ for each judge $u$, where $(x,x') \in C_u$ represents that judge $u$ would view it as \emph{unfair} if individuals $x$ and $x'$ were treated substantially differently (i.e. given a positive classification with a substantially different probability). We impose no structure on how judges form their views, or the relationship between the views of different judges --- i.e. the sets $\{C_u\}_{u \in \mathcal{U}}$ are allowed to be arbitrary (for example, they need not satisfy a triangle inequality), and need not be mutually consistent. We write $C = \cup_u C_u$.
We then formulate a constrained optimization problem, that has two different ``knobs'' with which we can quantitatively relax our fairness constraint. Suppose that we say that a $\gamma$-fairness violation corresponds to classifying a pair of individuals $(x, x') \in C$ such that their probabilities of receiving a positive label differ by more than $\gamma$ (our first knob): $\left\vert\E\left[h(x) - h(x')\right]\right\vert \leq \gamma$. In this expression, the expectation is taken only over the randomness of the classifier $h$. We might ask that for \emph{no} pair of individuals do we have a $\gamma$-fairness violation: $\max_{(x,x') \in C}\left\vert\E\left[h(x) - h(x')\right]\right\vert \leq \gamma$. On the other hand, we could ask for the weaker constraint that \emph{over a random draw of a pair of individuals}, the expected fairness violation is at most $\eta$ (our second knob): $\E_{(x,x') \sim \mathcal{P}^2}\left[\left\vert h(x)-h(x') \right\vert \cdot \ind\left[(x,x') \in C\right]\right] \leq \eta$. We can also combine both relaxations to ask that the in expectation over random pairs, the ``excess'' fairness violation, on top of an allowed budget of $\gamma$, is at most $\eta$. Subject to these constraints, we would like to find the distribution over classifiers that minimizes classification error: given a setting of the parameters $\gamma$ and $\eta$, this defines a benchmark with which we would like to compete.
\paragraph{Our Theoretical Results}
Even absent fairness constraints, learning to minimize 0/1 loss (even over linear classifiers) is computationally hard in the worst case (see e.g. \cite{agnostic1,agnostic2}). Despite this, learning seems to be empirically tractable in most cases. To capture the \emph{additional} hardness of learning subject to fairness constraints, we follow several recent papers \cite{reductions,KNRW18} in aiming to develop \emph{oracle efficient} learning algorithms. Oracle efficient algorithms are assumed to have access to an \emph{oracle} (realized in experiments using a heuristic --- see the next section) that can solve weighted classification problems. Given access to such an oracle, oracle efficient algorithms must run in polynomial time. We show that our fairness constrained learning problem is computationally no harder than unconstrained learning by giving such an oracle efficient algorithm (or reduction), and show moreover that its guarantees generalize from in-sample to out-of-sample in the usual way --- with respect to both accuracy and the frequency and magnitude of fairness violations. Our algorithm is simple and amenable to implementation, and we use it in our experimental results.
\paragraph{Our Experimental Results}
Finally, we implement our algorithm and run a set of experiments on the COMPAS recidivism prediction dataset, using fairness constraints elicited from 43 human subjects. We establish that our algorithm converges quickly (even when implemented with fast learning heuristics, rather than ``oracles''). We also explore the Pareto curves trading off error and fairness violations for different human judges, and find empirically that there is a great deal of variability across subjects in terms of their conception of fairness, and in terms of the degree to which their expressed preferences are in conflict with accurate prediction. Finally we find that most of the difficulty in
balancing accuracy with the elicited fairness constraints can be attributed to a small fraction of the reported constraints.
\subsection{Related work}
\citet{awareness} first proposed the notion of individual metric-fairness that we take inspiration from, imagining fairness as a Lipschitz constraint on a randomized algorithm, with respect to some ``task-specific metric'' to be provided to the algorithm designer. Since the original proposal, the question of where the fairness metric should come from has been one of the primary obstacles to its adoption, and the focus of subsequent work. \cite{zemel} attempt to automatically learn a representation for the data (and hence, implicitly, a similarity metric) that causes a classifier to label an equal proportion of two protected groups as positive. They provide a heuristic approach and an experimental evaluation. \cite{compawareness} consider a group-fairness like relaxation of individual metric-fairness, asking that on average, individuals in pre-specified groups are classified with probabilities proportional to the average distance between individuals in those groups. They show how to learn such classifiers given access to an oracle which can evaluate the distance between two individuals according to the metric. Compared to our work, they assume the existence of an exact fairness metric which can be accessed using a quantitative oracle, and they use this metric to define a statistical rather than individual notion of fairness. Most related to our work, \cite{unknown} assumes access to an oracle which simply identifies fairness violations across pairs of individuals. Under the assumption that the oracle is exactly consistent with a metric in a simple linear class, \cite{unknown} gives a polynomial time algorithm to compete with the best fair policy in an online linear contextual bandits problem. In contrast to the unrealistic assumptions that \cite{unknown} is forced to make in order to derive a polynomial time algorithm (consistency with a simple class of metrics), we make essentially no assumptions at all on the structure of the ``fairness'' constraints. \cite{I2019} studies the problem of metric learning with the goal of using only a small number of numeric valued queries, which are hard for human beings to answer, relying more on comparison queries. Finally, \cite{RY18} prove similar generalization guarantees to ours in the context of individual-metric fairness. In the setting that they consider, the metric fairness constraint is given.
\mk{Above and throughout: when we use a reference not as part of a sentence, we should put it in parens, e.g. "(Foo et al. 2001)". We should avoid instances where we use it both in the sentence and then as a ref, as in
"Zemel et al. (Zemel et al.)" above}\cj{fixed}
\section{Problem formulation}
Let $S$ denote a set of labeled examples
$\{z_i = (x_i, y_i)\}_{i=1}^n,$ where $x_i\in \mathcal{X}$ is a
feature vector and $y_i\in\mathcal{Y}$ is a label. We will also write $S_X = \{x_i\}_{i=1}^n$
and $S_Y = \{y_i\}_{i=1}^n$. Throughout the paper, we will restrict attention to binary labels, so let $\mathcal{Y} = \{0,1\}$. Let $\mathcal{P}$ denote the unknown distribution over $\mathcal{X} \times \mathcal{Y}$. Let
$\mathcal{H}$ denote a hypothesis class containing binary classifiers $h:\mathcal{X} \to \mathcal{Y}$. We
assume that $\mathcal{H}$ contains a constant classifier (which
will imply that the ``fairness constrained'' ERM problem that we define is always feasible). We'll denote classification error of hypothesis $h$ by $\err(h, \mathcal{P}) := \Pr_{(x,y) \sim \mathcal{P}}(h(x) \neq y)$ and its empirical classification error by $\err(h, S) := \frac{1}{n}\sum_{i=1}^n \ind(h(x_i) \neq y_i)$.
We assume there is a set of one or more judges $\mathcal{U}$, such that each judge $u \in \mathcal{U}$ is identified with a set of
pairs of individuals $C_u \subseteq \mathcal{X}^2$ that she thinks should be ``treated similarly'' i.e. ideally that for the learned classifier $h$, $h(x) = h(x')$ (we will ask that this hold in expectation if the classifier is randomized, and will relax it in various ways).
For each pair $(x, x')$, let $w_{x,x'}$ be the fraction of judges who would like individual $x$ and $x'$ to be treated similarly
-- that is $w_{x,x'} = \frac{|\{u | (x,x') \in C_u\}|}{|\mathcal{U}|}$. Note that $w_{x,x'} = w_{x',x}$.
In practice, we will not have direct access to the sets of pairs $C_u$ corresponding to the judges $u$, but we may ask them whether particular pairs are in this set (see Section \ref{sec:experiment} for details about how we actually query human subjects). We model this by imagining that we present each judge with a random set of pairs $A \subseteq [n]^2$\footnote{We will always assume that this pair set is closed under symmetry}, and for each pair $(x_i,x_j)$, ask if the pair should be treated similarly or not; we learn the set of pairs in $A \cap C_u$ for each $u$. Define the empirical constraint set $\hat{C}_u = \{(x_i,x_j) \in C_u \}_{\forall (i,j) \in A}$ and $\hat{w}_{x_i x_j} = \frac{|\{u | (x,x') \in \hat{C}_u\}|}{|\mathcal{U}|}$, if $(i,j) \in A$ and 0 otherwise. For simplicity, we will sometimes write $w_{ij}$ instead of $w_{x_i,x_j}$. Note that $\hat{w}_{ij} = w_{ij}$ for every $(i, j) \in A$.
Our goal will be to find the distribution over classifiers from
$\mathcal{H}$ that minimizes classification error, while satisfying the
judges' fairness requirement $C$. To do so, we'll try to find $D$, a probability
distribution over $\mathcal{H}$, that minimizes the training error and satisfy the judges' empirical fairness constraints,
$\hat{C}$. For convenience, we denote $D$'s expected classification error as $err(D, \cP) := \E_{h \sim D}[err(h,\cP)]$ and likewise its expected empirical classification error as $err(D, S) := \E_{h \sim D}[err(h,S)]$. We say that any distribution $D$ over classifiers satisfies $(\gamma, \eta)$-approximate subjective fairness if it is a feasible solution to the following constrained empirical risk minimization problem:
\begin{align}
\label{min} \min_{D\in\Delta\mathcal{H}, \alpha_{ij} \geq 0} &err(D, S)\\
\label{abs} \text{such that } \forall(i,j)\in{[n]^2}: \quad &\E_{h\sim D}\left[h(x_i) - h(x_j)\right] \leq \alpha_{ij} + \gamma\\
\label{sum} &\sum_{(i,j) \in [n]^2} \frac{\hat{w}_{ij} \alpha_{ij}}{|A|} \leq \eta.
\end{align}
%\ar{Now that the $\eta$ constraint is normalized by $|A|$, aren't the weights $\hat{w}_{i,j}$ incorrectly scaled?}
%\cj{Yeah, the normalization constant should be $\sum_{(i,j) \in A} \hat{w}_{ij}$}\ar{No it shouldn't? I think the normalization constant should be 1. e.g. the problem is still well defined even if the judges don't care about any pairs. Removing normalization for now.}
This ``Fair ERM'' problem, whose feasible region we denote by $\Omega(S, \hat{w}, \gamma, \eta)$, has decision variables $D$ and $\{\alpha_{ij}\}$, representing the distribution over classifiers and the ``fairness violation'' terms for each pair of training points, respectively. The parameters $\gamma$ and $\eta$ are constants which represent the two different ``knobs'' we have at our disposal to quantitatively relax the fairness constraint, in an $\ell_\infty$ and $\ell_1$ sense respectively. To understand each of them, it helps to consider them in isolation. First, imagine that we set $\eta = 0$. $\gamma$ controls the worst-case disparity between the probability that any pair $(x_i,x_j) \in \hat C$ is classified as positive. (Note that although we have constraints for \emph{every} pair $x_i,x_j$, not just those in $\hat C$, because $\hat{w}_{i,j} = 0$ if $(x_i,x_j) \not\in \hat{C}$, a solution to the above program is free to set the slack parameter $\alpha_{i,j} = 1$ for any such pair. When $\eta = 0$, the slack parameter $\alpha_{i,j}$ is constrained to be $0$ whenever $\hat{w}_{i,j} > 0$ --- i.e. whenever $(x_i,x_j) \in \hat C$.) Next imagine that $\gamma = 0$. The parameter $\eta$ controls the \emph{expected} difference in probability that a randomly selected pair $(x_i,x_j) \in A$ is classified positively, weighted by the number of judges $u$ who feel they should be classified the same way --- i.e. the expected degree of dissatisfaction of the panel of judges $\mathcal{U}$, over the random choice of a pair of individuals and the randomness of their classification\footnote{To see this, recall that $(x_i,x_j) \in C \Rightarrow (x_j,x_i) \in C$, and so constraint \ref{abs} can be rewritten as $|\E_{h\sim D}[h(x_i) - h(x_j)]| \leq \alpha_{ij} + \gamma$, and $\hat{w}_{i,j} = 0$ if $(i,j) \not\in A$, and so the sum in constraint \ref{sum} can equivalently be taken over $A$ rather than $[n]^2$.}.
%\ar{Are these next clarifications needed?}
%\cj{I agree. I think we can take them out}
%The above formulation asks that the weighted average of fairness violation on each pair (with a little bit of slack $\gamma$) can't be more than $\eta$.
% Observe that by symmetry the constraint in \eqref{abs} can also be written as
%\[
% \forall(i,j)\in{[n]^2}: \quad \left|\E_{h\sim D}[h(x_i) -
% h(x_j)]\right| \leq \alpha_{ij} + \gamma.
%\]
%Note also that the constraint in \eqref{sum} is equivalent to
%\[
%\frac{\sum_{(i,j) \in A} w_{ij} \alpha_{ij}}{|A|} \le \eta
%\] because for $(i,j) \notin A$, $\hat{w}_{ij}=0$.
%Lastly, if we set $\eta = 0$, then the set constraints is equivalent to\cj{I think it's worth describing the difference between $\eta=0$ and $\gamma=0$ more in detail}
%\[
% \forall(i,j)\in{[n]^2} \mbox{ such that } w_{ij} > 0: \quad
% \E_{h\sim D}[h(x_i) - h(x_j)] \leq \gamma.
%\]
\mk{Before I forget, according to Saeed, NIPS format asks that all section headers be lower case except for first word and proper nouns}
\subsection{Fairness loss}
Our goal is to develop an algorithm that will minimize its empirical error $err(D, S)$, while satisfying the empirical fairness constraints $\hat{C}$. The standard VC dimension argument states that empirical classification error will concentrate around the true classification error, and we hope to show the same kind of generalization for fairness as well. To do so, we first define fairness loss here.
For some fixed randomized hypothesis $D \in \Delta\mathcal{H}$ and $w$, define $\gamma$-fairness loss between a pair as \ar{Can we put absolute values around $\E_{h \sim D}[h(x) - h(x')]$ in the fairness loss expression? This would make it make more sense.}\cj{Yes, I think that makes more sense than enforcing $M$ is closed under symmetry}
\[
\Pi_{D,w,\gamma}\left(\left(x,x'\right)\right) = w_{x, x'} \max\left(0, \left\vert\E_{h \sim D}\left[h(x) - h(x')\right]\right\vert - \gamma \right)
\]
For a set of pairs $M \subset \mathcal{X} \times \mathcal{X}$, the $\gamma$-fairness loss of $M$ is defined to be:
\[
\Pi_{D,w,\gamma}(M) = \frac{1}{|M|}\sum_{(x,x') \in M} \Pi_{D,w,\gamma}((x,x'))
\]
This is the expected degree to which the difference in classification probability for a randomly selected pair exceeds the allowable budget $\gamma$, weighted by the fraction of judges who think that the pair should be treated similarly. By construction, the empirical fairness loss is bounded by $\eta$ (i.e. $\Pi_{D,w,\gamma}(M) \le \sum_{ij} \frac{\hat{w}_{ij} \alpha_{ij}}{|A|} \le \eta$), and we show in Section \ref{sec:generalization},
\mk{I think all references to objects should be capitalized throughout, e.g. Section 4, Equation (1), Theorem 3, etc.}\cj{Just went over everything to make sure they are all capitalized. Question: what about the subsection title in appendix, e.g. ``Omitted details in Theorem 3.14" or ``Omitted details in theorem 3.14"?}
the empirical fairness should concentrate around the true fairness loss $\Pi_{D,w,\gamma}(\cP) := \E_{x,x' \sim \cP^2}\left[\Pi_{D,w,\gamma}(x,x')\right]$.
\subsection{Cost-sensitive classification}
In our algorithm, we will make use of a cost-sensitive classification (CSC) oracle. An instance of CSC problem can be described by a set of costs $\{(x_i, c^0_i, c^1_i)\}_{i=1}^n$ and a hypothesis class, $\mathcal{H}$. $c^0_i$ and $c^1_i$ correspond to the cost of labeling $x_i$ as 0 and 1 respectively. Invoking a CSC oracle on $\{(x_i, c^0_i, c^1_i)\}_{i=1}^n$ returns a hypothesis $h^*$ such that
\[
h^* \in \argmin_{h \in \mathcal{H}} \sum_{i=1}^n \left(h(x_i)c_i^1 + \left(1-h(x_i)\right)c_i^0\right)
\]
We say that an algorithm is \emph{oracle-efficient} if it runs in polynomial time assuming access to a CSC oracle.
\section{Empirical risk minimization}\label{sec:erm}
\mk{Oracle-efficient should be hyphenated everywhere. Also, have we explained this concept earlier?
While I realize we have limited space, the very terse outline in this section invites the reader to
think that everything we're doing here is standard and straightforward; if there were natural places
to add sentences highlighting the tricker steps it might be helpful.}
In this section, we give an oracle-efficient algorithm for approximately solving our (in-sample) constrained empirical risk minimization problem.
\subsection{Outline of the solution}
We frame the problem of solving our constrained ERM problem as finding an approximate equilibrium of a zero-sum game between a primal player and a dual player, trying to minimize and maximize respectively the Lagrangian of the constrained optimization problem.
The Lagrangian for our optimization problem is
\begin{equation}
\Lagr(D, \alpha, \lambda, \tau) = \err(D,S) + \sum_{(i,j) \in [n]^2} \lambda_{ij} \left( \E_{h\sim D}\left[h(x_i) - h(x_j)\right] - \alpha_{ij} -\gamma \right) +
\tau \left( \frac{1}{|A|}\sum_{(i, j)\in [n]^2} w_{ij} \alpha_{ij} - \eta \right)
\end{equation}
For the constraint in equation \eqref{abs}, corresponding to each pair of individuals $(x_i,x_j)$, we introduce a dual variable $\lambda_{ij}$. For the constraint \eqref{sum}, we introduce a dual variable $\tau$. The primal player's action space is $(D, \alpha) \in (\Delta\mathcal{H}, [0,1]^{n^2})$, and the dual player's action space is $(\lambda, \tau) \in (\cR^{n^2}, \cR)$.%, which corresponds to how much he wants to charge each constraint for the violation.
Solving our constrained ERM problem equivalent to finding a minmax equilibrium of $\Lagr$:
\[
\argmin_{(D,\alpha) \in \Omega(S, \hat{w}, \gamma, \eta)} \err(D, S) = \argmin_{D \in \Delta\mathcal{H}, \alpha \in [0,1]^{n^2}} \max_{\lambda \in \cR^{n^2}, \tau \in \cR} \Lagr(D, \alpha, \lambda, \tau)
\]
Because $\Lagr$ is linear in terms of its parameters, Sion's minimax theorem \citep{sion1958general} gives us
\[
\min_{D \in \Delta\mathcal{H}, \alpha \in [0,1]^{n^2}} \max_{\lambda \in \cR^{n^2}, \tau \in \cR} \Lagr(D, \alpha, \lambda, \tau) = \max_{\lambda \in \cR^{n^2}, \tau \in \cR} \min_{D \in \Delta\mathcal{H}, \alpha \in [0,1]^{n^2}} \Lagr(D, \alpha, \lambda, \tau).
\]
By a classic result of \cite{freund1996game}, one can compute an approximate equilibrium by simulating ``no-regret'' dynamics between the primal and dual player. Our algorithm can be viewed as simulating the following \textit{no-regret dynamics} between the primal and the dual players over $T$ rounds. Over each of the rounds, the dual player updates dual variables $\{ \lambda, \tau \}$ according to \textit{no-regret} learning algorithms (exponentiated gradient descent \citep{kivinenwarmuth1997exponentiated} and online gradient descent \citep{zinkevich2003online} respectively). At every round, the primal player then best responds with a pair $\{ D, \alpha \}$ using a CSC oracle. The time-averaged play of both players converges to an approximate equilibrium of the zero-sum game, where the approximation is controlled by the regret of the dual player.
\input{primal}
\subsection{Dual player's no regret updates}\label{subsec:no-regret}
%\input{noregret}
In order to reason about convergence we need to restrict the dual player's action space to lie within a bounded $\ell_1$ ball, defined by the parameters $C_\tau$ and $C_\lambda$ that appear in our theorem --- and serve to trade off running time with approximation quality:
\[
\Lambda = \left\{ \lambda \in \cR_{+}^{n^2} : \norm{\lambda}_1 \le C_\lambda \right\}, \Tau = \left\{\tau \in \cR_{+} : \norm{\tau}_1 \le C_\tau\right\}.
\]
The dual player will use exponentiated gradient descent \citep{kivinenwarmuth1997exponentiated} to update $\lambda$ and online gradient descent \citep{zinkevich2003online} to update $\tau$, where the reward function will be defined as:
$r_\lambda\left(\lambda^t\right) = \sum_{(i,j) \in [n]^2} \lambda^t_{ij} \left( \E_{h\sim D}\left[h(x_i) - h(x_j)\right] - \alpha_{ij} -\gamma \right)$
and
$r_\lambda\left(\tau^t\right) = \tau^t \left( \frac{1}{|A|}\sum_{(i, j)\in [n]^2} w_{ij} \alpha_{ij} - \eta \right).$
\begin{algorithm}
\caption{No-Regret Dynamics}\label{alg:no-regret}
\begin{algorithmic}[ht!]
\State \textbf{Input:} training examples $\{ x_i, y_i \}_{i=1}^n,$ bounds $C_\lambda$ and $C_\tau$, time horizon $T$, step sizes $\mu_\lambda$ and $\{{\mu_\tau^t}\}^{t=1}_{T}$,
\State Set $\theta^{0}_1 = \mathbf{0} \in \mathbb{R}^{n^2}$
\State Set $\tau^0 = 0$
\For{$t = 1, 2, \dots, T$}
\State Set $\lambda^t_{ij} = C_{\lambda} \frac{\exp{\theta^{t-1}_{ij}}}{1 + \sum_{i',j' \in [n]^2} \exp{\theta^{t-1}_{i'j'}}}$ for all pairs $(i,j) \in [n]^2$
\State Set $\tau^t = \proj_{[0, C_\tau]}\left(\tau^{t-1} + \mu^t_{\tau} \left(\frac{1}{|A|} \sum_{i,j} w_{ij} \alpha^{t-1}_{ij} - \eta\right)\right)$
\State $D^t, \alpha^t \leftarrow \textnormal{BEST}_{\rho}(\lambda^t, \tau^t)$
%
% \State $\overline{D}_t \leftarrow \frac{1}{t} \sum_{t'=1}^t D_{t'}$
% \State $\overline{\alpha}_t \leftarrow \frac{1}{t} \sum_{t'=1}^t \alpha_{t'}, \quad \Lagr_{\max} \leftarrow \Lagr \left( \overline{D}_t, \overline{\alpha}_t, \textnormal{BEST}_{\lambda, \tau} (\overline{D}_t, \overline{\alpha}_t) \right)$
% \State $\overline{\lambda}_t \leftarrow \frac{1}{t} \sum_{t'=1}^t \lambda_{t'}$
% \State $\overline{\tau}_t \leftarrow \frac{1}{t} \sum_{t'=1}^t \tau_{t'}, \quad \Lagr_{\min} \leftarrow \Lagr \left( \textnormal{BEST}_{D, \alpha} ( \overline{\lambda}_t, \overline{\tau}_t), \overline{\lambda}_t, \overline{\tau}_t \right)$
% \State $\nu_t \leftarrow \max \{ \Lagr(\overline{D_t}, \overline{\alpha_t}, \overline{\lambda_t}, \overline{\tau_t}) - \Lagr_{\min}, \quad \Lagr_{\max} - \Lagr(\overline{D_t}, \overline{\alpha_t}, \overline{\lambda_t}, \overline{\tau_t} \}$
% \State
% \If {$\nu_t \leq \nu$}
% \State Return $(\overline{D}_t, \overline{\alpha}_t, \overline{\lambda}_t, \overline{\tau}_t)$
% \EndIf
\For {$(i,j) \in [n]^2$}
\State $\theta^{t}_{ij} = \theta^{t-1}_{ij} + \mu^{t-1}_{\lambda} \left( \E_{h\sim D^t}\left[h(x_i) - h(x_j)\right] - \alpha^t_{ij} - \gamma \right)$
\EndFor
\State
\EndFor
\State \textbf{Output:} $\frac{1}{T}\sum_{t=1}^T D^t$
\end{algorithmic}
\end{algorithm}
\begin{lemma}\label{lem:tau-no-regret}
Running online gradient descent for $\tau^t$, i.e. $\tau^t = \proj_{[0,C_{\tau}]}\left(\tau^{t-1} + \mu^{t-1} \cdot \nabla\Lagr_{D^t, \alpha^t}^{\psi_2}\left(\tau^{t-1}\right)\right)$, with step size $\mu^t = \frac{C_{\tau}}{\sqrt{T}}$ yields the following regret
\[\max_{\tau \in \Tau} \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_2}(\tau) - \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_2}\left(\tau^t\right) \leq C_{\tau}\sqrt{T}.\]
\end{lemma}
\begin{proof}
First, note that $\nabla\Lagr_{D^t, \alpha^t}^{\psi_2}\left(\tau^{t-1}\right) = \frac{1}{W} \sum_{ij} w_{ij} \alpha_{ij}^{t-1} - \eta$ and
\[\tau^t = proj_{[0,C_{\tau}]} \left(\tau^{t-1} + \mu_{\tau}^t \left(\frac{1}{W} \sum_{ij} w_{ij} \alpha_{ij}^{t-1} - \eta\right)\right).\]
From \cite{zinkevich2003online}, we find that the regret of this online gradient descent (translated into the terms of our paper) is bounded as follows:
\begin{equation}\label{eq:zinkevich}
\max_{\tau \in \Tau} \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_2}(\tau) - \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_2}\left(\tau^t\right) \leq \frac{C_{\tau}^2}{2 \mu_{\tau}^T} + \frac{\left\vert\left\vert\nabla \Lagr_{D,\alpha}^{\psi_2}\right\vert\right\vert^2}{2} \sum_{t=1}^T \mu_{\tau}^t,
\end{equation}
where the bound on our target $\tau$ term is $C_{\tau}$, the gradient of our cost function at round $t$ is $\nabla \Lagr^{\psi_2}_{D^t,\alpha^t}\left(\tau^{t-1}\right)$, and the bound $\left\vert\left\vert\nabla \Lagr_{D,\alpha}^{\psi_2}\right\vert\right\vert = \sup_{\tau \in \mathcal{T},\ t\in{[T]}} \left\vert\left\vert\nabla\Lagr_{D^t, \alpha^t}^{\psi_2}\left(\tau^{t-1}\right)\right\vert\right\vert.$ To prove the above lemma, we first need to show that this bound $\left\vert\left\vert\nabla \Lagr_{D,\alpha}^{\psi_2}\right\vert\right\vert \leq 1.$
Since $w_{ij}, \alpha_{ij}, \eta \in [0,1]$ for all pairs $(i,j)$, the Lagrangian $\frac{1}{|A|} \sum_{ij} w_{ij} \alpha_{ij} - \eta = \frac{\sum_{ij} w_{ij} \alpha_{ij}}{|A|} - \eta \leq 1.$ For all $t$, the gradient
\[\left\vert\nabla\Lagr_{D^t, \alpha^t}^{\psi_2}\left(\tau^{t-1}\right)\right\vert = \frac{\sum_{ij} w_{ij} \alpha_{ij}^{t-1}}{|A|} - \eta \leq 1.\]
Thus, \[\left\vert\nabla\Lagr_{D, \alpha}^{\psi_2}\right\vert \leq 1.\]
Note that if we define $\mu_{\tau}^t= \frac{C_{\tau}}{\sqrt{T}},$ then the summation of the step sizes is equal to
\[
\sum_{t=1}^T \mu_{\tau}^t = C_{\tau} \sqrt{T}
\]
Substituting these two results into inequality~\eqref{eq:zinkevich}, we get that the regret
\[\max_{\tau \in \Tau} \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_2}(\tau) - \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_2}\left(\tau^t\right) \leq \frac{C_{\tau}^2}{2 \left(C_{\tau}\ /\sqrt{T}\right)} + \frac{1}{2} C_{\tau}\sqrt{T} = C_{\tau}\sqrt{T} \]
\end{proof}
\begin{lemma}\label{lem:lambda-no-regret}
Running exponentiated gradient descent for $\lambda^t$ yields the following regret:
\[
\max_{\lambda \in \Lambda} \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_1}(\lambda) - \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_1}\left(\lambda^t\right) \leq 2C_{\lambda} \sqrt{T \log n}.
\]
\end{lemma}
\begin{proof}
In each round, the dual player gets to charge either some $(i,j)$ constraint or no constraint at all. In other words, he is presented with $n^2+1$ options. Therefore, to account for the option of not charging any constraint, we define vector $\lambda' = \left(\lambda, 0 \right)$, where the last coordinate, which will always be $0$, corresponds to the option of not charging any constraint.
Next, we define the reward vector $\zeta^t$ for $\lambda'^t$ as
\[
\zeta^t=\left(\left(\E\limits_{h\sim D^t}\left[h(x_i) - h(x_j)\right] - \alpha^t_{ij} -\gamma\right)_{i,j \in [n]^2}, 0\right).
\]
Hence, the reward function is \[r(\lambda'^t) = \zeta^t \cdot \lambda'^t = \Lagr_{D^t, \alpha^t}^{\psi_1}\left(\lambda^t\right).\]
The gradient of the reward function is
\[ \nabla r(\lambda'^t) = \left(\left(\nabla r(\lambda^t)\right)_{i,j \in [n^2]}, 0\right) = \left(\zeta^t, 0\right)\]
Note that the $\ell_\infty$ norm of the gradient is bounded by 1, i.e. \[\left\vert\left\vert\nabla r(\lambda'^t)\right\vert\right\vert_{\infty} \leq 1\]
because for any $t$, each respective component of the gradient, $\E\limits_{h\sim D^t}\left[h(x_i) - h(x_j)\right] - \alpha^t_{ij} -\gamma$, is bounded by 1.\\
Here, by the regret bound of \cite{kivinenwarmuth1997exponentiated}, we obtain the following regret bound:
\[
\max_{\lambda \in \Lambda} \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_1}(\lambda) - \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_1}\left(\lambda^t\right) \leq \frac{\log{n}}{\mu} + \mu \left\vert\left\vert\lambda'\right\vert\right\vert_1^2 \left\vert\left\vert\nabla r(\lambda') \right\vert\right\vert_{\infty}^2 T \le \frac{\log{n}}{\mu} + \mu C_{\lambda}^2 T.
\]
If we take $\mu = \frac{1}{C_{\lambda}} \sqrt{\frac{\log n}{T}},$ the regret is bounded as follows:
\begin{equation*}
\max_{\lambda \in \Lambda} \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_1}(\lambda) - \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_1}\left(\lambda^t\right) \leq 2C_{\lambda} \sqrt{T \log n}.
\end{equation*}
\swcomment{please move the step size def into the pseudo-code}
\end{proof}
\subsection{Guarantee}
Now, we appeal to \cite{freund1996game} to show that our no-regret dynamics converge to an approximate minmax equilibrium of $\Lagr$. Then, we show that an approximate minmax equilibrium corresponds to an approximately optimal solution to our original constrained optimization problem.
\begin{theorem}[\cite{freund1996game}]\label{thm:freund1996game}
Let $(D^1, \alpha^1), \dots, (D^T, \alpha^T)$ be the primal player's sequence of actions, and $(\lambda^1, \tau^1), \dots, (\lambda^T, \tau^T)$ be the dual player's sequence of actions. Let $\bar{D} = \frac{1}{T}\sum_{t=1}^T D^t$, $\bar{\alpha} = \frac{1}{T}\sum_{t=1}^T \alpha^t$, $\bar{\lambda} = \frac{1}{T}\sum_{t=1}^T \lambda^t$, and $\bar{\tau} = \frac{1}{T} \sum_{t=1}^T \tau^t$. Then, if the regret of the dual player satisfies
\[
\max_{\lambda \in \Lambda, \tau \in \Tau} \sum_{t=1}^T \Lagr\left(D^t, \alpha^t, \lambda^t, \tau^t\right) - \sum_{t=1}^T \Lagr\left(D^t, \alpha^t, \lambda^t, \tau^t\right)\le \xi_\psi T,
\]
and the primal player best responds in each round ($D^t, \alpha^t = \argmax_{D \in \Delta(H), \alpha \in [0,1]^{n^2}} \Lagr\left(D, \alpha, \lambda^t, \tau^t\right)$),
then $\left(\bar{D}, \bar{\alpha}, \bar{\lambda}, \bar{\tau}\right)$ is an $\xi_\psi$-approximate solution
\end{theorem}
\swcomment{wait, we have approximate equilibrium, which is different from approximate solution for the Lagrangian, right?}
\cj{I think they are equivalent, no? Theorem \ref{thm:approx-equilibrium} uses approximate lagrangian anyway}
\begin{remark}
If the primal learner's approximate best response satisfies
\[
\sum_{t=1}^T \Lagr\left(D^t, \alpha^t, \lambda^t, \tau^t\right) - \min_{D \in \Delta(H), \alpha \in [0,1]^{n^2}} \sum_{t=1}^T \Lagr\left(D, \alpha, \lambda^t, \tau^t\right) \le \xi_\rho T
\] along with dual player's regret of $\xi_\rho T$, then $\left(\bar{D}, \bar{\alpha}, \bar{\lambda}, \bar{\tau}\right)$ is an $\left(\xi_\rho + \xi_\psi\right)$-approximate solution
\end{remark}
\begin{theorem}\label{thm:approx-equilibrium}
Let $\left(\hat{D}, \hat{\alpha}, \hat{\lambda}, \hat{\tau}\right)$ be a $v$-approximate solution to the Lagrangian problem. More specifically,
\[\Lagr\left(\hat{D}, \hat{\alpha}, \hat{\lambda}, \hat{\tau}\right) \le \min_{D \in \Delta(\mathcal{H}), \alpha \in [0,1]^{n^2}} \Lagr\left(D, \alpha, \hat{\lambda}, \hat{\tau}\right) + v,\]
and
\[\Lagr\left(\hat{D}, \hat{\alpha}, \hat{\lambda}, \hat{\tau}\right) \ge \max_{\lambda \in \Lambda, \tau \in \Tau} \Lagr\left(\hat{D}, \hat{\alpha}, \lambda, \tau\right) - v.\]
Then, $err(\hat{D}, S) \le OPT + 2v$. And as for the constraints, $\E_{h\sim \hat{D}}\left[h(x_i) - h(x_j)\right] \le \hat{\alpha}_{ij} + \gamma + \frac{1+2v}{C_\lambda}, \forall (i,j) \in [n]^2$ and
$\frac{1}{|A|}\sum_{(i, j)\in [n]^2} \hat{w}_{ij} \hat{\alpha}_{ij} \le \eta + \frac{1+2v}{C_\tau}$.
\end{theorem}
\input{approx-equilibrium-thm-proof}
\begin{theorem}\label{thm:algorithm-guarantee}
Fix parameters $\nu, C_{\tau}, C_\lambda$ that serve to trade off running time with approximation error. Running Algorithm \ref{alg:no-regret} for $T=\left(\frac{2C_{\lambda}\sqrt{\log(n)} +C_{\tau}}{\nu}\right)^2$ outputs a solution $(\hat{D}, \hat{\alpha})$ with the following guarantee. The objective value is approximately optimal:
\[err(\hat{D}, S) \le \min_{(D,\alpha) \in \Omega(S, \hat{w}, \gamma, \eta)} \err(D, S) + 2\nu.\]
And the constraints are approximately satisfied: $\E_{h\sim \hat{D}}\left[h(x_i) - h(x_j)\right] \le \hat{\alpha}_{ij} + \gamma + \frac{1+2\nu}{C_\lambda}, \forall (i,j) \in [n]^2$ and
$\frac{1}{|A|}\sum_{(i, j)\in [n]^2} \hat{w}_{ij} \hat{\alpha}_{ij} \le \eta + \frac{1+2\nu}{C_\tau}.$
\end{theorem}
\begin{proof}
Observe that
\[
\Lagr(D, \alpha, \lambda, \tau) = err(D,S) + \Lagr_{D, \alpha}^{\psi_1}(\lambda) + \Lagr_{D, \alpha}^{\psi_2}(\tau)
\]
By how we constructed $\Lagr_{D, \alpha}^{\psi_1}$ and $\Lagr_{D, \alpha}^{\psi_2}$, combining Lemma \ref{lem:tau-no-regret} and \ref{lem:lambda-no-regret} yields
\begin{align*}
&\max_{\lambda \in \Lambda, \tau \in \Tau} \sum_{t=1}^T \Lagr\left(D^t, \alpha^t, \lambda^t, \tau^t\right) - \sum_{t=1}^T \Lagr\left(D^t, \alpha^t, \lambda^t, \tau^t\right)\\
&=\max_{\tau \in \Tau} \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_2}(\tau) - \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_2}\left(\tau^t\right) + \max_{\lambda \in \Lambda} \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_1}(\lambda) - \sum_{t=1}^T \Lagr_{D^t, \alpha^t}^{\psi_1}\left(\lambda^t\right)\\
&\le \xi_\psi T,
\end{align*}
where $\xi_\psi = \frac{2C_{\lambda} \sqrt{T \log n} + C_{\tau}\sqrt{T}}{T}$.
Then, theorem \ref{thm:freund1996game} tells us that $\bar{D}, \bar{\alpha}, \bar{\lambda}, \bar{\alpha}$ form a $\xi_\psi$-approximate equilibrium, where $\bar{D} = \frac{1}{T}\sum_{t=1}^T D^t$, $\bar{\alpha} = \frac{1}{T}\sum_{t=1}^T \alpha^t$, $\bar{\lambda} = \frac{1}{T}\sum_{t=1}^T \lambda^t$, and $\bar{\tau} = \frac{1}{T} \sum_{t=1}^T \tau^t$. And finally, with $T= \left(\frac{2C_{\lambda}\sqrt{\log(n)} +C_{\tau}}{v}\right)^2$ results in $\xi_\psi = \nu$, Theorem \ref{thm:approx-equilibrium} gives
\[err(\hat{D}, S) \le \min_{(D,\alpha) \in \Omega(S, \hat{w}, \gamma, \eta)} \err(D, S) + 2\nu.\]
And as for the constraints, \[\E_{h\sim \hat{D}}\left[h(x_i) - h(x_j)\right] \le \hat{\alpha}_{ij} + \gamma + \frac{1+2\nu}{C_\lambda}, \forall (i,j) \in [n]^2\] and
\[\frac{1}{|A|}\sum_{(i, j)\in [n]^2} \hat{w}_{ij} \hat{\alpha}_{ij} \le \eta + \frac{1+2v}{C_\tau}.\]
\end{proof}
%In this section, we provide our result that says the in-sample constrained optimization can be solved approximately (i.e. the objective $\err(D,S)$ will be close to optimal and the constraints will be violated only by a little) and provide a sketch of our solution.
%\subsection{Guarantee}
%The average of the the primal player's actions over sufficiently many rounds of no-regret dynamics converges to an approximate optimal solution.
%\begin{theorem}\label{thm:algorithm-guarantee}
%Running the algorithm for at least $T=(\frac{2C_{\lambda}\sqrt{\log(n)} +C_{\tau}}{\nu})^2$ rounds outputs $(\hat{D}, \hat{\alpha})$ with the following guarantee.
%\[err(\hat{D}, S) \le \min_{(D,\alpha) \in \Omega(S, \hat{w}, \gamma, \eta)} \err(D, S) + 2\nu.\]
% And as for the constraints, $\E_{h\sim \hat{D}}[h(x_i) - h(x_j)] \le \hat{\alpha}_{ij} + \gamma + \frac{1+2\nu}{C_\lambda}, \forall (i,j) \in [n]^2$ and
%$\frac{1}{|A|}\sum_{(i, j)\in [n]^2} \hat{w}_{ij} \hat{\alpha}_{ij} \le \eta + \frac{1+2v}{C_\tau}.$
%\end{theorem}
\subsection{Primal player's best response}\label{subsec:primal_best_response}
In each round $t$, given the actions chosen by the dual player $\left(\lambda^t, \tau^t\right)$, the primal player needs to best respond by choosing $\left(D^t,\alpha^t\right)$ such that
\[\left(D^t,\alpha^t\right) \in \argmin_{D \in \Delta\mathcal{H}, \alpha \in [0,1]^{n^2}} \Lagr\left(D, \alpha, \lambda^t, \tau^t\right).\]
We do so by leveraging a CSC oracle. Given $\lambda^t$, we can set the costs as follows
\[c_i^{0} = \frac{1}{n} \mathbb{E}_{h\sim D}\left[\ind(y_i \neq 0)\right] \text{ and } c_i^{1} = \frac{1}{n} \mathbb{E}_{h\sim D} \left[\ind(y_i \neq 1)\right] + \left(\lambda^t_{ij} - \lambda^t_{ji}\right).\]
Then, $D^t = h^t = CSC(\{(x_i, c^0_i, c^1_i)\}_{i=1}^n)$ (we note that the best response is always a deterministic classifier $h^t$). As for $\alpha^t$, we set $\alpha^t_{ij}=1$ if $\tau^t \frac{w_{ij}}{|A|} - \lambda^t_{ij} \le 0$ and 0 otherwise.
\begin{algorithm}
\caption{Best Response, $BEST_{\rho}(\lambda, \tau)$, for the primal player}
\begin{algorithmic}[ht!]
\State \textbf{Input:} training examples $S=\{ x_i, y_i \}_{i=1}^n,$ $\lambda \in \Lambda$, $\tau \in \Tau$, CSC oracle $CSC$
\For{$i = 1, \dots, n$}
\If{$y_i = 0$}
\State Set $c^0_{i}=0$
\State Set $c^1_{i}=\frac{1}{n} + \sum_{j \neq i} \lambda_{ij} - \lambda_{ji}$
\Else
\State Set $c^0_{i}=\frac{1}{n}$
\State Set $c^1_{i}=\sum_{j \neq i} \lambda_{ij} - \lambda_{ji}$
\EndIf
\EndFor
\State $D = CSC(S, c)$
\For{$(i,j) \in [n]^2$}
\State $\alpha_{ij} = \begin{cases}
1 : & \tau \frac{w_{ij}}{|A|} - \lambda_{ij} \leq 0 \\
0 : & \tau \frac{w_{ij}}{|A|} - \lambda_{ij} > 0.
\end{cases}
$
\EndFor
\State \textbf{Output:} $D, \alpha$
\end{algorithmic}
\end{algorithm}
\begin{lemma}\label{lem:primal_separate}
For fixed $\lambda, \tau$, the best response optimization for the primal player is separable, i.e.
\[\argmin_{D,\alpha} \Lagr(D, \alpha, \lambda, \tau) = \argmin_{D} \Lagr_{\lambda, \tau}^{\rho_1}(D) \times \argmin_{\alpha} \Lagr_{\lambda, \tau}^{\rho_2}(\alpha),\]
where
\[\Lagr_{\lambda, \tau}^{\rho_1}(D) = \err(h,D) + \sum_{(i,j) \in [n]^2} \lambda_{ij} \E\limits_{h\sim D} \left[h(x_i) - h(x_j)\right]\] and
\[\Lagr_{\lambda, \tau}^{\rho_2}(\alpha) = \sum_{(i,j) \in [n]^2} \lambda_{ij} \left(-\alpha_{ij}\right) + \tau \left( \frac{1}{|A|}\sum_{(i,j) \in [n]^2} w_{ij}\alpha_{ij} \right)\]
\end{lemma}
\begin{proof} First, note that $\alpha$ is not dependent on $D$ and vice versa. Thus, we may separate the optimization $\argmin_{D,\alpha} \Lagr$ as such:
\begin{align*}
&\argmin_{D,\alpha} \Lagr(D, \alpha, \lambda, \tau)\\
&= \argmin_{D, \alpha} \err(D,S)
+ \sum_{(i,j) \in [n]^2} \lambda_{ij} \left( \E\limits_{h\sim D}[h(x_i) - h(x_j)] - \alpha_{ij} -\gamma \right) + \tau \left( \frac{1}{|A|}\sum_{(i,j) \in [n]^2} w_{ij} \alpha_{ij} - \eta \right)\\
&= \argmin_{D} \err(D,S) + \sum_{(i,j) \in [n]^2} \lambda_{ij} \E\limits_{h\sim D} \left[h(x_i) - h(x_j)\right]
\times \sum_{(i,j) \in [n]^2} \lambda_{ij} \left(-\alpha_{ij}\right) + \tau \left(\frac{1}{|A|}\sum_{(i,j) \in [n]^2} w_{ij}\alpha_{ij} \right)\\
&= \argmin_{D} \Lagr_{\lambda, \tau}^{\rho_1}(D) \times \argmin_{\alpha}\Lagr_{\lambda, \tau}^{\rho_2}(\alpha)
\end{align*}
\end{proof}
\begin{lemma}\label{lem:primal_alpha}
For fixed $\lambda$ and $\tau$, the output $\alpha$ from $BEST_{\rho}(\lambda, \tau)$ minimizes $\Lagr_{\lambda, \tau}^{\rho_2}$
\end{lemma}
\begin{proof}
The optimization
\begin{align*}
\argmin_{\alpha} \Lagr_{\lambda, \tau}^{\rho_2}
&= \argmin_{\alpha} \sum_{(i,j) \in [n]^2} \lambda_{ij} \left(-\alpha_{ij}\right) + \tau \left( \frac{1}{|A|}\sum_{(i,j) \in [n]^2} w_{ij}\alpha_{ij} \right)\\
&= \argmin_{\alpha} \ \sum_{(i,j) \in [n]^2} -\lambda_{ij} \alpha_{ij}
+ \sum_{(i,j) \in [n]^2} \tau \frac{w_{ij}}{|A|} \alpha_{ij}\\
&= \argmin_{\alpha} \ \sum_{(i,j) \in [n]^2} \alpha_{ij} \left(\tau \frac{w_{ij}}{|A|} -\lambda_{ij}\right).
\end{align*}
Note that for any pair $(i,j) \in [n]^2$, the term $\alpha_{ij} \in [0,1]$. Thus, when the constant $\tau \frac{w_{ij}}{|A|} -\lambda_{ij} \leq 0,$ we assign $\alpha_{ij}$ as the maximum bound, $1$, in order to minimize $\Lagr_{\rho_2}$. Otherwise, when $\tau \frac{w_{ij}}{|A|} -\lambda_{ij} > 0,$ we assign $\alpha_{ij}$ as the minimum bound, 0.
\end{proof}
\begin{lemma}\label{lem:primal_D}
For fixed $\lambda$ and $\tau$, the output $D$ from $BEST_{\rho}(\lambda, \tau)$ minimizes $\Lagr_{\lambda, \tau}^{\rho_1}$
\end{lemma}
\begin{proof}
\begin{align*}
\argmin_{D} \Lagr_{\lambda, \tau}^{\rho_1} &= \argmin_{D} \err(D,S) + \sum_{(i,j) \in [n]^2} \lambda_{ij} \E\limits_{h\sim D} \left[h(x_i) - h(x_j)\right]\\
&= \argmin_{D} \frac{1}{n} \sum_{i=1}^n \mathbb{E}_{h\sim D} \left[\ind(h(x_i)\neq y_i)\right]
+ \sum_{(i,j) \in [n]^2} \lambda_{ij} \E\limits_{h\sim D} \left[h(x_i) - h(x_j)\right]\\
&= \argmin_D \ \sum_{i=1}^n \left( \frac{1}{n} \mathbb{E}_{h\sim D} \left[\ind(h(x_i)\neq y_i)\right]
+ \sum_{j \neq i} \lambda_{ij} h(x_i)
- \sum_{j \neq i} \lambda_{ji} h(x_i)\right)\\
&= \argmin_D \ \sum_{i=1}^n \left(\frac{1}{n} \mathbb{E}_{h\sim D} \left[\ind(h(x_i)\neq y_i)\right]
+ \sum_{j \neq i} h(x_i)\left(\lambda_{ij} - \lambda_{ji}\right) \right).
\end{align*}
For each $i \in [n],$ we assign the cost \[c_i^{h(x_i)} = \frac{1}{n} \mathbb{E}_{h\sim D} \left[\ind(h(x_i)\neq y_i)\right] + h(x_i)\left(\lambda_{ij} - \lambda_{ji}\right).\]
Note that the cost depends on whether $y_i = 0$ or 1. For example, take $y_i=1$ and $h(x_i)=0$. The cost
\begin{align*}
c_i^{h(x_i)} = c_i^0 &= \frac{1}{n} \mathbb{E}_{h\sim D} \left[\ind(h(x_i)\neq y_i)\right]
+ \sum_{j \neq i} h(x_i)\left(\lambda_{ij} - \lambda_{ji}\right)\\
&= \frac{1}{n} \cdot 1 + \sum_{j \neq i} 0 \cdot \left(\lambda_{ij} - \lambda_{ji}\right)
= \frac{1}{n}
\end{align*}
\end{proof}
\begin{proof}
Let $(D^*, \alpha^*)=\argmin_{(D,\alpha) \in \Omega(S, \hat{w}, \gamma, \eta)} \err(D, S)$, the optimal solution to the Fair ERM. Also, define
\[
penalty_{S,w}\left(D, \alpha, \lambda, \tau\right) := \sum_{(i,j)} \lambda_{ij} \left( \E\limits_{h\sim D}\left[h(x_i) - h(x_j)\right] - \alpha_{ij} -\gamma \right) + \tau \left( \frac{1}{|A|}\sum_{(i,j)} \hat{w}_{ij} \alpha_{ij} - \eta \right).
\]
Note that for any $D$ and $\alpha$, $\max_{\lambda \in \Lambda, \tau \in \Tau} penalty_{S, \hat{w}}(D, \alpha, \lambda, \tau) \ge 0$ because one can always set $\lambda = 0$ and $\tau = 0$.
\begin{align*}
\max_{\lambda \in \Lambda, \tau \in \Tau} \Lagr\left(\hat{D}, \hat{\alpha}, \lambda, \tau\right) &\le \Lagr\left(\hat{D}, \hat{\alpha}, \hat{\lambda}, \hat{\tau}\right) + v\\
&\le \min_{D \in \Delta(\mathcal{H}), \alpha \in [0,1]^{n^2}} \Lagr\left(D, \alpha \hat{\lambda}, \hat{\tau}\right) + 2v\\
&\le \Lagr\left(D^*, \alpha^*, \hat{\lambda}, \hat{\tau}\right) + 2v\\
&= \err\left(D^*, S\right) + penalty_{S, \hat{w}}\left(D^*, \alpha^*, \hat{\lambda}, \hat{\tau}\right) + 2v\\
&\le \err\left(D^*, S\right) + 2v
\end{align*}
The first inequality and the third inequality are from the definition of $v$-approximate saddle point, and the second to last equality comes from the fact that $D^*,a^*$ is a feasible solution.
Now, we consider two cases when$\left(\hat{D},\hat{\alpha}\right)$ is a feasible solution and when it's not.
\begin{enumerate}
\item $\left(\hat{D},\hat{\alpha}\right) \in \Omega\left(S,\hat{w}, \gamma, \eta\right)$\\
In this case, $\max_{\lambda \in \Lambda, \tau \in \Tau} penalty_{S, \hat{w}}\left(\hat{D}, \hat{\alpha}, \lambda, \tau\right) = 0$ because by the definition of being a feasible solution, \[\E_{h\sim D}\left[h(x_i) - h(x_j)\right] \le \alpha_{ij} + \gamma, \forall (i,j) \in [n]^2\] and
\[\frac{1}{|A|}\sum_{(i, j)\in [n]^2} \hat{w}_{ij} \alpha_{ij} \le \eta.\]
Hence, $\max_{\lambda \in \Lambda, \tau \in \Tau} \Lagr\left(\hat{D}, \hat{\alpha}, \lambda, \tau\right) = err\left(\hat{D}, S\right)$. Therefore, we have $err\left(\hat{D}, S\right) \le err\left(D^*, S\right) + 2v$.
\item $\left(\hat{D},\hat{\alpha}\right) \notin \Omega\left(S,\hat{w}, \gamma, \eta\right)$\\
$\max_{\lambda \in \Lambda, \tau \in \Tau} \Lagr\left(\hat{D}, \hat{\alpha}, \lambda, \tau\right) = err\left(\hat{D}, S\right) + \max_{\lambda \in \Lambda, \tau \in \Tau} penalty_{S, \hat{w}}\left(\hat{D}, \hat{\alpha}, \lambda, \tau\right)$. Therefore, $\err\left(\hat{D}, S\right) \le \err\left(D^*, S\right) + 2v$ because $\max_{\lambda \in \Lambda, \tau \in \Tau} penalty_{S, \hat{w}}\left(\hat{D}, \hat{\alpha}, \lambda, \tau\right) \ge 0$.
Now, we show that even when $(\hat{D}, \hat{\alpha})$ is not a feasible solution, the constraints are violated only by so much.
\begin{align*}
\max_{\lambda \in \Lambda, \tau \in \Tau} \Lagr(\hat{D}, \hat{\alpha}, \lambda, \tau) = \err(\hat{D}, S) + \max_{\lambda \in \Lambda, \tau \in \Tau} penalty_{S, \hat{w}}(\hat{D}, \hat{\alpha}, \lambda, \tau) &\le \err(D^*, S) + 2v\\
\max_{\lambda \in \Lambda, \tau \in \Tau} penalty_{S, \hat{w}}(\hat{D}, \hat{\alpha}, \hat{\lambda}, \hat{\tau}) &\le \err(D^*, S)- err(\hat{D}, S) + 2v\\
\max_{\lambda \in \Lambda, \tau \in \Tau} penalty_{S, \hat{w}}(\hat{D}, \hat{\alpha}, \hat{\lambda}, \hat{\tau}) &\le 1 + 2v
\end{align*}
Let $\lambda^*, \tau^* = BEST_{\psi}\left(\hat{D}, \hat{\alpha}\right)$, which minimizes the function as shown in Lemma \ref{lem:dual_lambda} and \ref{lem:dual_tau} (in the appendix). Now, consider
\[
\sum_{(i,j)} \lambda^*_{ij} \left( \E\limits_{h\sim D}\left[h(x_i) - h(x_j)\right] - \alpha_{ij} -\gamma \right) + \tau^* \left( \frac{1}{|A|}\sum_{(i,j)} \hat{w}_{ij} \alpha_{ij} - \eta \right) \le 1 + 2v
\]
Say $\left(i^*,j^*\right) = \argmax_{(i,j) \in [n^2]} \E\limits_{h\sim D}\left[h(x_i) - h(x_j)\right] - \alpha_{ij} -\gamma$. Note that if $\E\limits_{h\sim D}\left[h(x_{i^*}) - h(x_{j^*})\right] - \alpha_{i^*j^*} -\gamma > 0$, then $\lambda^*_{i^*j^*} = C_{\tau}$ and 0 for the other coordinates and else, it's just a zero vector. Also, $\tau=C_{\tau}$ if $\sum_{(i,j)} \hat{w}_{ij} \alpha_{ij} - \eta > 0$ and 0 otherwise. Thus,
\[
\sum_{(i,j)} \lambda^*_{ij} \left( \E\limits_{h\sim D}\left[h(x_i) - h(x_j)\right] - \alpha_{ij} -\gamma \right) \ge 0 \text{ and } \tau^* \left( \frac{1}{|A|}\sum_{(i,j)} \hat{w}_{ij} \alpha_{ij} - \eta \right) \ge 0
\]
Therefore, we have
\[
\max_{i,j \in [n]^2}(\E\limits_{h\sim D}[h(x_{i}) - h(x_{j})] - \alpha_{ij} -\gamma) \le \frac{1+2v}{C_{\lambda}},
\]
and
\[
\frac{1}{|A|}\sum_{(i, j)\in [n]^2} \hat{w}_{ij} \hat{\alpha}_{ij} \le \eta + \frac{1+2v}{C_\tau}
\]
\end{enumerate}
\end{proof}
\section{Generalization}\label{sec:generalization}
In this section, we show that fairness loss generalizes out-of-sample. Error generalization follows from the standard VC-dimension bound, which --- because it is a uniform convergece statement is unaffected by the addition of fairness constraints.
\begin{theorem}[\cite{kearns1994introduction}]\label{thm:error-generalization}
Fix some hypothesis class $\mathcal{H}$ and distribution $\cP$. Let $S \sim P^n$ be a dataset consisting of $n$ examples $\{x_i, y_i\}_{i=1}^n$ sampled i.i.d. from $\cP$. Then, for any $0<\delta<1$, with probability $1-\delta$, for every $h \in \mathcal{H}$, we have
\[
\left\vert err(h, \cP) - err(h, S)\right\vert \le O\left( \sqrt{\frac{VCDIM(\mathcal{H})+ log(\frac{1}{\delta})}{n}}\right)
\]
\end{theorem}
Proving that the fairness loss generalizes doesn't follow immediately from a standard VC-dimension argument for several reasons: it is not linearly separable, but defined as an average over non-disjoint \emph{pairs} of individuals in the sample. The difference between empirical fairness loss and true fairness loss of a randomized hypothesis $D \in \Delta\mathcal{H}$ is also a non-convex function of the supporting hypotheses $h$, and so it is not sufficient to prove a uniform convergence bound merely for the base hypotheses in our hypothesis class $\mathcal{H}$. We circumvent these difficulties by making use of an $\epsilon$-net argument, together with an application of a concentration inequality, and an application of Sauer's lemma. Briefly, we show that with respect to fairness loss, the continuous set of distributions over classifiers have an $\epsilon$-net of sparse distributions. Using the two-sample trick and Sauer's lemma, we can bound the number of such sparse distributions.
\subsection{Fairness Loss}
At a high level, our argument proceeds as follows: using McDiarmid's inequality, for any \emph{fixed} hypothesis, its empirical fairness loss concentrates around its expectation. This argument extends to an infinite family of hypotheses with bounded VC-dimension via the standard two-sample trick, together with Sauer's lemma: the only catch is that we need to use a variant of McDiarmid's inequality that applies to sampling without replacement. However, proving that the fairness loss for each fixed hypothesis $h$ concentrates around its expectation is not sufficient to obtain the same result for arbitrary distributions over hypotheses, because the difference between a randomized classifier's fairness loss and its expectation is a non-convex function of the mixture weights. To circumvent this issue, we show that with respect to fairness loss, there is an $\epsilon$-net consisting of sparse distributions over hypotheses. Once we apply Sauer's lemma and the two-sample trick, there are only finitely many such distributions, and we can union bound over them.
We begin by stating the standard version of McDiarmid's inequality:
\begin{theorem}[McDiarmid's Inequality]
Suppose $X_1, \dots, X_n$ are independent and $f$ satisfies
\[\sup_{x_1, \dots, x_n, \hat{x}_i} \left\vert f(x_1, \dots, x_n) - f(x_1, \dots, x_{i-1}, \hat{x}_i, x_{i+1}, \dots, x_n)\right\vert \le c_i.\]
Then, for any $\epsilon > 0$,
\[
\Pr_{X^1, \dots, X^n }\left(\left\vert f(X_1, \dots, X_n) - \E_{X_1, \dots, X_n}\left[f(X_1, \dots, X_n)\right]\right\vert \ge \epsilon \right)\le 2\exp\left(-\frac{2\epsilon^2}{\sum_{i=1}^n c_i^2}\right)
\]
\end{theorem}
\swcomment{$\Pi_{D, w, \gamma}(S \times S)$ is not defined, right?}
\cj{It is. We've defined $\Pi_{D, w, \gamma}(A)$ for set of pairs in the preliminary}
\begin{lemma}\label{lem:sample_concentration} \swcomment{maybe set up some notations here; what's $n$, $S$, and we fix $D$ etc...}\cj{Done}
Fix a randomized hypothesis $D \in \Delta\mathcal{H}$. Over the randomness of $S \sim \cP^n$, we have
\[
\Pr_{S \sim \cP^n}\left( \left\vert\Pi_{D, w, \gamma}(S \times S) - \E_{S}\left[\Pi_{D, w, \gamma}(S \times S)\right]\right\vert \ge \epsilon \right)\le 2\exp\left(-2n\epsilon^2\right)
\]
\end{lemma}
\begin{proof}
Define a slightly modified fairness loss function that depends on each instance instead of pairs.
\[
\Pi'_{D,w,\gamma}(x_1, x_2, \dots, x_{n}) = \frac{1}{{n}^2}\sum_{(i,j) \in [n]^2} \Pi_{D,w,\gamma}\left((x_i, x_j)\right).
\]
Note that $\Pi'_{D,w,\gamma}(x_1, \dots, x_n) = \Pi_{D,w,\gamma}(S \times S)$. The sensitivity of $\Pi'_{D,w,\gamma}(x_1, x_2, \dots, x_{n})$ is $\frac{1}{n}$, so applying McDiarmid's inequality yields the above concentration.\swcomment{where did $x'_i$ come up?}\cj{Oops. Fixed}
\end{proof}
Now, following the argument described above, we show that the difference between empirical fairness loss over $S \times S$ and true fairness loss converges uniformly over $D \in \Delta\mathcal{H}$ with high probability.
\begin{theorem}\label{thm:fairness-loss-all-pairs}
If $n \ge \frac{2\ln(2)}{\epsilon^2}$,
\[
\Pr_{S \sim \cP^n}\left(\sup_{D \in \Delta\mathcal{H}}\left\vert\Pi_{D,w,\gamma}(S \times S) - \E_{x,x'}\left[\Pi_{D,w,\gamma}(x,x')\right]\right\vert > \epsilon \right) \le 8 \cdot \left(\frac{e\cdot 2n}{d}\right)^{dk} \exp\left(\frac{-n\epsilon^2}{32}\right)
\]
where $d$ is the VC-dimension of $\mathcal{H}$, and $k=\frac{\ln(2n^2)}{8\epsilon^2} + 1$. \swcomment{wait, what is $\epsilon'$?}\cj{I got rid of it.}
\end{theorem}
\begin{proof}
%This proof is a slightly modified version of the proof for VC-inequality; it uses McDiarmid's inequality instead of Hoeffding's in order to circumvent dependency between pairs and also incorporates a net argument in order to argue for uniform convergence over $\Delta\mathcal{H}$ and not just over $\mathcal{H}$.
First, by linearity of expectation, we note that $\E_{S}[\Pi_{D,w,\gamma}(S \times S)] = \E_{x,x'}\left[\Pi_{D,w,\gamma}(x,x')\right]$. Given $S$, let $D^*_S$ be some randomized classifier such that $\left\vert\Pi_{D^*_S,w,\gamma}(S \times S) - \E_{x,x'}\left[\Pi_{D^*_S,w,\gamma}(x,x')\right]\right\vert > \epsilon$; if such hypothesis does not exist, let it be some fixed hypothesis in $\mathcal{H}$.\swcomment{I don't understand this previous sentence: if not ``what condition''?} \cj{if there does exist a hypothesis such that }We now use the standard symmetrization argument, which allows us to bound the difference between the fairness loss of our sample $S$ and that of another independent `ghost' sample $S' = (x'_1, \dots, x'_n)$ instead of bounding the difference between the empirical fairness loss and its expected fairness loss.
\swcomment{not sure about the first equality; should we just fix $D^*$ and push through the inequalities, and then sup over $D$ follows directly?}\cj{Should be an inequality. I think the way I have it now works.}
\begin{align*}
&\Pr_{S \sim \cP^n,S' \sim \cP^n} \left(\sup_{D \in \Delta\mathcal{H}}\left\vert\Pi_{D,w,\gamma}(S \times S) - \Pi_{D,w,\gamma}(S' \times S') \right\vert> \frac{\epsilon}{2} \right)\\
&\ge\Pr_{S,S'} \left(\left\vert \Pi_{D^*_S,w,\gamma}(S \times S) - \Pi_{D^*_S,w,\gamma}(S' \times S')\right\vert > \frac{\epsilon}{2} \right)\\
&\ge \Pr_{S, S'} \left(\left\vert \Pi_{D^*_S,w,\gamma}(S \times S) - \E_{x,x'}\left[\Pi_{D^*_S,w,\gamma}(x,x')\right] \right\vert > \epsilon \text{ and } \left\vert \Pi_{D^*,w,\gamma}(S' \times S') - \E_{x,x'}\left[\Pi_{D^*,w,\gamma}(x,x')\right] \right\vert \le \frac{\epsilon}{2} \right) \\
&= \E_{S,S'}\left[ \ind\left(\left\vert \Pi_{D^*_S,w,\gamma}(S \times S) - \E_{x,x'}\left[\Pi_{D^*_S,w,\gamma}(x,x')\right]\right\vert > \epsilon\right) \cdot \ind\left(\left\vert\Pi_{D^*,w,\gamma}(S' \times S') - \E_{x,x'}\left[\Pi_{D^*,w,\gamma}(x,x')\right]\right\vert \le \frac{\epsilon}{2}\right) \right]\\
&= \E_{S}\left[ \ind\left(\left\vert \Pi_{D^*_S,w,\gamma}(S \times S) - \E_{x,x'}\left[\Pi_{D^*_S,w,\gamma}(x,x')\right]\right\vert > \epsilon\right) \cdot \Pr_{S' \vert S}\left(\left\vert\Pi_{D^*,w,\gamma}(S' \times S') - \E_{x,x'}\left[\Pi_{D^*,w,\gamma}(x,x')\right]\right\vert \le \frac{\epsilon}{2}\right) \right]\\
&\ge \Pr_{S}\left(\left\vert\Pi_{D^*_S,w,\gamma}(S \times S) - \E_{x,x'}\left[\Pi_{D^*_S,w,\gamma}(x,x')\right]\right\vert > \epsilon) \right) \cdot \left(1-\exp\left(-\frac{n\epsilon^2}{2}\right) \right)\\
&\ge \frac{1}{2} \Pr_{S}\left(\sup_{D \in \Delta\mathcal{H}}\left\vert\Pi_{D,w,\gamma}(S \times S) - \E_{x,x'}\left[\Pi_{D,w,\gamma}(x,x')\right]\right\vert > \epsilon \right)\\
\end{align*}
We used Lemma \ref{lem:sample_concentration} for the second to last inequality, and the last inequality follows from the theorem's condition (i.e. $n \ge \frac{2\ln(2)}{\epsilon^2}$) and how we defined $D^*_S$.
Now, imagine sampling $\bar{S} = 2n$ points from $\cP$, and uniformly choosing $n$ points without replacement to be $S$ and the remaining $n$ points to be $S'$. This process is equivalent to sampling $n$ points from $\cP$ to form $S$ and another independent set of $n$ points from $\cP$ to form $S'$.
\swcomment{we want ``without replacement''?}\cj{Fixed.}
\begin{align*}
&\Pr_{\bar{S}, S,S'}\left(\sup_{D \in \Delta\mathcal{H}}\left\vert\Pi_{D,w,\gamma}(S \times S) - \Pi_{D,w,\gamma}(S' \times S') \right\vert > \frac{\epsilon}{2}\right) \\
&= \sum_{\bar{S}} \Pr\left(\bar{S}\right) \Pr_{S,S'}\left(\sup_{D \in \Delta\mathcal{H}}\left\vert\Pi_{D,w,\gamma}(S \times S) - \Pi_{D,w,\gamma}(S' \times S') \right\vert > \frac{\epsilon}{2} \bmid \bar{S}\right) \\
\end{align*}
Now, we show that the continuous set of distributions over classifiers $\Delta\mathcal{H}$ can be approximated by an $\epsilon'$-net of sparse distributions. By sparse distribution, we mean uniform distributions over supports of at most $k$, for some fixed value $k$: $\hat{D} = \frac{1}{k}(h_1, \dots, h_k)$ where $h_i \in \mathcal{H}$ for $i \in [k]$. Because by Sauer's lemma, the set of hypotheses $\mathcal{H}$ induces at most $O(n^d)$ distinct labellings of a dataset of sizse $n$, we need to union bound over at most $O(n^{dk})$ distinct sparse distributions.
\begin{lemma}\label{lem:approximate_random}
For some fixed data sample $S$ of size $n$, any $D \in \Delta\mathcal{H}$ can be approximated by some uniform mixture over $k := \frac{2 \ln(2n^2)}{\epsilon'^2} + 1$ hypotheses $\hat{D} = \frac{1}{k} \{h_1, \dots, h_{k}\}$ such that for every $(x,x') \in S \times S$,
\[\left\vert \E_{h \sim D}\left[h(x) - h(x')\right] - \E_{h \sim \hat{D}}\left[h(x) - h(x')\right] \right\vert \le \epsilon'. \]
\end{lemma}
\begin{proof}
Fix some $(x,x') \in S \times S$. Randomly sample $k$ hypotheses from $D$: $\{h_i\}_{i=1}^k \sim D^k$. Because for each randomly drawn hypothesis $h_i \sim D$, the difference in its prediction for $x$ and $x'$ is exactly $\E_{h \sim D}[h(x) - h(x')]$, Hoeffding's inequality yields that
\[
\Pr_{h_i \sim D, i \in [k]}\left(\left\vert \E_{h \sim D}\left[h(x) - h(x')\right] - \frac{1}{k} \sum_{i=1}^k \left[h_i(x) - h_i(x')\right] \right\vert > \epsilon'\right) \le 2\exp\left(-\frac{2k^2\epsilon'^2}{4k}\right) = 2\exp\left(-\frac{k\epsilon'^2}{2}\right).
\]
However, there are $n^2$ fixed pairs in $S \times S$, and if we distribute the failure property between $n^2$ pairs and union bound over all of them, we get
\[
\Pr_{h_i \sim D, i \in [k]} \left( \max_{(x,x') \in S \times S} \left\vert \E_{h \sim D}\left[h(x) - h(x')\right] - \frac{1}{k} \sum_{i=1}^k \left[h_i(x) - h_i(x')\right] \right\vert > \epsilon' \right) \le 2n^2\exp\left(-\frac{k\epsilon'^2}{2}\right).
\]
In order to achieve non-zero probability of having \[\left\vert \E_{h \sim D}\left[h(x) - h(x')\right] - \frac{1}{k} \sum_{i=1}^k \left[h_i(x) - h_i(x')\right] \right\vert \le \epsilon', \forall (x,x') \in S \times S,\] we need to make sure $2n^2\exp\left(-\frac{k\epsilon'^2}{2}\right) < 1$ or $k > \frac{2 \ln\left(2n^2\right)}{\epsilon'^2}$.
\end{proof}
\begin{corollary}\label{cor:approximate-random}
For some fixed data sample $S$, any $D \in \Delta\mathcal{H}$ can be approximated by a uniform mixture of $k := \frac{2 \ln\left(2n^2\right)}{\epsilon'^2} + 1$ hypotheses $\hat{D} = \frac{1}{k} \{h_1, \dots, h_{k}\}$ such that
\[
\left\vert \Pi_{D, w, \gamma}(S \times S) - \Pi_{\hat{D}, w, \gamma}(S \times S)\right\vert \le \epsilon'
\]
\end{corollary}
\begin{proof}
It simply follows from Lemma \ref{lem:approximate_random} and the fact that $\max\left(0, \E_{h \sim D}\left[h(x_i) - h(x_j)\right] - \gamma \right)$ is 1-Lipschitz in terms of $\E_{h \sim D}\left[h(x_i) - h(x_j)\right]$.
\end{proof}
Using Corollary \ref{cor:approximate-random} and Sauer's lemma that bounds the total number of possible labelings by $\mathcal{H}$ over $2n$ points by $(\frac{e\cdot 2n}{d})^d$, we have
\begin{align*}
&\sum_{\bar{S}} \Pr\left(\bar{S}\right) \Pr_{S,S'}\left(\sup_{D \in \Delta\mathcal{H}}\left\vert\Pi_{D,w,\gamma}(S \times S) - \Pi_{D,w,\gamma}(S' \times S') \right\vert > \frac{\epsilon}{2} \bmid \bar{S}\right) \\
&\le \sum_{\bar{S}} \Pr\left(\bar{S}\right) \Pr_{S,S'}\left(\sup_{\hat{D} \in \mathcal{H}^k}\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - \Pi_{\hat{D},w,\gamma}(S' \times S') \right\vert > \frac{\epsilon}{2} + \epsilon' \bmid \bar{S}\right) \\
&\le \sum_{\bar{S}} \Pr\left(\bar{S}\right) \cdot \left(\frac{e\cdot 2n}{d}\right)^{dk} \sup_{\hat{D} \in \mathcal{H}^k} \Pr_{S,S'}\left(\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - \Pi_{\hat{D},w,\gamma}(S' \times S') \right\vert > \frac{\epsilon}{2} + \epsilon' \bmid \bar{S}\right) \\
\end{align*}
Now, for any $\hat{D}$, we will try to bound the probability that the difference in fairness loss between $S$ and $S'$ is big. We do so by union bounding over cases where both of them deviate from its mean by too much.
If $\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - E_{S | \bar{S}}\left[\Pi_{\hat{D},w,\gamma}(S \times S)\right] \right\vert \le \frac{\epsilon}{4} + \frac{\epsilon'}{2}$ and $\left\vert\Pi_{\hat{D},w,\gamma}(S' \times S') - E_{S | \bar{S}}\left[\Pi_{\hat{D},w,\gamma}(S \times S)\right] \right\vert \le \frac{\epsilon}{4} + \frac{\epsilon'}{2}$, then $\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - \Pi_{\hat{D},w,\gamma}(S' \times S') \right\vert \le \frac{\epsilon}{2} + \epsilon'$. In other words,
\begin{align*}
&\Pr_{S,S'}\left(\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - \Pi_{\hat{D},w,\gamma}(S' \times S') \right\vert \le \frac{\epsilon}{2} + \epsilon' \bmid \bar{S}\right)\\
&\ge \Pr_{S, S'}\left(\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - E_{S | \bar{S}}\left[\Pi_{\hat{D},w,\gamma}(S \times S)\right] \right\vert \le \frac{\epsilon}{4} + \frac{\epsilon'}{2} \text{ and } \left\vert\Pi_{\hat{D},w,\gamma}(S' \times S') - E_{S | \bar{S}}\left[\Pi_{\hat{D},w,\gamma}(S \times S)\right] \right\vert \le \frac{\epsilon}{4} + \frac{\epsilon'}{2} \bmid \bar{S}\right).
\end{align*}
Therefore, by looking at the compliment probabilities, we have
\begin{align*}
&\Pr_{S,S'}\left(\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - \Pi_{\hat{D},w,\gamma}(S' \times S') \right\vert > \frac{\epsilon}{2} + \epsilon' \bmid \bar{S}\right) \\
&\le \Pr_{S, S'}\left(\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - E_{S | \bar{S}}\left[\Pi_{\hat{D},w,\gamma}(S \times S)\right] \right\vert > \frac{\epsilon}{4} + \frac{\epsilon'}{2} \text{ or } \left\vert\Pi_{\hat{D},w,\gamma}(S' \times S') - E_{S | \bar{S}}\left[\Pi_{\hat{D},w,\gamma}(S \times S)\right] \right\vert > \frac{\epsilon}{4} + \frac{\epsilon'}{2} \bmid \bar{S}\right)\\
&\le 2\Pr_{S}\left(\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - E_{S | \bar{S}}\left[\Pi_{\hat{D},w,\gamma}(S \times S)\right] \right\vert > \frac{\epsilon}{4} + \frac{\epsilon'}{2} \bmid \bar{S}\right).
\end{align*}
Here, we can't appeal to McDiarmid's because $S$ is sampled without replacement from $\bar{S}$. However, we use the stochastic covering property to show concentration for sampling without replacement \citep{pemantle2014concentration} (a similar technique was used by \cite{neel2018use})
\begin{definition}[\cite{pemantle2014concentration}]
$Z_1, \dots, Z_n$ satisfy the stochastic covering property, if for any $I \subset [n]$ and $a \ge a' \in \{0,1\}^I$ coordinate-wise such that $||a' - a||_1 = 1$, there is a coupling $\nu$ of the distributions $\mu, \mu'$ of $(Z_j: j \in [n] \setminus I)$ conditioned on $Z_I = a$ or $Z_I = a'$, respectively, such that $\nu(x, y) = 0$ unless $x \le y$ coordinate-wise and $||x-y||_1 \le 1$. \swcomment{what does $a\ge a'$ mean? they r boolean vecs? }\cj{Added coordinate-wise}
\end{definition}
\begin{theorem}[\cite{pemantle2014concentration}]\label{thm:without-replacement-concentration} \swcomment{maybe not use $n$ here...}
Let $(Z_1, \dots, Z_n) \in \{0,1\}$ be random variables such that $\Pr(\sum_{i=1}^n Z_i = k)=1$ and the stochastic covering property is satisfied. Let $f: \{0,1\}^n \to \cR$ be an $c$-Lipschitz function. Then, for any $\epsilon > 0$,
\[
\Pr\left(\left\vert f(Z_1, \dots, Z_n) - \E[f(Z_1, \dots, Z_n)]\right\vert \ge \epsilon\right) \le 2\exp\left(\frac{-\epsilon^2}{8c^2k}\right)
\]
\swcomment{did we use $\cR$ for real numbers?}
\cj{yeah.. Just changed command cR so that it's mathbb R}
\end{theorem}
\begin{lemma}[\cite{neel2018use}]\label{lem:without-replacement-property}
Given a set $S$ of $n$ points, sample $k \le n$ elements without replacement. Let $Z_i = \{0,1\}$ indicate whether $i$th element has been chosen. Then, $(Z_1, \dots, Z_n)$ satisfy the stochastic covering property.
\end{lemma}
Let $\bar{S} = \{x_1, \dots, x_{2n}\}$. If we slightly change the definition of the fairness loss so that it depends on the indicator variables $Z_1, \dots, Z_{2n}$,
\[\Pi''_{\hat{D}, w, \gamma, \bar{S}}(Z_1, \dots, Z_{2n})= \frac{1}{n^2} \sum_{i,j \in [2n]^2} Z_i Z_j \Pi_{\hat{D}, w, \gamma}(x_i, x_j) = \Pi_{\hat{D}, w, \gamma}(S \times S).\]
We see that $\Pi''_{\hat{D}, w, \gamma, \bar{S}}$ is $\frac{1}{n}$-Lipschitz, so by theorem \ref{thm:without-replacement-concentration} and lemma \ref{lem:without-replacement-property}, we get
\[
\Pr_{S}\left(\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - E_{S | \bar{S}}\left[\Pi_{\hat{D},w,\gamma}(S \times S)\right] \right\vert > \frac{\epsilon}{4} + \frac{\epsilon'}{2} \bmid \bar{S}\right) \le 2\exp\left(\frac{-\left(\frac{\epsilon}{4} + \frac{\epsilon'}{2}\right)^2}{8\frac{1}{n^2} \cdot n}\right) = 2\exp\left(\frac{-n\left(\frac{\epsilon}{4} + \frac{\epsilon'}{2}\right)^2}{8}\right)
\]
Combining everything, we get
\begin{align*}
&\Pr_{S}\left(\sup_{D \in \Delta\mathcal{H}}\left\vert\Pi_{D,w,\gamma}(S \times S) - \E_{x,x'}\left[\Pi_{D,w,\gamma}(x,x')\right]\right\vert > \epsilon \right) \\
&\le 2\sum_{\bar{S}} \Pr\left(\bar{S}\right) \cdot \left(\frac{e\cdot 2n}{d}\right)^{dk} \sup_{\hat{D} \in \mathcal{H}^k} \Pr_{S,S'}\left(\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - \Pi_{\hat{D},w,\gamma}(S' \times S') \right\vert > \frac{\epsilon}{2} + \epsilon' \bmid \bar{S}\right)\\
&\le 4\sum_{\bar{S}} \Pr\left(\bar{S}\right) \cdot \left(\frac{e\cdot 2n}{d}\right)^{dk} \sup_{\hat{D} \in \mathcal{H}^k} \Pr_{S}\left(\left\vert\Pi_{\hat{D},w,\gamma}(S \times S) - E_{S | \bar{S}}\left[\Pi_{\hat{D},w,\gamma}(S \times S)\right] \right\vert > \frac{\epsilon}{4} + \frac{\epsilon'}{2} \bmid \bar{S}\right) \\
&\le 8 \cdot \left(\frac{e\cdot 2n}{d}\right)^{dk} \exp\left(\frac{-n\left(\frac{\epsilon}{4} + \frac{\epsilon'}{2}\right)^2}{8}\right)
\end{align*}
For convenience, we set $\epsilon'=\frac{\epsilon}{2}$.
%\cj{I might the probability as it is, and instead calculate sample complexity in the final theorem statement}
%For convenience, setting $\epsilon' = \frac{\epsilon}{2}$ and the failure probability to equal $\delta$, we have
%\begin{align*}
%\delta &= 8 \cdot (\frac{e\cdot 2n}{d})^{dk} \exp(\frac{-n\epsilon^2}{32})\\
%\ln(\delta) &= \ln(8) + dk \ln(\frac{e\cdot 2n}{d}) -\frac{n\epsilon^2}{32}\\
%\frac{n\epsilon^2}{32} &= \ln(8) + d (\frac{8 \ln(2n^2)}{\epsilon^2} + 1) (1 + \ln(2) + \ln(n) - \ln(d)) -\ln(\delta)
%\end{align*}
\end{proof}
However, in our case, instead of finding the average over all pairs in $S$, we calculate the fairness loss only over $m$ randomly chosen pairs. Fixing $S$, if $m$ is sufficiently large, our empirical fairness loss should concentrate around the fairness loss over all the pairs for $S$.
\begin{lemma}\label{lem:random-pairs}
For fixed $S$, randomly chosen pairs $M \subset S \times S$, and randomized hypothesis $D$,
\[\Pr_{M \sim (S \times S)^m}\left( \Pi_{D,w,\gamma}(M) - \Pi_{D,w,\gamma}(S \times S) \ge \epsilon \right) \le \exp\left(-2m\epsilon^2\right)\]
\end{lemma}
\begin{proof}\swcomment{abusing $i$ below...}\cj{Ooops, I will use $a$ to iterate over the pairs then}
Write a random variable $L_a = \Pi_{D,w,\gamma}((x_{2a-1}, x_{2a}))$ for the fairness loss of the $a$th pair.
Note that \[E[L_a] = \sum_{(i,j) \in [n]^2} \frac{1}{n^2} \Pi_{D,w,\gamma}\left((x_i,x_j)\right)=\Pi_{D,w,\gamma}(S \times S), \forall a \in [|M|].\]
Therefore, by Hoeffding's inequality, we have
\[
\Pr_{M}\left( \Pi_{D,w,\gamma}(M) - \Pi_{D,w,\gamma}(S \times S) \ge \epsilon \right) \le \exp\left(-2m\epsilon^2\right).
\]
\end{proof}
Once again, using $\epsilon$-net sparsification of $\Delta\mathcal{H}$, we show the above concentration converges uniformly over $D \in \Delta\mathcal{H}$
\begin{lemma}\label{lem:unif-conver-random-pairs}
For fixed $S$ and randomly chosen pairs $M \subset S \times S$,
\[\Pr_{M \sim (S \times S)^m}\left(\sup_{D \in \Delta\mathcal{H}} \left\vert \Pi_{D,w,\gamma}(M) - \Pi_{D,w,\gamma}(S \times S) \right\vert \ge \epsilon \right) \le \left(\frac{e\cdot 2n}{d}\right)^{dk'} \exp\left(-8m\epsilon^2\right),\]
where $k'=\frac{2 \ln(2m)}{\epsilon^2} + 1$.
\end{lemma}
\begin{proof}
\begin{align*}
&\Pr_{M \sim (S \times S)^m}\left(\sup_{D \in \Delta\mathcal{H}} \left\vert \Pi_{D,w,\gamma}(M) - \Pi_{D,w,\gamma}(S \times S) \right\vert \ge \epsilon \right)\\
&\le \Pr_{M \sim (S \times S)^m}\left(\sup_{\hat{D} \in \mathcal{H}^k} \left\vert \Pi_{D,w,\gamma}(M) - \Pi_{D,w,\gamma}(S \times S) \right\vert \ge \epsilon + \epsilon' \right)\\
&\le \sum_{\hat{D} \in \mathcal{H}^k} \Pr_{M \sim (S \times S)^m}\left( \left\vert \Pi_{D,w,\gamma}(M) - \Pi_{D,w,\gamma}(S \times S) \right\vert \ge \epsilon + \epsilon' \right)\\
&\le \left(\frac{e\cdot 2n}{d}\right)^{dk} \exp\left(-2m\left(\epsilon + \epsilon'\right)^2\right),
\end{align*}
where $k = \frac{2 \ln(2m)}{\epsilon'^2} + 1$. The last inequality is from Corollary \ref{cor:approximate-random} and Lemma \ref{lem:random-pairs}. For convenience, we just set $\epsilon' = \epsilon$.
\end{proof}
Combining theorem \ref{thm:fairness-loss-all-pairs} and lemma \ref{lem:unif-conver-random-pairs} yields the following theorem for fairness loss generalization.
\ar{Removed the standard statement of error generalization for space. We should make sure it appears in the supplement. }\cj{This looks awesome! Thanks. We have a theorem from \cite{kearns1994introduction} in the supplement}
%Because the error of a randomized hypothesis $D$ is just a convex combination of the error of the support hypotheses, $err(D, \cP) - err(D, S) =\E_{h \sim D}[err(h, \cP) - err(h, S)]$. The error generalization bound holds over $\Delta\mathcal{H}$ as well.
%\begin{corollary}[corollary of \cite{kearns1994introduction}]
%Fix some hypothesis class $\mathcal{H}$ and distribution $\cP$. Let $S \sim P^n$ be a dataset consisting of $n$ examples $\{x_i, y_i\}_{i=1}^n$ sampled i.i.d. from $\cP$. Then, for any $0<\delta<1$, with probability $1-\delta$, for every $D \in \Delta\mathcal{H}$, we have
%\[
%\left\vert err(D, \cP) - err(D, S)\right\vert \le O\left( \sqrt{\frac{VCDIM(\mathcal{H})+ log(\frac{1}{\delta})}{n}}\right)
%\]
%\end{corollary}
%\mk{Not really sure KV94 is the right citation for this, and the corollary above is really so standard that it might be preferable to omit it and use the
%space to provide a little interpretation of the theorem below, which right now is just dropped on the reader with no discussion.}
\begin{theorem}\label{thm:fairness-loss-generalization}
Let $S$ consists of $n$ i.i.d points drawn from $\cP$ and let $M$ represent a set of $m$ pairs randomly drawn from $S \times S$ \ar{Is $M$ the same as $A$ from before? Why use different notation?}\cj{$A=\{(i,j)\}$ where as $M=\{(x_i,x_j)\}$. This is confusing, and maybe, we should just say that we might abuse notation.}. Then we have:
\begin{align*}
&\Pr_{\substack{S \sim \cP^n \\ M \sim (S \times S)^m}}\left(\sup_{D \in \Delta\mathcal{H}}\left\vert\Pi_{D,w,\gamma}(M) - \E_{(x,x') \sim \cP^2}[\Pi_{D,w,\gamma}(x,x')]\right\vert > 2\epsilon \right) \\
&\le \left( 8 \cdot \left(\frac{e\cdot 2n}{d}\right)^{dk} \exp\left(\frac{-n\epsilon^2}{32}\right)+ \left(\frac{e\cdot 2n}{d}\right)^{dk'} \exp\left(-8m\epsilon^2\right) \right),
\end{align*}
where $k'=\frac{2 \ln(2m)}{\epsilon^2} + 1$, $k=\frac{\ln(2n^2)}{8\epsilon^2} + 1$, and $d$ is the VC-dimension of $\mathcal{H}$. \ar{Right?}\cj{yes!}
\end{theorem}
To interpret this theorem, note that the right hand side (the probability of a failure of generalization) begins decreasing exponentially fast in the data and fairness constraint sample parameters $n$ and $m$ as soon as $n \geq \Omega( d\log(n)\log(n/d))$ and $m \geq \Omega(d\log(m)\log(n/d))$. \ar{Added this, check its correct}
\begin{proof}[proof of theorem \ref{thm:fairness-loss-generalization}]
With probability $1-\left( 8 \cdot \left(\frac{e\cdot 2n}{d}\right)^{dk} \exp\left(\frac{-n\epsilon^2}{32}\right)+ \left(\frac{e\cdot 2n}{d}\right)^{dk'} \exp\left(-8m\epsilon^2\right) \right)$, where $k'=\frac{2 \ln(2m)}{\epsilon^2} + 1$ and $k=\frac{\ln(2n^2)}{8\epsilon^2} + 1$,
we have
\[\sup_{D \in \Delta\mathcal{H}} \left\vert \Pi_{D,w,\gamma}(M) - \Pi_{D,w,\gamma}(S \times S) \right\vert \le \epsilon\]
and
\[\sup_{D \in \Delta\mathcal{H}} \left\vert \Pi_{D,w,\gamma}(S \times S) - \E_{x,x'}[\Pi_{D,w,\gamma}(x,x') \right\vert \le \epsilon.\]
Then, by triangle inequality,
\begin{align*}
\sup_{D \in \Delta\mathcal{H}} \left\vert \Pi_{D,w,\gamma}(M) - \E_{x,x'}[\Pi_{D,w,\gamma}(x,x') \right\vert \le 2\epsilon.
\end{align*}
In other words, with probability $\left( 8 \cdot \left(\frac{e\cdot 2n}{d}\right)^{dk} \exp\left(\frac{-n\epsilon^2}{32}\right)+ \left(\frac{e\cdot 2n}{d}\right)^{dk'} \exp\left(-8m\epsilon^2\right) \right)$, we have
\[
\sup_{D \in \Delta\mathcal{H}} \left\vert \Pi_{D,w,\gamma}(M) - \E_{x,x'}[\Pi_{D,w,\gamma}(x,x') \right\vert > 2\epsilon.
\]
\end{proof}
%%% Local Variables:
%%% mode: latex
%%% TeX-master: "main"
%%% End:
\section{A Behavioral Study of Subjective Fairness: Preliminary Findings}
\label{sec:experiment}
\begin{figure*}[h]
\label{fig:webapp}
\label{app}
\centering
\includegraphics[width=0.85\textwidth]{webapp2}
\caption{Screenshot of sample subjective fairness elicitation question posed to human subjects.}
\end{figure*}
The framework and algorithm we have provided
can be viewed as a potentially powerful tool for empirically studying
subjective individual fairness as a {\em behavioral\/} phenomenon. In this section
we describe preliminary results from a human-subject study we performed in which subjective
fairness was elicited and then enforced by our algorithm.
Our study used the COMPAS recidivism data gathered by ProPublica
\footnote{
The data can be accessed on ProPublica's Github
page \href{https://github.com/propublica/compas-analysis/blob/master/compas-scores-two-years.csv}{here}.
We cleaned the data as in the ProPublica study, removing any records with
missing data. This left 5829 records, where the base rate of two-year recidivism was $46\%$.
}
in their celebrated analysis of
Northepointe's risk assessment algorithm \cite{propublica1}. This data consists of defendants
from Broward County in Florida between 2013 to 2014.
For each defendant the data consists of
sex (male, female), age (18-96), race (African-American, Caucasian, Hispanic, Asian, Native American),
juvenile felony count, juvenile misdemeanor count, number of other juvenile offenses,
number of prior adult criminal offenses, the severity of the crime for which they were incarcerated (felony or misdemeanor),
as well as the outcome of whether or not they did in fact recidivate.
Recidivism is defined as a new arrest within 2 years, not counting traffic violations
and municipal ordinance violations.
We implemented our fairness framework via a web app that
elicited subjective fairness notions from 43 undergraduates at a major research university.
After reading a document describing the data and recidivism prediction task,
each subject was presented with 50 randomly chosen pairs of records from the COMPAS data set,
as illustrated in Figure~\ref{fig:webapp}, and asked whether in their opinion the two individuals
should treated (predicted) equally or not. Importantly, the subjects were shown only the features for
the individuals, and not their actual recidivism outcomes, since we sought to elicit subjects' fairness notions
regarding the predictions of those outcomes. While absolutely no guidance was given to subjects regarding fairness,
the elicitation framework allows for rich possibilities. For example, subjects could choose to ignore demographic
factors or criminal histories entirely if they liked, or a subject who believes that minorities are more vulnerable
to overpolicing could discount their criminal histories relative to Caucasians in their pairwise elicitations.
For each subject, the pairs they identified to be treated equally were taken as constraints on
error minimization with respect to the actual recidivism outcomes over the entire COMPAS dataset, and our algorithm
was applied to solve this constrained optimization problem,
using a linear threhsold heuristic
as the underlying learning oracle
(\cite{KNRW18}).
We ran our algorithm with
$\eta = 0$ and variable $\gamma$ in Equations (\ref{min}) through (\ref{sum}), which represents the strongest
enforcement of subjective fairness --- the difference in predicted values must be at most $\gamma$ on {\em every\/} pair
selected by a subject. Because the issues we are most interested in here (convergence, tradeoffs with accuracy, and heterogeneity of fairness preferences) are orthogonal to generalization --- and because we prove VC-dimension based generalization theorems --- for simplicity, the results we report are in-sample.
\subsection{Results}
\begin{figure*}[h]
\label{fig:exp}
\centering
\includegraphics[width=.3\textwidth]{Trajectory.png}
\includegraphics[width=.3\textwidth]{ParetoVariety.png}
\includegraphics[width=.3\textwidth]{NumConstraintsErrorScatter.png}
\caption{
(a) Sample algorithm trajectory for a particular subject at $\gamma = 0.3$.
(b) Sample subjective fairness Pareto curves for a sample of subjects.
(c) Scatterplot of number of constraints specified vs. error at $\gamma = 0.3$.
}
\end{figure*}
Since our algorithm relies on a learning heuristic for which worst-case guarantees are not possible,
the first empirical question is whether the algorithm converges rapidly on the behavioral data.
We found that it did so consistently; a typical example is Figure~\ref{fig:exp}(a), where
we show the trajectory of model error vs. fairness violation for a particular subject's data in which
the allowed violation was $\gamma = 0.3$ (horizontal line). At 2000 iterations the algorithm has saturated the
allowed violation with the constrained error-optimal model.
Perhaps the most basic behavioral questions we might ask
involve the extent and nature of subject variability. For example,
do some subjects identify constraint pairs that are much harder to satifsy
than other subjects? And if so, what factors seem to account for such variation?
Figure~\ref{fig:exp}(b) shows that there is indeed considerable variation in subject difficulty.
For a representative subset of the 43 subjects, we have plotted
the error vs. fairness violation Pareto curves obtained
by varying $\gamma$ from 0 (pairs selected by subjects must have
identical probabilistic predictions of recidivism) to 1.0 (no fairness enforced whatsoever).
Since our model space is closed under probabilistic mixtures, the worst-case Pareto curve
is linear, obtained by all mixtures of the error-optimal model and random
predictions. Easier constaint sets are more convex.
We see in the figure that both extremes are exhibited behaviorally --- some subjects yield
linear or near-linear curves, while others permit huge reductions in unfairness for only
slight increases in error, and virtually all the possibilities in between are realized as well.
\footnote{The slight deviations from true convexity are due to only approximate convergence.}
Since each subject was presented with 50 random pairs and was free to constrain as many or
as few as they wished, it is natural to wonder if the variation in difficulty is explained
simply by the number of constraints chosen. In Figure~\ref{fig:exp}(c) we show a scatterplot
of the the number of constraints selected by a subject ($x$ axis)
versus the error obtained ($y$ axis) for $\gamma = 0.3$ (an intermediate value that exhibits considerable
variation in subject error rates) for all 43 subjects. While we see there is
indeed strong correlation (approximately 0.69), it is far from the case that the number of
constraints explains all the variability. For example, amongst subjects who selected approximately 16 constraints, the resulting error varies over a range of nearly 8\%, which is over 40\% of the range from the optimal error (0.32) to the worst fairness-constrained error (0.5).
It is also interesting to consider the collective force of the 1432 constraints selected by all
43 subjects together, which we can view as a ``fairness panel'' of sorts. Given that there are already
individual subjects whose constraints yield the worst-case Pareto curve, it is unsurprising that
the collective constraints do as well. But we can exploit the flexibility of our optimization
framework in
Equations (\ref{min}) through constraint (\ref{sum}), and let
$\gamma = 0.0$ and vary only $\eta$, thus giving the learner
discretion in which subjects' constraints to discount or discard at a given budget
$\eta$. In doing so
we find that the unconstrained optimal error can be obtained while having the average (exact) pairwise constraint be violated by only roughly 25\%, meaning roughly that only 25\% of the collective constraints account for all the difficulty.
We leave the fuller investigation of our behavioral study for future work,
including the detailed nature of subject variability and
the comparison of behavioral subjective fairness to more standard algorithmic fairness notions.
\section{Omitted details in Section~\ref{sec:erm}}
\input{dual-best-response}
\subsection{Dual player's best response}
\begin{lemma}\label{lem:dual_separate}
For fixed $D$ and $\alpha$, the best response optimization for the dual player is separable, i.e.
\[\argmax_{\lambda \in \Lambda, \tau \in \Tau} \Lagr(D, \alpha, \lambda, \tau) = \argmax_{\lambda \in \Lambda} \Lagr_{D, \alpha}^{\psi_1}(\lambda) \times \argmax_{\tau \in \Tau} \Lagr_{D, \alpha}^{\psi_2}(\tau), \]
where
\[\Lagr_{D, \alpha}^{\psi_1}(\lambda) = \sum_{(i,j) \in [n]^2} \lambda_{ij} \left( \E_{h\sim D}\left[h(x_i) - h(x_j)\right] - \alpha_{ij} -\gamma \right)\]
and
\[\Lagr_{D, \alpha}^{\psi_2}(\tau)=\tau \left( \frac{1}{|A|} \sum_{(i, j)\in [n]^2} w_{ij} \alpha_{ij} - \eta \right).\]
\end{lemma}
\begin{proof}
\begin{align*}
& \argmax_{\lambda \in \Lambda, \tau \in \Tau} \Lagr(D, \alpha, \lambda, \tau)\\
&=\argmax_{\lambda \in \Lambda, \tau \in \Tau} \E_{h\sim D}\left[\err(h, S)\right] + \sum_{(i,j) \in [n]^2} \lambda_{ij} \left( \E_{h\sim D}\left[h(x_i) - h(x_j)\right] - \alpha_{ij} -\gamma \right) +
\tau \left( \frac{1}{|A|} \sum_{(i, j)\in [n]^2} w_{ij} \alpha_{ij} - \eta \right)\\
&=\argmax_{\lambda \in \Lambda} \sum_{(i,j) \in [n]^2} \lambda_{ij} \left( \E_{h\sim D}\left[h(x_i) - h(x_j)\right] - \alpha_{ij} -\gamma \right) \times \argmax_{\tau \in \Tau}
\tau \left( \frac{1}{|A|} \sum_{(i, j)\in [n]^2} w_{ij} \alpha_{ij} - \eta \right)\\
&= \argmax_{\lambda \in \Lambda} \Lagr_{D, \alpha}^{\psi_1}(\lambda) \times \argmax_{\tau \in \Tau} \Lagr_{D, \alpha}^{\psi_2}(\tau)
\end{align*}
\end{proof}
\begin{algorithm}
\caption{Best Response, $BEST_{\psi}(D, \alpha)$, for the dual player}
\begin{algorithmic}[ht!]
\State \textbf{Input:} training examples $S=\{ x_i, y_i \}_{i=1}^n,$ $D \in \Delta(H)$, $\alpha \in [0,1]^{n^2}$
\State $\lambda = 0 \in \mathbb{R}^{n^2}$
\State $(i^*, j^*) = \argmax_{(i,j) \in [n]^2} \E_{h\sim D}\left[h(x_i) - h(x_j)\right] -\alpha_{ij}-\gamma$
\If {$\E_{h\sim D}\left[h(x_{i^*}) - h(x_{j^*})\right] -\alpha_{i^*j^*}-\gamma \le 0$}
\State $\lambda_{i^*j^*} = C_{\lambda}$
\EndIf
\State set $\tau = \begin{cases}
0 & \frac{1}{|A|}\sum_{(i, j)\in [n]^2} w_{ij} \alpha_{ij} - \eta \le 0\\
C_{\tau} & o.w.
\end{cases}$
\State \textbf{Output:} $\lambda, \tau$
\end{algorithmic}
\end{algorithm}
\begin{lemma}\label{lem:dual_lambda}
For fixed $D$ and $\alpha$, the output $\lambda$ from $BEST_{\psi}(D, \alpha)$ minimizes $\Lagr_{D, \alpha}^{\psi_1}$
\end{lemma}
\begin{proof}
Because $\Lagr_{D, \alpha}^{\psi_1}$ is linear in terms of $\lambda$ and the feasible region is the non-negative orthant bounded by 1-norm, the optimal solution must include putting all the weight to the pair $(i,j)$ where $\E_{h\sim D}\left[h(x_i) - h(x_j)-\alpha_{ij}\right]$ is maximized.
\end{proof}
\begin{lemma}\label{lem:dual_tau}
For fixed $D$ and $\alpha$, the output $\tau$ from $BEST_{\psi}(D, \alpha)$ minimizes $\Lagr_{D, \alpha}^{\psi_2}$
\end{lemma}
\begin{proof}
Because $\Lagr_{D, \alpha}^{\psi_2}$ is linear in terms of $\tau$, the optimal solution is trivially to set $\tau$ at either $C_{\tau}$ or 0 depending on the sign.
\end{proof}
\begin{thebibliography}{18}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Agarwal et~al.(2018)Agarwal, Beygelzimer, Dudik, Langford, and
Wallach]{reductions}
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna
Wallach.
\newblock A reductions approach to fair classification.
\newblock In \emph{International Conference on Machine Learning}, pages 60--69,
2018.
\bibitem[Dwork et~al.(2012)Dwork, Hardt, Pitassi, Reingold, and
Zemel]{awareness}
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel.
\newblock Fairness through awareness.
\newblock In \emph{Proceedings of the 3rd innovations in theoretical computer
science conference}, pages 214--226. ACM, 2012.
\bibitem[Feldman et~al.(2009)Feldman, Gopalan, Khot, and Ponnuswami]{agnostic2}
Vitaly Feldman, Parikshit Gopalan, Subhash Khot, and Ashok~Kumar Ponnuswami.
\newblock On agnostic learning of parities, monomials, and halfspaces.
\newblock \emph{SIAM Journal on Computing}, 39\penalty0 (2):\penalty0 606--645,
2009.
\bibitem[Feldman et~al.(2012)Feldman, Guruswami, Raghavendra, and
Wu]{agnostic1}
Vitaly Feldman, Venkatesan Guruswami, Prasad Raghavendra, and Yi~Wu.
\newblock Agnostic learning of monomials by halfspaces is hard.
\newblock \emph{SIAM Journal on Computing}, 41\penalty0 (6):\penalty0
1558--1590, 2012.
\bibitem[Freund and Schapire(1996)]{freund1996game}
Yoav Freund and Robert~E Schapire.
\newblock Game theory, on-line prediction and boosting.
\newblock In \emph{COLT}, volume~96, pages 325--332. Citeseer, 1996.
\bibitem[Gillen et~al.(2018)Gillen, Jung, Kearns, and Roth]{unknown}
Stephen Gillen, Christopher Jung, Michael Kearns, and Aaron Roth.
\newblock Online learning with an unknown fairness metric.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
2600--2609, 2018.
\bibitem[Ilvento(2019)]{I2019}
C~Ilvento.
\newblock Metric learning for individual fairness.
\newblock Manuscript submitted for publication, 2019.
\bibitem[Kearns et~al.(2018)Kearns, Neel, Roth, and Wu]{KNRW18}
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei~Steven Wu.
\newblock Preventing fairness gerrymandering: Auditing and learning for
subgroup fairness.
\newblock In \emph{International Conference on Machine Learning}, pages
2569--2577, 2018.
\bibitem[Kearns and Vazirani(1994)]{kearns1994introduction}
Michael~J Kearns and Umesh~Virkumar Vazirani.
\newblock \emph{An introduction to computational learning theory}.
\newblock MIT press, 1994.
\bibitem[Kim et~al.(2018)Kim, Reingold, and Rothblum]{compawareness}
Michael Kim, Omer Reingold, and Guy Rothblum.
\newblock Fairness through computationally-bounded awareness.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
4842--4852, 2018.
\bibitem[Kivinen and Warmuth(1997)]{kivinenwarmuth1997exponentiated}
Jyrki Kivinen and Manfred~K. Warmuth.
\newblock Exponentiated gradient versus gradient descent for linear predictors.
\newblock \emph{Information and Computation}, 132:\penalty0 1--63, 1997.
\bibitem[Larson et~al.(2019)Larson, Angwin, Kirchner, and Mattu]{propublica1}
Jeff Larson, Julia Angwin, Lauren Kirchner, and Surya Mattu.
\newblock How we analyzed the compas recidivism algorithm, Mar 2019.
\newblock URL
\url{https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm}.
\bibitem[Neel et~al.(2018)Neel, Roth, and Wu]{neel2018use}
Seth Neel, Aaron Roth, and Zhiwei~Steven Wu.
\newblock How to use heuristics for differential privacy.
\newblock \emph{arXiv preprint arXiv:1811.07765}, 2018.
\bibitem[Pemantle and Peres(2014)]{pemantle2014concentration}
Robin Pemantle and Yuval Peres.
\newblock Concentration of lipschitz functionals of determinantal and other
strong rayleigh measures.
\newblock \emph{Combinatorics, Probability and Computing}, 23\penalty0
(1):\penalty0 140--160, 2014.
\bibitem[Rothblum and Yona(2018)]{RY18}
Guy~N Rothblum and Gal Yona.
\newblock Probably approximately metric-fair learning.
\newblock \emph{arXiv preprint arXiv:1803.03242}, 2018.
\bibitem[Sion et~al.(1958)]{sion1958general}
Maurice Sion et~al.
\newblock On general minimax theorems.
\newblock \emph{Pacific Journal of mathematics}, 8\penalty0 (1):\penalty0
171--176, 1958.
\bibitem[Zemel et~al.(2013)Zemel, Wu, Swersky, Pitassi, and Dwork]{zemel}
Rich Zemel, Yu~Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork.
\newblock Learning fair representations.
\newblock In \emph{International Conference on Machine Learning}, pages
325--333, 2013.
\bibitem[Zinkevich(2003)]{zinkevich2003online}
Martin Zinkevich.
\newblock Online convex programming and generalized infinitesimal gradient
ascent.
\newblock In \emph{Proceedings of the Twentieth International Conference on
Machine Learning (ICML-2003)}, 2003.
\end{thebibliography}