Fairness With Minimal Harm: A Pareto-Optimal Approach For Healthcare
Extracted source text ยท arXiv 1911.06935
\documentclass{article}
% if you need to pass options to natbib, use, e.g.,
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2019
% ready for submission
% \usepackage{neurips_2019}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
% \usepackage[preprint]{neurips_2019}
% to compile a camera-ready version, add the [final] option, e.g.,
\usepackage[final]{neurips_2019}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2019}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{graphicx,subcaption}
\usepackage{amsthm}
\usepackage{placeins}
\input{math_commands.tex}
\usepackage[ruled,vlined]{algorithm2e}
\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]
\newtheorem{theorem}{Theorem}[section]
\theoremstyle{remark}
\newtheorem*{remark}{Remark}
\newtheorem{lemma}[theorem]{Lemma}
\usepackage[ruled,vlined]{algorithm2e}
\SetKwInput{KwInit}{Init}
\SetKwInput{KwGiven}{Given}
\SetKwInput{KwReturn}{Return}
\SetKwComment{Comment}{start}{end}
\usepackage{booktabs}
\usepackage{multirow}
\newcommand{\cmmnt}[1]{\ignorespaces}
\usepackage{mdwlist}
\linespread{.975}
% figure space
\addtolength{\belowcaptionskip}{-1mm}
\addtolength{\abovecaptionskip}{-1mm}
\addtolength{\dblfloatsep}{-1mm}
\addtolength{\dbltextfloatsep}{-1mm}
\addtolength{\intextsep}{-1mm}
% math space
\addtolength{\abovedisplayskip}{-1.2mm}
\addtolength{\belowdisplayskip}{-1.2mm}
%\addtolength{\arraycolsep}{-1.2mm}
%list space
\addtolength{\topsep}{-1mm}
\addtolength{\partopsep}{-1mm}
\addtolength{\itemsep}{-1mm}
\setlength{\tabcolsep}{4pt}
\usepackage{authblk}
\newcommand*{\affaddr}[1]{#1} % No op here. Customize it for different styles.
\newcommand*{\affmark}[1][*]{\textsuperscript{#1}}
\newcommand*{\email}[1]{\texttt{#1}}
\newcommand\blfootnote[1]{%
\begingroup
\renewcommand\thefootnote{}\footnote{#1}%
\addtocounter{footnote}{-1}%
\endgroup
}
\usepackage{hyperref}
\usepackage{url}
\title{Fairness With Minimal Harm:\\ A Pareto-Optimal Approach For Healthcare}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
% Natalia Martinez
% \author{%
% Natalia Martinez$^\dagger$ \thanks{$\dagger$ Equal contributions} \\
% % E.C.E. Department\\
% Duke University\\
% % Durham, N.C., U.S.A \\
% \texttt{natalia.martinez@duke.edu} \\
% % examples of more authors
% \And
% Martin Bertran$^\dagger$ \\
% % E.C.E. Department\\
% Duke University\\
% % Durham, N.C., U.S.A \\
% \texttt{martin.bertran@duke.edu} \\
% \And
% Guillermo Sapiro \\
% % E.C.E. Department\\
% Duke University\\
% % Durham, N.C., U.S.A \\
% \texttt{guillermo.sapiro@duke.edu} \\
% % \AND
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% % \And
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% % \And
% % Coauthor \\
% % Affiliation \\
% % Address \\
% % \texttt{email} \\
% }
\author{%
Natalia Martinez$^\dagger$, Martin Bertran$^\dagger$, Guillermo Sapiro\\
\affaddr{Department of Electrical and Computer Engineering}\\
\affaddr{Duke University}\\
\email{\{natalia.martinez,martin.bertran,guillermo.sapiro\}@duke.edu}\\
}
\usepackage{fancyhdr}
\fancyhf{}
\rfoot{Machine Learning for Health (ML4H) at NeurIPS 2019 - Extended Abstract}
\begin{document}
\maketitle
\begin{abstract}
% Common fairness definitions in machine learning focus on balancing notions of fairness and utility. In this work, we study fairness in the context of risk disparity among sub-populations. We focus on Pareto-optimal fairness, where the goal of reducing risk disparity gaps is secondary only to the principle of not doing unnecessary harm. This is particularly relevant to high-stakes domains such as healthcare, where non-maleficence is a core principle. We provide analysis, based on recent works in fairness, to exemplify scenarios were perfect fairness might not be feasible without doing harm. We present a methodology for training neural networks by dynamically re-balancing subgroups risks to encourage minimal risk disparity while avoiding unnecessary performance degradation. We argue that even in domains where fairness at cost is required, no-harm fairness can be seen as the optimal initial step. Additionally, this same methodology can also be applied to any unbalanced classification task, where we want to dynamically equalize the misclassification risks across outcomes without degrading overall performance any more than strictly necessary. Finally, we demonstrate this methodology on real case-studies of predicting ICU patient mortality, and classifying skin lesions from dermatoscopic images.
Common\blfootnote{$\dagger$ Equal contributions} fairness definitions in machine learning focus on balancing notions of disparity and utility. In this work, we study fairness in the context of risk disparity among sub-populations. We are interested in learning models that minimize performance discrepancies across sensitive groups without causing unnecessary harm. This is relevant to high-stakes domains such as healthcare, where non-maleficence is a core principle. We formalize this objective using Pareto frontiers, and provide analysis, based on recent works in fairness, to exemplify scenarios were perfect fairness might not be feasible without doing unnecessary harm. We present a methodology for training neural networks that achieve our goal by dynamically re-balancing subgroups risks. We argue that even in domains where fairness at cost is required, finding a non-unnecessary-harm fairness model is the optimal initial step. We demonstrate this methodology on real case-studies of predicting ICU patient mortality, and classifying skin lesions from dermatoscopic images.
\end{abstract}
\section{Introduction}
Machine learning algorithms play an important role in decision making in society. When these algorithms are used to make high-impact healthcare decisions such as deciding whether or not to classify a skin lesion as benign or not, or predicting mortality for intensive care unit patients, it is paramount to guarantee that these decisions are accurate and unbiased with respect to sensitive attributes such as gender or ethnicity. A model that is trained naively may not have these properties by default (\cite{barocas2016big}). %A closely related problem, that is also of interest in health, is the unbalanced classification task, where one would like to minimize the performance gap for the different classes without degrading the overall performance any more than strictly necessary.
It is desirable in these critical applications to impose some fairness criteria. There are several lines of work in fairness in machine learning, such as Demographic Parity (\cite{louizos2015variational,zemel2013learning,feldman2015certifying}), Equality of Odds, Equality of Opportunity (\cite{hardt2016equality, woodworth2017learning}), or Calibration (\cite{pleiss2017fairness}). These notions of fairness are appropriate in many scenarios, but in domains where quality of service is paramount, such as healthcare, we argue that it is necessary to strive for models that are as close to fair as possible without introducing unnecessary harm to any subgroup (\cite{ustun2019fairness}). %Even if the overall fairness goal is a potentially harmful, zero-gap classifier, pursuing first a no-harm fair classifier and later applying other harmful methodologies ensures that all possible non-harmful trade-offs are covered before explicitly degrading performance, as explored by \cite{woodworth2017learning}.
In this work, we measure discrimination (unfairness) in terms of difference in predictive risks across sub-populations defined by our sensitive attributes. This concept has been explored in other works such as \cite{calders2010three,dwork2012fairness,feldman2015certifying,chen2018my,ustun2019fairness}. We examine the subset of models from our hypothesis class that have the best trade-offs between sub-population risks, and select from this set the one with the smallest risk disparity gap. \cmmnt{The goal is to select an algorithm from our hypothesis class that has the smallest risk disparity gap between subgroups, while at the same time being as close to optimal performance in every subgroup as possible.} This is in contrast to common post-hoc correction methods like the ones proposed in \cite{hardt2016equality,woodworth2017learning} where randomness is potentially added to the decisions of all sub-populations. While this type of approach diminishes the accuracy-disparity gap, it does so by potentially introducing randomness into the final decision (i.e., with some probability, disregard classifier output and produce arbitrary decision based solely on sensitive label), which leads to performance degradation in most common risk metrics. Since our proposed methodology does not require test-time access to sensitive attributes, and can be applied to any standard classification or regression task, it can also be used to reduce risk disparity between outcomes, acting as an adaptive risk equalization loss compatible with unbalanced classification scenarios.
\paragraph{Main Contributions} We formalize the notion of no-unnecessary-harm fairness using Pareto optimality, a state of resource allocations from which it is impossible to reallocate without making one subgroup worse. We show that finding a Pareto-fair classifier is equivalent to finding a model in our hypothesis class that is both Pareto optimal with respect to the sub-population risks (no unnecessary harm) and minimizes risk disparity. This notion is already amenable to non-binary sensitive attributes. We analyze Pareto fairness on an illustrative example, and compare it to alternative approaches. We provide an algorithm that promotes fair solutions belonging to the Pareto front; this algorithm can be applied to any standard classifier or regression task. Finally, we show how our methodology performs on real tasks such as predicting ICU mortality rates in the MIMIC-III dataset from hospital notes \cite{johnson2016mimic}, and classifying skin lesions in the HAM10000 dataset \cite{tschandl2018ham10000}.
\section{Problem Statement}
Consider we have access to a dataset $\mathcal{D} = \{(x_i,y_i,a_i)\}_{i=1}^n$ containing $n$ independent triplet samples drawn from a joint distribution $(x_i,y_i,a_i) \sim P(X,Y,A)$ where $x_i \in \mathcal{X}$ are our input features (e.g., images, tabular data, etc.), $y_i \in \mathcal{Y}$ is our target variable, and $a_i \in \mathcal{A}$ indicates group membership or sensitive status (e.g., ethnicity, gender); our input features $X$ may or may not explicitly contain $A$.
Let $h \in \mathcal{H}$ be a classifier from our hypothesis class $\mathcal{H}$ trained to infer $y$ from $x$, $h:\mathcal{X} \rightarrow \mathcal{Y}$; and a loss function $\ell: \mathcal{Y}\times \mathcal{Y} \rightarrow \mathbb{R}$. We define the class-specific risk of classifier $h$ on subgroup $a$ as $R_a(h) = \E_{X,Y|A=a}[\ell(Y,h(X))]$. The risk discrimination gap between two subgroups $a,a' \in \mathcal{A}$ is measured as $\Gamma_{a,a'}(h) = |R_a(h)- R_{a'}(h)|$, and we define the pairwise discrimination gap as $\Vec{\Gamma}_\mathcal{A}(h) = \{\Gamma_{a,a'}(h)\}_{a,a' \in \mathcal{A}}$. Our goal is to obtain a classifier $h \in \mathcal{H}$ that minimizes this gap without causing unnecessary harm to any particular group. To formalize this notion, we define:
%\begin{definition}{Dominant classifier:}
%Classifier $h'$ is said to dominate $h$ if $R_a(h) \ge R_a(h'), \forall a \in \mathcal{A}$ and $\exists a' \in \mathcal{A}: R_{a'}(h) > R_{a'}(h')$.
%\label{def:dominance}
%\end{definition}
\begin{definition}{Pareto front:}
The set of Pareto front classifiers is defined as $\mathcal{P}(\mathcal{H},\mathcal{A}) = \{ h \in \mathcal{H}: \not\exists h' \in \mathcal{H}, \not\exists a' \in \mathcal{A}: [R_{a'}(h') < R_{a'}(h)] \land [R_a(h') \le R_a(h)], \forall a \in \mathcal{A} \} $
\label{def:paretofront}
\end{definition}
\begin{definition}{Pareto-fair classifier and Pareto-fair vector:}
A classifier $h^*$ is an optimal no-harm classifier if it minimizes the discrimination gap among all Pareto front classifiers, $h^* = \argmin\limits_{h \in \mathcal{P}(\mathcal{H},\mathcal{A})} || \Vec{\Gamma}_\mathcal{A}(h) ||_{\infty} $. The Pareto-fair vector is defined as $\bm{r}^* \in \mathbb{R}^{|\mathcal{A}|} :$ $\bm{r}^* = \{R_a(h^*)\}_{a\in \mathcal{A}}$.
\label{def:noharm}
\end{definition}
The Pareto front defines the best achievable trade-offs between population risks $R_a(h)$, while the Pareto-fair classifier gives the trade-off with least disparity. Building on analysis done by \cite{chen2018my,domingos2000unified}, it is possible to decompose the risk in bias, variance and noise for some given loss functions. The noise represents the smallest achievable risk for infinitely large datasets. If it differs between sensitive groups, zero-discrimination (perfect fairness) can be only achieved by introducing bias or variance, hence doing harm. Figure \ref{fig:Gaussians} shows a scenario where the Pareto-front does not intersect the equality of risk line for the case of two sensitive groups $a \in \{0,1\}$ and a binary output variable $y \in \{0,1\}$. Here the level of noise between subgroups differs, and the Pareto-fair vector $\bm{r}^*$ would not be achieved by either a naive classifier (minimizes expected global risk), or a classifier where low-occurrence subgroups are over-sampled (re-balanced naive classifier). %\textbf{We also observe that the amount of performance degradation required to enforce perfect fairness starting from the naive classifier is significantly higher than when starting from the Pareto-fair point.}
\begin{figure}[ht]
\centering
% include first image
\includegraphics[width=.9\linewidth]{Picture1.png}
\caption{Example of binary target ($y$) and sensitive ($a$) variable. Bottom left figure shows conditional distributions of observation variable ($x$) $p(x|a)$, while top left shows distribution of $y$ as a function of $x,a$, $p(y|x,a)$, these distributions are simple piece-wise constants with two levels $\rho^{low}$ and $\rho^{high}$, $\Delta$ represents the gap between level transitions for each group. Middle image shows the Pareto front between group risks, since noise levels are not the same across subgroups, perfect fairness is not attainable. Equality of Risk requires pure degradation of service for group $a=1$. Both Naive and Re-balanced Naive classifiers do not attain the best possible no-harm classifier. Pareto-fair point is shown in green. Rightmost figure shows the trade-offs attainable between discrimination and mean risks, an equivalent problem to the risk trade-off figure.}
\label{fig:Gaussians}
\end{figure}
\section{Methods}
Any loss function that is monotonically decreasing with $R_{a}(h), \forall a$ is minimized by a classifier in the Pareto front. Since we want to minimize the discrimination gap, we will build an adaptive loss function that is shares this property with the following form:
\begin{equation}
\begin{array}{cc}
\phi(h;\bm{\mu},c) &= \sum\limits_{a\in\mathcal{A}} R_{a}(h) +\mu_a (R_{a}(h)-c)^{+2},
\end{array}
\label{eq:adaptiveloss}
\end{equation}
with $c < \min_{a}R_{a}(h)$ and $\bm{\mu} = \{ \mu_a\}_{a\in \mathcal{A}}$. It can be shown that for convex Pareto sets (with respect to risk vectors $\bm{r} = \{R_{a}(h)\}_{a \in \mathcal{A}}$), there exist a set of $\bm{\mu}^*$ such that the Pareto-fair vector $\bm{r}^*$ is a unique solution. In Algorithm \ref{algo:MainAlgo} we jointly search for $\bm{\mu}^*$ ,$\bm{r}^*$, and $h^*$.
%If we had replaced $h_{\theta^{\text{target}}}$ by $h_{\theta}$, this loss would be identical to the penalty method objective for problem $\argmin\limits_{h \in \mathcal{H}} \phi(\{R_{a}(h)\}_{a \in \mathcal{A}}) \; s.t.: || \Vec{\Gamma}_\mathcal{A}(h) ||_{\infty} \le \eps$ where we dynamically adjust $\bm{\mu} = \{ \mu_a\}_{a\in \mathcal{A}}$ to minimize $|| \Vec{\Gamma}_\mathcal{A}(h) ||_{\infty}$. However, note that $\theta^{\text{target}}$ is a delayed copy of the classifier parameters, and do not directly propagate gradients with respect to $\theta$, which makes $\phi(h_{\theta};\vec{\mu},h_{\theta^{\text{target}}})$ a properly monotonic function of $R_a(h)$ at every iteration. An overview of the algorithm is given in Algorithm \ref{algo:MainAlgo}
\begin{algorithm}[H]
\DontPrintSemicolon
\SetNoFillComment
\footnotesize
\SetAlgoLined
\KwGiven{$h_\theta, \mathcal{D}, \gamma>0, \xi\in(0,1), \zeta\in(0,1), \text{lr}, $}
$\bm{\mu} \leftarrow \bm{0},\; \bm{\mu}^* \leftarrow \bm{0},\;,c\leftarrow 0,\; h^* \leftarrow h_\theta $\;
\While{ Improving}{
$h_\theta \leftarrow \text{SGDwithEarlyStopping}(h_\theta,\phi, \bm{\mu},c, \mathcal{D}) $\tcp*[l]{Optimize adaptive loss with fixed $\bm{\mu}$ on dataset $\mathcal{D}$ }
$r^{\text{val}} \leftarrow \text{EvaluateRisk}(h_\theta, \mathcal{D}^{\text{val}})$\;
\uIf{$ || \Vec{\Gamma}_\mathcal{A}(h) ||_{\infty} < \Gamma^*$ and $\bm{r}^{\text{Val}}$ is not dominated by previous validation risks }{
$h^* \leftarrow h_\theta,\; \Gamma^* \leftarrow || \Vec{\Gamma}_\mathcal{A}(h) ||_{\infty},\; c_{\text{old}} \leftarrow c,\; c \leftarrow \min_a \frac{r_{a}^{\text{Val}}}{k}$\;
$ \bm{\mu}^* \leftarrow \bm{\mu}\cdot \frac{(\bm{r}^{\text{Val}}-c_{\text{old}})^+}{(\bm{r}^{\text{Val}}-c)^+},\; a' \leftarrow \argmax_a r_{a}^{\text{Val}}$}
\Else{lr $\leftarrow$ $\zeta$ lr, $\bm{\mu} \leftarrow \bm{\mu}^*,\; \gamma \leftarrow \gamma \xi,\; h_\theta \leftarrow h^*$}
$\mu_{a'}\leftarrow (1+\gamma) \mu_{a'}$
}
\KwReturn{$h^*$ \tcp*[1]{Exit loop due to excessive iterations or no improvement in fairness}}
\caption{ParetoFairOptimization}
\label{algo:MainAlgo}
\end{algorithm}
% \begin{algorithm}[H]
% \DontPrintSemicolon
% \SetNoFillComment
% \footnotesize
% \SetAlgoLined
% \KwGiven{$h_\theta, \mathcal{D},n_\mu, \gamma>1, \text{lr}, \text{B}$}
% $\vec{\mu} \leftarrow \vec{0}, \; \mu_{\text{count}}\leftarrow 0, \; \eps^*\leftarrow \infty,\; h^* \leftarrow h_\theta $\;
% \While{$ \mu_{\text{count}} \le n_\mu$}{
% $\mu_{\text{count}} \leftarrow \mu_{\text{count}} +1 $ \;
% $h_\theta \leftarrow \text{SGDwithEarlyStopping}(h_\theta,\phi, \vec{\mu}, \mathcal{D}) $\tcp*[l]{Optimize adaptive loss with fixed $\vec{\mu}$ on dataset $\mathcal{D}$ }
% \tcp{Check that solution is no-harm and reduces fairness gap}
% \If{$ || \Vec{\Gamma}_\mathcal{A}(h) ||_{\infty} <= \eps^*$ and $\vec{\bar{R}}^{\text{val}}(h)$ is not dominated by previous validation risks }{
% $\mu_{\text{count}}\leftarrow 0, \;h^* \leftarrow h_\theta , \;\eps^* \leftarrow || \Vec{\Gamma}_\mathcal{A}(h) ||_{\infty} $\;}
% \tcp{Update $\mu$ values}
% \If{$\mu_{\text{count}} \le n_\mu$}{
% \For{$ a \neq argmin_{a'}\bar{R}_{a'}^{\text{val}}(h) $}{
% $\mu_a \leftarrow \max(\gamma \mu_a, \frac{\bar{R}_{a}^{\text{val}}(h)}{\eps^*})$\;
% }
% }
% }
% \tcp{Exit loop due to excessive iterations or no improvement in fairness gap}
% \KwReturn{$h^*$}
% \caption{ParetoFairOptimization}
% \label{algo:MainAlgo}
% \end{algorithm}
% The simpler monotonic function $\phi(\{R_{a}(h)\}) = \sum\limits_{a\in \mathcal{A}} w_a R_a(h)$ with adaptive weights $w_a$ is known as the weighted-sum method, this function was discarded due to the fact that not all Pareto-optimal points are reachable with it (for non-convex Pareto fronts), and because small changes in weights $w_a$ can translate into large changes in the final solution. An in-depth discussion of the disadvantages of the weighted-sum method can be found in \cite{geoffrion1968proper,miettinen2012nonlinear, miettinen2008introduction}.
\section{Experiments and Results}
We evaluate our method on mortality prediction and skin lesion classification and show empirically how we reduce accuracy and risk disparity without unnecessary harm. We compare it against a naive classifier, class-rebalancing (equally sampled sub-groups), the post-processing framework presented in \cite{hardt2016equality}, and the Disparate Mistreatment framework of \cite{zafar2017parity}.
\subsection{Predicting Mortality in Intensive Care Patients}
We analyze clinical notes from adult ICU patients at the Beth Israel Deaconess Medical Center (MIMIC-III dataset) \cite{johnson2016mimic} to predict patient mortality. We follow the pre-processing methodology outlined in \cite{chen2018my} and use tf-idf statistics on the $10,000$ most frequent words in clinical notes as input features. Fairness is measured with respect to age (under/over 55 years old), ethnicity, and outcome. We used a fully connected neural network with two 2048-unit hidden layers trained with Brier score (BS) loss. Table \ref{table:MIMICResults} shows accuracy and BS of all tested methodologies. We observe from columns (PF BS) and (PF Acc) that our model has the best accuracy and BS discrepancies. We can reduce accuracy disparities further by applying Hardt post processing (HPF Acc), at the cost of reducing overall performance. Note that, as expected, it is better to apply this post processing on our method than on the rebalanced naive classifier (HPF Acc vs HRen Acc). This illustrates the goal of the proposed Pareto-fair paradigm: develop the fairest algorithm with no unnecessary harm, and if (e.g., due to policy) the resulting fairness level needs to be improved, use other post-processing techniques on the Pareto-fair classifier such as \cite{hardt2016equality}.
\begin{table}[ht]
\scriptsize
\centering
\begin{tabular}{lc|cccc|cc|cccc}
\toprule
Out/Age/Race & Ratio & Na Acc & ReN Acc & Zafar Acc & PF Acc & HReN Acc & HPF Acc & ReN BS & PF BS \\ %& ReN Risk & PF Risk
\midrule
A/A/NW &5.7\%& 99.1$\pm$0.4\% & 86.3$\pm$1.5\% & 93.0$\pm$1.4\% & 83.4$\pm$2.6\% &76.3$\pm$1.9\% &
71.6$\pm$3.2\% & 0.2 $\pm$ 0.02 & 0.25 $\pm$ 0.03
\\
A/A/W &13.3\% & 98.8$\pm$0.5\% & 86.3$\pm$1.1\%& 90.0$\pm$1.3\% & 83.2$\pm$1.5\% & 76.7$\pm$1.6\% &
71.8$\pm$1.9\% & 0.2 $\pm$ 0.01 & 0.25 $\pm$ 0.02 \\
A/S/NW &12.9\% & 97.5$\pm$0.6\% & 76.5$\pm$1.7\%& 81.8$\pm$1.7\% & 71.4$\pm$3.0\% & 76.4$\pm$2.2\% &
71.3$\pm$3.1\% & 0.31 $\pm$ 0.02 & 0.36 $\pm$ 0.03 \\
A/S/W &56.7\% & 97.9$\pm$0.3\% & 79.0$\pm$0.6\% & 77.4$\pm$0.7\%& 74.6$\pm$1.6\% & 76.2$\pm$2.2\% &
72.1$\pm$1.2\% & 0.28 $\pm$ 0.01 & 0.34 $\pm$ 0.02 \\
D/A/NW &0.4\% & 23.4$\pm$9.4\% & 76.1$\pm$8.5\% & 47.7$\pm$9.6\% & 78.6$\pm$6.1\% & 66.6$\pm$9.9\% &
74.1$\pm$9.2\% & 0.36 $\pm$ 0.06 & 0.34 $\pm$ 0.04 \\
D/A/W &0.9\% & 32.6$\pm$3.7\% & 80.1$\pm$3.3\%& 60.5$\pm$6.3\% & 83.3$\pm$4.2\% & 66.4$\pm$2.4\% &
73.6$\pm$4.3\% & 0.29 $\pm$ 0.04 & 0.28 $\pm$ 0.03 \\
D/S/NW &1.8\% & 21.4$\pm$2.2\% & 66.9$\pm$2.4\% & 48.2$\pm$2.0\% & 73.3$\pm$2.9\% & 64.8$\pm$2.1\% &
71.2$\pm$2.9\% & 0.42 $\pm$ 0.02 & 0.37 $\pm$ 0.03 \\
D/S/W &8.3\% & 23.4$\pm$2.2\% & 67.4$\pm$1.9\% & 57.1$\pm$2.2\% & 72.5$\pm$3.6\% &66.2$\pm$2.9\% &
72.4$\pm$3.5\% & 0.42 $\pm$ 0.02 & 0.37 $\pm$ 0.04 \\
\textbf{Sample Mean} & - & \textbf{89.5$\pm$0.2\%} & 78.9$\pm$0.7\% & 78.0$\pm$0.7\% & 75.7$\pm$1.1\% & \textbf{75.1$\pm$1.8\%} &
71.9$\pm$1.2\% & \textbf{0.29 $\pm$ 0.01} & 0.33 $\pm$ 0.01 \\
\textbf{Group Mean} & 12.5\% & 61.8$\pm$1.5\% & 77.3$\pm$1.3\% & 69.4$\pm$1.3\% & \textbf{77.5$\pm$0.7\%} & 71.2$\pm$1.1\% &
\textbf{72.3$\pm$1.0\%} & \textbf{0.31 $\pm$ 0.01} & 0.32 $\pm$ 0.01 \\
\textbf{Discrepancy} & 56.3\% & 81.1$\pm$2.5\% & 22.5$\pm$2.5\% & 49.3$\pm$4.4\% & \textbf{17.1$\pm$2.6\%} &18.6$\pm$3.2\% &
\textbf{13.1$\pm$5.6\%} & 0.24 $\pm$ 0.03 & \textbf{0.16 $\pm$ 0.03} \\
\bottomrule
\end{tabular}
\caption{\footnotesize Result summary table on MIMIC dataset. Sensitive attributes are the combination of Outcome (Alive or Deceased), Age (Adult or Senior) and Race (White or Non-White). Target attribute is Outcome. We report Brier scores (BS) and Accuracies (Acc) for Naive (Na), Rebalanced Naive (ReN), Zafar (Zafar), and Pareto-Fair (PF) classifiers; the prefix (H) indicates that Hardt post-processing was applied to control for Age and Race. Zafar was applied on top of the embeddings of the Naive classifier, as that produced the best results.}
\label{table:MIMICResults}
\end{table}
% \begin{table}
% \caption{Accuracies on test set for MIMIC dataset. Standard deviations are computed across 5 splits.}
% \centering
% %\small
% \scriptsize
% \begin{tabular}{lllllll}
% \toprule
% type & ratio & Naive & NaiveB & ZafarCE & ParetoFair CE & ParetoFair BS & ParetoFair BS H\\
% \midrule
% A/A/NW & 5.7\% & 99.1$\pm$0.4\% & 86.3$\pm$1.5\% & 93.0$\pm$1.4\% & 80.4$\pm$2.8\% & 83.4$\pm$2.6\% & 71.6$\pm$3.2\% \\
% A/A/W & 13.3\% & 98.8$\pm$0.5\% & 86.3$\pm$1.1\% & 90.0$\pm$1.3\% & 81.0$\pm$1.5\% & 83.2$\pm$1.5\% & 71.8$\pm$1.9\% \\
% A/S/NW & 12.9\% & 97.5$\pm$0.6\% & 76.5$\pm$1.7\% & 81.8$\pm$1.7\% & 69.5$\pm$2.2\% & 71.4$\pm$3.0\% & 71.3$\pm$3.1\%\\
% A/S/W & 56.7\% & 97.9$\pm$0.3\% & 79.0$\pm$0.6\% & 77.4$\pm$0.7\% & 73.5$\pm$1.3\% & 74.6$\pm$1.6\% & 72.1$\pm$1.2\% \\
% D/A/NW & 0.4\% & 23.4$\pm$9.4\% & 76.1$\pm$8.5\% & 47.7$\pm$9.6\% & 76.6$\pm$5.3\% & 78.6$\pm$6.1\% & 74.1$\pm$9.2\%\\
% D/A/W & 0.9\% & 32.6$\pm$3.7\% & 80.1$\pm$3.3\% & 60.5$\pm$6.3\% & 85.0$\pm$3.8\% & 83.3$\pm$4.2\% & 73.6$\pm$4.3\%\\
% D/S/NW & 1.8\% & 21.4$\pm$2.2\% & 66.9$\pm$2.4\% & 48.2$\pm$2.0\% & 72.6$\pm$5.1\% & 73.3$\pm$2.9\% \\
% D/S/W & 8.3\% & 23.4$\pm$2.2\% & 67.4$\pm$1.9\% & & 71.2$\pm$2.9\% 57.1$\pm$2.2\% & 71.0$\pm$4.2\% & 72.5$\pm$3.6\% \\
% sample mean & - \textbf{89.5$\pm$0.2\%} & 78.9$\pm$0.7\% & 78.0$\pm$0.7\% & 74.3$\pm$0.8\% & 75.7$\pm$1.1\% \\
% group mean & 12.5\% & 61.8$\pm$1.5\% & 77.3$\pm$1.3\% & 69.4$\pm$1.3\% & 76.2$\pm$0.5\% & \textbf{77.5$\pm$0.7\%} \\
% disc & 56.3\% & 81.1$\pm$2.5\% & 22.5$\pm$2.5\% & 49.3$\pm$4.4\% & 18.5$\pm$4.2\% & \textbf{17.1$\pm$2.6\%} \\
% \bottomrule
% \end{tabular}
% \end{table}
\subsubsection{Skin Lesion Classification}
The HAM10000 dataset \cite{tschandl2018ham10000} collects over $10,000$ dermatoscopic images of skin lesions over a diverse population, lesions are classified in 7 categories. We used a pretrained DenseNet121 \citet{huang2017densely} as our base classifier, and measured fairness with respect to diagnosis class, casting balanced risk minimization as a particular use-case for Pareto fairness. Table \ref{tab:HAMResults} shows our empirical results.
\begin{table}[!htb]
% \caption{Accuracies (left) and confidence (right) on test set for HAM10000 dataset.}
\centering
\scriptsize
\begin{tabular}{lc|ccc|ccc}%{lllllll}
% \caption{Accuracy on MIMIC Dataset}
\toprule
Groups & Ratio & PF Acc & ReN Acc & Na Acc & PF BS & ReN BS & Na BS \\
\midrule
akiec& 2.5\% & 51.9\% & 55.6\% & 3.7\% & 0.741 & 0.671 & 1.289 \\
bcc& 2.7\% & 56.7\% & 76.7\% & 56.7\%& 0.549 & 0.341 & 0.613 \\
bkl& 7.2\% & 59.5\% & 58.2\% & 36.7\% & 0.62 & 0.606 & 0.931 \\
df& 0.5\% & 66.7\% & 33.3\% & 0.0\% & 0.536 & 0.898 & 1.721 \\
nv& 81.0\% & 83.5\% & 90.9\% & 96.0\%& 0.241 & 0.128 & 0.054 \\
vasc& 1.3\% & 71.4\% & 85.7\% & 0.0\%& 0.36 & 0.246 & 1.604 \\
mel& 4.8\% & 53.8\% & 48.1\% & 32.7\%& 0.586 & 0.657 & 0.941 \\
Sample Mean & - & 78.6\% & \textbf{84.8}\% & 83.6\%& 0.308 & \textbf{0.213} & 0.234 \\
Group Mean & 14.3\% & 63.4\% & \textbf{64.1\%} & 32.3\% & 0.519 & \textbf{0.507} & 1.022 \\
Discrepancy & 80.4\% & \textbf{31.7}\% & 57.5\% & 96.0\% & \textbf{0.501} & 0.769 & 1.667 \\
\bottomrule
\end{tabular}
% \caption{Accuracy}
% \begin{subtable}{.35\linewidth}
% \centering
% \begin{tabular}{lllll}
% \toprule
% type & ratio & ParetoFair & NaiveB & Naive \\
% \midrule
% akiec& 2.5\% & 0.741 & 0.671 & 1.289 \\
% bcc& 2.7\% & 0.549 & 0.341 & 0.613 \\
% bkl& 7.2\% & 0.62 & 0.606 & 0.931 \\
% df& 0.5\% & 0.536 & 0.898 & 1.721 \\
% nv& 81.0\% & 0.241 & 0.128 & 0.054 \\
% vasc& 1.3\% & 0.36 & 0.246 & 1.604 \\
% mel& 4.8\% & 0.586 & 0.657 & 0.941 \\
% sample mean & -& 0.308 & \textbf{0.213} & 0.234 \\
% group mean & 14.3\% & 0.519 & \textbf{0.507} & 1.022 \\
% group max & 81.0\% & \textbf{0.741} & 0.898 & 1.721 \\
% group min & 0.5\% & 0.241 & 0.128 & \textbf{0.054} \\
% disc & 80.4\% & \textbf{0.501} & 0.769 & 1.667 \\
% \bottomrule
% \end{tabular}
% \caption{Confidence}
% \end{subtable}
\caption{\footnotesize{} Group-specific risks and accuracies for Naive (Na), Rebalanced Naive (ReN), and Pareto-Fair (PF) classifiers. Lesion Type was chosen as both target and sensitive variable, they are classified as Actinic keratoses and intraepithelial carcinoma (akiec), basal cell carcinoma (bcc), benign keratosis-like lesions (bkl), dermatofibroma (df), melanoma (mel), melanocytic nevi (nv) and vascular lesions (vasc). The Pareto-fair classifier exhibits the lowest discrepancy on Acc and BS while still being Pareto-optimal.}
\label{tab:HAMResults}
\end{table}
% \begin{table}[ht]
% \centering
% \scriptsize
% \begin{tabular}{lrrrrrr}
% \toprule
% Lesion Type & Naive Risk & ReN Risk & PF Risk & Naive Acc & ReN Acc & PF Acc \\
% \midrule
% akiec & 0.110 & 0.118 & 0.103 & 48\% & 56\% & 52\% \\
% bcc & 0.096 & 0.059 & 0.071 & 60\% & 73\% & 57\% \\
% bkl & 0.110 & 0.105 & 0.077 & 51\% & 58\% & 71\% \\
% df & 0.142 & 0.143 & 0.076 & 50\% & 33\% & 67\% \\
% mel & 0.011 & 0.009 & 0.076 & 95\% & 96\% & 67\% \\
% nv & 0.043 & 0.057 & 0.051 & 86\% & 71\% & 79\% \\
% vasc & 0.156 & 0.125 & 0.072 & 37\% & 50\% & 60\% \\
% \textbf{Mean} & 0.095 & 0.088 & \textbf{0.075} & 61\% & 63\% & \textbf{64}\% \\
% \textbf{Discrepancy} & 0.145 & 0.134 & \textbf{0.052} & 58\% & 63\% & \textbf{27}\% \\
% \bottomrule
% \end{tabular}
% \caption{\footnotesize{} Group-specific risks and accuracies for Naive (Na), Rebalanced Naive (ReN), and Pareto-Fair (PF) classifiers. Lesion Type was chosen as both target and sensitive variable; an application of Pareto-fairness to adaptive risk-equalization. Lesion types are classified as Actinic keratoses and intraepithelial carcinoma (akiec), basal cell carcinoma (bcc), benign keratosis-like lesions (bkl), dermatofibroma (df), melanoma (mel), melanocytic nevi (nv) and vascular lesions (vasc). The Pareto-fair classifier exhibits the lowest balanced mean risk and highest balanced accuracy, it also produces the smallest risk and accuracy discrepancies.}
% \label{tab:HAMResults}
% \end{table}
% \begin{figure}[ht]
% \centering
% \begin{tabular}{l}
% \begin{subfigure}{.95\textwidth}
% \centering
% % include first image
% \includegraphics[width=.7\linewidth]{Figures/MIMIC_acctest.png}
% \caption{Sub-group accuracies on MIMIC dataset}
% \label{fig:MIMIC_ACC}
% \end{subfigure} \\
% \begin{subfigure}{.95\textwidth}
% \centering
% % include first image
% \includegraphics[width=.7\linewidth]{Figures/HAM_acctest.png}
% \caption{Sub-group accuracies on HAM dataset}
% \label{fig:HAM_ACC}
% \end{subfigure}
% \end{tabular}
% \caption{Accuracy disparities per sensitive attribute on Naive, Naive Rebalanced, and Pareto-Fair classifiers. Class rebalancing is only partially effective at reducing accuracy disparity, while Pareto-Fairness proves adequate.}
% \label{fig:FigResults}
% \end{figure}
% \FloatBarrier
\section{Discussion}
Here we explore the problem of reducing the risk disparity gaps in the most ethical way possible (i.e., minimizing unnecessary harm). We provide an algorithm that can be used with any standard classification or regression loss to bridge risk disparity gaps without introducing unnecessary harm. We show its performance on two real-world case studies; we take advantage of the fact that our method does not require test-time access to sensitive attributes to frame balanced classification as a fairness problem. In future work, we wish to analyze if we can automatically identify high-risk sub-populations as part of the learning process and attack risk disparities as they arise, rather than relying on preexisting notions of disadvantaged groups or populations. We believe that no-unnecessary-harm notions of fairness are of great interest for several applications, especially so on domains such as healthcare.
% \section{Submission of papers to NeurIPS 2019}
% NeurIPS requires electronic submissions. The electronic submission site is
% \begin{center}
% \url{https://cmt.research.microsoft.com/NeurIPS2019/}
% \end{center}
% Please read the instructions below carefully and follow them faithfully.
% \subsection{Style}
% Papers to be submitted to NeurIPS 2019 must be prepared according to the
% instructions presented here. Papers may only be up to eight pages long,
% including figures. Additional pages \emph{containing only acknowledgments and/or
% cited references} are allowed. Papers that exceed eight pages of content
% (ignoring references) will not be reviewed, or in any other way considered for
% presentation at the conference.
% The margins in 2019 are the same as since 2007, which allow for $\sim$$15\%$
% more words in the paper compared to earlier years.
% Authors are required to use the NeurIPS \LaTeX{} style files obtainable at the
% NeurIPS website as indicated below. Please make sure you use the current files
% and not previous versions. Tweaking the style files may be grounds for
% rejection.
% \subsection{Retrieval of style files}
% The style files for NeurIPS and other conference information are available on
% the World Wide Web at
% \begin{center}
% \url{http://www.neurips.cc/}
% \end{center}
% The file \verb+neurips_2019.pdf+ contains these instructions and illustrates the
% various formatting requirements your NeurIPS paper must satisfy.
% The only supported style file for NeurIPS 2019 is \verb+neurips_2019.sty+,
% rewritten for \LaTeXe{}. \textbf{Previous style files for \LaTeX{} 2.09,
% Microsoft Word, and RTF are no longer supported!}
% The \LaTeX{} style file contains three optional arguments: \verb+final+, which
% creates a camera-ready copy, \verb+preprint+, which creates a preprint for
% submission to, e.g., arXiv, and \verb+nonatbib+, which will not load the
% \verb+natbib+ package for you in case of package clash.
% \paragraph{Preprint option}
% If you wish to post a preprint of your work online, e.g., on arXiv, using the
% NeurIPS style, please use the \verb+preprint+ option. This will create a
% nonanonymized version of your work with the text ``Preprint. Work in progress.''
% in the footer. This version may be distributed as you see fit. Please \textbf{do
% not} use the \verb+final+ option, which should \textbf{only} be used for
% papers accepted to NeurIPS.
% At submission time, please omit the \verb+final+ and \verb+preprint+
% options. This will anonymize your submission and add line numbers to aid
% review. Please do \emph{not} refer to these line numbers in your paper as they
% will be removed during generation of camera-ready copies.
% The file \verb+neurips_2019.tex+ may be used as a ``shell'' for writing your
% paper. All you have to do is replace the author, title, abstract, and text of
% the paper with your own.
% The formatting instructions contained in these style files are summarized in
% Sections \ref{gen_inst}, \ref{headings}, and \ref{others} below.
% \section{General formatting instructions}
% \label{gen_inst}
% The text must be confined within a rectangle 5.5~inches (33~picas) wide and
% 9~inches (54~picas) long. The left margin is 1.5~inch (9~picas). Use 10~point
% type with a vertical spacing (leading) of 11~points. Times New Roman is the
% preferred typeface throughout, and will be selected for you by default.
% Paragraphs are separated by \nicefrac{1}{2}~line space (5.5 points), with no
% indentation.
% The paper title should be 17~point, initial caps/lower case, bold, centered
% between two horizontal rules. The top rule should be 4~points thick and the
% bottom rule should be 1~point thick. Allow \nicefrac{1}{4}~inch space above and
% below the title to rules. All pages should start at 1~inch (6~picas) from the
% top of the page.
% For the final version, authors' names are set in boldface, and each name is
% centered above the corresponding address. The lead author's name is to be listed
% first (left-most), and the co-authors' names (if different address) are set to
% follow. If there is only one co-author, list both author and co-author side by
% side.
% Please pay special attention to the instructions in Section \ref{others}
% regarding figures, tables, acknowledgments, and references.
% \section{Headings: first level}
% \label{headings}
% All headings should be lower case (except for first word and proper nouns),
% flush left, and bold.
% First-level headings should be in 12-point type.
% \subsection{Headings: second level}
% Second-level headings should be in 10-point type.
% \subsubsection{Headings: third level}
% Third-level headings should be in 10-point type.
% \paragraph{Paragraphs}
% There is also a \verb+\paragraph+ command available, which sets the heading in
% bold, flush left, and inline with the text, with the heading followed by 1\,em
% of space.
% \section{Citations, figures, tables, references}
% \label{others}
% These instructions apply to everyone.
% \subsection{Citations within the text}
% The \verb+natbib+ package will be loaded for you by default. Citations may be
% author/year or numeric, as long as you maintain internal consistency. As to the
% format of the references themselves, any style is acceptable as long as it is
% used consistently.
% The documentation for \verb+natbib+ may be found at
% \begin{center}
% \url{http://mirrors.ctan.org/macros/latex/contrib/natbib/natnotes.pdf}
% \end{center}
% Of note is the command \verb+\citet+, which produces citations appropriate for
% use in inline text. For example,
% \begin{verbatim}
% \citet{hasselmo} investigated\dots
% \end{verbatim}
% produces
% \begin{quote}
% Hasselmo, et al.\ (1995) investigated\dots
% \end{quote}
% If you wish to load the \verb+natbib+ package with options, you may add the
% following before loading the \verb+neurips_2019+ package:
% \begin{verbatim}
% \PassOptionsToPackage{options}{natbib}
% \end{verbatim}
% If \verb+natbib+ clashes with another package you load, you can add the optional
% argument \verb+nonatbib+ when loading the style file:
% \begin{verbatim}
% \usepackage[nonatbib]{neurips_2019}
% \end{verbatim}
% As submission is double blind, refer to your own published work in the third
% person. That is, use ``In the previous work of Jones et al.\ [4],'' not ``In our
% previous work [4].'' If you cite your other papers that are not widely available
% (e.g., a journal paper under review), use anonymous author names in the
% citation, e.g., an author of the form ``A.\ Anonymous.''
% \subsection{Footnotes}
% Footnotes should be used sparingly. If you do require a footnote, indicate
% footnotes with a number\footnote{Sample of the first footnote.} in the
% text. Place the footnotes at the bottom of the page on which they appear.
% Precede the footnote with a horizontal rule of 2~inches (12~picas).
% Note that footnotes are properly typeset \emph{after} punctuation
% marks.\footnote{As in this example.}
% \subsection{Figures}
% \begin{figure}
% \centering
% \fbox{\rule[-.5cm]{0cm}{4cm} \rule[-.5cm]{4cm}{0cm}}
% \caption{Sample figure caption.}
% \end{figure}
% All artwork must be neat, clean, and legible. Lines should be dark enough for
% purposes of reproduction. The figure number and caption always appear after the
% figure. Place one line space before the figure caption and one line space after
% the figure. The figure caption should be lower case (except for first word and
% proper nouns); figures are numbered consecutively.
% You may use color figures. However, it is best for the figure captions and the
% paper body to be legible if the paper is printed in either black/white or in
% color.
% \subsection{Tables}
% All tables must be centered, neat, clean and legible. The table number and
% title always appear before the table. See Table~\ref{sample-table}.
% Place one line space before the table title, one line space after the
% table title, and one line space after the table. The table title must
% be lower case (except for first word and proper nouns); tables are
% numbered consecutively.
% Note that publication-quality tables \emph{do not contain vertical rules.} We
% strongly suggest the use of the \verb+booktabs+ package, which allows for
% typesetting high-quality, professional tables:
% \begin{center}
% \url{https://www.ctan.org/pkg/booktabs}
% \end{center}
% This package was used to typeset Table~\ref{sample-table}.
% \begin{table}
% \caption{Sample table title}
% \label{sample-table}
% \centering
% \begin{tabular}{lll}
% \toprule
% \multicolumn{2}{c}{Part} \\
% \cmidrule(r){1-2}
% Name & Description & Size ($\mu$m) \\
% \midrule
% Dendrite & Input terminal & $\sim$100 \\
% Axon & Output terminal & $\sim$10 \\
% Soma & Cell body & up to $10^6$ \\
% \bottomrule
% \end{tabular}
% \end{table}
% \section{Final instructions}
% Do not change any aspects of the formatting parameters in the style files. In
% particular, do not modify the width or length of the rectangle the text should
% fit into, and do not change font sizes (except perhaps in the
% \textbf{References} section; see below). Please note that pages should be
% numbered.
% \section{Preparing PDF files}
% Please prepare submission files with paper size ``US Letter,'' and not, for
% example, ``A4.''
% Fonts were the main cause of problems in the past years. Your PDF file must only
% contain Type 1 or Embedded TrueType fonts. Here are a few instructions to
% achieve this.
% \begin{itemize}
% \item You should directly generate PDF files using \verb+pdflatex+.
% \item You can check which fonts a PDF files uses. In Acrobat Reader, select the
% menu Files$>$Document Properties$>$Fonts and select Show All Fonts. You can
% also use the program \verb+pdffonts+ which comes with \verb+xpdf+ and is
% available out-of-the-box on most Linux machines.
% \item The IEEE has recommendations for generating PDF files whose fonts are also
% acceptable for NeurIPS. Please see
% \url{http://www.emfield.org/icuwb2010/downloads/IEEE-PDF-SpecV32.pdf}
% \item \verb+xfig+ "patterned" shapes are implemented with bitmap fonts. Use
% "solid" shapes instead.
% \item The \verb+\bbold+ package almost always uses bitmap fonts. You should use
% the equivalent AMS Fonts:
% \begin{verbatim}
% \usepackage{amsfonts}
% \end{verbatim}
% followed by, e.g., \verb+\mathbb{R}+, \verb+\mathbb{N}+, or \verb+\mathbb{C}+
% for $\mathbb{R}$, $\mathbb{N}$ or $\mathbb{C}$. You can also use the following
% workaround for reals, natural and complex:
% \begin{verbatim}
% \newcommand{\RR}{I\!\!R} %real numbers
% \newcommand{\Nat}{I\!\!N} %natural numbers
% \newcommand{\CC}{I\!\!\!\!C} %complex numbers
% \end{verbatim}
% Note that \verb+amsfonts+ is automatically loaded by the \verb+amssymb+ package.
% \end{itemize}
% If your file contains type 3 fonts or non embedded TrueType fonts, we will ask
% you to fix it.
% \subsection{Margins in \LaTeX{}}
% Most of the margin problems come from figures positioned by hand using
% \verb+\special+ or other commands. We suggest using the command
% \verb+\includegraphics+ from the \verb+graphicx+ package. Always specify the
% figure width as a multiple of the line width as in the example below:
% \begin{verbatim}
% \usepackage[pdftex]{graphicx} ...
% \includegraphics[width=0.8\linewidth]{myfile.pdf}
% \end{verbatim}
% See Section 4.4 in the graphics bundle documentation
% (\url{http://mirrors.ctan.org/macros/latex/required/graphics/grfguide.pdf})
% A number of width problems arise when \LaTeX{} cannot properly hyphenate a
% line. Please give LaTeX hyphenation hints using the \verb+\-+ command when
% necessary.
% \subsubsection*{Acknowledgments}
% Use unnumbered third level headings for the acknowledgments. All acknowledgments
% go at the end of the paper. Do not include acknowledgments in the anonymized
% submission, only in the final paper.
\bibliography{cites.bib}
\bibliographystyle{plainnat}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{16}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Barocas and Selbst(2016)]{barocas2016big}
Solon Barocas and Andrew~D Selbst.
\newblock Big data's disparate impact.
\newblock \emph{Calif. L. Rev.}, 104:\penalty0 671, 2016.
\bibitem[Calders and Verwer(2010)]{calders2010three}
Toon Calders and Sicco Verwer.
\newblock Three naive bayes approaches for discrimination-free classification.
\newblock \emph{Data Mining and Knowledge Discovery}, 21\penalty0 (2):\penalty0
277--292, 2010.
\bibitem[Chen et~al.(2018)Chen, Johansson, and Sontag]{chen2018my}
Irene Chen, Fredrik~D Johansson, and David Sontag.
\newblock Why is my classifier discriminatory?
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
3539--3550, 2018.
\bibitem[Domingos(2000)]{domingos2000unified}
Pedro Domingos.
\newblock A unified bias-variance decomposition.
\newblock In \emph{Proceedings of 17th International Conference on Machine
Learning}, pages 231--238, 2000.
\bibitem[Dwork et~al.(2012)Dwork, Hardt, Pitassi, Reingold, and
Zemel]{dwork2012fairness}
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel.
\newblock Fairness through awareness.
\newblock In \emph{Proceedings of the 3rd innovations in theoretical computer
science conference}, pages 214--226. ACM, 2012.
\bibitem[Feldman et~al.(2015)Feldman, Friedler, Moeller, Scheidegger, and
Venkatasubramanian]{feldman2015certifying}
Michael Feldman, Sorelle~A Friedler, John Moeller, Carlos Scheidegger, and
Suresh Venkatasubramanian.
\newblock Certifying and removing disparate impact.
\newblock In \emph{Proceedings of the 21th ACM SIGKDD International Conference
on Knowledge Discovery and Data Mining}, pages 259--268. ACM, 2015.
\bibitem[Hardt et~al.(2016)Hardt, Price, Srebro, et~al.]{hardt2016equality}
Moritz Hardt, Eric Price, Nati Srebro, et~al.
\newblock Equality of opportunity in supervised learning.
\newblock In \emph{Advances in neural information processing systems}, pages
3315--3323, 2016.
\bibitem[Huang et~al.(2017)Huang, Liu, Van Der~Maaten, and
Weinberger]{huang2017densely}
Gao Huang, Zhuang Liu, Laurens Van Der~Maaten, and Kilian~Q Weinberger.
\newblock Densely connected convolutional networks.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 4700--4708, 2017.
\bibitem[Johnson et~al.(2016)Johnson, Pollard, Shen, Li-wei, Feng, Ghassemi,
Moody, Szolovits, Celi, and Mark]{johnson2016mimic}
Alistair~EW Johnson, Tom~J Pollard, Lu~Shen, H~Lehman Li-wei, Mengling Feng,
Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo~Anthony Celi, and
Roger~G Mark.
\newblock Mimic-iii, a freely accessible critical care database.
\newblock \emph{Scientific data}, 3:\penalty0 160035, 2016.
\bibitem[Louizos et~al.(2015)Louizos, Swersky, Li, Welling, and
Zemel]{louizos2015variational}
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel.
\newblock The variational fair autoencoder.
\newblock \emph{arXiv preprint arXiv:1511.00830}, 2015.
\bibitem[Pleiss et~al.(2017)Pleiss, Raghavan, Wu, Kleinberg, and
Weinberger]{pleiss2017fairness}
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian~Q
Weinberger.
\newblock On fairness and calibration.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
5680--5689, 2017.
\bibitem[Tschandl et~al.(2018)Tschandl, Rosendahl, and
Kittler]{tschandl2018ham10000}
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.
\newblock The ham10000 dataset, a large collection of multi-source
dermatoscopic images of common pigmented skin lesions.
\newblock \emph{Scientific data}, 5:\penalty0 180161, 2018.
\bibitem[Ustun et~al.(2019)Ustun, Liu, and Parkes]{ustun2019fairness}
Berk Ustun, Yang Liu, and David Parkes.
\newblock Fairness without harm: Decoupled classifiers with preference
guarantees.
\newblock In \emph{International Conference on Machine Learning}, pages
6373--6382, 2019.
\bibitem[Woodworth et~al.(2017)Woodworth, Gunasekar, Ohannessian, and
Srebro]{woodworth2017learning}
Blake Woodworth, Suriya Gunasekar, Mesrob~I Ohannessian, and Nathan Srebro.
\newblock Learning non-discriminatory predictors.
\newblock \emph{arXiv preprint arXiv:1702.06081}, 2017.
\bibitem[Zafar et~al.(2017)Zafar, Valera, Rodriguez, Gummadi, and
Weller]{zafar2017parity}
Muhammad~Bilal Zafar, Isabel Valera, Manuel Rodriguez, Krishna Gummadi, and
Adrian Weller.
\newblock From parity to preference-based notions of fairness in
classification.
\newblock In \emph{Advances in Neural Information Processing Systems}, pages
229--239, 2017.
\bibitem[Zemel et~al.(2013)Zemel, Wu, Swersky, Pitassi, and
Dwork]{zemel2013learning}
Rich Zemel, Yu~Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork.
\newblock Learning fair representations.
\newblock In \emph{International Conference on Machine Learning}, pages
325--333, 2013.
\end{thebibliography}