Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation
Extracted source text · arXiv 2211.01939
\section{Conditinal Average Treatment Effect Estimators}
\label{sec:cate-estimators}
\subsection{S-Learner, T-Learner, \& X-Learner}
As stated in the problem formulation section, under the assumptions of consistency, exchangeability, and overlap, the CATE is identifiable and can be computed from observational data as follows:
\begin{equation*}
T(x) = \mathbb{E}(Y | W=1, X=x) - \mathbb{E}(Y | W=0, X=x)
\end{equation*}
The \textbf{T-Learner} approach is based on approximating the conditional expected outcome functions ($\mathbb{E}(Y | W=0, X=x), \mathbb{E}(Y | W=1, X=x)$) and using them for CATE estimation. It approximates $\mathbb{E}(Y | W=0, X=x) $ as $\hat{\mu}_0$ by regressing $Y$ on $X$ using samples from the un-treated population $\{X_i, Y_i \}_{W_i=0}$, and $\mathbb{E}(Y | W=1, X=x)$ as $\hat{\mu}_1$ by regressing $Y$ on $X$ using samples $\{X_i, Y_i \}_{W_i=1}$ from the treated population. Then the CATE is computed as follows:
\begin{equation}
\label{eq:ite-t-learner}
\hat{T}(X)= \hat{\mu}_1(X) - \hat{\mu}_0(X)
\end{equation}
The \textbf{S-Leaner} approach is also similar but it learns a single regression model, regressing $Y$ jointly on features $X$ and treatment assignments $W$, i.e., $\hat{\mu}(X, W)$ from samples $\{ X_i, W_i, Y_i \}$. Then the CATE is computed as follows:
\begin{equation}
\label{eq:ite-s-learner}
\hat{T}(X)= \hat{\mu}(X, 1) - \hat{\mu}(X, 0)
\end{equation}
If the nuisances models ($\hat{\mu}_0(X), \hat{\mu}_1(X), \hat{\mu}(X,W)$) are universal function approximators, then S-Learner and T-Leaner would offer unbiased estimates of CATE. However, there will be statistical error in practice due to the nuisance model class not capturing the true mean outcome function. This will be amplified more as compared to standard regression tasks as we also do model-extrapolation while computing CATE, i.e. use $\hat{\mu}_{w}(x)$ to make predictions for data points with same features as observed data points but different treatment assignment~\cite{kennedy2020optimal}.
Further, we also use the \textbf{Projected S-Learner}~\cite{econml}, which first learns the nuisance model $\hat{\mu}(X, W)$ like the S-Learner but then solves the following regression problem, where the final regression model ($f$) is used for CATE prediction.
\begin{equation}
\label{eq:ite-proj-s-learner}
\centering
\min_{\theta} \sum_{i} \left((\hat{\mu}(X_i, 1) - \hat{\mu}(X_i, 0)) - f_{\theta}(X_i)\right)^2
\end{equation}
This introduces an extra regularization step which could avoid potential overfitting, especially if we have some prior belief on the functional form or the smoothness of the CATE function.
Finally, we also use the \textbf{X-Learner}~\cite{kunzel2019metalearners} approach where we first train the nuisance models ($\hat{\mu}_0(X), \hat{\mu}_1(X)$) same as the T-Learner approach. But rather than directly using these nuisance models for predicting CATE, we approximate the CATE separately for the control and the treatment group using final regression models ($f_{\theta}^{0}, f_{\theta}^1$) as follows.
\begin{equation*}
\begin{aligned}
& \min_{\theta} \sum_{i, W_i= 0} \left((\mu_{1}(X_i) - Y_i) - f^{0}_{\theta}(X_i)\right)^2 \\
& \min_{\theta} \sum_{i, W_i= 1} \left((Y_i - \mu_{0}(X_i)) - f^{1}_{\theta}(X_i)\right)^2
\end{aligned}
\end{equation*}
Finally, we use propensity model ($\hat{\pi}_{W}(X)$) and combine the CATE from the control and treatment group as follows for final CATE prediction:
\begin{equation}
\label{eq:ite-x-learner}
\hat{T}(X)= \hat{\pi}_{W=0}(X) * f_{\theta}^0(X) + \hat{\pi}_{W=1}(X) * f_{\theta}^1(X)
\end{equation}
\subsection{Doubly Robust Learner (DR Learner)}
To avoid the heavy dependence on the regression function, recent works have proposed generalizations of the doubly robust estimator (traditionally used for ATE inference), for CATE estimation~\cite{foster2019orthogonal, kennedy2020optimal}. The doubly robust learner is a mixture of the S-learner with an inverse propensity based approach. First we state the vanilla inverse propensity based learner and then the combination. The inverse propensity weighted (IPW) estimator, uses the true propensity model ($\pi_{t}(x) =\mathbb{P}_{\theta} [W = t | X=x]$) and defines the IPW outcome as follows:
\begin{equation}
\label{eq:ite-ipw}
Y^{IPW}_{i,t}= \frac{Y_i}{ \pi_{t}(x) } \cdot \mathbbm{1}(W_i = t)
\end{equation}
It can be shown that $\mathbb{E}[ Y^{IPW}_{i, t} | X_{i} ] = \mathbb{E}[Y_{i}(t)| X_{i}]$~\cite{peters2017elements}. Hence, $Y^{IPW}_{i,t}$ gives an unbiased estimate of the potential outcome $Y_{i}(t)$ for the individual i. Therefore, $Y^{IPW}_{i,1} - Y^{IPW}_{i,0}$ can be used to approximate the CATE, and in similar spirit to supervised learning, we can learn a regression model that maps the features $\{X_i\}$ to the targets $\{ Y^{IPS}_{i,1} - Y^{IPS}_{i,0} \}$, i.e., $\min_{\theta} \sum_{i} || Y^{IPS}_{i,1} - Y^{IPS}_{i,0} - f_{\theta}(X_i) ||^2$, where $f(X)$ can be used for CATE estimation for new samples.
Unlike the S-Learner the IPW learner does not depend on any outcome regression model, but requires estimating a model $\hat{\pi}_t$ of the propensity model ($\pi_{W}(X)$) and heavily depends on the approximation error of that model. Further, there are numerical stability issues if the true probability of a certain treatment is low, as then we are dividing by very small numbers. Hence, the IPW estimator can have very high variance.
Doubly robust learning~\cite{foster2019orthogonal, kennedy2020optimal} overcomes these issues. It can be interpreted as a hybrid approach, where it first directly approximates the expected outcome function ($\hat{\mu}(x, w)$) like S-Learner, and then it debiases the estimate using the inverse propensity weighting based approach on the residual ($Y - \hat{\mu}(x, w)$), as shown below:
\begin{equation}
\label{eq:ite-dr}
Y^{DR}_{i,t} = \hat{\mu}(X_i, t) + \frac{Y_i - \hat{\mu}(X_i, W_i) }{\hat{\pi}_{t}(X_i)}\cdot\mathbbm{1}(W_i = t)
\end{equation}
Finally, we learn a regression model $f$ that maps the features $\{X_i\}$ to the targets $\{ Y^{DR}_{i,1} - Y^{DR}_{i,0} \}$, i.e., $\min_{\theta} \sum_{i} || Y^{DR}_{i,1} - Y^{DR}_{i,0} - f_{\theta}(X_i) ||^2$. Hence, the final regression model $f$ is trained to predict CATE on samples in observation data, and can be used for CATE prediction on new samples.
\subsection{Double Machine Learning (R-Learner)}
The Double ML approach to CATE estimation~\cite{chernozhukov2016double, nie2021quasi}, also known in the literature as the R-Learner~\cite{nie2021quasi}, solves the CATE estimation task by learning the following nuisance models ($\hat{\pi}, \hat{\mu}$), where $\hat{\pi}$ is an estimate of the true propensity model, but $\hat{\mu}$ is an estimate of $\mathbb{E}(Y|X)$, i.e, predicting outcome only from the features. Then it computes the residuals from both the outcome prediction task ($Y-\hat{\mu}(X)$) and the treatment prediction task ($W-\hat{\pi}(X)$). In the final step it learns a CATE model $f$, by minimizing the following loss function:
\begin{equation}
\label{eq:ite-dml}
\centering
\min_{\theta} \sum_{i} \left((Y_i - \hat{\mu}(X_{i})) - f_{\theta}(X_i) \cdot ( W_i - \hat{\pi}(X_i) ) \right)^2
\end{equation}
The latter can be viewed equivalently as a weighted regression problem, where we regress the target $\frac{Y-\hat{\mu}(X)}{W - \hat{\pi}(X)}$ on the features $X$, with sample weights $(W-\hat{\pi}(X))^2$. Hence, any sample-weighted regression approach can be applied to solve the R-Learner problem and from that perspective the R-Learner can also be viewed as a meta-learner.
This loss can be justified by simply observing that for a binary treatment, we can write
\begin{equation*}
\mathbb{E}[Y\mid X,W] = T(X)\cdot W \Rightarrow \mathbb{E}[Y\mid X] = T(X)\cdot \mathbb{E}[W\mid X]
\end{equation*}
Thus we have that:
\begin{align*}
\mathbb{E}[Y - \mathbb{E}[Y\mid X] \mid X,W] = T(X)\cdot (W - \mathbb{E}[W\mid X])
\end{align*}
and the R-Learner loss is the square loss that corresponds to the latter regression equation. The R-Learner has been shown to enjoy theoretical robustness properties similar to the DR learner (i.e. that the impact of errors in the nuisance models $\hat{\pi},\hat{\mu}$) on the final estimate is alleviated, albeit for the R-Learner the accuracy of the propensity is more important and the method is not consistent if the propensity model is not consistent (while the DR Learner would be if the regression model was consistent but the propensity model was inconsistent). Moreover, if the model space ($f_\theta$) does not contain the true CATE model ($T$), then the outcome of the DR Learner can be interpreted asymptotically as a projection of $T$ on the model space, with respect to the MSE norm, while the outcome of the R-Learner is a weighted projection, weighted by the variance of the treatment; hence approximating the true $T$ more accurately in regions where the treatment is more random.
% \textbf{Causal Forest DML Learner:} \divyat{TODO}
\section{GENERAL FORMATTING INSTRUCTIONS}
The camera-ready versions of the accepted papers are 8 pages,
plus any additional pages needed for references.
Papers are in 2 columns with the overall line width of 6.75~inches (41~picas).
Each column is 3.25~inches wide (19.5~picas). The space
between the columns is .25~inches wide (1.5~picas). The left margin is 0.88~inches (5.28~picas).
Use 10~point type with a vertical spacing of
11~points. Please use US Letter size paper instead of A4.
Paper title is 16~point, caps/lc, bold, centered between 2~horizontal rules.
Top rule is 4~points thick and bottom rule is 1~point thick.
Allow 1/4~inch space above and below title to rules.
Author descriptions are center-justified, initial caps. The lead
author is to be listed first (left-most), and the Co-authors are set
to follow. If up to three authors, use a single row of author
descriptions, each one center-justified, and all set side by side;
with more authors or unusually long names or institutions, use more
rows.
Use one-half line space between paragraphs, with no indent.
\section{FIRST LEVEL HEADINGS}
First level headings are all caps, flush left, bold, and in point size
12. Use one line space before the first level heading and one-half line space
after the first level heading.
\subsection{Second Level Heading}
Second level headings are initial caps, flush left, bold, and in point
size 10. Use one line space before the second level heading and one-half line
space after the second level heading.
\subsubsection{Third Level Heading}
Third level headings are flush left, initial caps, bold, and in point
size 10. Use one line space before the third level heading and one-half line
space after the third level heading.
\paragraph{Fourth Level Heading}
Fourth level headings must be flush left, initial caps, bold, and
Roman type. Use one line space before the fourth level heading, and
place the section text immediately after the heading with no line
break, but an 11 point horizontal space.
%%%
\subsection{Citations, Figure, References}
\subsubsection{Citations in Text}
Citations within the text should include the author's last name and
year, e.g., (Cheesman, 1985).
%Apart from including the author's last name and year, citations can follow any style, as long as the style is consistent throughout the paper.
Be sure that the sentence reads
correctly if the citation is deleted: e.g., instead of ``As described
by (Cheesman, 1985), we first frobulate the widgets,'' write ``As
described by Cheesman (1985), we first frobulate the widgets.''
The references listed at the end of the paper can follow any style as long as it is used consistently.
%Be sure to avoid
%accidentally disclosing author identities through citations.
\subsubsection{Footnotes}
Indicate footnotes with a number\footnote{Sample of the first
footnote.} in the text. Use 8 point type for footnotes. Place the
footnotes at the bottom of the column in which their markers appear,
continuing to the next column if required. Precede the footnote
section of a column with a 0.5 point horizontal rule 1~inch (6~picas)
long.\footnote{Sample of the second footnote.}
\subsubsection{Figures}
All artwork must be centered, neat, clean, and legible. All lines
should be very dark for purposes of reproduction, and art work should
not be hand-drawn. Figures may appear at the top of a column, at the
top of a page spanning multiple columns, inline within a column, or
with text wrapped around them, but the figure number and caption
always appear immediately below the figure. Leave 2 line spaces
between the figure and the caption. The figure caption is initial caps
and each figure should be numbered consecutively.
Make sure that the figure caption does not get separated from the
figure. Leave extra white space at the bottom of the page rather than
splitting the figure and figure caption.
\begin{figure}[h]
\vspace{.3in}
\centerline{\fbox{This figure intentionally left non-blank}}
\vspace{.3in}
\caption{Sample Figure Caption}
\end{figure}
\subsubsection{Tables}
All tables must be centered, neat, clean, and legible. Do not use hand-drawn tables.
Table number and title always appear above the table.
See Table~\ref{sample-table}.
Use one line space before the table title, one line space after the table title,
and one line space after the table. The table title must be
initial caps and each table numbered consecutively.
\begin{table}[h]
\caption{Sample Table Title} \label{sample-table}
\begin{center}
\begin{tabular}{ll}
\textbf{PART} &\textbf{DESCRIPTION} \\
\hline \\
Dendrite &Input terminal \\
Axon &Output terminal \\
Soma &Cell body (contains cell nucleus) \\
\end{tabular}
\end{center}
\end{table}
\section{SUPPLEMENTARY MATERIAL}
If you need to include additional appendices during submission, you can include them in the supplementary material file.
You can submit a single file of additional supplementary material which may be either a pdf file (such as proof details) or a zip file for other formats/more files (such as code or videos).
Note that reviewers are under no obligation to examine your supplementary material.
If you have only one supplementary pdf file, please upload it as is; otherwise gather everything to the single zip file.
You must use \texttt{aistats2023.sty} as a style file for your supplementary pdf file and follow the same formatting instructions as in the main paper.
The only difference is that it must be in a \emph{single-column} format.
You can use \texttt{supplement.tex} in our starter pack as a starting point.
Alternatively, you may append the supplementary content to the main paper and split the final PDF into two separate files.
\section{SUBMISSION INSTRUCTIONS}
To submit your paper to AISTATS 2023, please follow these instructions.
\begin{enumerate}
\item Download \texttt{aistats2023.sty}, \texttt{fancyhdr.sty}, and \texttt{sample\_paper.tex} provided in our starter pack.
Please, do not modify the style files as this might result in a formatting violation.
\item Use \texttt{sample\_paper.tex} as a starting point.
\item Begin your document with
\begin{flushleft}
\texttt{\textbackslash documentclass[twoside]\{article\}}\\
\texttt{\textbackslash usepackage\{aistats2023\}}
\end{flushleft}
The \texttt{twoside} option for the class article allows the
package \texttt{fancyhdr.sty} to include headings for even and odd
numbered pages.
\item When you are ready to submit the manuscript, compile the latex file to obtain the pdf file.
\item Check that the content of your submission, \emph{excluding} references, is limited to \textbf{8 pages}. The number of pages containing references alone is not limited.
\item Upload the PDF file along with other supplementary material files to the CMT website.
\end{enumerate}
\subsection{Camera-ready Papers}
%For the camera-ready paper, if you are using \LaTeX, please make sure
%that you follow these instructions.
% (If you are not using \LaTeX,
%please make sure to achieve the same effect using your chosen
%typesetting package.)
If your papers are accepted, you will need to submit the camera-ready version. Please make sure that you follow these instructions:
\begin{enumerate}
%\item Download \texttt{fancyhdr.sty} -- the
%\texttt{aistats2023.sty} file will make use of it.
\item Change the beginning of your document to
\begin{flushleft}
\texttt{\textbackslash documentclass[twoside]\{article\}}\\
\texttt{\textbackslash usepackage[accepted]\{aistats2023\}}
\end{flushleft}
The option \texttt{accepted} for the package
\texttt{aistats2023.sty} will write a copyright notice at the end of
the first column of the first page. This option will also print
headings for the paper. For the \emph{even} pages, the title of
the paper will be used as heading and for \emph{odd} pages the
author names will be used as heading. If the title of the paper
is too long or the number of authors is too large, the style will
print a warning message as heading. If this happens additional
commands can be used to place as headings shorter versions of the
title and the author names. This is explained in the next point.
\item If you get warning messages as described above, then
immediately after $\texttt{\textbackslash
begin\{document\}}$, write
\begin{flushleft}
\texttt{\textbackslash runningtitle\{Provide here an alternative
shorter version of the title of your paper\}}\\
\texttt{\textbackslash runningauthor\{Provide here the surnames of
the authors of your paper, all separated by commas\}}
\end{flushleft}
Note that the text that appears as argument in \texttt{\textbackslash
runningtitle} will be printed as a heading in the \emph{even}
pages. The text that appears as argument in \texttt{\textbackslash
runningauthor} will be printed as a heading in the \emph{odd}
pages. If even the author surnames do not fit, it is acceptable
to give a subset of author names followed by ``et al.''
%\item Use the file sample\_paper.tex as an example.
\item The camera-ready versions of the accepted papers are 8
pages, plus any additional pages needed for references.
\item If you need to include additional appendices,
you can include them in the supplementary
material file.
\item Please, do not change the layout given by the above
instructions and by the style file.
\end{enumerate}
\subsubsection*{Acknowledgements}
All acknowledgments go at the end of the paper, including thanks to reviewers who gave useful comments, to colleagues who contributed to the ideas, and to funding agencies and corporate sponsors that provided financial support.
To preserve the anonymity, please include acknowledgments \emph{only} in the camera-ready papers.
\section{Introduction}
\label{sec:intro}
% Consider the task of determining whether to prescribe a medicine to an individual. To make this decision, we would optimize for a certain outcome of the individual, say whether the medicine would cure the individual. Hence, we compare the effects of prescribing the medicine versus not prescribing the medicine on the health of the individual. This is a causal inference task, where we are interested in the causal effect of prescribing the medicine on the health of the individual. Many decision problems where we have to choose between intervening versus not intervening are studied as causal inference tasks, where interventions can range from targeted advertisements, recommendations, healthcare policies, etc.
What is the personalized effect of an intervention on an individual as a function of observable characteristics? This heterogeneous effect estimation problem is at the heart of every decision making problem, ranging from personalized medicine to personalized policies in the operations domain and digital platforms. With the emergence of rich data sets, decision makers are more and more interested in such personalized effect models.
The challenging task is to identify which individuals would benefit the most from intervention/treatment. This would require us to measure the treatment effects at an individual level as opposed to the average treatment effect across the whole population. This is because if we assign interventions solely based on the average treatment effect then we do not take the heterogeneity of the data into account, which might result in sub-optimal outcomes for many individuals~\cite{Segal2012UnderstandingHO}. The emphasis on individual treatments effects has been shown in multiple application domains, from personalised healthcare~\cite{foster2011subgroup} to social sciences~\cite{xie2012estimating}.
This has lead to an explosion of recent techniques in the causal machine learning literature to estimate flexible and accurate models of effect heterogeneity. These approaches range from adapting neural networks~\cite{shi2019adapting}, random forests~\cite{wager2018estimation}, double machine learning~\cite{chernozhukov2016double, foster2019orthogonal, nie2021quasi}, instrumental variables~\cite{hartford2017deep}, meta learners~\cite{kunzel2019metalearners}, etc. However, the issue is that these estimators often contradict each other and we would need to perform model selection. While in some situations we can choose between the estimators based on domain knowledge and application requirements, it is desirable to have to a model-free (not specific to any particular type of causal estimator) approach for model selection. Further, the commonly used practice of cross-validation in supervised learning problems~\cite{bengio2013representation} cannot be used for model selection in causal inference, as we never observe both of the potential outcomes for an individual (fundamental problem of causal inference~\cite{holland1986statistics}). Hence, no access to counterfactual data in observational studies makes the task of model selection in causal inference very challenging.
Recently, there have been good advances in solving this problem. Earlier proposals for model selection were based on evaluating the nuisance models (outcome/propensity models) associated with the estimators, and the utility of decision policy based on the heterogeneous treatment effects of the estimator. Further, there are proposals on accurately estimating the ground truth effect~\cite{rolling2013matching, nie2021quasi, saito2020counterfactual} and computing deviation of the estimator's treatment effect from the approximate ground truth effect, which have also been shown to be effective than other metrics~\cite{schuler2018comparison, alaa2019validating}. However, most of these evaluation studies have been performed only on few synthetic datasets, as there we can compare the performance of each metric against the oracle metric (access to counterfactual data). Hence, the trend in such studies could be questionable as we have not incorporated a variety of datasets. Also, there is a lack of fair comparison between the various metrics as some of them are excluded as part of the baseline when authors evaluate their proposed metrics.
We attempt to fix these issues with prior evaluation studies by performing a comprehensive evaluation study over \textbf{144 datasets} and \textbf{9 evaluation} metrics, where the model selection task is made challenging by training a large number of estimators (\textbf{415 CATE estimators}) for each dataset. We also make use of recent advances in generative modeling for causal inference~\cite{neal2020realcause} and use them to model interventional distributions from realistic datasets. This helps us to generate counterfactual data for the realistic datasets as well and include realistic benchmarks in our analysis, unlike prior evaluation studies~\cite{schuler2018comparison, alaa2019validating, saito2020counterfactual}. Further, we ensure a consistent hyperparameter selection strategy for nuisance models using AutoML~\cite{Wang_FLAML_A_Fast_2021}. An interesting finding from our analysis is that you often do not need complex evaluation metrics, and simple evaluation metrics like Tau T score are very good on average.
Our code repository can be accessed from the following link: \href{https://github.com/divyat09/cate-estimator-selection}{divyat09/cate-estimator-selection}.
% The paper is organized as follows. We first formulate the model selection problem formally in Section~\ref{sec:formulation}. We then provide a background of the causal estimators (Section~\ref{sec:cate-estimators}) and the evaluation metrics (Section~\ref{sec:eval-metrics}) that we use for our analysis. Then we provide details about our methodology in Section \ref{sec:methodology}, and discuss how it alleviates the shortcomings of prior studies. Finally, we provide the results of our analysis in Section~\ref{sec:results}.
\section{Results}
\label{sec:results}
\begin{table*}[t]
\centering
\small
\begin{tabular}{lrrrrr}
\toprule
Dataset Group & ACIC 2016 & LaLonde CPS & LaLonde PSID & ACIC 2018 & TWINS \\
\midrule
Value Score & 1147378.75 & 1.03 & 0.50 & 1.43e+06 & 1355516.83 \\
Value Dr Score & 4215.48 & 0.61 & 3.66 & 3.57e+04 & 1384464.32 \\
Value DR Clip Prop Score & 3722.94 & 3253.65 & 3.97 & 2.25e+05 & 1080046.10 \\
Tau T Score & \textbf{0.32} & \textbf{0.09} & \textbf{0.18} & 9.43e+01 & 0.46 \\
Tau S Score & \textbf{0.34} & 0.45 & 0.49 & \textbf{9.80e-01} & 0.32 \\
Tau Match Score & 1.71 & \textbf{0.10} & \textbf{0.19} & 1.59e+02 & 0.35 \\
Tau IPTW Score & 1.72 & 0.30 & \textbf{0.17} & 2.54e+08 & \textbf{0.20} \\
Tau IPTW Clip Prop Score & 1.57 & 0.84 & 0.86 & 1.82e+04 & \textbf{0.19} \\
Tau DR Score & \textbf{0.29} & \textbf{0.07} & \textbf{0.20} & 2.54e+08 & 0.61 \\
Tau DR Clip Prop Score & \textbf{0.30} & 0.91 & 0.87 & 1.09e+02 & 0.62 \\
Influence Score & 85496.76 & 1.14 & 0.58 & 5.62e+08 & 0.99 \\
Influence Clip Prop Score & 5.23 & 0.45 & 0.52 & 3.45e+02 & 1.02 \\
R Score & \textbf{0.35} & 0.38 & 0.38 & 8.83e+01 & 0.46 \\
\bottomrule
\end{tabular}
\caption{Normalized PEHE: Percentage difference of the PEHE of the best estimators chosen by each metric from the oracle PEHE; results report the mean across 3 seeds and also across dataset instances for the ACIC 2016 \& ACIC 2018 datasets. \textbf{Lower value is better.}}
\label{tab:normalized-pehe}
\end{table*}
\begin{table*}[th]
\centering
\small
\begin{tabular}{lrrrrr}
\toprule
Dataset Group & ACIC 2016 & LaLonde CPS & LaLonde PSID & ACIC 2018 & TWINS \\
\midrule
Value Score & 0.05 & \textbf{0.67} & 0.00 & 0.02 & 0.00 \\
Value DR Score & 0.10 & 0.00 & 0.00 & 0.04 & \textbf{0.33} \\
Value DR Clip Prop Score & 0.11 & 0.00 & 0.00 & 0.05 & 0.00 \\
Tau T Score & 0.48 & \textbf{0.67} & \textbf{ 0.67} & 0.17 & 0.00 \\
Tau S score & 0.26 & 0.00 & 0.00 & \textbf{0.49} & \textbf{ 0.33} \\
Tau Match Score & 0.06 & \textbf{0.67} & \textbf{ 0.67 }& 0.06 & \textbf{0.33}\\
Tau IPTW Score & 0.10 & 0.33 & 0.33 & 0.06 & \textbf{0.33 }\\
Tau IPTW Clip Prop Score & 0.10 & 0.00 & 0.00 & 0.04 & \textbf{0.33} \\
Tau DR Score & 0.51 & 0.33 & 0.00 & 0.26 & 0.00 \\
Tau DR Clip Prop Score & \textbf{0.52} & 0.00 & 0.00 & 0.28 & 0.00 \\
Influence Score & 0.01 & 0.00 & 0.00 & 0.03 & 0.00 \\
Influence Clip Prop Score & 0.01 & 0.00 & 0.00 & 0.03 & 0.00 \\
R Score & 0.23 & 0.00 & 0.00 & 0.06 & 0.00 \\
\bottomrule
\end{tabular}
\caption{Win Rate: Percentage of times the best estimators chosen by each metric dominate the best estimators chosen by other metrics in terms of PEHE; results report the mean across 3 seeds and also across dataset instances for the ACIC 2016 \& ACIC 2018 datasets. \textbf{Higher value is better}.}
\label{tab:win-rate}
\end{table*}
\textbf{Observations across all tables:}
In every table, Tau S score outperforms all other scores on the ACIC 2018 datasets. This is an indication that the data generating process for them might be too tight a match to S-learner, which would imply that it is a bad benchmark. Further, Tau T score and Tau DR score are consistently top performers. Interestingly, many simple scores such as Tau T score, Tau S score, etc.\ do quite well; while complex scores such as Influence score (clipped) do not perform as well as the simpler ones. If you want one score that does best, on average, it appears to be the Tau T score, but it does not dominate all the other scores. We weight Section \ref{sec:normalized-pehe} heaviest in this conclusion.
\subsection{Normalized PEHE Table}
\label{sec:normalized-pehe}
We first examine the normalized PEHE (in Table \ref{tab:normalized-pehe}), which is just the percentage change of PEHE of the best estimators selected by a metric from the PEHE of the best estimator chosen by oracle (absolute value for PEHE can be found in Appendix Section 2). We bold the values that are reasonably close to each other within columns, considering those are the winning scores for that column (dataset). We will slice it both by dataset group (column-wise) and by score (row-wise).
\textbf{Slicing by dataset (column-wise):} For the \textbf{ACIC 2016} dataset, Tau DR score (clipped), Tau T score, Tau S score, and R Score all do well, getting within 35\% of the oracle. In \textbf{ACIC 2018} dataset, Tau S score beats all other scores by at least 1 order of magnitude. For the \textbf{LaLonde CPS} dataset, Tau DR score, Tau T score, and Tau match score all do well, getting within 10\% of the oracle. In \textbf{LaLonde PSID} dataset, Tau IPTW score, Tau T score, Tau match score, and Tau DR score all do well, getting within 20\% of the oracle. Finally, for the \textbf{Twins} dataset, Tau IPTW score and the clipped version of it both do quite well, getting within 20\% of the oracle.
\textbf{Slicing by score (row-wise):} Tau T score is the winner, winning three datasets and doing well on the other two, and getting within 50\% of the oracle on all datasets. Tau DR score is the runner-up for winning three datasets and doing well on one of the others. Honorable mentions for winning two datasets: Tau S score, Tau match score, Tau IPTW score. Honorable mentions for getting within 50\% of the oracle on all datasets except ACIC 2018: Tau T score, Tau S score, R score.
\begin{table*}[th]
\centering
\small
\begin{tabular}{lrrrrr}
\toprule
Dataset Group & ACIC 2016 & LaLonde CPS & LaLonde PSID & ACIC 2018 & TWINS \\
\midrule
Value Score & 0.58 & 0.42 & 0.11 & 0.31 & 0.49 \\
Value DR Score & 0.80 & 0.37 & -0.05 & 0.50 & 0.39 \\
Value DR Clip Prop Score & 0.80 & -0.09 & -0.39 & 0.52 & 0.45 \\
Tau T Score & \textbf{0.98} & \textbf{0.85} & 0.91 & 0.84 & 0.72 \\
Tau S Score & \textbf{0.98} & 0.39 & -0.47 & \textbf{0.94} & 0.85 \\
Tau Match Score & 0.71 & \textbf{0.85} & 0.83 & 0.75 & 0.85 \\
Tau IPTW Score & 0.81 & 0.78 & \textbf{0.92} & 0.68 & 0.88 \\
Tau IPTW Clip Prop Score & 0.82 & 0.36 & -0.16 & 0.68 & \textbf{0.89} \\
Tau DR Score & \textbf{0.98} & \textbf{0.85} & 0.90 & 0.85 & 0.85 \\
Tau DR Clip Prop Score & \textbf{0.98} & 0.36 & -0.27 & 0.85 & 0.85 \\
Influence Score & 0.34 & 0.09 & -0.60 & 0.42 & 0.80 \\
Influence Clip Prop Score & 0.37 & 0.25 & -0.08 & 0.41 & 0.80 \\
R Score & 0.96 & 0.54 & 0.17 & 0.86 & 0.79 \\
\bottomrule
\end{tabular}
\caption{Spearman Rank correlation between evaluation metrics and oracle metric; results report the mean across 3 seeds and also across dataset instances for the ACIC 2016 \& ACIC 2018 datasets. \textbf{Higher value is better.}}
\label{tab:rank-correlation}
\end{table*}
\subsection{Win Rate Table}
We now examine the the win rates of strategies that use the different scores to choose the best estimators in Table \ref{tab:win-rate}. A given score ``chooses'' the estimator(s) that minimize that score.
%Again, we first slice by dataset (column) and then by score (row).
\textbf{Slicing by dataset:} For the \textbf{ACIC 2016} dataset, Tau DR score (clipped) and Tau T score perform best, winning about half the time. In \textbf{ACIC 2018}, Tau S score blows everything out of the water, winning about half the time, while the next closest only wins about $\nicefrac{1}{4}$ of the time.
For the \textbf{LaLonde} datasets, Tau T score and Tau match score perform best, each winning in two out of three seeds in both the LaLonde datasets. Finally, for the \textbf{Twins} dataset, Value DR score, Tau S score, Tau Match score, and Tau IPTW (clipped) score all perform okay, each winning only one out of three seeds.
\textbf{Slicing by score:} The standouts that do well on multiple datasets are Tau T score, Tau S score, Tau Match score, and Tau DR score.
\subsection{Rank Correlation Table}
Here we do not check for the quality of the best estimators chosen by each metric, but the performance (spearman rank correlation) of the metrics for ranking all the estimators w.r.t the ground truth ranking of estimators given by oracle score.
\textbf{Slicing by dataset:} You can see the winners pretty clearly in bold in Table \ref{tab:rank-correlation}.
\textbf{Slicing by score:} Both Tau DR score and Tau T score do well across all datasets, winning two datasets each and doing well on the rest.
\subsection{Chosen Estimators vs Inherent Bias of the Metric}
\begin{figure}[ht]
\centering
\includegraphics[width=0.45 \linewidth]{figures/RScore.eps}
\includegraphics[width=0.45 \linewidth]{figures/TauDRScore.eps}
% \subfloat[]{\includegraphics[width=0.45 \linewidth]{figures/RScore.eps}}
% \subfloat[]{\includegraphics[width=0.45 \linewidth]{figures/TauDRScore.eps}}\\
% \subfigure[]{
% }
% \subfigure[]{
% }
\caption{Percentage of times each estimator is chosen as the best estimator by R Score (left) and Tau DR Score (right).}
\label{fig:top-est-chosen-plots}
\end{figure}
We check for the percentage of times each type of CATE estimator is chosen as the best estimator by a metric, and find that it does not necessarily match with the underlying bias of the metric. Figure~\ref{fig:top-est-chosen-plots} (left) shows that the top estimators chosen by R Score is not DML Learner, and Figure~\ref{fig:top-est-chosen-plots} (right) shows that the top estimators chosen by Tau DR Score is not DR Learner.
\begin{figure}[t]
\centering
\includegraphics[width=0.80 \textwidth]{figures/pipeline.eps}
\caption{Overview of our evaluation methodology.}
\label{fig:pipeline}
\end{figure}
\section{Evaluation Process}
\label{sec:methodology}
While there have been prior studies~\cite{schuler2018comparison} on evaluating different metrics for model selection, they only used the Infant Health Development Program (IHDP) dataset and no realistic datasets where we do not have access to counterfactual data. Further, even the most recently proposed metrics~\cite{saito2020counterfactual, alaa2019validating} do not evaluate on multiple datasets and only stick to synthetic datasets. Also, there is a lack of fair comparison between different metrics. For example, the Influence Score~\cite{alaa2019validating} is not compared against other well established metrics like R Score~\ref{eq:r-score}. Given these issues with the prior works, we perform an extensive evaluation of all the major evaluation metrics over a range of datasets.
One of the benchmarks we use is RealCause~\cite{neal2020realcause}. Compared to other benchmarks, RealCause is useful as it closely models the distribution of real datasets such as Lalonde~\cite{lalonde1986evaluating} and Twins~\cite{louizos2017causal}. It does this using state-of-the-art generative modeling techniques such as normalizing flows~\cite{huang2018neural} and verifies its closeness to the original datasets using a variety of visualizations and statistical tests. They model the selection mechanism ($\mathbb{P}(W|X)$) and the output mechanism ($\mathbb{P}(Y|W, X)$) using generative models ($\mathbb{P}_{model}(W|X)$, $\mathbb{P}_{model}(Y|W, X)$), where the covariates $X$ are sampled from the observed realistic dataset. This gives us access to the interventional distributions ($\mathbb{P}_{model}(Y|do(W=0), X)$, $\mathbb{P}_{model}(Y|do(W=1), X)$), hence both potential outcomes for every individual in realistic datasets. We incorporate three realistic datasets; LaLonde PSID, LaLonde CPS~\cite{lalonde1986evaluating}, and Twins~\cite{louizos2017causal} using RealCause. Note that we do not need counterfactual data for evaluation metrics as they are a function of only the observational data. But we need counterfactual data to compute the oracle (PEHE) metric~\ref{eq:pehe}, as it would be used to ground the performance of other evaluation metrics. Further, we use semi-synthetic datasets from the ACIC 2016~\cite{dorie2019automated} and the ACIC 2018 competition~\cite{shimoni2018benchmarking}, where we discard datasets that have variance in ground truth CATE lower than $0.01$ in order to ensure heterogeneity. This leaves us with 75 datasets from the ACIC 2016 competition and 66 datasets from the ACIC 2018 competition.
For CATE estimators, we use the following 7 different type of estimators; S-Learner~\ref{eq:ite-s-learner}, Projected S-Learner~\ref{eq:ite-proj-s-learner}, T-Learner~\ref{eq:ite-t-learner}, X Learner~\ref{eq:ite-x-learner}, DR Learner~\ref{eq:ite-dr}, DML Learner~\ref{eq:ite-dml}, Causal Forest DML Learner~\ref{eq:ite-dml} (Section~\ref{sec:cate-estimators}). For for the various nuisance models associated with these CATE estimators, we use AutoML, specifically FLAML~\cite{Wang_FLAML_A_Fast_2021} to select the best performing nuisance model class as well its hyperparameters. AutoML is able to select much better nuisance models than grid search or random search would for the same amount of compute. Specifically, we have five different type of nuisance models, propensity model ($\hat{\pi}_{W}(X)$), outcome model used in S Learner, Projected S-Learner ($\hat{\mu}(X, W)$), outcome models used in T Learner, DR Learner, X Learner ($\hat{\mu}_{W=0}(X)$, $\hat{\mu}_{W=1}(X)$), and outcome model for DML Learner, Causal Forest DML Learner ($\hat{\mu}(X)$). For each dataset and for each type of nuisance model, we gave FLAML a budget of 30 minutes to select the nuisance model, including its hyperparameter.
Further for the CATE estimators with final models ($f_{\theta}$), i.e., DML Learner, DR Learner, X Learner, \& Projected S Learner, we use a total of 103 different regression models with variation across regression model class and associated hyperparamters. More details regarding them can be found in the Appendix (Section 1). Hence, we have 103 different estimators for each type of CATE estimator with final models, and a single estimator for each type of CATE estimator which do not have final models. Note that Causal Forest DML Learner technically has a final regression model, but we use the default random forest provided in the EconML implementation~\cite{econml}, hence effectively we do not consider variations across its final model.
Finally, for model selection metrics we use the following 9 metrics; Value Score~\ref{eq:value-score}, Value DR Score~\ref{eq:value-dr-score}, Tau T Score~\ref{eq:tau-t-score}, Tau S Score~\ref{eq:tau-s-score}, Tau Match Score~\ref{eq:matching-score}, Tau IPTW Score~\ref{eq:tau-iptw-score}, Tau DR Score~\ref{eq:tau-dr-score}, R Score~\ref{eq:r-score}, and Influence Score~\ref{eq:influence-score} (Section~\ref{sec:eval-metrics}). We follow the same procedure using AutoML for selecting the nuisance models for metrics as we did for selecting the nuisance models of the CATE estimators. Note that the nuisance model selection using AutoML only takes into account the training split of the dataset ($D_{tr}$) and never uses the validation split of the dataset ($D_{val}$), which is only used for computing the metrics. Also, since we can have extreme propensity values on some datasets, therefore, for all of the metrics that are an inverse function of the propensity model (Tau IPTW Score, Tau DR Score, Value DR Score, Influence Score), we compute a propensity clipped version of each metric by discarding data points with extreme propensity score.
Hence, we train a total of \textbf{415 CATE estimators} for each of the \textbf{144 datasets} over 3 random seeds, and evaluate with \textbf{9 different metrics} for model selection. Further, to compare the performance of different metrics, we also compute the oracle (PEHE) score~\ref{eq:pehe} and compare the PEHE of the best estimators chosen by each metric. If there are multiple best estimators for a metric, then we use the mean PEHE of them for comparison. Also, to better understand the finer trends, we report performance on the different dataset groups separately. A summary of our evaluation scheme can be also be found in Figure~\ref{fig:pipeline}. Further details can be found in Appendix Section 1.
% For each dataset, we evaluate the proposed metrics (Section~\ref{sec:eval-metrics}) along with the oracle PEHE metric (eq~\ref{eq:pehe}). Further, we propose a new metric that learns a convex combination of the existing metrics. Let us denote the set of existing metrics as $\{ M_{i} \}_{i=1}^{|M|}$. We learn weights $\{\alpha_{i}\}_{i=1}^{|M|}$ by optimizing the following objective:
% \begin{equation}
% \label{eq:ensemble}
% \min_{\alpha \; | \; \alpha_i > 0 \& \sum_{i}^{} \alpha_i =1 } \sum_{j=1}^{D} \big( l_{\theta}(\hat{T}(X_j)) - (\sum_{i=1}^{|M|} \alpha_i * M_{i}(X_j)) \big)^2
% \end{equation}
% where $l_{\theta}(\hat{T}(X_j))$ denotes the oracle PEHE metric (eq \ref{eq:pehe}) and the constraint $\alpha_i > 0 \; \& \; \sum_{i}^{} \alpha_i =1$ ensures that $\sum_{i=1}^{|M|} \alpha_i * M_{i}(X_j)$ is a convex combination of $M_{i}(X_j)$. Therefore, we learn a convex combination over existing metrics ($\sum_{i=1}^{|M|} \alpha_i * M_{i}(X_j)$) in a supervised fashion with targets as the oracle PEHE metric. We sample data from different datasets ($j=1:D$) while learning this ensemble, as this would help us in being robust to different trends across different datasets.
% We denote this metric as Ensemble Score in our analysis and compare its performance with the other metrics. To train the Ensemble Score, we do not use the realistic datasets (Twins, LaLonde) and use half of the LBIDD datasets. Also, when evaluating the Ensemble Score on the fraction of LBIDD datasets it was trained upon, we use different samples as compared to the samples used for training the Ensemble Score.
% Plugging above trained statistical models into causal estimators (estimating E[Y | do(T=1)], E[Y | do(T=0)], E[Y | do(T=1), W] and E[Y | do(T=0), W] or just their differences)
% For loop over RealCause dataset names
% For loop over samples (seeds) from dataset name (make sure that the samples here are exactly the same as the samples for other iterations of the loop)
% For loop over EconML meta-estimators
% For loop over final models (if applicable) and meta-estimator hyperparameters (model_final)
% For loop over (combinators of) sklearn models that are relevant for a given meta-estimator (regression models for modeling E[Y | T, W] and E[Y | W] and classification models for modeling P(T = 1 | W))
% Loop over hyperparameters in the grid
% Pass in untrained modelsPlug in pre-calculated models (can do this without saving the model by moving this loop to outside the relevant EconML meta-estimators loop)
% Record all factual scores and causal scores (see next section) (and [cached] predictive metric from cross_val_score)
% Do the above, but vary the RealCause knobs (degree of positivity, heterogeneity, and causal effect size) somehow (probably motivated by theory / Vasilis’ intuition)
% \begin{algorithm}[t]
% \footnotesize
% \begin{algorithmic}[1]
% \For{ Dataset $D$ in List of datasets }
% \For{ Seed $S$ in List of random seeds }
% \For{ CATE Estimator $E$ in List of CATE Estimators }
% \For{ Final Model $F$ in List of regression models }
% \For{ Outcome Model $O$ in List of regression models }
% \For{ Propensity Model $P$ in List of classification models }
% \State{Train CATE estimator $E$ on dataset $D$}
% \State{Append $E$ to the list of Trained-Estimators}
% \EndFor
% \EndFor
% \EndFor
% \EndFor
% \EndFor
% \EndFor
% \end{algorithmic}
% \caption{Proposed evaluation methodology}
% \label{alg:bic-erm}
% \end{algorithm}
% \textbf{Selecting nuisance models for the metrics:} When using metrics like RScore, DR Score, etc we have to train the associated nuisance models. Hence, we first need to do model selection amongst the various nuisance models for the purpose of computing these metrics. Since these metrics are going to be computed on a validation set, there are a different strategies available to us.
% Consider each dataset to be paritioned into train and validation split as follows, $D= (D_{tr}, D_{val})$, and the metrics would be computed on $D_{val}$. Consider the task of selecting the optimal regression model ($p(y|x)$) (needed for the computation of metric) from the list of available regression models $\{M_i\}$. We have the following options:
% \begin{itemize}
% \item For each regression model $m \sim \{M\}$, train $m$ on $D_{val}$, and compute its generalization performance considering the $D_{tr}$ split. We select the model with the best generalization performance. [\textbf{Recommended by Schuler et al.}]
% \item For each regression model $m \sim \{M\}$, train $m$ on $D_{tr}$, and compute its generalization performance considering the $D_{val}$ split. We select the model with the best generalization performance.
% \end{itemize}
% So far I do not empirically observe much difference in both the schemes, however, the second scheme seems better to me as we have much more data in the $D_{tr}$ split, hence it would be better to train the nuisance models on that. We can even think of cross-validation performance on the whole dataset $D$ for training these models?
\section{Evaluation Metrics}
\label{sec:eval-metrics}
We use metrics that take as input any CATE estimate $\hat{T}$ and produce a real-valued score. Typically, these scores will themselves depende on training auxiliary nuisance models (e.g. outcome model, propensity model). In order to differentiate the nuisance model estimates going into the score calculation from the nuisance model estimates going into the CATE estimators, we represent nuisance models of the metrics with downward hats ($\check{\mu}$, $\check{\pi}$) and nuisance models of the estimators with upward hats ($\hat{\mu}$, $\hat{\pi}$).
% Further, as we would explain ahead in section~\ref{sec:methodology}, the outcome nuisance models in our study are selected using AutoML and do not vary across estimators, which would imply a constant value for Mu Score across all the estimators.
% \subsubsection{Nuisance model performance based metrics}
% \divyat{Remove these scores with few lines of justification}
% These metrics are based on checking the quality of the outcome nuisance model ($\hat{\mu}_{W=t}(X)$). As stated in Meta Learner section~\ref{sec:cate-estimators}, if $\hat{\mu}_{W=t}(X)$ approximates $\mathbb{E}[ Y | W=t, X]$ well, then the ITE estimates are more likely to be close to the ground truth values.
% \begin{equation}
% \label{eq:mu-score}
% \centering
% \text{Mu Score} = \frac{1}{N} \sum_{i=1}^{N} ( Y_i - \hat{\mu}_{W_i}(X_i) ) ^2
% \end{equation}
% There is another adaptation of this metric, Mu IPTW Score~\cite{van2003unified}, where the residual of the conditional outcome function ($Y - \hat{\mu}_{W=t}(X)$) is scaled using propensity score estimates ($\check{\pi}_{t}(X)$) from a model trained on the validation set. Hence, the propensity estimates are not obtained from the estimator and rather they are a property of the metric.
% \begin{equation}
% \label{eq:mu-iptw-score}
% \centering
% \text{Mu IPTW Score} = \frac{1}{N} \sum_{i=1}^{N} \frac{( Y_i - \hat{\mu}_{W_i}(X_i) ) ^2}{ \check{\pi}_{W_i}(X_i)}
% \end{equation}
\subsection{Value of the decision policy based metrics}
Given CATE estimates ($\hat{T(X)}$) we can derive a decision policy $\hat{d(X)}$, where $\hat{d}(X) = \mathbbm{1}( \hat{T}(X) >0 )$ (assuming positive CATE is more desirable) or $\hat{d}(X) = \mathbbm{1}( \hat{T}(X) < 0 )$ ((assuming negative CATE is more desirable)). These metrics are based on the decision policy ($\hat{d}(X)$) derived from the estimator, where it computes the value of the decision policy ($\mathbb{E}[Y|W=\hat{d}(X)]$). The intuition behind the metric is that if CATE estimates are accurate, then expected outcome under the decision policy derived from CATE estimates should be better.
One such metric, Value Score~\cite{zhao2017uplift}, first converts outcomes into IPW-outcomes~\ref{eq:ite-ipw} and then computes the average outcome under current decision policy.
\begin{equation}
\label{eq:value-score}
\centering
\text{Value Score} = \frac{1}{N}\sum_{i=1}^{N} \frac{ Y_i}{\check{\pi}_{W_i}(X_i)} * \mathbbm{1} (W_i = \hat{d}(X_i)))
\end{equation}
An issue with the Value Score is that it only utilizes a portion of the data where the decision policy and the observed treatment agree. This is handled in the Value DR metric~\cite{athey2017policy}, where we also use the doubly robust outcomes (Equation~\ref{eq:ite-dr}) instead of the IPW outcomes used in Value Score. Note that the nuisance models required for obtaining the doubly robust outcomes are not a property of the estimator, and are trained on validation data.
\begin{equation}
\label{eq:value-dr-score}
\centering
\text{Value DR Score} = \frac{1}{N}\sum_{i=1}^{N} \hat{d}(X_i) * Y^{DR}_{i, W_{i}}
\end{equation}
\subsection{Ground truth CATE approximation based metrics}
Consider the oracle (PEHE) metric~\ref{eq:pehe}, which is not computable from observational data as we cannot compute the ground truth CATE ($T(X)$). There are a class of metrics that aim to approximate the ground truth CATE from validation data. Once we have a reliable approximation ($\tilde{T}(X)$), then we can perform model selection using the following approximate PEHE metric.
\begin{equation}
\label{eq:pehe-approx}
\tilde{l}(\hat{T})= \mathbb{E}_{X}[ ( \hat{T}(X) - \tilde{T}(X) )^2 ]
\end{equation}
One approach to estimate $\tilde{T}(X)$ is by using the matching method~\cite{rolling2013matching}. For each data point $X_i$, we find its nearest neighbour from the opposite treatment data points, i.e., $\tilde{i}= \argmin_{j | W_j \neq W_i } || X_j - X_i ||^2 $. Then we can approximate the ground truth CATE as follows: $\tilde{T}(X_i))= Y_{i} - Y_{\tilde{i}}$.
\begin{equation}
\label{eq:matching-score}
\centering
\text{Matching Score} = \frac{1}{N}\sum_{i=1}^{N} ( \hat{T}(X_i) - (Y_{i} - Y_{\tilde{i}}) )^2
\end{equation}
Another way to approximate ground truth CATE is by scaling the observed outcomes via the IPW technique~\ref{eq:ite-ipw}:
\begin{equation}
\label{eq:tau-iptw-score}
\centering
\text{Tau IPTW Score} = \frac{1}{N}\sum_{i=1}^{N} ( \hat{T}(X_i) - ( Y^{IPW}_{i, 1} - Y^{IPW}_{i, 0} ) )^2
\end{equation}
Similarly, instead of using the IPW-outcomes, we can use the DR-outcomes~\ref{eq:ite-dr}~\cite{saito2020counterfactual}.
%A variant of this score has been proposed in Saito et al.~\cite{saito2020counterfactual}.
\begin{equation}
\label{eq:tau-dr-score}
\centering
\text{Tau DR Score} = \frac{1}{N}\sum_{i=1}^{N} ( \hat{T}(X_i) - ( Y^{DR}_{i, 1} - Y^{DR}_{i, 0} ) )^2
\end{equation}
Further, there is work on using influence functions to approximate the oracle metric~\cite{alaa2019validating}. We first compute a plug-in PEHE estimate ($\tilde{l}(\hat{T})$, eq~\ref{eq:pehe-approx}) using T-Learner~\ref{eq:ite-t-learner} for computing the approximate ground truth CATE, i.e., $\tilde{T}(X) = \check{\mu}_1(X) - \check{\mu}_0(X) $. Then using influence functions, we debias the plug-in PEHE estimate. Following Theorem 2 from their work~\cite{alaa2019validating}, the correction term is defined as follows:
\begin{equation*}
\begin{aligned}
\text{IF-Correction} = & (1-B) (\tilde{T}(X))^2 + BY( \tilde{T}(X) - \hat{T}(X) ) \\
& - (A+1) \tilde{l}(\hat{T}) + (\hat{T}(X))^2
\end{aligned}
\end{equation*}
where $A= W - \check{\pi}(X)$, $B= 2*W*A*C^{-1}$, and $C= \check{\pi}(X)*(1-\check{\pi}(X))$
This is summarized in the equation below, where we add influence correction to the plug-in estimate.
\begin{equation}
\label{eq:influence-score}
\centering
\begin{aligned}
\text{Influence Score} = & \frac{1}{N}\sum_{i=1}^{N} \left[ ( \hat{T}(X_i) - ( \check{\mu}_1(X_i) - \check{\mu}_0(X_i) ) )^2 \right] \\
& + \text{IF-Correction}
\end{aligned}
\end{equation}
Additionally, there are metrics that use Double Machine Learning (DML) to approximate ground truth CATE. RScore~\cite{nie2021quasi}, originally proposed as an estimation technique, was adapted for model selection in the evaluation study of Schuler et al.~\cite{schuler2018comparison}. Using the approximation $\tilde{T}(X)$ via DML method, we know that $\tilde{T}(X)$ would be solution to the regression problem~\ref{eq:ite-dml}. Hence, in order to compute the deviation of estimator's CATE ($\hat{T}(X)$) from $\tilde{T}(X)$, we substitute $\hat{T}(X)$ in the DML regression problem~\ref{eq:ite-dml} and compute its error, as shown below:
\begin{equation}
\label{eq:r-score}
\centering
\text{RScore} = \frac{1}{N}\sum_{i=1}^{N} \big( (Y_i - \check{g}(X_{i})) - \hat{T}(X_i)*( W_i - \check{\pi}(X_i) ) \big)^2
\end{equation}
Finally, we also use two simple approximations for the ground truth CATE using the S-Learner~\ref{eq:ite-s-learner} and T-Learner~\ref{eq:ite-t-learner}, which are similar to the plug-in PEHE estimates~\cite{alaa2019validating}.
\begin{equation}
\small
\label{eq:tau-s-score}
\centering
\text{Tau S Score} = \frac{1}{N}\sum_{i=1}^{N} ( \hat{T}(X_i) - (\hat{\mu}(X_i, W_i=1) - \hat{\mu}(X_i, W_i=0)) )^2
\end{equation}
\begin{equation}
\label{eq:tau-t-score}
\centering
\text{Tau T Score} = \frac{1}{N}\sum_{i=1}^{N} ( \hat{T}(X_i) - (\hat{\mu}_1(X_i) - \hat{\mu}_0(X_i)) )^2
\end{equation}
% RealCause~\cite{neal2020realcause} is a recent approach that uses generative modeling to sample from interventional distributions in non-synthetic / realistic datasets. They model the selection mechanism ($\mathbb{P}(W|X)$) and the output mechanism ($\mathbb{P}(Y|W, X)$) using generative models ($\mathbb{P}_{model}(W|X)$, $\mathbb{P}_{model}(Y|W, X)$), where the covariates $X$ are sampled from the observed realistic dataset. They use normalizing flows~\cite{huang2018neural} for both $\mathbb{P}_{model}(W|X)$, $\mathbb{P}_{model}(Y|W, X)$ as they have been shown to be universal density models, hence they can learn complex distributions. Also, the input features ($X$) into the generative models ($\mathbb{P}_{model}(W|X)$, $\mathbb{P}_{model}(Y|W, X)$) are first transformed using neural networks. Further, instead of concatenating $X, W$, the authors learn separate generative models for different treatments ($\mathbb{P}_{model}(Y|W=0, X)$, $\mathbb{P}_{model}(Y|W=1, X)$), as with concatenation we have the risk that the generative model does not take $W$ into account when the features $X$ are high-dimensional. After training the generative models, we have access to interventional distributions ($\mathbb{P}_{model}(Y|do(W=0), X)$, $\mathbb{P}_{model}(Y|do(W=1), X)$), which provides us with access to both potential outcomes $Y(0), Y(1)$ in realistic datasets.
\section{Problem Formulation}
\label{sec:formulation}
\subsection{Conditional Average Treatment Effect (CATE) estimation and Identification}
We study the problem of causal inference under the potential outcomes framework~\cite{rubin2005causal} with conditional unconfoundedness. The observed data consist of samples of random variables $(Y, W, X)$, where $X$ represents the feature/co-variates/confounders/controls, $W$ represents the treatment assignment, and $Y$ represents the outcome of interest. We consider only discrete binary treatments where $W \in \{0, 1\}$. Hence, we we have two potential outcomes denoted as ($Y(0), Y(1)$) corresponding to the interventions $do(W=0)$ and $do(W=1)$.
The observational data $(X, W, Y)$ are sampled from an unknown distribution $P_{\theta}(X, W, Y(0), Y(1))$. We will denote with $\theta$ the collection of (potentially infinite dimensional) parameters (expected potential outcome, propensity score) corresponding to the nuisance models described below:
\begin{equation*}
\small
\begin{aligned}
&\mu_{0}(x)= \mathbb{E}_{\theta} [Y(0) | X=x] \\
&\mu_{1}(x)= \mathbb{E}_{\theta} [Y(1) | X=x] \\
&\pi(x)= \mathbb{P}_{\theta} [W = 1 | X=x]
\end{aligned}
\end{equation*}
Hence, $\theta$ can be seen as a collection of $\mu_0, \mu_1, \pi, \eta$, where $\eta$ corresponds to $\mathbb{P}_{\theta}(X)$. Our target of inference is the conditional average treatment effect (CATE).
\begin{equation}
\label{eq:ite}
\text{CATE:} \quad T(x)= \mathbb{E}_{\theta}[ Y(1) - Y(0) | X = x ]
\end{equation}
% Note that given samples from the observational data ($\{ (X_i, W_i, Y_i) \}_{i=1}^{N}$), the empirical estimate can simply be written as follows: $\frac{1}{N} \sum_{i=1}^{N} Y_{i}(1) - Y_{i}(0)$.
Also, observe that we can get the objective of average treatment effect estimation (ATE) from CATE as follows: ATE = $\mathbb{E}_{X}[ \mathbb{E}_{\theta}[ Y(1) - Y(0) | X = x ] ]$, which is basically expected value of CATE. Throughout our analysis we would be concerned with the CATE objective and not focus on ATE.
Since in general the estimation of treatment effects is not identifiable from observational data, we make the following standard assumptions that ensure identifiability~\cite{peters2017elements}; a) \textbf{Consistency:} $\;\; Y= W \cdot Y(1) + (1-W) \cdot Y(0) $ ; b) \textbf{Exchangeability:} $\;\; Y(0), Y(1) \; \indep \; W \; | \; X $ ; c) \textbf{Overlap:} $\;\; 0 < \pi(x) < 1 \;\; \forall x \in X$.
Under these assumptions, CATE is identified as:
$$ T(x) = \mathbb{E}_{\theta}\left[Y | W=1, X=x\right] - \mathbb{E}_{\theta}\left[Y | W=0, X=x\right]$$
The idealized measure of accuracy that we will use throughout for any CATE estimate ($\hat{T}$) is the mean squared error, also termed as precision of heterogeneous effects (PEHE)~\cite{hill2011bayesian}.
\begin{equation}
\label{eq:pehe}
\centering
l_{\theta}(\hat{T})= \mathbb{E}_{X}[ ( \hat{T}(X) - T(X) )^2 ]
\end{equation}
Note that if we had access to hypothetical counterfactual data (i.e. observed both $Y(0), Y(1)$ for every sample), then up to a constant (i.e. estimate $\hat{T}$-independent) additive term, the PEHE is equivalent to the hypothetical loss:
\begin{align*}
\mathbb{E}_{X}[ ( \hat{T}(X) - (Y(1) -Y(0)) )^2 ]
\end{align*}
Thus one could have used this idealized loss for model selection. Hence, the main difficulty stems from not observing both the potential outcomes.
\subsection{Model selection between multiple CATE estimators}
The problem of model selection is defined as follows: given a set of CATE estimates $\{\hat{T}_{1}, .., \hat{T}_{M}\}$ from estimators $\{E_{1}, .., E_{M}\}$, we want to select the best estimator. $E_{m^{*}}$ s.t. $ l_{\theta}(\hat{T}_{m^{*}}) = \min_{i} l_{\theta}(\hat{T}_{i})$. However, the main challenge is that we cannot compute PEHE (eq~\ref{eq:pehe}) using only observation data, as we either observe $Y_{i}(0)$ or $Y_{i}(1)$ for any individual ($i$). Hence, the ground truth effect ($T(X)$) cannot be computed using only observational data, which implies we cannot use methods like cross-validation, a widely used procedure for model selection in machine learning, where we split the observational data into training and validation splits, and evaluate the performance of the model on the validation split as we know the ground truth labels. Hence, our goal is to design new validation/evaluation metrics ($M(\hat{T})$) which does not rely on counterfactual data and resemble the oracle metric (PEHE~\ref{eq:pehe}) in performance.
\documentclass{article}
\usepackage[preprint]{style_file}
% \documentclass{article}
% \usepackage{aistats2023}
% If your paper is accepted, change the options for the package
% aistats2023 as follows:
%
%\usepackage[accepted]{aistats2023}
%
% This option will print headings for the title of your paper and
% headings for the authors names, plus a copyright note at the end of
% the first column of the first page.
% If you set papersize explicitly, activate the following three lines:
%\special{papersize = 8.5in, 11in}
%\setlength{\pdfpageheight}{11in}
%\setlength{\pdfpagewidth}{8.5in}
% If you use natbib package, activate the following three lines:
% \usepackage[round]{natbib}
% \renewcommand{\bibname}{References}
% \renewcommand{\bibsection}{\subsubsection*{\bibname}}
% If you use BibTeX in apalike style, activate the following line:
%\bibliographystyle{apalike}
% \usepackage[verbose]{backref}
% \backrefsetup{verbose=false}
%Customized Packages imported for paper
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{enumitem}
%Figures and Tables
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{multirow}
\usepackage{booktabs} % professional-quality tables
\usepackage{wrapfig}
\usepackage{xcolor} % colors
\usepackage{microtype} % microtypography
%Math
\usepackage{bbm}
\usepackage{mathtools}
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
%Customized new commands declarations
\newcommand{\vecx}{\mathbf{x}}
\newcommand{\vecz}{\mathbf{z}}
\newcommand{\indep}{\perp \!\!\! \perp}
\DeclarePairedDelimiter\ceil{\lceil}{\rceil}
\DeclarePairedDelimiter\floor{\lfloor}{\rfloor}
\DeclareMathOperator*{\argmin}{arg\,min}
\newcommand{\divyat}[1]{\textcolor{brown}{[#1]}}
\usepackage{color-edits}
%\usepackage[suppress]{color-edits}
\addauthor{vs}{red}
\addauthor{dm}{blue}
\title{Empirical Analysis of Model Selection for Heterogenous Causal Effect Estimation}
\author{%
Divyat Mahajan\\
Mila, Université de Montréal\\
\texttt{divyat.mahajan@mila.quebec} \\
\And
Ioannis Mitliagkas\\
Mila, Université de Montréal\\
\texttt{ioannis@mila.quebec}\\
\And
Brady Neal\thanks{Equal Contribution}\\
Mila, Université de Montréal\\
Dataiku\\
\texttt{bradyneal11@gmail.com}\\
\And
Vasilis Syrgkanis\footnotemark[1] \\
Stanford University\\
\texttt{vsyrgk@stanford.edu}\\
}
\begin{document}
% If your paper is accepted and the title of your paper is very long,
% the style will print as headings an error message. Use the following
% command to supply a shorter title of your paper so that it can be
% used as headings.
%
%\runningtitle{I use this title instead because the last one was very long}
% If your paper is accepted and the number of authors is large, the
% style will print as headings an error message. Use the following
% command to supply a shorter version of the authors names so that
% they can be used as headings (for example, use only the surnames)
%
%\runningauthor{Surname 1, Surname 2, Surname 3, ...., Surname n}
% \twocolumn[
% \aistatstitle{Empirical Analysis of Model Selection for Heterogenous Causal Effect Estimation}
% \aistatsauthor{ Author 1 \And Author 2 \And Author 3 }
% \aistatsaddress{ Institution 1 \And Institution 2 \And Institution 3 }
% ]
\maketitle
\begin{abstract}
We study the problem of model selection in causal inference, specifically for the case of conditional average treatment effect (CATE) estimation under binary treatments. Unlike model selection in machine learning, we cannot use the technique of cross-validation here as we do not observe the counterfactual potential outcome for any data point. Hence, we need to design model selection techniques that do not explicitly rely on counterfactual data. As an alternative to the cross-validation, there have been a variety of proxy metrics proposed in the literature, that depend on auxiliary nuisance models also estimated from the data (propensity score model, outcome regression model). However, the effectiveness of these metrics has only been studied on synthetic datasets as we can observe the counterfactual data for them. We conduct an extensive empirical analysis to judge the performance of these metrics, where we utilize the latest advances in generative modeling to incorporate multiple realistic datasets. We evaluate 9 metrics on 144 datasets for selecting between 415 estimators per dataset, including datasets that closely mimic real world datasets. Further, we use the latest techniques from AutoML to ensure consistent hyperparameter selection for nuisance models for a fair comparison across metrics.
% \vsdelete{Finally, we also present a technique for constructing an ensemble of estimators using the metrics.}
%Further, some recent works propose using influence functions to correct for the bias associated with using these metrics. We analyze these metrics on realistic datasets by using the latest advances in generative modeling for causal inference. Generative modeling of realistic datasets allows us to infer the counterfactual potential outcomes, effectively making them semi-synthetic. We conduct an extensive empirical analysis to judge the performance of these metrics using multiple realistic datasets and estimators.
%Our results show that the metrics which incorporate bias-correction using doubly robust functions and influence functions are better than metrics that rely solely on the performance of nuisance models. Further, we find there is no metric that consistently outperforms the other metrics on all the datasets. This motivates our approach of learning an ensemble of the existing metrics, which achieves more consistent results as compared to the other metrics.
% A3:
% Unlike in machine learning, model selection in causal inference is fundamentally hard because we do not observe counterfactuals. People have proposed several causally-conscious scores to use for model selection. We evaluate 10 scores on 33 datasets for selecting between 200 estimators per dataset, including datasets that closely mimic real world datasets. None of these scores dominate the others. We present a new score that dominates all others and recommend that others use it for model selection in causal inference.
\end{abstract}
\input{intro.tex}
\input{problem_statement.tex}
\input{estimators_review.tex}
\input{metrics_review.tex}
\input{methodology.tex}
\input{logs.tex}
\section{Conclusion}
Our work shows the importance of a consistent evaluation across a wide range of datasets for CATE model selection metrics, which leads to more detailed trends as opposed to prior works. We also see that no score completely dominates the other scores across all the datasets, which suggests opportunities for designing better model selection metrics.
% \clearpage
\bibliographystyle{apalike}
\bibliography{sample_paper}
\newpage
\appendix
\input{supplement.tex}
% \newpage
% \input{AISTATS2023PaperPack/supplement.tex}
% \subsubsection*{References}
% References follow the acknowledgements. Use an unnumbered third level
% heading for the references section.
% \begin{thebibliography}{}
% \setlength{\itemindent}{-\leftmargin}
% \makeatletter\renewcommand{\@biblabel}[1]{}\makeatother
% \bibitem{} J.~Alspector, B.~Gupta, and R.~B.~Allen (1989).
% \newblock Performance of a stochastic learning microchip.
% \newblock In D. S. Touretzky (ed.),
% \textit{Advances in Neural Information Processing Systems 1}, 748--760.
% San Mateo, Calif.: Morgan Kaufmann.
% \bibitem{} F.~Rosenblatt (1962).
% \newblock \textit{Principles of Neurodynamics.}
% \newblock Washington, D.C.: Spartan Books.
% \bibitem{} G.~Tesauro (1989).
% \newblock Neurogammon wins computer Olympiad.
% \newblock \textit{Neural Computation} \textbf{1}(3):321--323.
% \end{thebibliography}
\end{document}
% \documentclass[twoside]{article}
% \usepackage{aistats2023}
% % If your paper is accepted, change the options for the package
% % aistats2023 as follows:
% %
% %\usepackage[accepted]{aistats2023}
% %
% % This option will print headings for the title of your paper and
% % headings for the authors names, plus a copyright note at the end of
% % the first column of the first page.
% % If you set papersize explicitly, activate the following three lines:
% %\special{papersize = 8.5in, 11in}
% %\setlength{\pdfpageheight}{11in}
% %\setlength{\pdfpagewidth}{8.5in}
% % If you use natbib package, activate the following three lines:
% %\usepackage[round]{natbib}
% %\renewcommand{\bibname}{References}
% %\renewcommand{\bibsection}{\subsubsection*{\bibname}}
% % If you use BibTeX in apalike style, activate the following line:
% %\bibliographystyle{apalike}
% %Customized Packages imported for paper
% \usepackage{hyperref} % hyperlinks
% \usepackage{url} % simple URL typesetting
% \usepackage{enumitem}
% %Figures and Tables
% \usepackage{graphicx}
% \usepackage{multirow}
% \usepackage{booktabs} % professional-quality tables
% \usepackage{wrapfig}
% \usepackage{xcolor} % colors
% \usepackage{microtype} % microtypography
% %Math
% \usepackage{bbm}
% \usepackage{mathtools}
% \usepackage{amsfonts} % blackboard math symbols
% \usepackage{nicefrac} % compact symbols for 1/2, etc.
% %Customized new commands declarations
% \newcommand{\vecx}{\mathbf{x}}
% \newcommand{\vecz}{\mathbf{z}}
% \newcommand{\indep}{\perp \!\!\! \perp}
% \DeclarePairedDelimiter\ceil{\lceil}{\rceil}
% \DeclarePairedDelimiter\floor{\lfloor}{\rfloor}
% \DeclareMathOperator*{\argmin}{arg\,min}
% \newcommand{\divyat}[1]{\textcolor{brown}{[#1]}}
% \usepackage{color-edits}
% %\usepackage[suppress]{color-edits}
% \addauthor{vs}{red}
% \addauthor{dm}{blue}
% \begin{document}
% % If your paper is accepted and the title of your paper is very long,
% % the style will print as headings an error message. Use the following
% % command to supply a shorter title of your paper so that it can be
% % used as headings.
% %
% %\runningtitle{I use this title instead because the last one was very long}
% % If your paper is accepted and the number of authors is large, the
% % style will print as headings an error message. Use the following
% % command to supply a shorter version of the authors names so that
% % they can be used as headings (for example, use only the surnames)
% %
% %\runningauthor{Surname 1, Surname 2, Surname 3, ...., Surname n}
% % Supplementary material: To improve readability, you must use a single-column format for the supplementary material.
% \onecolumn
% \aistatstitle{Empirical Analysis of Model Selection for Heterogenous Causal Effect Estimation
% : Supplementary Materials}
% \aistatsauthor{ Author 1 \And Author 2 \And Author 3 }
% \aistatsaddress{ Institution 1 \And Institution 2 \And Institution 3 }
\section{Evaluation Setup Details}
\label{supp:exp-details}
\subsection{Dataset Statistics}
\begin{table}[th]
\centering
\begin{tabular}{llrrrr}
\toprule
Dataset Group & ACIC 2016 & LaLondeCPS & LaLonde PSID & ACIC 2018 & TWINS \\
\midrule
Training Data Size & 3841 & 8089 & 1338 & 4000 & 5992 \\
Evaluation Data Size & 961 & 1618 & 268 & 1000 & 1198 \\
Covariate Dimension & 82 & 8 & 8 & 117 & 75 \\
\bottomrule
\end{tabular}
\caption{Statistics for the various datasets used in our analysis}
\label{tab:dataset_statistics}
\end{table}
We provide details regarding the training and evaluation sample size, along with the covariate dimensions in Table~\ref{tab:dataset_statistics}.
\subsection{Final Regression Models for CATE Estimators}
We use the following regression model classes, along with the hyperparameter range considered for each class specified below. We used sklearn for implementing all the regression models and use the same notation from sklearn for representing the regresssion model class and the correpsonding hyperparameter names. For representing the hyperparameter ranges, we use the numpy logspace function to sample from a range (unless specified otherwise), with the syntax (start value, end value, total entries).
\begin{itemize}
\item Linear Regression; No Hyperparameter
\item Linear Regression; Degree 2 polynomial features; No interaction terms; No Hypereparameter
\item Linear Regression; Degree 2 polynomial features; Interaction term; No Hyperparameter
\item Ridge Regression; Hyperparameters ($\alpha$): $np.logspace(-4, 5, 10)$
\item Kernel Ridge Regression; Hyperparameters ($\alpha$): $np.logspace(-4, 5, 10)$
\item Lasso Regression; Hyperparameters ($\alpha$): $np.logspace(-4, 5, 10)$
\item Elastic Net Regression; Hyperparameters ($\alpha$): $np.logspace(-4, 5, 10)$
\item SVR; Sigmoid Kernel; Hyperparameters ($C$): $np.logspace(-4, 5, 10)$
\item SVR; RBF Kernel; Hyperparameters ($C$): $np.logspace(-4, 5, 10)$
\item Linear SVR; Hyperparameters ($C$): $np.logspace(-4, 5, 10)$
\item Decision Tree: Hyperparameters ($max-depth$): $list(range(2, 11)) + [None]$
\item Random Forest: Hyperparameters ($max-depth$): $list(range(2, 11)) + [None]$
\item Gradient Boosting: Hyperparameters ($max-depth$): $list(range(2, 11)) + [None]$
\end{itemize}
\subsection{Propensity clipping for metrics}
As mentioned in the main text, for metrics that involve inverse of propensity score, we compute a propensity clipped version for those metrics where we remove the data points with extreme propensity values. We decide on a threshold $\alpha$ and only retain data points that have propensity values ($\pi$) in the following range: $\alpha < \pi < 1 - \alpha$. For the experiments in the main paper, we used $\alpha=0.1$, which might have been a little harsh on the LaLonde datasets since they have severe imbalance w.r.t control vs treatment population. Therefore, we also provide results with $\alpha=0.01$ in the Appendix Section~\ref{supp:small-prop-resutls}.
\subsection{Performance comparison of different metrics}
Conditioned on a particular dataset and seed, let us denote the list of estimators as $E$ and the value of the metric $M$ on an estimator $e$ as $M(e)$. We denote the PEHE of an estimator $e$ as $O(e)$, consistent with the notation that the oracle metric ($O$) computes the PEHE to select estimators. We use $E^{*}_{M}$ to denote the set of estimators that are optimal w.r.t metric $M$, i.e., $E^{*}_{M}= \argmin_{e \in E} M(e)$. Further, $M(E)$ denotes the array of the values of metric $M$ on all the estimators $e \in E$. \\
\textbf{Absolute/Normalized PEHE:}
We report mean PEHE of the best estimators w.r.t metric $M$, i.e., PEHE($E^{*}_{M}$)= $ \frac{1}{|E^{*}_{M}|} * \sum_{e \in E^{*}_{M}}^{} O(e)$.
For computing the normalized PEHE, we also find the set of best estimators ($E^{\dagger}$) w.r.t the oracle metric ($O$), i.e., $E^{\dagger}= \argmin_{e \in E} O(e)$ and compute the percent change of PEHE($E^{*}_{M}$) from PEHE($E^{\dagger}$).
Hence, Normalized-PEHE($E^{*}_{M}$)= $\;$ [ PEHE($E^{*}_{M}$) - PEHE($E^{\dagger}$) ] / PEHE($E^{\dagger}$) \\
\textbf{Win Rate:}
We compare the absolute PEHE of the best estimators w.r.t metric $M$, i.e., PEHE($E^{*}_{M}$) against the absolute PEHE of the best estimators of other metrics $M^{'}$, i.e, PEHE($E^{*}_{M'}$); and define the scenario PEHE($E^{*}_{M}$) $<$ PEHE($E^{*}_{M'}$) as a win since lower value of PEHE is better.
Hence, win rate denotes the average number of wins for the metric $M$ or the average number of metrics $M^{'}$ worse than metric $M$ in terms of absolute PEHE of their best estimators. Note that in the event of ties, i.e., PEHE($E^{*}_{M}$) $=$ PEHE($E^{*}_{M'}$), we declare both $M$ and $M^{'}$ as the winner. \\
\textbf{Rank Correlation:}
For a metric $M$, we compute the spearman rank correlation between $M(E)$ and $O(E)$, which acts as a measure for the correlation between the rankings assigned to different estimators $e \in E$ by the metric $M$ and the oracle metric $O$.
\section{Additional Results}
\label{supp:extra-results}
\begin{table*}[h!]
\centering
\small
\begin{tabular}{lrrrrr}
\toprule
Dataset Group & ACIC 2016 & LaLonde CPS & LaLonde PSID & ACIC 2018 & TWINS \\
\midrule
Value Score & 916271.97 & 13314.36 & 62014.10 & 8.57e+05 & 112959.82 \\
Value DR Score & 3367.19 & 10578.04 & 192449.67 & 2.13e+04 & 115372.11 \\
Value DR Clip Prop Score & 2973.85 & 21383976.18 & 205376.47 & 1.34e+05 & 90003.93 \\
Tau T Score & \textbf{1.05} & \textbf{7135.93} & \textbf{48716.10} & 5.69e+01 & 0.12 \\
Tau S Score & \textbf{1.07} & 9553.56 & 61467.51 & \textbf{1.18e+00} & 0.11 \\
Tau Match Score & 2.17 & \textbf{7206.48} & \textbf{49003.09} & 9.58e+01 & 0.11 \\
Tau IPTW Score & 2.17 & 8514.23 & \textbf{48287.59} & 1.52e+08 & \textbf{0.10} \\
Tau IPTW Clip Prop Score & 2.05 & 12064.69 & 76990.28 & 1.09e+04 & \textbf{0.10} \\
Tau DR Score & \textbf{1.03} & \textbf{7044.04} & \textbf{49548.59} & 1.52e+08 & 0.13 \\
Tau DR Clip Prop Score & \textbf{1.03} & 12565.76 & 77138.89 & 6.57e+01 & 0.14 \\
Influence Score & 68276.61 & 14035.74 & 65098.26 & 3.36e+08 & 0.17 \\
Influence Clip Prop Score & 4.97 & 9542.50 & 62808.15 & 2.07e+02 & 0.17 \\
R Score & \textbf{1.08} & 9077.27 & 57093.07 & 5.34e+01 & 0.12 \\
\midrule
Oracle & 0.80 & 6570.28 & 41284.94 & 6.00e-01 & 0.08 \\
\bottomrule
\end{tabular}
\caption{Absolute PEHE: PEHE of the best estimators chosen by each metric; results report the mean across 3 seeds and also across dataset instances for the ACIC 2016 \& ACIC 2018 datasets. \textbf{Lower value is better.}}
\label{tab:absolute-pehe}
\end{table*}
\subsection{Absolute PEHE Table}
Table~\ref{tab:absolute-pehe} is analogous to the Normalised PEHE table in the main paper, instead we provide the aboslute values for the PEHE of the best chosen estimators instead of normalizing it by the PEHE of the oracle score. Since the same normalization scheme is applied to all the scores, the trends do not change if we compare the absolute PEHE vs normalized PEHE, and all the conclusions we listed in the main paper for the normalized PEHE table hold here as well.
% \subsection{Chosen estimator vs inherent bias of metric}
\begin{table*}[th]
\centering
\small
\begin{tabular}{lrrrrr}
\toprule
Dataset Group & ACIC 2016 & LaLonde CPS & LaLonde PSID & ACIC 2018 & TWINS \\
\midrule
Value Score & 1147378.75 & 1.03 & 0.50 & 1.43e+06 & 1355516.83 \\
Value Dr Score & 4215.48 & 0.61 & 3.66 & 3.57e+04 & 1384464.32 \\
Value DR Clip Prop Score & 4215.48 & 0.45 & 3.67 & 4.08e+06 & 1384464.32 \\
Tau T Score & \textbf{0.32} & \textbf{0.09} & \textbf{0.18} & 9.43e+01 & 0.46 \\
Tau S Score & \textbf{0.34} & 0.45 & 0.49 & \textbf{9.80e-01} & 0.32 \\
Tau Match Score & 1.71 & \textbf{0.10} & \textbf{0.19} & 1.59e+02 & 0.35 \\
Tau IPTW Score & 1.72 & 0.30 & \textbf{0.17} & 2.54e+08 & \textbf{0.20} \\
Tau IPTW Clip Prop Score & 1.64 & 0.38 & \textbf{0.18} & 2.92e+03 & \textbf{0.20} \\
Tau DR Score & \textbf{0.29} & \textbf{0.07} & \textbf{0.20} & 2.54e+08 & 0.61 \\
Tau DR Clip Prop Score & \textbf{0.31} & 0.26 & 0.27 & 1.44e+02 & 0.61 \\
Influence Score & 85496.76 & 1.14 & 0.58 & 5.62e+08 & 0.99 \\
Influence Clip Prop Score & 5.16 & 0.64 & 0.55 & 3.16e+02 & 0.99 \\
R Score & \textbf{0.35} & 0.38 & 0.38 & 8.83e+01 & 0.46 \\
\bottomrule
\end{tabular}
\caption{Normalized PEHE with $\alpha= 0.01$: Percentage difference of the PEHE of the best estimators chosen by each metric from the oracle PEHE; results report the mean across 3 seeds and also across dataset instances for the ACIC 2016 \& ACIC 2018 datasets. \textbf{Lower value is better.}}
\label{tab:normalized-pehe-low-prop}
\end{table*}
\begin{table*}[th]
\centering
\small
\begin{tabular}{lrrrrr}
\toprule
Dataset Group & ACIC 2016 & LaLonde CPS & LaLonde PSID & ACIC 2018 & TWINS \\
\midrule
Value Score & 0.05 & \textbf{0.67} & 0.00 & 0.02 & 0.00 \\
Value DR Score & 0.11 & 0.00 & 0.00 & 0.03 & \textbf{0.33} \\
Value DR Clip Prop Score & 0.11 & 0.00 & 0.00 & 0.06 & \textbf{0.33} \\
Tau T Score & 0.49 & \textbf{0.67} & \textbf{ 0.67} & 0.17 & 0.00 \\
Tau S score & 0.27 & 0.00 & 0.00 & \textbf{0.50} & \textbf{ 0.33} \\
Tau Match Score & 0.06 & \textbf{0.67} & \textbf{ 0.67 }& 0.06 & \textbf{0.33}\\
Tau IPTW Score & 0.10 & 0.33 & 0.33 & 0.06 & \textbf{0.33 }\\
Tau IPTW Clip Prop Score & 0.10 & 0.00 & 0.33 & 0.04 & \textbf{0.33} \\
Tau DR Score & 0.51 & 0.33 & 0.00 & 0.26 & 0.00 \\
Tau DR Clip Prop Score & \textbf{0.52} & 0.00 & 0.00 & 0.26 & 0.00 \\
Influence Score & 0.01 & 0.00 & 0.00 & 0.03 & 0.00 \\
Influence Clip Prop Score & 0.01 & 0.00 & 0.00 & 0.02 & 0.00 \\
R Score & 0.25 & 0.00 & 0.00 & 0.06 & 0.00 \\
\bottomrule
\end{tabular}
\caption{Win Rate with $\alpha=0.01$: Percentage of times the best estimators chosen by each metric dominate the best estimators chosen by other metrics in terms of PEHE; results report the mean across 3 seeds and also across dataset instances for the ACIC 2016 \& ACIC 2018 datasets. \textbf{Higher value is better}.}
\label{tab:win-rate-low-prop}
\end{table*}
\subsection{Smaller propensity clipping for metrics}
\label{supp:small-prop-resutls}
We now present results with lower propensity clipping ($\alpha=0.01$) and compare them against the results in the main paper, which had propensity clipping of $\alpha=0.1$. Note that the results for the metric that do not involve propensity clipping would not change, except in the win rate table~\ref{tab:win-rate-low-prop} as it involves comparing against the other metrics.
We find that reducing propensity clipping helps for the LaLonde datasets as they had the worst treatment vs control population imbalance; the Normalized PEHE (Table~\ref{tab:normalized-pehe-low-prop}) for Tau IPTW Clip Score and Tau DR Clip Score have significantly improved. Tau DR Clip Score improves from 0.91 to 0.26 on LaLonde CPS dataset, also it improves from 0.87 to 0.27 on LaLonde PSID dataset. Similarly, Tau IPTW Clip Score improves from 0.84 to 0.38 on LaLonde CPS dataset, also it improves from 0.86 to 0.18 on LaLonde PSID dataset. Further, the Rank Correlation (Table~\ref{tab:rank-correlation-low-prop}) has also shows significant improvement on the LaLonde datasets. Tau DR Clip Score improves from -0.27 to 0.73 on LaLonde PSID dataset. Similarly, Tau IPTW Clip Score improves from -0.16 to 0.80 on LaLonde PSID dataset.
\begin{table*}[th]
\centering
\small
\begin{tabular}{lrrrrr}
\toprule
Dataset Group & ACIC 2016 & LaLonde CPS & LaLonde PSID & ACIC 2018 & TWINS \\
\midrule
Value Score & 0.58 & 0.42 & 0.11 & 0.31 & 0.49 \\
Value DR Score & 0.80 & 0.37 & -0.05 & 0.50 & 0.39 \\
Value DR Clip Prop Score & 0.80 & 0.17 & -0.12 & 0.52 & 0.39 \\
Tau T Score & \textbf{0.98} & \textbf{0.85} & 0.91 & 0.84 & 0.72 \\
Tau S Score & \textbf{0.98} & 0.39 & -0.47 & \textbf{0.94} & 0.85 \\
Tau Match Score & 0.71 & \textbf{0.85} & 0.83 & 0.75 & 0.85 \\
Tau IPTW Score & 0.81 & 0.78 & \textbf{0.92} & 0.68 & \textbf{0.88} \\
Tau IPTW Clip Prop Score & 0.81 & 0.67 & 0.80 & 0.70 & \textbf{0.88} \\
Tau DR Score & \textbf{0.98} & \textbf{0.85} & 0.90 & 0.85 & 0.85 \\
Tau DR Clip Prop Score & \textbf{0.98} & 0.61 & 0.73 & 0.85 & 0.85 \\
Influence Score & 0.34 & 0.09 & -0.60 & 0.42 & 0.80 \\
Influence Clip Prop Score & 0.34 & 0.41 & -0.52 & 0.42 & 0.80 \\
R Score & 0.96 & 0.54 & 0.17 & 0.86 & 0.79 \\
\bottomrule
\end{tabular}
\caption{Spearman Rank correlation ($\alpha= 0.01$) between evaluation metrics and oracle metric; results report the mean across 3 seeds and also across dataset instances for the ACIC 2016 \& ACIC 2018 datasets. \textbf{Higher value is better.}}
\label{tab:rank-correlation-low-prop}
\end{table*}
\end{document}
\begin{thebibliography}{}
\bibitem[Alaa and Van Der~Schaar, 2019]{alaa2019validating}
Alaa, A. and Van Der~Schaar, M. (2019).
\newblock Validating causal inference models via influence functions.
\newblock In {\em International Conference on Machine Learning}, pages
191--201. PMLR.
\bibitem[Athey and Wager, 2017]{athey2017policy}
Athey, S. and Wager, S. (2017).
\newblock Policy learning with observational data.
\newblock {\em arXiv preprint arXiv:1702.02896}.
\bibitem[Battocchi et~al., 2019]{econml}
Battocchi, K., Dillon, E., Hei, M., Lewis, G., Oka, P., Oprescu, M., and
Syrgkanis, V. (2019).
\newblock {EconML}: {A Python Package for ML-Based Heterogeneous Treatment
Effects Estimation}.
\newblock https://github.com/microsoft/EconML.
\newblock Version 0.x.
\bibitem[Bengio et~al., 2013]{bengio2013representation}
Bengio, Y., Courville, A., and Vincent, P. (2013).
\newblock Representation learning: A review and new perspectives.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
35(8):1798--1828.
\bibitem[Chernozhukov et~al., 2016]{chernozhukov2016double}
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey,
W., and Robins, J. (2016).
\newblock Double/debiased machine learning for treatment and causal parameters.
\newblock {\em arXiv preprint arXiv:1608.00060}.
\bibitem[Dorie et~al., 2019]{dorie2019automated}
Dorie, V., Hill, J., Shalit, U., Scott, M., and Cervone, D. (2019).
\newblock Automated versus do-it-yourself methods for causal inference: Lessons
learned from a data analysis competition.
\newblock {\em Statistical Science}, 34(1):43--68.
\bibitem[Foster and Syrgkanis, 2019]{foster2019orthogonal}
Foster, D.~J. and Syrgkanis, V. (2019).
\newblock Orthogonal statistical learning.
\newblock {\em arXiv preprint arXiv:1901.09036}.
\bibitem[Foster et~al., 2011]{foster2011subgroup}
Foster, J.~C., Taylor, J.~M., and Ruberg, S.~J. (2011).
\newblock Subgroup identification from randomized clinical trial data.
\newblock {\em Statistics in medicine}, 30(24):2867--2880.
\bibitem[Hartford et~al., 2017]{hartford2017deep}
Hartford, J., Lewis, G., Leyton-Brown, K., and Taddy, M. (2017).
\newblock Deep iv: A flexible approach for counterfactual prediction.
\newblock In {\em International Conference on Machine Learning}, pages
1414--1423. PMLR.
\bibitem[Hill, 2011]{hill2011bayesian}
Hill, J.~L. (2011).
\newblock Bayesian nonparametric modeling for causal inference.
\newblock {\em Journal of Computational and Graphical Statistics},
20(1):217--240.
\bibitem[Holland, 1986]{holland1986statistics}
Holland, P.~W. (1986).
\newblock Statistics and causal inference.
\newblock {\em Journal of the American statistical Association},
81(396):945--960.
\bibitem[Huang et~al., 2018]{huang2018neural}
Huang, C.-W., Krueger, D., Lacoste, A., and Courville, A. (2018).
\newblock Neural autoregressive flows.
\newblock In {\em International Conference on Machine Learning}, pages
2078--2087. PMLR.
\bibitem[Kennedy, 2020]{kennedy2020optimal}
Kennedy, E.~H. (2020).
\newblock Optimal doubly robust estimation of heterogeneous causal effects.
\newblock {\em arXiv preprint arXiv:2004.14497}.
\bibitem[K{\"u}nzel et~al., 2019]{kunzel2019metalearners}
K{\"u}nzel, S.~R., Sekhon, J.~S., Bickel, P.~J., and Yu, B. (2019).
\newblock Metalearners for estimating heterogeneous treatment effects using
machine learning.
\newblock {\em Proceedings of the national academy of sciences},
116(10):4156--4165.
\bibitem[LaLonde, 1986]{lalonde1986evaluating}
LaLonde, R.~J. (1986).
\newblock Evaluating the econometric evaluations of training programs with
experimental data.
\newblock {\em The American economic review}, pages 604--620.
\bibitem[Louizos et~al., 2017]{louizos2017causal}
Louizos, C., Shalit, U., Mooij, J.~M., Sontag, D., Zemel, R., and Welling, M.
(2017).
\newblock Causal effect inference with deep latent-variable models.
\newblock {\em Advances in neural information processing systems}, 30.
\bibitem[Neal et~al., 2020]{neal2020realcause}
Neal, B., Huang, C.-W., and Raghupathi, S. (2020).
\newblock Realcause: Realistic causal inference benchmarking.
\newblock {\em arXiv preprint arXiv:2011.15007}.
\bibitem[Nie and Wager, 2021]{nie2021quasi}
Nie, X. and Wager, S. (2021).
\newblock Quasi-oracle estimation of heterogeneous treatment effects.
\newblock {\em Biometrika}, 108(2):299--319.
\bibitem[Peters et~al., 2017]{peters2017elements}
Peters, J., Janzing, D., and Sch{\"o}lkopf, B. (2017).
\newblock {\em Elements of causal inference: foundations and learning
algorithms}.
\newblock The MIT Press.
\bibitem[Rolling and Yang, 2013]{rolling2013matching}
Rolling, C. and Yang, Y. (2013).
\newblock Model selection for estimating treatment effects.
\newblock {\em Journal of the Royal Statistical Society: Series B
(StatisticalMethodology)}, 76:749--769.
\bibitem[Rubin, 2005]{rubin2005causal}
Rubin, D.~B. (2005).
\newblock Causal inference using potential outcomes: Design, modeling,
decisions.
\newblock {\em Journal of the American Statistical Association},
100(469):322--331.
\bibitem[Saito and Yasui, 2020]{saito2020counterfactual}
Saito, Y. and Yasui, S. (2020).
\newblock Counterfactual cross-validation: Stable model selection procedure for
causal inference models.
\newblock In {\em International Conference on Machine Learning}, pages
8398--8407. PMLR.
\bibitem[Schuler et~al., 2018]{schuler2018comparison}
Schuler, A., Baiocchi, M., Tibshirani, R., and Shah, N. (2018).
\newblock A comparison of methods for model selection when estimating
individual treatment effects.
\newblock {\em arXiv preprint arXiv:1804.05146}.
\bibitem[Segal et~al., 2012]{Segal2012UnderstandingHO}
Segal, J.~B., Weiss, C.~O., and Varadhan, R. (2012).
\newblock Understanding heterogeneity of treatment effects in pragmatic trials.
\bibitem[Shi et~al., 2019]{shi2019adapting}
Shi, C., Blei, D., and Veitch, V. (2019).
\newblock Adapting neural networks for the estimation of treatment effects.
\newblock {\em Advances in neural information processing systems}, 32.
\bibitem[Shimoni et~al., 2018]{shimoni2018benchmarking}
Shimoni, Y., Yanover, C., Karavani, E., and Goldschmnidt, Y. (2018).
\newblock Benchmarking framework for performance-evaluation of causal inference
analysis.
\newblock {\em arXiv preprint arXiv:1802.05046}.
\bibitem[Wager and Athey, 2018]{wager2018estimation}
Wager, S. and Athey, S. (2018).
\newblock Estimation and inference of heterogeneous treatment effects using
random forests.
\newblock {\em Journal of the American Statistical Association},
113(523):1228--1242.
\bibitem[Wang et~al., 2021]{Wang_FLAML_A_Fast_2021}
Wang, C., Wu, Q., Weimer, M., and Zhu, E. (2021).
\newblock {FLAML: A Fast and Lightweight AutoML Library}.
\bibitem[Xie et~al., 2012]{xie2012estimating}
Xie, Y., Brand, J.~E., and Jann, B. (2012).
\newblock Estimating heterogeneous treatment effects with observational data.
\newblock {\em Sociological methodology}, 42(1):314--347.
\bibitem[Zhao et~al., 2017]{zhao2017uplift}
Zhao, Y., Fang, X., and Simchi-Levi, D. (2017).
\newblock Uplift modeling with multiple treatments and general response types.
\newblock In {\em Proceedings of the 2017 SIAM International Conference on Data
Mining}, pages 588--596. SIAM.
\end{thebibliography}