Counterfactual Survival Q-learning via Buckley-James Boosting, with Applications to ACTG 175 and CALGB 8923
Extracted source text · arXiv 2508.11060
% interacttfqsample.tex
% v1.05 - August 2017
\documentclass[]{interact}
\usepackage{epstopdf}% To incorporate .eps illustrations using PDFLaTeX, etc.
% Support for small, `sub' figures and tables
%\usepackage[nolists,tablesfirst]{endfloat}% To `separate' figures and tables from text if required
%\usepackage[doublespacing]{setspace}% To produce a `double spaced' document if required
%\setlength\parindent{24pt}% To increase paragraph indentation when line spacing is doubled
%\setlength\bibindent{2em}% To increase hanging indent in bibliography when line spacing is doubled
\usepackage{epstopdf}% To incorporate .eps illustrations using PDFLaTeX, etc.
% Support for small, `sub' figures and tables
%\usepackage[nolists,tablesfirst]{endfloat}% To `separate' figures and tables from text if required
\usepackage[doublespacing]{setspace}% To produce a `double spaced' document if required
%\setlength\parindent{24pt}% To increase paragraph indentation when line spacing is doubled
%\setlength\bibindent{2em}% To increase hanging indent in bibliography when line spacing is doubled
\usepackage{natbib}% Citation support using natbib.sty
% Document Layout and Formatting
\usepackage{appendix}
\usepackage{fancyhdr}
\usepackage{float}
\usepackage{setspace}
\usepackage{hyperref} % Must be the last package
% Tables and Algorithms
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{algorithm}
\usepackage[noend]{algpseudocode}
% Math and Symbols
\usepackage{mathtools}
\usepackage{amsmath, amssymb, bbm}
% Figures and Graphics
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{tikz}
\usetikzlibrary{positioning}
% EPS Compatibility
\usepackage{epsfig} % Only if EPS files are strictly necessary
\theoremstyle{plain}% Theorem-like structures provided by amsthm.sty
\newtheorem{theorem}{Theorem}[section]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{proposition}[theorem]{Proposition}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}
\begin{document}
\articletype{ARTICLE TEMPLATE}% Specify the article type or omit as appropriate
\title{Counterfactual Survival Q-Learning for Longitudinal Randomized Trials via Buckley–James Boosting}
\author{
\name{Jeongjin Lee\textsuperscript{a} and Jong-Min Kim\textsuperscript{b}\thanks{Corresponding Author: Jong-Min Kim. Email: jongmink@morris.umn.edu}}
\affil{\textsuperscript{a}Division of Biostatistics, The Ohio State University, 281 W Lane Ave, Columbus, OH 43210, U.S.A.;
\textsuperscript{b}Division of Science and Mathematics, University of Minnesota-Morris, Morris, MN 56267, U.S.A.}
}
\maketitle
\begin{abstract}
We propose a Buckley–James (BJ) Boost Q-learning framework for estimating optimal dynamic treatment regimes under right-censored survival data, tailored for longitudinal randomized clinical trial settings. The method integrates accelerated failure time models with iterative boosting techniques—componentwise least squares and regression trees—within a counterfactual Q-learning framework. By directly modeling conditional survival time, BJ Boost Q-learning avoids the restrictive proportional hazards assumption and enables unbiased estimation of stage-specific Q-functions. Grounded in potential outcomes, this framework ensures identifiability of the optimal treatment regime under standard causal assumptions. Compared to Cox-based Q-learning, which relies on hazard modeling and may suffer from bias under misspecification, our approach provides robust and flexible estimation. Simulation studies and analysis of the ACTG175 HIV trial demonstrate that BJ Boost Q-learning yields higher accuracy in treatment decision-making, especially in multi-stage settings where bias can accumulate.
\end{abstract}
\begin{keywords}
Boosting; Reinforcement Learning; Imputation; Survival Analysis
\end{keywords}
\section{Introduction}
\label{s:intro}
\sloppy
In the evolving field of contemporary healthcare, individualized treatment strategies \citep{moodie2012q, wahed2013evaluating, chakraborty2014dynamic, kosorok2015adaptive, song2015penalized, simoneau2020estimating, cho2023multi} have become a central approach for tailoring interventions to optimize patient outcomes. This approach is especially important in the presence of complex censored data, where event times may be partially unobserved due to issues such as patient dropout or loss to follow-up. Such challenges have motivated the development of methodological frameworks capable of handling incomplete outcome information.
Survival analysis, which focuses on time-to-event data, is crucial in medical research. A major challenge in this field is the frequent occurrence of censoring, where the event time for certain observations is unknown, making it difficult to make well-informed decisions from these datasets. Cox regression \citep{cox1972regression} is commonly used for analyzing time-to-event data with censoring. However, it has limitations, including the assumption of proportional hazards and a less intuitive interpretation compared to linear regression. Moreover, hazard ratios (HRs) derived from the Cox model are relative measures that compare the hazard rates of two groups and depend on the underlying hazard function, which may not always be straightforward to interpret. The Cox PH model assumes that the hazard function for an individual is a product of a baseline hazard function and an exponential function of a linear combination of the covariates. This linearity assumption may not hold in all situations, especially when the relationships between covariates and the hazard are complex and non-linear.
An alternative approach is the accelerated failure time (AFT) model \citep{bu79, wei1992accelerated, ji03, ji06, ze07, ch21}, which offers several advantages. The AFT model directly models survival time using a linear regression form, providing more interpretable effects than hazard ratios in Cox models. Moreover, the AFT model is robust to violations of the proportional hazards assumption. The Buckley–James (BJ) method \citep{bu79, ji06}, a semiparametric estimator for the AFT model, accommodates arbitrary censoring mechanisms and remains consistent under mild conditions. Due to these advantages, several penalized extensions of BJ estimation have been developed to address high-dimensional data \citep{jo08, wa08, jo09, li14, lee2024censored}.
To extend the Buckley–James (BJ) framework to non-linear and high-dimensional settings, \citet{wang2010buckley} proposed the BJ Boosting algorithm, which iteratively updates the predictive function using flexible base learners such as componentwise least squares or regression trees.
Unlike Cox-based machine learning approaches that model hazard functions, BJ Boosting directly targets the log-transformed survival time under the accelerated failure time (AFT) model, enabling more interpretable and robust inference.
It effectively accommodates right-censoring and tied event times while capturing complex, non-linear relationships. In terms of computational efficiency and estimation accuracy, BJ Boosting has shown superior performance over classical BJ and Cox models.
Its iterative structure adaptively selects relevant covariates and mitigates overfitting, making it particularly well-suited for biomarker-driven survival analysis and dynamic treatment regime estimation.
Reinforcement learning, particularly Q-Learning \citep{wa92}, has shown great potential in personalized treatment optimization due to its ability to adapt and learn from sequential decision-making processes. In healthcare, treatment decisions are often made in stages, considering the evolving state of a patient's health. Q-Learning can model this dynamic process by learning optimal policies that maximize long-term health outcomes based on cumulative rewards. This adaptive capability makes Q-Learning especially promising for personalized treatment, where the goal is to tailor interventions to individual patient needs over time. Despite its promise in personalized treatment optimization, directly applying Q-Learning to censored survival data proves challenging due to severe censoring.
To address these challenges, we propose the BJ Boost Q-Learning under a counterfactual framework, which integrates Q-learning with the Buckley--James (BJ) boosting algorithm to accommodate non-linear relationships under right-censored survival data.
This approach is motivated by real-world clinical studies such as the ACTG 175 trial, a randomized clinical trial designed to compare monotherapy with zidovudine or didanosine against combination therapies involving zidovudine and didanosine or zidovudine and zalcitabine, in HIV-positive adults with CD4 T cell counts between 200 and 500 cells/mm\textsuperscript{3}.
In such trials, each patient is assigned to one treatment arm, but to evaluate optimal dynamic treatment regimes, it is necessary to estimate potential outcomes under all possible treatment strategies, requiring counterfactual reasoning.
While \citet{lee2025qlearninglinear} introduced a linear BJ Q-learning approach under a counterfactual framework, its reliance on linear associations between covariates and survival outcomes limits its applicability in complex, real-world clinical contexts.
Furthermore, the linear Buckley-James estimator \citep{bu79, ji06} is known to exhibit convergence issues when the true data-generating mechanism is non-linear.
By employing flexible base learners such as componentwise least squares or regression trees, the BJ boosting approach enables stable and accurate imputation of censored survival times even in non-linear settings.
Coupled with the recursive structure of Q-learning, this iterative boosting mechanism adaptively refines treatment value estimation at each stage, thereby improving the reliability of learned dynamic treatment regimes and ultimately enhancing patient-specific clinical decision-making.
\section{Method}
\subsection{Preliminary}
We consider a longitudinal (or sequentially) randomized clinical trial consisting of \( K \) decision stages, indexed by \( k = 1, \ldots, K \), where each stage corresponds to the interval between two consecutive patient visits.
These visits may be scheduled regularly (e.g., every 30 days) or triggered by clinical events (e.g., symptom onset or adverse reaction).
At each stage \( k \), clinicians observe patient-specific, time-varying covariates \( X_{i,k} \) (e.g., biomarker levels, symptom scores), and randomly assign a treatment \( A_{i,k} \in \mathcal{A} \) according to the trial protocol. The treatment set \( \mathcal{A} \) may consist of binary (e.g., treatment vs. control), ordinal (e.g., low vs. high dose), or categorical options. The observed history up to stage \( k \) for patient \( i \) is:
\begin{equation}
H_{i,k} = \left(B_{i,0}, X_{i,1}, A_{i,1}, \ldots, X_{i,k}, A_{i,k} \right),
\end{equation}
where \( B_{i,0} \) includes baseline covariates (e.g., age, sex), and \( \{X_{i,l}, A_{i,l}\}_{l=1}^k \) represents the longitudinal covariate and treatment history up to stage \( k \).
Let \( T_{i,k} \) denote the true stage-specific survival time, i.e., the time between stages \( k{-}1 \) and \( k \). Let \( C_i \) denote the censoring time due to dropout or end of follow-up. The observed survival time and censoring indicator are:
\begin{equation}
Y_{i,k} = \min(T_{i,k}, C_i), \quad \delta_{i,k} = \mathbbm{1}(T_{i,k} \leq C_i),
\end{equation}
where \( \delta_{i,k} = 1 \) implies the event is observed, and \( \delta_{i,k} = 0 \) indicates right-censoring.
\begin{figure}[h!]
\centering
\includegraphics[scale=0.4]{Example.png}
\caption{Illustration of a patient trajectory over three stages, with observed and censored outcomes.}
\label{fig:patient}
\end{figure}
Figure~\ref{fig:patient} illustrates the follow-up trajectory of a sample patient across three clinical visits.
At each stage, covariate and treatment information (\( H_{i,k} \)) is recorded. The outcomes at Visits 1 and 2 are fully observed (\( \delta_{i1} = 1, \delta_{i2} = 1 \)), indicating that the event of interest occurred before censoring.
In contrast, the event at Visit 3 is censored (\( \delta_{i3} = 0 \)), signifying that the study ended or the patient was lost to follow-up before the event occurred.
This figure highlights how longitudinal studies simultaneously capture observed and censored time-to-event data over successive stages.
Dynamic treatment regimes (DTRs) aim to learn optimal treatment rules that adapt to evolving patient information. The cumulative survival time for patient \( i \) is:
\begin{equation}
T_{i,\text{cum}} = \sum_{k=1}^K \eta_{i,k} T_{i,k},
\end{equation}
where \( \eta_{i,k} = 1 \) if patient \( i \) reaches stage \( k \), and \( 0 \) otherwise.
For instance, \( \eta_{i,k} = 0 \) if the physician discontinues treatment due to adverse effects or lack of clinical benefit before stage \( k \).
Let \( T_{i,k}(a_k) \) denote the potential survival time at stage \( k \) under treatment \( a_k \), and define the cumulative potential survival time under treatment sequence \( a = (a_1, \ldots, a_K) \) as:
\begin{equation}
T_{i,\text{cum}}(a) = \sum_{k=1}^K \eta_{i,k} T_{i,k}(a_k).
\end{equation}
The optimal DTR is the sequence \( d_i^{\text{true}} = \{d_{i,1}^{\text{true}}, \ldots, d_{i,K}^{\text{true}}\} \) that maximizes the expected cumulative survival:
\begin{equation}
d_i^{\text{true}} = \arg\max_{(a_1, \ldots, a_K) \in \mathcal{A}^K} \mathbb{E}\left[\sum_{k=1}^K \eta_{i,k} T_{i,k}(a_k) \right].
\end{equation}
To recursively estimate the optimal rule, we define the Q-function at stage \( k \) as the expected cumulative survival from stage \( k \) onward, conditional on the patient history and a hypothetical treatment \( a_k \):
\begin{equation}
\label{eq:true_q_function}
Q_k^*(H_{i,k}(a_k)) = \mathbb{E} \left[ T_{i,k}(a_k) + \max_{a_{k+1} \in \mathcal{A}} Q_{k+1}^*(H_{i,k+1}(a_{k+1})) \,\big|\, H_{i,k}(a_k) \right],
\end{equation}
where \( H_{i,k}(a_k) \) replaces \( A_{i,k} \) with \( a_k \), and \( H_{i,k+1}(a_{k+1}) \) extends this to the next stage. The stage-\( k \) optimal treatment decision is:
\begin{equation}
d_{i,k}^{\text{true}} = \arg\max_{a_k \in \mathcal{A}} Q_k^*(H_{i,k}(a_k)).
\end{equation}
Since treatment assignment \( A_{i,k} \) is randomized by design, the estimation of Q-functions benefits from the unconfoundedness between treatments and potential outcomes, allowing for unbiased learning of optimal dynamic treatment strategies.
\subsection{Buckley-James Boost Q-Learning Framework}
The Buckley-James (BJ) Boost Q-Learning framework offers a structured approach to optimizing treatment regimes by leveraging robust imputation techniques for handling censored survival data. Central to this framework is the accurate estimation of stage-specific survival times, which directly influences the derivation of optimal treatment policies.
For notational simplicity, we denote by \( H_{i,k} \) the observed history of covariates and treatments for individual \( i \) up to stage \( k \) when describing the Buckley–James boosting (BJ Boosting) method \citep{wang2010buckley, wang2023package}.
In subsequent subsections, within the counterfactual framework, we adopt the notation \( H_{i,k}(a_k) \) to represent the hypothetical history that would have been observed had treatment \( a_k \) been assigned at stage \( k \).
\subsubsection{Stage-Specific Survival Time Imputation with BJ-Boosting}
We utilize the Buckley-James (BJ) boosting method \citep{wang2010buckley, wang2023package} to impute censored survival (or visit) times. Specifically, we employ two algorithms: BJ Twin Boosting with componentwise least squares and BJ Boosting with regression trees.
Both BJ boosting methods iteratively update function estimates and impute censored survival times, assuming conditional independence between failure and censoring times given the covariates.
%\begin{equation}
% T_{i,k} \perp C_{i,k} \mid B_{i,0}, X_{i,1},\ldots,X_{i,k}, A_{i,1}, \ldots ,A_{i,k}.
%\end{equation}
These algorithms provide a robust and efficient framework for addressing right-censored survival data, enabling accurate imputation and facilitating reliable subsequent analysis.
\begin{algorithm}[h!]
\caption{BJ Twin Boosting with Componentwise Least Squares at Stage \( k \)}
\label{alg:bjtwin}
\small
\setstretch{1}
\begin{algorithmic}[1]
\State \textbf{Initialization:}
\begin{itemize}
\item Set the initial function estimate at stage \( k \): \( \hat{f}_{0,k} = \bar{Y}_k \), where \( \bar{Y}_k = \frac{1}{n} \sum_{i=1}^{n} Y_{i,k} \) is the mean of the observed event times at stage \( k \).
\item Set the iteration counter \( m = 0 \).
\end{itemize}
\State \textbf{Boosting Iterations:}
\For{\( m = 1 \) to \( M \)}
\State Compute residuals at stage \( k \):
\[
U_{i,m,k} = Y_{i,k} - \hat{f}_{m,k}(H_{i,k}),
\]
where \( Y_{i,k} \) is the observed event time, \( \hat{f}_{m,k}(H_{i,k}) \) is the current function estimate, and \( U_{i,m,k} \) are the residuals.
\State \textbf{Variable Selection:} For each covariate \( j \), fit a linear model to the residuals \( U_{i,m,k} \) using least squares:
\[
g_{m,k}^{(j)}(H_{j,k}) = \beta_{j,m,k} H_{j,k}, \quad \text{where} \quad \beta_{j,m,k} = \frac{\sum_{i=1}^{n} H_{i,j,k} U_{i,m,k}}{\sum_{i=1}^{n} H_{i,j,k}^2}.
\]
Select the covariate \( j^* \) that minimizes the residual sum of squares (RSS):
\[
j^* = \arg\min_{j} \sum_{i=1}^{n} \left( U_{i,m,k} - g_{m,k}^{(j)}(H_{j,k}) \right)^2.
\]
The selected covariate \( j^* \) has the greatest contribution to reducing residuals at iteration \( m \).
\State Update the function estimate at stage \( k \):
\[
\hat{f}_{m+1,k}(H_{k}) = \hat{f}_{m,k}(H_{k}) + \nu g_{m,k}^{(j^*)}(H_{j^*,k}),
\]
where \( \nu \) is the learning rate (\( 0 < \nu \leq 1 \)), and \( g_{m,k}^{(j^*)}(H_{j^*,k}) \) is the selected function.
\State Impute censored survival times at stage \( k \):
\[
Y_{i,k}^* = Y_{i,k} \delta_{i,k} + (1 - \delta_{i,k}) \left( \hat{f}_{m,k}(H_{i,k}) + \frac{\int_{Y_{i,k} - \hat{f}_{m,k}(H_{i,k})}^{\infty} t \, d\hat{F}(t)}{1 - \hat{F}(Y_{i,k} - \hat{f}_{m,k}(H_{i,k}))} \right),
\]
where \( \hat{F} \) is the Kaplan-Meier estimator of the residuals.
\State Increase \( m \) by one and repeat until the stopping criterion is met.
\EndFor
\end{algorithmic}
\end{algorithm}
\begin{algorithm}[h!]
\caption{BJ Boosting with Regression Trees at Stage \( k \)}
\label{alg:bjtree}
\begin{algorithmic}[1]
\small
\setstretch{1}
\State \textbf{Initialization:}
\begin{itemize}
\item Set the initial function estimate at stage \( k \): \( \hat{f}_{0,k} = \bar{Y}_k \), where \( \bar{Y}_k = \frac{1}{n} \sum_{i=1}^{n} Y_{i,k} \) is the mean of the observed event times at stage \( k \).
\item Set the iteration counter \( m = 0 \).
\end{itemize}
\State \textbf{Boosting Iterations:}
\begin{itemize}
\For{\( k \) in \( 1 \) to \( K \)}
\State Compute residuals at stage \( k \):
\[
U_{i,m,k} = Y_{i,k} - \hat{f}_{m,k}(H_{i,k}),
\]
where \( Y_{i,k} \) is the observed event time, \( \hat{f}_{m,k}(H_{i,k}) \) is the current function estimate, and \( U_{i,m,k} \) are the residuals.
\State Fit a regression tree \( g_{m,k}(H_{k}) \) to the residuals \( U_{i,m,k} \). The tree can vary in complexity:
\begin{itemize}
\item \textbf{Regression Stumps}: Trees with only two terminal nodes (\textit{degree} = 1).
\item \textbf{Higher Degree Trees}: Trees with more terminal nodes (e.g., \textit{degree} = 2) to capture interactions between covariates.
\end{itemize}
\State Update the function estimate at stage \( k \):
\[
\hat{f}_{m+1,k}(H_{k}) = \hat{f}_{m,k}(H_{k}) + \nu g_{m,k}(H_{k}),
\]
where \( \nu \) is the learning rate (\( 0 < \nu \leq 1 \)), and \( g_{m,k}(H_{k}) \) is the fitted regression tree.
\State Impute censored survival times at stage \( k \):
\begin{align*}
Y_{i,k}^* &= \hat{f}_{m,k}(H_{i,k}) + \left( Y_{i,k} - \hat{f}_{m,k}(H_{i,k}) \right) \delta_{i,k} \\
&+ \left(1 - \delta_{i,k}\right) \left( \hat{f}_{m,k}(H_{i,k})
+ \frac{\int_{Y_{i,k} - \hat{f}_{m,k}(H_{i,k})}^{\infty} t \, d\hat{F}(t)}
{1 - \hat{F}(Y_{i,k} - \hat{f}_{m,k}(H_{i,k}))} \right),
\end{align*}
where \( \hat{F} \) is the Kaplan-Meier estimator of the residuals.
\State Increase \( m \) by one and repeat until the stopping criterion is met.
\EndFor
\end{itemize}
\end{algorithmic}
\end{algorithm}
The BJ twin boosting with componentwise least squares algorithm (Algorithm~\ref{alg:bjtwin}) addresses right-censored survival data by iteratively selecting and updating covariates in a componentwise framework at each stage \( k \). At each iteration \( m \), the residuals \( U_{i,m,k} = Y_{i,k} - \hat{f}_{m,k}(H_{i,k}) \) are computed for each individual \( i = 1, \ldots, n \), where \( Y_{i,k} \) denotes the observed (or imputed) survival outcome at stage \( k \) for individual \( i \), and \( \hat{f}_{m,k}(H_{i,k}) \) represents the current function estimate based on the covariate vector \( H_{i,k} = (H_{i,1,k}, \ldots, H_{i,p,k})^\top \).
For each covariate \( j = 1, \ldots, p \), a univariate linear model
\begin{equation}
g_{m,k}^{(j)}(H_{i,j,k}) = \beta_{j,m,k} H_{i,j,k}
\end{equation}
is fit to predict the residuals \( U_{i,m,k} \), where \( H_{i,j,k} \) denotes the value of covariate \( j \) for individual \( i \) at stage \( k \) and \( \beta_{j,m,k} \) is the corresponding coefficient at iteration \( m \). The covariate \( j^* \) that minimizes the residual sum of squares (RSS) is selected according to
\begin{equation}
j^* = \arg\min_{j} \sum_{i=1}^{n} \left( U_{i,m,k} - g_{m,k}^{(j)}(H_{i,j,k}) \right)^2,
\end{equation}
ensuring that the variable contributing most to reducing the residuals is updated at each step.
The coefficient \( \beta_{j,m,k} \) represents the contribution of covariate \( j \) to the function update at stage \( k \) and iteration \( m \). At initialization (\( m = 0 \)), the coefficients \( \beta_{j,k,\text{init}} \) are set to provide a proper starting magnitude for updates. A careful choice of \( \beta_{j,k,\text{init}} \) is critical: if \( \beta_{j,k,\text{init}} \) is too large, the model may overfit early and become unstable, whereas if it is too small, convergence may be slow or ineffective. In particular, setting \( \beta_{j,k,\text{init}} = 0 \) would prevent any updates and render boosting ineffective. To balance stability and adaptability, \( \beta_{j,k,\text{init}} \) is typically initialized based on the least squares fit of \( Y_k \) on \( H_k \) at early iterations, providing a well-scaled starting point while allowing flexible updates throughout the boosting process.
Although \( g_{m,k}^{(j)}(H_{i,j,k}) \) updates only a single covariate \( j^* \) at each iteration, the cumulative function estimate
\begin{equation}
\hat{f}_{m+1,k}(H_{i,k}) = \hat{f}_{m,k}(H_{i,k}) + \nu g_{m,k}^{(j^*)}(H_{i,j^*,k}),
\end{equation}
where \( \nu \in (0,1] \) is the learning rate, aggregates these updates over iterations, thereby constructing a multivariable model that effectively captures complex survival-covariate relationships across stages.
The BJ boosting with regression trees (algorithm \ref{alg:bjtree}) addresses right-censored survival data by iteratively boosting regression trees to model the data. At each stage \( t \), residuals are computed, and a regression tree is fitted to these residuals. The function estimate is updated based on the fitted regression tree, and censored survival times are imputed. This process is repeated for a specified number of iterations or until convergence, allowing for the capture of complex interactions between covariates and accurate imputation of censored survival times.
The BJ boosting with regression trees algorithm \ref{alg:bjtree} addresses right-censored survival data by iteratively boosting regression trees to model the relationship between covariates and survival times. At each stage \( k \), residuals are computed based on the difference between observed event times and the current function estimate. A regression tree \( g_{m,k}(H_k) \) is then fitted to these residuals to capture nonlinear dependencies and potential interactions between covariates. The function estimate is updated by incorporating the newly fitted tree with a learning rate \( \nu \), controlling the contribution of each iteration to prevent overfitting. The flexibility of this approach allows for different tree complexities, ranging from regression stumps (trees with only two terminal nodes) to deeper trees that model higher-order interactions. After updating the function estimate, censored survival times are imputed using a weighted adjustment based on the Kaplan-Meier estimator of residuals. This iterative process continues for a specified number of boosting iterations or until convergence, refining the function estimate over time and allowing the model to adaptively capture complex survival-covariate relationships.
Tuning parameter selection is essential for optimizing BJ Twin Boosting and BJ Boosting with Regression Trees. Key parameters include the number of boosting iterations (\( M \)), learning rate (\( \nu \)), and tree complexity (for BJ Boosting with Trees), selected via cross-validation to minimize prediction error and prevent overfitting.
BJ Twin Boosting iteratively updates function estimates while performing variable selection. The number of iterations \( M \) controls refinement, while \( \nu \) regulates update size, balancing convergence and stability.
BJ Boosting with Trees extends this framework by using decision trees as base learners to capture nonlinear relationships. In addition to \( M \) and \( \nu \), tree complexity (e.g., depth or size) is tuned to prevent overfitting.
Cross-validation ensures optimal parameter selection based on prediction error minimization, though AIC may also be used for linear base learners as suggested by \citet{wang2010buckley}.
To guide the selection between these techniques, BJ Twin Boosting is recommended when the primary goal is variable selection and interpretability in structured data, whereas BJ Boosting with Trees is more appropriate for capturing complex interactions and nonlinear effects. For further details, see \citet{wang2010buckley}.
\subsubsection{Recursive Estimation of Q-Functions}
The Q-function \( Q_k(H_{i,k}(a_k)) \) represents the expected cumulative survival time starting from stage \( k \), conditional on the counterfactual history \( H_{i,k}(a_k) \) where treatment \( a_k \) is hypothetically assigned at stage \( k \).
At the final stage \( K \), the Q-function is initialized using the Buckley--James imputed counterfactual survival time:
\begin{equation}
Q_K(H_{i,K}(a_K)) = Y_{i,K}^*(a_K),
\end{equation}
where \( Y_{i,K}^*(a_K) \) is the BJ-imputed survival time under treatment \( a_K \), estimated using a boosting procedure (Algorithms~\ref{alg:bjtwin} or~\ref{alg:bjtree}).
For earlier stages \( k = K-1, \ldots, 1 \), the pseudo-outcome under counterfactual treatment \( a_k \) is defined as:
\begin{equation}
\tilde{Y}_{i,k}(a_k) = Y_{i,k}^*(a_k) + \max_{a_{k+1}} Q_{k+1}(H_{i,k+1}(a_{k+1})),
\end{equation}
where \( Y_{i,k}^*(a_k) \) is the imputed survival time under \( a_k \), and \( Q_{k+1}(H_{i,k+1}(a_{k+1})) \) is the estimated Q-function at the next stage.
Then, the Q-function at stage \( k \) is estimated by solving:
\begin{equation}
\min_{\beta_k} \sum_{i=1}^n \left( \tilde{Y}_{i,k}(A_{i,k}) - Q_k(H_{i,k}(A_{i,k}); \beta_k) \right)^2,
\end{equation}
where the model is evaluated at the observed treatment \( A_{i,k} \), and the Q-function is parameterized as:
\begin{equation}
Q_k(H_{i,k}(a_k); \beta_k) = g(H_{i,k}(a_k), a_k; \beta_k),
\end{equation}
where \( g(H_{i,k}(a_k), a_k; \beta_k) \) is a flexible (possibly nonlinear) function capturing the joint effects of the covariate history and treatment.
\subsubsection{Optimal Treatment Decision at Each Stage}
The optimal treatment rule at stage \( k \), based on counterfactual Q-values, is defined as:
\begin{equation}
d_{i,k}^{\text{opt}} = \arg\max_{a_k \in \mathcal{A}} Q_k(H_{i,k}(a_k)).
\end{equation}
Since each patient only receives one observed treatment in practice, we estimate \( Q_k(H_{i,k}(a_k)) \) for all \( a_k \in \mathcal{A} \) using BJ-imputed survival times \( Y_{i,k}^*(a_k) \). These counterfactual estimates enable valid comparisons across treatment options, allowing data-driven, individualized treatment recommendations that aim to maximize cumulative survival.
\subsubsection{Assumptions for Identifiability of the Q-function}
To ensure the identifiability of the Q-function in a longitudinal randomized clinical trial, we make the following standard assumptions:
\begin{enumerate}
\item \textbf{Consistency:} If a patient receives treatment \( A_{i,k} = a_k \), then the observed survival time equals the corresponding potential outcome:
\begin{equation}
T_{i,k} = T_{i,k}(a_k) \quad \text{for all } k.
\end{equation}
\item \textbf{Unconfoundedness via Randomization:}
\begin{equation}
A_{i,k} \perp T_{i,k}(a_k) \mid H_{i,k}^{-} \quad \text{for all } a_k \in \mathcal{A},\, k,
\end{equation}
which holds by design in a longitudinal randomized clinical trial, where $H_{i,k}^{-}$ denotes the pre-treatment history up to stage $k$. This assumption implies that, conditional on the observed history, the treatment assignment is independent of the potential outcome under any treatment option.
\item \textbf{Positivity:}
\begin{equation}
\mathbb{P}(A_{i,k} = a_k) > 0 \quad \text{for all } a_k \in \mathcal{A},\, k,
\end{equation}
ensuring that every treatment option has a non-zero probability of being assigned.
\item \textbf{Stable Unit Treatment Value Assumption (SUTVA):}
\begin{equation}
T_{i,k}(a_k) \text{ is unaffected by } A_{j,k} \quad \text{for all } i \neq j,
\end{equation}
indicating no interference between patients.
\item \textbf{Correct Model Specification:}
\begin{equation}
Q_k(H_{i,k}(a_k); \beta_k) = \mathbb{E}[T_{i,k}(a_k) + \max_{a_{k+1} \in \mathcal{A}} Q_{k+1}(H_{i,k+1}(a_{k+1})) \mid H_{i,k}(a_k)],
\end{equation}
stating that the statistical model used to estimate the Q-function correctly reflects the conditional expectation of future outcomes.
\end{enumerate}
\section{Simulation Study}
This simulation study evaluates the performance of five Q-value estimation approaches for right-censored survival data in the context of a multistage clinical trial designed to inform dynamic treatment regimes (DTRs).
Specifically, we compare the accuracy of treatment decisions derived from the true (oracle) Q-values, Buckley–James Q-learning with linear regression (BJ-Q), as well as its extensions using twin boosting (BJ-LS Q), regression trees (BJ-Tree Q), and Cox proportional hazards models (Cox-Q). The BJ-Q and Cox-Q methods follow the framework proposed by \citet{lee2025qlearninglinear}.
We simulated data for \( n \in \{500, 1000\} \) patients enrolled in a two-stage longitudinal randomized clinical trial.
Each patient was followed over \( K = 2 \) clinical decision stages, indexed by \( k = 1, 2 \), and characterized by four baseline and time-varying covariates: sex, tumor size, body mass index (BMI), and age.
The binary sex variable was generated as \( \text{Sex}_i \sim \text{Bernoulli}(0.5) \). Tumor size at each stage was drawn independently from a uniform distribution, \( \text{TumorSize}_{i,k} \sim \text{Unif}(1, 3) \), and transformed to induce nonlinearity:
\[
\text{TumorSize}_{i,k}^{\text{trans}} = \left(\text{TumorSize}_{i,k}\right)^{2.3} - \text{median}\left\{ \left(\text{TumorSize}_{j,k}\right)^{2.3} \right\}_{j=1}^n.
\]
BMI and age were both generated as time-invariant covariates: \( \text{BMI}_i \sim \mathcal{N}(25,\, 5^2) \) and \( \text{Age}_i \sim \mathcal{N}(50,\, 10^2) \).
At each stage \( k \), treatment \( A_{i,k} \in \{0, 1\} \) was randomly assigned with equal probability. The potential survival time under treatment \( a_k \) was generated from the nonlinear model:
\begin{align*}
T_{i,k}(a_k) =\ & \beta_0 + \beta_1 \cdot \text{Sex}_i + \beta_2 \cdot \text{TumorSize}_{i,k}^{\text{trans}} + \beta_3 \cdot \log(\text{BMI}_i) \\
& + \beta_4 \cdot \sqrt{\text{Age}_i} + \beta_5 \cdot \mathbbm{1}(a_k = 1) + \beta_6 \cdot \text{TumorSize}_{i,k}^{\text{trans}} \cdot \mathbbm{1}(a_k = 1) + \varepsilon_{i,k},
\end{align*}
where \( \varepsilon_{i,k} \sim \mathcal{N}(0, 1) \). The parameters were set to \( \beta_0 = 10, \beta_1 = 0.4, \beta_2 = -1, \beta_3 = -0.4, \beta_4 = -0.01, \beta_5 = 0.05, \beta_6 = 1.3 \).
Right censoring was introduced by sampling a patient-specific censoring time \( C_i \sim \text{Unif}(q_{0.2}(T_{i,1}),\ q_{0.8}(T_{i,1})) \) in the single-stage setting, and \( C_i \sim \text{Unif}(q_{0.2}(T_{i,1} + T_{i,2}),\ q_{0.8}(T_{i,1} + T_{i,2})) \) in the two-stage setting, where \( T_{i,1} \) and \( T_{i,2} \) denote the uncensored survival times at each stage.
Here, \( q_p(\cdot) \) denotes the \( p \)-th empirical quantile computed across the simulated sample.
The observed survival time and event indicator at each stage were then defined as:
\[
Y_{i,k} = \min(T_{i,k},\, C_i), \quad \delta_{i,k} = \mathbbm{1}(T_{i,k} \leq C_i).
\]
\subsection{Single Stage Setting}
We first consider the single-stage setting (\(k = 1\)).
The true Q-values under each treatment arm are defined as:
\[
\begin{aligned}
Q_{i,1}^{(1)} =\ & \beta_0 + \beta_1 \cdot \text{Sex}_i + \beta_2 \cdot \text{TumorSize}_{i,1}^{\text{trans}} + \beta_3 \cdot \log(\text{BMI}_i) \\
& + \beta_4 \cdot \sqrt{\text{Age}_i} + \beta_5 + \beta_6 \cdot \text{TumorSize}_{i,1}^{\text{trans}}, \\
Q_{i,1}^{(0)} =\ & \beta_0 + \beta_1 \cdot \text{Sex}_i + \beta_2 \cdot \text{TumorSize}_{i,1}^{\text{trans}} + \beta_3 \cdot \log(\text{BMI}_i) \\
& + \beta_4 \cdot \sqrt{\text{Age}_i}.
\end{aligned}
\]
However, under right censoring, the observed survival times are incomplete, and special techniques are required to consistently estimate Q-functions. In the Buckley–James Boost Q-learning framework, censored outcomes are handled through iterative imputation and model fitting. The Q-functions are estimated as:
\[
\begin{aligned}
\widehat{Q}_{i,1}^{(1)} &= \hat{f}_1(\text{Sex}_i, \text{TumorSize}_{i,1}, \text{BMI}_i, \text{Age}_i), \\
\widehat{Q}_{i,1}^{(0)} &= \hat{f}_0(\text{Sex}_i, \text{TumorSize}_{i,1}, \text{BMI}_i, \text{Age}_i),
\end{aligned}
\]
where \( \hat{f}_1(\cdot) \) and \( \hat{f}_0(\cdot) \) are flexible functions fitted separately for each treatment group. These functions can be estimated using the classical Buckley-James linear model (BJ) \citep{ji06} or more flexible boosting-based methods, such as componentwise least squares (BJ-LS) or regression trees (BJ-Tree), both of which impute censored survival times before model fitting (see Algorithms~\ref{alg:bjtwin} and~\ref{alg:bjtree}).
\begin{figure}[h!]
\centering
\begin{subfigure}[b]{0.8\linewidth}
\centering
\includegraphics[width=1\textwidth]{Q500.png}
\caption{($n$, Censoring Rate) = (500, 0.5)}
\end{subfigure}
\vskip\baselineskip
\begin{subfigure}[b]{0.8\linewidth}
\centering
\includegraphics[width=1\textwidth]{Q1000.png}
\caption{($n$, Censoring Rate) = (1000, 0.5)}
\end{subfigure}
\caption{Comparison of Estimated Q-values for Treatments A and B under a Single-Stage Dynamic Treatment Regime. Each panel represents a single simulated replicate with approximately 50\% right-censoring. Boxplots display the distribution of true Q-values versus those estimated using Buckley–James Q-learning with linear regression (BJ), twin boosting (BJ-LS), regression trees (BJ-Tree), and Cox-based Q-learning (Cox), across varying sample sizes.}
\label{fig:qvalue_comparison_single}
\end{figure}
Figure~\ref{fig:qvalue_comparison_single} presents boxplots comparing the true and estimated Q-values for Treatments~A and~B under a single-stage dynamic treatment regime, with sample sizes \( n = 500 \) and \( n = 1000 \), each subject to a 50\% censoring rate.
The methods evaluated include the oracle (True), Buckley-James Q-learning with linear regression (BJ), twin boosting with componentwise least squares (BJ-LS), regression tree (BJ-Tree), and Cox-based Q-learning (Cox).
Among these, BJ-Tree most closely recovers the distribution of the true Q-values, while the other methods exhibit noticeable deviations due to their limited capacity to capture nonlinear effects.
To evaluate treatment decision accuracy, we compared estimated optimal treatment assignments against oracle decisions derived from the true Q-functions prior to censoring. The true optimal decision rule assigns Treatment~A if the expected counterfactual survival under Treatment~A exceeds that under Treatment~B:
\[
d_{i,1}^{\text{true}} = \mathbbm{1}\left\{ Q_{i,1}^{(1)} > Q_{i,1}^{(0)} \right\},
\]
where \( Q_{i,1}^{(1)} \) and \( Q_{i,1}^{(0)} \) denote the true potential outcomes defined according to the known data-generating mechanism.
The estimated treatment decision rule is defined analogously based on the model-based Q-function estimates:
\[
\widehat{d}_{i,1}^{\dagger} = \mathbbm{1}\left\{ \widehat{Q}_{i,1}^{(1), \dagger} > \widehat{Q}_{i,1}^{(0), \dagger} \right\},
\]
where \( \dagger \in \{ \text{BJ}, \text{BJ-LS}, \text{BJ-Tree}, \text{Cox} \} \) denotes the estimation method used. Each method fits a separate function for each treatment group:
\[
\begin{aligned}
\widehat{Q}_{i,1}^{(1), \dagger} &= \hat{f}_1^{\dagger}(\text{Sex}_i, \text{TumorSize}_{i,1}, \text{BMI}_i, \text{Age}_i), \\
\widehat{Q}_{i,1}^{(0), \dagger} &= \hat{f}_0^{\dagger}(\text{Sex}_i, \text{TumorSize}_{i,1}, \text{BMI}_i, \text{Age}_i),
\end{aligned}
\]
where \( \hat{f}_1^{\dagger} \) and \( \hat{f}_0^{\dagger} \) are fitted Q-functions where \( \dagger \in \{ \text{BJ}, \text{BJ-LS}, \text{BJ-Tree}, \text{Cox} \} \) denotes the estimation method used.
Each rule assigns the treatment with the higher estimated survival benefit. Decision accuracy was defined as the proportion of individuals whose estimated optimal treatment matched the oracle treatment:
\[
\text{Accuracy}^{\dagger} = \frac{1}{n} \sum_{i=1}^{n} \mathbbm{1} \left\{ \widehat{d}_{i,1}^{\dagger} = d_{i,1}^{\text{true}} \right\}.
\]
\begin{table}[H]
\small
\centering
\caption{Treatment Decision Accuracy by Method and Sample Size under Single Stage Setting across 100 replications}
\begin{tabular}{llcccccc}
\toprule
\textbf{Method} & \textbf{Sample Size} & \textbf{Min} & \textbf{1st Qu.} & \textbf{Median} & \textbf{Mean} & \textbf{3rd Qu.} & \textbf{Max} \\
\midrule
BJ & 500 & 0.8700 & 0.8720 & 0.8720 & 0.8917 & 0.9180 & 0.9240 \\
& 1000 & 0.8840 & 0.8908 & 0.9005 & 0.9002 & 0.9100 & 0.9160 \\
BJ-LS & 500 & 0.8540 & 0.8620 & 0.8700 & 0.8832 & 0.9120 & 0.9140 \\
& 1000 & 0.8730 & 0.8775 & 0.8895 & 0.8882 & 0.9002 & 0.9010 \\
BJ-Tree & 500 & 0.9000 & 0.9100 & 0.9180 & 0.9221 & 0.9400 & 0.9480 \\
& 1000 & 0.9130 & 0.9167 & 0.9200 & 0.9285 & 0.9317 & 0.9610 \\
Cox & 500 & 0.4240 & 0.4360 & 0.4380 & 0.4402 & 0.4500 & 0.4500 \\
& 1000 & 0.4340 & 0.4392 & 0.4420 & 0.4405 & 0.4432 & 0.4440 \\
\bottomrule
\end{tabular}
\label{tab:accuracy-comparison-singlestage}
\end{table}
Table~\ref{tab:accuracy-comparison-singlestage} presents treatment decision accuracy across 100 replicates under a single-stage dynamic treatment regime (\(k = 1\)) for two sample sizes, \(n = 500\) and \(n = 1000\), using the Buckley–James Q-learning framework.
Among all methods, BJ-Tree achieved the highest decision accuracy. Median accuracy exceeded 91\% for both sample sizes and improved with larger sample size, with reduced variability across replicates. This demonstrates the method’s robustness in capturing complex, nonlinear covariate-treatment interactions under right-censored survival outcomes.
BJ-LS, which implements boosting with componentwise least squares, also performed well, achieving median accuracy close to 89\% at \(n = 1000\), although consistently lower than that of BJ-Tree.
In contrast, Cox-based Q-learning resulted in markedly lower accuracy, with median values below 45\% across both sample sizes. This poor performance likely stems from the model’s reliance on the proportional hazards assumption, which fails to capture nonlinear and heterogeneous covariate effects.
\subsection{Two-Stage Setting}
We then extended the analysis to the two-stage setting (\(k \in \{1,2\}\)), where stage-specific covariates and treatment decisions are observed at each decision point.
We assume that all patients continue to the second stage.
The true cumulative Q-value for a treatment sequence \((a_1, a_2)\), based on the known data-generating mechanism, is defined as:
\[
Q_i^{(a_1, a_2)} = Q_{i,1}^{(a_1)} + Q_{i,2}^{(a_2)},
\]
where \( Q_{i,k}^{(a_k)} \) denotes the counterfactual survival outcome at stage \(k\) under treatment \(a_k \in \{0,1\}\).
Correspondingly, at each stage \(k\), Q-functions were estimated separately for each treatment arm, producing model-based estimates \(\widehat{Q}_{i,k}^{(a_k), \dagger}\), where \( \dagger \) denotes the modeling approach used (e.g., BJ, BJ-LS, or BJ-Tree).
The same method was applied at both stages to ensure consistency in estimation. The estimated cumulative Q-value under treatment sequence \((a_1, a_2)\) is then given by:
\[
\widehat{Q}_i^{(a_1, a_2), \dagger} = \widehat{Q}_{i,1}^{(a_1), \dagger} + \widehat{Q}_{i,2}^{(a_2), \dagger}.
\]
\begin{figure}[h!]
\centering
\begin{subfigure}[b]{0.8\linewidth}
\centering
\includegraphics[width=1\textwidth]{Q500stage2.png}
\caption{($n$, Censoring Rate) = (500, 0.5)}
\end{subfigure}
\vskip\baselineskip
\begin{subfigure}[b]{0.8\linewidth}
\centering
\includegraphics[width=1\textwidth]{Q1000stage2.png}
\caption{($n$, Censoring Rate) = (1000, 0.5)}
\end{subfigure}
\caption{Comparison of Estimated Cumulative Q-values for Treatment Sequences $(A_1, A_2)$ under a Two-Stage Dynamic Treatment Regime. Each panel represents a single simulated replicate with approximately 50\% right-censoring at each stage. Boxplots display the distribution of true cumulative Q-values versus those estimated using Buckley–James Q-learning with linear regression (BJ), twin boosting (BJ-LS), regression trees (BJ-Tree), and Cox-based Q-learning (Cox), for each of the four treatment sequences: AA, AB, BA, and BB.}
\label{fig:qvalue_comparison_double}
\end{figure}
Figure~\ref{fig:qvalue_comparison_double} displays the distribution of estimated cumulative Q-values for the four possible treatment sequences—AA, AB, BA, and BB—under a two-stage dynamic treatment regime. The figure compares the true cumulative Q-values, derived from the known data-generating mechanism, with estimates obtained using Buckley–James Q-learning methods with linear regression (BJ), twin boosting (BJ-LS), regression trees (BJ-Tree), and Cox-based Q-learning (Cox). Among all methods, the BJ-Tree approach demonstrated the closest alignment with the true Q-value distributions across all treatment sequences, effectively capturing the nonlinear structure of the underlying survival outcomes. In contrast, the Cox-based Q-learning method consistently deviated from the truth, displaying systematic bias that reflects its limitations in modeling complex, non-proportional hazard structures inherent in the data. Notably, the magnitude of deviation from the true Q-values is greater in the two-stage setting compared to the single-stage results shown in Figure~\ref{fig:qvalue_comparison_single}, reflecting increased modeling difficulty due to compounding estimation errors across stages.
Finally, the optimal sequence is estimated by selecting the treatment pair that maximizes this total, and decision accuracy is defined similarly:
\[
\text{Accuracy} = \frac{1}{n} \sum_{i=1}^n \mathbbm{1} \left( \arg\max_{(a_1,a_2)} \widehat{Q}_i^{(a_1,a_2), \dagger} = \arg\max_{(a_1,a_2)} Q_i^{(a_1,a_2)} \right).
\]
Table~\ref{tab:accuracy-comparison-stage2} summarizes treatment decision accuracy under a two-stage dynamic treatment regime across 100 simulation replications, for sample sizes \( n = 500 \) and \( n = 1000 \). Among all methods, BJ-Tree consistently achieved the highest accuracy, with median values exceeding 84\% across both sample sizes. Accuracy further improved as the sample size increased. Both BJ and BJ-LS also performed competitively, with median accuracies above 78\%, though consistently below that of BJ-Tree. In contrast, Cox-based Q-learning demonstrated poor performance, with median accuracies below 20\% regardless of sample size, highlighting its inadequacy in settings involving nonlinear covariate-treatment interactions. These findings emphasize the importance of using flexible, nonparametric models such as BJ-Tree when estimating optimal treatment sequences under complex censoring mechanisms.
\begin{table}[H]
\small
\centering
\caption{Treatment Decision Accuracy by Method and Sample Size under Stage 2 Setting across 100 replications}
\begin{tabular}{llcccccc}
\toprule
\textbf{Method} & \textbf{Sample Size} & \textbf{Min} & \textbf{1st Qu.} & \textbf{Median} & \textbf{Mean} & \textbf{3rd Qu.} & \textbf{Max} \\
\midrule
BJ & 500 & 0.7530 & 0.7668 & 0.7850 & 0.7876 & 0.8075 & 0.8280 \\
& 1000 & 0.7640 & 0.7825 & 0.8020 & 0.8052 & 0.8325 & 0.8460 \\
BJ-LS & 500 & 0.7430 & 0.7558 & 0.7780 & 0.7759 & 0.7965 & 0.8060 \\
& 1000 & 0.7560 & 0.7620 & 0.7860 & 0.7874 & 0.8055 & 0.8260 \\
BJ-Tree & 500 & 0.7990 & 0.8360 & 0.8440 & 0.8475 & 0.8675 & 0.8760 \\
& 1000 & 0.7980 & 0.8380 & 0.8410 & 0.8482 & 0.8660 & 0.9000 \\
Cox & 500 & 0.1750 & 0.1780 & 0.1875 & 0.1897 & 0.1933 & 0.2220 \\
& 1000 & 0.1740 & 0.1920 & 0.1930 & 0.1932 & 0.2000 & 0.2020 \\
\bottomrule
\end{tabular}
\label{tab:accuracy-comparison-stage2}
\end{table}
These findings highlight the advantage of Buckley–James (BJ) boosting methods in accurately estimating optimal dynamic treatment regimes under right censoring.
By combining flexible and iterative function estimation with modeling of censored survival outcomes, BJ boosting effectively captures individualized Q-functions without relying on restrictive hazard-based assumptions.
Unlike the commonly used Cox-Q model, which indirectly models survival through proportional hazards and is prone to bias under model misspecification, BJ boosting directly targets conditional survival time.
This leads to improved robustness and estimation precision.
The benefit of BJ boosting becomes increasingly important as the number of decision stages \( K \) increases, since cumulative bias from inaccurate survival estimation can substantially reduce the accuracy of treatment decisions.
In complex longitudinal clinical settings with right-censored data, BJ boosting methods such as twin boosting and regression tree boosting provide a powerful and interpretable framework for learning personalized and stage-specific treatment strategies.
\section{Application to the ACTG175 Dataset}
\subsection{Single-Stage Analysis Using Real Outcomes}
We apply our proposed methodology to the analysis of the ACTG175 dataset, available from the \texttt{speff2trial} R package \citep{juraska2022package}. This dataset includes data from 2,139 HIV-infected individuals who were randomized to one of four treatment strategies: AZT monotherapy, combination therapy with AZT and didanosine (ddI), combination therapy with AZT and zalcitabine (ddC), or ddI monotherapy. The key outcome variable, \texttt{days}, measures time to a clinically significant event, such as a drop in CD4 T cell count by at least 50 cells/mm\(^3\), progression to AIDS, or death. The censoring indicator \texttt{cens} equals 1 for observed events and 0 for censored observations. Notably, the dataset exhibits a high censoring rate of approximately 75\% and contains missing values in several covariates, adding analytical complexity. Further details on the variables and data structure are provided in \cite{juraska2022package}. The primary objective of the trial is to compare the efficacy of monotherapy versus combination therapy in patients with baseline CD4 T cell counts between 200 and 500 cells/mm\(^3\) \citep{hammer1996trial}.
Previous findings \citep{hammer1996trial} have indicated that patients previously treated with AZT tend to experience improved outcomes when switched to ddI, either alone or in combination with AZT, compared to continuing AZT monotherapy.
In our analysis, we focus on comparing ddI monotherapy (\(A_i = 0\)) and combination therapy with AZT and ddI (\(A_i = 1\)). We employ the Buckley–James Boost Q-learning framework under an accelerated failure time model to estimate optimal treatment strategies that maximize the expected counterfactual survival time for each individual. The Q-functions are modeled using BJ boosting methods that accommodate right-censored data through iterative updates, without relying on proportional hazards assumptions. These estimated Q-functions are then used to determine individualized optimal treatment rules based on long-term survival prospects.
\begin{figure}[h!]
\centering
\includegraphics[scale=0.3]{Qvalue1.png}
\caption{
Each panel compares the estimated survival time (Q-value) between the two treatment strategies: ddI monotherapy (\(A_i = 0\)) and combination therapy with AZT and ddI (\(A_i = 1\)).
The left panel displays observed survival times for uncensored individuals, which show limited differentiation due to the presence of censoring.
The middle and right panels present model-based Q-values imputed via Buckley--James Q-learning using twin boosting (BJ-LS) and regression trees (BJ-Tree), respectively.
These imputed outcomes incorporate censored observations and reveal clearer differences between treatment strategies.
The BJ-LS method produces greater variability and more extreme survival estimates than BJ-Tree, highlighting differences in model behavior under censoring.
}
\label{fig:real}
\end{figure}
Figure~\ref{fig:real} displays boxplots of observed and imputed survival outcomes stratified by treatment group, using three approaches: uncensored survival time, Buckley--James Q-learning with twin boosting (BJ-LS), and with regression trees (BJ-Tree).
The left panel shows only uncensored survival times, which exhibit limited separation between treatment groups due to the presence of censoring.
In contrast, the center and right panels illustrate imputed survival outcomes under the BJ-LS and BJ-Tree models, respectively.
Both imputation strategies reveal a clear survival advantage for the combination therapy group (\(A_i = 1\)), with notably higher median and upper-tail survival compared to monotherapy (\(A_i = 0\)).
This treatment effect aligns with our simulation study, where the BJ-LS method exhibited greater variability than the BJ-Tree approach.
These results demonstrate that the BJ-Q learning framework can recover treatment effects obscured by censoring, thereby enabling more accurate comparisons of potential outcomes under different treatment strategies.
\subsection{Synthetic Two-Stage Analysis}
To evaluate the performance of the proposed BJ-Q learning framework in a multi-stage setting, we constructed a two-stage treatment scenario using the ACTG175 dataset.
Specifically, we divided each patient's observed survival time into two stages based on a randomly assigned cutoff between 120 and 180 days. The first-stage outcome (\(T_{i1}, \delta_{i1}\)) was defined as the minimum of the observed survival time and the cutoff, with an event indicator equal to 1 if the observed event occurred before the cutoff without censoring. The second-stage outcome (\(T_{i2}, \delta_{i2}\)) captured the residual survival time beyond the cutoff for individuals who remained uncensored past stage 1.
Given that the ACTG175 dataset originates from a randomized clinical trial in which participants were randomly assigned to either ddI monotherapy or a combination therapy with AZT and ddI, we used the original randomized treatment as the first-stage indicator \(A_{i1}\).
To simulate dynamic treatment regimes, the second-stage treatment \(A_{i2}\) was generated by retaining the first-stage treatment with 70\% probability and switching with 30\% probability. This probabilistic assignment introduced realistic heterogeneity in treatment sequences by allowing patients to either remain on the same treatment or switch at the second stage.
As a result, four distinct two-stage treatment paths were created:
\textbf{00} (ddI $\rightarrow$ ddI),
\textbf{01} (ddI $\rightarrow$ AZT+ddI),
\textbf{10} (AZT+ddI $\rightarrow$ ddI), and
\textbf{11} (AZT+ddI $\rightarrow$ AZT+ddI).
This setup mimics real-world clinical scenarios in which treatment decisions may be dynamically adjusted over time based on evolving patient responses, allowing us to evaluate the BJ-Q learning framework's ability to recover cumulative survival benefits across diverse longitudinal treatment strategies.
We then applied the Buckley--James Q-learning framework separately at each stage using both BJ-LS and BJ-Tree to impute stage-specific conditional survival times. For individuals who survived to the second stage, we computed imputed second-stage survival outcomes based on their treatment path and covariates. The total counterfactual Q-value for each individual was obtained by summing the imputed survival times across both stages.
\begin{figure}[h!]
\centering
\includegraphics[scale=0.5]{Qvalue2.png}
\caption{
Each panel compares the estimated cumulative survival time (Q-value) across three grouped two-stage treatment strategies: 00 (ddI → ddI), 01 or 10 (switch between ddI and AZT+ddI), and 11 (AZT+ddI → AZT+ddI).
The left panel displays Q-values imputed using the Buckley--James Q-learning framework with twin boosting (BJ-LS), while the right panel shows results using regression trees (BJ-Tree).
Both methods reveal that regimens involving at least one stage of combination therapy (01, 10, or 11) tend to yield improved survival outcomes compared to monotherapy (00), with the 11 path showing the highest median Q-values.
The BJ-LS model exhibits greater variability and more extreme upper-tail values, while BJ-Tree provides more stable estimates.
}
\label{fig:real2stage}
\end{figure}
Figure~\ref{fig:real2stage} illustrates the cumulative Q-value distributions across three grouped two-stage treatment trajectories: continued ddI monotherapy (00), switching between monotherapy and combination therapy (01 or 10), and continued combination therapy (11).
These Q-values represent imputed survival outcomes under the Buckley--James Q-learning framework, using either twin boosting (BJ-LS, left panel) or regression trees (BJ-Tree, right panel).
The results reveal a clear trend in which treatment paths involving combination therapy in at least one stage (01, 10, or 11) are associated with higher median cumulative Q-values compared to sustained monotherapy (00).
Among these, the 11 trajectory—representing combination therapy at both stages—yields the most favorable outcomes, with the highest median and upper-tail survival times. This pattern is consistently observed across both estimation approaches.
While both models capture the survival advantage of combination therapy, the BJ-LS approach exhibits greater variability and more extreme upper-tail estimates, reflecting its higher sensitivity to covariate effects.
In contrast, the BJ-Tree model produces more regularized and stable Q-value distributions.
These findings underscore the flexibility of the BJ-Q learning framework in accommodating multi-stage treatment regimes and recovering dynamic treatment effects that are often obscured by censoring in survival data.
Moreover, the observed patterns are consistent with our simulation studies, where the BJ-LS method demonstrated higher variability and sensitivity, while the BJ-Tree approach provided more stable estimates across treatment trajectories.
\section{Discussion}
The BJ Boost Q-Learning framework demonstrates strong empirical performance in estimating optimal dynamic treatment regimes under right-censored data. Our simulation study, conducted under a moderate censoring rate of approximately 50\%, showed that the method achieves high decision accuracy and reliable Q-value estimation across multiple model specifications. Among these, BJ-Tree learner consistently delivered the most accurate results, underscoring its strength in modeling nonlinear treatment effects.
Despite these promising results, limitations remain. The framework’s reliability under higher levels of censoring has not been systematically assessed. Since the Buckley–James estimator relies on imputation of censored outcomes, excessive censoring could degrade the quality of the imputed values and compromise downstream Q-learning performance. Future studies should examine the method’s robustness under various censoring scenarios.
Model misspecification also presents a concern, particularly in the early stages of the boosting process. Although regression trees offer flexibility, early-stage bias may be propagated through successive boosting iterations. Investigating regularization techniques or early stopping criteria may improve robustness in such cases.
Lastly, this study assumes randomized treatment assignment. In observational settings, where confounding is a key challenge, the BJ Boost Q-Learning framework should be extended to incorporate causal adjustment techniques such as inverse probability of treatment weighting (IPTW) or propensity score matching. These enhancements would expand the method’s applicability to real-world clinical data where treatment allocation is nonrandom.
\section*{Data and Code}
The implementation of our method, along with detailed information on data generation and real data analysis, is available at \url{https://github.com/jeongjin95/BJ-Boost-Q-Learning} for reproducibility.
\section*{Acknowledgments}
We thank Professor Ji-Hyun Lee from the Department of Biostatistics at the University of Florida for her valuable guidance on survival analysis and real data analysis.
\section*{Conflict of interest}
The authors declare that they have no conflict of interest.
\bibliographystyle{agsm}
\bibliography{interacttfqsample}
\end{document}
\begin{thebibliography}{xx}
\harvarditem{Buckley \harvardand\ James}{1979}{bu79}
Buckley, J. \harvardand\ James, I. \harvardyearleft 1979\harvardyearright , `Linear regression with censored data', {\em Biometrika} {\bf 66}(3),~429--436.
\harvarditem{Chakraborty \harvardand\ Murphy}{2014}{chakraborty2014dynamic}
Chakraborty, B. \harvardand\ Murphy, S.~A. \harvardyearleft 2014\harvardyearright , `Dynamic treatment regimes', {\em Annual review of statistics and its application} {\bf 1}(1),~447--464.
\harvarditem[Cho et~al.]{Cho, Holloway, Couper \harvardand\ Kosorok}{2023}{cho2023multi}
Cho, H., Holloway, S.~T., Couper, D.~J. \harvardand\ Kosorok, M.~R. \harvardyearleft 2023\harvardyearright , `Multi-stage optimal dynamic treatment regimes for survival outcomes with dependent censoring', {\em Biometrika} {\bf 110}(2),~395--410.
\harvarditem[Choi et~al.]{Choi, Kim \harvardand\ Choi}{2021}{ch21}
Choi, T., Kim, A.~K. \harvardand\ Choi, S. \harvardyearleft 2021\harvardyearright , `Semiparametric least-squares regression with doubly-censored data', {\em Computational Statistics \& Data Analysis} {\bf 164}(C),~107306.
\harvarditem{Cox}{1972}{cox1972regression}
Cox, D.~R. \harvardyearleft 1972\harvardyearright , `Regression models and life-tables', {\em Journal of the Royal Statistical Society: Series B (Methodological)} {\bf 34}(2),~187--202.
\harvarditem[Hammer et~al.]{Hammer, Katzenstein, Hughes, Gundacker, Schooley, Haubrich, Henry, Lederman, Phair, Niu et~al.}{1996}{hammer1996trial}
Hammer, S.~M., Katzenstein, D.~A., Hughes, M.~D., Gundacker, H., Schooley, R.~T., Haubrich, R.~H., Henry, W.~K., Lederman, M.~M., Phair, J.~P., Niu, M. et~al. \harvardyearleft 1996\harvardyearright , `A trial comparing nucleoside monotherapy with combination therapy in hiv-infected adults with cd4 cell counts from 200 to 500 per cubic millimeter', {\em New England Journal of Medicine} {\bf 335}(15),~1081--1090.
\harvarditem[Jin et~al.]{Jin, Lin, Wei \harvardand\ Ying}{2003}{ji03}
Jin, Z., Lin, D., Wei, L. \harvardand\ Ying, Z. \harvardyearleft 2003\harvardyearright , `Rank-based inference for the accelerated failure time model', {\em Biometrika} {\bf 90}(2),~341--353.
\harvarditem[Jin et~al.]{Jin, Lin \harvardand\ Ying}{2006}{ji06}
Jin, Z., Lin, D. \harvardand\ Ying, Z. \harvardyearleft 2006\harvardyearright , `On least-squares regression with censored data', {\em Biometrika} {\bf 93}(1),~147--161.
\harvarditem{Johnson}{2009}{jo09}
Johnson, B.~A. \harvardyearleft 2009\harvardyearright , `On lasso for censored data', {\em Electronic Journal of Statistics} {\bf 3}(none),~485--506.
\harvarditem[Johnson et~al.]{Johnson, Lin \harvardand\ Zeng}{2008}{jo08}
Johnson, B.~A., Lin, D.~Y. \harvardand\ Zeng, D. \harvardyearleft 2008\harvardyearright , `Penalized estimating functions and variable selection in semiparametric regression models', {\em Journal of the American Statistical Association} {\bf 103}(482),~672--680.
\harvarditem{Juraska \harvardand\ Juraska}{2022}{juraska2022package}
Juraska, M. \harvardand\ Juraska, M.~M. \harvardyearleft 2022\harvardyearright , `Package ‘speff2trial’', {\em CRAN Repository} .
\harvarditem{Kosorok \harvardand\ Moodie}{2015}{kosorok2015adaptive}
Kosorok, M.~R. \harvardand\ Moodie, E.~E. \harvardyearleft 2015\harvardyearright , {\em Adaptive treatment strategies in practice: planning trials and analyzing data for personalized medicine}, SIAM.
\harvarditem[Lee et~al.]{Lee, Choi \harvardand\ Choi}{2024}{lee2024censored}
Lee, J., Choi, T. \harvardand\ Choi, S. \harvardyearleft 2024\harvardyearright , `Censored broken adaptive ridge regression in high-dimension', {\em Computational Statistics} pp.~1--26.
\harvarditem{Lee \harvardand\ Kim}{2025}{lee2025qlearninglinear}
Lee, J. \harvardand\ Kim, J.-M. \harvardyearleft 2025\harvardyearright , `Counterfactual q-learning via the linear buckley--james method for longitudinal survival data', {\em Journal of the Royal Statistical Society: Series A (Statistics in Society)} .
\newline\harvardurl{https://doi.org/10.1093/jrsssa/qnaf123}
\harvarditem[Li et~al.]{Li, Dicker \harvardand\ Zhao}{2014}{li14}
Li, Y., Dicker, L. \harvardand\ Zhao, S. \harvardyearleft 2014\harvardyearright , `The {D}antzig selector for censored linear regression models', {\em Statistica Sinica} {\bf 24}(1),~251--268.
\harvarditem[Moodie et~al.]{Moodie, Chakraborty \harvardand\ Kramer}{2012}{moodie2012q}
Moodie, E.~E., Chakraborty, B. \harvardand\ Kramer, M.~S. \harvardyearleft 2012\harvardyearright , `Q-learning for estimating optimal dynamic treatment rules from observational data', {\em Canadian Journal of Statistics} {\bf 40}(4),~629--645.
\harvarditem[Simoneau et~al.]{Simoneau, Moodie, Nijjar, Platt, Investigators et~al.}{2020}{simoneau2020estimating}
Simoneau, G., Moodie, E.~E., Nijjar, J.~S., Platt, R.~W., Investigators, S. E. R. A. I.~C. et~al. \harvardyearleft 2020\harvardyearright , `Estimating optimal dynamic treatment regimes with survival outcomes', {\em Journal of the American Statistical Association} {\bf 115}(531),~1531--1539.
\harvarditem[Song et~al.]{Song, Wang, Zeng \harvardand\ Kosorok}{2015}{song2015penalized}
Song, R., Wang, W., Zeng, D. \harvardand\ Kosorok, M.~R. \harvardyearleft 2015\harvardyearright , `Penalized q-learning for dynamic treatment regimens', {\em Statistica Sinica} {\bf 25}(3),~901.
\harvarditem{Wahed \harvardand\ Thall}{2013}{wahed2013evaluating}
Wahed, A.~S. \harvardand\ Thall, P.~F. \harvardyearleft 2013\harvardyearright , `Evaluating joint effects of induction--salvage treatment regimes on overall survival in acute leukaemia', {\em Journal of the Royal Statistical Society Series C: Applied Statistics} {\bf 62}(1),~67--83.
\harvarditem[Wang et~al.]{Wang, Nan, Zhu \harvardand\ Beer}{2008}{wa08}
Wang, S., Nan, B., Zhu, J. \harvardand\ Beer, D.~G. \harvardyearleft 2008\harvardyearright , `Doubly penalized {B}uckley-{J}ames method for survival data with high-dimensional covariates', {\em Biometrics} {\bf 64}(1),~132--140.
\harvarditem{Wang \harvardand\ Wang}{2010}{wang2010buckley}
Wang, Z. \harvardand\ Wang, C. \harvardyearleft 2010\harvardyearright , `Buckley-james boosting for survival analysis with high-dimensional biomarker data', {\em Statistical Applications in Genetics and Molecular Biology} {\bf 9}(1).
\harvarditem[Wang et~al.]{Wang, Wang \harvardand\ Suggests}{2023}{wang2023package}
Wang, Z., Wang, M.~Z. \harvardand\ Suggests, T. \harvardyearleft 2023\harvardyearright , `Package ‘bujar’', {\em CRAN Repository} .
\harvarditem{Watkins \harvardand\ Dayan}{1992}{wa92}
Watkins, C.~J. \harvardand\ Dayan, P. \harvardyearleft 1992\harvardyearright , `Q-learning', {\em Machine learning} {\bf 8},~279--292.
\harvarditem{Wei}{1992}{wei1992accelerated}
Wei, L.-J. \harvardyearleft 1992\harvardyearright , `The accelerated failure time model: a useful alternative to the cox regression model in survival analysis', {\em Statistics in medicine} {\bf 11}(14-15),~1871--1879.
\harvarditem{Zeng \harvardand\ Lin}{2007}{ze07}
Zeng, D. \harvardand\ Lin, D. \harvardyearleft 2007\harvardyearright , `Efficient estimation for the accelerated failure time model', {\em Journal of the American Statistical Association} {\bf 69}(4),~507--564.
\end{thebibliography}