Efficient Poisson Subsampling for the Partially Linear Additive Cox Model
Abstract
To address the computational and storage challenges often encountered in large-scale survival data analysis, we propose an efficient Poisson subsampling method for the partially linear additive Cox model. This model provides a flexible yet interpretable framework by incorporating linear covariate effects, additive nonparametric components for nonlinear covariates, and a nonparametric baseline hazard function. The proposed method adopts B-spline basis functions to approximate the nonparametric components and employs the decorrelated score technique to construct a Poisson subsampling-based estimation equation, based on which we establish the asymptotic normality of the resulting estimator and derive the optimal subsampling probabilities according to the L-optimality criterion. Furthermore, we design a two-step adaptive algorithm for practical implementation. The proposed approach enables computationally efficient statistical inference for large-scale survival analysis without processing the full dataset. We validate the performance of the proposed method through extensive simulation studies and a real-world application to a lymphoma cancer dataset, demonstrating its efficiency and accuracy in large-scale settings.
Keywords: Massive data; Optimal subsampling; Partially linear additive Cox model; Survival data.
1 Introduction
With the rapid growth of data volume in modern application scenarios, traditional statistical methods frequently face computationally infeasible challenges, particularly in resource-constrained environments. Subsampling has emerged as a powerful and widely adopted technique in statistical and computational research. By carefully selecting a representative subset from the full dataset, subsampling allows for parameter estimation with reduced resource consumption while preserving the statistical validity of the resulting estimates. This makes it a practical solution for modern large-scale data challenges.
Extensive research efforts have been devoted to the development of subsampling strategies for various statistical models. Specifically, 11 and 10 introduced L-optimality based methods for logistic and quantile regression, respectively; 15 studied the softmax regression via Poisson subsampling; 7 investigated A-optimality methods for the functional generalized linear model; Shao et al. (9) studied the partially linear model by constructing an unbiased Neyman-orthogonal score function; 20, Zuo et al. (24); 16 proposed optimal subsample selection approaches in distributed computing environments, and 18 developed an online strategy for the streaming multinomial logistic model. In high-dimensional settings, 21 provided a cross-validation procedure for selecting optimal subsamples. Comprehensive reviews of this field can be found in 14 (14) and Yu et al. (17) and the references therein.
Survival data arise frequently in many fields, such as biomedicine 2, engineering 13, and social sciences (Lee et al., 5). Although recent years have seen the development of subsampling methods for such data, these methods are limited to settings where covariates have parametric effects. For instance, Zuo et al. (23) investigated the additive hazards model 6, and Zhang et al. (22) and Shao et al. (8) studied the Cox model 1, with all these studies employing L-optimality subsampling; In distributed settings, Zhang et al. (19) further proposed a divide-and-conquer based subsampling approach for the Cox model. The partially linear additive Cox (PLA-Cox; 4, 4) model, which includes the Cox model as a special case, integrates both parametric and nonparametric covariate effects. Given its wide applicability, there is an urgent need for subsampling methods to alleviate the computational burden of fitting this model to large-scale data.
In this article, we propose an efficient Poisson subsampling method tailored for the PLA-Cox model. Our approach approximates nonparametric components using B-spline basis functions, constructs Poisson subsampling-based estimation equations via the decorrelated score technique, and derives the optimal subsampling probabilities based on the L-optimality criterion. For practical implementation, we design a two-stage adaptive algorithm. In the first stage, a pilot subsample is drawn via uniform Poisson subsampling to obtain pilot estimators. These estimators are then used to determine the optimal subsampling probabilities for selecting the final subsample and estimating the target parameters in the second stage. The differences between our work and prior studies on subsampling in survival analysis are as follows: First, as mentioned above, existing subsampling methods are designed for models with fully parametric covariate effects (e.g., the additive hazards model and the Cox model), whereas our method targets the PLA-Cox model, which includes both parametric and nonparametric covariate effects. Second, we develop an efficient Poisson subsampling scheme for the PLA-Cox model, which implements sampling without replacement (thus avoiding duplicate observations), distinguishing it from existing with-replacement subsampling strategies. Third, to address the nonparametric components of covariate effects, we adopt B-spline basis functions for approximation and employ the decorrelated score technique to eliminate the confounding influence of nuisance parameters. This distinguishes our estimation strategy from existing subsampling methods. Moreover, the introduction of this nonparametric component substantially complicates the theoretical development, as it necessitates validating the B-spline approximation and adjusting for the decorrelation procedure.
The remainder of this paper is outlined as follows: In Section 2, we develop a Poisson subsampling method for the PLA-Cox model, along with the asymptotic normality of the corresponding estimator. In Section 3, we focus on determining the optimal subsampling probabilities, and propose a two-stage algorithm that enables the adaptive implementation of the optimal subsampling method. In Section 4, we conduct simulation studies to examine the finite-sample performance of the proposed method. An application to a real dataset is provided in Section 5. Concluding remarks are presented in Section 6. All proofs are deferred to the Supplementary Material.
2 Poisson Subsampling-Based Estimation
Let and be the failure time and censoring time, respectively. Denote as the covariate vector for the parametric components, and as the covariates for the nonparametric components. Conditional on and , we assume that and are independent. The observed time is defined as , with event indicator , where denotes the indicator function. The full dataset consists of independent and identically distributed observations from the population, denoted as . The PLA-Cox model assumes
| (1) |
where is the conditional hazard function given and , is an unspecified baseline hazard function, is a -dimensional regression parameter vector, and the nonparametric components consists of smooth functions of , the th component of , satisfying for .
We approximate each by using B-spline basis functions. Suppose that each takes values in , where and are finite numbers. Define as the space of polynomial splines of degree . Consider a knot sequence with interior knots
where may increase with sample size . For each , we approximate by a linear combination of B-spline basis functions of order . The nonparametric components in model (1) can be approximated as , where is the B-spline basis vector, and is the corresponding coefficient vector.
The negative log pseudo-partial likelihood function for the full data is
where , is a prespecified positive constant, and is a counting process, . However, when the full dataset is large, minimizing brings an enormous computational burden, severely limiting the practical application of the PLA-Cox model in massive data scenarios. We employ the Possion subsampling method to address this challenge. Specifically, the th observation is assigned a sampling probability satisfying with , where represents the expected subsample size. Let be independent uniform random variables on that are independent of the observed data. For each individual , we include the tuple in the subsample if . This yields the subsampled dataset , where for each observation , are the linear-effect covariates, are the nonparametric confounders, is the event indicator, is the response variable, and is the sampling probability. The realized subsample size satisfies . Based on , a negative weighted log pseudo-partial likelihood is given as follows:
where , .
In the PLA-Cox model, is the target parameter requiring rigorous inference, whereas is nuisance parameter which merely capture nonparametric confounding effects. As a result, we adopt the decorrelated score technique to eliminate the influence of on . Let
and
If is available, we propose the following decorrelated score function
where is the projection matrix of onto . Nevertheless, since and are unavailable, we replace them with pilot estimators obtained from a pilot subsample. Given the expected pilot subsample size , we draw a pilot subsample via uniform Poisson subsampling with , . Based on this subsample, we compute and , where
and , . Finally, we obtain the subsampling estimator by solving .
Our method inherits two core statistical advantages. First, from the computational perspective, the proposed method significantly reduces the computational burden by only utilizing a small fraction of the full dataset, avoiding the high computational cost of full-data optimization. Second, in terms of statistical inference, the orthogonality between the decorrelated score function and the gradient vector of the nuisance parameter eliminates the interference of on , ensuring valid inference for the target parameter in the presence of nonparametric covariate effects. Our approach differs from existing subsampling methods in survival analysis, which are designed exclusively for models with fully parametric covariate effects and thus cannot be directly extended to our setting involving both parametric and nonparametric covariate components.
For any vector , define , and . Let
| and | ||||
where , and = 0, 1. The following conditions are required for establishing the asymptotic properties of .
- (C1)
, and .
- (C2)
The covariate vector is bounded.
- (C3)
The limit of is nonsingular.
- (C4)
, and .
Condition (C1) is a regularity assumption in survival analysis. Condition (C2) is a boundedness constraint on , which is a conventional requirement in the literature 3. This assumption maintains practical relevance for biomedical survival data, where typical covariates, such as treatment indicators, blood pressure measurements, age, and gender, are naturally bounded. Condition (C3) is imposed to guarantee the invertibility of the matrix . Condition (C4) specifies that the minimal subsampling probability scales at order 1/n, which is consistent with Assumption 5 in Wang et al. (12). Moreover, and ensure that the subsample sizes are much smaller than the full data size, a natural requirement to alleviate the computational burden. Theorem 1 below develops the asymptotic normality of .
Theorem 1
Under conditions (C1)-(C4), conditional on in probability, we have
where denotes convergence in distribution, and with
and , .
Next, we aim to find the optimal probabilities that minimize the covariance matrix in Theorem 1. Because is equivalent to in the Loewner ordering, this objective reduces to minimizing . However, since two positive definite matrices are not necessarily comparable, we instead minimize , a criterion justified by the fact that implies . This strategy is known as the L-optimality criterion, which has been widely adopted in the field of subsampling 10. We denote these optimal probabilities as , and their closed-form expressions are presented in Theorem 2.
Theorem 2
Suppose that conditions (C1)-(C4) are satisfied. If the subsampling probabilities are chosen as
| (2) |
then attains its minimum, where ,
are the order statistics of , , , and is an integer such that
and
with .
Remark 1
As shown by Wang et al. (12), we can practically use , and the performance of the estimator with is comparable to that obtained with the optimal sampling probabilities. This strategy has also been adopted in other Poisson subsampling-based methods (Wang et al. 12; 16 16; Yao et al. 15) and can improve computational efficiency.
3 Two-Step Subsampling Algorithm
As shown in Section 2, the L-optimal subsampling probabilities depend on several unobservable quantities: the full data estimator , the projection matrix , , , , and . To overcome the unavailability of these quantities in advance, we propose the following two-step subsampling algorithm.
Step 1: Pilot Subsampling and Hybrid Probability Calculation. A pilot subsample of expected size is drawn from via uniform Poisson subsampling. Based on this pilot subsample, we compute pilot estimators and . We estimate the cumulative baseline hazard by the following Breslow estimator computed on the pilot subsample,
where , . In addition, we replace , , and with their pilot subsample counterparts:
| and | ||||
which are computed by using the pilot subsample. Apply these substitutions to (2), yielding the implementable optimal subsampling probabilities,
where
with , .
To address numerical instability caused by near-zero , we introduce a hybrid subsampling scheme: , where (e.g., ) controls the proportion of mixture. This scheme balances optimality and stability, where the term retains the optimal subsampling feature (selecting informative observations) and guarantees a non-zero subsampling probability for all observations.
Step 2: Final Subsampling and Estimator Computation. We generate independent uniform random variables for each observation in , and include observation in the final subsample if , where is the expected final subsample size. The two-step Poisson subsampling estimator is then computed by solving the decorrelated score equation
where has the same expression as except that is replaced with . Our proposed two-step subsampling method is summarized in the following Algorithm 1.
Input: The full data ; the expected pilot subsample size ; the expected final subsample size ; the proportion of mixture .
Step 1: Pilot Subsampling and Hybrid Probability Calculation
- 1.
Draw a uniform pilot subsample: , where is the actual pilot subsample size.
- 2.
Compute , and .
- 3.
Calculate hybrid subsampling probabilities:
Step 2: Final Subsampling and Estimator Computation
- 1.
For each :
- •
Generate ;
- •
If , we include in the subsample.
- •
- 2.
By solving the estimating equation , we obtain the estimator .
We establish the asymptotic normality of in the next Theorem 3.
Theorem 3
Assume conditions (C1)-(C4) hold. Conditional on and , the two-step Poisson subsampling estimator satisfies
where and
For the purposes of conducting statistical inference, we estimate the covariance matrix of by
where
and
together with , ,
| and | ||||
.
The proposed two-step subsampling algorithm enables data-dependent sampling probabilities and adapts to the intrinsic features of complex large-scale survival datasets. Notably, the covariance estimator is merely constructed from the final subsample, which effectively alleviates computational burden while providing reliable uncertainty quantification for to support statistical inference.
4 Numerical Simulations
In this section, simulation studies were conducted to evaluate the estimation accuracy and computational efficiency of the proposed Poisson subsampling estimator. Let , and . We considered two distinct data-generating processes for covariate :
-
Case I: followed a multivariate normal distribution , where denotes the entry of , with if and if .
-
Case II: followed a mixed multivariate normal distribution , where .
Let , with and independently drawn from a uniform distribution on . The nonparametric functions were specified as and . The interior knots were . The censoring time was simulated from a uniform distribution on , where was selected to give a certain censoring rate (CR). We set CR at and , respectively. All the simulation results presented below are based on 1000 replications.
4.1 Target 1: Sensitivity to the Pilot Subsample Size
We began by examining the effect of the pilot subsample size under and . Define as the empirical standard error of estimator computed over 1000 simulation repetitions, where is the th component of , . The average standard error (ASE) was then given by . Table 1 displays ASE for the subsampling estimator across pilot subsample sizes and final subsample sizes . From Table 1, the ASE shows minimal fluctuations as increases from 500 to 700. For example, in Case I with CR=20% and the final subsample size , the ASE is when , and decreases only slightly to for and , corresponding to a maximum reduction of just . This indicates that is sufficient to generate stable pilot estimators, as further increasing does not meaningfully improve the precision of . In addition, as expected, the ASE consistently decreases with the increase of the subsample size , while the higher censoring rate (CR=60%) generally leads to a larger ASE compared to the lower censoring rate (CR=20%).
| CR= | CR= | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Case I | 500 | 0.0733 | 0.0616 | 0.0538 | 0.0883 | 0.0748 | 0.0658 | ||
| 600 | 0.0720 | 0.0607 | 0.0535 | 0.0865 | 0.0740 | 0.0651 | |||
| 700 | 0.0714 | 0.0603 | 0.0535 | 0.0858 | 0.0729 | 0.0646 | |||
| Case II | 500 | 0.0725 | 0.0619 | 0.0543 | 0.1117 | 0.0815 | 0.0696 | ||
| 600 | 0.0715 | 0.0611 | 0.0538 | 0.0871 | 0.0730 | 0.0645 | |||
| 700 | 0.0708 | 0.0603 | 0.0528 | 0.0858 | 0.0721 | 0.0633 | |||
4.2 Target 2: Sensitivity to the Mixing Parameter
We analyzed how ASE values vary with under and . The approximate sampling distribution exhibits two boundary behaviors: it converges to the optimal subsampling distribution as approaches 0, and approaches the uniform subsampling distribution when approaches 1. Table 2 shows that achieves the lowest ASE across all scenarios, striking the optimal balance between optimality and stability. For example, in Case I with a 20% censoring rate and , its ASE of 0.0538 is about 1.5% lower than that of and 1.3% lower than that of . Thus, eliminates the numerical instability caused by near zero probabilities without sacrificing efficiency. As increases to 0.3 or 0.5, the ASE generally rises, reflecting a loss of optimality as the sampling distribution becomes too uniform. The advantage of is greater under challenging conditions such as a high censoring rate of 60% or asymmetric covariates as in Case II, where stability is critical. We therefore recommend for practical implementation of our method.
| CR= | CR= | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Case I | 600 | 0.0740 | 0.0713 | 0.0718 | 0.0723 | 0.0907 | 0.0883 | 0.0887 | 0.0939 | ||
| 800 | 0.0629 | 0.0616 | 0.0609 | 0.0624 | 0.0768 | 0.0748 | 0.0758 | 0.0787 | |||
| 1000 | 0.0546 | 0.0538 | 0.0545 | 0.0556 | 0.0682 | 0.0658 | 0.0669 | 0.0698 | |||
| Case II | 600 | 0.0729 | 0.0725 | 0.0722 | 0.0734 | 0.1139 | 0.1017 | 0.1002 | 0.1066 | ||
| 800 | 0.0620 | 0.0619 | 0.0609 | 0.0621 | 0.0996 | 0.0815 | 0.0811 | 0.0845 | |||
| 1000 | 0.0550 | 0.0543 | 0.0547 | 0.0549 | 0.0861 | 0.0696 | 0.0727 | 0.0755 | |||
4.3 Target 3: Sensitivity to the Spline Degree
Following the preceding investigation, we assessed the sensitivity of the proposed estimator to the spline degree under and . The spline degree controls the flexibility of the B-spline basis used to approximate the nonparametric component . A lower degree, such as for a cubic spline, is computationally efficient but may underfit complex nonparametric relationships. A higher degree, such as for a quintic spline, can capture more intricate patterns but risks overfitting and higher computational cost. Table 3 shows that the ASE values for =3,4 and 5 are nearly identical, with maximum differences of less than 5%, showing that the spline degree has minimal impact on the estimation accuracy of . For example, in Case I with a 20% censoring rate and r=1000, the ASE value for =3 is 0.0538, which is only 3.4% lower than the value of 0.0558 for =5. Given the minimal differences in ASE across values, the robustness of the estimator to the spline degree, and the computational efficiency of lower degrees, =3 is the optimal choice.
| CR= | CR= | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Case I | 600 | 0.0733 | 0.0741 | 0.0753 | 0.0883 | 0.0915 | 0.1252 | ||
| 800 | 0.0616 | 0.0625 | 0.0631 | 0.0748 | 0.0777 | 0.0952 | |||
| 1000 | 0.0538 | 0.0556 | 0.0558 | 0.0658 | 0.0687 | 0.0833 | |||
| Case II | 600 | 0.0725 | 0.0736 | 0.0738 | 0.0892 | 0.0913 | 0.0996 | ||
| 800 | 0.0619 | 0.0620 | 0.0627 | 0.0815 | 0.0786 | 0.0877 | |||
| 1000 | 0.0543 | 0.0548 | 0.0553 | 0.0696 | 0.0735 | 0.0838 | |||
4.4 Target 4: Performance Comparison of Subsampling Methods
To assess the performance advantage of our L-optimality criterion-based subsampling approach (Lopt), we compared it with uniform subsampling (UNIF) and the method of 22 (denoted as “Oracle”). We set , and . Under the UNIF scheme, each observation receives equal subsampling probability for . The Oracle method assumes prior knowledge of the nonparametric functions and . Performance evaluation employs four metrics: estimated bias (BIAS) computed as the sample mean of the estimates minus the full data estimator, empirical standard error (SE), mean estimated standard error (ESE), and empirical 95% coverage probability (CP) based on normal approximation.
| Subsample Size | Methods | BIAS | SE | ESE | CP | |
|---|---|---|---|---|---|---|
| Lopt | -0.0008 | 0.0755 | 0.0778 | 0.958 | ||
| UNIF | -0.0250 | 0.0858 | 0.0960 | 0.964 | ||
| Oracle | -0.0010 | 0.0765 | 0.0737 | 0.952 | ||
| Lopt | -0.0045 | 0.0619 | 0.0662 | 0.962 | ||
| UNIF | -0.0167 | 0.0719 | 0.0827 | 0.974 | ||
| Oracle | 0.0024 | 0.0653 | 0.0629 | 0.942 | ||
| Lopt | -0.0030 | 0.0549 | 0.0587 | 0.978 | ||
| UNIF | -0.0177 | 0.0638 | 0.0740 | 0.966 | ||
| Oracle | 0.0001 | 0.0559 | 0.0558 | 0.940 |
| Subsample Size | Methods | BIAS | SE | ESE | CP | |
|---|---|---|---|---|---|---|
| Lopt | 0.0054 | 0.0910 | 0.0938 | 0.958 | ||
| UNIF | -0.0329 | 0.1323 | 0.1429 | 0.958 | ||
| Oracle | 0.0096 | 0.0911 | 0.0904 | 0.944 | ||
| Lopt | 0.0025 | 0.0773 | 0.0800 | 0.946 | ||
| UNIF | -0.0023 | 0.1134 | 0.1225 | 0.960 | ||
| Oracle | 0.0094 | 0.0785 | 0.0770 | 0.954 | ||
| Lopt | 0.0038 | 0.0696 | 0.0710 | 0.950 | ||
| UNIF | -0.0228 | 0.0997 | 0.1093 | 0.952 | ||
| Oracle | 0.0075 | 0.0685 | 0.0684 | 0.938 |
Tables 4-7 show that all three methods produce near-zero BIAS values, implying the unbiasedness of the subsampling estimator for . The results for the other components of are similar and omitted. For Lopt, the ESE closely matches the corresponding SE, demonstrating that the proposed covariance estimator is reliable. For instance, at , SE = 0.0549 and ESE = 0.0587, a relative difference of only 6.9%. In contrast, UNIF shows a larger discrepancy, as seen at where SE = 0.0638 and ESE = 0.0740, a relative difference of 16.0%. This is because uniform sampling ignores the information of observations, leading to less accurate standard error estimation. This advantage of Lopt grows with the subsample size , as more high-information observations are retained.
| Subsample Size | Methods | BIAS | SE | ESE | CP | |
|---|---|---|---|---|---|---|
| Lopt | -0.0117 | 0.0727 | 0.0745 | 0.960 | ||
| UNIF | -0.0257 | 0.0865 | 0.0924 | 0.956 | ||
| Oracle | -0.0034 | 0.0740 | 0.0722 | 0.948 | ||
| Lopt | -0.0090 | 0.0631 | 0.0634 | 0.944 | ||
| UNIF | -0.0188 | 0.0761 | 0.0794 | 0.956 | ||
| Oracle | -0.0001 | 0.0634 | 0.0617 | 0.942 | ||
| Lopt | -0.0005 | 0.0565 | 0.0563 | 0.946 | ||
| UNIF | -0.0159 | 0.0672 | 0.0709 | 0.950 | ||
| Oracle | -0.0008 | 0.0530 | 0.0548 | 0.954 |
| Subsample Size | Methods | BIAS | SE | ESE | CP | |
|---|---|---|---|---|---|---|
| Lopt | -0.0066 | 0.0932 | 0.0915 | 0.958 | ||
| UNIF | -0.0337 | 0.1287 | 0.1384 | 0.962 | ||
| Oracle | 0.0088 | 0.0891 | 0.0891 | 0.958 | ||
| Lopt | -0.0026 | 0.0791 | 0.0766 | 0.950 | ||
| UNIF | -0.0241 | 0.1103 | 0.1181 | 0.960 | ||
| Oracle | 0.0108 | 0.0738 | 0.0760 | 0.958 | ||
| Lopt | -0.0027 | 0.0693 | 0.0678 | 0.958 | ||
| UNIF | -0.0165 | 0.0998 | 0.1053 | 0.962 | ||
| Oracle | 0.0098 | 0.0659 | 0.0674 | 0.960 |
Figure 1 visualizes the ASE comparisons across all scenarios, further validating the superiority of Lopt. In all four subfigures, the ASE of Lopt is substantially lower than that of UNIF, and nearly identical to that of Oracle, especially when is large. For example, in Case I with CR=20%, the ASE of Lopt at is almost indistinguishable from Oracle, while UNIF’s ASE is significantly higher. This demonstrates that Lopt achieves near-optimal estimation accuracy without requiring prior knowledge of nonparametric functions.
4.5 Target 5: Computational Efficiency Evaluation
We conducted additional simulations to evaluate the computational efficiency of our proposed method. Data generation followed the Case I specification with covariates constructed as previously described, except that the regression coefficient vector was with dimensions and . We set , , and . All computations were performed in R on a laptop with 32GB RAM. We report in Table 8 the CPU times (in seconds) for computing , averaged over 100 repetitions (excluding data generation time). From Table 8 we see that both Lopt and UNIF achieve substantial computational savings compared to full data analysis. For example, when and , the CPU time of Lopt is 7.459 seconds, which is only 4.2% of that of full data analysis at 178.67 seconds. This implies that subsampling effectively reduces the computational burden by focusing on a small subset of informative observations, rather than processing the entire large dataset. UNIF is slightly faster than Lopt in all scenarios, which is attributed to the fact that UNIF does not require calculating optimal subsampling probabilities or allocation sizes. While UNIF offers faster computation, it suffers from a larger SE. In contrast, Lopt balances computational efficiency and statistical performance. The computational difference between subsampling methods and full data analysis widens with increasing . This is because full data analysis requires processing all observations with covariates, leading to an increase in computational cost as grows, while subsampling methods only process a small subset of all observations, making the impact of on CPU time much smaller.
| Method | ||||||||
|---|---|---|---|---|---|---|---|---|
| =5 | Lopt | 1.538 | 4.363 | 7.459 | ||||
| UNIF | 0.982 | 1.259 | 1.494 | |||||
| Full data | 15.25 | 79.73 | 178.67 | |||||
| =50 | Lopt | 12.906 | 28.844 | 44.644 | ||||
| UNIF | 9.197 | 10.922 | 37.068 | |||||
| Full data | 58.95 | 292.14 | 628.01 |
5 A Real Data Example
We illustrated the practical performance of the proposed two-step Poisson subsampling method by using a large-scale lymphoma cancer dataset from the Surveillance, Epidemiology, and End Results (SEER) program. The SEER is a authoritative and widely used source for population-level cancer survival research in the United States. The dataset included complete records of lymphoma patients diagnosed between 1975 and 2007. In this survival analysis, the follow-up time was censored at 60 months (5 years) after diagnosis. Within the cohort, 46,067 individuals experienced the event of interest, leading to an empirical censoring rate of , which was consistent with the moderate-to-high censoring settings considered in our simulation studies. The parametric covariate was biological sex (coded as for male and for female), and the nonparametric nuisance covariate was age at diagnosis (centered and standardized). The full data estimator using all observations was , which served as the gold-standard benchmark for evaluating the subsampling estimator.
We set , , and used interior knots at . We generated independent subsamples at four incremental sizes: . For each subsample size, we obtained both Lopt and UNIF estimators, and reported the point estimate, SE, ESE based on 1000 replications. The results were summarized in Table 9.
| Method | Estimate | SE | ESE | |
|---|---|---|---|---|
| Lopt | 0.18310 | 0.11297 | 0.10990 | |
| UNIF | 0.18983 | 0.12164 | 0.13277 | |
| Lopt | 0.18353 | 0.09584 | 0.09434 | |
| UNIF | 0.18848 | 0.10632 | 0.11429 | |
| Lopt | 0.18548 | 0.08281 | 0.08395 | |
| UNIF | 0.18905 | 0.09645 | 0.10204 | |
| Lopt | 0.18684 | 0.07391 | 0.07648 | |
| UNIF | 0.19012 | 0.08991 | 0.09302 |
Both Lopt and UNIF produce point estimates close to the full data estimator , confirming that subsampling-based inference is reliable for large-scale survival data. Notably, the Lopt estimates are systematically closer to the full data estimator than UNIF across all subsample sizes. As increases from 600 to 1200, the Lopt estimates improve steadily from to , gradually converging toward . In contrast, the UNIF estimates remain relatively unstable and deviate more noticeably from the benchmark. The Lopt method achieves consistently smaller SE than UNIF at every subsample size. For instance, at , SE of Lopt is , which is lower than SE of UNIF (). At , the efficiency gain increases to . This indicates that the L-optimal subsampling strategy effectively prioritizes informative observations, leading to more precise estimation than uniform sampling.
Moreover, ESE closely matches the SE for Lopt across all subsample sizes, with relative differences below . In contrast, ESE of UNIF noticeably overstates uncertainty and deviates from SE, especially at small subsample sizes. This validates that the proposed covariance estimator is reliable for practical applications. Moreover, as the subsample size increases, both methods improve in precision, with SE and ESE decreasing monotonically. The Lopt method maintains its efficiency advantage across all levels of , demonstrating its stability and practical usefulness even with relatively small subsamples.
6 Concluding Remarks
In this article, we proposed an efficient Poisson subsampling method for the PLA-Cox model to address the computational and storage challenges of large-scale survival data analysis. The proposed method uses B-spline basis functions to approximate the additive nonparametric components and adopts the decorrelated score technique to construct a subsampling-based estimation equation. We established the asymptotic distribution of the resulting estimator and derived optimal subsampling probabilities under the L-optimality criterion. A two-step adaptive algorithm was developed for practical implementation. Extensive simulation studies and a real-world application to a lymphoma cancer dataset were conducted to validate the performance of the proposed method.
This research has several limitations that provide directions for future work. First, the hybrid subsampling scheme introduces a tuning parameter , whose selection is currently based on empirical experience (e.g., ). Developing a data-driven method to select adaptively would further improve the method’s robustness. Second, our framework focuses on independent and identically distributed datasets. Extending the Poisson subsampling framework to dependent survival data (e.g., clustered survival data or longitudinal survival data) is a valuable direction for future research. Third, the current method uses B-spline approximations for the nonparametric component . Exploring alternative nonparametric approximations (e.g., neural networks) may enhance the flexibility in capturing complex nonlinear relationships.
AI Use Statement
During the preparation of this work, the authors use Deepseek-V2 exclusively for English language editing and readability improvement.
Disclosure Statement
The authors report that there are no competing interests to declare.
SUPPLEMENTARY MATERIAL
-
The supplementary material contains proofs of all the theoretical results.
Acknowledgments
Dongxiao Han’s research was partly supported by the National Natural Science Foundation of China (12471259 and 12231011), and Academy for Advanced Interdisciplinary Studies of Nankai University, and Tianjin Municipal Natural Science Foundation (23JCYBJC01270). Liuquan Sun’s research was partially supported by the National Natural Science Foundation of China (12571299). Chunjie Wang’s research was partly supported by the National Natural Science Foundation of China (12271060), and the Outstanding Youth Fund Project of Jilin Provincial Natural Science Foundation (20250101011JJ). Dehui Wang’s research was partly supported by the National Natural Science Foundation of China (12271231, 1247012719).
References
- Regression models and life-tables (with discussions). Journal of the Royal Statistical Society, Series B (Statistical Methodology) 34, pp. 187–220. Cited by: §1.
- Stable cox regression for survival analysis under distribution shifts. Nature Machine Intelligence 6, pp. 1525–1541. Cited by: §1.
- Oracle inequalities for the lasso in the cox model. The Annals of Statistics 41, pp. 1142–1165. External Links: Document Cited by: §2.
- Efficient estimation of the partly linear additive cox model. The Annals of Statistics 27, pp. 1536–1563. Cited by: §1.
- Discrete-time survival analysis incorporating time structure in developmental research. Structural Equation Modeling: A Multidisciplinary Journal 32, pp. 929–940. Cited by: §1.
- Semiparametric analysis of the additive risk model. Biometrika 81, pp. 61–71. Cited by: §1.
- Functional L-optimality subsampling for functional generalized linear models with massive data. Journal of Machine Learning Research 24, pp. 1–41. Cited by: §1.
- Optimal decorrelated score subsampling for Cox regression with massive survival data. Neurocomputing 638, pp. 1–8. Cited by: §1.
- Optimal subsampling for high-dimensional partially linear models via machine learning methods. Journal of Machine Learning Research 26, pp. 1–70. Cited by: §1.
- Optimal subsampling for quantile regression in big data. Biometrika 108, pp. 99–112. Cited by: §1, §2.
- Optimal subsampling for large sample logistic regression. Journal of the American Statistical Association 113, pp. 829–844. Cited by: §1.
- Sampling with replacement vs poisson sampling: a comparative study in optimal subsampling. IEEE Transactions on Information Theory 68, pp. 6605–6630. Cited by: §2, Remark 1.
- Glioma survival analysis empowered with data engineering—a survey. IEEE Access 9, pp. 43168–43191. Cited by: §1.
- A review on optimal subsampling methods for massive datasets. Journal of Data Science 19, pp. 151–172. Cited by: §1.
- Optimal poisson subsampling for softmax regression. Journal of Systems Science and Complexity 36, pp. 1609–1625. Cited by: §1, Remark 1.
- Optimal distributed subsampling for maximum quasi-likelihood estimators with massive data. Journal of the American Statistical Association 117, pp. 265–276. Cited by: §1, Remark 1.
- A review on design inspired subsampling for big data. Statistical Papers 65, pp. 467–510. Cited by: §1.
- Optimal subsampling for data streams with measurement constrained categorical responses. Journal of Computational and Graphical Statistics 34, pp. 994–1004. Cited by: §1.
- DsubCox: a fast subsampling algorithm for Cox model with distributed and massive survival data. International Journal of Biostatistics 21, pp. 53–65. Cited by: §1.
- Distributed subdata selection for big data via sampling-based approach. Computational Statistics and Data Analysis 153, pp. 107072. External Links: Document Cited by: §1.
- Refitted cross-validation estimation for high-dimensional subsamples from low-dimension full data. Computational Statistics 41, pp. 1–15. Cited by: §1.
- Approximating partial likelihood estimators via optimal subsampling. Journal of Computational and Graphical Statistics 33, pp. 276–288. Cited by: §1, §4.4.
- Sampling-based estimation for massive survival data with additive hazards model. Statistics in Medicine 40, pp. 441–450. Cited by: §1.
- Optimal subsample selection for massive logistic regression with distributed data. Computational Statistics 36, pp. 2535–2562. Cited by: §1.