Beyond Final-Token Classification: Heterogeneous Readouts for Evidence-Grounded Suicide Risk Detection

Subtask 1 predicts Indicator, Ideation, Behavior, or Attempt. The official risk metric is weighted F1. Evidence is a semicolon-separated set of verbatim phrases and is evaluated by phrase F1 after case-insensitive normalization. A predicted phrase matches a gold phrase when either contains the other, subject to a three-times token-length cap and one-to-one matching. Subtask 2 predicts any subset of 24 factors and is evaluated by macro F1. The overall competition score is

The leaderboard reports a normalized Subtask 1 score and the factor score separately.

We use only the official training set: 1,635 posts from 153 anonymized users. It contains 611 Indicator, 519 Ideation, 391 Behavior, and 114 Attempt instances. The median post length is 38 whitespace-delimited tokens, the mean is 68.8, and the maximum is 2,066. After de-duplicating repeated factor annotations within a post, each post has 2.92 factors on average. Label support ranges from 745 instances for hopelessness to eight for sexual orientation related issues. This imbalance makes accuracy and micro F1 inadequate for factor model selection.

We form three user-grouped folds of sizes 548, 543, and 544. All posts from a user remain in one fold. Retrieval indices, semantic caches that use training labels, thresholds, and probe fitting respect this partition. No external labeled examples are used. Publicly available pretrained models and manually authored operational label cards are declared components of the system.

Prior work has used prompting, pseudo-labeling, and model ensembles when suicide labels are limited [8]. The CLPsych 2024 shared task specifically studied evidence of suicide risk in online posts [4]; participating systems included knowledge self-generation and output refinement with an LLM [5]. In contrast, our evidence decoder is non-generative: every returned phrase is an exact substring of the input. The 24-factor taxonomy is adapted from work that jointly models risk and protective factors over time [1]. We use these constructs as annotation variables, not as clinical diagnoses.

Natural-language label descriptions can transform classification into entailment between an input and a label hypothesis [9]. This is particularly relevant for long-tailed ontologies because parameters can be shared across labels while their semantics remain explicit. Recent analysis further shows that autoregressive multi-label generation behaves differently from independent classification and may suppress otherwise plausible labels [3]. We consequently ask one binary question per factor and read a continuous margin rather than generate a factor list.

The tuned lens decodes predictions from frozen transformer blocks through learned affine maps [6]. RISE trains robust internal classifiers for suicide risk and uses them for ensembling, abstention, and early exit [7]. Our work is complementary: the deployed model still runs to its final layer, and we study a mismatch between the final hidden state and the final answer-token boundary. Label-selective routing is conceptually related to learning to defer [10], but our “experts” are two readout interfaces of one model and the route is frozen at the label level rather than selected per test instance. Finally, probe accuracy alone does not establish that a feature is causally used by the model [11, 12]; we therefore use the term predictively accessible and do not interpret probe weights as psychological mechanisms.

Extractive rationale models separate selection from prediction [13] and can enforce faithfulness by construction when the predictor is restricted to selected text [14]. Sufficiency, comprehensiveness, and compactness regularization have also been studied for multi-label rationale extraction [15]. Our exact-copy constraint prevents fabricated evidence, but the risk model can still access the full post. We therefore claim verbatim grounding, not causal faithfulness.

For a post $x_{i}$ and label $l$, a card $c_{l}$ contains a name, operational definition, inclusion rule, exclusion rule, and confusable labels. A factor prompt has the form

Does the post support the supplied factor?
A = PRESENT; B = ABSENT.

Risk uses four analogous one-vs-rest cards with YES/NO answers. The final-token margin is

where $z^{+}$ and $z^{-}$ are the logits for the positive and negative answer tokens at the answer position.

For long posts, we preserve the opening and ending clauses and add the clauses most similar to the card embedding. E5-large-v2 embeddings [18] provide this relevance score. Factor prompts additionally include one fold-safe positive precedent and one semantically similar negative precedent. Eligibility excludes the validation fold, the current user, exact duplicate text, and the query itself. Negatives prioritize boundary confusions specified on the card. This retrieval is used as context during SFT and scoring; it is not a post-hoc label vote.

Let $h_{i,r}^{(k)}$ be the answer-position state from decoder layer $k$ when post $i$ is queried with risk card $r$. We treat the four cards as binary pairs and fit a shared logistic probe

where $e_{r}$ is a four-dimensional risk identity vector. The predicted risk is $\arg\max_{r}q_{i,r}$. Standardization and logistic regression are fit only on the outer-training users.

Layers $\{15,31,47,63\}$ and $C\in\{0.001,0.01,0.1\}$ were evaluated through inner grouped cross-validation on development Fold 0. Layer 63 (zero indexed) and $C=0.01$ were frozen, then evaluated without retuning on Folds 1 and 2. The confirmation experiment in Sec. VI-B evaluates the full latent decision. For the public submission, we used a more conservative test-time rule: an Indicator prediction was changed to Ideation only when all three fold-specific probes predicted Ideation and their mean Ideation-minus-Indicator probability margin was at least 0.08. This frozen rule changed one test row and left the evidence and factor fields unchanged.

A global latent factor probe uses

where $e_{l}$ is a 24-dimensional label identity vector. The probe is deliberately shared rather than fitting 24 independent heads. It adds only a standardized logistic readout to the frozen 27B representation.

The latent interface is not uniformly better. On development Fold 0, global replacement raises average precision but changes macro F1 by only $+0.0002$ and reduces tail macro F1. We therefore freeze a label route $\rho_{l}$:

Seven labels pass the development gate: psychological capital, social support, poor school performance, sense of responsibility, traumatic experience, low socio-economic status, and interpersonal difficulty. Layer 63, $C=0.001$, the route set, and all threshold procedures are then fixed. Folds 1 and 2 are used only for confirmation. This is a static label route, not a sample-adaptive expert selector.

The base factor score is a probability blend of a Qwen3-14B shared verifier [19] (weight 0.25) and the Qwen3.8-27B verifier (weight 0.75). The 14B component is retained because its errors are not perfectly correlated with the 27B model. Per-label thresholds are estimated from user-grouped out-of-fold predictions. The latent route replaces only the seven designated label scores before thresholding.

Evidence extraction is formulated as selection from a candidate inventory $C(x)$ rather than token generation. The inventory combines sentence and clause boundaries, cue-centered word windows, a fold-trained suicide lexicon, and token regions proposed by a ModernBERT encoder [20]. Every candidate records its exact character offsets and therefore remains a verbatim substring.

The Task 1 adapter receives the provisional risk card, the post, and one candidate span, and returns a support margin. A low-capacity logistic calibrator estimates candidate relevance from the frozen margin and test-time-observable metadata: candidate source, three predicted risk fields, span length, relative position, candidate count, margin rank, distance from the top margin, baseline phrase count, and post length. It does not access gold evidence at inference time.

Candidate probabilities are converted into an evidence set by a frozen threshold of 0.40, exact-string deduplication, and suppression of overlapping events within 40 characters. Risk-conditional caps return zero phrases for Indicator, at most two for Ideation, three for Behavior, and two for Attempt. This prevents the common failure mode of maximizing recall by returning long or redundant spans and mirrors the official one-to-one phrase matching rule.

Our large design space makes ordinary pooled cross-validation susceptible to selection leakage. We therefore separate development from confirmation, as shown in Fig. 2. Fold 0 selects the hidden layer, regularization, route labels, and decoder hyperparameters. These choices are serialized before Folds 1 and 2 are evaluated. A component is accepted only if it satisfies a pre-declared pooled gain and does not rely on one favorable fold.

Risk acceptance requires pooled weighted-F1 gain of at least 0.006, nonnegative pooled macro-F1 gain, no fold below $-0.003$ weighted F1, and paired-bootstrap probability of positive gain above 0.90. Factor routing requires at least $+0.006$ pooled macro F1 and macro AP, tail macro F1 above a $-0.005$ floor, and strictly positive macro-F1 gain on each confirmation fold.

The evidence experiment has a weaker methodological status. Its candidate-level calibrator is frozen before outer Fold 1/2 evaluation, and predictions are row-level out of fold. However, the Fold 0 candidate generator was trained using Folds 1/2. Thus evidence numbers are valid for competition model selection but are not a fully nested external-test estimate. We preserve this distinction rather than describing all three components as equivalently untouched.

Both 27B adapters use one epoch, learning rate $10^{-4}$, rank 16, LoRA scale 32, dropout 0.05, maximum length 1,536, batch size 1, and gradient accumulation 32. All 64 language layers are adapted. Training uses 4-bit NF4 with double quantization, bfloat16 computation, gradient checkpointing, FlashAttention-2 for full-attention blocks, and optimized causal-convolution/linear-attention kernels. Experiments run locally in Google Colab Pro on a single NVIDIA A100 80GB. The factor verifier uses balanced positive/negative pairs with a 0.7 hard-negative fraction. Random seed 42 is fixed for model and probe training.

Table II gives the score of the submitted system. The leaderboard does not reveal gold labels or separate risk/evidence components, so it is used only as an external distribution check. All method selection reported below uses training-set folds.

Table III reports the frozen confirmation. The latent readout improves weighted F1 on both folds, yielding $+0.0135$ pooled weighted F1 and $+0.0219$ macro F1. A paired post-level bootstrap gives a 95% interval of $[0.0012,0.0261]$ for weighted-F1 gain and probability 0.983 that the gain is positive. Behavior recall rises from 0.7315 to 0.7704; Attempt recall rises from 0.6712 to 0.7397. Mean absolute severity error falls from 0.2052 to 0.1858.

Table IV compares the frozen 0.25/0.75 token blend with the seven-label route on Folds 1/2. Routing improves macro AP by 0.0082, macro F1 by 0.0105, and tail macro F1 by 0.0189. The gain is positive on both folds. Head-label macro F1 is unchanged because none of the pooled head labels is routed.

Fold 1 macro F1 rises from 0.6619 to 0.6736; Fold 2 rises from 0.6788 to 0.6895. The readout advantage is therefore label dependent: selected factors benefit from the answer state, while the token boundary remains preferable for the other labels.

Table V decomposes the frozen route by label. Three labels improve on both confirmation folds: poor school performance, psychological capital, and traumatic experience. Their supports span tail and mid-frequency strata, so frequency alone does not explain the gain. The remaining routed labels have mixed fold-level effects even though the aggregate route improves both folds. This rules out a broad claim that latent states are uniformly better for rare or abstract constructs. The supported interpretation is narrower: a shared latent correction improves particular decision boundaries, while development-only route selection remains noisy at the individual-label level.

The frozen candidate calibrator improves pooled phrase F1 from 0.7488 to 0.7609 (Table VI). Its main effect is to trade redundant recall for precision: mean predicted phrases fall from 1.35 to 1.01, while phrase precision rises from 0.7237 to 0.7723. Fold 1 improves by 0.0234 and Fold 2 by 0.0007. The positive but unequal fold gains motivate the methodological qualification in Section V-A.

Table VII records the main rejected alternatives. An additional hard-error SFT epoch increases factor ranking AP but sharply reduces thresholded macro F1. Global factor latent replacement also improves AP without improving macro F1. Pairwise ranking and structured CoT do not repair the boundary, and a risk event compiler lowers the projected Task 1 score. The seven-label shared probe is the factor readout that passes frozen confirmation.

The ablations narrow the interpretation of the gain. Extra SFT does not help, so the result is not explained by additional adaptation alone. Global replacement fails, so hidden states are not uniformly superior to token margins. Seven independent factor probes also fail, showing that the successful route depends on regularized cross-label sharing rather than unrestricted per-label capacity.

The result establishes predictive accessibility, not a mechanistic account of suicidal reasoning. The layer-63 state contains information from which a low-capacity classifier generalizes better than the answer-token boundary on selected outputs. Because the probes are supervised, some performance can originate in the probe itself. We limit this risk through linear models, strong regularization, grouped inner cross-validation, frozen confirmation folds, and comparison against the final-token margin. Control-task and intervention analyses would be needed to show that the same features causally drive the model’s original computation [11, 12].

The three output spaces impose incompatible constraints. Risk requires a global comparison among four severity cards; factors require independent decisions so that one generated label does not suppress another; evidence requires exact offsets and a variable-size set. HRD does not force these tasks into one generation grammar. Semantic verification is shared, while the final decision rule follows the output structure. The latent components are linear models over cached states and do not require additional 27B networks.

Label cards make inclusion and exclusion boundaries explicit and inspectable. They can reveal whether a prediction follows a future appraisal (hopelessness), self-evaluation (low self-esteem), resource availability (social support), or material deprivation (low socioeconomic status). Nevertheless, the cards are operational annotations rather than validated psychometric instruments. Their predictive validity in this dataset does not establish construct validity across populations, platforms, or cultures.

Table VIII separates what each experiment establishes from what it does not. This distinction is especially important here because a probe can expose predictive information without identifying the mechanism that produced it, and an exact copied span can be traceable without being causally sufficient for the prediction.

Table V shows that the fixed factor route is useful in aggregate but heterogeneous across labels. This suggests replacing the hand-selected route with a regularized instance–label meta-router. For post $x_{i}$ and label card $c_{l}$, let $m_{i,l}$ denote the answer-token margin and $g_{i,l}$ the shared latent-probe score. A router predicts

and compiles the final score as

The meta-features $\boldsymbol{\phi}$ should be observable at test time and estimated without crossing user folds. Candidate features include post length and clause dispersion; label support, prevalence, lexical specificity, and card-embedding abstraction; token-margin magnitude and entropy; and token–latent disagreement. A hierarchical prior could share evidence across labels while shrinking rare-label routes toward the token-margin baseline. To avoid converting routing noise into overfitting, router fitting, feature normalization, and route-threshold selection would all occur inside nested user-grouped folds, with an abstention region when neither readout is reliably preferred.

The broader claim also needs external validation. We would test the same frozen protocol across multiple Qwen parameter scales and at least one bidirectional encoder, separating model-scale transfer from architecture transfer. Cross-domain evaluation on compatible CLPsych suicide-risk and evidence corpora [4] would then test platform transfer; because their ontologies do not necessarily match the present 24 factors, risk/evidence transfer and factor-ontology transfer should be reported separately rather than forced into one score. These experiments would establish whether heterogeneous readout selection is a general property or a dataset–backbone interaction.