Traverse: Learning When to Remember, Reset, and Redirect for Long-Horizon Web Search

Our goal is to maximize agent autonomy in long-horizon search. We first address the unavoidable problem of context exhaustion by letting the agent decide both when to compress its context and what information to carry forward. This capability is implemented through two tools: the Seal Memory tool, which compresses and stores memory, and the Read Memory tool, which retrieves finer-grained details when needed. We further organize the search process as a state machine. Given a query $q$, the agent first enters the Rubric state and constructs $R=\{r_{i}\}_{i=1}^{m}$. These rubrics are then passed to the Answer state, where the agent gathers evidence with search and browse tools while autonomously deciding whether to compress its context or produce an answer $y$. In the Verify state, the agent evaluates $y$ conditioned on $q$ and $R$, and outputs $d\in\{\text{Pass},\text{Revise Answer}\}$. The process terminates on Pass; otherwise, the agent returns to the Answer state with verifier feedback and continues searching. Our framework is summarized in Figure 1.

Context management is unavoidable in long-horizon agentic search. When an agent follows an incorrect search path, or must verify an answer across multiple pieces of evidence, its finite context can easily become exhausted. We observe, however, that the agent itself often has sufficient signal to recognize when search has stalled and when the evidence is strong enough to discard accumulated noise. We therefore introduce the Seal Memory Tool, an agent-invoked context management mechanism that lets the agent autonomously decide whether to continue searching or reset its context. When invoking the tool, the agent is required to summarize the information worth preserving according to a structured memory template. It reviews its interaction history, incorporates evidence collected so far, and, if available, the memory produced by the previous segment. We expose several summary fields as tool arguments to guide this process. Let $h_{k}$ denote the interaction history in segment $k$, and $m_{k}$ the memory carried into that segment. The next memory is generated as

with $m_{0}=\varnothing$. After sampling $m_{k+1}$, the context is reset and the next segment starts from

The agent then resumes search from $c_{k+1}$. The trajectory terminates once the agent chooses to produce a final answer instead of invoking the memory tool again. Meanwhile, after each tool call, the agent is informed of its current token usage, allowing it to explicitly track the remaining context budget rather than blindly continuing until it hits the context limit. We also provide a Read Memory Tool to retrieve detailed information from the stored memory $m_{k}$ when necessary. More details about the memory tools are provided in Appendix F.

In this section, we explore how to equip the agent with self-verification capabilities. Inspired by DeepSeekMath-V2 (Shao et al., 2025), we design a workflow that spans question decomposition through answer verification, enabling the agent to assess the correctness of its own answer and decide whether the search process should terminate.

Rubric State. We first ask the agent to decompose the question in the Rubric State. Given a query $q$, the agent identifies the criteria that must be satisfied for an answer to be considered correct and produces a rubric set $R=\{R_{i}\}_{i=1}^{n}$, where the number of criteria $n$ is determined adaptively by the agent. The rubric turns the constraints in the query into an explicit, structured checklist that can guide subsequent search and verification. For the multi-constraint, short-answer tasks considered in this work, $R$ typically contains one content criterion for each distinct clue in the question, together with an additional criterion specifying the required answer format. Each content criterion describes an independently verifiable property of the target entity and is designed to be checked against Web evidence. To avoid injecting assumptions at this early stage, the criteria remain simple, objective, and faithful to the original query, preserving approximate expressions, numerical ranges, and indirect descriptions at their original level of specificity. We represent each criterion as an indexed natural-language description and carry the resulting rubric forward to subsequent states.

Answer State. Next, the agent enters the Answer State and begins the actual search. Given the rubric set $R$ produced in the previous state, the agent is equipped with search, Web browsing, Seal Memory, and Read Memory tools. Each tool response is augmented with the current resource usage:

Based on the remaining budget and the noisiness of the accumulated context, the agent decides whether to invoke the Seal Memory Tool to compact its context. Within each segment, it repeatedly chooses between continuing the search through compaction or terminating with an answer $y$. When compaction resets the context, the token budget is refreshed, while the turn budget is preserved across segments.

Verification State. Finally, the agent enters the Verify State, where it receives the query $q$, the candidate answer $y$, and the rubric set $R$, and independently assesses whether $y$ satisfies the required criteria. We equip the verifier with the same four tools as the Answer State—search, Web browsing, Seal Memory, and Read Memory—allowing it to gather external evidence for each rubric. When the verification context becomes overly long or noisy, the verifier may also invoke Seal Memory to compact its context. At the end of verification, the agent outputs a decision $d\in\{\textsc{Pass},\textsc{ReviseAnswer}\}$. If $d=\textsc{Pass}$, the process terminates and $y$ is returned as the final answer. Otherwise, the verifier produces a suggestion explaining why the current answer is insufficient, which rubrics remain unsatisfied, and where the subsequent search should focus, before transitioning back to the Answer State. Upon re-entry, the Answer State is additionally informed of previously proposed answers and their failures, together with the verifier’s latest feedback and unmet rubrics, enabling the agent to resume search with a more targeted direction rather than starting from scratch.

Taken together, the agent cycles through the Rubric, Answer, and Verify states, potentially undergoing multiple rounds of answering and verification until it can no longer identify an error and decides to submit the answer. Throughout this process, the agent autonomously manages its context via the Seal Memory Tool and independently determines whether the current answer is sufficiently verified or whether further search is necessary.

Query Construction and Data Curation. To train our model, we first synthesize challenging yet verifiable information-seeking questions using knowledge graphs, following WebShaper (Tao et al., 2025), and apply rigorous verification and cleaning to retain only solvable instances. We then pair these questions with a strong teacher model and our harness to collect agent trajectories. Since teacher rollouts can still contain undesirable behaviors, we filter both rule violations, such as malformed or unparsable tool calls, and behavioral errors, such as invoking the Seal Memory Tool prematurely or after the answer has already been found. Rather than discarding an entire trajectory due to a few flawed turns, we mask those turns out during training while preserving the remaining valid supervision. Further details about data synthesis are provided in Appendix A.

Masked SFT. To incorporate the filtering masks above, we augment the standard next-token prediction objective with a turn-level training mask. Specifically, for each turn $i$, we assign a mask $\mathcal{M}_{i}\in\{0,1\}$, which is broadcast to all tokens in that turn. All system, user, and tool turns are always assigned $\mathcal{M}_{i}=0$. For assistant turns, $\mathcal{M}_{i}=0$ if the turn is identified as undesirable and $1$ otherwise. The SFT objective becomes

where $N$ is the number of turns, $T_{i}$ is the number of tokens in turn $i$, and $x_{i,j}$ denotes its $j$-th token. This allows the agent to avoid imitating bad patterns while still learning how to recover after it.

After distilling context-management behaviors from strong-teacher trajectories, we further ask whether the agent can learn them from environmental feedback. Context compaction introduces a structural challenge for RL: every context reset fragments the original trajectory into a new training sample. A trajectory $\tau_{i}$ may therefore contain multiple segments $\tau_{i}=\{\tau_{i}^{(1)},\ldots,\tau_{i}^{(K_{i})}\}$, where $K_{i}$ is the number of segments induced by compaction.

A natural strategy is to assign each trajectory its final correctness as reward $r_{i}$, compute the group-normalized advantage $\hat{A}_{i}=(r_{i}-\mu_{\mathcal{G}})/\sigma_{\mathcal{G}}$, where $\mu_{\mathcal{G}}$ and $\sigma_{\mathcal{G}}$ are the group reward mean and standard deviation, and broadcast it to every segment and optimized token: $\hat{A}_{i,k,t}=\hat{A}_{i}$. This follows the philosophy of GRPO, where all optimized tokens in a trajectory share the same advantage.

This seemingly natural choice creates two problems. First, it obscures credit assignment. A successful trajectory may recover only after its final compaction removes misleading evidence from earlier exploration. Conversely, a failed trajectory may contain useful evidence in earlier segments yet make a wrong commitment only in the last. Broadcasting the final outcome therefore rewards or penalizes many actions only weakly related to the result, injecting substantial optimization noise. Second, fragmentation implicitly reweights trajectories. The RL objective can be written as

where $N$ is the number of trajectories, $\mathcal{L}$ is an RL objective such as PPO, and $\pi_{\theta}$ is the policy parameterized by $\theta$. Suppose one trajectory is split into $K$ segments while the remaining $N-1$ trajectories are not compacted. The batch now contains $K+N-1$ training samples. Each non-compacted trajectory receives weight $1/(K+N-1)$ instead of $1/N$; thus, the normalizer changes with $K$.

Addressing both issues simultaneously is non-trivial. One must either spend additional computation to obtain more accurate value estimates, thereby enabling finer-grained credit assignment, or resort to more sophisticated algorithmic designs whose effectiveness must then be carefully validated under this dynamically evolving training process. Instead, we take a deliberately simple route. We propose a simple yet effective strategy that sidesteps both challenges: we optimize only the final segment produced by context compaction. Despite its simplicity, this design is motivated by two key observations:

1. 1.

   Implicit Learning. At the end of each segment, the agent must either produce an answer or invoke the Seal Memory Tool, otherwise, the segment terminates due to context overflow and becomes the final segment, where failure is penalized. Thus, even without explicitly rewarding memory tool calls, the agent is implicitly encouraged to invoke the Seal Memory Tool when further search is needed, while learning when to stop and answer within the current segment.

2. 2.

   Reward Signal. The final segment is temporally closest to the trajectory-level reward and therefore receives a less noisy learning signal, reducing the optimization variance induced by broadcasting the same advantage across all preceding segments.

In this way, training only the final segment actually couples two decisions: whether to continue searching through compaction, and whether enough evidence has been gathered to answer. We therefore formulate our RL objective as:

where only the last segment $\tau_{i}^{(K_{i})}$ is optimized. We compute within-group advantages using the GRPO-style Z-score normalization. For policy optimization, we adopt GSPO (Zheng et al., 2025a) to stabilize MoE training. Specifically, GSPO aggregates token-level policy ratios into a sequence-level importance ratio, $\rho_{i}=\exp\left(\frac{1}{T_{i}}\sum_{j=1}^{T_{i}}\log\frac{\pi_{\theta}(x_{i,j}\mid x_{<i},x_{i,<j})}{\pi_{\theta_{\mathrm{old}}}(x_{i,j}\mid x_{<i},x_{i,<j})}\right)$. We additionally apply penalties for tool usage, trajectory length, and token-budget consumption. Detailed reward configurations are provided in Appendix C.

Implementation Details. We use Qwen3.5-35B-A3B as our base model and adopt GSPO to stabilize RL training for the MoE architecture. We set the maximum number of agent turns to 512, the context length to 256K, and the sampling temperature to 1.0. Our RL training uses 96 GPUs, with 32 GPUs allocated to policy optimization and 64 GPUs dedicated to asynchronous rollout generation. For all ablation studies, we keep the search backend, training hyperparameters, and sampling configuration identical to ensure fair comparisons. During SFT, we collect and train on trajectories from the rubric, answer, and verification states, whereas RL is applied only to the answer state. For BrowseComp, we additionally enable a Discard-All context reset when the context is exhausted, following common baseline practice; this fallback is orthogonal to Seal Memory and can be combined with the agent-triggered compression mechanism.

Baselines. We compare our method with the closed-source models like GPT-5.5 (OpenAI, 2026), Claude Opus 5 (Anthropic, 2026), and Gemini 3.1 Pro (Google DeepMind, 2026), as well as the open-source models like GLM-5.1 (GLM-5 Team and others, 2026), Kimi K3 (Team et al., 2026a), DeepSeek-V4-Pro (DeepSeek-AI et al., 2026), Qwen3.5-35B-A3B and Qwen3.5-122B-A10B (Team, 2026), MiroThinker-1.7-mini (Team et al., 2026b), QUEST-35B (Xie et al., 2026), FORT-Searcher (Deng et al., 2026), and AREX-Turbo (Lu et al., 2026).

Benchmarks. We evaluate our model on five search benchmarks: BrowseComp for deep information seeking, BrowseComp-ZH (BC-ZH) (Zhou et al., 2025) for multilingual search, WideSearch (Wong et al., 2025) for breadth-oriented retrieval, and xbench-DeepSearch (xbench) (Chen et al., 2025a) and DeepSearchQA (DSQA) (Gupta et al., 2026) for multi-step evidence collection and synthesis. Together, they cover diverse deep-search capabilities across languages and task formats. We further evaluate generalization on GAIA (Mialon et al., 2024), FinSearchComp (Hu et al., 2025), and ShoppingComp (Tou et al., 2025). Additional benchmark and evaluation details are provided in Appendix F.3.

In this section, we evaluate our model across a diverse suite of benchmarks and compare it against strong existing systems. Beyond widely used search benchmarks, we further probe its out-of-domain generalization on finance and e-commerce-oriented tasks. As shown in Table 1, Traverse-35B achieves a score of 72.83 on BrowseComp, reaching performance comparable to FORT-Searcher. Although it trails slightly on BrowseComp-ZH and xbench, it remains highly competitive. More notably, on DeepSearchQA and WideSearch, our model achieves state-of-the-art performance among search-specialized models. Table 2 further highlights the robustness of this capability beyond the training distribution: on both financial and e-commerce benchmarks, Traverse-35B delivers substantial gains over the base model, suggesting that the learned search behaviors transfer effectively to previously unseen domains. Our method improves GAIA accuracy by 18.45 percentage points, FinSearchComp accuracy by 19.70 percentage points, and ShoppingComp SoP by 0.1331 over the base model.

In this section, we investigate the effectiveness of our approach from three perspectives. We first validate the proposed RL algorithm, then quantify the gains brought by RL over the SFT model, and finally compare our method against Auto Compaction. For BrowseComp, we evaluate on a 185-question subset to reduce evaluation cost, which we refer to as BC185. Further details about BC185 are provided in Appendix D.

Is Training the Last Segment Effective? We compare final-segment RL (Section 3.2) with two alternatives that broadcast the trajectory-level advantage to all segments: standard all-segment RL and a $1/K_{i}$-weighted variant that equalizes total trajectory weights. As shown in Figure 2, standard all-segment RL reduces BC185 Avg@3 from 60.4 for SFT to 51.7. Weighting partially recovers it to 57.8, still below SFT and final-segment RL (65.2). On WideSearch, the weighted variant achieves 40.1/59.5 Row-F1/Item-F1, compared with 46.3/66.2 for standard all-segment RL and 49.7/71.1 for final-segment RL. Thus, correcting trajectory reweighting alone does not resolve the degradation from all-segment training.

Figure 3 reveals distinct training failures. Standard all-segment RL initially improves but later collapses: Seal calls are delayed, usage approaches zero, and hard context-budget termination rises sharply. The $1/K_{i}$-weighted variant shows accuracy degradation from around step 20, followed by a surge in Seal usage around steps 25–30, increasingly early Seal calls, and repeated Search/LinkSummary loops. These patterns are consistent with unresolved credit-assignment difficulties despite trajectory reweighting. Final-segment RL maintains stable accuracy, Seal usage, and search behavior while optimizing only one segment per trajectory, avoiding growth in training-segment count as context resets increase.

How Much Does RL Improve over SFT? Here, we quantify the gains brought by RL over SFT. We compare the SFT and RL models on BC185 and WideSearch. On BC185, RL improves the score by 4.8 points while reducing the number of agent turns required to reach the answer. We observe the same trend on WideSearch, where RL improves Row F1 by 6.0 points and Item F1 by 7.9 points, again with fewer search turns. These results show that RL yields substantial improvements in both effectiveness and search efficiency, while further validating the effectiveness of optimizing only the final segment.

Can Seal Memory Outperform Auto Compaction? We further compare our proposed Seal Memory mechanism with the widely used Auto Compaction strategy. We implement an Auto Compaction harness that triggers summarization when token usage reaches 90% of the context window, i.e., roughly 230K tokens in a 256K context. For a fair comparison, the summarizer is the same model used by the main agent: once the threshold is reached, it receives the full interaction history and produces a compact memory in the same format as the Seal Memory Tool, after which the agent continues from the user prompt augmented with this memory. As shown in Table 3, allowing the agent to decide when to compact its context yields a substantial performance gain: Active Seal improves Avg@3 by 10.28 points, with consistent gains on Pass@3 and Maj@3. More importantly, it reorganizes the search process much earlier, with the median first Seal occurring at 67 turns and 67.5K tokens, compared with 217 turns and 231.4K tokens for Auto Compaction. The flexibility of Seal Memory enables the agent to explore a broader range of possibilities and pursue more diverse search paths. Its average trajectory length increases from 112.2 to 217 turns, ultimately achieving substantially higher final-answer accuracy than fixed-threshold compaction.

Large language models have increasingly been equipped with search and browsing tools to acquire external information during reasoning. Early systems such as WebGPT (Nakano et al., 2022) and ReAct (Yao et al., 2023) established the foundations for grounded answer generation and interleaved reasoning and action. Recent search agents extend this paradigm to longer and more autonomous interactions involving query reformulation, webpage navigation, evidence aggregation, and answer synthesis (Li et al., 2025b; Wu et al., 2025; Li et al., 2025a). Meanwhile, benchmarks such as GAIA (Mialon et al., 2024), BrowseComp (Wei et al., 2025), BrowseComp-ZH (Zhou et al., 2025), WideSearch (Wong et al., 2025), and DeepSearchQA (Gupta et al., 2026) evaluate complementary aspects of tool use, persistent browsing, multilingual retrieval, and broad information collection. Our framework organizes search into explicit Rubric, Answer, and Verify states, enabling the agent to construct task-specific evaluation criteria, conduct evidence-seeking interactions, and revise its answer according to verification outcomes.

Recent work applies outcome-based reinforcement learning to teach language models when and how to search, allowing search behavior to emerge without intermediate reasoning annotations (Jin et al., 2025; Song et al., 2025; Chen et al., 2025b). Extending this approach to long-horizon search introduces two coupled challenges: the interaction history may exceed the active context budget, and a terminal reward provides ambiguous supervision for the many exploratory actions preceding the final answer. Context summarization and folding methods address the first challenge by compressing interaction histories and restructuring trajectories around context updates (Wu et al., 2026; Zhang et al., 2026; Sun et al., 2025). However, trajectory-level advantages are commonly propagated across multiple reconstructed segments, even though early search often contains irrelevant retrievals, candidate elimination, and abandoned directions. Our method uses autonomous Seal Memory operations to define segment boundaries and applies the terminal learning signal only to the final segment. Earlier evidence remains available through the sealed memory, while uncertain exploratory segments receive no direct gradient.