GIFT: Games as Informal Training for Generalizable LLMs
Extracted source text · arXiv 2601.05633
\documentclass[11pt]{article}
% Change "review" to "final" to generate the final (sometimes called camera-ready) version.
% Change to "preprint" to generate a non-anonymous version with page numbers.
\usepackage[final]{acl}
% Standard package includes
\usepackage{times}
\usepackage{latexsym}
\usepackage{multirow}
\usepackage{amsmath}
\usepackage{amssymb}
% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets
% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}
% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
\usepackage{microtype}
% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
\usepackage{inconsolata}
%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}
\usepackage{tcolorbox}
\tcbuselibrary{breakable, skins, listings}
\usepackage{listings}
\usepackage{makecell}
\newtcblisting{promptbox}{
colback=gray!5,
colframe=black!30,
boxrule=0.5pt,
arc=2pt,
left=4pt,
right=4pt,
top=4pt,
bottom=4pt,
width=\columnwidth,
breakable,
enhanced,
listing only,
listing options={
basicstyle=\ttfamily\footnotesize,
breaklines=true, % 自动换行
breakatwhitespace=false
}
}
\usepackage[table]{xcolor}
\definecolor{grey_purple}{RGB}{225,225,245}
\definecolor{highlight}{RGB}{255, 235, 210}
\definecolor{grey_blue}{RGB}{235,240,248}
\definecolor{grey}{RGB}{240,240,240}
\usepackage{booktabs}
% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.
\newcommand\yyg[1]{{\color{red} [YYG: #1]}}
% GIFT: Games as Informal Training for Generalizable Language Models
\title{GIFT: Games as Informal Training for Generalizable LLMs}
% \author{
% Nuoyan Lyu \and Bingbing Xu \\
% State Key Laboratory of AI Safety, Institute of Computing Technology, CAS \\
% \texttt{\{lvnuoyan23z,xubingbing\}@ict.ac.cn}
% \And
% Weihao Meng\\
% %Affiliation 2 \\
% % wmeng9@jh.edu
% \texttt{wmeng9@jh.edu}
% \And
% Author E \\
% Affiliation 3 \\
% \texttt{e@domain3}
% }
% Zhiyong Huang” Seng Chua Tat 沈华伟
\author{
\textbf{
Nuoyan Lyu$^{\clubsuit \heartsuit}$,
Bingbing Xu$^{\clubsuit }$\thanks{Corresponding author.},
Weihao Meng, % weihao单位怎么写？
Yige Yuan$^{\clubsuit \heartsuit}$, % yige师姐单位怎么写？
} \\
\textbf{Yang Zhang$^{\diamond}$,Zhiyong Huang$^{\diamond}$,
Tat-Seng Chua$^{\diamond}$,
Huawei Shen$^{\clubsuit \heartsuit}$}
\\
$^{\clubsuit}$ State Key Laboratory of AI Safety, Institute of Computing Technology, CAS \\
$^{\heartsuit}$ University of Chinese Academy of Sciences \\
$^{\diamond}$ National University of Singapore \\
\texttt{\{lvnuoyan23z,xubingbing,yuanyige20z\}@ict.ac.cn, wmeng9@jh.edu} \\
% \texttt{wmeng9@jh.edu, \{zhangy,dcshuang,dcscts\}@nus.edu.sg} \\
% \texttt{\{zhangy,dcshuang,dcscts\}@nus.edu.sg}
}
\begin{document}
\maketitle
\begin{abstract}
% Reasoning is a fundamental capability for enhancing the general intelligence of large language models (LLMs). However, existing reasoning enhancement methods rely heavily on rule-based rewards and large amounts of annotated, domain-specific data, which are both costly and limited in terms of Out-of-Distribution (OOD) generalization.
% Games are diverse, reasoning-intensive, and annotation-free environments that offer a promising alternative for improving reasoning ability. In this work, we explore a critical question: \textbf{“Can diverse games serve as a path toward general reasoning without human annotations?”}
% To address this question, we present a unified, multi-game framework that supports training and evaluating LLMs across diverse game environments. Through systematic analysis, we find that mixing multiple games naively introduces trade-offs across different reasoning dimensions. To mitigate this, we propose \textbf{XXX}, a method that harmonizes learning signals across games and enables the mutual enhancement of reasoning capabilities. Experimental results demonstrate that our approach effectively improves OOD reasoning performance. The framework and implementation are publicly available at \url{https://github.com/XXX/XXXX}.
While Large Language Models (LLMs) have achieved remarkable success in formal learning tasks such as mathematics and code generation, they still struggle with the "practical wisdom" and generalizable intelligence, such as strategic creativity and social reasoning, that characterize human cognition. This gap arises from a lack of informal learning, which thrives on interactive feedback rather than goal-oriented instruction. In this paper, we propose treating Games as a primary environment for LLM informal learning, leveraging their intrinsic reward signals and abstracted complexity to cultivate diverse competencies. To address the performance degradation observed in multi-task learning, we introduce a Nested Training Framework. Unlike naive task mixing optimizing an implicit "OR" objective, our framework employs sequential task composition to enforce an explicit "AND" objective, compelling the model to master multiple abilities simultaneously to achieve maximal rewards. Using GRPO-based reinforcement learning across \textit{Matrix Games}, \textit{TicTacToe}, and \textit{Who's the Spy} games, we demonstrate that integrating game-based informal learning not only prevents task interference but also significantly bolsters the model’s generalization across broad ability-oriented benchmarks.
The framework and implementation are publicly available\footnote{\url{https://github.com/XXX/XXXX}}.
\end{abstract}
\section{Introduction}
Human intelligence arises from the interplay between formal and informal learning \cite{scribner1973cognitive}. Formal learning emphasizes structured, goal-oriented education for acquiring task-specific knowledge, whereas informal learning unfolds in everyday environments through the unstructured interactions involving iterative experience and implicit feedback, %the unstructured interactions with iterations of practice and feedback,
enabling the acquisition of implicit knowledge and practical wisdom that further support general and transferable intelligence \cite{callanan2011informal}. In parallel, recent large language models (LLMs) have achieved remarkable success on formal learning tasks \cite{liu2025reinforcement, xu2025towards}, including mathematical reasoning \cite{wang2025reinforcement, zhang2025deeptheorem, chen2025seed} and code generation \cite{seed2025seed, yang2025code, wang2025large}. However, the broader competencies expected of general-purpose models, including creativity, social reasoning, etc., lie beyond the scope of formal learning and call for learning mechanisms akin to informal learning.
Therefore, this prompts us to think what is the informal learning environment for LLMs.
In this paper, we propose a new perspective: treating \textbf{games} as a fundamental environment for informal learning due to the following three key properties:
%it emerges from unstructured interaction, experience, and feedback; it proceeds without explicit instruction and predefined learning objectives; and it spans diverse and heterogeneous scenarios.
1) emerging from unstructured interactions involving iterative experience;
%unstructured interactions with experience and feedback; %providing intrinsic feedback and reward signals through interactions; % , naturally supporting reinforcement learning;
2) proceeding without explicit instructions, enabling learning without reliance on manually annotated data, overcoming the limitations of high-cost datasets;
3) serving as highly abstracted sandboxes of complex real-world interactions \cite{edwards2019tools, roungas2019game, kriz2022gaming}, unifying a diverse set of tasks and closely aligning with the goals of informal learning \cite{innes1999consensus, dutta1999strategies, aumann1992handbook}.
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{image/intro-new.pdf}
\caption{
Overview of formal and informal learning paradigms, and a comparison between naive mixed training and the proposed nested training framework.
}
\label{fig:nest-intro}
\end{figure*}
Building on this insight, we introduce a new training strategy of LLM that integrates formal learning with game-based informal learning. Specifically, we train LLMs using GRPO-based reinforcement learning \cite{shao2024deepseekmath} across mathematical tasks (formal learning) and multiple game environments (informal learning). We design three representative categories of games to cover diverse capacities: single-turn games (\textit{Matrix Games}), multi-turn two-player games (\textit{TicTacToe}) and multi-turn multi-agent social games (\textit{Who’s the Spy}), collectively covering a broad spectrum of cognitive abilities, including abstract reasoning \cite{lucas1981game, aumann1992handbook}, long-horizon planning \cite{mishra2025ttt, crowley1993flexible}, creativity and social intelligence \cite{zhang1997nature, wei2025exploring}.
A straightforward approach is to train on a mixture of tasks. However, we observe that naive mixing often leads to performance degradation due to trade-offs among tasks. This issue arises because mixed training implicitly optimizes an \textbf{OR}-style objective, where high rewards can be achieved by excelling at only a subset of tasks. To address this limitation, we propose a \textbf{nested training framework} that transforms the implicit \textbf{OR} objective into an explicit \textbf{AND} objective (Fig.~\ref{fig:nest-intro}). In this framework, nested tasks are constructed by sequentially composing multiple sub-tasks, and the model receives maximal reward only when it performs well across all components, thereby explicitly encouraging the simultaneous acquisition of multiple abilities with more stable gradients and higher entropy.
%Experimental results show that,
Experimental results across a broad set of ability-oriented benchmarks show that augmenting formal learning with game-based informal learning consistently improves general performance. On average, general ability increases from 38.34\% to 42.43\% for 1.5B models, while a substantially larger gain is observed for 7B models, rising from 42.00\% to 55.84\%. Moreover, greater diversity in nested game types further enhances generalization, with performance improving from 40.40\% to 42.43\% for 1.5B models and from 54.95\% to 55.84\% for 7B models.
% Experimental results on a broad set of ability-oriented benchmarks demonstrate that,
% compared to formal learning alone, incorporating game-based informal learning leads to substantial improvements across a wide range of evaluation dimensions, with the average general ability rising from 38.34\% to 42.43\% for 1.5B models, and exhibiting a much larger gain from 42.00\% to 55.84\% for 7B models. Furthermore, increasing the diversity of nested game types consistently enhances generalization, yielding a progression from 40.40\% to 42.43\% in 1.5B and from 54.95\% to %54.65\%, and ultimately reaching
% 55.84\% in 7B.
% Compared to formal learning alone, incorporating game-based informal learning leads to substantial improvements across a wide range of evaluation dimensions, * with the average general ability rising from 38.34\% to 42.43\% for 1.5B models, and exhibiting a much larger gain from 42.00\% to 55.84\% for 7B models. * Furthermore, increasing the diversity of nested game types consistently enhances generalization, * yielding a steady progression from 54.95\% to 54.65\%, and ultimately reaching 55.84\%. *
%We validate the effectiveness of our approach on a broad suite of ability-oriented benchmarks.
% Consequently, rethinking what informal learning entails for LLMs motivates a new perspective: we propose treating games as a fundamental environment for informal learning, for three main reasons:
%An effective informal learning environment should satisfy three key properties:
%Achieving such general intelligence in LLMs therefore calls for moving beyond single-task, goal-driven training toward an informal learning paradigm that enables self-evolution. An effective informal learning environment should satisfy three key properties:
% 1) agents can obtain feedback and rewards through their own actions, supporting iterative improvement via reinforcement learning; 2) learning does not rely on manually annotated data, thereby overcoming the limitations of finite, high-cost datasets; and 3) the environment promotes the acquisition of diverse, transferable abilities rather than overfitting to a single task.
%within a single interactive environment; and
% 3) are inherently diverse and reasoning-intensive, encompassing planning, creativity, strategic reasoning, and social interaction,
%Motivated by these considerations, we propose a new perspective: treating \textbf{games} as a fundamental environments for informal learning as games naturally constitute highly abstracted sandboxes of complex real-world interactions, encapsulating a diverse collection of tasks within a unified, interactive environment.
% They provide intrinsic feedback and reward signals through interaction, making them well suited for reinforcement learning without manual annotation.
% Moreover, games are inherently diverse and reasoning-intensive, spanning planning, creativity, strategic reasoning, and social interaction, closely aligning with the objectives of informal learning.
%xbb:***********************
\iffalse
Human intelligence emerges from the interplay between formal and informal learning \cite{scribner1973cognitive}: while formal learning relies on structured, goal-oriented instruction to acquire task-specific knowledge such as mathematics, informal learning occurs in rich everyday environments through interaction, practice, and feedback, enabling the accumulation of implicit knowledge and practical wisdom that supports more general and transferable intelligence \cite{callanan2011informal}. In parallel, recent large language models (LLMs) have demonstrated remarkable success on tasks characteristic of formal learning, including mathematical reasoning and code generation \cite{liu2025reinforcement, xu2025towards}; however, the broader capabilities expected of general-purpose models, including creativity, social reasoning, and understanding complex real-world scenarios, lie beyond these domains and likely require mechanisms analogous to informal learning to develop.
Human intelligence emerges from the combination of formal and informal learning\cite{scribner1973cognitive}. Formal learning emphasizes structured, goal-oriented instruction that enables the acquisition of task-specific knowledge, such as mathematics. Informal learning, which takes place in rich, everyday environments through interaction, practice, and feedback, allows humans to acquire implicit knowledge and practical wisdom, ultimately fostering more general and transferable intelligence \cite{callanan2011informal}.
Recent large language models (LLMs) have achieved remarkable success on formal-learning tasks, such as mathematical reasoning and code generation \cite{liu2025reinforcement, xu2025towards}. However, the capabilities we ultimately expect from general-purpose models extend far beyond these domains, encompassing creativity, social reasoning, and the understanding of complex real-world scenarios, which needs to develop through informal learning.
To achieve genuine general intelligence in LLMs, it is necessary to move beyond single-task, goal-driven training and instead seek a non-formal learning environment that supports self-evolution. An ideal informal learning environment should satisfy three key properties: 1) models can naturally obtain feedback and rewards through their own actions, enabling iterative learning via reinforcement learning, (2) learning does not depend on manually labeled data, thereby breaking the limitations imposed by finite datasets, and (3) LLMs can acquire a broad range of transferable abilities rather than overfitting to a specific task.
%xbb: interactive不需要标注数据吗，我感觉比如code也可以做成interactive的过程，但是还是需要标注数据
%lny: code理论上也可以不需要标注数据（也有相关的零数据做code生成的工作），我之前一直倾向于需要标注数据是math的特性、不是formal learning的普遍问题，不需要标注数据是game的特性、不是informal learning的普遍问题；不过理论上来说interactive的从环境中交互获得反馈这个更像传统强化学习，从这个角度来看interactive不需要标注数据也合理
%This motivates us to find what constitutes informal learning for LLMs. An ideal informal learning environment should satisfy two key properties: (1) LLMs can naturally obtain feedback and rewards through interaction, enabling reinforcement learning, and (2) the environment promotes the development of multiple abilities rather than overfitting to a single task.
% 这个地方不需要提及annotation-free，但实际上interactive就不需要标注数据了
Inspired by this perspective, we propose a promising paradigm: rethinking \textbf{Games} as a fundamental medium for informal learning in intelligent models. The key insight is that games naturally serve as highly abstracted sandboxes of complex real-world interactions, encapsulating a diverse collection of tasks within a unified, interactive environment.
%Based on these criteria, we propose using \textbf{Games} as a form of informal learning environment for LLM-based agents.
Games provide interactive environments and feedbacks, naturally supporting reinforcement learning, and are highly diverse and inherently reasoning-intensive, covering a wide range of abilities, corresponding to the informal learning scenarios. Importantly, unlike high-cost data in math and code, games naturally provide reward signals through interactions, eliminating the need for manual annotation.
Building on this insight, we introduce a training paradigm that integrates formal learning with game-based informal learning. We train LLMs via GRPO reinforcement learning across math and multiple game environments. We design three representative categories of games: single-turn games \textit{Matrix Games}, multi-turn two-player games \textit{TicTacToe}, and multi-turn multi-agent social games \textit{Who's the Spy}, collectively covering reasoning, planning, creativity, and social intelligence.
% Figure 1: Comparison between Naïve Task Mixing and the Nested Training Framework.In naïve mixing, each task is optimized independently, and training trajectories tend to follow the easiest objective, potentially neglecting harder tasks. In contrast, the nested framework organizes multiple tasks into a unified hierarchical objective, encouraging the model to consistently progress toward a shared global optimum.
A straightforward solution is to directly mix these tasks during training. However, we find that naive task mixing often leads to performance degradation due to task trade-offs. This failure arises because the mixed training optimizes an implicit \textbf{"OR"} objective where high rewards can be achieved by excelling at only one task. To address this issue, we propose a \textbf{nested training framework} that transforms the \textbf{"OR"} objective into an explicit \textbf{"AND"} objective, as shown in Fig.~\ref{fig:nest-intro}. Nested tasks are constructed by sequentially completing sub-tasks, where the model receives the highest reward only when it performs well on all tasks. This formulation explicitly encourages the simultaneous acquisition of multiple abilities.
Experimental results on a broad set of ability-oriented benchmarks demonstrate that, compared to training with formal learning alone, incorporating game-based informal learning significantly improves performance across multiple evaluation dimensions. Moreover, increasing the diversity of nested game types further enhances generalization.
%We validate our approach .
\fi
Our contributions are summarized as follows:
\begin{itemize}
\item[1)] \textbf{Informal learning Paradigm}: We propose a novel perspective that conceptualizes games as the fundamental environment for the informal learning of LLMs, providing a scalable, interactive environment without the need for manual annotation;
%Inspired by human learning processes, we systematically investigate informal learning for LLMs and propose games as an annotation-free environment that naturally supports multi-dimensional ability acquisition;
\item[2)]\textbf{Methodological Innovation}: We identify the "OR-style" optimization trap in naive mixed training and propose a Nested Training Framework, which transforms the objective into an explicit "AND" logic and ensures the simultaneous acquisition of diverse abilities;
%We introduce a nested training framework that converts naive multi-task "OR" optimization into an "AND" objective, effectively mitigating task trade-off;
%这个地方实际并不是mitigate task trade-off，更明显的是提升了多种能力
\item[3)] \textbf{Empirical Validation}: Through extensive experiments on ability-oriented benchmarks, we demonstrate that game-based informal learning significantly enhances LLMs beyond formal learning only.
%We achieve strong empirical results across three types of game settings, demonstrating improvements in creative writing and social reasoning beyond formal learning alone.
\end{itemize}
% Recent large language models (LLMs) have achieved impressive performance on goal-driven tasks such as mathematical reasoning and code generation. However, training paradigms centered on task-specific objectives~\yyg{Is “goal-driven, task-specific” an accurate description here? “Goal-driven” feels a bit under-specified in the earlier context, so I’d suggest briefly clarifying what you mean by it. Alternatively, you could frame it as a distinction between \textbf{verifiable and non-verifiable} objectives/tasks.} often struggle to endow models with more general abilities, including creativity, social reasoning, and understanding of complex real-world scenarios~\yyg{Add evidence/references here}. While supervised fine-tuning(SFT) and reinforcement learning(RL) are effective for targeted tasks, they frequently lead to overfitting to specific distributions and limited transfer across capability dimensions~\yyg{Is the critique of RL and SFT reasonable here? RL and SFT are training algorithms, so I assume you mean that the objectives they optimize don’t generalize well, rather than that the algorithms themselves are inherently non-generalizable. After all, if you don’t use RL or SFT, what other training approach would you use? Also, add evidence/references to support this claim.}.
% Human intelligence, in contrast, emerges from the combination of formal and informal learning~\yyg{This paragraph feels a bit disconnected from the previous one, especially with the sudden introduction of the terms “formal learning” and “informal learning.”}. Formal learning focuses on acquiring task-specific knowledge through structured objectives~\yyg{The “goal-driven, task-specific” framing in the previous paragraph should be closely related to the “formal vs. informal learning” here. Why not introduce the terms “formal learning” and “informal learning” in the previous paragraph? } such as math and code scenarios, whereas informal learning arises from rich environmental interactions, enabling the acquisition of implicit knowledge and more general reasoning skills. This observation motivates a central question: \textbf{what constitutes informal learning for LLMs?}
% % 我们一直说的纳什均衡实际上算的是矩阵博弈，不确定哪个更合适，实际上是矩阵博弈-寻求纳什均衡——这么说还是应该用矩阵博弈的名字
% In this work, we propose using \textsc{games} as a form of informal learning for LLM-based agents. Game environments are diverse, interactive, and inherently reasoning-intensive, while requiring no human annotation, since agents can obtain reward signals naturally through interaction and self-play. We consider three representative categories of games: single-turn games (e.g., Matrix Games) , multi-turn two-player games(e.g., TicTacToe), and multi-turn multi-agent games (e.g., Who’s the Undercover), which collectively cover reasoning, planning, creativity, and social intelligence.
% A straightforward solution is to directly mix formal learning tasks with multiple game environments during training. However, we find that naive task mixing often fails to produce synergistic improvements and can even lead to performance degradation due to task trade-offs. This occurs because naive mixing implicitly optimizes an “OR” objective, where the model focus on a subset of tasks while neglecting others.
% To address this issue, we introduce a \textbf{nested training framework} that converts the \textbf{OR} objective into an explicit \textbf{AND} objective. By sequentially composing multiple games into a single nested task, the agent receives the highest reward only when it performs well across all constituent games, thereby encouraging the simultaneous acquisition of multiple abilities. Experiments demonstrate that incorporating game-based informal learning on top of formal learning leads to consistent improvements across diverse evaluation dimensions.
\section{Related Works}
\subsection{Games and Large Language Models}
% LLM For Games: Training LLM to master games.
% Game For LLMs: Use game to boost LLM's abilities. Overlook the multiple games settings.
Recent studies on LLMs and games can be broadly categorized into two research directions. The first direction, commonly referred as LLM for Games, investigates the training and evaluation of LLMs within specific game environments. Representative works examine LLM behavior in negotiation games, social deduction settings, and multi-agent text-based games, aiming to assess strategic consistency, equilibrium behavior, or task-specific performance \cite{ fan2024can, mao2025alympics, bianchi2024well, guertler2025textarena, akata2025playing}. These studies primarily focus on understanding or improving LLM performance within particular games, rather than enhancing general reasoning or learning capabilities across tasks. In contrast, the Game for LLM line of research treats games as structured interaction frameworks for improving broader LLM abilities. Prior works demonstrate that self-play, repeated interactions, and multi-agent game dynamics can facilitate improvements in reasoning, alignment, and strategic adaptation \cite{tang2025game, liu2025spiral, xie2025play}. However, existing studies remain limited in game diversities and provide insufficient analysis on multi-task settings. % different categories of games differentially shape model behavior and capability development.
% Recent benchmarks further systematize this perspective by evaluating LLMs under game-theoretic pressures across diverse strategic settings \cite{ huang2025competing, chen2025llmspark }.
\subsection{Multi-Task RL Training in LLMs}
Reinforcement learning (RL) has been shown to play a critical role in enhancing the reasoning capabilities of large language models (LLMs) \cite{liu2024deepseek, wang2024reinforcement, khatri2025art, guo2025deepseek, xu2025towards}. A growing body of work demonstrates that RL-based training can significantly improve performance on reasoning-intensive tasks, such as mathematical problem solving \cite{zeng2025simplerl, wang2025reinforcement} and code generation\cite{zhao2025absolute}.More recently, several works have begun to investigate multi-task RL training for LLMs\cite{zeng2025rlve}. Some studies observe that naively mixing heterogeneous tasks often leads to performance trade-offs\cite{wu2025imbalanced}. To mitigate this issue, OMNI-THINKER adopts a curriculum-based training strategy and mixed reward designs \cite{li2025omni} and AgentRL replaces the group-based advantage in GRPO with a task-aware advantage formulation \cite{zhang2025agentrl}.
% 这个地方最好加上和我们的区别，或者不要提这么多？
In contrast to these approaches, which primarily address multi-task instability through curriculum design, reward shaping, or advantage reweighting, our work explores a complementary direction by reformulating multi-task optimization itself, enabling synergistic ability acquisition without relying on manual designs.
%感觉这个地方不需要说太多，后续方法部分会详细介绍 We leverage game-based non-formal learning and introduce a nested training framework that explicitly enforces an \emph{AND}-style objective across heterogeneous tasks, enabling synergistic ability acquisition without relying on manual annotation.
% \section{Preliminary}
% In this section, we introduce the conceptual and technical foundations of our approach, including the distinction between formal and non-formal learning, the task formulation and notation used throughout the paper, and the reinforcement learning objectives on which our method is built.
\begin{table*}[t]
\centering
\small
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{l l l}
\hline
\textbf{Category} & \textbf{Game} & \textbf{Targeted Abilities} \\
\hline
Single-turn
& \textit{Matrix Games} % (Nash Equilibrium)
& Abstract \& Strategic reasoning; \\
Multi-turn two-player
& \textit{TicTacToe}
& Long-horizon planning; Sequential decision making; \\
Multi-turn multi-player
& \textit{Who's the Spy}
& Theory of Mind; Creative language generation \\
\hline
\end{tabular}
\caption{Representative game environments and the reasoning abilities they promote.}
\label{tab:game_types}
\end{table*}
\section{Method}
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{image/game.pdf}
\caption{
% Illustration of formal and informal learning tasks, the nested training frameworks and detailed training process. In the nested training, we sequentially generate sub-tasks from the task list, the model continuously completes the sub-tasks, and a full trajectory includes all tasks sequentially.
Overview of the proposed nested training framework with formal and informal learning tasks.
The middle panel presents a high-level abstraction of the nested task structure,
while the right panel details the multi-turn nested training process with iterative model-environment interactions.
}
\label{fig:main-architect}
\end{figure*}
In this section, we explore the motivation of introducing formal and informal learning and how informal learning signals can be integrated into model training through reinforcement learning.% , enabling models to balance trade-offs and develop more general and transferable capabilities.
\subsection{Motivation}
\label{sec:learning}
% 介绍认知学基础——显然写的太长应该精简一下
In educational theory, human learning is commonly categorized into three types: \emph{formal learning}, \emph{non-formal learning}, and \emph{informal learning} \cite{coombs1974attacking,johnson2022formal}. Formal learning refers to institutionalized education systems with well-defined curricula, while non-formal learning is the structured education outside the standard formal education system. Both formal and non-formal learning reflect accumulated task-specific knowledge \cite{cattell1963theory, horn1967age}. In contrast, informal learning is defined as learning that arises from everyday activities and takes place through immersion in interactive environments, where learning is conducted through practice and feedback. This contributes to solving problems that are independent of specific tasks and supports general abilities that adapt to unseen situations \cite{ ziegler2012openness, thorsen2014influence}. Since non-formal learning shares similar characteristics with formal learning, we exclude it in this work.
Despite the central role of formal learning in current LLM training, particularly through structured tasks such as mathematics and code, educational research shows that informal learning activities play a more significant role \cite{fevre2001necessary, za2014organisational}. Informal learning has been strongly associated with the development of effective problem-solving skills in technology-rich environments \cite{nygren2019lifelong} and is more critical in skill development than formal training courses in workplaces \cite{de2024importance,fevre2001necessary}. Moreover, previous studies indicate that the combination of formal and informal learning is particularly beneficial \cite{gerber2001relationships}. These findings suggest that informal learning is a crucial yet underexplored component to developing generalizable intelligence and the combination of formal and informal learning is necessary.
% In contrast, non-formal learning can be operationalized through \textsc{Game}-based environments, which provide knowledge acquisition via interactive environmental feedback rather than explicit curricula. Such environments require neither rule-based rewards nor annotated data, making them nearly cost-free while naturally fostering the development of fluid intelligence and deeper logical reasoning. For example, the \emph{Prisoner's Dilemma}\footnote{https://en.wikipedia.org/wiki/Prisoner\%27s\_dilemma} encourages strategic reasoning and payoff-aware decision making; in the game \emph{Who’s the Undercover}\footnote{https://www.undercovergame.com/rules}, the description phase enhances a model’s expressive ability, while the voting and inference phases promote its capacity to analyze and reason over textual evidence. These properties make game environments a compelling instantiation of non-formal learning for LLM training.
% Since human cognition relies on both formal and non-formal learning—where the latter fosters fluid intelligence that is crucial for solving novel OOD problems—we aim to integrate both forms of learning into LLM training. Our goal is to develop models that not only master structured, curriculum-aligned in-domain tasks but also generalize robustly to unfamiliar OOD settings.
% 对应上math以及GAME两个场景
%这个小标题有些奇怪
% \subsection{Task Settings and Notation}
% \subsection{Reinforcement Learning Objective}
% GRPO and StarPO
% Formal learning are structured knowledge-based learning, so we formulate the formal learning into the \textit{Math} scenario. Motivated by the importance of informal learning in human cognition, we wonder what constitutes informal learning for LLM-based agents.
\subsection{Game as Informal Learning Environments}
Formal learning is characterized by structured acquisition of explicit knowledge. In LLM training, such learning paradigms are naturally instantiated by mathematics, where models are trained to solve well-defined problems under supervision. Accordingly, we formulate \textit{Math} as the formal learning environment in our study.
Motivated by the central role of informal learning in human cognition, we consider what constitutes informal learning for LLMs. Informal learning is characterized by three key properties: it emerges from unstructured interaction, experience, and feedback; it proceeds without explicit instruction and predefined learning objectives; and it spans diverse and heterogeneous scenarios.
From this perspective, \textsc{game}-based environments provide a natural and effective abstraction for modeling informal learning. Games are inherently interaction-driven and governed by intrinsic rules that generate feedback and rewards through agent-environment dynamics. Learning signals arise directly from interaction outcomes, eliminating the need for human-annotated data. Moreover, games distill essential structures of real-world scenarios into controlled yet expressive settings, enabling models to explore strategies, adapt to feedback, and reason about others across a wide range of simulated scenarios.
% Motivated by the importance of informal learning in human cognition, we wonder what constitutes informal learning for LLMs. Unlike formal learning, informal learning arises from unstructured interaction, experience, and feedback beyond explicit curricula. %In the following, we argue that game-based environments provide a natural and effective instantiation of informal learning for LLMs.
% Informal learning emerges from diverse forms of experience that vary in both temporal scope and social interaction.
% Some experiences involve isolated decisions with immediate feedback, while others require long-horizon planning across multiple steps. Likewise, certain learning processes are individual, whereas others depend critically on reasoning about the actions, beliefs, and intentions of others.
% Viewed from this perspective, \textsc{Game}-based environments provide a natural abstraction and sandbox for modeling informal learning. Games distill essential structures of real-world decision making and social interaction into controlled but expressive settings, allowing models to explore strategies, adapt to feedback, and reason about others in a wide range of simulated scenarios. As abstractions of real-world environments, games can naturally capture strategic, social, and communicative dynamics without the complexity and noise of fully open-ended settings.
% Importantly, game environments offer intrinsic advantages for scalable learning. They do not require human-annotated data or handcrafted supervision; instead, learning signals arise directly from interaction outcomes. This property enables virtually unlimited data generation and supports scalable reinforcement learning, making games particularly well suited for modeling informal learning beyond the constraints of curated datasets.
To systematically capture the diversity of informal learning experiences, we organize game environments along the degree of social interaction complexity (single-turn, multi-turn two-player, and multi-turn multi-player settings).We design and select three representative game environments, summarized in Table~\ref{tab:game_types}, each targeting complementary aspects of informal learning and ability acquisition \cite{noda2019effectiveness, liang2025multi}. The detailed game information is introduced in Appendix~\ref{app:game_rule}.
% The key distinction between naive mixed training and the proposed nested training framework lies not in the optimization algorithm itself, but in how trajectory-level rewards are constructed and composed across multiple tasks, as described in the following sections.
\subsection{Naive Mixed Training}
Given the diversity of formal and informal learning tasks, which instantiate informal learning and target complementary abilities, a natural strategy is to jointly train a single model on all tasks.
Such naive mixed training exposes the model to heterogeneous learning signals and is expected to facilitate the acquisition of diverse capabilities.
However, our empirical results show that directly mixing formal and informal learning tasks often leads to suboptimal outcomes, as summarized in Table~\ref{tab:main_results_1.5B} and Table~\ref{tab:main_results_7B}.
Although each task is effective in isolation, naive mixed training frequently suffers from unstable optimization and negative interference across tasks.In practice, the model tends to over-optimize a subset of tasks while neglecting others, hindering the simultaneous improvement of diverse abilities, illustrated in Fig.~\ref{fig:stable-train}.
%Under the $\text{F + I}_2$ setting, which includes \textit{Math}, \textit{Matrix Games}, and \textit{TicTacToe}, the success rate on \textit{TicTacToe} sharply drops from 48.00\% to 2.00\% during training, while performance on \textit{Matrix Games} continues to improve from 46.00\% to 65.00\%.
From an optimization perspective, mixed training aggregates learning signals across tasks, where achieving success on \emph{any} sub-task is sufficient to increase the overall objective.
Concretely, given $K$ sub-tasks with task-specific rewards $R_k(\tau_k)$, where $\tau_k$ denotes the trajectory for task $k$, mixed training optimizes the following additive objective:
\begin{equation}
\max_{\theta} \;
\mathbb{E}_{\tau \sim \pi_\theta}
\left[
\sum_{k=1}^K R_k(\tau_k)
\right].
\end{equation}
% Under this formulation, progress on a single sub-task directly contributes to the total reward, irrespective of performance on the remaining tasks.
As a result, %mixed objectives implicitly implement an \textbf{OR-type condition}, where improvement on one task can mask failures on others.
once a particular sub-task reaches high performance, its reward may dominate the gradient signal, causing gradients associated with other sub-tasks to diminish or vanish, without yielding commensurate learning gains \cite{wu2025imbalanced}.
% As shown in Fig.~\ref{fig:entropy-grad}, with optimization concentrating on a subset of tasks, gradient magnitudes become highly variable and policy entropy rapidly collapses, indicating unstable optimization and reduced exploration.
As shown in Fig.~\ref{fig:entropy-grad}, the OR-type objective allows individual tasks to converge independently, leading to low policy entropy within each task.
At the same time, it does not enforce coordination across tasks, causing different tasks to alternately dominate the reward signal.
Due to varying reward scales and success probabilities, the dominant optimization signal shifts across batches, resulting in highly variable gradient magnitudes.
Together, these dynamics undermine optimization stability and ultimately lead to poor joint generalization.
Importantly, this limitation is intrinsic to the OR-type objective structure, which allows sub-tasks to be optimized independently instead of encouraging a unified solution.
\subsection{Nested Training Framework}
% Moreover, when one sub-task achieves success, it do not provide more gradients, neglecting the overall abilities to solve all sub-tasks.
% To address this limitation, we propose a \textbf{Nested} training framework, which constructs a composite task by hierarchically combining multiple sub-tasks, defining task success in terms of the \textbf{joint satisfaction} of all sub-tasks with total reward as average success of all-subtasks.
% Concretely, the model attains the highest reward only when \emph{all} sub-tasks are successfully completed.
To address this limitation, we propose a \textbf{Nested} training framework that constructs a composite task by hierarchically combining multiple sub-tasks, as illustrated in Fig.~\ref{fig:main-architect}.
Task success is defined by the \textbf{joint satisfaction} of all sub-tasks under a \textbf{global, order-invariant objective}, where the model attains the highest reward only when \textbf{all} sub-tasks are completed.
Partial success does not saturate the objective and continues to provide informative gradients for optimization.
% To address this limitation, we propose a \textbf{Nested} training framework that constructs a composite task by hierarchically combining multiple sub-tasks.
% Crucially, task success is defined in terms of the \textbf{joint satisfaction} of all sub-tasks, rather than the independent success of any individual one.
% The nested objective enforces a \textbf{global success condition}, under which the model receives the highest reward only when \emph{all} sub-tasks are successfully completed, independent of the task order.
% Partial success, where only a subset of sub-tasks is solved, does not saturate the objective and continues to provide informative gradients for optimization.
In this way, nested training replaces the OR-type objective in mixed training with an explicit \textbf{AND-type} success condition, formalized as
\begin{equation}
\max_{\theta} \;
\mathbb{E}_{\tau \sim \pi_\theta}
\left[
R(\tau_1, \tau_2, \cdots, \tau_K)
\right].
\end{equation}
Because partial success does not saturate the AND-type objective, the policy avoids premature commitment, preserving exploration across sub-tasks and maintaining higher entropy throughout training.
Meanwhile, by coupling optimization signals across all sub-tasks, the AND-type objective prevents any single task from dominating the gradient, yielding balanced and stable optimization dynamics.
As shown in Fig.~\ref{fig:entropy-grad}, sustained entropy and stable gradients together enable steady, coordinated improvement across tasks and result in superior joint generalization.
% As partial success does not saturate the AND-type objective, the policy is discouraged from prematurely committing to a narrow subset of behaviors. This preserves exploration across sub-tasks and results in consistently higher policy entropy throughout training.
% Moreover, the AND-type optimization including signals on all sub-tasks, preventing any single task from independently dominating the gradient.
% Because progress on all sub-tasks is required to increase the overall reward, gradient contributions remain balanced and consistent across training steps, leading to stable optimization dynamics.
% As shown in Fig.~\ref{fig:entropy-grad}, sustained entropy and stable gradients enable steady, coordinated improvement across tasks and yield superior joint generalization.
% Crucially, the AND-type objective couples the optimization signals across sub-tasks, preventing any single task from independently dominating the gradient.
% As progress on all sub-tasks is required to increase the overall reward, gradient contributions remain balanced and consistent across training steps, leading to more stable optimization dynamics.
% Moreover, because partial success does not collapse the objective, the policy is discouraged from prematurely committing to a narrow subset of behaviors.
% This property preserves exploration across sub-tasks and results in consistently higher policy entropy throughout training, as observed in Fig.~\ref{fig:entropy-grad}.
% Together, stable gradients and sustained entropy enable steady and coordinated improvement across tasks, yielding superior joint generalization.
Importantly, nested training is neither a stricter reward scaling nor a multiplicative objective.
Rather, it elevates the learning problem to a higher level: the optimization target is no longer individual task performance, but the acquisition of a \textbf{joint capability} that is sufficient to solve all sub-tasks simultaneously.
Detailed information about nested training settings are in Appendix~\ref{app:nested}.
% Importantly, nested training is neither a stricter reward scaling nor a multiplicative objective.
% Rather, it elevates the learning problem to a higher level: the optimization target is no longer individual task performance, but the acquisition of a \textbf{joint capability} that is sufficient to solve all sub-tasks simultaneously. Detailed information about nested training settings are in Appendix~\ref{app:nested}.
% To address this limitation, we propose \textbf{Nested} training framework, which constructs a new composite task by hierarchically combining multiple sub-tasks. Instead of aggregating rewards additively, nested training defines task success in terms of the \textbf{joint satisfaction} of all sub-tasks. Concretely, the model attains the highest reward only when \emph{all} sub-tasks are successfully completed. Partial success, solving only a subset of sub-tasks, does not saturate the objective and preserves informative gradients for further optimization. In this way, nested training replaces the OR-type objective of mixed training with an explicit \textbf{AND-type} success condition, as optimization in the form:
% \begin{equation}
% \max_{\theta} \;
% \mathbb{E}_{\tau \sim \pi_\theta}
% \left[
% R(\tau_1, \tau_2, \cdots, \tau_k)
% \right],k\in[1,K]
% \end{equation}
% This formulation is not a stricter reward scaling or a multiplicative objective. Instead, nested training elevates the learning problem to a higher level: the optimization target is no longer individual task performance, but the acquisition of a \textbf{joint capability} that is sufficient to solve all sub-tasks simultaneously.
% \subsection{Training Protocol and Trajectory Generation}
% 另一种组织方式，在在这里再介绍RL训练方案——感觉也是合理的
\section{Experiment}
\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{@{}l l c c c c | c c c c c@{}}
% \hline
% \multirow{2}{*}{Setting}
% & \multirow{2}{*}{Model}
% & \multicolumn{4}{c}{\textbf{In-Domain Tasks}}
% & \multicolumn{4}{c}{\textbf{Out-of-Distribution Tasks}} \\
% \cmidrule(lr){3-6} \cmidrule(lr){7-10}
% &
\toprule
\makecell[c]{\textbf{Setting}}
& \makecell[c]{\textbf{Model}}
& \makecell[c]{\textbf{MATH}}
& \makecell[c]{\textbf{Matrix}}
& \makecell[c]{\textbf{TicTacToe}}
& \makecell[c]{\textbf{Spy}}
& \makecell[c]{\textbf{MMLU}}
& \makecell[c]{\textbf{MMLU-Pro}}
& \makecell[c]{\textbf{Common}}
& \makecell[c]{\textbf{Social}}
& \makecell[c]{\textbf{Avg.}}\\
\midrule
Base
& Qwen2.5-1.5B
& 17.20 & 7.00 & 1.00 & 2.00 & 37.87 & 13.49 & 7.79 & 27.64 & 27.10 \\
\midrule
Formal
& \textit{Math}
& \cellcolor{grey_purple} 43.20 & 21.00 & 4.00 & 9.00 & 51.38 & 20.97 & 15.08 & 65.92 & 38.34 \\
\midrule
\multirow{3}{*}{Informal}
& \textit{Matrix Games}
& 19.20 & \cellcolor{grey_purple} 44.00 & 34.00 & 14.00 & 43.79 & 18.00 & 9.55 & 64.12 & 33.87 \\
& \textit{TicTacToe}
& 22.40 & 32.00 & \cellcolor{grey_purple} 75.00 & 21.00 & 47.89 & 16.45 & 11.56 & 64.12 & 35.00 \\
& \textit{Who's the Spy}
& 19.20 & 21.00 & 0.00 & \cellcolor{grey_purple} 33.00 & 43.85 & 15.68 & 20.85 & 60.70 & 35.27 \\
\midrule
\multirow{2}{*}{F + $\text{I}_1$}
& mixed
& \cellcolor{grey_purple} 37.80 & \cellcolor{grey_purple} 29.00 & 0.00 & 16.00 & 45.19 & 17.00 & 12.81 & 59.11 & 33.53 \\
& nested
& \cellcolor{grey_purple} 40.00 & \cellcolor{grey_purple} 65.00 & 8.00 & 24.00 & \textbf{52.94} & \textbf{21.03} & \textbf{21.11} & \textbf{66.53} & \textbf{40.40} \\
\midrule
\multirow{2}{*}{F + $\text{I}_2$}
& mixed
& \cellcolor{grey_purple} 28.00 & \cellcolor{grey_purple} 30.00 & \cellcolor{grey_purple} 50.00 & 20.00 & 51.35 & 19.47 & 17.84 & 65.30 & 38.49 \\
& nested
& \cellcolor{grey_purple} 16.20 & \cellcolor{grey_purple} 20.00 & \cellcolor{grey_purple} 16.00 & 26.00 & \textbf{53.08} & \textbf{20.20} & \textbf{20.85} & \textbf{67.09} & \textbf{40.31} \\
\midrule
\multirow{2}{*}{F + $\text{I}_3$}
& mixed
& \cellcolor{grey_purple} 34.60 & \cellcolor{grey_purple} 55.00 & \cellcolor{grey_purple} 34.00 & \cellcolor{grey_purple} 37.00 & \textbf{53.55} & \textbf{21.19} & 24.37 & 67.60 & 41.68 \\
& nested
& \cellcolor{grey_purple} 22.20 & \cellcolor{grey_purple} 57.00 & \cellcolor{grey_purple} 46.00 & \cellcolor{grey_purple} 35.00 & 53.27 & 20.58 & \textbf{28.14} & \textbf{67.71} & \textbf{42.43} \\
\bottomrule
\end{tabular}
\caption{Performance of single-tasks, mixed multi-tasks, and nested multi-tasks training for base model Qwen2.5-1.5B-Instruct, where F + $\text{I}_k$ denotes the combination of formal learning and $k$ informal learning tasks. %We use MATH, Matrix, Spy, Common and Social to denote MATH500, \textit{Matrix Games}, \textit{Who's the Spy}, CommonGen and SocialIQA respectively. Avg represents the average score of general abilities, including MMLU, MMLU-Pro, CommonGen and SocialIQA.
Purple shading highlights the in-domain tasks for each setting.}% Purple shading highlights the improvement of nested training. }
\label{tab:main_results_1.5B}
\end{table*}
% -------------- 7B models -1 ---------------
% 这个{table*}[t]可能就是要分散在两页里，可能需要把7B结果放在前面？到时候看一下
\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{@{}l l c c c c | c c c c c@{}}
% \hline
% \multirow{2}{*}{Setting}
% & \multirow{2}{*}{Model}
% & \multicolumn{4}{c}{\textbf{In-Domain Tasks}}
% & \multicolumn{4}{c}{\textbf{Out-of-Distribution Tasks}} \\
% \cmidrule(lr){3-6} \cmidrule(lr){7-10}
% &
\toprule
\makecell[c]{\textbf{Setting}}
& \makecell[c]{\textbf{Model}}
& \makecell[c]{\textbf{MATH}}
& \makecell[c]{\textbf{Matrix}}
& \makecell[c]{\textbf{TicTacToe}}
& \makecell[c]{\textbf{Spy}}
& \makecell[c]{\textbf{MMLU}}
& \makecell[c]{\textbf{MMLU-Pro}}
& \makecell[c]{\textbf{Common}}
& \makecell[c]{\textbf{Social}}
& \makecell[c]{\textbf{Avg.}}\\
\midrule
Base
& Qwen2.5-7B
& 54.60 & 40.00 & 40.00 & 25.00
& 71.43 & 40.17 & 26.38 & 75.44 & 53.36 \\
\midrule
Formal
& \textit{Math}
& \cellcolor{grey_purple} 58.40 & 39.00 & 31.00 & 18.00
& 66.51 & 32.45 & 35.18 & 75.90 & 42.00 \\
\midrule
\multirow{3}{*}{Informal}
& \textit{Matrix Games}
& 58.20 & \cellcolor{grey_purple} 64.00 & 23.00 & 27.00
& 73.22 & 44.51 & 26.13 & 76.20 & 55.02 \\
& \textit{TicTacToe}
& 49.80 & 57.00 & \cellcolor{grey_purple} 78.00 & 33.00
& 73.37 & 46.35 & 28.89 & 76.41 & 56.26 \\
& \textit{Who's the Spy}
& 55.60 & 43.00 & 38.00 & \cellcolor{grey_purple} 37.00
& 71.33 & 41.17 & 28.39 & 74.92 & 53.95 \\
\midrule
\multirow{2}{*}{F + $\text{I}_1$}
& mixed
& \cellcolor{grey_purple} 58.20 & \cellcolor{grey_purple} 52.00 & 24.00 & 21.00
& 67.55 & 35.21 & 27.89 & 73.34 & 51.00\\
& nested
& \cellcolor{grey_purple} 57.40 & \cellcolor{grey_purple} 43.00 & 33.00 & 33.00
& \textbf{68.33} & \textbf{35.56} & \textbf{41.46} & \textbf{74.46} & \textbf{54.95} \\
\midrule
\multirow{2}{*}{F + $\text{I}_2$}
& mixed
& \cellcolor{grey_purple} 50.00 & \cellcolor{grey_purple} 63.00 & \cellcolor{grey_purple} 45.00 & 9.00
& 66.80 & 37.81 & \textbf{36.68} & 72.21 & 53.38 \\
& nested
& \cellcolor{grey_purple} 55.60 & \cellcolor{grey_purple} 40.00 & \cellcolor{grey_purple} 33.00 & 40.00
& \textbf{71.31} & \textbf{44.00} & 28.89 & \textbf{74.41} & \textbf{54.65}\\
\midrule
\multirow{2}{*}{F + $\text{I}_3$}
& mixed
& \cellcolor{grey_purple} 43.40 & \cellcolor{grey_purple} 49.00 & \cellcolor{grey_purple} 36.00 & \cellcolor{grey_purple} 62.00
& 68.64 & 36.87 & \textbf{35.68} & 74.41 & 53.90 \\
& nested
& \cellcolor{grey_purple} 52.20 & \cellcolor{grey_purple} 47.00 & \cellcolor{grey_purple} 47.00 & \cellcolor{grey_purple} 45.00
& \textbf{70.50} & \textbf{40.86} & 35.43 & \textbf{76.56} & \textbf{55.84} \\
\bottomrule
\end{tabular}
\caption{Performance of single-tasks, mixed multi-tasks, and nested multi-tasks training for base model Qwen2.5-7B-Instruct, where F + $\text{I}_k$ denotes the combination of formal learning and $k$ informal learning tasks.}
\label{tab:main_results_7B}
\end{table*}
In this section, we present experimental results to evaluate the effectiveness of combining formal and informal learning, as well as the impact of the proposed nested training framework.
\subsection{Environment Setup}
We briefly describe the environments, training configurations, and evaluation protocols used in our experiments, with details in the Appendix.
\subsubsection{Environments and Tasks}
We adopt \textit{Math} as the formal learning environment and a set of game-based environments as informal learning tasks, including \textit{Matrix Games}, \textit{TicTacToe}, and \textit{Who’s the Spy}. For \textit{Math} training, we use MathLv3-5 problems from the SimpleRL-Zoo-Data, following the setup in \cite{zeng2025simplerl}. For game-based training, we refer to Appendix~\ref{app:game_rule} for detailed environment descriptions and prompting strategies.
\subsubsection{Training Settings}
Following the RAGEN framework, we train language models using $\text{StarPO}^*$, with trajectory-based reinforcement learning formulation using GRPO. We evaluate two model scales, Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct \cite{qwen2,qwen2.5}, as base models. In two-player and multi-player games, we employ Qwen3-14B as opponents \cite{qwen3technicalreport}. Full training parameters and implementation are reported in Appendix~\ref{app:experiment}.
We report three main experimental settings: single-task training, multi-task mixed training, and multi-task nested training. For multi-task configurations, we investigate progressive combinations of formal and informal learning and denote the setups as $\text{F + I}_k$, where $k$ indicates the number of informal learning components included in training, reflecting the depth and complexity of informal learning. Specifically, $\text{I}_1$, $\text{I}_2$, and $\text{I}_3$ correspond to \textit{Matrix Games}, \textit{Matrix Games} combined with \textit{TicTacToe}, and the full combination including \textit{Matrix Games}, \textit{TicTacToe}, and \textit{Who's the Spy}, respectively.
\subsubsection{Evaluation Metrics}
We evaluate in-domain performance on the training tasks, including the MATH500 benchmark \cite{lightman2023lets}, \textit{Matrix Games}, \textit{TicTacToe}, and \textit{Who’s the Spy}. For games with opponents, including \textit{TicTacToe} and \textit{Who’s the Spy}, we report the average success rate against Gemini-2.5-Flash in 100 rounds \cite{comanici2025gemini}. For general and diverse abilities, we choose MMLU and MMLU-Pro for multi-domain reasoning \cite{hendrycks2021ethics,hendryckstest2021}, CommonGen for creative language generation \cite{lin-etal-2020-commongen} and SocialIQA for social reasoning \cite{sap2019socialIQa}. In CommonGen tasks, we use GPT-4o \cite{hurst2024gpt} to compare the generated outputs against ground truth references, counting better and semantically equivalent generations (ties) as success. %All reported results are percentages, and the percentage symbol (\%) is omitted for brevity.
The details are in Appendix~\ref{app:common}.
\paragraph{Notation.}
We use MATH, Matrix, Spy, Common and Social to denote MATH500, \textit{Matrix Games}, \textit{Who's the Spy}, CommonGen and SocialIQA respectively.
Avg denotes the average performance over general ability benchmarks,
including MMLU, MMLU-Pro, CommonGen and SocialIQA. We report math performance using accuracy, game performance using success rates and general-ability performance using accuracy;
all values are reported in percentage form, with the percentage symbol omitted for brevity in the tables. %We report the success rate of games and accuracy of general abilities, omitting the 百分号.
\subsection{Main Results}
\begin{figure}[t]
\includegraphics[width=\columnwidth]{image/mix-nest.pdf}
\caption{Comparison between mixed and nested training in F + $\text{I}_2$ setting with 7B base model. Vertical axis denotes the performance, purple color denotes the ID tasks and gray color denotes the general abilities.}
\label{fig:stable-train}
\end{figure}
Table~\ref{tab:main_results_1.5B} and~\ref{tab:main_results_7B} summarize the main experimental results across two model scales. We analyze these results from two perspectives: the role of formal versus informal learning and the effectiveness of nested training.
\paragraph{Formal vs. Informal Learning.}
We first examine whether distinguishing formal and informal learning is empirically meaningful. Results from single-task training show that both paradigms contribute to general ability improvement, but in complementary ways. Formal learning primarily benefits structured reasoning tasks (MMLU and MMLU-Pro), while informal learning yields stronger gains on creative writing and socially grounded benchmarks (CommonGen and SocialIQA), especially in 7B base model. These observations validate the necessity of separating formal and informal learning signals and are consistent with insights from cognitive science.
\paragraph{Effectiveness of Nested Training.}
Although both formal and informal learning are beneficial when applied in isolation, naively mixing their learning signals does not consistently translate into performance gains.
In contrast, nested training substantially alleviates performance degradation on general abilites across \textbf{all} settings.
The effect is particularly pronounced in the $\text{F + I}_1$ setting with the 1.5B base model, where nested training yields a \textbf{6.87\% absolute improvement}.
To further analyze this behavior, we compare the training dynamics of mixed and nested training in Fig.~\ref{fig:stable-train} and Fig.~\ref{fig:entropy-grad}.
Although mixed training achieves higher in-domain performance at early stages, it gradually collapses toward specific tasks (e.g., \textit{Matrix Games}), accompanied by unstable gradients and rapidly decreasing entropy, which ultimately degrades generalization performance.
As a result, mixed training may exhibit stronger in-domain results in the table, while nested training prioritizes balanced optimization across tasks. %leading to more stable and evenly improved performance.
In contrast, nested training maintains stable gradients and higher entropy throughout training, leading to steady and consistent improvements across most abilities.
These results demonstrate the superior optimization stability and overall effectiveness of the proposed nested training framework.
% \paragraph{Effectiveness of Nested Training.}
% Although both formal and informal learning are beneficial when applied in isolation, naively mixing their learning signals does not consistently translate into performance gains.
% In contrast, nested training substantially alleviates performance degradation on general abilites across \textbf{all} settings.
% The effect is particularly pronounced in the $\text{F + I}_1$ setting with the 1.5B base model, where nested training yields a \textbf{6.87\% absolute improvement}.
% To further analyze this behavior, we compare the training dynamics of mixed and nested training in Fig.~\ref{fig:stable-train} and Fig.~\ref{fig:entropy-grad}.
% Although mixed training achieves higher in-domain performance at early stages, it gradually collapses toward specific tasks (e.g., \textit{Matrix Games}), accompanied by unstable gradients and rapidly decreasing entropy, which ultimately degrades generalization performance.
% In contrast, nested training maintains stable gradients and higher entropy throughout training, leading to steady and consistent improvements across most abilities.
% These results demonstrate the superior optimization stability and overall effectiveness of the proposed nested training framework.
% To further understand this behavior, we analyze the training dynamics of mixed and nested training, as illustrated in Fig.~\ref{fig:stable-train} and Fig.~\ref{fig:entropy-grad}.
% Although mixed training achieves higher in-domain success at early stages, the optimization process gradually collapses toward specific task (\textit{Matrix Games}) while increasingly neglecting others such as \textit{TicTacToe}.
% This collapse is accompanied by unstable gradients and rapidly shrinking entropy, %indicating reduced exploration and poor optimization stability,
% and ultimately leads to consistent performance degradation across all generalization benchmarks.
% In contrast, although nested training exhibits lower in-domain performance at the early stage, it maintains stable gradients and consistently higher entropy throughout training.
% As a result, the model demonstrates steady and progressive improvements across most abilities.
% These results highlight the superior optimization stability and overall effectiveness of the proposed nested training framework.
\begin{figure}[t]
\includegraphics[width=\columnwidth]{image/entropy-grad.pdf}
\caption{Comparison between mixed and nested training in F + $\text{I}_2$ setting with 7B base model. }
\label{fig:entropy-grad}
\end{figure}
% To further understand this behavior, we analyze the training dynamics of mixed and nested training, as shown in Fig.~\ref{fig:stable-train} and~\ref{fig:entropy-grad}.
% While mixed training achieves higher in-domain success at early stages, the model gradually collapses toward simpler tasks (e.g., \textit{Matrix Games}) and increasingly abandons more complex ones such as \textit{TicTacToe}, with unstable gradients and shrinking entropy, leading to consistent performance degradation across all generalization benchmarks.
% In contrast, although nested training exhibits lower in-domain performance at the early stage, it demonstrates steady and progressive improvements across most abilities throughout training, with stable gradients and consistently high entropy.
% These results highlight the superior stability and effectiveness of the proposed nested training framework.
% Although both learning paradigms are beneficial in isolation, naively mixing formal and informal learning signals does not lead to consistent performance gains. Nested training substantially alleviates the degradation in all settings, especially in $\text{F + I}_1$ settings in 1.5B base model, where the nested training results in 6.87\% improvement. Moreover, we analyze the training dynamics of mixed and nested training, as illustrated in Fig.~\ref{fig:stable-train}. Although mixed training achieves higher in-domain success in the early stages, the model gradually collapses toward simpler tasks (e.g., \textit{Matrix Games}) while abandoning more complex ones such as \textit{TicTacToe}. This collapse is accompanied by consistent performance degradation across all generalization benchmarks. In contrast, although nested training get lower in-domain results in the beginning, it exhibits steady and progressive improvements across most abilities throughout training, demonstrating the stability and effectiveness of the proposed nested training framework. % In addition, we explore the performance of strong closed-source models in Appendix~\ref{app:api}.
% Main results are reported in Tables~\ref{tab:main_results_1.5B} and~\ref{tab:main_results_7B}.
% Across both model scales, single-task training based on either formal learning or informal learning leads to measurable improvements in general abilities, confirming that the two learning paradigms capture complementary aspects of model capability. In particular, formal learning primarily benefits structured reasoning benchmarks, while informal learning consistently yields stronger gains on socially grounded and creative tasks.
% Comparing model scales, we observe that in the 1.5B setting, both formal and informal learning improve generalization performance, whereas in the 7B setting, informal learning exhibits stronger and more consistent generalization effects than formal learning alone. We conjecture that this difference is related to the stronger instruction-following and representation capacity of larger models: for smaller models, formal learning tasks such as mathematics may additionally enhance instruction adherence, indirectly benefiting downstream evaluations, while larger models already possess sufficient instruction-following ability and therefore benefit more from experience-driven informal learning.
% Overall, these results demonstrate that the distinction between formal and informal learning is both empirically meaningful and conceptually aligned with findings from cognitive science: formal learning primarily strengthens task-specific knowledge and procedural skills, whereas informal learning contributes more to diverse and out-of-distribution scenarios.
\subsection{Case Study}
\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{@{}l l c c c c | c c c c c@{}}
% \hline
% \multirow{2}{*}{Setting}
% & \multirow{2}{*}{Model}
% & \multicolumn{4}{c}{\textbf{In-Domain Tasks}}
% & \multicolumn{4}{c}{\textbf{Out-of-Distribution Tasks}} \\
% \cmidrule(lr){3-6} \cmidrule(lr){7-10}
% &
\toprule
\makecell[c]{\textbf{Setting}}
& \makecell[c]{\textbf{Model}}
& \makecell[c]{\textbf{MATH}}
& \makecell[c]{\textbf{Matrix}}
& \makecell[c]{\textbf{TicTacToe}}
& \makecell[c]{\textbf{Spy}}
& \makecell[c]{\textbf{MMLU}}
& \makecell[c]{\textbf{MMLU-Pro}}
& \makecell[c]{\textbf{Common}}
& \makecell[c]{\textbf{Social}}
& \makecell[c]{\textbf{Avg.}}\\
\midrule
\multirow{2}{*}{F + $\text{I}_2$}
& mixed
& \cellcolor{grey_purple} 28.00 & \cellcolor{grey_purple} 30.00 & \cellcolor{grey_purple} 50.00 & 20.00 & 51.35 & 19.47 & 17.84 & 65.30 & 38.49 \\
& nested
& \cellcolor{grey_purple} 16.20 & \cellcolor{grey_purple} 20.00 & \cellcolor{grey_purple} 16.00 & 26.00 & 53.08 & 20.20 & 20.85 & 67.09 & 40.31 \\
\midrule
\multirow{2}{*}{$\text{I}_2$}
& mixed & 9.40 & \cellcolor{grey_purple} 49.00 & \cellcolor{grey_purple} 51.00 & 26.00 & 39.57 & 15.78 & 17.34 & 59.98 & 33.17 \\
& nested & 20.20 & \cellcolor{grey_purple} 24.00 & \cellcolor{grey_purple} 22.00 & 9.00 & 43.80 & 16.25 & 17.09 & 60.70 & 34.46 \\
\bottomrule
\end{tabular}
\caption{Effect of formal learning and nested training under the informal-only setting $\text{I}_2$ with 1.5B base model.}
\label{tab:ablation_informal}
\end{table*}
\begin{figure}[t]
\includegraphics[width=\columnwidth]{image/case-mini.pdf}
\caption{Case Studies on MATH500 and CommonGen.}
%\caption{Cases of MATH500 answers between base and model after \textit{Matrix Games}, and CommonGen answers between mixed and nested training 7B models in $\text{F + I}_2$ setting.}
\label{fig:case-mini}
\end{figure}
To further illustrate the effectiveness of informal learning, we present a compact case study in Fig.~\ref{fig:case-mini}, which summarizes two representative examples, with detailed comparisons in the Appendix (Fig.~\ref{fig:case-matrix-math} and~\ref{fig:case-mnc-common}).
On the MATH500 benchmark, the base 7B model follows the correct high-level reasoning but makes a subtle arithmetic error, leading to an incorrect result, whereas the \textit{Matrix Games}-trained model maintains a more explicit and verifiable derivation. Moreover, it chooses $a=1$ creatively to simplify the calculation.
On the CommonGen benchmark, while mixed training on $\text{F + I}_2$ setting produces a semantically reasonable sentence, nested training encourages deeper semantic integration by first validating the given concepts and then constructing a coherent and detailed scene.
Together, these cases demonstrate that informal learning with the nested training framework promotes more explicit, robust, and creative reasoning across both mathematical and generative tasks.
% 证明混合进去的non-formal learning确实提升了模型效果
% 几个点：informal 有意义，combination有意义
% To demonstrate the effectiveness of informal learning, we present a case study illustrating the improvement brought by \textit{Matrix Games} on the MATH500 benchmark in Fig.~\ref{fig:case-matrix-math} (Appendix). While the base 7B model follows the correct high-level reasoning, % identifying the conjugate root and constructing the corresponding quadratic polynomial,
% it commits a local arithmetic error, $(x-4)^2 - 11=x^2 -8x + 3$, % incorrectly computing the constant term from $16 - 11$. This subtle mistake propagates to the subsequent evaluation of $p(3)$ and $p(4)$,
% leading to an incorrect result. In contrast, the \textit{Matrix Games}-trained model maintains a more structured and explicit derivation by writing $(x-4)^2 - 11 = x^2 - 8x + 16 - 11$. %, which makes the critical arithmetic step transparent and easy to verify.
% Moreover, it correctly and creatively observes that the leading coefficient $a$ can be any integer and thus chooses $a=1$ for simplicity. This case demonstrates that informal learning through interactive environments promotes more explicit, verifiable, and creative reasoning, helping models avoid local mistakes.
% Furthermore, we analyze the difference between mixed and nested training strategies through a representative case study on the CommonGen benchmark, as shown in Fig.~\ref{fig:case-mnc-common} (Appendix). Although the model trained with mixed objectives already captures the semantics of individual words and produces a reasonable sentence, the model trained with nested objectives first briefly verifies the given concepts and then elaborates a coherent and detailed common scenario that integrates them naturally. % This leads to a more vivid and contextually appropriate generation, achieving win-tie results against strong human-annotated references.
% This highlights that nested training encourages deeper semantic integration and scene-level reasoning, resulting in more expressive and human-aligned generations than naive mixed training.
\subsection{Ablation Study}
% 大量消融实验，这个地方可以放几个问句那种
We conduct a series of ablation studies to further analyze the generality, necessity, and sensitivity of the proposed nested training framework.
\subsubsection{Nested Training Beyond Formal+Informal}
To examine whether the nested training framework generalizes beyond the combination of formal and informal learning, we evaluate it under an informal-learning-only setting, denoted as $\text{I}_2$, which consists of \textit{Matrix Games} and \textit{TicTacToe}. As shown in Table~\ref{tab:ablation_informal}, nested training remains effective in this setting, improving MMLU performance from 39.57\% to 43.80\% and increasing the average general ability score from 33.17\% to 34.46\%. These results demonstrate that the nested framework provides stable gains even when applied solely to informal learning environments, indicating its robustness beyond the formal-informal combination.
\subsubsection{Necessity of Formal vs Informal Learning}
% While our main results demonstrate that the combination of formal and informal learning is better than formal learning alone, we further examine whether informal learning alone is sufficient.
% Using the same $\text{I}_2$ configuration on the 1.5B model, we compare informal-only training with the $\text{F + I}_2$ setting.
% As shown in Table~\ref{tab:ablation_informal} (Appendix), informal-only training consistently underperforms the combined setting across most benchmarks under both mixed and nested training.
% In particular, removing \textit{Math} signals results in a 9.28\% drop on MMLU and a 5.85\% decrease in average general ability, highlighting that formal learning remains indispensable and cannot be fully replaced by informal learning.
The main experiments already show that combining formal and informal learning yields stronger general abilities than formal learning alone. Here, we further examine whether informal learning is sufficient, as assumed in many game-centric LLM training approaches.
Using the same $\text{I}_2$ configuration on the 1.5B base model, we compare informal-only training against the formal+informal($\text{F + I}_2$) setting. As shown in Table~\ref{tab:ablation_informal}, models trained solely on informal learning tasks consistently underperform those trained with both formal and informal learning across most benchmarks, under both mixed and nested training. In particular, removing \textit{Math} signals leads to a performance drop of 9.28\% on MMLU and 5.85\% on the average general ability score. These results highlight the critical role of formal learning, suggesting that it cannot be fully replaced by informal learning alone.
\subsubsection{Opponent Sensitivity in Multi-player Games}
We study the effect of opponent choice by training the 1.5B model against different opponents, as shown in Table~\ref{tab:ablation_opponent} (Appendix). We find that Qwen3-14B outperforms Gemini-2.5-Flash in direct evaluations, achieving success rates of 66\% in \textit{TicTacToe} and 52\% in \textit{Who’s the Spy}. Training against the Qwen3-14B opponent yields better \textit{TicTacToe} performance, aligning with the intuition that stronger opponents encourage the learning of more robust strategies. Moreover, open-sourced Qwen3-14B can be deployed locally with lower inference latency. Therefore, it is adopted as the default opponent in all experiments.
% This aligns with the intuition that stronger opponents encourage the learning of more robust strategies.
% Consistently, Qwen3-14B outperforms Gemini-2.5-Flash in direct evaluations, achieving success rates of 66\% in \textit{TicTacToe} and 52\% in \textit{Who’s the Spy}, and is therefore adopted as the default opponent in subsequent experiments.
% As shown in Table~\ref{tab:ablation_opponent} (Appendix), we investigate the impact of opponent choice by training the 1.5B base model against different opponent models. Specifically, we compare training with Qwen3-14B and Gemini-2.5-Flash as opponents. We observe that models trained against Qwen3-14B achieve better performance on \textit{TicTacToe} when evaluated against Gemini-2.5-Flash.
% Intuitively, training against a stronger opponent should encourage the model to learn more robust strategies, whereas models trained against weaker opponents are less likely to perform well in stronger adversaries. To further verify this assumption, we directly evaluate the performance between Qwen3-14B and Gemini-2.5-Flash in both \textit{TicTacToe} and \textit{Who’s the Spy}. The results show that Qwen3-14B achieves success rates of 66\% and 52\% in the two games, respectively.
% % Although these game scenarios are largely outside the distribution of typical LLM training corpora, Qwen3-14B outperforms Gemini-2.5-Flash.
% This finding suggests that Qwen3-14B serves as a strong and effective opponent for training, and we therefore adopt it as the default opponent model in subsequent experiments.
\subsubsection{Order Sensitivity of Nested Training}
%In nested training, the overall reward is defined as the average success rate across all sub-tasks.
From the nested training reward definition, the nested objective is theoretically invariant to the execution order of sub-tasks. To empirically validate this property, we conduct an order-sensitivity ablation under the $\text{F + I}_1$ setting using the 7B base model. Specifically, we compare two nested configurations: \textit{Math} $\rightarrow$ \textit{Matrix Games} and \textit{Matrix Games} $\rightarrow$ \textit{Math}. As shown in Table~\ref{tab:ablation_order} (Appendix), the two configurations yield comparable performance on average general abilities, between 54.95\% and 54.59\%. These results indicate that the proposed nested training framework is largely robust to the ordering of sub-tasks, confirming that its effectiveness does not rely on a specific order.
\section{Conclusion}
Inspired by theories in cognitive science, we model formal learning as structured \textit{Math} reasoning tasks and informal learning as interactive, game-based environments. We design three representative game settings: single-turn \textit{Matrix Games}, multi-turn two-player \textit{TicTacToe}, and multi-turn multi-player \textit{Who’s the Spy}. While a naive mixed training strategy leads to performance degradation, we show that the nested training framework enables effective integrations. Extensive experiments demonstrate that the combination of formal and informal learning is necessary and compared to mixed training, nested training improves both generalization and training stability across diverse settings, resulting in LLMs with broader abilities.
\section{Limitations}
This work considers a limited set of formal and informal learning environments, and the game designs represent only a subset of possible interactive settings. While the nested framework generalizes beyond the studied combinations, its effectiveness in more complex or real-world interactive environments remains to be explored. Moreover, our experiments use fixed opponent models in multi-agent games, and extending to self-evolve settings or adaptive opponents is left for future work.
% This work focuses on a limited set of formal and informal learning environments, and the game designs considered here represent only a subset of possible interactive settings. Although our nested framework generalizes beyond the specific combinations studied, its effectiveness under more complex, open-ended, or real-world interactive environments remains to be explored. In addition, our experiments rely on fixed opponent models in multi-agent games; extending the framework to adaptive or learning opponents is a promising direction for future work.
% Bibliography entries for the entire Anthology, followed by custom entries
%\bibliography{anthology,custom}
% Custom bibliography entries only
\bibliography{custom}
\appendix
% \section{Results of Naively Mixing}
% \label{sec:exp_mix}
% 这个地方可以用图表的形式，折线图之类的展示mix训练不稳定的问题
\section{Game Rules}
\label{app:game_rule}
We categorize games into three types according to their interaction structure: single-turn games, multi-turn two-player games, and multi-turn multi-player games. For each category, we select a representative game, namely \textit{Matrix Games}, \textit{TicTacToe}, and \textit{Who's the Spy}, respectively.
Following the terminology in VeRL~\cite{sheng2024hybridflow}, we adopt the concept of \textbf{multi-turn} rather than \textbf{multi-step}. Here, a \textbf{turn} is defined as one complete interaction round in which the trained LLM is queried to produce an action or decision, contributing to the overall game trajectory, while multi-turn refers to a game trajectory with multiple turns.
For example, \textit{Who's the Spy} is considered a three-turn game, as each player, as well as the trained LLM, participates in three major interaction rounds: two description turns and one final voting turn. Each turn corresponds to a distinct LLM query, and the sequence of these turns together constitutes a full trajectory.
\subsection{Matrix Games}
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{image/matrix2.pdf}
\caption{Detailed payoff matrices used in matrix game environments.}
\label{fig:matrix}
\end{figure*}
\textit{Matrix Games} are single-turn strategic reasoning games. We adopt a set of classic matrix games to train LLMs in strategic reasoning, specifically their ability to infer opponents’ actions and optimize their own policies toward Nash Equilibrium. In these games, the model must select an action based solely on an abstract payoff table, without access to domain-specific semantics or external knowledge. This setting encourages \emph{abstract reasoning} by requiring the model to interpret symbolic payoffs, compare outcomes across action pairs, and reason about best responses under different opponent choices. At the same time, it fosters \emph{strategic reasoning}: since rewards depend on both players’ decisions, the model must anticipate the opponent’s likely move, consider mutual incentives, and choose actions that remain robust under strategic interaction, rather than maximizing immediate payoff in isolation. Through reinforcement learning over repeated interactions, the model is trained to align its action selection with equilibrium-consistent behavior, learning to balance self-interest with opponent-aware reasoning.
Following \cite{hua2024game}, we select a collection of well-studied matrix games that are verified to admit Nash Equilibrium solutions, including Prisoner’s Dilemma, Battle of the Sexes, Game of Chicken, Stag Hunt, Radio Station, IESDS, Duopolistic Game, GAME, and Weakly Dominated Game. The corresponding payoff matrices are illustrated in Fig.~\ref{fig:matrix}.
To prevent the model from overfitting to a fixed numerical scale or specific prompt format, we apply random transformations to the payoff matrices during training. Specifically, each matrix is randomly multiplied by $-1$ or left unchanged, and an offset of $\pm 100$ is added to all entries. These transformations preserve the strategic structure and equilibrium properties of the games while encouraging the model to focus on relative payoffs and strategic relationships rather than absolute values.
In addition, we design multiple instruction prompt templates to present matrix games under diverse linguistic and contextual formulations. This further improves robustness to prompt variations and discourages reliance on superficial patterns. An illustrative example of such a prompt template is shown below:
\begin{promptbox}
{role}.
Rows = Player 1's actions [{p1_actions_list}]; Columns = Player 2's actions [{p2_actions_list}].
### P1's payoff
{p1_payoff_table}
### P2's payoff
{p2_payoff_table}
{instr}
\end{promptbox}
% \begin{lstlisting}[basicstyle=\ttfamily\footnotesize,breaklines=true]
% Rows = Player 1's actions [{p1_actions_list}]; Columns = Player 2's actions [{p2_actions_list}].
% \end{lstlisting}
\subsection{TicTacToe}
\textit{TicTacToe} is a multi-turn two-player board game in which players alternately place symbols (e.g., \texttt{X} and \texttt{O}) on a $3 \times 3$ grid, with the objective of forming a straight line of three identical symbols horizontally, vertically, or diagonally. Successful play requires each player to anticipate the opponent’s potential actions and strategically plan subsequent moves, thereby encouraging the model to reason about others’ intentions and future behaviors. In addition, \textit{TicTacToe} imposes strict constraints on valid actions: only unoccupied grid positions can be selected at each turn. These explicit legality requirements provide a clear supervision signal for action validity, which further strengthens the model’s instruction-following and rule-compliance capabilities. Prior cognitive and educational studies have also shown that turn-based board games like \textit{TicTacToe} are effective for training planning and strategic reasoning skills \cite{noda2019effectiveness}, supporting its suitability as an informal learning environment.
To enhance training robustness and reduce prompt-specific bias, we design four distinct initial prompts that describe the game setting and interaction protocol from different perspectives. Moreover, we observe that the win conditions are not always trivially recognized by LLMs, especially in multi-turn settings. To address this issue, we further introduce an auxiliary win-condition prompt that explicitly explains the winning criteria, including illustrative examples such as horizontal, vertical, and diagonal line completions. During training, the initial prompt is randomly sampled from the prompt set, and the win-condition prompt is also randomly included to improve generalization. Qwen3-14B is employed as the opponent model during training, considering its strong overall capabilities and efficient local inference speed. An illustrative example of the prompt template with the win-condition description is provided below:
\begin{promptbox}
##Game Rules: TicTacToe
**Objective**: Be the first player to connect 3 of your pieces in a continuous line.
**Player Pieces**:
- Player 1: 'O'
- Player 2: 'X'
- Empty Slot: '.'
**How to Play**:
1. The game is played on a 3x3 vertical grid.
2. Players take turns setting one of their pieces into any available slot.
**Winning Conditions**:
The game ends when a player forms a line of 3 of their own pieces. The line can be:
1. **Horizontal** (side-by-side in a row)
*Example of a horizontal win for Player 1 ('O'):*
```
X . .
O O O <-- 3 'O's in row 2
. X .
```
2. **Vertical** (stacked on top of each other in a column)
*Example of a vertical win for Player 2 ('X'):*
```
. X O
O X O <-- 3 'X's in column 2
. X .
```
3. **Diagonal** (connected at an angle)
*Example of a diagonal win (bottom-left to top-right) for Player 1:*
```
. . O
. O X <-- 3 'O's in a diagonal line
O X .
```
*Example of another diagonal win (top-left to bottom-right) for Player 2:*
```
X . O
. X O <-- 3 'X's in a diagonal line
. O X
```
**Draw Condition**:
If the entire grid is filled with pieces and no player has won, the game is a draw.
## Current Game State
{state_prompt}
## Your Turn
You are {player}.
The available actions are: {actions}.
\end{promptbox}
\subsection{Who's the Spy}
\textit{Who's the Spy} is a multi-player, multi-turn social deduction game involving $N$ players, among whom $N-1$ are assigned as \textit{Civilians} and one as the \textit{Undercover}. All players are secretly assigned a word: the civilians share the same word, while the undercover receives a different but semantically related word. Importantly, players do not know anyone's identity throughout the game.
The game proceeds in multiple speaking rounds. In each round, players take turns generating short descriptions of their assigned word, aiming to convey its meaning without revealing too much explicit information. After two full speaking rounds, all players simultaneously vote to eliminate one suspected undercover. The player receiving the most votes is removed from the game. The civilians win if the undercover is successfully eliminated, while the undercover wins if at least one civilian is voted out.
Winning \textit{Who's the Spy} requires a delicate balance between informativeness and concealment. Civilian players must describe the shared word accurately enough to signal alignment with other civilians, while avoiding overly explicit descriptions that could allow the undercover to infer the civilians’ word and adapt accordingly. Conversely, the undercover must generate plausible but strategically ambiguous descriptions to blend in and avoid detection. This interaction demands players to reason about others’ beliefs, intentions, and linguistic strategies, thereby strongly engaging theory-of-mind capabilities. At the same time, the open-ended nature of word description encourages flexible and creative language generation.
In our experiments, we fix the number of players to four, consisting of three civilians and one undercover, and set the Qwen3-14B model as training opponents. For each game instance, the identity and speaking order of the trained LLM are randomly assigned. A trajectory is considered successful if the side corresponding to the identity assigned to the trained model wins the game. The word list used in our experiments is adopted from an open-source resource\footnote{\url{https://github.com/xzx34/SocialMaze/tree/main/find_the_spy}}. To improve training robustness and reduce sensitivity to prompt phrasing, we design four distinct rule prompts that describe the game mechanics and player objectives from different perspectives. During training, one rule prompt is randomly sampled for each episode. An example of such a rule prompt is shown below:
\begin{promptbox}
Game: Who's the Undercover Agent
Roles:
- 3 Civilians share one word.
- 1 Undercover has a related but different word.
Goal:
- Civilians: Find the undercover.
- Undercover: Stay hidden until only 2 players remain.
How to Play:
Each player describes their word in one sentence (without saying the word itself).
Be subtle yet clear. After two-turn speak, everyone votes out one player. The one with most votes is eliminated.
Win:
- Civilians win if the undercover is voted out.
- Undercover wins if one civilian is voted out.
\end{promptbox}
During the testing phase, we observe that the model tends to repeatedly generate highly similar or identical descriptions across turns, which reduces linguistic diversity and weakens the effectiveness of social deduction training. To mitigate this issue, we introduce an additional hint prompt during the description phase to encourage more varied and informative language generation:
\begin{promptbox}
### Additional Rules for Description (Very Important)
- Your description MUST be clearly different from any descriptions you have given in earlier rounds.
- Do NOT reuse similar words, sentence structures, or ideas. Avoid describing it as a "process" again.
- Each round, pick a NEW angle of interpretation (e.g., its effect, its form, its symbolism, its usage, etc.)
- The new description should have LOW semantic similarity with your previous descriptions.
- Act as if you don't remember your previous answers, but you MUST ensure this answer is not similar to them.
- Always produce a new, creative, and distinct sentence.
- If the word has multiple meanings, assume 100\% that the basic meaning is intended. Never choose the less common or technical ones.
\end{promptbox}
\section{Detailed Training settings}
\label{app:experiment}
\subsection{Optimization Objective}
We adopt a trajectory-based reinforcement learning formulation using Group Relative Policy Optimization (GRPO) as the optimization backbone following the RAGEN framework\cite{wang2025ragen}. In multi-turn settings, the model interacts with the environment over a full trajectory $\tau_i$, and a scalar reward is assigned at the trajectory level. The resulting advantage is normalized and distributed across all token positions within the trajectory.
Given a group of $G$ sampled trajectories $\{\tau_i\}_{i=1}^{G}$, we define the policy ratio at token position $t$ as
\begin{equation}
r_{i,t}(\theta)
=
\frac{\pi_\theta(\tau_{i,(t)} \mid \tau_{i,<t})}
{\pi_{\text{old}}(\tau_{i,(t)} \mid \tau_{i,<t})},
\end{equation}
and its clipped version as
\begin{equation}
\tilde{r}_{i,t}(\theta)
=
\text{clip}\!\left(
r_{i,t}(\theta),
1-\epsilon,\,1+\epsilon
\right).
\end{equation}
The trajectory-level GRPO objective is then given by
\begin{equation}
\begin{aligned}
\frac{1}{G}
\sum_{i=1}^{G}
\frac{1}{|\tau_i|}
\sum_{t=1}^{|\tau_i|}
\min \Big[
r_{i,t}(\theta)\hat{A}_{i,t},
\;
\tilde{r}_{i,t}(\theta)\hat{A}_{i,t}
\Big],
\end{aligned}
\end{equation}
where $\hat{A}_{i,t}$ denotes the normalized advantage at token position $t$ within trajectory $\tau_i$, and $\epsilon$ is the clipping threshold.
This objective follows a standard PPO-style formulation and serves as a unified optimization backbone for all training settings in this work.
\subsection{Nested Training Framework}
\label{app:nested}
Nested training constructs a composite learning task by sequentially composing multiple sub-tasks into a single trajectory. Specifically, a full trajectory in nested training consists of the concatenation of trajectories from all constituent sub-tasks, executed in sequence within one episode. Each sub-task preserves its original interaction protocol and success condition, while the model is required to solve all sub-tasks within a unified rollout.
The reward for a nested trajectory is defined as the average success across all sub-tasks, rather than a strict conjunction (i.e., requiring all sub-tasks to succeed simultaneously). This design choice avoids making the optimization problem excessively difficult, especially for 1.5B models in early training stages, while still enforcing joint task competence. Partial success on a subset of sub-tasks yields intermediate rewards, providing informative gradients that encourage balanced learning across tasks.
This formulation fundamentally differs from naive task mixing. In mixed training, each trajectory corresponds to a single sub-task sampled from a task pool, and the optimization objective aggregates rewards across trajectories from different tasks. In contrast, nested training embeds all sub-tasks within the same trajectory, forcing the model to reason across multiple task contexts in a single rollout. As a result, gradients are jointly influenced by all sub-tasks, reducing the risk that learning is dominated by easier tasks and improving credit assignment for harder ones.
From an optimization perspective, nested trajectories exhibit higher action and state diversity compared to single-task trajectories, leading to increased entropy during training. This property empirically contributes to improved gradient stability and more robust exploration. Moreover, since the nested reward is computed as an average over sub-task rewards, the overall objective is invariant to the ordering of sub-tasks within a trajectory. We empirically verify this order-invariance property in our ablation studies, where different nesting orders yield comparable performance.
%In our experiments, nested training uses the same set and number of sub-tasks as mixed training to ensure a fair comparison. The maximum trajectory length in nested training is set to the sum of the maximum turn counts of all sub-tasks, while each sub-task retains its task-specific termination conditions. These design choices allow nested training to preserve task diversity while maintaining stable and efficient optimization.
\subsection{Training Configuration and Implementation}
Following the RAGEN framework based on VeRL~\cite{sheng2024hybridflow}, we utilize the stable $\text{StarPO}^*$ algorithm. The key improvements in $\text{StarPO}^*$ are DAPO \cite{yu2025dapo} and trajectory filtering. DAPO removes the KL-term and employs a clip-higher strategy. The trajectory filtering is based on variance, where only the top 25\% of trajectories with the highest variance are retained for each round.
Regarding hyperparameters, each setting runs a maximum of 250 rollout-upodate iterations, with the group size fixed to 16. During training, we fixed the random seed to generate the task prompts to ensure reproducibility.
The maximum number of turns for each sub-task is determined by the nature of the task. For single-turn tasks such as \textit{Math} and \textit{Matrix Games}, the maximum number of turns is 1. For multi-turn tasks, the maximum turns for \textit{TicTacToe} is set to 5, since the game involves a 3x3 grid, and the game will always end after the 5th turn. In \textit{Who's the Spy}, the maximum turns is 3, consisting of two description rounds and one voting round. In mixed training, the \texttt{max\_turn} is set to the maximum number of turns across all sub-tasks. In nested training, the \texttt{max\_turn} is the sum of the maximum number of turns for all sub-tasks.
The model is optimized using Generalized Advantage Estimation (GAE) with $\gamma = 1.0$ and $\lambda = 1.0$, along with the Adam optimizer where $\beta_1 = 0.9$ and $\beta_2 = 0.999$. We also apply entropy regularization with a coefficient $\beta = 0.001$. The reward is defined as 1 for success and 0 for failure, with a reward of 0.5 assigned to draws in \textit{TicTacToe}. The format penalty of $-0.1$ is applied.
In addition, for the $\text{F + I}_3$ setting with the 1.5B model, nested training is performed after a brief warm-up stage using a small amount of mixed-task data. This design is motivated by the observation that directly training the 1.5B model from the base initialization on the full combination of all tasks leads to unstable optimization and frequent failure to complete all sub-tasks. The warm-up phase serves as a cold-start mitigation strategy, enabling the model to acquire basic competence across individual tasks before being exposed to the more challenging nested training objective.
%In addition, in $\text{F + I}_3$ scenarios on 1.5B setting, the nested training is conducted after the model trained on a small amount of mixed data, since the combination of all-tasks is too hard for the 1.5B models to accomplish from base.
% The reward is defined 1 for success and 0 for failure (0.5 for draw in \textit{TicTacToe}), with a format penalty of -0.1.
In multi-task settings, both mixed and nested training scenarios involve an equal number of tasks, ensuring fairness in task comparison.
\section{Detailed Evaluation Settings}
\label{app:common}
We introduce the detailed evaluation settings including math, game and general abilities scenarios.
\subsection{Math Evaluation}
We evaluate the model’s math reasoning ability using the MATH500 benchmark. Model outputs are compared against ground-truth answers using the Math-Verify toolkit\footnote{\url{https://github.com/huggingface/Math-Verify}}, which provides a robust verification pipeline for mathematical equivalence and correctness.
\subsection{Game Evaluation}
To assess game-playing ability, we evaluate the model’s success rate over 100 independent game rounds, with the opponent model fixed to Gemini-2.5-Flash for \textit{TicTacToe} and \textit{Who's the Spy} tasks. For each game, the success criterion is defined in accordance with the corresponding training objective. Specifically, in \textit{Matrix Games}, a trajectory is considered successful if the selected action satisfies the Nash Equilibrium condition. In \textit{TicTacToe}, success is defined as achieving a valid three-in-a-row configuration (horizontal, vertical, or diagonal) for the trained model’s pieces. To make the evaluation of \textit{TicTacToe} fair, models are tested on an empty board and the first player is randomly selected. In \textit{Who's the Spy}, a trajectory is counted as successful if the side corresponding to the trained model’s assigned identity wins the game.
\subsection{General Abilities Evaluation}
To evaluate the model’s generalization across diverse ability dimensions, we select a set of widely used benchmarks, including MMLU, MMLU-Pro, CommonGen, and SocialIQA.
\paragraph{MMLU and MMLU-Pro.}
MMLU covers a broad range of academic subjects and evaluates knowledge and reasoning through four-choice questions. MMLU-Pro extends MMLU by increasing both task difficulty and answer space, expanding each question from four options to ten options, thereby posing a more challenging evaluation setting.
\paragraph{CommonGen.}
The CommonGen benchmark is designed to evaluate the model’s language generation ability by requiring it to produce a coherent and plausible sentence that incorporates a given set of concept words with specified semantic roles, as illustrated in Fig.~\ref{fig:case-mnc-common}. Since CommonGen is an open-ended generation task, we follow the evaluation protocol provided by allenai\footnote{\url{https://github.com/allenai/CommonGen-Eval}}, using few-shot prompting and GPT-4o as an automatic evaluator to compare model outputs against human-annotated reference sentences. To reduce positional bias during judgment, we randomly permute the order of the model-generated output and the reference sentence, and instruct the evaluator to carefully compare both candidates. Since the human-annotated references represent a strong upper bound that is difficult to consistently surpass, we report the success rate using a win-or-tie criterion. The detailed evaluation prompt is provided below:
\begin{promptbox}
# Data
Given several concepts (i.e., nouns or verbs), we ask models to write a short and simple sentence that contains *all* the required words.
The sentence should describe a common scene in daily life, and the concepts should be used in a natural way.
Concepts: "{concept_list}"
Model A: "{candidate_A}"
Model B: "{candidate_B}"
# Your Task
Your task is to choose a better sentence from the two candidates. Decide which model's sentence is better in terms of the naturalness and commonness of the scenes they describe.
## Rules:
- A better sentence should describe a common scene in daily life, and all concepts should be used in a natural way.
- You should prefer sentences that use all given concepts with correct part-of-speech tags.
- A simpler and shorter sentence is preferred if it describes the same scene as the other sentence.
- If you think both sentences are equally good or bad, please choose "tie".
Now, please output your choice ("A" or "B" or "tie").
Your choice:
\end{promptbox}
\paragraph{SocialIQA.}
SocialIQA is a question-answering benchmark designed to evaluate social reasoning through three-choice questions. Unlike prior benchmarks that emphasize physical or taxonomic knowledge, SocialIQA focuses on understanding people’s actions and their social motivations. Given a described action, the model must infer the most plausible social intent or implication among multiple candidates. The dataset covers a wide range of everyday social situations, with answer options consisting of both human-written and adversarially filtered machine-generated candidates.
\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{@{}l l c c c c | c c c c c@{}}
% \hline
% \multirow{2}{*}{Setting}
% & \multirow{2}{*}{Model}
% & \multicolumn{4}{c}{\textbf{In-Domain Tasks}}
% & \multicolumn{4}{c}{\textbf{Out-of-Distribution Tasks}} \\
% \cmidrule(lr){3-6} \cmidrule(lr){7-10}
% &
\toprule
\makecell[c]{\textbf{Setting}}
& \makecell[c]{\textbf{Opponent}}
& \makecell[c]{\textbf{MATH}}
& \makecell[c]{\textbf{Matrix}}
& \makecell[c]{\textbf{TicTacToe}}
& \makecell[c]{\textbf{Spy}}
& \makecell[c]{\textbf{MMLU}}
& \makecell[c]{\textbf{MMLU-Pro}}
& \makecell[c]{\textbf{Common}}
& \makecell[c]{\textbf{Social}}
& \makecell[c]{\textbf{Avg.}}\\
\midrule
\multirow{2}{*}{\textit{TicTacToe}}
& Gemini-2.5-Flash
& 25.80 & 17.00 & \cellcolor{grey_purple} 46.00 & 12.00 & 39.00 & 13.79 & 14.32 & 51.18 & 29.57 \\
& \textbf{Qwen3-14B}
& 22.40 & 32.00 & \cellcolor{grey_purple} 75.00 & 21.00 & 47.89 & 16.45 & 11.56 & 64.12 & 35.00 \\
\midrule
\multirow{2}{*}{\textit{Who's the Spy}}
& Gemini-2.5-Flash
& 19.20 & 19.00 & 2.00 & \cellcolor{grey_purple} 38.00 & 46.86 & 17.21 & 20.85 & 59.57 & 36.12 \\
& \textbf{Qwen3-14B}
& 19.20 & 21.00 & 0.00 & \cellcolor{grey_purple} 33.00 & 43.85 & 15.68 & 20.85 & 60.70 & 35.27 \\
\bottomrule
\end{tabular}
\caption{Opponent sensitivity analysis of \textit{TicTacToe} and \textit{Who's the Spy} training for base 1.5B model, comparing Gemini-2.5-Flash and Qwen3-14B as opponents.}
\label{tab:ablation_opponent}
\end{table*}
\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{@{}l l c c c c | c c c c c@{}}
% \hline
% \multirow{2}{*}{Setting}
% & \multirow{2}{*}{Model}
% & \multicolumn{4}{c}{\textbf{In-Domain Tasks}}
% & \multicolumn{4}{c}{\textbf{Out-of-Distribution Tasks}} \\
% \cmidrule(lr){3-6} \cmidrule(lr){7-10}
% &
\toprule
\makecell[c]{\textbf{Model}}
& \makecell[c]{\textbf{Order}}
& \makecell[c]{\textbf{MATH}}
& \makecell[c]{\textbf{Matrix}}
& \makecell[c]{\textbf{TicTacToe}}
& \makecell[c]{\textbf{Spy}}
& \makecell[c]{\textbf{MMLU}}
& \makecell[c]{\textbf{MMLU-Pro}}
& \makecell[c]{\textbf{Common}}
& \makecell[c]{\textbf{Social}}
& \makecell[c]{\textbf{Avg.}}\\
\midrule
\multirow{2}{*}{nested}
& \textit{Math} $\rightarrow$ \textit{Matrix Games}
& \cellcolor{grey_purple} 57.40 & \cellcolor{grey_purple} 43.00 & 33.00 & 33.00 & 68.33 & 35.56 & 41.46 & 74.46 & \textbf{54.95} \\
& \textit{Matrix Games} $\rightarrow$ \textit{Math}
& \cellcolor{grey_purple} 58.40 & \cellcolor{grey_purple} 58.00 & 33.00 & 45.00 & 70.48 & 38.56 & 35.43 & 73.90 & \textbf{54.59} \\
\bottomrule
\end{tabular}
\caption{Order sensitivity analysis of nested training on \textit{Math} and \textit{Matrix Games} using the 7B model.}
\label{tab:ablation_order}
\end{table*}
\section{Detailed Ablation Study}
The detailed ablation study tables are provided here, including the opponent sensitivity analysis and the order sensitivity analysis.
\paragraph{Opponent Sensitivity.}
We first analyze the sensitivity of training performance to the choice of opponent model. Specifically, we compare Gemini-2.5-Flash and Qwen3-14B as fixed opponents for \textit{TicTacToe} and \textit{Who's the Spy} training under the 1.5B setting, as shown in Table~\ref{tab:ablation_opponent}.
Overall, using Qwen3-14B as the opponent consistently leads to stronger performance across most in-domain and out-of-domain benchmarks. In particular, for \textit{TicTacToe}, training against Qwen3-14B yields substantially higher game success rates and improved generalization on MMLU, SocialIQA, and the averaged score. Similar trends are observed for \textit{Who's the Spy}, where Qwen3-14B as the opponent results in competitive outcomes.
Beyond performance gains, Qwen3-14B also offers practical advantages in training efficiency. As a locally deployed model, it provides significantly faster and more stable inference compared to closed-source APIs, enabling scalable and reproducible self-play training. These results suggest that a strong yet efficient local opponent can provide higher-quality interaction signals, leading to more effective informal learning.
\paragraph{Order Sensitivity.}
We further investigate whether the ordering of sub-tasks in nested training affects final performance. Table~\ref{tab:ablation_order} reports results for two different nesting orders of \textit{Math} and \textit{Matrix Games} using the 7B model.
While slight differences can be observed across individual benchmarks depending on the task order, the overall performance remains highly consistent. In particular, the averaged scores across all evaluation metrics are nearly identical for different orders, indicating that nested training is insensitive to the specific sequencing of sub-tasks. This empirical robustness aligns with the design of the nested objective, where rewards are computed as an average over sub-task successes, making the optimization objective invariant to task order.
Together, these ablation results demonstrate that the proposed framework is robust to both opponent selection and task ordering, further supporting the stability and general applicability of nested training.
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{image/case-matrix-math.pdf}
\caption{
An example of the effectiveness of informal learning on the MATH500 benchmark, with Qwen2.5-7B-Instruct base model trained on \textit{Matrix Games}.
}
\label{fig:case-matrix-math}
\end{figure*}
\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{image/case-mnc-common.pdf}
\caption{
An example of the effectiveness of nested training on the CommonGen benchmark, comparing mixed training with nested training on $\text{F + I}_1$ settings.
}
\label{fig:case-mnc-common}
\end{figure*}
\section{Detailed Case Studies}
\label{app:case-study}
\paragraph{Effectiveness of Informal Learning on Mathematical Reasoning.}
To provide a more detailed illustration of how informal learning improves mathematical reasoning, we present a representative case study on the MATH500 benchmark in Fig.~\ref{fig:case-matrix-math}. The example compares the base \textit{Qwen2.5-7B-Instruct} model with the same model after training on \textit{Matrix Games}.
In this example, both models correctly identify that if a quadratic polynomial with integer coefficients has $4-\sqrt{11}$ as a root, then its conjugate $4+\sqrt{11}$ must also be a root. The base model further recognizes the high-level structure of the solution and attempts to construct the corresponding polynomial. However, it commits a subtle arithmetic error when expanding $(x-4)^2 - 11$, incorrectly computing $16 - 11$ as $3$. This local mistake propagates to the subsequent evaluation of $p(3)$ and $p(4)$, ultimately leading to an incorrect final answer, despite the overall reasoning strategy being correct.
In contrast, the model trained with \textit{Matrix Games} exhibits a more structured and explicit derivation. Instead of directly simplifying intermediate expressions, it expands the polynomial step by step as
\[
(x-4)^2 - 11 = x^2 - 8x + 16 - 11,
\]
making the critical arithmetic operation transparent and easy to verify. Moreover, it explicitly observes that the leading coefficient $a$ can be any integer and chooses $a=1$ for simplicity, demonstrating greater flexibility and creativity in solution construction. This explicit and verifiable reasoning process helps the model avoid local arithmetic errors and arrive at the correct final result. This case suggests that informal learning through interactive game environments encourages more cautious, structured, and self-checking reasoning behaviors in mathematical problem solving.
\paragraph{Effectiveness of Nested Training on Semantic Generation.}
We further analyze the difference between mixed and nested training strategies through a detailed case study on the CommonGen benchmark, as shown in Fig.~\ref{fig:case-mnc-common}. The task requires generating a coherent daily-life sentence that naturally incorporates all given concepts.
The model trained with mixed objectives is able to correctly include the required concepts and produce a grammatically valid sentence. However, its generation primarily focuses on satisfying individual lexical constraints, resulting in a relatively minimal and less vivid description. While the output is semantically acceptable, it lacks explicit scene construction and deeper integration among the concepts.
In contrast, the model trained with nested objectives demonstrates a more deliberate generation process. Before producing the final sentence, it briefly verifies the required concepts and then constructs a coherent common scenario that naturally binds them together, such as people sitting at a table and placing food in front of each other. This leads to a more expressive, contextually rich, and human-aligned sentence. As a result, the nested-trained model achieves a win-or-tie outcome when compared against strong human-annotated references.
This comparison highlights that nested training encourages deeper semantic integration and scene-level reasoning, going beyond word-level constraint satisfaction. By requiring the model to jointly consider multiple objectives within a single trajectory, nested training promotes more coherent and expressive language generation than naive mixed training.
\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{@{} l c c c c | c c c c c@{}}
% \hline
% \multirow{2}{*}{Setting}
% & \multirow{2}{*}{Model}
% & \multicolumn{4}{c}{\textbf{In-Domain Tasks}}
% & \multicolumn{4}{c}{\textbf{Out-of-Distribution Tasks}} \\
% \cmidrule(lr){3-6} \cmidrule(lr){7-10}
% &
\toprule
\makecell[c]{\textbf{Model}}
& \makecell[c]{\textbf{MATH}}
& \makecell[c]{\textbf{Matrix}}
& \makecell[c]{\textbf{TicTacToe}}
& \makecell[c]{\textbf{Spy}}
& \makecell[c]{\textbf{MMLU}}
& \makecell[c]{\textbf{MMLU-Pro}}
& \makecell[c]{\textbf{Common}}
& \makecell[c]{\textbf{Social}}
& \makecell[c]{\textbf{Avg.}}\\
\midrule
GPT-4o
& 59.80 & 48.00 & 63.00 & 70.00 & 85.08 & 66.00 & 53.02 & 79.02 & 70.78 \\
Gemini3-Flash
& 49.80 & 64.00 & 86.00 & 16.00 & 78.42 & 52.01 & 41.96 & 74.87 & 61.82 \\
DeepSeek-V3.2 & 55.80 & 61.00 & 88.00 & 69.00 & 86.73 & 63.69 & 67.34 & 80.19 & 74.49 \\
\bottomrule
\end{tabular}
\caption{Additional results of close-source models. }
\label{tab:add_api}
\end{table*}
\section{Additional Results of Closed-Source models}
\label{app:api}
% We further conduct experiments on strong closed-source models, including GPT-4o, Gemini3-Flash-Preview and DeepSeekV3.2(chat mode, without thinking) \cite{liu2025deepseek}. The evaluation methods are aligned with the main experiments. Results are shown in Table.~\ref{tab:add_api}.
% We further evaluate a set of strong closed-source models, including GPT-4o, Gemini-3-Flash-Preview, and DeepSeek-V3.2 (chat mode, without reasoning)~\cite{liu2025deepseek}. All evaluation protocols are strictly aligned with those used in the main experiments. The corresponding results are reported in Table~\ref{tab:add_api}.
We further evaluate a set of strong closed-source models, including GPT-4o, Gemini-3-Flash-Preview, and DeepSeek-V3.2 (chat mode, without reasoning)~\cite{liu2025deepseek}. All evaluation protocols are strictly aligned with those used in the main experiments. The corresponding results are reported in Table~\ref{tab:add_api}.
Overall, these results demonstrate that strong closed-source models exhibit competitive performance across both formal and informal learning benchmarks, while showing distinct strengths on different task categories. GPT-4o achieves balanced performance across mathematical reasoning, social reasoning, and general knowledge benchmarks, reflecting its strong general-purpose capability. Gemini-3-Flash-Preview performs particularly well on interactive game tasks such as \textit{TicTacToe} and \textit{Matrix Games}, but shows relatively weaker performance on social deduction and general language generation tasks. DeepSeek-V3.2 achieves the highest overall average score, with consistently strong results on math, game, and general ability benchmarks, indicating that competitive generalization can be achieved even without explicit reasoning traces.These observations suggest that while closed-source models benefit from large-scale training and strong base capabilities, their performance profiles vary substantially across different ability dimensions.
Moreover, we observe that the 1.5B and 7B models after RL training can, under certain domain-specific or nested training settings, achieve performance comparable to or even surpass strong closed-source models on targeted benchmarks. As shown in the main results, a 1.5B model trained under the $\text{F + I}_1$ setting attains 65.00\% accuracy on \textit{Matrix Games}, exceeding the performance of all evaluated closed-source models in this domain.
Similar phenomena are also observed for 7B models on several general ability benchmarks, including MMLU-Pro, CommonGen, and SocialIQA, where models trained with nested or domain-aligned tasks achieve results on par with those of closed-source APIs. These gains are particularly pronounced on tasks that require structured reasoning, interaction, or social inference, which are directly targeted by the corresponding informal learning environments.
Taken together, these results indicate that, despite the substantially smaller model size and more limited training data, our proposed training framework can effectively elicit strong domain-specific and transferable capabilities through structured and nested reinforcement learning. This comparison with closed-source models provides further evidence that carefully designed interactive training objectives can partially compensate for scale and serve as an efficient alternative for capability enhancement.
\end{document}
% \section{Document Body}
% \subsection{Footnotes}
% Footnotes are inserted with the \verb|\footnote| command.\footnote{This is a footnote.}
% \subsection{Tables and figures}
% See Table~\ref{tab:accents} for an example of a table and its caption.
% \textbf{Do not override the default caption sizes.}
% \begin{table}
% \centering
% \begin{tabular}{lc}
% \hline
% \textbf{Command} & \textbf{Output} \\
% \hline
% \verb|{\"a}| & {\"a} \\
% \verb|{\^e}| & {\^e} \\
% \verb|{\`i}| & {\`i} \\
% \verb|{\.I}| & {\.I} \\
% \verb|{\o}| & {\o} \\
% \verb|{\'u}| & {\'u} \\
% \verb|{\aa}| & {\aa} \\\hline
% \end{tabular}
% \begin{tabular}{lc}
% \hline
% \textbf{Command} & \textbf{Output} \\
% \hline
% \verb|{\c c}| & {\c c} \\
% \verb|{\u g}| & {\u g} \\
% \verb|{\l}| & {\l} \\
% \verb|{\~n}| & {\~n} \\
% \verb|{\H o}| & {\H o} \\
% \verb|{\v r}| & {\v r} \\
% \verb|{\ss}| & {\ss} \\
% \hline
% \end{tabular}
% \caption{Example commands for accented characters, to be used in, \emph{e.g.}, Bib\TeX{} entries.}
% \label{tab:accents}
% \end{table}
% As much as possible, fonts in figures should conform
% to the document fonts. See Figure~\ref{fig:experiments} for an example of a figure and its caption.
% Using the \verb|graphicx| package graphics files can be included within figure
% environment at an appropriate point within the text.
% The \verb|graphicx| package supports various optional arguments to control the
% appearance of the figure.
% You must include it explicitly in the \LaTeX{} preamble (after the
% \verb|\documentclass| declaration and before \verb|\begin{document}|) using
% \verb|\usepackage{graphicx}|.
% \begin{figure}[t]
% \includegraphics[width=\columnwidth]{example-image-golden}
% \caption{A figure with a caption that runs for more than one line.
% Example image is usually available through the \texttt{mwe} package
% without even mentioning it in the preamble.}
% \label{fig:experiments}
% \end{figure}
% \begin{figure*}[t]
% \includegraphics[width=0.48\linewidth]{example-image-a} \hfill
% \includegraphics[width=0.48\linewidth]{example-image-b}
% \caption {A minimal working example to demonstrate how to place
% two images side-by-side.}
% \end{figure*}
% \subsection{Hyperlinks}
% Users of older versions of \LaTeX{} may encounter the following error during compilation:
% \begin{quote}
% \verb|\pdfendlink| ended up in different nesting level than \verb|\pdfstartlink|.
% \end{quote}
% This happens when pdf\LaTeX{} is used and a citation splits across a page boundary. The best way to fix this is to upgrade \LaTeX{} to 2018-12-01 or later.
% \subsection{Citations}
% \begin{table*}
% \centering
% \begin{tabular}{lll}
% \hline
% \textbf{Output} & \textbf{natbib command} & \textbf{ACL only command} \\
% \hline
% \citep{Gusfield:97} & \verb|\citep| & \\
% \citealp{Gusfield:97} & \verb|\citealp| & \\
% \citet{Gusfield:97} & \verb|\citet| & \\
% \citeyearpar{Gusfield:97} & \verb|\citeyearpar| & \\
% \citeposs{Gusfield:97} & & \verb|\citeposs| \\
% \hline
% \end{tabular}
% \caption{\label{citation-guide}
% Citation commands supported by the style file.
% The style is based on the natbib package and supports all natbib citation commands.
% It also supports commands defined in previous ACL style files for compatibility.
% }
% \end{table*}
% Table~\ref{citation-guide} shows the syntax supported by the style files.
% We encourage you to use the natbib styles.
% You can use the command \verb|\citet| (cite in text) to get ``author (year)'' citations, like this citation to a paper by \citet{Gusfield:97}.
% You can use the command \verb|\citep| (cite in parentheses) to get ``(author, year)'' citations \citep{Gusfield:97}.
% You can use the command \verb|\citealp| (alternative cite without parentheses) to get ``author, year'' citations, which is useful for using citations within parentheses (e.g. \citealp{Gusfield:97}).
% A possessive citation can be made with the command \verb|\citeposs|.
% This is not a standard natbib command, so it is generally not compatible
% with other style files.
% \subsection{References}
% \nocite{Ando2005,andrew2007scalable,rasooli-tetrault-2015}
% The \LaTeX{} and Bib\TeX{} style files provided roughly follow the American Psychological Association format.
% If your own bib file is named \texttt{custom.bib}, then placing the following before any appendices in your \LaTeX{} file will generate the references section for you:
% \begin{quote}
% \begin{verbatim}
% \bibliography{custom}
% \end{verbatim}
% \end{quote}
% You can obtain the complete ACL Anthology as a Bib\TeX{} file from \url{https://aclweb.org/anthology/anthology.bib.gz}.
% To include both the Anthology and your own .bib file, use the following instead of the above.
% \begin{quote}
% \begin{verbatim}
% \bibliography{anthology,custom}
% \end{verbatim}
% \end{quote}
% Please see Section~\ref{sec:bibtex} for information on preparing Bib\TeX{} files.
% \subsection{Equations}
% An example equation is shown below:
% \begin{equation}
% \label{eq:example}
% A = \pi r^2
% \end{equation}
% Labels for equation numbers, sections, subsections, figures and tables
% are all defined with the \verb|\label{label}| command and cross references
% to them are made with the \verb|\ref{label}| command.
% This an example cross-reference to Equation~\ref{eq:example}.
% \subsection{Appendices}
% Use \verb|\appendix| before any appendix section to switch the section numbering over to letters. See Appendix~\ref{sec:appendix} for an example.
% \section{Bib\TeX{} Files}
% \label{sec:bibtex}
% Unicode cannot be used in Bib\TeX{} entries, and some ways of typing special characters can disrupt Bib\TeX's alphabetization. The recommended way of typing special characters is shown in Table~\ref{tab:accents}.
% Please ensure that Bib\TeX{} records contain DOIs or URLs when possible, and for all the ACL materials that you reference.
% Use the \verb|doi| field for DOIs and the \verb|url| field for URLs.
% If a Bib\TeX{} entry has a URL or DOI field, the paper title in the references section will appear as a hyperlink to the paper, using the hyperref \LaTeX{} package.
% \section*{Limitations}
% This document does not cover the content requirements for ACL or any
% other specific venue. Check the author instructions for
% information on
% maximum page lengths, the required ``Limitations'' section,
% and so on.
% \section*{Acknowledgments}
% This document has been adapted
% by Steven Bethard, Ryan Cotterell and Rui Yan
% from the instructions for earlier ACL and NAACL proceedings, including those for
% ACL 2019 by Douwe Kiela and Ivan Vuli\'{c},
% NAACL 2019 by Stephanie Lukin and Alla Roskovskaya,
% ACL 2018 by Shay Cohen, Kevin Gimpel, and Wei Lu,
% NAACL 2018 by Margaret Mitchell and Stephanie Lukin,
% Bib\TeX{} suggestions for (NA)ACL 2017/2018 from Jason Eisner,
% ACL 2017 by Dan Gildea and Min-Yen Kan,
% NAACL 2017 by Margaret Mitchell,
% ACL 2012 by Maggie Li and Michael White,
% ACL 2010 by Jing-Shin Chang and Philipp Koehn,
% ACL 2008 by Johanna D. Moore, Simone Teufel, James Allan, and Sadaoki Furui,
% ACL 2005 by Hwee Tou Ng and Kemal Oflazer,
% ACL 2002 by Eugene Charniak and Dekang Lin,
% and earlier ACL and EACL formats written by several people, including
% John Chen, Henry S. Thompson and Donald Walker.
% Additional elements were taken from the formatting instructions of the \emph{International Joint Conference on Artificial Intelligence} and the \emph{Conference on Computer Vision and Pattern Recognition}.