Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization

Extracted source text · arXiv 2510.19325

% Template for ICASSP-2026 paper; to be used with:
%          spconf.sty  - ICASSP/ICIP LaTeX style file, and
%          IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
\documentclass{article}
\usepackage{spconf,amsmath,graphicx,hyperref}
\usepackage{amsmath}  % For mathematical symbols and equations
\usepackage{amssymb}  % For additional symbols (like \mathbb)
\usepackage{graphicx}  % 在导言区导入 graphicx 包
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{amsmath}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{enumitem}
\usepackage{graphicx} % 需要引入这个包来使用图形功能
\usepackage{amsmath}  % 如果有数学公式需要使用此包
\usepackage{multirow} % 用来合并单元格的包（如果需要）
% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

% Title.
% ------
\title{Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization}
%
% Single address.
% ---------------
\name{Junjie Song\sthanks{Equal Contribution}, 
Yiwen Liu$^*$,
Dapeng Li$^*$,
Yin Sun,
Shukun Fu,
Siqi Chen,
Yuji Cao\sthanks{Corresponding Author}
}
\address{AI-4-Business of Li Auto Inc.  \\ Beijing, China}

% For example:
% ------------
%\address{School\\
%	Department\\
%	Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
% \twoauthors
%  {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
% 	{School A-B\\
% 	Department A-B\\
% 	Address A-B}
%  {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
% 	while at ...}}
% 	{School C-D\\
% 	Department C-D\\
% 	Address C-D}
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
Text summarization is a crucial task that requires the simultaneous optimization of multiple objectives, including consistency, coherence, relevance, and fluency, which presents considerable challenges. Although large language models (LLMs) have demonstrated remarkable performance, enhanced by reinforcement learning (RL), few studies have focused on optimizing the multi-objective problem of summarization through RL based on LLMs. In this paper, we introduce hypervolume optimization (HVO), a novel optimization strategy that dynamically adjusts the scores between groups during the reward process in RL by using the hypervolume method. This method guides the model’s optimization to progressively approximate the pareto front, thereby generating balanced summaries across multiple objectives.
% In this paper, we introduce hypervolume optimization (\textbf{HVO}), a novel optimization strategy that dynamically adjusts the scores between groups during the reward
% process in RL using the hypervolume method, guiding the model’s
% optimization towards progressively approximating the Pareto front to generate balanced summaries across multiple objectives. 
Experimental results on several representative summarization datasets demonstrate that our method outperforms group relative policy optimization (GRPO) in overall scores and shows more balanced performance across different dimensions. Moreover, a 7B foundation model enhanced by HVO performs comparably to GPT-4 in the summarization task, while maintaining a shorter generation length. Our code is publicly available at \url{https://github.com/ai4business-LiAuto/HVO.git}
\end{abstract}
%
\begin{keywords}
Multi-objective Reinforcement Learning,
Text Summarization,
Hypervolume Optimization
\end{keywords}
%
\section{Introduction}
\label{sec:intro}
Text summarization is a core and challenging task in natural language processing (NLP)~\cite{DBLP:journals/csur/ZhangYZ25}. To comprehensively evaluate the quality of generated summaries, researchers typically examine multiple dimensions, such as coherence, consistency, fluency, and relevance~\cite{DBLP:conf/emnlp/LiuIXWXZ23}. 
However, optimizing the objectives of these dimensions simultaneously is challenging, as improvements along one dimension may lead to compromises in others~\cite{abo2023automatic}, resulting in imbalanced summaries.
\begin{figure}[htb]
\begin{minipage}[b]{1\linewidth}
  \centering
  \centerline{\includegraphics[width=1\textwidth]{figures/radar2.pdf}}
%  \vspace{2.0cm}
\end{minipage}
%
\caption{The radar chart shows the scores of different models in each dimension, evaluated by UniEval~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22} on the CNN/DailyMail~\cite{nallapati-etal-2016-abstractive}. The same meanings are represented in the same color and the highlighted areas show where the HVO summary performs better. Underline indicates improper phrasing.}
\label{fig:cnn_rader}
%
\vspace{-15pt}  % 负值减小距离，正值增大距离
\end{figure}

With the development of large language models (LLMs), utilizing LLMs for zero-shot generation of summaries has become one of the mainstream approaches~\cite{DBLP:journals/corr/abs-2309-09558}. As shown in Figure~\ref{fig:cnn_rader}, the zero-shot summaries leave room for improvement in all dimensions, especially in consistency. With the demonstrated success of reinforcement learning (RL) in post-training~\cite{li2025efficient,li2024verco}, it has been established as an effective enhancement method in text summarization~\cite{DBLP:journals/csur/ZhangYZ25}. Most studies use RL methods to enhance summarization models by relying on a single reward signal~\cite{DBLP:conf/acl/YadavGAD20,DBLP:journals/nle/FikriOY24,roit2023factually}. However, they do not integrate multi-dimensional metrics as rewards for summarization, and those methods still encounter challenges in generating high-quality summaries while balancing multiple dimensions. To achieve well-balanced summaries, MDO~\cite{DBLP:conf/acl/RyuD0LO24} incorporates multi-dimensional rewards and explores optimal strategies for multi-objective RL. In detail, MDO uses PCGrad optimization to reduce gradient interference across different dimensions, facilitating the discovery of Pareto improvements by balancing multiple objectives. However, this method requires pairwise gradient projections between different dimensions. Due to the high computational cost, it is not feasible to integrate into LLMs.

% At present, there are several methods used for multi-dimensional evaluation of a summary. UniEval, a unified multi-dimensional evaluation tool, shows a significantly stronger correlation with human judgment, outperforming existing metrics~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22}. It performs exceptionally well in the text summarization task, and numerous studies in text summarization have conducted experiments based on this evaluation framework~\cite{DBLP:conf/coling/ZhuL025, DBLP:conf/acl/RyuD0LO24, DBLP:conf/icassp/JungSCR024}.

In this work, we propose hypervolume optimization (HVO), a multi-objective reinforcement learning (MORL) optimization strategy designed for text summarization, which is based on group
relative policy optimization (GRPO)~\cite{shao2024deepseekmathpushinglimitsmathematical} and well-suited for LLMs. HVO incorporates multi-dimensional rewards into a hypervolume~\cite{guerreiro2020hypervolume} computation framework, optimizing the model towards hypervolume maximization, progressively
approaching the Pareto optimal frontier. As illustrated in Figure~\ref{fig:cnn_rader}, our proposed HVO method significantly outperforms LLM-based baseline approaches by achieving higher scores across multiple evaluation dimensions, while maintaining a more balanced and stable optimization. The main contributions of this work are summarized as follows:

\begin{itemize}[noitemsep, topsep=0pt]
\item We introduce HVO, a multi-objective reinforcement learning strategy for text summarization based on
GRPO, which efficiently balances multiple evaluation
dimensions without requiring supervised fine-tuning or
a cold start.

\item On representative datasets, HVO outperforms GRPO with better hypervolume and UniEval scores. The 7B LLM version of HVO performs similarly to GPT-4 on two benchmarks.


\item To address the training instability and summary length collapse issues in vanilla GRPO, a new length constraint mechanism is introduced to enhance the training stability.
\end{itemize}


\section{METHOD}
\label{sec:intro}

In order to preserve the core summarization capabilities of LLMs while enhancing performance across multiple dimensions. We introduce HVO based on the R1-Zero-like~\cite{deepseekai2025deepseekr1incentivizingreasoningcapability} training paradigm, which directly applies GRPO to base LLMs without relying on supervised fine-tuning (SFT) as a preliminary step. Specifically, HVO applies hypervolume evaluation to improve the multi-objective optimization process, where a policy model generates summaries optimized with multi-dimensional rewards calculated via UniEval~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22} , enhanced by a length constraint mechanism. UniEval is a multi-dimensional evaluation tool that strongly correlates with human judgment~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22} and is commonly used in text summarization research~\cite{ DBLP:conf/acl/RyuD0LO24, DBLP:conf/icassp/JungSCR024}. The entire process is illustrated in Figure \ref{fig:structure}.


\subsection{Problem Formulation}

Given a set of documents $\{p_1, p_2, \dots, p_N\}$, LLMs generate the corresponding summaries $\{s_1, s_2, \dots, s_N\}$ using a prompt $\alpha$. Text summarization can be modeled as a Question Answering (QA) problem, where the question is a prompted document $q_i=\textit{f}(p_i;\alpha) $ and the answer is the corresponding summary $s_i$. The dataset $\mathcal{D}$ is formed as $\{(q_i, s_i)\}_{i=1}^N$. For a specific question $q \in \mathcal{D}$
the policy model $\pi_\theta$ generates a group of $G$ individual
summaries $\{o_i\}_{i=1}^G$. For each evaluation dimension $D_k$ in $\{D_1, D_2, \dots, D_M\}$, the reward model $R$ computes the reward $r^{k}_i$ for $o_i$ at the 
$k$-th dimension. Our objective is to optimize the policy model's parameters $\theta$ using multi-objective reinforcement learning to maximize the expected reward guided by hypervolume maximization based on GRPO. 
\subsection{HVO}
\begin{figure}[ht]
  \centering
  \includegraphics[width=0.5\textwidth]{figures/struct2.pdf}
  \caption{The entire process of HVO. In subplot (c), the points on the $f_1$ line represent the same linear weighted sum score for $D_1$ and $D_2$, while the points on the $f_2$ line represent the same hypervolume value for $D_1$ and $D_2$.}
  \label{fig:structure}
\end{figure}


HVO is based on GRPO in which the optimization of policy $\pi_\theta$ is achieved by maximizing the following objective function:
\begin{align}
\mathcal{J}(\theta) &= \mathbb{E}_{(q, a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\text{old}}\nonumber (\cdot | q)} \Bigg[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \Bigg( \\
& \min \left( f_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}( f_{i,t}(\theta), 1-\epsilon, 1+\epsilon ) \hat{A}_{i,t} \right) \\
& - \beta D_{\text{KL}}(\pi_{\sigma} || \pi_{\text{ref}}) \Bigg) \Bigg], \nonumber
\end{align}
where
\[
\begin{aligned}
f_{i,t}(\theta) &= \frac{\pi_{\theta}(o_{i,t} | q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} | q, o_{i,<t})}, \quad
\hat{A}_{i,t} &= \frac{r_i - \text{mean}(\{r_i\}_{i=1}^G)}{\text{std}(\{r_i\}_{i=1}^G)}.
\end{aligned}
\]
The default and direct way to extend GRPO to multi-objective optimization is to compute the reward \( r_i \) through a weighted linear combination:
\begin{align}
r_i &= \sum_{k=1}^{M} w^{k} \cdot r_{i}^{k},
\end{align}
where \( w^k \) represents the weight for the reward $r_{i}^{k}$, which requires manual configuration. It is worth noting that while the weighted linear combination method is simple, it has non-trivial limitations, particularly in handling issues of inter-dependencies between objectives, which can result in imbalanced or incomplete optimization outcomes~\cite{das1997closer}.

The hypervolume method is an evaluation metric in multi-objective optimization that measures the volume of the hypercube occupied by a set of solutions in the objective space. As shown in Figure \ref{fig:structure} (c), taking the two-dimensional case as an example, when the weighted linear combination scores of the samples are similar, samples with more balanced dimensions have higher hypervolume values. Moreover, hypervolume-based evaluation has been shown to be a Pareto-consistent evaluation method~\cite{DBLP:journals/tec/ZitzlerTLFF03}. Therefore, optimizing towards hypervolume maximization ensures continuous improvement in the quality of the solution set, progressively approaching the Pareto optimal frontier. Inspired by this, we integrate hypervolume evaluation into the multi-dimensional rewards of GRPO and use the commonly adopted approach of selecting a slightly worse reference point than the nadir point. The specific method of HVO is as follows:
\begin{align} r_i &= \prod_{k=1}^{M} \left[ \min \left( \epsilon, r_i^k - \min(\{r_i^k\}_{i=1}^{G}) + \delta \right) \right ]^{-w^k},
\label{eq:hypervolume}
\end{align}
where \( \delta, \epsilon \in (0, 1) \),  $\delta$ is a small constant used to avoid zero values and
$\epsilon$ is an upper bound, which ensures that the term \( r^k_{i} - \min_{i=1}^{G}(r^k_{i}) \) is restricted within the range \( [\delta, \epsilon] \), allowing monotonic adjustment of $w^k$ for $D_k$. 
\renewcommand{\arraystretch}{0.80}

\begin{table*}[ht]
\centering
% % 调整列间距，默认是6pt，这里设置为3pt
\setlength{\tabcolsep}{4pt}
% \begin{tabular}{l | c | c c c c c c c | c  c c c c c c }
% \toprule
% \multirow{2}{*}{\textbf{Model}}&\multirow{2}{*}{\textbf{Method}}&\multicolumn{7}{|c|}{CNN/DailyMail}&\multicolumn{7}{c}{BillSum} \\
% \cline{3-16}
%  && \textbf{C} & \textbf{Cons} & \textbf{F} & \textbf{R} & \textbf{HV} & \textbf{O} & \textbf{S} & \textbf{C} & \textbf{Cons} & \textbf{F} & \textbf{R} & \textbf{HV} & \textbf{O} & \textbf{S} \\
% \midrule
% PEGASUS & SFT & 0.936 & \textcolor{red}{\textbf{0.939}} & 0.815 & 0.684 & 0.000 & 0.843 & 0.121 & 0.823 & 0.832 & 0.849 & 0.814 & 0.000 & 0.830 & 0.015 \\
% Qwen2.5 1.5B & Zero-shot & 0.871 & 0.819 & 0.936 & 0.861 & 0.086 & 0.872 & 0.048 & 0.941 & 0.826 & 0.905 & 0.914 & 0.473 & 0.896 & \textcolor{red}{\textbf{0.050}} \\
% Qwen2.5 7B & Zero-shot & 0.890 & 0.820 & 0.932 & 0.874 & 0.136 & 0.879 & 0.046 & 0.958 & \textcolor{blue}{\underline{0.843}} & 0.820 & 0.948 & 0.346 & 0.892 & 0.071 \\
% Qwen2.5 14B & Zero-shot & 0.931 & 0.826 & 0.859 & 0.907 & 0.152 & 0.881 & 0.047 & 0.953 & 0.805 & 0.919 & 0.940 & 0.520 & 0.904 & 0.068 \\
% Qwen2.5 32B & Zero-shot & 0.918 & 0.843 & 0.933 & 0.893 & 0.298 & 0.897 & 0.040 & 0.948 & 0.799 & \textcolor{red}{\textbf{0.941}} & 0.933 & 0.512 & 0.905 & 0.071 \\
% GPT4 & Zero-shot & \textcolor{red}{\textbf{0.967}} & 0.840 & \textcolor{blue}{\underline{0.945}} & \textcolor{blue}{\underline{0.934}} & 0.535 & 0.921 & 0.056 & \textcolor{red}{\textbf{0.973}} & \textcolor{blue}{\underline{0.843}} & 0.831 & \textcolor{red}{\textbf{0.971}} & 0.477 & 0.904 & 0.078 \\
% Qwen2.5 7B & GRPO & 0.908 & 0.903 & 0.922 & \textcolor{red}{\textbf{0.954}} & \textcolor{blue}{\underline{0.544}} & \textcolor{blue}{\underline{0.922}} & \textcolor{blue}{\underline{0.023}} & 0.959 & 0.823 & 0.912 & \textcolor{blue}{\underline{0.955}} & \textcolor{blue}{\underline{0.663}} & \textcolor{blue}{\underline{0.912}} & 0.063 \\
% Qwen2.5 7B & HVO & \textcolor{blue}{\underline{0.961}} & \textcolor{blue}{\underline{0.926}} & \textcolor{red}{\textbf{0.951}} & \textcolor{blue}{\underline{0.934}} & \textcolor{red}{\textbf{1.000}} & \textcolor{red}{\textbf{0.943}} & \textcolor{red}{\textbf{0.016}} & \textcolor{blue}{\underline{0.964}} & \textcolor{red}{\textbf{0.853}} & \textcolor{blue}{\underline{0.939}} & \textcolor{blue}{\underline{0.955}} & \textcolor{red}{\textbf{1.000}} & \textcolor{red}{\textbf{0.928}} & \textcolor{blue}{\underline{0.051}} \\
% \bottomrule
% \end{tabular}
{\normalsize
\begin{tabular}{c |r  r|  c c c c c c c  }
\toprule
 % Dataset&Model&Method& \textbf{C} & \textbf{Cons} & \textbf{F} & \textbf{R} & \textbf{HV} & \textbf{O} & \textbf{S} 
 % \\
\textbf{Dataset}&\textbf{Model} & \textbf{Method} & \textbf{Coherence} & \textbf{Consistency} & \textbf{Fluency} & \textbf{Relevance} & \textbf{HV score} & \textbf{Overall}  & \textbf{STD}\\
 \hline
  \multirow{8}{*}{\rotatebox{90}{\textbf{BillSum}}} &PEGASUS & SFT & 0.823 & 0.832 & 0.849 & 0.814 & 0.171 & 0.830 & \textcolor{red}{\textbf{0.015}} \\
  &Qwen2.5 1.5B & Zero-shot & 0.941 & 0.826 & 0.905 & 0.914 & 1.018 & 0.896 & \textcolor{blue}{\underline{0.050}} \\
  &Qwen2.5 7B & Zero-shot & 0.958 & \textcolor{blue}{\underline{0.843}} & 0.820 & 0.948 & 0.790 & 0.892 & 0.071 \\
  &Qwen2.5 14B & Zero-shot & 0.953 & 0.805 & 0.919 & 0.940 & 1.101 & 0.904 & 0.068 \\
  &Qwen2.5 32B & Zero-shot & 0.948 & 0.799 & \textcolor{red}{\textbf{0.941}} & 0.933 & 1.088 & 0.905 & 0.071 \\
  &GPT4 & Zero-shot & \textcolor{red}{\textbf{0.973}} & \textcolor{blue}{\underline{0.843}} & 0.831 & \textcolor{red}{\textbf{0.971}} & 1.025 & 0.904 & 0.078 \\
  &Qwen2.5 7B & GRPO & 0.959 & 0.823 & 0.912 & \textcolor{blue}{\underline{0.955}} & \textcolor{blue}{\underline{1.358}} & \textcolor{blue}{\underline{0.912}} & 0.063 \\
  &Qwen2.5 7B & HVO & \textcolor{blue}{\underline{0.964}} & \textcolor{red}{\textbf{0.853}} & \textcolor{blue}{\underline{0.939}} & \textcolor{blue}{\underline{0.955}} & \textcolor{red}{\textbf{1.961}} & \textcolor{red}{\textbf{0.928}} & {{0.051}} \\
\midrule
 \multirow{8}{*}{\rotatebox{90}{\textbf{CNN/DailyMail}}}&PEGASUS & SFT & 0.936 & \textcolor{red}{\textbf{0.939}} & 0.815 & 0.684 & 0.364 & 0.843 & 0.121 \\
 &Qwen2.5 1.5B & Zero-shot & 0.871 & 0.819 & 0.936 & 0.861 & 0.612 & 0.872 & 0.048 \\
 &Qwen2.5 7B & Zero-shot & 0.890 & 0.820 & 0.932 & 0.874 & 0.757 & 0.879 & 0.046 \\
 &Qwen2.5 14B & Zero-shot & 0.931 & 0.826 & 0.859 & 0.907 & 0.805 & 0.881 & 0.047 \\
 &Qwen2.5 32B & Zero-shot & 0.918 & 0.843 & 0.933 & 0.893 & 1.226 & 0.897 & 0.040 \\
 &GPT4 & Zero-shot & \textcolor{red}{\textbf{0.967}} & 0.840 & \textcolor{blue}{\underline{0.945}} & \textcolor{blue}{\underline{0.934}} & 1.913 & 0.921 & 0.056 \\
 &Qwen2.5 7B & GRPO & 0.908 & 0.903 & 0.922 & \textcolor{red}{\textbf{0.954}} & \textcolor{blue}{\underline{1.938}} & \textcolor{blue}{\underline{0.922}} & \textcolor{blue}{\underline{0.023}} \\
 &Qwen2.5 7B & HVO & \textcolor{blue}{\underline{0.961}} & \textcolor{blue}{\underline{0.926}} & \textcolor{red}{\textbf{0.951}} & \textcolor{blue}{\underline{0.934}} & \textcolor{red}{\textbf{3.258}} & \textcolor{red}{\textbf{0.943}} & \textcolor{red}{\textbf{0.016}} \\ \bottomrule 

\end{tabular}
}
\caption{The results of multi-dimensional evaluation measured on both the CNN/DailyMail and BillSum datasets. Within the same dimension, the bold denotes the highest score, and the underline denotes the second-highest score. The HV score is expressed in units of $10^{-3}$ calculated according to Equation \ref{eq:hypervolume}.
}
\label{tab:combined_results}
\end{table*}

We use the scores of UniEval as a reward that focuses on coherence, consistency, fluency, and relevance in text summarization. However, recent studies have shown that GRPO encounters issues with training instability~\cite{yu2025dapo}, which leads to issues with summary length collapse during training when using UniEval as the sole reward. Consequently, we propose a new length constraint method to help maintain training stability. The calculation is as follows:
\begin{align}
R_{\text{conciseness}}\left( o_i\right) &= \frac{1}{1 + \left( x_i/\rho \right)^{\lambda}},
\end{align}
where $
x_i = \left| \left|p_i\right|/\left|o_i\right| - \text{mean}\left( \left\{ \left|p_j\right|/\left|s_j\right| \right\} _{j=1}^{V} \right) \right|
$, $V$ is the size of the training set.
It represents the absolute difference between the compression ratio (CR) of the generated summary $o_i$ and the average compression ratio of human-generated summaries in the training set.
The $\lambda$ represents the steepness, indicating the degree of rapid decrease, while $\rho$ represents the offset, indicating how far from $x_i = 0$ the value starts to decrease sharply. This ensures that the score decreases slowly around $x_i = 0$, maintaining sufficient exploration space, and declines rapidly when $x_i$ is large.

\section{Experiment}
\subsection{Experiment Setup}
\subsubsection{Dataset and Baselines}
We use two text summarization datasets: CNN/DailyMail~\cite{nallapati-etal-2016-abstractive} for news summarization, with 287K training and 11.5K test samples, and BillSum~\cite{kornilova-eidelman-2019-billsum} for legislative content, with 18.9K training and 3.2K test samples.

We used the instruct version of the Qwen 2.5 as the baseline, as it is known for its strong performance in various benchmarks~\cite{qwen2025qwen25technicalreport}. In the experiment, the 7B model was selected for comparing GRPO and HVO, balancing performance and resource usage. In addition, we used GPT-4-turbo for GPT-4 and employed the already fine-tuned versions of PEGASUS on the BillSum\footnote{\href{https://huggingface.co/google/pegasus-billsum}{https://huggingface.co/google/pegasus-billsum}} and CNN/DailyMail\footnote{\href{https://huggingface.co/google/pegasus-cnn_dailymail}{https://huggingface.co/google/pegasus-cnn\_dailymail}} as baselines.

\subsubsection{Setup and Metrics}
For training the GRPO, the parameters are set as follows: train\_batch\_size = 64, num\_generations = 8, max\_grad\_norm = 0.4, and learning\_rate = 5e-7. All rewards are weighted equally with a weight of 1.0. Additionally, for HVO, we default to use $\epsilon$ = 0.99, $\delta$ = 0.1, $\rho$ = 16 and $\lambda$ = 2, $w^k$ is defaulted to -1 to align with GRPO. We primarily use UniEval (coherence, consistency, fluency, relevance, and the average of them) and hypervolume (HV) scores according to Equation~\ref{eq:hypervolume} \, as the evaluation metric.
\begin{figure*}
  \centering
  \includegraphics[width=\textwidth]{figures/eval.pdf}
  \vspace{-20pt}  % 负值减小距离，正值增大距离
  \caption{The inter-group means and standard deviations on the 500-validation set during the training process on CNN/DailyMail, with HVO method recording scores prior to hypervolume calculation and being comparable to GRPO.}
  \label{fig:plot}
\end{figure*}
\subsection{Results}
\begin{figure}[ht]
  \centering
  \includegraphics[width=0.4\textwidth]{figures/plot_output2.pdf}
  \caption{Scatter plot illustrating the relationship between the overall score and completion length, as evaluated by UniEval, for different models on two datasets: CNN/DailyMail and BillSum.}
  \label{fig:scattor}
  \vspace{-20pt}  % 负值减小距离，正值增大距离
\end{figure}

Based on the experimental results from Table~\ref{tab:combined_results}, HVO outperforms all other methods in both datasets, demonstrating the highest hypervolume (HV) scores and overall scores. 

GPT-4, while excelling in coherence (0.967) and fluency (0.945) on the CNN/DailyMail dataset and coherence (0.973) and relevance (0.971) on the BillSum dataset, does not perform as well in overall performance and dimension balance compared to the Qwen 2.5 7B (HVO), which shows that the HVO method can achieve results comparable to GPT-4 in the summarization task. 

Though Qwen2.5 7B enhanced by both GRPO and HVO, shows balanced performance, HVO achieves much better overall scores. From Figure \ref{fig:plot}, in the early stages of training, the GRPO algorithm prioritizes fluency and relevance, with less emphasis on consistency. As a result, the optimization of consistency is constrained. In contrast, HVO optimizes all objectives more evenly and it is worth noting that the standard deviation of HVO is large and persists over a longer period of training, which makes the advantage signal more significant. This provides the opportunity to explore a larger strategy space, increasing the probability of approaching Pareto optimality, leading to more comprehensive and stable performance across all metrics.
% From the BillSum results, we observe that though the UE scores of Qwen 2.5 14B, Qwen2.5 32B, and GPT-4 are close, with no obvious Pareto dominance between the individual objectives, the HV score better accounts for the balance among all the objectives.
The HV score serves as a proxy for how closely results approximate the pareto front. As shown in Table \ref{tab:combined_results}, HVO achieves a superior HV score. This indicates that our approach enables the policy model to approximate the pareto front more closely compared to existing baselines. 

Finally, we computed the generation length of different models and plotted a scatter plot of the overall score against completion length, as shown in Figure~\ref{fig:scattor}. Our method not only achieves the highest overall score but also maintains a shorter completion length, ensuring better conciseness.

\section{Conclusion}
In this paper, we introduce hypervolume optimization enhanced GRPO (HVO), a multi-objective reinforcement learning framework for text summarization that directly optimizes the hypervolume indicator in high-dimensional objective space. By balancing multiple evaluation metrics, HVO achieves a more stable and efficient trajectory toward the pareto frontier. Experiments on CNN/DailyMail and BillSum show that HVO attains state-of-the-art hypervolume and overall scores, outperforming existing methods and rivaling GPT-4, without supervised fine-tuning or cold-start initialization. These results confirm HVO’s effectiveness in managing complex trade-offs and generating high-quality summaries, offering a robust solution for multi-objective text summarization.
% Future work will explore its extension to other NLP tasks and more intricate reward structures.


% In this paper, we proposed a new method called Hypervolume Optimization (HVO) for multi-objective reinforcement learning in text summarization. Unlike traditional methods, HVO integrates multiple evaluation metrics to balance different objectives, improving the stability of the optimization process.

% Our experiments show that HVO outperforms existing methods on benchmark datasets like CNN/DailyMail and BillSum, achieving the highest hypervolume (HV) scores and overall score. This demonstrates HVO’s ability to balance trade-offs between different objectives. Additionally, HVO performs similarly to or better than GPT-4 without the need for supervised fine-tuning or a cold start, highlighting its effectiveness in generating high-quality summaries.

% Overall, HVO offers an effective solution for multi-objective text summarization, improving both summary quality and optimization stability. Future work could explore applying this method to other NLP tasks and further improving it for more complex reward structures.





\label{ssec:subhead}
\label{sec:intro}
% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEbib}
\bibliography{strings,refs}

\end{document}

% Template for ICASSP-2026 paper; to be used with:
%          spconf.sty  - ICASSP/ICIP LaTeX style file, and
%          IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
% \documentclass{article}
\usepackage{spconf,amsmath,graphicx,hyperref}

% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

% Title.
% ------
\title{AUTHOR GUIDELINES FOR ICASSP 2026 PROCEEDINGS MANUSCRIPTS}
%
% Single address.
% ---------------
\name{Author(s) Name(s)\sthanks{Thanks to XYZ agency for funding.}}
\address{Author Affiliation(s)}
%
% For example:
% ------------
%\address{School\\
%	Department\\
%	Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
%\twoauthors
%  {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
%	{School A-B\\
%	Department A-B\\
%	Address A-B}
%  {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
%	while at ...}}
%	{School C-D\\
%	Department C-D\\
%	Address C-D}
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
The abstract should appear at the top of the left-hand column of text, about
0.5 inch (12 mm) below the title area and no more than 3.125 inches (80 mm) in
length.  Leave a 0.5 inch (12 mm) space between the end of the abstract and the
beginning of the main text.  The abstract should contain about 100 to 150
words, and should be identical to the abstract text submitted electronically
along with the paper cover sheet.  All manuscripts must be in English, printed
in black ink.
\end{abstract}
%
\begin{keywords}
One, two, three, four, five
\end{keywords}
%
\section{Introduction}
\label{sec:intro}

These guidelines include complete descriptions of the fonts, spacing, and
related information for producing your proceedings manuscripts. Please follow
them and if you have any questions, direct them to Conference Management
Services, Inc.: Phone +1-979-846-6800 or email
to \\\texttt{papers@2026.ieeeicassp.org}.

\section{Formatting your paper}
\label{sec:format}

All printed material, including text, illustrations, and charts, must be kept
within a print area of 7 inches (178 mm) wide by 9 inches (229 mm) high. Do
not write or print anything outside the print area. The top margin must be 1
inch (25 mm), except for the title page, and the left margin must be 0.75 inch
(19 mm).  All {\it text} must be in a two-column format. Columns are to be 3.39
inches (86 mm) wide, with a 0.24 inch (6 mm) space between them. Text must be
fully justified.

\section{PAGE TITLE SECTION}
\label{sec:pagestyle}

The paper title (on the first page) should begin 1.38 inches (35 mm) from the
top edge of the page, centered, completely capitalized, and in Times 14-point,
boldface type.  The authors' name(s) and affiliation(s) appear below the title
in capital and lower case letters.  Papers with multiple authors and
affiliations may require two or more lines for this information. Please note
that papers should not be submitted blind; include the authors' names on the
PDF.

\section{TYPE-STYLE AND FONTS}
\label{sec:typestyle}

To achieve the best rendering both in printed proceedings and electronic proceedings, we
strongly encourage you to use Times-Roman font.  In addition, this will give
the proceedings a more uniform look.  Use a font that is no smaller than nine
point type throughout the paper, including figure captions.

In nine point type font, capital letters are 2 mm high.  {\bf If you use the
smallest point size, there should be no more than 3.2 lines/cm (8 lines/inch)
vertically.}  This is a minimum spacing; 2.75 lines/cm (7 lines/inch) will make
the paper much more readable.  Larger type sizes require correspondingly larger
vertical spacing.  Please do not double-space your paper.  TrueType or
Postscript Type 1 fonts are preferred.

The first paragraph in each section should not be indented, but all the
following paragraphs within the section should be indented as these paragraphs
demonstrate.

\section{MAJOR HEADINGS}
\label{sec:majhead}

Major headings, for example, "1. Introduction", should appear in all capital
letters, bold face if possible, centered in the column, with one blank line
before, and one blank line after. Use a period (".") after the heading number,
not a colon.

\subsection{Subheadings}
\label{ssec:subhead}

Subheadings should appear in lower case (initial word capitalized) in
boldface.  They should start at the left margin on a separate line.
 
\subsubsection{Sub-subheadings}
\label{sssec:subsubhead}

Sub-subheadings, as in this paragraph, are discouraged. However, if you
must use them, they should appear in lower case (initial word
capitalized) and start at the left margin on a separate line, with paragraph
text beginning on the following line.  They should be in italics.

\section{PRINTING YOUR PAPER}
\label{sec:print}

Print your properly formatted text on high-quality, 8.5 x 11-inch white printer
paper. A4 paper is also acceptable, but please leave the extra 0.5 inch (12 mm)
empty at the BOTTOM of the page and follow the top and left margins as
specified.  If the last page of your paper is only partially filled, arrange
the columns so that they are evenly balanced if possible, rather than having
one long column.

In LaTeX, to start a new column (but not a new page) and help balance the
last-page column lengths, you can use the command ``$\backslash$pagebreak'' as
demonstrated on this page (see the LaTeX source below).

\section{PAGE NUMBERING}
\label{sec:page}

Please do {\bf not} paginate your paper.  Page numbers, session numbers, and
conference identification will be inserted when the paper is included in the
proceedings.

\section{ILLUSTRATIONS, GRAPHS, AND PHOTOGRAPHS}
\label{sec:illust}

Illustrations must appear within the designated margins.  They may span the two
columns.  If possible, position illustrations at the top of columns, rather
than in the middle or at the bottom.  Caption and number every illustration.
All halftone illustrations must be clear black and white prints.  Colors may be
used, but they should be selected so as to be readable when printed on a
black-only printer.

Since there are many ways, often incompatible, of including images (e.g., with
experimental results) in a LaTeX document, below is an example of how to do
this \cite{Lamp86}.

\section{FOOTNOTES}
\label{sec:foot}

Use footnotes sparingly (or not at all!) and place them at the bottom of the
column on the page on which they are referenced. Use Times 9-point type,
single-spaced. To help your readers, avoid using footnotes altogether and
include necessary peripheral observations in the text (within parentheses, if
you prefer, as in this sentence).

% Below is an example of how to insert images. Delete the ``\vspace'' line,
% uncomment the preceding line ``\centerline...'' and replace ``imageX.ps''
% with a suitable PostScript file name.
% -------------------------------------------------------------------------
\begin{figure}[htb]

\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[width=8.5cm]{image1}}
%  \vspace{2.0cm}
  \centerline{(a) Result 1}\medskip
\end{minipage}
%
\begin{minipage}[b]{.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{image3}}
%  \vspace{1.5cm}
  \centerline{(b) Results 3}\medskip
\end{minipage}
\hfill
\begin{minipage}[b]{0.48\linewidth}
  \centering
  \centerline{\includegraphics[width=4.0cm]{image4}}
%  \vspace{1.5cm}
  \centerline{(c) Result 4}\medskip
\end{minipage}
%
\caption{Example of placing a figure with experimental results.}
\label{fig:res}
%
\end{figure}


% To start a new column (but not a new page) and help balance the last-page
% column length use \vfill\pagebreak.
% -------------------------------------------------------------------------
%\vfill
%\pagebreak

\section{COPYRIGHT FORMS}
\label{sec:copyright}

You must submit your fully completed, signed IEEE electronic copyright release
form when you submit your paper. We {\bf must} have this form before your paper
can be published in the proceedings.

\section{RELATION TO PRIOR WORK}
\label{sec:prior}

The text of the paper should contain discussions on how the paper's
contributions are related to prior work in the field. It is important
to put new work in  context, to give credit to foundational work, and
to provide details associated with the previous work that have appeared
in the literature. This discussion may be a separate, numbered section
or it may appear elsewhere in the body of the manuscript, but it must
be present.

You should differentiate what is new and how your work expands on
or takes a different path from the prior studies. An example might
read something to the effect: "The work presented here has focused
on the formulation of the ABC algorithm, which takes advantage of
non-uniform time-frequency domain analysis of data. The work by
Smith and Cohen \cite{Lamp86} considers only fixed time-domain analysis and
the work by Jones et al \cite{C2} takes a different approach based on
fixed frequency partitioning. While the present study is related
to recent approaches in time-frequency analysis [3-5], it capitalizes
on a new feature space, which was not considered in these earlier
studies."

\vfill\pagebreak

\section{REFERENCES}
\label{sec:refs}

List and number all bibliographical references at the end of the
paper. The references can be numbered in alphabetic order or in
order of appearance in the document. When referring to them in
the text, type the corresponding reference number in square
brackets as shown at the end of this sentence \cite{C2}. An
additional final page (the fifth page, in most cases) is
allowed, but must contain only references to the prior
literature.

Please follow the IEEE Citation Guidelines, \url{https://ieee-dataport.org/sites/default/files/analysis/27/IEEE\%20Citation\%20Guidelines.pdf} for formatting of references.

% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEbib}
\bibliography{strings,refs}

\end{document}

% Template for ICASSP-2026 paper; to be used with:
%          spconf.sty  - ICASSP/ICIP LaTeX style file, and
%          IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
% \documentclass{article}
\usepackage{spconf,amsmath,graphicx,hyperref}
\usepackage{amsmath}  % For mathematical symbols and equations
\usepackage{amssymb}  % For additional symbols (like \mathbb)
\usepackage{graphicx}  % 在导言区导入 graphicx 包
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{amsmath}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{enumitem}

% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

% Title.
% ------
\title{Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization}
%
% Single address.
% ---------------
\name{Author(s) Name(s)\thanks{Thanks to XYZ agency for funding.}}
\address{Author Affiliation(s)}
%
% For example:
% ------------
%\address{School\\
%	Department\\
%	Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
%\twoauthors
%  {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
%	{School A-B\\
%	Department A-B\\
%	Address A-B}
%  {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
%	while at ...}}
%	{School C-D\\
%	Department C-D\\
%	Address C-D}
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
Reinforcement Learning (RL) methods have been proven effective in improving the performance of Large Language Models (LLMs), particularly methods such as Group Relative Policy Optimization (GRPO) that demonstrate outstanding capabilities in many tasks. However, despite the potential of RL in text summarization, few studies have focused on leveraging it to tackle the challenges of optimizing objectives that involve potential conflicts. In this paper, we introduce a strategy called Hypervolume Optimization (HVO), the first multi-objective optimization approach based on GRPO in the task of text summarization without supervised fine-tuning or cold-start. In addition, our method achieves superior performance across multiple benchmark datasets, demonstrating the best results while ensuring a more balanced distribution of scores across dimensions, effectively handling the trade-offs between objectives and enhancing overall optimization stability.

\end{abstract}
%
\begin{keywords}
Multi-objective Reinforcement Learning,
Text Summarization,
Hypervolume Contributions
\end{keywords}
%
\section{Introduction}
\label{sec:intro}
Text summarization is one of the most critical and challenging tasks in Natural Language Processing (NLP)~\cite{DBLP:journals/csur/ZhangYZ25}.To comprehensively evaluate the quality of generated summaries, researchers typically examine multiple dimensions, such as coherence, consistency, fluency, and relevance~\cite{DBLP:conf/emnlp/LiuIXWXZ23}. However, optimizing these dimensions simultaneously is inherently challenging, as improvements along one dimension may lead to compromises in others~\cite{abo2023automatic}, resulting in imbalanced summaries.
\begin{figure}[htb]
\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[width=1\textwidth]{figures/cnn_radar_chart (18).pdf}}
%  \vspace{2.0cm}
\end{minipage}
%
\caption{The radar chart presents the UniEval scores of the summary produced by the baseline model across four dimensions.}
\label{fig:cnn_rader}
%
\end{figure}

With the development of LLMs, LLM-based methods for summarization now demonstrate strong performance~\cite{DBLP:journals/corr/abs-2309-09558}. Building on this progress, reinforcement learning (RL) has emerged as a complementary paradigm, enabling further improvements in model performance across various tasks, including summarization~\cite{DBLP:journals/csur/ZhangYZ25}. Recently, most studies use RL methods to enhance models for summarization by relying on a single reward signal. Roit et al.~\cite{DBLP:conf/acl/RoitFSACDGGHKMG23} use RL with textual-entailment rewards to optimize for factual consistency; Fikri et al.~\cite{DBLP:journals/nle/FikriOY24} developed an abstractive summarization method using RL with semantic similarity rewards; QTR~\cite{DBLP:conf/acl/YadavGAD20} computes the F-Score of question types between the generated summary and the gold summary as a reward by enhancing a BERT model. However, they do not consider integrating multi-dimensional metrics as rewards for summarization, making it challenging to achieve high-quality summary generation by balancing multiple dimensions; moreover, The integration of multi-reward mechanisms into reinforcement learning-based text summarization has received limited attention in the existing literature.~\cite{DBLP:conf/acl/RyuD0LO24}.

Currently, there are several methods used for multi-dimensional evaluation of a summary. UniEval, a unified multi-dimensional evaluation tool, shows a significantly stronger correlation with human judgment, outperforming existing metrics~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22}. It performs exceptionally well in the text summarization task, and numerous studies in text summarization have conducted experiments based on this evaluation framework~\cite{DBLP:conf/coling/ZhuL025, DBLP:conf/acl/RyuD0LO24, DBLP:conf/icassp/JungSCR024}.

To incorporate multi-dimensions into the reward for generating balanced and high-quality summaries and perform multi-objective optimization, the latest multi-objective optimization method in the field of summarization, MDO~\cite{DBLP:conf/acl/RyuD0LO24} uses PCGrad optimization to reduce gradient interference across dimensions, facilitating the discovery of Pareto improvements by balancing multiple objectives. This requires pairwise gradient projections between different dimensions, which becomes computationally complex with the large-scale parameters of LLMs.

Therefore, we propose an optimization strategy, HVO for multi-objective reinforcement learning. As illustrated in Figure~\ref{fig:cnn_rader}, our proposed HVO method significantly outperforms LLM-based baseline approaches by achieving higher scores across multiple evaluation dimensions, while maintaining a more balanced and stable optimization.

\begin{itemize}[noitemsep, topsep=0pt]
\item We introduce HVO, the first multi-objective reinforcement learning strategy based on Hypervolume and GRPO, which efficiently balances multiple evaluation dimensions without requiring supervised fine-tuning or a cold start.

\item Our method outperforms GPT-4o-level results on two benchmark datasets using the 7B parameter model on 2 benchmark datasets, demonstrating the generalization and effectiveness of HVO across different domains.

\item To address the training instability and summary length collapse issues in traditional GRPO, a new length constraint mechanism is introduced, enhancing training stability.
\end{itemize}


\section{METHOD}
\label{sec:intro}
In text summarization, generating high-quality summaries necessitates a careful balance across multiple evaluative dimensions. To achieve a balance among these competing objectives, we introduce HVO, improving the multi-objective optimization method based on GRPO, where a policy model generates summaries optimized with multi-dimensional rewards computed via the UniEval. The entire process is illustrated in Figure 2.


\subsection{Problem Formulation}

Given a set of documents $\{P_1, P_2, \dots, P_N\}$ and their corresponding human expert summaries $\{S_1, S_2, \dots, S_N\}$, using the provided summary prompt, a corresponding question set $\{Q_1, Q_2, \dots, Q_N\}$ is obtained. The dataset $\mathcal{D}$ is formed as $\{(Q_i, S_i)\}_{i=1}^N$. For a specific question-answer pair $(q, s)$,
the policy model $\pi_\theta$ generates a group of $G$ individual
summaries $\{o_i\}_{i=1}^G$. For each evaluation dimension $D_k$ in $\{D_1, D_2, \dots, D_M\}$, the reward model $R$ computes the reward $r^{k}_i$ for the generated summary $o_i$. Our objective is to optimize the policy model's parameters $\theta$ using reinforcement learning to maximize the expected reward. 

\subsection{HVO}
\begin{figure}[ht]
  \centering
  \includegraphics[width=0.48\textwidth]{figures/structure.pdf}
  \caption{The entire process of HVO. $D_i$ denotes the evaluation dimension of the summary, which corresponds to four commonly used criteria in the literature: coherence, consistency, fluency, and relevance.}
  \label{fig:cnn_rader}
\end{figure}


The HVO method is based on current state-of-the-art RL algorithms, exemplified by GRPO~\cite{zheng2025group}. In the GRPO method, the optimization of the policy $\pi_\theta$ is achieved by maximizing the following objective function:
\[
\begin{aligned}
\mathcal{J}(\theta) &= \mathbb{E}_{(q, a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\text{old}}(\cdot | q)} \Bigg[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \Bigg( \\
& \min \left( f_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}( f_{i,t}(\theta), 1-\epsilon, 1+\epsilon ) \hat{A}_{i,t} \right) \\
& - \beta D_{\text{KL}}(\pi_{\sigma} || \pi_{\text{ref}}) \Bigg) \Bigg],
\end{aligned}
\]
where
\[
\begin{aligned}
f_{i,t}(\theta) &= \frac{\pi_{\theta}(o_{i,t} | q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} | q, o_{i,<t})}, \quad
\hat{A}_{i,t} &= \frac{r_i - \text{mean}(\{r_i\}_{i=1}^G)}{\text{std}(\{r_i\}_{i=1}^G)}.
\end{aligned}
\]
In GRPO, The reward \( r_i \) is computed as a weighted linear combination of rewards from different dimensions:
\[
r_i = \sum_{k=1}^{M} w^{k} \cdot r_{i}^{k},
\]
where \( w^k \) represents the weight for the reward corresponding to the \( k \)-th dimension $D_k$. 

It is worth noting that while the weighted linear combination method is simple and commonly used, it has limitations, particularly in handling interdependencies between objectives, which can result in unbalanced or incomplete optimization outcomes~\cite{das1997closer}. To address this, we optimize the problem based on the Hypervolume method.

The Hypervolume method~\cite{DBLP:journals/tec/ZitzlerT99} is an evaluation metric in multi-objective optimization used to measure the volume of the hypercube occupied by a set of solutions in the objective space. By calculating the covered area of the solution set relative to a reference point, Hypervolume considers both the quality and diversity of the solutions. A larger covered area indicates better performance of the solution set. It is commonly used to compare the performance of different algorithms or solution sets, especially in the evaluation of Pareto fronts. Moreover, hypervolume-based evaluation has been shown to be a Pareto-consistent evaluation method~\cite{DBLP:journals/tec/ZitzlerTLFF03}. Therefore, optimizing towards hypervolume maximization ensures continuous improvement in the quality of the solution set, progressively approaching the Pareto optimal front.

Inspired by the hypervolume method, we integrate hypervolume evaluation into the multi-dimensional rewards of GRPO and propose the HVO strategy, optimizing the model towards hypervolume maximization, progressively approaching the Pareto optimal front, thereby achieving more balanced and effective summarization. The specific method of HVO is as follows:

\[
r_i = \prod_{k=1}^{M} \left( \min \left( \epsilon, \left( r_i^k - \min_{i=1}^{G} (r_i^k) + \delta \right) \right) \right)^{-w^k},
\]
where $ \min_{i=1}^{G}(r_i^k)$ is the minimum value of the $ k-th$ dimension across all \( G \) samples.
\( \delta \in (0, 1) \) is a small constant used to avoid zero values.

To ensure that \( w_d \) maintains the monotonic effect on the entire hypercube edge length, the term \( r^d_{i} - \min_{i=1}^{G}(r^d_{i}) \) is constrained within the range \( [\delta, \epsilon] \).



\begin{table*}[ht]
\centering
\begin{tabular}{l | c | c c c c c c c }
\toprule
\textbf{Model} & \textbf{Method} & \textbf{Coherence} & \textbf{Consistency} & \textbf{Fluency} & \textbf{Relevance} & \textbf{HV score} & \textbf{Overall}  & \textbf{STD} \\
\midrule
PEGASUS & SFT & 0.936  & \textcolor{red}{\textbf{0.939}}  & 0.815  & 0.684  & 0.000 & 0.843 & 0.121  \\
Qwen2.5 1.5B & Zero-shot & 0.871 & 0.819 & 0.936 & 0.861 & 0.086 & 0.872 & 0.048  \\
Qwen2.5 7B   & Zero-shot & 0.890 & 0.820 & 0.932 & 0.874 & 0.136 & 0.879 & 0.046 \\
Qwen2.5 14B  & Zero-shot       & 0.931 & 0.826 & 0.859 & 0.907 & 0.152 & 0.881 & 0.047  \\
Qwen2.5 32B  & Zero-shot       & 0.918 & 0.843 & 0.933 & 0.893 & 0.298 & 0.897 & 0.040  \\
GPT4         & Zero-shot & \textcolor{red}{\textbf{0.967}} & 0.840 & \textcolor{blue}{\underline{0.945}} & \textcolor{blue}{\underline{0.934}} & 0.535 & 0.921 & 0.056  \\
Qwen2.5 7B   & GRPO   & 0.908 & 0.903 & 0.922 & \textcolor{red}{\textbf{0.954}} & \textcolor{blue}{\underline{0.544}} & \textcolor{blue}{\underline{0.922}} & \textcolor{blue}{\underline{0.023}}  \\
Qwen2.5 7B   & HVO (Ours)  & \textcolor{blue}{\underline{0.961}} & \textcolor{blue}{\underline{0.926}} & \textcolor{red}{\textbf{0.951}} & \textcolor{blue}{\underline{0.934}} & \textcolor{red}{\textbf{1.000}} & \textcolor{red}{\textbf{0.943}} & \textcolor{red}{\textbf{0.016}} \\
\toprule
\end{tabular}
\caption{The results of multi-dimension evaluation measured on the CNN/DailyMail dataset. Within the same dimension, the bold denotes the highest score, and the underline denotes the second-highest score. The HV score is a normalized score that makes the comparison clearer.}
\label{tab:cnn_daily_mail_results}
\end{table*}

\begin{table*}[ht]
\centering
\begin{tabular}{l | c | c c c c c c c }
\toprule
\textbf{Model} & \textbf{Method} & \textbf{Coherence} & \textbf{Consistency} & \textbf{Fluency} & \textbf{Relevance} & \textbf{HV score} & \textbf{Overall}  & \textbf{STD} \\
\midrule
PEGASUS & SFT & 0.823 & 0.832 & 0.849 & 0.814 & 0.000 & 0.830 & 0.015 \\
Qwen2.5 1.5B & Zero-shot & 0.941 & 0.826 & 0.905 & 0.914 & 0.473 & 0.896 & \textcolor{red}{\textbf{0.050}}  \\
Qwen2.5 7B   & Zero-shot & 0.958 & \textcolor{blue}{\underline{0.843}} & 0.820 & 0.948 & 0.346  & 0.892 & 0.071 \\
Qwen2.5 14B  & Zero-shot & 0.953 & 0.805 & 0.919 & 0.940 & 0.520 & 0.904 & 0.068  \\
Qwen2.5 32B  & Zero-shot & 0.948 & 0.799 & \textcolor{red}{\textbf{0.941}} & 0.933 & 0.512 & 0.905 & 0.071 \\
GPT4         & Zero-shot & \textcolor{red}{\textbf{0.973}} & \textcolor{blue}{\underline{0.843}} & 0.831 & \textcolor{red}{\textbf{0.971}} & 0.477 & 0.904 & 0.078 \\
Qwen2.5 7B   & GRPO & 0.959 & 0.823 & 0.912 & \textcolor{blue}{\underline{0.955}} & \textcolor{blue}{\underline{0.663}} & \textcolor{blue}{\underline{0.912}} & 0.063 \\
Qwen2.5 7B   &  HVO (Ours) & \textcolor{blue}{\underline{0.964}} & \textcolor{red}{\textbf{0.853}} & \textcolor{blue}{\underline{0.939}} & \textcolor{blue}{\underline{0.955}} & \textcolor{red}{\textbf{1.000}} & \textcolor{red}{\textbf{0.928}} & \textcolor{blue}{\underline{0.051}} \\
\toprule
\end{tabular}
\caption{The results of multi-dimension evaluation measured on the BillSum dataset. Within the same dimension, the bold denotes the highest score, and the underline denotes the second-highest score. The HV score is a normalized score that makes the comparison clearer.}
\label{tab:Unieval_mail_results}
\end{table*}

\subsection{Multi-dimensional rewards}
\label{ssec:subhead}
UniEval redefines multi-dimensional evaluation in summarization as a Boolean Question-Answering (QA) task. It calculates scores based on the model's "Yes/No" answers to specific questions for each dimension. Using a unified QA format, UniEval combines multi-task learning, integrates external knowledge, and employs continual learning to reduce negative transfer. This enables multi-dimensional evaluation with a single model, focusing on coherence, consistency, fluency, and relevance in text summarization.

Recent studies have shown that GRPO encounters issues with training instability~\cite{yu2025dapo,zheng2025group}. This leads to issues with summary length collapse during training when using UniEval as the sole reward. To address this, we propose a new length constraint method to help maintain training stability.

\[
R_{\text{conciseness}}\left( o_i\right) = \frac{1}{1 + \left( \frac{x_i}{\rho} \right)^{\lambda}},
\]
where
\[
x_i = \left| \frac{\left|P_i\right|}{\left|o_i\right|} - \text{mean} \left( \left\{ \frac{\left|P_j\right|}{\left|A_j\right|} \right\}_{j=1}^{V} \right) \right|,
\]
It represents the absolute difference between the compression ratio of the generated summary $o_i$ and the average compression ratio of human-generated summaries in the training set.
This ensures that the score decreases slowly around $x_i = 0$, maintaining sufficient exploration space, and declines rapidly when $x_i$ is large.
The $\lambda$ represents the steepness, indicating the degree of rapid decrease, while $\rho$ represents the offset, indicating how far the value starts to decrease sharply when $x_i = 0$.

\section{Experiment}
\subsection{HVO}
\begin{figure}[ht]
  \centering
  \includegraphics[width=0.48\textwidth]{figures/plot_output.pdf}
  \caption{ .}
  \label{fig:cnn_rader}
\end{figure}
\subsection{Dataset}

To address the potential impact of document complexity, we utilize two text summarization datasets: the CNN/Daily Mail dataset for news summarization and the BillSum dataset for legislative content. The CNN/DM dataset~\cite{nallapati-etal-2016-abstractive} comprises a training set of 287K and a test set of 11.5K, while BillSum~\cite{kornilova-eidelman-2019-billsum} has a training set of 18.9K and a test set of 3.2K.

\subsection{Baseline models}
We use the Qwen 2.5 series model as our baseline model. Qwen 2.5 has demonstrated top-tier performance on a wide range of benchmarks evaluating language understanding, reasoning, mathematics, coding, human preference
alignment, etc~\cite{qwen2025qwen25technicalreport}. In the experiment, we used the instruct version exclusively, which enhanced the accuracy and relevance of task execution, ensuring more precise and efficient results. We used the 7B model for comparing Naive GRPO and GRPO + HVO due to its good balance between performance and resource usage. In addition, we used GPT-4-turbo for GPT-4 and employed the already fine-tuned versions of PEGASUS on the BillSum and CNN/DM datasets as baselines.


\subsection{Setup and Metrics}
For training the Naive GRPO, we randomly select only 10K samples from the training set of each dataset. The training parameters are set as follows: train\_batch\_size = 64, num\_generations = 8, max\_grad\_norm = 0.4, and learning\_rate = 5e-7. All rewards are weighted equally with a weight of 1.0 . Additionally, for HVO, we default to use $\epsilon$ = 0.99 and $\delta$ = 0.1, We primarily use Unieval as the evaluation metric.

\subsection{Results}
Based on the experimental results from Table~\ref{tab:cnn_daily_mail_results} and Table~\ref{tab:Unieval_mail_results}, HVO Method outperforms all other methods in both datasets (CNN/DailyMail and BillSum), demonstrating the highest HpyerVolume (HV) scores and overall scores. It also achieves the first and second lowest standard deviations on the CNN/DailyMail and BillSum datasets, indicating superior performance in balancing multiple evaluation dimensions.

GPT-4, while excelling in coherence (0.967) and fluency (0.945) on the CNN/DailyMail dataset and coherence (0.973) and relevance (0.971) on the BillSum dataset, does not perform as well in overall performance and dimension balance compared to the Qwen 2.5 7B HVO method. This also shows that the HVO method, without needing SFT cold-start, can achieve results comparable to or better than GPT-4 in the summarization task.

Qwen2.5 7B under both GRPO and HVO methods shows balanced performance across coherence, consistency, fluency, and relevance, with HVO achieving the best results. This highlights the limitations of using weighted linear combination method for multi-objective optimization and demonstrates the effectiveness of our approach. To analyze this, from the BillSum results, we observe that when the overall scores of Qwen 2.5 14B, Qwen2.5 32B, and GPT-4 are close, with no obvious Pareto dominance between the individual objectives, the HV score better accounts for the balance among all the objectives.

These results indicate that the HVO method has a significant advantage in multi-objective optimization tasks, effectively improving summary generation quality and achieving balanced optimization across dimensions.

\section{Conclusion}

In this paper, we proposed a new method called Hypervolume Optimization (HVO) for multi-objective reinforcement learning in text summarization. Unlike traditional methods, HVO integrates multiple evaluation metrics to balance different objectives, improving the stability of the optimization process.

Our experiments show that HVO outperforms existing methods on benchmark datasets like CNN/DailyMail and BillSum, achieving the highest hypervolume (HV) scores and overall score. This demonstrates HVO’s ability to balance trade-offs between different objectives. Additionally, HVO performs similarly to or better than GPT-4 without the need for supervised fine-tuning or a cold start, highlighting its effectiveness in generating high-quality summaries.

Overall, HVO offers an effective solution for multi-objective text summarization, improving both summary quality and optimization stability. Future work could explore applying this method to other NLP tasks and further improving it for more complex reward structures.





\label{ssec:subhead}
\label{sec:intro}
% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEbib}
\bibliography{strings,refs}

\end{document}

% Template for ICASSP-2026 paper; to be used with:
%          spconf.sty  - ICASSP/ICIP LaTeX style file, and
%          IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
% \documentclass{article}
\usepackage{spconf,amsmath,graphicx,hyperref}
\usepackage{amsmath}  % For mathematical symbols and equations
\usepackage{amssymb}  % For additional symbols (like \mathbb)
\usepackage{graphicx}  % 在导言区导入 graphicx 包
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{amsmath}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{multirow}

\usepackage{enumitem}

% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

% Title.
% ------
\title{Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization}
%
% Single address.
% ---------------
\name{Author(s) Name(s)\thanks{Thanks to XYZ agency for funding.}}
\address{Author Affiliation(s)}
%
% For example:
% ------------
%\address{School\\
%	Department\\
%	Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
%\twoauthors
%  {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
%	{School A-B\\
%	Department A-B\\
%	Address A-B}
%  {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
%	while at ...}}
%	{School C-D\\
%	Department C-D\\
%	Address C-D}
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
Reinforcement Learning (RL) methods have been proven effective in improving the performance of Large Language Models (LLMs), particularly methods such as Group Relative Policy Optimization (GRPO) that demonstrate outstanding capabilities in many tasks. However, despite the potential of RL in text summarization, few studies have focused on leveraging it to tackle the challenges of optimizing objectives that involve potential conflicts. In this paper, we introduce a strategy called Hypervolume Optimization (HVO), the first multi-objective optimization approach based on GRPO in the task of text summarization without supervised fine-tuning or cold-start. In addition, our method achieves superior performance across multiple benchmark datasets, demonstrating the best results while ensuring a more balanced distribution of scores across dimensions, effectively handling the trade-offs between objectives and enhancing overall optimization stability.

\end{abstract}
%
\begin{keywords}
Multi-objective Reinforcement Learning,
Text Summarization,
Hypervolume Contributions
\end{keywords}
%
\section{Introduction}
\label{sec:intro}
Text summarization is one of the most critical and challenging tasks in Natural Language Processing (NLP)~\cite{DBLP:journals/csur/ZhangYZ25}. To comprehensively evaluate the quality of generated summaries, researchers typically examine multiple dimensions, such as coherence, consistency, fluency, and relevance~\cite{DBLP:conf/emnlp/LiuIXWXZ23}. To evaluate these various aspects, UniEval, a unified evaluation tool, shows strong performanceand~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22} is widely adopted for assessing various models~\cite{DBLP:conf/coling/ZhuL025, DBLP:conf/acl/RyuD0LO24, DBLP:conf/icassp/JungSCR024}.
However, optimizing these dimensions simultaneously is inherently challenging, as improvements along one dimension may lead to compromises in others~\cite{abo2023automatic}, resulting in imbalanced summaries.
\begin{figure}[htb]
\begin{minipage}[b]{1.0\linewidth}
  \centering
  \centerline{\includegraphics[width=1\textwidth]{figures/cnn_radar_chart (18).pdf}}
%  \vspace{2.0cm}
\end{minipage}
%
\caption{The radar chart presents the UniEval scores of the summary produced by the baseline model across four dimensions.}
\label{fig:cnn_rader}
%
\end{figure}

With the development of LLMs, utilizing LLMs for zero-shot generation of summaries has become one of the mainstream approaches~\cite{DBLP:journals/corr/abs-2309-09558}. As shown in Figure~\ref{fig:cnn_rader}, the zero-shot summarization leave room for improvement in overall performance and show imbalanced performance across different dimensions. With the demonstrated success of Reinforcement Learning (RL) in post-training, it has been established as a viable solution to the aforementioned problems~\cite{DBLP:journals/csur/ZhangYZ25}, despite the fact that most studies use RL methods to enhance summarization models by relying on a single reward signal~\cite{DBLP:conf/acl/YadavGAD20,DBLP:journals/nle/FikriOY24,DBLP:conf/acl/RoitFSACDGGHKMG23}. Because they do not integrate multi-dimensional metrics as rewards for summarization, their models still have challenges in generating high-quality summaries under balancing multiple dimensions. To incorporate multi-dimensions into the reward for generating balanced and high-quality summaries, MDO~\cite{DBLP:conf/acl/RyuD0LO24} uses PCGrad optimization to reduce gradient interference across dimensions, facilitating the discovery of Pareto improvements by balancing multiple objectives. Their method requires pairwise gradient projections between different dimensions. Due to the high computational cost, it is not feasibly to be integrated into LLMs.

% At present, there are several methods used for multi-dimensional evaluation of a summary. UniEval, a unified multi-dimensional evaluation tool, shows a significantly stronger correlation with human judgment, outperforming existing metrics~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22}. It performs exceptionally well in the text summarization task, and numerous studies in text summarization have conducted experiments based on this evaluation framework~\cite{DBLP:conf/coling/ZhuL025, DBLP:conf/acl/RyuD0LO24, DBLP:conf/icassp/JungSCR024}.



To overcome these challenges, we propose HVO, a multi-objective reinforcement learning method designed for text summarization that is well-suited for large language models (LLMs). HVO incorporates multi-dimensional rewards into a hypercube volume computation framework, guided by several regulatory mechanisms. As illustrated in Figure~\ref{fig:cnn_rader}, our proposed HVO method significantly outperforms LLM-based baseline approaches by achieving higher scores across multiple evaluation dimensions, while maintaining a more balanced and stable optimization. The main contributions of this work are summarized as follows:

\begin{itemize}[noitemsep, topsep=0pt]
\item We introduce HVO, the first multi-objective reinforcement learning strategy for text summarization based on GRPO, which efficiently balances multiple evaluation dimensions without requiring supervised fine-tuning or a cold start.

\item Using a 7B parameter foundation model, HVO outperforms GPT-4o on two benchmarks, demonstrating the generalization and effectiveness of HVO across different domains.

\item To address the training instability and summary length collapse issues in traditional GRPO, a new length constraint mechanism is introduced, enhancing training stability.
\end{itemize}


\section{METHOD}
\label{sec:intro}
In text summarization, generating high-quality summaries necessitates a careful balance across multiple evaluative dimensions. To achieve a balance among these competing objectives, we introduce HVO, improving the multi-objective optimization method based on GRPO, where a policy model generates summaries optimized with multi-dimensional rewards computed via the UniEval. The entire process is illustrated in Figure 2.


\subsection{Problem Formulation}

Given a set of documents $\{P_1, P_2, \dots, P_N\}$ and their corresponding human expert summaries $\{S_1, S_2, \dots, S_N\}$, using the provided summary prompt, a corresponding question set $\{Q_1, Q_2, \dots, Q_N\}$ is obtained. The dataset $\mathcal{D}$ is formed as $\{(Q_i, S_i)\}_{i=1}^N$. For a specific question-answer pair $(q, s)$,
the policy model $\pi_\theta$ generates a group of $G$ individual
summaries $\{o_i\}_{i=1}^G$. For each evaluation dimension $D_k$ in $\{D_1, D_2, \dots, D_M\}$, the reward model $R$ computes the reward $r^{k}_i$ for the generated summary $o_i$. Our objective is to optimize the policy model's parameters $\theta$ using reinforcement learning to maximize the expected reward. 

\subsection{HVO}
\begin{figure}[ht]
  \centering
  \includegraphics[width=0.48\textwidth]{figures/structure.pdf}
  \caption{The entire process of HVO. $D_i$ denotes the evaluation dimension of the summary, which corresponds to four commonly used criteria in the literature: coherence, consistency, fluency, and relevance.}
  \label{fig:structure}
\end{figure}


The HVO method is based on current state-of-the-art RL algorithms, exemplified by GRPO~\cite{zheng2025group}. In the GRPO method, the optimization of the policy $\pi_\theta$ is achieved by maximizing the following objective function:
\[
\begin{aligned}
\mathcal{J}(\theta) &= \mathbb{E}_{(q, a) \sim \mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\text{old}}(\cdot | q)} \Bigg[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \Bigg( \\
& \min \left( f_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}( f_{i,t}(\theta), 1-\epsilon, 1+\epsilon ) \hat{A}_{i,t} \right) \\
& - \beta D_{\text{KL}}(\pi_{\sigma} || \pi_{\text{ref}}) \Bigg) \Bigg],
\end{aligned}
\]
where
\[
\begin{aligned}
f_{i,t}(\theta) &= \frac{\pi_{\theta}(o_{i,t} | q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} | q, o_{i,<t})}, \quad
\hat{A}_{i,t} &= \frac{r_i - \text{mean}(\{r_i\}_{i=1}^G)}{\text{std}(\{r_i\}_{i=1}^G)}.
\end{aligned}
\]
In GRPO, The reward \( r_i \) is computed as a weighted linear combination of rewards from different dimensions:
\[
r_i = \sum_{k=1}^{M} w^{k} \cdot r_{i}^{k},
\]
where \( w^k \) represents the weight for the reward corresponding to the \( k \)-th dimension $D_k$. 

It is worth noting that while the weighted linear combination method is simple and commonly used, it has limitations, particularly in handling interdependencies between objectives, which can result in unbalanced or incomplete optimization outcomes~\cite{das1997closer}. To address this, we optimize the problem based on the Hypervolume method.

The Hypervolume method~\cite{DBLP:journals/tec/ZitzlerT99} is an evaluation metric in multi-objective optimization used to measure the volume of the hypercube occupied by a set of solutions in the objective space. By calculating the covered area of the solution set relative to a reference point, Hypervolume considers both the quality and diversity of the solutions. A larger covered area indicates better performance of the solution set. It is commonly used to compare the performance of different algorithms or solution sets, especially in the evaluation of Pareto fronts. Moreover, hypervolume-based evaluation has been shown to be a Pareto-consistent evaluation method~\cite{DBLP:journals/tec/ZitzlerTLFF03}. Therefore, optimizing towards hypervolume maximization ensures continuous improvement in the quality of the solution set, progressively approaching the Pareto optimal front.

Inspired by the hypervolume method, we integrate hypervolume evaluation into the multi-dimensional rewards of GRPO and propose the HVO strategy, optimizing the model towards hypervolume maximization, progressively approaching the Pareto optimal front, thereby achieving more balanced and effective summarization. The specific method of HVO is as follows:

\[
r_i = \prod_{k=1}^{M} \left( \min \left( \epsilon, \left( r_i^k - \min_{i=1}^{G} (r_i^k) + \delta \right) \right) \right)^{-w^k},
\]
where $ \min_{i=1}^{G}(r_i^k)$ is the minimum value of the $ k-th$ dimension across all \( G \) samples.
\( \delta \in (0, 1) \) is a small constant used to avoid zero values.

To ensure that \( w_d \) maintains the monotonic effect on the entire hypercube edge length, the term \( r^d_{i} - \min_{i=1}^{G}(r^d_{i}) \) is constrained within the range \( [\delta, \epsilon] \).



\begin{table*}[ht]
\centering
\begin{tabular}{C |r  r  c c c c c c c  }
\toprule
 Dataset&Model&Method& \textbf{C} & \textbf{Cons} & \textbf{F} & \textbf{R} & \textbf{HV} & \textbf{O} & \textbf{S} 
 \\
 \hline
  \multirow{8}{*}{\rotatebox{90}{BillSum}} &PEGASUS & SFT & 0.823 & 0.832 & 0.849 & 0.814 & 0.000 & 0.830 & 0.015 \\
  &Qwen2.5 1.5B & Zero-shot & 0.941 & 0.826 & 0.905 & 0.914 & 0.473 & 0.896 & \textcolor{red}{\textbf{0.050}} \\
  &Qwen2.5 7B & Zero-shot & 0.958 & \textcolor{blue}{\underline{0.843}} & 0.820 & 0.948 & 0.346 & 0.892 & 0.071 \\
  &Qwen2.5 14B & Zero-shot & 0.953 & 0.805 & 0.919 & 0.940 & 0.520 & 0.904 & 0.068 \\
  &Qwen2.5 32B & Zero-shot & 0.948 & 0.799 & \textcolor{red}{\textbf{0.941}} & 0.933 & 0.512 & 0.905 & 0.071 \\
  &GPT4 & Zero-shot & \textcolor{red}{\textbf{0.973}} & \textcolor{blue}{\underline{0.843}} & 0.831 & \textcolor{red}{\textbf{0.971}} & 0.477 & 0.904 & 0.078 \\
  &Qwen2.5 7B & GRPO & 0.959 & 0.823 & 0.912 & \textcolor{blue}{\underline{0.955}} & \textcolor{blue}{\underline{0.663}} & \textcolor{blue}{\underline{0.912}} & 0.063 \\
  &Qwen2.5 7B & HVO & \textcolor{blue}{\underline{0.964}} & \textcolor{red}{\textbf{0.853}} & \textcolor{blue}{\underline{0.939}} & \textcolor{blue}{\underline{0.955}} & \textcolor{red}{\textbf{1.000}} & \textcolor{red}{\textbf{0.928}} & \textcolor{blue}{\underline{0.051}} \\
\hline
 \multirow{8}{*}{\rotatebox{90}{CNN/DailyMail}}&PEGASUS & SFT & 0.936 & \textcolor{red}{\textbf{0.939}} & 0.815 & 0.684 & 0.000 & 0.843 & 0.121 \\
 &Qwen2.5 1.5B & Zero-shot & 0.871 & 0.819 & 0.936 & 0.861 & 0.086 & 0.872 & 0.048 \\
 &Qwen2.5 7B & Zero-shot & 0.890 & 0.820 & 0.932 & 0.874 & 0.136 & 0.879 & 0.046 \\
 &Qwen2.5 14B & Zero-shot & 0.931 & 0.826 & 0.859 & 0.907 & 0.152 & 0.881 & 0.047 \\
 &Qwen2.5 32B & Zero-shot & 0.918 & 0.843 & 0.933 & 0.893 & 0.298 & 0.897 & 0.040 \\
 &GPT4 & Zero-shot & \textcolor{red}{\textbf{0.967}} & 0.840 & \textcolor{blue}{\underline{0.945}} & \textcolor{blue}{\underline{0.934}} & 0.535 & 0.921 & 0.056 \\
 &Qwen2.5 7B & GRPO & 0.908 & 0.903 & 0.922 & \textcolor{red}{\textbf{0.954}} & \textcolor{blue}{\underline{0.544}} & \textcolor{blue}{\underline{0.922}} & \textcolor{blue}{\underline{0.023}} \\
 &Qwen2.5 7B & HVO & \textcolor{blue}{\underline{0.961}} & \textcolor{blue}{\underline{0.926}} & \textcolor{red}{\textbf{0.951}} & \textcolor{blue}{\underline{0.934}} & \textcolor{red}{\textbf{1.000}} & \textcolor{red}{\textbf{0.943}} & \textcolor{red}{\textbf{0.016}} \\ \bottomrule 

\end{tabular}

\caption{The results of multi-dimension evaluation measured on both the CNN/DailyMail and BillSum datasets. Within the same dimension, the bold denotes the highest score, and the underline denotes the second-highest score. The HV score is a normalized score that makes the comparison clearer.}
\label{tab:combined_results}
\end{table*}



\begin{table*}[ht]
\centering
\begin{tabular}{l | c | c c c c c c c }
\toprule
\textbf{Model} & \textbf{Method} & \textbf{Coherence} & \textbf{Consistency} & \textbf{Fluency} & \textbf{Relevance} & \textbf{HV score} & \textbf{Overall}  & \textbf{STD} \\
\midrule
PEGASUS & SFT & 0.936  & \textcolor{red}{\textbf{0.939}}  & 0.815  & 0.684  & 0.000 & 0.843 & 0.121  \\
Qwen2.5 1.5B & Zero-shot & 0.871 & 0.819 & 0.936 & 0.861 & 0.086 & 0.872 & 0.048  \\
Qwen2.5 7B   & Zero-shot & 0.890 & 0.820 & 0.932 & 0.874 & 0.136 & 0.879 & 0.046 \\
Qwen2.5 14B  & Zero-shot       & 0.931 & 0.826 & 0.859 & 0.907 & 0.152 & 0.881 & 0.047  \\
Qwen2.5 32B  & Zero-shot       & 0.918 & 0.843 & 0.933 & 0.893 & 0.298 & 0.897 & 0.040  \\
GPT4         & Zero-shot & \textcolor{red}{\textbf{0.967}} & 0.840 & \textcolor{blue}{\underline{0.945}} & \textcolor{blue}{\underline{0.934}} & 0.535 & 0.921 & 0.056  \\
Qwen2.5 7B   & GRPO   & 0.908 & 0.903 & 0.922 & \textcolor{red}{\textbf{0.954}} & \textcolor{blue}{\underline{0.544}} & \textcolor{blue}{\underline{0.922}} & \textcolor{blue}{\underline{0.023}}  \\
Qwen2.5 7B   & HVO (Ours)  & \textcolor{blue}{\underline{0.961}} & \textcolor{blue}{\underline{0.926}} & \textcolor{red}{\textbf{0.951}} & \textcolor{blue}{\underline{0.934}} & \textcolor{red}{\textbf{1.000}} & \textcolor{red}{\textbf{0.943}} & \textcolor{red}{\textbf{0.016}} \\
\toprule
\end{tabular}
\caption{The results of multi-dimension evaluation measured on the CNN/DailyMail dataset. Within the same dimension, the bold denotes the highest score, and the underline denotes the second-highest score. The HV score is a normalized score that makes the comparison clearer.}
\label{tab:cnn_daily_mail_results}
\end{table*}

\begin{table*}[ht]
\centering
\begin{tabular}{l | c | c c c c c c c }
\toprule
\textbf{Model} & \textbf{Method} & \textbf{Coherence} & \textbf{Consistency} & \textbf{Fluency} & \textbf{Relevance} & \textbf{HV score} & \textbf{Overall}  & \textbf{STD} \\
\midrule
PEGASUS & SFT & 0.823 & 0.832 & 0.849 & 0.814 & 0.000 & 0.830 & 0.015 \\
Qwen2.5 1.5B & Zero-shot & 0.941 & 0.826 & 0.905 & 0.914 & 0.473 & 0.896 & \textcolor{red}{\textbf{0.050}}  \\
Qwen2.5 7B   & Zero-shot & 0.958 & \textcolor{blue}{\underline{0.843}} & 0.820 & 0.948 & 0.346  & 0.892 & 0.071 \\
Qwen2.5 14B  & Zero-shot & 0.953 & 0.805 & 0.919 & 0.940 & 0.520 & 0.904 & 0.068  \\
Qwen2.5 32B  & Zero-shot & 0.948 & 0.799 & \textcolor{red}{\textbf{0.941}} & 0.933 & 0.512 & 0.905 & 0.071 \\
GPT4         & Zero-shot & \textcolor{red}{\textbf{0.973}} & \textcolor{blue}{\underline{0.843}} & 0.831 & \textcolor{red}{\textbf{0.971}} & 0.477 & 0.904 & 0.078 \\
Qwen2.5 7B   & GRPO & 0.959 & 0.823 & 0.912 & \textcolor{blue}{\underline{0.955}} & \textcolor{blue}{\underline{0.663}} & \textcolor{blue}{\underline{0.912}} & 0.063 \\
Qwen2.5 7B   &  HVO (Ours) & \textcolor{blue}{\underline{0.964}} & \textcolor{red}{\textbf{0.853}} & \textcolor{blue}{\underline{0.939}} & \textcolor{blue}{\underline{0.955}} & \textcolor{red}{\textbf{1.000}} & \textcolor{red}{\textbf{0.928}} & \textcolor{blue}{\underline{0.051}} \\
\toprule
\end{tabular}
\caption{The results of multi-dimension evaluation measured on the BillSum dataset. Within the same dimension, the bold denotes the highest score, and the underline denotes the second-highest score. The HV score is a normalized score that makes the comparison clearer.}
\label{tab:Unieval_mail_results}
\end{table*}

\subsection{Multi-dimensional rewards}
\label{ssec:subhead}
UniEval redefines multi-dimensional evaluation in summarization as a Boolean Question-Answering (QA) task. It calculates scores based on the model's "Yes/No" answers to specific questions for each dimension. Using a unified QA format, UniEval combines multi-task learning, integrates external knowledge, and employs continual learning to reduce negative transfer. This enables multi-dimensional evaluation with a single model, focusing on coherence, consistency, fluency, and relevance in text summarization.

Recent studies have shown that GRPO encounters issues with training instability~\cite{yu2025dapo,zheng2025group}. This leads to issues with summary length collapse during training when using UniEval as the sole reward. To address this, we propose a new length constraint method to help maintain training stability.

\[
R_{\text{conciseness}}\left( o_i\right) = \frac{1}{1 + \left( \frac{x_i}{\rho} \right)^{\lambda}},
\]
where
\[
x_i = \left| \frac{\left|P_i\right|}{\left|o_i\right|} - \text{mean} \left( \left\{ \frac{\left|P_j\right|}{\left|A_j\right|} \right\}_{j=1}^{V} \right) \right|,
\]
It represents the absolute difference between the compression ratio of the generated summary $o_i$ and the average compression ratio of human-generated summaries in the training set.
This ensures that the score decreases slowly around $x_i = 0$, maintaining sufficient exploration space, and declines rapidly when $x_i$ is large.
The $\lambda$ represents the steepness, indicating the degree of rapid decrease, while $\rho$ represents the offset, indicating how far the value starts to decrease sharply when $x_i = 0$.

\section{Experiment}

\subsection{Dataset}
To address the potential impact of document complexity, we utilize two text summarization datasets: the CNN/Daily Mail dataset for news summarization and the BillSum dataset for legislative content. The CNN/DM dataset~\cite{nallapati-etal-2016-abstractive} comprises a training set of 287K and a test set of 11.5K, while BillSum~\cite{kornilova-eidelman-2019-billsum} has a training set of 18.9K and a test set of 3.2K.

\subsection{Baseline models}
We use the Qwen 2.5 series model as our baseline model. Qwen 2.5 has demonstrated top-tier performance on a wide range of benchmarks evaluating language understanding, reasoning, mathematics, coding, human preference
alignment, etc~\cite{qwen2025qwen25technicalreport}. In the experiment, we used the instruct version exclusively, which enhanced the accuracy and relevance of task execution, ensuring more precise and efficient results. We used the 7B model for comparing Naive GRPO and GRPO + HVO due to its good balance between performance and resource usage. In addition, we used GPT-4-turbo for GPT-4 and employed the already fine-tuned versions of PEGASUS on the BillSum and CNN/DM datasets as baselines.


\subsection{Setup and Metrics}
For training the Naive GRPO, we randomly select only 10K samples from the training set of each dataset. The training parameters are set as follows: train\_batch\_size = 64, num\_generations = 8, max\_grad\_norm = 0.4, and learning\_rate = 5e-7. All rewards are weighted equally with a weight of 1.0 . Additionally, for HVO, we default to use $\epsilon$ = 0.99 and $\delta$ = 0.1, We primarily use Unieval as the evaluation metric.

\subsection{Results}
\begin{figure}[ht]
  \centering
  \includegraphics[width=0.48\textwidth]{figures/plot_output.pdf}
  \caption{Scatter plot illustrating the relationship between the overall score and completion length, as evaluated by UniEval, for different models on two datasets: CNN-DailyMail (blue) and BillSum (green).}
  \label{fig:scattor}
\end{figure}
Based on the experimental results from Table~\ref{tab:cnn_daily_mail_results} and Table~\ref{tab:Unieval_mail_results}, HVO Method outperforms all other methods in both datasets (CNN/DailyMail and BillSum), demonstrating the highest HpyerVolume (HV) scores and overall scores. It also achieves the first and second lowest standard deviations on the CNN/DailyMail and BillSum datasets, indicating superior performance in balancing multiple evaluation dimensions.

GPT-4, while excelling in coherence (0.967) and fluency (0.945) on the CNN/DailyMail dataset and coherence (0.973) and relevance (0.971) on the BillSum dataset, does not perform as well in overall performance and dimension balance compared to the Qwen 2.5 7B HVO method. This also shows that the HVO method, without needing SFT cold-start, can achieve results comparable to or better than GPT-4 in the summarization task.

Qwen2.5 7B under both GRPO and HVO methods shows balanced performance across coherence, consistency, fluency, and relevance, with HVO achieving the best results. This highlights the limitations of using weighted linear combination method for multi-objective optimization and demonstrates the effectiveness of our approach. To analyze this, from the BillSum results, we observe that when the overall scores of Qwen 2.5 14B, Qwen2.5 32B, and GPT-4 are close, with no obvious Pareto dominance between the individual objectives, the HV score better accounts for the balance among all the objectives.

Finally, we computed the generation length of different models and plotted a scatter plot of the overall score against Completion Length, as shown in the figure ~\ref{fig:scattor}. Our method not only achieves the highest overall score but also maintains a shorter Completion Length, ensuring better conciseness.


These results indicate that the HVO method has a significant advantage in multi-objective optimization tasks, effectively improving summary generation quality and achieving balanced optimization across dimensions.

\section{Conclusion}

In this paper, we proposed a new method called Hypervolume Optimization (HVO) for multi-objective reinforcement learning in text summarization. Unlike traditional methods, HVO integrates multiple evaluation metrics to balance different objectives, improving the stability of the optimization process.

Our experiments show that HVO outperforms existing methods on benchmark datasets like CNN/DailyMail and BillSum, achieving the highest hypervolume (HV) scores and overall score. This demonstrates HVO’s ability to balance trade-offs between different objectives. Additionally, HVO performs similarly to or better than GPT-4 without the need for supervised fine-tuning or a cold start, highlighting its effectiveness in generating high-quality summaries.

Overall, HVO offers an effective solution for multi-objective text summarization, improving both summary quality and optimization stability. Future work could explore applying this method to other NLP tasks and further improving it for more complex reward structures.





\label{ssec:subhead}
\label{sec:intro}
% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEbib}
\bibliography{strings,refs}

\end{document}

% Template for ICASSP-2026 paper; to be used with:
%          spconf.sty  - ICASSP/ICIP LaTeX style file, and
%          IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
% \documentclass{article}
\usepackage{spconf,amsmath,graphicx,hyperref}
\usepackage{amsmath}  % For mathematical symbols and equations
\usepackage{amssymb}  % For additional symbols (like \mathbb)
\usepackage{graphicx}  % 在导言区导入 graphicx 包
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{amsmath}
% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

% Title.
% ------
\title{Multi-Objective Text Summarization with GRPO: A Hypervolume Maximization Framework}
%
% Single address.
% ---------------
\name{Author(s) Name(s)}
% \thanks{Thanks to XYZ agency for funding.}}
\address{Author Affiliation(s)}
%
% For example:
% ------------
%\address{School\\
%	Department\\
%	Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
%\twoauthors
%  {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
%	{School A-B\\
%	Department A-B\\
%	Address A-B}
%  {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
%	while at ...}}
%	{School C-D\\
%	Department C-D\\
%	Address C-D}
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
The summarization task typically involves multiple objectives, such as consistency, coherence, relevance, and fluency. Due to the difficulty in predefining the optimal trade-offs between these objectives, finding Pareto improvements becomes a particularly challenging task when facing such inherent trade-offs. In GRPO, the manual setting of weights relies on expert knowledge, which may fail to fully account for the interdependencies between multiple objectives, thereby hindering the achievement of an optimal trade-off. To address this, we propose a novel optimization strategy that dynamically adjusts the scores between groups during the reward process using the hypervolume method, guiding the model's optimization towards progressively approximating the Pareto front. Unlike previous ROUGE-based rewards that rely on reference summaries, we use a QA-based reward model that aligns with human preferences. Our method achieves superior performance across multiple benchmark datasets, demonstrating the best results. Additionally, it ensures a more balanced distribution of scores across dimensions, effectively addressing the trade-offs between objectives and improving the overall optimization stability.


\end{abstract}
%
\begin{keywords}
Multi-objective Reinforcement Learning,
Text Summarization,
Hypervolume Contributions
\end{keywords}
%
\section{Introduction}
\label{sec:intro}
Abstractive summarization~\cite{DBLP:journals/corr/abs-2403-02901} is a natural language processing task aimed at capturing the core meaning of the original content while ensuring fluency and coherence. With the advancement of large models, summarization tasks now demonstrate strong performance and high accuracy~\cite{DBLP:journals/corr/abs-2309-09558}. However, there is still room for improvement in several key metrics, particularly in reducing hallucinations and improving fidelity~\cite{DBLP:conf/coling/LiLZ24}. Current evaluation metrics for summarization primarily focus on similarity-based measures, such as ROUGE~\cite{ROUGE-2004ACL-LIN} and BARTSCORE~\cite{yuan2021bartscore}. However, these metrics fail to fully evaluate abstractive summaries, neglecting factors like semantic consistency and fluency~\cite{DBLP:conf/acl/SiledarNMRNBBPS24}. To mitigate these limitations, UNIEVAL~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22} is a unified multi-dimensional evaluation tool that performs exceptionally well in the text summarization task, and numerous studies have conducted experiments based on this evaluation framework~\cite{DBLP:conf/coling/ZhuL025, DBLP:conf/acl/RyuD0LO24, DBLP:conf/icassp/JungSCR024}.

Reinforcement learning (RL) has been widely applied in large language models (LLMs), enabling them to achieve better performance on various tasks, including abstractive summarization tasks~\cite{DBLP:journals/csur/ZhangYZ25}. Consequently, researchers have leveraged reinforcement learning (RL) to fine-tune models for the summarization task to improve summary generation performance. However, these studies often do not fully consider the balance across multiple dimensions of summary quality~\cite{DBLP:conf/acl/RoitFSACDGGHKMG23, DBLP:conf/emnlp/ChoiCSJK24}. MDO~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22} has been applied to multi-objective optimization in summarization tasks; however, it has not yet been extended to LLMs. Moreover, it overlooks the inefficiency and instability of Proximal Policy Optimization (PPO)~\cite{DBLP:journals/corr/SchulmanWDRK17} with sparse reward signals in LLM training.

Therefore, we propose an optimization strategy based on GRPO~\cite{DBLP:journals/corr/abs-2402-03300} for multi-objective reinforcement learning, aiming to produce balanced and high-quality summaries. Furthermore, we demonstrate the superiority of our strategy through experiments on several benchmark public datasets, showing that it outperforms existing methods in terms of performance.

\section{BACKGROUND}
\label{sec:background}
\subsection{Multi-dimensional reward}
UniEval redefines multi-dimensional evaluation in natural language generation (NLG) as a Boolean Question-Answering (QA) task. It guides the model to output "Yes/No" judgments by designing specific questions for different dimensions (e.g., "Is this a coherent summary?"), and then calculates the dimension scores based on the model's probability of generating a "Yes" response. Additionally, through the unified Boolean QA format, UniEval incorporates intermediate multi-task learning, integrating external knowledge from tasks like natural language inference and linguistic tasks, while employing continual learning to mitigate the negative transfer issue. This enables multi-dimensional evaluation with a single model. In the text summarization task, it primarily evaluates four core metrics: Coherence, Consistency, Fluency, and Relevance.

\subsection{GRPO Algorithm}

We employ the Group Relative Policy Optimization (GRPO) method to optimize the summarization model using UniEval metric reward. At each training step, for a specific summarization task \(q\), a set of candidate outputs \(\{o_1, o_2, \dots, o_G\}\) is drawn from the policy model \(\pi_{\theta_{old}}\). The advantage \(A_i\) for each candidate output is calculated as follows:


\[
A_i = \frac{r_i - \text{mean}(\{r_1, r_2, \dots, r_G\})}{\text{std}(\{r_1, r_2, \dots, r_G\})},
\]


GRPO then maximizes the following objective function to optimize \(\pi_{\theta}\):

\[
\begin{aligned}
J_{GRPO}(\theta) &= \mathbb{E}_{q \sim P(Q), \{o_i\}_i \sim \pi_{\theta_{old}}(O|q)} \Bigg[ \\
&\quad \frac{1}{G} \sum_{i=1}^{G} \min \left( \frac{\pi_{\theta}(o_i | q)}{\pi_{\theta_{old}}(o_i | q)} A_i, \right. \\
&\quad \left. \text{clip} \left( \frac{\pi_{\theta}(o_i | q)}{\pi_{\theta_{old}}(o_i | q)}, 1 - \epsilon, 1 + \epsilon \right) A_i \right) \Bigg] \\
&\quad - \beta D_\text{KL}(\pi_{\theta} || \pi_{ref}),
\end{aligned}
\]
Where \(\epsilon\) and \(\beta\) are hyperparameters controlling the clipping threshold and the weight of the Kullback-Leibler (KL) divergence penalty, respectively. The KL divergence approximation term is given by:

\[
D_\text{KL}(\pi_{\theta} || \pi_{ref}) = \sum_{i} \pi_{ref}(o_i|q) \log \left( \frac{\pi_{ref}(o_i|q)}{\pi_{\theta}(o_i|q)} \right),
\]
This formulation ensures stable and efficient RL training by optimizing the policy while respecting the constraints of the reward function.

\section{Method}
\label{sec:method}
The application of large-scale Reinforcement Learning (RL) has significantly enhanced the reasoning abilities of Large Language Models (LLMs)~\cite{openai2024b}.In abstractive summarization tasks, applying reinforcement learning (RL) for post-training large language models (LLMs) is crucial. The core objective is to preserve the model's original summarization capabilities after fine-tuning while further optimizing its performance across various evaluation dimensions. This involves enhancing semantic consistency, fluency, and information coverage, while maintaining a balanced optimization across multiple metrics to avoid overemphasis or bias towards any single dimension. A multi-objective optimization strategy driven by RL can effectively improve the model’s overall performance across all dimensions, leading to a comprehensive enhancement of summarization quality.To achieve this, we employ GRPO for optimization alongside Unieval, By incorporating four distinct dimensions into the reward process, we introduce a method called HyperVolume Optimization (HVO).

\subsection{GRPO Rewards Strategy}
In GRPO, The reward \( r \) is computed as a weighted sum of rewards from different dimensions:

\[
r_i = \sum_{d=1}^{M} w^{d} \cdot r_{i}^{d},
\]
where \( w_d \) represents the weight for the reward corresponding to the \( d \)-th dimension. \( r_{i}^{d} \) represents the score for the \( d \)-th dimension of the \( i \)-th sample, such as fluency in the case of Unieval.


\subsection{HVO}
The Hypervolume method~\cite{DBLP:journals/tec/ZitzlerT99} is an evaluation metric in multi-objective optimization used to measure the volume of the hypercube occupied by a set of solutions in the objective space. By calculating the covered area of the solution set relative to a reference point, Hypervolume considers both the quality and diversity of the solutions. A larger covered area indicates better performance of the solution set. It is commonly used to compare the performance of different algorithms or solution sets, especially in the evaluation of Pareto fronts. Moreover, hypervolume-based evaluation has been shown to be a Pareto-consistent evaluation method~\cite{DBLP:journals/tec/ZitzlerTLFF03}. Therefore, optimizing towards hypervolume maximization ensures continuous improvement in the quality of the solution set, progressively approaching the Pareto optimal front.

Inspired by the hypervolume method, we integrate hypervolume evaluation into the multi-dimensional rewards of GRPO and propose the HVO strategy, optimizing the model towards hypervolume maximization, thereby achieving more balanced and effective optimization, progressively approaching the Pareto optimal front, thereby achieving more balanced and effective summarization. The specific method of HVO is as follows:

\[
r_i = \prod_{d=1}^{M} \left( \max \left( \epsilon, \min \left( r_i^d - \min_{i=1}^{G} (r_i^d), 1 - \epsilon \right) \right) \right)^{-w_d},
\]
Where:
\begin{itemize}
    \item \( \min_{i=1}^{G}(r_i^d) \) is the minimum value of the \( d \)-th dimension across all \( G \) samples.
    \item \( \epsilon \in (0, 1) \) is a small constant used to avoid zero values.
\end{itemize}
To ensure that \( w_d \) maintains the monotonic effect on the entire hypercube edge length, the term \( r_{id} - \min_{i=1}^{G}(r_{id}) \) is constrained within the range \( [\epsilon, 1 - \epsilon] \) using the \( \min \) and \( \max \) functions.




\begin{algorithm}
\caption{Calculation of HVO}
\begin{algorithmic}[1]
\State \textbf{Input:} paragraphs = $\{P_1, P_2, \dots, P_G\}$, initial\_policy $\pi_{\theta}$, reward\_function $r$, group\_size $M$, clipping\_epsilon $\varepsilon$, KL\_penalty $\beta$
\State \textbf{Output:} Optimized policy $\pi_{\theta^*}$
\For{each training iteration}
    \State reference\_policy $\gets \pi_{\theta}$ 
    \For{each prompt $P_i$ in paragraphs}
        \State Generate $M$ outputs $\{o_{i,1}, o_{i,2}, \dots, o_{i,M}\}$ using $\pi_{\theta}$
        \State Compute rewards $\{r_{i,1}, r_{i,2}, \dots, r_{i,M}\}$ for each output using $r$
        \State \textbf{Calculate the weighted reward:}
        \[
        r_i = \sum_{d=1}^{M} w^{d} \cdot r_{i}^{d}
        \]
        \State $\mu_i \gets \text{mean}(r_{i,1}, r_{i,2}, \dots, r_{i,M})$
        \State $\sigma_i \gets \text{std}(r_{i,1}, r_{i,2}, \dots, r_{i,M})$
        \For{each output $o_{i,j}$}
            \State $A_{i,j} \gets \frac{r_{i,j} - \mu_i}{\sigma_i}$ \Comment{Normalized advantage}
        \EndFor
    \EndFor
    \State Update $\pi_{\theta}$ by maximizing the objective:
    \State Update $\theta$ using gradient ascent
\EndFor

\end{algorithmic}
\end{algorithm}

\section{Experiment}
\label{sec:exp}
% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEbib}
\bibliography{strings,refs}

\end{document}

% Template for ICASSP-2026 paper; to be used with:
%          spconf.sty  - ICASSP/ICIP LaTeX style file, and
%          IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
% \documentclass{article}
\usepackage{spconf,amsmath,graphicx,hyperref}
\usepackage{amsmath}  % For mathematical symbols and equations
\usepackage{amssymb}  % For additional symbols (like \mathbb)
\usepackage{graphicx}  % 在导言区导入 graphicx 包
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{amsmath}
% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

% Title.
% ------
\title{Multi-Objective Text Summarization with GRPO: A Hypervolume Maximization Framework}
%
% Single address.
% ---------------
\name{Author(s) Name(s)\thanks{Thanks to XYZ agency for funding.}}
\address{Author Affiliation(s)}
%
% For example:
% ------------
%\address{School\\
%	Department\\
%	Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
%\twoauthors
%  {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
%	{School A-B\\
%	Department A-B\\
%	Address A-B}
%  {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
%	while at ...}}
%	{School C-D\\
%	Department C-D\\
%	Address C-D}
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
Reinforcement Learning (RL) methods have been shown to improve the performance of Large Language Models (LLMs), particularly with models such as DeepSeek-R1 demonstrating outstanding capabilities in many tasks. However, despite the potential of RL in text summarization, few studies have focused on leveraging it to tackle the challenges of optimizing objectives that involve potential conflicts. In this paper, we introduce a strategy called Hypervolume Optimization (HVO), the first multi-objective optimization approach based on R1-like reinforcement learning in the task of text summarization without supervised fine-tuning or cold-start. In addition, our method achieves superior performance across multiple benchmark datasets, demonstrating the best results while ensuring a more balanced distribution of scores across dimensions, effectively handling the trade-offs between objectives and enhancing overall optimization stability.


\end{abstract}
%
\begin{keywords}
Multi-objective Reinforcement Learning,
Text Summarization,
Hypervolume Contributions
\end{keywords}
%
\section{Introduction}
\label{sec:intro}
Abstractive summarization~\cite{DBLP:journals/corr/abs-2403-02901} is a natural language processing task aimed at capturing the core meaning of the original content while ensuring fluency and coherence. However, a high-quality summary is not determined by a single factor. it is typically assessed across multiple dimensions, including coherence, consistency, fluency, and relevance~\cite{DBLP:conf/emnlp/LiuIXWXZ23}.With the advancement of LLMs, summarization tasks now demonstrate strong performance and high accuracy~\cite{DBLP:journals/corr/abs-2309-09558}. However, there is still room for improvement in several key metrics, particularly in reducing hallucinations and improving fidelity~\cite{DBLP:conf/coling/LiLZ24}. 

Current evaluation metrics for summarization primarily focus on similarity-based measures, such as ROUGE~\cite{ROUGE-2004ACL-LIN} and BARTSCORE~\cite{yuan2021bartscore}. However, these metrics fail to fully evaluate abstractive summaries, neglecting factors like semantic consistency and fluency~\cite{DBLP:conf/acl/SiledarNMRNBBPS24}. To mitigate these limitations, various studies have focused on addressing this issue~\cite{DBLP:conf/naacl/PasunuruB18,DBLP:conf/acl/ParnellUP22}. In comparison, UNIEVAL, a unified multi-dimensional evaluation tool, shows a significantly stronger correlation with human judgment, outperforming existing metrics~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22} . It performs exceptionally well in the text summarization task, and numerous studies have conducted experiments based on this evaluation framework~\cite{DBLP:conf/coling/ZhuL025, DBLP:conf/acl/RyuD0LO24, DBLP:conf/icassp/JungSCR024}.

Reinforcement learning (RL) has been widely applied in large language models (LLMs), enabling them to achieve better performance on various tasks, including abstractive summarization tasks~\cite{DBLP:journals/csur/ZhangYZ25}. Consequently, researchers have leveraged reinforcement learning (RL) to fine-tune models for the summarization task to improve summary generation performance~\cite{DBLP:conf/acl/ParnellUP22}. However, these studies often rely on a single dimension or do not fully consider the balance across multiple dimensions of summary quality.~\cite{DBLP:conf/acl/RoitFSACDGGHKMG23, DBLP:conf/emnlp/ChoiCSJK24,DBLP:conf/acl/RoitFSACDGGHKMG23}. The latest multi-objective optimization method in the field of summarization, MDO~\cite{DBLP:conf/emnlp/Zhong0YMJLZJH22} has been applied to multi-objective optimization in summarization tasks; however, It has not yet been extended to LLMs. Moreover, it overlooks the inefficiency and instability of Proximal Policy Optimization (PPO)~\cite{DBLP:journals/corr/SchulmanWDRK17} with sparse reward signals in LLM training, the need for SFT cold-start, and the high computational complexity of the PCGrad optimizer~\cite{DBLP:conf/nips/YuK0LHF20}, all of which contribute to the significant challenges in training LLMs.

Therefore, we propose an optimization strategy, HVO, for multi-objective reinforcement learning, with the following contributions:

(1) We propose HVO, a multi-objective optimization strategy for text summarization. It guides the model to approximate the Pareto front, generating high-quality summaries.


(2) HVO is the first multi-objective optimization approach based on R1-like reinforcement learning in the task of text summarization, without requiring supervised fine-tuning or cold-start.

(3) Our method outperforms GPT-4O-level results on multiple benchmark datasets, ensuring a balanced score distribution, effectively handling trade-offs, and improving overall optimization stability.

\section{BACKGROUND}
\label{sec:intro}
\subsection{Multi-dimensional reward}
\label{ssec:subhead}
UniEval redefines multi-dimensional evaluation in NLG as a Boolean Question-Answering (QA) task. It calculates scores based on the model's "Yes/No" answers to specific questions for each dimension. Using a unified QA format, UniEval combines multi-task learning, integrates external knowledge, and employs continual learning to reduce negative transfer. This enables multi-dimensional evaluation with a single model, focusing on Coherence, Consistency, Fluency, and Relevance in text summarization.

\subsection{GRPO Algorithm}

We employ the Group Relative Policy Optimization (GRPO) method to optimize the summarization model using UniEval metric reward. At each training step, for a specific summarization task \(q\), a set of candidate outputs \(\{o_1, o_2, \dots, o_G\}\) is drawn from the policy model \(\pi_{\theta_{old}}\). The advantage \(A_i\) for each candidate output is calculated as follows:


\[
A_i = \frac{r_i - \text{mean}(\{r_1, r_2, \dots, r_G\})}{\text{std}(\{r_1, r_2, \dots, r_G\})},
\]


GRPO then maximizes the following objective function to optimize \(\pi_{\theta}\):

\[
\begin{aligned}
J_{GRPO}(\theta) &= \mathbb{E}_{q \sim P(Q), \{o_i\}_i \sim \pi_{\theta_{old}}(O|q)} \Bigg[ \\
&\quad \frac{1}{G} \sum_{i=1}^{G} \min \left( \frac{\pi_{\theta}(o_i | q)}{\pi_{\theta_{old}}(o_i | q)} A_i, \right. \\
&\quad \left. \text{clip} \left( \frac{\pi_{\theta}(o_i | q)}{\pi_{\theta_{old}}(o_i | q)}, 1 - \epsilon, 1 + \epsilon \right) A_i \right) \Bigg] \\
&\quad - \beta \text{DKL}(\pi_{\theta} || \pi_{ref}),
\end{aligned}
\]
Where \(\epsilon\) and \(\beta\) are hyperparameters controlling the clipping threshold and the weight of the Kullback-Leibler (KL) divergence penalty, respectively. The KL divergence approximation term is given by:

\[
\text{DKL}(\pi_{\theta} || \pi_{ref}) = \sum_{i} \pi_{ref}(o_i|q) \log \left( \frac{\pi_{ref}(o_i|q)}{\pi_{\theta}(o_i|q)} \right),
\]
This formulation ensures stable and efficient RL training by optimizing the policy while respecting the constraints of the reward function.

\section{Method}
\label{sec:intro}
The application of large-scale Reinforcement Learning (RL) has significantly enhanced the reasoning abilities of Large Language Models (LLMs)~\cite{openai2024b}.In abstractive summarization tasks, applying reinforcement learning (RL) for post-training large language models (LLMs) is crucial. The core objective is to preserve the model's original summarization capabilities after fine-tuning while further optimizing its performance across various evaluation dimensions. This involves enhancing semantic consistency, fluency, and information coverage, while maintaining a balanced optimization across multiple metrics to avoid overemphasis or bias towards any single dimension. A multi-objective optimization strategy driven by RL can effectively improve the model’s overall performance across all dimensions, leading to a comprehensive enhancement of summarization quality.To achieve this, we employ GRPO for optimization alongside Unieval, By incorporating four distinct dimensions into the reward process, we introduce a method called HyperVolume Optimization (HVO).

\subsection{GRPO Rewards Strategy}
In GRPO, The reward \( r \) is computed as a weighted sum of rewards from different dimensions:

\[
r_i = \sum_{d=1}^{M} w^{d} \cdot r_{i}^{d},
\]
where \( w_d \) represents the weight for the reward corresponding to the \( d \)-th dimension. \( r_{i}^{d} \) represents the score for the \( d \)-th dimension of the \( i \)-th sample, such as fluency in the case of Unieval.


\subsection{HVO}
The Hypervolume method~\cite{DBLP:journals/tec/ZitzlerT99} is an evaluation metric in multi-objective optimization used to measure the volume of the hypercube occupied by a set of solutions in the objective space. By calculating the covered area of the solution set relative to a reference point, Hypervolume considers both the quality and diversity of the solutions. A larger covered area indicates better performance of the solution set. It is commonly used to compare the performance of different algorithms or solution sets, especially in the evaluation of Pareto fronts. Moreover, hypervolume-based evaluation has been shown to be a Pareto-consistent evaluation method~\cite{DBLP:journals/tec/ZitzlerTLFF03}. Therefore, optimizing towards hypervolume maximization ensures continuous improvement in the quality of the solution set, progressively approaching the Pareto optimal front.

Inspired by the hypervolume method, we integrate hypervolume evaluation into the multi-dimensional rewards of GRPO and propose the HVO strategy, optimizing the model towards hypervolume maximization, thereby achieving more balanced and effective optimization, progressively approaching the Pareto optimal front, thereby achieving more balanced and effective summarization. The specific method of HVO is as follows:

\[
r_i = \prod_{d=1}^{M} \left( \max \left( \epsilon, \min \left( r_i^d - \min_{i=1}^{G} (r_i^d), 1 - \epsilon \right) \right) \right)^{-w_d},
\]
Where:
\begin{itemize}
    \item \( \min_{i=1}^{G}(r_i^d) \) is the minimum value of the \( d \)-th dimension across all \( G \) samples.
    \item \( \epsilon \in (0, 1) \) is a small constant used to avoid zero values.
\end{itemize}
To ensure that \( w_d \) maintains the monotonic effect on the entire hypercube edge length, the term \( r_{id} - \min_{i=1}^{G}(r_{id}) \) is constrained within the range \( [\epsilon, 1 - \epsilon] \) using the \( \min \) and \( \max \) functions.


\begin{algorithm}
\caption{Calculation of HVO}
\begin{algorithmic}[1]
\State \textbf{Input:} paragraphs = $\{P_1, P_2, \dots, P_G\}$, initial\_policy $\pi_{\theta}$, reward\_function $r$, group\_size $M$, clipping\_epsilon $\varepsilon$, KL\_penalty $\beta$
\State \textbf{Output:} Optimized policy $\pi_{\theta^*}$
\For{each training iteration}
    \State reference\_policy $\gets \pi_{\theta}$ 
    \For{each paragraph $P_i$ in paragraphs}
        \State Generate $N$ outputs $\{o_{i,1}, o_{i,2}, \dots, o_{i,N}\}$ using $\pi_{\theta}$
        \For{each output $o_{i,j}$}
            \State Compute $M$ dimensional scores $r_{i,j}^d$ for the output using $r$ (i.e., $r_{i,j}^d$ for $d=1, 2, \dots, M$)
        \EndFor
        \State \textbf{Calculate the weighted reward:}
        \[
        r_i = \sum_{d=1}^{M} w^{d} \cdot r_{i}^{d}
        \]
        \State $\mu_i \gets \text{mean}(r_{i,1}^1, r_{i,1}^2, \dots, r_{i,1}^M)$
        \State $\sigma_i \gets \text{std}(r_{i,1}^1, r_{i,1}^2, \dots, r_{i,1}^M)$
        \For{each output $o_{i,j}$}
            \State $A_{i,j} \gets \frac{r_{i,j} - \mu_i}{\sigma_i}$ \Comment{Normalized advantage}
        \EndFor
    \EndFor
    \State Update $\pi_{\theta}$ by maximizing the objective:
    \State Update $\theta$ using gradient ascent
\EndFor
\end{algorithmic}
\end{algorithm}



\section{Experiment}
\label{sec:intro}
% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEbib}
\bibliography{strings,refs}

\end{document}

\begin{thebibliography}{10}

\bibitem{DBLP:journals/csur/ZhangYZ25}
Haopeng Zhang, Philip~S. Yu, and Jiawei Zhang,
\newblock ``A systematic survey of text summarization: From statistical methods to large language models,''
\newblock {\em {ACM} Comput. Surv.}, vol. 57, no. 11, pp. 277:1--277:41, 2025.

\bibitem{DBLP:conf/emnlp/LiuIXWXZ23}
Yang Liu, Dan Iter, Yichong Xu, et~al.,
\newblock ``G-eval: {NLG} evaluation using gpt-4 with better human alignment,''
\newblock in {\em Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, {EMNLP} 2023, Singapore, December 6-10, 2023}, Houda Bouamor, Juan Pino, and Kalika Bali, Eds. 2023, pp. 2511--2522, Association for Computational Linguistics.

\bibitem{abo2023automatic}
H~Abo-Bakr and SA~Mohamed,
\newblock ``Automatic multi-documents text summarization by a large-scale sparse multi-objective optimization algorithm,''
\newblock {\em Complex \& Intelligent Systems}, vol. 9, no. 4, pp. 4629--4644, 2023.

\bibitem{DBLP:conf/emnlp/Zhong0YMJLZJH22}
Ming Zhong, Yang Liu, Da~Yin, et~al.,
\newblock ``Towards a unified multi-dimensional evaluator for text generation,''
\newblock in {\em Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, {EMNLP} 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022}, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, Eds. 2022, pp. 2023--2038, Association for Computational Linguistics.

\bibitem{nallapati-etal-2016-abstractive}
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, et~al.,
\newblock ``Abstractive text summarization using sequence-to-sequence {RNN}s and beyond,''
\newblock in {\em Proceedings of the 20th {SIGNLL} Conference on Computational Natural Language Learning}, Stefan Riezler and Yoav Goldberg, Eds., Berlin, Germany, Aug. 2016, pp. 280--290, Association for Computational Linguistics.

\bibitem{DBLP:journals/corr/abs-2309-09558}
Xiao Pu, Mingqi Gao, and Xiaojun Wan,
\newblock ``Summarization is (almost) dead,''
\newblock {\em CoRR}, vol. abs/2309.09558, 2023.

\bibitem{li2025efficient}
Dapeng Li, Na~Lou, Zhiwei Xu, Bin Zhang, and Guoliang Fan,
\newblock ``Efficient communication in multi-agent reinforcement learning with implicit consensus generation,''
\newblock in {\em Proceedings of the AAAI Conference on Artificial Intelligence}, 2025, vol.~39, pp. 23240--23248.

\bibitem{li2024verco}
Dapeng Li, Hang Dong, Lu~Wang, Bo~Qiao, Si~Qin, Qingwei Lin, Dongmei Zhang, Qi~Zhang, Zhiwei Xu, Bin Zhang, et~al.,
\newblock ``Verco: Learning coordinated verbal communication for multi-agent reinforcement learning,''
\newblock {\em arXiv preprint arXiv:2404.17780}, 2024.

\bibitem{DBLP:conf/acl/YadavGAD20}
Shweta Yadav, Deepak Gupta, Asma~Ben Abacha, et~al.,
\newblock ``Reinforcement learning for abstractive question summarization with question-aware semantic rewards,''
\newblock in {\em Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, {ACL/IJCNLP} 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021}. 2021, pp. 249--255, Association for Computational Linguistics.

\bibitem{DBLP:journals/nle/FikriOY24}
Figen~Beken Fikri, Kemal Oflazer, and Berrin Yanikoglu,
\newblock ``Abstractive summarization with deep reinforcement learning using semantic similarity rewards,''
\newblock {\em Nat. Lang. Eng.}, vol. 30, no. 3, pp. 554--576, 2024.

\bibitem{roit2023factually}
Paul Roit, Johan Ferret, Lior Shani, et~al.,
\newblock ``Factually consistent summarization via reinforcement learning with textual entailment feedback,''
\newblock {\em arXiv preprint arXiv:2306.00186}, 2023.

\bibitem{DBLP:conf/acl/RyuD0LO24}
Sangwon Ryu, Heejin Do, Yunsu Kim, et~al.,
\newblock ``Multi-dimensional optimization for text summarization via reinforcement learning,''
\newblock in {\em Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), {ACL} 2024, Bangkok, Thailand, August 11-16, 2024}, Lun{-}Wei Ku and Andre~Martins andUniEval Vivek~Srikumar, Eds. 2024, pp. 5858--5871, Association for Computational Linguistics.

\bibitem{shao2024deepseekmathpushinglimitsmathematical}
Zhihong Shao, Peiyi Wang, Qihao Zhu, et~al.,
\newblock ``Deepseekmath: Pushing the limits of mathematical reasoning in open language models,'' 2024.

\bibitem{guerreiro2020hypervolume}
Andreia~P Guerreiro, Carlos~M Fonseca, and Lu{\'\i}s Paquete,
\newblock ``The hypervolume indicator: Problems and algorithms,''
\newblock {\em arXiv preprint arXiv:2005.00515}, 2020.

\bibitem{deepseekai2025deepseekr1incentivizingreasoningcapability}
DeepSeek-AI et~al.,
\newblock ``Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,'' 2025.

\bibitem{DBLP:conf/icassp/JungSCR024}
Jee{-}Weon Jung, Roshan~S. Sharma, William Chen, et~al.,
\newblock ``Augsumm: Towards generalizable speech summarization using synthetic labels from large language models,''
\newblock in {\em {IEEE} International Conference on Acoustics, Speech and Signal Processing, {ICASSP} 2024, Seoul, Republic of Korea, April 14-19, 2024}. 2024, pp. 12071--12075, {IEEE}.

\bibitem{das1997closer}
Indraneel Das and John~E Dennis,
\newblock ``A closer look at drawbacks of minimizing weighted sums of objectives for pareto set generation in multicriteria optimization problems,''
\newblock {\em Structural optimization}, vol. 14, no. 1, pp. 63--69, 1997.

\bibitem{DBLP:journals/tec/ZitzlerTLFF03}
Eckart Zitzler, Lothar Thiele, Marco Laumanns, et~al.,
\newblock ``Performance assessment of multiobjective optimizers: an analysis and review,''
\newblock {\em {IEEE} Trans. Evol. Comput.}, vol. 7, no. 2, pp. 117--132, 2003.

\bibitem{yu2025dapo}
Qiying Yu, Zheng Zhang, Ruofei Zhu, et~al.,
\newblock ``Dapo: An open-source llm reinforcement learning system at scale,''
\newblock {\em arXiv preprint arXiv:2503.14476}, 2025.

\bibitem{kornilova-eidelman-2019-billsum}
Anastassia Kornilova and Vladimir Eidelman,
\newblock ``{B}ill{S}um: A corpus for automatic summarization of {US} legislation,''
\newblock in {\em Proceedings of the 2nd Workshop on New Frontiers in Summarization}, Lu~Wang, Jackie Chi~Kit Cheung, Giuseppe Carenini, and Fei Liu, Eds., Hong Kong, China, Nov. 2019, pp. 48--56, Association for Computational Linguistics.

\bibitem{qwen2025qwen25technicalreport}
Qwen, An~Yang, Baosong Yang, and et~al.,
\newblock ``Qwen2.5 technical report,'' 2025.

\end{thebibliography}