Automated Summarization of Software Documents: An LLM-based Multi-Agent Approach
Extracted source text · arXiv 2606.24689
%%%%%%%%%%%%%%%%%%%%%%% file template.tex %%%%%%%%%%%%%%%%%%%%%%%%%
%
% This is a general template file for the LaTeX package SVJour3
% for Springer journals. Springer Heidelberg 2010/09/16
%
% Copy it to a new file with a new name and use it as the basis
% for your article. Delete % signs as needed.
%
% This template includes a few options for different layouts and
% content for various journals. Please consult a previous issue of
% your journal as needed.
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
% First comes an example EPS file -- just ignore it and
% proceed on the \documentclass line
% your LaTeX will extract the file if required
\RequirePackage{fix-cm}
%
%\documentclass{svjour3} % onecolumn (standard format)
%\documentclass[smallcondensed]{svjour3} % onecolumn (ditto)
\documentclass[smallextended]{svjour3} % onecolumn (second format)
%\documentclass[twocolumn]{svjour3} % twocolumn
%
\smartqed % flush right qed marks, e.g. at end of proof
%
\usepackage{comment}
%\usepackage{graphicx}
\usepackage[graphicx]{realboxes}
\usepackage{url}
%\usepackage{subfigure}
\usepackage{tcolorbox}
\usepackage{multirow}
\usepackage{amssymb,amsfonts}
\usepackage{pifont}%for \xmark and \cmark
\usepackage{xcolor,colortbl} %for \rowcolor{}
\usepackage{listings}
\usepackage{hyperref}
\usepackage[figuresright]{rotating}
\usepackage{rotating}
\usepackage{pdflscape}
\hypersetup{colorlinks,breaklinks,
linkcolor=darkblue,urlcolor=darkblue,
anchorcolor=darkblue,citecolor=darkblue}
\usepackage{natbib}
\setcitestyle{authoryear,open={(},close={)}}
\usepackage{marvosym}
\usepackage{algorithm}
\usepackage{amsmath}
\usepackage[noend]{algpseudocode}
%\usepackage{subcaption}
\usepackage[tight,footnotesize]{subfigure}
\usepackage{xcolor, soul}
\colorlet{lightgray}{gray!30}
\sethlcolor{lightgray} %change the highlight color
\definecolor{darkgray}{gray}{0.78}
\definecolor{lightgray}{gray}{0.85}
\definecolor{verylightgray}{gray}{0.95}
\definecolor{lightgray}{gray}{0.85}
\definecolor{verylightgray}{gray}{0.95}
\definecolor{darkblue}{rgb}{0.1,0.6,1.0}
\definecolor{red}{rgb}{1.0, 0.01, 0.24}
\definecolor{gray}{rgb}{0.8, 0.8, 0.8}
\newcommand\revised[1]{\textcolor{blue}{#1}}
\newcommand\rev[1]{\textcolor{black}{#1}}
\newcommand\tick{\ding{52}}
\makeatletter
\def\BState{\State\hskip-\ALG@thistlm}
\makeatother
%
% \usepackage{mathptmx} % use Times fonts if available on your TeX system
%
% insert here the call for the packages your document requires
%\usepackage{latexsym}
% etc.
%
% please place your own definitions here and don't use \def but
% \newcommand{}{}
%
% Insert the name of "your journal" with
% \journalname{myjournal}
%
\input{commands}
\begin{document}
\title{\ME: An LLMs-based Multi-Agent System for the Summarization of \GH \RM Files}
\title{\ME: Empowering the Synergy Among LLMs for \GH \RM Summarization}
\title{\ME: Augmenting the Synergy Among LLMs for \GH \RM Summarization}
\title{\ME: Augmenting the Synergy Among Large Language Models for \RM Summarization}
\title{Turning \RM files into About Descriptions: An Approach with LLMs-based Multi-Agent Systems}
\title{Turning \RM files into About Descriptions: An LLMs-based Multi-Agent Approach}
\title{Turning Long \RM Files into Short About Descriptions: An LLMs-based Multi-Agent Approach}
\title{Automated Summarization of Software Documents: An LLM-based Multi-Agent Approach}
\titlerunning{An LLM-based MAS for the Summarization of Software Documents} % if too long for running head
\author{Duc S. H. Nguyen \and Minh T. Nguyen \and Phuong T. Nguyen \and Juri Di Rocco \and Davide Di Ruscio}
%<paola.inverardi@univaq.it>
%<patrizio.migliarini@graduate.univaq.it>
%\authorrunning{Short form of author list} % if too long for running head
%\institute{Davide Di Ruscio, Paola Inverardi, Patrizio Migliarini, Phuong T. Nguyen \at
% DISIM - University of L'Aquila (Italy)\\
% \email{firstname.lastname@univaq.it}
\institute{
Duc S. H. Nguyen \at
Hanoi University of Science and Technology, Vietnam \\ %67100 --
\email{duc.nsh231061m@sis.hust.edu.vn}
\and
Minh T. Nguyen \at
Hanoi University of Science and Technology, Vietnam \\ %67100 --
\email{minh.nt225450@sis.hust.edu.vn}
\and
Phuong T. Nguyen \at
University of L'Aquila, Italy \\ %67100 --
\email{phuong.nguyen@univaq.it}
\and
Juri Di Rocco \at
University of L'Aquila, Italy \\ %67100 --
\email{juri.dirocco@univaq.it}
\and
\Letter~Davide Di Ruscio \at
University of L'Aquila, Italy \\ %67100 --
% University of L'Aquila, Via Vetoio 2, L'Aquila, Italy \\ %67100 --
% Tel.: +39-0862-433735\\
%Fax: +123-45-678910\\
\email{davide.diruscio@univaq.it}
}
% \\
% \emph{Present address:} of F. Author % if needed
\maketitle
\begin{abstract}
Large Language Models (LLMs) and LLM-based Multi-Agent Systems (MAS) are revolutionizing software engineering (SE) by advancing automation, decision-making, and knowledge processing. Their recent application to SE tasks has already shown promising results. In this paper, we focus on summarization as a key application area. We present \ME, an LLM-based MAS designed to generate concise and accurate summaries of software documentation.
\ME employs a Teacher–Student architecture where multiple LLM agents collaborate to enhance relevance and precision of produced summaries. An empirical evaluation on real-world datasets demonstrates \ME's effectiveness in streamlining workflows, outperforming the considered baselines. The evaluation provides evidence that \ME improves summarization for requirements analysis and technical documentation.
Our findings underscore the transformative potential of these technologies in SE, while identifying challenges and future research directions for their seamless integration.
%
\begin{comment}
Large Language Models (LLMs) and LLMs-based Multi-Agent Systems are transforming software engineering (SE) by enhancing automation, decision-making, and knowledge processing.
This paper explores their applications in SE, focusing on their role in %code generation, bug detection, software testing, and
summarization tasks. In particular, we highlight the capability of LLMs to generate concise and accurate summaries of complex software documentation, %, requirements, and codebases,
thereby improving developer productivity and collaboration. %In this paper,
We developd \ME--an LLMs-based Multi Agent Systems for the
% introduce a novel approach to
summarization of software documents. Built on top of various LLMs, \ME is able to increase the synergies among them, making the recommendations more relevant. %By leveraging LLMs-based Multi-Agent Systems, we
An empirical evaluation using real-world datasets has been conducted to study \ME's performance. The evaluation demonstrates how collaborative agents can streamline workflows, facilitate real-time problem-solving, and enhance summarization for requirements analysis and technical documentation. Our findings underscore the potential of these technologies to revolutionize SE practices while identifying challenges and future research directions for their effective integration.
\end{comment}
\begin{comment}
Repositories in \GH are equipped with
%In each repository, \GH allows it users to create
\RM files %as a means to demonstrate
to expose their main functionalities to the public.
%to help \GH visitors to understand the main functionalities.
%Interestingly,
Nonetheless, while providing practical information, \RM files are often verbose, necessitating time and effort to read and understand. To this end, \GH enables its users to add a short description called ``About'', providing %visitors with
a brief but informative summary about a repository. %This helps visitors to quickly comprehend the main content, and decide whether to continue reading.
However, this field is usually overlooked and left blank, impeding reading comprehension. In this respect, the automatic summarization of \RM to yield the ``About'' description is of great help. %a short but informative description is useful
%as it helps developers capture the intention of the repository.
%However, due to various reasons--not excluding laziness and tiredness--this field is usually left unfilled.
Large Language Models (LLMs) have demonstrated their ability in different application domains. The combination of various LLMs ... agentic LLMs have become mainstream in the recent months.
This paper introduces \ME--a practical approach %workable solution
to the summarization of \RM. %, automatically filling the blank ``About'' field for repositories. %Starting from a set of \RM,
\end{comment}
%\GS is built on top of BART and T5, two cutting-edge deep learning techniques, learning from existing data to perform recommendations for repositories with a missing description. We test its performance using two datasets collected from \GH. The evaluation %experimental results
%shows that \GS can generate relevant predictions, outperforming a well-established baseline.
\keywords{Summarization \and LLMs \and Multi-Agent Systems \and GitHub \and Google Play}
% \PACS{PACS code1 \and PACS code2 \and more}
% \subclass{MSC code1 \and MSC code2 \and more}
\end{abstract}
\section{Introduction}
\label{sec:Introduction}
\input{src/Introduction}
\section{Motivation and Background}
% \section{\revised{Motivating example}}
\label{sec:Background}
\input{src/Background}
% \section{Existing studies and the current research gap} \label{sec:LiteratureReview}
% \input{src/LiteratureReview}
%
\section{Proposed Approach}
\label{sec:Proposed_Approach}
\input{src/Proposed_Approach}
\section{Evaluation}
\label{sec:Evaluation}
\input{src/Evaluation}
\section{Experimental Results}
\label{sec:Results}
\input{src/Results}
%
\section{Discussion}
\label{sec:Discussion}
\input{src/Discussion}
\section{Related Work}
\label{sec:RelatedWork}
\input{src/RelatedWork.tex}
%
%
\section{Conclusion and Future Work}
\label{sec:Conclusions}
\input{src/Conclusions}
%
% \section{Proof of Concept}
% \subsection{Research Questions}
% \begin{itemize}
% \item \textbf{RQ1:} How does a dynamic iteration strategy compare to a fixed-loop (non-dynamic) strategy in terms of summarization quality and training efficiency?
% \item \textbf{RQ2:} How does the generated "About" text perform across different test subsets in terms of semantic alignment and content overlap?
% \end{itemize}
% \subsection{Data Overview and Selection Criteria}
% \input{src/DataOverview}
% \section{Results and Discussion}
% \subsection{Comparison Between Dynamic and Non-Dynamic Strategies}
% \input{src/Comparison_Strategies}
% \begin{mdframed}
% \textit{\textbf{Answer to RQ1}:} The dynamic method creates summaries that are nearly as good as the non-dynamic one, but it needs much fewer training steps (260 vs. 620). This means it saves time and resources while still keeping quality, making it a more efficient option.
% \end{mdframed}
%
% \subsection{Evaluation Summary Across Test Sets}
% \input{src/TestSet_Evaluation}
%
% \begin{mdframed}
% \textit{\textbf{Answer to RQ2}:} The generated summaries work well across different test sets. They are often close in meaning to the original app descriptions and include the right content. This shows that the system can create accurate and useful "About" texts even with different types of input.
% \end{mdframed}
\section*{Declarations}
\textbf{Conflict of interest.} All the authors declare that they have no conflict of interest. Furthermore, they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
\vspace{.2cm}
\noindent
\textbf{Ethical approval.} This article does not contain any studies with human participants or animals performed by any of the authors.
% \setcounter{tocdepth}{2}
% \tableofcontents
% \section{Background}
% \label{sec:Motivation}
% \input{src/Context}
%
% \section{The CROSSMINER project}
% \label{sec:CROSSMINER}
% \input{src/Crossminer}
%
%
%
%
% \section{Challenges and lessons learned from eliciting the requirements of the \projectName \rs}
% \label{sec:requirements}
% \input{src/requirements}
%
%
% \section{Challenges and lessons learned from developing the \projectName \rs}
% \label{sec:DevelopingRs}
% \input{src/Developing}
%
%
% \section{Challenges and lessons learned from the evaluation of the \projectName \rs}
% \label{sec:evaluation}
% \input{src/evaluation}
% \section{Lessons learned}
% \label{sec:Lessons}
% \input{src/Lessons}
% \section{Assessing the quality of recommendation system}
% \label{sec:Evaluation}
% \section{Related work}
% \label{sec:RelatedWork}
% \input{src/RelatedWork.tex}
%
% \section{Conclusion and future work}
% \label{sec:Conclusions}
% \input{src/Conclusions}
\begin{acknowledgements}
This paper has been partially supported by the MOSAICO project (Management, Orchestration and Supervision of AI-agent COmmunities for reliable AI in software engineering) that has received funding from the European Union under the Horizon Research and Innovation Action (Grant Agreement No. 101189664). %The work has been partially supported by the EMELIOT national research project, which has been funded by the MUR under the PRIN 2020 program (Contract 2020W3A5FY).
The work has been also partially supported by the European Union--NextGenerationEU through the Italian Ministry of University and Research, Projects PRIN 2022 PNRR \emph{``FRINGE: context-aware FaiRness engineerING in complex software systEms''} grant n. P2022553SL. We acknowledge the Italian ``PRIN 2022'' project TRex-SE: \emph{``Trustworthy Recommenders for Software Engineers,''} grant n. 2022LKJWHC.
% The research described in this paper has been carried out as part of the
% CROSSMINER Project, which has received funding from the European Union's
% Horizon 2020 Research and Innovation Programme under Grant 732223
% The research described in this paper has been carried out as part of the ... %CROSSMINER Project, which has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant 732223. We thank the anonymous reviewers for their valuable comments and suggestions that helped us improve the paper.
\end{acknowledgements}
% Authors must disclose all relationships or interests that
% could have direct or potential influence or impart bias on
% the work:
%
% \section*{Conflict of interest}
%
% The authors declare that they have no conflict of interest.
% BibTeX users please use one of
%\bibliographystyle{spbasic} % basic style, author-year citations
% \bibliographystyle{spmpsci} % mathematics and physical sciences
\bibliographystyle{abbrvnat}
%\bibliographystyle{spphys} % APS-like style for physics
%\bibliography{} % name your BibTeX data base
\bibliography{main}
\end{document}
% Usual suspects
\usepackage{xspace}
\usepackage{amssymb}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage{xcolor}
\newcommand{\nb}[2]{
\fbox{\bfseries\sffamily\scriptsize#1}
{\sf$\blacktriangleright$\textit{#2}$\blacktriangleleft$}
}
\newcommand*{\ie}{i.e.,\@\xspace}
\newcommand*{\eg}{e.g.,\@\xspace}
\newcommand*{\cf}{cf.\@\xspace}
\newcommand*{\RM}{README\@\xspace}
%\newcommand*{\RM}{README.MD\@\xspace}
\newcommand*{\RMs}{README.MDs\@\xspace}
\newcommand*{\SO}{StackOverflow\@\xspace}
\newcommand*{\GH}{GitHub\@\xspace}
\newcommand*{\ME}{\texttt{Metagente}\@\xspace}
\newcommand*{\GS}{\texttt{GitSum}\@\xspace}
\newcommand{\shaded}[1]{\texttt{\hl{\small #1}}}
\newcommand*{\AL}{\texttt{Analysis Agent}\@\xspace}
\newcommand*{\CA}{\texttt{Prompt Creator Agent}\@\xspace}
\newcommand*{\EA}{\texttt{Extractor Agent}\@\xspace}
\newcommand*{\SA}{\texttt{Summarizer Agent}\@\xspace}
\newcommand*{\TA}{\texttt{Teacher Agent}\@\xspace}
\newcommand*{\LLa}{\texttt{LLaMA-2}\@\xspace}
\newcommand*{\Fo}{\texttt{GPT-4o}\@\xspace}
\renewcommand\rev[1]{\textcolor{black}{#1}}
\makeatletter
\newcommand*{\etc}{%
\@ifnextchar{.}%
{etc}%
{etc.\@\xspace}%
}
\makeatother
\newcommand*{\etal}{\emph{et~al.}\@\xspace}
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%
%\newcommand{\nb}[2]{\fbox{\bfseries\sffamily\scriptsize#1}{\sf\small$\blacktriangleright$\textit{#2}$\blacktriangleleft$}}
%\newcommand{\nb}[2]{\fbox{\bfseries\sffamily\scriptsize#1} {\sf\small
%\textit{#2} }}
\newcommand*\circled[1]{\tikz[baseline=(char.base)]{\color{black}
\node[shape=circle,draw=cyan,fill=black!10!white,inner sep=.3pt] (char)
{{{\texttt\textbf #1}}};}}
\def\checkmark{\tikz\fill[scale=0.4](0,.35) -- (.25,0) -- (1,.7) -- (.25,.15)
-- cycle;}
\newcommand\PN[1]{\textcolor{blue}{\nb{Phuong}{#1}}}
\newcommand\DDR[1]{\textcolor{red}{\nb{DDR} {#1}}}
\newcommand\JDR[1]{\textcolor{purple}{\nb{Juri} {#1}}}
%\newcommand\revised[1]{\textcolor{blue}{#1}}
\newcommand{\code}[1]{{\texttt{#1}}}
\newcommand*{\rs}{recommendation systems\@\xspace}
\newcommand{\challenge}[2]{
\begin{tcolorbox}[boxrule=0.5pt]
{\bfseries #1}: {#2}
\end{tcolorbox}
}
\definecolor{darkgray}{gray}{0.78}
\definecolor{lightgray}{gray}{0.85}
\definecolor{verylightgray}{gray}{0.95}
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
\makeatletter
\p@=1pt
\makeatother
%
%
%\lstset{numbers=left,
% inputencoding=latin1,
% basicstyle=\footnotesize\ttfamily,
% keywordstyle=\color{blue},
% breaklines=true,
% showtabs=false,
% showstringspaces=false,
% numberstyle=\tiny\color{gray}
%}
\newcommand{\mybox}[4]{
\begin{figure}[h]
\centering
\begin{tikzpicture}%[boxrule=0.86pt,left=0.3em, right=0.3em,top=0.1em, bottom=0.05em]
\node[anchor=text,text width=\columnwidth-0.5cm, draw, rounded corners, line width=1pt, fill=#3, inner sep=2mm] (big) {\\#4};
\node[draw, rounded corners, line width=.3pt, fill=#2, anchor=west, xshift=3mm] (small) at (big.north west) {#1};
\end{tikzpicture}
\end{figure}
}
%Code listing style named "mystyle"
\lstdefinestyle{java}{
backgroundcolor=\color{backcolour}, commentstyle=\color{codegreen},
keywordstyle=\color{magenta},
numberstyle=\tiny\color{codegray},
stringstyle=\color{codepurple},
basicstyle=\ttfamily\scriptsize,
breakatwhitespace=false,
breaklines=true,
captionpos=b,
keepspaces=true,
numbers=left,
numbersep=5pt,
showspaces=false,
showstringspaces=false,
showtabs=false,
tabsize=2
}
%"mystyle" code listing set
\lstset{style=java}
%\item \textbf{RQ1:} How does a dynamic iteration strategy compare to a fixed-loop (non-dynamic) strategy in terms of summarization quality and training efficiency?
%\item \textbf{RQ2:} How does the generated "About" text perform across different test subsets in terms of semantic alignment and content overlap?
\newcommand{\rqfirst}{\textbf{RQ$_1$}: \emph{How does \ME perform compared to single LLM-based agents?}} %Can we trust the self assessment given by users
\newcommand{\rqsecond}{\textbf{RQ$_2$}: \emph{Does the dynamic iteration strategy contribute to %to a fixed-loop (non-dynamic) strategy in terms of summarization quality and
training efficiency?}} %Can we trust the self assessment given by users
%\newcommand{\rqthird}{\textbf{RQ$_3$}: \emph{How does the generated ``About'' text perform across different test subsets in terms of semantic alignment and content overlap?}}
\newcommand{\rqthird}{\textbf{RQ$_3$}: \emph{Which LLM contributes to a better performance of \ME?}}
%\newcommand{\rqfourth}{\textbf{RQ$_4$}: \emph{How does the self-improvement process evolve across the dataset?}} % with supervised classification
%How does \tool succeed in terms of accuracy?
%\newcommand{\rqsecond}{\textbf{RQ$_2$}: \emph{Which set of features contributes to a better characterization of privacy profiles?}} %Which set of privacy is the best for clustering user profiles?
%\newcommand{\rqthird}{\textbf{RQ$_3$}: \emph{How well can we recommend privacy settings to users, given that they already specified some of their preferences?}} %Can we trust the self assessment given by users
%\newcommand{\rqthird}{\textbf{RQ$_3$}: \emph{How well can we predict the category for a user based on their settings with supervised classification?}} %Can we trust the self assessment given by users
\begin{comment}
In recent years, \GH %is the largest open source software store, %
has become one of the most popular open source platforms, offering--free of charge--%allowing developers to approach and reuse
a large number of %code %open source
repositories of code written in several programming language%in which the majority of the software repositories are publicly shared with the world,
~\citep{BORGES2018112}.
The ability to allow
%\GH has gained traction thanks to a collaborative development environment that allows
developers to work %code and collaborate
in a collaborative %development
environment is among the key factors that make %. The ability to allow developers to share their source code and collaborate on software projects has made
\GH become favoured %favorable %one of the most favored platforms
by software developers. \GH equips developers with different utilities, enabling them to seamlessly manage and work with repositories.
%To understand a repository in GitHub,
%Each \GH repository has one or more
\end{comment}
In Software Engineering, there has been an increasing use of documents to provide detailed descriptions for software artifacts. Among others, \RM files in \GH are written in the Markdown format, containing information about a repository, \eg instructions, help, or
updates. A well-written \RM facilitates reading comprehension, helping visitors to grasp the scope of a repository.
This is also the case of mobile app stores like Google Play Store,\footnote{\url{https://play.google.com/store/apps}} in which apps are normally equipped with a long HTML document, allowing visitors to understand the apps' functionalities.
An empirical study~\citep{LIU2022106924} showed that \RM is often the very first item that visitors take a look at
when it comes to becoming acquainted with a \GH repository. Similarly, apps' descriptions are of high importance, as they can be used as sources to mine domain knowledge~\citep{LIU2017126}.
By several \GH repositories and Google Play apps, instruction files are usually lengthy, and reading and understanding them can
require time and effort, thus discouraging visitors from continuing with the repositories/apps.
Therefore, apart from \RM, \GH also allows its users to add a short description called ``About'' to each repository, offering a succinct summary of the main functionalities. In a similar manner, besides a full \RM file, apps in Google Play Store feature a brief description, resembling the ``About'' summary in \GH repositories.
Unfortunately, in these platforms, while being useful, this field is usually overlooked and left unfilled, posing difficulties for those who want to explore the repositories, on the fly. Recently, \GS~\citep{10.1145/3593434.3593448} has been proposed as the first approach to summarize long \RM files to yield brief but concise ``About'' descriptions. Being built on top of BART and T5, \GS successfully produces relevant summaries starting from long \RM files. However, while obtaining an encouraging recommendation performance, the tool still suffers from a low accuracy for input files that are of mixed fields, including code and text. This triggers the need for a more effective way to summarize \RM files.
Large Language Models (LLMs) have been %widely
applied in %various application domains, including
Software Engineering~\citep{DBLP:journals/software/Ozkaya23b,NGUYEN2024112059} to solve a wide range of tasks. To name but a few, researchers have employed LLMs %have been used
in code summarization~\citep{sun2024source,haldar2024analyzing}, debugging~\citep{lee2024github,tian2024debugbench}, testing~\citep{arawjo2024chainforge,li2025enhancing}, or %most notably,
code generation~\citep{gu2023llm,fakhoury2024llm,huang2024bias}.
However, LLMs are not without limitations: Existing work~\citep{He_Treude_Lo_2024} has revealed that while they are gaining popularity, their performance on tasks requiring domain-specific expertise or complex multi-step reasoning remains limited. In this context, prompt-tuning is a practical approach, in which prompts are iteratively refined to improve the performance of the pre-trained language model without modifying its internal design. Noteworthy, the prompt-tuning process can be prone to subjective bias and scalability issues, making it difficult to generalize across diverse tasks \citep{white2023promptpatterncatalogenhance}. Moreover, it is not easily accessible for several developers as it might require advanced technical expertise. %Still, this requires access via APIs, making it become less accessible for several developers. \DDR{The last sentence is not clear. The API access is required also in the case of MAS. Maybe I missing some point here. I would say instead that the "prompt-tuning process is not easily accessible for several developers as it might require advanced technical expertise", or as it was written in previous text something like "the prompt-tuning process can be prone to subjective bias and scalability issues, making it difficult to generalize across diverse tasks \citep{white2023promptpatterncatalogenhance}"}
\begin{comment}
Furthermore, manual prompt engineering techniques, such as Chain-of-Thought (CoT) reasoning, require significant human effort to refine prompts iteratively. This labor-intensive process is prone to subjective bias and scalability issues, making it difficult to generalize across diverse tasks \citep{white2023promptpatterncatalogenhance}.
\end{comment}
To cope with the limitations of single LLMs,
multi-agent systems have been proposed to enable specialized LLMs to collaborate within a shared framework~\citep{He_Treude_Lo_2024,DBLP:journals/fcsc/WangMFZYZCTCLZWW24}. These systems magnify the unique strengths of various LLMs, where agents specialize in tasks such as code generation, debugging, or domain-specific problem-solving \citep{He_Treude_Lo_2024,DBLP:journals/corr/abs-2407-01489}. Challenges such as effective coordination, efficient communication, and the overhead of integrating multiple agents persist, even though dedicated frameworks like \texttt{LangChain}\footnote{\url{https://www.langchain.com/}} or \texttt{LLamaIndex}\footnote{\url{https://www.llamaindex.ai/}} are now increasingly adopted in Software Engineering to mitigate such issues. Nevertheless, MASs represent a compelling alternative to relying solely on the capabilities of individual models~\citep{10.1145/3691620.3695291}.
In our previous work \citep{10.1145/3696630.3728511}, we developed %introduced
\ME, a multi-agent framework composed of four LLM-based agents, which %: the \EA selects relevant content, the \SA generates candidate summaries, the \TA evaluates outputs and suggests prompt refinements, and the \CA synthesizes generalizable prompts. These agents
interact in a collaborative manner using a teacher-student loop, enabling prompt optimization with minimal supervision.
An evaluation using datasets collected from \GH demonstrated that \ME obtains a good recommendation performance, showing that the proposed approach is suitable for generating summaries for software documents.
In this paper, we enhance \ME across several dimensions. \emph{First}, to improve efficiency, we propose a dynamic iteration strategy that adaptively halts processing of low-potential samples, reducing computational cost without compromising quality. \emph{Second}, we extend the framework to incorporate multiple LLMs as the recommendation engine, enhancing its flexibility. \emph{Third}, we broaden the evaluation to include summarization of structured HTML-based application descriptions into concise texts. This task is particularly challenging due to the heterogeneous content of HTML files, which can complicate summarization. Our evaluation on curated datasets demonstrates that \ME achieves strong semantic alignment and competitive ROUGE scores while requiring fewer training iterations, outperforming various baselines.
In summary, the main contributions of this work include the novel dynamic iteration strategy, the integration of diverse LLMs, and the expanded evaluation scope for complex summarization tasks, elaborated as follows.
\begin{comment}
In this paper, we further improve \ME in different aspects. \emph{First}, aiming for efficiency, we propose a \emph{dynamic iteration strategy} that adaptively halts low-potential samples, reducing computation without sacrificing quality. \emph{Second}, the framework has been extended to incorporate different LLMs as the recommendation engine. \emph{Third}, we expanded the evaluation to summarization of %explore this potential in the context of summarizing
structured HTML-based app descriptions into concise summarized texts. Such type of task is particularly challenging, as the heterogeneous content of HMTL files may render the summarization cumbersome. %This type of data poses significant difficulties for summarization tools. %(``About'').
Our evaluation on curated datasets shows that \ME achieves strong semantic alignment and competitive ROUGE scores, while requiring fewer training iterations, %than a fixed-loop baseline.
outperforming various baselines.
\end{comment}
%The main contributions of this work are summarized as follows:
\begin{itemize}
\item \textbf{Solution.} We introduced an extended version of \ME, a collaborative LLM-based multi-agent system for summarizing app descriptions. Aiming for efficiency, a dynamic stopping mechanism has been deployed, so as to optimize the fine-tuning process. % to enhance training efficiency.
% \item \textbf{Efficiency.} We introduce a dynamic stopping mechanism to enhance training efficiency.
\item \textbf{Evaluation.} Using datasets collected for mobile apps' documents, we conducted an empirical evaluation to study the performance of \ME. Moreover, we also compared dynamic vs. non-dynamic strategies, evaluating generalization across test subsets.
\item \textbf{Reproducibility.} The replication package including code, data, and prompts has been released to foster reproducibility.\footnote{\url{https://github.com/MDEGroup/Metagente}}
\end{itemize}
\noindent
\textbf{Structure.} Section~\ref{sec:Background} provides some background related to the importance of \RM and ``About'' descriptions in %\GH and
Google Play. Section~\ref{sec:Proposed_Approach} introduces \ME, our proposed approach to the summarization of \RM files using LLM-based MAS. The evaluation is elaborated in Section~\ref{sec:Evaluation}, and the results are reported and analyzed in Section~\ref{sec:Results}. In Section~\ref{sec:Discussion}, there are discussion on the impacts, and the threats to the validity of the findings. We review the related work in Section~\ref{sec:RelatedWork}. Finally, Section~\ref{sec:Conclusions} sketches future work, and concludes the paper.
%Background goes here.
Section~\ref{sec:MotivatingExample} presents a motivating example to demonstrate the necessity of ``About'' descriptions in the Google Play ecosystem. Afterwards, the ROUGE scores are presented in Section~\ref{sec:Metrics} as a base for the presentation of \ME in Section~\ref{sec:Proposed_Approach}.
\subsection{Motivating Examples}
\label{sec:MotivatingExample}
\begin{comment}
By large-scale projects, where there are several software artifacts and components, %\RM needs to cover a large amount of %convey the containing
the amount of information that a
\RM files in \GH, while providing useful information, are usually lengthy and require effort to read and comprehend. %needs to convey is large, and thus making it become very lengthy, requiring much efforts to read and understand. Altogether,
This may discourage visitors from continuing with the repository. Fortunately, besides \RM, \GH equips a brief summary called ``About'' to each repository, providing visitors with
a comprehensive recap of the repository's main functionalities. %This helps visitors to quickly become acquainted with the repository.
\end{comment}
\begin{figure*}[t!]
\centering
\begin{tabular}{c c}
\subfigure[Front page]{\label{fig:Telegram-Front}
\includegraphics[width=0.35\linewidth]{figs/Example_Telegram_Front.jpg}} &
\subfigure[README]{\label{fig:Telegram-Back}
\includegraphics[width=0.35\linewidth]{figs/Example_Telegram_Back.jpg}}
\end{tabular}
% \vspace{-.2cm}
\caption{\texttt{Telegram} features a brief description for the app in its front page.}
\label{fig:Telegram}
% \vspace{-.2cm}
\end{figure*}
Figure~\ref{fig:Telegram-Front} shows %an example with
the \texttt{Telegram} app, seen from the frontpage on Google Play Store. %\footnote{PowerToys is a project developed by Microsoft \url{https://github.com/microsoft/PowerToys}}
Apart from essential information including the number of reviews (15M with 4.0 as rating), number of downloads ($>$1B), %tags (\eg \texttt{windows}, \texttt{color-picker}, \texttt{desktop}),
%number of forks (7.1k), number of stars (120k),
there is also a short description, which is actually the ``About'' field, describing the main functionalities, \ie ``\emph{Telegram is a messaging app with focus on speed and security}.'' Such a description is brief but informative, and it helps %Thanks to this,
users--especially first-time visitors--gain a quick orientation to decide whether to continue with the app, without reading the corresponding long \RM, which resides behind the frontpage shown in Figure~\ref{fig:Telegram-Back}. %
\begin{comment}
\begin{figure}[t!]
\centering
\includegraphics[width=0.68\columnwidth]{figs/PowerToys.png}
% \vspace{-.2cm}
\caption{The \texttt{PowerToys} repository.}
\label{fig:PowerToys}
\vspace{-.2cm}
\end{figure}
\end{comment}
\begin{figure*}[t!]
\centering
\begin{tabular}{c c}
\subfigure[Front page]{\label{fig:Courtside1891-Front}
\includegraphics[width=0.35\linewidth]{figs/Example_Courtside1891_Front.jpg}} &
\subfigure[README]{\label{fig:Courtside1891-Back}
\includegraphics[width=0.35\linewidth]{figs/Example_Courtside1891_Back.jpg}}
\end{tabular}
% \vspace{-.2cm}
\caption{Despite a lengthy \RM file, \texttt{Courtside 1891} does not offer any informative descriptions.}
\label{fig:Courtside1891}
% \vspace{-.2cm}
\end{figure*}
%The repository in
Figure~\ref{fig:Courtside1891} depicts another example of %the starting page of
%a repository named
\texttt{Courtside 1891}\footnote{\url{https://play.google.com/store/apps/details?id=com.pl.fiba&hl=en}} that is also a well-maintained app. However, while having a long and detailed \RM (see Figure~\ref{fig:Courtside1891-Back}), the app does not feature any short %``About''
description (see Figure~\ref{fig:Courtside1891-Front}). Due to such a lack, visitors need to spend time reading the \RM in order to understand the app's functionalities. Essentially--in some extent--%certain cases,
this may discourage visitors from continuing with the app.
The examples motivate us to come up with an approach to summarize long \RM documents to yield a short description. In our previous work~\citep{10.1145/3593434.3593448}, we conceived \GS as a first approach to translate long \GH \RM files to obtain ``About'' descriptions. Using BART and T5 as the recommendation engine, \GS can generate
relevant summaries for \GH repositories.
However, while earning a high accuracy in different testing samples, \GS still struggles with input files composed of miscellaneous fields, including code and text.
%
In this paper, we aim to overcome the current limitations, leveraging the synergies of different task-specific LLMs that are properly orchestrated to enhance the recommendation quality. %Developing a multi-agent system on top of
%Using a series of LLMs, %large language models (LLMs),
In particular, we employ an LLM to perform the extraction of the input data, filtering out noisy fields and retaining the most meaningful ones. More importantly, by means of a Teacher-Student architecture applying collaborative LLMs, we iteratively refine a set of prompts to produce the best fit prompt, which then can be used to generate the final summary. %the synergies among different LLMs so as to optimize the summaries.
%Essentially, the LLMs-based agents work in a collaborative manner %with each other
%to perform a common task, with the aim of refining the summaries.
The next subsection describes the metrics used by \ME to steer the adjustment phase of LLMs-based agents.
%Summarization of \GH \RM.
%A short description is useful as it helps developers quickly comprehend the intent of the repository.
%https://github.com/google/tcmalloc
%https://github.com/google/fuzztest
%https://github.com/google/crubit
%https://github.com/google/budoux
\begin{comment}
\begin{figure}[t!]
\centering
\includegraphics[width=1.0\columnwidth]{figs/silifuzz.png}
% \vspace{-.2cm}
\caption{The \texttt{silifuzz} repository without any About description.}
\label{fig:silifuzz}
\vspace{-.2cm}
\end{figure}
\end{comment}
\subsection{Metrics for Evaluating Summarization Tasks}\label{sec:Metrics}
%, are used to evaluate \ME, as they
%The ROUGE metrics \citep{rouge2004} including ROUGE-1, ROUGE-2, ROUGE-L have been widely used to evaluate the in text summarization tasks \citep{Chen2020,itiger2022}
Recall-Oriented Understudy for Gisting Evaluation (ROUGE) \citep{rouge2004} is a set of metrics applied in evaluating automatic summarization and machine translation %software.
%in natural language processing (NLP)
\citep{Chen2020,itiger2022}. Given a pair of a produced summary %or translation
and a ground-truth text, ROUGE metrics judge %compare to judge
their similarity based on the overlap between them. %The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation.
ROUGE metrics are in the range of 0 and 1, in which a higher value corresponds to a higher similarity between the automatically produced summary and the reference one.
The metrics are computed as follows: %. The precision, recall, and F1-score for ROUGE-N are calculated as follows:
\vspace{-.1cm}
\begin{equation*}
% \begin{split}
% Precision_{rouge-n}=\frac{count(overlapped\_N\_grams)}{count(N\_grams \in ref\_summary)} \\
% Recall_{rouge-n}=\frac{count(overlapped\_N\_grams)}{count(N\_grams \in gen\_summary)} \\
% F1_{rouge-n}=2*\frac{Precision_{rouge-n} * Recall_{rouge-n}}{Precision_{rouge-n} + Recall_{rouge-n}}
% \end{split}
\begin{aligned}
Precision_{rouge-n} =&\frac{ \sum_{R,G \in S} \sum_{gram_n \in R} Count_G(gram_n) }{ \sum_{R,G \in S} \sum_{gram_n \in G} Count_G(gram_n) } \\
Recall_{rouge-n}=&\frac{ \sum_{R,G \in S} \sum_{gram_n \in R} Count_G(gram_n) }{ \sum_{R,G \in S} \sum_{gram_n \in R} Count_R(gram_n) } \\
F1_{rouge-n}=&\frac{2* Precision_{rouge-n} * Recall_{rouge-n}}{Precision_{rouge-n} + Recall_{rouge-n}}
\end{aligned}
\end{equation*}
%\vspace{-.4cm}
where $R$, $G$, and $S$ are the reference summary, generated summary, and the test set, respectively. $gram_n$ is an n-gram phase, where $n$ is the length of a word sequence; $Count_R(gram_n)$ and $Count_G(gram_n)$ are the occurrence number of $gram_n$ in $R$ and $G$. %, respectively.
% We perform these above equations with $N=1,2$ to
%We compute
ROUGE-1 and ROUGE-2 are calculated with $N=1,2$, %corresponding to
\ie uni-gram and bi-grams. $\sum_{gram_n \in R} Count_G(gram_n)$ represents the number of N-grams existing in both the ground-truth summary and generated summary. The aforementioned equations are interpreted as follows. Given two sets of N-grams produced from a generated summary and its original summary, respectively, Precision evaluates the proportion of N-grams in the first set that is found in the second set, while Recall measures the proportion of N-grams in the second set that exists in the first set. %For a more comprehensive assessment,
%As F1-score considers both Precision and Recall, we use it as the main metric in the evaluation.
%Slightly different from ROUGE-N, ROUGE-L
ROUGE F1-score is based on Longest Common Subsequence (LCS) defined as follows: %, given below:
\begin{equation*}
\begin{aligned}
Precision_{rouge-l}=&\frac{LCS(R,G)}{length(G)}; Recall_{rouge-l}=&\frac{LCS(R,G)}{length(R)} \\
F1_{rouge-l}=&\frac{2* Precision_{rouge-l} * Recall_{rouge-l}}{Precision_{rouge-l} + Recall_{rouge-l}}
\end{aligned}
\end{equation*}
where $LCS(R,G)$ represents the length of the longest common subsequence of %$R$ and $G$, presenting for a
reference summary and its generated summary ($R$ and $G$), respectively. ROUGE-L indicates the natural similarity between the two given sequences in sentence-level structure. It is also considered to be useful in evaluating summarization performance.
In our proposed approach, ROUGE scores are used as means to guide the learning process, allowing the recommendation engine to refine the summaries, making them more relevant. The detailed architecture of \ME together with its constituent components is described in the next section.
%To address the effectiveness of T5 pre-trained model in our summarization task, we established a comparison with BART baseline model in terms of ROUGE-1, ROUGE-2, and ROUGE-L.
% On the one hand, given a set of N-grams produced from a
% generated summary, Precision measures the proportion of N-grams
% in this set that exists in the reference summary. On the other hand,
% given a reference summary which could be transformed into a set
%The emergence of Large Language Models (LLMs) has opened new frontiers in artificial intelligence, enabling the creation of systems that can reason, plan, and interact using natural language. Among the emerging applications of LLMs is their integration into autonomous agents, which are capable of independently performing complex tasks in dynamic environments. As these LLM-based agents become more sophisticated, the need for structured frameworks that support multi-agent coordination has grown.
This section introduces the \ME approach to the summarization of software documents using LLMs-based agents. %, whose %. The proposed
%architecture is
To deal with noise in the input data, an LLM has been deployed as an extractor component, %to perform the extraction of the input data,
filtering out noisy and retaining the most meaningful fields. More importantly, by means of a Teacher-Student architecture applying collaborative LLMs, we iteratively refine a set of prompts to produce the best fit prompt, which then can be used to generate the final summary. As shown %illustrated
in Fig.~\ref{fig:pipeline_combined}, there are two main phases, \ie Optimization and Evaluation, explained as follows.
%the motivation, problem space, and research questions that guide this thesis, which focuses on a comparative analysis of frameworks used to build LLM-based multi-agent systems.
\subsection{Optimization} % Pipeline %Architecture
\begin{comment}
\begin{figure}[t!]
\centering
\begin{subfigure}[b]{\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/Optimize-pipeline.drawio.png}
\caption{Optimization Pipeline}
\label{fig:opt_pipeline}
\end{subfigure}
\vspace{1em} % Optional spacing between the two images
\begin{subfigure}[b]{\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/Evaluation_pipeline.png}
\caption{Evaluation Pipeline}
\label{fig:eval_pipeline}
\end{subfigure}
\caption{Pipelines used in the multi-agent system: (a) optimization and (b) evaluation.}
\label{fig:pipeline_combined}
\end{figure}
\end{comment}
\begin{figure*}[t!]
\centering
% \vspace{-3.25cm}
\begin{tabular}{c}
\subfigure[Optimization Pipeline]{\label{fig:opt_pipeline}\includegraphics[width=0.88\textwidth]{figures/Optimize-pipeline.drawio.png}} \\
\subfigure[Evaluation Pipeline]{\label{fig:eval_pipeline}\includegraphics[width=0.88\textwidth]{figures/Evaluation_pipeline.png}}
\end{tabular}
\caption{Pipelines used in \ME.} %ROC curves: (a) optimization and (b) evaluation
\label{fig:pipeline_combined}
\end{figure*}
The pipeline shown in Fig.~\ref{fig:opt_pipeline} involves four specialized LLM-powered agents: \EA, \SA, \TA, and \CA. These agents collaborate through prompt-driven communication, and their outputs are evaluated using ROUGE metrics to iteratively improve the quality of produced summaries as presented below.
% Due to space limitations, all prompt templates are provided in the online appendix~\cite{metagente-repo}.
%\textbf{Model Deployment Strategy.}
%\begin{itemize}
% \item
\smallskip
\noindent
\textbf{\EA.} This agent is responsible for isolating the most informative parts of a raw description that %. Informative parts of the description given as input,
can be in different formats, \eg HTML and Markdown. An application description is typically structured text document that may contain sections such as %An application's HTML description is a structured text document that may contain sections such as
\textit{introduction}, \textit{key features}, \textit{advantages}, \textit{detailed use cases}, \textit{setup instructions}, \textit{compatibility}.
%\DDR{This is confusing because we do not have always have HTML descriptions. In the case of GitHub we have MD files. I propose as follows: ... informative parts of the description given as input, which can be in different formats \eg HTML and Markdown. An application description is typically structured text document that may contain sections such as ...}.
%
\EA filters out irrelevant or boilerplate content, focusing only on the segments that explain the core function and value proposition of the application. This filtered version, rather than the entire original document, is then passed on for downstream processing, significantly improving both computational efficiency and focus during training.
\smallskip
\noindent
\textbf{\SA.} Given the refined description, this agent generates a concise and meaningful ``About'' statement for the app. Starting from an initial prompt, it iteratively produces summaries that are then improved over multiple rounds using updated prompts generated by \TA. By means of a dedicated prompt, \SA focuses on capturing the app’s primary goal, context, and differentiating features, while excluding peripheral or overly detailed information. The final result is a short and clear description that accurately represents the essence of the application.
\smallskip
\noindent
\textbf{\TA.} It evaluates the quality of each generated summary using the following four elements: \emph{(1)} the current summarization prompt; \emph{(2)} the generated short description; \emph{(3)} the ground-truth target description; and \emph{(4)} the computed ROUGE-L scores. Based on these inputs, it produces an improved prompt for \SA. This optimization follows a multi-step reasoning structure designed to pinpoint mismatches and refine instructions accordingly.
\smallskip
\noindent
\textbf{\CA.} After multiple training iterations across diverse samples, this agent aggregates and generalizes the refined prompts into a single, robust instruction set. It extracts recurring patterns and conditional logic to form a unified prompt suitable for inference. This final prompt helps \SA adapt to varying input description styles and content structures, ensuring consistent output quality.
%\end{itemize}
%\textbf{Model Deployment Strategy.}
\smallskip
\ME has been designed so as to allow for a flexible integration of different LLMs. In particular, for this implementation, five LLMs are considered as the engine, including: \texttt{GPT-4o}, \texttt{GPT-3.5-turbo}, \texttt{Gemma}, \texttt{Mistral}, and \texttt{Llama}. For the configuration with \texttt{GPT}, to optimize resource usage, \EA and \SA are implemented using the lightweight \texttt{GPT-4o-mini} version, which offers an efficient balance of performance and cost. In contrast, \TA and \CA leverage the more capable \Fo model to perform higher-level evaluation and reasoning. This hierarchical configuration enables the system to scale effectively while maintaining strong overall performance.
%\smallskip
%\noindent
%$\triangleright$
%\subsection{Optimization Workflow}
As illustrated in Figure~\ref{fig:opt_pipeline}, the workflow of the agents unfolds as follows:
\begin{enumerate}%[label=\arabic*.]
\item A %An HTML \DDR{Again, I would remove the reference to HTML. If you agree, we need to update the figure accordingly. Many other occurrences occur in the text.}
description is provided to \EA, which generates a concise version of the text to be used as input by \SA.
% \vspace{-.2cm}
\mybox{\textbf{\small{\EA Prompt}}}{gray!10}{gray!10}{\small{Your task is to shorten and extract only the introduction and description information from an app. You are given the following description for an app:
$<$Description$>$ \\
readme\_text \\
$<$/Description$>$
\# Steps \\
- **Identify the structure of the app's description**: The app's description is a structure text file that might contains many sections such as introduction, description, key features, advantages, detailed use case, setup instructions, compatibility,... \\
- **Remove all sections that are not relevant to the app's description**: Irrelevant sections might include technical guidance (installing/running/specification... instruction), compatibility, troubleshooting,... \\
- **Remove all unnecessary links/tags**: Identify all links/tags that DO NOT contribute to the description of the app. You must remove all of these reference links and tags. \\
- **Return only text that is relevant to the description of the app**: The output should only contains the text that is relevant to the introduction/description of the app, including the app name/title, app feature description/purpose statement/overview. DO NOT include any output identifications such as: "Here's the ..." or "Extracted App's description:"
"""
}}
\item Optimization Loop:
\begin{enumerate}%[label=2.\arabic*]
\item \SA receives the extracted text and produces a summary based on its current prompt, shown below.
\mybox{\textbf{\small{\SA Prompt}}}{gray!10}{gray!10}{\small{Your task is to shorten and extract only the introduction and description information from an app. You are given the following description for an app: \\
``''" \\
Summarize the following extracted text from an app's description into a short term/phrase introducing the app: \\
$<$EXTRACTED\_APP'S\_DESCRIPTION$>$ \\
\$extracted\_text \\
$<$/EXTRACTED\_APP'S\_DESCRIPTION$>$ \\
The output should include only a short term/phrase introducing the app. \\
``''"
}}
\item A ROUGE-L score is then calculated by comparing the generated summary with the ground truth summary. If the score is equal to or exceeds a predefined threshold, the optimization loop is interrupted, and the current prompt used by \SA is saved as a candidate prompt.
\item If the ROUGE-L score is below the threshold, \TA receives the inputs from \EA and \SA along with the ROUGE-L score. It uses this information to generate a new prompt, which is then passed to \SA for the next iteration. The prompt used by \TA is as follows.
\mybox{\textbf{\small{\TA Prompt}}}{gray!10}{gray!10}{\small{You are a professional Prompt Engineer. You are working on a system using a Large Language Model (LLM) to help developers automatically generate a short Description term/phrase contain key concept/idea from an extracted text of the description of an app. Your task is to modify and improve the current prompt of the LLM based on the result of testing on a data include a description and a ground truth description. \\
\# Steps: \\
- **Analyze the data for testing**: Analyze the following data include an extracted text from a description and a ground truth description from an app: \\
$<$EXTRACTED\_TEXT$>$ \\
\$extracted\_text \\
$<$/EXTRACTED\_TEXT$>$ \\
$<$GROUND\_TRUTH DESCRIPTION$>$ \\
\$description \\
$<$/GROUND\_TRUTH DESCRIPTION$>$ \\
- **Review the current result**: Review the generated description using the extracted text its ROUGE score on the ground truth description to identify improvements that could be made: \\
$<$GENERATED\_DESCRIPTION$>$ \\
\$generated\_about \\
$<$/GENERATED\_DESCRIPTION$>$ \\
$<$ROUGE\_SCORE$>$ \\
\$rouge\_score \\
$<$/ROUGE\_SCORE$>$ \\
}}
\item The loop continues until one of the following three conditions is satisfied: \emph{(i)} the ROUGE-L score meets or exceeds the threshold; \emph{(ii)} the maximum number of predefined iterations is reached; or \emph{(iii)} the ROUGE-L score stops improving or begins to decline.
\end{enumerate}
\item Once the loop ends, a new HTML description is passed through the same pipeline to generate another candidate prompt.
\item After all HTML descriptions have been processed, \CA collects all candidate prompts that achieved a ROUGE-L score above the defined threshold and aggregates them into a single, final prompt. The agent performs its activities by means of the following prompt.
\mybox{\textbf{\small{\CA Prompt}}}{gray!10}{gray!10}{\small{You are a professional Prompt Engineer. You are working on a system using a Large Language Model (LLM) to help developers automatically generate a short Description term/phrase contain key concept/idea from an extracted text of the description of an app. Your task is to combine several candidate prompts for the LLM into a final prompt. \\
\# Steps: \\
- **Review all candidate prompts**: Analyze the following prompts to identify common parts to be included in the final prompt and also includes specific details or conditional key points from these prompts to be included in the final prompt \\
$<$CANDIDATE\_PROMPTS$>$ \\
\$summarizer\_list \\
$<$/CANDIDATE\_PROMPTS$>$ \\
- **Generate a final prompt**: Based on the common parts and conditional key points, generate a final prompt for the LLM. \\
\# Output Format:\\
Do not include any reasoning/explanation like "Based on the result of the above review:", "Here's the", ... or any output identifiers like "Prompt:", "New Prompt", ... The output should only include a string representing the prompt for the LLM
"""
}}
\end{enumerate}
\subsection{Orchestration of Agents}
\ME operates as a collaborative pipeline in which multiple LLM-based agents interact toward a common objective. The orchestration of these agents is structured around three main phases: \textit{Agent Communication}, \textit{Iterative Refinement}, and \textit{Prompt Consolidation}, explained as follows.
\begin{itemize}
\item \textit{Agent Communication}. Each agent functions not in isolation but through controlled interaction with its environment and its peers. To facilitate this, we utilize the \texttt{LangChain} framework as the communication backbone. In addition, outputs are structured in a machine-readable format, enabling agents to reliably pass information to one another. This design ensures consistency and seamless cooperation throughout the pipeline.
\item \textit{Iterative Refinement}. The process begins with \SA receiving a filtered version of the application's HTML description from \EA. Using the current summarization prompt, it generates a candidate ``About'' summary. The result is evaluated using the ROUGE-L metric, which serves as a primary indicator of semantic and structural alignment with the ground-truth summary. This feedback, along with the current prompt and outputs, is passed to \TA, which formulates an improved prompt for the next round.
Unlike traditional fixed-iteration loops as proposed in our previous work \citep{10.1145/3696630.3728511}, in this paper we implement a \textit{dynamic stopping criterion} to optimize the training efficiency as follows:
\begin{enumerate}
\item Each sample starts with a baseline allowance of 15 iterations.
\item If the ROUGE-L score stagnates across 3 consecutive iterations or decreases in 2 iterations (with each drop also counted as stagnation), the process halts early for that sample.
\item If ROUGE-L reaches a predefined threshold (0.7), then the process stops.
\end{enumerate}
We expect this adaptive mechanism prevents unnecessary computation on low-potential samples, while still offering more iteration budget to promising ones. In addition, we examine the impact of this dynamic iteration strategy in comparison to a non-dynamic (fixed-loop) setting to evaluate gains in performance and efficiency.
\item \textit{Prompt Consolidation}. After the optimization process completes, we gather a pool of prompt versions, each tailored to a specific training instance. These prompts are analyzed by \CA, which extracts common structures and key conditional instructions to synthesize a generalized summarization prompt. This final prompt is then used during inference to ensure that \SA performs effectively across a diverse range of HTML description inputs, maintaining both coherence and relevance in the generated ``About'' summaries.
\end{itemize}
\subsection{Evaluation Pipeline}
Once the optimized prompt has been obtained from the Optimization Pipeline, the evaluation phase can then be initiated. As illustrated in Figure~\ref{fig:eval_pipeline}, the Evaluation Pipeline is comparatively simple and consists of only two agents, \ie \EA and \SA. The former uses the same prompt and performs the same function as in the optimization phase, processing the HTML description to extract a concise description. The latter, however, uses the final optimized prompt produced in the previous phase for all summarization. After the summarization, ROUGE-1, ROUGE-2, ROUGE-L and cosine similarity were collected to evaluate the quality of the generated summaries against their respective ground truth references. These metrics are collected from the evaluation of several HTML descriptions so that the optimization task could be assessed.
This section presents the empirical study conducted to evaluate the proposed approach's performance. We introduce the %three
research questions in Section~\ref{sec:ResearchQuestions}, %. Afterwards, we
and describe the datasets in Section~\ref{sec:Dataset}. %and metrics %used in our evaluation.
%Finally, the metrics used to evaluate the outcomes are presented in Section~\ref{sec:Metrics}.
\subsection{Research Questions} \label{sec:ResearchQuestions}
To evaluate \ME we answer the following research questions, and compare it with baselines.
\begin{itemize}
% \vspace{.1cm}
% \noindent
% $\rhd$~
% \item \rqfirst~%In our previous work, we compared
% \ME has been compared with \GS and \texttt{LLaMA-3.2}, demonstrating a superior performance \citep{10.1145/3696630.3728511}. In this paper, apart from the previous baselines, we further evaluated our tool with \texttt{Mistral-7b-instruct}. %, and ...
\item \rqfirst~\\%In our previous work, we compared
We investigate if the use of a multi-agent architecture is really needed, given that a single agent might already be sufficient to get a decent recommendation. For this RQ, we consider \texttt{Mixtral-8x7B-Instruct-v0.1}, \texttt{Llama-2-7b-hf}, \texttt{GPT-4o}, and \texttt{Gemma-2-2b-it} as baselines for comparison as they have been widely used in summarization tasks. %has been compared with two single LLM agents, \ie \texttt{LLaMA-3.2} and \texttt{Mistral-7b-instruct}.
\item \rqsecond~\\This RQ evaluates whether the newly proposed dynamic iteration strategy is beneficial to the generation of short summaries, \ie effective and/or efficient, compared to the static one conceived in our previous work \citep{10.1145/3696630.3728511}. %As users in the considered dataset \citep{sanchez_recommendation_2020} have been allowed to self-assess their privacy category, we examine if such a self-evaluation %is credible, \ie it
%reflects their real category. %of a user. %Moreover, we attempt to find a compact set of questions that can be used to cluster users.
% \vspace{.1cm}
% \noindent
% $\rhd$~
\item \rqthird~The previous version of \ME relies on \texttt{Original LLM set} including \texttt{GPT-4o} and \texttt{GPT-4o-mini} %\DDR{what do we mean with original LLM set here? We need to be explicit.}
as the only engine for agents. In this paper, we further extended the architecture to sustain other LLMs, including \texttt{Mistral-7B-Ins\-truct-v0.3}, \texttt{Llama-3.2-3B-Instruct}, and \texttt{Gemma-2-2b-it}, to validate the extensibility of \ME.
% \item \rqfourth~The previous version of \ME relies on GPT-4o as the only engine for agents. In this paper, we further extended the architecture to host other LLMs, including ...
% \item \rqfourth~
\end{itemize}
\subsection{Datasets} \label{sec:Dataset}
In this section, we explain in detail the process conducted to curate, clean, and restructure the datasets used in the evaluation.
\subsubsection{Data Curation}
To assess the effectiveness of \ME in a domain beyond traditional software documentation, we curated a novel dataset of mobile applications derived from the AndroZoo\footnote{\url{https://androzoo.uni.lu/}} repository~\citep{Allix:2016:ACM:2901739.2903508}, a large-scale archive comprising over 25 million Android applications mined from various sources.
We initially selected a representative sample of 20{,}000 applications from the 5GB AndroZoo summary metadata file, filtering for apps that \textit{(i)} were sourced from the Google Play Store; and \textit{(ii)} had a release date after January 1st, 2020 (we consider apps in the most recent 5 years). %\DDR{Is there any motivation supporting such filtering criteria?}
Application metadata was retrieved programmatically through the AndroZoo API, which provides multiple versions for each application along with associated attributes (e.g., \texttt{name}, \texttt{description\_short}, \texttt{description\_html}, \texttt{version\_\-string}).
The following filtering criteria were applied to refine the corpus:
\begin{itemize}
\item Apps with only one version available were excluded.
\item For multi-version apps, only the metadata of the most recent version was retained.
\item Apps without a star rating (\ie no user reviews) were discarded.
\item Using the \texttt{langdetect}\footnote{\url{https://github.com/Mimino666/langdetect}} library for language identification, we excluded apps whose \texttt{description\_html} or \texttt{description\_short} fields are not written in English.
\end{itemize}
After filtering, the final dataset consisted of 2{,}980 mobile applications. For each application, we used the \texttt{description\_html} field as input and the corresponding \texttt{description\_short} as the ground-truth summary. %To characterize the dataset, we analyzed the distribution of user star ratings across applications.
Table~\ref{tab:star-distribution} reports the number of applications falling into predefined rating intervals.
\begin{table}[h]
\centering
\caption{Distribution of applications by average star rating.}
\label{tab:star-distribution}
\begin{tabular}{|l|c|} \hline
\textbf{Star Rating Range} & \textbf{Number of Applications} \\ \hline
0--1 & 0 \\ \hline
1--2 & 68 \\ \hline
2--3 & 361 \\ \hline
3--4 & 854 \\ \hline
4--5 & 1,696 \\ \hline
\end{tabular}
\end{table}
As shown in the table, the majority of the apps in the dataset have a rating above 3.0, with over half being rated between 4 and 5. This distribution reflects a natural skew toward well-rated apps, which are also more likely to have informative and well-structured descriptions. Apps without any rating were excluded as part of our data quality filtering process, %. We computed descriptive statistics on the length of input and output texts, estimating token counts by assuming one token per six characters \DDR{Is this assumption sustainable? Do we have any reference for this?},
and the statistics are depicted in Table~\ref{tab:dataset-stats}. % summarizes the main characteristics of the dataset.
\begin{table}[h]
\centering
\caption{Summary statistics of the mobile application dataset (token-level estimates).}
\label{tab:dataset-stats}
\begin{tabular}{|l|r|r|} \hline
\textbf{Metric} & \texttt{description\_html} & \texttt{description\_short} \\ \hline
Average Length & 289 tokens & 10 tokens \\ \hline
Median Length & 244 tokens & 11 tokens \\ \hline
Minimum Length & 2 tokens & 1 token \\ \hline
Maximum Length & 798 tokens & 17 tokens \\ \hline
Standard Deviation & 198 tokens & 3 tokens \\ \hline
\end{tabular}
\end{table}
A distributional analysis reveals that input texts exhibit high variance and right-skewness, consistent with the verbose and heterogeneous nature of mobile app descriptions. In contrast, the ground-truth summaries are concise, with low variance and a narrow range, confirming the suitability of this dataset for evaluating the summarization capabilities of \ME.
\subsubsection{Data Preprocessing} %Overview and Selection Criteria
%The dataset consists of approximately 3,000 samples, each containing two main columns of interest: the raw HTML text and its corresponding summary.
Data quality plays a crucial role in the fine tuning and inference phases. To assess the quality of these pairs, we conducted an exploratory data analysis (EDA) by computing ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L) and Cosine Similarity between \texttt{description\_html} and \texttt{description\_short}, and the scores are shown in Figure~\ref{fig:rouge_cosine_scatter}. %.the HTML and summary texts.
\begin{figure}[ht]
\centering
\includegraphics[width=\linewidth]{figures/EDA_OG_data.png}
\caption{The relationship between Cosine Similarity and ROUGE scores.} % (ROUGE-1, ROUGE-2, ROUGE-L)
\label{fig:rouge_cosine_scatter}
\end{figure}
As seen in the figure, there is a clear positive correlation between ROUGE and Cosine Similarity scores. In other words, a higher semantic similarity %measured by Cosine Similarity
tends to coincide with a denser n-gram overlap. % measured by the ROUGE scores.
This observation allows us to come up with the following %guides our
data filtering strategy.
\begin{itemize}
\item \textbf{Training set}. Given the goal of building a system that learns effectively from a small amount of high-quality data, we selected training samples with 0.85 $\leq$ Cosine Similarity $<$ 1.0.
%Cosine Similarity~$\geq$~0.85 and~$<$~1.0.
This range ensures that each training pair is highly relevant yet not identical, thereby reducing noise and enhancing learning efficiency.
\item \textbf{Test set}. It was constructed from the remaining samples not used in training. The following filtering rule was applied:
\begin{center}
\textit{(ROUGE-1~$\geq$~0.1) or (ROUGE-2~$\geq$~0.1) or (ROUGE-L~$\geq$~0.1)} \textbf{and} \textit{Cosine Similarity~$\geq$~0.4}
\end{center}
\end{itemize}
These thresholds were selected based on empirical inspection of samples at various cutoff points, ensuring a reasonable level of summary relevance while maintaining diversity. This filtered set contains over 800 samples, and it is used to derive the following final sets for the evaluation. %From this test set, the following four final datasets were derived.
\begin{itemize}
\item \textbf{D$_1$}: 400 randomly sampled test cases.
\item \textbf{D$_2$}: 600 randomly sampled test cases.
\item \textbf{D$_3$}: Top 200 samples with the highest Cosine Similarity scores.
\item \textbf{D$_4$}: Top 200 samples with the highest ROUGE-L scores.
\end{itemize}
These subsets enable us to evaluate the performance of \ME across both general and high-quality cases, facilitating robust and diverse performance analysis. The next section elaborates on %reports and analyzes
the results obtained from the empirical evaluation.
We report and analyze the experimental results by answering the %three aforementioned
research questions introduced in Section~\ref{sec:ResearchQuestions}.
\subsection{\rqfirst}
In our previous work \citep{10.1145/3696630.3728511}, \ME was compared with various studies, and the %\GS and \texttt{LLaMA-2}.
results demonstrated that it consistently outperformed the baselines in generating repository descriptions from \GH \RM files. %Building on this, r
Recent studies have evaluated the performance of different LLMs on summarization tasks across various domains~\citep{Aly2025,takeshita-etal-2025-irsum,10.1145/3677389.3702588,10986332}. These studies highlight that models such as \texttt{Mixtral-8x7B-Instruct-v0.1} and \texttt{Llama-2-7b-hf} achieve competitive or superior results, especially in zero-shot settings. To further deepen %broaden
the comparison with single-agent LLMs, we extended our evaluation to include two widely adopted general-purpose LLMs that are frequently used in summarization tasks including \texttt{GPT-4o} and \texttt{Gemma-2-2b-it}.
We evaluate \ME against the baselines using two %\SA's output across four
datasets, \ie \textbf{D$_1$} and \textbf{D$_2$} and the results are shown in Figure~\ref{fig:RQ1-ROUGE-D1D2}. %¸
%two randomly sampled groups and two high-quality groups based on top cosine similarity and ROUGE-L.
%Notably, in both random test sets, the generated outputs consistently outperform ground-truth in cosine similarity (e.g., 0.7429 vs. 0.6797 in Random 600 samples), with minimal ROUGE differences. For top-ranked subsets, although ground-truth slightly leads in ROUGE, the generated outputs show greater flexibility in length and stable quality.
Overall, the generated %“About”
summaries exhibit higher or comparable semantic similarity (cosine) and competitive ROUGE scores compared to ground-truth summaries. As shown in Figure~\ref{fig:RQ1-ROUGE-D1D2}, it is evident that both the baselines, \ie \texttt{Mixtral-8x7B-Instruct-v0.1} and \texttt{Llama-2-7b-hf}, achieve a lower and less stable performance on the two datasets. An extremely low recommendation quality is seen by \texttt{Mixtral-8x7B-Ins\-truct-v0.1}, \ie most of the ROUGE scores are smaller than 0.15. Their outputs also show a greater variance--some summaries are overly generic, while others omit key aspects of the original descriptions. This underscores the baselines' limitations when used for summarization without iterative refinement or agent collaboration.
The figure shows that \ME consistently outperforms both baselines across all the evaluation metrics, \ie %By contrast,
it delivers more consistent and higher-quality results across both random subsets. The advantage is especially clear in ROUGE-L, which captures structural and semantic similarity with the reference summaries. In particular, by the \textbf{D$_1$} dataset (see Figure~\ref{fig:RQ1-ROUGE-D1}), most of the ROUGE-L scores obtained by \ME range from 0.3 to 1.0. Meanwhile by \texttt{Mixtral-8x7B-Instruct-v0.1} and \texttt{Llama-2-7b-hf}, the corresponding scores are much smaller, \eg less than 0.2. The same trend is witnessed with \textbf{D$_2$} (see Figure~\ref{fig:RQ1-ROUGE-D2}), in which the ROUGE scores obtained by \ME are always superior to those of \texttt{Mixtral-8x7B-Instruct-v0.1} and \texttt{Llama-2-7b-hf}.
This suggests that \ME\ generates summaries that are not only concise but also more faithful to the original content, reflecting both the overall purpose and the unique details of the applications described. These improvements point to the value of the multi-agent optimization loop in producing summaries that are easier to read and more informative. The stability of these improvements across metrics and datasets provides strong evidence that the Teacher–Student refinement loop and prompt consolidation strategy not only improve accuracy but also enhance robustness and generalizability in diverse summarization settings.
%\vspace{.2cm}
\noindent\fbox{\begin{minipage}{0.98\columnwidth}
\paragraph{\textbf{Answer to RQ$_1$:}} \ME consistently outperforms various single LLM-based agents in terms of all the ROUGE scores, % across the \textbf{D$_1$} and \textbf{D$_2$} datasets,
demonstrating the benefit of collaboratively joint LLM-based agents. %its effectiveness in generating higher-quality summaries for mobile apps.
\end{minipage}}
%``\emph{allow}''
%Based on
%we realized that there is a high level of similarity among different users. This inspires us to
%We utilized the same dataset in evaluating the performance of \PR.
%the users already specified a value of 2
%(\emph{allow}) or 1 (\emph{deny}) for a setting.
%When users start using an app, or accessing a website,
%they are expected to specify their privacy preferences, \eg by
%, and oftentimes makes users become bored.
%At a certain point, they just
%This simulates real usage, where users already specified some settings, and the system is supposed to provide addition ones that are relevant.
%that may lead users to either consent or reject them all~\citep{10.1145/2702123.2702210}, without being aware of what benefits or risks they are up to~\citep{liu_follow_2016}.
%=========================================
%Specifying privacy preferences is requested when users start using an app, or accessing a website. This is done by turning on or off several radio boxes, which is indeed time consuming and boring. \PR has been developed to alleviate such a fatigue %tiredness
%when setting preferences.%a %burden. %users in such a process. %Oftentimes users just.
%we use %our proposed recommender system, \ie
%the execution is done ten times:
%=========================================
%
%================================================
%The clusters obtained from \textbf{RQ$_2$} allow us to identify users with similar privacy preferences. Based on the obtained categorization, given an input user, the conceived neural network is used to assign the user to a specific category. Afterward, we build a graph only for this category following the paradigm in Figure~\ref{fig:Graph}. Such a subgraph contains less nodes and edges compared to a full graph for all categories. This is important in practice, since for a large number of users, the graph can be huge, which may slow down the computation. In this respect, a pre-defined categorization serves as a means to optimize the computation. On top of this, \PR recommends missing settings to users.
%================================================
%\vspace{.1cm}
%\noindent
%$\rhd$~\textbf{\rqthird}
%\begin{figure}[h!]
% \centering
% \vspace{-.4cm}
% \includegraphics[width=0.25\textwidth]{figs/NewROC.png}
% \vspace{-.4cm}
% \caption{Receiver Operating Characteristic.}
% \label{fig:NewROC}
% \vspace{-.2cm}
%\end{figure}
%================================================
%In the dataset, there are several on/off features, for instance, \emph{s\_location} can be $1$ (\emph{allow}) or $0$ (\emph{deny}),\footnote{The actual values are 2 and 1, respectively. For the sake of computation, they were normalized to 1 and 0.} dictating if the user is willing to share location or not. %We filtered out and selected only the features related to privacy settings.
%================================================
\begin{landscape}
\begin{figure*}[t!]
\centering
% \vspace{-3.25cm}
\begin{tabular}{c}
\subfigure[\textbf{D$_1$}]{\label{fig:RQ1-ROUGE-D1}\includegraphics[width=0.95\linewidth]{figures/RQ1-ROUGE-D1.png}} \\
\subfigure[\textbf{D$_2$}]{\label{fig:RQ1-ROUGE-D2}\includegraphics[width=0.95\linewidth]{figures/RQ1-ROUGE-D2.png}}
\end{tabular}
\caption{Comparison between \ME and LLM-based single agents on the \textbf{D$_1$} and \textbf{D$_2$} datasets.} %ROC curves: (a) optimization and (b) evaluation
\label{fig:RQ1-ROUGE-D1D2}
\end{figure*}
\end{landscape}
\begin{comment}
\begin{landscape}
\begin{figure*}[t!]
\centering
% \vspace{-3.25cm}
\begin{tabular}{c}
\subfigure[\textbf{D$_1$}]{\label{fig:RQ1-ROUGE-D1}\includegraphics[width=0.5\linewidth]{figures/RQ1-ROUGE-1-D1.jpg}} \\
\subfigure[\textbf{D$_2$}]{\label{fig:RQ1-ROUGE-D2}\includegraphics[width=0.5\linewidth]{figures/RQ1-ROUGE-2-D1.jpg}}
\end{tabular}
\caption{Performance comparison on \textbf{D$_1$} and \textbf{D$_2$}.} %ROC curves: (a) optimization and (b) evaluation
\label{fig:RQ1-ROUGE-D1D2}
\end{figure*}
\end{landscape}
\end{comment}
\begin{comment}
\begin{figure*}[t!]
\centering
\begin{tabular}{ccc}
\subfigure[ROUGE-1]{\label{fig:RQ1-ROUGE-1-D1}
\includegraphics[width=0.3\linewidth]{figures/RQ1-ROUGE-1-D1.jpg}} &
\subfigure[ROUGE-2]{\label{fig:RQ1-ROUGE-2-D1}
\includegraphics[width=0.3\linewidth]{figures/RQ1-ROUGE-2-D1.jpg}} &
\subfigure[ROUGE-L]{\label{fig:RQ1-ROUGE-L-D1}
\includegraphics[width=0.3\linewidth]{figures/RQ1-ROUGE-L-D1.jpg}} \\
\subfigure[ROUGE-1]{\label{fig:RQ1-ROUGE-1-D2}
\includegraphics[width=0.3\linewidth]{figures/RQ1-ROUGE-1-D2.jpg}} &
\subfigure[ROUGE-2]{\label{fig:RQ1-ROUGE-2-D2}
\includegraphics[width=0.3\linewidth]{figures/RQ1-ROUGE-2-D2.jpg}} &
\subfigure[ROUGE-L]{\label{fig:RQ1-ROUGE-L-D2}
\includegraphics[width=0.3\linewidth]{figures/RQ1-ROUGE-L-D2.jpg}}
\end{tabular}
\caption{Comparison between \ME and the baselines on the \textbf{D$_1$} and \textbf{D$_2$} datasets.}
\label{fig:ComparisonMetagente}
\end{figure*}
\end{comment}
\begin{landscape}
\begin{figure*}[t!]
\centering
% \vspace{-3.25cm}
\begin{tabular}{c}
\subfigure[Comparison of generated summary with ground-truth \texttt{description\_short}]{\label{fig:RQ2-GroundTruth_DescriptionShort}\includegraphics[width=0.95\linewidth]{figures/RQ2_GroundTruth_DescriptionShort.jpeg}} \\
\subfigure[Comparison of generated summary with ground-truth \texttt{description\_html}]{\label{fig:RQ2-GroundTruth_README}\includegraphics[width=0.95\linewidth]{figures/RQ2_GroundTruth_README.jpeg}}
\end{tabular}
\caption{Dynamic vs. Non-Dynamic strategies.} %ROC curves: (a) optimization and (b) evaluation
\label{fig:RQ2-Comparison}
\end{figure*}
\end{landscape}
%\subsection{Comparison Between Dynamic and Non-Dynamic Strategies}
\subsection{\rqsecond}
%\textbf{RQ1: How does a dynamic iteration strategy compare to a fixed-loop (non-dynamic) strategy in terms of summarization quality and training efficiency?}
We conducted a comparative analysis between the dynamic and non-dynamic prompting strategies using ROUGE scores and cosine similarity. The results are depicted in Figure~\ref{fig:RQ2-Comparison}.
\begin{comment}
\begin{figure}[h!]
\centering
\includegraphics[width=0.48\textwidth]{figures/ROUGE_1.png}
\includegraphics[width=0.48\textwidth]{figures/ROUGE_2.png}
\includegraphics[width=0.48\textwidth]{figures/ROUGE_L.png}
\includegraphics[width=0.48\textwidth]{figures/Cosine.png}
\caption{Histograms comparing Dynamic vs. Non-Dynamic strategies on ROUGE-1, ROUGE-2, ROUGE-L, and Cosine Similarity.}
\label{fig:histograms}
\end{figure}
\end{comment}
Quantitatively, the non-dynamic approach shows slightly higher averages in ROUGE-2 and ROUGE-L, suggesting stronger n-gram overlap with reference summaries. In contrast, the dynamic method performs on par in ROUGE-1 and demonstrates stronger semantic alignment in cosine similarity, highlighting its ability to capture meaning beyond surface-level overlap.
Interestingly, despite comparable performance, the dynamic strategy completed training in only 260 iterations, while the non-dynamic method requires 620 iterations to converge. The comparison of efficiency between non-dynamic and dynamic strategies is shown in Figure~\ref{fig:Efficiency}. %reach convergence.
\begin{comment}
\begin{table}[h]
\centering
\caption{Execution Time.}
\label{tab:Timing}
\begin{tabular}{|l|r|r|r|r|} \hline
\textbf{Agent} & \texttt{\# Input Token} & \texttt{\# Output Token} & \texttt{Avg. Duration} & \texttt{Avg. Flow Duration} \\ \hline
EA & 401.03 & 90.45 & 3.39 & 12.89 \\ \hline
CA & 1,482.00 & 90.45 & 1.43 & 12.89 \\ \hline
SA & 177.59 & 12.30 & 1.12 & 12.89 \\ \hline
TA & 206.29 & 96.55 & 1.47 & 12.89 \\ \hline
\end{tabular}
\end{table}
\end{comment}
\begin{figure*}[h!]
\centering
\begin{tabular}{c c}
\subfigure[Average Flow Duration]{\label{fig:FlowDuration}
\includegraphics[width=0.45\linewidth]{figures/RQ2_StepDuration.jpg}} &
\subfigure[Average Step Duration]{\label{fig:StepDuration}
\includegraphics[width=0.40\linewidth]{figures/RQ2_FlowDuration.jpg}} \\
\subfigure[Number of input tokens]{\label{fig:InputTokens}
\includegraphics[width=0.45\linewidth]{figures/RQ2_InputTokens.jpg}} &
\subfigure[Number of output tokens]{\label{fig:OutputTokens}
\includegraphics[width=0.45\linewidth]{figures/RQ2_OutputTokens.jpg}}
\end{tabular}
% \vspace{-.2cm}
\caption{Comparison of efficiency between non-dynamic and dynamic strategies.}
\label{fig:Efficiency}
% \vspace{-.2cm}
\end{figure*}
The figure shows that compared to the non-dynamic method, the dynamic one is always efficient with respect to the timing and resources used. This suggests that the dynamic method is significantly more efficient in learning useful prompts, adapting early based on intermediate feedback. %\DDR{Do we have any number \eg related to execution time or even consumed token to reinforce this aspect? Number of iterations is ok but I think we are more convincing if we explicitly mention the impact of such a reduction of the iterations.}
%\textbf{Distribution analysis} in the histograms
Figure~\ref{fig:RQ2-Comparison} further highlights that both strategies follow similar trends in ROUGE and cosine similarity distributions. However, the dynamic version tends to produce outputs with higher semantic consistency and broader generalization, whereas the non-dynamic one exhibits sharper peaks in overlap-based metrics.
%\vspace{.2cm}
The evaluation reveals that, despite producing results comparable to the non-dynamic setting, the dynamic iteration strategy requires significantly fewer training steps. This means that adaptive stopping criteria help avoid wasted computation while still capturing meaningful improvements. Such efficiency is especially important in real-world deployments, where computational resources are often limited.
%In summary, while the non-dynamic approach slightly leads in ROUGE-2 and ROUGE-L, the dynamic strategy strikes a better balance between effectiveness %performance
%and efficiency. %This makes it a preferable option in low-resource or iterative refinement settings, especially within the \ME\ framework using LLM-based agents.
\begin{comment}
\begin{tcolorbox}[boxrule=0.86pt,left=0.3em, right=0.3em,top=0.1em, bottom=0.05em]
\textbf{Answer to RQ$_{1}$.} The dynamic method creates summaries that are nearly as good as the non-dynamic one, but it needs much fewer training steps (260 vs. 620). This means it saves time and resources while still keeping quality, making it a more efficient option.
\end{tcolorbox}
\end{comment}
\vspace{.2cm}
\noindent\fbox{\begin{minipage}{0.98\columnwidth}
\paragraph{\textbf{Answer to RQ$_2$:}}
The dynamic method creates summaries %that are
comparable to the those generated by the %nearly as good as the
non-dynamic ones. However, the dynamic method needs %it needs
much fewer training steps (260 vs. 620). %This means it saves time and resources while still keeping quality, making it a more efficient option in real-world scenarios.
This means that it improves efficiency by reducing unnecessary training steps while maintaining comparable quality, making the framework suitable for resource-constrained environments in real-world scenarios.
\end{minipage}}
\begin{comment}
\subsection{\rqthird}
% \textbf{RQ2: How does the generated “About” text perform across different test subsets in terms of semantic alignment and content overlap?}
We evaluated \SA's output across four subsets: two randomly sampled groups and two high-quality groups based on top cosine similarity and ROUGE-L. Overall, the generated “About” summaries exhibit higher or comparable semantic similarity (cosine) and competitive ROUGE scores compared to ground-truth summaries.
Notably, in both random test sets, the generated outputs consistently outperform ground-truth in cosine similarity (e.g., 0.7429 vs. 0.6797 in Random 600 samples), with minimal ROUGE differences. For top-ranked subsets, although ground-truth slightly leads in ROUGE, the generated outputs show greater flexibility in length and stable quality.
\end{comment}
%\begin{figure}[h!]
% \centering
% \includegraphics[width=0.48\textwidth]{figures/Rand_600_CS.png}
% \includegraphics[width=0.48\textwidth]{figures/CS_Top_200_RL.png}
% \caption{Cosine Similarity vs. Length in Random 600 and ROUGE-L Top 200 subsets.}
% \label{fig:rq2_summary_fig}
%\end{figure}
\begin{comment}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{0.88\textwidth}
\includegraphics[width=\textwidth]{figures/Rand_600_CS.png} % Replace with your first image file
\caption{Random 600 samples.}
\label{fig:subfig1}
\end{subfigure}
\vspace{10pt} % Adjust vertical space between subfigures
\begin{subfigure}[b]{0.88\textwidth}
\includegraphics[width=\textwidth]{figures/CS_Top_200_RL.png} % Replace with your second image file
\caption{Top 200 samples.}
\label{fig:subfig2}
\end{subfigure}
\caption{Cosine Similarity vs. Length in Random 600 and ROUGE-L Top 200 subsets.}
\label{fig:main}
\end{figure}
\end{comment}
\begin{comment}
\begin{figure*}[t!]
\centering
% \vspace{-3.25cm}
\begin{tabular}{c}
\subfigure[Random 600 samples]{\label{fig:Random600}\includegraphics[width=0.88\textwidth]{figures/Rand_600_CS.png}} \\
\subfigure[Top 200 samples]{\label{fig:TopCS200}\includegraphics[width=0.88\textwidth]{figures/CS_Top_200_RL.png}}
\end{tabular}
\caption{Pipelines used in \ME.} %ROC curves: (a) optimization and (b) evaluation
\label{fig:DataProcessing}
\end{figure*}
\end{comment}
\begin{comment}
\vspace{.2cm}
\noindent\fbox{\begin{minipage}{0.98\columnwidth}
\paragraph{\textbf{Answer to RQ$_3$:}} Conclusions go here.
% The dynamic method creates summaries that are nearly as good as the non-dynamic one, but it needs much fewer training steps (260 vs. 620). This means it saves time and resources while still keeping quality, making it a more efficient option.
\end{minipage}}
\begin{tcolorbox}[boxrule=0.86pt,left=0.3em, right=0.3em,top=0.1em, bottom=0.05em]
\textbf{Answer to RQ$_{2}$.} The generated summaries work well across different test sets. They are often close in meaning to the original app descriptions and include the right content. This shows that the system can create accurate and useful "About" texts even with different types of input.
\end{tcolorbox}
\end{comment}
\begin{comment}
\begin{figure}[H]
\centering
\includegraphics[width=0.48\textwidth]{figures/rouge_violin_box_all_200_rouge.png}
\includegraphics[width=0.48\textwidth]{figures/rouge_violin_box_all_200_cosine.png}
\includegraphics[width=0.48\textwidth]{figures/rouge_violin_box_all_400.png}
\includegraphics[width=0.48\textwidth]{figures/rouge_violin_box_all_600.png}
\caption{Evaluating LLMs Across Multiple Datasets and ROUGE Metrics.}
\label{fig:llm_test}
\end{figure}
\end{comment}
% \begin{figure}[H]
% \centering
% \includegraphics[width=\textwidth]{figures/llm_test.png}
% \caption{Evaluating LLMs Across Multiple Datasets and ROUGE Metrics.}
% \label{fig:llm_test}
% \end{figure}
%\begin{figure*}[t!]
% \centering
% \begin{tabular}{c c}
% \subfigure[Front page]{\label{fig:ROUGE-200}
% \includegraphics[width=0.80\linewidth]{figures/rouge_violin_box_all_200_rouge.png}} &
% \subfigure[README]{\label{fig:Courtside1891-Back}
% \includegraphics[width=0.80\linewidth]{figures/rouge_violin_box_all_200_cosine.png}}
% \end{tabular}
% % \vspace{-.2cm}
% \caption{Evaluating LLMs Across Multiple Datasets and ROUGE Metrics.}
% \label{fig:ROUGE-LLMs}
% % \vspace{-.2cm}
%\end{figure*}
\subsection{\rqthird} %Can Our Pipeline Work with Any LLM?
To evaluate the compatibility and adaptability of our pipeline across various LLMs, we implemented \ME using multiple models--ranging from lightweight open-source LLMs to commercial APIs--on the same datasets and measured their performance using ROUGE metrics. In particular, the following models are considered in our evaluation: \texttt{Original LLM set (a combination of GPT-4o and GPT-4o-mini)}, \texttt{GPT-3.5-turbo}, \texttt{Gemma-2-2b-it}, \texttt{Mistral-7b-I\-nstruct-v0.3}, and \texttt{Llama-3.2-3B-Instruct}. %\DDR{What do we report in Figure 7 and 8 concerning the Original LLM? The average scores? The set includes two different LLMs, this is why reviewer can wonder what is reported in Figure 7 and 8} %, and the original LLM set (i.e., \Fo and its smaller variant).
Figure~\ref{fig:ROUGE-D1D2} and Figure \ref{fig:ROUGE-D3D4} depict violin boxplots, comparing the average ROUGE-1, ROUGE-2, and ROUGE-L scores across four evaluation subsets: two random test sets, \ie \textbf{D$_1$} and \textbf{D$_2$} with 400 and 600 samples, respectively, and two top-ranked subsets based on Cosine Similarity and ROUGE-L, \textbf{D$_3$} and \textbf{D$_4$} each with 200 samples.
As shown in the figures, our pipeline demonstrates compatibility with all tested LLMs %, including \texttt{GPT-3.5-turbo}, \texttt{Gemma-2-2b}, \texttt{LLaMA-3.2-3b}, \texttt{Mistral-7b-instruct}, and the original LLM set (i.e., \Fo and its smaller variant).
although performance varies--with the original LLM set consistently outperforming others--the system remains operational and effective with all models. The framework remains robust across different LLM backbones, indicating strong portability and adaptability in diverse real-world scenarios. Even lighter models such as \texttt{Gemma-2-2b-it} and \texttt{Mistral-7B-Ins\-truct-v0.3} were able to produce summaries meeting minimum ROUGE thresholds, indicating robustness in prompt processing and adaptation.
\begin{landscape}
\begin{figure*}[t!]
\centering
% \vspace{-3.25cm}figures/rouge_violin_box_all_random_400_colored.png
\begin{tabular}{c}
\subfigure[\textbf{D$_1$}]{\label{fig:ROUGE-D1}\includegraphics[width=0.95\linewidth]{figures/rouge_violin_box_D1.png}} \\
\subfigure[\textbf{D$_2$}]{\label{fig:ROUGE-D2}\includegraphics[width=0.95\linewidth]{figures/rouge_violin_box_D2.png}}
\end{tabular}
\caption{Performance comparison on \textbf{D$_1$} and \textbf{D$_2$}.} %ROC curves: (a) optimization and (b) evaluation
\label{fig:ROUGE-D1D2}
\end{figure*}
\begin{figure*}[t!]
\centering
% \vspace{-3.25cm}
\begin{tabular}{c}
\subfigure[\textbf{D$_3$}]{\label{fig:ROUGE-D3}\includegraphics[width=0.95\linewidth]{figures/rouge_violin_box_D3.png}} \\
\subfigure[\textbf{D$_4$}]{\label{fig:ROUGE-D4}\includegraphics[width=0.95\linewidth]{figures/rouge_violin_box_D4.png}}
\end{tabular}
\caption{Performance comparison on \textbf{D$_3$} and \textbf{D$_4$}.} %ROC curves: (a) optimization and (b) evaluation
\label{fig:ROUGE-D3D4}
\end{figure*}
\end{landscape}
%\DDR{When we say the usage of some LLM, we mean that all the four agents are using the same LLM? In the FSE paper we had two agents using a small LLM and other two agents using a bigger one, isn't it?}
In Figure~\ref{fig:ROUGE-D1D2}, concerning the ROUGE scores obtained on \textbf{D$_1$}, we can see that using \texttt{Gemma-2-2b-it}, \ME yields a mediocre performance compared to that when using the other LLMs. \texttt{Mistral-7B-Ins\-truct-v0.3} as the agent is better than \texttt{Gemma-2-2b-it}, still it underperforms the remaining LLMs including \texttt{Original LLM set}, \texttt{GPT-3.5-turbo} and \texttt{Llama-3.2-3B-Instruct}. Among others, the \texttt{Original LLM set} with two \texttt{GPT-4o} agents and \texttt{GPT-4o-mini} agents contributes to the best performance to \ME. The same trend is seen on \textbf{D$_2$} (shown in Figure~\ref{fig:ROUGE-D2}). These findings suggest that our pipeline does not depend on a specific backbone model, making it portable and flexible for different deployment scenarios, including low-resource settings.
\begin{figure*}[t!]
\centering
% \vspace{-3.25cm}
\begin{tabular}{c}
\subfigure[ROUGE-1]{\label{fig:avg_ROUGE-1}\includegraphics[width=0.60\textwidth]{figures/avg_curve_R1_fair.png}} \\
\subfigure[ROUGE-2]{\label{fig:avg_ROUGE-2}\includegraphics[width=0.60\textwidth]{figures/avg_curve_R2_fair.png}} \\
\subfigure[ROUGE-L]{\label{fig:avg_ROUGE-L}\includegraphics[width=0.60\textwidth]{figures/avg_curve_RL_fair.png}}
\end{tabular}
\caption{Learning curves.} %ROC curves: (a) optimization and (b) evaluation
\label{fig:LearningCurves}
\end{figure*}
\begin{comment}
\begin{tcolorbox}[boxrule=0.86pt,left=0.3em, right=0.3em,top=0.1em, bottom=0.05em]
\textbf{Answer to RQ$_{3}$.} Using \texttt{GPT-4.0} as the engine for agents helps \ME obtain a superior recommendation performance.
%Our pipeline is compatible with a wide range of LLMs. While higher-capacity models yield better scores, the framework runs successfully across both commercial and open-source LLMs with no architectural changes.
\end{tcolorbox}
\end{comment}
%\subsection{\rqfourth}%How Does the Self-Improvement Process Evolve Across the Dataset?
%\begin{figure}[H]
% \centering
% \includegraphics[width=0.680\textwidth]{figures/avg_curve_L_fair.png}
% \caption{Average Learning Curve — ROUGE-L (fair up to iter 7).}
% \label{fig:avg_L}
%\end{figure}
%
%\begin{figure}[H]
% \centering
% \includegraphics[width=0.680\textwidth]{figures/avg_curve_R2_fair.png}
% \caption{Average Learning Curve — ROUGE-2 (fair up to iter 7).}
% \label{fig:avg_R2}
%\end{figure}
%
%\begin{figure}[H]
% \centering
% \includegraphics[width=0.680\textwidth]{figures/avg_curve_R1_fair.png}
% \caption{Average Learning Curve — ROUGE-1 (fair up to iter 7).}
% \label{fig:avg_R1}
%\end{figure}
We further investigated how the system improves summary quality over time using the proposed Teacher-Student architecture. For this, we keep track of the ROUGE scores across iterations for each model and computed the average learning curves for ROUGE-L, ROUGE-2, and ROUGE-1 respectively, up to iteration 7. The final results are depicted in Figure~\ref{fig:LearningCurves}. % display the results.
The figure %results
reveals consistent upward trends in ROUGE-L, ROUGE-1, and ROUGE-2 for most models within the early iterations. \texttt{Original LLM set} exhibits the most stable and effective self-improvement behavior, quickly reaching and maintaining high scores. Other models such as \texttt{Mistral-7B-Ins\-truct-v0.3} and \texttt{GPT-3.5-turbo} also improve over iterations, though with more fluctuations. \texttt{Llama-3.2-3B-Instruct} and \texttt{Gemma-2-2b-it} show limited progress, suggesting that lower-capacity models may struggle to utilize refined prompts effectively. This pattern validates the effectiveness of our prompt refinement loop: after each round, \TA leverages feedback (including ROUGE scores and summary outputs) to update the prompt for \SA, leading to consistent performance gains. The early stopping mechanism also helps terminate training for stagnant samples, reducing redundant computation.
The cross-model evaluation demonstrates that \ME achieves reasonable performance across a wide spectrum of LLMs, including lightweight open-source models and high-capacity proprietary systems. This implies %suggests
that the framework is not bound to a specific engine, but rather can flexibly adapt to available resources and deployment contexts.
\vspace{.2cm}
\noindent\fbox{\begin{minipage}{0.98\columnwidth}
\paragraph{\textbf{Answer to RQ$_3$:}} Using \texttt{Original LLM set} as the agent engine %for agents
helps \ME obtain a superior recommendation performance. The self-improvement process is effective across the datasets, with most models showing iterative gains in ROUGE scores. Higher-capacity LLMs benefit more from the prompt refinement mechanism, but even smaller models improve initially, validating the robustness of our training loop.
\end{minipage}}
%\vspace{.2cm}
%\noindent\fbox{\begin{minipage}{0.98\columnwidth}
% \paragraph{\textbf{Answer to RQ$_4$:}} The self-improvement process is effective across the dataset, with most models showing iterative gains in ROUGE scores. Higher-capacity LLMs benefit more from the prompt refinement mechanism, but even smaller models improve initially, validating the robustness of our training loop.
%
%\end{minipage}}
%
%\begin{tcolorbox}[boxrule=0.86pt,left=0.3em, right=0.3em,top=0.1em, bottom=0.05em]
% \textbf{Answer to RQ$_{4}$.} The self-improvement process is effective across the dataset, with most models showing iterative gains in ROUGE scores. Higher-capacity LLMs benefit more from the prompt refinement mechanism, but even smaller models improve initially, validating the robustness of our training loop.
%\end{tcolorbox}
This section provides discussion related to the implications and possible extensions of our work, as well as the threats to validity of the findings.
% \subsection{Implications}
% The experimental results demonstrated that the incorporation of different LLMs in the \ME framework makes differences in the recommendation results.
% The framework could be deployed for other application domains, such as code recommendation, Stack Overflow post generation.
% %code comment generation,
% \subsection{Threats to validity}
% \rev{We are aware of the existence of threats that might harm the validity of the performed experiments as they are presented as follows.}
% \begin{itemize}
% %In particular,
% \item Threats to \emph{internal validity} are related to any confounding factors that could influence our findings. For the comparison with the baselines, we made use of the original implementations.
% \item Threats to \emph{external validity} are ...
% \item Threats to \emph{construct validity} are related to any factor that can compromise the validity of the given observations.
% \end{itemize}
\subsection{Implications}
Our experimental results provide several implications for both research and practice as follows.
\vspace{.2cm}
\noindent
\textbf{Usage.} The findings demonstrate that \ME, as a multi-agent framework, consistently outperforms various single-agent LLMs in summarization tasks. This suggests that the collaborative Teacher--Student refinement loop and the dynamic stopping strategy are effective mechanisms for generating concise and faithful summaries. %The comparison between \ME and single-agent baselines highlights the strength of multi-agent collaboration.
By assigning specialized roles to different agents and enabling iterative refinement, \ME is able to generate summaries that are both more accurate and more faithful to the original content. This indicates that the benefit of multi-agent systems lies not only in diversity of models but also in the structured cooperation among them. In practice, such a system could be integrated into SE workflows to automatically produce short descriptions of lengthy documentation, thereby reducing cognitive load for developers and end users.
\vspace{.2cm}
\noindent
\textbf{Use cases.} While our evaluation focuses on Markdown %\RM
files and application descriptions, the approach could potentially be extended to other domains. For example, the framework could support software documentation summarization, where lengthy requirements or technical reports are converted into concise and accessible descriptions. It will also enhance developer support platforms, such as automatically generating short ``About'' sections for repositories or applications, improving accessibility for first-time users. Furthermore, it might benefit community-driven Q\&A platforms by condensing verbose answers into shorter and more searchable forms.
\begin{comment}
\vspace{.2cm}
\noindent\fbox{\begin{minipage}{0.98\columnwidth}
\paragraph{\textbf {Finding 1.}} Multi-agent collaboration leads to more stable and higher-quality summaries compared to single-agent LLMs, underscoring the practical advantage of role specialization and collaborative refinement.
\end{minipage}}
\end{comment}
\begin{comment}
\vspace{.2cm}
The evaluation of the dynamic iteration strategy shows that, despite producing results comparable to the non-dynamic setting, it requires significantly fewer training steps. This reveals that adaptive stopping criteria help avoid wasted computation while still capturing meaningful improvements. Such efficiency is especially important in real-world deployments, where computational resources are often limited.
\vspace{.2cm}
\noindent\fbox{\begin{minipage}{0.98\columnwidth}
\paragraph{\textbf {Finding 2.}} The dynamic iteration strategy improves efficiency by reducing unnecessary training steps while maintaining comparable quality, making the framework suitable for resource-constrained environments.
\end{minipage}}
\end{comment}
%\vspace{.2cm}
%The cross-model evaluation demonstrates that \ME achieves reasonable performance across a wide spectrum of LLMs, including lightweight open-source models and high-capacity proprietary systems. This implies %suggests
%that the framework is not bound to a specific engine, but rather can flexibly adapt to available resources and deployment contexts.
\begin{comment}
\vspace{.2cm}
\noindent\fbox{\begin{minipage}{0.98\columnwidth}
\paragraph{\textbf {Finding 2.}} The framework remains robust across different LLM backbones, indicating strong portability and adaptability in diverse real-world scenarios.
\end{minipage}}
\end{comment}
\vspace{.2cm}
\noindent
\textbf{Broader implications.} Beyond the specific task of summarizing \RM and application descriptions, our findings suggest that LLM-based multi-agent systems could serve as a general approach to improve the reliability, scalability, and controllability of AI-assisted summarization. The principles of dynamic prompt refinement and agent specialization can also inspire solutions in adjacent tasks such as code recommendation, bug report summarization, and automated knowledge management.
\subsection{Concerns and Limitations}
While the empirical evaluation demonstrates the effectiveness of \ME, there are various concerns and limitations as follows.
\begin{itemize}
\item \textbf{Diversity of datasets and application contexts.} The evaluation primarily focused on \GH \RM files and Google Play application descriptions. Although these are representative sources of software documentation, they may not capture the full diversity of real-world documents such as technical reports, requirement specifications, or bug reports. Consequently, the generalizability of \ME to heterogeneous domains still requires further investigation.
\item \textbf{Task-oriented and user-centered validation.} Our %current
evaluation relied on benchmark datasets and quantitative metrics (\eg ROUGE, cosine similarity). While these measures have been widely used, they may not fully reflect the utility of summaries in practical scenarios where developers, %students,
or end-users consume documentation. Conducting user studies, for instance through crossover experiments comparing human- vs. agent-generated summaries, would provide a deeper understanding of \ME's real-world impact.
\item \textbf{Dependence on training data and potential biases.} Although our dataset curation aimed to ensure quality and diversity, the reliance on app descriptions from the Google Play ecosystem may introduce inherent biases, such as domain-specific language or stylistic conventions. Moreover, overlaps with LLM pretraining corpora could unintentionally inflate performance, raising concerns about overfitting. To address this, future evaluations should incorporate datasets from independent sources, including industrial documentation and student-generated artifacts.
\item \textbf{Computational cost and scalability.}
Despite the efficiency gains achieved by the dynamic iteration strategy, the orchestration of multiple LLM-based agents inevitably introduces computational overhead. Scaling \ME to larger datasets or integrating it into production pipelines may require further optimization or hybrid strategies to balance accuracy and resource consumption.
\end{itemize}
\subsection{Future Research Directions}
Future work on \ME can be extended along several promising directions:
\begin{itemize}
\item \textbf{Expanding application domains.} While the current evaluation has focused on Markdown files and application descriptions, future work could apply \ME to additional domains such as code summarization, technical documentation, engineering reports, or biomedical texts. Such extensions would allow the framework to demonstrate its adaptability to highly diverse and domain-specific documents.
\item \textbf{Reinforcement learning from feedback.} Another direction is to incorporate reinforcement learning with human feedback (RLHF) or agent feedback (RLAIF) to optimize the prompt refinement and summarization process. This could provide stronger alignment with user expectations and continuously improve the quality of generated summaries.
\item \textbf{Retrieval-augmented generation.} Integrating external retrieval mechanisms into the multi-agent loop would allow agents to access contextual information beyond the input document. This hybrid approach could reduce the risk of missing critical background knowledge and enhance factual accuracy.
\item \textbf{Cost-aware optimization.} Future work should also explore efficient allocation of computational resources by combining lightweight open-source LLMs with larger proprietary ones in a hybrid manner. Such resource-aware scheduling could reduce cost while sustaining high-quality results, making the system more practical in production environments.
\item \textbf{Explainability and transparency.} Finally, improving interpretability by designing logging, tracing, and decision-explanation mechanisms for each agent could strengthen trust in industrial deployments. Providing insights into why a specific summary was produced would enhance both reliability and accountability in real-world software engineering workflows.
\end{itemize}
\subsection{Threats to Validity}
\begin{itemize}
\item \textbf{Internal Validity} is related to the factors within the experimental design that may bias or distort the results. In our approach, the Teacher--Student refinement loop and dynamic stopping strategy are sensitive to hyperparameters such as iteration thresholds and ROUGE cutoffs. Suboptimal settings might have influenced convergence speed and output quality. %Furthermore, overlaps between the curated dataset and the pretraining corpora of the LLMs could have artificially inflated performance, as some models might have been exposed to similar content during training. Although filtering and diverse sampling were applied, we cannot entirely rule out leakage.
Another internal factor is the selection of baseline models: stronger or weaker baselines may lead to different perceived relative improvements. Moreover, single-agent baselines are highly sensitive to model setup (e.g., prompt design, hyperparameter configuration), which may lead to variability in their performance and affect the fairness of comparison. Future experiments should incorporate cross-validation with randomized seeds, multiple runs per configuration, and more systematic baselines to minimize such risks.
\item \textbf{External Validity} concerns the generalizability of our findings to other domains, document types, and usage contexts. While \GH \RM and Google Play descriptions are representative forms of software documentation, they do not capture the broader diversity of real-world documents such as industrial requirement specifications, technical reports, or bug-tracking entries. The language style, structural heterogeneity, and domain-specific jargon in these contexts may pose additional challenges not addressed by our current evaluation. Similarly, the experiments were limited to English-language datasets; generalizing to multilingual documentation or low-resource languages remains unexplored. %Finally, the computational overhead of coordinating multiple agents may restrict the framework’s scalability in production-scale deployments. Future research should validate the approach in diverse domains, industrial-scale corpora, and multilingual contexts to ensure robust applicability.
\item \textbf{Construct Validity} is whether the employed measures and datasets adequately capture the concept of summarization quality. In our evaluation, we relied primarily on ROUGE and cosine similarity metrics, which are well-established but may fail to reflect semantic fidelity, factual consistency, or user-perceived usefulness of the generated summaries. For instance, a summary with high n-gram overlap may still be misleading in meaning. Additionally, we adopted the short ``About'' descriptions from Google Play as ground truth references; however, such descriptions are often written with a marketing or promotional purpose, rather than serving as strictly informative summaries. This introduces the risk that models learn to mimic stylistic tendencies rather than optimizing for clarity or completeness. While we attempted to mitigate this by curating high-quality subsets and applying filtering thresholds, human-centered evaluations (\eg user studies with developers or app users) would provide a more comprehensive assessment of summary utility.
% \item \textbf{Conclusion Validity} relates to the soundness of inferences drawn from the observed results. Our findings were supported by distributional analyses and comparative metrics, which consistently favored the proposed multi-agent framework. However, the study did not employ statistical significance testing across all evaluation subsets. Without formal hypothesis testing, there remains a risk of overinterpreting performance differences that may be due to variance in the data. Moreover, while we observed consistent trends across multiple datasets, \ie D$_1$--D$_4$, the lack of longitudinal or task-specific replications limits the certainty of general conclusions. To reinforce conclusion validity, future studies should integrate statistical hypothesis testing (e.g., McNemar’s test, paired t-tests), report confidence intervals, and replicate experiments across independent datasets.
\end{itemize}
\begin{comment}
\subsection{Visionary Outlook}
Metagente points toward a paradigm shift in \textbf{software knowledge management}:
from static documentation to continuously orchestrated \textit{``living documentation''}
maintained by specialized agents. Rather than treating summarization as the final
goal, we envision \textbf{knowledge orchestration} as an infrastructural layer in
software engineering---where artifacts (README files, specifications, tickets,
change logs) remain synchronized with source code evolution and CI/CD pipelines.
Beyond the near-term use cases discussed in the Implications section, the long-term
vision is a \textbf{transformation of human roles}: developers move from being
\textit{documentation authors} to \textbf{curators of intents and quality standards},
specifying acceptance constraints while agents autonomously update, validate,
and recover inconsistencies. This transition entails the definition of
\textbf{standardized communication schemas} (e.g., documentation intents, provenance,
and lineage of summaries) as well as \textbf{agent governance mechanisms}
(auditability, accountability, and boundaries of delegation) to ensure safe
adoption at enterprise scale.
Looking further ahead, we foresee the emergence of \textbf{adaptive SE ecosystems}:
knowledge pipelines that self-regulate based on project context, organizational
policies, and compliance risks. Evaluation will increasingly move beyond
offline metrics toward \textbf{task-grounded assessment} embedded in workflows,
capturing real impact on productivity and quality of software changes while
complementing---rather than replacing---traditional measures such as ROUGE or
cosine similarity.
\end{comment}
% \subsection{Summarization in Software Engineering}
% \subsection{LLM-based Multi Agent Systems} % in Software Engineering
Recently, there has been a surge in the applications of LLM-based MAS in Software Engineering. This section reviews some of the most notable applications, % of LLM-based MAS in Software Engineering,
as well as MAS for summarization and prompt optimization in agent collaborations.
\subsection{LLM-based Multi-Agent Systems in Software Engineering}
A number of representative applications highlight the potential of LLM-based multi-agent systems (LaMAS) across the SE lifecycle. In program synthesis, systems like \textsc{CodePori}~\citep{rasheed2024codepori} and \textsc{AgentMesh}~\citep{khanzadeh2025agentmesh} demonstrate how assigning specialized roles to multiple agents—such as manager, developers, reviewers, and verification agents--enables the generation of functional, production-level software directly from natural language requirements. \textsc{CodePori} shows strong performance on the \textit{HumanEval} benchmark and further scales to end-to-end applications spanning thousands of lines of code, while \textsc{AgentMesh} showcases artifact-centric communication and iterative plan–code–review loops for reliable development. Moving beyond coding, \textsc{MAGIS}~\citep{tao2024magis} addresses repository-level issue resolution on GitHub by coordinating multiple agents to locate files, handle long contexts, and implement complex fixes, significantly outperforming single-agent LLMs on the \textit{SWE-bench} benchmark.
In industrial contexts, \textsc{GoNoGo}~\citep{Khoee2024GoNoGo} introduces a planner-actor setup to support automotive software release decisions, already deployed at Volvo to improve accuracy and reduce manual overhead in risk-sensitive processes. Similarly, \textsc{CogniSim}~\citep{cinkusz2025agile} integrates cognitive agents into Agile and DevOps workflows, showing improvements in backlog refinement, testing, and deployment, though it also raises questions about transparency and human–agent collaboration. At higher abstraction levels, frameworks such as \textsc{MAAD}~\citep{li2025maad} and multi-agent design/refactoring models automate architecture generation, design trade-offs, and quality balancing by assigning domain-specialized agents (e.g., performance, security, UI/UX) and employing structured evaluation protocols.
LaMAS is emerging as a promising way to streamline software engineering--from requirements to deployment--while also highlighting challenges such as scalability, reliability, security, and the role of human oversight in collaborative, intelligent development.
\subsection{Summarization} %Multi-Agent Systems for
\ME exemplifies the growing shift toward multi-agent systems (MAS) to enhance the quality, factual consistency, and controllability of LLM-based summarization. By assigning specialized roles, \ie \EA, \SA, and \TA, \ME decomposes the summarization task into modular subtasks with tailored prompts. This role-based specialization is complemented by adaptive strategies, including early halting of low-potential samples and real-time prompt adjustments, enabling efficient and high-quality generation.
%
This design builds upon prior MAS approaches like MAMM-RE\-FINE~\citep{pan2024mammrefine}, which introduced a \textsc{Detect}-\textsc{Critique}-\textsc{Refine} pipeli\-ne where collaborative agents iteratively improve summaries. Notably, its Multi-Agent Single-Model (MASM) configuration with reranking significantly outperforms single-agent baselines. Metagente extends these ideas by integrating adaptive prompt control and dynamic role coordination throughout the generation process.
Other MAS-based summarization systems further demonstrate the versatility of collaborative agents. Chain-of-Agents~\citep{zhang2024chainofagents} processes long-form input via chained agents under manager oversight, while D\&R~\citep{zhou2025debate} distills stronger models through structured debates and preference optimization. MADRA~\citep{wang2023apollo} enhances agent reasoning with retrieval-augmented evidence to reduce hallucination, and SR-DCR~\citep{zhou2025selfreflective} leverages asymmetric debates and token-level confidence for arbitration. Even in adjacent domains, such as legal argumentation, the Reflective Multi-Agent framework~\citep{zhang2025reflective} highlights the benefits of iterative role-driven collaboration for improving factual and ethical integrity.
Beyond summarization, several MAS frameworks illustrate the power of structured agent collaboration across broader reasoning and task-solving settings. OPRO~\citep{yang2024opro} employs different LLMs as optimizers, using natural language prompts to iteratively generate and refine solutions. APE~\citep{zhou2023ape} adopts a similar philosophy by generating candidate instructions and refining them via semantic similarity and evaluation metrics. Camel~\citep{li2024camel} introduces a role-playing framework where agents are guided by inception prompting, enabling instruction-following cooperation and generation of multi-agent conversational data. \texttt{MetaGPT}~\citep{hong2024metagpt} embeds human workflows into modular agent behaviors to address hallucination and coordination challenges. Complementary to these systems, \texttt{AutoGen}~\citep{wu2024autogen} and \texttt{LangChain}\footnote{\url{https://docs.langchain.com}} facilitate the development of LLM-based applications via conversation-driven agents and modular programming abstractions. \texttt{AutoGen} promotes human-in-the-loop interactions through multi-turn dialog and agent orchestration, while \texttt{LangChain} streamlines integration with external tools and data sources.
Altogether, these studies contextualize and reinforce \ME's contributions: dynamic agent specialization, modular workflows, and adaptive control as key mechanisms for building faithful and effective multi-agent summarization systems.
\subsection{Prompt Optimization in Agent Collaboration}
\ME emphasizes prompt optimization as a core mechanism for effective multi-agent collaborations. Its architecture incorporates dynamic, role-specific prompt refinement—most notably via \TA, which continuously updates summarization prompts based on ROUGE-L feedback. This closed-loop adaptation enables agents to specialize and respond to diverse inputs with increased coherence, factuality, and control.
%
This design is in line with broader efforts to optimize prompts across agent-based systems. MASS~\citep{zhou2025mass} introduces a framework that jointly searches for both communication topologies and agent prompts, demonstrating that interleaving local and global prompt tuning leads to better system-wide coordination. Similarly, NEXUSSUM~\citep{kim2025nexussum} applies modular prompt designs across hierarchical agent roles--Preprocessor, Summarizer, and Compressor—using Chain-of-Thought reasoning and few-shot examples to guide generation.
Further advances in Automated Prompt Optimization (APO) support the principles underpinning \ME. MARS~\citep{zhang2025mars} proposes a multi-agent dialogue framework, where agents like Planner and Teacher-Critic-Student collaboratively evolve prompts. Shen et al.~\citep{shen2025optimizing} explored feedback-driven refinement, showing that group-level, online optimization consistently improves collaboration quality. Meanwhile, SEE~\citep{cui2025see} and EXPO~\citep{kong2025meta} treated prompt tuning as a high-dimensional or dynamic problem, applying metaheuristics and bandit learning to refine both instructions and exemplars.
Collectively, these studies reinforce the design choices in \ME: prompt modularity, agent-level specialization, and adaptive optimization loops are essential to building robust, controllable LLM-based multi-agent systems.
%Conclusions go here.
This paper presented \ME--a novel approach to summarization of software documents leveraging LLM-based multi agents. Using datasets collected for mobile apps' documents, we conducted an empirical evaluation to study the performance of \ME. Moreover, we also compared dynamic vs. non-dynamic strategies, evaluating generalization across test subsets. The experimental results showed that \ME is able to generate highly relevant summaries, outperforming single LLMs. Our work showcases the potential of LLM-based MAS in the generation of software documents. %in the Software Engineering domain.
Future work will focus on the deployment of the proposed architecture to a wider range of application domains, beyond the scenarios investigated in this study. In particular, we plan to explore tasks such as code recommendation, automated code comment generation, and other software engineering activities where collaborative LLM-based agents can provide added value. A key challenge in these extensions will be to refine and optimize the architecture of the LLM-based MAS, with the dual objective of improving the overall effectiveness of the system and reducing its computational cost, thus making it more suitable for real-world integration in developer workflows.
In addition, an important research direction concerns the definition of benchmarking framework tailored to LLM-based MAS in Software Engineering. Such a framework should not only enable rigorous comparisons across different agent architectures and coordination strategies, but also include metrics that capture aspects such as accuracy, efficiency, robustness, scalability, and fairness. Establishing these benchmarks will be instrumental for the community to assess progress in this emerging area and to guide the development of more reliable, efficient, and trustworthy multi-agent solutions.