ToolMem: Enhancing Multimodal Agents with Learnable Tool Capability Memory
Extracted source text · arXiv 2510.06664
\documentclass{article}
\usepackage[preprint]{neurips_2025}
\usepackage{tabularx}
\usepackage{microtype}
\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{lineno}
\usepackage{multirow}
\usepackage{makecell}
\usepackage[table]{xcolor}
\usepackage{enumitem}
\usepackage{wrapfig}
\usepackage{bbm}
\usepackage{dsfont}
\usepackage{amsmath}
\usepackage[most]{tcolorbox} % Load the package
\usepackage{color-edits}
\addauthor{zw}{orange}
\addauthor{sx}{green}
\usepackage{xspace}
\newcommand{\tm}{\textsc{ToolMem}\xspace}
\newcommand{\gr}{\textsc{Generic}\xspace}
\newcommand{\fs}{\textsc{Few-Shot}\xspace}
\newcommand{\gb}{\textsc{GenAI-Bench}\xspace}
\newcommand{\bb}{\textsc{BiGGen Bench}\xspace}
\definecolor{darkblue}{rgb}{0, 0, 0.5}
\hypersetup{colorlinks=true, citecolor=darkblue, linkcolor=darkblue, urlcolor=darkblue}
\title{\tm: Enhancing Multimodal Agents with \\Learnable Tool Capability Memory}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
% \author{%
% David S.~Hippocampus\thanks{Use footnote for providing further information
% about author (webpage, alternative address)---\emph{not} for acknowledging
% funding agencies.} \\
% Department of Computer Science\\
% Cranberry-Lemon University\\
% Pittsburgh, PA 15213 \\
% \texttt{hippo@cs.cranberry-lemon.edu} \\
% }
\author{%
Yunzhong Xiao$^{1}$ \quad
Yangmin Li$^{1}$ \quad
Hewei Wang$^{1}$ \quad
Yunlong Tang$^{2}$ \quad
Zora Zhiruo Wang$^{1}$ \\
$^{1}$Carnegie Mellon University \quad
$^{2}$University of Rochester \\
\texttt{yunzhonx@alumni.cmu.edu, yangmin2@alumni.cmu.edu, heweiw@alumni.cmu.edu,} \\
\texttt{yunlong.tang@rochester.edu, zhiruow@andrew.cmu.edu} \\
}
\begin{document}
\maketitle
\begin{abstract}
% background
Agents utilizing tools powered by large language models (LLMs) or vision-language models (VLMs) have demonstrated remarkable progress in diverse tasks across text and visual modalities.
% neural tools
Unlike traditional tools such as calculators, which give deterministic outputs, neural tools perform uncertainly across task scenarios. While different tools for a task may excel in varied scenarios, existing agents typically rely on fixed tools, thus limiting the flexibility in selecting the most suitable tool for specific tasks.
% motivation: human
In contrast, humans snowball their understanding of the capabilities of different tools by interacting with them, and apply this knowledge to select the optimal tool when solving a future task.
% our method
To build agents that similarly benefit from this process, we propose \tm that enables agents to develop memories of tool capabilities from previous interactions, by summarizing their strengths and weaknesses and storing them in memory; at inference, the agent can retrieve relevant entries from \tm, and select the best tool to solve individual tasks more accurately.
% results
We evaluate \tm on learning varied text generation and text-to-image generation neural tools. Compared to no-memory, generic agents, we find \tm-augmented agents predict tool performance 14.8\% and 28.7\% more accurately across text and multimodal generation scenarios.
Moreover, \tm facilitates optimal tool selection among multiple choices by 21\% and 24\% absolute increases in respective scenarios.
\end{abstract}
\input{sections/1_intro}
\input{sections/2_toolmem}
\input{sections/3_perf-predict}
\input{sections/4_advanced-scenarios}
\input{sections/related_work}
\input{sections/conclusion}
\bibliographystyle{plainnat}
\bibliography{references}
\appendix
\newpage
% \begin{center}
% \Large\textbf{Appendix}
% \end{center}
\input{appendix/image-eval-topk}
% \input{appendix/text-gen-details}
% \input{appendix/model-selection}
% \input{appendix/ablation_study}
\input{appendix/prompts}
\input{appendix/case-study}
\input{appendix/computation}
\input{appendix/impact_limitation}
% \newpage
% \input{sections/checklist}
\end{document}
\section{Introduction}
% tool-powered agents are great
Recent advances in agents have drastically reshaped the landscape of generative tasks, especially by utilizing powerful tools supported by large language models (LLMs) \citep{schick2023toolformerlanguagemodelsteach} or vision-language models (VLMs) \citep{gao2025multimodalagenttuningbuilding,Carrasco_2025,clip}.
These agents supported with task-specific neural tools have demonstrated impressive capabilities across various downstream applications such as instruction following and reasoning in text modality \citep{kim2025biggenbenchprincipledbenchmark,ouyang2022training}, as well as visual understanding and editing in cross-modal scenarios \citep{shen2023hugginggptsolvingaitasks,wang2024genartistmultimodalllmagent, tang2025captionvideofinegrainedobjectcentric}.
% deterministic -> neural tools, uncertain
Unlike traditional tools such as \texttt{calculator} or \texttt{sql\_executor} that always produce deterministic outputs for a given input, tools supported by neural models vastly expand the spectrum of tasks that cannot be tackled by deterministic tools, such as answering ad-hoc questions \citep{chen2017readingwikipediaansweropendomain,karpukhin2020densepassageretrievalopendomain} or search relevant news on the web \citep{nakano2022webgptbrowserassistedquestionansweringhuman,dunn2017searchqanewqadataset}. Nonetheless, what are brought along with this wider applicability of neural tools is the uncertainty of their outputs---there is no guarantee that a \texttt{QA} tool can answer an arbitrary question correctly, especially when the answer to that question is open-ended \citep{lewis2021retrievalaugmentedgenerationknowledgeintensivenlp} or changes over time \citep{guu2020realmretrievalaugmentedlanguagemodel}.
% use fixed tools --> no selection
Moreover, current agents typically solve tasks using a pre-designated set of tools by human experts, and only get to learn to use these tools at test time by contextualizing on static textual descriptions for tool functionalities \citep{qin2023toolllm,guo2024stabletoolbench}. More often than not, the agent may be provided with multiple tools possessing similar functionalities \citep{tang2023toolalpaca,li2023api}, thus making it harder to distinguish the best tool to use for a certain task, because the agent has no prior knowledge about the individual expertise of these seemingly similar tools \citep{yao2023reactsynergizingreasoningacting}. Taking \autoref{fig:teaser} as an example, the two text-to-image tools have different expertise: while both might be good at representing lively environment, the later one may excel at rendering text. Given the task of generating image with text, the agent better pick the later tool to ensure better performance. % Beyond accuracy, efficiency may be another consideration --- if two tools perform similarly well on simple visual question-answering tasks, it is more desirable if the agent learns to pick the more efficient tool to optimize cost or latency.
% existing methods on learning tool prior knowledge
Human gradually learn to distinguish and select between tools by interacting with tools and gathering knowledge about their individual properties through past experiences.
Recent methods have explored some techniques to gather explicit knowledge from experiences by reflecting on \citep{shinn2023reflexion} and refining \citep{madaan2023selfrefineiterativerefinementselffeedback} generated solutions, or constructing short-term workflows \citep{wang2024agentworkflowmemory} and long-term memory \citep{xu2025amemagenticmemoryllm}, and even curating new tools to assist target downstream tasks \citep{wang2023voyageropenendedembodiedagent,wang2024troveinducingverifiableefficient}.
Besides explicitly storing the knowledge, approaches like Toolformer \citep{schick2023toolformerlanguagemodelsteach} and ToolLLM \citep{qin2023toolllmfacilitatinglargelanguage} also explored parametric updates via self‑supervised fine‑tuning and instruction tuning on tool-using experiences.
However, these methods focus on learning task-specific procedures with a fixed tool, instead of distinguishing between a group of functionally similar tools to optimize downstream tool selection and performance. More concretely, agents lack a mechanism to build and update an internal, dynamic memory that encapsulates the strengths and weaknesses of diverse generative tools.
\begin{figure}[t]
% \vspace{-5mm}
\centering
\includegraphics[width=\linewidth]{figures/teaser.pdf}
\caption{Example of identifying better-performing tools based on learned \tm. For example, compared to the image generation tool instantiated by {\it Midjourney}, {\it SDXL-Base} tool is better at rendering short text such as the queried ``Eco Market''. We thus store this information in \tm and prioritize selecting {\it SDXL-base} in later tasks that ask to render short texts.}
\label{fig:teaser}
\vspace{-2mm}
\end{figure}
% our method
To bridge this gap, we introduce \tm, a framework that empowers agents to learn and apply tool-specific capability memories (\S\ref{sec:2:toolmem}). \tm is built upon three core components: (1) a structured capability memory initialized with a taxonomy that categorizes tool behaviors by proficiency levels, enabling consistent updates and targeted retrieval; (2) a feedback generation process that evaluates tool outputs using either human annotations or automated metrics—such as LLM-based judgment scores—to extract fine-grained insights into tool performance; and (3) a dynamic memory update mechanism that incorporates new experiences through retrieval-augmented generation (RAG), allowing the agent to refine, revise, and expand its tool knowledge over time. During inference, \tm-augmented agents retrieve relevant memory entries based on the current task and inject them into the input context, enabling more accurate tool selection and solution generation.
We first study: \textit{Can agents learn and estimate tool capabilities through interaction?} (\S\ref{sec:3:perf-predict})
We construct \tm about image generation tools on \gb \citep{li2024genaibenchevaluatingimprovingcompositional} and various text-oriented tools on \bb \citep{kim2025biggenbenchprincipledbenchmark}, then evaluate if \tm-augmented agent can accurately estimate the performance of these tools. We find that \tm can predict solution quality using varied tools 14.8--28.7\% more accurately than \gr agent without prior knowledge about tools.
Next, we ask: \textit{Can agents leverage \tm to optimize performance through tool selection?} (\S\ref{sec:4:advanced-scenarios})
We examine \tm-augmented agents on selecting the best-performing tool for individual tasks among multiple tool candidates, and find it outperforming \gr agent approach by 21\% and 24\% on \gb and \bb benchmarks (\S\ref{sec:4:advanced-scenarios}).
% Besides solely optimizing task performance, we explore another scenario where we take computation cost into joint consideration---\tm-augmented agents can similarly yield \zwcomment{xx.x\% and xx.x\%} better choices than baselines (\S\ref{sec:4.2:cost-efficiency}), demonstrating the practical utility of \tm.
% summary
In short, our work builds memory-adaptive tool-using agents, and uniquely addresses the dynamic nature of neural tools by enabling agents to grasp an evolving understanding of accessible tools.
\section{\tm: Learning Tool Capability Memory}
\label{sec:2:toolmem}
In this section, we introduce our \tm framework that initializes a structured memory (\S\ref{sec:2.1:mem-init}), learns and updates tool capability knowledge through experiences (\S\ref{sec:2.2:learn-toolmem}), and eventually solves new tasks according to retrieved memory entries (\S\ref{sec:2.3:task-solving}).
Overall, \tm equips agents to build tool capability memories progressively learned from past tool usage experiences, and leverages this dynamic knowledge repository to enhance its decision-making.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Structured Memory Initialization}
\label{sec:2.1:mem-init}
The evolution of \tm is an iterative process that builds upon previous states. As such, the initial memory plays a critical role in guiding subsequent developments.
Our goal is for agents $\pi$ to learn the capabilities of a set of tools $T = \{t\}$ with similar functionalities to solve a particular task (e.g., text-to-image generation, instruction following), where each tool $t$ is instantiated with a neural model (e.g., Midjourney, GPT).
For each tool $t$, we initialize its capability memory $\mathcal{M}_t = \emptyset$ and aim to collect a set of knowledge entries $M_t = \{m_t\}$, each describing some properties of tool behaviors in natural language (NL).
While a tool $t$ can be good at certain scenarios and bad at others, we categorize memory entries based on the varied proficiency levels the agent possesses, namely $C$ = $\{${\it proficient at} ($^p$), {\it good at} ($^g$), {\it bad at} ($^b$), {\it weak at} ($^w$) $\}$. %, as illustrated in \autoref{fig:memory_taxonomy}.
Moreover, these proficiency-aware categories can be associated with concrete numerical measures of tool performance, $+2$, $+1$, $-1$, $-2$, when more accurate measures are useful. We thus denote \tm as the union of all memory categories $\mathcal{M} = \cup_{c \in C} \mathcal{M}^{c}$. This structured memory initialization offers flexibility in memory update and retrieval, which we will provide more details on next.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Learning Tool Capabilities from Experiences}
\label{sec:2.2:learn-toolmem}
\noindent \textbf{Constructing Memory from Experiences} \quad
% an experience to learn from
Agents learn more about the tool capabilities by interacting with them in task-solving experiences.
More concretely, the agent begins by receiving a task $q$ (e.g., an NL instruction), as well as a corresponding solution $s_t$ using a designated tool $t$ (e.g., an image generated by a text-to-image tool $t$, an answer generated by a text generation tool $t$).
As an informative learning signal of this task, a reward system $R$ provides feedback $r_t = R(q, s_t)$ on the quality of the solution $s_t$ in tackling the task $q$, which could be numerical quality measures (e.g., 1-5 Likert scale) or NL feedback (as exemplified in \autoref{fig:feedback_generation}).
Together, the task $q$, tool-using solution $s_t$, and its quality feedback $r_t$ form an experience $e_t = (q, s_t, r_t)$ from interaction with tool $t$, upon which we can construct capability memories about.
In our study, we focus on building \tm from experiences annotated by human experts for image generation tools and by llm-as-a-judge for text generation tools. These annotations are provided as part of the training datasets, typically accompanied by high-quality solutions and reliable feedback.
% induce memory from an experience
We introduce a memory induction module $I_{\it LM}$ instantiated with an LM, to summarize the capabilities of tool $t$ from a pertaining experience $e_t$ via $I_{\it LM}(e_t) \rightarrow m_t$.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{figures/pipeline.pdf}
\caption{Illustration of the \tm construction process from past experiences (left), as well as \tm-augmented task-solving and capability analysis at downstream (right).}
\label{fig:feedback_generation}
\end{figure}
\noindent \textbf{Updating Agent Memory} \quad
Given a new memory entry $m$ derived from an experience $e$, a naive memory update would directly augment the existing memory repository by $\mathcal{M} \leftarrow \mathcal{M} \cup \{m\}$. However, this simplistic approach often introduces redundancy and inflates computational overhead. Specifically, entries that are highly similar or overlapping may accumulate without being appropriately consolidated, and prior incomplete or suboptimal entries might persist without receiving necessary updates.
To enable flexible memory refinement capabilities, we introduce a structured and refined memory updating procedure. Formally, we denote the agent's current memory as
$\mathcal{M} = \cup_{c \in C}\mathcal{M}^c$
partitioned into $C$ categories introduced in \S\ref{sec:2.1:mem-init}. Given the newly constructed experience $e = (q, s, r)$, we first perform a targeted retrieval within each memory category $\mathcal{M}^{c}$ to identify the top-$k$ most semantically relevant memory entries.
$\mathcal{M}^{c}_{\text{retrieved}} = \textit{Retriever}(e, \mathcal{M}^{c}), \forall c \in C$ where $\textit{Retriever}(\cdot, \cdot)$ calculates some similarity measure (e.g., cosine similarity) computed between embedded memory entries.
Next, we construct a condensed contextual set from these retrieved entries across all categories:
$\mathcal{M}_{\text{retrieved}} = \bigcup_{c \in C} \mathcal{M}^{c}_{\text{retrieved}}$.
Subsequently, the condensed contextual set $\mathcal{M}_{\text{context}}$ along with the new experience $e = (q, s, r)$ is passed into the \tm induction module $I$ to for entry refinement
$\mathcal{M}_{\text{updated}} = I(\mathcal{M}_{\text{retrieved}}, e)$ then memory update $\mathcal{M} \leftarrow (\mathcal{M} \setminus \mathcal{M}_{\text{context}}) \cup \mathcal{M}_{\text{updated}}$
The refinement process involves adding novel insights, updating incomplete entries, removing redundant information, and merging semantically related entries (see exact prompts in \S\ref{app:prompts}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Solving Tasks with Retrieved Memory}
\label{sec:2.3:task-solving}
Analogous to how humans draw upon past experiences when confronted with a new task, for a given task $q'$ at test time, our task-solving agents can (i) retrieve top-$k$ relevant memory entries from \tm utilizing a retriever model ${\it Retriever}(q', \mathcal{M}) \rightarrow \mathcal{M}_{q'}$, and then (ii) augmented these relevant memory entries to input contexts to generate solutions $\pi(q', \mathcal{M}_{q'}) \rightarrow s'$ or estimate tool performance $\pi(q', \mathcal{M}_{q'}) \rightarrow r'$. As \tm contains multiple categories $\mathcal{M} = \cup_{c} \mathcal{M}^c$, we separately retrieve top-$k$ relevant entries from each category $\mathcal{M}^c$ and provide all of them in context.
This RAG strategy addresses two key challenges: (1) relevance filtering: focusing on contextually useful entries and discarding irrelevant ones; (2) scalability: preventing computation overload by restricting retrieval to a small number of entries per category.
\section{Experiments: Tool Performance Prediction}
\label{sec:3:perf-predict}
In this section, we first introduce the baseline approaches in comparison to our \tm (\S\ref{sec:3.1:baselines}), then demonstrate \tm's effectiveness on two representative tasks --- text generation (\S\ref{sec:3.4:text-gen}) and text-to-image generation evaluation (\S\ref{sec:3.3:image-eval}).
%
We focus on \tm usage under the {\it performance prediction} scenario in this section. Later in \S\ref{sec:4:advanced-scenarios}, we further demonstrate its utility when extrapolating to \textit{model selection} under accuracy and efficiency considerations.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Baseline and \tm Approaches}
\label{sec:3.1:baselines}
We introduce the two baseline approaches and our \tm method for experimentation.
\noindent \textbf{Generic Agent} \quad
Our first baseline is a vanilla agent without tool-specific memory beyond general information (e.g., tool's name, basic skills, input/output) at inference. This setup is similar to the current tool-selection approach adopted by state-of-the-art generative agents \citep{wang2024genartistmultimodalllmagent}. Agents must rely heavily on the parametric knowledge acquired through the training of their backbone LM to estimate tool capabilities.
\noindent \textbf{Few-Shot Memory} \quad
Without the need to induce \tm, we propose another stronger baseline on top of \gr that also retrieves top-12 training examples, i.e., (task, score, rubric) triplets, that are most relevant to the test example at hand.
We denote this the \fs setting and compare it to our \tm-augmented LM, to show the effectiveness of derived capability memory against implicit information from raw examples.
\noindent \textbf{\tm} \quad
Instead of employing memory with generic or raw experiences, our \tm contains carefully induced and updated memory from past experiences. Particularly, we induce memory entries from the same set of training examples used for retrieval in \fs setting, and test \tm-augmented LM on the same test set.
%
For generations, we use GPT-4o (the \texttt{gpt-4o-mini-2024-07-18} checkpoint) for all approaches by default; we use a temperature $t = 0.0$ and sample $n=1$ output.
For retrieval, we use OpenAI’s \texttt{text-embedding-ada-002}\footnote{\url{https://platform.openai.com/docs/models/text-embedding-ada-002}} model to generate embeddings and implement RAG using ChromaDB\footnote{\url{https://www.trychroma.com/}} that measures task-memory similarity using (negative) cosine distance.
During \tm construction, we retrieve $k=6$ entries from each memory category and perform memory update on them.
At test time, the agent retrieves $k=12$ entries from each memory category and grounds on them to generate task solutions; this setting scores the best in our ablation study on $k$ value choices in \S\ref{app:topK}.
\input{sections/experiment/31_text-gen}
\input{sections/experiment/32_image-eval}
\subsection{Text Generation}
\label{sec:3.4:text-gen}
% \noindent \textbf{Dataset: \bb} \quad
\paragraph{Dataset: \bb}
We evaluate \tm on diverse text-generation tasks from the \bb benchmark \citep{kim2025biggenbenchprincipledbenchmark}, which includes 696 task instances spanning 8 capability dimensions, such as reasoning, safety, and tool usage, evaluated across 103 language models. We split the data into 211 training and 485 testing examples for our experiments.
Each task $q$ consists of a natural language instruction composed of a system prompt and user input. The solution $s_t$ is the textual response generated by a specific language model (tool) $t$. The quality feedback $r_t = R(q, s_t)$ includes a numerical score (on a 5-point Likert scale), a detailed score rubric, and NL feedback by GPT-4. We use feedback generated by GPT-4 to construct \tm for each tool, which are then used to improve downstream score prediction accuracy for new prompts.
We focus on six language model tools with varying capabilities and accessibility levels:
(1) Proprietary models: \textit{Claude-3-Sonnet}, \textit{GPT-3.5-Turbo};\footnote{Specific checkpoints are: {\tt claude-3-sonnet-20240229}, {\tt gpt-3.5-turbo-1106}.}
(2) Top-tier open-source models: \textit{Meta-Llama-3-70B-Instruct}, \textit{Qwen-110B-Chat};
(3) Smaller models: the strongest sub-7B model, \textit{Gemma-1.1-2B-It}, and the smallest chat-capable model, \textit{Qwen1.5-0.5B-Chat}.
% \noindent \textbf{Evaluation: Score Prediction} \quad
\paragraph{Evaluation: Score Prediction}
We evaluate \tm-augmented LMs under a score prediction evaluation setup on how accurately they measure the tool generation quality.
In our vanilla evaluation setup, each method is tasked with predicting a quality score for the generated image or text given a task prompt.
The goal is to predict the quality score of a prompt-response pair $q$, using the task-specific rubrics provided in the dataset, also rated on a 1--5 Likert scale. We take the \texttt{gpt4-04-turbo-score} as ground-truth, as no human scores are provided.
We measure the consistency of predicted scores $s_{\it pred}$ to the ground-truth scores $s$ by
(i) the average mean absolute error (MAE) across all $N$ test examples ${\it MAE} = \frac{1}{N} \sum_{i=1}^{N} |s^{(i)}_{\it pred} - s^{(i)}|$,
(ii) the root mean squared error by ${\it RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (s^{(i)}_{\it pred} - s^{(i)})^2}$,
(iii) the Pearson correlation coefficient between predicted and true scores:
${\it Pearson} = \frac{\sum_{i=1}^{N} (s^{(i)}_{\it pred} - \bar{s}_{\it pred})(s^{(i)} - \bar{s})}{\sqrt{\sum_{i=1}^{N} (s^{(i)}_{\it pred} - \bar{s}_{\it pred})^2} \sqrt{\sum_{i=1}^{N} (s^{(i)} - \bar{s})^2}}$, where $\bar{s}_{\it pred}$ and $\bar{s}$ are the means of the predicted and ground-truth scores, respectively.
% %%%%%%%%%%%%%%%%
% %%%%%%%%%%%%%%%%
% \noindent \textbf{Results and Analysis} \quad
\paragraph{Results and Analysis}
As shown in \autoref{table:qa_score_full_comparison}, \tm reduces MAE by 14.8\% and RMSE by 14.5\% on average than \gr baseline;
and improves ranking fidelity—Pearson correlation increases by 76.7\%.
% Without Spearman's version
\noindent \textit{Tools with varied sizes} \quad
For the two smallest and least capable tools (\textit{Qwen1.5-0.5B-Chat} and \textit{gemma-1.1-2B-it}), \tm boosts Pearson correlation significantly, lifting it from near-random (${-}\!0.007$ and $0.120$) to $0.405$ and $0.324$, highlighting the valuable, tool-specific priors that are initially lack for these lightweight models and are augmented through \tm.
Even for the strongest open-weight tool (\textit{Meta-Llama-3-70B-Instruct}), \tm achieves a notable 38.9\% increase in Pearson correlation, from $0.175$ to $0.243$, while slightly reducing RMSE.
\begin{wraptable}[24]{r}{0.68\textwidth}
\vspace{-6mm}
\centering
\small
\caption{
Performance prediction of text-generation tools.
}
\vspace{2mm}
\label{table:qa_score_full_comparison}
% \renewcommand{\arraystretch}{1.3}
\resizebox{0.68\textwidth}{!}{%
\begin{tabular}{ll|ccc}
\toprule
\multicolumn{1}{c}{\bf Tool} & \multicolumn{1}{c|}{\bf Method} & \bf MAE $\downarrow$ & \bf RMSE $\downarrow$ & \bf Pearson $\uparrow$ \\
\midrule
\multirow{3}{*}{Llama3 70B}
& \gr & \textbf{0.571} & 1.036 & 0.175 \\
& \fs & 1.115 & 1.566 & 0.163 \\
& \tm & 0.610 & \textbf{1.025} & \textbf{0.243} \\
\midrule
\multirow{3}{*}{Qwen 110B}
& \gr & 0.602 & 0.991 & 0.228 \\
& \fs & 1.155 & 1.605 & 0.146 \\
& \tm & \textbf{0.600} & \textbf{0.988} & \textbf{0.246} \\
\midrule
\multirow{3}{*}{Gemma 2B}
& \gr & 1.004 & 1.528 & 0.120 \\
& \fs & 1.008 & 1.429 & 0.271 \\
& \tm & \textbf{0.932} & \textbf{1.252} & \textbf{0.324} \\
\midrule
\multirow{3}{*}{Qwen1.5 0.5B}
& \gr & 1.769 & 2.136 & -0.007 \\
& \fs & 1.381 & 1.762 & 0.102 \\
& \tm & \textbf{1.080} & \textbf{1.400} & \textbf{0.405} \\
\midrule
\multirow{3}{*}{Claude3}
& \gr & \textbf{0.546} & 0.983 & \textbf{0.313} \\
& \fs & 1.085 & 1.555 & 0.140 \\
& \tm & 0.551 & \textbf{0.979} & 0.310 \\
\midrule
\multirow{3}{*}{GPT-3.5}
& \gr & 0.625 & 1.095 & 0.224 \\
& \fs & 0.924 & 1.392 & 0.204 \\
& \tm & \textbf{0.588} & \textbf{0.998} & \textbf{0.332} \\
\bottomrule
\end{tabular}
}
\vspace{0mm}
\end{wraptable}
\noindent \textit{Top closed tools have diminishing returns} \quad
For \textit{gpt-3.5-turbo-1106} and \textit{claude-3-sonnet}, baseline predictions already achieve relatively strong Pearson correlations ($\geq 0.224$), while \tm still manages to enhance correlation for GPT by 48.2\%.
% \noindent \textbf{Tools with varied sizes} \quad
% For the two smallest and least capable tools (\textit{Qwen1.5-0.5B-Chat} and \textit{gemma-1.1-2B-it}), \tm slashes RMSE by 18--34\% and lifts Spearman from near-random (${-}\!0.04$ and $0.08$) to $0.42$ and $0.33$, showcasing the crucial, tool-specific priors that these lightweight models lack and are augmented via \tm.
% Even for the strongest open-weight tool (\textit{Meta-Llama-3-70B-Instruct}), \tm
% increases Pearson by 39\% and slightly reduces RMSE.
% \noindent \textbf{Top closed tools have diminishing returns} \quad
% On \textit{gpt-3.5-turbo-1106} and \textit{claude-3-sonnet}, baselines already predict scores fairly well (MAE $\leq 0.63$, Spearman $\geq 0.23$), while \tm still cuts RMSE by up to 9\% and raises correlation on GPT (+48 \% Pearson), it offers marginal changes for Claude, indicating that capacity ceilings limit memory-based gains.
\noindent \textit{Instability of Raw Experiences} \quad
Across all tools, the \fs strategy underperforms \gr across metrics and often doubles the error for the two top models (e.g., MAE $+95\%$ on Meta-Llama-3-70B).
In contrast, \tm delivers robust improvements without such regressions, underscoring the value of a compact, induced capability memory over raw example reference.
\subsection{Text-to-Image Evaluation}
\label{sec:3.3:image-eval}
%%%%%%%%%%%%%%
% \noindent \textbf{Dataset: \gb} \quad
\paragraph{Dataset: \gb}
We evaluate text-to-image generation tools using the \gb benchmark \citep{li2024genaibenchevaluatingimprovingcompositional}, which contains 1,600 diverse natural language instructions spanning compositional scenarios such as object counting, spatial relations, and negation.
We randomly sample 200 training and 800 testing examples for experiments.
Each instruction is paired with six images generated by different tools, including \textit{MidJourney}, \textit{DALL{\small \textperiodcentered}E~3}, \textit{SDXL Turbo}, \textit{DeepFloyd I-XL v1}, \textit{SDXL 2.1}, and \textit{SDXL Base}.
Each text-image pair is annotated by three human raters on a 1–5 Likert scale, reflecting how well the image aligns with the prompt. As multiple human annotators can introduce variance in prediction scores, we take the first annotator's rating as the ground-truth score to ensure label consistency.
Our goal is to predict the quality score $r_t$ based on the task input $q$ (image generation prompt) and the solution $s_t$ (the generated image using tool $t$) with agents having no memory, few-shot examples, or \tm.
%%%%%%%%%%%%%%%%%
% \noindent {\bf Evaluation: Score \& Description Prediction} \quad
\paragraph{Evaluation: Score \& Description Prediction}
We similarly evaluate \tm-augmented LM to predict the alignment score of a prompt-image pair $q$, rated on a 5-point Likert scale. %, consistent with human annotations.
% The ground-truth score $s$ is defined as the first human rating.
While score prediction may be subject to individual human judgment and less robust, we alternatively evaluate image quality through text-image alignment: given a text prompt $q$ (e.g. ``Three birds flying in the sky''), we ask agent to predict a text description $d_{\it pred}$ for the image $s^{\it image}$ generated by tool $t$. We then evaluate the alignment between the image and the predicted description using the VQA score \citep{li2024genaibenchevaluatingimprovingcompositional}, computed as $s_{\it vqa} = {\it VQA}(s^{\it image}, d_{\it pred})$. A higher VQA score indicates stronger alignment between the image and the textual description. We report average VQA Score in \autoref{table:score-pred-basic}.
%%%%%%%%%%%%%%%%%
% \noindent \textbf{Results and Analysis} \quad
\paragraph{Results and Analysis}
\autoref{table:score-pred-basic} lists the six image–generation tools in descending order of capability.
Overall, \tm consistently improves over the \gr baseline by decreasing MAE by 28.7\% and RMSE by 26.6\% across tools.
The gains, however, are not uniform:
\noindent \textit{Top-tier closed models (DALLE 3, Midjourney 6).} \quad
\tm beats the \gr agent (–7.4\% and –12.2\% MAE, respectively) but trails \fs by 3.7\% to 6.7\%. This suggests that when the underlying generator is highly capable, large in-context exemplars are sufficient and an external capability memory brings limited extra benefit.
However, \tm delivers consistent gains in VQA scores, with improvements of +2.1\% for DALLE 3 and +2.0\% for Midjourney 6. These steady gains highlight \tm's ability to fine-tune descriptions even for top-performing models, effectively capturing subtle yet crucial details that baseline methods might overlook.
\noindent \textit{Mid/low-tier open models (DeepFloyd\_I\_XL\_v1, SDXL-Base, SDXL-Turbo, SD-2-1).} \quad
Here \tm clearly shows its strength: it achieves the lowest MAE and RMSE on all four models, reducing MAE by 26.1–42.6\% and RMSE by 25.5–36.6\% relative to \gr, and outperforming \fs by 6.7–18.0\% MAE.
%%%%%%%%%%%%%%%%%
\begin{wraptable}[24]{r}{0.7\textwidth}
\vspace{-7mm}
\centering
\small
\caption{Predicting text-to-image tool performance on \gb.}
\label{table:score-pred-basic}
\vspace{1mm}
\begin{tabular}{ll|ccc}
\toprule
\multicolumn{1}{c}{\multirow{2}{*}{\bf Tool}} & \multicolumn{1}{c|}{\multirow{2}{*}{\bf Method}} & \multicolumn{2}{c}{\bf Score Prediction} & \multicolumn{1}{c}{\bf Desc. Pred.} \\
\cmidrule(lr){3-4} \cmidrule(lr){5-5}
& & \bf MAE $\downarrow$ & \bf RMSE $\downarrow$ & \bf VQA Score (\%) $\uparrow$ \\
\midrule
\multirow{3}{*}{DALLE 3}
& \gr & 1.009 & 1.405 & 81.30 \\
& \fs & \textbf{0.875} & \textbf{1.180} & 81.64 \\
& \tm & 0.934 & 1.204 & \textbf{82.98} \\
\midrule
\multirow{3}{*}{Midjourney 6}
& \gr & 1.085 & 1.434 & 81.19 \\
& \fs & \textbf{0.919} & 1.227 & 82.21 \\
& \tm & 0.953 & \textbf{1.212} & \textbf{82.84} \\
\midrule
\multirow{3}{*}{DeepFloyd}
& \gr & 1.334 & 1.703 & 79.17 \\
& \fs & 1.058 & 1.393 & 79.74 \\
& \tm & \textbf{0.986} & \textbf{1.268} & \textbf{81.30} \\
\midrule
\multirow{3}{*}{SDXL-Base}
& \gr & 1.481 & 1.841 & 80.47 \\
& \fs & 1.111 & 1.422 & 81.37 \\
& \tm & \textbf{0.978} & \textbf{1.285} & \textbf{82.03} \\
\midrule
\multirow{3}{*}{SDXL-Turbo}
& \gr & 1.566 & 1.932 & 79.22 \\
& \fs & 1.101 & 1.400 & 80.01 \\
& \tm & \textbf{0.996} & \textbf{1.331} & \textbf{81.73} \\
\midrule
\multirow{3}{*}{SDXL-2-1}
& \gr & 1.645 & 1.981 & 75.70 \\
& \fs & 1.151 & 1.477 & 75.91 \\
& \tm & \textbf{0.944} & \textbf{1.255} & \textbf{78.90} \\
\bottomrule
\end{tabular}
\vspace{-2mm}
\end{wraptable}
%%%%%%%%%%%%%%%%%
\tm also exhibits significant improvements for VQA scores, particularly in mid- and low-tier models where gains reach +3.17\% for SDXL-Turbo and +4.23\% for SDXL-2-1. This pattern reflects \tm's ability to adapt to each model's specific strengths and weaknesses, resulting in more accurate and context-aware descriptions. The learnable memory is especially effective for weaker tools that lack robust prior knowledge, helping narrow the gap between less capable generators and their stronger counterparts.
In summary, \tm offers the greatest benefit where it is most needed—narrowing the gap between weaker generators and their stronger peers—while remaining competitive on state-of-the-art closed models. Its dual effectiveness in both score prediction and precise description prediction highlights its capacity to enhance model evaluation by leveraging learned memory, ultimately providing higher VQA scores and more precise model-specific descriptions.
\section{Experiments: Performant Tool Selection}
\label{sec:4:advanced-scenarios}
% In this section, we further explore a more advanced \tm usage for selecting between tools under performance optimization goals. % (\S\ref{sec:4.1:model-selection}) and with joint efficiency considerations (\S\ref{sec:4.2:cost-efficiency}).
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \subsection{Selecting Performant Tools}
% \label{sec:4.1:model-selection}
In addition to predicting a single tool's performance, we can select the better-performing tool among multiple choices based on the \tm learned for these tools.
%
We construct \tm and evaluate on the same datasets on {\sc GenAI-Bench} and {\sc BigGen-Bench}, and compare the same three methods (\gr, \fs, \tm) as in \S\ref{sec:3:perf-predict}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \noindent \textbf{Evaluation Metrics} \quad
\paragraph{Evaluation Metrics}
Let $s_A$ and $s_B$ denote the ground-truth scores of tools A and B, and $s_{\it pred,A}$ and $s_{\it pred,B}$ be their predicted scores. Since for equal pairs selecting either tool is equally acceptable, we evaluate tool selection performance over the subset of \emph{unequal} pairs $D = \{i: s_A^i \ne s_B^i\}$. For each $i \in D$, we define $TP_< = |\{i \in D: s_A^i < s_B^i \wedge s_{\it pred,A}^i < s_{\it pred,B}^i\}|$, $P_< = |\{i \in D: s_{\it pred,A}^i < s_{\it pred,B}^i\}|$, and $R_< = |\{i \in D: s_A^i < s_B^i\}|$. We define $TP_>$, $P_>$, and $R_>$ analogously for the “$>$” case. Then we compute $F_{1<} = \frac{2 TP_<}{P_< + R_<}$, $F_{1>} = \frac{2 TP_>}{P_> + R_>}$, and $\mathrm{Acc} = \frac{TP_< + TP_>}{|D|}$, where $F_{1<}$ (resp.\ $F_{1>}$) is the F$_1$ score for predicting $s_A < s_B$ (resp.\ $s_A > s_B$) over the non-equal set $D$, and Acc is the proportion of cases where the predicted and ground-truth preference directions match.
% \paragraph{Evaluation Metrics}
% We evaluate tool selection performance by the alignment between ground-truth and predicted choices. Specifically, let $s_A$ and $s_B$ denote the ground-truth scores of tools A and B, and $s_{\it pred,A}$ and $s_{\it pred,B}$ be their predicted scores.
% % We assign a score of 1, 0, -1 to each ground-truth/predicted pair if $s_A > s_B$, $s_A = s_B$, and $s_A < s_B$, i.e., ${\it score}(s_A, s_B) = \mathbbm{1}(s_A > s_B) - \mathbbm{1}(s_A < s_B)$.
% On the subset We then compute the F$_1$ score for the cases where $s_A < s_B$ is the positive label (denoted as F$_{1<}$) and where $s_A > s_B$ is positive label (F$_{1>}$).
% Additionally, we report the accuracy of mapped scores ${\it score}(s_{\it pred,A}, s_{\it pred,B})$ with respect to the ground-truth scores ${\it score}(s_A, s_B)$.
% Note that we only measure over the subset of \textit{unequal pairs}, i.e., the ground-truth scores of two models are not equal ($s_A \ne s_B$), as otherwise selecting either tool is equally acceptable.
% See \S\ref{app:model-selection} for more details and evaluation examples.
%%%%%%%%%%
\paragraph{Text Generation Evaluation}
On \bb, we evaluate six tool pairs as listed in \autoref{table:model-selection-textgen}.
The selected pairs include comparisons between top open-source and closed-source models (e.g., Meta-Llama-3-70B vs.\ Claude-3-Sonnet), similarly strong closed models (Claude-3-Sonnet vs.\ GPT-3.5-Turbo), and strong open models (Meta-Llama-3-70B vs.\ Qwen110B), models from the same family with large capacity gaps (Qwen110B vs.\ Qwen0.5B), and small-scale model comparisons (e.g., Gemma-2B vs.\ Qwen0.5B).
These combinations cover diverse settings across model families, sizes, and access types.
% analysis
As shown in \autoref{table:model-selection-textgen}, we find that \tm offers clear and consistent accuracy advantages over \gr and \fs settings by 21\% and 18\% on average.
%
\tm increases the F$_{1<}$ score, which measures the ability to identify the cases where weaker tool excels, increases from 0.00 to 0.28 from \gr in the Meta-Llama-3-70B vs.\ Claude-3-Sonnet comparison, on average it improves from \gr's 0.03 to 0.09.
%
F$_{1>}$ improves more significantly from 0.12 to 0.39 on average, with \tm achieving the highest score on five of the six tool pairs.
%
\tm boosts performance most substantially when tool pairs exhibit large capability gaps, e.g., \tm improves accuracy by +0.55 for Qwen110B vs.\ Qwen0.5B and +0.28 for GPT-3.5-Turbo vs.\ Gemma-2B.
Even for tool pairs with smaller capability gaps (e.g., Meta-Llama-3-70B vs.\ Claude-3-Sonnet), \tm maintains an edge over both baselines.
\input{tables/tool-selection/text-gen}
%%%%%%%%%%
% \noindent \textbf{Text-to-Image Generation Evaluation} \quad
\paragraph{Text-to-Image Generation Evaluation}
On \gb, we examine six representative tool pairs (in \autoref{table:model-selection-image-eval}) to test whether \tm-augmented LM can accurately identify the superior tool for individual tasks. More capable model is on the left side of each pair.
To showcase \tm utility in varied scenarios, we choose varied types of tool pairs. Specifically:
(i) distinguishing comparably top-performing proprietary tools (DALL$\cdot$E 3, Midjourney) and against open-weight tools (DALL$\cdot$E 3, DeepFloyd), (Midjourney, DeepFloyd);
(ii) comparing tools with huge gaps from the same (SDXL-Base, SDXL-Turbo) and different model families (DALL$\cdot$E 3, SDXL-2.1).
% analysis
\autoref{table:model-selection-image-eval} shows the consistent edge of \tm across varied tool pairs. Averaged over five tool pairs, accuracy on unequal pairs climbs from 0.09 (\gr) and 0.27 (\fs) to 0.33 with \tm, an absolute gain of +0.24 over \gr (+266\%) and +0.06 over \fs (+22\%).
% between two groups
The hardest class F$_{1<}$, which measures the situation to find worse cases for a more capable model, improves most dramatically---rising from 0.09 to 0.32 compared to \gr and \fs approaches.
Meanwhile, F$_{1>}$ also edges up from 0.15 to 0.46 when augmented with \tm.
% between tool pairs
The benefit brought by \tm shows more prominently as the capability gap of the tool pair grows. For instance, between comparable DALL$\cdot$E 3 and Midjourney tools, the gain of Acc. for \tm compare to \gr is relatively small (+0.12), yet comparing DALL$\cdot$E 3 to the weaker SDXL-2.1, \tm boosts accuracy from 0.08 to 0.53, achieves much larger gains.
% within same model family
Even within the same model family, \tm can effectively tell tools' capability differences, showcased by the +0.18 gain in the SDXL-Base vs.\ SDXL-Turbo comparison.
%
These results confirm that \tm consistently chooses better tools when the tool choices exhibit different properties.
\input{tables/tool-selection/image-gen}
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \subsection{Performance-Efficiency Joint Consideration} \sxcomment{Consider moving this experiment to appendix to save some page?}
% \label{sec:4.2:cost-efficiency}
% In addition to selecting superior tools based on their task performance, in practical usage, it is often crucial to jointly consider their efficiency.
% Thus, we showcase how \tm could take tool efficiency into joint consideration, by comparing multiple tools that achieve comparable task performance yet require varied computational costs.
% We approximate the cost of a tool by the parameter size of its underlying neural model. For example, SDXL-Base has 2.6B parameters and \zwcomment{Llama 3} has \zwcomment{x.x}B parameters. For Diffusion Models we estimate the cost with UNet size, SDXL-Base has 2.6B and SD2-1 has a 0.865B U-Net, and they share same diffusion steps
% \sxcomment{We should elaborate more on this part, but diffusion model cost estimate really depends on model architecture (text encoder, UNet, VAE decoder...) and diffusion steps. For now we chose those two with similar architecture and diffusion steps so we just estimate with their UNet sizes, which is the main compute part}
% %\zwcomment{what if the model is closed-source, e.g., gpt?}
% At tool selection time, if our \tm-augmented LM predicts one tool to achieve a higher performance score, it selects that tool. If two tools are judged as equally performant, we choose the cheaper tool (i.e., the underlying model has fewer parameters).
% \paragraph{Text-to-Image Generation}
% On text-to-image evaluation tasks, we examine the (SDXL-Base, SD-2-1) tool pair as they are from the same model family with similar architecture so their relative cost can be estimated with their U-Net sizes,
% %achieve the closest performance scores.
% % \begin{wrapfigure}[13]{r}{0.5\textwidth}
% % \centering
% % \includegraphics[width=0.5\linewidth]{images/cost_score_optimization.png}
% % \caption{Performance-efficiency joint optimization using different methods.\zwcomment{use pdf format instead of png; crop out the white spaces using the mac preview software.}}
% % \label{fig:cost_score_optimization}
% % \end{wrapfigure}
% \begin{figure}
% \centering
% \includegraphics[width=\linewidth]{images/task3.png}
% \caption{Qwen1.5-0.5B-Chat vs gemma-1.1-2b-it. Shawn: I just put those two pics (fig 4 and fig 5) here to showcase that our efficiency usually is not the best, but our avg score is higher than fewshot and generic baseline. And we are below the line between selecting all A and selecting all B. What's the best way to report this?}
% \label{fig:enter-label}
% \end{figure}
% \begin{figure}
% \centering
% \includegraphics[width=\linewidth]{images/task3_mixtral.png}
% \caption{Mixtral-8x22B vs 8x7B}
% \label{fig:enter-label}
% \end{figure}
% \autoref{fig:cost_score_optimization} plots total score against total cost for various approaches. A single-tool strategy using only DeepFloyd yields a higher overall score (2692) \zwcomment{is this the total score across all examples? we should report average score.} but at a higher cost (3440.0), whereas relying solely on SDXL-Base is cheaper (2800.0) but scores slightly lower (2583). The \tm approach achieves 2620 points at a moderate cost of 2925.6, placing it near the Pareto frontier—improving upon the baseline (2585 points at 2808.8 cost) while keeping costs substantially below DeepFloyd-only usage. Hence, \tm demonstrates its capacity to adaptively trade off performance and cost.
% \paragraph{Text Generation}
% \zwcomment{specify {\bf single} pair selection here and explain the reason.}
% \zwcomment{it is weird that for text-to-image we only select one pair, but for text generation we have so many pairs. We could simply just select one pair (with the closest performance) and similarly plot the figure, instead of having the big table now.}
% \sxcomment{ToolMem is not the most cost efficient method, since select which model really impact total effficency. a 70B model that scores 5 gets 5/70=0.071 and a 14B model that scores 3 get 3/14=0.214 efficiency.}
% \sxcomment{And if we follow task 2, which experiment on non-equal pairs, the results looks better, we are able to achieve even better avg score improvement. @hewei will add 3 pic from all pairs and 3 from non equal pairs, for 3 model pairs for comparison}
% To access ToolMem’s ability to balance quality and compute, we selected six pairs that together span extreme and incremental capacity gaps, different model families, and open vs. closed‑source access. We include one extreme scale contrast (Meta‑Llama‑3‑70B vs.\ Phi‑3‑mini‑4k), a heavyweight vs.\ ultra‑lightweight cross‑family match (Meta‑Llama‑3‑70B vs.\ Qwen1.5‑0.5B), two mid‑range vs.\ small closed‑source comparisons (Meta‑Llama‑3‑70B vs.\ GPT‑3.5‑Turbo and vs.\ Gemma‑1.1‑2B), another extreme from tiny to very large (Phi‑3‑mini‑4k vs.\ Qwen1.5‑72B), and a dramatic intra‑family jump (Qwen1.5‑0.5B vs.\ Qwen‑110B). This mix ensures we cover both subtle and stark trade‑offs across diverse model lines.
% Table~\ref{tab:cost-efficiency-extended} reveals that ToolMem yields the biggest efficiency wins when slight performance gains align with cost reductions—for example, cutting average cost by 27 percent (11.44→8.31) and boosting efficiency by 38 percent (0.345→0.477) on the Meta‑Llama‑3 vs.\ Phi‑3 pair. In contrast, in pairs with extreme capacity mismatches or minimal score gaps—such as Meta‑Llama‑3 vs.\ Qwen1.5‑0.5B or vs.\ Qwen‑110B—ToolMem’s occasional over‑investment in the larger model drives cost up by several hundred percent for under 1 point of score improvement, resulting in efficiency below 0.10. These findings show that while ToolMem excels when cost and quality scale together, it can over‑allocate compute when performance differences are too small to justify heavier models.
% \zwcomment{analyze results here}
% %For text-generation tools, we found that \tm achieves much better
% \zwcomment{``much better'' is very vague, support by concrete numbers, e.g., xx.x\% improvement.} %cost efficiency when two tool options's capability varies more. For example Qwen1.5-0.5B-Chat vs Qwen1.5-72B-Chat,
% \begin{table*}[htbp]
% \centering
% \renewcommand{\arraystretch}{1.2}
% \caption{Cost/efficiency comparison of selected model pairs under Baseline, Few‑Shot, and ToolMem.}
% \label{tab:cost-efficiency-extended}
% \resizebox{\textwidth}{!}{%
% \begin{tabular}{l ccc ccc ccc}
% \toprule
% \multirow{2}{*}{\textbf{Model Pair}}
% & \multicolumn{3}{c}{\textbf{Baseline}}
% & \multicolumn{3}{c}{\textbf{Few‑Shot}}
% & \multicolumn{3}{c}{\textbf{ToolMem}} \\
% \cmidrule(lr){2-4} \cmidrule(lr){5-7} \cmidrule(lr){8-10}
% & \textbf{AvgCost} & \textbf{AvgScore} & \textbf{Efficiency}
% & \textbf{AvgCost} & \textbf{AvgScore} & \textbf{Efficiency}
% & \textbf{AvgCost} & \textbf{AvgScore} & \textbf{Efficiency} \\
% \midrule
% Meta-Llama-3-70B-Instruct vs Phi-3-mini-4k-instruct
% & 11.444 & 3.949 & 0.3449
% & 9.396 & 3.956 & 0.4209
% & 8.308 & 3.962 & 0.4770 \\
% \midrule
% Meta‑Llama‑3‑70B‑Instruct vs Qwen1.5‑0.5B‑Chat
% & 9.667 & 2.542 & 0.2629
% & 4.941 & 2.410 & 0.4877
% & 42.498 & 3.500 & 0.0824 \\
% \midrule
% Meta‑Llama‑3‑70B‑Instruct vs gpt‑3.5‑turbo‑1106
% & 13.341 & 3.729 & 0.2794
% & 14.835 & 3.752 & 0.2530
% & 16.804 & 3.758 & 0.2234 \\
% \midrule
% Meta‑Llama‑3‑70B‑Instruct vs gemma‑1.1‑2b‑it
% & 12.382 & 3.402 & 0.2748
% & 7.234 & 3.310 & 0.4576
% & 37.586 & 3.788 & 0.1008 \\
% \midrule
% Phi-3-mini-4k-instruct vs Qwen1.5-72B-Chat
% & 8.158 & 3.926 & 0.4809
% & 8.724 & 3.942 & 0.4518
% & 7.312 & 3.956 & 0.5409 \\
% \midrule
% Qwen1.5‑0.5B‑Chat vs qwen‑110b‑chat
% & 6.101 & 2.431 & 0.3984
% & 4.481 & 2.395 & 0.5343
% & 41.184 & 3.358 & 0.0815 \\
% \bottomrule
% \end{tabular}%
% }
% \end{table*}
\begin{table}[t!]
\centering
\small
\caption{Selecting better text generation tools. Best results for each tool pair are bold.}
\label{table:model-selection-textgen}
% \resizebox{\textwidth}{!}{
\begin{tabular}{ll|ccc|ccc|ccc}
\toprule
\multicolumn{2}{c|}{\multirow{2}{*}{\bf Tool Pair}} & \multicolumn{3}{c}{\bf \gr} & \multicolumn{3}{c}{\bf \fs} & \multicolumn{3}{c}{\bf \tm} \\
{} & {} & F$_{1<}$ & F$_{1>}$ & Acc. & F$_{1<}$ & F$_{1>}$ & Acc. & F$_{1<}$ & F$_{1>}$ & Acc. \\
\midrule
Llama3 & Claude3 & 0.00 & 0.09 & 0.03 & 0.20 & 0.06 & 0.07 & \textbf{0.28} & \textbf{0.10} & \textbf{0.11} \\
Claude3 & GPT-3.5-Turbo & 0.03 & 0.06 & 0.03 & \textbf{0.21} & 0.22 & 0.13 & 0.06 & \textbf{0.30} & \textbf{0.15} \\
Qwen 110B & Qwen 0.5B & 0.00 & 0.13 & 0.07 & 0.00 & 0.09 & 0.05 & 0.00 & \textbf{0.77} & \textbf{0.62} \\
Llama3 & Qwen 110B & 0.04 & 0.19 & 0.07 & 0.12 & \textbf{0.17} & 0.08 & \textbf{0.16} & 0.14 & \textbf{0.09} \\
Gemma 2B & Qwen 0.5B & 0.10 & 0.04 & 0.03 & 0.05 & 0.12 & 0.06 & \textbf{0.06} & \textbf{0.39} & \textbf{0.23} \\
GPT-3.5-Turbo & Gemma 2B & 0.00 & 0.24 & 0.11 & \textbf{0.08} & 0.27 & 0.14 & 0.00 & \textbf{0.62} & \textbf{0.39} \\
\midrule
\multicolumn{2}{c|}{\bf Average} & 0.03 & 0.12 & 0.06 & \textbf{0.11} & 0.16 & 0.09 & 0.09 & \textbf{0.39} & \textbf{0.27} \\
\bottomrule
\end{tabular}
% }
\end{table}
\begin{table}[ht]
\small
\centering
\caption{Selecting better text-to-image generation tools. Best results for each tool pair are bold.}
\label{table:model-selection-image-eval}
% \vspace{1mm}
% \resizebox{\textwidth}{!}{
\begin{tabular}{ll|ccc|ccc|ccc}
\toprule
\multicolumn{2}{c|}{\multirow{2}{*}{\bf Tool Pair}} & \multicolumn{3}{c|}{\bf \gr} & \multicolumn{3}{c|}{\bf \fs} & \multicolumn{3}{c}{\bf \tm} \\
{} & {} & F$_{1<}$ & F$_{1>}$ & Acc. & F$_{1<}$ & F$_{1>}$ & Acc. & F$_{1<}$ & F$_{1>}$ & Acc. \\
\midrule
DALLE 3 & DeepFloyd & 0.07 & 0.19 & 0.10 & 0.19 & \textbf{0.52} & 0.31 & \textbf{0.31} & \textbf{0.52} & \textbf{0.35} \\
DALLE 3 & Midjourney & 0.00 & 0.31 & 0.14 & 0.17 & 0.42 & 0.23 & \textbf{0.26} & \textbf{0.43} & \textbf{0.26} \\
DALLE 3 & SD-2-1 & 0.00 & 0.16 & 0.08 & 0.20 & 0.66 & 0.47 & \textbf{0.32} & \textbf{0.72} & \textbf{0.53} \\
SDXL-Base & SDXL-Turbo & 0.14 & 0.03 & 0.05 & 0.24 & \textbf{0.29} & 0.18 & \textbf{0.36} & 0.26 & \textbf{0.23} \\
Midjourney & DeepFloyd & 0.23 & 0.07 & 0.08 & 0.26 & 0.28 & 0.18 & \textbf{0.35} & \textbf{0.37} & \textbf{0.26} \\
\midrule
\multicolumn{2}{c|}{\bf Average} & 0.09 & 0.15 & 0.09 & 0.21 & 0.43 & 0.27 & \textbf{0.32} & \textbf{0.46} & \textbf{0.33} \\
\bottomrule
\end{tabular}
% }
\vspace{-3mm}
\end{table}
\section{Related Work}
% \noindent \textbf{Tool‑Using Multimodal Agents} \quad
\paragraph{Tool‑Using Multimodal Agents}
Multimodal agents combine LLMs and LVLMs to solve complex tasks across modalities \citep{xie2024largemultimodalagentssurvey}. Early systems like Visual Programming \citep{gupta2022visualprogrammingcompositionalvisual} and ViperGPT \citep{surís2023vipergptvisualinferencepython} use LLMs to compose expert modules for visual reasoning. Visual ChatGPT \citep{wu2023visualchatgpttalkingdrawing} and HuggingGPT \citep{shen2023hugginggptsolvingaitasks} improve agent flexibility via prompt routing and tool selection from a VFM pool. Later methods such as Toolformer \citep{schick2023toolformerlanguagemodelsteach}, AssistGPT \citep{gao2023assistgptgeneralmultimodalassistant}, and ToolLLM \citep{qin2023toolllm} learn to invoke tools through self-supervision or in-context examples. While effective, these approaches assume fixed tool behaviors or rely on limited, static memory.
In contrast, \tm enables adaptive, context-aware tool selection among choices, by building capability memories that capture tools' empirical strengths and weaknesses through experiences.
% \noindent \textbf{Generative AI Tools} \quad
\paragraph{Generative AI Tools}
Unlike deterministic tools that produce predictable outputs, tools supported by generative neural models (e.g., \textit{DALLE-3}, \textit{GPT}) exhibit varied performances across task specifications \citep{tang2025generativeaicelanimationsurvey}. Some tools may excel at object counting, while others are better at spatial understanding \citep{sim2024evaluatinggenerationspatialrelations}. While prior work has relied on human expertise to navigate these nuances, current agent systems typically neglect such variance during tool selection \citep{wang2024genartistmultimodalllmagent}.
In contrast, \tm records empirical performance signals for each tool, enabling agents to make accuracy-aware decisions during task solving.
% \noindent \textbf{Memory-Augmented Agents} \quad
\paragraph{Memory-Augmented Agents}
Effective memory systems enable agents to operate over extended tasks. Foundational work like neural Turing machines \citep{graves2014neuralturingmachines} introduced external memory modules for querying past states. Recent systems such as Reflexion \citep{shinn2023reflexion} store in-session NL-formatted feedback as memory, while AWM \citep{wang2024agentworkflowmemory} captures reusable workflows across tasks. MemGPT \citep{packer2024memgptllmsoperatingsystems} and MemoryBank \citep{zhong2023memorybankenhancinglargelanguage} separate memory into short- and long-term slots with dynamic update strategies. While these methods enhance general reasoning and task procedures of agents, they do not model the behavior of external tools, especially those with functionalities unknown prior to testing. In contrast, \tm explicitly tracks tool-specific strengths and weaknesses, facilitating smarter tool selection for use.
\section{Conclusion and Future Work}
We introduced \tm, a closed-loop framework that equips multimodal agents with a learnable and evolving memory of tool capabilities. By integrating structured memory initialization based on a proficiency-level taxonomy, feedback-driven learning from LLM-generated critiques, and retrieval-augmented generation for memory refinement, \tm enables agents to continually improve their understanding of tool behaviors. Our experiments demonstrate that agents augmented with \tm achieve more accurate tool performance estimation and make better-informed tool choices, leading to improved outcomes in both text and image generation tasks.
This work opens up several promising directions for future exploration. One compelling avenue is extending ToolMem to support \textit{a wider array of generative and decision-making tools} across domains such as multimodal editing and code synthesis.
Another direction is developing \textit{more advanced automated feedback} mechanisms to support autonomous memory construction.
Additionally, analyzing the \textit{theoretical underpinnings of memory} maintenance can offer new insights into long-term memory scalability.
Finally, incorporating \textit{human-in-the-loop} and \textit{principled memory consolidation} strategies may enhance learning from sparse or noisy experiences.
We hope \tm can serve as a foundation for building adaptive agents that learn from prior interactions without repeated retraining, paving the way for more efficient, resilient, and generalizable AI systems.
% \section{Analysis: Number of Memory Entries to Retrieve}
% \label{app:topK}
% We further conduct an in-depth analysis on top-$k$ retrieval granularity, by retrieving varying numbers of memory entries from \tm.
% As shown in \autoref{fig:topK_trend},
% We also compare performance when using varying \textit{top-k} values (0--24)\footnote{When \textit{top-k} = 0, predictions are made using only the tool name and its overview, without retrieving any memory items.}.
% \autoref{fig:topK_trend} shows that larger \textit{top-k} values generally reduce MAE and RMSE but eventually plateau or degrade due to noise from excessive retrieval. Accuracy follows a similar trend, often peaking around \textit{top-k}=10--14. For example, \tm’s predictions for Midjourney\_6 improve from an MAE of 0.99 at \textit{top-k}=0 to 0.82 at \textit{top-k}=18. These results suggest that leveraging a moderate number of high-relevance memory items helps balance predictive accuracy and computational efficiency.
% \begin{figure}[ht]
% \centering
% \includegraphics[width=\linewidth]{images/mae_rmse_acc_vs_topk.pdf}
% \caption{\tm's score prediction accuracy on text-to-image generation across different top-$k$ memory retrieval sizes. Evaluation on 100 samples using additive memory updates (directly augment the existing memory repository by $\mathcal{M} \leftarrow \mathcal{M} \cup \{m\}$ without RAG). The dashed line at $k=12$ indicates most models' empirically optimal memory size.}
% \label{fig:topK_trend}
% \end{figure}
\section{Analysis: Number of Memory Entries to Retrieve}
\label{app:topK}
We further conduct an in-depth analysis on top-$k$ retrieval granularity, by retrieving varying numbers of memory entries from \tm.
As shown in \autoref{fig:topK_trend},
We also compare performance when using varying \textit{top-k} values (0--24)\footnote{When \textit{top-k} = 0, predictions are made using only the tool name and its overview, without retrieving any memory items.}.
\autoref{fig:topK_trend} shows that larger \textit{top-k} values generally reduce MAE and RMSE but eventually plateau or degrade due to noise from excessive retrieval. Accuracy follows a similar trend, often peaking around \textit{top-k} = 10--14. For example, \tm’s predictions for Midjourney\_6 improve from an MAE of 0.98 at \textit{top-k} = 0 to 0.91 at \textit{top-k} = 18; SDXL\_Turbo’s MAE falls from 1.24 to 1.06 over the same range. These results suggest that leveraging a moderate number of high-relevance memory items helps balance predictive accuracy and computational efficiency.
\begin{figure}[ht]
\centering
\includegraphics[width=\linewidth]{images/mae_rmse_acc_vs_topk.pdf}
\caption{\tm's score prediction accuracy on six text-to-image models across different top-$k$ memory retrieval sizes, evaluated on 100 samples. The dashed line at $k=12$ indicates most models' empirically optimal memory size.}
\label{fig:topK_trend}
\end{figure}
\section{Prompt Templates}
\label{app:prompts}
% \newcommand{\tm}{\textsc{ToolMem}\xspace}
% \newcommand{\gr}{\textsc{Generic}\xspace}
% \newcommand{\fs}{\textsc{Few-Shot}\xspace}
\subsection{Text to Image Tools}
% \paragraph{Feedback Generation}
% \label{feedback_prompt}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Prompt for Feedback Generation, fonttitle=\bfseries]
The text prompt used to generate the image is ``\texttt{\{prompt\}}''. The Scoring metric used here is a 5-point Likert scale:
\textit{1 (Does not match at all)}: The generated image or video completely fails to align with the text prompt. It either lacks the required elements or depicts a scenario that contradicts the prompt.
\textit{2 (Has significant discrepancies)}: There are major mismatches between the text prompt and the visual output. Key elements may be missing, or relationships and attributes are significantly misrepresented.
\textit{3 (Has several minor discrepancies)}: The output aligns with the prompt in a general sense but has noticeable errors or omissions in key details. While the main elements are present, finer aspects are not accurately rendered.
\textit{4 (Has a few minor discrepancies)}: The output is mostly aligned with the prompt, with only a few small and less critical details being inaccurate or missing. The visual matches the description well but not perfectly.
\textit{5 (Matches exactly)}: The generated image or video perfectly matches the text prompt. All elements, relationships, attributes, and details align seamlessly with the description, with no noticeable discrepancies.
This image scored \texttt{\{score\}} based on the human ratings. Now tell me why you think the image is or isn't following the text prompt.
\end{tcolorbox}
% \subsubsection{Memory Refinement}
% \label{memory_refinement_prompt}
\newpage
\newpage
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Prompt for Memory Refinement, fonttitle=\bfseries, breakable]
You are an assistant responsible for maintaining an incremental memory of the tool's capabilities, limitations, and use cases. \\
You will receive a current memory and new feedback. Your job is to update the memory by integrating the new feedback without losing previously stored information. \\
\textbf{Rules:} \\
1. Start with the current memory. \\
2. Consider the new prompt and feedback, but do not simply repeat the feedback, think deep and get general insights. \\
3. Incrementally refine and update the memory to reflect any new insights. \\
4. Do not remove previously known capabilities unless the new feedback explicitly contradicts them. \\
5. If no new insights are provided, leave the memory unchanged. \\
\textbf{Instructions:} \\
Follow this procedure strictly: \\
- Review the current memory for model \texttt{\{tool\_name\}}: \\
\quad ``\texttt{\{current\_memory\}}'' \\
- Input task prompt to the tool was: \\
\quad ``\texttt{\{task\_prompt\}}'' \\
- Tool's answer was: \\
\quad ``\texttt{\{response\}}'' \\
- Score rubric was: \\
\quad ``\texttt{\{score\_rubric\}}'' \\
- The score for tool's response was: \\
\quad ``\texttt{\{score\}}'' \\
- Feedback for this answer received: \\
\quad ``\texttt{\{feedback\}}'' \\
Integrate the key insights of the tool's limitations and capabilities from the feedback into the existing memory. Use complete sentences starting with phrases like \texttt{good at}, \texttt{bad at}, \texttt{proficient at}, or \texttt{poor at} to describe your findings. \\
- If the feedback suggests new limitations, capabilities, or clarifications, add them. \\
- If the feedback suggests that something previously stated is incorrect, correct it. \\
- If the feedback suggests the tool struggles with a certain aspect (like counting multiple objects), note it. \\
- If no contradiction or new insight is provided, keep the memory as is. \\
- Think deeply and determine what the feedback actually means for the model's capability, rather than staying on the surface of a specific example. \\
In your final answer, output \textbf{ONLY} the updated overall memory of the tool. Do not include extra explanations. \\
\end{tcolorbox}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Predict Image Description - \gr, fonttitle=\bfseries, breakable]
You are familiar with how the model \texttt{\{model\_name\}} generates images. \\
Your current memory of \texttt{\{model\_name\}} is: ``A text to image model'' \\
Based on this memory, predict how an image generated by \texttt{\{model\_name\}} for the following prompt would look: \\
Prompt: ``\texttt{\{prompt\}}'' \\
Describe the expected image in no more than 50 English words. Be concise and specific. \\
Only return the description text.
\end{tcolorbox}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Predict Image Description - \fs, fonttitle=\bfseries, breakable]
You are familiar with how the model \texttt{\{self.model\_name\}} generates images. \\
Your current memory of \texttt{\{self.model\_name\}} is: ``\texttt{\{current\_memory\}}'' \\
Here are some examples of the model's performance on specific prompts: \\
\texttt{\{few\_shot\_memory\}} \\
The scoring metric used here is a 5-point Likert scale: \\
\textit{1 (Does not match at all)}: The generated image or video completely fails to align with the text prompt. It either lacks the required elements or depicts a scenario that contradicts the prompt. \\
\textit{2 (Has significant discrepancies)}: There are major mismatches between the text prompt and the visual output. Key elements may be missing, or relationships and attributes are significantly misrepresented. \\
\textit{3 (Has several minor discrepancies)}: The output aligns with the prompt in a general sense but has noticeable errors or omissions in key details. While the main elements are present, finer aspects are not accurately rendered. \\
\textit{4 (Has a few minor discrepancies)}: The output is mostly aligned with the prompt, with only a few small and less critical details being inaccurate or missing. The visual matches the description well but not perfectly. \\
\textit{5 (Matches exactly)}: The generated image or video perfectly matches the text prompt. All elements, relationships, attributes, and details align seamlessly with the description, with no noticeable discrepancies. \\
Based on these examples and your memory of \texttt{\{self.model\_name\}}, predict how an image generated by \texttt{\{self.model\_name\}} for the following prompt would look: \\
Prompt: ``\texttt{\{prompt\}}'' \\
Describe the expected image in no more than 50 English words. Be concise and specific. \\
Only return the description text.
\end{tcolorbox}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Predict Image Description - \tm, fonttitle=\bfseries, breakable]
You are familiar with how the model \texttt{\{model\_name\}} generates images. \\
Your current memory of \texttt{\{model\_name\}} is: ``\texttt{\{current\_memory\}}'' \\
Based on this memory, predict how an image generated by \texttt{\{model\_name\}} for the following prompt would look: \\
Prompt: ``\texttt{\{prompt\}}'' \\
Describe the expected image in no more than 50 English words. Be concise and specific. \\
Only return the description text.
\end{tcolorbox}
% \subsubsection{Predict Image Description Baseline}
% \label{predict_image_description}
% \begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
% title=Prompt for Memory Refinement, fonttitle=\bfseries, breakable]
% You are familiar with how the model \texttt{\{self.model\_name\}} generates images.\\
% Your current memory of \texttt{\{self.model\_name\}} is: ``\texttt{\{current\_memory\}}''\\
% Here are some examples of the model's performance on specific prompts: \\
% \texttt{\{few\_shot\_memory\}}\\
% Based on these examples and your memory of \texttt{\{self.model\_name\}}, predict how an image generated by \texttt{\{self.model\_name\}} for the following prompt would look:
% Prompt: ``\texttt{\{prompt\}}''\\
% Provide a concise and specific description of the expected image in no more than 50 words.\\
% Return only the description text.
% \end{tcolorbox}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Predict Score - \gr, fonttitle=\bfseries, breakable]
Your current memory (knowledge) of the tool \texttt{\{model\_name\}} is: ``A text to image model'' \\
Based on your memory of model \texttt{\{model\_name\}}, predict the score for the following prompt: ``\texttt{\{prompt\}}''. \\
The scoring metric used here is a 5-point Likert scale: \\
\textit{1 (Does not match at all)}: The generated image or video completely fails to align with the text prompt. It either lacks the required elements or depicts a scenario that contradicts the prompt. \\
\textit{2 (Has significant discrepancies)}: There are major mismatches between the text prompt and the visual output. Key elements may be missing, or relationships and attributes are significantly misrepresented. \\
\textit{3 (Has several minor discrepancies)}: The output aligns with the prompt in a general sense but has noticeable errors or omissions in key details. While the main elements are present, finer aspects are not accurately rendered. \\
\textit{4 (Has a few minor discrepancies)}: The output is mostly aligned with the prompt, with only a few small and less critical details being inaccurate or missing. The visual matches the description well but not perfectly. \\
\textit{5 (Matches exactly)}: The generated image or video perfectly matches the text prompt. All elements, relationships, attributes, and details align seamlessly with the description, with no noticeable discrepancies. \\
Now return a score between 1 and 5 for prompt \texttt{\{prompt\}}, based on your memory of the tool \texttt{\{model\_name\}}. Return a single number only, nothing else. \\
\end{tcolorbox}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title= Predict Score - \fs, fonttitle=\bfseries]
Your current memory (knowledge) of the tool \texttt{\{self.model\_name\}} is: ``A text to image model''.
Here are some examples of the model's performance on specific prompts:
\texttt{\{samples\_prompt\}}.
Based on these examples and your memory of \texttt{\{self.model\_name\}}, predict the score this model will get for answering the following task prompt: ``\texttt{\{prompt\}}''.
The scoring metric used here is a 5-point Likert scale:
\textit{1 (Does not match at all)}: The model's response completely fails to align with the task prompt. It either lacks the required elements or presents information that contradicts the prompt.
\textit{2 (Has significant discrepancies)}: There are major mismatches between the task prompt and the model's response. Key elements may be missing, or relationships and attributes are significantly misrepresented.
\textit{3 (Has several minor discrepancies)}: The response aligns with the prompt in a general sense but has noticeable errors or omissions in key details. While the main elements are present, finer aspects are not accurately rendered.
\textit{4 (Has a few minor discrepancies)}: The response is mostly aligned with the prompt, with only a few small and less critical details being inaccurate or missing. The response matches the description well but not perfectly.
\textit{5 (Matches exactly)}: The model's response perfectly matches the task prompt. All elements, relationships, attributes, and details align seamlessly with the description, with no noticeable discrepancies.
Now, return a score between 1 and 5 based on your memory of \texttt{\{self.model\_name\}}'s capabilities and limitations. Return a single number only, nothing else.
\end{tcolorbox}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Predict Score - \tm, fonttitle=\bfseries, breakable]
Your current memory (knowledge) of the tool \texttt{\{model\_name\}} is: ``\texttt{\{current\_memory\}}'' \\
Based on your memory of model \texttt{\{model\_name\}}, predict the score for the following prompt: ``\texttt{\{prompt\}}''. \\
The scoring metric used here is a 5-point Likert scale: \\
\textit{1 (Does not match at all)}: The generated image or video completely fails to align with the text prompt. It either lacks the required elements or depicts a scenario that contradicts the prompt. \\
\textit{2 (Has significant discrepancies)}: There are major mismatches between the text prompt and the visual output. Key elements may be missing, or relationships and attributes are significantly misrepresented. \\
\textit{3 (Has several minor discrepancies)}: The output aligns with the prompt in a general sense but has noticeable errors or omissions in key details. While the main elements are present, finer aspects are not accurately rendered. \\
\textit{4 (Has a few minor discrepancies)}: The output is mostly aligned with the prompt, with only a few small and less critical details being inaccurate or missing. The visual matches the description well but not perfectly. \\
\textit{5 (Matches exactly)}: The generated image or video perfectly matches the text prompt. All elements, relationships, attributes, and details align seamlessly with the description, with no noticeable discrepancies. \\
Now return a score between 1 and 5 for prompt \texttt{\{prompt\}}, based on your memory of the tool \texttt{\{model\_name\}}. Return a single number only, nothing else. \\
\end{tcolorbox}
\subsection{Text Generation Tools}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Prompt for Memory Refinement, fonttitle=\bfseries, breakable]
You are an assistant responsible for maintaining an incremental memory of the tool's capabilities, limitations, and use cases. \\
You will receive a current memory and new feedback. Your job is to update the memory by integrating the new feedback without losing previously stored information. \\
\textbf{Rules:} \\
1. Start with the current memory. \\
2. Consider the new prompt and feedback, but do not simply repeat the feedback, think deep and get general insights. \\
3. Incrementally refine and update the memory to reflect any new insights. \\
4. Do not remove previously known capabilities unless the new feedback explicitly contradicts them. \\
5. If no new insights are provided, leave the memory unchanged. \\
\textbf{Instructions:} \\
Follow this procedure strictly: \\
- Review the current memory for model \texttt{\{tool\_name\}}: \\
\quad ``\texttt{\{current\_memory\}}'' \\
- Input task prompt to the tool was: \\
\quad ``\texttt{\{task\_prompt\}}'' \\
- Tool's answer was: \\
\quad ``\texttt{\{response\}}'' \\
- Score rubric was: \\
\quad ``\texttt{\{score\_rubric\}}'' \\
- The score for tool's response was: \\
\quad ``\texttt{\{score\}}'' \\
- Feedback for this answer received: \\
\quad ``\texttt{\{feedback\}}'' \\
Integrate the key insights of the tool's limitations and capabilities from the feedback into the existing memory. Use complete sentences starting with phrases like \texttt{good at}, \texttt{bad at}, \texttt{proficient at}, or \texttt{poor at} to describe your findings. \\
- If the feedback suggests new limitations, capabilities, or clarifications, add them. \\
- If the feedback suggests that something previously stated is incorrect, correct it. \\
- If the feedback suggests the tool struggles with a certain aspect (like counting multiple objects), note it. \\
- If no contradiction or new insight is provided, keep the memory as is. \\
- Think deeply and determine what the feedback actually means for the model's capability, rather than staying on the surface of a specific example. \\
In your final answer, output \textbf{ONLY} the updated overall memory of the tool. Do not include extra explanations. \\
\end{tcolorbox}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Predict Score - \gr, fonttitle=\bfseries, breakable]
Your current memory (knowledge) of the tool \texttt{\{model\_name\}} is: ``A large language model'' \\
Based on your memory of model \texttt{\{model\_name\}}, predict the score this model will get for answering the following task prompt: ``\texttt{\{prompt\}}''. \\
The scoring metric used here is a 5-point Likert scale: \\
\texttt{\{rubric\}} \\
Now return a score between 1 and 5, based on your memory of \texttt{\{model\_name\}}'s capabilities and limitations. \\
Return a single number only, nothing else.
\end{tcolorbox}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Predict Score - \fs, fonttitle=\bfseries, breakable]
Your current memory (knowledge) of the tool \texttt{\{model\_name\}} is: A large language model. \\
Here are some examples of the model's performance on some tasks: \\
Task Prompt: \texttt{\{task\_prompt\}} \\
Rubric: \texttt{\{rubric\}} \\
Model's Score: \texttt{\{score\}} \\
Task Prompt: \texttt{\{task\_prompt\}} \\
Rubric: \texttt{\{rubric\}} \\
Model's Score: \texttt{\{score\}} \\
Task Prompt: \texttt{\{task\_prompt\}} \\
Rubric: \texttt{\{rubric\}} \\
Model's Score: \texttt{\{score\}} \\
...
Based on those samples and your memory of \texttt{\{model\_name\}}, predict the score this model will get for answering the following task prompt: ``\texttt{\{task\_prompt\}}''. \\
The scoring metric used here is a 5-point Likert scale: \\
\texttt{\{rubric\}} \\
Now return a score between 1 and 5, based on your memory of \texttt{\{model\_name\}}'s capabilities and limitations. \\
Return a single number only, nothing else.
\end{tcolorbox}
\begin{tcolorbox}[colback=gray!10, colframe=gray!60, coltitle=black,
title=Predict Score - \tm, fonttitle=\bfseries, breakable]
Your current memory (knowledge) of the tool \texttt{\{model\_name\}} is: \texttt{\{current\_memory\}} \\
Based on your memory of model \texttt{\{model\_name\}}, predict the score this model will get for answering the following task prompt: ``\texttt{\{prompt\}}''. \\
The scoring metric used here is a 5-point Likert scale: \\
\texttt{\{rubric\}} \\
Now return a score between 1 and 5, based on your memory of \texttt{\{model\_name\}}'s capabilities and limitations. \\
Return a single number only, nothing else.
\end{tcolorbox}
\begin{figure}[htbp]
\centering
\includegraphics[width=\linewidth]{images/case_study.png}
\caption{Qualitative examples for \tm. The left image illustrates the prediction for ``An oil painting of a majestic lion hanging above the fireplace,'' while the right example shows the prediction for ``A pair of socks, one is striped, but the other is not.''}
\label{fig:case_study}
\end{figure}
\section{Case Study}
\label{app:case_study}
To better understand \tm's capability in performance prediction, we present two representative examples in Figure~\ref{fig:case_study}, where \tm is prompted to explicitly forecast the quality and fidelity of the output images generated by the SDXL-Turbo tool.
\paragraph{Example 1: Majestic Lion Above the Fireplace.}
For the prompt ``\emph{An oil painting of a majestic lion hanging above the fireplace},'' \tm assigns a predicted score of 4 (minor discrepancies). In practice, SDXL-Turbo successfully produces a visually appealing scene with a detailed lion portrait. However, our system anticipates that fine spatial alignment may not be perfectly captured (e.g., the fireplace might be cropped or misaligned). Indeed, the actual generated image exhibits a slightly off positioning of the lion, confirming \tm's prediction. Notably, the final score aligns closely with human evaluations, underlining \tm's proficiency in assessing model outputs.
\paragraph{Example 2: Pair of Socks with Different Patterns.}
For the prompt ``\emph{A pair of socks, one is striped, but the other is not},'' \tm predicts a score of 3 (several minor discrepancies). While SDXL-Turbo generally handles simple object prompts and textures adequately, \tm notes the model’s recurring difficulty in distinguishing subtle visual contrasts between highly similar objects. In the actual output, both socks often appear identical (both striped), matching \tm’s forecast. However, human evaluators deemed these inaccuracies more severe, giving a lower score of 2. This discrepancy suggests that although \tm detects the correct type of error, it occasionally assigns higher tolerance to such mismatches than human annotators.
\paragraph{Discussion.}
These examples highlight both the strengths and limitations of \tm. On one hand, \tm demonstrates strong alignment with human judgments when identifying whether the main elements of a prompt are preserved or substantially distorted. On the other hand, its scoring mechanism sometimes diverges from that of human evaluators, underscoring the subjective nature of visual quality assessments.
\section{Compute Resource and Reproducibility Details}
\label{app:computation}
The core computation in our experiments is conducted via OpenAI’s API, GPT-4o-mini serves as the backbone language model for \tm, and text-embedding-ada-002 for RAG tasks. As our framework relies on API calls, it does not require any local GPU hardware for inferences.
All other models involved---such as Claude-3, DALL·E 3, Midjourney, SDXL, and others---are part of the precomputed outputs provided by the \gb and \bb datasets. We do not rerun these tools; we only analyze and build memory around their existing outputs.
The only locally executed model in our pipeline is the clip-flant5-xl model used to calculate VQA Score, which evaluates image-description alignment. This model is run on an NVIDIA A100 GPU, and is highly efficient—each VQA score can be computed in about 0.2 seconds. As such, the compute requirement is minimal, and large batches can be processed in real time.
Overall, our pipeline is lightweight and fully reproducible. All heavy model inference is handled externally via OpenAI APIs, and the only local requirement is for lightweight VQA scoring, making \tm practical to deploy and evaluate with modest resources.
\section{Broader Impact and Limitation}
\label{app:impact_limitation}
This paper investigates a new capability for agents powered by Large Language Models (LLMs): building and utilizing structured memory representations of tool capabilities to improve tool selection and task-solving performance. By enabling agents to systematically record and retrieve performance feedback from past tool interactions, our proposed \tm framework pushes toward more autonomous and adaptive decision-making systems.
The broader impact of this line of work lies in its potential to make complex multi-tool reasoning more robust, transparent, and efficient. Applications that involve numerous generative or decision-support tools—such as multimodal assistants, educational platforms, or creative design systems—could benefit from better memory-guided tool selection, reducing trial-and-error usage and model waste. \tm may also lower the barrier for non-experts to leverage advanced model capabilities, as the system autonomously adapts based on prior observations.
However, several limitations and risks must be acknowledged. First, the memory quality heavily depends on the accuracy and reliability of the feedback used to construct it. In this work, we rely on GPT-4-generated scores and rubrics, which, although effective, may introduce biases or inconsistencies not present in human-grounded evaluations. Moreover, the framework assumes tool capabilities remain relatively stable over time. In real-world deployments, frequent model updates or usage context shifts may render historical memory entries stale or misleading.
Another concern is over-reliance on memorized preferences, which may lead to suboptimal or unfair tool choices in edge cases not well-represented in the agent’s prior experiences. Without proper monitoring, such systems might reinforce early biases or miss the opportunity to explore potentially better alternatives.
Lastly, our experiments are conducted in controlled benchmark environments with known tool outputs and synthetic feedback. Generalizing \tm to open-ended tasks, dynamic tool ecosystems, or safety-critical applications requires additional safeguards and validation, which we leave for future work.
While promising, \tm represents an early step toward memory-augmented tool reasoning, and further exploration is needed to ensure both its robustness and its alignment with human-centered values in broader deployment contexts.
\begin{thebibliography}{38}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Carrasco et~al.(2025)Carrasco, Nedungadi, Rodriguez-Fernandez, and Linares]{Carrasco_2025}
Alejandro Carrasco, Marco Nedungadi, Victor Rodriguez-Fernandez, and Richard Linares.
\newblock Visual language models as operator agents in the space domain.
\newblock In \emph{AIAA SCITECH 2025 Forum}. American Institute of Aeronautics and Astronautics, January 2025.
\newblock \doi{10.2514/6.2025-1543}.
\newblock URL \url{http://dx.doi.org/10.2514/6.2025-1543}.
\bibitem[Chen et~al.(2017)Chen, Fisch, Weston, and Bordes]{chen2017readingwikipediaansweropendomain}
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes.
\newblock Reading wikipedia to answer open-domain questions, 2017.
\newblock URL \url{https://arxiv.org/abs/1704.00051}.
\bibitem[Dunn et~al.(2017)Dunn, Sagun, Higgins, Guney, Cirik, and Cho]{dunn2017searchqanewqadataset}
Matthew Dunn, Levent Sagun, Mike Higgins, V.~Ugur Guney, Volkan Cirik, and Kyunghyun Cho.
\newblock Searchqa: A new q\&a dataset augmented with context from a search engine, 2017.
\newblock URL \url{https://arxiv.org/abs/1704.05179}.
\bibitem[Gao et~al.(2023)Gao, Ji, Zhou, Lin, Chen, Fan, and Shou]{gao2023assistgptgeneralmultimodalassistant}
Difei Gao, Lei Ji, Luowei Zhou, Kevin~Qinghong Lin, Joya Chen, Zihan Fan, and Mike~Zheng Shou.
\newblock Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn, 2023.
\newblock URL \url{https://arxiv.org/abs/2306.08640}.
\bibitem[Gao et~al.(2025)Gao, Zhang, Li, Ma, Yuan, Fan, Wu, Jia, Zhu, and Li]{gao2025multimodalagenttuningbuilding}
Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, and Qing Li.
\newblock Multi-modal agent tuning: Building a vlm-driven agent for efficient tool usage, 2025.
\newblock URL \url{https://arxiv.org/abs/2412.15606}.
\bibitem[Graves et~al.(2014)Graves, Wayne, and Danihelka]{graves2014neuralturingmachines}
Alex Graves, Greg Wayne, and Ivo Danihelka.
\newblock Neural turing machines, 2014.
\newblock URL \url{https://arxiv.org/abs/1410.5401}.
\bibitem[Guo et~al.(2024)Guo, Cheng, Wang, Liang, Qin, Li, Liu, Sun, and Liu]{guo2024stabletoolbench}
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu.
\newblock Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2024.
\bibitem[Gupta and Kembhavi(2022)]{gupta2022visualprogrammingcompositionalvisual}
Tanmay Gupta and Aniruddha Kembhavi.
\newblock Visual programming: Compositional visual reasoning without training, 2022.
\newblock URL \url{https://arxiv.org/abs/2211.11559}.
\bibitem[Guu et~al.(2020)Guu, Lee, Tung, Pasupat, and Chang]{guu2020realmretrievalaugmentedlanguagemodel}
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.
\newblock Realm: Retrieval-augmented language model pre-training, 2020.
\newblock URL \url{https://arxiv.org/abs/2002.08909}.
\bibitem[Karpukhin et~al.(2020)Karpukhin, Oğuz, Min, Lewis, Wu, Edunov, Chen, and tau Yih]{karpukhin2020densepassageretrievalopendomain}
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih.
\newblock Dense passage retrieval for open-domain question answering, 2020.
\newblock URL \url{https://arxiv.org/abs/2004.04906}.
\bibitem[Kim et~al.(2025)Kim, Suk, Cho, Longpre, Kim, Yoon, Son, Cho, Shafayat, Baek, Park, Hwang, Jo, Cho, Shin, Lee, Oh, Lee, Ho, Joo, Ko, Lee, Chae, Shin, Jang, Ye, Lin, Welleck, Neubig, Lee, Lee, and Seo]{kim2025biggenbenchprincipledbenchmark}
Seungone Kim, Juyoung Suk, Ji~Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue~Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se~June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill~Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo.
\newblock The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models, 2025.
\newblock URL \url{https://arxiv.org/abs/2406.05761}.
\bibitem[Lewis et~al.(2021)Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, tau Yih, Rocktäschel, Riedel, and Kiela]{lewis2021retrievalaugmentedgenerationknowledgeintensivenlp}
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.
\newblock Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021.
\newblock URL \url{https://arxiv.org/abs/2005.11401}.
\bibitem[Li et~al.(2024)Li, Lin, Pathak, Li, Fei, Wu, Ling, Xia, Zhang, Neubig, and Ramanan]{li2024genaibenchevaluatingimprovingcompositional}
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, and Deva Ramanan.
\newblock Genai-bench: Evaluating and improving compositional text-to-visual generation, 2024.
\newblock URL \url{https://arxiv.org/abs/2406.13743}.
\bibitem[Li et~al.(2023)Li, Zhao, Yu, Song, Li, Yu, Li, Huang, and Li]{li2023api}
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li.
\newblock {API}-bank: A comprehensive benchmark for tool-augmented {LLM}s.
\newblock In \emph{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing}, 2023.
\newblock URL \url{https://aclanthology.org/2023.emnlp-main.187/}.
\bibitem[Madaan et~al.(2023)Madaan, Tandon, Gupta, Hallinan, Gao, Wiegreffe, Alon, Dziri, Prabhumoye, Yang, Gupta, Majumder, Hermann, Welleck, Yazdanbakhsh, and Clark]{madaan2023selfrefineiterativerefinementselffeedback}
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa~Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.
\newblock Self-refine: Iterative refinement with self-feedback, 2023.
\newblock URL \url{https://arxiv.org/abs/2303.17651}.
\bibitem[Nakano et~al.(2022)Nakano, Hilton, Balaji, Wu, Ouyang, Kim, Hesse, Jain, Kosaraju, Saunders, Jiang, Cobbe, Eloundou, Krueger, Button, Knight, Chess, and Schulman]{nakano2022webgptbrowserassistedquestionansweringhuman}
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu~Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman.
\newblock Webgpt: Browser-assisted question-answering with human feedback, 2022.
\newblock URL \url{https://arxiv.org/abs/2112.09332}.
\bibitem[Ouyang et~al.(2022)Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray, Schulman, Hilton, Kelton, Miller, Simens, Askell, Welinder, Christiano, Leike, and Lowe]{ouyang2022training}
Long Ouyang, Jeff Wu, Xu~Jiang, Diogo Almeida, Carroll~L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.
\newblock Training language models to follow instructions with human feedback, 2022.
\newblock URL \url{https://arxiv.org/abs/2203.02155}.
\bibitem[Packer et~al.(2024)Packer, Wooders, Lin, Fang, Patil, Stoica, and Gonzalez]{packer2024memgptllmsoperatingsystems}
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir~G. Patil, Ion Stoica, and Joseph~E. Gonzalez.
\newblock Memgpt: Towards llms as operating systems, 2024.
\newblock URL \url{https://arxiv.org/abs/2310.08560}.
\bibitem[Qin et~al.(2023{\natexlab{a}})Qin, Liang, Ye, Zhu, Yan, Lu, Lin, Cong, Tang, Qian, Zhao, Hong, Tian, Xie, Zhou, Gerstein, Li, Liu, and Sun]{qin2023toolllmfacilitatinglargelanguage}
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.
\newblock Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2307.16789}.
\bibitem[Qin et~al.(2023{\natexlab{b}})Qin, Liang, Ye, Zhu, Yan, Lu, Lin, Cong, Tang, Qian, Zhao, Tian, Xie, Zhou, Gerstein, Li, Liu, and Sun]{qin2023toolllm}
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.
\newblock Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023{\natexlab{b}}.
\bibitem[Radford et~al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever]{clip}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
\newblock Learning transferable visual models from natural language supervision, 2021.
\newblock URL \url{https://arxiv.org/abs/2103.00020}.
\bibitem[Schick et~al.(2023)Schick, Dwivedi-Yu, Dessì, Raileanu, Lomeli, Zettlemoyer, Cancedda, and Scialom]{schick2023toolformerlanguagemodelsteach}
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
\newblock Toolformer: Language models can teach themselves to use tools, 2023.
\newblock URL \url{https://arxiv.org/abs/2302.04761}.
\bibitem[Shen et~al.(2023)Shen, Song, Tan, Li, Lu, and Zhuang]{shen2023hugginggptsolvingaitasks}
Yongliang Shen, Kaitao Song, Xu~Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.
\newblock Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023.
\newblock URL \url{https://arxiv.org/abs/2303.17580}.
\bibitem[Shinn et~al.(2023)Shinn, Cassano, Berman, Gopinath, Narasimhan, and Yao]{shinn2023reflexion}
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
\newblock Reflexion: Language agents with verbal reinforcement learning, 2023.
\bibitem[Sim et~al.(2024)Sim, Lee, Tan, and Tan]{sim2024evaluatinggenerationspatialrelations}
Shang~Hong Sim, Clarence Lee, Alvin Tan, and Cheston Tan.
\newblock Evaluating the generation of spatial relations in text and image generative models, 2024.
\newblock URL \url{https://arxiv.org/abs/2411.07664}.
\bibitem[Surís et~al.(2023)Surís, Menon, and Vondrick]{surís2023vipergptvisualinferencepython}
Dídac Surís, Sachit Menon, and Carl Vondrick.
\newblock Vipergpt: Visual inference via python execution for reasoning, 2023.
\newblock URL \url{https://arxiv.org/abs/2303.08128}.
\bibitem[Tang et~al.(2023)Tang, Deng, Lin, Han, Liang, Cao, and Sun]{tang2023toolalpaca}
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le~Sun.
\newblock Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.
\newblock \emph{arXiv preprint arXiv:2306.05301}, 2023.
\bibitem[Tang et~al.(2025{\natexlab{a}})Tang, Bi, Huang, Liang, Shimada, Hua, Xiao, Song, Liu, Feng, Guo, Liu, Song, Vosoughi, He, He, Zhang, Luo, and Xu]{tang2025captionvideofinegrainedobjectcentric}
Yunlong Tang, Jing Bi, Chao Huang, Susan Liang, Daiki Shimada, Hang Hua, Yunzhong Xiao, Yizhi Song, Pinxin Liu, Mingqian Feng, Junjia Guo, Zhuo Liu, Luchuan Song, Ali Vosoughi, Jinxi He, Liu He, Zeliang Zhang, Jiebo Luo, and Chenliang Xu.
\newblock Caption anything in video: Fine-grained object-centric captioning via spatiotemporal multimodal prompting, 2025{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2504.05541}.
\bibitem[Tang et~al.(2025{\natexlab{b}})Tang, Guo, Liu, Wang, Hua, Zhong, Xiao, Huang, Song, Liang, Song, He, Bi, Feng, Li, Zhang, and Xu]{tang2025generativeaicelanimationsurvey}
Yunlong Tang, Junjia Guo, Pinxin Liu, Zhiyuan Wang, Hang Hua, Jia-Xing Zhong, Yunzhong Xiao, Chao Huang, Luchuan Song, Susan Liang, Yizhi Song, Liu He, Jing Bi, Mingqian Feng, Xinyang Li, Zeliang Zhang, and Chenliang Xu.
\newblock Generative ai for cel-animation: A survey, 2025{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2501.06250}.
\bibitem[Wang et~al.(2023)Wang, Xie, Jiang, Mandlekar, Xiao, Zhu, Fan, and Anandkumar]{wang2023voyageropenendedembodiedagent}
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.
\newblock Voyager: An open-ended embodied agent with large language models, 2023.
\newblock URL \url{https://arxiv.org/abs/2305.16291}.
\bibitem[Wang et~al.(2024{\natexlab{a}})Wang, Li, Li, and Liu]{wang2024genartistmultimodalllmagent}
Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu.
\newblock Genartist: Multimodal llm as an agent for unified image generation and editing, 2024{\natexlab{a}}.
\newblock URL \url{https://arxiv.org/abs/2407.05600}.
\bibitem[Wang et~al.(2024{\natexlab{b}})Wang, Fried, and Neubig]{wang2024troveinducingverifiableefficient}
Zhiruo Wang, Daniel Fried, and Graham Neubig.
\newblock Trove: Inducing verifiable and efficient toolboxes for solving programmatic tasks, 2024{\natexlab{b}}.
\newblock URL \url{https://arxiv.org/abs/2401.12869}.
\bibitem[Wang et~al.(2024{\natexlab{c}})Wang, Mao, Fried, and Neubig]{wang2024agentworkflowmemory}
Zora~Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig.
\newblock Agent workflow memory, 2024{\natexlab{c}}.
\newblock URL \url{https://arxiv.org/abs/2409.07429}.
\bibitem[Wu et~al.(2023)Wu, Yin, Qi, Wang, Tang, and Duan]{wu2023visualchatgpttalkingdrawing}
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.
\newblock Visual chatgpt: Talking, drawing and editing with visual foundation models, 2023.
\newblock URL \url{https://arxiv.org/abs/2303.04671}.
\bibitem[Xie et~al.(2024)Xie, Chen, Zhang, Wan, and Li]{xie2024largemultimodalagentssurvey}
Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li.
\newblock Large multimodal agents: A survey, 2024.
\newblock URL \url{https://arxiv.org/abs/2402.15116}.
\bibitem[Xu et~al.(2025)Xu, Liang, Mei, Gao, Tan, and Zhang]{xu2025amemagenticmemoryllm}
Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang.
\newblock A-mem: Agentic memory for llm agents, 2025.
\newblock URL \url{https://arxiv.org/abs/2502.12110}.
\bibitem[Yao et~al.(2023)Yao, Zhao, Yu, Du, Shafran, Narasimhan, and Cao]{yao2023reactsynergizingreasoningacting}
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
\newblock React: Synergizing reasoning and acting in language models, 2023.
\newblock URL \url{https://arxiv.org/abs/2210.03629}.
\bibitem[Zhong et~al.(2023)Zhong, Guo, Gao, Ye, and Wang]{zhong2023memorybankenhancinglargelanguage}
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He~Ye, and Yanlin Wang.
\newblock Memorybank: Enhancing large language models with long-term memory, 2023.
\newblock URL \url{https://arxiv.org/abs/2305.10250}.
\end{thebibliography}