ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis
Extracted source text · arXiv 2605.25160
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
\PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2026
% The authors should use one of these tracks.
% Before accepting by the NeurIPS conference, select one of the options below.
% 0. "default" for submission
\usepackage[preprint]{neurips_2026}
% the "default" option is equal to the "main" option, which is used for the Main Track with double-blind reviewing.
% 1. "main" option is used for the Main Track
% \usepackage[main]{neurips_2026}
% 2. "position" option is used for the Position Paper Track
% \usepackage[position]{neurips_2026}
% 3. "eandd" option is used for the Evaluations & Datasets Track
% \usepackage[eandd]{neurips_2026}
% 4. "creativeai" option is used for the Creative AI Track
% \usepackage[creativeai]{neurips_2026}
% 5. "sglblindworkshop" option is used for the Workshop with single-blind reviewing
% \usepackage[sglblindworkshop]{neurips_2026}
% 6. "dblblindworkshop" option is used for the Workshop with double-blind reviewing
% \usepackage[dblblindworkshop]{neurips_2026}
% After being accepted, the authors should add "final" behind the track to compile a camera-ready version.
% 1. Main Track
% \usepackage[main, final]{neurips_2026}
% 2. Position Paper Track
% \usepackage[position, final]{neurips_2026}
% 3. Evaluations & Datasets Track
% \usepackage[eandd, final]{neurips_2026}
% 4. Creative AI Track
% \usepackage[creativeai, final]{neurips_2026}
% 5. Workshop with single-blind reviewing
% \usepackage[sglblindworkshop, final]{neurips_2026}
% 6. Workshop with double-blind reviewing
% \usepackage[dblblindworkshop, final]{neurips_2026}
% Note. For the workshop paper template, both \title{} and \workshoptitle{} are required, with the former indicating the paper title shown in the title and the latter indicating the workshop title displayed in the footnote.
% For workshops (5., 6.), the authors should add the name of the workshop, "\workshoptitle" command is used to set the workshop title.
% \workshoptitle{WORKSHOP TITLE}
% "preprint" option is used for arXiv or other preprint submissions
% \usepackage[preprint]{neurips_2026}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2026}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage[HTML]{xcolor} % colors
\usepackage{mwe}
\usepackage{array}
\usepackage{float}
\usepackage{makecell}
\usepackage{pifont}
\usepackage{subcaption}
\usepackage{xspace}
\usepackage{minted}
\usepackage{wrapfig}
\usepackage{tcolorbox}
\usepackage{enumitem}
\tcbuselibrary{breakable, skins}
% Note. For the workshop paper template, both \title{} and \workshoptitle{} are required, with the former indicating the paper title shown in the title and the latter indicating the workshop title displayed in the footnote.
\title{SimuWoB: Simulating Real-World Mobile Apps for Fast and Faithful GUI Agent Benchmarking}
% \author{%
% Guohong Liu \\
% Institute for AI Industry Research (AIR)\\
% Tsinghua University
% \And Jialei Ye \\
% University of Electronic Science and Technology of China \\
% \And Pengzhi Gao \\
% MiLM Plus, Xiaomi Inc.
% \And Wei Liu \\
% MiLM Plus, Xiaomi Inc.
% \And Jian Luan \\
% MiLM Plus, Xiaomi Inc.
% \And Yunxin Liu \\
% Institute for AI Industry Research (AIR)\\
% Tsinghua University
% \And Yuanchun Li \\
% Institute for AI Industry Research (AIR)\\
% Tsinghua University
% }
\author{%
\textbf{Guohong Liu}$^{1}$,
Jialei Ye$^{2}$,
Pengzhi Gao$^{3}$,
Wei Liu$^{3}$,
Jian Luan$^{3}$,
Yunxin Liu$^{1}$,
Yuanchun Li$^{1\ddagger}$ \\
\textmd{} \\
$^{1}$ Institute for AI Industry Research (AIR), Tsinghua University \\
$^{2}$ University of Electronic Science and Technology of China \\
$^{3}$ MiLM Plus, Xiaomi Inc. \\
$^{\ddagger}$ Corresponding author
}
\newcommand{\sys}{SimuWoB}
\newcommand{\eg}{e.g.\xspace}
\newcommand{\etc}{etc.\xspace}
\definecolor{myframe}{HTML}{118C7E}
\newcolumntype{P}[1]{>{\raggedright\arraybackslash}p{#1}}
\newcolumntype{C}[1]{>{\centering\arraybackslash}p{#1}}
\newcolumntype{R}[1]{>{\raggedleft\arraybackslash}p{#1}}
\begin{document}
\maketitle
\input{tex/abstract.tex}
\section{Introduction}
\label{sec:intro}
\input{tex/introduction.tex}
\section{Related Work}
\label{sec:related_work}
\input{tex/related_work.tex}
\section{\sys}
\label{sec:method}
\input{tex/method.tex}
\section{Experiments}
\label{sec:experiments}
\input{tex/experiments.tex}
\section{Limitations}
\label{sec:limitations}
\input{tex/limitations.tex}
\section{Conclusion}
\label{sec:conclusion}
\input{tex/conclusion.tex}
{\small\bibliographystyle{unsrtnat}
\bibliography{ref}}
\appendix
\input{tex/appendix.tex}
\newpage
\input{tex/checklist.tex}
\end{document}
\begin{abstract}
Mobile GUI agents powered by large language models have progressed rapidly,
creating urgent needs for realistic and comprehensive
evaluation. Existing benchmarks prioritize reproducibility but are often limited to open-source apps
or file-operation tasks for the difficulty of constructing rewards on real applications,
leaving a gap between benchmark settings and real-world usage.
Moreover, most benchmarks focus on basic grounding and navigation, with limited coverage of complex, long-horizon interactions.
To address these limitations, we introduce \sys, a fully synthetic benchmark for mobile GUI agents
with 120 challenging tasks spanning diverse types and difficulty levels.
We build a robust virtual environment generation framework that synthesizes high-fidelity tasks and environments,
and automatically provides valid rewards for each task.
Each environment is deployed as a backend-free webpage accessible via URL, enabling
efficient and reproducible evaluation.
We conduct comprehensive experiments on several state-of-the-art mobile GUI agents.
The average success rate is only 27.92\%,
dropping to 17.82\% on long-horizon tasks, which reveals substantial weaknesses in current agents
under complex scenarios.
Evaluation result comparison with real-world sample tasks demonstrate that agent assessments based on
our synthetic environment generalize well. We further provide diagnostic insights across key capability
dimensions and discuss implications for future mobile GUI agent development.
\end{abstract}
Mobile GUI agents~%
\cite{autodroid,UI-TARS,Aguvis,OS-Copilot,aria-ui,android-in-the-zoo,mobile-agent-v3,agent-q,agent-s,autoglm,step-gui,mobileagentv3.5,MAI-UI}
powered by large language models~(LLMs) and vision language models~(VLMs) have progressed rapidly in recent years.
Given a task and an interactive interface, an autonomous agent must interpret the current screen, reason over task state,
execute actions step by step, and use environment feedback until completion.
To measure these capabilities, many benchmarks~%
\cite{android-lab,android-world,os-world,CRAB,ScienceBoard,windows-agent-arena,mobile-agent-bench,MobileWorld,%
MobileEnv,weblinux,WebVoyager,pixelhelp}
have been proposed across platforms and task settings.
Yet existing benchmarks still fall short of fast and faithful evaluation in realistic mobile scenarios in several perspectives.
\textbf{(1)~Limited realism and diversity of environments/tasks.}
To maintain reproducibility and robust reward checking, many benchmarks are constrained to use open-source
applications with public backends~\cite{android-world,MobileWorld}, file-operation tasks~\cite{EvoCUA},
or execution-pattern matching that requires substantial manual efforts~\cite{android-lab,os-world,windows-agent-arena}.
These constraints narrow task coverage and create a gap between benchmark tasks and real-world app usage scenarios.
\textbf{(2)~Limited task complexity.}
As the capabilities of GUI agents continue to advance, benchmark difficulty should correspondingly
increase to align with this trend and yield more valuable evaluation results.
However, many still emphasize vision grounding~\cite{SeeClick,OS-ATLAS,ScreenSpot-pro} and
simple operations~(\eg, basic actions and navigations), with limited stress testing of long-horizon execution,
intermediate information management, and multi-step reasoning, \etc
This limits their ability to guide the next stage of agent development.
\textbf{(3)~Limited evaluation efficiency.}
Many interactive benchmarks depend on emulators, virtual devices, or Dockerized systems~\cite{android-world,os-world,windows-agent-arena,WebArena}
for state loading, resetting and recoveries.
While effective, this setup increases system complexity and runtime overhead, slowing large-scale
evaluation and downstream tasks such as online RL trainings.
To address these limitations, we propose {\sys}.
\begin{figure}[H]
\centering
\includegraphics[width=0.9\textwidth]{figures/overview.pdf}
\caption{An overview of {\sys}. After collecting real-world tasks along with related applications,
our framework iteratively builds and refines the environment until the task and reward validator
are ready. Mobile GUI agents can then directly interact with the environments and perform tasks.}
\label{fig:overview}
\end{figure}
{\sys} is a fully synthetic benchmark for mobile GUI agents, built on a \emph{simulated environment generation framework}
that leverages LLM code generation abilities%
~\cite{llm-codegen-survey,code-foundation-model-agents,software-dev-life-cycle,challenges-paths-ai-software}
as illustrated in Figure~\ref{fig:overview}.
Given a natural-language task description and optional screenshots of the target workflow,
the framework iteratively generates and refines a backend-free webpage that simulates a real-world mobile application
with aligned interaction logic over mock data.
During the generation process, it also validates task solvability and produces executable
and valid reward functions for each task.
This design directly improves \textbf{environment coverage}, \textbf{reward reliability}, and \textbf{evaluation speed},
circumventing the challenge of constructing rewards inherent in real-world apps,
enabling more faithful benchmarking at a much lower operational cost.
Built on this framework, \sys~includes 120 tasks across 63 simulated app environments,
taking up 20 of 33 Google Play Store app categories.
All tasks and scenarios are collected based on a user study covering
86 participants across 77 industries.
Every environment is able to be served via a URL, which enables lightweight deployment and efficient evaluation in parallel
with near-zero setup overhead. To probe different capability dimensions, we organize tasks into three categories:
\emph{simple}, \emph{long-horizon}, and \emph{math-related}.
Half of the tasks require more than 20 interaction steps, and the most challenging ones require over 50 steps to finish.
We perform comprehensive experiments on several recent state-of-the-art mobile GUI agents%
~\cite{UI-TARS,autoglm,step-gui,Seed1.8,gemini}.
Results show substantial headroom for current agents: the average success rate is 27.92\% over all tasks and only 17.82\% on the long-horizon subset.
Besides, evaluation result comparison with 20 sample tasks selected from the real world demonstrate that agent assessments based on our synthetic environment generalize well.
Our contributions are as follows:
\begin{enumerate}
\item We develop a scalable LLM-based framework that generates interactive, verifiable mobile app environments and tasks from natural-language descriptions for efficient agent evaluation.
\item Using this framework, we synthesize 63 simulated mobile applications and 120 tasks spanning multiple languages, task formats, and difficulty levels.
\item We benchmark five state-of-the-art mobile GUI agents and show large performance gaps on complex tasks, especially long-horizon tasks, with detailed analysis of failure modes and implications.
\end{enumerate}
GUI agents operate digital interfaces in the same loop as humans:
observe the current UI state, reason about intent and progress, and execute actions across desktop, web, and mobile platforms.
With the rise of LLMs and VLMs~\cite{llm-survey,survey-of-llms}, recent work has rapidly expanded agent capabilities~%
\cite{autodroid,UI-TARS,Aguvis,aria-ui,mobile-agent-v3,agent-q,agent-s,autoglm,step-gui,mobileagentv3.5,MAI-UI,CogAgent}.
Early systems mainly relied on text-only interface representations~\cite{autodroid,mind2web,AutoWebGLM,WebAgent},
while newer agents increasingly consume screenshots and other visual signals~%
\cite{UI-TARS,aria-ui,mobile-agent-v3,autoglm,step-gui,OS-ATLAS,CogAgent,OmniParser,UGround}.
From a systems perspective, existing approaches broadly include modular pipelines and more end-to-end policies that directly map multimodal observations to actions.
Recent surveys~\cite{cua-survey,osagents} further highlight that robust computer-use agents require stronger long-horizon planning, memory, grounding reliability, and stable execution under noisy interface states.
Benchmarks for GUI agents can be grouped into {static datasets} and {interactive environments}.
Static datasets~\cite{pixelhelp,SeeClick,OS-ATLAS,ScreenSpot-pro,mind2web,aitw,gaia,seq2act,motif,meta-gui,OmniACT,AndroidControl}
are valuable for scalable offline evaluation of grounding, instruction following, and action prediction.
However, they generally do not capture closed-loop interaction dynamics (\eg, recovery from mistakes, delayed feedback, or stateful multi-step dependencies).
Interactive benchmarks~\cite{android-lab,android-world,os-world,mobile-agent-bench,MobileWorld,WebArena,agentbench,miniwob++,webshop,visual-WebArena,WorkArena,wikihow,Android-Agent-Arena}
provide executable environments and therefore better measure end-to-end success.
These have substantially advanced reproducible evaluation, but also expose a practical trade-off:
higher realism often brings higher engineering cost in environment setup, state reset, and resource consumption.
Many benchmarks depend on containers, emulators, or VM snapshots to preserve recoverability, which can limit evaluation speed and concurrency.
Another bottleneck is {reward construction under realistic app constraints}.
For reliable automatic scoring, many benchmarks either focus on environments with accessible internal state (\eg, open-source apps, synthetic web worlds, or file-based tasks)~\cite{android-world,MobileWorld,EvoCUA}
or use extensive manual rule engineering for execution checking~\cite{android-lab,os-world,windows-agent-arena}.
This makes it difficult to scale toward faithful simulations of real-world workflows, especially when task state is complex and not directly observable.
At the same time, a non-trivial portion of current tasks remains short-horizon or structurally simple, reducing discriminative power for stronger agents and leaving long-horizon failure modes underexplored.
Our work is positioned at this intersection.
{\sys}~emphasizes \emph{faithful simulation of real-world mobile apps} and \emph{fast benchmarking throughput} by synthesizing backend-free, URL-accessible environments together with executable rewards.
Compared with prior mobile and cross-platform benchmarks, this design aims to improve realism-task coverage, reward scalability, and evaluation efficiency simultaneously, while explicitly stressing complex task categories such as long-horizon, ambiguous, composite, and reasoning-heavy workflows.
In this section, we introduce {\sys}, a fully synthetic benchmark with 120 tasks
derived from real-world use cases and paired simulated environments.
In Section~\ref{subsec:env-gen}, we describe how environments are generated with our LLM-powered
pipeline and refined through an automatic feedback loop.
In Section~\ref{subsec:bench-construct}, we present the task selection and benchmark construction process.
\subsection{Environment Generation}
\label{subsec:env-gen}
Our goal is to build executable mobile environments that are both realistic enough to reflect real-app
interaction patterns and structured enough for large-scale, reliable rewards and evaluation.
In practice, this requires jointly handling UI layout, interaction logic, persistent data state,
and task-level verification, while keeping generation cost manageable across apps and tasks.
We therefore formulate environment synthesis as a 2-stage process rather than a single-pass generation,
as shown in Figure~\ref{fig:gen}.
The design principle is to first construct a realistic app simulation, and then inject benchmark tasks and validators.
This separation improves both quality control and generation efficiency: Stage 1 focuses on app fidelity and functional completeness, while Stage 2 focuses on task executability and precise automatic checking.
The following paragraphs describe these two stages and the subsequent validation-and-repair loop used to ensure final usability.
\textbf{Stage 1: Minimal working environment construction.}
We first collect application metadata from public sources, including app name,
visual style, feature summary, and core interaction logic.
When available, additional screenshots will be provided to better align layout patterns,
iconography, and information hierarchy with real applications.
Given these inputs, a code-generation LLM (\eg, Gemini~\cite{gemini} or Claude~\cite{Claude})
runs an iterative build loop.
In each iteration, the model:
\emph{(i)} drafts or updates a PRD (Product Requirements Document)
that designs the application's pages, features, design styles, \etc
\emph{(ii)} implements or revises page structure, data schema, and interaction logic,
and \emph{(iii)} performs a self-review pass over completeness and consistency.
The review output is fed back into the PRD, which drives the next implementation round.
After a predefined number of iterations, we obtain a stable minimal working environment (MWE) with executable UI logic, initial data entities, and seed mock records.
Examples of PRD and refinements are listed in Appendix~\ref{appendix:env-synthesizing}.
\begin{figure}[H]
\centering
\includegraphics[width=0.85\textwidth]{figures/gen.pdf}
\caption{Two-stage environment synthesis pipeline of \sys.
% Stage 1 takes app metadata and optional screenshots,
% then iteratively refines UI pages, data entities, and interaction logic to produce a minimal working environment.
% Stage 2 injects task requirements, further edits the environment for task executability, and synthesizes task-specific reward validators.
}
\label{fig:gen}
\end{figure}
\textbf{Stage 2: Task injection and reward synthesis.}
Starting from the MWE, we first expand the database with richer mock content (texts, images, and structured records) using the same schema and style constraints as Stage 1.
We then provide task specifications that include expected execution intent and verification criteria.
A task-injection agent scans the codebase, patches task-relevant logic when necessary, and synthesizes executable validators for each task.
Because we control fine-grained environment state transitions, validators can check success conditions with perfect precision rather than relying only on approximate pattern matching.
Besides, the two-stage design also improves environment quality.
By constructing app logic before task-specific editing, the generated environment is less likely to overfit onto a single target trajectory.
In other words, the environment remains broadly usable beyond one scripted path.
To further reduce task-path overfitting, we co-generate related tasks under the same app context instead of injecting isolated tasks one by one.
\begin{figure}[H]
\centering
\includegraphics[width=0.8\textwidth]{figures/auto-validation.pdf}
\caption{Automatic issue inspection and correction workflow.
% For each generated app bundle, a validation agent executes benchmark tasks in the synthesized environment.
% Failed trajectories are escalated to human experts for root-cause diagnosis.
% If the issue is environment-side, experts provide targeted fix instructions, and the bundle re-enters the repair loop.
% Bundles that pass validation are added to the final pool, followed by random quality audits.
}
\label{fig:auto-validation}
\end{figure}
Considering that large language models, when generating complex environments, are constrained by their
inherent capabilities and potential hallucinations, the resulting environments cannot guarantee 100\% usability.
For instance, they may contain flawed interaction logic or UI design issues that prevent certain tasks from being completed.
To address this, we designed a human-in-the-loop issue detection and repair mechanism that remains scalable and
applicable even in large-scale generation scenarios. The main workflow of this is shown in Figure~\ref{fig:auto-validation}.
For each generated app bundle, we run a multi-step verification procedure.
For every task in the bundle, a validation agent executes the task interactively, and the synthesized validator determines success or failure.
Successful tasks are provisionally accepted.
Failed trajectories, together with environment artifacts, are sent to human experts for triage.
Experts determine whether failure is caused by agent behavior or by environment/task defects.
If environment-side defects are identified, experts provide targeted feedback, and the bundle returns to the generation pipeline for repair and re-validation.
Only bundles that pass this loop are moved into the candidate benchmark pool.
We then perform additional manual quality control via random sampling to inspect usability, logical consistency, and task reasonableness.
For the final benchmark release, \textbf{all environments and tasks are manually verified} to ensure rigorous quality and reliable experimental conclusions.
Figure~\ref{fig:task-example} shows an example generated environment and task.
\begin{figure}[H]
\centering
\includegraphics[width=0.9\textwidth]{figures/task-example.pdf}
\caption{Example task in \sys.
The agent is asked to top up the wallet by 100 euros in a simulated hotel-booking app.
The generated environment closely matches real-app appearance and interaction behavior, and persists state via local storage for consistent multi-step execution.}
\label{fig:task-example}
\end{figure}
\subsection{\sys~Benchmark}
\label{subsec:bench-construct}
\subsubsection{Tasks \& Environments}
\sys~is constructed with a user-need-driven pipeline.
To anchor the benchmark in real-world mobile use cases, we conducted a user study and collected open-ended task requests describing participants' daily demands.
After filtering malformed records, the study pool contains 260 valid requests from 86 participants across 77 industries.
Each record includes a natural-language user command, background pain points, and an annotator judgment of whether the request is currently feasible for a mobile agent.
We then transform all raw requirement records into benchmark tasks through four steps:
\emph{(i)}~We normalize requests by merging semantically equivalent intents and removing near-duplicates.
\emph{(ii)}~We perform feasibility screening and exclude requests that require unavailable permissions, unsupported cross-platform integrations, or non-executable conditions.
\emph{(iii)}~We operationalize retained intents into executable task specifications with explicit completion criteria.
\emph{(iv)}~We balance the benchmark across language, app domains, and difficulty dimensions.
Check Appendix~\ref{appendix:user-study} for more details of this study.
Every task is then fed into the pipeline as stated in Section~\ref{subsec:env-gen}.
In addition to the target task, we also leverage LLMs to propose other reasonable tasks within the app,
incorporating them into the generation process to ensure the final application is more comprehensive and diverse in functionality.
% Table~\ref{tab:user-study-example} provides an example of collected real-world user requests,
% and the corresponding verified evaluation tasks included in \sys.
\begin{table}[H]
\centering
\caption{Examples of tasks with and without return values.
For return-required tasks, the agent receives an additional JSON schema and must output a schema-compliant JSON object for verification.}
\begin{tabular}{P{6em}P{16em}C{10em}}
\toprule
\textbf{Task Type} & \textbf{Task} & \textbf{Return Format} \\
\midrule
\emph{w/ returns} & In app ..., how much have I spent on ... in 2023? & \texttt{\{"price":130.2\}} \\
\emph{w/o returns} & In app ..., please archive all emails that meet the ... condition. & \texttt{(null)} \\
\bottomrule
\end{tabular}
\label{tab:return-example}
\end{table}
Following this process, \sys~contains 120 executable tasks from 63 distinct virtual apps~(\eg, Gmail, Reddit, Spotify, Telegram),
covering representative commercial scenarios and daily use cases.
We compiled statistics on the number of app categories covered by various mobile GUI agent benchmarks
(based on the 33 app categories defined by the Google Play Store), where \sys~covers 20 out of the 33 app categories,
exceeding 60\%; in contrast, other benchmarks~\cite{android-world,android-lab,MobileWorld} cover only around 30\%,
predominantly consisting of communication and tools applications.
This indicates that \sys~offers a more comprehensive evaluation, resulting in a much smaller gap relative to real-world usage scenarios.
The required interaction length of the final task set ranges from about 10 to over 50 steps.
Tasks in~\sys~are categorized from multiple perspectives as follows.
\emph{(1)}~\textbf{Language}. In order to evaluate cross-lingual robustness and account for language-specific UI design conventions,
\sys~includes both Chinese and English tasks.
\emph{(2)}~\textbf{Returns}. Real-world mobile tasks often require both UI execution and structured information output.
For example, an agent may need to compute a summary value after completing several interactions.
To reflect this requirement, \sys~includes tasks with explicit return values.
For these tasks, the agent receives a JSON schema at the start and must return a schema-compliant JSON object;
both the system state and the return value provided will be considered to compute the reward.
Examples are shown in Table~\ref{tab:return-example}.
\emph{(3)}~\textbf{Task Categories}. To ensure diversity in task difficulty,
\sys~covers simple navigation/operation tasks as well as more practical, challenging workflows.
We group tasks into 3 categories according to the main source of difficulty:
\emph{simple} (naive navigation and operations),
\emph{long-horizon} (long step chains/loops, or more information involved),
and \emph{math-related} (information aggregation and calculations).
This taxonomy enables a finer-grained analysis of agent capability under different failure modes.
Example tasks of each category and average steps are listed in Table~\ref{tab:complexity-example}.
In general, \emph{long-horizon} tasks require more steps (near 25) than the other categories.
\begin{table}[H]
\centering
\caption{Example tasks from different categories.}
\begin{tabular}{P{6em}P{25em}P{3.5em}}
\toprule
\textbf{Task Type} & \textbf{Task} & \textbf{\# Steps} \\
\midrule
\emph{Simple} & Add xxx item to the cart. & 14.96\\
\emph{Long-Horizon} & View the first 15 images recommended in ``For You'', summarize their titles, posters, and the posters' follower counts. & 24.73\\
\emph{Math-Related} & During my purchase history, how much did I spend on shipping in total? & 11.27\\
\bottomrule
\end{tabular}
\label{tab:complexity-example}
\end{table}
\subsubsection{Evaluation}
\label{subsubsec:method-eval}
To support standardized and efficient evaluation, each environment exposes several lightweight DOM-level API functions:
\emph{(1)}~\texttt{window.getTasks}: returns task objects containing task description,
unique id, and an optional JSON return schema.
\emph{(2)}~\texttt{window.evaluateTask}: evaluates whether the current environment state
satisfies a target task id, with optional returned content for schema-based checking.
\emph{(3)}~\texttt{window.reset}: restores the environment to its initial state before
action execution, preventing cross-task interference within the same application instance.
Examples of these functions are listed in Appendix~\ref{appendix:env-synthesizing}.
% Because environments are browser-based, evaluation naturally supports headless execution and parallel workers.
Depending on hardware resources, the system can typically run 8--16 concurrent workers or more.
\subsection{Settings}
\label{subsec:settings}
We evaluate recent mobile GUI agents on \sys.
Because \sys~is currently implemented on web-based tech stacks, it provides only screenshot observations and no structured UI signals (e.g., Android Accessibility trees). Therefore, we include only agents that can solve tasks from visual input alone.
The evaluated agents are UI-TARS-1.5~\cite{UI-TARS}, doubao-seed~\cite{Seed1.8}, Gemini 3 Pro~\cite{Gemini3}, MAI-UI~\cite{MAI-UI}, and Mobile-Agent-v3.5~\cite{mobileagentv3.5}.
The first three are API-based models, while the latter two are open-source fine-tuned ones.
For API models, specific checkpoints are \texttt{doubao-seed-1.8-251228}, \texttt{doubao-ui-tars-250428}, and \texttt{gemini-3-pro-preview}; for local models, we deploy \texttt{GUI-Owl-1.5-8B-Instruct} and \texttt{MAI-UI-8B}.
The experiment is run with 8 parallel workers.
In preliminary trials, local fine-tuned models failed to produce schema-valid JSON outputs under prompting;
accordingly, for local models we report results only on tasks without return-value requirements.
Our primary metric is success rate (SR).
To prevent infinite loops, we cap each trajectory at 100 steps.
This cap is sufficient for task completion while keeping evaluation cost manageable.
We also evaluate human performance on these tasks as well.
All results are averaged over two runs.
Furthermore, to test whether the discriminative power of our benchmark for agents
aligns with real tasks in actual applications, we designed 20 similar tasks from
17 apps in a real-world environment and evaluated these agents.
The results from the real-world environment were manually assessed to ensure accuracy.
The details of experiment setting up, real-world validation tasks, \etc are at Appendix~\ref{appendix:result}.
\subsection{Experimental Results}
\begin{figure}[H]
\centering
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/nips-full.pdf}
\caption{GUI agent SR on different benchmarks}
\label{subfig:full-sr}
\end{subfigure}
\begin{subfigure}[b]{0.48\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/nips-params.pdf}
\caption{SR on w/ \& w/o return-value subsets}
\label{subfig:return-value-sr}
\end{subfigure}
\caption{Experimental results of different agents on \sys.
For local models, we evaluate only tasks without return-value requirements.}
% Since Gemini 3 Pro does not have an official report, we take the score of Gemini 2.5 Pro as its lower bound.}
\label{fig:main-result}
\end{figure}
The main results are shown in Figure~\ref{fig:main-result}. Overall, \texttt{seed-1.8} achieves
the best SR on \sys, followed by \texttt{Gemini 3 Pro}, \texttt{UI-TARS-1.5},
\texttt{MAI-UI} and \texttt{GUI-Owl-1.5}.
For tasks without return-value requirements, \texttt{seed-1.8} reaches 50.00\%,
\texttt{Gemini 3 Pro} reaches 45.27\% and \texttt{UI-TARS-1.5}
reaches 39.86\%; for tasks with return-value requirements, their SR drops to 30.43\%, 28.26\% and 13.04\%, respectively;
this phenomenon may stem from the fact that tasks requiring return values are relatively more difficult.
For local models, \texttt{MAI-UI} outperforms \texttt{GUI-Owl-1.5}.
Although all agents report much higher scores on AndroidWorld (64.2\%--71.6\%), their SR on
\sys~is substantially lower, indicating the higher challenge posed by \sys.
While humans earn an average score of 92.08\%, which proves there is still huge ability gap between current mobile GUI agents
and humans.
Notably, the performance trends observed on sample tasks from real apps are largely consistent with those of \sys,
meaning the relative performance rankings among models remain the same.
This demonstrates that evaluation results obtained in synthetic environments can generalize to real-world apps,
thereby validating the feasibility and generalization ability of this approach.
\subsection{Analysis}
\begin{figure}[H]
\centering
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/nips-simple.pdf}
\caption{Simple Subset}
\end{subfigure}
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/nips-long.pdf}
\caption{Long-horizon Subset}
\end{subfigure}
\begin{subfigure}[b]{0.32\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/nips-math.pdf}
\caption{Math-related Subset}
\end{subfigure}
\caption{Success rate of different task categories across evaluated agents in \sys.}
\label{fig:category-result}
\end{figure}
\textbf{Tasks in \sys~are challenging for mobile GUI agents.}
As shown in Figure~\ref{subfig:full-sr}, all evaluated agents achieve much higher SR
on AndroidWorld (64.2\%--71.6\%) than on \sys.
On \sys, overall SR drops to 42.50\% for the best model (\texttt{seed-1.8})
and to as low as 9.17\% for \texttt{GUI-Owl-1.5-8B}; even on the easier subset without
return-value requirements, the best SR is only 50.00\%.
At the aggregate level, the average SR across all models is 27.92\%, versus
69.38\% on AndroidWorld, and this drop is consistent for all agents.
In addition, \sys~exhibits clear discriminative power: the best--worst spread in overall SR reaches
33.33 points (42.50\% vs 9.17\%), and even within stronger API models the spread remains substantial
(42.50\% vs 29.58\%). These results show that \sys~is substantially more challenging for current mobile
GUI agents and provides stronger separation across capability levels.
From Figure~\ref{subfig:return-value-sr}, we can tell that tasks with return values are relatively harder,
leading to an obvious SR drop for agents.
% \textbf{Mobile GUI agents struggle in tasks requiring return values.}
% Agents perform better on pure operational
% tasks that do not require a return value~(50.68\% vs 30.43\%, 39.68\% vs 13.04\%).
% Existing GUI agent models are often trained and optimized specifically for GUI operations,
% lacking training in capabilities related to information integration and feedback.
% Although some agents can implement tool calls through basic MCP~\cite{MAI-UI,autoglm}, these tools are often used to request user input.
% Many agents support the ``answer'' command~(\eg, GUI-Owl and MAI-UI), but they typically only could return simple descriptions in
% natural language. When faced with scenarios that require providing return values in a structured format,
% this approach is difficult to constrain and evaluate.
\begin{figure}[H]
\centering
\includegraphics[width=0.95\textwidth]{figures/case1.pdf}
\caption{Case study of a \emph{long-horizon} failure: the agent executes UI operations correctly
but does not persist key information in context, leading to an incorrect final answer.}
\label{fig:case1}
\end{figure}
\textbf{Mobile GUI agents fall short in \emph{long-horizon} tasks.}
Results in Figure~\ref{fig:category-result} show a clear drop on \emph{long-horizon} tasks compared with \emph{simple} tasks
for all agents (\eg, 56.48\% vs 29.09\%), with an average SR of only 17.82\%.
This gap suggests the core bottleneck is not UI operation, but state management across many steps:
agents must identify and preserve critical intermediate information while discarding low-value ones under limited context budgets.
The case in Figure~\ref{fig:case1} illustrates this failure mode.
The task asks the agent to traverse historical trips and aggregate service fees and taxes.
The agent executes each UI step correctly (opening ``Trips'', entering history,
opening each trip invoice), but fails to write extracted values into a persistent internal state.
It then moves to the next trip assuming previously seen details are still available,
and eventually produces an incorrect final response.
Because most agents cannot keep all prior screenshots and observations in context at every step,
\emph{long-horizon} performance depends on explicit memory strategies
(what to store, when to update, and how to reuse it), which current models still handle poorly.
Similar issues also exist for \emph{math-related} tasks because agents also need to take down
key information for further calculations.
SR of human also falls on these subsets.
\begin{figure}[H]
\centering
\includegraphics[width=0.95\textwidth]{figures/case2.pdf}
\caption{Case study of a \emph{vague-description} failure: the agent fails to locate
the task entry point due to a lack of proactive exploration capabilities following initial failures.}
\label{fig:case2}
\end{figure}
\textbf{Tasks with vague descriptions or inconspicuous functional entry points can confuse the agent.}
Our analysis shows that agent performance degrades when instructions are underspecified and the true entry point is visually inconspicuous~(\eg, deeply nested in the UI and not exposed on the home screen).
In these settings, the key bottleneck is weak proactive exploration after an initial strategy fails.
The case in Figure~\ref{fig:case2} illustrates this behavior.
The agent is asked to find a ``Client Sync'' event in a simulated Gmail app and return its start time and participants.
Because the instruction does not specify whether the event should be searched in email or calendar views (the latter requires opening the side menu), the agent first searches in email and finds no result.
Instead of systematically switching hypotheses and exploring alternative entry points, it continues with low-yield scrolling and eventually fails.
This pattern suggests that robust exploration, trial-and-error adaptation, and online experience reuse remain underdeveloped in current agents, especially when task cues are ambiguous and interface affordances are not explicit.
\begin{wrapfigure}{r}{0.4\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/nips-microp.pdf}
\caption{{Fine-grained control} results.}
\label{fig:case3}
\end{wrapfigure}
\textbf{Agents perform poorly on tasks that require fine-grained control.}
Fine-grained control is a common requirement in real-world mobile tasks, including dragging a slider to a target position, setting date/time values with pickers, confirming payment via drag gestures, and invoking context menus through long presses.
Compared with standard click-and-type tasks, these operations impose much stricter spatial and temporal constraints: small coordinate errors or incorrect interaction timing are often enough to cause failure.
We curated 11 such tasks in \sys~and summarize model performance in Figure~\ref{fig:case3}.
While this subset is quite easy for humans to finish (full score on all tasks),
the results on agents show a clear weakness: Gemini 3 Pro completes 5/11 tasks (still below 50\%), while the other agents complete only about 0.5 to 1.5 tasks on average.
We attribute this failure mode to two factors.
{(1)}~Insufficient action precision. For drag-and-drop interactions, many predicted coordinates are not accurate enough, leading to repeated unsuccessful attempts to reach the target state.
{(2)}~Limited interaction-strategy knowledge. Control semantics vary across interfaces: some scrollbars respond to taps, others require dragging, and some hidden actions are exposed only by long press.
Without robust UI priors or effective online trial-and-error adaptation, agents struggle to discover and execute the correct control policy.
While our benchmark is built on LLM-synthesized environments and automatically constructed rewards,
and already provides informative signals for agent evaluation, several limitations remain.
{(1)}~Observation space. Since our environments are web-based, they currently expose only
visual observations and interaction affordances available on the page, rather than richer structured
signals (\eg, accessibility trees) that can be accessed on real Android systems.
{(2)}~Cross-app task support. At present, \sys~focuses on single-app workflows and does not yet
cover tasks that require coordinated operations across multiple applications.
However, these constraints should be interpreted as scope boundaries rather than invalidating factors.
They do not undermine the benchmark's practical value, and they do not affect the main empirical
conclusion of this work: current mobile GUI agents still show substantial gaps on realistic,
long-horizon tasks.
More broadly, the field continues to face objective bottlenecks in benchmark construction,
including environment selection, reliable reward design, and evaluation efficiency.
Our synthesis pipeline provides a technically viable foundation to address both limitations above,
and we leave these to future work.
In this paper, we present \sys, a fully synthetic benchmark for evaluating mobile GUI agents.
Built on an LLM-driven generation framework, \sys~automatically constructs backend-free,
interactive mobile environments and task-specific reward validators from natural-language task descriptions.
% This design improves realism, reproducibility, and evaluation efficiency, while reducing deployment and reset overhead.
\sys~contains 63 simulated applications and 120 tasks derived from real-world usage scenarios,
covering 3 specific task types, proved to have a good generalization effect.
Comprehensive experiments on recent mobile GUI agents show substantial headroom for current systems,
especially on long-horizon tasks. Our analysis provide further insights on the weaknesses of
mobile GUI agents on proactive exploration and fine-grained action control.
% Our analysis further indicates that current agents are limited not only by long-horizon state management,
% but also by proactive exploration and fine-grained action control.
% These findings suggest that future mobile GUI agents should prioritize stronger memory mechanisms,
% better online adaptation, and more precise action grounding.
We hope \sys~can serve as a scalable and practical testbed for measuring progress and guiding
next generation of mobile GUI agent research.
\section{\sys~Environment Synthesizing}
\label{appendix:env-synthesizing}
Following the pipeline of Figure~\ref{fig:gen}, we first had the model draft a detailed PRD document
based on the given metadata, then asked it to write code based on the document.
Here follows an example of PRD document when generating simulated application iQIYI, which consists of
many parts that defines the workflow, UI transition logic, data entity, styles, \etc
\begin{tcolorbox}[title=Product Overview,colframe=myframe,breakable,fontupper=\footnotesize]
\textbf{Product Positioning}
\vspace{0.5em} \\
iQIYI is a leading high-quality video entertainment service platform.
Its core positioning is to provide long-form video streaming services (movies, TV series, variety shows, anime),
while gradually integrating short-form video feeds, pan-entertainment communities, and IP-derived consumption
(literature, e-commerce, ticketing).
\vspace{0.5em}
\textbf{Core Value}
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item \emph{Enjoy Quality}: Emphasizes high-definition picture quality, Dolby audio, and exclusive original content.
\item \emph{Immersive Experience}: Enhances viewing focus with dark mode and distraction-free playback design.
\item \emph{VIP Privileges}: Establishes a comprehensive membership tier and benefits system, providing differentiated services.
\end{itemize}
\end{tcolorbox}
\begin{tcolorbox}[title=Design Philosophy \& Style,colframe=myframe,breakable,fontupper=\footnotesize]
\textbf{Overall Style}
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item \textit{Browsing Mode (Light)}: On pages such as ``Home'', ``Discovery'', and ``Profile'', a large area of white and
light gray backgrounds is used. The design pursues breathability, reduces the use of dividing lines, and uses
card shadows and spacing to distinguish content, reducing visual pressure caused by information density.
\item \textit{Immersive Mode (Dark/Gold)}: On the ``Playback'' page and ``VIP Member'' page,
the interface switches to dark or black-gold color schemes. The dark background highlights the video,
creating a cinema-like atmosphere; the black-gold tones reinforce the sense of premium
status and privileges for VIP users.
\end{itemize}
\textbf{Visual Metaphors}
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item \textit{Cards}: Simulate physical posters, carrying content covers.
\item \textit{Flow}: Waterfall flow layout, metaphorically representing an endless supply of content.
\end{itemize}
\end{tcolorbox}
\begin{tcolorbox}[title=Main Interface Architecture,colframe=myframe,breakable,fontupper=\footnotesize]
\textbf{Home - Content Distribution Center}
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item \textit{Layout Logic}: Top search + navigation → Header banner carousel → Double-column waterfall recommendation feed.
\item \textit{Key Elements}:
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item Top Function Area: Contains search box (displaying trending search terms), history entry, and upload/publish entry.
\item Navigation Bar: Horizontally scrollable tabs (Recommendations, TV Series, Movies, Anime, \etc).
\item Content Cards: Uses 16:9 cover images, with the drama title, subtitle (e.g., "All 36 Episodes"), and badges (e.g., "Exclusive," "Free for a Limited Time") displayed at the bottom.
\end{itemize}
\item \textit{Design Purpose}: Maximize content exposure and guide user clicks through algorithmic recommendations.
\end{itemize}
\textbf{Free / Suike (Short Video) - Immersive Short Videos} (...)
\vspace{0.5em} \\
\textbf{VIP (Membership) - Monetization Center} (...)
\vspace{0.5em} \\
\textbf{Discovery - Community \& Topics} (...)
% \vspace{0.5em} \\
% \textbf{Profile - Personal Service Hub} (...)
\end{tcolorbox}
\begin{tcolorbox}[title=Core User Flows,colframe=myframe,breakable,fontupper=\footnotesize]
\textbf{Long-Form Video Watching Flow}
\begin{enumerate}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item \textit{Trigger}: User clicks on a drama/movie cover on the [Home] page.
\item \textit{Transition}: The page slides right to enter the [Playback Details Page].
\item \textit{Loading}:
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item Non-member: Pre-roll ad plays before the video (showing countdown and ``Skip ad with membership'' button).
\item Member: Enters the main video directly, with a prompt saying ``Exclusive VIP, ads have been skipped for you''.
\end{itemize}
\item \textit{Actions}: Single-tap on the video area to evoke the control layer; use gravity sensor or tap the button to switch to [full-screen landscape mode].
\end{enumerate}
\textbf{Membership Subscription Flow} (...)
\vspace{0.5em} \\
\textbf{Short Video Browsing Flow} (...)
\end{tcolorbox}
\begin{tcolorbox}[title=Visual Interface Guidelines,colframe=myframe,breakable,fontupper=\footnotesize]
\textbf{Color Palette}
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item \textit{Primary Brand Color}: iQIYI Green (\#00CC36). Represents vitality and youthfulness. Used for the logo, selected states, and primary action buttons.
\item \textit{Secondary Colors}:
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item VIP Gold (\#E2B376): Used for membership identifiers and paid buttons.
\item Alert Red (\#FF3B30): Used for notification badges and limited-time flash sales.
\end{itemize}
\item \textit{Neutral Colors}:
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item Backgrounds: \#FFFFFF, \#F5F5F5, \#1A1A1A
\item Text: \#333333, \#666666, \#999999
\end{itemize}
\end{itemize}
\textbf{Typography}: Chinese Fonts: ..., English Fonts: ..., Hierarchy Specifications: ...
\vspace{0.5em} \\
\textbf{Iconography \& Components}
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item Icon Style: Primarily line icons, transitioning to solid style with the brand
color when selected. Lines are rounded and approachable.
\item Badges: Located in the top-right or bottom-right corner of the cover, using
semi-transparent backgrounds or brand-colored backgrounds, with extremely small font sizes,
used to convey status.
\item Buttons: Full pill-shaped buttons, emphasizing tactile comfort.
\end{itemize}
\end{tcolorbox}
\begin{tcolorbox}[title=Design Summary,colframe=myframe,breakable,fontupper=\footnotesize]
The design of the iQIYI App successfully balances the tension between massive content distribution and user experience.
\begin{enumerate}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item Clear Information Hierarchy: Through the card-style design featuring ``large images with minimal text'',
users can quickly capture the visual focus while scrolling rapidly.
\item Contextual Design: Strictly distinguishes between the ``browsing for content'' scenario (bright, efficient)
and the ``watching content'' scenario (dark, immersive), aligning with user mental models.
\item Monetization Integration: The VIP membership design is not just a functional entry point but an independent
visual system that effectively stimulates users' desire to pay through color psychology.
\item Ecosystem Loop: Cleverly embeds short videos (Suike) and community (Discovery) into the bottom navigation,
forming a content consumption loop of ``long-form video attracts → community discussion → short-form video kills time''.
\end{enumerate}
\end{tcolorbox}
After writing,
it reviewed the existing codebase, proposed a series of items to be added or modified,
updated the PRD document accordingly, and iterated repeatedly. During iterations, we collect
2 examples of self-reflection outputs of the model to reveal this progress:
\begin{itemize}[leftmargin=*, labelsep=1em, itemindent=0pt]
\item \textit{(Iteration 1)} I have enhanced the `Store' to handle more complex user states (search history, following, daily check-in),
updated the `Router' to handle view cleanup (essential for timers like carousels),
and significantly upgraded the `HomeView', `PlayerView', `SearchView', `ShortView', and `WalletView'
to be fully interactive and logic-driven.
\item \textit{(Iteration 2)} I have added the missing `Collections', `About', and `Feedback' views,
updated the `Player' to handle episodes dynamically, and ensured the `Profile' and `Settings' flows
are fully connected. I also updated the `Store' to seed some initial collection data for a better first-run experience.
\end{itemize}
After 15 iterations, we get the minimal working example of the application, then the data will be further enriched,
and task requirements will be provided to the model to inject to the app as well.
Finally, all provided tasks will be bundled into the environment through three DOM-based API functions,
as we introduced in Section~\ref{subsubsec:method-eval}.
\begin{listing}[H]
\begin{minted}[fontsize=\footnotesize,frame=single]{js}
window.getTasks = function () {
return [
{
taskId: 0,
task: "Search for the TV series The Knockout and add it to your collections.",
params: {
type: "object",
properties: {
videoId: { type: "integer", const: 1 },
},
required: ["videoId"],
},
},
{
taskId: 1,
task: ```Purchase the "The Knockout Official Artbook" from the Mall. During checkout,
create and use a new shipping address with the recipient name 'TestUser'.```,
params: {
type: "object",
properties: {
recipientName: { type: "string", const: "TestUser" },
},
required: ["recipientName"],
},
},
{
taskId: 2,
task: ```Customize the app appearance by changing the theme to Pink and set the
default playback quality to '4K' in Settings.```,
params: {
type: "object",
properties: {
targetTheme: { type: "string", const: "#FF3366" },
targetQuality: { type: "string", const: "4K" },
},
required: ["targetTheme", "targetQuality"],
},
},
{
taskId: 3,
task: ```Go to the Wallet's Points Mall and redeem the 1-Day VIP Pass using
your points.```,
params: {
type: "object",
properties: {
productTitle: { type: "string", const: "1-Day VIP Pass" },
},
required: ["productTitle"],
},
},
];
};
\end{minted}
\caption{Example of function \texttt{window.getTasks}.}
\label{lst:gettasks}
\end{listing}
Listing~\ref{lst:gettasks} shows an example of \texttt{window.getTasks} function in the environment.
This function returns a series of dictionary objects, each containing information about a task supported by each environment,
including the task ID, description, and the JSON schema of the expected return value.
Note that the model tends to generate schemas that only contain `const' fields;
during evaluation, we automatically filter out such fields to avoid confusing the agent being evaluated.
\begin{listing}[H]
\begin{minted}[fontsize=\footnotesize,frame=single]{js}
window.evaluateTask = function (params) {
if (!params || params.taskId === undefined) return { success: false };
const user = window.AppStore.getUser();
switch (params.taskId) {
case 0: {
// Task: Collect video 1
const isCollected = window.AppStore.isCollected(params.videoId);
return { success: isCollected, score: isCollected ? 100 : 0 };
}
case 1: {
// Task: Buy product with specific address name
const orders = user.orders || [];
// Find order for the specific book
const order = orders.find((o) => o.title.includes("The Knockout Official Artbook"));
if (!order) return { success: false, score: 0 };
// Check if address contains the required name
const addressMatch =
order.address && order.address.includes(params.recipientName);
return { success: addressMatch, score: addressMatch ? 100 : 0 };
}
case 2: {
// Task: Theme Pink and Quality 4K
const currentTheme = user.settings.themeColor;
const currentQuality = user.settings.playback
? user.settings.playback.defaultQuality
: "";
const success =
currentTheme === params.targetTheme &&
currentQuality === params.targetQuality;
return { success: success, score: success ? 100 : 0 };
}
case 3: {
// Task: Redeem VIP Card
const transactions = user.transactions || [];
const redeemed = transactions.some((t) =>
t.title.includes(params.productTitle),
);
return { success: redeemed, score: redeemed ? 100 : 0 };
}
}
};
\end{minted}
\caption{Example of function \texttt{window.evaluateTask}.}
\label{lst:evaltask}
\end{listing}
Listing~\ref{lst:evaltask} shows an example of \texttt{window.evaluateTask} function.
For each task, the environment has corresponding low-level code that can directly access its relevant data state;
we use local storage to manage the underlying data and ensure data consistency. This code is very concise and intuitive,
with correct logic, so no additional manually written pattern matching or similar logic is needed to complete the evaluation.
\begin{listing}[H]
\begin{minted}[fontsize=\footnotesize,frame=single]{js}
window.reset = function () {
localStorage.clear();
window.location.reload();
};
\end{minted}
\caption{Example of function \texttt{window.reset}.}
\label{lst:evalreset}
\end{listing}
Listing~\ref{lst:evalreset} shows an example of \texttt{window.reset} function.
Using this function, we can reset various states in the environment, ensuring that state modifications from previous
task executions do not interfere with subsequent task executions. Additionally, this function also resets the
page position, ensuring the page returns to its initial state.
\section{Comprehensive Experimental Results}
\label{appendix:result}
Table~\ref{tab:main-result} and table~\ref{tab:category-scores} displays the full experimental results on \sys.
During evaluation, all GUI agents were tested using the prompts, code, and other resources from their official implementations.
The experiment is run with 8 parallel workers.
We provide our evaluation code in supplemental material with reproduction documentation.
The full benchmark dataset is available at this
\href{https://dataverse.harvard.edu/previewurl.xhtml?token=d9570e08-95f9-46e3-a6a3-1b542e7273e5}{anonymous link}.
\begin{table}[H]
\centering
\caption{Experimental results of different agents on \sys.}
\begin{tabular}{P{8em}C{6.5em}C{6.5em}C{6.5em}C{4.5em}}
\toprule
\textbf{Agent} & \textbf{SR} & \textbf{SR}~\scriptsize{(\emph{w/ returns})} & \textbf{SR}~\scriptsize{(\emph{w/o returns})} & \textbf{SR}~\scriptsize{(\emph{AW})} \\
\midrule
\textbf{\# of Tasks} & 120 & 46 & 74 & 116\\
\midrule
\texttt{seed-1.8} & \underline{42.50}~\color{gray}{(\(\pm\)7.07)} & \underline{30.43}~\color{gray}{(\(\pm\)9.22)} & \underline{50.00}~\color{gray}{(\(\pm\)5.73)} & 70.7 \\
\texttt{Gemini 3 Pro} & 38.75~\color{gray}{(\(\pm\)0.59)} & 28.26~\color{gray}{(\(\pm\)3.07)} & 45.27~\color{gray}{(\(\pm\)0.96)} & > 69.7 \\
\texttt{UI-TARS-1.5} & 29.58~\color{gray}{(\(\pm\)2.95)} & 13.04~\color{gray}{(\(\pm\)0)} & 39.86~\color{gray}{(\(\pm\)4.77)} & 64.2 \\
\texttt{GUI-Owl-1.5-8B} & 9.17~\color{gray}{(\(\pm\)2.35)} & - & 14.86~\color{gray}{(\(\pm\)3.83)} & 71.6 \\
\texttt{MAI-UI-8B} & 19.58~\color{gray}{(\(\pm\)0.59)} & - & 31.76~\color{gray}{(\(\pm\)0.95)} & 70.7 \\
\midrule
\textbf{Average} & 27.92 & 23.91 & 36.35 & 69.38 \\
\midrule
\textbf{Human} & 92.08~\color{gray}{(\(\pm\)0.59)} & 85.87~\color{gray}{(\(\pm\)1.54)} & 95.95~\color{gray}{(\(\pm\)1.92)} & - \\
\bottomrule
\end{tabular}
\label{tab:main-result}
\end{table}
\sys~provides unified interfaces for agent-environment interactions.
The observation channel currently exposes screenshots of the environment state.
The action interface maps model outputs (action type and parameters) to executable simulator operations.
Supported operations include click, swipe, long press, type, clear text, enter, wait, \etc
This design is compatible with most existing mobile GUI agents.
\begin{table}[H]
\centering
\caption{Success rate of different task categories across evaluated agents. We use the first 4 letters
to represent the corresponding category: \emph{simple}~(simp), \emph{long-horizon}~(long), and \emph{math-related}~(math).}
% Given that the base numbers are too small, we don't use percentages in this table, but instead present the results directly in the form of \texttt{x/y}.}
\begin{tabular}{P{8em}C{7.5em}C{7.5em}C{7.5em}}
\toprule
\textbf{Agent} & \textbf{SR}~\scriptsize{(\emph{simp.})} & \textbf{SR}~\scriptsize{(\emph{long.})} & \textbf{SR}~\scriptsize{(\emph{math.})} \\
\midrule
\textbf{\# of Tasks} & 54 & 55 & 11 \\
\midrule
\texttt{seed-1.8} & 56.48~\color{gray}{(\(\pm\)11.7)} & 29.09~\color{gray}{(\(\pm\)2.57)} & 40.91~\color{gray}{(\(\pm\)6.42)} \\
\texttt{Gemini 3 Pro} & 51.85~\color{gray}{(\(\pm\)2.62)} & 24.55~\color{gray}{(\(\pm\)1.28)} & 45.45~\color{gray}{(\(\pm\)12.86)} \\
\texttt{UI-TARS-1.5} & 41.67~\color{gray}{(\(\pm\)3.92)} & 21.82~\color{gray}{(\(\pm\)2.57)} & 9.09~\color{gray}{(\(\pm\)0)} \\
\texttt{GUI-Owl-1.5-8B} & 16.67~\color{gray}{(\(\pm\)0)} & 3.64~\color{gray}{(\(\pm\)5.14)} & 0.00~\color{gray}{(\(\pm\)0)} \\
\texttt{MAI-UI-8B} & 33.33~\color{gray}{(\(\pm\)2.62)} & 10.00~\color{gray}{(\(\pm\)3.86)} & 0.00~\color{gray}{(\(\pm\)0)} \\
\midrule
\textbf{Average} & 40.00 & 17.82 & 19.09 \\
\midrule
\textbf{Human} & 97.22~\color{gray}{(\(\pm\)3.93)} & 89.09~\color{gray}{(\(\pm\)2.57)} & 81.82~\color{gray}{(\(\pm\)0)} \\
\bottomrule
\end{tabular}
\label{tab:category-scores}
\end{table}
To verify that our benchmark effectively simulates real-world app usage scenarios,
we select a total of 20 tasks across 17 apps in a real mobile environment and evaluate
these GUI agents on them. The test results are manually verified to ensure the accuracy of the evaluation.
These tasks still cover the categories of \textit{simple}, \textit{long-horizon}, and \textit{math-related},
as shown in Table~\ref{tab:sample-tasks}.
As demonstrated in the main text of our paper, we found that the performance trends of GUI agents on
these sample tasks are largely consistent with those observed on our benchmark.
This indicates that our benchmark's evaluation of agent capabilities generalizes well
to real-world usage scenarios.
\section{Broader Impacts}
\label{appendix:impact}
The broader impact of \sys~is primarily tied to its role as an evaluation and development
testbed for mobile GUI agents. On the positive side, a fast, reproducible, and controlled
benchmark can help researchers identify failures in long-horizon reasoning, memory, exploration,
and fine-grained control before such agents are deployed in real applications. Because the
environments are synthetic and backend-free, evaluation can be conducted without exposing real
user accounts, private records, payments, messages, or other sensitive app data. This may lower
the barrier to safer and more transparent analysis of mobile-agent capabilities.
At the same time, stronger GUI agents may also enable harmful automation, such as unauthorized
account operations, spam, fraud, scraping, or privacy-invasive workflows, if deployed without
appropriate safeguards. Benchmarks like \sys~could indirectly accelerate these capabilities by
making agent weaknesses easier to diagnose and improve. We therefore encourage future users of
\sys~to treat it as a controlled research benchmark rather than evidence of readiness for
unsupervised real-world deployment, and to pair progress on task success with safeguards such as
permission checks, human confirmation for sensitive actions, rate limits, and auditing.
\begin{table}[H]
\centering
\caption{Real-World Validation Sample Tasks}
\small
\begin{tabular}{P{6em}P{35em}}
\toprule
\textbf{App} & \textbf{Task Description} \\
\midrule
\textbf{RedNote} & Find Lei Jun's homepage and like the second post. \\
\textbf{Douyin} & Search for ``ice cream'' in the group-buying section and add the first item to favorites. \\
\textbf{Weibo} & Like the latest Weibo post by the user ``Cherry''. \\
\textbf{AMap} & Find public transportation routes from the current location to Wangfujing, and start navigation for the route with the shortest travel time. \\
\textbf{Settings} & Without using the search function, open the ``Show pointer location'' setting in Developer Options. \\
\textbf{Wikipedia} & Open the article explaining Europa Universalis V. \\
\textbf{Instagram} & Scroll through Instagram Reels until you find one with over 2M likes, and like it. \\
\textbf{Reddit} & Search for news about the new model from Anthropic and upvote to a relevant post with more than 1k upvotes. \\
\textbf{Google Play} & Check the most relevant ratings and reviews of app ``YouTube Kids''. \\
\textbf{Spotify} & Search for Jay Chow and start playing his album ``Opus 12'' in random order. \\
\textbf{RedNote} & Browse the first 10 posts recommended on the homepage, and record the author, number of likes, and number of comments for the post with the highest number of likes. \\
\textbf{Pinduoduo} & Browse the first 10 items in the ``Billions Subsidies'' channel, and output the store name, price, and number of product reviews. \\
\textbf{Netease Music} & Browse the songs in ``Daily Recommendations'' and output the title, artist, and album of the top three songs with the longest duration. \\
\textbf{Hema Fresh} & In the categories (excluding "Recommended for You"), view the subcategories under each category, and output which category has the most subcategories along with the count. \\
\textbf{Trip} & Search for flights from Beijing to Shanghai, browse the top 5 flights for each of the next 3 days, record the flight with the lowest price for each day, and output the date, departure time, airline, and ticket price. \\
\textbf{Booking} & Search for hotels in Xiamen from May 15th to May 16th at Siming District. View the top 15 hotels and output the ones with free cancellation and over 1000 reviews. \\
\textbf{Tiktok} & Watch 15 videos and list the creator, like count, and comment count for the top 3 most-liked videos. \\
\textbf{Spotify} & Search for Vae and check all his releases. Output the name, number of songs and total time of each album. \\
\textbf{eBay} & Search for iPhone and check the top 20 items. Tell me the names, shop names and maximum prices of Brand New items. \\
\textbf{Google Play} & Check top 15 Top Charts Games, and return the app with the most raters. Tell me its name, rating score, and numbers of downloads. \\
\bottomrule
\end{tabular}
\label{tab:sample-tasks}
\end{table}
\section{User Study}
\label{appendix:user-study}
We conducted a volunteer-based user study to collect realistic mobile-agent task requests.
Participants were asked to describe daily mobile-app tasks they would like an agent to help with,
along with brief background pain points. Participation was voluntary and uncompensated.
The study did not ask participants to provide account credentials, private messages, payment
records, or other sensitive personal data. Collected requests were manually filtered,
normalized, and transformed into synthetic benchmark tasks; no real user accounts or real
personal records are included in \sys.
All examples reported in the paper are anonymized.
The detailed questions are displayed in Table~\ref{tab:user-study-question}.
After the information was collected, we organized the feedback provided by the volunteers and
designed corresponding tasks based on these actual needs, as shown in Table~\ref{tab:user-study-example}.
This helps us to better understands the actual pain points of daily usage on mobile applications,
thus designing more appropriate tasks to evaluate GUI agents on a benchmark which is much closer to
the real world scenario.
\begin{table}[H]
\centering
\caption{Questions we asked for volunteers in our user study.}
\small
\begin{tabular}{P{2em}P{38em}}
\toprule
\textbf{No.} & \textbf{Question} \\
\midrule
1 & What are the main apps you typically use, and what activities do you usually perform with them? \\
2 & While using these apps, are there any aspects you find unsatisfactory or overly cumbersome? \\
3 & Are there any needs in your daily life or work that you wish your smartphone could support but currently cannot? \\
4 & Are there any mobile phone features or usage methods that you know others around you use, which you feel could be helpful to you, but you don't know how to use yourself? \\
5 & Apart from your current smartphone, do you use any other smart devices regularly (e.g., a second phone, computer, tablet, etc.)? What do you use each of these devices for? \\
6 & On a scale of 1 to 9, how would you rate your current smartphone user experience? (1 = ``Completely unable to meet my needs, useless''; 9 = ``Meets all my needs, no need for improvement''). Please explain your reason. \\
\bottomrule
\end{tabular}
\label{tab:user-study-question}
\end{table}
\begin{table}[H]
\centering
\small
\caption{Example of requests from user study and actual tasks included in~\sys.}
\begin{tabular}{P{1.5em}P{4em}P{16em}P{15em}}
\toprule
\textbf{ID} & \textbf{Category} & \textbf{Original Request} & \textbf{Verified Task} \\
\midrule
118 & Cost & Help me record all online expenses, including credit card payment records. & During my history purchases, how much did I spend on shipping in total? \\
14 & Work & Help me check which students in the WeChat group have not submitted their assignments and remind them to complete their work. & Check the ``2025 Student'' chat, identify students who have not completed the ``Completed Assignment Relay'' and send each of them a message. \\
\bottomrule
\end{tabular}
\label{tab:user-study-example}
\end{table}
\section*{NeurIPS Paper Checklist}
%%% BEGIN INSTRUCTIONS %%%
% The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: {\bf The papers not including the checklist will be desk rejected.} The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page
% limit.
% Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
% \begin{itemize}
% \item You should answer \answerYes{}, \answerNo{}, or \answerNA{}.
% \item \answerNA{} means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
% \item Please provide a short (1--2 sentence) justification right after your answer (even for \answerNA).
% % \item {\bf The papers not including the checklist will be desk rejected.}
% \end{itemize}
% {\bf The checklist answers are an integral part of your paper submission.} They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will also be asked to include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
% The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While \answerYes{} is generally preferable to \answerNo{}, it is perfectly acceptable to answer \answerNo{} provided a proper justification is given (e.g., error bars are not reported because it would be too computationally expensive'' or ``we were unable to find the license for the dataset we used''). In general, answering \answerNo{} or \answerNA{} is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer \answerYes{} to a question, in the justification please point to the section(s) where related material for the question can be found.
% IMPORTANT, please:
% \begin{itemize}
% \item {\bf Delete this instruction block, but keep the section heading ``NeurIPS Paper Checklist"},
% \item {\bf Keep the checklist subsection headings, questions/answers and guidelines below.}
% \item {\bf Do not modify the questions and only use the provided macros for your answers}.
% \end{itemize}
%%% END INSTRUCTIONS %%%
\begin{enumerate}
\item {\bf Claims}
\item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We've made it clear in the abstract and introduction about the main idea and core contributions of this paper properly: (1) We develop a scalable LLM-based framework that generates interactive, verifiable mobile app environments and tasks from natural-language descriptions for efficient agent evaluation. (2) Using this framework, we synthesize 63 simulated mobile applications and 120 tasks spanning multiple languages, task formats, and difficulty levels. (3) We benchmark five state-of-the-art mobile GUI agents and show large performance gaps on complex tasks, especially long-horizon tasks, with detailed analysis of failure modes and implications.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the abstract and introduction do not include the claims made in the paper.
\item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A \answerNo{} or \answerNA{} answer to this question will not be perceived well by the reviewers.
\item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
\item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
\end{itemize}
\item {\bf Limitations}
\item[] Question: Does the paper discuss the limitations of the work performed by the authors?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: The paper discusses limitations of the work in Section~\ref{sec:limitations}.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper has no limitation while the answer \answerNo{} means that the paper has limitations, but those are not discussed in the paper.
\item The authors are encouraged to create a separate ``Limitations'' section in their paper.
\item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
\item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
\item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
\item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
\item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
\item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
\end{itemize}
\item {\bf Theory assumptions and proofs}
\item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: This paper does not include formal theoretical results or proofs.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not include theoretical results.
\item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
\item All assumptions should be clearly stated or referenced in the statement of any theorems.
\item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
\item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
\item Theorems and Lemmas that the proof relies upon should be properly referenced.
\end{itemize}
\item {\bf Experimental result reproducibility}
\item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We provide full details on experiment settings such as model usage, repetition, etc. in Section~\ref{sec:experiments}; the code and data is also available in supplemental material.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not include experiments.
\item If the paper includes experiments, a \answerNo{} answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
\item If the contribution is a dataset and\slash or model, the authors should describe the steps taken to make their results reproducible or verifiable.
\item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
\item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
\begin{enumerate}
\item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
\item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
\item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
\item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
\end{enumerate}
\end{itemize}
\item {\bf Open access to data and code}
\item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We plan to publicly release the full dataset and codebase via GitHub upon acceptance of the paper.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that paper does not include experiments requiring code.
\item Please see the NeurIPS code and data submission guidelines (\url{https://neurips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item While we encourage the release of code and data, we understand that this might not be possible, so \answerNo{} is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
\item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://neurips.cc/public/guides/CodeSubmissionPolicy}) for more details.
\item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
\item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
\item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
\item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
\end{itemize}
\item {\bf Experimental setting/details}
\item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We've covered the experiment setting in Section~\ref{subsec:settings} and Appendix~\ref{appendix:result}.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not include experiments.
\item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
\item The full details can be provided either with the code, in appendix, or as supplemental material.
\end{itemize}
\item {\bf Experiment statistical significance}
\item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We report error bars of all experimental results in the paper.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not include experiments.
\item The authors should answer \answerYes{} if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
\item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
\item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
\item The assumptions made should be given (e.g., Normally distributed errors).
\item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
\item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
\item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
\item If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
\end{itemize}
\item {\bf Experiments compute resources}
\item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: Our experiments do not require to have GPU available. We perform all experiments on an Ubuntu 22.04 machine with 24 CPU processors (12th Gen Intel(R) Core(TM) i9-12900K) with 64GB RAM plus 8GB Swap.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not include experiments.
\item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
\item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
\item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
\end{itemize}
\item {\bf Code of ethics}
\item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We carefully reviewed the Code of Ethics and affirm our work fully complies with the principles.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the authors have not reviewed the NeurIPS Code of Ethics.
\item If the authors answer \answerNo, they should explain the special circumstances that require a deviation from the Code of Ethics.
\item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
\end{itemize}
\item {\bf Broader impacts}
\item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We have discussed the impacts in Appendix~\ref{appendix:impact}.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that there is no societal impact of the work performed.
\item If the authors answer \answerNA{} or \answerNo, they should explain why their work has no societal impact or why the paper does not address societal impact.
\item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
\item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
\item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
\item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
\end{itemize}
\item {\bf Safeguards}
\item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
\item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We don't release any datasets or models that could have such risks.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper poses no such risks.
\item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
\item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
\item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
\end{itemize}
\item {\bf Licenses for existing assets}
\item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We use publicly available pretrained models such as API-based models (doubao-seed, UI-TARS, Gemini 3 Pro) as well as local fine-tuned models (GUI-Owl, MAI-UI). All models used are properly cited in our paper, with their licenses and terms followed.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not use existing assets.
\item The authors should cite the original paper that produced the code package or dataset.
\item The authors should state which version of the asset is used and, if possible, include a URL.
\item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
\item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
\item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
\item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
\item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
\end{itemize}
\item {\bf New assets}
\item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We have provided the code and data in supplemental material with comprehensive documentations.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not release new assets.
\item Researchers should communicate the details of the dataset\slash code\slash model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
\item The paper should discuss whether and how consent was obtained from people whose asset is used.
\item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
\end{itemize}
\item {\bf Crowdsourcing and research with human subjects}
\item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We describe the volunteer-based user study protocol, collected fields,
anonymization procedure, and compensation status in Appendix~\ref{appendix:user-study}.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not involve crowdsourcing nor research with human subjects.
\item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
\item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
\end{itemize}
\item {\bf Institutional review board (IRB) approvals or equivalent for research with human subjects}
\item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
\item[] Answer: \answerNo{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: We conducted a volunteer-based, low-risk user study to collect open-ended mobile-agent task requests, but did not obtain formal IRB approval or equivalent institutional
review. Participants voluntarily provided task descriptions and background pain points; we did not collect sensitive personal information, and all reported examples are anonymized and used only
to derive benchmark task scenarios.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the paper does not involve crowdsourcing nor research with human subjects.
\item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
\item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
\item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
\end{itemize}
\item {\bf Declaration of LLM usage}
\item[] Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does \emph{not} impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
%this research?
\item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
\item[] Justification: In our work, we use LLM to generate simulated mobile app environments and validate them; and LLMs are crucial role in our automatic benchmark preparation pipeline as we stated in the paper.
\item[] Guidelines:
\begin{itemize}
\item The answer \answerNA{} means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
\item Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.
\end{itemize}
\end{enumerate}