Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning
Extracted source text · arXiv 2510.27266
\documentclass{article} % For LaTeX2e
\usepackage{iclr2026_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage{wrapfig}
\usepackage{hyperref}
\usepackage{url}
\usepackage{float}
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs}
\usepackage{amsmath}
\usepackage{bbm}
\usepackage{pifont}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{makecell}
\usepackage{amssymb}
\usepackage{xspace}
\usepackage{tcolorbox}
\definecolor{ForestGreen}{RGB}{34,139,34}
\definecolor{BrickRed}{rgb}{.72,0,0}
\definecolor{LakeBlue}{RGB}{0,61,153}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{MiOrange}{RGB}{255,225,204}
\usepackage{paralist}
\usepackage{booktabs}
\hypersetup{colorlinks,allcolors=blue}
\usepackage{cleveref}
\usepackage{tabularx}
% \usepackage[table]{xcolor}
\usepackage{colortbl}
\newcommand{\greencheck}{\textcolor{ForestGreen}{\ding{51}}}
\newcommand{\redcross}{\textcolor{BrickRed}{\ding{55}}}
\definecolor{veronica-red}{RGB}{196,30,58}
\title{HyperClick: Advancing Reliable GUI Grounding via Uncertainty Calibration}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{
\makecell[l]{Shaojie~Zhang$^*$~
Pei~Fu$^*$~
Ruoceng~Zhang~
Jiahui~Yang~
Anan~Du~
Xiuwen~Xi~ \\
Shaokang~Wang~
Ying~Huang~
Bin~Qin~
Zhenbo~Luo$^\dagger$~
Jian~Luan} \\
~~ \\
MiLM Plus, Xiaomi Inc\\
\texttt{\{zhangshaojie5, fupei1, luozhenbo, luanjian\}@xiaomi.com}
\vspace{-6pt} \\
}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
{\let
\thefootnote \relax
\footnote{$^*$ Equal contribution; $\dagger$ Corresponding author.} \\
\footnote{$^{\S}$ https://github.com/xiaomi-research/hyperclick}
}
\maketitle
\begin{abstract}
Autonomous Graphical User Interface (GUI) agents rely on accurate GUI grounding, which maps language instructions to on-screen coordinates, to execute user commands. However, current models, whether trained via supervised fine-tuning (SFT) or reinforcement fine-tuning (RFT), lack self-awareness of their capability boundaries, leading to overconfidence and unreliable predictions. We first systematically evaluate probabilistic and verbalized confidence in general and GUI-specific models, revealing a misalignment between confidence and actual accuracy, which is particularly critical in dynamic GUI automation tasks, where single errors can cause task failure. To address this, we propose HyperClick, a novel framework that enhances reliable GUI grounding through uncertainty calibration. HyperClick introduces a dual reward mechanism, combining a binary reward for correct actions with a truncated Gaussian–based spatial confidence modeling, calibrated using the Brier score. This approach jointly optimizes grounding accuracy and confidence reliability, fostering introspective self-criticism. Extensive experiments on seven challenge benchmarks show that HyperClick achieves state-of-the-art performance while providing well-calibrated confidence. By enabling explicit confidence calibration and introspective self-criticism, HyperClick reduces overconfidence and supports more reliable GUI automation.
\end{abstract}
\begin{figure}[!htbp]
\centering
\includegraphics[width=1.0\linewidth]{figs/Motivation.pdf}
\vspace{-1em}
\caption{Overview of accuracy and confidence evaluation on ScreenSpot-Pro. (a): Illustration of probabilistic and verbalized confidence. Probabilistic confidence represents the probability of the model generating the next token corresponding to the target coordinates, while verbalized confidence indicates the model’s self-reported certainty about its output in natural language. (b): Comparisons of accuracy, probabilistic confidence, and verbalized confidence for several general-purpose and GUI-specific models on the ScreenSpot-Pro benchmark. The models exhibit a higher confidence in their answers than in the accuracy that they actually achieve.}
\label{fig: motivation}
\end{figure}
\input{sec/1_introduction}
\input{sec/2_related_work}
\input{sec/3_method}
\input{sec/4_experiments}
\input{sec/5_conclusion}
\section*{Ethics and Reproducibility statement}
This research focuses on building a policy model for reliable GUI grounding. The data used are obtained by synthesizing or reprocessing previously released datasets, with all datasets or benchmarks properly cited. In this paper, there are no discrimination, bias, or fairness issues that need to be addressed. In addition, our models are not expected to generate potentially harmful content. To ensure reproducibility, we provide all experimental and data details in Section \ref{sec: experiments} and the corresponding appendices. We will release the source code and model checkpoints to support
reproducibility.
\bibliography{ref}
\bibliographystyle{iclr2026_conference}
\input{sec/6_appendix}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
% ==========================================================================================
\section{Introduction}
The revolution of autonomous Graphical User Interface (GUI) agents is transforming human-computer interaction, enabling users to control mobile applications, web platforms, and complex desktop software directly through natural language instructions~\citep{wang2024gui,nguyen2024gui}. At the heart of these agents lies the GUI grounding, the ability to accurately map textual commands to precise pixel coordinates on user interface elements~\citep{cheng2024seeclick,tang2025gui}. This fundamental task determines whether an agent can successfully execute user commands, making it the cornerstone of reliable GUI automation.
Recent progress in GUI grounding has been driven by supervised fine-tuning (SFT) with curated large-scale datasets~\citep{wu2024atlas,gou2025uground,xu2024aguvis} and reinforcement fine-tuning (RFT) with verifiable GUI-specific rewards~\citep{lu2025ui,luo2025gui,liu2025infigui}. Although these techniques yield strong performance, they share a critical weakness: the lack of self-awareness of their capability boundary, making it difficult to judge when predictions are reliable.
A reliable GUI agent should be aware of its limitations and accurately distinguish between what it can and cannot do~\citep{ding2025lvlms}. While this ability has been extensively studied in large language models (LLMs)~\citep{xiong2023can,tian2023just}, it remains underexplored in GUI agents. The reliability level of an agent is assessed by the alignment between its confidence and actual performance~\citep{ding2025lvlms}. In this paper, we first evaluate probabilistic and verbalized confidence for several general models~\citep{openai2024hello,bai2025qwen2,guo2025seed1,team2025kimi,coreteam2025mimovltechnicalreport} and GUI-specific models~\citep{qin2025ui} on the ScreenSpot-Pro benchmark~\citep{li2025screenspot}, which emphasizes high-resolution displays, smaller target sizes, and complex environments. Specifically, probabilistic confidence reflects token-level likelihoods for predicted coordinates~\citep{guo2017calibration,desai2020calibration}, while verbalized confidence captures self-reported certainty in natural language~\citep{lin2022teaching,yang2024alignment}.
As shown in Figure~\ref{fig: motivation}, the models exhibit a higher confidence in their answers than in the accuracy that they actually achieve. In other words, even on challenging tasks, these agents remain overconfident in their predictions both probabilistically and from a self-assessed perspective. We argue that this is analogous to the hallucination problem commonly observed in LLMs and vision-language models (VLMs), where the model produces fluent, yet factually erroneous outputs while maintaining high confidence~\citep{ji2023survey,ji2023towards,kalai2025language}. This limitation is particularly critical in real-world GUI tasks, where their dynamic and continuous nature means that even a single error in an intermediate step can result in overall task failure.
To address this limitation, we propose HyperClick, a novel framework that enhances reliable GUI grounding through uncertainty calibration. Unlike prior approaches that treat grounding as a pure hit-or-miss classification problem, HyperClick explicitly integrates verbalized confidence estimation into the grounding process. Each prediction consists not only of a selected UI element, but also of a natural-language confidence statement, providing a self-assessment of reliability.
Specifically, we introduce two complementary rule-based reward mechanisms that optimize both action accuracy and uncertainty calibration. A binary reward enforces correct grounding actions, while a truncated Gaussian–based distribution models spatial confidence over the entire screenshot. The predicted confidence is then calibrated against this distribution using the Brier score~\citep{glenn1950verification,damani2025beyond}. This dual mechanism enables HyperClick to achieve two intertwined goals: accurate GUI grounding and well-calibrated confidence. More importantly, it fosters a form of introspectiveness, where the model not only acts but also critiques its own reliability. This self-criticism capacity reduces overconfidence, supports safer decision-making, and gradually expands the agent’s boundaries of reliable operation.
Our contributions are summarized as follows:
\begin{itemize}
\item We systematically reveal that existing GUI grounding models are prone to overconfidence, analogous to hallucinations in LLMs and VLMs, and highlight their critical implications for reliable GUI automation.
\item We propose HyperClick, the first GUI grounding framework that explicitly integrates uncertainty calibration, introducing a dual reward mechanism that jointly optimizes grounding accuracy and confidence reliability via binary correctness and truncated Gaussian–based confidence modeling.
\item Through extensive evaluations on challenging GUI grounding benchmarks, HyperClick not only achieves state-of-the-art (SOTA) accuracy but also establishes well-calibrated confidence, enabling introspective self-criticism and more reliable GUI agents.
\end{itemize}
% ==========================================================================================
\section{Related Work}
% ======================================================================
\subsection{GUI Agents and Grounding}
GUI agents, as autonomous intelligent systems specialized in interacting with graphical user interfaces, have emerged as a key technology to automate complex desktop and mobile tasks~\citep{wang2024gui,nguyen2024gui,zhang2024large}. Recently, VLM-based GUI agents~\citep{cheng2024seeclick,wu2024atlas,qin2025ui} have demonstrated strong GUI comprehension by integrating visual perception with language reasoning, allowing them to handle diverse interface styles across applications. At the heart of VLM-based GUI agents lies the task of GUI grounding, which bridges natural language instructions with precise interface elements, thereby underpinning reliable GUI automation.
Early works~\citep{cheng2024seeclick, lin2025showui, yang2024aria} primarily focused on acquiring GUI-specific capabilities by collecting large-scale GUI corpora for SFT, thereby developing models customized for GUI tasks. SeeClick~\citep{cheng2024seeclick} first introduced VLM to complete GUI tasks with only visual inputs. OS-Atlas~\citep{wu2024atlas}, UGround~\citep{gou2025uground}, and Aguvis~\citep{xu2024aguvis} aim to enhance perception by fine-tuning pre-trained models on a dataset constructed from diverse environments. UI-TARS~\citep{qin2025ui} develops a native end-to-end GUI agent through large-scale GUI screenshots to enhance perception and reasoning for unified action modeling across platforms.
With the success of DeepSeek-R1-Zero~\citep{guo2025deepseek}, RFT has drawn increased attention in the GUI-specific domain. UI-R1~\citep{lu2025ui}, GUI-R1~\citep{luo2025gui}, InfiGUI-R1~\citep{liu2025infigui}, and BTL-UI~\citep{zhang2025btl} naively replicate techniques from DeepSeek-R1, prompting the model to think before generating an answer and optimizing the policy model with Verifiable GUI-specific reward functions. However, these native R1-based GUI agents overlook an important insight: Chain-of-Thought (CoT) reasoning degrades performance in GUI grounding, where precise spatial perception matters more than deep reasoning. Subsequently, GUI-G1~\citep{zhou2025gui} revisits the limitations of current R1-based GUI agents by introducing controllable box-size rewards for grounding tasks. SE-GUI~\citep{yuan2025enhancing} proposes self-evolution approaches and continuous rewards to guide model learning. GUI-G$^2$~\citep{tang2025gui} further introduced Gaussian reward modeling for GUI grounding. However, existing GUI grounding approaches primarily focus on improving grounding accuracy, while largely overlooking the importance of confidence calibration.
% ======================================================================
\subsection{Uncertainty Calibration}
The concept of uncertainty originates from the error analysis theory, where it quantifies the degree of confidence associated with a measurement~\citep{oberkampf2002error}. This notion has been widely adopted in computer vision tasks such as object detection~\citep{ren2015faster,redmon2016you} and semantic segmentation~\citep{long2015fully,he2017mask}, helping to assess the reliability of model predictions. With the rise of large language models (LLMs) and vision-language models (VLMs), several representative types of confidence signals have been proposed to capture the uncertainty of generated natural language: \textbf{(1) Probabilistic confidence}~\citep{guo2017calibration,desai2020calibration}, which uses token generation probabilities as a measure of uncertainty; \textbf{(2) Answer consistency confidence}~\citep{zhang2023sac3,manakul2023selfcheckgpt,fu2025deep}, which quantifies uncertainty based on semantic consistency between multiple model outputs rather than token-level probabilities; and \textbf{(3) Verbalized confidence}~\citep{lin2022teaching,yang2024alignment}, where the model explicitly reports its confidence in natural language, providing an intuitive model-agnostic signal without requiring repeated sampling. Building on these advances, uncertainty estimation has been shown to improve the robustness and reliability of neural network systems by providing calibrated confidence for downstream decision-making.
\vspace{-0.5em}
% ==========================================================================================
\section{Method}
\begin{figure}[htbp]
\centering
\includegraphics[width=1.0\linewidth]{figs/Framework.pdf}
\caption{Framework of the proposed HyperClick, optimized with Group Relative Policy Optimization (GRPO). Given a screenshot and an instruction, the policy generates $N$ predictions, which are evaluated by a verifiable reward mechanism. The correctness reward measures grounding precision, while the calibration reward assesses uncertainty. For clarity, the reference model is omitted.}
\label{fig: Framework}
\end{figure}
% ======================================================================
\subsection{Problem Formulation}
GUI grounding can be formalized as the problem of mapping a natural language instruction to spatial coordinates corresponding to the target UI element on a given screenspot. From the perspective of policy optimization, this task can be instantiated in two ways: location formulation~\citep{wu2024atlas,tang2025gui} and click formulation~\citep{xu2024aguvis,luo2025gui,yuan2025enhancing}.
\begin{itemize}
\item Location formulation: Given a screenshot $s$ and an instruction $q$, the policy model is optimized by predicting the bounding box $\hat{\mathbf{b}} = (\hat{x}_1, \hat{y}_1, \hat{x}_2, \hat{y}_2)$, where $(\hat{x}_1, \hat{y}_1)$ and $(\hat{x}_2, \hat{y}_2)$ denote the top-left and bottom-right corners of the UI element referred to by $q$.
\item Click formulation: Alternatively, the policy model predicts a single point $\hat{\mathbf{p}} = (\hat{x}, \hat{y})$, corresponding to the center of the target element, which directly simulates a clicking action.
\end{itemize}
In this work, we adopt the click formulation as our primary paradigm, as it naturally aligns with executable actions in GUI interaction, simplifies the action space compared to bounding-box prediction, and provides a direct objective for reinforcement learning.
% ======================================================================
\subsection{Confidence Modeling}
\label{confidence_modeling}
Building on the introduction of the Gaussian distribution in error analysis theory~\citep{gauss1809theoria,gauss1877theoria,mackenzie1988history} and recent advances in GUI-G$^2$~\citep{tang2025gui}, we model the confidence distribution in GUI grounding using a Gaussian formulation. Furthermore, as shown in Figure~\ref{fig: Framework}, since most UI element annotations are represented as bounding boxes, and to jointly account for correctness and confidence, we adopt a truncated Gaussian distribution~\citep{galli1994pros} to model confidence.
\textbf{Truncated Gaussian Representation.} For each UI element with bounding box $\mathbf{b}=(x_1,y_1,x_2,y_2)$, the 2D Gaussian distribution on the screenshot interface can be denoted as:
\begin{equation}
\mathcal{N}(\mathbf{x};\bm{\mu},\Sigma) = \frac{1}{2\pi|\Sigma|^{\frac{1}{2}}}\text{exp}(-\frac{1}{2}(\mathbf{x}-\bm{\mu})^{\text{T}} \Sigma^{-1}(\mathbf{x}-\bm{\mu})),
\end{equation}
where $\mathbf{x}$ means any point on the 2D interface, $\bm{\mu}=(\mu_x,\mu_y)=(\frac{x_1+x_2}{2}, \frac{y_1+y_2}{2})$ represents the center point of the UI element, and $\Sigma=\begin{pmatrix} \sigma_x^2 & 0 \\ 0 & \sigma_y^2\end{pmatrix}$ is the covariance matrix. The diagonal structure assumes independence between the dimensions $x$ and $y$, simplifying the computation while maintaining expressiveness.
We preliminarily formulate the uncertainty distribution based on the constructed 2D Gaussian distribution on the interface. For each prediction point $\hat{\mathbf{p}}$ on the screenshot, the confidence value can be computed:
\begin{equation}
\mathcal{C}(\hat{\mathbf{p}}) = 2\pi|\Sigma|^{\frac{1}{2}} \cdot \mathcal{N}(\hat{\mathbf{p}};\bm{\mu};\Sigma) = \text{exp}(-\frac{1}{2}[\frac{(\hat{x}-\mu_x)^2}{\sigma_x^2} + \frac{(\hat{y}-\mu_y)^2}{\sigma_y^2}]).
\end{equation}
For the center point $(\mu_x,\mu_y)$ of the grounding truth bounding box $\mathbf{b}$, the constructed value naturally reaches its maximum value of 1. This means that when the policy model predicts the point $(\mu_x,\mu_y)$, the model should have the highest confidence in its response.
Furthermore, we truncate the constructed confidence distribution by restricting it to the region defined by the bounding box $\mathbf{b}$, which aligns with the discriminative nature of the task. Specifically, confidence is assigned only when the predicted point $\hat{\mathbf{p}}$ is within $\mathbf{b}$; otherwise, the confidence is set to zero. In summary, the confidence distribution is modeled as a truncated Gaussian:
\begin{equation}
\overline{\mathcal{C}}(\hat{\mathbf{p}}) =
\begin{cases}
\mathcal{C}(\hat{\mathbf{p}}), & (x_1 < \hat{x} < x_2) \land (y_1 < \hat{y} < y_2), \\
0, & \text{otherwise}.
\end{cases}
\end{equation}
\textbf{Adaptive Variance.}
Previous approaches~\citep{zhou2025gui,tang2025gui} have highlighted the difficulty bias in GUI grounding, where target elements with a smaller relative box size on the screenshot are more challenging. To handle UI elements with a wide range of sizes, we adopt the adaptive variance mechanism to control the confidence distribution on various platforms and screenshots:
\begin{equation}
\sigma_x = \alpha \cdot (x_2 - x_1), \quad \sigma_y = \alpha \cdot (y_2 - y_1),
\end{equation}
which $\alpha$ is a scaling factor that controls the relative influence of the element size on the standard deviations.
\subsection{Training Objective}
\textbf{Correctness Reward.}
As shown in Figure~\ref{fig: Framework}, we adopt the binary reward mechanism to guide the prediction point of the policy model $\hat{\mathbf{p}}$ within the bounding box $\mathbf{b}$. This discrete supervision directly aligns the policy objective with the success or failure of the grounding. Therefore, the correctness reward is expressed as follows:
\begin{equation}
R_{\text{correctness}} = \mathbbm{1}_{\hat{\mathbf{p}} \in \mathbf{b}} =
\begin{cases}
1, & (x_1 < \hat{x} < x_2) \land (y_1 < \hat{y} < y_2), \\
0, & \text{otherwise}.
\end{cases}
\end{equation}
\textbf{Confidence Reward.}
The purpose of the confidence reward is to encourage the policy model to evaluate and criticize the prediction generated $\hat{\mathbf{p}}$, making the confidence in the model output more precise. Thus, the confidence $\hat{c}$ of the model output should be aligned with the confidence distribution constructed in section~\ref{confidence_modeling}. To achieve this, we introduce the Brier score~\citep{glenn1950verification} to build the reward function, which can be thought of as a measure of the calibration of a set of probabilistic forecasts. The confidence reward can be formulated as follows.
\begin{equation}
R_{\text{confidence}} = 1 - (\hat{c} - \overline{\mathcal{C}}(\hat{x}, \hat{y}))^2 .
\end{equation}
This formulation provides several key properties. First, the closer the prediction confidence of the policy model $\hat{c}$ to the value corresponding to the constructed confidence distribution, the model will receive more reward. Second, when the model's prediction is incorrect and has a low confidence value for its generation, the policy model can still obtain a high confidence reward, which aligns with the model's motivation to self-criticize through confidence.
In summary, the final reward signal for the policy model combines a format reward $R_{\text{format}}$ with the correctness reward $R_{\text{correctness}}$ and the confidence reward $R_{\text{confidence}}$. The total reward is thus:
\begin{equation}
R = R_{\text{format}} + R_{\text{correctness}} + R_{\text{confidence}}.
\end{equation}
We optimize HyperClick with Group Relative Policy Optimization (GRPO)~\citep{shao2024deepseekmath}, which extends the idea of relative advantage estimation to a group of predictions. Unlike standard policy gradient methods that rely on a single sampled return, GRPO leverages multiple candidate outputs to construct a relative reward signal, leading to more stable and informative optimization. Given $N$ generations $\{o_i\}_{i=1}^N$, each is evaluated by the reward function $R$. GRPO normalizes these rewards within the group to obtain relative advantages:
\begin{equation}
A _i = \frac{R(o_i) - \text{mean}(\{R(o_j)\}_{j=1}^N)}{\text{std}(\{R(o_j)\}_{j=1}^N)}
\end{equation}
The training objective of GRPO is then defined as
\begin{equation}
\begin{aligned}
\mathcal{J}(\theta) & = \mathbb{E}_{\{o_{i}\}_{i=1}^{N} \sim \pi_{\theta_{\text{old}}}(\cdot | s,q)}\\
& \frac{1}{N} \sum_{i=1}^{N} \left\{
\text{min}\left[
\frac{\pi_{\theta}(o_i|s,q)}{\pi_{\theta_{\text{old}}}(o_i|s,q)}A_i,
\text{clip}\left( \frac{\pi_{\theta}(o_i|s,q)}{\pi_{\theta_{\text{old}}}(o_i|s,q)}, 1-\epsilon, 1+\epsilon \right)A_i
\right]
- \beta \cdot \text{KL}(\pi_{\theta}||\pi_{\text{ref}})
\right\},
\end{aligned}
\end{equation}
where $\pi_{\theta}$ denotes the policy model parameterized by $\theta$, $\epsilon$ is a hyperparameter that controls $\text{clip}(\cdot,1-\epsilon,1+\epsilon)$ and $\beta$ weights the KL regularization~\citep{schulman2017proximal,shao2024deepseekmath} to stabilize training.
% ==========================================================================================
\section{Experiments}
\label{sec: experiments}
\input{tab/main}
% ======================================================================
\subsection{Implementation Details}
We implement HyperClick on top of Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct. Training data is sampled from multiple public GUI datasets, including OS-Atlas~\citep{wu2024atlas}, Widget Caption~\citep{li2020widget}, UI-Refexp~\citep{bai2021uibert}, and OmniAct~\citep{kapoor2024omniact}, resulting in approximately 30K samples. Model training is conducted within the VLM-R1~\citep {shen2025vlm} codebase. We train for one epoch on 16 NVIDIA H100 GPUs, using a learning rate linearly decayed from 1e-6 to 0 with a cosine scheduler, a global batch size of 16, 8 generations per instance, and a KL constraint coefficient of $\beta=0.04$. To improve efficiency, we leverage FlashAttention-2~\citep{dao2023flashattention}, adopt bfloat16 precision, and enable gradient checkpointing. During inference, the temperature is fixed to 0 to ensure reproducibility.
% ======================================================================
\subsection{Evaluation Benchmarks}
We comprehensively evaluate the GUI grounding capability of HyperClick on ScreenSpot~\citep{cheng2024seeclick} (SS), ScreenSpot-V2~\citep{wu2024atlas} (SS2), ScreenSpot-Pro~\citep{li2025screenspot} (SSP), MMBench-GUI~\citep{wang2025mmbench} (MMG), UI-I2E-Bench~\citep{liu2025ui} (I2E), CAGUI~\citep{zhang2025agentcpm} (CAG) and UI-Vision~\citep{nayak2025ui} (UIV). More details about each evaluation benchmark are described in the Appendix.
% ======================================================================
\subsection{Main Results}
\textbf{Comparisons with Baselines.}
The main experimental results of HyperClick and comparisons with general models and GUI-specific models are shown in Table~\ref{tab: main}. HyperClick achieves consistently strong performance across all benchmarks. In particular, HyperClick-7B reaches new SOTA results in SS2 (93.7), SSP (48.2), MMG (79.6), I2E (76.5), CAG (82.9), and UIV (25.7), surpassing previous RFT-based approaches such as GUI-G$^2$~\citep{tang2025gui} and SE-GUI~\citep{yuan2025enhancing}. In ScreenSpot (SS), HyperClick-7B obtains 91.5, which is highly competitive and comparable to the best results (92.0) of GUI-G$^2$. Moreover, HyperClick also demonstrates strong performance, outperforming much larger GUI-specific models, such as UI-TARS-72B~\citep{wu2024atlas} and Aguvis-72B~\citep{xu2024aguvis}.
A key source of HyperClick's improvement lies in the introduction of uncertainty calibration, which equips the model with a self-criticism mechanism. Unlike GUI grounding models that rely solely on sparse binary~\citep{lu2025ui,luo2025gui} or continuous~\citep{yuan2025enhancing,tang2025gui} correctness rewards, HyperClick leverages a calibrated confidence distribution to explicitly distinguish between reliable and uncertain predictions. This enables the policy to penalize overconfident errors while reinforcing well-calibrated clicks. As shown in Table~\ref{tab: main}, such self-criticism translates into consistent gains across benchmarks, highlighting that calibrated confidence improves the model’s ability to generalize across diverse UI environments. These results confirm that confidence-aware grounding not only enhances accuracy but also makes the model more robust to task difficulty and annotation variability.
\textbf{The confidence of HyperClick is reliable.}
To evaluate whether HyperClick is truly reliable, we introduce the average precision (AP) of object detection~\citep{lin2014microsoft}, which adopts $\texttt{confidence} \in \{0.5, 0.75, 0.9, 0.95\}$ as the boundary positive for counting positive and negative samples. As shown in Table~\ref{tab: ap}, HyperClick consistently maintains high AP across all thresholds, and as the confidence threshold increases, the AP also gradually increases, which indicates that the model not only makes accurate predictions but also assigns well-calibrated confidence scores, rather than overestimating or underestimating its certainty. Furthermore, compared to baseline models, HyperClick shows a clear margin of improvement, particularly in the high-confidence regime (\texttt{AP$^{\text{conf}=90}$} and \texttt{AP$^{\text{conf}=95}$}). This suggests that HyperClick is capable of self-criticizing its predictions: when the model outputs a high confidence score, the prediction is highly reliable; when the score is low, it effectively signals uncertainty. Such behavior is crucial for practical deployment in GUI automation, where wrong but overconfident predictions may lead to catastrophic task failures.
\input{tab/AP}
% ======================================================================
\subsection{Ablation Study}
\input{tab/Ablation}
We conducted an ablation study on ScreenSpot-Pro to verify the effectiveness of key components of HyperClick.
\textbf{Reward Mechanism.}
The results in Table~\ref{tab: ablation_reward} demonstrate the importance of combining correctness and confidence rewards. Using only the format or correctness reward yields relatively limited improvements (47.5\% and 47.7\%, respectively). Introducing the confidence reward alone already achieves stronger performance (48.0\%), while the combination of correctness and confidence rewards further increases the precision to 48.2\%. This validates our motivation that confidence calibration acts as a self-critical signal, discouraging overconfident errors and reinforcing reliable predictions.
\textbf{Confidence Modeling.}
Table~\ref{tab: ablation_confidence}, investigates the effect of the adaptive variance factor $\alpha$, Without confidence modeling based on the truncated Gaussian distribution ($\alpha$=0), which means only binary confidence is used for uncertainty calibration. Therefore, when $\alpha$=0, the confidence reward is represented as:
\begin{equation}
R_{\text{confidence}} = 1 - (\hat{c}-\mathbbm{1}_{\hat{\mathbf{p} \in \mathbf{b}}})^2,
\end{equation}
the policy model reaches 47.7\%, which is weaker than the truncated Gaussian variants. Moreover, we set $\alpha$ according to the principle 3$\sigma$ of the Gaussian distribution. Take the x direction as an example, $k\cdot\sigma_x=\frac{1}{2}(x_2-x_1)$, where $k \in \{1,2,3\}$ and subtract the scaling factor $\alpha \in \{\frac{1}{2}, \frac{1}{4}, \frac{1}{6}\}$. As shown in the results, while too large ($\alpha=\tfrac{1}{2}$) or too small ($\alpha=\tfrac{1}{6}$) variances lead to suboptimal performance. Specifically, when $\alpha=\tfrac{1}{2}$, the variance is too large and the Gaussian distribution is excessively truncated within the bounding box. As a result, the confidence mass is overly concentrated near the center, which weakens the model’s sensitivity to the boundary regions of the element. In contrast, when $\alpha=\tfrac{1}{6}$, the variance is too small, leading to a distribution that is too truncated. Consequently, the confidence at the edge of the bounding box is nearly zero, making the calibration too strict and reducing the tolerance to minor prediction deviations.
For comparison, $\alpha=\tfrac{1}{4}$ provides a balanced trade-off between concentration and spread, providing the most effective uncertainty modeling and the highest precision (48.2\%).
\textbf{Extension to other baselines.}
As shown in Table~\ref{tab: ablation_baseline}, we further extend HyperClick to MiMo-VL~\citep{coreteam2025mimovltechnicalreport}, a strong general-purpose VLM. With our training framework, MiMo-VL improves from 39.9\% to 49.5\%, demonstrating that HyperClick serves as a plug-and-play training paradigm for GUI grounding. Similarly, applying HyperClick to Qwen2.5-VL also brings substantial improvement (from 26.8\% to 48.2\%), confirming the generality and scalability of our approach across different foundation models.
% ======================================================================
\subsection{Visualization}
\begin{figure}[!t]
\centering
\includegraphics[width=0.99\linewidth]{figs/Visualization.pdf}
\caption{Visualization of the confidence distribution output by HyperClick. We inject the coordinates on the interface into the assistant's generation and enforce it to continue to output the confidence for the click position. The darker the color, the higher the confidence value.}
\label{fig: visualization}
\end{figure}
To better understand the effect of uncertainty calibration, we visualize the confidence distributions predicted by HyperClick in Figure~\ref{fig: visualization}. For each instruction, we inject the coordinates on the interface into the assistant's generation and enforce the policy model, continuing to output the click position. Thus, the heatmap represents its confidence in the possible click positions on the interface. We observe that the confidence is sharply concentrated around the ground-truth elements, while irrelevant regions exhibit low or near-zero confidence. This aligns with our design of truncated Gaussian modeling, where confidence only exists inside valid bounding boxes. Moreover, the adaptive variance mechanism adjusts the spread of the confidence distribution according to the element size: smaller UI elements yield tighter confidence peaks, whereas larger ones result in more diffuse heatmaps. These visualizations intuitively demonstrate how HyperClick achieves reliable and robust GUI grounding by avoiding overconfident but incorrect clicks.
\begin{table}[!t]
\centering
\label{tab: main}
\caption{GUI grounding accuracy on seven benchmarks including {ScreenSpot}~\citep{cheng2024seeclick} (SS), {ScreenSpot-V2}~\citep{wu2024atlas} (SS2), {ScreenSpot-Pro}~\citep{li2025screenspot} (SSP), {MMBench-GUI}~\citep{wang2025mmbench} (MMG), {UI-I2E-Bench}~\citep{liu2025ui} (I2E), {CAGUI}~\citep{zhang2025agentcpm} (CAG) and {UI-Vision}~\citep{nayak2025ui} (UIV). \textbf{Bold} and \underline{underline} indicate the best and second-best results. \textbf{The detailed experimental results on each benchmark are in the appendix}.}
\resizebox{\linewidth}{!}
{
\begin{tabular}{lcccccccc}
\toprule
% \textbf{Model} & \textbf{Size} & \textbf{ScreenSpot} & \textbf{ScreenSpot-V2} & \textbf{ScreenSpot-Pro} & \textbf{MMBench-GUI} & \textbf{UI-I2E-Bench} & \textbf{CAGUI} & \textbf{UI-Vision} \\
\textbf{Model} & \textbf{Size} & \textbf{SS} & \textbf{SS2} & \textbf{SSP} & \textbf{MMG} & \textbf{I2E} & \textbf{CAG} & \textbf{UIV} \\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{General Models}}} \\
GPT-4o~\citep{openai2024hello}
& - & 18.8 & 20.1 & 0.8 & 2.9 & - & 21.0 & 1.4 \\
Claude~\citep{anthropic2024claude}
& - & 83.0 & - & 17.1 & 4.7 & - & - & 8.3 \\
Qwen2-VL~\citep{wang2024qwen2}
& 7B & 42.9 & - & - & - & 48.7 & - & 2.7 \\
\multirow{2}{*}{Qwen2.5-VL~\citep{bai2025qwen2}}
& 3B & 55.5 & 80.9 & 16.1 & - & 41.7 & - & - \\
& 7B & 84.7 & 88.8 & 26.8 & 33.9 & 53.8 & 59.6 & 0.9 \\
\multirow{2}{*}{Intern3VL~\citep{zhu2025internvl3}}
& 8B & 79.5 & 81.4 & - & - & - & - & - \\
& 38B & 85.6 & 88.3 & - & - & - & - & - \\
MiMo-VL~\citep{coreteam2025mimovltechnicalreport}
& 7B & 87.2 & 90.5 & 41.9 & - & - & - & - \\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (SFT)}}} \\
CogAgent~\citep{hong2024cogagent} &
18B & 47.4 & - & 7.7 & - & - & - & 8.9 \\
SeeClick~\citep{cheng2024seeclick}
& 9.6B & 53.4 & 55.1 & 1.1 & - & 26.4 & - & 5.4 \\
Aria-UI~\citep{yang2024aria}
& 25.3B & 82.4 & - & 11.3 & - & - & - & 10.1 \\
ShowUI~\citep{lin2025showui}
& 2B & 75.1 & 77.3 & 7.7 & 16.0 & 41.5 & - & 5.9 \\
UGround~\citep{gou2025uground}
& 7B & 73.3 & - & 16.5 & - & 16.5 & - & 8.8 \\
\multirow{2}{*}{UGround-V1~\citep{gou2025uground}}
& 2B & 77.7 & - & - & - & 57.4 & - & 12.9 \\
& 7B & 86.3 & - & 31.1 & 65.7 & 70.3 & - & 23.2 \\
\multirow{2}{*}{OS-Atlas~\citep{wu2024atlas}}
& 4B & 70.1 & 71.9 & 3.7 & - & 44.3 & - & - \\
& 7B & 82.5 & 84.1 & 18.9 & 41.4 & 58.6 & 57.2 & 9.0 \\
\multirow{2}{*}{Aguvis~\citep{xu2024aguvis}}
& 7B & 84.4 & - & - & 45.7 & 53.2 & 68.7 & 13.7 \\
& 72B & 89.2 & - & - & - & - & - & - \\
\multirow{3}{*}{UI-TARS~\citep{qin2025ui}}
& 2B & 82.3 & 84.7 & 27.7 & - & 27.7 & - & - \\
& 7B & 89.5 & 91.6 & 35.7 & - & 61.4 & 61.8 & 17.6 \\
& 72B & 88.4 & 90.3 & 38.1 & \underline{74.3} & \underline{73.7} & - & \underline{25.5} \\
\multirow{2}{*}{TongUI~\citep{zhang2025tongui}}
& 3B & 83.6 & 85.5 & 18.0 & - & - & - & 15.4 \\
& 7B & 86.0 & 88.7 & 24.7 & - & - & - & 18.0 \\
\multirow{2}{*}{GUI-Actor~\citep{wu2025gui}}
& 2B & 86.5 & 88.6 & 42.2 & - & - & - & - \\
& 7B & 88.3 & 89.5 & 44.6 & - & - & - & - \\
\multirow{2}{*}{JEDI~\citep{xie2025scaling}}
& 3B & - & 88.6 & 36.1 & - & - & - & 19.0 \\
& 7B & - & 91.7 & 39.5 & - & - & - & 25.2 \\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (RFT)}}} \\
UI-R1~\citep{lu2025ui}
& 3B & 83.3 & 85.4 & 17.8 & - & 58.5 & - & - \\
UI-R1-E~\citep{lu2025ui}
& 3B & 89.2 & 89.5 & 33.5 & - & - & - & - \\
\multirow{2}{*}{GUI-R1~\citep{luo2025gui}}
& 3B & - & - & 28.6 & - & - & - & - \\
& 7B & - & - & 31.3 & - & - & - & - \\
InfiGUI-R1~\citep{liu2025infigui}
& 3B & 87.5 & - & 35.7 & - & 69.7 & - & - \\
GUI-G1~\citep{zhou2025gui}
& 3B & 90.3 & - & 37.1 & - & - & - & - \\
SE-GUI~\citep{yuan2025enhancing}
& 7B & 88.2 & 90.3 & 47.3 & - & - & - & - \\
LPO~\citep{tang2025lpo}
& 8B & - & 90.5 & - & - & - & - & - \\
GUI-G$^2$\xspace~\citep{tang2025gui}
& 7B & \textbf{92.0} & \underline{93.3} & \underline{47.5} & - & - & - & -\\
% AgentCPM-GUI~\citep{zhang2025agentcpm}
% & 8B \\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{Ours}}} \\
\multirow{2}{*}{HyperClick}
& 3B & 88.5 & 90.6 & 41.3 & 71.4 & 71.8 & \underline{81.0} & 19.6 \\
& 7B & \underline{91.5} & \textbf{93.7} & \textbf{48.2} & \textbf{79.6} & \textbf{76.5} & \textbf{82.9} & \textbf{25.7} \\
\bottomrule
\end{tabular}
}
\vspace{-1.0em}
\end{table}
\begin{table}[!t ]
\centering
\caption{The evaluation of HyperClick on ScreenSpot-Pro is conducted under reliable and reproducible settings. The “Original” accuracy refers to the results reported in the corresponding papers or reproduced by subsequent studies, while the “Replicated” accuracy denotes our reproduction using the vllm-project~\citep{kwon2023efficient} with the official model weights. The observed performance gaps may stem from differences in prompt design or in whether unparsed outputs are included during evaluation.}
\label{tab: ap}
\resizebox{\linewidth}{!}
{
\begin{tabular}{lccccccc}
\toprule
\multirow{2}{*}{\textbf{Model}}
& \multirow{2}{*}{\textbf{Size}}
& \multicolumn{2}{c}{\textbf{Accuracy}}
& \multirow{2}{*}{\textbf{AP}$^{\text{conf}=50}$}
& \multirow{2}{*}{\textbf{AP}$^{\text{conf}=75}$}
& \multirow{2}{*}{\textbf{AP}$^{\text{conf}=90}$}
& \multirow{2}{*}{\textbf{AR}$^{\text{conf}=95}$} \\
\cmidrule(lr){3-4}
& & Original & Replicated \\
\midrule
GPT-4o~\citep{openai2024hello}
& - & 0.8 & 0.8 & 0.9 & 0.9 & 1.2 & 1.0 \\
Doubao~\citep{guo2025seed1}
& - & - & 13.0 & 13.6 & 15.8 & 21.2 & 21.5 \\
Qwen2.5-VL~\citep{bai2025qwen2}
& 7B & 26.8 & 22.5 & 24.9 & 24.9 & 24.8 & 24.7 \\
KiMi-VL~\citep{team2025kimi}
& 16B & 34.5 & 35.4 & 34.8 & 34.8 & 25.8 & 40.6 \\
MiMo-VL~\citep{coreteam2025mimovltechnicalreport}
& 7B & 39.9 & 38.3 & 29.5 & 28.9 & 28.8 & 30.0 \\
UI-TARS~\citep{qin2025ui}
& 7B & 35.7 & 37.6 & 37.5 & 37.5 & 37.4 & 39.3 \\
UI-TARS-1.5~\citep{qin2025ui}
& 7B & - & 37.2 & 37.6 & 37.5 & 37.5 & 40.4 \\
\midrule
\multirow{2}{*}{HyperClick}
& 3B & - & 41.3 & \textbf{70.6} & \textbf{76.0} & \textbf{78.0} & 78.0 \\
& 7B & - & \textbf{48.2} & 61.3 & 64.6 & 71.2 & \textbf{78.7} \\
\bottomrule
\end{tabular}
}
\vspace{-1.0em}
\end{table}
\begin{table}[!t]
\centering
\begin{minipage}{0.42\textwidth}
\centering
\caption{Ablation study of reward configurations.}
\label{tab: ablation_reward}
\small
\begin{tabular}{cccc}
\toprule
$R_{\text{format}}$
& $R_{\text{corectness}}$
& $R_{\text{confidence}}$
& \textbf{Acc(\%)} \\
\midrule
& \checkmark & & 47.5 \\
\checkmark & \checkmark & & 47.7 \\
& \checkmark & \checkmark & 48.0 \\
\rowcolor{MiOrange} \checkmark & \checkmark & \checkmark & 48.2 \\
\bottomrule
\end{tabular}
\end{minipage}
\hfill
\begin{minipage}{0.24\textwidth}
\centering
\caption{Ablation study of confidence.}
\label{tab: ablation_confidence}
\small
\begin{tabular}{cc}
\toprule
\makebox[0.30\linewidth][c]{$\alpha$}
& {\textbf{Acc(\%)}} \\
\midrule
0 & 47.7 \\
1/2 & 48.0 \\
\rowcolor{MiOrange} 1/4 & 48.2 \\
1/6 & 45.7 \\
\bottomrule
\end{tabular}
\end{minipage}
\hfill
\begin{minipage}{0.28\textwidth}
\centering
\caption{Ablation of baseline.}
\label{tab: ablation_baseline}
\small
\begin{tabular}{cc}
\toprule
\textbf{Model} & \textbf{Acc(\%)} \\
\midrule
Qwen2.5-VL & 26.8 \\
\rowcolor{MiOrange} HyperClick & 48.2 \\
\midrule
MiMo-VL & 39.9 \\
\rowcolor{MiOrange} HyperClick & 49.5 \\
\bottomrule
\end{tabular}
\end{minipage}
\vspace{-1.0em}
\end{table}
% ==========================================================================================
\section{Conclusion}
In this work, we address the critical issue of overconfidence in GUI grounding models, which undermines the reliability of autonomous GUI agents. We introduce HyperClick, a novel framework that augments grounding with explicit uncertainty calibration. By combining binary correctness rewards with truncated Gaussian–based spatial confidence modeling, HyperClick enhances grounding accuracy while producing well-calibrated confidence estimates, enabling the agent to assess its own reliability introspectively. Extensive experiments on challenging benchmarks demonstrate that HyperClick achieves SOTA performance in both accuracy and calibration, substantially enhancing the reliability of GUI agents. Looking ahead, this framework can be extended to broader multimodal agentic settings, where reliable confidence estimation is essential for safe and reliable human-AI interaction.
% ==========================================================================================
\newpage
\appendix
\section{Appendix}
% ======================================================================
\subsection{Use of LLMs}
In this paper, LLMs are employed solely as auxiliary tools for text refinement. Specifically, they are used to edit, polish and improve the clarity and readability of the manuscript, without contributing to the design of methods, the execution of experiments, or the analysis of results. All conceptual development, technical implementation, and empirical evaluation were independently conducted by the authors. The use of LLMs is therefore limited to linguistic enhancement, ensuring that the work's presentation is more precise and accessible to readers.
% ======================================================================
\subsection{Limitation}
Although the effect of the uncertainty calibration mechanism proposed in this work has been verified, it has not been extended to GUI planning tasks. We believe that the reliability of planning is even more critical for the overall success of GUI automation, since inaccurate or overconfident planning decisions can propagate errors across multiple steps and ultimately lead to task failure. In future work, we plan to investigate how uncertainty calibration can be incorporated into planning modules, enabling agents to not only ground actions reliably but also make trustworthy high-level decisions throughout complex multi-step interactions.
% ======================================================================
\subsection{Prompt}
In this section, we detail the prompt for the replicated evaluation in ScreenSpot-Pro~\citep{li2025screenspot}. We follow the instructions they originally provided to reproduce and analyze the experimental results. The prompts are shown as follows:
\begin{tcolorbox}[colframe=gray, colback=MiOrange, title=GPT-4o's Prompt]
Locate the UI element most related to the instruction \{\texttt{problem}\} on the screenshot. Output only a JSON in the format [\{``point\_2d'': [...]\}].
\end{tcolorbox}
\begin{tcolorbox}[colframe=gray, colback=MiOrange, title=Doubao's Prompt]
Locate the UI element most related to the instruction \{\texttt{problem}\} on the screenshot. Output only a JSON in the format [\{``point\_2d'': [...]\}].
\end{tcolorbox}
\begin{tcolorbox}[colframe=gray, colback=MiOrange, title=Qwen2.5-VL's Prompt]
Locate the UI element most related to the instruction \{\texttt{problem}\} on the screenshot. Output only a JSON in the format [\{``point\_2d'': [...], ``label'': ... \}].
\end{tcolorbox}
\begin{tcolorbox}[colframe=gray, colback=MiOrange, title=KiMi-VL's Prompt]
Point to the UI element most related to the instruction \{\texttt{problem}\} on the screenshot.
\end{tcolorbox}
\begin{tcolorbox}[colframe=gray, colback=MiOrange, title=MiMo-VL's Prompt]
Locate the UI element most related to the instruction \{\texttt{problem}\} on the screenshot. Output a JSON format [\{``bbox\_2d'': [...], ``label'': ...\}]./no\_think
\end{tcolorbox}
\begin{tcolorbox}[colframe=gray, colback=MiOrange, title=UI-TARS' and UI-TARS-1.5's Prompt]
Point to the element related to the instruction \{\texttt{problem}\} on the screenshot.
\end{tcolorbox}
Due to UI-TARS~\citep{qin2025ui} and UI-TARS-1.5~\citep{qin2025ui} being trained with a large amount of GUI-specific data, the ability to follow instructions is relatively poor. To prompt such models to generate verbalized confidence in their predictions, we adopt a multi-round conversation to output confidence for their answer. Specifically, policy models use the above prompts for GUI grounding in the first round and in the second round, generate the verbalized confidence of the prediction according to the prompt below:
\begin{tcolorbox}[colframe=gray, colback=MiOrange, title=Confidence Prompt]
Output only a float number ranging from 0 to 1, representing your confidence with your provided answer, without any format.
\end{tcolorbox}
% ======================================================================
\subsection{Stability of confidence}
To evaluate the reliability of HyperClick's confidence, we verified that the model's confidence in the same answer remains stable. We first let HyperClick predict the coordinates without doing a sample. Then, we inject the coordinates into the assistant’s generation and instruct it to continue outputting confidence at a temperature of 1.0 for 8 times. As shown in Table~\ref{tab: variance}, we report the mean variance for different sample sizes. The results indicate that both HyperClick-3B and HyperClick-7B maintain very low variance across different sampling scales, with the larger 7B model showing slightly more stable outputs. This suggests that the confidence estimation of HyperClick is well-calibrated, ensuring consistent reliability even under repeated sampling.
\begin{table}[htbp]
\centering
\caption{Stability evaluation of the model for the same prediction.}
\label{tab: variance}
\begin{tabular}{lcccccc}
\toprule
\multirow{2}{*}{\textbf{Model}} & \multicolumn{6}{c}{\textbf{Variance}} \\
\cmidrule(lr){2-7}
& 10 & 50 & 100 & 500 & 1000 & 1581 \\
\midrule
HyperClick-3B & 0.020 & 0.028 & 0.023 & 0.020 & 0.020 & 0.020 \\
HyperClick-7B & 0.014 & 0.020 & 0.020 & 0.019 & 0.019 & 0.019 \\
\bottomrule
\end{tabular}
\end{table}
% ======================================================================
\subsection{Data Details}
To provide a comprehensive grounding resource across diverse platforms, we construct a dataset containing 30K samples distributed across three representative domains: Mobile, Web, and Desktop. Each domain contains a balanced set of grounding instances that pair natural language commands with corresponding UI elements. The number of samples collected from each dataset is shown below.
\input{tab/Data}
To construct high-quality samples for RFT, we first employ Qwen2.5-VL-7B~\citep{bai2025qwen2} to generate raw data with the temperature set to 0, and identify cases where the model produces incorrect predictions. For each of these error cases, we then perform eight additional inferences with temperature 0.9 and extract the correctly predicted results as the final training data.
In addition, prior to RFT, we incorporate an equal number of correctly predicted samples from Stage 1 to provide a cold start. This initialization not only stabilizes the training but also helps the model adhere to the target output format: \texttt{<point>[x,y]</point><confidence>conf</confidence>}.
% ======================================================================
\subsection{Evaluation Benchmarks and Detailed Experimental Results}
In this section, we detail the benchmarks adopted in this work.
\textbf{ScreenSpot} evaluates GUI grounding across mobile, desktop, and web platforms. Provides a diverse set of interface types, enabling the comparison of model robustness across common user scenarios. Detailed experimental results and comparisons with baselines are shown in Table~\ref{tab: screenspot}.
\input{tab/ScreenSpot}
\textbf{ScreenSpot-V2} extends ScreenSpot with more challenging tasks and refined annotations. Additionally, it tests grounding accuracy in various real-world environments. Detailed experimental results and comparisons with baselines are shown in Table~\ref{tab: screenspot_v2}.
\input{tab/ScreenSpot_V2}
\textbf{ScreenSpot-Pro} focuses on high-resolution professional settings with expert-annotated tasks. Covers 23 applications, five industries, and three operating systems, making it one of the most comprehensive GUI grounding benchmarks. Detailed experimental results and comparisons with baselines are shown in Table~\ref{tab: screenspot_pro}.
\input{tab/ScreenSpot_Pro}
\textbf{MMBench-GUI} organizes tasks into a hierarchical structure of basic and advanced instructions. This design enables the systematic evaluation of model performance across varying levels of instruction complexity. Detailed experimental results and comparisons with baselines are shown in Table~\ref{tab: mmbench_gui}.
\input{tab/MMbench_GUI}
\textbf{UI-I2E-Bench} introduces implicit instructions that require both semantic understanding and spatial reasoning. Highlights the limitations of direct grounding and encourages models to adopt more sophisticated reasoning. Detailed experimental results and comparisons with baselines are shown in Table~\ref{tab: ui_i2e}.
\input{tab/UI_I2E_Bench}
\textbf{CAGUI} is a Chinese benchmark for mobile GUI grounding. It emphasizes the grounding of textual elements and functional operations within Chinese-language applications. Detailed experimental results and comparisons with baselines are shown in Table~\ref{tab: cagui}.
\input{tab/CAGUI}
\textbf{UI-Vision} evaluates the generalization of cross-applications in diverse desktop environments. By incorporating previously unseen applications, it tests the model's robustness and adaptability. Detailed experimental results and comparisons with baselines are shown in Table~\ref{tab: ui_vision}.
\input{tab/UI_Vision}
\begin{table}[htbp]
\centering
\caption{Statistics and sources of the grounding dataset adopted in HyperClick.}
\label{tab: data}
\begin{tabular}{lccccccc}
\toprule
\textbf{Source} & OmniAct & ShowUI-Web & UI-Refexp & Widgent-Caption & OS-Atlas & In-House\\
\midrule
\textbf{Size} & 119 & 19172 & 280 & 3672 & 26114 & 1664 \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[htbp]
\centering
\caption{GUI grounding accuracy on the ScreenSpot~\citep{cheng2024seeclick} benchmarks over the Mobile, Desktop, and Web sub-tasks. \textbf{Bold} and \underline{underline} indicate the best and second-best results.}
\resizebox{\linewidth}{!}
{
\begin{tabular}{lcccccccc}
\toprule
\multirow{4}{*}{\textbf{Model}} & \multirow{4}{*}{\textbf{Size}} & \multicolumn{6}{c}{\textbf{ScreenSpot v1}} & \multirow{4}{*}{\makecell{\textbf{SSv1} \\ \textbf{Avg.}}} \\
\cmidrule(lr){3-8}
& & \multicolumn{2}{c}{\textbf{Mobile}} & \multicolumn{2}{c}{\textbf{Desktop}} & \multicolumn{2}{c}{\textbf{Web}} \\
\cmidrule(lr){3-4}
\cmidrule(lr){5-6}
\cmidrule(lr){7-8}
& & Text & Icon & Text & Icon & Text & Icon \\
& & (273) & (229) & (194) & (140) & (230) & (206) \\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{General Models}}} \\
GPT-4o~\citep{openai2024hello}
& - & 30.5 & 23.2 & 20.6 & 19.4 & 11.1 & 7.8 & 18.8 \\
Claude~\citep{anthropic2024claude}
& - & - & - & - & - & - & - & 83.0 \\
Qwen2-VL~\citep{wang2024qwen2}
& 7B & 61.3 & 39.3 & 52.0 & 45.0 & 33.0 & 21.8 & 42.9 \\
\multirow{2}{*}{Qwen2.5-VL~\citep{bai2025qwen2}}
& 3B & - & - & - & - & - & - & 55.5 \\
& 7B & - & - & - & - & - & - & 84.7 \\
\multirow{2}{*}{InternVL3~\citep{zhu2025internvl3}}
& 8B & - & - & - & - & - & - & 79.5 \\
& 38B & - & - & - & - & - & - & 85.6\\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (SFT)}}} \\
CogAgent~\citep{hong2024cogagent} &
18B & 67.0 & 24.0 & 74.2 & 20.0 & 70.4 & 28.6 & 47.4 \\
SeeClick~\citep{cheng2024seeclick}
& 9.6B & 78.0 & 52.0 & 72.2 & 30.0 & 55.7 & 32.5 & 53.4 \\
Aria-UI~\citep{yang2024aria}
& 25.3B & 92.3 & 73.8 & 93.3 & 64.3 & 86.5 & 76.2 & 82.4 \\
ShowUI~\citep{lin2025showui}
& 2B & 92.3 & 75.5 & 76.3 & 61.1 & 81.7 & 63.6 & 75.1 \\
UGround~\citep{gou2025uground}
& 7B & 82.8 & 60.3 & 82.5 & 63.6 & 80.4 & 70.4 & 73.3 \\
\multirow{2}{*}{UGround-V1~\citep{gou2025uground}}
& 2B & 89.4 & 72.0 & 88.7 & 65.7 & 81.3 & 68.9 & 77.7 \\
& 7B & 93.0 & 79.9 & 93.8 & 76.4 & 90.9 & 84.0 & 86.3 \\
\multirow{2}{*}{OS-Atlas~\citep{wu2024atlas}}
& 4B & 85.7 & 58.5 & 72.2 & 45.7 & 82.6 & 63.1 & 70.1 \\
& 7B & 93.0 & 72.9 & 91.8 & 62.9 & 90.89 & 74.3 & 82.5 \\
\multirow{2}{*}{Aguvis~\citep{xu2024aguvis}}
& 7B & 95.6 & 77.7 & 93.8 & 67.1 & 88.3 & 75.2 & 84.4 \\
& 72B & 94.5 & 85.2 & 95.4 & 77.9 & 91.3 & 85.9 & 89.2 \\
\multirow{3}{*}{UI-TARS~\citep{qin2025ui}}
& 2B & 93.0 & 75.5 & 90.7 & 68.6 & 84.3 & 74.8 & 82.3 \\
& 7B & 94.5 & 85.2 & 95.9 & 85.7 & 90.0 & 83.5 & 89.5 \\
& 72B & 94.9 & 82.5 & 89.7 & 88.6 & 88.7 & 85.0 & 88.4 \\
\multirow{2}{*}{TongUI~\citep{zhang2025tongui}}
& 3B & 92.6 & 77.7 & 92.3 & 77.1 & 87.8 & 74.8 & 83.6 \\
& 7B & 91.9 & 79.5 & 93.8 & 80.0 & 89.1 & 81.6 & 86.0 \\
\multirow{2}{*}{GUI-Actor~\citep{wu2025gui}}
& 2B & 93.0 & 79.9 & 88.1 & 78.6 & 90.9 & 84.0 & 86.5 \\
& 7B & 94.9 & 82.1 & 91.8 & 80.0 & 91.3 & 85.4 & 88.3 \\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (RFT)}}} \\
UI-R1~\citep{lu2025ui}
& 3B & 95.6 & 84.7 & 90.2 & 59.3 & 85.2 & 73.3 & 83.3 \\
UI-R1-E~\citep{lu2025ui}
& 3B & 97.1 & 83.0 & 95.4 & 77.9 & 91.7 & 85.0 & 89.2 \\
\multirow{2}{*}{GUI-R1~\citep{luo2025gui}}
& 3B & - & - & 93.8 & 64.8 & 89.6 & 72.1 & - \\
& 7B & - & - & 91.8 & 73.6 & 91.3 & 75.7 & - \\
InfiGUI-R1~\citep{liu2025infigui}
& 3B & 97.1 & 81.2 & 94.3 & 77.1 & 91.7 & 77.6 & 87.5 \\
GUI-G1~\citep{zhou2025gui}
& 3B & 98.6 & 85.8 & 96.4 & 80.7 & 91.4 & 82.3 & 90.3 \\
SE-GUI~\citep{yuan2025enhancing}
& 7B & - & - & - & - & - & - & 88.2 \\
GUI-G$^2$\xspace~\citep{tang2025gui}
& 7B & 96.7 & 90.8 & 95.9 & 88.6 & 90.9 & 86.9 & \textbf{92.0} \\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{Ours}}} \\
\multirow{2}{*}{HyperClick}
& 3B & 96.7 & 83.9 & 92.8 & 80.7 & 88.7 & 83.5 & 88.5 \\
& 7B & 95.6 & 91.7 & 93.8 & 82.9 & 92.2 & 88.4 & \underline{91.5} \\
\bottomrule
\end{tabular}
}
\label{tab: screenspot}
\end{table}
\begin{table}[htbp]
\centering
\caption{GUI grounding accuracy on the ScreenSpot~\citep{cheng2024seeclick} and ScreenSpot-V2 benchmarks over the Mobile, Desktop, and Web sub-tasks. \textbf{Bold} and \underline{underline} indicate the best and second-best results.}
\resizebox{\linewidth}{!}
{
\begin{tabular}{lcccccccc}
\toprule
\multirow{4}{*}{\textbf{Model}} & \multirow{4}{*}{\textbf{Size}} & \multicolumn{6}{c}{\textbf{ScreenSpot V2}} & \multirow{4}{*}{\makecell{\textbf{SSv2} \\ \textbf{Avg.}}} \\
\cmidrule(lr){3-8}
& & \multicolumn{2}{c}{\textbf{Mobile}} & \multicolumn{2}{c}{\textbf{Desktop}} & \multicolumn{2}{c}{\textbf{Web}} \\
\cmidrule(lr){3-4}
\cmidrule(lr){5-6}
\cmidrule(lr){7-8}
& & Text & Icon & Text & Icon & Text & Icon \\
& & (290) & (211) & (194) & (140) & (234) & (203) \\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{General Models}}} \\
GPT-4o~\citep{openai2024hello}
& - & 26.6 & 24.2 & 24.2 & 19.3 & 12.8 & 11.8 & 20.1 \\
\multirow{2}{*}{Qwen2.5-VL~\citep{bai2025qwen2}}
& 3B & 93.4 & 73.5 & 88.1 & 58.6 & 88.0 & 71.4 & 80.9 \\
& 7B & 97.6 & 87.2 & 90.2 & 74.2 & 93.2 & 81.3 & 88.8\\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (SFT)}}} \\
SeeClick~\citep{cheng2024seeclick}
& 9.6B & 78.4 & 50.7 & 70.1 & 29.3 & 55.2 & 32.5 & 55.1\\
UGround~\citep{gou2025uground}
& 7B & 75.1 & 84.5 & 85.1 & 61.4 & 84.6 & 71.9 & 76.3\\
\multirow{2}{*}{OS-Atlas~\citep{wu2024atlas}}
& 4B & 87.2 & 59.7 & 72.7 & 46.4 & 85.9 & 63.1 & 71.9 \\
& 7B & 95.2 & 75.8 & 90.7 & 63.6 & 90.6 & 77.3 & 84.1 \\
\multirow{3}{*}{UI-TARS~\citep{qin2025ui}}
& 2B & 95.2 & 79.1 & 90.7 & 68.6 & 87.2 & 78.3 & 84.7 \\
& 7B & 96.9 & 89.1 & 95.4 & 85.0 & 93.6 & 85.2 & 91.6 \\
& 72B & 94.8 & 86.3 & 91.2 & 87.9 & 91.5 & 87.7 & 90.3 \\
\multirow{2}{*}{TongUI~\citep{zhang2025tongui}}
& 3B & 94.4 & 79.6 & 92.8 & 75.0 & 87.6 & 77.8 & 85.5 \\
& 7B & 93.1 & 81.5 & 96.4 & 82.9 & 90.2 & 84.7 & 88.7 \\
\multirow{2}{*}{GUI-Actor~\citep{wu2025gui}}
& 2B & 95.0 & 82.2 & 92.2 & 81.8 & 92.9 & 82.7 & 88.6 \\
& 7B & 96.5 & 84.3 & 91.7 & 84.1 & 93.9 & 82.3 & 89.5 \\
\multirow{2}{*}{JEDI~\citep{xie2025scaling}}
& 3B & 96.6 & 81.5 & 96.9 & 78.6 & 88.5 & 83.7 & 88.6 \\
& 7B & 96.9 & 87.2 & 95.9 & 87.9 & 94.4 & 84.2 & 91.7 \\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (RFT)}}} \\
UI-R1~\citep{lu2025ui}
& 3B & 96.2 & 84.3 & 92.3 & 63.6 & 89.2 & 75.4 & 85.4 \\
UI-R1-E~\citep{lu2025ui}
& 3B & 98.2 & 83.9 & 94.8 & 75.0 & 83.7 & 93.2 & 89.5 \\
SE-GUI~\citep{yuan2025enhancing}
& 7B & - & - & - & - & - & - & 90.3 \\
LPO~\citep{tang2025lpo}
& 8B & 97.9 & 82.9 & 95.9 & 86.4 & 95.6 & 84.2 & 90.5 \\
GUI-G$^2$\xspace~\citep{tang2025gui}
& 7B & 98.3 & 91.9 & 95.4 & 89.3 & 94.0 & 87.7 & \underline{93.3} \\
\midrule
\multicolumn{9}{l}{\cellcolor{MiOrange}{\textit{Ours}}} \\
\multirow{2}{*}{HyperClick}
& 3B & 98.6 & 86.3 & 95.4 & 90.6 & 82.2 & 84.7 & 90.6 \\
& 7B & 98.3 & 93.4 & 96.9 & 85.7 & 96.2 & 86.7 & \textbf{93.7} \\
\bottomrule
\end{tabular}
}
\label{tab: screenspot_v2}
\end{table}
\begin{table}[htbp]
\centering
\caption{GUI grounding accuracy on the ScreenSpot-Pro~\citep{li2025screenspot} benchmark over the CAD, Development, Creative, Scientific, Office, and OS sub-tasks. \textbf{Bold} and \underline{underline} indicate the best and second-best results.}
\resizebox{\linewidth}{!}
{
\begin{tabular}{lcccccccccccccc}
\toprule
\multirow{3}{*}{\textbf{Model}} & \multirow{3}{*}{\textbf{Size}} & \multicolumn{2}{c}{\textbf{CAD}} & \multicolumn{2}{c}{\textbf{Development}} & \multicolumn{2}{c}{\textbf{Creative}} & \multicolumn{2}{c}{\textbf{Scientific}} & \multicolumn{2}{c}{\textbf{Office}} & \multicolumn{2}{c}{\textbf{OS}} & \multirow{3}{*}{\textbf{Avg.}} \\
\cmidrule(lr){3-4}
\cmidrule(lr){5-6}
\cmidrule(lr){7-8}
\cmidrule(lr){9-10}
\cmidrule(lr){11-12}
\cmidrule(lr){13-14}
& & Text & Icon & Text & Icon & Text & Icon & Text & Icon & Text & Icon & Text & Icon \\
& & (197) & (64) & (154) & (145) & (198) & (143) & (144) & (110) & (177) & (53) & (107) & (89) \\
\midrule
\multicolumn{15}{l}{\cellcolor{MiOrange}{\textit{General Models}}} \\
GPT-4o~\citep{openai2024hello}
& - & 2.0 & 0.0 & 1.3 & 0.0 & 1.0 & 0.0 & 2.1 & 0.0 & 1.1 & 0.0 & 0.0 & 0.0 & 0.8 \\
Claude~\citep{anthropic2024claude}
& - & 14.5 & 3.7 & 22.0 & 3.9 & 25.9 & 3.4 & 33.9 & 15.8 & 30.1 & 16.3 & 11.0 & 4.5 & 17.1 \\
\multirow{2}{*}{Qwen2.5-VL~\citep{bai2025qwen2}}
& 3B & 9.1 & 7.3 & 22.1 & 1.4 & 26.8 & 2.1 & 38.2 & 7.3 & 33.9 & 15.1 & 10.3 & 1.1 & 16.1 \\
& 7B & 16.8 & 1.6 & 46.8 & 4.1 & 35.9 & 7.7 & 49.3 & 7.3 & 52.5 & 20.8 & 37.4 & 6.7 & 26.8 \\
\midrule
\multicolumn{15}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (SFT)}}} \\
CogAgent~\citep{hong2024cogagent}
& 18B & 7.1 & 3.1 & 14.9 & 0.7 & 9.6 & 0.0 & 22.2 & 1.8 & 13.0 & 0.0 & 5.6 & 0.0 & 7.7 \\
SeeClick~\citep{cheng2024seeclick}
& 9.6B & 2.5 & 0.0 & 0.6 & 0.0 & 1.0 & 0.0 & 3.5 & 0.0 & 1.1 & 0.0 & 2.8 & 0.0 & 1.1 \\
ShowUI~\citep{lin2025showui}
& 2B & 2.5 & 0.0 & 16.9 & 1.4 & 9.1 & 0.0 & 13.2 & 7.3 & 15.3 & 7.5 & 10.3 & 2.2 & 7.7 \\
Aria-UI~\citep{yang2024aria}
& 25.3B & 7.6 & 1.6 & 16.2 & 0.0 & 23.7 & 2.1 & 27.1 & 6.4 & 20.3 & 1.9 & 4.7 & 0.0 & 11.3\\
UGround~\citep{gou2025uground}
& 7B & 14.2 & 1.6 & 26.6 & 2.1 & 27.3 & 2.8 & 31.9 & 2.7 & 31.6 & 11.3 & 17.8 & 0.0 & 16.5 \\
UGround-V1~\citep{gou2025uground}
& 7B & 15.8 & 1.2 & 51.9 & 2.8 & 47.5 & 9.7 & 57.6 & 14.5 & 60.5 & 13.2 & 38.3 & 7.9 & 45.2 \\
\multirow{2}{*}{OS-Atlas~\citep{wu2024atlas}}
& 4B & 2.0 & 0.0 & 7.1 & 0.0 & 3.0 & 1.4 & 9.0 & 5.5 & 5.1 & 3.8 & 5.6 & 0.0 & 3.7 \\
& 7B & 12.2 & 4.7 & 33.1 & 1.4 & 28.8 & 2.8 & 37.5 & 7.3 & 33.9 & 5.7 & 27.1 & 4.5 & 18.9 \\
\multirow{3}{*}{UI-TARS~\citep{qin2025ui}}
& 2B & 17.8 & 4.7 & 47.4 & 4.1 & 42.9 & 6.3 & 56.9 & 17.3 & 50.3 & 17.0 & 21.5 & 5.6 & 27.7 \\
& 7B & 20.8 & 9.4 & 58.4 & 12.4 & 50.0 & 9.1 & 63.9 & 31.8 & 63.3 & 20.8 & 30.8 & 16.9 & 35.7 \\
& 72B & 18.8 & 12.5 & 62.9 & 17.2 & 57.1 & 15.4 & 64.6 & 20.9 & 63.3 & 26.4 & 42.1 & 15.7 & 38.1 \\
\multirow{2}{*}{TongUI~\citep{zhang2025tongui}}
& 3B & 11.7 & 1.6 & 32.5 & 0.7 & 24.8 & 2.8 & 43.1 & 12.7 & 32.8 & 7.6 & 15.0 & 1.1 & 18.0 \\
& 7B & 17.3 & 9.4 & 40.9 & 3.5 & 31.3 & 7.0 & 50.7 & 12.7 & 45.8 & 13.2 & 28.0 & 6.7 & 24.7 \\
\multirow{2}{*}{GUI-Actor~\citep{wu2025gui}}
& 2B & - & - & - & - & - & - & - & - & - & - & - & - & 36.7 \\
& 7B & - & - & - & - & - & - & - & - & - & - & - & - & 40.7 \\
\multirow{2}{*}{JEDI~\citep{xie2025scaling}}
& 3B & 27.4 & 9.4 & 61.0 & 13.8 & 53.5 & 8.4 & 54.2 & 18.2 & 64.4 & 32.1 & 38.3 & 9.0 & 36.1 \\
& 7B & 38.0 & 14.1 & 42.9 & 11.0 & 50.0 & 11.9 & 72.9 & 25.5 & 75.1 & 47.2 & 33.6 & 16.9 & 39.5 \\
\midrule
\multicolumn{15}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (RFT)}}} \\
UI-R1~\citep{lu2025ui}
& 3B & 11.2 & 6.3 & 22.7 & 4.1 & 27.3 & 3.5 & 42.4 & 11.8 & 32.2 & 11.3 & 13.1 & 4.5 & 17.8 \\
UI-R1-E~\citep{lu2025ui}
& 3B & 37.1 & 12.5 & 46.1 & 6.9 & 41.9 & 4.2 & 56.9 & 21.8 & 65.0 & 26.4 & 32.7 & 10.1 & 33.5 \\
\multirow{2}{*}{GUI-R1~\citep{luo2025gui}}
& 3B & 26.4 & 7.8 & 33.8 & 4.8 & 40.9 & 5.6 & 61.8 & 17.3 & 53.6 & 17.0 & 28.1 & 5.6 & 28.6 \\
& 7B & 23.9 & 6.3 & 49.4 & 4.8 & 38.9 & 8.4 & 55.6 & 11.8 & 58.7 & 26.4 & 42.1 & 16.9 & 31.3 \\
InfiGUI-R1~\citep{liu2025infigui}
& 3B & 33.0 & 14.1 & 51.3 & 12.4 & 44.9 & 7.0 & 58.3 & 20.0 & 65.5 & 28.3 & 43.9 & 12.4 & 35.7 \\
GUI-G1~\citep{zhou2025gui}
& 3B & 39.6 & 9.4 & 50.7 & 10.3 & 36.6 & 11.9 & 61.8 & 30.0 & 67.2 & 32.1 & 23.5 & 10.6 & 37.1 \\
\multirow{2}{*}{SE-GUI~\citep{yuan2025enhancing}}
& 3B & 38.1 & 12.5 & 55.8 & 7.6 & 47.0 & 4.9 & 61.8 & 16.4 & 59.9 & 24.5 & 40.2 & 12.4 & 35.9 \\
& 7B & 51.3 & 42.2 & 68.2 & 19.3 & 57.6 & 9.1 & 75.0 & 28.2 & 78.5 & 43.4 & 49.5 & 25.8 & 47.3 \\
GUI-G$^2$\xspace~\citep{tang2025gui}
& 7B & 55.8 & 12.5 & 68.8 & 17.2 & 57.1 & 15.4 & 77.1 & 24.5 & 74.0 & 32.7 & 57.9 & 21.3 & \underline{47.5} \\
\midrule
\multicolumn{15}{l}{\cellcolor{MiOrange}{\textit{Ours}}} \\
\multirow{2}{*}{HyperClick}
& 3B & 43.7 & 23.5 & 62.4 & 20.0 & 50.5 & 12.6 & 55.6 & 30.0 & 63.9 & 37.8 & 41.1 & 20.2 & 41.3 \\
& 7B & 51.3 & 20.3 & 70.2 & 22.1 & 57.6 & 20.3 & 76.4 & 30.9 & 70.1 & 30.2 & 56.1 & 22.5 & \textbf{48.2} \\
\bottomrule
\end{tabular}
}
\label{tab: screenspot_pro}
\end{table}
\begin{table}[htbp]
\centering
\caption{GUI grounding accuracy on the MMBench-GUI~\citep{wang2025mmbench} benchmark over the Windows, MacOS, Linux, iOS, Android, and Web sub-stasks. \textbf{Bold} and \underline{underline} indicate the best and second-best results.}
\resizebox{\linewidth}{!}
{
\begin{tabular}{lcccccccccccccc}
\toprule
\multirow{3}{*}{\textbf{Model}} & \multirow{3}{*}{\textbf{Size}} & \multicolumn{2}{c}{\textbf{Windows}} & \multicolumn{2}{c}{\textbf{MacOS}} & \multicolumn{2}{c}{\textbf{Linux}} & \multicolumn{2}{c}{\textbf{iOS}} & \multicolumn{2}{c}{\textbf{Android}} & \multicolumn{2}{c}{\textbf{Web}} & \multirow{3}{*}{\textbf{Avg.}} \\
\cmidrule(lr){3-4}
\cmidrule(lr){5-6}
\cmidrule(lr){7-8}
\cmidrule(lr){9-10}
\cmidrule(lr){11-12}
\cmidrule(lr){13-14}
& & Basic & Adv. & Basic & Adv.& Basic & Adv.& Basic & Adv.& Basic & Adv.& Basic & Adv. & \\
& & (271) & (272) & (345) & (346) & (191) & (196) & (314) & (330) & (356) & (355) & (310) & (308) \\
\midrule
\multicolumn{15}{l}{\cellcolor{MiOrange}{\textit{General Models}}} \\
GPT-4o~\citep{openai2024hello}
& - & 1.5 & 1.1 & 8.7 & 4.3 & 1.1 & 1.0 & 5.1 & 3.3 & 2.5 & 1.4 & 3.2 & 2.9 & 2.9 \\
Claude~\citep{anthropic2024claude}
& - & 1.5 & 0.7 & 12.5 & 7.5 & 1.1 & 0.0 & 13.7 & 10.6 & 1.4 & 1.4 & 3.2 & 2.3 & 4.7 \\
Qwen-Max-VL~\citep{baiqwenvl}
& - & 43.9 & 36.8 & 58.8 & 56.1 & 53.9 & 30.1 & 77.4 & 59.1 & 79.5 & 70.1 & 74.8 & 58.8 & 58.0 \\
\multirow{2}{*}{Qwen2.5-VL~\citep{bai2025qwen2}}
& 7B & 31.4 & 16.5 & 31.3 & 22.0 & 21.5 & 12.2 & 66.6 & 55.2 & 35.1 & 35.2 & 40.3 & 32.5 & 33.9 \\
& 72B & 55.7 & 33.8 & 49.9 & 30.1 & 40.3 & 20.9 & 56.1 & 28.2 & 55.6 & 25.4 & 68.4 & 45.8 & 41.8 \\
InternVL3~\citep{zhu2025internvl3}
& 72B & 70.1 & 42.6 & 75.7 & 52.3 & 59.2 & 41.3 & 93.6 & 80.6 & 92.7 & 78.6 & 90.7 & 65.9 & 72.2 \\
\midrule
\multicolumn{15}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (SFT)}}} \\
ShowUI~\citep{lin2025showui}
& 2B & 9.2 & 4.4 & 24.1 & 10.4 & 25.1 & 11.7 & 29.0 & 19.7 & 17.4 & 8.7 & 22.9 & 12.7 & 16.0 \\
OS-Atlas~\citep{wu2024atlas}
& 7B & 36.9 & 18.8 & 44.4 & 21.7 & 31.4 & 13.3 & 74.8 & 48.8 & 69.6 & 46.8 & 61.3 & 35.4 & 41.4 \\
Aguvis~\citep{xu2024aguvis}
& 7B & 37.3 & 21.7 & 48.1 & 33.3 & 33.5 & 25.0 & 67.5 & 65.2 & 61.0 & 51.0 & 61.6 & 45.5 & 45.7 \\
UGround-V1~\citep{gou2025uground}
& 7B & 66.8 & 39.0 & 71.3 & 48.6 & 56.5 & 31.1 & 92.7 & 70.9 & 93.5 & 71.0 & 88.7 & 64.6 & 65.7 \\
UI-TARS~\citep{qin2025ui}
& 72B & 78.6 & 51.8 & 80.3 & 62.7 & 68.6 & 51.5 & 90.8 & 81.2 & 93.0 & 80.0 & 88.1 & 68.5 & \underline{74.3} \\
\midrule
\multicolumn{15}{l}{\cellcolor{MiOrange}{\textit{Ours}}} \\
\multirow{2}{*}{HyperClick}
& 3B & 73.8 & 45.6 & 80.3 & 52.9 & 66.5 & 35.7 & 91.4 & 72.7 & 92.4 & 74.9 & 89.1 & 60.1 & 71.4 \\
& 7B & 82.3 & 61.4 & 82.9 & 67.1 & 66.5 & 48.0 & 94.0 & 82.1 & 95.8 & 85.1 & 93.2 & 85.1 & \textbf{79.6} \\
\bottomrule
\end{tabular}
}
\label{tab: mmbench_gui}
\end{table}
\begin{table}[htbp]
\centering
\caption{GUI grounding accuracy on the UI-I2E-Bench~\citep{liu2025ui} benchmark over the platforms of mobile, desktop, and web with various implicitness. \textbf{Bold} and \underline{underline} indicate the best and second-best results.}
\resizebox{\linewidth}{!}
{
\begin{tabular}{lccccccc}
\toprule
\multirow{3}{*}{\textbf{Model}} & \multirow{2}{*}{\textbf{Size}} & \multicolumn{3}{c}{\textbf{Platform}} & \multicolumn{2}{c}{\textbf{Implicitness}} & \multirow{3}{*}{\textbf{Avg.}} \\
\cmidrule(lr){3-5}
\cmidrule(lr){6-7}
& & Mobile & Desktop & Web & Explicit & Implicit \\
& & (705) & (519) & (253) & (917) & (560) \\
\midrule
\multicolumn{8}{l}{\cellcolor{MiOrange}{\textit{General Models}}} \\
\multirow{3}{*}{Qwen2.5-VL~\citep{bai2025qwen2}}
& 3B & 44.5 & 38.7 & 39.9 & 51.4 & 35.8 & 41.7 \\
& 7B & 61.7 & 41.6 & 56.9 & 58.4 & 51.0 & 53.8 \\
& 72B & 55.3 & 47.2 & 49.0 & 49.6 & 52.5 & 51.4 \\
\midrule
\multicolumn{8}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (SFT)}}} \\
ShowUI~\citep{lin2025showui}
& 2B & 53.9 & 30.4 & 29.6 & 51.3 & 35.6 & 41.5 \\
SeeClick~\citep{cheng2024seeclick}
& 9.6B & 37.2 & 15.8 & 18.2 & 37.1 & 19.9 & 26.4 \\
Aguvis~\citep{xu2024aguvis}
& 7B & 60.3 & 47.6 & 45.1 & 61.1 & 48.4 & 53.2 \\
OmniParser~\citep{wan2024omniparser}
& - & 67.6 & 45.5 & 30.8 & 54.3 & 52.4 & 53.1 \\
OmniParser~\citep{yu2025omniparser}
& - & 69.4 & 42.4 & 40.7 & 57.0 & 53.5 & 54.8 \\
\multirow{2}{*}{OS-Atlas~\citep{wu2024atlas}}
& 4B & 58.6 & 19.9 & 54.6 & 51.5 & 39.9 & 44.3 \\
& 7B & 68.1 & 48.9 & 52.2 & 63.2 & 55.8 & 58.6 \\
\multirow{3}{*}{UGround-V1~\citep{gou2025uground}}
& 2B & 59.9 & 49.5 & 66.4 & 72.9 & 47.9 & 57.4 \\
& 7B & 73.5 & 65.7 & 70.8 & 81.3 & 63.6 & 70.3 \\
& 72B & 78.2 & 74.6 & 74.7 & 84.5 & 71.3 & \underline{76.3} \\
\multirow{3}{*}{UI-TARS~\citep{qin2025ui}}
& 2B & 66.7 & 54.0 & 62.2 & 74.1 & 54.5 & 62.0 \\
& 7B & 65.7 & 58.0 & 56.5 & 71.4 & 55.3 & 61.4 \\
& 72B & 75.5 & 69.8 & 77.1 & 80.9 & 69.4 & 73.7 \\
\multirow{2}{*}{UI-I2E-VLM~\citep{liu2025ui}}
& 4B & 61.4 & 38.9 & 60.9 & 61.9 & 48.3 & 53.4 \\
& 7B & 76.2 & 64.0 & 62.1 & 72.0 & 67.9 & 69.5 \\
\midrule
\multicolumn{8}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (RFT)}}} \\
UI-R1~\citep{lu2025ui}
& 3B & 67.8 & 46.2 & 58.1 & 67.9 & 52.8 & 58.5\\
\midrule
\multicolumn{8}{l}{\cellcolor{MiOrange}{\textit{Ours}}} \\
\multirow{2}{*}{HyperClick}
& 3B & 77.9 & 59.0 &81.0 & 81.1 & 66.1 & 71.8 \\
& 7B & 80.4 & 67.5 & 84.2 & 84.8 & 71.4 & \textbf{76.5} \\
\bottomrule
\end{tabular}
}
\label{tab: ui_i2e}
\end{table}
\begin{table}[htbp]
\centering
\caption{GUI grounding accuracy on the CAGUI~\citep{zhang2025agentcpm} benchmark over the Fun2Point, Text2Point, and Bbox2Text sub-tasks. \textbf{Bold} and \underline{underline} indicate the best and second-best results.}
% \resizebox{\linewidth}{!}
% {
\begin{tabular}{lcccc}
\toprule
\multirow{2}{*}{\textbf{Model}} & \multirow{2}{*}{\textbf{Size}} & \textbf{Fun2Point} & \textbf{Text2Point} & \multirow{2}{*}{\textbf{Avg.}} \\
& & (1500) & (1500) \\
\midrule
\multicolumn{5}{l}{\cellcolor{MiOrange}{\textit{General Models}}} \\
GPT-4o~\citep{openai2024hello}
& - & 22.1 & 19.9 & 21.0 \\
Qwen2.5-VL~\citep{bai2025qwen2}
& 7B & 59.8 & 59.3 & 59.6 \\
\multirow{2}{*}{InternVL2.5~\citep{chen2024expanding}}
& 8B & 17.2 & 24.2 & 20.7 \\
& 26B & 14.8 & 16.6 & 15.7 \\
\midrule
\multicolumn{5}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (SFT)}}} \\
OS-Genesis~\citep{sun2024genesis}
& 7B & 8.3 & 5.8 & 7.1 \\
OS-Altas~\citep{wu2024atlas}
& 7B & 53.6 & 60.7 & 57.2 \\
Aguvis~\citep{xu2024aguvis}
& 7B & 60.8 & 76.5 & 68.7 \\
UI-TARS~\citep{qin2025ui}
& 7B & 56.8 & 66.7 & 61.8 \\
\midrule
\multicolumn{5}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (RFT)}}} \\
AgentCPM-GUI~\citep{zhang2025agentcpm}
& 8B & 79.1 & 76.5 & 77.8 \\
\midrule
\multicolumn{5}{l}{\cellcolor{MiOrange}{Ours}} \\
\multirow{2}{*}{HyperClick}
& 3B & 80.9 & 81.2 & \underline{81.0} \\
& 7B & 82.7 & 83.1 & \textbf{82.9} \\
\bottomrule
\end{tabular}
% }
\label{tab: cagui}
\end{table}
\begin{table}[htbp]
\centering
\caption{GUI grounding accuracy on the UI-Vision~\citep{nayak2025ui} benchmark over the Education (Ed.), Browsers (Br.), Development (De.), Productivity (Pr.), Creativity (Cr.), and Entertainment (En.) subtasks. \textbf{Bold} and \underline{underline} indicate the best and second-best results.}
\resizebox{\linewidth}{!}
{
\begin{tabular}{lccccccccccc}
\toprule
\multirow{3}{*}{\textbf{Model}} & \multirow{3}{*}{\textbf{Size}} & \multicolumn{3}{c}{\textbf{Setting}} & \multicolumn{6}{c}{\textbf{Category}} & \multirow{3}{*}{\textbf{Avg.}} \\
\cmidrule(lr){3-5}
\cmidrule(lr){6-11}
& & Basic & Functional & Spatial & Ed. & Br. & De. & Pr. & Cr. & En. \\
& & (1772) & (1772) & (1935) & (642) & (143) & (1090) & (1950) & (1462) & (192) \\
\midrule
\multicolumn{12}{l}{\cellcolor{MiOrange}{\textit{General Models}}} \\
GPT-4o~\citep{openai2024hello}
& - & 1.6 & 1.5 & 1.0 & 1.5 & 0.0 & 2.2 & 1.1 & 0.8 & 4.2 & 1.4 \\
Gemini-1.5-pro~\citep{team2024gemini}
& - & 0.8 & 0.3 & 0.6 & 0.5 & 0.6 & 0.9 & 0.5 & 0.4 & 0.0 & 0.6 \\
Claude~\citep{anthropic2024claude}
& - & 9.5 & 7.7 & 7.6 & 6.1 & 9.8 & 8.0 & 9.4 & 7.7 & 8.3 & 8.3 \\
Qwen2.5-VL~\citep{wang2024qwen2}
& 7B & 1.2 & 0.8 & 0.5 & 0.5 & 0.0 & 1.2 & 0.9 & 0.5 & 1.0 & 0.9\\
InternVL2.5~\citep{chen2024expanding}
& 8B & 2.5 & 2.8 & 1.0 & 1.1 & 7.0 & 3.0 & 1.8 & 1.2 & 5.2 & 2.1 \\
MiniCPM-V~\citep{yao2024minicpm}
& 8B & 7.1 & 5.3 & 1.5 & 3.0 & 16.8 & 5.4 & 3.8 & 2.1 & 13.0 & 4.3 \\
\midrule
\multicolumn{12}{l}{\cellcolor{MiOrange}{\textit{GUI-specific Models (SFT)}}} \\
CogAgent~\citep{hong2024cogagent}
& 9B & 12.0 & 12.2 & 2.6 & 8.7 & 11.2 & 8.6 & 10.3 & 5.6 & 15.6 & 8.9 \\
SeeClick~\citep{cheng2024seeclick}
& 9.6B & 9.4 & 4.7 & 2.1 & 4.2 & 13.3 & 7.3 & 4.3 & 4.0 & 11.0 & 5.4 \\
AriaUI~\citep{yang2024aria}
& 25.3B & 12.2 & 14.0 & 4.0 & 9.0 & 18.9 & 11.2 & 10.4 & 6.5 & 19.3 & 10.1 \\
ShowUI~\citep{lin2025showui}
& 2B & 8.1 & 7.7 & 2.1 & 3.7 & 13.3 & 7.5 & 6.5 & 2.5 & 15.6 & 5.9 \\
OS-Atlas~\citep{wu2024atlas}
& 7B & 12.2 & 11.2 & 3.7 & 8.7 & 16.8 & 10.3 & 9.2 & 5.6 & 16.2 & 9.0 \\
\multirow{2}{*}{UGround-V1~\citep{nayak2025ui}}
& 7B & 15.4 & 17.1 & 6.3 & 10.4 & 28.7 & 17.5 & 12.2 & 8.6 & 18.2 & 12.9 \\
& 72B & 27.9 & 26.7 & 14.9 & 22.4 & 35.7 & 27.6 & 21.6 & 18.3 & 38.0 & 23.2 \\
Aguvis~\citep{xu2024aguvis}
& 7B & 17.8 & 18.3 & 5.1 & 13.1 & 30.8 & 17.1 & 12.1 & 9.6 & 24.0 & 13.7 \\
\multirow{2}{*}{UI-TARS~\citep{qin2025ui}}
& 7B & 20.1 & 24.3 & 8.4 & 14.2 & 35.0 & 19.7 & 18.3 & 11.1 & 38.5 & 17.6 \\
& 72B & 31.4 & 30.5 & 14.7 & 24.8 & 40.5 & 27.9 & 26.8 & 26.8 & 17.8 & \underline{25.5} \\
\midrule
\multicolumn{12}{l}{\cellcolor{MiOrange}{\textit{Ours}}} \\
\multirow{2}{*}{HyperClick}
& 3B & 28.7 & 24.4 & 6.8 & 19.6 & 30.8 & 20.6 & 21.1 & 12.7 & 40.6 & 19.6 \\
& 7B & 35.3 & 32.1 & 11.0 & 24.3 & 47.6 & 26.5 & 27.1 & 18.3 & 50.0 & \textbf{25.7} \\
\bottomrule
\end{tabular}
% \begin{tabular}{lccccccccccccccccccccccc}
% \toprule
% \multirow{3}{*}{\textbf{Model}} & \multirow{3}{*}{\textbf{Size}} & \multicolumn{7}{c}{\textbf{Basic}} & \multicolumn{7}{c}{\textbf{Functional}} & \multicolumn{7}{c}{\textbf{Spatial}} & \multirow{3}{*}{\textbf{Avg.}}\\
% \cmidrule(lr){3-9}
% \cmidrule(lr){10-16}
% \cmidrule(lr){17-23}
% &
% & \multirow{2}{*}{\makecell{Ed \\(215)}}
% & \multirow{2}{*}{\makecell{Br \\(56)}}
% & \multirow{2}{*}{\makecell{De \\(376)}}
% & \multirow{2}{*}{\makecell{Pr \\(605)}}
% & \multirow{2}{*}{\makecell{Cr \\(438)}}
% & \multirow{2}{*}{\makecell{En \\(82)}}
% & \multirow{2}{*}{\makecell{Overall \\ (1772)}}
% & \multirow{2}{*}{\makecell{Ed \\(215)}}
% & \multirow{2}{*}{\makecell{Br \\(56)}}
% & \multirow{2}{*}{\makecell{De \\(376)}}
% & \multirow{2}{*}{\makecell{Pr \\(605)}}
% & \multirow{2}{*}{\makecell{Cr \\(438)}}
% & \multirow{2}{*}{\makecell{En \\(82)}}
% & \multirow{2}{*}{\makecell{Overall \\ (1772)}}
% & \multirow{2}{*}{\makecell{Ed \\(212)}}
% & \multirow{2}{*}{\makecell{Br \\(31)}}
% & \multirow{2}{*}{\makecell{De \\(338)}}
% & \multirow{2}{*}{\makecell{Pr \\(740)}}
% & \multirow{2}{*}{\makecell{Cr \\(586)}}
% & \multirow{2}{*}{\makecell{En \\(28)}}
% & \multirow{2}{*}{\makecell{Overall \\ (1935)}} \\
% & \\
% \midrule
% \multicolumn{24}{l}{\cellcolor{MiOrange}{\textit{General Models}}} \\
% GPT-4o~\citep{openai2024hello}
% & - & 2.2 & 0.0 & 1.9 & 1.2 & 1.1 & 4.9 & 1.6 & 1.4 & 0.0 & 3.2 & 0.8 & 0.9 & 3.7 & 1.5 & 0.9 & 0.0 & 1.5 & 1.2 & 0.5 & 3.6 & 1.0 & 1.4 \\
% \bottomrule
% \end{tabular}
}
\label{tab: ui_vision}
\end{table}
\begin{thebibliography}{76}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Anthropic(2024)]{anthropic2024claude}
Anthropic.
\newblock Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku, 2024.
\bibitem[Bai et~al.(2021)Bai, Zang, Xu, Sunkara, Rastogi, Chen, et~al.]{bai2021uibert}
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et~al.
\newblock Uibert: Learning generic multimodal representations for ui understanding.
\newblock \emph{arXiv preprint arXiv:2107.13731}, 2021.
\bibitem[Bai et~al.(2023)Bai, Bai, Yang, Wang, Tan, Wang, Lin, Zhou, and Zhou]{baiqwenvl}
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.
\newblock Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.
\newblock \emph{arXiv preprint arXiv:2308.12966}, 2023.
\bibitem[Bai et~al.(2025)Bai, Chen, Liu, Wang, Ge, Song, Dang, Wang, Wang, Tang, et~al.]{bai2025qwen2}
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et~al.
\newblock Qwen2. 5-vl technical report.
\newblock \emph{arXiv preprint arXiv:2502.13923}, 2025.
\bibitem[Chen et~al.(2024)Chen, Wang, Cao, Liu, Gao, Cui, Zhu, Ye, Tian, Liu, et~al.]{chen2024expanding}
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et~al.
\newblock Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.
\newblock \emph{arXiv preprint arXiv:2412.05271}, 2024.
\bibitem[Cheng et~al.(2024)Cheng, Sun, Chu, Xu, YanTao, Zhang, and Wu]{cheng2024seeclick}
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li~YanTao, Jianbing Zhang, and Zhiyong Wu.
\newblock {S}ee{C}lick: Harnessing {GUI} grounding for advanced visual {GUI} agents.
\newblock In \emph{Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, pp.\ 9313--9332, 2024.
\bibitem[Damani et~al.(2025)Damani, Puri, Slocum, Shenfeld, Choshen, Kim, and Andreas]{damani2025beyond}
Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas.
\newblock Beyond binary rewards: Training lms to reason about their uncertainty.
\newblock \emph{arXiv preprint arXiv:2507.16806}, 2025.
\bibitem[Dao(2023)]{dao2023flashattention}
Tri Dao.
\newblock Flashattention-2: Faster attention with better parallelism and work partitioning.
\newblock \emph{arXiv preprint arXiv:2307.08691}, 2023.
\bibitem[Desai \& Durrett(2020)Desai and Durrett]{desai2020calibration}
Shrey Desai and Greg Durrett.
\newblock Calibration of pre-trained transformers.
\newblock \emph{arXiv preprint arXiv:2003.07892}, 2020.
\bibitem[Ding et~al.(2025)Ding, Ni, and Bi]{ding2025lvlms}
Zhikai Ding, Shiyu Ni, and Keping Bi.
\newblock Do lvlms know what they know? a systematic study of knowledge boundary perception in lvlms.
\newblock \emph{arXiv preprint arXiv:2508.19111}, 2025.
\bibitem[Fu et~al.(2025)Fu, Wang, Tian, and Zhao]{fu2025deep}
Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao.
\newblock Deep think with confidence.
\newblock \emph{arXiv preprint arXiv:2508.15260}, 2025.
\bibitem[Galli et~al.(1994)Galli, Beucher, Le~Loc’h, Doligez, and Group]{galli1994pros}
A~Galli, H~Beucher, G~Le~Loc’h, B~Doligez, and Heresim Group.
\newblock The pros and cons of the truncated gaussian method.
\newblock In \emph{Geostatistical Simulations: Proceedings of the Geostatistical Simulation Workshop, Fontainebleau, France, 27--28 May 1993}, pp.\ 217--233. Springer, 1994.
\bibitem[Gauss(1809)]{gauss1809theoria}
Carl~Friedrich Gauss.
\newblock \emph{Theoria motus corporum coelestium in sectionibus conicis solem ambientium auctore Carolo Friderico Gauss}.
\newblock sumtibus Frid. Perthes et IH Besser, 1809.
\bibitem[Gauss(1877)]{gauss1877theoria}
Carl~Friedrich Gauss.
\newblock \emph{Theoria motus corporum coelestium in sectionibus conicis solem ambientium}, volume~7.
\newblock FA Perthes, 1877.
\bibitem[Glenn et~al.(1950)]{glenn1950verification}
W~Brier Glenn et~al.
\newblock Verification of forecasts expressed in terms of probability.
\newblock \emph{Monthly weather review}, 78\penalty0 (1):\penalty0 1--3, 1950.
\bibitem[Gou et~al.(2025)Gou, Wang, Zheng, Xie, Chang, Shu, Sun, and Su]{gou2025uground}
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu~Su.
\newblock Navigating the digital world as humans do: Universal visual grounding for {GUI} agents.
\newblock In \emph{Proceedings of the International Conference on Learning Representations}, 2025.
\bibitem[Guo et~al.(2017)Guo, Pleiss, Sun, and Weinberger]{guo2017calibration}
Chuan Guo, Geoff Pleiss, Yu~Sun, and Kilian~Q Weinberger.
\newblock On calibration of modern neural networks.
\newblock In \emph{Proceedings of the IEEE International Conference on Machine Learning}, pp.\ 1321--1330. PMLR, 2017.
\bibitem[Guo et~al.(2025{\natexlab{a}})Guo, Yang, Zhang, Song, Zhang, Xu, Zhu, Ma, Wang, Bi, et~al.]{guo2025deepseek}
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et~al.
\newblock Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2501.12948}, 2025{\natexlab{a}}.
\bibitem[Guo et~al.(2025{\natexlab{b}})Guo, Wu, Zhu, Leng, Shi, Chen, Fan, Wang, Jiang, Wang, et~al.]{guo2025seed1}
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et~al.
\newblock Seed1. 5-vl technical report.
\newblock \emph{arXiv preprint arXiv:2505.07062}, 2025{\natexlab{b}}.
\bibitem[He et~al.(2017)He, Gkioxari, Doll{\'a}r, and Girshick]{he2017mask}
Kaiming He, Georgia Gkioxari, Piotr Doll{\'a}r, and Ross Girshick.
\newblock Mask r-cnn.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer Vision}, pp.\ 2961--2969, 2017.
\bibitem[Hong et~al.(2024)Hong, Wang, Lv, Xu, Yu, Ji, Wang, Wang, Dong, Ding, et~al.]{hong2024cogagent}
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et~al.
\newblock Cogagent: A visual language model for gui agents.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 14281--14290, 2024.
\bibitem[Ji et~al.(2023{\natexlab{a}})Ji, Lee, Frieske, Yu, Su, Xu, Ishii, Bang, Madotto, and Fung]{ji2023survey}
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye~Jin Bang, Andrea Madotto, and Pascale Fung.
\newblock Survey of hallucination in natural language generation.
\newblock \emph{ACM computing surveys}, 55\penalty0 (12):\penalty0 1--38, 2023{\natexlab{a}}.
\bibitem[Ji et~al.(2023{\natexlab{b}})Ji, Yu, Xu, Lee, Ishii, and Fung]{ji2023towards}
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung.
\newblock Towards mitigating llm hallucination via self reflection.
\newblock In \emph{Findings of the Association for Computational Linguistics: EMNLP 2023}, pp.\ 1827--1843, 2023{\natexlab{b}}.
\bibitem[Kalai et~al.(2025)Kalai, Nachum, Vempala, and Zhang]{kalai2025language}
Adam~Tauman Kalai, Ofir Nachum, Santosh~S Vempala, and Edwin Zhang.
\newblock Why language models hallucinate.
\newblock \emph{arXiv preprint arXiv:2509.04664}, 2025.
\bibitem[Kapoor et~al.(2024)Kapoor, Butala, Russak, Koh, Kamble, AlShikh, and Salakhutdinov]{kapoor2024omniact}
Raghav Kapoor, Yash~Parag Butala, Melisa Russak, Jing~Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov.
\newblock Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web.
\newblock In \emph{Proceedings of the European Conference on Computer Vision}, pp.\ 161--178. Springer, 2024.
\bibitem[Kwon et~al.(2023)Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, and Stoica]{kwon2023efficient}
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody~Hao Yu, Joseph~E. Gonzalez, Hao Zhang, and Ion Stoica.
\newblock Efficient memory management for large language model serving with pagedattention.
\newblock In \emph{Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles}, 2023.
\bibitem[Li et~al.(2025)Li, Meng, Lin, Luo, Tian, Ma, Huang, and Chua]{li2025screenspot}
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua.
\newblock Screenspot-pro: Gui grounding for professional high-resolution computer use.
\newblock \emph{arXiv preprint arXiv:2504.07981}, 2025.
\bibitem[Li et~al.(2020)Li, Li, He, Zheng, Li, and Guan]{li2020widget}
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan.
\newblock Widget captioning: Generating natural language description for mobile user interface elements.
\newblock \emph{arXiv preprint arXiv:2010.04295}, 2020.
\bibitem[Lin et~al.(2025)Lin, Li, Gao, Yang, Wu, Bai, Lei, Wang, and Shou]{lin2025showui}
Kevin~Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan~Weixian Lei, Lijuan Wang, and Mike~Zheng Shou.
\newblock Showui: One vision-language-action model for gui visual agent.
\newblock In \emph{Proceedings of the Computer Vision and Pattern Recognition Conference}, pp.\ 19498--19508, 2025.
\bibitem[Lin et~al.(2022)Lin, Hilton, and Evans]{lin2022teaching}
Stephanie Lin, Jacob Hilton, and Owain Evans.
\newblock Teaching models to express their uncertainty in words.
\newblock \emph{arXiv preprint arXiv:2205.14334}, 2022.
\bibitem[Lin et~al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Doll{\'a}r, and Zitnick]{lin2014microsoft}
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll{\'a}r, and C~Lawrence Zitnick.
\newblock Microsoft coco: Common objects in context.
\newblock In \emph{Proceedings of the European conference on computer vision}, pp.\ 740--755. Springer, 2014.
\bibitem[Liu et~al.(2025{\natexlab{a}})Liu, Zhang, Zhang, and Lu]{liu2025ui}
Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, and Yan Lu.
\newblock Ui-e2i-synth: Advancing gui grounding with large-scale instruction synthesis.
\newblock \emph{arXiv preprint arXiv:2504.11257}, 2025{\natexlab{a}}.
\bibitem[Liu et~al.(2025{\natexlab{b}})Liu, Li, Xie, Hu, Han, Zhang, Yang, and Wu]{liu2025infigui}
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu.
\newblock Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners.
\newblock \emph{arXiv preprint arXiv:2504.14239}, 2025{\natexlab{b}}.
\bibitem[Long et~al.(2015)Long, Shelhamer, and Darrell]{long2015fully}
Jonathan Long, Evan Shelhamer, and Trevor Darrell.
\newblock Fully convolutional networks for semantic segmentation.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pp.\ 3431--3440, 2015.
\bibitem[Lu et~al.(2025)Lu, Chai, Guo, Yin, Liu, Wang, Xiao, Ren, Xiong, and Li]{lu2025ui}
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi~Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li.
\newblock Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2503.21620}, 2025.
\bibitem[Luo et~al.(2025)Luo, Wang, He, and Xia]{luo2025gui}
Run Luo, Lu~Wang, Wanwei He, and Xiaobo Xia.
\newblock Gui-r1: A generalist r1-style vision-language action model for gui agents.
\newblock \emph{arXiv preprint arXiv:2504.10458}, 2025.
\bibitem[MacKenzie(1988)]{mackenzie1988history}
Donald MacKenzie.
\newblock The history of statistics: the measurement of uncertainty before 1900 by stephen m. stigler.
\newblock \emph{Technology and Culture}, 29\penalty0 (2):\penalty0 299--300, 1988.
\bibitem[Manakul et~al.(2023)Manakul, Liusie, and Gales]{manakul2023selfcheckgpt}
Potsawee Manakul, Adian Liusie, and Mark~JF Gales.
\newblock Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.
\newblock \emph{arXiv preprint arXiv:2303.08896}, 2023.
\bibitem[Nayak et~al.(2025)Nayak, Jian, Lin, Rodriguez, Kalsi, Awal, Chapados, {\"O}zsu, Agrawal, Vazquez, et~al.]{nayak2025ui}
Shravan Nayak, Xiangru Jian, Kevin~Qinghong Lin, Juan~A Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M~Tamer {\"O}zsu, Aishwarya Agrawal, David Vazquez, et~al.
\newblock Ui-vision: A desktop-centric gui benchmark for visual perception and interaction.
\newblock \emph{arXiv preprint arXiv:2503.15661}, 2025.
\bibitem[Nguyen et~al.(2024)Nguyen, Chen, Wang, Wu, Park, Hu, Lyu, Wu, Aponte, Xia, et~al.]{nguyen2024gui}
Dang Nguyen, Jian Chen, Yu~Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu~Xia, et~al.
\newblock Gui agents: A survey.
\newblock \emph{arXiv preprint arXiv:2412.13501}, 2024.
\bibitem[Oberkampf et~al.(2002)Oberkampf, DeLand, Rutherford, Diegert, and Alvin]{oberkampf2002error}
William~L Oberkampf, Sharon~M DeLand, Brian~M Rutherford, Kathleen~V Diegert, and Kenneth~F Alvin.
\newblock Error and uncertainty in modeling and simulation.
\newblock \emph{Reliability Engineering \& System Safety}, 75\penalty0 (3):\penalty0 333--357, 2002.
\bibitem[OpenAI(2024)]{openai2024hello}
OpenAI.
\newblock Hello gpt-4o, 2024.
\bibitem[Qin et~al.(2025)Qin, Ye, Fang, Wang, Liang, Tian, Zhang, Li, Li, Huang, et~al.]{qin2025ui}
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et~al.
\newblock Ui-tars: Pioneering automated gui interaction with native agents.
\newblock \emph{arXiv preprint arXiv:2501.12326}, 2025.
\bibitem[Redmon et~al.(2016)Redmon, Divvala, Girshick, and Farhadi]{redmon2016you}
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.
\newblock You only look once: Unified, real-time object detection.
\newblock In \emph{Proceedings of the IEEE conference on computer vision and pattern recognition}, pp.\ 779--788, 2016.
\bibitem[Ren et~al.(2015)Ren, He, Girshick, and Sun]{ren2015faster}
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
\newblock Faster r-cnn: Towards real-time object detection with region proposal networks.
\newblock volume~28, 2015.
\bibitem[Schulman et~al.(2017)Schulman, Wolski, Dhariwal, Radford, and Klimov]{schulman2017proximal}
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
\newblock Proximal policy optimization algorithms.
\newblock \emph{arXiv preprint arXiv:1707.06347}, 2017.
\bibitem[Shao et~al.(2024)Shao, Wang, Zhu, Xu, Song, Bi, Zhang, Zhang, Li, Wu, et~al.]{shao2024deepseekmath}
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK~Li, Yang Wu, et~al.
\newblock Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
\newblock \emph{arXiv preprint arXiv:2402.03300}, 2024.
\bibitem[Shen et~al.(2025)Shen, Liu, Li, Fang, Ma, Liao, Shen, Zhang, Zhao, Zhang, et~al.]{shen2025vlm}
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et~al.
\newblock Vlm-r1: A stable and generalizable r1-style large vision-language model.
\newblock \emph{arXiv preprint arXiv:2504.07615}, 2025.
\bibitem[Sun et~al.(2024)Sun, Cheng, Ding, Jin, Wang, Xu, Wu, Jia, Chen, Liu, et~al.]{sun2024genesis}
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et~al.
\newblock Os-genesis: Automating gui agent trajectory construction via reverse task synthesis.
\newblock \emph{arXiv preprint arXiv:2412.19723}, 2024.
\bibitem[Tang et~al.(2025{\natexlab{a}})Tang, Gu, Lu, Liu, Shen, Meng, Wang, Zhang, Shen, Lu, et~al.]{tang2025gui}
Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, et~al.
\newblock Gui-g2: Gaussian reward modeling for gui grounding.
\newblock \emph{arXiv preprint arXiv:2507.15846}, 2025{\natexlab{a}}.
\bibitem[Tang et~al.(2025{\natexlab{b}})Tang, Xia, Wu, Hu, Chen, Chen, Xu, Wu, Lu, Ma, et~al.]{tang2025lpo}
Jiaqi Tang, Yu~Xia, Yi-Feng Wu, Yuwei Hu, Yuhui Chen, Qing-Guo Chen, Xiaogang Xu, Xiangyu Wu, Hao Lu, Yanqing Ma, et~al.
\newblock Lpo: Towards accurate gui agent interaction via location preference optimization.
\newblock \emph{arXiv preprint arXiv:2506.09373}, 2025{\natexlab{b}}.
\bibitem[Team et~al.(2024)Team, Georgiev, Lei, Burnell, Bai, Gulati, Tanzer, Vincent, Pan, Wang, et~al.]{team2024gemini}
Gemini Team, Petko Georgiev, Ving~Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et~al.
\newblock Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
\newblock \emph{arXiv preprint arXiv:2403.05530}, 2024.
\bibitem[Team et~al.(2025)Team, Du, Yin, Xing, Qu, Wang, Chen, Zhang, Du, Wei, et~al.]{team2025kimi}
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et~al.
\newblock Kimi-vl technical report.
\newblock \emph{arXiv preprint arXiv:2504.07491}, 2025.
\bibitem[Tian et~al.(2023)Tian, Mitchell, Zhou, Sharma, Rafailov, Yao, Finn, and Manning]{tian2023just}
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher~D Manning.
\newblock Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.
\newblock \emph{arXiv preprint arXiv:2305.14975}, 2023.
\bibitem[Wan et~al.(2024)Wan, Song, Yu, Liu, Cheng, Huang, Bai, Yao, and Yang]{wan2024omniparser}
Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang.
\newblock Omniparser: A unified framework for text spotting key information extraction and table recognition.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 15641--15653, 2024.
\bibitem[Wang et~al.(2024{\natexlab{a}})Wang, Bai, Tan, Wang, Fan, Bai, Chen, Liu, Wang, Ge, et~al.]{wang2024qwen2}
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et~al.
\newblock Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution.
\newblock \emph{arXiv preprint arXiv:2409.12191}, 2024{\natexlab{a}}.
\bibitem[Wang et~al.(2024{\natexlab{b}})Wang, Liu, Chen, Zhou, Gan, Zeng, Che, Yu, Hao, Shao, et~al.]{wang2024gui}
Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, et~al.
\newblock Gui agents with foundation models: A comprehensive survey.
\newblock \emph{arXiv preprint arXiv:2411.04890}, 2024{\natexlab{b}}.
\bibitem[Wang et~al.(2025)Wang, Wu, Xie, Ding, Yang, Li, Liu, Li, Dong, Chen, et~al.]{wang2025mmbench}
Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, et~al.
\newblock Mmbench-gui: Hierarchical multi-platform evaluation framework for gui agents.
\newblock \emph{arXiv preprint arXiv:2507.19478}, 2025.
\bibitem[Wu et~al.(2025)Wu, Cheng, Yang, Zhang, Yang, Jiang, Mu, Peng, Qiao, Tan, et~al.]{wu2025gui}
Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo~Qiao, Reuben Tan, et~al.
\newblock Gui-actor: Coordinate-free visual grounding for gui agents.
\newblock \emph{arXiv preprint arXiv:2506.03143}, 2025.
\bibitem[Wu et~al.(2024)Wu, Wu, Xu, Wang, Sun, Jia, Cheng, Ding, Chen, Liang, et~al.]{wu2024atlas}
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul~Pu Liang, et~al.
\newblock Os-atlas: A foundation action model for generalist gui agents.
\newblock 2024.
\bibitem[Xiaomi(2025)]{coreteam2025mimovltechnicalreport}
LLM-Core-Team Xiaomi.
\newblock Mimo-vl technical report, 2025.
\newblock URL \url{https://arxiv.org/abs/2506.03569}.
\bibitem[Xie et~al.(2025)Xie, Deng, Li, Yang, Wu, Chen, Hu, Wang, Xu, Wang, et~al.]{xie2025scaling}
Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et~al.
\newblock Scaling computer-use grounding via user interface decomposition and synthesis.
\newblock \emph{arXiv preprint arXiv:2505.13227}, 2025.
\bibitem[Xiong et~al.(2023)Xiong, Hu, Lu, Li, Fu, He, and Hooi]{xiong2023can}
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi.
\newblock Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.
\newblock \emph{arXiv preprint arXiv:2306.13063}, 2023.
\bibitem[Xu et~al.(2024)Xu, Wang, Wang, Lu, Xie, Saha, Sahoo, Yu, and Xiong]{xu2024aguvis}
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong.
\newblock Aguvis: Unified pure vision agents for autonomous gui interaction.
\newblock \emph{arXiv preprint arXiv:2412.04454}, 2024.
\bibitem[Yang et~al.(2024{\natexlab{a}})Yang, Wang, Li, Luo, Chen, Huang, and Li]{yang2024aria}
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li.
\newblock Aria-ui: Visual grounding for gui instructions.
\newblock \emph{arXiv preprint arXiv:2412.16256}, 2024{\natexlab{a}}.
\bibitem[Yang et~al.(2024{\natexlab{b}})Yang, Chern, Qiu, Neubig, and Liu]{yang2024alignment}
Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu.
\newblock Alignment for honesty.
\newblock volume~37, pp.\ 63565--63598, 2024{\natexlab{b}}.
\bibitem[Yao et~al.(2024)Yao, Yu, Zhang, Wang, Cui, Zhu, Cai, Li, Zhao, He, et~al.]{yao2024minicpm}
Yuan Yao, Tianyu Yu, Ao~Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et~al.
\newblock Minicpm-v: A gpt-4v level mllm on your phone.
\newblock \emph{arXiv preprint arXiv:2408.01800}, 2024.
\bibitem[Yu et~al.(2025)Yu, Yang, Wan, Song, Tang, Cheng, Liu, and Bai]{yu2025omniparser}
Wenwen Yu, Zhibo Yang, Jianqiang Wan, Sibo Song, Jun Tang, Wenqing Cheng, Yuliang Liu, and Xiang Bai.
\newblock Omniparser v2: Structured-points-of-thought for unified visual text parsing and its generality to multimodal large language models.
\newblock \emph{arXiv preprint arXiv:2502.16161}, 2025.
\bibitem[Yuan et~al.(2025)Yuan, Zhang, Li, Cai, Yao, Chen, Wang, Hou, Chen, Jiang, et~al.]{yuan2025enhancing}
Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et~al.
\newblock Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning.
\newblock \emph{arXiv preprint arXiv:2505.12370}, 2025.
\bibitem[Zhang et~al.(2025{\natexlab{a}})Zhang, Shang, Gao, Zhang, Xie, Ma, Yuan, Wu, Zhu, and Li]{zhang2025tongui}
Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song-Chun Zhu, and Qing Li.
\newblock Tongui: Building generalized gui agents by learning from multimodal web tutorials.
\newblock \emph{arXiv preprint arXiv:2504.12679}, 2025{\natexlab{a}}.
\bibitem[Zhang et~al.(2024)Zhang, He, Qian, Li, Li, Qin, Kang, Ma, Liu, Lin, et~al.]{zhang2024large}
Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si~Qin, Yu~Kang, Minghua Ma, Guyue Liu, Qingwei Lin, et~al.
\newblock Large language model-brained gui agents: A survey.
\newblock \emph{arXiv preprint arXiv:2411.18279}, 2024.
\bibitem[Zhang et~al.(2023)Zhang, Li, Das, Malin, and Kumar]{zhang2023sac3}
Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley~A Malin, and Sricharan Kumar.
\newblock Sac3: reliable hallucination detection in black-box language models via semantic-aware cross-check consistency.
\newblock \emph{arXiv preprint arXiv:2311.01740}, 2023.
\bibitem[Zhang et~al.(2025{\natexlab{b}})Zhang, Zhang, Fu, Wang, Yang, Du, Cui, Qin, Huang, Luo, et~al.]{zhang2025btl}
Shaojie Zhang, Ruoceng Zhang, Pei Fu, Shaokang Wang, Jiahui Yang, Xin Du, Shiqi Cui, Bin Qin, Ying Huang, Zhenbo Luo, et~al.
\newblock Btl-ui: Blink-think-link reasoning model for gui agent.
\newblock \emph{arXiv preprint arXiv:2509.15566}, 2025{\natexlab{b}}.
\bibitem[Zhang et~al.(2025{\natexlab{c}})Zhang, Lu, Fu, Huo, Yang, Wu, Si, Cong, Chen, Lin, et~al.]{zhang2025agentcpm}
Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, et~al.
\newblock Agentcpm-gui: Building mobile-use agents with reinforcement fine-tuning.
\newblock \emph{arXiv preprint arXiv:2506.01391}, 2025{\natexlab{c}}.
\bibitem[Zhou et~al.(2025)Zhou, Dai, Wang, Zhou, Jia, and Xu]{zhou2025gui}
Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu.
\newblock Gui-g1: Understanding r1-zero-like training for visual grounding in gui agents.
\newblock \emph{arXiv preprint arXiv:2505.15810}, 2025.
\bibitem[Zhu et~al.(2025)Zhu, Wang, Chen, Liu, Ye, Gu, Tian, Duan, Su, Shao, et~al.]{zhu2025internvl3}
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et~al.
\newblock Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.
\newblock \emph{arXiv preprint arXiv:2504.10479}, 2025.
\end{thebibliography}