Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents
Extracted source text · arXiv 2607.18659
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% arXiv version — USENIX Security template
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\documentclass[letterpaper,twocolumn,10pt]{article}
\usepackage{usenix}
\usepackage{tikz}
\usepackage{amsmath}
\usepackage{filecontents}
\usepackage{epsfig}
\usepackage{endnotes}
\usepackage{fontawesome}
\usepackage{graphicx}
\graphicspath{ {./images/} }
\usepackage{caption}
\usepackage{subcaption}
\captionsetup[table]{position=above,skip=4pt}
\captionsetup[figure]{position=below,skip=4pt}
\usepackage{listings}
\usepackage{msc}
\usepackage{svg}
\usepackage{comment}
\usepackage{xcolor}
\usepackage{xspace}
\usepackage{booktabs}
\usepackage{colortbl}
\usepackage{multirow}
\usepackage{lscape}
\usepackage{rotating}
\usepackage{float}
\usepackage{url}
\usepackage{seqsplit}
\usepackage{color}
\definecolor{lightgray}{rgb}{.9,.9,.9}
\definecolor{darkgray}{rgb}{.4,.4,.4}
\definecolor{purple}{rgb}{0.65, 0.12, 0.82}
\usetikzlibrary{arrows.meta, positioning, fit, calc}
\usepackage{tcolorbox}
\tcbuselibrary{listingsutf8}
\usepackage{amssymb}
\usepackage{pifont}
\usepackage{makecell}
\definecolor{bypass}{RGB}{220,245,220}
\definecolor{fullpass}{RGB}{140,200,140}
\definecolor{promptbg}{rgb}{0.97,0.97,0.97}
\definecolor{promptborder}{rgb}{0.2,0.6,0.86}
\newtcblisting{promptbox}[2][]{
colback=promptbg,
colframe=promptborder,
listing only,
listing options={
basicstyle=\ttfamily\small,
breaklines=true,
columns=fullflexible,
language=,
escapeinside=||,
},
left=0.5mm,
right=0.5mm,
top=0.5mm,
bottom=0.5mm,
boxrule=0.8pt,
arc=2pt,
enhanced,
title={#2},
#1
}
\newcommand{\circled}[1]{%
\tikz[baseline=(char.base)]{
\node[shape=circle, draw, fill=black, text=white, scale=0.7, inner sep=2pt] (char) {#1};}}
\lstdefinelanguage{JavaScript}{
keywords={typeof, new, true, false, catch, function, return, null, catch, switch, var, if, in, while, do, else, case, break},
keywordstyle=\color{blue}\bfseries,
ndkeywords={class, export, boolean, throw, implements, import, this},
ndkeywordstyle=\color{darkgray}\bfseries,
identifierstyle=\color{black},
sensitive=false,
comment=[l]{//},
morecomment=[s]{/*}{*/},
commentstyle=\color{purple}\ttfamily,
stringstyle=\color{red}\ttfamily,
morestring=[b]',
morestring=[b]"
}
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.95,0.95,0.92}
\newcommand{\cmark}{\textbf{\ding{51}}}
\newcommand{\xmark}{\textcolor{gray!60}{\ding{55}}}
\newcommand{\ph}[1]{{\color{red}$wip$} }
\newcommand{\ak}[1]{$<$ \textbf{\textit{AK: }}{\color{blue} #1} $>$}
\newcommand{\bo}[1]{$<$ \textbf{\textit{BO: }}{\color{olive} #1} $>$}
\newcommand{\lr}[1]{$<$ \textbf{\textit{LR: }}{\color{codepurple} #1} $>$}
\newcommand{\todo}[1]{{\color{red} \textsf{\bf [TODO: #1]}}}
\newcommand{\fix}[1]{{\color{blue} \textsf{\bf [FIX: #1]}}}
\newcommand{\update}[1]{{\color{magenta} #1}}
\def\checkmark{\tikz\fill[scale=0.4](0,.35) -- (.25,0) -- (1,.7) -- (.25,.15) -- cycle;}
\floatplacement{table}{ht}
\floatplacement{figure}{ht}
\lstdefinestyle{mystyle}{
backgroundcolor=\color{backcolour},
commentstyle=\color{codegreen},
keywordstyle=\color{magenta},
numberstyle=\tiny\color{codegray},
stringstyle=\color{codepurple},
basicstyle=\ttfamily\footnotesize,
breakatwhitespace=false,
breaklines=true,
captionpos=b,
keepspaces=true,
numbers=left,
numbersep=5pt,
showspaces=false,
showstringspaces=false,
showtabs=false,
tabsize=2
}
\lstset{style=mystyle}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\begin{document}
\title{Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents}
\author{
{\rm Behzad Ousat, Nikita Turkmen, Lalchandra Rampersaud, Dillan Bailey, and Amin Kharraz}\\
\{bousat, nturk005, lramp004, dbail042, mkharraz\}@fiu.edu \\ \\
Florida International University
}
\maketitle
\thispagestyle{plain}
\pagestyle{plain}
\input{01-abstract}
\input{02-introduction}
\input{03-background}
\input{04-methodology}
\input{05-experiments}
\input{06-discussion}
\input{09-conclusion}
\newpage
\bibliographystyle{plain}
\bibliography{refs}
\input{10-appendix}
\end{document}
\begin{abstract}
LLM-based browser agents are rapidly changing the threat landscape for web security. Unlike traditional automation frameworks that execute predefined scripts, these agents can autonomously navigate websites, reason about page content, and interact with web interfaces using natural-language instructions. This evolution raises fundamental questions about the effectiveness of bot management systems, widely deployed to defend against automated web abuse.
In this paper, we present a systematic measurement study evaluating the resilience of both interactive challenge-based defenses and non-interactive trust-based defenses against two attacker classes: commercial Captcha-solving services and LLM-based browser agents. Our evaluation spans seven solver services and six agents, including cloud-hosted, self-hosted, AI-assisted, and browser-extension configurations, tested against hCaptcha, reCaptcha v2, reCaptcha v3, and Cloudflare Turnstile.
Our results show that challenge-based defenses are broadly ineffective against commercial solvers, which achieve near-perfect bypass at negligible cost. The challenges can similarly be defeated by LLM-based agents when a dedicated solver module is available.
Non-interactive defenses such as reCaptcha v3 exhibit stronger resistance, but our analysis reveals that this resilience does not reflect a fundamental security property. Through fine-grained interaction trace analysis, we find that two agents with nearly indistinguishable behavioral footprints yield divergent outcomes, one bypassing the defense and one failing, isolating execution-environment authenticity, rather than agent behavior, as the determining factor. These findings suggest that the security boundary of non-interactive defenses lies at the environment layer, with significant implications for how bot management systems are designed and evaluated.
\end{abstract}
\section{Introduction}
\noindent Modern web defenses increasingly depend on mechanisms that distinguish legitimate human use from automated abuse. Challenge-based Captchas and non-interactive bot-management systems~\cite{google_recaptcha_about} have been central to this defense, reducing the impact of many common forms of scripted and high-volume automated traffic. These systems have also adapted over time by combining explicit challenges with behavioral analysis, browser execution signals, reputation information, and risk-based decision-making.
This adaptation has sustained the competition between attackers and defenders, but the boundary these systems rely on is becoming less stable. Many deployed defenses assume that explicit challenges impose meaningful friction and that automated actors remain behaviorally distinguishable from legitimate users. Third-party solver services weaken the first assumption by allowing attackers to outsource Captcha challenges at low cost~\cite{gao2022demystifying,jin2023secure}.
Credential stuffing, account takeover, and large-scale abuse campaigns routinely use these services as an operational step, allowing high-volume automation to continue despite deployed anti-bot controls~\cite{barkworth2022detecting,singh2024effective,li2021good}.
More recently, LLM-based browser agents challenge the second by operating through ordinary web interfaces: they can navigate websites, interpret page content, submit forms, adapt to feedback, and continue multi-step workflows from natural-language instructions. As a result, modern automation is no longer limited to brittle scripts that fail at hard interaction points; it can combine solver services, real-browser execution, feedback adaptation, and goal-directed task completion.
In this paper, we first establish what commercial solver services can and cannot defeat across seven web bot management systems, identifying their effective ceiling for both challenge-based and non-interactive systems (RQ1). We then evaluate six LLM-based browser agents spanning cloud-hosted, self-hosted, AI-assisted browser, and browser-extension deployment modes to characterize how off-the-shelf agents perform against the same configurations (RQ2). Finally, we investigate why default agents fail against non-interactive defenses (RQ3).
The results reveal that solver services achieve near-perfect bypass against challenge-based Captchas at costs as low as \$0.10 per 1,000 solves, making these defenses largely ineffective against cost-driven adversaries. However, they reach only 23\% success on average against reCaptcha v3.
On the other hand, the evaluated LLM-based browser agents generally fail on the challenge-based defenses, unless in cases where a dedicated Captcha solver module is available~\cite{skyvern_captcha_blog}. The same agents mostly fail on non-interactive defenses too, not because they cannot complete the required interaction, but because their browser execution environments produce detectable artifacts that the system flags.
We show that this failure is not a durable security property, but rather a consequence of environmental authenticity signals independent of agent behavior. Our interaction analysis reveals that behavioral similarity does not guarantee bypass: Browser-Use and NanoBrowser produce nearly indistinguishable event traces, yet only NanoBrowser, which operates within a real browser profile with persistent cookies, browsing history, and stable fingerprinting signals, consistently achieves bypass. This indicates that the trust gap originates not from agent behavior, but from the absence of long-term environmental legitimacy in clean, instrumented browser environments.
\noindent In summary, this paper makes the following contributions:
\noindent \textbf{First}, we present a systematic measurement study of seven commercial solver services against different configurations of hCaptcha, reCaptcha v2, reCaptcha v3, and Cloudflare Turnstile, covering seven overall settings with varying difficulty and interaction requirements. Our results show that challenge-based defenses are almost universally defeated at negligible cost, while non-interactive defenses such as reCaptcha v3 exhibit meaningful resistance against solver services.
\noindent \textbf{Second}, we evaluate six LLM-based browser agents against the same defense configurations, finding that agents consistently fail on challenge-based defenses without dedicated solver modules, and similarly fail on non-interactive defenses despite successfully completing the required interaction workflows. This establishes that the agent failure boundary is not task comprehension, but execution environment detectability.
\noindent \textbf{Third}, we investigate the root cause of this failure through fine-grained interaction trace analysis, comparing Browser-Use and NanoBrowser, two agents with nearly indistinguishable behavioral footprints, and finding that environmental authenticity signals, rather than behavioral similarity, determine bypass success. This reveals that the resilience of non-interactive defenses is an artifact of execution environment properties rather than a fundamental security guarantee.
% \noindent \textbf{Open Science.}
% \noindent The experiment configuration templates and other artifacts are available at \href{https://anonymous.4open.science/r/llm-solvers-6E43/}{\texttt{\seqsplit{https://anonymous.4open.science/r/llm-solvers-6E43/}}}.
% \noindent To reduce misuse risk, public artifacts exclude target-specific automation recipes.
\section{Background and Related Work}
\label{sec:background}
% \subsection{Threat Model}
\subsection{Automated Traffic and Contemporary Defenses}
Automated web traffic spans a wide spectrum of intent. Beneficial bots underpin critical
infrastructure such as web crawlers, conversational agents, and monitoring
systems~\cite{khder2021web, kucherbaev2018human, yamanoue2017monitoring}, while malicious
automation drives abuse, including spam, credential stuffing, click fraud, and large-scale
vulnerability scanning~\cite{ferrara2019history, barkworth2022detecting, li2021good,
kondracki2022uninvited}. Li et al.~\cite{li2021good} found that 57\% of bots observed over seven months were plainly malicious, exploiting newly disclosed vulnerabilities within hours of public disclosure and evading detection via browser modification, spoofed User-Agent strings, and proxy routing.
Recent incident reports on account takeover and credential stuffing campaigns reveal the scale and structure of this abuse.
Akamai recorded 193~billion credential stuffing attempts in 2024, a 66\% increase over the prior year~\cite{akamai2024soti}.
Analysis of 22 active credential-stuffing groups found that 65\% of attacks leveraged Captcha solver services and residential proxies, and that 85\% of targeted organizations had bot detection in place yet were still successfully compromised~\cite{kasada2025ato}.
Furthermore, Verizon's 2025 DBIR corroborates this at the breach level: credential stuffing constituted 19\% of all authentication attempts on a median daily basis in SSO provider logs, and stolen credentials were the initial access vector in 22\% of confirmed breaches~\cite{verizon2025dbir_stuffing}.
% In these campaigns, Captcha bypass is a key operational enabler that allows high-volume abuse to continue despite visible anti-bot controls.
Understanding why these attacks persist requires examining the defenses they are designed to defeat.
Bot management systems remain a primary countermeasure for distinguishing human users from automated clients at web interaction points.
These defenses can be categorized into two main groups. Challenge-based systems rely on explicit user interaction; reCaptcha v2 and hCaptcha are two popular challenge-based defenses that can render image recognition or puzzle tasks and are often tuned by operators to balance user friction with abuse resistance.
% In operational settings, these systems are exposed to outsourcing and API-mediated solving economies~\cite{motoyama2010re,weng2019towards,gao2022demystifying,jin2023secure}.
Non-interactive defenses, on the other hand, reduce explicit challenges and rely more on background telemetry. These systems emphasize environmental integrity checks and behavioral scoring with site-specific thresholds. This shift changes the attack surface from ``can the bot solve a puzzle'' to ``can the automation stack look sufficiently human and trustworthy.''
\subsection{Automated Solvers and Solving Services}
A substantial body of work demonstrates that the web bot management systems are vulnerable to automated solvers across all major challenge types. Text-based Captcha schemes have been broken using GANs, CNNs, and RNNs, with success rates exceeding 94\%~\cite{zhang2022counteracting, ye2018yet, tang2018research, dionysiou2020sok}. Image-based reCaptcha challenges have similarly been defeated, with deep learning-based solvers achieving 70\% accuracy by exploiting image repetition~\cite{sivakorn2016m}, and audio variants bypassed via off-the-shelf speech recognition~\cite{ieee11129490}.
Even reCaptcha~v3's behavioral scoring has been compromised using reinforcement learning: Tsingenopoulos~et~al.~\cite{tsingenopoulos2022captcha} trained an agent over months of score feedback to simulate human-like browsing, achieving evasion rates up to 99.6\%. Across all these approaches, the common bottleneck is data dependency, requiring each solver to be trained or fine-tuned as providers update their challenge pools.
Automated crawlers and solvers work together to bypass the defense; the crawler detects a Captcha challenge, delegates it to the solver, and resumes automated traversal upon receiving a solution or a valid token when required. In~\cite{ousat2024matter}, the authors create a pipeline to integrate a pre-trained text-based Captcha solver into a real-world crawler and achieve 63\% successful bypass in only five attempts for each target page.
In practice, most malicious actors are cost-driven and will gravitate toward whichever bypass method minimizes overhead and cost. Commodity solver services meet this demand by abstracting the problem entirely via hybrid human/automation pipelines~\cite{motoyama2010re, weng2019towards, gao2022demystifying, jin2023secure}, eliminating the need for any model expertise.
The user simply submits a site key, page URL, and challenge type to an API, polls for a solution token, and injects it into the target form. This capability is even integrated into open-source frameworks such as OpenBullet~2, making bypass a default feature of commodity credential-stuffing toolkits with no specialist expertise required~\cite{startupdefense2026stuffing}.
% \begin{figure}[t]
% \centering
% \includegraphics[width=\linewidth]{images/Third-Party Solver Sequence Diagram.pdf}
% \caption{Sequence diagram for third-party Captcha solver services.}
% \label{fig:3rd_party_solver_sequence_diagram}
% \end{figure}
% \begin{figure}[t]
% \centering
% \resizebox{\columnwidth}{!}{
% \begin{tikzpicture}[
% font=\normalsize,
% box/.style={
% draw,
% align=center,
% inner sep=6pt,
% minimum width=2.6cm,
% minimum height=0.8cm
% },
% arrow/.style={
% -{Latex[length=1.6mm]},
% thick
% },
% lifeline/.style={
% dashed,
% thick
% }
% ]
% % ── Actors (header boxes) ────────────────────────────────────────────
% \node[box] (web) at (0, 0) {\textbf{Target}\\\textbf{Web Page}};
% \node[box] (crawler) at (5.5, 0) {\textbf{Automated}\\\textbf{Crawler}};
% \node[box] (solver) at (11, 0) {\textbf{Third-Party}\\\textbf{Solver}};
% % ── Lifelines ────────────────────────────────────────────────────────
% \draw[lifeline] (web.south) -- ++(0,-9.5);
% \draw[lifeline] (crawler.south) -- ++(0,-9.5);
% \draw[lifeline] (solver.south) -- ++(0,-9.5);
% % ── Step 1: Visit Page ───────────────────────────────────────────────
% \draw[arrow] ([yshift=-1.0cm]crawler.south)
% -- node[above,font=\small]
% {\textbf{\textcircled{\raisebox{-0.9pt}{1}}}~~Visit Page}
% ([yshift=-1.0cm]web.south);
% % ── Step 2: Get Parameters ───────────────────────────────────────────
% \draw[arrow] ([yshift=-2.5cm]web.south)
% -- node[below,font=\small,align=left]
% {\textbf{\textcircled{\raisebox{-0.9pt}{2}}}~~Get Parameters\\
% \hspace*{1.4em}$-$ SiteKey\\
% \hspace*{1.4em}$-$ Captcha Type}
% ([yshift=-2.5cm]crawler.south);
% % ── Step 3: Create Task ──────────────────────────────────────────────
% \draw[arrow] ([yshift=-4.8cm]crawler.south)
% -- node[above,font=\small,align=left]
% {\textbf{\textcircled{\raisebox{-0.9pt}{3}}}~~Create Task\\
% \hspace*{1.4em}$-$ SiteKey\\
% \hspace*{1.4em}$-$ Captcha Type\\
% \hspace*{1.4em}$-$ Service Parameters}
% ([yshift=-4.8cm]solver.south);
% % ── Step 4: Send Solution ────────────────────────────────────────────
% \draw[arrow] ([yshift=-6.6cm]solver.south)
% -- node[above,font=\small,align=left]
% {\textbf{\textcircled{\raisebox{-0.9pt}{4}}}~~Send Solution\\
% \hspace*{1.4em}$-$ Token}
% ([yshift=-6.6cm]crawler.south);
% % ── Step 5: Use the Solution ─────────────────────────────────────────
% \draw[arrow] ([yshift=-7.8cm]crawler.south)
% -- node[above,font=\small,align=left]
% {\textbf{\textcircled{\raisebox{-0.9pt}{5}}}~~Use the solution\\
% \hspace*{1.4em}$-$ Form Values\\
% \hspace*{1.4em}$-$ Token}
% ([yshift=-7.8cm]web.south);
% % ── Step 6: Send Results ─────────────────────────────────────────────
% \draw[arrow] ([yshift=-9.2cm]web.south)
% -- node[above,font=\small]
% {\textbf{\textcircled{\raisebox{-0.9pt}{6}}}~~Send Results:
% Success or Failure}
% ([yshift=-9.2cm]crawler.south);
% \end{tikzpicture}
% }
% \caption{Sequence diagram of a third-party Captcha solver integrated into an automated crawler. The crawler extracts challenge parameters, delegates solving to the service, and injects the returned token before submitting the target form.}
% \label{fig:3rd_party_solver_sequence_diagram}
% \end{figure}
% The full solver workflow is detailed in Section~\ref{sec:solver-service-eval-pipeline}.
\subsection{LLM-Based Solvers and Browser Agents}
LLMs reduce the cost and complexity of bypass even further. Recent work shows that LLM-based solvers can circumvent traditional Captchas without challenge-specific training~\cite{wang2023bot}, and that pairing LLMs with domain-specific scaffolding extends this to reasoning-based challenges~\cite{ding2025illusioncaptcha}.
Similarly, \cite{deng2024oedipus} describes an end-to-end framework for automated reasoning Captcha solving decomposes human-easy, AI-hard challenges into a sequence of simpler LLM-solvable sub-steps using chain-of-thought prompting, achieving an average success rate of 63.5\% and showing adaptability to new challenge designs.
Beyond direct solving, the rise of the agentic web, where LLM-backed agents autonomously navigate, interpret, and act on web content~\cite{yang2025agenticweb}, introduces a broader threat surface. These systems couple an LLM backend with a browser controller to close an iterative perception-planning-execution loop (Figure~\ref{fig:llm_browser_agent_pipeline}), and while designed for legitimate automation, they can be readily repurposed for adversarial tasks. Teoh~et~al.~\cite{teohcaptchas} and Luo~et~al.~\cite{luo2025open} demonstrate this directly, showing that LLM agents can frame image-based challenges as optimization problems and solve reCaptcha~v2 and hCaptcha successfully. Li~et~al.~\cite{li2025webcloak} further illustrate the scale of the threat: evaluating 32 scraper variants across real-world webpages, they find that LLM-powered frameworks achieve visual content exfiltration recall rates of 88.7\% with minimal human effort. That work characterizes what agents can \emph{extract} once on a page.
Our paper addresses the upstream question of how agents can \emph{bypass} the bot management barriers that gate access in the first place.
\begin{figure}[t]
\centering
% \includegraphics[width=\linewidth]{images/LLM Based Browser Agents Diagram.pdf}
\includegraphics[width=\linewidth]{images/Short_LLM_Based_Browser_Agents_Diagram.pdf}
\caption{LLM-based Browser Agents Workflow Diagram. A legitimate LLM-powered browser can be repurposed for malicious activities at scale.}
\label{fig:llm_browser_agent_pipeline}
\end{figure}
\section{Threat Model and Research Questions}
% We define the threat model to clarify how the tools evaluated in this paper fit within the adversarial scenario and what capabilities are assumed on each side.
\subsection{Threat Model}
We model an economically motivated web attacker who seeks to bypass the bot management barrier to execute a downstream automated action, such as login attempts, form submissions, or large-scale scraping.
The attacker can: (i) call commercial solver APIs, (ii) run cloud or self-hosted LLM browser agents, and (iii) automate repeated trials under a fixed budget.
We do not aim to propose a new solver model. We systematically analyze end-to-end bypass feasibility across two stacks, including commercial solver services and LLM-based browser agents.
\subsection{Research Questions}
\label{sec:rq}
This paper investigates how modern automation stacks can evade web bot management systems, including both interactive and non-interactive defenses. We first characterize commodity attacks via commercial solver services (RQ1), then evaluate LLM-based browser agents in their default configurations across multiple deployment models (RQ2), and finally investigate what limits agent effectiveness (RQ3).
% \begin{itemize}
\noindent \textbf{RQ1.} How effective are commercial third-party solver services across
challenge-based, non-interactive, and behavioral defense systems, and where does the
solver ecosystem reach its ceiling?
\noindent \textbf{RQ2.} How do off-the-shelf LLM-based browser agents perform in bypassing
web bot management systems across cloud, self-hosted, AI-assisted browser, and browser-extension deployment modes?
\noindent \textbf{RQ3.} What factors limit the effectiveness of off-the-shelf LLM-based
browser agents against modern defenses?
% \end{itemize}
% RQ4. Do targeted case studies on live websites reveal gaps between nominal reCaptcha v3 deployment and effective enforcement, and what failure modes are most consequential in practice?
\begin{figure}[t]
\centering
\includegraphics[width=0.9\linewidth]{images/Captcha_Login_Sample_Combined.pdf}
\caption{Sample Login Pages Protected with hCaptcha and reCaptcha v2.}
\label{fig:sample_login_v2_hCaptcha}
\end{figure}
\section{Experiments Setup}
% To empirically evaluate the capabilities of third-party services and LLM-based browser agents against popular defense solutions, we designed a series of experiments.
This section details the evaluation environment and the systems under evaluation, including the selected defenses, commercial services, and LLM-based agents.
\subsection{Bot Mitigation Deployments}
\label{sec:selected_captcha}
We constructed a controlled public testbed consisting of seven protected web applications, each deployed on a dedicated subdomain and configured with a distinct bot-mitigation setting. Each application implements a standard HTML form protected by one of the defense configurations listed below. Holding the application logic constant allows us to attribute differences in agent behavior and success rates to the deployed defense mechanism rather than to variation in page structure, task complexity, or backend behavior.
%We deployed seven distinct websites on a dedicated server with public access, ensuring a stable and representative evaluation environment. Each site is an application containing a standard HTML form protected by one of the seven defense configurations described below.
Configurations that differ in difficulty level or interaction mode (e.g., hCaptcha Easy/Hard, Turnstile Invisible/Default) are deployed as separate subdomains, with the appropriate site key and difficulty parameter set on the backend. This controlled setup allows us to monitor interactions in isolation without affecting public services.
\noindent \textbf{hCaptcha}~\cite{hcaptcha_about} is deployed in two difficulty tiers that share a common challenge interface but differ in task complexity. The \textit{Easy} tier consists of relatively unambiguous object-recognition tasks that can typically be solved by human users within seconds. The \textit{Hard} tier introduces more ambiguous, fine-grained image classification and puzzles intended to increase resistance against automated solvers while preserving human solvability. This higher complexity also increases the operational cost of outsourcing attacks, making the system harder to commoditize.
\noindent \textbf{reCaptcha v2}~\cite{recaptcha_v2_about} is deployed in two variants that share the same underlying risk-scoring engine but differ in how they present verification to the user. The \textit{Checkbox} variant requires an explicit ``I'm not a robot'' interaction; when background signals are inconclusive, an additional image-grid challenge is surfaced. The \textit{Invisible} variant presents no visible widget: verification runs entirely in the background during normal page interaction, and a grid-image challenge is only triggered when the risk score is high. The Invisible mode is increasingly adopted by sites seeking passive protection without interrupting user flow.
% Despite being relatively older, both variants remain prevalent across the web.
\noindent \textbf{reCaptcha v3}~\cite{recaptcha_v3_about} operates without any visible challenge or user interaction. Instead, it silently assigns a continuous risk score between 0.0 and 1.0 to each session based on behavioral and environmental signals accumulated during page interaction. Sites use this score to gate access or trigger step-up authentication, with the default acceptance threshold set at 0.5~\cite{google_recaptcha_v3}. This design makes reCaptcha v3 fundamentally different from challenge-response systems: there is no puzzle to solve, and success depends entirely on producing interaction signals that the scoring model associates with legitimate human users.
% \noindent \textbf{reCaptcha v3-enterprise}. A more advanced version of reCaptcha v3 designed for enterprise applications. It provides enhanced risk analysis and integration capabilities with security tools.
\noindent \textbf{Cloudflare Turnstile}~\cite{cloudflare_turnstile} is deployed in two modes that differ in the degree of visible user interaction required. In the \textit{Managed} (checkbox) mode, users are presented with a visible checkbox widget. Turnstile frequently verifies users automatically and checkmarks the box without requiring meaningful interaction. Upon page load, Turnstile performs background integrity checks on the browser environment, including JavaScript execution, device and network signals, browser consistency checks, and behavioral telemetry, before deciding whether additional verification is necessary.
In the \textit{Invisible} mode, the explicit checkbox is removed entirely, and verification occurs silently in the background while the same integrity and behavioral checks execute. Both modes rely primarily on environment-authenticity and risk-analysis signals rather than explicit challenge solving, placing Turnstile closer to reCaptcha v3 in its threat model than to challenge-centric systems such as reCaptcha v2 or hCaptcha.
All experimental pages remained publicly accessible for approximately six months prior to evaluation and received regular benign interactions multiple times per week. Maintaining long-lived deployments exposed the systems to realistic background traffic, including potential automated scanning and unsolicited access attempts commonly observed on publicly reachable web services. This extended deployment period also reduced the likelihood of evaluating freshly initialized or insufficiently calibrated configurations, which is particularly important for behavioral and risk-based systems such as reCaptcha v3 that rely on historical telemetry and reputation signals for risk assessment.
These bot management systems were selected to span the full spectrum of defense paradigms, from explicit challenge-response mechanisms (reCaptcha v2, hCaptcha) to non-interactive verification (Turnstile, reCaptcha v3), enabling a comprehensive assessment of each attacker stack's capabilities and limitations.
% The sample login pages with incorporated reCaptcha v2 and hCaptcha are presented in Figure \ref{fig:sample_login_v2_hCaptcha}.
% \begin{figure}[t]
% \centering
% \includegraphics[width=\linewidth]{images/Third-Party Solver Sequence Diagram.pdf}
% \caption{Sequence diagram for third-party Captcha solver services.}
% \label{fig:3rd_party_solver_sequence_diagram}
% \end{figure}
\subsection{Selected Solving Services}
To evaluate the effectiveness of third-party solving services against the deployed defense solutions, we selected a representative set of services based on their prevalence and ecosystem footprint. Selection was guided by public web-traffic signals and integration breadth, prioritizing services with high user activity commonly deployed in automation pipelines~\cite{motoyama2010re,weng2019towards,gao2022demystifying,jin2023secure}. As summarized in Table~\ref{tab:captcha-services}, the chosen services include a high monthly user base (190K--1.2M) and collectively support a variety of Captcha types through human labor, machine learning models, or hybrid approaches~\cite{motoyama2010re,jin2023secure}.
\begin{table}[t]
\centering
\footnotesize
\caption{Commercial solving services and their popularity. User volume and ranking extracted from SimiliarWeb~\cite{similair_web}.}
\label{tab:captcha-services}
\begin{tabular}{llll}
\toprule
\textbf{Service} & \textbf{Monthly Users} & \textbf{Global Rank} & \textbf{Price/1K} \\
\midrule
2Captcha & 1.2M & 37,810 & \$1.45--\$2.99 \\
Anti-Captcha & 450K & 84,392 & \$2.00--\$5.00 \\
AZCaptcha & 190K & 145,771 & \$0.10--\$1.00 \\
BestCaptchaSolver & 300K & 110,531 & \$1.80--\$3.00 \\
Capsolver & 650K & 72,405 & \$1.00--\$3.00 \\
CapMonster & 980K & 49,219 & \$0.60--\$1.30 \\
ImageTyperz & 210K & 132,884 & \$1.80--\$2.10 \\
\bottomrule
\end{tabular}
\end{table}
Pricing varies across services and reflects both market positioning and the operational complexity of supported defense types (Table~\ref{tab:captcha-services}). AZCaptcha offers the lowest cost at \$0.10--\$1.00 per 1,000 solves, while Anti-Captcha is the most expensive at \$2.00--\$5.00. The two services that support hCaptcha---2Captcha (\$1.45--\$2.99/1K) and ImageTyperz (\$1.80--\$2.10/1K)---occupy a higher price tier than services that do not, such as AZCaptcha (\$0.10--\$1.00/1K) and CapMonster (\$0.60--\$1.30/1K). This pricing gap partially reflects the higher operational cost of hCaptcha solving, which demands human annotators or sophisticated models capable of fine-grained image classification, in contrast to the token-injection approaches used for reCaptcha~v2 and Turnstile.
% As we discuss in Section~\ref{sec:solver-service-eval-pipeline}, hCaptcha's limited solver-ecosystem coverage is itself a security property that defenders can exploit.
\subsection{Selected LLM-Based Browser Agents}
\label{sec:selected_agents}
To assess the real-world capabilities of modern large language model (LLM)-driven browser agents in bypassing bot management solutions, we selected a diverse set of representative systems spanning both research prototypes and production deployments. Our selection prioritizes agents capable of multi-step web reasoning and interaction, while ensuring reproducibility through publicly accessible implementations. We further emphasize architectural diversity across DOM-based, vision-based, and hybrid designs, as well as variation in execution environments ranging from cloud-hosted services to fully self-hosted and browser-native systems.
We organize the evaluated systems into four deployment settings: cloud-hosted agents, self-hosted agents, AI-assisted browsers, and browser-extension-based agents. These settings primarily differ in execution environment authenticity, browser instrumentation level, and access to proprietary capabilities.
\noindent \textbf{Cloud Agents.} We evaluate cloud-hosted deployments of \textsc{Browser-Use}~\cite{browseruse2024}, \textsc{Skyvern}~\cite{skyvern2024}, and \textsc{Manus}~\cite{meta_manus}. \textsc{Browser-Use} integrates structured DOM interaction with visual grounding in a managed cloud execution environment. \textsc{Skyvern} combines DOM extraction and visual context for multi-step automation, and its commercial deployment includes proprietary components such as a dedicated Captcha-solving pipeline~\cite{skyvern_captcha_blog}. \textsc{Manus} represents a commercial end-to-end autonomous browsing system designed for general-purpose task execution within a hosted infrastructure. Collectively, these systems reflect a low-barrier threat model in which adversaries rely solely on access to hosted agent interfaces without maintaining local infrastructure.
\noindent \textbf{Self-Hosted Agents.} We evaluate self-hosted configurations of \textsc{Browser-Use}~\cite{browseruse2024}, \textsc{Skyvern}~\cite{skyvern2024}, \textsc{OpenManus}~\cite{openmanus2025}, and \textsc{SeeAct}~\cite{zheng2024seeact}. These systems expose internal prompts, execution traces, and browser interactions, enabling fine-grained behavioral analysis. Unlike commercial deployments, they do not include proprietary integrations or external solving services. All self-hosted agents are evaluated in default configurations without modifying browser fingerprints, extending tool access, or altering the action space.
\noindent \textbf{AI-Assisted Browsers.} We evaluate \textsc{BrowserOS}~\cite{browseros2025} and \textsc{Perplexity Comet}~\cite{perplexity2025comet}. Unlike standalone autonomous agents, these systems embed LLM capabilities directly into the browsing interface, where the browser remains the primary execution environment, and the model acts as an assistive layer. \textsc{BrowserOS} represents a system-level browser control framework integrating reasoning with native browser operations, while \textsc{Comet} is a browser-integrated assistant designed for conversational and task-oriented web interaction. This setting captures an emerging class of systems where automation is coupled with the interface.
\noindent \textbf{Browser-Extension Agents.} We evaluate \textsc{NanoBrowser}~\cite{nanobrowser2024}, a browser-extension-based agent operating directly within a standard user browser environment. Unlike instrumented automation frameworks, this configuration inherits native browser state, including cookies, history, and extensions, significantly reducing fingerprinting artifacts. This setting isolates whether failures arise from reasoning limitations or from environmental detectability introduced by automation frameworks.
Overall, these systems span hybrid perception, DOM-centric interaction, cloud-hosted autonomous agents, browser-native assistants, and system-level browser control. Across these deployment settings, we include substantial variation in execution environment authenticity, instrumentation level, and proprietary capability access, enabling a controlled analysis of how these factors influence the interaction with defense systems and the outcomes.
The next section presents the framework for experiments and evaluations across different bot management systems and agent configurations.
\subsection{Experiment Framework}
% \begin{figure}[t]
% \centering
% \resizebox{\columnwidth}{!}{
% \begin{tikzpicture}[
% font=\normalsize,
% box/.style={
% draw,
% rounded corners=2pt,
% align=center,
% inner sep=3pt,
% minimum height=0.65cm
% },
% widebox/.style={
% box,
% minimum width=5.0cm
% },
% smallbox/.style={
% box,
% minimum width=2.35cm,
% minimum height=0.85cm
% },
% module/.style={
% box,
% minimum width=2.35cm,
% minimum height=1.55cm
% },
% arrow/.style={
% -{Latex[length=1.6mm]},
% thick
% },
% dashedarrow/.style={
% -{Latex[length=1.6mm]},
% thick,
% dashed
% }
% ]
% % Top runtime
% \node[widebox] (agent) {
% \textbf{Autonomous Agent Runtime}
% };
% \node[widebox, below=0.45cm of agent] (analysis) {
% \textbf{Verification Analysis}\\
% Identify defense type
% };
% % Verification branches
% \node[smallbox, below left=0.65cm and 0.15cm of analysis] (explicit) {
% \textbf{Interactive}\\
% reCaptcha v2, hCaptcha
% };
% \node[smallbox, below right=0.65cm and 0.15cm of analysis] (implicit) {
% \textbf{Non-Interactive}\\
% reCaptcha v3, v2-invis
% };
% % Modules
% \node[module, below=0.55cm of explicit] (solver) {
% \textbf{Solver}\\[2pt]
% OCR / VLM\\
% Vision
% };
% % \node[module, below=2.2cm of analysis] (thirdparty) {
% \node[module, below=1.1cm of $(explicit)!0.5!(implicit)$] (thirdparty) {
% \textbf{Third-Party}\\
% \textbf{Solver}\\[2pt]
% Token APIs\\
% Human farms
% };
% \node[module, below=0.55cm of implicit] (auth) {
% \textbf{Authenticity}\\[2pt]
% Real profile\\
% Fingerprints\\
% Cookies
% };
% % Behavioral layer
% \node[widebox, below=1cm of auth] (behavior) {
% \textbf{Behavioral Realism}\\
% Cursor -- Timing -- Scroll
% };
% % Submission and outcome
% \node[widebox, below=3cm of thirdparty] (submit) {
% % \node[widebox, below=2cm of $(solver)!0.5!(auth)$] (submit) {
% \textbf{Submission / Validation}
% };
% \node[smallbox, below=0.5cm of submit] (success) {
% \textbf{Bypass} \\
% Successful
% };
% % Escalation aligned with solver
% \node[smallbox, below=1cm of solver] (escalation) {
% \textbf{Escalation}\\
% suspicious
% };
% % Main flow
% \draw[arrow] (agent) -- (analysis);
% \draw[arrow] (analysis.south west) -- (explicit.north);
% \draw[arrow] (analysis.south east) -- (implicit.north);
% \draw[arrow] (explicit.east) -- (thirdparty);
% \draw[arrow] (implicit.west) -- (thirdparty);
% \draw[arrow] (explicit) -- (solver);
% \draw[arrow] (implicit) -- (auth);
% % Merge into behavioral layer
% % \draw[arrow] (solver.east) -| ([xshift=-1.25cm]behavior.north);
% \draw[arrow] (auth.south) -- (behavior.north);
% \draw[arrow] (behavior) -- (submit);
% \draw[arrow] (solver) -- ([xshift=-1.2cm] submit.north);
% \draw[arrow] (thirdparty) -- (submit);
% \draw[arrow] (submit) -- (success);
% % Escalation loop
% \draw[dashedarrow] (submit.west) -| (escalation.south);
% \draw[dashedarrow] (escalation.north) -- (solver.south);
% \end{tikzpicture}
% }
% \caption{
% Experiment framework for autonomously bypassing bot management defenses. Interactive challenges require solver capabilities, while non-interactive systems rely on environment authenticity and behavioral realism. Suspicious activity may trigger escalation into interactive challenges.
% }
% \label{fig:automated_captcha_solving_flowchart}
% \end{figure}
To answer the research questions introduced in Section~\ref{sec:rq}, we evaluate the results for each pair of defense and bypass mechanisms.
% Figure~\ref{fig:automated_captcha_solving_flowchart} summarizes the design space of the automated bypass scenario.
Our goal is to systematically evaluate the practical effectiveness of the mechanisms currently available to automated systems. Depending on the type of defense encountered, an agent may rely on different capabilities to progress through the protected workflow.
The solution that is applicable across both interactive and non-interactive defense solutions is the use of third-party solving services. As discussed in Section~\ref{sec:background}, these services abstract away the underlying verification process and provide a token that can be injected into the protected workflow. For interactive Captchas, the token is typically obtained by solving the presented challenge through human labor or specialized solving infrastructure. For non-interactive systems such as reCaptcha v3, the service instead attempts to generate a token associated with a sufficiently high trust score.
In addition to third-party services, interactive challenges (e.g., reCaptcha v2 and hCaptcha) can also be bypassed through agent-native solving capabilities. In this setting, the agent must interact with and solve the presented challenge using mechanisms such as OCR, vision-language models, or other perception modules before continuing the workflow.
In contrast, non-interactive systems do not present a challenge to solve. Success instead depends on convincing the verification system that the interaction originates from a legitimate user. We therefore characterize bypasses in terms of two requirements: \emph{authenticity}, which captures browser- and profile-level trust signals, and \emph{behavioral realism}, which captures human-like interaction patterns such as mouse movement, scrolling behavior, and timing characteristics.
Guided by this framework, we first evaluate the effectiveness of commercial third-party solving services across multiple defense solutions. We then analyze the ability of autonomous browser agents to complete protected form submission workflows and identify the factors that contribute to successful bypasses. Together, these experiments provide a comprehensive view of the current capabilities and limitations of automated circumvention.
\begin{table*}[t!]
\centering
\small
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1.15}
\caption{Performance of third-party solvers. Succ.\ = success rate, Time = solving time.
--: not supported. We observe a nearly perfect success rate for challenged-based defenses but only 23\% success rate on average against reCaptcha v3.}
\label{tab:solver-performance}
\begin{tabular}{l l cc cc cc cc cc cc cc cc}
\toprule
& &
\multicolumn{2}{c}{\textbf{2Captcha}} &
\multicolumn{2}{c}{\textbf{Anti-Captcha}} &
\multicolumn{2}{c}{\textbf{AZCaptcha}} &
\multicolumn{2}{c}{\textbf{BestCaptchaSolver}} &
\multicolumn{2}{c}{\textbf{CapMonster}} &
\multicolumn{2}{c}{\textbf{Capsolver}} &
\multicolumn{2}{c}{\textbf{ImageTyperz}} \\
\cmidrule(lr){3-4}\cmidrule(lr){5-6}\cmidrule(lr){7-8}\cmidrule(lr){9-10}
\cmidrule(lr){11-12}\cmidrule(lr){13-14}\cmidrule(lr){15-16}
\textbf{Captcha} & \textbf{Variant}
& \scriptsize Succ. & \scriptsize Time
& \scriptsize Succ. & \scriptsize Time
& \scriptsize Succ. & \scriptsize Time
& \scriptsize Succ. & \scriptsize Time
& \scriptsize Succ. & \scriptsize Time
& \scriptsize Succ. & \scriptsize Time
& \scriptsize Succ. & \scriptsize Time \\
& & \scriptsize (\%) & \scriptsize (s)
& \scriptsize (\%) & \scriptsize (s)
& \scriptsize (\%) & \scriptsize (s)
& \scriptsize (\%) & \scriptsize (s)
& \scriptsize (\%) & \scriptsize (s)
& \scriptsize (\%) & \scriptsize (s)
& \scriptsize (\%) & \scriptsize (s) \\
\midrule
\multirow{2}{*}{\textbf{hCaptcha}}
& Easy
& \textbf{100} & 67.0 & -- & -- & -- & --
& -- & -- & -- & -- & -- & --
& \textbf{100} & 42.3 \\
& Hard
& \textit{58} & 102.5 & -- & -- & -- & --
& -- & -- & -- & -- & -- & --
& \textbf{100} & 82.7 \\
\midrule
\multirow{2}{*}{\textbf{reCaptcha v2}}
& Checkbox
& \textbf{100} & 10.8 & \textbf{100} & 51.0 & \textbf{100} & 132.2
& \textbf{100} & 60.2 & \textbf{100} & 8.4 & \textbf{100} & 11.1
& \textbf{100} & 49.4 \\
& Invisible
& \textbf{100} & 13.7 & \textbf{100} & 52.7 & \textbf{100} & 101.7
& \textbf{100} & 53.5 & \textbf{100} & 7.9 & \textbf{100} & 7.2
& \textbf{100} & 64.3 \\
\midrule
\multirow{1}{*}{\textbf{reCaptcha v3}}
& Score ($>$0.5)
& \textit{26} & 22.1 & \textit{30} & 12.1 & \textit{0} & 17.5
& 63 & 39.7 & \textit{0} & 4.5 & \textit{0} & 1.6
& \textit{41} & 40.4 \\
\midrule
\multirow{2}{*}{\makecell{\textbf{Cloudflare}\\\textbf{Turnstile}}}
& Managed
& \textbf{100} & 7.2 & \textbf{100} & 20.3 & -- & --
& \textbf{100} & 71.0 & \textbf{100} & 4.8 & \textbf{100} & 5.5
& \textbf{100} & 43.3 \\
& Invisible
& \textbf{100} & 7.8 & \textbf{100} & 26.1 & -- & --
& \textbf{100} & 88.8 & \textbf{100} & 4.5 & \textbf{100} & 4.0
& \textbf{100} & 46.8 \\
\bottomrule
\end{tabular}
\end{table*}
\section{Evaluations and Results}
\label{sec:evaluations}
In this section, we describe the experiment details and evaluate the results to gather insight and answer each research question.
% In this section, we describe the experimental methodology used to evaluate automated Captcha bypass techniques and analyze the results to answer the research questions introduced in Section~\ref{sec:rq}.
% To make results comparable across services and agent classes, we use a fixed execution protocol and consistent success criteria. For each solver service and supported Captcha type, we run repeated attempts under the same task configuration and record success outcome, solve latency, and cost fields where available. For agent evaluations, each agent--Captcha pair is executed under a consistent prompt template and testbed configuration to minimize experimental variance.
% \subsection{Reproducibility and Measurement Protocol}
% \noindent \textbf{Trial accounting.}
% TODO(numbers): add final counts for attempts per condition and total runs.
% \noindent \textbf{Validation source of truth.} We treat backend verification or observed page state as ground truth. Agent self-reports are not used as success evidence when they conflict with verification outcomes.
% \noindent \textbf{Failure handling.} Timeouts, unsupported challenge types, and API failures are logged separately from explicit bypass failures. In tables, unsupported conditions are marked as ``--'' and incomplete runs are marked as pending where applicable.
% \noindent \textbf{Randomness and variance.} Captcha systems are adaptive and can change behavior across repeated interactions. We therefore report aggregate outcomes (success rate and average solve time) rather than claiming deterministic per-run behavior.
\subsection{RQ1: Evaluating Third-Party Solving Services}
\label{sec:solver-service-eval-pipeline}
Third-party solving services represent one of the primary bypass mechanisms available to automated systems.
% Rather than solving challenges locally or establishing trust through browser authenticity, these services abstract the verification process behind a simple API that returns a verification token to the requester.
Importantly, this approach applies to both interactive Captchas and non-interactive defenses.
% To quantify the effectiveness of this bypass strategy, we evaluate a diverse set of commercial solving services across multiple Captcha providers and measure their success rates, response times, and operational costs.
To evaluate the reliability and effectiveness of third-party solving services, we designed a parallelized evaluation pipeline in which each solver service is assigned a dedicated worker process. Each worker continuously submits tasks until 100 instances have been evaluated for every target website protected by the defenses described in Section~\ref{sec:selected_captcha}.
The workflow starts by constructing a solving task through the corresponding service API and terminates once a valid solution is returned or a generous timeout threshold of 4 minutes is reached.
Whenever a token is received, its validity is verified through the official verification interfaces provided by the vendor, including hCaptcha's \texttt{siteverify} interface, Google's reCaptcha verification API, and Cloudflare Turnstile's verification endpoint.
% For solving services that support feedback mechanisms, the pipeline reports the verification outcome back to the provider, enabling automatic refunds for invalid responses.
The collected metrics, including solving success rate and average response time, are summarized in Table~\ref{tab:solver-performance}, allowing a direct comparison across solver platforms.
For reCaptcha v2, hCaptcha (Easy mode), and Cloudflare Turnstile, success rates were consistently at 100\% across most evaluated services.
However, not every service supports every Captcha type, which explains the smaller number of data points for hCaptcha and Turnstile relative to reCaptcha. Only two of the seven evaluated providers (2Captcha and ImageTyperz) offer hCaptcha-solving capabilities. This limited adoption is likely driven by the higher operational complexity of hCaptcha challenges, which often require either human annotation or more sophisticated image-recognition pipelines. As a result, solving hCaptcha incurs greater cost and engineering effort compared to the largely token-based workflows commonly used for reCaptcha v2 and Turnstile.
Nevertheless, the empirical results demonstrate that adversaries can integrate these solving services into automated crawlers with minimal effort and cost. As shown in Table~\ref{tab:captcha-services}, solving prices range from approximately \$0.10 to \$5.00 per 1,000 requests depending on the provider and defense type.
The combination of low operational cost and consistently high solving reliability substantially reduces the practical security provided by challenge-based Captcha deployments.
In contrast, performance against behavioral reCaptcha v3 was lower across all evaluated services. Following Google's deployment recommendations, we consider an attempt successful when the returned score exceeds the default threshold of 0.5~\cite{google_recaptcha_v3}.
Several services, including AZCaptcha and Anti-Captcha, mostly failed to generate tokens meeting this threshold. Two other services (CapMonster and Capsolver) consistently failed to generate verifiable tokens. Even the best-performing provider, BestCaptchaSolver, achieved a success rate of only 63\%.
While third-party services can reliably outsource the solution of interactive challenges, they are considerably less effective when success depends on behavioral reputation and environmental trust signals.
% In such cases, possession of a syntactically valid token alone is often insufficient to satisfy the verification process.
\begin{tcolorbox}[
colback=gray!10,
colframe=black!40,
title=\textbf{Key Observation},
fonttitle=\bfseries,
]
Third-party solving services achieve near-perfect success rates against challenge-based Captchas at negligible cost, enabling adversaries to outsource verification with minimal technical effort. The effectiveness drops substantially against behavioral systems such as reCaptcha v3, suggesting that behavioral reputation and environmental trust signals remain difficult to commoditize through token-generation services alone.
\end{tcolorbox}
% PLACED HERE FOR BETTER OUTPUT
\begin{table*}[t!]
\centering
\footnotesize
\renewcommand{\arraystretch}{1.2}
\setlength{\tabcolsep}{3pt}
\caption{
Evaluation of cloud-based agents, self-hosted agents, AI browsers, and browser-extension agents across Captcha types.
\cmark: successful Captcha bypass and submission.
{$\lozenge$}: successful Captcha bypass without successful submission.
\xmark: failure.
% \ph: evaluation pending.
}
\label{tab:combined_agent_eval}
\begin{tabular}{l l ccc cccc cc c}
\toprule
& &
\multicolumn{3}{c}{\textbf{Cloud-based}} &
\multicolumn{4}{c}{\textbf{Self-hosted with gpt-4o backend}} &
\multicolumn{2}{c}{\textbf{AI Browser}} &
\multicolumn{1}{c}{\textbf{Extension}} \\
\cmidrule(lr){3-5}
\cmidrule(lr){6-9}
\cmidrule(lr){10-11}
\cmidrule(lr){12-12}
\textbf{Captcha}
& \textbf{Variant}
& \textbf{Browser-Use}
& \textbf{Skyvern}
& \textbf{Manus}
& \textbf{Browser-Use}
& \textbf{Skyvern}
& \textbf{OpenManus}
& \textbf{SeeAct}
& \textbf{BrowserOS}
& \textbf{Comet}
& \textbf{NanoBrowser} \\
\midrule
\multirow{2}{*}{\textbf{hCaptcha}}
& Easy
& \xmark
& \cellcolor{bypass}$\lozenge$
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark \\
& Hard
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark \\
\midrule
\multirow{2}{*}{\textbf{reCaptcha v2}}
& Checkbox
& \xmark
& \cellcolor{fullpass}\cmark
& \xmark
& \xmark
& \xmark
& \cellcolor{bypass}$\lozenge$
& \xmark
& \xmark
& \xmark
& \xmark \\
& Invisible
& \xmark
& \cellcolor{fullpass}\cmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \cellcolor{fullpass}\cmark \\
\midrule
\multirow{1}{*}{\textbf{reCaptcha v3}}
& Score $>$0.5
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \cellcolor{fullpass}\cmark \\
\midrule
\multirow{2}{*}{\makecell{\textbf{Cloudflare}\\\textbf{Turnstile}}}
& Managed
& \cellcolor{fullpass}\cmark
& \xmark
& \xmark
& \cellcolor{fullpass}\cmark
& \xmark
& \xmark
& \xmark
& \xmark
& \xmark
& \cellcolor{fullpass}\cmark \\
& Invisible
& \cellcolor{fullpass}\cmark
& \cellcolor{fullpass}\cmark
& \xmark
& \cellcolor{fullpass}\cmark
& \xmark
& \xmark
& \xmark
& \cellcolor{fullpass}\cmark
& \xmark
& \cellcolor{fullpass}\cmark \\
\bottomrule
\end{tabular}
\end{table*}
Overall, these results answer RQ1 and establish an important baseline for the remainder of this study. The third-party solver ecosystem is highly effective against challenge-based Captchas, yet substantially less effective against behavioral systems such as reCaptcha v3. A likely explanation is that solver services merely inject a valid token without reproducing the behavioral signals and browser-environment characteristics evaluated by reCaptcha v3. Overcoming this limitation requires an attacker capable of generating realistic interaction traces while simultaneously operating within a browser environment that satisfies environmental integrity checks. In the following section, we investigate whether off-the-shelf LLM-based browser agents, which interact with web pages directly, can satisfy these requirements and successfully bypass such defenses (RQ2).
% \begin{table}[t!]
% \centering
% \scriptsize
% \setlength{\tabcolsep}{3pt} % reduce column spacing
% \renewcommand{\arraystretch}{1.15}
% \caption{Third-party Captcha solvers. Succ. = success rate (\%), Time = seconds. \textbf{Bold}: 100\% success. \textit{Italic}: $\leq$50\%. \xmark{}: pending. --: not supported.}
% \label{tab:third_party_services_results}
% \begin{tabular}{l l c c c}
% \toprule
% \textbf{Solver} & \textbf{Captcha} & \textbf{Success (\%)} & \textbf{Time (s)} & \textbf{\$/1K} \\
% \midrule
% \multirow{7}{*}{2Captcha} & reCaptcha v2 (Check) & \textbf{100} & 10.8 & 2.99 \\
% & reCaptcha v2 (Inv) & \textbf{100} & 13.7 & 2.99 \\
% & reCaptcha v3 & \textit{26} & 22.1 & 2.99 \\
% & hCaptcha (E) & \textbf{100} & 67.0 & 2.99 \\
% & hCaptcha (H) & \textit{50} & 102.5 & 2.99 \\
% & Turnstile (Inv) & \textbf{100} & 7.2 & 1.45 \\
% & Turnstile (NI) & \textbf{100} & 7.8 & 1.45 \\
% \midrule
% \multirow{7}{*}{Anti-Captcha} & reCaptcha v2 (Check) & \textbf{100} & 51.0 & 2.00 \\
% & reCaptcha v2 (Inv) & \textbf{100} & 52.7 & 2.00 \\
% & reCaptcha v3 & \textit{30} & 12.1 & 5.00 \\
% & hCaptcha (E) & -- & -- & -- \\
% & hCaptcha (H) & -- & -- & -- \\
% & Turnstile (Inv) & \textbf{100} & 20.3 & 2.00 \\
% & Turnstile (NI) & \textbf{100} & 26.1 & 2.00 \\
% \midrule
% \multirow{5}{*}{AZCaptcha} & reCaptcha v2 (Check) & \textbf{100} & 132.2 & 1.00 \\
% & reCaptcha v2 (Inv) & \textbf{100} & 101.7 & 1.00 \\
% & reCaptcha v3 & \textit{0} & 17.5 & 1.00 \\
% & hCaptcha & -- & -- & -- \\
% & Turnstile & -- & -- & -- \\
% \midrule
% \multirow{7}{*}{BestCaptchaSolver} & reCaptcha v2 (Check) & \textbf{100} & 60.2 & 3.00 \\
% & reCaptcha v2 (Inv) & \textbf{100} & 53.5 & 3.00 \\
% & reCaptcha v3 & 63 & 39.7 & 3.00 \\
% & hCaptcha (E) & -- & -- & -- \\
% & hCaptcha (H) & -- & -- & -- \\
% & Turnstile (Inv) & \textbf{100} & 71.0 & 1.80 \\
% & Turnstile (NI) & \textbf{100} & 88.8 & 1.80 \\
% \midrule
% \multirow{7}{*}{CapMonster} & reCaptcha v2 (Check) & \textbf{100} & 8.4 & 0.60 \\
% & reCaptcha v2 (Inv) & \textbf{100} & 7.9 & 0.60 \\
% & reCaptcha v3 & \textit{0} & 4.5 & 1.00 \\
% & hCaptcha (E) & -- & -- & -- \\
% & hCaptcha (H) & -- & -- & -- \\
% & Turnstile (Inv) & \textbf{100} & 4.8 & 1.30 \\
% & Turnstile (NI) & \textbf{100} & 4.5 & 1.30 \\
% \midrule
% \multirow{6}{*}{Capsolver} & reCaptcha v2 (Check) & \textbf{100} & 11.1 & 1.00 \\
% & reCaptcha v2 (Inv) & \textbf{100} & 7.2 & 1.00 \\
% & reCaptcha v3 & \textit{0} & 1.6 & 3.00 \\
% & hCaptcha (M) & -- & -- & -- \\
% & Turnstile (Inv) & \textbf{100} & 5.5 & 1.20 \\
% & Turnstile (NI) & \textbf{100} & 4.0 & 1.20 \\
% \midrule
% \multirow{7}{*}{ImageTyperz} & reCaptcha v2 (Check) & \textbf{100} & 49.4 & 2.10 \\
% & reCaptcha v2 (Inv) & \textbf{100} & 64.3 & 2.10 \\
% & reCaptcha v3 & \textit{41} & 40.4 & 3.00 \\
% & hCaptcha (E) & \textbf{100} & 42.3 & 1.80 \\
% & hCaptcha (H) & \textbf{100} & 82.7 & 1.80 \\
% & Turnstile (Inv) & \textbf{100} & 43.3 & 1.80 \\
% & Turnstile (NI) & -- & -- & -- \\
% \bottomrule
% \end{tabular}
% \end{table}
% \begin{figure}
% \centering
% \includegraphics[width=1.1\linewidth]{images/time_vs_succes.png}
% \caption{Captcha Solving: Time vs Success Rate}
% \label{fig:third_party_services_results}
% \end{figure}
\subsection{RQ2: Off-the-Shelf LLM Browsing Agents}
\label{sec:cloud_llm_results}
In this section, we evaluate the selected LLM-based browser agents introduced in Section~\ref{sec:selected_agents} under their default, unmodified configurations. The goal of this evaluation is not only to measure whether agents can bypass the deployed defense mechanisms, but also to characterize the operational factors that enable or prevent successful bypass across different verification paradigms.
For all experiments, we follow a unified evaluation procedure across agents and deployment configurations. Each agent is provided with a simple task prompt (Figure~\ref{fig:cracking_prompt}) and is executed without modifications to its underlying model, tools, or browser environment beyond its default setup.
All interactions are executed end-to-end, and full browser sessions are recorded for post-hoc analysis.
\begin{figure}[h]
\centering
\begin{tcolorbox}[listing only,
colback=promptbg,
colframe=promptborder,
listing options={
basicstyle=\ttfamily\small,
breaklines=true,
columns=fullflexible,
language=,
escapeinside=||,
},
left=0.5mm,
right=0.5mm,
top=0.5mm,
bottom=0.5mm,
boxrule=0.8pt,
arc=2pt]
Navigate to https://example-login.com and fill out the form with the following information: \\
-----\\
username: admin, password: P@ssword \\
-----\\
Submit the form by clicking the login button.
\end{tcolorbox}
\caption{\normalsize
{Prompt used to instruct the LLM-based browser agents during the experiment. The URL, username, and password have been replaced with placeholders.}}
\label{fig:cracking_prompt}
\end{figure}
For each attempt, we define two sequential stages of the workflow:
\begin{itemize}
\item \textbf{Bypass:} The agent successfully satisfies the Captcha verification mechanism, including solving interactive challenges or achieving acceptance in non-interactive verification systems.
\item \textbf{Submission:} The agent completes the expected workflow by submitting the form.
\end{itemize}
A successful attempt requires completion of both stages. If an agent submits a form without successfully passing the verification mechanism, the attempt is considered a failure.
We record full browser interaction traces for each run, screenshots (where applicable), and agent action logs. Outcomes are manually verified to ensure the correctness of both the challenge resolution and final submission state.
We use a two-stage evaluation protocol. First, each agent--defense configuration is evaluated in a screening phase of five attempts to determine whether the agent is capable of satisfying the verification mechanism and to characterize the dominant failure mode. Configurations that failed all attempts in this phase were marked as failed and not expanded further, since additional trials would only increase cost without changing the qualitative conclusion. Configurations that proved promising (i.e., able to interact with the challenge or satisfy the threshold score) were then evaluated with ten additional trials to assess repeatability.
This protocol allows us to distinguish deterministic failures from configurations with genuine bypass capability, while keeping the evaluation reproducible and cost-effective. The results were consistent across both phases in the vast majority of cases. The exception was in challenge-based configurations (reCaptcha v2 checkbox and easy hCaptcha), where success was sensitive to the specific visual challenge content served. Throughout our evaluation, we adopt a conservative success criterion: a bypass outcome is marked as successful if the agent achieves bypass in \emph{at least one} attempt in each phase. This reflects a realistic threat model in which an adversary may retry until successful, and it ensures that intermittent but genuine bypass capability is not obscured by aggregated failure rates. Accordingly, the $\lozenge$ symbol in Table~\ref{tab:combined_agent_eval} denotes configurations where bypass was achieved at least once but form submission was not completed in any trial.
% \bo{Add number of trials?}
% Importantly, we verify Completion against the actual page state rather than the agent's self-report, as cloud agents in particular frequently hallucinate successful submission even without Captcha bypass.
% \subsubsection{Results Across Agent Classes}
% summarizes the evaluation results across all agent classes and Captcha systems.
The overall results, provided in Table~\ref{tab:combined_agent_eval}, reveal that successful bypass is not determined solely by the reasoning capabilities of the underlying LLM. Different defense systems pressure distinct layers of the autonomous agent stack, including challenge-solving capability, execution-environment authenticity, and behavioral analysis.
Notably, Comet declined to fill in or submit login forms containing, citing concerns about bypassing authentication flows. This suggests that Comet's failures reflect an intentional safety policy rather than a capability gap, and its bypass results should be interpreted accordingly.
\noindent \textbf{Interactive Verification Systems.}
Interactive systems, including reCaptcha v2 and hCaptcha, primarily pressure the perception and solver capabilities of autonomous agents. Across these systems, agents frequently identify the presence of the verification mechanism and repeatedly interact with the corresponding page elements, yet fail to complete the required challenge successfully. This indicates that the agents are capable of locating and recognizing the Captcha interface, but lack the specialized perception and solver capabilities necessary to satisfy visual or interactive verification tasks.
Successful bypass in these systems depends less on planning or reasoning ability and more on access to specialized challenge-solving integrations such as OCR pipelines, vision-language models, or external token-solving services. This distinction is particularly visible in the cloud-hosted Skyvern deployment, which achieves the highest success rate across challenge-based systems. Skyvern successfully bypasses both reCaptcha v2 variants and partially succeeds against hCaptcha. This behavior aligns with Skyvern's documented integration of dedicated Captcha-solving capabilities~\cite{skyvern_captcha_blog}.
In contrast, self-hosted agents consistently fail against challenge-based systems despite often correctly recognizing the presence of the Captcha and attempting interaction. The dominant failure mode is therefore not task recognition, but the absence of specialized perception and solver capabilities required to satisfy the challenge itself.
Figure~\ref{fig:hCaptcha_samples} illustrates representative visual puzzles encountered during hCaptcha evaluation where the agent failed to successfully complete the task. Notably, frameworks such as \cite{deng2024oedipus} have begun lowering the technical barrier to develop solvers, and as visual reasoning capabilities continue to improve, constructing capable solvers for such challenges is becoming an increasingly tractable problem.
\begin{figure}[t]
\centering
\includegraphics[width=\linewidth]{images/hCaptcha_Samples.pdf}
\caption{Samples of visual puzzles encountered during hCaptcha evaluation.
All agents failed to solve these challenges due to the absence
of visual puzzle-solving capabilities in their default action spaces.
}
\label{fig:hCaptcha_samples}
\end{figure}
\noindent \textbf{Non-Interactive Verification Systems.}
In contrast to challenge-based systems, invisible reCaptcha v2, Cloudflare Turnstile, and reCaptcha v3 primarily rely on non-interactive verification mechanisms that evaluate browser authenticity and behavioral trust signals rather than solving challenges. Across these systems, the dominant failure mode is different. Agents operating within default automated browser environments are frequently detected and assigned high-risk scores despite successfully navigating and submitting the workflow. In these cases, the failure does not originate from reasoning or task execution, but rather from detectable artifacts introduced by the browser environment and execution context itself.
Across all cloud-based and self-hosted deployments, agents frequently complete the interaction workflow and submit the form successfully. However, the submissions consistently fail the verification threshold, indicating that the agents can perform the required task but fail to generate sufficiently legitimate behavioral and environmental signals.
Turnstile does not uniformly reject all automated environments. Certain configurations, particularly Browser-Use and NanoBrowser, successfully bypass the system and complete the target workflow. This indicates that Turnstile's effectiveness is highly sensitive to browser-environment characteristics and implementation details rather than purely to the presence of automation itself.
reCaptcha v3 exhibits the strongest dependence on execution-environment authenticity. Unlike challenge-based verification systems, reCaptcha v3 does not present a visible challenge. Instead, it silently evaluates browser, environmental, and behavioral signals to assign a trust score during submission.
The browser-extension deployment (NanoBrowser) is the only configuration that consistently achieves successful submission with an acceptable trust score. Operating directly within a real browser profile, NanoBrowser inherits persistent state, including cookies, browsing history, installed extensions, and stable fingerprinting characteristics. This substantially reduces automation artifacts and enables the interaction to satisfy reCaptcha v3's trust-scoring model.
% These observations suggest that challenge difficulty alone no longer provides a robust barrier against autonomous agents. Once solver integrations become available, challenge-based Captchas increasingly reduce to a modular tooling problem rather than a fundamental limitation of agent reasoning.
\begin{tcolorbox}[
colback=gray!10,
colframe=black!40,
title=\textbf{Key Observation},
fonttitle=\bfseries,
]
Successful bypass is increasingly determined by the availability of specialized capabilities rather than the reasoning ability of the underlying agent. Challenge-based systems primarily depend on solver integrations, while non-interactive systems rely on environment authenticity and behavioral trust signals.
% As a result, Captcha robustness is shifting from challenge difficulty toward detecting automation artifacts across the execution environment and interaction behavior.
\end{tcolorbox}
% Across all evaluated reCaptcha v3 configurations, only NanoBrowser achieves successful bypass. The remaining deployments consistently complete the workflow while failing verification. This behavior strongly suggests that the dominant failure mode is not reasoning or planning failure, but rather execution-environment detectability.
% \paragraph{Environment-Based Captchas (Cloudflare Turnstile).}
% Cloudflare Turnstile exhibits a different behavior. Rather than presenting an explicit task, Turnstile enforces validation at the interaction level and prevents the form submission unless verification is satisfied.
% Across both cloud-based and self-hosted agents, this generally results in failure at the Completion stage. In most cases, agents engage with the mechanism but repeatedly attempt interactions without progressing, leading to Engagement without Completion.
% However, this behavior is not consistent across all configurations. We observe that certain self-hosted
% setups (e.g., Browser-Use) are able to successfully bypass Turnstile in some instances, indicating that the system does not reliably detect all automated environments. This suggests that Turnstile’s effectiveness is sensitive to specific browser characteristics and implementation details, rather than acting as a robust, universal defense against automation.
% The browser-extension agent (NanoBrowser) consistently succeeds across all stages, including Completion, further reinforcing that execution within a real browser context significantly reduces detection signals.
% \paragraph{Behavioral Captchas (reCaptcha v3).}
% reCaptcha v3 fundamentally is different from challenge-based Captchas, as it does not present an explicit task for the agent to interact with. Instead, it silently evaluates user behavior and assigns a score upon form submission. As such, the Engagement metric is not applicable in this setting, and the evaluation focuses on Bypass and Completion.
% Across both cloud-based and self-hosted agents, Completion is frequently achieved as agents proceed to submit the form. However, Bypass consistently fails, indicating that submissions do not meet the acceptance threshold (0.5). This reflects that agents can execute the workflow, but cannot generate behavior consistent with legitimate users.
% The browser extension agent (NanoBrowser) is the only configuration that achieves successful Bypass and Completion. Operating within a real browser profile, it inherits persistent state such as browsing history, cookies, and stable fingerprint characteristics, as well as more natural interaction patterns.
% This enables it to produce behavioral signals that satisfy reCaptcha v3’s scoring model.
% Across all 8 agent-deployment configurations evaluated against reCaptcha v3, only 1, NanoBrowser operating inside a real user browser, achieves Bypass. The other 7 configurations achieve Completion but fail Bypass, a pattern consistent across all LLM backends and agent architectures tested. This confirms that the failure mode is not a reasoning or planning limitation but an execution environment limitation: agents know what to do, but their browser fingerprints reveal their automation.
% These results directly answer RQ2: off-the-shelf agents do not close the reCaptcha v3 gap established by the solver baseline in RQ1, unless they operate within a real browser context. This finding motivates the targeted modifications explored in the next section (RQ3).
%
These findings directly answer RQ2. Off-the-shelf autonomous browser agents are capable of navigating complex workflows and interacting with modern challenge-based systems, yet their success increasingly depends on the availability of modular capabilities beyond core reasoning alone. In particular, environment authenticity emerges as the dominant bottleneck for non-interactive systems such as reCaptcha v3. This observation motivates the targeted environment and behavior modifications explored in the following section.
\subsection{RQ3: Failures in LLM-based Browsers}
The results of the previous section suggest that modern verification systems rely on fundamentally different properties than traditional challenge-based Captchas. While challenge-oriented systems primarily depend on solver capabilities, reCaptcha v3 remains resistant even when agents successfully complete the target workflow. The results also suggest that this resistance originates from the environment authenticity layer rather than the solver layer. In particular, the successful bypass achieved by NanoBrowser indicates that browser identity and accumulated trust signals play a significant role. In this section, we investigate this hypothesis by systematically analyzing reCaptcha v3's trust model and evaluating what causes agents to fail in the bypass scenario.
We observe that all agents, except the browser-extension-based NanoBrowser, consistently fail to bypass the reCaptcha v3 defense despite completing the interaction workflow, with the system consistently assigning low trust scores (0.1--0.3) even when valid end-to-end interactions are performed. To understand what distinguishes the succeeding agent from the failing ones, we conduct a fine-grained analysis of the browser events triggered during form submission sessions. Specifically, we instrument the websites environment to collect interaction traces, including click events, keyboard inputs, mouse movements, and form submissions, and compare their distribution across agents. This allows us to isolate whether the trust score disparity stems from differences in behavioral patterns or from factors outside the interaction layer entirely.
We focus on Browser-Use as the representative failing agent because, among all evaluated agents, it produces the closest behavioral approximation to NanoBrowser, triggering a comparable range of browser events across clicks, key presses, and form submissions, and completing sessions within similar time ranges (approximately 11 seconds). Most other agents trigger substantially fewer event types or interact with the page in ways that are more immediately identifiable as automated. Despite this behavioral proximity, Browser-Use consistently receives low trust scores (0.1--0.3) while NanoBrowser succeeds. This contrast isolates the variable of interest: if behavioral similarity were sufficient for bypass, Browser-Use would succeed too. The side-by-side comparison of triggered browser events is provided in Figure~\ref{fig:interaction_heatmap_bu_nb}.
Despite nearly indistinguishable behavioral traces, only NanoBrowser achieves a successful bypass. This observation reveals a strong dependence on \emph{environmental authenticity}. NanoBrowser operates within a real user browser profile, inheriting persistent cookies, browsing history, extensions, and stable fingerprinting signals. In contrast, Browser-Use, although running in headful mode, relies on a clean and instrumented browser environment that lacks these long-term signals of legitimacy.
% With these observations, reCaptcha v3 is better understood not as just a behavioral classifier, but as a \emph{trust accumulation system over browser identity}.
\begin{figure}[h]
\centering
\includegraphics[width=0.8\linewidth]{images/interaction_heatmap_bu_nb.pdf}
\caption{Interaction Event Heatmap for Browser-Use and NanoBrowser.}
\label{fig:interaction_heatmap_bu_nb}
\end{figure}
\begin{tcolorbox}[
colback=gray!10,
colframe=black!40,
title=\textbf{Key Observation},
fonttitle=\bfseries,
]
The primary barrier to bypassing reCaptcha v3 is not the difficulty of generating human-like interactions, but the ability to operate from a trusted browser environment. With authentic browser state, cookies, and fingerprinting signals, agents can successfully bypass the defense.
\end{tcolorbox}
\section{Discussion}
\noindent \textbf{The Fragility of Bot Detection Solutions}
The key technical takeaway is not that all defenses fail equally, but that many deployed defenses are fragile under composition attacks. Bypass outcomes are strongly shaped by whether defenders rely on challenge complexity or on environment-integrity signals. Challenge-based controls can be outsourced to solver ecosystems; behavior-scoring controls are vulnerable when automation stacks successfully imitate trustworthy browser and session characteristics.
This shifts the strategic question for defenders from ``how hard is the puzzle'' to ``how costly is authentic impersonation.'' Defenses that verify execution context, enforce backend token validation, and combine risk signals with workflow-specific policies are more robust than those relying on isolated front-end challenges.
\noindent \textbf{Security and Safety of LLM-based Agents}
This study is a concrete example of dual-use risk in LLM agents. Capabilities desirable for productivity, including goal decomposition, adaptive planning, and robust web interaction, also lower the barrier for adversarial automation. Misuse does not require novel model training; practical abuse can be achieved by composing publicly available agents, solver APIs, and minor code modifications.
For AI safety and platform governance, this implies that risk cannot be managed at the model layer alone. Browser instrumentation choices, action-space design, default prompting behavior, and integration guardrails all materially influence whether an agent is easily repurposed for abuse. Agent developers should consider whether their default action spaces expose human-emulation primitives that an adversary could invoke with a natural-language prompt, and whether their deployment environments resist fingerprinting by anti-bot systems.
% \subsection{Future Directions}
% We anticipate an ``AI-versus-AI'' dynamic in which both attackers and defenders improve automation realism and detection quality in parallel. Future work should: (i) evaluate adaptive defense policies that respond to repeated adversarial interaction rather than using static thresholds; (ii) quantify false-positive trade-offs for high-friction reCaptcha v3 configurations; (iii) study how emerging browser attestation mechanisms affect the attacker cost curve; and (iv) characterize the additional lift provided by LLM agent modifications against Captcha types beyond reCaptcha v3.
\noindent \textbf{Ethics Considerations}
\noindent This work studies security weaknesses in widely deployed anti-automation mechanisms and is intended to improve defensive practice. Our experiments are designed to minimize harm by prioritizing controlled environments, avoiding disruptive behavior
% , and restricting real-world validation to limited, non-destructive interactions.
We do not release exploit-ready automation scripts, sensitive endpoint details, or operational instructions that would materially lower misuse barriers.
\noindent The study follows a dual-use risk model. We report enough technical detail for scientific reproducibility and defensive validation, while withholding implementation specifics that would directly enable abuse at scale. Findings are presented as defender guidance (verification, policy hardening, and monitoring) rather than offensive playbooks.
% \noindent In accordance with responsible disclosure principles, the authors reported the reCaptcha findings accompanied by a proof-of-concept, to Google's security team prior to publication via the Google Vulnerability Reward Program. The report was accepted, escalated to severity S2/priority P2, and forwarded to the reCaptcha product team, where it remains actively in progress at the time of submission.
\section{Conclusion}
This paper presents a systematic evaluation of bypass capabilities against bot management systems, covering seven commercial solver services and six LLM-based browser agents across challenge-based and non-interactive defense configurations. Our results show that challenge-based defenses are broadly ineffective against cost-driven adversaries, with commercial solvers achieving near-perfect bypass at negligible cost. Non-interactive defenses such as reCaptcha v3 exhibit stronger resistance, but our interaction trace analysis reveals that this resilience is not grounded in challenge complexity.
Instead, bypass outcomes are determined by execution-environment authenticity: agents that operate within real browser profiles with accumulated trust signals succeed, while behaviorally similar agents running in clean, instrumented environments consistently fail.
These findings reframe the threat model for anti-bot defenses. The security boundary does not lie at the solver or reasoning layer, but at the environment authenticity layer, a distinction with significant implications for how defenses are designed and evaluated. As LLM-based agents become more capable and more widely deployed, the assumption that non-interactive defenses provide durable protection warrants reexamination. Effective bot management will increasingly depend on robust server-side validation and resistance to environmental spoofing, rather than on interaction complexity alone.
% \newpage
% \subsection{Browser Interaction Comaprison}