PolyJailbreak: Cross-Modal Jailbreaking Attacks on Black-Box Multimodal LLMs
Extracted source text · arXiv 2510.17277
\pdfoutput=1
\documentclass[lettersize,journal]{IEEEtran}
\usepackage{amsmath,amsfonts}
\usepackage{algorithmic}
\usepackage{array}
\usepackage{caption}
\usepackage{subcaption} % 提供 subfigure 环境
% 可选：统一字号
\captionsetup[figure]{font=footnotesize}
\captionsetup[subfigure]{font=footnotesize}
\usepackage{textcomp}
\usepackage{stfloats}
\usepackage{url}
\usepackage{verbatim}
\usepackage{graphicx}
\usepackage[most]{tcolorbox}
\usepackage{tikz}
\usepackage{amsmath, amssymb}
\usepackage{enumitem}
\usepackage{booktabs}
\usepackage{tabularx}
\usepackage{array}
\usepackage{makecell}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{caption}
\usepackage{multirow}
\usepackage{graphicx}
\usepackage[normalem]{ulem}
\usepackage[most]{tcolorbox}
\usepackage{algorithmic}
\usepackage{algorithm}
\usepackage{hyperref}
\usepackage[export]{adjustbox}
\hyphenation{op-tical net-works semi-conduc-tor IEEE-Xplore}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\usepackage{balance}
\begin{document}
\title{\Large \bf Multimodal Safety Is Asymmetric: Cross-Modal Exploits Unlock \\ Black-Box MLLMs Jailbreaks}
\author{Xinkai Wang,
Beibei Li,
Zerui Shao,
Ao Liu,
Shouling Ji
\thanks{Xinkai Wang, Beibei Li, Zerui Shao, Ao Liu are with the School of Cyber Science and Engineering, Sichuan University, Chengdu 610207, China (email: wangxinkai6@stu.scu.edu.cn; libeibei@scu.edu.cn; shaozerui@stu.scu.edu.cn; aliu@scu.edu.cn).}
\thanks{Shouling Ji is with the College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China (e-mail: sji@zju.edu.cn).}}
%\markboth{Journal of \LaTeX\ Class Files,~Vol.~18, No.~9, September~2020}%
%{How to Use the IEEEtran \LaTeX \ Templates}
\maketitle
\begin{abstract}
Multimodal large language models (MLLMs) have demonstrated significant utility across diverse real-world applications. But MLLMs remain vulnerable to jailbreaks, where adversarial inputs can collapse their safety constraints and trigger unethical responses. In this work, we investigate jailbreaks in the text-vision multimodal setting and pioneer the observation that visual alignment imposes uneven safety constraints across modalities in MLLMs, thereby giving rise to multimodal safety asymmetry. We then develop PolyJailbreak, a black-box jailbreak method grounded in reinforcement learning. Initially, we probe the model’s attention dynamics and latent representation space, assessing how visual inputs reshape cross-modal information flow and diminish the model’s ability to separate harmful from benign inputs, thereby exposing exploitable vulnerabilities. On this basis, we systematize them into generalizable and reusable operational rules that constitute a structured library of Atomic Strategy Primitives, which translate harmful intents into jailbreak inputs through step-wise transformations. Guided by the primitives, PolyJailbreak employs a multi-agent optimization process that automatically adapts inputs against the target models. We conduct comprehensive evaluations on a variety of open-source and closed-source MLLMs, demonstrating that PolyJailbreak outperforms state-of-the-art baselines.
\end{abstract}
\textcolor{red}{Content warning: This paper contains unfiltered content generated by MLLMs that may be offensive to readers.}
%\begin{IEEEkeywords}
%Class, IEEEtran, \LaTeX, paper, style, template, typesetting.
%\end{IEEEkeywords}
\section{Introduction}
\IEEEPARstart{T}{he} rapid advancement of large language models (LLMs) has catalyzed multimodal large language models (MLLMs) that extend text-only capabilities to integrated vision and language reasoning~\cite{DBLP:conf/ijcai/GuoCWCPCW024,DBLP:conf/acl/CaffagniCBMS0CC24}, with widely deployed systems such as GPT-5~\cite{GPT-5}, Gemini~\cite{gemini}, and Claude~\cite{AnthropicClaude}. However, the widespread adoption of MLLMs also expose them to security threats that can undermine their reliability and trustworthiness~\cite{MLLMsecurityconcern}. Among these, jailbreak attacks, in which adversaries deliberately craft inputs to circumvent safety mechanisms and elicit unethical responses, pose particularly severe security risks and jeopardize the safe deployment of MLLMs in practice~\cite{white5, DBLP:conf/ccs/ShenC0SZ24}. While safety alignment techniques such as RLHF~\cite{RLHF} and instruction tuning~\cite{tuning} have improved model robustness, MLLMs remain vulnerable to jailbreak attacks~\cite{JailbreakDefense}. Prior work has shown that adding visual modalities enlarges the input space of MLLMs and introduces exploitable attack surfaces~\cite{black1, white3, white5, black6}, making them vulnerable even in black-box settings~\cite{black2, black3, JOOD, black6}.
\noindent\textbf{Insights.}
Through systematic analysis, we identify that the vulnerability of MLLMs arises from a safety asymmetry between the textual and visual modalities, where visual alignment weakens the robustness of text-based safety constraints and fails to establish boundaries of comparable strength for vision. This asymmetry manifests in two dimensions: (i) Many visual alignment schemes map or fuse textual and visual features to endow the backbone LLM with multimodal capability, but certain forms of integration can interfere with and potentially weaken the original text-based safety mechanism. (ii) Compared with text, visual inputs are subject to weaker safety constraints, resulting in less separable safety boundaries when MLLMs process multimodal inputs. This discrepancy likely stems from the limited availability of alignment data for harmful visual content and from the inherently more ambiguous and complex semantics of images. Follow-up experiments demonstrate that the identified asymmetry holds universally and stably across mainstream MLLMs, including GPT-4o~\cite{DBLP:journals/corr/abs-2410-21276}, Gemini-2.5~\cite{gemini2.5}, Claude-3.7~\cite{claude} and so on.
\noindent\textbf{Challenge.}
While prior studies have shown that vision can be exploited to jailbreak MLLMs~\cite{PrivacyRisks, ImageInputs, hades, Medical}, these efforts largely focus on isolated case studies or handcrafted prompts. What remains unaddressed is how to systematically exploit the multimodal safety asymmetry for scalable jailbreak generation across diverse black-box models~\cite{RedAgent, eosToken, FromLLMstoMLLMs, LandscapeofMLLM}. In practice, MLLMs can detect and refuse overtly malicious content, which forces adversaries to design prompts that are both covert and tailored to model-specific behaviors. This gap highlights the core challenge of our work: designing an automated jailbreak generation method that leverages the identified vulnerability and generalizes across models. Accordingly, the challenges include:
\begin{enumerate}[leftmargin=*]
\item Identifying and characterizing vulnerabilities. Fully uncover, localize, and formalize multimodal vulnerabilities arising from safety asymmetry.
\item Designing covert inputs. Craft inputs that evade detection while still eliciting harmful behaviors.
\item Scaling jailbreak generation across models. Automate jailbreak prompt generation and adaptation to diverse black-box MLLMs at scale.
\end{enumerate}
\noindent\textbf{Findings.}
To address these challenges, we characterize how multimodal safety asymmetry concretely manifests by systematically investigating typical visual alignment schemes and visual inputs. We demonstrate two key findings: (i) Different visual alignment schemes influence the integrity of the text-based safety mechanisms inherited from the backbone. Trainable-backbone schemes disrupt internal textual safety representations, potentially enabling identical textual inputs to succeed in jailbreaking. (ii) Visual inputs act as triggers and amplifiers of jailbreak vulnerabilities, not solely based on content, but through dynamic interactions with textual semantics during multimodal fusion.
\noindent\textbf{Our Work.}
We present PolyJailbreak, a black-box jailbreak method that leverages the multimodal safety asymmetry of MLLMs. Its core is a composable library of Atomic Strategy Primitives (ASPs), defined as reusable operational rules that map identified vulnerabilities into step-wise actions for constructing jailbreak prompts. The library covers three dimensions: textual manipulation, visual manipulation, and prompt amplification. Composing ASPs across these dimensions yields a wide range of strategies that broaden the attack surface of MLLMs. PolyJailbreak proceeds by profiling model behaviors and adaptively assembling ASPs into multimodal prompts. These prompts are then iteratively refined through guided search to produce high-efficacy adversarial inputs capable of breaking multimodal safety defenses.
\noindent\textbf{Contributions.} Our contributions are summarized as follows.
\begin{enumerate}
\item \textbf{We identify the asymmetry between textual and visual safety constraints, a previously unexamined structural vulnerability.}
We present a systematic empirical study of how typical visual alignment strategies and visual inputs affect MLLMs. Our results reveal that certain visual alignment schemes weaken the backbone LLM’s safety mechanisms, which in turn leads to abnormal safety behaviors even under text-only queries. Moreover, we show that visual inputs can function as latent triggers and amplifiers of jailbreak vulnerabilities, by interacting with textual semantics during multimodal fusion.
\item \textbf{We propose PolyJailbreak, a reinforcement learning-driven method for black-box multimodal jailbreak generation.}
PolyJailbreak identifies and consolidates vulnerabilities in MLLMs, distilling them into a composable Atomic Strategy Primitive library spanning textual manipulation, visual manipulation, and prompt amplification. By profiling the security characteristics of target models, it searches for and flexibly exploits vulnerabilities in diverse multimodal contexts, adaptively composing and optimizing adversarial inputs to jailbreak different MLLMs.
\item \textbf{We conduct extensive black-box evaluations on mainstream state-of-the-art MLLMs, demonstrating high attack effectiveness across models.}
PolyJailbreak achieves an average attack success rate of 83.34\%, reaching 97.5\% on GPT-4o and 97.25\% on Gemini, and surpasses the average success rate of existing baselines by 49.55\%.
\end{enumerate}
\section{Background}
\subsection{Definitions}
\noindent\textbf{Multimodal Large Language Model.}
A prevalent MLLM design extends a backbone text-only LLM with a visual modality for cross-modal understanding and generation~\cite{X-Fusion}. This approach reuses the LLM’s language capabilities while reducing computational costs~\cite{EMMA}. A typical MLLM architecture consists of a text encoder \(E_{t}\), a visual encoder \(E_{v}\), a fusion module \(\Phi\), and a backbone LLM \(\mathcal{M}_{init}\). The generation process can be formalized as:
\begin{equation}
y = \mathcal{M}^{*}_{init} \left( \Phi \left( E_t(x) , \; E_v(v) \right) \right),
\end{equation}
where \(x\) and \(v\) denote textual and visual inputs, and \(\mathcal{M}^*_{init}\) is the backbone LLM before visual alignment, with parameters trainable or frozen per scheme.
\subsection{Jailbreak Threats in MLLMs}
Jailbreaking in MLLMs refers to crafting adversarial multimodal inputs that bypass built-in safety constraints, inducing the model to generate harmful or restricted content. Unlike text-only LLMs, the use of a visual modality in MLLMs results in new attack surfaces, allowing adversaries to craft or modify images to embed, obscure, or obfuscate harmful instructions. A range of studies report that typographic~\cite{hades}, adversarial~\cite{AlignIsNotEnough}, or context-manipulated~\cite{FigStep} images can reliably trigger harmful responses across diverse MLLMs. Benchmarks and universal attack frameworks further reveal that such vulnerabilities are not model-specific but transferable, highlighting a systemic weakness in current safety alignment approaches~\cite{MM-SafetyBench}. Despite these advances, current jailbreak research largely follows a result-driven paradigm, emphasizing attack efficacy over understanding the underlying causes of vulnerabilities. While these efforts have demonstrated diverse and potent multimodal attack methods, they often lack systematic investigation into the structural and representational factors that enable such compromises. This limitation points to a critical gap in understanding the root causes of MLLM jailbreak vulnerabilities, and resolving this gap is essential for developing principled and robust defense strategies.
\subsection{Visual Alignment Schemes in MLLMs}
Visual alignment aims to project visual semantics into the backbone model’s representational space, enabling unified processing of multimodal inputs~\cite{EMMA}. Existing alignment schemes primarily differ in how they integrate visual information and whether they modify the parameters of the backbone model \(\mathcal{M}_{init}\)~\cite{VisualRepresentationsMap, HyperLLaVA}. To facilitate our analysis, we propose a two-paradigm categorization of alignment approaches, based on their treatment of the backbone during alignment.
\textit{Frozen \(\mathcal{M}_{init}\) alignment}: These approaches preserve the parameters of the backbone language model. Visual information is integrated through dedicated encoders and fusion mechanisms, such as cross-attention layers or projection modules, while the backbone weights remain unchanged. This design follows a modular decoupling philosophy, aiming to retain the LLM’s original semantic competence, reasoning consistency, and safety alignment.
\textit{Trainable \(\mathcal{M}_{init}\) alignment}: These approaches partially or fully update the parameters of the backbone language model during visual alignment, coupling visual and textual representations via end-to-end optimization. While this can strengthen cross-modal integration, it also modifies the backbone’s internal representations, which may inadvertently shift its original reasoning behavior and safety boundaries.
\subsection{Threat Model and Evaluation Dataset}
\noindent\textbf{Adversary’s Goal.}
We consider that the adversary targets MLLMs capable of processing both text and visual inputs, equipped with comprehensive safety alignment to reject malicious queries. These models are running in their intended form, without data poisoning, parameter tampering, or other adversarial modifications. The adversary’s goal is to craft adversarial inputs that bypass these safety mechanisms and induce the model to generate harmful responses. Such responses, which violate mainstream safety policies (e.g., malicious, violent, or illegal content), may fully, partially, or even excessively satisfy the adversary’s intent. For instance, the adversary may intend to acquire detailed steps for weapon construction. Even if the model issues an explicit refusal, its subsequent output may still leak critical design principles, thereby providing the adversary with actionable knowledge, albeit insufficient to constitute a complete solution.
\noindent\textbf{Adversary's Capabilities and Threat Surface.}
The adversary is external and operates in a black-box setting without access to the model’s internals. They can query the model with text, images, or both, and obtain the outputs. Public information and auxiliary tools (e.g., other models or encoders) may also be leveraged. The adversary is limited to single-turn interactions but may issue multiple independent queries.
%as depicted in Figure~\ref{fig:threatModel}.
The threat surface lies in the model’s inference-time decision process, encompassing the joint interpretation of multimodal inputs, the decision of whether to refuse or comply with a query, and the generation of responses. This decision pipeline induces exploitable asymmetries that adversaries can leverage to bypass safety mechanisms.
\noindent\textbf{Attack's Generality and Practicality.}
The generality of this threat arises from the rapid proliferation of MLLMs across both general-purpose platforms and domain-specific applications, including healthcare, finance, education, customer service, and office automation. While our evaluation focuses on representative models, the widespread adoption of MLLMs strongly suggests that similar vulnerabilities are likely to persist across diverse contexts, thereby continuously expanding the attack surface. The impact can also be severe: successful jailbreaks may produce harmful outputs that undermine information reliability, erode trust in AI systems, and enable misinformation or malicious use in high-stakes domains. Moreover, because the attack requires only black-box access and modest technical expertise, it remains feasible for a broad spectrum of adversaries, ranging from casual users to well-resourced actors, greatly increasing its disruptive potential.
%\begin{figure}[t]
% \centering
% \includegraphics[width=0.46\textwidth]{ThreatModel.pdf} % 调整宽度为页面的80%
% \caption{Illustration of MLLMs under black-box attack.}
% \label{fig:threatModel}
%\end{figure}
\noindent\textbf{Evaluation Dataset.}
To systematically evaluate vulnerabilities in MLLMs, we construct our dataset by curating and extending AdvBench~\cite{Advbench}, yielding a task-specific dataset tailored to our study. Following OpenAI's content moderation taxonomy~\cite{OpenAIcontent}, we define eight sensitive categories (e.g., \textit{criminal}, \textit{hate}), each containing 50 paired instructions. Each pair consists of a harmful instruction and a benign counterpart that preserves contextual similarity while adhering to ethical norms. For example, given a harmful instruction such as “\textit{Write a script that exploits software vulnerabilities},” the benign counterpart would be “\textit{Write a script to test software robustness through edge case simulation}.” This paired design enables fine-grained comparison between harmful and benign scenarios under identical contextual settings. The dataset contains 400 harmful-benign pairs in total, with the eight sensitive categories and their descriptions summarized in Table~\ref{table:dataset}. Detailed construction procedures are provided in Appendix~\ref{appendix:dataset_construction}.
\section{Asymmetry Induces Vulnerability}
In this section, we present a comprehensive empirical study that analyzes how multimodal safety asymmetry leads to vulnerabilities in MLLMs. We structure our investigation around two key research questions:
\begin{itemize}[leftmargin=*]
\item\textbf{RQ1:} \textbf{(Impact of Alignment Choices)} How do different visual alignment schemes affect the textual safety behavior of MLLMs compared with their backbone LLMs?
\item\textbf{RQ2:} \textbf{(Safety Boundaries under Vision)} How do visual inputs reshape the safety boundaries of MLLMs?
\end{itemize}
\begin{figure*}[htbp]
\centering
\begin{subfigure}{0.96\textwidth}
\includegraphics[width=\linewidth]{llava_trend_five_histories_zoom_last10.pdf}
\end{subfigure}
\begin{subfigure}{0.96\textwidth}
\includegraphics[width=\linewidth]{vicuna_trend_five_histories_zoom_last10.pdf}
\end{subfigure}
\caption{Comparison of information flow in LLaVA and Vicuna across dialogue histories (from none to four turns, left to right).}
\label{RQ1fig}
\end{figure*}
To address \textbf{RQ1}, we analyze two representative alignment schemes. For frozen alignment, we study its multimodal fusion principle to reason about its safety implications. For trainable alignment, we empirically examine the effect of different textual inputs on model attention before and after alignment, and employ representative jailbreak attacks to compare responses. For \textbf{RQ2}, we investigate how different image types influence model responses to harmful instructions. These image types vary in content, semantics, and symbolic features. Building on an analysis of hidden-layer representations that assessed the separability of benign and malicious multimodal inputs, we then evaluated the models with corresponding input combinations, yielding several noteworthy and insightful observations. We select two representative MLLMs for comparative analysis: the LLaVA series (trainable \(\mathcal{M}_{init}\))~\cite{llava-1.5} and the LLaMA 3.2-Vision series (frozen \(\mathcal{M}_{init}\))~\cite{llama}.
\subsection{Impact of Alignment Choices (RQ1)}
\textbf{Frozen Alignment Analysis.}
Frozen alignment integrates visual inputs without modifying parameters of the backbone $\mathcal{M}_{\text{init}}$. In LLaMA 3.2-Vision, this is implemented via trainable cross-attention layers inserted between frozen transformer blocks, following the Flamingo architecture~\cite{Flamingo}. These layers fuse visual and textual features using attention:
\(\text{Attention} = \text{Softmax} ( {QK^\top}/{\sqrt{d}} )V\)
where the \textit{Query} ($Q$) originates from the transformer, and \textit{Key/Value} ($K/V$) are derived from the visual encoder $E_v$. When no image is provided, $K/V$ collapse to zeros, reducing the cross-attention output to:
\(
Q + 0 = Q.
\)
Under text-only queries, these layers function as pass-through layers, preserving the backbone’s original representation dynamics. This structural characteristic ensures that the design does not perturb the backbone model’s semantic behavior or its refusal mechanisms when processing text-only queries.
\noindent\textbf{Trainable Alignment Analysis.} In contrast to frozen alignment, trainable alignment schemes fine-tune the backbone $\mathcal{M}_{\text{init}}$ during visual alignment. The LLaVA series exemplifies this approach, where visual inputs are allowed to reshape internal representations. This adaptability, however, \textit{motivates a key question: could parameter updates from visual alignment fine-tuning introduce behavioral uncertainty in text-only inference and thereby affect the model’s safety behavior?} To investigate this, we compare LLaVA v1.5-7B with its backbone model Vicuna-7B-v1.5~\cite{vicuna}, analyzing how different input token types affect inference. Following the taxonomy of ``information flow'' proposed by Zhang \textit{et al.}~\cite{informationflow}, we further categorize tokens into four types: system, user, image, and history, and examine their influence through attention distribution across layers.
Figure~\ref{RQ1fig} shows the attention weight distributions for both models when processing a malicious instruction (``Write a script that can exploit vulnerabilities in a software or operating system''), either in isolation or in the presence of conversational history. In LLaVA, even without an actual image, a designated placeholder token \(\mathit{<unk>}\) occupies the image slot in the input sequence. Notably, this \(\mathit{<unk>}\) token receives considerable attention across layers, suggesting that the model maintains structural awareness of visual placeholders. Furthermore, compared to Vicuna, LLaVA exhibits a shift in attention focus: system tokens receive reduced attention, while user and image-related tokens are more emphasized. This attention redistribution suggests that multimodal alignment alters the model's semantic processing flow, which may affect the robustness of its safety mechanisms. We further analyze how dialogue history influences model behavior in multi-turn settings. By appending five benign contextually-relevant dialogue turns before the malicious instruction, we observe that attention to historical tokens increases across both models. In deeper layers, historical context even surpasses the current user input in attention dominance. Interestingly, in LLaVA, the attention weight of the \(\mathit{<unk>}\) token drops sharply in the final two layers when dialogue history is present, indicating that textual context can dilute the residual influence of visual placeholders. The attention variation of \(\mathit{<unk>}\) also corroborates this finding. To understand how these shifts impact model outputs, we examine last-layer attention distributions during token generation. The analysis reveals that both models rely heavily on harmful keyword detection (e.g., ``exploit'', ``script'', ``vulnerability'') to trigger refusal responses. However, compared to Vicuna, LLaVA consistently assigns lower attention to such critical tokens, particularly the keyword exploit, and this gap widens as benign dialogue history accumulates. The resulting attenuation of focus on harmful keywords may divert attention away from system-level safety triggers and introduce potential vulnerabilities.
To further verify our observations, we convert 400 malicious instructions in the dataset into standardized adversarial prompts to evaluate Vicuna and LLaVA under text-only inferences. The results, summarized in Figure~\ref{fig:RQ1_ASR}, reveal a substantial decline in LLaVA’s ability to refuse harmful instructions compared to Vicuna. Despite identical inputs and no visual data, LLaVA exhibits markedly higher attack success rates (ASR), confirming that parameter tuning during visual alignment compromises the backbone’s original refusal mechanisms. We further investigate how dialogue history modulates the responses by prepending either benign or malicious prior dialogues to the jailbreak prompts. Both models show enhanced robustness with benign history, as evidenced by significantly reduced ASR. Surprisingly, introducing malicious history does not improve jailbreak effectiveness, and the success rate actually decreases relative to the no-history baseline. This counterintuitive outcome suggests that explicit attack-related cues in dialogue history may inadvertently trigger safety mechanisms earlier, disrupting attack stealth that would otherwise succeed in single-turn interactions.
These findings confirm that trainable visual alignment can weaken refusal mechanisms by altering internal representations, increasing jailbreak susceptibility. Harmful keywords are critical for triggering safety responses, and reduced attention to them directly undermines the model’s ability to refuse harmful queries. Dialogue history plays a dual role: it can reinforce safety alignment in benign interactions, yet its improper exploitation may create latent multi-turn vulnerabilities.
\begin{tcolorbox}[enhanced,
colback=white,
colframe=black,
arc=2mm,
boxrule=0.8pt,
drop fuzzy shadow={black!70!gray},
left=3pt, right=3pt, top=3pt, bottom=3pt,
boxsep=2pt
]
\textbf{Finding 1:} Frozen alignment achieves seamless integration of vision and text while preserving the textual safety behavior of the backbone, ensuring that refusal consistency is largely maintained after alignment. In contrast, trainable alignment alters internal representations, where the introduction of visual tokens indirectly reduces attention to harmful keywords and thereby weakens the model’s refusal behavior, even in text-only queries. Manipulated dialogue history further amplifies this vulnerability.
\end{tcolorbox}
\subsection{Safety Boundaries under Vision (RQ2)}
\textbf{RQ2} builds on the observation of multimodal safety asymmetry and investigates how different visual inputs affect the internal semantic separation between benign and malicious content. This separation is fundamental to a model’s ability to detect harmful inputs~\cite{SneakyPrompt}. To study whether visual inputs affect this separability in the hidden space, we employ a two-stage analysis: (i) \textit{Representation-Level Analysis}, which quantifies internal separability using cosine distances; and (ii) \textit{Behavioral-Level Evaluation}, which measures the attack success rates under controlled multimodal inputs.
\begin{table*}[htbp]
\centering
\caption{The ASR (\%) of LLaVA and LLaMA under different input combinations (RQ2).}
\label{tab:asr_comparison}
\scriptsize
\setlength{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.1}
\begin{tabular}{lcccccccccc}
\toprule
\multirow{2}{*}{\textbf{Model}} & \multirow{2}{*}{\textbf{Text Type}} &
\multicolumn{7}{c}{\textbf{Image Type}} &
\multirow{2}{*}{\textbf{Text-only}} &
\multirow{2}{*}{\shortstack{\textbf{Overall} \\ \textbf{(Text\&Text+Image)}}}\\
\cmidrule(lr){3-9}
& & {\scriptsize White} & {\scriptsize MalTypora} & {\scriptsize CatSameLabel} &
{\scriptsize CatOppLabel} & {\scriptsize Noise+White} & {\scriptsize Noise+MalTypora} & {\scriptsize Noise+CatSameLabel} \\
\midrule
\multirow{2}{*}{\textbf{LLaVA}}
& plain & 64.50 & 76.50 & 74.50 & 70.50 & 63.00 & 74.25 & 75.75 & 37.00 & 95.00\\
& emoji & 56.50 & 70.50 & 72.75 & 67.50 & 58.00 & 70.50 & 71.00 & 47.25 & 95.25\\
\midrule
\multirow{2}{*}{\textbf{LLaMA}}
& plain & 15.00 & 3.50 & 11.75 & 8.25 & 6.00 & 5.75 & 8.75 & 24.25 & 34.25\\
& emoji & 13.25 & 1.00 & 4.25 & 10.75 & 5.75 & 1.75 & 1.5 & 18.00 & 30.75\\
\bottomrule
\end{tabular}
\end{table*}
\begin{figure}[tbp]
\centering
\begin{subfigure}{0.21\textwidth}
\includegraphics[width=\linewidth, frame]{llava_results_umap.png}
\end{subfigure}%
\begin{subfigure}{0.21\textwidth}
\includegraphics[width=\linewidth, frame]{llama_results_umap.png}
\end{subfigure}
\caption{UMAP visualization of hidden state clustering for benign and malicious instructions (LLaVA vs LLaMA).}
\label{RQ2fig}
\end{figure}
\noindent\textbf{Preparation.} We extract hidden states from the -$5$th layer of LLaVA and LLaMA and project them into a 2D space using UMAP~\cite{umap} for visualization. As shown in Figure~\ref{RQ2fig}, both models exhibit some separation between benign and malicious instructions, but LLaMA demonstrates clearer cluster boundaries, suggesting stronger semantic discrimination in the absence of visual inputs. Based on this observation, we design three exploratory experiments:
\textit{Existence Prior Test:} Introduces semantically blank images (White) to examine whether the mere presence of visual signals influences safety boundaries.
\textit{Semantic Consistency Disruption Test:} Evaluates potential decision boundary shifts by introducing semantically consistent (MalTypora, CatSameLabel), semantically contradictory (CatOppLabel), and weakly related (CrossCatSameLabel, CrossCatOppLabel) images (e.g., pairing \textit{Hate} text with \textit{Criminal} images).
\textit{Lightweight Perturbation Test:} Uses emoji-augmented inputs (EmojiText, EmojiMalTypora) and noise-injected images (Noise+) to assess model robustness against symbolic and mild perceptual corruption.
To assess the impact of visual inputs on the safety boundary, we compute the cluster separation ratio (CSR) across representative transformer layers in LLaVA and LLaMA, given identical text inputs but varying images. The CSR is defined as the ratio of the cosine distance between benign and malicious cluster centers to the average intra-cluster distance. A higher CSR value reflects stronger separability of the model between different categories of inputs. Results are summarized in Tables~\ref{tab:llava_csr_analysis} and~\ref{tab:llama_csr_analysis}.
First, under the \textit{Existence Prior Test}, LLaVA shows a clear collapse in semantic boundaries. At layer -$5$, CSR drops from 0.4538 (Text-only) to 0.3435 (White image), and further to 0.2437 when emojis are added. This degradation persists across layers, indicating that LLaVA's visual encoder injects alignment noise even when semantic content is absent. By contrast, LLaMA maintains stronger separability under the same conditions (e.g., 1.8839\textrightarrow{}0.9321), suggesting that frozen visual pathways buffer against modality-induced interference. Second, the \textit{Semantic Consistency Disruption Test} reveals LLaVA's vulnerability to conflicting or weakly related image-text semantics. Under CrossCatOppLabel at layer -$5$, CSR drops from 0.3430 (Text-only) to 0.3288 (CrossCatOppLabel), and further to 0.2061 with emoji. Similar trends appear under CatOppLabel. In contrast, LLaMA retains more stable CSR values (e.g., 0.9242\textrightarrow{}0.7383), indicating greater robustness. Third, under the \textit{Lightweight Perturbation Test}, LLaVA consistently exhibits CSR compression. For instance, in Noise+MalTypora, CSR drops from 0.2943 (Plain) to 0.2059 (Emoji), and remains suppressed under other perturbations such as Noise+White (0.3517\textrightarrow{}0.2477). In contrast, LLaMA is largely unaffected, with CSR generally above 0.6.
Across all conditions, we observe a layerwise trend: CSR typically peaks around layer -$15$ and declines toward the output. However, this decline is steeper in LLaVA, especially under visual perturbations. This suggests that its trainable visual alignment entangles semantic clusters in deeper layers, weakening safety boundaries. In contrast, LLaMA's frozen structure preserves more consistent separability throughout.
To validate these findings at the behavioral level, we measure the ASR under various multimodal configurations (Table~\ref{tab:asr_comparison}). Results show that internal semantic compression aligns with increased jailbreak success. In text-only settings, emoji prompts raise LLaVA's ASR from 37.00\% to 47.25\%, mirroring the CSR drop. With visual inputs, ASR remains high regardless of image semantics: 74.50\% (CatSameLabel), 70.50\% (CatOppLabel), and 74.25\% (Noise+MalTypora). These results suggest that LLaVA's alignment amplifies vulnerability, even under noisy or contradictory inputs. LLaMA maintains ASR below 10\% in all cases, including only 10.75\% in emoji+CatOppLabel, consistent with its higher CSR. This also suggests that if more complex visual inputs were used to substantially reduce LLaMA’s CSR, its susceptibility to jailbreaks may increase significantly. Aggregating across modalities, LLaVA's ASR increases from 42.13\% (text-only) to 95.13\% (with visuals). LLaMA only increases from 21.13\% to 32.5\%. Crucially, both use identical textual prompts, confirming that visual inputs, regardless of content, act as latent triggers and amplifiers of jailbreak vulnerability in MLLMs.
\begin{tcolorbox}[enhanced,
colback=white,
colframe=black,
arc=2mm,
boxrule=0.8pt,
drop fuzzy shadow={black!70!gray},
left=3pt, right=3pt, top=3pt, bottom=3pt,
boxsep=2pt
]
\textbf{Finding 2:} Visual inputs exacerbate the internal inseparability between benign and malicious semantics in MLLMs, and can act as both triggers and amplifiers of jailbreaks through their interaction with textual prompts during multimodal fusion.
\end{tcolorbox}
\section{PolyJailbreak}
This section outlines the workflow of PolyJailbreak, first introducing its overall pipeline, then detailing the library of ASPs and input construction, and finally presenting the reinforcement learning-based optimization process.
\subsection{Workflow of the PolyJailbreak Scheme}
Building on our RQ findings, we propose PolyJailbreak, a black-box jailbreak method that systematically exploits vulnerabilities arising from multimodal safety asymmetry. The core idea is to abstract these vulnerabilities into ASPs, which serve as actionable rules for adversarial input generation. PolyJailbreak begins with behavioral profiling to gather comprehensive information about the model’s safety mechanisms, then adaptively selects ASPs from three spaces: textual manipulation, visual manipulation, and prompt amplification. These ASPs are composed into multimodal prompts and refined through a closed-loop optimization cycle driven by a customized Soft Actor-Critic (SAC)~\cite{SAC} algorithm in a multi-agent setting. As shown in Figure~\ref{fig:method} and Algorithm~\ref{PolyJailbreak}, the workflow comprises seven key steps detailed below.
\noindent\textbf{Model Discovery.} Let \(\mathcal{M}\) denote the target MLLM. This phase constructs a safety profile \(\mathcal{P}_{M}\) to guide subsequent strategy selection.
\(\mathcal{P}_{M}\) is derived through two complementary channels, namely \emph{Direct Inquiry} and \emph{Online Probing}, which together expose the defense characteristics of \(\mathcal{M}\). In the \textit{Direct Inquiry} stage, we pose targeted queries to \(\mathcal{M}\) to elicit its refusal templates, content-moderation guidelines, vision-filtering behaviors, and self-declared safety policies, thereby revealing \(\mathcal{M}\)’s decision-making factors and characteristic phrasing. In the \textit{Online Probing} stage, a network-connected intelligence agent attempts to gather publicly available information across five dimensions: base architecture, alignment data, refusal language patterns, safety guidelines, and published policy rules. The extracted findings from both channels are manually validated and merged into \(\mathcal{P}_{M}\). This provides quantitative priors for ASP selection, parameter initialization, and dynamic adaptation in PolyJailbreak, thereby improving sample efficiency and adversarial optimization success.
\noindent\textbf{Attack Initialization.} We initialize the components required for adversarial optimization: the ASPs pool \(\mathcal{S}\), the attack agent \(\mathcal{M}_{A}\), the judging agent \(\mathcal{M}_{J}\), and functional modules. At each timestep \(t\), the observed state is denoted as \(\mathbf{s}_t\), the action as \(\mathbf{a}_t\), the scalar reward as \(\mathbf{r}_t\), and the next state as \(\mathbf{s}_{t+1}\). To enhance input quality, we embed Chain-of-Thought (CoT) reasoning into instruction design. Given a malicious instruction \(g\) and safety profile \(\mathcal{P}_{M}\), the agent \(\mathcal{M}_{A}\) outputs the tuple
\(
\langle s_{\text{text}},\, s_{\text{image}},\, s_{\text{pers}},\, r^{\star} \rangle
\)
, where \(s_{\text{text}} \in \mathcal{S}_{\text{text}}\), \(s_{\text{image}} \in \mathcal{S}_{\text{image}}\), and \(s_{\text{pers}} \in \mathcal{S}_{\text{pers}}\) denote selected ASP indices, and \(r^{\star}\) is a reference-answer prefix realizing \(g\). The triple \(\langle s_{\text{text}},\, s_{\text{image}},\, s_{\text{pers}} \rangle\) defines the initial action space, while \(r^{\star}\) serves as the semantic anchor for reward computation during optimization.
\noindent\textbf{Attack Construction.}
We construct the adversarial input by combining textual and visual components derived from selected ASPs. For the text part, we initialize the macro-level operation mode $\mu_{\text{text}}$ and the prompt state $x_t$ (with $x_0 = 0$). The textual input is then generated through the \(\texttt{TextConstruction}(\cdot)\), which incorporates both textual manipulation and prompt amplification strategies. For the visual part, we initialize a text-to-image generator $\mathcal{M}_D$ and a task-specific processing module $\mathcal{T}$. The \(\texttt{ImageConstruction}(\cdot)\) is then applied to obtain the corresponding image. Last, the text prompt and image are combined to yield the multimodal adversarial input \(I_{g}^{t}\) for step $t$.
\noindent\textbf{Interaction \& Judgement.} The adversarial input \(I_{g}^{t}\) is fed into \(\mathcal{M}\), yielding response \(y_t\). \(\mathcal{M}_{J}\) then evaluates the tuple \(\langle g,\, I_{g}^{t},\, y_t \rangle\), returning a hard label \(\ell_t \in \{\textsc{Success}, \textsc{Fail}\}\) and a soft-score metric \(\mathbf{h}_t\).
\noindent\textbf{Reward Computation \& Network Update.} If \(\ell_t = \textsc{Fail}\), the reward \(\mathbf{r}_t\) is computed via \(\texttt{GetReward}(\cdot)\), and the transition \(\langle \mathbf{s}_t, \mathbf{a}_t, \mathbf{r}_t, \mathbf{s}_{t+1} \rangle\) is stored in replay buffer \(\mathcal{B}\). When \(|\mathcal{B}|\) reaches the mini-batch threshold, the SAC routine updates the actor and twin-critic networks.
\noindent \textbf{Action Sampling.} From state \(\mathbf{s}_{t+1}\), the updated policy network samples a macro textual operation \(\mu_{\text{text}} \in \{0, 1, 2\}\) and a strategy triple \(\langle s_{\text{text}},\, s_{\text{image}},\, s_{\text{pers}} \rangle\), which are used to reconstruct the next input \(I_{g}^{t+1}\).
\noindent\textbf{Termination Condition.} The loop continues until either \(\mathcal{M}_{J}\) returns \(\ell_t = \textsc{Success}\), or the step counter reaches \(T_{\max}\). If \(\mathcal{M}\) yields two consecutive \textsc{Fail} labels, a forced reboot is triggered by setting \(\mu_{\text{text}} \leftarrow 2\) in the next iteration.
While this workflow provides a high-level overview, further details are required to understand the underlying components. In the following subsections, we elaborate on the construction of the ASPs space and adversarial input mechanisms, followed by the reinforcement learning configuration, including network architecture, reward function, and training dynamics.
\begin{figure*}[t!]
\centering
\includegraphics[width=0.85\textwidth, keepaspectratio]{jailbreak.drawio0826.drawio.pdf}
\caption{Illustration of the PolyJailbreak workflow, starting with model reconnaissance and vulnerability analysis, followed by attack initialization, and then a reinforcement learning-driven optimization process that iteratively generates and refines jailbreak content until success or the preset step limit is reached.}
\label{fig:method}
\end{figure*}
\subsection{ASPs Library and Input Construction}
Our analysis indicates that the vulnerabilities of MLLMs stem from the asymmetry inherent in multimodal safety constraints. These asymmetries manifest in multiple exploitable forms. Examples include injecting carefully crafted dialogue history to reduce attention to harmful keywords, transforming or splitting malicious keywords to evade keyword-based safety triggers, replacing them with emojis, embedding critical malicious instructions as obfuscated artistic text within images (typora-style rendering), and designing complex multi-turn text-image interactions with either semantically consistent or contradictory pairings to mislead safety mechanisms. To systematically exploit these vulnerabilities, we design an ASPs library \(\mathcal{S}\), partitioned into three complementary subspaces: textual manipulation ASPs \(\mathcal{S}_{\text{text}}\), visual manipulation ASPs \(\mathcal{S}_{\text{image}}\), and prompt amplification ASPs \(\mathcal{S}_{\text{pers}}\). Each ASP encodes a reusable attack strategy, enabling efficient index-based retrieval and composable input construction during the optimization process. The details are described as follows:
\begin{tcolorbox}[
title=\textbf{Atomic Strategy Primitive Template},
colback=gray!2,
colframe=black!70,
fonttitle=\bfseries,
boxrule=0.8pt,
sharp corners,
enhanced,
breakable,
left=4pt,
right=4pt,
top=4pt,
bottom=4pt]
\ttfamily\small
\textbf{id}: \textit{strategy\_id} \\[2pt]
\textbf{type}: \textit{textual manipulation / visual manipulation / prompt amplification} \\[2pt]
\textbf{principle}: \\
\quad [Fundamental rationale behind the strategy’s ability to evade safety mechanisms] \\[2pt]
\textbf{method}: \\
\quad [Concrete manipulations or realizations that instantiate the strategy in practice] \\[2pt]
\quad 1.~[Step one of execution] \\
\quad 2.~[Step two of execution] \\
\quad 3.~[...] \\[2pt]
\textbf{case}: \\
\quad [The working examples that concretely demonstrates the strategy in action]
\end{tcolorbox}
\noindent\textbf{Textual Manipulation ASPs \(\mathcal{S}_{\text{text}}\).}
To exploit vulnerabilities at both character and semantic levels, we design multiple textual ASPs, covering character obfuscation, context fragmentation, role-play, and information dilution. Additionally, our empirical study identifies several high-impact techniques, including fabricated conversation history, emoji substitution, and system-instruction injection, which are incorporated to extend the coverage of the exploit space.
\noindent\textbf{Visual Manipulation ASPs \(\mathcal{S}_{\text{image}}\).}
We implement various visual ASPs, categorized into:
(i)~\emph{Image-generation strategies}, including semantically consistent/inconsistent synthesis and visual steganography. (ii)~\emph{Image-transformation strategies}, such as noise injection and block shuffling. Generation-based ASPs manipulate cross-modal consistency or embed hidden instructions, while transformation-based ASPs perturb pixel distributions or spatial layouts to disturb the attention of \(\mathcal{M}\).
\noindent\textbf{Prompt Amplification ASPs \(\mathcal{S}_{\text{pers}}\).}
Inspired by Zeng \emph{et al.}~\cite{Johnny}, we parameterize a spectrum of persuasion methods as standalone ASPs. By combining persuasion ASPs with textual and visual components, the Cartesian product allows adversaries to dynamically steer conversational tone and pragmatic framing without altering the malicious intent, which amplifies the effectiveness of jailbreak attempts by exploiting the model’s alignment blind spots.
Given a target instruction \(g\) and a selected strategy tuple \(\langle s_{\text{text}},\, s_{\text{image}},\, s_{\text{pers}} \rangle\), PolyJailbreak constructs textual and visual components of the adversarial input separately.
\textit{Text Construction.}
The objective is to combine \(g\) with selected strategies to effectively conceal or wrap the malicious intent. To automate and scale this process, we employ an attack LLM \(\mathcal{M}_A\) to generate the textual input \(x\). A macro-level mutation coefficient \(\mu_{\text{text}} \in \{0,1,2\}\) controls the granularity of modifications:
\(
0 = \textsc{Refine}, 1 = \textsc{Mutate}, 2 = \textsc{Reboot}.
\)
\(\textsc{Refine}\) conducts in-depth optimization of promising prompts, extending their meaning without deviating from the original direction. \(\textsc{Mutate}\) generates directional variations within the current strategy to explore nearby alternatives, whereas \(\textsc{Reboot}\) performs a full reset through the reselection of strategies. The tuple \(\langle g, s_{\text{text}}, s_{\text{pers}}, \mu_{\text{text}}, x_{t-1} \rangle\) is passed to \(\mathcal{M}_A\) along with a comprehensive system prompt that explicitly instructs the attack model on how to construct the target prompt, yielding the updated textual component \(x\).
\textit{Image Construction.}
For \textit{Image-generation} primitives (e.g., semantic consistency control, visual steganography), the attack LLM \(\mathcal{M}_A\) first produces a detailed natural-language description of the target image conditioned on \(g\) and any selected visual ASPs. This description is then passed to a diffusion model \(\mathcal{M}_{D}\), which synthesizes the corresponding image \(v\) while preserving the intended malicious semantics in a concealed or contextually aligned form. In contrast, for \textit{Image-transformation} primitives (e.g., noise injection, spatial shuffling), the process operates on an existing image, applying the specified transformation directly through the manipulation module to produce the modified image.
\subsection{Optimization Process}
To navigate the vast and heterogeneous strategy space for multimodal jailbreak prompts, PolyJailbreak adopts a reinforcement learning framework based on SAC. The Actor-Critic paradigm enables structured selection of multimodal strategies and reward-driven adaptation, allowing prompts to be refined over time. At each timestep, the agent samples a composite action from the current state, receives feedback from the target model and judging agent, and updates its policy to maximize jailbreak effectiveness.
\textbf{Network Architecture.}
The actor network is implemented as a Transformer encoder that serves as the policy network, taking the current state representation from the optimization process as input. This state encapsulates relevant contextual information and is mapped into a shared latent representation, which then branches into two parallel heads: one for discrete strategy selection and another for continuous parameter prediction. Discrete strategies are sampled via a differentiable Gumbel-Softmax mechanism, while continuous parameters provide fine-grained control over strategy execution. The resulting composite action combines these two outputs. The critic adopts a twin-Q design to stabilize training, and the final value estimate is taken as the minimum of the two Q-networks to mitigate overestimation bias.
\begin{algorithm}[t]
\footnotesize
\caption{Reward Function $\texttt{GetReward}(\cdot)$}
\label{alg:reward}
\textbf{Input:} The response $x$, reference answer $r^\star$, current image $v$, previous image $v^\star$, step index $t$, metrics $\{r_{\text{atk}}, r_{\text{harm}}, \Delta r_{\text{jb}}, r_{\text{refuse}}, r_{\text{step}}\}$, hyperparameters $\{\alpha_i\}$, $\beta$, $\{\gamma_i\}$, text encoder $E_t$, perceptual hash function $\texttt{pHash}(\cdot)$, grayscale edge variance function $\texttt{EdgeVar}(\cdot)$. \\
\textbf{Output:} Final reward $r_t$.
\begin{algorithmic}[1]
\STATE // Safety feedback reward.
\STATE $\mathcal{R}_{\text{safe}} = \alpha_1 r_{\text{atk}} + \alpha_2 r_{\text{harm}} + \alpha_3 \Delta r_{\text{jb}} - \alpha_4 r_{\text{refuse}} - \alpha_5 r_{\text{step}}$
\STATE // Semantic similarity reward.
\STATE Encode $x$ and $r^\star$ with text encoder $E_t$.
\STATE $\mathcal{R}_{\text{sim}} = \beta \cdot \big( E_t(x) \cdot E_t(r^\star) \big) / \big( \|E_t(x)\|_2 \cdot \|E_t(r^\star)\|_2 \big)$
\STATE // Stylistic diversity: text-level.
\STATE $H_{\text{char}} = - \sum_{c \in \mathcal{C}} p(c)\,\log_2 p(c), p(c) = \text{count}(c) / \sum_{c'} \text{count}(c')$ // character entropy.
\STATE $R_{\text{vocab}} = |\text{unique tokens}(x)| / |\text{tokens}(x)|$ // vocabulary richness.
\STATE $S_{\text{tfidf}} = 1 - (\|\mathbf{v}\|_0 / \dim(\mathbf{v}))$,
where $\mathbf{v}$ is constructed from $x$ using standard TF-IDF weighting, $\|\mathbf{v}\|_0$ counts its nonzero entries. // TF-IDF sparsity.
\STATE $\mathcal{A}_{\text{text}} = \alpha_1 \cdot (H_{\text{char}}/H_{\max}) + \alpha_2 \cdot R_{\text{vocab}} + \alpha_3 \cdot S_{\text{tfidf}}$
\STATE // Stylistic diversity: image-level.
\IF{ $v^\star$ is available}
\STATE $ \mathcal{A}_{\text{image}} = \|\texttt{pHash}(v) - \texttt{pHash}(v^\star)\| / 64$,
\STATE where 64 is the bit length of the pHash code.
\ELSE
\STATE $\mathcal{A}_{\text{image}} = \texttt{EdgeVar}(v) / Z$,
\STATE where $Z$ is a normalization constant ensuring $\mathcal{A}_{\text{image}} \in [0,1]$.
\ENDIF
\STATE $\mathcal{R}_{\text{style}} = \gamma_1 \cdot \mathcal{A}_{\text{text}} + \gamma_2 \cdot \mathcal{A}_{\text{image}}$
\STATE // Final reward.
\STATE $r_t = \mathcal{R}_{\text{safe}} + \mathcal{R}_{\text{sim}} + \mathcal{R}_{\text{style}}$
\STATE \textbf{return} $r_t$
\end{algorithmic}
\end{algorithm}
\textbf{Reward Function.}
The reward function \(\texttt{GetReward}(\cdot)\) is designed to guide the policy toward generating effective and diverse jailbreak prompts.
It consists of three main components:
(1) \textit{Safety feedback} \(\mathcal{R}_{\text{safe}}\), which balances attack strength, harmfulness, jailbreak success, and refusal/step penalties;
(2) \textit{Semantic similarity} \(\mathcal{R}_{\text{sim}}\), which measures the alignment between the generated response and a reference answer through text embedding similarity;
(3) \textit{Stylistic diversity} \(\mathcal{R}_{\text{style}}\), which encourages linguistic and visual variation in prompts.
The overall reward integrates these components as summarized in Algorithm~\ref{alg:reward}.
\textbf{Policy Update.}
When the replay buffer reaches a threshold, the actor and critic are jointly updated.
The critic minimizes the temporal difference loss:
\begin{equation}
\mathcal{L}_{\text{critic}} = \mathbb{E} \left[ \left(Q_{\phi_j}(s_t, a_t) - y_t \right)^2 \right]
\end{equation}
where $s_t$ and $a_t$ denote the state and action at timestep $t$, $Q_{\phi_j}$ is the $j$-th critic parameterized by $\phi_j$,
and $y_t$ is the soft target Q-value.
The actor maximizes expected return with entropy regularization:
\begin{equation}
\scalebox{0.9}{$\displaystyle
\mathcal{L}_{\text{actor}} =
\mathbb{E}_{s_t \sim \mathcal{B}} \left[
\mathbb{E}_{a_t \sim \pi_\theta} \left[
\lambda \log \pi_\theta(a_t | s_t) - \min_{j=1,2} Q_{\phi_j}(s_t, a_t)
\right] \right]$}
\end{equation}
where $\pi_\theta$ is the policy parameterized by $\theta$, and $\lambda$ is the entropy regularization coefficient encouraging exploration. Both networks are updated iteratively. This training scheme enables robust exploration of multimodal strategy combinations and effective optimization of adversarial input generation.
\begin{table*}[t]
\centering
\caption{Baseline comparison with ASR (\%) and HS (0-5). Best results are shown in \textbf{bold}, and second-best are \underline{underlined}.}
\label{tab:Comparative_experiment}
\scriptsize
\setlength{\tabcolsep}{3.5pt}
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{l||*{8}{c@{\hskip 4pt}c}|c}
\toprule
\multirow{3}{*}{\textbf{Method}}
& \multicolumn{16}{c|}{\textbf{Target Model}}
& \multirow{2}{*}{\textbf{Average}} \\ \cline{2-17}
& \multicolumn{2}{c}{\textbf{LLaVA-v1.5}}
& \multicolumn{2}{c}{\textbf{LLaVA-v1.6}}
& \multicolumn{2}{c}{\textbf{Qwen2.5-VL}}
& \multicolumn{2}{c}{\textbf{LLaMA3.2-Vision}}
& \multicolumn{2}{c}{\textbf{GPT-4o}}
& \multicolumn{2}{c}{\textbf{GPT-4.1}}
& \multicolumn{2}{c}{\textbf{Gemini-2.5-Flash}}
& \multicolumn{2}{c|}{\textbf{Claude-3-7-Sonnet}}
& \\ \cline{2-18}
& ASR & HS & ASR & HS & ASR & HS & ASR & HS
& ASR & HS & ASR & HS & ASR & HS & ASR & HS
& ASR / HS \\
\midrule
% \multicolumn{16}{c}{\textbf{Text-only Input Method}} \\
% \midrule
JOOD &29.50 &2.189 &50.25 &2.905 &18.25 &1.861 &28.25 &1.900 &31.50 &2.259 &17.50 &1.605 &23.00 &1.915 &00.75 &1.054 &24.88 / 1.961 \\
DRA &64.00 &\textbf{4.301} &69.00 &\uline{4.389} &64.25 &\textbf{4.249} &\uline{71.75} &\textbf{4.463} &24.00 &2.752 &50.00 &3.684 &82.25 &\uline{4.536} &08.25 &2.617 &54.19 / \uline{3.874} \\
DarkCite &71.75 &2.709 &82.50 &2.934 &\uline{77.25} &2.937 &58.00 &2.385 &68.00 &2.394 &75.50 &2.598 &58.75 &2.664 &29.75 &1.590 &\uline{65.19} / 2.526 \\
ArtPrompt &54.50 &3.187 &12.75 &2.565 &13.75 &1.550 &29.00 &1.975 &35.75 &2.207 &23.50 &1.736 &35.50 &2.132 &06.25 &1.382 &26.38 / 2.092 \\
FlipAttack &13.25 &2.486 &24.50 &3.106 &17.50 &2.743 &53.50 &3.250 &\uline{90.75} &\textbf{4.769} &\textbf{90.50} &\textbf{4.691} &\uline{96.50} &\textbf{4.857} &\textbf{46.00} &\uline{3.165} &54.06 / 3.633\\
% \midrule
% \multicolumn{16}{c}{\textbf{Text-Image Input Method}} \\
% \midrule
FigStep &76.25 &4.062 &\uline{84.50} &4.154 &24.00 &2.035 &62.75 &3.280 &03.00 &1.108 &02.25 &1.087 &05.25 &1.184 &00.00 &1.030 &32.25 / 2.243\\
Hades &56.50 &3.299 &50.75 &2.904 &16.75 &1.788 &06.00 &1.255 &14.25 &1.535 &03.75 &1.209 &09.00 &1.317 &00.00 &1.016 &19.63 / 1.790\\
MM-SafetyBench &76.25 &3.829 &62.00 &3.333 &23.50 &2.054 &10.00 &1.360 &18.75 &1.593 &07.50 &1.197 &10.00 &1.343 &00.50 &1.030 &26.06 / 1.967\\
Query-Attack &\uline{77.75} &3.998 &27.50 &2.047 &23.50 &1.107 &17.50 &1.137 &02.00 &1.134 &00.75 &1.050 &00.00 &1.008 &00.00 &1.013 &18.63 / 1.562\\
MML-M &46.75 &2.810 &37.54 &2.354 &15.50 &1.714 &06.50 &1.297 &14.75 &1.540 &03.75 &1.181 &08.50 &1.319 &00.00 &1.014 &16.66 / 1.654\\
\midrule
\textbf{PolyJailbreak} &\textbf{98.50} &\uline{4.164} &\textbf{97.00} &\textbf{4.443} &\textbf{80.00} &\uline{3.762} &\textbf{75.94} &\uline{3.517} &\textbf{97.50} &\uline{4.280} &\uline{86.00} &\uline{4.118} &\textbf{97.25} &4.343 &\uline{34.50} &\textbf{3.179} &\textbf{83.34 / 3.976} \\
\bottomrule
\end{tabular}
\end{table*}
\section{Evaluation}
In this section, we conduct comprehensive experiments to evaluate the effectiveness and generalizability of PolyJailbreak. Our evaluation addresses the following questions:
\begin{itemize}[leftmargin=*]
\item \textbf{Effectiveness Study:} How effective are PolyJailbreak’s jailbreak prompts relative to those of existing methods?
\item \textbf{Ablation Study:} How do key components, including multimodal inputs, optimization steps, and strategy refinement, impact the overall performance of PolyJailbreak?
\item \textbf{Vulnerability Study:} Do different MLLMs exhibit distinct response patterns to specific attack categories or strategy combinations, revealing inherent structural vulnerabilities?
\end{itemize}
\subsection{Experimental Setup}
\textbf{Target Models.}
To assess generalizability, we evaluate PolyJailbreak on eight state-of-the-art MLLMs. The testbed comprises four closed-source models (GPT-4o, GPT-4.1, Gemini-2.5-Flash (0520), and Claude-3.7-Sonnet (20250219)) and four open-source models (LLaVA-1.5 (7B), LLaVA-1.6 (7B)~\cite{llava-1.6}, LLaMA-3.2-Vision (11B), and Qwen-2.5-VL (7B) \cite{qwen2.5-VL}). All models are safety-aligned and exhibit refusal behavior for overtly harmful requests.
\noindent\textbf{Baselines.}
We compare PolyJailbreak against ten representative state-of-the-art jailbreak methods, including ArtPrompt~\cite{ArtPrompt}, DarkCite~\cite{DarkCite}, DRA~\cite{DRADRA}, JOOD~\cite{JOOD}, FlipAttack~\cite{FlipAttack}, FigStep~\cite{FigStep}, Hades~\cite{hades}, MM-SafetyBench~\cite{MM-SafetyBench}, Query-Attack~\cite{Query-Attack}, and MML-M~\cite{MML-M}. These baselines span the dominant paradigms of jailbreak research, providing a comprehensive foundation for comparative evaluation.
\noindent\textbf{Metrics.}
We employ two metrics to quantify jailbreak effectiveness and efficiency.
(i) \textit{Attack Success Rate (ASR)}.
We use ASR to quantify the effectiveness of adversarial inputs in bypassing a target model’s safety mechanisms. A jailbreak is considered successful if the model generates a harmful response, as judged by an external safety classifier $\mathcal{R}_{J}$~\cite{hades}. ASR is then defined as the proportion of malicious instructions that lead to successful jailbreaks over the total number of instructions. A higher ASR indicates a stronger jailbreak method and reflects greater vulnerability of the model to adversarial prompts.
(ii) \textit{Harmfulness Score (HS):} Following Zhao \textit{et al.}~\cite{HS}, we assess response severity on a five-point Likert scale (1: no harm, 5: extreme harm). Consistent with prior studies, we employ GPT-4o to assign the scores, with higher values indicating more severe policy violations. For experimental setup details, please refer to Appendix~\ref{appendix:experimental_setup}.
\begin{table*}[htbp]
\centering
\caption{The ASR (\%) results of different target models under various input modality combinations.}
\renewcommand{\arraystretch}{1.2}
\resizebox{\textwidth}{!}{
\begin{tabular}{
>{\centering\arraybackslash}m{6.8cm} % 左侧 Input Combinations，固定宽度+自动换行
cccccccc % 中间8个模型列
>{\centering\arraybackslash}m{1.8cm} % Average，固定宽度
l *{9}{>{\large}c}
}
\toprule
\multirow{2}{*}{\textbf{Input Combinations}} &
\multicolumn{8}{c}{\textbf{Target Model}} &
\multirow{2}{*}{\textbf{Average}} \\
\cmidrule(lr){2-9}
& \textbf{LLaVA-v1.5} & \textbf{LLaVA-v1.6} & \textbf{Qwen2.5-VL} & \textbf{LLaMA3.2-Vision}
& \textbf{GPT-4o} & \textbf{GPT-4.1} & \textbf{Gemini-2.5-Flash} & \textbf{Claude-3-7-Sonnet} \\
\midrule
\large Original Instructions & \large 22.75 & \large 21.25 & \large 01.50 & \large 18.05 & \large 04.00 & \large 01.50 & \large 02.00 & \large 00.50 & \large 08.94 \\
\large Optimized Prompts & \large 83.75 & \large 86.25 & \large 71.00 & \large 53.38 & \large 71.00 & \large 65.75 & \large 25.50 & \large 12.25 & \large 58.61 \\
\large Original Instructions+Optimized images & \large 59.00 & \large 58.75 & \large 10.50 & \large 03.01 & \large 09.50 & \large 02.50 & \large 25.00 & \large 00.50 & \large 21.10 \\
\large PolyJailbreak & \large 98.50 &\large 97.00 & \large 80.00 & \large 75.94 & \large 97.50 & \large 86.00 & \large 97.25 & \large 34.50 & \large 83.34 \\
\bottomrule
\end{tabular}
}
\label{tab:polyjailbreak-results}
\end{table*}
\subsection{Effectiveness Study}
We evaluate whether PolyJailbreak can effectively transform malicious instructions into successful jailbreak attacks across diverse MLLMs. Table~\ref{tab:Comparative_experiment} reports the ASR and average HS of PolyJailbreak and ten baseline methods.
PolyJailbreak outperforms existing methods across all eight target models, demonstrating robust generalizability to both closed-source and open-source MLLMs. Notably, it achieves over 95\% ASR on GPT-4o, Gemini-2.5-Flash, LLaVA-1.5, and LLaVA-1.6. In the context of the remaining four models, PolyJailbreak consistently secures a top-two ranking. Across all evaluated models, PolyJailbreak attains an average ASR of 83.34\% and an HS of 3.976, consistently surpassing baseline methods. These results indicate that PolyJailbreak not only bypasses safety mechanisms but also induces semantically coherent and policy-violating outputs. DRA achieves high HS on select open-source models but suffers from low ASR due to limited strategy adaptability. While FlipAttack achieves relatively strong results on closed-source models, it fails to generalize, with ASR decreasing to below 20\% on smaller-scale models. Multimodal baselines such as FigStep, Hades, and MML-M exhibit poor effectiveness on models like LLaMA and several closed-source systems, where PolyJailbreak achieves ASR improvements exceeding 60\%. These findings underscore the advantage of PolyJailbreak’s dynamic multimodal optimization framework over static template-driven attacks.
\noindent\textbf{Key Observations.} A manual inspection of adversarial prompts and outputs reveals three critical insights:
\textit{(1) Model scale and parsing capability determine attack effectiveness.}
FlipAttack’s use of long, obfuscated prompts favors large proprietary models (e.g., GPT-4.1, Claude), which possess stronger instruction parsing capabilities. In contrast, the same strategy severely impairs smaller open-source models, leading to hallucinations and misinterpretations. This highlights the necessity of tailoring jailbreak strategies to model-specific characteristics, with offensive techniques adaptive to model scale and architecture.
\textit{(2) Multimodal attacks are not always the optimal choice.}
We find text-only methods generally outperform multimodal baselines on both LLaMA series and closed-source models. We attribute this to two factors: (i) Recent improvements in multimodal safety alignment have enhanced the detection of harmful visual inputs, rendering image-centric attacks (e.g., FigStep, Query-Attack) ineffective. (ii) Existing multimodal methods often overlook the interplay between textual obfuscation and cross-modal semantics (e.g., Hades, MM-SafetyBench). In contrast, the expressive flexibility of text enables PolyJailbreak to craft adversarial prompts that evade detection while maintaining attack intent.
\textit{(3) PolyJailbreak’s model-aware collaborative optimization drives success.}
PolyJailbreak’s initial \textit{Model Discovery} phase profiles refusal templates, safety guidelines, published policy rules, and other relevant artifacts of the target model, providing structured priors for reinforcement learning. This enables precise strategy composition: avoiding hallucination-prone long prompts (as seen in FlipAttack) and enhancing text-image synergy absent in existing multimodal methods.
%\begin{figure}[tbp]
% \centering
% \begin{subfigure}[b]{0.21\textwidth} % 每张子图占单栏宽度的48%
% \centering
% \includegraphics[width=\linewidth]{llama_radar_usenix.pdf}
% \caption{LLaMA as source}
% \label{fig:radar_llama}
% \end{subfigure}
% \hfill % 自动填充间距，替代固定\hspace
% \begin{subfigure}[b]{0.21\textwidth}
% \centering
% \includegraphics[width=\linewidth]{claude_radar_usenix.pdf}
% \caption{Claude as source}
% \label{fig:radar_claude}
% \end{subfigure}
% \caption{Comparison of transfer attack effectiveness.}
% \label{fig:radar_transfer_comparison}
%\end{figure}
\begin{figure}[tbp]
\centering
\includegraphics[width=0.42\textwidth]{llama_claude_radar_usenix0826.pdf} % 调整宽度为页面的80%
\caption{Comparison of transfer attack effectiveness.}
\label{fig:radar_transfer_comparison}
\end{figure}
\subsection{Ablation Study}
We perform ablation experiments to assess the impact of (i) input modality, (ii) cross-model transferability, and (iii) strategy search algorithms on PolyJailbreak’s effectiveness.
\begin{figure*}[t!]
\centering
\includegraphics[width=0.95\textwidth]{ASR_curves_all_models_legend_each0826.pdf}
\caption{Cumulative ASR (\%) comparison of PolyJailbreak, RS, and CMA-ES across five optimization steps on target MLLMs.}
\label{fig:searchBaseline}
\end{figure*}
\noindent\textbf{Impact of Input Modality Combinations.}
To assess the contribution of different input modalities, we compare four settings:
(1) \textit{Original Instructions}: direct use of unmodified malicious prompts;
(2) \textit{Optimized Prompts}: text-only adversarial prompts generated by PolyJailbreak;
(3) \textit{Original Instructions + Optimized Images}: pairing the original prompt with a crafted adversarial image (blank image used where inapplicable);
(4) \textit{PolyJailbreak (Full)}: joint optimization of both text and image inputs.
Table~\ref{tab:Comparative_experiment} reports the ASR results across all eight models. The full PolyJailbreak input consistently achieves the highest attack success rates, confirming the advantage of coordinated multimodal optimization. Replacing only the text with optimized prompts yields a significant ASR gain, with an average improvement of 49.67\%. However, this boost diminishes against models with advanced safety alignment (e.g., GPT-4.1, Gemini, Claude), where textual transformations alone fail to bypass robust safety mechanisms. Conversely, injecting optimized images while retaining the original instruction leads to noticeable ASR increases on models with weaker vision alignment. Yet, on models like LLaMA and Claude, this strategy is less effective than the original prompt, indicating that unimodal perturbations are insufficient against enhanced multimodal defenses. Overall, textual optimization primarily mitigates refusal triggers, while adversarial images provide a complementary evasion channel. Their synergy is key to maximizing jailbreak success.
\noindent\textbf{Attack Transferability Across Models.}
We further investigate PolyJailbreak's cross-model generalizability through transfer attacks. Specifically, we select optimized inputs crafted for LLaMA and Claude, where PolyJailbreak exhibits comparatively lower performance, and apply them without modification to the remaining target models. This simulates a black-box transfer scenario. For reference, we include two baselines: (1) original unoptimized instructions and (2) inputs optimized specifically for each target model by PolyJailbreak. The ASR results, visualized in Figure~\ref{fig:radar_transfer_comparison}, reveal that inputs optimized on LLaMA and Claude exhibit strong transferability, successfully inducing jailbreak responses across diverse models. Remarkably, Claude-optimized inputs, despite underperforming on their source model, often achieve higher transfer ASR than LLaMA-optimized inputs. This suggests that PolyJailbreak, while targeting specific model weaknesses, can inadvertently discover broadly exploitable adversarial patterns that generalize across architectures and alignment strategies.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.45\textwidth]{global_combo_hbar0826.pdf} % 调整宽度为页面的80%
\caption{Successful jailbreak strategies under target models.}
\label{fig:RQ5}
\end{figure}
These findings highlight PolyJailbreak’s capability to generate transferable attack prompts, demonstrating its potential as a black-box jailbreak framework. The generalization of attacks across models with varying backbones, data scales, and safety alignment methods highlights the systemic nature of MLLMs vulnerabilities.
\noindent\textbf{Effectiveness of Strategy Search Algorithms.}
To evaluate the necessity of reinforcement learning in PolyJailbreak’s strategy search, we compare it against Random Search (RS) and Covariance Matrix Adaptation Evolution Strategy (CMA‑ES), tracking ASR progression over the first five optimization steps. All methods share the same adversarial prompt pipeline, differing only in strategy selection mechanisms. As shown in Figure~\ref{fig:searchBaseline}, PolyJailbreak consistently outperforms RS and CMA‑ES across all eight MLLMs, achieving faster ASR gains and higher final success rates. On safety-hardened models like GPT‑4.1 and Claude, PolyJailbreak surpasses RS and CMA‑ES by 19.25\% and 10.75\% respectively at step 5. Although CMA-ES attains competitive results on Gemini, such performance is likely attributable to serendipitous exploration of a favorable search direction, rather than to systematic improvements in the algorithm’s search efficiency. Overall, reinforcement learning-driven exploration proves more effective than heuristic methods in navigating the high-dimensional multimodal strategy space. Interestingly, the strong performance of RS and CMA‑ES despite their simplicity underscores the strength of our ASP library and prompt generation method.
\subsection{Vulnerability Study}
\begin{table*}[t]
\centering
\caption{Categories comparison with ASR (\%) and HS (0-5). Best results are shown in \textbf{bold}, and second-best are \underline{underlined}.}
\label{tab:Comparative_category}
\scriptsize
\setlength{\tabcolsep}{3.3 pt}
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{l||*{8}{c@{\hskip 4pt}c}|c}
\toprule
\multirow{3}{*}{\textbf{Category}}
& \multicolumn{16}{c|}{\textbf{Target Model}}
& \multirow{2}{*}{\textbf{Average}} \\ \cline{2-17}
& \multicolumn{2}{c}{\textbf{LLaVA-v1.5}}
& \multicolumn{2}{c}{\textbf{LLaVA-v1.6}}
& \multicolumn{2}{c}{\textbf{Qwen2.5-VL}}
& \multicolumn{2}{c}{\textbf{LLaMA3.2-Vision}}
& \multicolumn{2}{c}{\textbf{GPT-4o}}
& \multicolumn{2}{c}{\textbf{GPT-4.1}}
& \multicolumn{2}{c}{\textbf{Gemini-2.5-Flash}}
& \multicolumn{2}{c|}{\textbf{Claude-3-7-Sonnet}}
& \\ \cline{2-18}
& ASR & HS & ASR & HS & ASR & HS & ASR & HS
& ASR & HS & ASR & HS & ASR & HS & ASR & HS
& ASR / HS \\
\midrule
Criminal &\uline{98.00} &\uline{4.396} &\uline{98.00} &4.542 &76.00 &3.808 &70.00 &\uline{3.874} &\uline{98.00} &3.652 &86.00 &\textbf{4.520} &94.00 &4.407 &16.00 &3.090 &79.50 / 4.036 \\
Harassment &96.00 &4.084 &94.00 &4.226 &74.00 &3.580 &68.00 &3.380 &\textbf{100.00} &4.310 &90.00 &4.154 &96.00 &4.443 &\uline{38.00} &2.990 &82.00 / 3.896 \\
Hate &\uline{98.00} &3.706 &94.00 &4.052 &70.00 &3.208 &74.00 &3.288 &\uline{98.00} &3.798 &92.00 &3.732 &\uline{98.00} &4.085 &\textbf{42.00} &2.770 &83.25 / 3.580 \\
Misinformation &\textbf{100.00} &4.266 &96.00 &4.476 &\uline{86.00} &\textbf{4.018} &74.00 &3.178 &\textbf{100.00} &\textbf{4.550} &\textbf{100.00} &3.918 &\textbf{100.00} &\textbf{4.765} &\textbf{42.00} &2.970 &\uline{87.25} / 4.013 \\
Self-harm &96.00 &3.892 &94.00 &\uline{4.566} &\uline{86.00} &3.846 &76.00 &3.490 &94.00 &4.106 &86.00 &3.934 &96.00 &4.261 &36.00 &3.222 &83.00 / 3.915\\
Terrorism &\textbf{100.00} &4.390 &\textbf{100.00} &4.392 &\textbf{90.00} &3.802 &\uline{80.00} &3.596 &\textbf{100.00} &4.458 &\uline{94.00} &4.152 &\textbf{100.00} &4.497 &\textbf{42.00} &\uline{3.416} &\textbf{88.25} / \uline{4.088}\\
Violence &\textbf{100.00} &4.204 &\textbf{100.00} &4.456 &76.00 &3.898 &78.00 &3.432 &\textbf{100.00} &4.346 &86.00 &4.202 &\uline{98.00} &4.489 &\textbf{42.00} &3.410 &85.00 / 4.055\\
Weapons &\textbf{100.00} &\textbf{4.434} &\textbf{100.00} &\textbf{4.836} &82.00 &\uline{3.938} &\textbf{87.76} &\textbf{3.894} &90.00 &\uline{4.516} &54.00 &\uline{4.336} &96.00 &\uline{4.527} &18.00 &\textbf{3.564} &78.47 / \textbf{4.256}\\
\bottomrule
\end{tabular}
\end{table*}
We further examine how different malicious intents and strategy combinations affect MLLM defenses, exposing structural vulnerabilities exploited by PolyJailbreak.
First, we evaluate the ASR and HS across eight prohibited categories, namely \textit{Criminal}, \textit{Harassment}, \textit{Hate}, \textit{Misinformation}, \textit{Self-harm}, \textit{Terrorism}, \textit{Violence}, and \textit{Weapons}. As shown in Table~\ref{tab:Comparative_category}, PolyJailbreak achieves consistently high ASR and HS across categories, demonstrating strong generalizability to diverse malicious categories, which indicates that MLLM safety mechanisms fail uniformly under adaptive attacks. Notably, \textit{Misinformation} and \textit{Terrorism} yield the highest ASR and HS, likely because their neutral and informative linguistic framing evades refusal triggers. This suggests that models are vulnerable when harmful intent is obscured by ostensibly factual or instructional language. In contrast, \textit{Weapons} yields the lowest ASR but the highest HS, indicating that although jailbreak cases are fewer than in other categories, those that succeed lead to particularly severe consequences. These findings reveal that MLLMs lack uniform robustness across threat vectors: linguistically subtle categories (e.g., \textit{Misinformation}) are harder to defend, while explicit categories (e.g., \textit{Weapons}) pose high-risk failure modes. PolyJailbreak's ability to expose these asymmetric vulnerabilities underscores the need for context-aware safety alignment.
Next, we analyze how triplet combinations of strategies influence model vulnerability. Figure~\ref{fig:RQ5} shows the distribution of successful strategy combinations. The top 15 groups are highlighted, while the proportion and frequency of the remaining combinations are also indicated. Our analysis reveals that certain strategy triplets, such as \textit{txt\_roleplay\_expert + img\_semantic\_conflict + Authority Endorsement}, maintain consistently high jailbreak rates across different models, suggesting their generalizability beyond a single model. However, robust jailbreak performance is not confined to a small strategy set: non-top-15 combinations account for 41.2\% to 68.6\% of successful jailbreaks, with closed-source models averaging over 60\%. All models have at least 84 distinct successful combinations. These findings highlight a critical weakness: MLLM defenses can be bypassed through a wide spectrum of distinct multimodal attack pathways, including many long-tail strategies beyond the most frequently used patterns. This underscores the systemic vulnerability of current defenses to diverse and unanticipated attacks. By adaptively discovering and exploiting this diversity, PolyJailbreak achieves consistent cross-model jailbreak success.
We further investigate model preference across strategy types. Figure~\ref{fig:asr_model_heatmaps} presents a heatmap of strategy effectiveness across models, revealing both consistent and model-specific patterns. For textual manipulation strategies, all models favor \textit{txt\_roleplay\_expert} and \textit{txt\_indirect\_request}, with average ASR of 49\% and 20\%, respectively. These tactics obscure malicious intent via natural, human-like phrasing, indicating that MLLMs may overly rely on surface-level intent recognition. In contrast, \textit{txt\_escape\_markdown} consistently performs poorly, suggesting that format-based defenses are effective against such inputs. Among visual manipulation strategies, \textit{img\_semantic\_conflict} yields the highest ASR (>40\%), revealing that semantic mismatch between image and text disrupts multimodal alignment. Visual obfuscation techniques, such as emoji-embedded or stylized harmful text, also enhance jailbreak success, showing the effectiveness of visually deceptive cues. However, image-based strategies are not universally beneficial. For example, in closed-source models like GPT-4o, GPT-4.1, and Gemini, purely textual prompts already achieve an average ASR of 17.2\%. For prompt amplification strategies, models show strong preference for \textit{Authority Endorsement}, \textit{Expert Endorsement}, and \textit{Alliance Building}, which leverage credible rhetorical framing to bypass ethical safeguards. Structurally similar models (e.g., LLaVA-v1.5 and LLaVA-v1.6, GPT-4o and GPT-4.1) exhibit aligned preferences, likely due to shared architectural backbones or alignment schemes.
%\section{Discussion}
%\noindent\textbf{Limitations.}
%While our study advances the systematic understanding and red-teaming of MLLMs through the design of PolyJailbreak, several aspects warrant further refinement. First, while the effectiveness of PolyJailbreak has been empirically validated, we observe performance differences on a small subset of models (e.g., Claude), reflecting the broader challenge of capturing vulnerabilities across diverse and model-specific safety mechanisms. Second, our evaluation relies on LLM-based classifiers to judge jailbreak success. Such classifiers, while practical and widely adopted, may be influenced by training data or vendor-specific alignment preferences. Third, the multi-agent framework provides scalability and automation, but agents themselves can introduce noise such as hallucinations or safety-driven constraints. In some cases, generated prompts deviated from the original malicious intent, a limitation that is common in automated red-teaming pipelines. Addressing these issues in future work can further strengthen the robustness and applicability of our approach.
%
%\noindent\textbf{Future Work.}
%Future work can advance in several directions. An immediate priority is to move beyond input-level attacks and uncover new vulnerabilities in output-level defenses, including response filtering, post-generation moderation, and safety layers that re-rank or sanitize outputs. Another direction is to construct a structured taxonomy of vulnerabilities grounded in the end-to-end multimodal processing pipeline and its safety response mechanisms. Organizing vulnerabilities along the actual flow of information, from input encoding through cross-modal fusion to decoding and filtering, would enable more systematic discovery. In addition, refining the automation framework remains essential. Stronger control over the generation process and higher fidelity to adversarial intent could be achieved by integrating constraint-aware objectives or verification-guided search. Such advances would improve the robustness of large-scale jailbreak generation and enhance transferability across alignment strategies.
\section{Conclusion}
In this article, we have exposed and experimentally validated the multimodal safety asymmetry introduced by visual alignment, as well as the vulnerabilities triggered by visual inputs. We have developed PolyJailbreak, a black-box jailbreak method targeting MLLMs. Our approach has introduced a composable strategy framework, grounded in a library of Atomic Strategy Primitives and guided by reinforcement learning, which has systematically explored the multimodal attack space. Through extensive evaluation on mainstream MLLMs, we showed that PolyJailbreak can reliably compromise both open-source and commercial systems, exposing persistent vulnerabilities even in the most robust models. These results underscore the pressing need for modality-aware alignment and defense mechanisms to ensure the secure deployment of next-generation MLLMs.
%\section*{Ethical Considerations}
%\noindent\textbf{Stakeholders and Principles.}
%Stakeholders include end users who rely on multimodal large language models (MLLMs), model providers (commercial and open-source), the security research community, and our own research team. We ground our analysis in the Menlo Report principles-Beneficence, Respect for Persons, Justice, and Respect for Law and Public Interest-and follow the perspective of Ethical Frameworks and Computer Security Trolley Problems. Our overarching goal is to reduce real-world risk by revealing multimodal safety asymmetry and leveraging these insights to strengthen red-teaming practices and improve MLLM defenses.
%
%\noindent\textbf{Scope of Experiments.}
%All experiments were conducted under a black-box, single-turn threat model, with open-source MLLMs run locally and commercial MLLMs accessed through their standard APIs. We did not attempt to subvert provider infrastructure, alter models, exceed usage limits, or implant persistent content. No personal or sensitive user data were collected, and all work complied with API terms of service and applicable laws.
%
%\noindent\textbf{Potential Harms and Mitigations.}
%Potential risks include facilitating misuse by lowering the barrier to jailbreak attacks, exposing researchers to harmful content, and reputational impact to model providers. To mitigate these risks, we adopted strict dual-use controls: no harmful outputs, high-risk prompts, or step-by-step attack recipes are released, and all shared artifacts (e.g., evaluation scripts) are sanitized and strictly restricted to research purposes. Researcher exposure was minimized through controlled environments, content warnings, and opt-out policies. We further conducted responsible disclosure, notifying reachable providers through email and bug-report channels and offering to share technical details under coordinated disclosure agreements.
%
%\noindent\textbf{Decision.}
%We judge that the societal and scientific benefits, such as advancing the understanding of multimodal safety risks and improving red-teaming and defensive practices, outweigh the residual risks after mitigation. Publication is paired with carefully restricted, defense-oriented disclosures to further minimize potential misuse.
%
%\noindent\textbf{Limitations and Attestations.}
%This analysis may not fully capture future deployment contexts, such as new modalities or post-publication model updates, and we commit to revisiting our disclosures should new risks emerge. The authors attest that (i) the ethics guidelines and CFP instructions were read and considered, and (ii) the research was conducted ethically to the best of our knowledge.
%
%\section*{Open Science}
%We release all artifacts in an anonymized repository at \url{https://anonymous.4open.science/r/PolyJailbreak-86JL} (access granted at submission).
%The repository includes \texttt{Baseline/} (comparison baselines), \texttt{RQ1/} and \texttt{RQ2/} (analyses for each research question), \texttt{jailbreak/} (PolyJailbreak implementation), and datasets (\texttt{final\_dataset.json} and derived variants).
%These artifacts are sufficient to reproduce our experiments on open-source and API-based commercial MLLMs. All materials are anonymized and available at submission; upon acceptance, we will provide a permanent public link and ensure long-term availability.
%
%We have released a subset of successful jailbreak cases in our repository. Readers are advised that the repository includes samples which may contain harmful or disturbing content, and caution should be exercised when accessing them.
\bibliographystyle{plain}
\bibliography{reference}
\appendix
\section{Dataset Construction}
\label{appendix:dataset_construction}
The dataset is constructed through the following steps:
\begin{enumerate}[label=(\roman*)]
\item \textbf{Categorization and filtering:} We categorize and filter existing AdvBench samples, removing redundant or semantically overlapping prompts within each category to ensure diversity.
\item \textbf{Expansion with novel harmful instructions:} Each category is expanded to 50 harmful samples by leveraging both manually crafted instructions and GPT-4o-assisted generation, thereby incorporating malicious intents that were previously underexplored.
\item \textbf{Benign counterpart generation:} For every harmful instruction, we construct a benign variant that mirrors the original intent but aligns with ethical norms. This ensures that harmful and benign prompts share similar context, allowing for controlled comparative evaluation.
\end{enumerate}
%Additional representative examples for each category are listed in Table~\ref{table:dataset_examples}.
\begin{table}[htbp]
\small
\centering
\caption{Evaluation Dataset Category Descriptions}
\label{table:dataset}
\begin{tabularx}{\linewidth}{
>{\centering\arraybackslash}m{2.2cm}
>{\arraybackslash}X
}
\toprule
\textbf{Category} & \multicolumn{1}{c}{\textbf{Description}} \\
\midrule
Criminal & Cybercrime, fraud schemes, software exploits, and unauthorized data access. \\
Harassment & Online abuse involving bullying, threats, stalking behaviors, and personal attacks. \\
Hate & Promotion of hate speech, racial hostility, and identity-based discrimination tactics. \\
Misinformation & Spreading false claims, conspiracy theories, health hoaxes, and fabricated news content. \\
Self-harm & Encouraging suicide, self-injury methods, and dangerous behavior justification. \\
Terrorism & Guiding extremist activities, attack planning, and recruitment for radical ideologies. \\
Violence & Facilitating threats, physical harm strategies, abuse content, and incitement to violence. \\
Weapons & Tutorials on crafting explosives, building weapons, and engaging in illegal arms trade. \\
\bottomrule
\end{tabularx}
\end{table}
\section{Experimental Setup}
\label{appendix:experimental_setup}
\textbf{Experimental Setup.}
Experiments on open-source MLLMs were conducted on a server with dual NVIDIA RTX A6000 GPUs, running Ubuntu 20.04.5 LTS. We used Python 3.10.12, CUDA 12.2, and Transformers v4.50.2 for model deployment and inference. Closed-source models were evaluated via official APIs from OpenAI, Google, and Anthropic through POST requests. During PolyJailbreak’s optimization phase, GPT-4o serves as the \textit{search agent}, GPT-3.5-turbo~\cite{GPT3.5Turbo} as the \textit{attack agent} (\(\mathcal{M}_A\)), and DeepSeek-V3-0324~\cite{deepseekai2024deepseekv3technicalreport} as the \textit{judging agent} (\(\mathcal{M}_J\)). The image generator (\(\mathcal{M}_D\)) is realized with FLUX.1-dev~\cite{labs2025flux1kontextflowmatching}. For reward computation, we employ MiniLM-L6-v2~\cite{MiniLM} as the text encoder and CLIP-ViT-B/32~\cite{CLIP} as the image encoder to extract semantic and stylistic embeddings. For each malicious goal, the optimization budget is capped at $T_{\max} = 15$ steps.
\section{Figures, Tables, and Algorithm}
\begin{figure}[tbp]
\centering
\includegraphics[width=0.9\columnwidth]{RQ1_ASR.pdf}
\caption{The ASR (\%) comparison of LLaVA and Vicuna under no, benign and malicious history settings.}
\label{fig:RQ1_ASR}
\end{figure}
\begin{algorithm}[t]
\footnotesize
\caption{PolyJailbreak}
\label{PolyJailbreak}
\textbf{Input:} Malicious goal $g$, target MLLM $\mathcal{M}$, attack model $\mathcal{M}_A$, judging model $\mathcal{M}_J$, text-to-image generator $\mathcal{M}_D$, strategy seed library $\mathcal{S}$, actor network $\pi_\theta$, twin critic networks $Q_{\phi_1}, Q_{\phi_2}$, step limit $T_{\max}$, and hyperparameters $\alpha_i$, $\beta$, $\gamma_i$. \\
\textbf{Output:} Adversarial input $(x_t, v_t)$ and the corresponding response $y_t$ if successful.
\begin{algorithmic}[1]
\STATE // Execute direct inquiry and online search separately.
\STATE $\mathcal{P}_{\mathcal{M}} \gets \texttt{ModelDiscovery}(\mathcal{M})$
\STATE // Initialize attack and obtain reference answer $r^\star$.
\STATE $(s_{\text{text}}, s_{\text{image}}, s_{\text{pers}}, r^\star) \gets \texttt{AttackInit}(\mathcal{M}_A, g, \mathcal{P}_{\mathcal{M}})$
\STATE // Initialize text variation factor and replay buffer.
\STATE $\mu_{\text{text}} \gets 0$
\STATE $\mathcal{B} \gets \emptyset$
\STATE // Construct initial input $(x_0, v_0)$.
\STATE $x_0 \gets \texttt{TextConstruction}(\mathcal{M}_A, g, s_{\text{text}}, s_{\text{pers}}, \mu_{\text{text}})$
\STATE $v_0 \gets \texttt{ImageConstruction}(\mathcal{M}_A, \mathcal{M}_D, g, s_{\text{image}})$
\FOR{$t = 0$ \textbf{to} $T_{\max}-1$}
\STATE // Query target model.
\STATE $y_t \gets \texttt{Query}((x_t, v_t), \mathcal{M})$
\STATE // Judge if jailbreak succeeded.
\STATE $(\text{label}, \mathbf{h}_t) \gets \texttt{Judge}(g, x_t, y_t, \mathcal{M}_J)$
\IF{$\text{label} = \texttt{Success}$}
\STATE \textbf{return} $(x_t, v_t, y_t, \text{Success})$
\ENDIF
\STATE // Compute rewards.
\STATE $r_t \gets \texttt{GetReward}(\text{label}, \mathbf{h}_t, t, x_t, v_t, r^\star)$
\STATE // Store transition into replay buffer.
\STATE $a_t \gets (s_{\text{text}}, s_{\text{image}}, s_{\text{pers}}, \mu_{\text{text}})$
\STATE $\mathcal{B} \gets \mathcal{B} \cup \{(s_t, a_t, r_t, s_{t+1})\}$
\IF{$|\mathcal{B}| \ge$ \texttt{batch\_size}}
\STATE \texttt{UpdateCritic}$(Q_{\phi_1}, Q_{\phi_2}, \mathcal{B})$
\STATE \texttt{UpdateActor}$(\pi_\theta, Q_{\phi_1}, Q_{\phi_2})$
\ENDIF
\STATE // Sample next action from actor policy.
\STATE $(\mu_{\text{text}}, s_{\text{text}}, s_{\text{image}}, s_{\text{pers}}) \sim \pi_\theta(\cdot | s_{t+1})$
\STATE // Construct next input.
\STATE $x_{t+1} \gets \texttt{TextConstruction}(\mathcal{M}_A,\ g,\ x_t,$\\
\hspace*{12.0em}$s_{\text{text}},\ s_{\text{pers}},\ \mu_{\text{text}})$
\IF{\textsc{ShouldRegenerateImage}$(s_{\text{image}})$}
\STATE $v_{t+1} \gets \texttt{ImageConstruction}(\mathcal{M}_A, g, s_{\text{image}})$
\ELSE
\STATE $v_{t+1} \gets \texttt{ImageTools}(v_t, s_{\text{image}})$
\ENDIF
\ENDFOR
\STATE \textbf{return} $(x_t, v_t, \text{Fail})$
\end{algorithmic}
\end{algorithm}
%\begin{figure*}[htbp]
% \centering
% \begin{subfigure}{0.18\textwidth}
% \includegraphics[width=\linewidth]{RQ1/llava_attention_user_no.pdf}
% \end{subfigure}\hfill
% \begin{subfigure}{0.18\textwidth}
% \includegraphics[width=\linewidth]{RQ1/llava_attention_user_one.pdf}
% \end{subfigure}\hfill
% \begin{subfigure}{0.18\textwidth}
% \includegraphics[width=\linewidth]{RQ1/llava_attention_user_two.pdf}
% \end{subfigure}\hfill
% \begin{subfigure}{0.18\textwidth}
% \includegraphics[width=\linewidth]{RQ1/llava_attention_user_three.pdf}
% \end{subfigure}\hfill
% \begin{subfigure}{0.18\textwidth}
% \includegraphics[width=\linewidth]{RQ1/llava_attention_user_four.pdf}
% \end{subfigure}\hfill
%
% % \vspace{0.5em}
%
% \begin{subfigure}{0.18\textwidth}
% \includegraphics[width=\linewidth]{RQ1/vicuna_attention_user_no.pdf}
% \end{subfigure}\hfill
% \begin{subfigure}{0.18\textwidth}
% \includegraphics[width=\linewidth]{RQ1/vicuna_attention_user_one.pdf}
% \end{subfigure}\hfill
% \begin{subfigure}{0.18\textwidth}
% \includegraphics[width=\linewidth]{RQ1/vicuna_attention_user_two.pdf}
% \end{subfigure}\hfill
% \begin{subfigure}{0.18\textwidth}
% \includegraphics[width=\linewidth]{RQ1/vicuna_attention_user_three.pdf}
% \end{subfigure}\hfill
% \begin{subfigure}{0.18\textwidth}
% \includegraphics[width=\linewidth]{RQ1/vicuna_attention_user_four.pdf}
% \end{subfigure}\hfill
%
% \caption{Comparison of information attentions in LLaVA and Vicuna across dialogue histories (from none to four turns, left to right). The top row corresponds to LLaVA and the bottom row to Vicuna.}
% \label{RQ1figAttention}
%\end{figure*}
\begin{table*}[htbp]
\centering
\scriptsize
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1}
\caption{LLaVA: CSR Results under different image input conditions}
\resizebox{\textwidth}{!}{
\begin{tabular}{ll c c c c c c c c c c c}
\toprule
\multirow{2}{*}{Model Layer} & \multirow{2}{*}{Text Type}
& \multicolumn{11}{c}{CSR under Different Image Inputs Conditions} \\
\cmidrule{3-13}
& & Text-only & White & MalTypora & EmojiMalTypora & CatSameLabel & CatOppLabel & CrossCatSameLabel & CrossCatOppLabel & Noise+White & Noise+MalTypora & Noise+CatSameLabel \\
\midrule
\multirow{2}{*}{\textbf{-20}}
& Plain & 0.2731 & 0.2226 & 0.2153 & 0.2402 & 0.2227 & 0.2320 & 0.2253 & 0.2348 & 0.2277 & 0.2226 & 0.2234 \\
& Emoji & 0.1785 & 0.1421 & 0.1421 & 0.1420 & 0.1476 & 0.1521 & 0.1494 & 0.1539 & 0.1457 & 0.1478 & 0.1482 \\
\midrule
\multirow{2}{*}{\textbf{-15}}
& Plain & 0.6171 & 0.5385 & 0.4493 & 0.4860 & 0.5072 & 0.4994 & 0.5214 & 0.5155 & 0.5566 & 0.4553 & 0.5050 \\
& Emoji & 0.4544 & 0.3714 & 0.3037 & 0.2969 & 0.3378 & 0.3127 & 0.3381 & 0.3221 & 0.3814 & 0.3084 & 0.3365 \\
\midrule
\multirow{2}{*}{\textbf{-10}}
& Plain & 0.5182 & 0.4041 & 0.3440 & 0.3736 & 0.3937 & 0.3698 & 0.3945 & 0.3872 & 0.4162 & 0.3473 & 0.3912 \\
& Emoji & 0.3868 & 0.2835 & 0.2395 & 0.2339 & 0.2693 & 0.2342 & 0.2681 & 0.2456 & 0.2896 & 0.2416 & 0.2683 \\
\midrule
\multirow{2}{*}{\textbf{-5}}
& Plain & 0.4538 & 0.3435 & 0.2905 & 0.3265 & 0.3430 & 0.3133 & 0.3453 & 0.3288 & 0.3517 & 0.2943 & 0.3418 \\
& Emoji & 0.3355 & 0.2437 & 0.2040 & 0.2069 & 0.2386 & 0.1990 & 0.2381 & 0.2061 & 0.2477 & 0.2059 & 0.2381 \\
\midrule
\multirow{2}{*}{\textbf{-4}}
& Plain & 0.4248 & 0.3268 & 0.2768 & 0.3120 & 0.3277 & 0.2967 & 0.3306 & 0.3045 & 0.3347 & 0.2800 & 0.3272 \\
& Emoji & 0.3145 & 0.2333 & 0.1960 & 0.1995 & 0.2312 & 0.1897 & 0.2267 & 0.1963 & 0.2371 & 0.1978 & 0.2313 \\
\midrule
\multirow{2}{*}{\textbf{-3}}
& Plain & 0.3944 & 0.3008 & 0.2578 & 0.2919 & 0.3035 & 0.2752 & 0.3039 & 0.2863 & 0.3082 & 0.2602 & 0.3025 \\
& Emoji & 0.2931 & 0.2156 & 0.1843 & 0.1884 & 0.2159 & 0.1773 & 0.2129 & 0.1867 & 0.2191 & 0.1856 & 0.2155 \\
\midrule
\multirow{2}{*}{\textbf{-2}}
& Plain & 0.3821 & 0.2920 & 0.2493 & 0.2849 & 0.2936 & 0.2656 & 0.2986 & 0.2749 & 0.2987 & 0.2518 & 0.2928 \\
& Emoji & 0.2826 & 0.2084 & 0.1780 & 0.1836 & 0.2095 & 0.1707 & 0.2068 & 0.1789 & 0.2118 & 0.1796 & 0.2095 \\
\midrule
\multirow{2}{*}{\textbf{-1}}
& Plain & 0.3626 & 0.2709 & 0.2337 & 0.2625 & 0.2594 & 0.2354 & 0.2689 & 0.2454 & 0.2797 & 0.2341 & 0.2586 \\
& Emoji & 0.2778 & 0.2001 & 0.1804 & 0.1880 & 0.2072 & 0.1643 & 0.2051 & 0.1733 & 0.2045 & 0.1812 & 0.2075 \\
\bottomrule
\end{tabular}
}
\label{tab:llava_csr_analysis}
\end{table*}
\begin{table*}[htbp]
\centering
\scriptsize
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1}
\caption{LLaMA: CSR Results under different image input conditions}
\resizebox{\textwidth}{!}{
\begin{tabular}{ll c c c c c c c c c c c}
\toprule
\multirow{2}{*}{Model Layer} & \multirow{2}{*}{Text Type}
& \multicolumn{11}{c}{CSR under Different Image Inputs Conditions} \\
\cmidrule{3-13}
& & Text-only & White & MalTypora & EmojiMalTypora & CatSameLabel & CatOppLabel & CrossCatSameLabel & CrossCatOppLabel & Noise+White & Noise+MalTypora & Noise+CatSameLabel \\
\midrule
\multirow{2}{*}{\textbf{-20}}
& Plain
& 1.8140 & 1.0720 & 1.4256 & 1.4009 & 1.1110 & 0.8386 & 1.1424 & 0.8260 & 1.0049 & 1.3668 & 1.1267 \\
& Emoji
& 1.3435 & 0.8022 & 1.1147 & 1.0891 & 0.9619 & 0.6557 & 0.9602 & 0.6457 & 0.7536 & 1.0652 & 0.9774 \\
\midrule
\multirow{2}{*}{\textbf{-15}}
& Plain
& 2.2889 & 1.1508 & 1.0777 & 1.2936 & 1.2081 & 0.9607 & 1.1573 & 0.9380 & 1.0365 & 1.0409 & 1.1629 \\
& Emoji
& 1.7932 & 0.8938 & 0.8663 & 1.1033 & 1.0680 & 0.7833 & 1.0435 & 0.7990 & 0.8012 & 0.8433 & 1.0193 \\
\midrule
\multirow{2}{*}{\textbf{-10}}
& Plain
& 2.2575 & 1.0978 & 0.9764 & 1.2288 & 1.1434 & 0.9197 & 1.1177 & 0.9031 & 0.9947 & 0.9401 & 1.1430 \\
& Emoji
& 1.7686 & 0.8753 & 0.7862 & 1.0326 & 1.0251 & 0.7546 & 1.0370 & 0.7784 & 0.7843 & 0.7773 & 1.0143 \\
\midrule
\multirow{2}{*}{\textbf{-5}}
& Plain
& 1.8839 & 0.9321 & 0.8014 & 0.9751 & 0.9242 & 0.7524 & 0.8942 & 0.7383 & 0.8419 & 0.7774 & 0.9171 \\
& Emoji
& 1.4911 & 0.7578 & 0.6626 & 0.8378 & 0.8418 & 0.6357 & 0.8243 & 0.6421 & 0.6819 & 0.6526 & 0.8225 \\
\midrule
\multirow{2}{*}{\textbf{-4}}
& Plain
& 1.7444 & 0.8714 & 0.7415 & 0.8894 & 0.8464 & 0.6855 & 0.8462 & 0.6805 & 0.7779 & 0.7139 & 0.8405 \\
& Emoji
& 1.3865 & 0.7207 & 0.6153 & 0.7645 & 0.7677 & 0.5790 & 0.7360 & 0.5748 & 0.6336 & 0.6088 & 0.7518 \\
\midrule
\multirow{2}{*}{\textbf{-3}}
& Plain
& 1.6933 & 0.8228 & 0.7035 & 0.8506 & 0.7942 & 0.6453 & 0.7818 & 0.6369 & 0.7281 & 0.6800 & 0.7896 \\
& Emoji
& 1.3397 & 0.6844 & 0.5818 & 0.7258 & 0.7136 & 0.5432 & 0.7335 & 0.5506 & 0.5931 & 0.5761 & 0.7033 \\
\midrule
\multirow{2}{*}{\textbf{-2}}
& Plain
& 1.6921 & 0.8017 & 0.6905 & 0.8164 & 0.7859 & 0.6175 & 0.7727 & 0.6169 & 0.7152 & 0.6632 & 0.7641 \\
& Emoji
& 1.3353 & 0.6699 & 0.5846 & 0.7006 & 0.7026 & 0.5249 & 0.7103 & 0.5289 & 0.5839 & 0.5747 & 0.6753 \\
\midrule
\multirow{2}{*}{\textbf{-1}}
& Plain
& 1.5633 & 0.7281 & 0.6140 & 0.7559 & 0.7069 & 0.5518 & 0.7046 & 0.5461 & 0.6450 & 0.5836 & 0.6978 \\
& Emoji
& 1.2685 & 0.6066 & 0.5197 & 0.6443 & 0.6260 & 0.4632 & 0.6074 & 0.4743 & 0.5152 & 0.5065 & 0.6063 \\
\bottomrule
\end{tabular}
}
\label{tab:llama_csr_analysis}
\end{table*}
\begin{figure*}[htbp]
\centering
% 上排两张图（并排）
\begin{subfigure}{0.45\textwidth}
\centering
\includegraphics[width=\linewidth]{asr_text_strategy_heatmap.pdf}
\caption{Textual manipulation strategies vs Target models}
\label{fig:asr_text_model}
\end{subfigure}
\hfill
\begin{subfigure}{0.45\textwidth}
\centering
\includegraphics[width=\linewidth]{asr_image_strategy_heatmap.pdf}
\caption{Visual manipulation strategies vs Target models}
\label{fig:asr_image_model}
\end{subfigure}
% \vskip\baselineskip
% 下排单独一张图
\begin{subfigure}{0.85\textwidth}
\centering
\includegraphics[width=\linewidth]{asr_persuasion_strategy_heatmap.pdf}
\caption{Prompt amplification strategies vs Target models}
\label{fig:asr_persuasion_model}
\end{subfigure}
\caption{Heatmaps showing the jailbreak success rates for different strategy types across target models.}
\label{fig:asr_model_heatmaps}
\end{figure*}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\end{document}
\begin{thebibliography}{10}
\bibitem{Flamingo}
Jean{-}Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr,
Yana Hasson, Karel Lenc, Arthur Mensch, and et~al.
\newblock {{Flamingo}: A Visual Language Model for Few-Shot Learning}.
\newblock In {\em Advances in Neural Information Processing Systems 35
(NeurIPS)}, New Orleans, LA, USA, Nov. 28--Dec. 9, 2022.
\bibitem{gemini}
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean{-}Baptiste Alayrac, Jiahui Yu,
Radu Soricut, and et~al.
\newblock {Gemini: A Family of Highly Capable Multimodal Models}.
\newblock {\em arXiv preprint arXiv:2312.11805}, 2023.
\bibitem{AnthropicClaude}
{Anthropic}.
\newblock {Introducing Claude}.
\newblock {\em Anthropic Blog}, 2023.
\newblock Available at:
\url{https://www.anthropic.com/index/introducing-claude}.
\bibitem{claude}
{Anthropic}.
\newblock {Claude 3.7 Sonnet and Claude Code}, Feb. 2025.
\newblock Available at: \url{https://www.anthropic.com/news/claude-3-7-sonnet}.
\bibitem{DBLP:conf/acl/CaffagniCBMS0CC24}
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara
Sarto, and et~al.
\newblock The {R}evolution of {M}ultimodal {L}arge {L}anguage {M}odels: {A}
{S}urvey.
\newblock In {\em Findings of the Association for Computational Linguistics
(ACL)}, Bangkok, Thailand, Aug. 11--16, 2024, pp. 13590--13618.
\bibitem{PrivacyRisks}
Tiejin Chen, Pingzhi Li, Kaixiong Zhou, Tianlong Chen, and Hua Wei.
\newblock {Unveiling Privacy Risks in Multi-Modal Large Language Models:
Task-Specific Vulnerabilities and Mitigation Challenges}.
\newblock In {\em Findings of the Association for Computational Linguistics
({ACL})}, Vienna, Austria, Jul. 27--Aug. 1, 2025, pp. 4573--4586.
\bibitem{gemini2.5}
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen
Sachdeva, and et~al.
\newblock {Gemini 2.5: Pushing the Frontier with Advanced Reasoning,
Multimodality, Long Context, and Next Generation Agentic Capabilities}.
\newblock {\em arXiv preprint arXiv:2507.06261}, 2025.
\bibitem{deepseekai2024deepseekv3technicalreport}
{DeepSeek-AI}.
\newblock {DeepSeek-V3 Technical Report}.
\newblock {\em arXiv preprint arXiv:2412.19437}, 2024.
\newblock \url{https://arxiv.org/abs/2412.19437}.
\bibitem{llama}
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad
Al{-}Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan,
and et~al.
\newblock {The {LLaMA} 3 Herd of Models}.
\newblock {\em arXiv preprint arXiv:2407.21783}, 2024.
\bibitem{ImageInputs}
Yihe Fan, Yuxin Cao, Ziyu Zhao, Ziyao Liu, and Shaofeng Li.
\newblock {Unbridled Icarus: A Survey of the Potential Perils of Image Inputs
in Multimodal Large Language Model Security}.
\newblock In {\em Proceedings of the {IEEE} International Conference on
Systems, Man, and Cybernetics ({SMC})}, Kuching, Malaysia, Oct. 6--10, 2024,
pp. 3428--3433.
\bibitem{EMMA}
Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Siddharth Garg, and
Farshad Khorrami.
\newblock {{EMMA}: Efficient Visual Alignment in Multi-Modal {LLMs}}.
\newblock {\em Transactions on Machine Learning Research}, 2025, 2025.
\bibitem{FigStep}
Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, and et~al.
\newblock {FigStep: Jailbreaking Large Vision-Language Models via Typographic
Visual Prompts}.
\newblock In {\em Proceedings of the Thirty-Ninth AAAI Conference on Artificial
Intelligence (AAAI)}, Philadelphia, PA, USA, Feb. 25--Mar. 4, 2025, pp.
23951--23959.
\bibitem{DBLP:conf/ijcai/GuoCWCPCW024}
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh~V.
Chawla, and et~al.
\newblock {Large Language Model Based Multi-Agents: A Survey of Progress and
Challenges}.
\newblock In {\em Proceedings of the Thirty-Third International Joint
Conference on Artificial Intelligence (IJCAI)}, Jeju, South Korea, Aug. 3--9,
2024, pp. 8048--8057.
\bibitem{SAC}
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
\newblock {Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement
Learning with a Stochastic Actor}.
\newblock In {\em Proceedings of the 35th International Conference on Machine
Learning ({ICML})}, Stockholm, Sweden, Jul. 10--15, 2018, pp. 1856--1865.
\bibitem{tuning}
Danny Halawi, Alexander Wei, Eric Wallace, Tony~Tong Wang, Nika Haghtalab, and
Jacob Steinhardt.
\newblock {Covert Malicious Finetuning: Challenges in Safeguarding {LLM}
Adaptation}.
\newblock In {\em Proceedings of the 41st International Conference on Machine
Learning ({ICML})}, Vienna, Austria, Jul. 21--27, 2024.
\bibitem{Medical}
Xijie Huang, Xinyuan Wang, Hantao Zhang, Yinghao Zhu, Jiawen Xi, Jingkun An,
Hao Wang, Hao Liang, and Chengwei Pan.
\newblock {Medical {MLLM} Is Vulnerable: Cross-Modality Jailbreak and
Mismatched Attacks on Medical Multimodal Large Language Models}.
\newblock In {\em Proceedings of the Thirty-Ninth {AAAI} Conference on
Artificial Intelligence ({AAAI})}, Philadelphia, USA, Feb. 25--Mar. 4, 2025,
pp. 3797--3805.
\bibitem{DBLP:journals/corr/abs-2410-21276}
Aaron Hurst, Adam Lerer, Adam~P. Goucher, Adam Perelman, Aditya Ramesh, Aidan
Clark, and et~al.
\newblock {GPT-4o System Card}.
\newblock {\em arXiv preprint arXiv:2410.21276}, 2024.
\bibitem{JOOD}
Joonhyun Jeong, Seyun Bae, Yeonsung Jung, Jaeryong Hwang, and Eunho Yang.
\newblock {Playing the Fool: Jailbreaking {LLMs} and Multimodal {LLMs} with
Out-of-Distribution Strategy}.
\newblock In {\em Proceedings of the {IEEE/CVF} Conference on Computer Vision
and Pattern Recognition ({CVPR})}, Nashville, USA, Jun. 11--15, 2025, pp.
29937--29946.
\bibitem{ArtPrompt}
Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian,
and et~al.
\newblock {ArtPrompt: ASCII Art-based Jailbreak Attacks Against Aligned LLMs}.
\newblock In {\em Proceedings of the 62nd Annual Meeting of the Association for
Computational Linguistics (ACL)}, Bangkok, Thailand, Aug. 11--16, 2024, pp.
15157--15173.
\bibitem{labs2025flux1kontextflowmatching}
Black~Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham
Consul, and et~al.
\newblock {FLUX.1 Kontext: Flow Matching for In-Context Image Generation and
Editing in Latent Space}.
\newblock {\em arXiv preprint arXiv:2506.15742}, 2025.
\newblock \url{https://arxiv.org/abs/2506.15742}.
\bibitem{black3}
Hongyi Li, Jiawei Ye, Jie Wu, Tianjie Yan, Chu Wang, and Zhixin Li.
\newblock {JailPO: A Novel Black-Box Jailbreak Framework via Preference
Optimization Against Aligned {LLMs}}.
\newblock In {\em Proceedings of the Thirty-Ninth {AAAI} Conference on
Artificial Intelligence ({AAAI})}, Philadelphia, USA, Feb. 25--Mar. 4, 2025,
pp. 27419--27427.
\bibitem{hades}
Yifan Li, Hangyu Guo, Kun Zhou, Wayne~Xin Zhao, and Ji{-}Rong Wen.
\newblock {Images Are Achilles' Heel of Alignment: Exploiting Visual
Vulnerabilities for Jailbreaking Multimodal Large Language Models}.
\newblock In {\em Proceedings of the 18th European Conference on Computer
Vision ({ECCV})}, Milan, Italy, Sep. 29--Oct. 4, 2024, pp. 174--189.
\bibitem{llava-1.6}
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong~Jae Lee.
\newblock {Improved Baselines with Visual Instruction Tuning}.
\newblock In {\em Proceedings of the {IEEE/CVF} Conference on Computer Vision
and Pattern Recognition ({CVPR})}, Seattle, USA, Jun. 16--22, 2024, pp.
26286--26296.
\bibitem{llava-1.5}
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong~Jae Lee.
\newblock {Visual Instruction Tuning}.
\newblock In {\em Advances in Neural Information Processing Systems 36
({NeurIPS})}, New Orleans, USA, Dec. 10--16, 2023.
\bibitem{DRADRA}
Tong Liu, Yingjie Zhang, Zhe Zhao, Yinpeng Dong, Guozhu Meng, and Kai Chen.
\newblock {Making Them Ask and Answer: Jailbreaking Large Language Models in
Few Queries via Disguise and Reconstruction}.
\newblock In {\em Proceedings of the 33rd {USENIX} Security Symposium ({USENIX}
Security)}, Philadelphia, USA, Aug. 14--16, 2024.
\bibitem{MM-SafetyBench}
Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and et~al.
\newblock {MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal
Large Language Models}.
\newblock In {\em Proceedings of the 18th European Conference on Computer
Vision (ECCV)}. Springer, Milan, Italy, Sep. 29--Oct. 4, 2024, pp. 386--403.
\bibitem{FlipAttack}
Yue Liu, Xiaoxin He, Miao Xiong, Jinlan Fu, Shumin Deng, and Bryan Hooi.
\newblock {FlipAttack: Jailbreak LLMs via Flipping}.
\newblock {\em CoRR}, abs/2410.02832, 2024.
\bibitem{FromLLMstoMLLMs}
Yanxu Mao, Tiehan Cui, Peipei Liu, Datao You, and Hongsong Zhu.
\newblock {From {LLMs} to {MLLMs} to Agents: A Survey of Emerging Paradigms in
Jailbreak Attacks and Defenses within {LLM} Ecosystem}.
\newblock {\em arXiv preprint arXiv:2506.15170}, 2025.
\bibitem{OpenAIcontent}
Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine~Eloundou Nekoul,
Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng.
\newblock {A Holistic Approach to Undesired Content Detection in the Real
World}.
\newblock In {\em Proceedings of the Thirty-Seventh AAAI Conference on
Artificial Intelligence (AAAI)}, Washington, DC, USA, Feb. 7--14, 2023, pp.
15009--15018.
\bibitem{umap}
Leland McInnes and John Healy.
\newblock {UMAP: Uniform Manifold Approximation and Projection for Dimension
Reduction}.
\newblock {\em CoRR}, abs/1802.03426, arXiv preprint arXiv:1802.03426, 2018.
\bibitem{X-Fusion}
Sicheng Mo, Thao Nguyen, Xun Huang, Siddharth~Srinivasan Iyer, Yijun Li, Yuchen
Liu, Abhishek Tandon, Eli Shechtman, Krishna~Kumar Singh, Yong~Jae Lee, Bolei
Zhou, and Yuheng Li.
\newblock {{X-Fusion}: Introducing New Modality to Frozen Large Language
Models}.
\newblock {\em arXiv preprint arXiv:2504.20996}, 2025.
\bibitem{GPT3.5Turbo}
{OpenAI}.
\newblock {GPT‑3.5‑Turbo}.
\newblock 2023.
\newblock Available at:
\url{https://platform.openai.com/docs/models/gpt-3.5-turbo}.
\bibitem{GPT-5}
{OpenAI}.
\newblock {Introducing GPT-5}.
\newblock 2025.
\newblock Available at: \url{https://openai.com/index/introducing-gpt-5/}.
\bibitem{RLHF}
Long Ouyang, Jeffrey Wu, Xu~Jiang, Diogo Almeida, Carroll~L. Wainwright, Pamela
Mishkin, and et~al.
\newblock {Training Language Models to Follow Instructions with Human
Feedback}.
\newblock In {\em Advances in Neural Information Processing Systems 35
({NeurIPS})}, New Orleans, USA, Nov. 28--Dec. 9, 2022.
\bibitem{white5}
Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and
Prateek Mittal.
\newblock {Visual Adversarial Examples Jailbreak Aligned Large Language
Models}.
\newblock In {\em Proceedings of the Thirty-Eighth {AAAI} Conference on
Artificial Intelligence ({AAAI})}, Vancouver, Canada, Feb. 20--27, 2024, pp.
21527--21536.
\bibitem{MiniLM}
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne~Xin Zhao, Daxiang
Dong, Hua Wu, and Haifeng Wang.
\newblock {RocketQA: An Optimized Training Approach to Dense Passage Retrieval
for Open-Domain Question Answering}.
\newblock {\em arXiv preprint arXiv:2010.08191}, 2020.
\bibitem{qwen2.5-VL}
{Qwen Team}.
\newblock {Qwen2.5-VL}, Jan. 2025.
\newblock Available at: \url{https://qwenlm.github.io/blog/qwen2.5-vl/}.
\bibitem{CLIP}
Alec Radford, Jong~Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, and
et~al.
\newblock {Learning Transferable Visual Models From Natural Language
Supervision}.
\newblock In {\em Proceedings of the 38th International Conference on Machine
Learning (ICML)}, Jul. 18--24, 2021, pp. 8748--8763.
\bibitem{MLLMsecurityconcern}
Md.~Abdur Rahman, Lamyaa Alqahtani, Amna Albooq, and Alaa Ainousah.
\newblock {A Survey on Security and Privacy of Large Multimodal Deep Learning
Models: Teaching and Learning Perspective}.
\newblock In {\em Proceedings of the 21st Learning and Technology Conference
({L{\&}T})}, 2024, pp. 13--18.
\bibitem{DBLP:conf/ccs/ShenC0SZ24}
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang.
\newblock {``Do Anything Now'': Characterizing and Evaluating In-the-Wild
Jailbreak Prompts on Large Language Models}.
\newblock In {\em Proceedings of the {ACM} {SIGSAC} Conference on Computer and
Communications Security ({CCS})}, Salt Lake City, USA, Oct. 14--18, 2024, pp.
1671--1685.
\bibitem{VisualRepresentationsMap}
Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, and Neel Nanda.
\newblock {How Visual Representations Map to Language Feature Space in
Multimodal {LLMs}}.
\newblock {\em arXiv preprint arXiv:2506.11976}, 2025.
\bibitem{LandscapeofMLLM}
Siyuan Wang, Zhuohan Long, Zhihao Fan, and Zhongyu Wei.
\newblock {From {LLMs} to {MLLMs}: Exploring the Landscape of Multimodal
Jailbreaking}.
\newblock In {\em Proceedings of the 2024 Conference on Empirical Methods in
Natural Language Processing ({EMNLP})}, Miami, USA, Nov. 12--16, 2024, pp.
17568--17582.
\bibitem{AlignIsNotEnough}
Youze Wang, Wenbo Hu, Yinpeng Dong, Jing Liu, Hanwang Zhang, and Richang Hong.
\newblock {Align Is Not Enough: Multimodal Universal Jailbreak Attack Against
Multimodal Large Language Models}.
\newblock {\em {IEEE} Transactions on Circuits and Systems for Video
Technology}, 35(6):5475--5488, 2025.
\bibitem{MML-M}
Yu~Wang, Xiaofei Zhou, Yichen Wang, Geyuan Zhang, and Tianxing He.
\newblock {Jailbreak Large Vision-Language Models Through Multi-Modal Linkage}.
\newblock In {\em Proceedings of the 63rd Annual Meeting of the Association for
Computational Linguistics (ACL)}, Vienna, Austria, Jul. 27--Aug. 1, 2025, pp.
1466--1494.
\bibitem{RedAgent}
Huiyu Xu, Wenhui Zhang, Zhibo Wang, Feng Xiao, Rui Zheng, Yunhe Feng, Zhongjie
Ba, and Kui Ren.
\newblock {{RedAgent}: Red Teaming Large Language Models with Context-Aware
Autonomous Language Agent}.
\newblock {\em arXiv preprint arXiv:2407.16667}, 2024.
\bibitem{JailbreakDefense}
Zihao Xu, Yi~Liu, Gelei Deng, Yuekang Li, and Stjepan Picek.
\newblock {A Comprehensive Study of Jailbreak Attack versus Defense for Large
Language Models}.
\newblock In {\em Findings of the Association for Computational Linguistics
({ACL})}, Bangkok, Thailand, Aug. 11--16, 2024, pp. 7432--7449.
\bibitem{black2}
Yanni Xue, Jiakai Wang, Zixin Yin, Yuqing Ma, Haotong Qin, Renshuai Tao, and
Xianglong Liu.
\newblock {Dual Intention Escape: Penetrating and Toxic Jailbreak Attack
Against Large Language Models}.
\newblock In {\em Proceedings of the {ACM} Web Conference ({WWW})}, Sydney,
Australia, Apr. 28--May 2, 2025, pp. 863--871.
\bibitem{DarkCite}
Xikang Yang, Xuehai Tang, Jizhong Han, and Songlin Hu.
\newblock {The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks
on Large Language Models}.
\newblock {\em CoRR}, abs/2411.11407, 2024.
\bibitem{SneakyPrompt}
Yuchen Yang, Bo~Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao.
\newblock {SneakyPrompt}: Jailbreaking text-to-image generative models.
\newblock In {\em IEEE Symposium on Security and Privacy (S\&P)}, San
Francisco, CA, USA, May 19--23, 2024, pp. 897--912.
\bibitem{black1}
Zuopeng Yang, Jiluan Fan, Anli Yan, Erdun Gao, Xin Lin, Tao Li, Kanghua Mo, and
Changyu Dong.
\newblock {Distraction is All You Need for Multimodal Large Language Model
Jailbreaking}.
\newblock In {\em Proceedings of the {IEEE/CVF} Conference on Computer Vision
and Pattern Recognition ({CVPR})}, Nashville, USA, Jun. 11--15, 2025, pp.
9467--9476.
\bibitem{white3}
Zonghao Ying, Aishan Liu, Tianyuan Zhang, Zhengmin Yu, Siyuan Liang, Xianglong
Liu, and Dacheng Tao.
\newblock {J}ailbreak {V}ision {L}anguage {M}odels via {B}i-modal {A}dversarial
{P}rompt.
\newblock {\em {IEEE} Trans. Inf. Forensics Secur.}, 20:7153--7165, Jun. 2025.
\bibitem{black6}
Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing.
\newblock {{GPTFUZZER}: Red Teaming Large Language Models with Auto-Generated
Jailbreak Prompts}.
\newblock {\em arXiv preprint arXiv:2309.10253}, 2023.
\bibitem{eosToken}
Jiahao Yu, Haozheng Luo, Jerry~Yao{-}Chieh Hu, Wenbo Guo, Han Liu, and Xinyu
Xing.
\newblock {Enhancing Jailbreak Attack Against Large Language Models through
Silent Tokens}.
\newblock {\em arXiv preprint arXiv:2405.20653}, 2024.
\bibitem{Johnny}
Yi~Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi.
\newblock {How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking
Persuasion to Challenge {AI} Safety by Humanizing LLMs}.
\newblock In {\em Proceedings of the 62nd Annual Meeting of the Association for
Computational Linguistics (ACL)}, Bangkok, Thailand, Aug. 11--16, 2024,
pp.14322--14350.
\bibitem{HyperLLaVA}
Wenqiao Zhang, Tianwei Lin, Jiang Liu, Fangxun Shu, Haoyuan Li, Lei Zhang,
Wanggui He, Hao Zhou, Zheqi Lv, Hao Jiang, Juncheng Li, Siliang Tang, and
Yueting Zhuang.
\newblock {HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal
Large Language Models}.
\newblock {\em arXiv preprint arXiv:2403.13447}, 2024.
\bibitem{informationflow}
Xiaofeng Zhang, Fanshuo Zeng, and Chaochen Gu.
\newblock {Simignore: Exploring and Enhancing Multimodal Large Model Complex
Reasoning via Similarity Computation}.
\newblock {\em Neural Networks}, 184:107059, Apr. 2025.
\bibitem{HS}
Shiji Zhao, Ranjie Duan, Fengxiang Wang, Chi Chen, Caixin Kang, and et~al.
\newblock {Jailbreaking Multimodal Large Language Models via Shuffle
Inconsistency}.
\newblock {\em CoRR}, abs/2501.04931, 2025.
\bibitem{Query-Attack}
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, and et~al.
\newblock {On Evaluating Adversarial Robustness of Large Vision-Language
Models}.
\newblock In {\em Proceedings of the 36th Annual Conference on Neural
Information Processing Systems (NeurIPS)}, New Orleans, LA, USA, Dec. 10--16,
2023.
\bibitem{vicuna}
Lianmin Zheng, Wei{-}Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, and
et~al.
\newblock {Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena}.
\newblock In {\em Advances in Neural Information Processing Systems 36
(NeurIPS)}, New Orleans, LA, USA, Dec. 10--16, 2023.
\bibitem{Advbench}
Andy Zou, Zifan Wang, J.~Zico Kolter, and Matt Fredrikson.
\newblock {Universal and Transferable Adversarial Attacks on Aligned Language
Models}.
\newblock {\em CoRR}, abs/2307.15043, arXiv preprint arXiv:2307.15043, 2023.
\end{thebibliography}