HRGR: Enhancing Image Manipulation Detection via Hierarchical Region-aware Graph Reasoning
Extracted source text · arXiv 2410.21861
\documentclass[conference]{IEEEtran}
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\usepackage{booktabs}
\usepackage{algorithm}
\usepackage{multirow}
\usepackage{bm}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
\newtheorem{example}{\textbf{Example}}
\newtheorem{theorem}{\textbf{Theorem}}
\newtheorem{defn}{\textbf{Definition}}
\newtheorem{remark}{\textbf{Remark}}
%%%%%%%%%%%%
\usepackage{float}
\usepackage{colortbl}
\def\red#1{\textcolor{red}{#1}}
\def\blue#1{\textcolor{blue}{#1}}
\def\green#1{\textcolor{green}{#1}}
\def\yellow#1{\textcolor{yellow}{#1}}
\def\orange#1{\textcolor{orange}{#1}}
\def\violet#1{\textcolor{violet}{#1}}
\def\blue#1{\textcolor{blue}{#1}}
\def\purple#1{\textcolor{purple}{#1}}
\def\magenta#1{\textcolor{magenta}{#1}}
\def\cyan#1{\textcolor{cyan}{#1}}
\definecolor{DeepBlue}{RGB}{0, 50, 150}
\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}
% \newcommand{\mc}[1]{\mathcal{#1}}
% \newcommand{\mf}[1]{\mathbf{#1}}
\def\eg{\emph{e.g.}}
\def\etal{\emph{et al. }}
\def\ie{\emph{i.e.}}
\definecolor{mygray}{gray}{.9}
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}
% Support for easy cross-referencing
\usepackage[capitalize]{cleveref}
\crefname{section}{Sec.}{Secs.}
\Crefname{section}{Section}{Sections}
\Crefname{table}{Table}{Tables}
\crefname{table}{Tab.}{Tabs.}
\begin{document}
\title{HRGR: Enhancing Image Manipulation Detection via Hierarchical Region-aware Graph Reasoning}
% \author{Anonymous ICME submission}
\author{\thanks{This work is supported in part by the National Natural Science Foundation of China (No.62402464), Shandong Provincial Natural Science Foundation (No.ZR2024QF035), and China Postdoctoral Science Foundation (No.2021TQ0314; No.2021M703036).}
Xudong Wang$^1$, Jiaran Zhou$^1$, Huiyu Zhou$^2$, Junyu Dong$^1$, Yuezun Li$^{1,\dagger}$\thanks{$\dagger$ Corresponding author} \\
$^1$ School of Computer Science and Technology, Ocean University of China \\
$^2$ School of Computing and Mathematical Sciences, University of Leicester}
\maketitle
\begin{abstract}
Image manipulation detection is to identify the authenticity of each pixel in images. One typical approach to uncover manipulation traces is to model image correlations.
The previous methods commonly adopt the grids, which are fixed-size squares, as graph nodes to model correlations. However, these grids, being independent of image content, struggle to retain local content coherence, resulting in imprecise detection.
To address this issue, we describe a new method named Hierarchical Region-aware Graph Reasoning (HRGR) to enhance image manipulation detection. Unlike existing grid-based methods, we model image correlations based on content-coherence feature regions with irregular shapes, generated by a novel Differentiable Feature Partition strategy. Then we construct a Hierarchical Region-aware Graph based on these regions within and across different feature layers. Subsequently, we describe a structural-agnostic graph reasoning strategy tailored for our graph to enhance the representation of nodes.
Our method is fully differentiable and can seamlessly integrate into mainstream networks in an end-to-end manner, without requiring additional supervision.
Extensive experiments demonstrate the effectiveness of our method in image manipulation detection, exhibiting its great potential as a plug-and-play component for existing architectures.
Codes and models are available at \url{https://github.com/OUC-VAS/HRGR-IMD}.
\end{abstract}
\begin{IEEEkeywords}
image manipualtion detection, feature clustering, graph reasoning
% component, formatting, style, styling, insert
\end{IEEEkeywords}
\section{Introduction}
Image manipulation involves various techniques such as copy-move, splicing, and object removal that can subtly alter the primary content of images. As image editing tools and AI technology continue to advance, image manipulation has become effortless, leading to significant societal issues such as misinformation spread, ethical dilemmas, and potential legal problems \cite{recent}.
\begin{figure}[!t]
\centering
\vspace{0.3cm}
\includegraphics[width=1\linewidth]{figures/overview1.pdf}
\vspace{-0.6cm}
\caption{\small The difference between (a) previous methods and (b) ours. Instead of using regular grids in previous methods, we model image correlations across different scales (\orange{orange} and {\textcolor{DeepBlue}{blue}} arrows) using connected local content-coherent feature regions. }
\label{fig:concept}
\vspace{-0.5cm}
\end{figure}
To combat this, countermeasures have been proposed and gained increasing attention in recent years. Deep learning-based methods have emerged as the dominant method for detecting manipulations \eg, \cite{ERMPC,PSCC,ObjectFormer,CAT-Net,ACBG,TruFor},. Leveraging the impressive learning capabilities of deep learning models. Based on deep neural networks, feature enhancement strategies have been explored to further emphasize manipulation traces.
Particularly, exploiting image correlations is a recent feature enhancement solution that has proven effective in enhancing detection ability \cite{ERMPC,PSCC}. The intuition is that manipulated regions can be better perceived when compared to pristine regions. To capture the correlations, these methods usually employ the mechanism of either attention mechanism \cite{PSCC,ObjectFormer} or graphs \cite{HGCN,ERMPC}. For example, PSCC-Net \cite{PSCC} proposed a spatial-channel correlation module to capture the correlations using attention. {ObjectFormer \cite{ObjectFormer} learned the correlation between intermediate features in objects and patches through cross-attention. HGCN \cite{HGCN} and ERMPC \cite{ERMPC} sought the relationship between pristine and manipulated regions by building a graph structure. }
\textbf{Despite their promising results, these methods share a common limitation: They all rely on the grid-based modeling of correlations.} It is important to note that these grids are regular squares with fixed sizes that are independent of the content. Typically, these methods partition the features into grids of size $n \times n$ and encapsulate the features within each grid as respective units for correlation modeling (see Fig.~\ref{fig:concept} (top)). However, as manipulations usually happen on the semantic-integrity objects (\eg, copy-move/splicing a whole person), the use of grids can hardly retain the local content coherence. This limitation can potentially hinder their representation and subsequently restrict the extraction of manipulation traces (\eg, leading to ambiguous detection boundaries (see Fig.~\ref{fig:concept} (top)).)
% \begin{center}
% \fcolorbox{black}{gray!5}{\parbox{0.95\linewidth}{
% \textit{\underline{Question}: ``Could we model the image correlations based on the units that are dependent on the image content rather than the regular grids?''}}
% }
% \end{center}
% \textit{``Could we model the correlations based on content-coherence entity instead of the regular grids?''}
% In this paper, we introduce two concepts, {\em Coherency in Relevancy} and {\em Relevancy for Coherency}, to address the above limitations. {\em Coherency} refers to the coherence of the content in graph nodes, while {\em Relevancy} refers to the graph modeling of these nodes. {\em Coherency in Relevancy} represents considering the coherency in the graph to improve the effectiveness of correlation modeling. Since the initial state may not perfectly maintain coherency, we introduce {\em Relevancy for Coherency} which aims to refine the coherency through the modeling of correlations. These two concepts are anticipated to be accomplished in a rolling mode, alternatively enhancing the coherency and relevancy (see Fig.~\ref{fig:concept}).
To overcome the limitation, we introduce a novel framework called {\em Hierarchical Region-aware Graph Reasoning (HRGR)} to enhance image manipulation detection. Instead of using regular grids to model correlations, we propose a Hierarchical Region-aware Graph that captures the correlations based on content-coherent \textbf{feature regions} (see Fig.~\ref{fig:concept} (bottom)).
% \begin{center}
% \fcolorbox{black}{gray!5}{\parbox{0.95\linewidth}{
% \textit{\underline{Definition}: The connected local content-coherent region on feature maps is called \textbf{Feature Region}. These regions are usually irregular and dependent on the image content.} \\
% \textit{\underline{Remark}: In contrast, the grids are regular and fixed squares that are independent of image content.}}
% }
% \end{center}
\begin{figure*}[!t]
\centering
\includegraphics[width=1\linewidth]{figures/method1.pdf}
\vspace{-0.9cm}
\caption{\small Overview of our method. We first extract the feature regions on feature layers from the encoder using a Differentiable Feature Partition (DFP) strategy. Then we build the hierarchical region-aware graph over all features and perform graph reasoning to update the features. The process of Differentiable Feature Partition and Hierarchical Graph Reasoning and Reasoning are shown in the right part.}
\label{fig:overview}
\vspace{-0.3cm}
\end{figure*}
To achieve this, we first describe a Differentiable Feature Partition strategy that divides the features into non-overlapped coherent regions while maintaining differentiability for end-to-end utilization. These feature regions are subsequently encapsulated as graph nodes using a Weighted Feature Aggregation strategy. In contrast to previous methods that typically construct graphs on single feature layers, we propose a Hierarchical Graph Modeling strategy to construct a graph within the single feature layers as well as across different feature layers. This is motivated by that the same manipulated instance at different scales should exhibit consistent manipulation traces. By modeling across different feature layers, we can identify this relation, thus further improving the identification of manipulation traces. After modeling the correlations, we describe a Structural-agnostic Graph Reasoning strategy to refine the representation of graph nodes, which are then mapped back to feature layers for feature enhancement. The overview of our method is illustrated in Fig.~\ref{fig:overview} (left).
It is noteworthy that our method is designed as a plug-and-play framework that can seamlessly integrate into mainstream networks without requiring additional supervision.
Experiments are conducted on commonly used datasets, and the results demonstrate the efficacy of the proposed method.
The contribution of this paper can be summarized as follows: \textbf{1)} To the best of our knowledge, we are the first to explore image correlations based on content-coherent irregular feature regions rather than regular grids, facilitated by a novel Differentiable Feature Partition strategy. \textbf{2)} Leveraging the feature regions, we describe a new Hierarchical Region-aware Graph Reasoning strategy that seeks correlations within the same feature layer and across different feature layers. \textbf{3)} Our framework is fully differentiable and can seamlessly integrate into mainstream networks in an end-to-end manner without requiring additional supervision.
% % \vspace{-0.3cm}
% \section{Related Works}
% \label{sec:related}
% % \smallskip
% % \noindent{\bf Image Manipulation Detection (IMD).}
% The recent forensics methods mainly rely on deep neural networks to expose the manipulations \cite{ManTraNet,SPAN,TransForensics,MVSS,PSCC,ObjectFormer,CAT-Net,PCL,SAFL-Net,NCL,HiFi,WSCL}. ManTra-Net~\cite{ManTraNet}, SPAN~\cite{SPAN}, TransForensics~\cite{TransForensics}, MVSS-Net~\cite{MVSS}, PSCC-Net~\cite{PSCC} and HiFi-Het~\cite{HiFi} are representative methods that are designed on specific DNNs. More recently, Vision Transformer (ViT)~\cite{ViT} are also adapted in our task, such as ObjectFormer~\cite{ObjectFormer} and EVP~\cite{EVP}.
% Within these methods, strategies {using graph spirits} for capturing the image correlations have been studied to further enhance feature effectiveness.
% \blue{TruFor\cite{TruFor} extracts high-level and low-level traces by combining RGB images with a transformer-based fusion architecture for learning noise-sensitive fingerprints. ACBG\cite{ACBG} ACBG proposes a new boundary-guided image manipulation detection method, which introduces the inherent bias of utilizing tampered region boundary information.}
% HGCN-Net~\cite{HGCN} introduced graph structures for manipulation detection tasks, which modeled region adjacency relationships on feature grids by considering four connectivity. PSCC-Net~\cite{PSCC} introduced a module for capturing spatial correlations using attention. This module can be conceptualized as a fully connected graph, where each element corresponds to a grid and is connected to every other element. ERMPC~\cite{ERMPC} extracted the inconsistency between real and manipulated regions by controlling message passing.
% It is worth highlighting that the aforementioned methods share a commonality in constructing graphs based on grids. However, due to the content independence and fixed sizes of these grids, they lack an understanding of the content coherence and often struggle to capture irregularly manipulated regions. As such, we describe the Hierarchical Region-aware Graph Reasoning method to address these limitations.
% For example,
% ManTra-Net~\cite{ManTraNet} designed a specific network and trained it using plenty of data augmentations, achieving good performance on public datasets. SPAN~\cite{SPAN} formulated the spatial correlations by constructing a pyramid of local self-attention blocks.
% {TransForensics~\cite{TransForensics} introduces dense self-attention encoders and dense correction modules, which enable it to have high generalization capabilities across different types of manipulation.}
% MVSS-Net~\cite{MVSS} utilized multi-view and multi-scale forgery cues to locate tampered regions and employed different supervision strategies for different forgery cues. PSCC-Net~\cite{PSCC} designed a top-down refined attention mechanism to progressively refine detection results. ObjectFormer~\cite{ObjectFormer} introduces a module based on the Vision Transformer (ViT)~\cite{ViT} architecture and learns the correlation between intermediate features in objects and patches through cross-attention.
% {EVP~\cite{EVP} proposed a method of Explicit Visual Prompting on a pre-trained Transformer, which can be applied to image manipulation detection.}
% CAT-Net~\cite{CAT-Net} focuses on learning the JPEG compression artifacts left during the image compression process to distinguish between authentic and manipulated regions.
% PCL~\cite{PCL} introduces a method of Proposal Contrastive Learning to extract the local feature relationships between manipulated and authentic regions.
% SAFL-Net~\cite{SAFL-Net} enhances its generalization on manipulation detection tasks by suppressing the semantic information of manipulated images. NCL-IML~\cite{NCL} introduces a non-mutually exclusive contrastive learning approach, mitigating the issue of limited data in manipulation detection tasks. HiFi-Het~\cite{HiFi} proposes a tree-structured detection method based on relationships among different forgery types, to learn intrinsic connections between different manipulation methods.
% WSCL~\cite{WSCL} proposes a weakly supervised self-consistency learning method, which is beneficial for enhancing generalization and alleviating data scarcity issues.
% \smallskip
% \noindent{\bf Modeling Image Correlations for IMD.}
% Within these methods, strategies {using graph spirits} for capturing the image correlations have been studied to further enhance feature effectiveness. HGCN-Net~\cite{HGCN} introduced graph structures for manipulation detection tasks, which modeled region adjacency relationships on feature grids by considering four connectivity. PSCC-Net~\cite{PSCC} introduced a module for capturing spatial correlations using attention. This module can be conceptualized as a fully connected graph, where each element corresponds to a grid and is connected to every other element. ERMPC~\cite{ERMPC} extracted the inconsistency between real and manipulated regions by controlling message passing.
% It is worth highlighting that the aforementioned methods share a commonality in constructing graphs based on grids. However, due to the content independence and fixed sizes of these grids, they lack an understanding of the content coherence and often struggle to capture irregularly manipulated regions. As such, we describe the Hierarchical Region-aware Graph Reasoning method to address these limitations.
% \blue{Regular grids may lead to difficulties in capturing irregularly tampered objects.} \blue{Since the grids are content-independent with fixed sizes and graph structures are static, they lack the understanding of coherence and relevancy, limiting their representations.}
% In principle, Vision Transformer (ViT)~\cite{ViT} is equivalent to a fully connected graph. IML-ViT~\cite{IML-ViT} pretrains ViT using the Masked Autoencoder~\cite{MAE}, mitigating to some extent the issue of limited data in manipulation detection tasks. EVP~\cite{EVP} fine-tunes a pre-trained ViT model using Explicit Visual Prompting, unifying foreground segmentations, including manipulation detection.
% All the mentioned methods, whether based on fully connected or 4-connected graphs, share a common issue: these modules are constructed on regular grids. Our Hierarchical region-aware Graph Reasoning addresses this problem from a different perspective.
% 1. introduce image manipulation detection methods, \\
% 2. introduce the application of current graph based solutions in image manipulation detection methods.
% Thus, we target the encoder and propose a new graph-based method to enhance the effectiveness of manipulation-concentrated feature extraction.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \vspace{-0.2cm}
% \smallskip
% \noindent{\bf Further Analysis of Existing Methods.} To capture the correlations, existing methods usually employ the mechanism of either attention \cite{PSCC,ObjectFormer} or graph \cite{HGCN,ERMPC}. For example, The attention xxxxxxxx? The graph xxxxxxx.
% Despite their showing promising results, they have a common limitation: \textbf{They all rely on the grids to model the correlations.}
% The grids are regular and fixed squares that are independent of the image content, \ie, for any input image, the grid is always regular and the same (See Fig.?).
% Typically, the existing methods first partition the feature maps into grids. Then the features inside grids are encapsulated as entities for modeling correlations.
% However, since the manipulation usually happens on the semantic-integrity objects (\eg, copy-move/splicing a whole person), using grids can hardly retain the content coherence in manipulated objects, resulting in the ambiguous detection boundary (See Fig.?).
% % Therefore, a natural question to ask is
% A straightforward question to ask is \textit{``Could we model the correlations based on content-coherence entity instead of regular grids?''}
% \begin{enumerate}
% \item[$\spadesuit$] They all rely on the grids to model the correlations. The grids are regular and fixed squares that are independent of the image content, \ie, for any input image, the grid is always regular and the same (See Fig.?). However, since the manipulation usually happens on the semantic-integrity objects (\eg, copy-move/splicing a whole person), using grids can hardly retain the semantic coherence in manipulated objects, resulting in the ambiguous detection boundary (See Fig.?).
% \item[$\spadesuit$] The correlations are usually conducted within single feature layers (see Fig.?). However, this operation overlooks the correlations across different feature scales, failing to seek further guidance from adjacent feature layers. For example,
% \end{enumerate}
% \begin{center}
% \fcolorbox{black}{gray!10}{\parbox{\linewidth}{
% \textbf{Question \#1}: How to ?}}
% \end{center}
% \begin{center}
% \fcolorbox{black}{gray!10}{\parbox{\linewidth}{
% \textbf{Question \#2}: How to ?}}
% \end{center}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\vspace{-0.2cm}
\section{Method}
% \subsection{Overview}
% In contrast to previous works that use grids for correlation modeling, we develop an end-to-end Hierarchical Region-aware Graph Reasoning method to effectively excavate the image correlations. The overview of our method is illustrated in Fig.~\ref{fig:overview} (left).
% Our method consists of two main components: Hierarchical Region-aware Graph and Structure-agnostic Graph Reasoning. The Hierarchical Region-aware Graph is a newly proposed graph structure constructed on content-coherence feature regions and the Structure-agnostic Graph Reasoning passes the messages through the constructed graph to enhance the effectiveness of features. These two components are elaborated in Sec.~\ref{sec:hrg} and Sec.~\ref{sec:sgr} respectively. Then we describe the training process of our method in Sec.~\ref{sec:training}. The overall process of our method is shown in Alg.~\ref{alg:process}.
% feature regions and construct a hierarchical region-aware graph on the regions from multiple feature layers. Then a structural-agnostic graph reasoning strategy is proposed to update the representations of graph nodes, which are then projected into corresponding pixels for feature enhancement. During the training process, the feature regions and graph model are alternatively updated, which reflects the essence of Coherency in Relevancy and Relevancy for Coherency.
% \subsection{Preliminary}
% Image manipulation detection aims to predict binary labels (authentic or manipulated) for each pixel.
% To achieve this, an encoder-decoder architecture is typically employed. The encoder transforms the input image to manipulation-concentrated features, and the decoder generates a detection map on the encoded features. Denote the encoder as $\mathcal{E}$ and the decoder as $\mathcal{D}$. Given an input image $\bm{x} \in \mathcal{X} = \{0,...,255\}^{H \times W \times 3}$, the detection result can be defined as $\bm{y} = \mathcal{D}(\mathcal{E}(\bm{x})), \bm{y} \in [0,1]^{H \times W \times 2}$, where $0,1$ denotes the pixel is authentic or manipulated.
% Since the effectiveness of features highly determines the performance, the recent efforts mainly focus on exploring the design of the encoders. This paper focuses on the typical solution of image correlation modeling, to enhance the effectiveness of encoders.
\subsection{Hierarchical Region-aware Graph}
\label{sec:hrg}
With feature maps, the gist of constructing this graph lies in two main steps: 1) formulating graph nodes and 2) modeling the correlations among these graph nodes. To ensure the end-to-end fashion of our method, accomplishing these two steps is not trivial. In the following, we first outline the overview of our method and then elaborate on the details of each step.
% \smallskip
% \noindent{\bf Overview.}
% To formulate the graph nodes, we propose a new Differentiable Feature Partition (DFP) strategy (described in Sec.\ref{sec:dfp}) to create feature regions. This strategy can partition the features into several non-overlapped and content-coherent regions in an unsupervised way, and can be integrated into the training in an end-to-end fashion.
% After obtaining the feature regions, we encapsulate the feature elements inside regions as the representation of graph nodes using a Weighted Feature Aggregation strategy (elaborated in Sec.\ref{sec:nr}).
% With the obtained graph nodes, we propose a Hierarchical Graph Modeling strategy to construct a graph on feature regions within the same and across different feature layers (described in Sec.\ref{sec:hgm}).
\subsubsection{Differentiable Feature Partition}
\label{sec:dfp}
To formulate the graph nodes, we propose a new Differentiable Feature Partition (DFP) strategy to create feature regions. This strategy can partition the features into several non-overlapped and content-coherent regions in an unsupervised way, and can be integrated into the training in an end-to-end fashion.
Intuitively, the conventional clustering algorithms (\eg, K-means) can be applied to feature layers by measuring the similarity among feature elements. However, this algorithm suffers from two difficulties. One is that local connectivity can not be guaranteed and the other is the clustering process is not differentiable. Inspired by the superpixel extraction methods \cite{SpixelFCN,SSN} and etc., we construct a soft feature element-region association matrix to make the clustering process differentiable.
Denote a feature map from the encoder $\mathcal{E}$ as $f \in \mathbb{R}^{h \times w \times c}$, where $h,w,c$ are the corresponding number of feature height, width, and channel. To ensure region connectivity, we append the position (\ie, x- and y- coordinates) of each feature element on its channel dimension as $f' \in \mathbb{R}^{h \times w \times c'},c'=c+2$.
Note that the time cost in clustering is proportional to feature dimensions. To reduce the computational overhead while refining the features, we design a convolutional block $\phi$ that transforms the features $f'' = \phi (f'), f'' \in \mathbb{R}^{h \times w \times c''} (c'' < c')$.
Based on $f''$, we initialize the region centers. Denote the number of regions as $m$. We evenly divide $f''$ into $m$ grids, and average the features inside each grid as the initial representation of region centers. Let the region centers be a matrix $\mathbf{R}'' = [ r''_1; ...; r''_m ] \in \mathbb{R}^{m \times c''}$. Each region center $r''_i$ has a size of $1 \times c''$. For clustering, we build the feature element-region association matrix to store the probability of assigning a feature element to a region, denoted as $\mathbf{D} \in \mathbb{R}^{n \times m}, n = h \times w$.
For the $i$-th feature element $e''_i \in f''$, we calculate the Euclidean distance between it and other region centers in $\mathbf{R}''$ and perform a softmax operation to normalize the probability, as
\begin{equation}
\begin{aligned}
\mathbf{D}(i, j) = \frac{{\exp}^{-\lVert e''_i - r''_j \rVert^{2}}}{\sum^{m}_{j=1} {\exp}^{-\lVert e''_i - r''_j \rVert^{2}}}.
\end{aligned}
\label{eq:prob}
\end{equation}
After obtaining the matrix $\mathbf{D}$, the intuitive way of obtaining regions is to assign each element to its nearest center, as
\begin{equation}
\begin{aligned}
\mathbf{D}(i, j) =
\left \{
\begin{array}{cl}
1 & \text{if} \quad j = \mathop{\arg\max}\limits_{j \in \{1,...,m \}} (\mathbf{D}(i,j)), \\
0 & \text{otherwise}.
\end{array}
\right.
\end{aligned}
\label{eq:assign}
\end{equation}
Then the region center $r''_j \in \mathbf{R}''$ can be updated by
\begin{equation}
\begin{aligned}
r''_j = \frac{\sum^{n}_{i=1} e''_i \cdot \mathbf{D}(i, j) }{\sum^{n}_{i=1} \mathbf{D}(i, j)}.
\end{aligned}
\label{eq:center}
\end{equation}
However, this assignment of Eq.~\eqref{eq:assign} is not differentiable due to the \texttt{argmax} operation, which thus can not be incorporated into our model.
% \begin{center}
% \fcolorbox{black}{gray!5}{\parbox{0.95\linewidth}{
% \textit{``How to make this process differentiable?''}}
% }
% \end{center}
\smallskip
\textit{\includegraphics[width=0.3cm]{figures/R.jpg}: ``How to make this process differentiable?''}
\smallskip
To address this issue, we use soft assignments, \ie, an element can be assigned to all regions with various probabilities. Specifically, we abandon Eq.~\eqref{eq:assign} and update the region centers using the probabilities in matrix $\mathbf{D}$, directly following Eq.~\eqref{eq:center}. The update process can be formulated as the operations between matrices as
\begin{equation}
\begin{aligned}
& \mathbf{F}'' \in \mathbb{R}^{n \times c''} \leftarrow \text{Reshape}({f''} \in \mathbb{R}^{h \times w \times c''}), \\
& \mathbf{R}'' = \frac{1}{\mathbf{N}_{\rm D}} \circ (\mathbf{D}^{\top} \mathbf{F}''),
\end{aligned}
\label{eq:center_m}
\end{equation}
where $\mathbf{N}_{\rm D} = [\sum^{n}_{i=1} \mathbf{D}(i, 1),...,\sum^{n}_{i=1} \mathbf{D}(i, m)]$ is the summation of each column of $\mathbf{D}$, $\circ$ denotes the element-wise product.
Note the feature regions are updated iteratively. Denote the current iteration as $t$. Once the region centers $\mathbf{R}''_{t}$ are updated, we can re-calculate the matrix $\mathbf{D}_{t+1}$ and then update $\mathbf{R}''_{t+1}$. This process is repeated until the maximum iteration $T$ is reached. The overview of this process is shown in Fig.~\ref{fig:overview} (right-top).
% The algorithm of this process is shown in Alg.~\ref{alg:process} (Line 1 - Line 12).
\subsubsection{Weighted Feature Aggregation for Node Representation}
\label{sec:nr}
After obtaining the feature regions, we encapsulate the feature elements inside regions as the representation of graph nodes using a Weighted Feature Aggregation strategy.
Denote the matrix of node representations as $\mathbf{R} = [ r_i;...;r_m ] \in \mathbb{R}^{m \times c}$.
{The encapsulation process can be defined with similar formulas as Eq.~\eqref{eq:center_m}, with the difference that here the encapsulation is applied to the original encoder features $f$, as}
\begin{equation}
\begin{aligned}
& \mathbf{F} \in \mathbb{R}^{n \times c} \leftarrow \text{Reshape}({f} \in \mathbb{R}^{h \times w \times c}), \\
& \mathbf{R} = \frac{1}{\mathbf{N}_{\rm D}} \circ (\mathbf{D}^{\top} \mathbf{F}).
\end{aligned}
\label{eq:center_f}
\end{equation}
% \begin{algorithm}[!t]
% \small
% \caption{\small Overall process of our method.}
% \textbf{Input}: Feature maps from the encoder $\{f^1,...,f^k\}$, iteration number $T$ \\
% \textbf{Output}: Enhanced $\{f^1,...,f^k\}$
% \begin{algorithmic}[1] %[1] enables line numbers
% \STATE {\texttt{\footnotesize{\violet{// In parallel}}}}
% \FOR{$f \in \{f^1,...,f^k\}$}
% \STATE Transforming $f$ to $f''$
% \STATE {\texttt{\footnotesize{\violet{// Differentiable Feature Partition}}}}
% \STATE Initial region centers $\mathbf{R}''_0$
% \FOR{$t = 1 \; {\rm to} \; T$}
% \STATE Calculating matrix $\mathbf{D}_{t}$ using Eq.\eqref{eq:prob},
% \STATE Updating region centers $\mathbf{R}''_{t}$ using Eq.\eqref{eq:center_m},
% \ENDFOR
% \STATE {\texttt{\footnotesize{\violet{// Graph Nodes Formulation}}}}
% \STATE Obtaining the representations of graph nodes $\mathbf{R}$ using Eq.\eqref{eq:center_f},
% \ENDFOR
% \STATE {\texttt{\footnotesize{\violet{// Hierarchical Graph Modeling}}}}
% \STATE Obtaining adjacency matrix $\mathbf{A}$,
% \STATE Obtaining graph nodes representations matrix $\mathbf{R}_{\rm G}$,
% \STATE {\texttt{\footnotesize{\violet{// Structure-agnostic Graph Reasoning}}}}
% \STATE Updating $\mathbf{R}_{\rm G}$ by graph reasoning in Eq.\eqref{eq:message_pass},
% \STATE {\texttt{\footnotesize{\violet{// Graph Features Inverse Mapping}}}}
% \STATE Mapping $\mathbf{R}_{\rm G}$ to $\{f^1,...,f^k\}$ using Eq.\eqref{eq:mapping_fuse},
% \STATE Fusing the mapped features with original features using Eq.\eqref{eq:remap} and Eq.\eqref{eq:mapping_fuse}.
% \end{algorithmic}
% \label{alg:process}
% \end{algorithm}
\subsubsection{Hierarchical Graph Modeling}
\label{sec:hgm}
With the obtained graph nodes, we propose a Hierarchical Graph Modeling strategy to construct a graph on feature regions within the same and across different feature layers.
Denote a set of feature maps from different layers as $\{f^1,...,f^k\}$. We first obtain the feature element-region matrix for each feature as $\{\mathbf{D}^1,...,\mathbf{D}^k\}$ and their corresponding node representations as $\{\mathbf{R}^1,...,\mathbf{R}^k\}$. Denote the adjacent matrix for this graph as $\mathbf{A} \in \mathbb{R}^{M \times M}$, where $M$ is the total number of nodes, \ie, $M = \sum^{k}_{i=1} | \mathbf{R}^i |$ (see Fig.~\ref{fig:relationships} for illustration).
% \begin{center}
% \fcolorbox{black}{gray!5}{\parbox{0.95\linewidth}{
% \textit{``How to seek the neighbors?''}}
% }
% \end{center}
\smallskip
\textit{\includegraphics[width=0.3cm]{figures/R.jpg}: ``How to seek the hierarchical neighborhood relationships?''}
\smallskip
Since the regions corresponding to nodes are irregular, modeling the relationship is more challenging than the conventional grid-based methods.
As such, we describe a 3D sliding-window based strategy to seek the correlations.
It is noteworthy that in this step, we require an assignment of feature elements to regions. Therefore, we reuse Eq.~\eqref{eq:assign} to divide each feature map into non-overlapped regions and assign each region a unique index from $\{1,...,M\}$. Then we can create the region index maps $\{J^1,...,J^k\}$ corresponding to $\{f^1,...,f^k\}$. We resize each index map to the same spatial size and concatenate them as a matrix $\mathbf{J} \in \mathbb{R}^{h \times w \times k}$. Then we perform the 3D sliding window with a size of $3 \times 3 \times 3$ in order. The criterion is that $\mathbf{A}(\mathbf{J}(p_0),\mathbf{J}(p_0+p_n)) = 1$, {if $\mathbf{J}(p_0) \neq \mathbf{J}(p_0+p_n)$, where $p_0$ is the position of the window center and $p_n$ is the offset within window.}
To accelerate this process, we formulate it into CUDA code.
% \blue{We build a hierarchical graph that establishes adjacency relationships within the same layer (intra-layer, orange arrows in Fig.~\ref{fig:concept} (b)) and also across different layers (inter-layer, blue arrows in Fig.~\ref{fig:concept} (b)), resulting in a single adjacent matrix $\mathbf{A}$. For simplicity, we use an example with two layers (highlighted in blue with one node and green with four nodes, see Fig~\ref{fig:relationships} (a)). First, we resample them to the same resolution (see Fig~\ref{fig:relationships} (b)). Then we stack them along channel dimension and apply the 3D sliding window strategy to obtain an adjacency matrix (see Fig~\ref{fig:relationships} (c,d)). }
\begin{figure}[!t]
\centering
\vspace{-0.35cm}
\includegraphics[width=1\linewidth]{figures/relationships.pdf}
\vspace{-0.6cm}
\caption{Illustration of adjacent matrix $\mathbf{A}$. We build a hierarchical graph that establishes adjacency relationships within the same layer (intra-layer, orange arrows in Fig.~\ref{fig:concept} (b)) and also across different layers (inter-layer, blue arrows in Fig.~\ref{fig:concept} (b)), resulting in a single adjacent matrix $\mathbf{A}$. For simplicity, we use an example with two layers (highlighted in blue with one node and green with four nodes, see Fig~\ref{fig:relationships} (a)). First, we resample them to the same resolution (see Fig~\ref{fig:relationships} (b)). Then we stack them along channel dimension and apply the 3D sliding window strategy to obtain an adjacency matrix (see Fig~\ref{fig:relationships} (c,d)). }
\vspace{-0.5cm}
\label{fig:relationships}
\end{figure}
% \textit{\includegraphics[width=0.3cm]{samples/figures/R.jpg}: ``Will the hard assignment in seeking neighbors break the differentiability?''}
% \blue{???. see Fig.~\ref{??}}
Besides the adjacent matrix $\mathbf{A}$,
we construct the node representation matrix for all regions.
Since $\mathbf{R}^1,...,\mathbf{R}^k$ has different channel numbers, we perform a linear transform for each matrix to uniform their channel number as $C$. Denote the node representation matrix as $\mathbf{R}_{\rm G} \in \mathbb{R}^{M \times C}$, which is obtained by
\begin{equation}
\begin{aligned}
\mathbf{R}_{\rm G} & = [\mathbf{R}^{1}_{\rm G};...;\mathbf{R}^{k}_{\rm G}] = [\mathbf{R}^1 \mathbf{W}^1;...;\mathbf{R}^k \mathbf{W}^k],
\end{aligned}
\end{equation}
where $\mathbf{W}^1,...,\mathbf{W}^k$ are learnable linear matrics.
% \begin{figure*}[!htb]
% \centering
% \includegraphics[width=1\linewidth]{build_graph.pdf}
% \caption{Hierachical regions to adjacency matrix}
% \label{fig:graph}
% \end{figure*}
\subsection{Structure-agnostic Graph Reasoning}
\label{sec:sgr}
% In the design of our method, the feature regions can be dynamically updated during training, which subsequently results in different graph structures.
% Therefore, the transductive graph convolutions \cite{GCN,hu2023leveraging,zhu2023transductive} can not be directly employed for message passing in our method.
% To process the dynamic graphs, we adapt structure-agnostic graph reasoning strategies \cite{GraphSAGE} and etc. to our method. Generally, this strategy consists of two steps: aggregation and update. The aggregation step gathers the information from surrounding nodes and the update step transforms the node representations based on the aggregated information with a learnable weight matrix.
In the design of our method, the feature regions can be dynamically updated during training, which subsequently results in different graph structures.
To process the dynamic graphs, we adapt structure-agnostic graph reasoning strategies \cite{GraphSAGE} and etc. to our method. Generally, this strategy consists of two
steps: aggregation and update. The aggregation step gathers the information from surrounding nodes and the update step transforms the node representations based on the aggregated information with a learnable weight matrix.
% \blue{To exhibit the efficacy of our hierarchical region-aware graph, we employ only a prototype of \cite{GraphSAGE,fey2019just,xiao2023adversarially}. Since the graph evolves with each training iteration, we discard random sampling operations to improve efficiency due to relatively restricted adjacencies repretented by $\mathbf{A}$. We also use averaging instead of complex mechanisms for feature aggregation.}
For each node, we simply average the representations of surrounding nodes and employ linear transformation to update the current node representations. Since both of the steps are independent of specific graph structures, our method can be applied even if node adjacency and the number of nodes vary.
Denote the learnable linear transformation matrix as $\mathbf{W_{G}} \in \mathbb{R}^{C \times C}$. The message passing process can be formulated as
\begin{equation}
\begin{aligned}
\hat{\mathbf{R}}_{\rm {G}}=\left ( \frac{1}{\mathbf{N_A}}\circ(\mathbf{A}\mathbf{R}_{\rm G}) \right ) \mathbf{W}_{\rm G},
\end{aligned}
\end{equation}
where $\mathbf{N_A} = [\sum^{M}_{i=1} \mathbf{A}(1, i);...;\sum^{M}_{i=1} \mathbf{A}(M,i)]$ is the summation of each row of $\mathbf{A}$.
To mitigate the over-smoothing effect, we employ a regularization on $\hat{\mathbf{R}}$ following \cite{ViG}, which is defined as
\begin{equation}
\begin{aligned} \mathbf{R}_{\rm G}=\sigma(\hat{\mathbf{R}}_{\rm {G}}\mathbf{W_{\alpha}})\mathbf{W_{\beta}} + \hat{\mathbf{R}}_{\rm {G}},
\end{aligned}
\label{eq:message_pass}
\end{equation}
where $\mathbf{W_{\alpha}}, \mathbf{W_{\beta}}$ are two learnable linear matrics and $\sigma$ is an activation function (\eg, GeLu).
\textit{Note that this process is only a prototype version to demonstrate the effectiveness of the proposed hierarchical region-aware graph. It can be upgraded with other advanced graph reasoning approaches, which may result in additional improvement.}
% \smallskip
% \noindent{\bf Graph Representations Inverse Mapping.}
After graph reasoning, we remap the updated representations back to the feature elements for feature enhancement. The essence is to revert Eq.~\eqref{eq:center_f}. For the feature $f^i$, the inverse mapping can be defined as
\begin{equation}
\begin{aligned}
\mathbf{F}^{i}_{\rm G}=\mathbf{D}^{i}\mathbf{R}^i, \quad \mathbf{R}^i = \mathbf{R}^{i}_{\rm G} {\mathbf{W}^{i}_{\rm inv}},
\end{aligned}
\label{eq:remap}
\end{equation}
where $\mathbf{R}_{\rm G}^i$ denotes the updated representations of graph nodes at $i$-th feature layer, $\mathbf{W}^{i}_{\rm inv}$ is a linear matrix to match the channel number of $f^i$, $\mathbf{D}^{i}$ is the feature element-region association matrix at $i$-th feature layer.
Then we integrate $\mathbf{F}^{i}_{\rm G}$ with the features $f^i$ as
\begin{equation}
\begin{aligned}
& f^{i}_{\rm G} \leftarrow \text{Reshape}(\mathbf{F}^{i}_{\rm G}), f^{i} = \mu f^{i}_{\rm G} + f^{i},
\end{aligned}
\label{eq:mapping_fuse}
\end{equation}
where $\mu$ is a learnable parameter that controls the feature integration.
% \subsection{Detection Decoder}
% \label{sec:dd}
% \begin{figure}[!t]
% \centering
% \includegraphics[width=0.7\linewidth]{figures/uperhead.pdf}
% \vspace{-0.4cm}
% \caption{\small Architecture of the detection decoder.}
% \vspace{-0.6cm}
% \label{fig:uperhead}
% \end{figure}
% To demonstrate the efficacy of our method, we only employ a conventional feature pyramid detector \cite{UPerNet}.
% The prior researches \cite{PSCC,ERMPC} have demonstrated the effectiveness of feature fusing strategies in the decoder. In our method, we follow this strategy and employ a feature pyramid structure described in \cite{UPerNet}. Specifically, we design a top-down decoder architecture with lateral connections to the encoder and perform a pyramid pooling on the last feature map of the encoder to capture multi-scale information. The overview of the architecture is shown in Fig.~\ref{fig:uperhead}. Details of this architecture can be found in {\em Supplementary}.
% \textit{It is noteworthy that the major contribution of this work lies in enhancing the feature effectiveness of the encoder using the proposed method. Thus we employ a plain architecture for the detection decoder to demonstrate the efficacy of our method. The detection performance can potentially be further improved if a more advanced decoder is employed.}
% \begin{figure*}[!htb]
% \centering
% \includegraphics[width=1\linewidth]{uperhead.pdf}
% \caption{Decoder Head}
% \label{fig:decoder}
% \end{figure*}
% \color{blue}
% Prior research \cite{PSCC} has demonstrated the effectiveness of top-down fusion for hierarchical features extracted by the encoder. We also adopt this top-down refinement strategy. However, in contrast to \cite{PSCC}, which performs top-down refinement on the predicted mask, inspired by feature pyramids \cite{?}, we directly apply top-down refinement to the features in the encoder. The overall structure of our decoder head is illustrated in Figure \ref{fig:decoder}. The decoder is divided into two parts. For all features except the highest level, we directly use feature pyramids \cite{FPN} for fusion. Regarding the highest-level feature, we first employ pyramid pooling \cite{PSP} before fusion.
% \noindent{\bf Pyramid pooling.}
% Although pooling can capture global information, it may lead to information loss. Different-scale pooling can partially compensate for this drawback by effectively extracting context information at different scales. For the input features, pooling is performed at different scales, resulting in resolutions of $1\times 1, 2\times 2, 3\times 3, 6\times 6$. Subsequently, $1\times 1$ convolutions are applied to these features at different scales to reduce them to $1/n$ of their original size, where $n$ is the number of pooling scales. Finally, these features are upsampled and concatenated with the input features. Empirically, pooling is performed on the deepest features of the network, where the resolution is the lowest, making it easier to extract global features.
% \noindent{\bf Feature Pyramid.}
% We utilize a top-down architecture with lateral connections to merge high-level semantic information into the lower levels. After passing through the Pyramid Pooling module, the features undergo a $1\times 1$ convolution to obtain the top-down highest-level features. Subsequently, we progressively upsample the top-down features and fuse them with the lateral encoder features layer by layer . A $3\times 3$ convolution is applied to each combined feature to produce the final feature map, aiming to mitigate the aliasing effect caused by upsampling \cite{?}. Finally, we upsample and concatenate the features from different top-down levels to obtain the fused feature.
% \noindent{\bf Segmentation head.}
% The fused features are passed through a $1\times 1$ convolution, reducing the number of channels to 2. A $\texttt{softmax}$ operation is performed along the channel dimension, yielding the probabilities in pixel level for both authentic and manipulated regions.
% \color{black}
\subsection{Objective Functions}
\label{sec:training}
% \smallskip
% \noindent{\bf Objective Functions.}
Note that the proposed method does not require extra supervision. Thus we only employ a task-specific loss function for training.
Considering that the area of the manipulated regions in images is usually less than the authentic surroundings, we employ focal loss \cite{focal} for balancing the importance of authentic and manipulated pixels, which is defined as
%%% here
\begin{equation}
\begin{aligned}
\mathbf{L}(\bm{y}, \hat{\bm{y}}) = - \sum & \left ( \alpha \cdot (1 - \bm{y})^{\gamma} \cdot \hat{\bm{y}} \log (\bm{y}) \right. \\
& \left. + (1 - \alpha) \bm{y}^{\gamma} (1 - \hat{\bm{y}}) \log (1 - \bm{y}) \right ),
\end{aligned}
\end{equation}
where $\hat{\bm{y}}$ represents the ground truth mask, $ \alpha $ is the weight for balancing, and $ \gamma $ is the difficulty adjustment factor. The larger value of $ \gamma $ indicates a focus on more challenging samples.
% \begin{equation}
% \begin{aligned}
% \text{FL}(p_t)=-\alpha_t(1-p_t)^{\gamma}\log(p_t)
% \end{aligned}
% \end{equation}
% where $ p_t \in [0,1] $ represents the model's predicted probability that sample $ t $ is a positive sample, $ \alpha_t $ represents the weight of sample $ t $, and $ \gamma $ is the difficulty adjustment factor, with a larger value indicating a focus on more challenging samples. Empirically, we set $ \alpha_t = 0.5 $ and $ \gamma = 2 $.
% \color{black}
% \smallskip
% \noindent{\bf Training Process.}
% {Given a batch of image features, the feature regions are obtained within the maximum iteration number $T$. This stage updates the feature element-region association matrix $\mathbf{D}$ and region center matrix $\mathbf{R}''$.} Then these feature regions are encapsulated as graph nodes $\mathbf{R}_{\rm G}$ for constructing the hierarchical graph model. Then the graph reasoning is performed to update $\mathbf{R}_{\rm G}$, by optimizing the parameters $\mathbf{W}_{\rm G},\mathbf{W}_{\alpha},\mathbf{W}_{\beta}$. Lastly, $\mathbf{R}_{\rm G}$ is mapped back to the original features and fused with them as the enhanced features. These enhanced features are then sent into the detection decoder for objective calculation. This process is repeated in the next training batch.
% \subsection{Rethinking the Insights}
% During the training process, the feature region and graph model are alternatively updated, which reflects the form of Coherency in Relevancy and Relevancy for Coherency.
% \smallskip
% \noindent{\bf Coherency in Relevancy.}
% The proposed region-aware graph model seeks the correlations among different coherent feature regions. Since the region-aware graph nodes naturally reflect the coherency, they have better representations, resulting in the effective construction of correlations.
% \smallskip
% \noindent{\bf Relevancy for Coherency.}
% After graph reasoning, the weight parameters of the network $\phi$ are updated, improving the effectiveness of features for clustering. Then the next round of graph modeling is performed again on these updated feature regions.
% \subsection{Highlights of Distinction with Existing Methods}
% \vspace{-0.3cm}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Experiments}
\subsection{Experimental Settings}
% \vspace{-0.2cm}
\smallskip
\noindent{\bf Datasets.}
% Following previous methods, we evaluate our method on five datasets, CASIA \cite{casia}, Coverage \cite{coverage}, NIST16 \cite{nist16}, Columbia \cite{columbia}, and IMD20 \cite{imd20}.
Following previous methods, we evaluate our method on five datasets, CASIA\cite{casia}, Coverage\cite{coverage}, NIST16\cite{nist16}, Columbia\cite{columbia}, and IMD20\cite{imd20}.
The CASIA dataset contains 921 images in CASIAv1 and 5123 images in CASIAv2. We use CASIAv2 for training and CASIAv1 for testing.
The coverage dataset comprises 100 images, with 75 images for training and 25 for testing. NIST16 has 564 images, where 404 images for training and 160 images for testing. The Columbia dataset consists of 180 images, and {IMD20} dataset consists of 2010 images. Note that these two datasets do not provide training sets. Thus we only use them for testing.
\smallskip
\noindent{\bf Evaluation Metrics.}
Following previous work \cite{PSCC,ObjectFormer,ERMPC,HiFi}, we utilize the widely used F1 score for evaluation. The F1 score is calculated using the Equal Error Rate (EER) as the threshold. Moreover, we also employ pixel-level Area Under Curve (AUC) as complementary.
% \smallskip
% \noindent{\bf Base networks.}
% To demonstrate the effectiveness of our method, we employ several base networks as the encoder and perform our method on them. These base networks are commonly used in image manipulation detection, including ResNet-50 \cite{resnet}, HRNet \cite{hrnet}, EfficientNet-b4 \cite{efficient} and InternImage-tiny \cite{intern}.
% \smallskip
% \noindent\textbf{Detection Decoder.}
% \begin{figure}[!t]
% \centering
% \includegraphics[width=0.7\linewidth]{figures/uperhead.pdf}
% \vspace{-0.4cm}
% \caption{\small Architecture of the detection decoder.}
% \vspace{-0.6cm}
% \label{fig:uperhead}
% \end{figure}
\smallskip
\noindent{\bf Implementation Details.}
Our method is implemented by PyTorch 11.3 with a Nvidia 3090 GPU. In the training phase, we utilize the AdamW optimizer with a learning rate of $6 \times 10^{-5}$ and a weight decay of $0.05$. We employ a linear warm-up learning strategy, where the learning rate is linearly increased during warm-up and then linearly decreased based on the iteration until it reaches $0$. The initial learning rate for warm-up is set to $1 \times 10^{-6}$, and warm-up is completed within $1.5k$ iterations. The training image size is set to $512 \times 512$ and the training iterations are set to $160k$. For feature integration, each layer's $\mu_i$ is learned separately with initialization of $1$. In the objective function, we set the hyperparameters as follows: $\alpha = 0.5, \gamma = 2$. We employ a plain feature pyramid detector \cite{UPerNet} as the decoder. The detection performance can be further improved if a more advanced decoder is employed.
% Experimental results show that the integration coefficients $\mu_i$ for each layer roughly converge to $1.05\pm0.05$. This provides some evidence for the effectiveness of the proposed module.
% \begin{table}[!t]
% \centering
% \small
% \caption{\small Fine-tuned scenario. AUC and F1 scores (\%) are reported.}
% \vspace{-0.3cm}
% \tabcolsep=0.4em
% \resizebox{\linewidth}{!}{
% \begin{tabular}{c|cccccc|cc}
% \hline
% \multirow{2}{*}{Base network} & \multicolumn{2}{c}{CASIA} & \multicolumn{2}{c}{Coverage} & \multicolumn{2}{c|}{NIST16} & \multicolumn{2}{c}{avg} \\
% ~ & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 \\
% \hline
% Res-50 & 87.0 & 54.0 & 90.5 & 69.8 & 97.1 & \textbf{64.7} & 91.5 & 62.8 \\
% + HRGR & \textbf{88.5} & \textbf{56.2} & \textbf{92.7} & \textbf{71.3} & \textbf{97.2} & 64.1 & \textbf{92.8} & \textbf{63.9} \\
% \hline
% HRNet & 90.1 & 62.2 & 90.3 & 68.6 & 97.4 & 66.5 & 92.6 & 65.8 \\
% + HRGR & \textbf{91.4} & \textbf{64.2} & \textbf{90.9} & \textbf{69.5} & \textbf{98.4} & \textbf{67.1} & \textbf{93.6} & \textbf{66.9} \\
% \hline
% EffNet-b4 & 92.5 & 66.2 & 93.3 & \textbf{76.1} & \textbf{97.5} & 61.7 & 94.4 & 68.0 \\
% + HRGR & \textbf{92.9} & \textbf{66.7} & \textbf{93.9} & 74.8 & 97.2 & \textbf{63.4} & \textbf{94.7} & \textbf{68.3} \\
% \hline
% Intern-tiny & 91.4 & 66.0 & 94.5 & 78.7 & 99.3 & 79.3 & 95.1 & 74.7 \\
% + HRGR & \textbf{93.4} & \textbf{69.9} & \textbf{94.8} & \textbf{82.1} & \textbf{99.3} & \textbf{84.6} & \textbf{95.8} & \textbf{78.9} \\
% \hline
% \end{tabular}}
% \label{tab:fine-tuned-ours}
% \vspace{-0.4cm}
% \end{table}
% Following previous works \cite{SPAN,PSCC}, we explore two scenarios: 1) Pre-trained scenario: training on synthetic images and directly testing on other datasets, and 2) Fine-tuned scenario: fine-tuning the pre-trained model on the training set of target datasets and evaluating on their testing set. The pre-trained scenario focuses on exhibiting the generalizability of detection, while the fine-tuned scenario represents the capacity to capture manipulation traces in a specific dataset.
% \smallskip
% \noindent{\bf Fine-tuned Scenario.}
% We also fine-tuned different base networks on these datasets. Since only CASIA, Coverage, and NIST16 datasets provide training sets, we validate our method on these datasets. The results are shown in Table~\ref{tab:fine-tuned-ours}, which reveals that our method can improve the performance by $0.8\%$ in AUC and $1.7\%$ in the F1 score on average. This experiment demonstrates that our method can help improve the capacity of learning specific manipulation traces.
\begin{table}[!t]
\centering
\small
\caption{\small Comparison of different methods. The \textbf{best} result is highlighted in bold while the \underline{second-best} result is marked by underlining.}
\vspace{-0.3cm}
\tabcolsep=0.4em
\resizebox{\linewidth}{!}{
\begin{tabular}{c|cccccc|cc}
\hline
\multirow{2}{*}{Method} & \multicolumn{2}{c}{CASIA} & \multicolumn{2}{c}{Coverage} & \multicolumn{2}{c|}{NIST16} & \multicolumn{2}{c}{avg} \\
~ & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 \\
\hline
HGCN \cite{HGCN} & - & 40.8 & - & 39.3 & - & 71.0 & - & 50.4 \\
% RGB-N \cite{RGB-N} & 79.5 & 40.8 & 81.7 & 43.7 & 93.7 & 72.2 & 85.0 & 52.2 \\
% SPAN \cite{SPAN} & 83.8 & 38.2 & 93.7 & 55.8 & 96.1 & 58.2 & 91.2 & 51.6 \\
% TransF.* \cite{TransForensics} & 85.0 & 62.7 & 88.4 & 67.4 & - & - & 86.7 & 65.1 \\
PSCC-Net \cite{PSCC} & 87.5 & 55.4 & 94.1 & 72.3 & 99.1 & 74.2 & 93.7 & 69.8 \\
ObjFormer \cite{ObjectFormer} & 88.2 & 57.9 & 95.7 & 75.8 & 99.6 & 82.4 & 94.5 & 72.0 \\
EVP* \cite{EVP} & 86.2 & 63.6 & - & - & - & - & 86.2 & 63.6 \\
PCL \cite{PCL} & 75.1 & 46.7 & 91.7 & 62.0 & 94.6 & 78.0 & 87.1 & 62.2 \\
NCL* \cite{NCL} & 86.4 & 59.8 & 92.8 & 80.1 & 91.2 & 83.1 & 90.1 & 74.3 \\
HiFi-Net \cite{HiFi} & 88.5 & 61.6 & \underline{96.1} & \underline{80.1} & 98.9 & \underline{85.0} & 94.6 & 75.5 \\
ERMPC \cite{ERMPC} & 90.4 & 58.6 & \textbf{98.4} & 77.3 & \textbf{99.7} & 83.6 & \textbf{96.1} & 73.1 \\
% DiffF. \cite{DiffForensics} & 86.8 & 51.7 & - & - & 82.8 & 41.5 & 84.8 & 46.6 \\
ACBG. \cite{ACBG} & - & 66.9 & - & 71.1 & - & \textbf{93.3} & - & 77.1 \\
% \hline
% Our & \textbf{93.4} & \textbf{69.9} & 94.8 & \textbf{82.1} & 99.3 & 84.6 & 95.8 & \textbf{78.9} \\
% \hline
\hline
PSCC-Net$^\dagger$ \cite{PSCC} & 85.2 & 53.6 & 91.1 & 71.6 & 96.4 & 70.1 & 90.3 & 65.1 \\
HiFi-Net$^\dagger$ \cite{HiFi} & 85.9 & 59.0 & 86.2 & 63.4 & 87.4 & 62.8 & 86.5 & 61.7 \\
CAT-Net$^\dagger$ \cite{CAT-Net} & 77.3 & 40.6 & 79.2 & 47.8 & 89.2 & 61.5 & 81.9 & 50.0 \\
MVSS-Net$^\dagger$ \cite{MVSS} & 86.4 & 60.2 & 92.7 & 76.3 & 96.1 & 69.3 & 91.7 & 68.6 \\
ObjFormer$^\dagger$ \cite{ObjectFormer} & 71.9 & 42.6 & 68.4 & 41.3 & 91.4 & 68.2 & 77.2 & 50.7 \\
TruFor$^\dagger$ \cite{TruFor} & 92.4 & \underline{67.2} & 96.2 & 79.0 & 98.6 & 82.6 & 95.7 & 76.2 \\
\hline
Res-50 \cite{resnet} & 87.0 & 54.0 & 90.5 & 69.8 & 97.1 & 64.7 & 91.5 & 62.8 \\
+ HRGR & 88.5 & 56.2 & 92.7 & 71.3 & 97.2 & 64.1 & 92.8 & 63.9 \\
\hline
HRNet \cite{hrnet} & 90.1 & 62.2 & 90.3 & 68.6 & 97.4 & 66.5 & 92.6 & 65.8 \\
+ HRGR & 91.4 & 64.2 & 90.9 & 69.5 & 98.4 & 67.1 & 93.6 & 66.9 \\
\hline
EffNet-b4 \cite{efficient} & 92.5 & 66.2 & 93.3 & 76.1 & 97.5 & 61.7 & 94.4 & 68.0 \\
+ HRGR & \underline{92.9} & 66.7 & 93.9 & 74.8 & 97.2 & 63.4 & 94.7 & 68.3 \\
\hline
Intern-tiny \cite{intern} & 91.4 & 66.0 & 94.5 & 78.7 & 99.3 & 79.3 & 95.1 & 74.7 \\
+ HRGR & \textbf{93.4} & \textbf{69.9} & 94.8 & \textbf{82.1} & 99.3 & 84.6 & \underline{95.8} & \textbf{78.9} \\
\hline
\end{tabular}}
\label{tab:finetune}
% \vspace{-0.4cm}
\end{table}
% \begin{table}[!th]
% \centering
% \small
% \caption{\small Performance without Fine-tune. AUC and F1 scores (\%) are reported.}
% \vspace{-0.3cm}
% \resizebox{\linewidth}{!}{
% \begin{tabular}{c|cccccccccc|cc}
% \hline
% \multirow{2}{*}{Base network} & \multicolumn{2}{c}{CASIA} & \multicolumn{2}{c}{Coverage} & \multicolumn{2}{c}{NIST16} &
% \multicolumn{2}{c}{Columbia} & \multicolumn{2}{c|}{IMD20} & \multicolumn{2}{c}{avg} \\
% ~ & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 \\
% \hline
% ResNet-50 & 82.0 & 41.2 & 78.3 & 42.5 & 80.6 & 28.1 & \textbf{92.8} & \textbf{80.3} & 78.6 & 26.3 & 82.4 & 43.6 \\
% + HRGR & \textbf{82.3} & \textbf{42.6} & \textbf{78.9 }& \textbf{43.0} & \textbf{80.7} & \textbf{29.0} & 92.1 & 80.1 & \textbf{79.2} & \textbf{27.1} & \textbf{82.6} & \textbf{44.3} \\
% \hline
% HRNet & 82.7 & 44.9 & 80.3 & 45.6 & 81.7 & 31.5 & 95.1 & 87.2 & 79.6 & 28.4 & 83.8 & 47.5 \\
% + HRGR & \textbf{84.0} & \textbf{46.3} & \textbf{81.0} & \textbf{46.0} & \textbf{82.3} & \textbf{34.6} & \textbf{95.7} & \textbf{88.2} & \textbf{80.4} & \textbf{29.9} & \textbf{84.6} & \textbf{49.0} \\
% \hline
% EffNet-b4 & 82.1 & 43.3 & 74.3 & 35.6 & \textbf{83.2} & 30.1 & 85.8 & 68.5 & \textbf{81.3} & 31.5 & 81.3 & 41.8 \\
% + HRGR & \textbf{83.5} & \textbf{44.9} & \textbf{75.8} & \textbf{37.8} & 81.9 & \textbf{32.2} & \textbf{86.4} & \textbf{68.7} & 80.9 & \textbf{31.7} & \textbf{81.7} & \textbf{43.0} \\
% \hline
% Intern-tiny & 84.1 & 48.3 & 80.3 & 49.3 & \textbf{86.6} & 39.3 & 94.7 & 88.8 & 79.5 & 29.5 & 85.0 & 51.0 \\
% + HRGR & \textbf{86.3} & \textbf{52.4} & \textbf{83.9} & \textbf{54.7} & 86.2 & \textbf{39.4} & \textbf{96.4} & \textbf{91.2} & \textbf{79.8} & \textbf{30.4} & \textbf{86.5} & \textbf{53.6} \\
% \hline
% \end{tabular}
% }
% \label{tab:pretrained-ours}
% \vspace{-0.3cm}
% \end{table}
\begin{figure}[!t]
\centering
\includegraphics[width=1\linewidth]{figures/visualization_5.pdf}
\vspace{-0.6cm}
\caption{\small Qualitative visualization on CASIA, IMD20, NIST16, Coverage, and Columbia datasets (from top to bottom).}
\label{fig:visualization}
\vspace{-0.5cm}
\end{figure}
% \smallskip
% \noindent{\bf Comparing with State-of-the-arts.}
% To demonstrate the effectiveness of our method, we perform it on several base networks commonly used as the encoder in image manipulation detection. These include ResNet-50 \cite{resnet}, HRNet \cite{hrnet}, EfficientNet-b4 \cite{efficient} and InternImage-tiny \cite{intern}. For comparison with state-of-the-arts, we employ Intern-tiny as our base network.
\subsection{Comparing with State-of-the-arts}
Following previous works \cite{PSCC,HiFi}, we explore the Fine-tuned scenario, that firstly pre-trains the methods using synthetic dataset in \cite{PSCC} and fine-tunes the pre-trained model on the training set of each dataset. Then we evaluate the performance on its testing set. This scenario represents the capacity to capture manipulation traces in a specific dataset.
Following previous methods, the comparison is conducted on CASIA, Coverage, and NIST16 datasets since they provide training sets.
Table \ref{tab:finetune} shows the results of our method compared to other state-of-the-art methods and graph-based methods under the fine-tuned scenario.
We would like to clarify that EVP and NCL were trained without using synthetic dataset as others, while TransF. was tested on the Coverage dataset without fine-tuning. These three methods are marked by $(*)$.
% \blue{Moreover, the methods with $(\dagger)$ represent those that we have reproduced, where PSCC-Net and HiFi-Net using the code provided by the authors, and the rest of the methods using IMDLBenCo\cite{benco}.} We have included the details of our reproduction in the \textit{Supplementary}.
Moreover, the methods marked with $(\dagger)$ are the ones that we have reproduced. Specifically, PSCC-Net and HiFi-Net were reproduced using the code provided by the respective authors, while the remaining methods were reproduced with IMDLBenCo \cite{benco}.
% The detailed information regarding our reproduction is presented in the \textit{Supplementary}.
% Limited by the insufficient details provided in the original paper, we were unable to fully reproduce their results.
We can observe that our method improves the performance on these base networks by $0.8\%$ in AUC and $1.7\%$ in the F1 score on average, demonstrating the efficacy of our method to improve the capacity of learning specific manipulation traces. Compared to state-of-the-arts, our method achieves an average F1 score of $78.9\%$ across the three datasets, surpassing the highest-performing HiFi-Net by $3.3\%$. Particularly on the CASIA dataset, our method outperforms HiFi-Net with a $3.0\%$ improvement in the AUC score and a $8.3\%$ increase in the F1 score. These results demonstrate the superiority of the proposed method in capturing subtle manipulation traces. Moreover, our method only introduces a few additional parameters (1.45M) and FLOPS (1.49G). Fig.~\ref{fig:visualization} shows several visualizations of different methods on five datasets.
% \textit{Due to constrained computing resources, we could not employ more powerful architectures to push performance further. Nonetheless, this experiment has demonstrated that our method can enhance the capacity of base networks, showing the potential for further performance improvements with stronger architectures.}
% \includegraphics[width=0.3cm]{igures/R.jpg}:
% \smallskip
% \noindent{\bf Qualitative Comparison.}
% Fig.~\ref{fig:visualization} shows several visualizations of different methods on five datasets. It can be observed that our method achieves favorable detection results, especially in challenging images with imperceptible or ambiguous manipulated regions. More examples are in {\em Supplementary}.
% \begin{table}[h]
% \centering
% % \small
% % \vspace{-0.3cm}
% % \tabcolsep=0.4em
% \resizebox{\linewidth}{!}{
% \begin{tabular}{c|cccccc|cc}
% \hline
% \multirow{2}{*}{Method} & \multicolumn{2}{c}{CASIA} & \multicolumn{2}{c}{Coverage} & \multicolumn{2}{c|}{NIST16} & \multicolumn{2}{c}{avg} \\
% ~ & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 \\
% \hline
% PSCC w/o HRGR & 89.4 & 59.6 & \textbf{93.1} & 70.0 & 95.8 & 60.7 & 92.7 & 63.4 \\
% PSCC w/ HRGR & \textbf{89.9} & \textbf{61.3} & 92.0 & \textbf{70.4} & \textbf{96.6} & \textbf{64.2} & \textbf{92.8} & \textbf{65.3} \\
% \hline
% \end{tabular}}
% % \label{tab:finetune}
% % \vspace{-0.4cm}
% \end{table}
% \begin{table}[t]
% \centering
% \small
% % \tabcolsep=1em
% \caption{\small Robustness analysis under various perturbations.}
% \vspace{-0.3cm}
% \begin{tabular}{ccccc}
% \hline
% \multirow{2}{*}{Method} & \multicolumn{2}{c}{CASIA} & \multicolumn{2}{c}{Coverage} \\
% ~ & AUC & F1 & AUC & F1 \\
% \hline
% None & 93.4 & 69.9 & 94.8 & 82.1 \\
% \hline
% Resize 0.78x & 91.8 & 65.9 & 93.4 & 75.4 \\
% Resize 0.25x & 78.7 & 36.5 & 75.9 & 36.6 \\
% \hline
% GaussianBlur $k$=3 & 92.1 & 67.3 & 88.5 & 61.4 \\
% GaussianBlur $k$=15 & 77.8 & 33.5 & 69.1 & 27.7 \\
% \hline
% GaussianNoise $\sigma$=3 & 92.7 & 68.5 & 94.2 & 79.7 \\
% GaussianNoise $\sigma$=15 & 90.7 & 62.2 & 84.9 & 58.4 \\
% \hline
% JPEGCompression $q$=100 & 92.9 & 67.9 & 94.3 & 77.4 \\
% JPEGCompression $q$=50 & 81.6 & 38.0 & 91.0 & 68.1 \\
% \hline
% \end{tabular}
% \label{tab:robust}
% \vspace{-0.3cm}
% \end{table}
% \begin{table}[htb]
% \centering
% \small
% \tabcolsep=0.3em
% \caption{Comparisons of manipulation localization AUC (\%)scores of different pre-trained models.}
% \begin{tabular}{c|ccccc|c}
% \hline
% Method & Columbia & Coverage & CASIA & NIST16 & IMD20 & avg \\
% \hline
% PSCC-Net & 98.2 & 84.7 & 82.9 & 85.5 & 80.6 & 86.3 \\
% ObjectFormer & 95.5 & 92.8 & 84.3 & 87.2 & 82.1 & 88.3 \\
% HiFi-Net & 98.4 & 92.4 & 86.6 & 86.9 & 83.4 & 89.6 \\
% ERMPC & 96.8 & 94.4 & 87.6 & 89.5 & 85.6 & 90.7 \\
% Our & 96.4 & 84.0 & 86.3 & 85.0 & 79.8 & 86.3 \\
% \hline
% \end{tabular}
% \label{tab:pretrain}
% \end{table}
% \noindent{\bf Feature Regions.}
% We visualize the results of feature clustering in Figure \ref{fig:regions}. We execute Eq.(\ref{eq:assign}) separately for different layers $ {D^1, ..., D^k}$, obtaining the region with the highest probability for each element, and assign random colors to different regions. It can be observed that for low-level features, a roughly regular grid pattern is presented. This indicates that the features at this stage have high similarity and fail to distinguish between manipulated and genuine regions. For high-level features, the shapes of the feature regions closely resemble those of the manipulated regions, demonstrating that the network has captured the location of manipulated regions.
% \begin{figure}[t]
% \centering
% \includegraphics[width=1\linewidth]{figures/regions.pdf}
% \caption{Visualizations of feature clustering. We observe that high-level features capture the shape of manipulated regions, while low-level features still exhibit the traditional grid pattern. Zoom in for Details.}
% \label{fig:regions}
% \end{figure}
% \begin{table}[htb]
% \centering
% \small
% % \tabcolsep=1em
% \caption{Comparisons of manipulation localization scores during pretrain and finetune using different backbones on CASIA dataset. AUC and F1 scores (\%) are reported.}
% \begin{tabular}{ccccc}
% \hline
% \multirow{2}{*}{Method} & \multicolumn{2}{c}{Pretrain} & \multicolumn{2}{c}{Finetune} \\
% ~ & AUC & F1 & AUC & F1 \\
% \hline
% ResNet-50 & 82.02 & 41.19 & 86.97 & 53.97 \\
% w/ our & \textbf{82.32} & \textbf{42.63} & \textbf{88.45} & \textbf{56.16} \\
% \hline
% HRNetv2-w48 & 82.72 & 44.95 & 90.05 & 62.20 \\
% w/ our & \textbf{84.08} & \textbf{46.33} & \textbf{91.42} & \textbf{64.18} \\
% \hline
% EfficientNet-b4 & 82.14 & 43.37 & 92.54 & 66.17 \\
% w/ our & \textbf{83.58} & \textbf{44.90} & \textbf{92.89} & \textbf{66.69} \\
% \hline
% InternImage-tiny & 84.10 & 48.27 & 91.42 & 66.02 \\
% w/ our & \textbf{86.32} & \textbf{52.44} & \textbf{93.40} & \textbf{69.89} \\
% \hline
% \end{tabular}
% \label{tab:backbone}
% \end{table}
\subsection{Analysis}
\smallskip
\noindent{\bf Ablation Study.}
This part investigates the effect of different settings: 1) \textbf{Baseline} indicates solely using Intern-tiny network. 2) \textbf{Feature region $\rightarrow$ Grid} indicates replacing the feature partition module with the original regular grid. 3) \textbf{Graph reasoning $\rightarrow$ Linear} refers to replacing graph reasoning with several linear layers (equivalent to fully connected graph reasoning). 4) \textbf{Graph reasoning $\rightarrow$ MHSA} refers to replacing graph reasoning with several Mutil-Head Self Attention layers. 5) \textbf{Hierarchical $\rightarrow$ Layer-wise} means replacing the hierarchical graph modeling with layer-wise independent graph modeling and performing graph reasoning. 6) \textbf{DFP w/o x- and y- coords} means that we do not additionally concatenate the x and y coordinates in the clustering process of the DFP module.
Table \ref{tab:ablation} displays the results of ablation experiments. It can be observed that when any module is replaced, the performance drops significantly, \eg, F1 scores decreasing by $2.9\%$, $2.6\%$, $3.1\%$, $1.9\%$, $1.0\%$ in the finetune stage. Nevertheless, only replacing one component still has an improvement compared to the baseline, which confirms the effectiveness of each component.
\begin{table}[!t]
\centering
\small
% \tabcolsep=1em
\caption{\small Ablation study of different settings on CASIA dataset.}
\vspace{-0.3cm}
\begin{tabular}{ccccc}
\hline
\multirow{2}{*}{Method} & \multicolumn{2}{c}{w/o Fine-tune} & \multicolumn{2}{c}{w/ Fine-tune} \\
~ & AUC & F1 & AUC & F1 \\
\hline
Baseline & 84.1 & 48.3 & 91.4 & 66.0 \\
Feature region $\rightarrow$ Grid & 85.1 & 49.6 & 92.1 & 67.0 \\
Graph reasoning $\rightarrow$ Linear & 85.8 & 50.8 & 92.1 & 67.3 \\
Graph reasoning $\rightarrow$ MHSA & 85.3 & 49.1 & 91.7 & 66.8 \\
Hierarchical $\rightarrow$ Layer-wise & 85.6 & 50.6 & 92.5 & 68.0 \\
DFP w/o x- and y- coords & 85.4 & 49.5 & 93.3 & 68.9 \\
Full & \textbf{86.3} & \textbf{52.4} & \textbf{93.4} & \textbf{69.9} \\
\hline
\end{tabular}
\label{tab:ablation}
\vspace{-0.3cm}
\end{table}
\smallskip
\noindent{\bf Performance without Fine-tune.}
This scenario represents that the methods are pre-trained on synthetic images and directly tested on other datasets, aiming to exhibit the generalizability of detection. Table~\ref{tab:pretrained-ours} shows the performance of the different base networks before and after using our method HRGR. These methods are pre-trained on the synthetic dataset from \cite{PSCC} and directly tested on all datasets.
For a fair comparison, we use the same configuration for each experimental group. It can be seen that the performance of base networks generally improves after using our method.
\begin{table}[!t]
\centering
\small
\caption{\small Performance without fine-tune.}
\vspace{-0.3cm}
\tabcolsep=0.4em
\resizebox{\linewidth}{!}{
\begin{tabular}{c|cccccccccc|cc}
\hline
\multirow{2}{*}{Base network} & \multicolumn{2}{c}{CASIA} & \multicolumn{2}{c}{Coverage} & \multicolumn{2}{c}{NIST16} &
\multicolumn{2}{c}{Columbia} & \multicolumn{2}{c|}{IMD20} & \multicolumn{2}{c}{avg} \\
~ & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 \\
\hline
ResNet-50 & 82.0 & 41.2 & 78.3 & 42.5 & 80.6 & 28.1 & \textbf{92.8} & \textbf{80.3} & 78.6 & 26.3 & 82.4 & 43.6 \\
+ HRGR & \textbf{82.3} & \textbf{42.6} & \textbf{78.9 }& \textbf{43.0} & \textbf{80.7} & \textbf{29.0} & 92.1 & 80.1 & \textbf{79.2} & \textbf{27.1} & \textbf{82.6} & \textbf{44.3} \\
\hline
HRNet & 82.7 & 44.9 & 80.3 & 45.6 & 81.7 & 31.5 & 95.1 & 87.2 & 79.6 & 28.4 & 83.8 & 47.5 \\
+ HRGR & \textbf{84.0} & \textbf{46.3} & \textbf{81.0} & \textbf{46.0} & \textbf{82.3} & \textbf{34.6} & \textbf{95.7} & \textbf{88.2} & \textbf{80.4} & \textbf{29.9} & \textbf{84.6} & \textbf{49.0} \\
\hline
EffNet-b4 & 82.1 & 43.3 & 74.3 & 35.6 & \textbf{83.2} & 30.1 & 85.8 & 68.5 & \textbf{81.3} & 31.5 & 81.3 & 41.8 \\
+ HRGR & \textbf{83.5} & \textbf{44.9} & \textbf{75.8} & \textbf{37.8} & 81.9 & \textbf{32.2} & \textbf{86.4} & \textbf{68.7} & 80.9 & \textbf{31.7} & \textbf{81.7} & \textbf{43.0} \\
\hline
Intern-tiny & 84.1 & 48.3 & 80.3 & 49.3 & \textbf{86.6} & 39.3 & 94.7 & 88.8 & 79.5 & 29.5 & 85.0 & 51.0 \\
+ HRGR & \textbf{86.3} & \textbf{52.4} & \textbf{83.9} & \textbf{54.7} & 86.2 & \textbf{39.4} & \textbf{96.4} & \textbf{91.2} & \textbf{79.8} & \textbf{30.4} & \textbf{86.5} & \textbf{53.6} \\
\hline
\end{tabular}}
\label{tab:pretrained-ours}
\vspace{-0.3cm}
\end{table}
% \begin{table}[!th]
% \centering
% \small
% \caption{\small Performance without Fine-tune.}
% \vspace{-0.3cm}
% \tabcolsep=0.4em
% \resizebox{0.9\linewidth}{!}{
% \begin{tabular}{c|cccccccc}
% \hline
% \multirow{2}{*}{Network} & \multicolumn{2}{c}{CASIA} & \multicolumn{2}{c}{Coverage} & \multicolumn{2}{c}{NIST16} &
% \multicolumn{2}{c}{Columbia} \\
% ~ & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 \\
% \hline
% ResNet-50 & 82.0 & 41.2 & 78.3 & 42.5 & 80.6 & 28.1 & \textbf{92.8} & \textbf{80.3} \\
% + HRGR & \textbf{82.3} & \textbf{42.6} & \textbf{78.9 }& \textbf{43.0} & \textbf{80.7} & \textbf{29.0} & 92.1 & 80.1 \\
% \hline
% HRNet & 82.7 & 44.9 & 80.3 & 45.6 & 81.7 & 31.5 & 95.1 & 87.2 \\
% + HRGR & \textbf{84.0} & \textbf{46.3} & \textbf{81.0} & \textbf{46.0} & \textbf{82.3} & \textbf{34.6} & \textbf{95.7} & \textbf{88.2} \\
% \hline
% EffNet-b4 & 82.1 & 43.3 & 74.3 & 35.6 & \textbf{83.2} & 30.1 & 85.8 & 68.5 \\
% + HRGR & \textbf{83.5} & \textbf{44.9} & \textbf{75.8} & \textbf{37.8} & 81.9 & \textbf{32.2} & \textbf{86.4} & \textbf{68.7} \\
% \hline
% Intern-tiny & 84.1 & 48.3 & 80.3 & 49.3 & \textbf{86.6} & 39.3 & 94.7 & 88.8 \\
% + HRGR & \textbf{86.3} & \textbf{52.4} & \textbf{83.9} & \textbf{54.7} & 86.2 & \textbf{39.4} & \textbf{96.4} & \textbf{91.2} \\
% \hline
% \end{tabular}
% }
% \label{tab:pretrained-ours}
% \vspace{-0.3cm}
% \end{table}
% \smallskip
% \noindent{\bf More Adaptations.}
% To be more comprehensive, we conduct two more experiments in
% Table~\ref{tab:pscc-based}. The first part adapts the strategy of PSCC-Net into our
% backbone (Intern-tiny). The second adds HRGR to HRNet-based
% PSCC-Net. Both experiments use the same training configuration
% as in our method, suggesting that HRGR can be applied to
% different backbones and other IMD methods.
% % \blue{We first HRGR and PSCC structures are used on the same backbone network respectively. Second, we add HRGR module to the original PSCC-Net. Both experiments use the same training configuration
% % as in main experiment. In Tab.~\ref{tab:pscc-based} we see that there is a significant performance improvement in HRGR in both cases respectively. This again validates the plug-and-play nature of our approach.}
% \begin{table}[!t]
% \centering
% % \small
% % \vspace{-0.3cm}
% % \tabcolsep=0.4em
% \caption{More comparison with PSCC-Net on CASIA dataset.}
% \vspace{-0.3cm}
% \resizebox{\linewidth}{!}{
% \begin{tabular}{c|cccccc|cc}
% \hline
% \multirow{2}{*}{Method} & \multicolumn{2}{c}{CASIA} & \multicolumn{2}{c}{Coverage} & \multicolumn{2}{c|}{NIST16} & \multicolumn{2}{c}{avg} \\
% ~ & AUC & F1 & AUC & F1 & AUC & F1 & AUC & F1 \\
% \hline
% Intern w/ PSCC & 89.4 & 60.0 & 92.2 & 76.8 & 98.7 & 77.1 & 93.4 & 71.3 \\
% Intern w/ HRGR & \textbf{93.4} & \textbf{69.9} & \textbf{94.8} & \textbf{82.1} & \textbf{99.3} & \textbf{84.6} & \textbf{95.8} & \textbf{78.9} \\
% \hline
% PSCC w/o HRGR & 89.4 & 59.6 & \textbf{93.1} & 70.0 & 95.8 & 60.7 & 92.7 & 63.4 \\
% PSCC w/ HRGR & \textbf{89.9} & \textbf{61.3} & 92.0 & \textbf{70.4} & \textbf{96.6} & \textbf{64.2} & \textbf{92.8} & \textbf{65.3} \\
% \hline
% \end{tabular}}
% \label{tab:pscc-based}
% % \vspace{-0.4cm}
% \end{table}
% \subsection{Further Analysis}
% \smallskip
% \noindent{\bf Visualization of Feature Regions.}
% % The visualization of feature regions on three layers from shallow to deep is shown in Fig.~\ref{fig:regions}. Specifically, Layer 1, Layer 2, and Layer 3 represent the \blue{??} layer, \blue{??} layer, and \blue{??} layer of our base network. It can be observed that the feature regions in different layers are dependent on image content. Particularly, the feature regions in Layer 3 have notable deformation to preserve the content coherence, thereby outlining the boundary of manipulated regions.
% {We use the third layer of our model for demonstration. As shown in Fig.~\ref{fig:regions}, our method generates non-overlapping and content-coherence feature regions, effectively delineating the boundary between manipulated and authentic regions, as highlighted in ``GT matched feature regions''. In contrast, the feature regions within the authentic area (highlighted in gray) mostly appear as regular rectangles, indicating that our model has not overfitted to {semantic features} but instead concentrates on extracting manipulation traces.
% }
% \begin{figure}[!t]
% \centering
% \includegraphics[width=1\linewidth]{figures/regions.pdf}
% \vspace{-0.9cm}
% \caption{\small Visualizations of feature partition. See text for details.}
% \label{fig:regions}
% \vspace{-0.3cm}
% \end{figure}
% \smallskip
% \noindent{\bf Freezing Parameters of DFP in Fine-tuning.}
% This part studies the effect of freezing the pre-trained parameters in the feature partition during the fine-tuned scenario. The motivation of this part is to verify whether fine-tuning the partition knowledge acquired in the pre-trained phase is necessary. As shown in Tab.~\ref{tab:freeze}, this freezing operation only achieves better performance on the ResNet-50 but drops the performance on other backbones, representing learnable parameters in feature partition is more applicable for capturing coherent feature regions.
% \begin{table}[!t]
% \centering
% \small
% % \tabcolsep=1em
% \caption{\small Effect of freezing parameters of DFP in fine-tuning on the CASIA dataset.}
% \vspace{-0.3cm}
% \begin{tabular}{ccccc}
% \hline
% \multirow{2}{*}{Backbones} & \multicolumn{2}{c}{w/o freeze} & \multicolumn{2}{c}{w/ freeze} \\
% ~ & AUC & F1 & AUC & F1 \\
% \hline
% Res-50 & 87.2 & 54.2 & \textbf{88.5} & \textbf{56.2} \\
% HRNet & \textbf{91.4} & \textbf{64.2} & 89.9 & 61.9 \\
% EffNet-b4 & \textbf{92.9} & \textbf{66.7} & 92.7 & 66.2 \\
% Intern-tiny & \textbf{70.3} &\textbf{93.5} & 69.9 & 93.4 \\
% \hline
% \end{tabular}
% \label{tab:freeze}
% \vspace{-0.4cm}
% \end{table}
% \smallskip
% \noindent{\bf {Parameter and Computational Complexity}.}
% \blue{Our method only needs a few additional parameters (1.45M) and FLOPS (1.49G). The main cost of our method is in the linear layer, and almost negligible additional costs are incurred by the \textit{pair-wise distance} in the Differentiable Feature Partition and the \textit{sliding window} in the Hierarchical Graph Modeling, since all these algorithms we used have a linear complexity. }
\smallskip
\noindent{\bf Robustness.}
To evaluate the robustness of our method, we follow the perturbation settings in \cite{PSCC}, which degrades manipulated images obtained from CASIA under different preprocesses, including resizing images with different scales, Gaussian Blurring with kernel size, adding Gaussian Noise with standard deviation ``Sigma'', and performing JPEG Compression with quality factor. The evaluation results in comparison to PSCC-Net and HiFi-Net are shown in Fig.~\ref{fig:robust}, which reveals that our method can resist certain perturbations.
% More analysis on other datasets can be found in \textit{Supplementary}.
\begin{figure}[!t]
\centering
\includegraphics[width=0.9\linewidth]{figures/robust_casia.pdf}
\vspace{-0.3cm}
\caption{\small Robustness analysis under various perturbations on CASIA dataset.}
\label{fig:robust}
\vspace{-0.4cm}
\end{figure}
% \smallskip
% \noindent{\bf {Limitations}.}
% Since our method is performed on the encoders, its effectiveness depends on the capacities of these encoders. Our experiments have demonstrated that when using mainstream base networks, our method can further enhance the detection performance. Nevertheless, its effectiveness may be limited when applied to base networks with restricted capacities.
% \blue{Our proposed model may lead to a decrease in performance for light base networks.
% % Specifically, our proposed modules generally adhere to the principle that the stronger the base networks, the greater the performance improvement.
% Specifically, our model essentially clusters features, and better base networks are more adept at extracting high-quality features, resulting in better clustering effects and ultimately better performance. For light base networks, we believe they may struggle to extract the semantic features illustrated in Fig.~\ref{fig:regions}, leading to poor clustering effects.}
% \vspace{-0.3cm}
\section{Conclusion}
This paper introduces a new method, Hierarchical Region-aware Graph Reasoning (HRGR), to enhance image manipulation detection. Specifically, we describe a differentiable feature partition strategy to formulate feature regions and construct a hierarchical region-aware graph on these regions. Then we introduce a structure-agnostic graph reasoning to improve the feature effectiveness. These processes are iteratively updated during training, leading to mutual boosting. Extensive experiments demonstrate the effectiveness of our method and its potential as a plug-and-play component for classical models.
%%%%%%%%% REFERENCES
{\small
\bibliographystyle{IEEEbib}
\bibliography{icme2025references}
}
\end{document}
\begin{thebibliography}{10}
\bibitem{recent}
Rahul Thakur and Rajesh Rohilla,
\newblock ``Recent advances in digital image manipulation detection techniques: A brief review,''
\newblock {\em Forensic science international}, 2020.
\bibitem{ERMPC}
Dong Li, Jiaying Zhu, Menglu Wang, Jiawei Liu, Xueyang Fu, and Zheng-Jun Zha,
\newblock ``Edge-aware regional message passing controller for image forgery localization,''
\newblock in {\em CVPR}, 2023.
\bibitem{PSCC}
Xiaohong Liu, Yaojie Liu, Jun Chen, and Xiaoming Liu,
\newblock ``Pscc-net: Progressive spatio-channel correlation network for image manipulation detection and localization.,''
\newblock {\em TCSTV}, 2022.
\bibitem{ObjectFormer}
Junke Wang, Zuxuan Wu, Jingjing Chen, Xintong Han, Abhinav Shrivastava, Ser-Nam Lim, and Yu-Gang Jiang,
\newblock ``Objectformer for image manipulation detection and localization,''
\newblock in {\em CVPR}, 2022.
\bibitem{CAT-Net}
Myung-Joon Kwon, Seung-Hun Nam, In-Jae Yu, Heung-Kyu Lee, and Changick Kim,
\newblock ``Learning jpeg compression artifacts for image manipulation detection and localization,''
\newblock {\em IJCV}, 2022.
\bibitem{ACBG}
Wenxi Liu, Hao Zhang, Xinyang Lin, Qing Zhang, Qi~Li, Xiaoxiang Liu, and Ying Cao,
\newblock ``Attentive and contrastive image manipulation localization with boundary guidance,''
\newblock {\em TIFS}, 2024.
\bibitem{TruFor}
Fabrizio Guillaro, Davide Cozzolino, Avneesh Sud, Nicholas Dufour, and Luisa Verdoliva,
\newblock ``Trufor: Leveraging all-round clues for trustworthy image forgery detection and localization,''
\newblock in {\em CVPR}, 2023.
\bibitem{HGCN}
Wenyan Pan, Zhili Zhou, Miaogen Ling, Xin Geng, and QM~Wu,
\newblock ``Learning hierarchical graph representation for image manipulation detection,''
\newblock {\em arXiv preprint arXiv:2201.05730}, 2022.
\bibitem{SpixelFCN}
Fengting Yang, Qian Sun, Hailin Jin, and Zihan Zhou,
\newblock ``Superpixel segmentation with fully convolutional networks,''
\newblock in {\em CVPR}, 2020.
\bibitem{SSN}
Varun Jampani, Deqing Sun, Ming-Yu Liu, Ming-Hsuan Yang, and Jan Kautz,
\newblock ``Superpixel sampling networks,''
\newblock in {\em ECCV}, 2018.
\bibitem{GraphSAGE}
Will Hamilton, Zhitao Ying, and Jure Leskovec,
\newblock ``Inductive representation learning on large graphs,''
\newblock {\em Advances in neural information processing systems}, 2017.
\bibitem{ViG}
Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu,
\newblock ``Vision gnn: An image is worth graph of nodes,''
\newblock {\em Advances in Neural Information Processing Systems}, 2022.
\bibitem{focal}
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll{\'a}r,
\newblock ``Focal loss for dense object detection,''
\newblock in {\em ICCV}, 2017.
\bibitem{casia}
Jing Dong, Wei Wang, and Tieniu Tan,
\newblock ``Casia image tampering detection evaluation database,''
\newblock in {\em ChinaSIP}, 2013.
\bibitem{coverage}
Bihan Wen, Ye~Zhu, Ramanathan Subramanian, Tian-Tsong Ng, Xuanjing Shen, and Stefan Winkler,
\newblock ``Coverage — a novel database for copy-move forgery detection,''
\newblock in {\em ICIP}, 2016.
\bibitem{nist16}
HY~Guan, YY~Lee, A~Yates, A~Delgado, D~Zhou, D~Joy, and A~Pereira,
\newblock ``Nist nimble 2016 datasets,'' 2016.
\bibitem{columbia}
Tian-Tsong Ng, Jessie Hsu, and Shih-Fu Chang,
\newblock ``Columbia image splicing detection evaluation dataset,''
\newblock {\em DVMM lab. Columbia Univ CalPhotos Digit Libr}, 2009.
\bibitem{imd20}
Adam Novozamsky, Babak Mahdian, and Stanislav Saic,
\newblock ``Imd2020: A large-scale annotated dataset tailored for detecting manipulated images,''
\newblock in {\em WACVW}, 2020.
\bibitem{HiFi}
Xiao Guo, Xiaohong Liu, Zhiyuan Ren, Steven Grosz, Iacopo Masi, and Xiaoming Liu,
\newblock ``Hierarchical fine-grained image forgery detection and localization,''
\newblock in {\em CVPR}, 2023.
\bibitem{UPerNet}
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun,
\newblock ``Unified perceptual parsing for scene understanding,''
\newblock in {\em ECCV}, 2018.
\bibitem{EVP}
Weihuang Liu, Xi~Shen, Chi-Man Pun, and Xiaodong Cun,
\newblock ``Explicit visual prompting for low-level structure segmentations,''
\newblock in {\em CVPR}, 2023.
\bibitem{PCL}
Yuyuan Zeng, Bowen Zhao, Shanzhao Qiu, Tao Dai, and Shu-Tao Xia,
\newblock ``Towards effective image manipulation detection with proposal contrastive learning,''
\newblock {\em TCSVT}, 2023.
\bibitem{NCL}
Jizhe Zhou, Xiaochen Ma, Xia Du, Ahmed~Y Alhammadi, and Wentao Feng,
\newblock ``Pre-training-free image manipulation localization through non-mutually exclusive contrastive learning,''
\newblock in {\em ICCV}, 2023.
\bibitem{MVSS}
Chengbo Dong, Xinru Chen, Ruohan Hu, Juan Cao, and Xirong Li,
\newblock ``Mvss-net: Multi-view multi-scale supervised networks for image manipulation detection,''
\newblock {\em TPAMI}, 2022.
\bibitem{resnet}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,
\newblock ``Deep residual learning for image recognition,''
\newblock in {\em CVPR}, 2016.
\bibitem{hrnet}
Jingdong Wang, Ke~Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et~al.,
\newblock ``Deep high-resolution representation learning for visual recognition,''
\newblock {\em TPAMI}, 2020.
\bibitem{efficient}
Mingxing Tan and Quoc Le,
\newblock ``Efficientnet: Rethinking model scaling for convolutional neural networks,''
\newblock in {\em ICML}, 2019.
\bibitem{intern}
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, and Hongsheng Li,
\newblock ``Internimage: Exploring large-scale vision foundation models with deformable convolutions,''
\newblock in {\em CVPR}, 2023.
\bibitem{benco}
Xiaochen Ma, Xuekang Zhu, Lei Su, Bo~Du, Zhuohang Jiang, Bingkui Tong, Zeyu Lei, Xinyu Yang, Chi-Man Pun, Jiancheng Lv, and Jizhe Zhou,
\newblock ``Imdl-benco: A comprehensive benchmark and codebase for image manipulation detection and localization,''
\newblock in {\em NeurIPS}, 2024.
\end{thebibliography}