Binarized Mamba-Transformer for Lightweight Quad Bayer HybridEVS Demosaicing
Extracted source text · arXiv 2503.16134
% CVPR 2025 Paper Template; see https://github.com/cvpr-org/author-kit
\documentclass[10pt,twocolumn,letterpaper]{article}
%%%%%%%%% PAPER TYPE - PLEASE UPDATE FOR FINAL VERSION
\usepackage{cvpr} % To produce the CAMERA-READY version
\usepackage{algorithmic}
\usepackage{xcolor}
% %
% \usepackage{newfloat}
% \usepackage{listings}
\usepackage{booktabs} %插入三线表格用的宏包
\usepackage{diagbox} %插入斜线表头的表格用的宏包
\usepackage{multirow} %插入多行表格用的宏包
% \usepackage{graphicx} %插入图片的宏包
\usepackage{makecell}
\usepackage{subcaption}
\usepackage{amsmath} % 引入数学宏包
% \usepackage{algorithm}
% \usepackage[cache=false]{minted}
% \usepackage{array}
\usepackage{tabularx}
\usepackage{rotating}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{url} % simple URL typesetting
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{ragged2e} % for \RaggedRight macro
\usepackage{graphicx}
\graphicspath{{figures/}}
\usepackage{wrapfig}
\usepackage{colortbl}
% Import additional packages in the preamble file, before hyperref
\input{preamble}
% It is strongly recommended to use hyperref, especially for the review version.
% hyperref with option pagebackref eases the reviewers' job.
% Please disable hyperref *only* if you encounter grave issues,
% e.g. with the file validation for the camera-ready version.
%
% If you comment hyperref and then uncomment it, you should delete *.aux before re-running LaTeX.
% (Or just hit 'q' on the first LaTeX run, let it finish, and you should be clear).
\definecolor{cvprblue}{rgb}{0.21,0.49,0.74}
\usepackage[pagebackref,breaklinks,colorlinks,allcolors=cvprblue]{hyperref}
%%%%%%%%% PAPER ID - PLEASE UPDATE
\def\paperID{12314} % *** Enter the Paper ID here
\def\confName{CVPR}
\def\confYear{2025}
%%%%%%%%% TITLE - PLEASE UPDATE
\title{Binarized Mamba-Transformer for Lightweight Quad Bayer HybridEVS Demosaicing}
%%%%%%%%% AUTHORS - PLEASE UPDATE
\author{
Shiyang Zhou$^{1}$\thanks{Equal contribution} \hspace{0.2cm} Haijin Zeng$^{2}$\footnotemark[1]\hspace{0.2cm} Yunfan Lu$^{3}$ \hspace{0.2cm} Tong Shao$^{1}$ \\
Ke Tang$^{1}$ \hspace{0.2cm} Yongyong Chen$^{1}$\footnotemark[2] \hspace{0.2cm} Jie Liu$^{1}$ \hspace{0.2cm} Jingyong Su$^{1}$\thanks{Corresponding author} \\
$^{1}$Harbin Institute of Technology, Shenzhen \hspace{0.2cm} $^{2}$Harvard University \\ $^{3}$Hong Kong University of Science and Technology, Guangzhou \\
}
\begin{document}
\maketitle
\input{sec/0_abstract}
\input{sec/1_intro}
\input{sec/2_relatedwork}
\input{sec/3_methods}
\input{sec/4_experiments}
\input{sec/5_conclusion}
\input{sec/6_acknowledgement}
{
\small
\bibliographystyle{ieeenat_fullname}
\bibliography{main}
}
% WARNING: do not forget to delete the supplementary pages from your submission
% \input{sec/X_suppl}
\end{document}
%
% --- inline annotations
%
% \newcommand{\red}[1]{{\color{red}#1}}
% \newcommand{\todo}[1]{{\color{red}#1}}
% \newcommand{\TODO}[1]{\textbf{\color{red}[TODO: #1]}}
% \definecolor{brinkpink}{rgb}{0.98, 0.38, 0.5}
% \usepackage{xcolor}
% --- disable by uncommenting
% \renewcommand{\TODO}[1]{}
% \renewcommand{\todo}[1]{#1}
\begin{abstract}
Quad Bayer demosaicing is the central challenge for enabling the widespread application of Hybrid Event-based Vision Sensors (HybridEVS). Although existing learning-based methods that leverage long-range dependency modeling have achieved promising results, their complexity severely limits deployment on mobile devices for real-world applications. To address these limitations, we propose a lightweight Mamba-based binary neural network designed for efficient and high-performing demosaicing of HybridEVS RAW images. First, to effectively capture both global and local dependencies, we introduce a hybrid Binarized Mamba-Transformer architecture that combines the strengths of the Mamba and Swin Transformer architectures. Next, to significantly reduce computational complexity, we propose a binarized Mamba (Bi-Mamba), which binarizes all projections while retaining the core Selective Scan in full precision. Bi-Mamba also incorporates additional global visual information to enhance global context and mitigate precision loss. We conduct quantitative and qualitative experiments to demonstrate the effectiveness of BMTNet in both performance and computational efficiency, providing a lightweight demosaicing solution suited for real-world edge devices. Our codes and models are available at \url{https://github.com/Clausy9/BMTNet}.
\end{abstract}
\section{Introduction}
\label{sec:intro}
By integrating traditional frame-based imaging with event-based detection, the Quad Bayer HybridEVS camera~\cite{kodama20231} has been designed as an advanced type of event camera for next-generation mobile phones. This integration captures detailed spatial information and high-speed temporal changes, offering superior performance in dynamic environments by effectively detecting motion and rapid events. Moreover, recent studies~\cite{yang2022mipi,wu2024mipi} have shown that non-Bayer color filter arrays (CFAs), like Quad Bayer, make great success in low-light scenes on mobile devices with limited sensors. These features make the Quad Bayer HybridEVS camera a cutting-edge device. However, despite its significant advantages in dynamic scenarios and low light conditions, it faces challenges in image demosaicing due to its limited sensor size, the complexity of the Quad Bayer CFA, and color loss caused by event pixels, as shown in Figure \ref{figure1}.
Recently, with the development of mobile imaging, artificial intelligence image signal processors (AI-ISPs) have become crucial for enhancing imaging quality on mobile devices. Specifically, for the critically demosaicing process, deep learning methods provide a strong fitting ability to reconstruct full RGB images from degraded mosaic images, significantly improving restoration result
\begin{figure}
% \vspace{-8mm}
\begin{center}
\begin{subfigure}{0.48\textwidth}
\includegraphics[width=\linewidth]{figures/chart1.pdf}
\vspace{-4mm}
\label{fig: comparison}
\end{subfigure}
\begin{subfigure}{0.44\textwidth}
\begin{subfigure}{\textwidth}
\includegraphics[width=\linewidth]{figures/pattern.pdf}
\label{fig: event}
\end{subfigure}
\end{subfigure}
\hfill
\end{center}
\vspace{-10mm}
\caption{\small Up-left: PSNR and parameters comparisons of our BMTNet and other BNNs on MIPI dataset. Up-right: PSNR and Parameters comparisons of our BMTNet and other FP methods on MIPI dataset. Down: CFA comparisons between Bayer, Quad Bayer, and Quad Bayer HybridEVS. Event pixel appears as a mixed color.}
\vspace{-6mm}
\label{figure1}
% \vspace{-2mm}
\end{figure}
and overcoming artifacts and aliasing issues on Quad Bayer CFA sensors~\cite{a2021beyond,zheng2024quad}. Although existing methods have demonstrated the effectiveness of deep learning in demosaicing, their deployment on edge devices remains challenging due to the high computational cost. Moreover, recent studies have shown that global information, such as long-range dependencies and global context~\cite{wu2024seesr,zhang2024recognize} plays a key role in enhancing image restoration tasks.
These tasks are typically based on Transformer architectures, which can effectively capture global dependencies. However, these methods suffer from high complexity and insufficient emphasis on global visual priors, making it challenging for demosaicing approaches to achieve high performance with limited resources.
To address this issue, various model compression methods have been proposed, such as 8-bit and 4-bit quantization~\cite{banner2019post, banner2018scalable}. Among these, binary neural networks (BNNs) represent the most extreme approach, compressing the model to only positive and negative values, which marks the upper limit of model compression~\cite{hubara2016binarized, rastegari2016xnor, cai2024binarized}.
These highly compressed methods demonstrate significant potential for deploying deep learning models onto resource-constrained devices. Prior works have shown BNN's capability in both natural language processing and vision tasks, including large language models~\cite{huang2024billm,yuanpb}, image classification~\cite{hubara2016binarized,rastegari2016xnor,qin2020forward}, and image restoration~\cite{zhang2024binarized,cai2024binarized}, etc. However, the application of binary networks for the demosaicing task remains an unexplored area for further research.
To fully leverage global information with fewer resources, state space models (SSMs)~\cite{gu2021efficiently,smith2022simplified} have emerged as a fundamental architecture, competing with conventional structures like convolutional neural networks (CNNs) and Transformers. Advanced SSMs like Mamba~\cite{gu2023Mamba} model have demonstrated significant capability in capturing long-range dependencies with lower computational costs than self-attention mechanisms. Nevertheless, deploying Mamba on resource-limited devices is still constrained by complex projection layers. BNN offers a promising optimization to reduce complexity in less critical layers of Mamba, yet most research on binarization mainly focused on CNNs and Transformers, leaving Mamba binarization relatively blank.
%and give a lightweight solution for the Quad Bayer HybridEVS camera
In this paper, we introduce a binarized Mamba-Transformer architecture called BMTNet to tackle the aforementioned issues, as shown in Figure \ref{figure2}. The basic unit called the Binarized Mamba-Transformer (BMT) block combines Mamba and Transformer in a binary format with linear complexity, which is pioneering the binarization of Mamba by binarizing all projections, which effectively reduces model complexity while maintaining core functions in full precision. The advanced combination effectively leverages both long and short dependencies while significantly decreasing parameters and computation costs by 97\% and 96\% on Mamba blocks.
To reduce precision loss and enhance global feature extraction, we incorporate an additional binarized global visual encoder specifically designed for Quad Bayer RAW images to capture global visual information. For the visual representation, we use an embedding mechanism that combines image features with global information to generate a control matrix for the input in the Selective Scan process of our proposed Binarized Mamba (Bi-Mamba). This property can effectively enhance the perceptual ability of BMTNet by improving the control matrix of input. Our proposed BMTNet outperforms other BNN methods and achieves results comparable to full precision methods, as shown in Figure \ref{figure1},
% Furthermore, An efficient distillation method with perceptual loss is designed to align distribution through the bottleneck of UNets. The alignment is performed in periodic epochs to avoid model degradation caused by traditional knowledge distillation while effectively reducing training time and resource consumption. BMTNet achieves an improvement of over 0.4dB on PNSR compared to previous state-of-the-art BNN methods over eight diverse datasets, delivering superior visual results with more vivid colors and finer details on both synthesized and real data.
In summary, the contributions of our work are that
\begin{itemize}
\item We introduce a novel BNN-based hybrid BMTNet for Quad Bayer HybridEVS demosaicing. To the best of our knowledge, this is the \textbf{first} research to explore binary Mamba and binarized Quad Bayer demosaicing.
\item We propose a hybrid binary Mamba-Transformer design that benefits from its dual-branch form and linear complexity in a binary format, efficiently capturing global and local dependencies with minimal computation.
\item An advanced binary Mamba that binarizes non-critical projections while keeping the core SSM in full precision, enabling meaningful global visual information to be embedded in the attention process and significantly reducing complexity by 79\% on parameters and 88\% on OPs.
% \item We design an efficient distillation strategy for UNet-like demosaicing networks, periodically implementing distribution alignment on the bottleneck. This approach effectively mitigates model degradation while reducing additional training time.
\end{itemize}
\section{Related Work}
\label{sec:relatedwork}
Demosaicing is a crucial step in the imaging process, aiming to reconstruct RGB images from RAW data, and is typically integrated into ISPs. Traditional demosaicing methods are mainly based on interpolation~\cite{hirakawa2005adaptive,li2008image}. In recent years, Convolution Neural Networks (CNNs) have led a great improvement in Bayer demosaicing, exhibiting impressive ability to overcome the color degradation~\cite{tan2017color,tan2017joint,syu2018learning,tan2018deepdemosaicking,liu2020joint}. Despite demosaicing on Bayer images having achieved great success, non-Bayer CFAs, which have more complicated color arrangements present new challenges. With the development of mobile imaging, Quad Bayer has been widely implemented on flagship smartphone cameras~\cite{yang2022mipi} and advanced event cameras~\cite{son2017640}, which demonstrate better imaging results on dark scenes. Still, as an emergent CFA, research on Quad Bayer is limited. Some research~\cite {jia2022learning,zeng2023inheriting} proposed two-stage strategies on remosaicing Quad Bayer to Bayer. Some GAN-based structures~\cite{a2021beyond,sharif2021sagan} have been proposed to achieve refined detail on Quad Bayer demosaicing. However, most of them neglect constrained computation resources on ISPs, making them difficult to deploy on edge devices.
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{figures/figure1.pdf}
\vspace{-6mm}
\caption{Overall architecture of BMTNet. A binary convolution-based simple subnetwork is initially employed for event pixel inpainting. The main branch incorporates our hybrid binary Mamba-Transformer Block, which pioneeringly integrates Bi-Mamba with Bi-Swin Transformer to capture both global and local features. An additional global visual branch is used to enhance global dependencies, with Bi-Mamba specifically handling the fusion of global features.}
\vspace{-4mm}
\label{figure2}
\label{fig: BMTNet}
\end{figure*}
Recently, event cameras have emerged as a bio-inspired vision sensor to capture changes in the scene, showing high dynamic range, high temporal resolution, and low latency~\cite{gallego2020event,son2017640,dong2024joint}. Some methods~\cite{zhang2022spiking,munda2018real} demonstrate advanced applications of event cameras. Pioneering methods~\cite{ma2024color4e} explore demosaicing for neuromorphic event sensors with real data. Nevertheless, research on the recently proposed Quad Bayer Hybrid Event Vision Sensor (HybridEVS) is still limited~\cite{kodama20231}. MIPI workshop~\cite{wu2024mipi} proposed some advanced methods for demosaicing. Some advanced methods~\cite{lu2024event,xu2024demosaicformer,yunfanrgb} are proposed to solve the cutting-edge problem. However, the substantial computational demands make these methods challenging to deploy on resource-constrained ISPs. The need for the exploration of lightweight demosaicing methods remains essential.
State space models (SSMs) have developed to be a powerful competitor to CNNs and Transformers that capture global information with linear complexity. Prior models~\cite{gu2021efficiently,smith2022simplified} introduce efficient parallel scanning to improve the effectiveness and speed of SSMs. The emergence of Mamba~\cite{gu2023Mamba} features a data-independent SSM layer that demonstrated impressive performance with linear complexity. Prior works have utilized Mamba on multiple computer vision tasks~\cite{zhu2024vision,liao2024lightm,wang2023selective,hatamizadeh2024mambavision} as well as image restoration~\cite{guo2024Mambair,xiao2024frequency}. The linear scalability of complexity and strong modeling ability for long-dependencies demonstrate extraordinary potential for visual tasks, with emerging OEM prototypes exploring Mamba's edge deployment.
Binary neural network (BNN) is an extreme compression technology that quantizes the network's weights and activation values on a 1-bit form, which can bring a significant reduction in computation loads. Early works~\cite{courbariaux2015binaryconnect,hubara2016binarized,qin2020forward,liu2018bi} introduced binarized CNNs, utilizing the sign function to binarize weights and activations. To mitigate the loss of precision due to binarization, Rastegari \emph{et al.}~\cite{rastegari2016xnor} adapted scaling factors for weights and activations. For BNNs in image restoration tasks, Xia \emph{et al.}~\cite{xia2022basic} determined some essential components of BNN in image restoration. Some advanced research~\cite{cai2024binarized,zhang2024binarized} employed BNNs for downstream restoration tasks. However, the binarized network for the demosaicing task still needs to be explored. Furthermore, besides binarization of CNNs, binarized Transformers~ are also explored~\cite{qin2022bibert,liu2022bit}, He \emph{et al.}~\cite{he2023bivit} expanded binarization into vision Transformers. \emph{Still, as for the recently proposed Mamba network, no work has yet explored its binarization.}
\section{Methods}
\label{sec:methods}
\begin{figure*}
\centering
\includegraphics[width=0.96\textwidth]{figures/figure2.pdf}
\vspace{-3mm}
\caption{Model details of the bi-visual encoder and Bi-Mamba. (a) We first adopted a pretrained large visual encoder from RAM~\cite{zhang2024recognize} to pretrain our binarized visual encoder fit for Quad Bayer RAW input. (b) During the training of BMTNet, the binarized visual encoder is frozen and produces global visual embeddings to Bi-Mamba after an adapter. (c) In the binarized Mamba, we binarize all projections while keeping the core selective scan calculation in full precision, effectively reducing computational load while maintaining performance. To further enhance the global capacity, we introduce extra global information into the control matrix $\mathbf{B}$ of input.}
\vspace{-4mm}
\label{figure3}
\label{fig: biMamba}
\end{figure*}
In this section, we first formulate the Quad Bayer HybridEVS demosaicing problem and outline the overall architecture of our network. We then provide a detailed explanation of the proposed binarized visual encoder and binarized Mamba-Transformer block.
\subsection{Network Architecture}
Demosaicing for Quad Bayer HybridEVS cameras is a cutting-edge challenge, particularly for real-world event camera applications. Unlike standard demosaicing tasks, the unique design of HybridEVS introduces the Quad Bayer CFA and the event pixel. The Quad Bayer CFA increases color distortion due to larger gaps between identical colors, while the event pixel leads to additional color loss, making demosaicing especially challenging. The task is to reconstruct the RGB image $\mathbf{I_R} \in \mathbb{R}^{H\times W\times 3}$ from the Quad Bayer RAW image $\mathbf{I_Q} \in \mathbb{R}^{H\times W\times 1}$. Additionally, HybridEVS RAW images experience color loss at event position $\mathbf{L} \in \mathbb{R}^{H\times W\times 1}$. The relationship between these factors can be expressed as:
\begin{equation}
\mathbf{I_Q} = \mathcal{M}(\mathbf{I_R})+\mathbf{L},
\end{equation}
where $\mathcal{M}$ indicates Quad Bayer mosaic process.
To address these challenges, we introduce the binarized hybrid Mamba-Transformer network (BMTNet), as shown in Figure \ref{fig: BMTNet}. Our approach starts with a subnetwork $\mathcal{N}_{1}$ based on BBCU~\cite{xia2022basic} to coarsely inpaint color loss. Then, the binarized Mamba-Transformer network $\mathcal{N}_{2}$ solves the demosaicing task. Additionally, we incorporate an encoder branch to leverage additional global visual information from Quad Bayer images. The entire process can be formulated as:
\begin{equation}
\mathbf{I_R} = \mathcal{N}_{2}(\mathcal{N}_{1}(\mathbf{I_Q})).
\end{equation}
Next, we present the binarized visual encoder and details of the hybrid binarized Mamba-Transformer block.
\subsection{Binarized Visual Encoder} \label{SE}
Recent research shows that global visual information supplies extra global information, improving the accuracy of image restoration, especially in finer details~\cite{wu2024seesr}. To strengthen Mamba’s global capacity, we incorporated a global visual encoder tailored for Quad Bayer RAW images in a binary format, as shown in Figure \ref{fig: BMTNet}, which offers implicit visual encoding with little computational load. This encoder starts with a pretraining phase (see Figure \ref{fig: biMamba} (a)), where a frozen large visual encoder (from RAM~\cite{zhang2024recognize}) serves as a teacher to train a compact visual encoder. This enables a direct fusion of global visual information into the main branch in vector form.
During main branch training and inference, the binarized visual encoder remains frozen to preserve its capacity for global visual extraction (see Figure \ref{fig: biMamba} (b)). The visual embeddings are then fed into each layer's Bi-Mamba module through a single-layer Bi-Linear adapter, as shown in Figure \ref{fig: BMTNet}. This adapter is essential for adapting the visual embeddings across layers with different levels of information. Overall, the additional visual encoder ensures global consistency throughout the encoding and decoding process, providing a distinct prior and preserving global structural information throughout these stages.
\subsection{Binarized Mamba-Transformer}\label{bmtb}
Mamba is an efficient mechanism for capturing long-range dependencies with linear complexity, but the numerous projection layers still make it challenging to deploy on resource-constrained mobile devices. The Swin Transformer has demonstrated strong capabilities in extracting local features but is also limited by high computational demands. To leverage the strengths of both architectures while expanding the applicability of BNN, we introduce a binarized Mamba-Transformer block (BMT block), as shown in Figure \ref{figure2}. The BMT block employs a two-branch design combining the binarized Mamba (Bi-Mamba) and binarized Swin Transformer (Bi-SwinT) to capture global and local dependencies in parallel. Additionally, it benefits from a reduced channel number in each sub-block, lowering the computation load and resulting in a more lightweight model.
Specifically, the Bi-Mamba fully binarizes non-critical projections while using full precise computations for the core selective scan (SS) function, along with an extra global visual embedding on the control matrix of input, the illustration of Bi-Mamba is shown in Figure \ref{figure3} (c). This approach minimizes model complexity while maintaining high performance with precise core computations. The quantization in Mamba mainly focuses on linear and convolution projections. In our approach, full precision linear weights $\mathbf{W}^f \in \mathbb{R}^{C_{in}\times C_{out}}$ are binarized using the Sign function and activation $\mathbf{A}^f \in \mathbb{R}^{H\times W\times C_{in}}$ are binarized using RSign function~\cite{liu2020reactnet}, resulting in values of \{+1, -1\}, which can be expressed as:
\vspace{-1mm}
\begin{equation}
\mathbf{W}^{b}=\operatorname{Sign}\left(\mathbf{W}^f\right)=\left\{\begin{array}{ll}
+1, & \mathbf{W}^f>0 \\
-1, & \mathbf{W}^f \leq 0
\end{array}.\right.
\end{equation}
\vspace{-1mm}
\begin{equation}
\mathbf{A}^{b}=\operatorname{RSign}\left(\mathbf{A}^f\right)=\left\{\begin{array}{ll}
+1, & \mathbf{A}^f>\alpha \\
-1, & \mathbf{A}^f \leq \alpha
\end{array},\right.
\end{equation}
where $\alpha \in \mathbb{R}^{C_{in}}$ represents learnable parameters. To mitigate precision loss due to binarization, we apply additional learnable scaling factors $\mathbf{S} \in \mathbb{R}^{C_{out}}$ ~\cite{he2023bivit}. The whole Bi-Linear process can be expressed as:
% \vspace{-1mm}
\begin{equation}
\begin{aligned}
\operatorname{Bi-Linear}(A^f) &= \mathbf{W}^b \ast \mathbf{A}^b \ast \mathbf{S} \\
&= \operatorname{bitcount}(\mathbf{XNOR}(\mathbf{W}^b, \mathbf{A}^b)) \ast \mathbf{S}.
\end{aligned}
\end{equation}
The Binarized convolution ($\operatorname{BConv2d}$) applies similar binarization on weights and activations but sets $\mathbf{S}$ as the average of $\mathbf{W}^f$. Specific to Bi-Mamba, given a full-precision input $\mathbf{X}^f \in \mathbb{R}^{H\times W\times C_{in}}$, it is first projected into feature $\mathbf{X'} \in \mathbb{R}^{H\times W\times d}$, where $d$ is the hidden dimension, which can be expressed as:
\begin{equation}
\mathbf{X'} =\operatorname{SiLU}(\operatorname{BConv2d}(\operatorname{Bi-Linear}(\mathbf{X}^f))).
\end{equation}
Features are then transformed into a one-dimensional format using multiple scan orders~\cite{liu2024vMambavisualstatespace}, which can be expressed as:
\begin{equation}
\begin{split}
\mathbf{X}_1, \mathbf{X}_2, \ldots, \mathbf{X}_n =
\operatorname{S_1}(\mathbf{X'}), \operatorname{S_2}(\mathbf{X'}), \ldots, \operatorname{S_n}(\mathbf{X'}),
\end{split}
\end{equation}
where $\operatorname{S}$ denotes the scan operation and $n$ is the total number of scan types. For each $\mathbf{X}_i \in \mathbb{R}^{L\times d}$, $i\in \{1, 2, \ldots, n\}$, $L=H*W$, we extract SSM parameters $\mathbf{B}_i \in \mathbb{R}^{L\times m}$, $\mathbf{C}_i \in \mathbb{R}^{L\times m}$ and $\mathbf{\Delta}_i \in \mathbb{R}^{L\times d}$. Unlike previous Mamba methods that derive all parameters solely from the input $\mathbf{X}_i$, we enhance the control matrix by concatenating the additional global visual vector $\mathbf{S}\in \mathbb{R}^{L\times 1}$ from the bi-visual encoder. The projection of SSM parameters is thus represented as:
\begin{equation}
\begin{split}
\mathbf{C}_i &= \operatorname{Bi-Linear}(\mathbf{X}_i),\\
\mathbf{\Delta}_i &= \operatorname{Bi-Linear}(\mathbf{X}_i),\\
\mathbf{B}_i &= \operatorname{Bi-Linear}(\operatorname{Concat}(\mathbf{X}_i, \mathbf{S})).
\end{split}
\end{equation}
With the learnable parameters $\mathbf{A}_i \in \mathbb{R}^{d\times m}$ and $\mathbf{D}_i \in \mathbb{R}^{d}$, the Selective Scan (SS) process in full precise can be expressed as:
\begin{equation}\label{eq:9}
\begin{split}
\overline{\mathbf{B}}_i &= \mathbf{B}_i^\top \otimes \mathbf{\Delta}_i, \\
\overline{\mathbf{A}}_i &= \operatorname{exp}(\mathbf{\Delta}_i \otimes \mathbf{A}_i), \\
% h_{ik} &= \overline{\mathbf{A}}_{ik} h_{i(k-1)} + \overline{\mathbf{B}}_{ik} x_{ik}, \\
% y_{ik} &= \mathbf{C}_{ik} h_{ik} + \mathbf{D}_i x_{ik}, \\
\mathbf{O}_i &= \operatorname{Reshape}(\operatorname{SSM}(\overline{\mathbf{A}}_i, \overline{\mathbf{B}}_i, \mathbf{C}_i, \mathbf{D}_i, \mathbf{X}_i)).
\end{split}
\end{equation}
This SSM~\cite{gu2023Mamba} function computes the most critical long-range attention dependencies. Retaining it in full precision enables our Bi-Mamba to achieve results comparable to full-precision Mamba with minimal computation cost. The global visual embedding in the control matrix $\mathbf{B}$ directly enriches the input $\mathbf{X}$ with extra global visual features, contributing to the recursive SS process in an additive manner. This approach preserves the original formula while effectively enhancing performance without significant disruption. The final output is computed by summing the SS results and performing a Hadamard product with the output from another Bi-Linear branch, which can be expressed as:
\begin{equation}
\mathbf{X}^{out} =
\operatorname{LN}(\sum_{i=1}^{n}\mathbf{O}_i) \ast \operatorname{SiLU}(\operatorname{Bi-Linear}(\mathbf{X}^f)),
\end{equation}
where $\operatorname{LN}$ represents layer normalization. Bi-Mamba reduces parameters by binarizing non-essential projections like linear and convolution while maintaining high performance through full-precision selective scanning and global visual embedding. This approach offers a practical way to enhance Mamba and significantly compress the model into a binary format, making it well-suited for demosaicing on resource-limited edge devices.
To further enhance local representation, we introduce the binarized Swin Transformer (Bi-SwinT) block. The Bi-SwinT block employs a binary format~\cite{he2023bivit} of the Swin Transformer~\cite{liu2021swin}, facilitating information exchange and improving local detail extraction. In summary, the integration of the binarized Mamba-Transformer block pioneeringly simplifies the Mamba architecture while effectively capturing both global and local dependencies with minimal computational loads, which significantly broadens the application of BNNs in vision Transformers and Mamba.
\begin{figure*}[ht]
\centering
% Row 1
\begin{subfigure}[b]{0.38\textwidth}
\centering
\includegraphics[width=\textwidth, trim=200 200 150 25, clip]{figures/visual1/gt.png}
\caption*{\makecell{067, Urban100~\cite{Cordts_2016_CVPR}\\PSNR/SSIM}}
\end{subfigure}%
\hfill
\begin{subfigure}{0.2\textwidth}
\begin{subfigure}[b]{\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/visual1/bbcu.pdf}
\caption*{\makecell{BBCU~\cite{xia2022basic}\\26.75/0.958}}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/visual1/react.pdf}
\caption*{\makecell{ReActNet~\cite{liu2020reactnet}\\26.04/0.961}}
\end{subfigure}
\end{subfigure}
\begin{subfigure}{0.2\textwidth}
\begin{subfigure}[b]{\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/visual1/bivit.pdf}
\caption*{\makecell{BiViT~\cite{he2023bivit}\\27.32/0.961}}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/visual1/bnn.pdf}
\caption*{\makecell{BNN~\cite{hubara2016binarized}\\22.31/0.894}}
\end{subfigure}
\end{subfigure}
\begin{subfigure}{0.2\textwidth}
\begin{subfigure}[b]{\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/visual1/btm.pdf}
\caption*{\makecell{BTM~\cite{jiang2021training}\\21.05/0.929}}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/visual1/bmtnet.pdf}
\caption*{\makecell{\textbf{BMTNet (Ours)}\\ \textbf{29.07/0.972}}}
\end{subfigure}
\end{subfigure}
% \begin{subfigure}[b]{0.38\textwidth}
% \centering
% \includegraphics[width=\textwidth, trim=15 10 10 10, clip]{figures/visual1/big.png}
% \caption*{\makecell{0801, Val Set, MIPI\\PSNR/SSIM}}
% \end{subfigure}%
% \hfill
% \begin{subfigure}{0.2\textwidth}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\textwidth, trim=10 20 10 10, clip]{figures/visual1/big.png}
% \caption*{\makecell{BNN\\35.56/0.9642}}
% \end{subfigure}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\textwidth, trim=10 20 10 10, clip]{figures/visual1/big.png}
% \caption*{\makecell{BNN\\35.56/0.9642}}
% \end{subfigure}
% \end{subfigure}
% \begin{subfigure}{0.2\textwidth}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\textwidth, trim=10 20 10 10, clip]{figures/visual1/big.png}
% \caption*{\makecell{BNN\\35.56/0.9642}}
% \end{subfigure}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\textwidth, trim=10 20 10 10, clip]{figures/visual1/big.png}
% \caption*{\makecell{BNN\\35.56/0.9642}}
% \end{subfigure}
% \end{subfigure}
% \begin{subfigure}{0.2\textwidth}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\textwidth, trim=10 20 10 10, clip]{figures/visual1/big.png}
% \caption*{\makecell{BNN\\35.56/0.9642}}
% \end{subfigure}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\textwidth, trim=10 20 10 10, clip]{figures/visual1/big.png}
% \caption*{\makecell{BNN\\35.56/0.9642}}
% \end{subfigure}
% \end{subfigure}
\vspace{-3mm}
\caption{Visualization on the Urban100 dataset across all compared BNN methods. The proposed BMTNet achieves the best visual quality, effectively reducing artifacts and color aliasing.}
\label{figure4}
\vspace{-2mm}
\end{figure*}
% \vspace{-2mm}
\section{Experiments}
\begin{table*}[ht]
\footnotesize
\centering
\fontsize{9}{11}\selectfont
\setlength{\tabcolsep}{1.4mm}\resizebox{1\linewidth}{!}{
\begin{tabular}{p{2.4cm}|cc|cccccc|c}
\toprule
\fontsize{10}{12}\selectfont{\multirow{2}{*}{Methods}} & Params & \fontsize{10}{12}\selectfont{OPs} & \multirow{2}{*}{MIPI} & \multirow{2}{*}{Kodak} & \multirow{2}{*}{McM} & \multirow{2}{*}{BSD100}& \multirow{2}{*}{Urban100} & \multirow{2}{*}{Wed} & \fontsize{10}{12}\selectfont{\multirow{2}{*}{Average}}
\\
& (M) & (G)&&&&&&&
\\
% \cmidrule(r){1-2}
\midrule
\rowcolor{blue!2}
DFormer~\cite{Xu_2024_CVPR} & 30.28 & 491.1 & \textbf{39.35}/0.981 & \textbf{39.32}/0.982 & \textbf{37.88}/\textbf{0.963} & \textbf{37.65}/0.982 & \textbf{37.64}/0.980 & \textbf{34.86}/0.968 & \textbf{38.15}/0.978 \\
\rowcolor{blue!2}
NAFNet~\cite{chen2022simple} & 29.16 & 32.19 & 37.91/0.980 & 38.60/0.984 & 36.18/0.961 & 37.12/0.985 & 35.63/0.978 & 35.24/\textbf{0.972} & 37.16/0.978 \\
\rowcolor{blue!2}
Restormer~\cite{zamir2022restormer} & 26.11 & 282.2 & 38.46/\textbf{0.984} & 39.16/\textbf{0.986} & 36.54/\textbf{0.967} & 37.11/\textbf{0.985} & 36.36/0.977 & 35.00/0.971 & 37.45/\textbf{0.980} \\
\rowcolor{blue!2}
SAGAN~\cite{sharif2021sagan} & 22.56 & 341.6 & 34.25/0.959 & 36.14/0.974 & 32.58/0.939 & 30.53/0.931 & 29.89/0.946 & 28.22/0.917 & 32.74/0.952 \\
\rowcolor{blue!2}
PIPNet~\cite{a2021beyond} & 3.46 & 68.8 & 33.73/0.950 & 32.20/0.960 & 31.34/0.928 & 31.97/0.950 & 28.92/0.942 & 29.19/0.929 & 31.97/0.951 \\
\rowcolor{blue!2}
CycleISP~\cite{zamir2020cycleisp} & 3.23 & 104.9 & 30.04/0.934 & 33.09/0.970 & 30.37/0.919 & 32.18/0.969 & 29.78/0.942 & 30.22/0.944 & 31.14/0.952 \\
\midrule
\rowcolor{green!5}
BNN~\cite{hubara2016binarized} & 1.42 & 6.45 & 32.53/0.930 & 33.83/0.955 & 28.59/0.889 & 31.43/0.953 & 29.56/0.935 & 29.32/0.926 & 31.07/0.926 \\
\rowcolor{green!5}
ReActNet~\cite{liu2020reactnet} & 1.47 & 6.12 & 36.47/0.971 & 37.25/0.978 & 34.55/0.946 & 35.25/0.978 & 33.86/0.970 & 33.53/0.962 & 35.08/0.970 \\
\rowcolor{green!5}
BBCU~\cite{xia2022basic} & 1.51 & 6.97 & 36.06/0.970 & 37.03/0.978 & 33.74/0.941 & 34.30/0.977 & 33.27/0.967 & 32.63/0.959 & 34.85/0.967 \\
\rowcolor{green!5}
BTM~\cite{jiang2021training} & 1.47 & \textbf{6.12} & 35.98/0.972 & 37.39/0.979 & 32.99/0.947 & 35.41/0.979 & 33.69/0.971 & 32.76/0.962 & 34.59/0.972 \\
\rowcolor{green!5}
BiViT~\cite{he2023bivit} & 1.36 & 6.51 & 34.72/0.963 & 36.55/0.975 & 30.44/0.932 & 33.48/0.974 & 32.79/0.965 & 30.40/0.952 & 33.33/0.963 \\
\rowcolor{gray!20}
\textbf{BMTNet (Ours)} & \textbf{1.28} & 6.56 & \textbf{36.95}/\textbf{0.975} & \textbf{37.69}/\textbf{0.980} & \textbf{34.79}/\textbf{0.950} & \textbf{36.11}/\textbf{0.981} & \textbf{34.45}/\textbf{0.973} & \textbf{33.95}/\textbf{0.965} & \textbf{35.52}/\textbf{0.975} \\
\bottomrule
\end{tabular}}
\vspace{-2mm}
\caption{Quantitative evaluation of our BMTNet compared to full-precision and other BNN methods across six image datasets, using PSNR (dB) / SSIM as evaluation metrics for visual quality. The blue background indicates full-precise methods, while the green background means binary neural networks. BMTNet outperforms other BNNs while achieving results comparable to full-precision models at a minimal computational cost.}
\label{tab:table1}
\vspace{-5mm}
\end{table*}
\label{sec:experiments}
In this section, we first specify the implementation details. Then, we evaluate our BMTNet across eight diverse datasets, including both single images and video sequences, demonstrating a comparison with full precision and BNN methods. Finally, we perform an ablation study to analyze our methods. Additional analysis of extended visual results is provided in the supplementary material.
% To round the numbers with four decimal places to three decimal places, we will look at the fourth decimal place and apply the standard rounding rules. Here's the updated table:
% \begin{figure*}[!h]
% \fontsize{6}{10}\selectfont
% \begin{subfigure}[b]{0.4\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/visual1/big.png}
% \caption*{\makecell{0801, Val Set, MIPI\\PSNR/SSIM}}
% \label{fig: f1ref}
% \end{subfigure}
% \hfill
% \hfill
% \begin{subfigure}[b]{0.17\textwidth}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/visual1/0801_bnn.png}
% \caption*{\makecell{BNN\\35.56/0.9642}}
% \label{fig: bnn1}
% \end{subfigure}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/visual1/0801_react.png}
% \caption*{\makecell{ReActNet\\35.83/0.9673}}
% \label{fig: react1}
% \end{subfigure}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.17\textwidth}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/visual1/0801_btm.png}
% \caption*{\makecell{BTM\\35.76/0.9659}}
% \label{fig: f3nafnet}
% \end{subfigure}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/visual1/0801_bivit.png}
% \caption*{\makecell{BiViT\\36.08/0.9692}}
% \label{fig: f3sagan}
% \end{subfigure}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.17\textwidth}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/visual1/0801_btm.png}
% \caption*{\makecell{BBCU\\35.85/0.9659}}
% \label{fig: f3pipnet}
% \end{subfigure}
% \begin{subfigure}[b]{\textwidth}
% \centering
% \includegraphics[width=\linewidth]{figures/visual1/0801_bmtnet.png}
% \caption*{\makecell{\textbf{BMTNet (Ours)}\\ \textbf{36.44/0.9719}}}
% \label{fig: f1gt}
% \end{subfigure}
% \end{subfigure}
% \label{fig: fig3}
% \vspace{-2mm}
% \caption{Visualization on the MIPI dataset across all compared BNN methods. The proposed BMTNet achieves the best visual quality, effectively reducing real-world sensor noise and artifacts.}
% \vspace{-4mm}
% \label{figure5}
% \end{figure*}
\subsection{Experimental Settings}
\paragraph{Datasets.}
The dataset used in our experiments comprises both simulated and real data. The simulated data includes ground truth, which can be utilized for training and quantitative analysis, while the real data lacks definitive values and is employed for qualitative analysis.
We train all models using the MIPI dataset from the CVPR Mobile Intelligent Photography \& Imaging Workshop 2024. This dataset includes 800 training pairs and 26 test pairs of RAW and RGB images, containing synthesized Gaussian noise and defect pixels~\cite{wu2024mipi}. To broaden the test scenes in different conditions, we simulate the HybridEVS test cases on additional seven datasets, including image datasets: Kodak~\cite{10.1145/1290082.1290117}, McM~\cite{Woo_2018_ECCV}, BSD100~\cite{937655}, Urban100~\cite{Cordts_2016_CVPR}, Wed~\cite{7752930}, and video datasets: REDS~\cite{nah2019ntire}, Vid4~\cite{liu2011bayesian}. These datasets cover a wide range of daily life scenes.
% The real data were captured using a HybridEVS camera.
The \textbf{real-world data} was collected using a hybrid-vision sensor (HVS) developed in collaboration with Alpsentek~\cite{alpsentek2024}, which is based on the ALPIX-Eiger chip, with a resolution of $2448 \times 3246$. Additionally, we collected scenes in both indoor and outdoor environments.
For the demosaicing task, we specifically captured a 2000-line resolution chart to evaluate the resolution performance of different methods.
To focus the evaluation on demosaicing, we employed long exposure to minimize the noise in the RAW images and applied white balance after processing the results to prevent color deviation.
Furthermore, we used MATLAB to perform a basic ISP on the RAW images to obtain a reference image for comparison.
% \resizebox{0.9\linewidth}{!}{
\begin{figure*}[!ht]
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual5/gt.pdf}
\caption{Reference}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual5/bnn.pdf}
\caption{BNN~\cite{hubara2016binarized}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual5/bbcu.pdf}
\caption{BBCU~\cite{xia2022basic}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual5/bivit.pdf}
\caption{BiViT~\cite{he2023bivit}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual5/btm.pdf}
\caption{BTM~\cite{jiang2021training}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual5/react.pdf}
\caption{ReActNet~\cite{liu2020reactnet}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual5/bmtnet.pdf}
\caption{\textbf{BMTNet (Ours)}}
\end{subfigure}
\vspace{-2mm}
\caption{Visualized results across all compared BNN methods on the Kodak (up) and Vid4 (down) datasets, with a corresponding heatmap showing the pixel value differences. The proposed BMTNet exhibits less color aliasing than other BNN methods.}
\label{figure5}
\vspace{-3mm}
\end{figure*}
\vspace{-3mm}
\paragraph{Implementation Details.}
During training, we randomly crop images into 128 $\times$ 128 patches with batch size = 32. The Adam optimizer with $L1$ loss is employed, with a learning rate from $2 \times 10^{-4}$ to $1 \times 10^{-7}$ in a cosine annealing scheme. Total iterations are set to $1 \times 10^{6}$. For BMTNet and other compared BNNs, we apply a pretraining step to utilize the two-stage structure. To preserve performance, upsampling and downsampling operations remain in FP.
\vspace{-3mm}
\paragraph{Computation Load Calculation of BNNs.} Following the prior works on BNN~\cite{hubara2016binarized,liu2018bi,xia2022basic}, binarized operations ($\operatorname{BOPs}^b$) is computed as $\operatorname{BOPs}^b = \operatorname{BOPs}^f/64$, with total operations calculated by $\operatorname{OPs} = \operatorname{BOPs}^b + \operatorname{OPs}^f$, where $\operatorname{OPs}^f$ means floating-point operations. The binarized parameters are calculated as $\operatorname{BParms}^b = \operatorname{BParms}^f/32$, and total parameters are computed as $\operatorname{Parms} = \operatorname{BParms}^b + \operatorname{Parms}^f$, where $\operatorname{Parms}^f$ indicates the number of float-point parameters. All computational load tests are conducted on a 256$\times$256 image.
\begin{table}[ht]
\footnotesize
\centering
\fontsize{9}{11}\selectfont
\setlength{\tabcolsep}{1.2mm}{
\begin{tabular}{p{2.4cm}|cc|c}
\toprule
% \multicolumn{2}{c}{Part} \\
% \cmidrule(r){1-2}
\fontsize{9}{11}Methods &REDS& Vid4& Avearge\\
% \cmidrule(r){1-2}
\midrule
\rowcolor{blue!2}
DFormer~\cite{Xu_2024_CVPR} & 42.45/0.991 & 36.01/0.979 & 39.23/0.985 \\
\rowcolor{blue!2}
NAFNet~\cite{chen2022simple} & 41.65/0.989 & 34.98/0.974 & 38.31/0.982 \\
\rowcolor{blue!2}
Restormer~\cite{zamir2022restormer} & 41.91/0.990 & 35.08/0.981 & 38.50/0.985 \\
\rowcolor{blue!2}
SAGAN~\cite{sharif2021sagan} & 38.13/0.984 & 32.16/0.963 & 35.15/0.974 \\
\rowcolor{blue!2}
PIPNet~\cite{a2021beyond} & 36.19/0.981 & 32.20/0.964 & 34.20/0.973 \\
\rowcolor{blue!2}
CycleISP~\cite{zamir2020cycleisp} & 32.96/0.975 & 30.46/0.964 & 31.71/0.965 \\
\rowcolor{blue!2}
\midrule
\rowcolor{green!5}
BNN~\cite{hubara2016binarized} & 36.55/0.978 & 31.23/0.957 & 33.89/0.967 \\
\rowcolor{green!5}
ReActNet~\cite{liu2020reactnet} & 40.28/0.991 & 33.90/0.975 & 37.09/0.983 \\
\rowcolor{green!5}
BBCU~\cite{xia2022basic} & 40.47/0.992 & 33.79/0.974 & 37.13/0.983 \\
\rowcolor{green!5}
BTM~\cite{jiang2021training} & 40.30/0.992 & 33.35/0.975 & 36.83/0.983 \\
\rowcolor{green!5}
BiViT~\cite{he2023bivit} & 39.96/0.990 & 33.59/0.972 & 36.78/0.981 \\
\rowcolor{gray!20}
\textbf{BMTNet (Ours)} & \textbf{41.15/0.993} & \textbf{34.24/0.976} & \textbf{37.70/0.985} \\
\bottomrule
\end{tabular}
% \label{results}
}
\vspace{-2mm}
\caption{Quantitative evaluation on two video datasets, using PSNR (dB) / SSIM as evaluation metrics. BMTNet outperforms all other BNNs by over 0.6dB in PSNR, surpasses several full-precision networks, and approaches state-of-the-art FP methods.}
\label{tab:table2}
\vspace{-5mm}
\end{table}
\vspace{-1mm}
\subsection{Comparison to State-of-the-Arts}
\vspace{-1mm}
\paragraph{Quantitative Comparison.}
We compared our BMTNet with other binarization methods by replacing BMT block to the corresponding BNN block, including BNN~\cite{hubara2016binarized}, ReActNet~\cite{liu2020reactnet}, BBCU~\cite{xia2022basic}, BTM~\cite{jiang2021training} and BiViT~\cite{he2023bivit}. Additionally, we compare our BMTNet with State-of-the-Arts image restoration and demosaicing networks, including DFormer~\cite{Xu_2024_CVPR}, NAFNet~\cite{chen2022simple}, Restormer~\cite{zamir2022restormer}, SAGAN~\cite{sharif2021sagan}, CycleISP~\cite{zamir2020cycleisp}, PIPNet~\cite{a2021beyond}.
\begin{figure*}[ht]
\centering
% Row 1
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual6/ref.pdf}
\caption{Reference}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual6/bnn.pdf}
\caption{BNN~\cite{hubara2016binarized}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual6/bbcu.pdf}
\caption{BBCU~\cite{xia2022basic}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual6/bivit.pdf}
\caption{BiViT~\cite{he2023bivit}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual6/btm.pdf}
\caption{BTM~\cite{jiang2021training}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual6/react.pdf}
\caption{ReActNet~\cite{liu2020reactnet}}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.135\textwidth}
\centering
\includegraphics[width=\linewidth]{figures/visual6/bmtnet.pdf}
\caption{\textbf{BMTNet (Ours)}}
\end{subfigure}
\vspace{-1mm}
\caption{Visualized comparison on real data of HybridEVS. Reference is acquired from the classic demosaic method. Our BMTNet reduces moiré artifacts on dense lines and achieves the best result among BNNs.}
\label{figure6}
\vspace{-2mm}
\end{figure*}
We evaluated the performance and computational load of the models across six image datasets and two video datasets, as illustrated in Table \ref{tab:table1} and Table \ref{tab:table2}. The upper section reports the performance of full precision models, while the lower section demonstrates the results of BNNs. Notably, some full-precision methods perform unsatisfied results due to the color loss caused by event pixels. In contrast, our BMTNet demonstrates robust performance with significant reductions in parameters and operations. Furthermore, our proposed binary Mamba-Transformer block outperformed other BNN methods on all image and video datasets, achieving superior results with the least parameters of 1.28M and a slight increase of operators of 0.44G compared with the minimized model BTM due to the full-precise Selective Scan. The results validate that our method effectively enhances information extraction across both local and global dimensions.
\vspace{-4mm}
\paragraph{Visual Comparison.}
Visual comparisons are presented in Figures \ref{figure4}, \ref{figure5}, and \ref{figure6}, showing results on image data, video data, and real HybridEVS images. Our BMTNet reaches superior visual performance, which effectively reduces color aliasing and moiré artifacts in the test datasets compared to other approaches. Our BMTNet also demonstrates the best visual results on real-world HybridEVS data, effectively reducing artifacts when facing dense lines on images.
% \paragraph{Visual Comparison.}
\begin{figure}
% \vspace{-2mm}
\begin{center}
\begin{subfigure}{0.235\textwidth}
\includegraphics[width=\linewidth]{figures/chart2-2.pdf}
\end{subfigure}
\hfill
\begin{subfigure}{0.235\textwidth}
\includegraphics[width=\linewidth]{figures/chart2-1.pdf}
\end{subfigure}
\end{center}
\vspace{-5mm}
\caption{\small Left: Parameters reduction of our Bi-Mamba. Right: Operations reduction of our Bi-Mamba.}
\vspace{-4mm}
\label{figure7}
% \vspace{-2mm}
\end{figure}
% \vspace{-4mm}
\subsection{Ablation Study}
% \vspace{-2mm}
% \paragraph{Binarized Mamba}
We demonstrate an ablation study on the proposed binarized Mamba block to validate its effectiveness, as shown in Table \ref{tab:table3} and Table \ref{tab:table4}, separately conducting the influence of binarization and global visual embeddings.
\vspace{-4mm}
\paragraph{Binarization of Mamba}
As shown in Table \ref{tab:table3}, our proposed Bi-Mamba achieves a significant PSNR improvement of 1.78dB, attributed to the enhancement of long-range dependencies. This highlights the effectiveness of our hybrid structure in capturing both global and local information. We replace Bi-Mamba with a full-precision Mamba block to evaluate the impact of binarization. Bi-Mamba achieves a 79\% reduction in parameters and an 88\% reduction in total computation costs, with a reasonable performance drop of 0.79dB in PSNR. Specific to the Mamba blocks, as shown in Figure \ref{figure7}, Bi-Mamba significantly compresses the projections, including linear and convolution layers, reducing parameters and computation costs by \textbf{96\%} and \textbf{97\%}.
\vspace{-4mm}
\paragraph{Global Visual Embeddings}
To further explore the global visual embeddings, we analyze the effects of embedding positions across different control matrices, as shown in Table \ref{figure4}. Our proposed global visual embedding method on $\mathbf{B}$ improves PSNR by 0.05dB and SSIM by 0.001, demonstrating that appropriately integrated extra global information can enhance performance. However, applying global visual embeddings to all control matrices or the $\Delta$ matrix results in a slight performance decline. As shown in Formula \ref{eq:9}, $\Delta$ controls both $\overline{\mathbf{B}}$ and $\overline{\mathbf{A}}$, meaning the latent vector $h_k$ is affected in a cumulative product form by global visual information, which reduces its impact and causes $h_k$ to become unstable. In contrast, embedding the global visual vector with $\mathbf{B}$ directly influences the input and impacts the latent vector through cumulative summation, preserving usable information more effectively. Otherwise, embedding into $\mathbf{C}$ avoids cumulative product but has a limited impact on $h_k$, yielding only a 0.01dB improvement in PSNR.
% \begin{table}[ht]
% \footnotesize
% \centering
% \fontsize{10}{12}\selectfont
% \setlength{\tabcolsep}{0.45mm}{
% \begin{tabularx}{0.43\textwidth}{p{1.5cm}|cc|c}
% \toprule
% % \multicolumn{2}{c}{Part} \\
% % \cmidrule(r){1-2}
% \fontsize{10}{12}Methods &REDS& Vid4& Avearge\\
% % \cmidrule(r){1-2}
% \midrule
% DFormer & 42.45/0.9907& 36.01/0.9787&39.23/0.9847\\
% NAFNet &41.65/0.9889 &34.98/0.9740 &38.31/0.9815 \\
% Restormer & 41.91/0.9895& 35.08/0.9807&38.50/0.9851\\
% SAGAN & 38.13/0.9844 & 32.16/0.9631&35.15/0.9738 \\
% PIPNet & 36.19/0.9812& 32.20/0.9637 &34.20/0.9725\\
% CycleISP & 32.96/0.9753 &30.46/0.9642&31.71/0.9648\\
% \midrule
% BNN &36.55/0.9776&31.23/0.9567&33.89/0.9672\\
% ReActNet &40.28/0.9912&33.90/0.9748&37.09/0.9830\\
% BBCU &40.47/0.9916&33.79/0.9736&37.13/0.9826\\
% BTM &40.30/0.9918&33.35/0.9745&36.83/0.9831\\
% BiViT &39.96/0.9897&33.59/0.9717&36.78/0.9807\\
% \textbf{BMTNet} &\textbf{41.15/0.9931}&\textbf{34.24/0.9762}&\textbf{37.70/0.9847}\\
% \bottomrule
% \end{tabularx}
% % \label{results}
% }
% \caption{Quantitative evaluation of BMTNet compared to full-precision and other BNN methods on two video datasets datasets. BMTNet outperforms other BNNs while achieving results comparable to full-precision models at a minimal computational cost.}
% \label{tab:table1}
% \end{table}
\begin{table}
\centering
\fontsize{10}{12}\selectfont
\centering
\vskip 2mm
\setlength{\tabcolsep}{1.0mm}{
\begin{tabular}{lcccc}
\toprule
% \multicolumn{2}{c}{Part} \\
% \cmidrule(r){1-2}
Modules&Params (M)&OPs (G) & PSNR/SSIM \\
% \cmidrule(r){1-2}
\midrule
w/o Mamba& 1.25 & 5.32 & 35.17/0.964 \\
FP Mamba& 6.15 & 54.07 & 37.73/0.971 \\
\textbf{Bi-Mamba (Ours)} & 1.28 & 6.56 & 36.95/0.975 \\
\bottomrule
\end{tabular}}
% \vspace{-2mm}
\caption{Ablation study on the Bi-Mamba block. Bi-Mamba is crucial for maintaining performance, while it significantly reduces both parameters and operations with reasonable performance loss compared with the full precision version.}
\label{tab:table3}
\vspace{-4mm}
\end{table}
\begin{table}
\centering
\fontsize{10}{12}\selectfont
\centering
\vskip 2mm
\setlength{\tabcolsep}{1.3mm}{
\begin{tabular}{lccccc}
\toprule
% \multicolumn{2}{c}{Part} \\
% \cmidrule(r){1-2}
SE Position & None& All &$\mathbf{C}$ &$\mathbf{\Delta}$ & \textbf{$\mathbf{B}$ (Ours)} \\
% \cmidrule(r){1-2}
\midrule
PSNR &36.90 & 36.89 & 36.91 & 36.83 & \textbf{36.95} \\
SSIM &0.974 & 0.975 & 0.974 & 0.974 & \textbf{0.975} \\
\bottomrule
\end{tabular}}
\vspace{-2mm}
\caption{Ablation study on the position of global visual embeddings. Embedding into the control matrix $\mathbf{B}$ enhances global capacity while maintaining a stable influence on the latent vector $h_k$.}
\label{tab:table4}
\vspace{-5mm}
\end{table}
% \begin{table}
% \centering
% \fontsize{10}{12}\selectfont
% \vskip 2mm
% \setlength{\tabcolsep}{0.5mm}{
% \begin{tabular}{cccc}
% \toprule
% % \multicolumn{2}{c}{Part} \\
% % \cmidrule(r){1-2}
% \multirow{2}{*}{\makecell{Mamba\\Components}} &\multicolumn{1}{c}{Projections}&\multicolumn{1}{c}{Selective Scan}&\multicolumn{1}{c}{Total} \\
% \cline{2-4}
% &\multicolumn{3}{c}{Parms (M) / OPs (G)}\\
% \midrule
% Before binarization& 1.15/15.53 &0/1.09 & 1.15/16.62 \\
% After binarization& \textbf{0.17/0.76} &0/1.09 & \textbf{0.17/1.85} \\
% \bottomrule
% \end{tabular}}
% \caption{Analysis of Mamba binarization. The proposed binary format significantly reduces parameters and operations.}
% \label{tab:table3}
% \end{table}
% \paragraph{Efficient Bottleneck Distillation}
% To demonstrate the effectiveness of our proposed distillation methods, we evaluate BMTNet using three training strategies: no distillation, traditional knowledge distillation (KD)~\cite{hinton2015distilling}, and our efficient bottleneck distillation (EBD), as shown in Table \ref{tab:table4}. All experiments are conducted on 4 RTX 4090 GPUs. The results indicate KD increases training time by 1.4 times, but it leads to a performance drop due to the representation mismatch between the teacher and student models. In contrast, our EBD method improves PSNR by 0.05 dB with only a 4\% increase in training time, showing the efficiency and capability of our EBD approach.
% \begin{table}[ht]
% \label{results}
% \footnotesize
% \centering
% \setlength{\tabcolsep}{1.4mm}{
% \begin{tabular}{ccc }
% \toprule
% % \multicolumn{2}{c}{Part} \\
% % \cmidrule(r){1-2}
% Distillation Methods &\makecell{Training Time\\(hours)}& \makecell{PSNR/SSIM} \\
% \midrule
% w/o Distillation & 20.83 & 36.46/0.9720 \\
% Knowledge Distillation & 58.33 & 36.28/0.9711 \\
% Efficient Bottleneck Distillation & 21.66 & \textbf{36.51/0.9722} \\
% \bottomrule
% \end{tabular}}
% \caption{Ablation on efficient bottleneck distillation. Our proposed EBD enhances performance with only a 4\% increase in training time, whereas traditional KD leads to model degradation and extends training duration by 1.4 times.}
% \label{tab:table4}
% \end{table}
\vspace{-1mm}
\section{Conclusion}
\vspace{-1mm}
\label{sec:conclusion}
In this paper, we presented a lightweight binarized Mamba-Transformer network (BMTNet) for Quad Bayer HybridEVS demosaicing. First, we presented a binarized global visual encoding branch to acquire additional global information, which effectively enhances Mamba's global capacity. Second, we introduced a binarized Mamba-Transformer structure to reduce the model complexity. The pioneering binarization of Mamba and the fusion of extra global visual embeddings reduce computational complexity by compressing non-essential projections while enhancing performance through precise integration. Experiments conducted on eight diverse datasets demonstrate that our BMTNet outperforms other BNN methods while achieving results comparable to full-precision models at a minimal computational cost. Our approach expands the capabilities of BNNs and offers an efficient and high-performing demosaicing solution for resource-constrained HybridEVS.
% In this paper, we presented a lightweight binarized Mamba-Transformer network (BMTNet) for Quad Bayer HybridEVS demosaicing. First, we presented a binarized global visual encoding branch to acquire additional global information, which effectively enhances Mamba's global capacity. Second, we introduced a binarized Mamba-Transformer structure to reduce the model complexity. The pioneering binarization of Mamba and the fusion of extra global visual embeddings reduce computational complexity by compressing non-essential projections while enhancing performance through precise integration. Experiments conducted on eight diverse datasets demonstrate that our BMTNet outperforms other BNN methods while achieving results comparable to full-precision models at a minimal computational cost. Our approach expands the capabilities of BNNs and offers an efficient and high-performing demosaicing solution for resource-constrained HybridEVS.
\vspace{-4mm}
\paragraph{Acknowledgments:} This work was supported by the National Natural Science Foundation of China (grant No. 62350710797), the Key Program of Technology Research from Shenzhen Science and Technology Innovation Committee under Grant JSGG20220831104402004.
% addressing the problem of strict resource constraints in real-world demosaicinG, combines the advantages of Mamba in capturing long-dependencies with Swin Transformer's focus on local details within an extremely lightweight binary format
% \vspace{-2mm}
% \section{Acknowledgements}
% % \vspace{-2mm}
% \label{sec:ack}
\begin{thebibliography}{66}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[A~Sharif et~al.(2021)A~Sharif, Naqvi, and Biswas]{a2021beyond}
SM A~Sharif, Rizwan~Ali Naqvi, and Mithun Biswas.
\newblock Beyond joint demosaicking and denoising: An image processing pipeline for a pixel-bin image sensor.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 233--242, 2021.
\bibitem[Alpsentek(2024)]{alpsentek2024}
Alpsentek.
\newblock Alpix-eiger product overview: \url{https://alpsentek.com/product}, 2024.
\newblock URL \url{https://alpsentek.com/product}. Accessed: 2024-05-19.
\bibitem[Banner et~al.(2018)Banner, Hubara, Hoffer, and Soudry]{banner2018scalable}
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry.
\newblock Scalable methods for 8-bit training of neural networks.
\newblock In \emph{Proceedings of the Advances in Neural Information Processing Systems}, 2018.
\bibitem[Banner et~al.(2019)Banner, Nahshan, and Soudry]{banner2019post}
Ron Banner, Yury Nahshan, and Daniel Soudry.
\newblock Post training 4-bit quantization of convolutional networks for rapid-deployment.
\newblock In \emph{Proceedings of the Advances in Neural Information Processing Systems}, 2019.
\bibitem[Cai et~al.(2024)Cai, Zheng, Lin, Yuan, Zhang, and Wang]{cai2024binarized}
Yuanhao Cai, Yuxin Zheng, Jing Lin, Xin Yuan, Yulun Zhang, and Haoqian Wang.
\newblock Binarized spectral compressive imaging.
\newblock In \emph{Proceedings of the Advances in Neural Information Processing Systems}, 2024.
\bibitem[Chen et~al.(2022)Chen, Chu, Zhang, and Sun]{chen2022simple}
Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun.
\newblock Simple baselines for image restoration.
\newblock In \emph{Proceedings of the European Conference on Computer Vision}, pages 17--33. Springer, 2022.
\bibitem[Cordts et~al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson, Franke, Roth, and Schiele]{Cordts_2016_CVPR}
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.
\newblock The cityscapes dataset for semantic urban scene understanding.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, 2016.
\bibitem[Courbariaux et~al.(2015)Courbariaux, Bengio, and David]{courbariaux2015binaryconnect}
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David.
\newblock Binaryconnect: Training deep neural networks with binary weights during propagations.
\newblock In \emph{Proceedings of the Advances in Neural Information Processing Systems}, 2015.
\bibitem[Dong et~al.(2024)Dong, Xiong, Zhao, Zhang, Fan, Zhu, and Huang]{dong2024joint}
Yanchen Dong, Ruiqin Xiong, Jing Zhao, Jian Zhang, Xiaopeng Fan, Shuyuan Zhu, and Tiejun Huang.
\newblock Joint demosaicing and denoising for spike camera.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, pages 1582--1590, 2024.
\bibitem[Gallego et~al.(2020)Gallego, Delbr{\"u}ck, Orchard, Bartolozzi, Taba, Censi, Leutenegger, Davison, Conradt, Daniilidis, et~al.]{gallego2020event}
Guillermo Gallego, Tobi Delbr{\"u}ck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew~J Davison, J{\"o}rg Conradt, Kostas Daniilidis, et~al.
\newblock Event-based vision: A survey.
\newblock \emph{IEEE Transactions on Pattern Analysis and Machine Intelligence}, 44\penalty0 (1):\penalty0 154--180, 2020.
\bibitem[Gu and Dao(2023)]{gu2023Mamba}
Albert Gu and Tri Dao.
\newblock Mamba: Linear-time sequence modeling with selective state spaces.
\newblock \emph{arXiv preprint arXiv:2312.00752}, 2023.
\bibitem[Gu et~al.(2021)Gu, Goel, and R{\'e}]{gu2021efficiently}
Albert Gu, Karan Goel, and Christopher R{\'e}.
\newblock Efficiently modeling long sequences with structured state spaces.
\newblock \emph{arXiv preprint arXiv:2111.00396}, 2021.
\bibitem[Guo et~al.(2024)Guo, Li, Dai, Ouyang, Ren, and Xia]{guo2024Mambair}
Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia.
\newblock Mambair: A simple baseline for image restoration with state-space model.
\newblock \emph{arXiv preprint arXiv:2402.15648}, 2024.
\bibitem[Hatamizadeh and Kautz(2024)]{hatamizadeh2024mambavision}
Ali Hatamizadeh and Jan Kautz.
\newblock Mambavision: A hybrid mamba-transformer vision backbone.
\newblock \emph{arXiv preprint arXiv:2407.08083}, 2024.
\bibitem[He et~al.(2023)He, Lou, Zhang, Liu, Wu, Zhou, and Zhuang]{he2023bivit}
Yefei He, Zhenyu Lou, Luoming Zhang, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang.
\newblock Bivit: Extremely compressed binary vision transformers.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 5651--5663, 2023.
\bibitem[Hirakawa and Parks(2005)]{hirakawa2005adaptive}
Keigo Hirakawa and Thomas~W Parks.
\newblock Adaptive homogeneity-directed demosaicing algorithm.
\newblock \emph{IEEE Transactions on Image Processing}, 14\penalty0 (3):\penalty0 360--369, 2005.
\bibitem[Huang et~al.(2024)Huang, Liu, Qin, Li, Zhang, Liu, Magno, and Qi]{huang2024billm}
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi.
\newblock Billm: Pushing the limit of post-training quantization for llms.
\newblock \emph{arXiv preprint arXiv:2402.04291}, 2024.
\bibitem[Hubara et~al.(2016)Hubara, Courbariaux, Soudry, El-Yaniv, and Bengio]{hubara2016binarized}
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio.
\newblock Binarized neural networks.
\newblock In \emph{Proceedings of the Advances in Neural Information Processing Systems}, 2016.
\bibitem[Jia et~al.(2022)Jia, Sun, Liu, Xiao, Xu, and Zhai]{jia2022learning}
Jun Jia, Hanchi Sun, Xiaohong Liu, Longan Xiao, Qihang Xu, and Guangtao Zhai.
\newblock Learning rich information for quad bayer remosaicing and denoising.
\newblock In \emph{Proceedings of the European Conference on Computer Vision}, pages 175--191. Springer, 2022.
\bibitem[Jiang et~al.(2021)Jiang, Wang, Xin, Li, Yang, and Gao]{jiang2021training}
Xinrui Jiang, Nannan Wang, Jingwei Xin, Keyu Li, Xi Yang, and Xinbo Gao.
\newblock Training binary neural network without batch normalization for image super-resolution.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, pages 1700--1707, 2021.
\bibitem[Kodama et~al.(2023)Kodama, Sato, Yorikado, Berner, Mizoguchi, Miyazaki, Tsukamoto, Matoba, Shinozaki, Niwa, et~al.]{kodama20231}
Kazutoshi Kodama, Yusuke Sato, Yuhi Yorikado, Raphael Berner, Kyoji Mizoguchi, Takahiro Miyazaki, Masahiro Tsukamoto, Yoshihisa Matoba, Hirotaka Shinozaki, Atsumi Niwa, et~al.
\newblock 1.22 $\mu$m 35.6 mpixel rgb hybrid event-based vision sensor with 4.88 $\mu$m-pitch event pixels and up to 10k event frame rate by adaptive control on event sparsity.
\newblock In \emph{2023 IEEE International Solid-State Circuits Conference (ISSCC)}, pages 92--94. IEEE, 2023.
\bibitem[Li et~al.(2008)Li, Gunturk, and Zhang]{li2008image}
Xin Li, Bahadir Gunturk, and Lei Zhang.
\newblock Image demosaicing: A systematic survey.
\newblock \emph{Visual Communications and Image Processing}, 6822:\penalty0 489--503, 2008.
\bibitem[Liao et~al.(2024)Liao, Zhu, Wang, Pan, Wang, and Ma]{liao2024lightm}
Weibin Liao, Yinghao Zhu, Xinyuan Wang, Cehngwei Pan, Yasha Wang, and Liantao Ma.
\newblock Lightm-unet: Mamba assists in lightweight unet for medical image segmentation.
\newblock \emph{arXiv preprint arXiv:2403.05246}, 2024.
\bibitem[Liu and Sun(2011)]{liu2011bayesian}
Ce Liu and Deqing Sun.
\newblock A bayesian approach to adaptive video super resolution.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, pages 209--216. IEEE, 2011.
\bibitem[Liu et~al.(2020{\natexlab{a}})Liu, Jia, Liu, and Tian]{liu2020joint}
Lin Liu, Xu Jia, Jianzhuang Liu, and Qi Tian.
\newblock Joint demosaicing and denoising with self guidance.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 2240--2249, 2020{\natexlab{a}}.
\bibitem[Liu et~al.(2024)Liu, Tian, Zhao, Yu, Xie, Wang, Ye, Jiao, and Liu]{liu2024vMambavisualstatespace}
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu.
\newblock Vmamba: Visual state space model.
\newblock In \emph{The Thirty-eighth Annual Conference on Neural Information Processing Systems}, 2024.
\bibitem[Liu et~al.(2018)Liu, Wu, Luo, Yang, Liu, and Cheng]{liu2018bi}
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng.
\newblock Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm.
\newblock In \emph{Proceedings of the European Conference on Computer Vision}, pages 722--737, 2018.
\bibitem[Liu et~al.(2020{\natexlab{b}})Liu, Shen, Savvides, and Cheng]{liu2020reactnet}
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng.
\newblock Reactnet: Towards precise binary neural network with generalized activation functions.
\newblock In \emph{Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XIV 16}, pages 143--159. Springer, 2020{\natexlab{b}}.
\bibitem[Liu et~al.(2021)Liu, Lin, Cao, Hu, Wei, Zhang, Lin, and Guo]{liu2021swin}
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.
\newblock Swin transformer: Hierarchical vision transformer using shifted windows.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages 10012--10022, 2021.
\bibitem[Liu et~al.(2022)Liu, Oguz, Pappu, Xiao, Yih, Li, Krishnamoorthi, and Mehdad]{liu2022bit}
Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krishnamoorthi, and Yashar Mehdad.
\newblock Bi\protect{T}: Robustly binarized multi-distilled transformer.
\newblock In \emph{Proceedings of the Advances in Neural Information Processing Systems}, pages 14303--14316, 2022.
\bibitem[Loui et~al.(2007)Loui, Luo, Chang, Ellis, Jiang, Kennedy, Lee, and Yanagawa]{10.1145/1290082.1290117}
Alexander Loui, Jiebo Luo, Shih-Fu Chang, Dan Ellis, Wei Jiang, Lyndon Kennedy, Keansub Lee, and Akira Yanagawa.
\newblock Kodak's consumer video benchmark data set: concept definition and annotation.
\newblock In \emph{Proceedings of the International Workshop on Multimedia Information Retrieval}, pages 245--254, 2007.
\bibitem[Lu et~al.(2024)Lu, Xu, Ma, Guo, and Xiong]{lu2024event}
Yunfan Lu, Yijie Xu, Wenzong Ma, Weiyu Guo, and Hui Xiong.
\newblock Event camera demosaicing via swin transformer and pixel-focus loss.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 1095--1105, 2024.
\bibitem[Ma et~al.(2017)Ma, Duanmu, Wu, Wang, Yong, Li, and Zhang]{7752930}
Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang.
\newblock Waterloo exploration database: New challenges for image quality assessment models.
\newblock \emph{IEEE Transactions on Image Processing}, 26\penalty0 (2):\penalty0 1004--1016, 2017.
\bibitem[Ma et~al.(2024)Ma, Duan, Hong, Zhou, Zhang, Ren, and Shi]{ma2024color4e}
Yi Ma, Peiqi Duan, Yuchen Hong, Chu Zhou, Yu Zhang, Jimmy Ren, and Boxin Shi.
\newblock Color4e: Event demosaicing for full-color event guided image deblurring.
\newblock In \emph{Proceedings of the 32nd ACM International Conference on Multimedia}, pages 661--670, 2024.
\bibitem[Martin et~al.(2001)Martin, Fowlkes, Tal, and Malik]{937655}
D. Martin, C. Fowlkes, D. Tal, and J. Malik.
\newblock A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.
\newblock In \emph{Proceedings Eighth IEEE International Conference on Computer Vision}, pages 416--423 vol.2, 2001.
\bibitem[Munda et~al.(2018)Munda, Reinbacher, and Pock]{munda2018real}
Gottfried Munda, Christian Reinbacher, and Thomas Pock.
\newblock Real-time intensity-image reconstruction for event cameras using manifold regularisation.
\newblock \emph{International Journal of Computer Vision}, 126\penalty0 (12):\penalty0 1381--1393, 2018.
\bibitem[Nah et~al.(2019)Nah, Baik, Hong, Moon, Son, Timofte, and Mu~Lee]{nah2019ntire}
Seungjun Nah, Sungyong Baik, Seokil Hong, Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu~Lee.
\newblock Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops}, pages 0--0, 2019.
\bibitem[Qin et~al.(2020)Qin, Gong, Liu, Shen, Wei, Yu, and Song]{qin2020forward}
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song.
\newblock Forward and backward information retention for accurate binary neural networks.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 2250--2259, 2020.
\bibitem[Qin et~al.(2022)Qin, Ding, Zhang, Yan, Liu, Dang, Liu, and Liu]{qin2022bibert}
Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu.
\newblock Bibert: Accurate fully binarized bert.
\newblock \emph{arXiv preprint arXiv:2203.06390}, 2022.
\bibitem[Rastegari et~al.(2016)Rastegari, Ordonez, Redmon, and Farhadi]{rastegari2016xnor}
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi.
\newblock Xnor-net: Imagenet classification using binary convolutional neural networks.
\newblock In \emph{Proceedings of the European Conference on Computer Vision}, pages 525--542. Springer, 2016.
\bibitem[Sharif et~al.(2021)Sharif, Naqvi, and Biswas]{sharif2021sagan}
SMA Sharif, Rizwan~Ali Naqvi, and Mithun Biswas.
\newblock Sagan: adversarial spatial-asymmetric attention for noisy nona-bayer reconstruction.
\newblock \emph{arXiv preprint arXiv:2110.08619}, 2021.
\bibitem[Smith et~al.(2022)Smith, Warrington, and Linderman]{smith2022simplified}
Jimmy~TH Smith, Andrew Warrington, and Scott~W Linderman.
\newblock Simplified state space layers for sequence modeling.
\newblock \emph{arXiv preprint arXiv:2208.04933}, 2022.
\bibitem[Son et~al.(2017)Son, Suh, Kim, Jung, Kim, Shin, Park, Lee, Park, Woo, et~al.]{son2017640}
B Son, Y Suh, S Kim, H Jung, JS Kim, C Shin, K Park, K Lee, J Park, J Woo, et~al.
\newblock A 640$\times$ 480 dynamic vision sensor with a 9 um pixel and 300 meps address-event representation.
\newblock In \emph{Proceedings of the IEEE International Conference on Solid-State Circuits, San Francisco, CA, USA}, pages 5--9, 2017.
\bibitem[Syu et~al.(2018)Syu, Chen, and Chuang]{syu2018learning}
Nai-Sheng Syu, Yu-Sheng Chen, and Yung-Yu Chuang.
\newblock Learning deep convolutional networks for demosaicing.
\newblock \emph{arXiv preprint arXiv:1802.03769}, 2018.
\bibitem[Tan et~al.(2018)Tan, Chen, and Hua]{tan2018deepdemosaicking}
Daniel~Stanley Tan, Wei-Yang Chen, and Kai-Lung Hua.
\newblock Deepdemosaicking: Adaptive image demosaicking via multiple deep fully convolutional networks.
\newblock \emph{IEEE Transactions on Image Processing}, 27\penalty0 (5):\penalty0 2408--2419, 2018.
\bibitem[Tan et~al.(2017{\natexlab{a}})Tan, Zeng, Lai, Liu, and Zhang]{tan2017joint}
Hanlin Tan, Xiangrong Zeng, Shiming Lai, Yu Liu, and Maojun Zhang.
\newblock Joint demosaicing and denoising of noisy bayer images with admm.
\newblock In \emph{2017 IEEE International Conference on Image Processing (ICIP)}, pages 2951--2955. IEEE, 2017{\natexlab{a}}.
\bibitem[Tan et~al.(2017{\natexlab{b}})Tan, Zhang, Zuo, and Zhang]{tan2017color}
Runjie Tan, Kai Zhang, Wangmeng Zuo, and Lei Zhang.
\newblock Color image demosaicking via deep residual learning.
\newblock In \emph{Proceedings of the IEEE International Conference on Multimedia and Expo}, page~6, 2017{\natexlab{b}}.
\bibitem[Wang et~al.(2023)Wang, Zhu, Wang, Yu, Liu, Omar, and Hamid]{wang2023selective}
Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid.
\newblock Selective structured state-spaces for long-form video understanding.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 6387--6397, 2023.
\bibitem[Woo et~al.(2018)Woo, Park, Lee, and Kweon]{Woo_2018_ECCV}
Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In~So Kweon.
\newblock Cbam: Convolutional block attention module.
\newblock In \emph{Proceedings of the European Conference on Computer Vision}, 2018.
\bibitem[Wu et~al.(2024{\natexlab{a}})Wu, Yang, Sun, Zhang, Li, and Zhang]{wu2024seesr}
Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang.
\newblock Seesr: Towards semantics-aware real-world image super-resolution.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 25456--25467, 2024{\natexlab{a}}.
\bibitem[Wu et~al.(2024{\natexlab{b}})Wu, Fan, Chu, Ren, Li, Yue, Li, Zhou, Feng, Dai, et~al.]{wu2024mipi}
Yaqi Wu, Zhihao Fan, Xiaofeng Chu, Jimmy~S Ren, Xiaoming Li, Zongsheng Yue, Chongyi Li, Shangcheng Zhou, Ruicheng Feng, Yuekun Dai, et~al.
\newblock Mipi 2024 challenge on demosaic for hybridevs camera: Methods and results.
\newblock \emph{arXiv preprint arXiv:2405.04867}, 2024{\natexlab{b}}.
\bibitem[Xia et~al.(2022)Xia, Zhang, Wang, Tian, Yang, Timofte, and Van~Gool]{xia2022basic}
Bin Xia, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Radu Timofte, and Luc Van~Gool.
\newblock Basic binary convolution unit for binarized image restoration network.
\newblock \emph{arXiv preprint arXiv:2210.00405}, 2022.
\bibitem[Xiao et~al.(2024)Xiao, Yuan, Jiang, Chen, Zhang, and Lin]{xiao2024frequency}
Yi Xiao, Qiangqiang Yuan, Kui Jiang, Yuzeng Chen, Qiang Zhang, and Chia-Wen Lin.
\newblock Frequency-assisted mamba for remote sensing image super-resolution.
\newblock \emph{arXiv preprint arXiv:2405.04964}, 2024.
\bibitem[Xu et~al.(2024{\natexlab{a}})Xu, Sun, Zhu, Zhu, Fu, and Zha]{Xu_2024_CVPR}
Senyan Xu, Zhijing Sun, Jiaying Zhu, Yurui Zhu, Xueyang Fu, and Zheng-Jun Zha.
\newblock Demosaicformer: Coarse-to-fine demosaicing network for hybridevs camera.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops}, pages 1126--1135, 2024{\natexlab{a}}.
\bibitem[Xu et~al.(2024{\natexlab{b}})Xu, Sun, Zhu, Zhu, Fu, and Zha]{xu2024demosaicformer}
Senyan Xu, Zhijing Sun, Jiaying Zhu, Yurui Zhu, Xueyang Fu, and Zheng-Jun Zha.
\newblock Demosaicformer: Coarse-to-fine demosaicing network for hybridevs camera.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 1126--1135, 2024{\natexlab{b}}.
\bibitem[Yang et~al.(2022)Yang, Yang, Jiang, Li, Feng, Zhou, Sun, Zhu, Loy, Gu, et~al.]{yang2022mipi}
Qingyu Yang, Guang Yang, Jun Jiang, Chongyi Li, Ruicheng Feng, Shangchen Zhou, Wenxiu Sun, Qingpeng Zhu, Chen~Change Loy, Jinwei Gu, et~al.
\newblock Mipi 2022 challenge on quad-bayer re-mosaic: Dataset and report.
\newblock In \emph{Proceedings of the European Conference on Computer Vision}, pages 21--35. Springer, 2022.
\bibitem[Yuan et~al.(2024)Yuan, Shang, and Dong]{yuanpb}
Zhihang Yuan, Yuzhang Shang, and Zhen Dong.
\newblock Pb-llm: Partially binarized large language models.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2024.
\bibitem[Yunfan et~al.(2024)Yunfan, Qian, Rao, Xiao, Chen, and Xiong]{yunfanrgb}
LU Yunfan, Yanlin Qian, Ziyang Rao, Junren Xiao, Liming Chen, and Hui Xiong.
\newblock Rgb-event isp: The dataset and benchmark.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2024.
\bibitem[Zamir et~al.(2020)Zamir, Arora, Khan, Hayat, Khan, Yang, and Shao]{zamir2020cycleisp}
Syed~Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad~Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.
\newblock Cycleisp: Real image restoration via improved data synthesis.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 2696--2705, 2020.
\bibitem[Zamir et~al.(2022)Zamir, Arora, Khan, Hayat, Khan, and Yang]{zamir2022restormer}
Syed~Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad~Shahbaz Khan, and Ming-Hsuan Yang.
\newblock Restormer: Efficient transformer for high-resolution image restoration.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 5728--5739, 2022.
\bibitem[Zeng et~al.(2023)Zeng, Feng, Cao, Huang, Zhao, Luong, Aelterman, and Philips]{zeng2023inheriting}
Haijin Zeng, Kai Feng, Jiezhang Cao, Shaoguang Huang, Yongqiang Zhao, Hiep Luong, Jan Aelterman, and Wilfried Philips.
\newblock Inheriting bayer's legacy-joint remosaicing and denoising for quad bayer image sensor.
\newblock \emph{International Journal of Computer Vision}, 2023.
\bibitem[Zhang et~al.(2024{\natexlab{a}})Zhang, Zhang, Yuan, and Fu]{zhang2024binarized}
Gengchen Zhang, Yulun Zhang, Xin Yuan, and Ying Fu.
\newblock Binarized low-light raw video enhancement.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 25753--25762, 2024{\natexlab{a}}.
\bibitem[Zhang et~al.(2022)Zhang, Dong, Zhang, Ding, Heide, Yin, and Yang]{zhang2022spiking}
Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang.
\newblock Spiking transformers for event-based single object tracking.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 8801--8810, 2022.
\bibitem[Zhang et~al.(2024{\natexlab{b}})Zhang, Huang, Ma, Li, Luo, Xie, Qin, Luo, Li, Liu, et~al.]{zhang2024recognize}
Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et~al.
\newblock Recognize anything: A strong image tagging model.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages 1724--1732, 2024{\natexlab{b}}.
\bibitem[Zheng et~al.(2024)Zheng, Li, Chen, Wang, Zhou, Hu, and Yan]{zheng2024quad}
Bolun Zheng, Haoran Li, Quan Chen, Tingyu Wang, Xiaofei Zhou, Zhenghui Hu, and Chenggang Yan.
\newblock Quad bayer joint demosaicing and denoising based on dual encoder network with joint residual learning.
\newblock In \emph{Proceedings of the AAAI Conference on Artificial Intelligence}, pages 7552--7561, 2024.
\bibitem[Zhu et~al.(2024)Zhu, Liao, Zhang, Wang, Liu, and Wang]{zhu2024vision}
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang.
\newblock Vision mamba: Efficient visual representation learning with bidirectional state space model.
\newblock \emph{arXiv preprint arXiv:2401.09417}, 2024.
\end{thebibliography}