Unsupervised Coordinate-Based Video Denoising
Extracted source text ยท arXiv 2307.00179
\documentclass{article}
\pdfoutput=1
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2023
% ready for submission
% \usepackage{neurips_2023}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
\usepackage[preprint]{neurips_2023}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2023}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2023}
\usepackage[toc,page]{appendix}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{makecell}
\usepackage{mathtools}
\usepackage{multirow}
\usepackage{hhline}
\usepackage{cellspace}
\setlength\cellspacetoplimit{5pt}
\setlength\cellspacebottomlimit{5pt}
\newcommand{\mtx}[1]{\mathbf{#1}}
\newcommand{\vct}[1]{\mathbf{#1}}
\def \lg {\langle}
\def \rg {\rangle}
\def \mA {\mtx{A}}
\def \mC {\mtx{C}}
\def \mI {\mtx{I}}
\def \mU {\mtx{U}}
\def \mS {\mtx{S}}
\def \mV {\mtx{V}}
\def \mW {\mtx{W}}
\def \mLambda {\mtx{\Lambda}}
\def \mX {\mtx{X}}
\def \mY {\mtx{Y}}
\def \mZ {\mtx{Z}}
\def \zero {\mathbf{0}}
\def \vzero {\vct{0}}
\def \vone {\vct{1}}
\def \vu {\vct{u}}
\def \vv {\vct{v}}
\def \vx {\vct{x}}
\def \vy {\vct{y}}
\def \vz {\vct{z}}
\def \vphi {\vct{\phi}}
\def \vmu {\vct{\mu}}
\def \R {\mathbb{R}}
\title{Unsupervised Coordinate-Based Video Denoising}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Mary Damilola Aiyetigbo\thanks{Clemson University, School of Computing} \\
\texttt{maiyeti@clemson.edu} \\
\And
Dineshchandar Ravichandran\footnotemark[1] \\
\texttt{dravich@clemson.edu} \\
\AND
Reda Chalhoub\thanks{MUSC, Department of Neuroscience} \\
\texttt{chalhoub@musc.edu} \\
\And
Peter Kalivas\footnotemark[2] \\
\texttt{kalivasp@musc.edu} \\
\And
Nianyi Li\footnotemark[1] \\
\texttt{nianyil@clemson.edu} \\
}
\begin{document}
\maketitle
\begin{abstract}
% Emerging applications in computer vision and medical image processing require algorithms capable of effectively denoising video data containing noise with various sources and distributions.
% Existing video denoising techniques either predominantly rely on synthetic noise-clean image pairs, or injecting specific noise patterns to the clean reference images during training, which shows limited efficacy on real-world video sequences due to the complexity of real-world noise.
In this paper, we introduce a novel unsupervised video denoising deep learning approach that can help to mitigate data scarcity issues and shows robustness against different noise patterns, enhancing its broad applicability. Our method comprises three modules: a Feature generator creating features maps, a Denoise-Net generating denoised but slightly blurry reference frames, and a Refine-Net re-introducing high-frequency details. By leveraging the coordinate-based network, we can greatly simplify the network structure while preserving high-frequency details in the denoised video frames.
Extensive experiments on both simulated and real-captured demonstrate that our method can effectively denoise real-world calcium imaging video sequences without prior knowledge of noise models and data augmentation during training.
% In response to the growing need for unsupervised deep learning algorithms in computer vision and medical image processing, we present a novel coordinate-based approach that effectively denoises video data without clean reference videos or noise introduction during training. Our method comprises two stages: an image generator producing color images for each frame index, and a denoiser outputting denoised reference frames. This approach addresses data scarcity challenges and demonstrates robustness to various noise models, enhancing its generalizability across diverse applications.
% By eliminating manual annotation and clean reference image requirements, our method significantly reduces data collection and preprocessing costs and time. We showcase its versatility by applying it to real-world imaging tasks, including calcium imaging, raw video, electron-microscopy, and noisy synthetic video, without prior knowledge of noise models. Our approach effectively denoises even short video sequences without data augmentation during training, highlighting its potential for improved performance and accurate decision-making in various applications.
% Unsupervised convolutional neural networks have been trained to denoise noisy video without a clean reference video. However, most of these methods denoise synthetic data by adding noise iteratively during training which introduces new noise value to the model, thereby potentially preventing the model from learning the implicit noise representation in the dataset. We propose an unsupervised coordinate-based approach that takes as input the pixel coordinates of each video frame and outputs the corresponding clean frame. Unlike other conventional techniques, our method can perform video denoising without adding noise to each frame during training. Our method has two stages: the image generator, which produces the corresponding color image for each frame index, and a denoiser which outputs the denoised reference frame. We demonstrate that our approach applies to denoising real-world imaging such as calcium imaging, raw video, electron-microscopy, and noisy synthetic video without prior knowledge of the noise model. The performance of our method shows that it can denoise even short video sequences without data augmentation during training.
\end{abstract}
\input{sections/introduction}
\input{sections/relatedwork}
\input{sections/method}
\input{sections/trainingdetails}
\input{sections/results}
\input{sections/discussion}
\small
\bibliographystyle{plain}
\bibliography{denoise}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\appendix
\input{sections/supplement_new}
\end{document}
\section{Introduction}
Video denoising is a critical task in computer vision, with various applications such as video surveillance, medical imaging, and autonomous driving. The goal is to remove the noise present in videos, which can be caused by various factors such as sensor noise, compression, and low-light conditions. Supervised Convolutional Neural Networks (CNNs) have been successful in solving this problem by learning the mapping between noisy and clean images from large datasets \cite{dvdnet,fastdvdnet,rvidenet,deformableattention}. However, obtaining such datasets can be challenging and time-consuming, especially when the noise is complex and diverse.
To address this challenge, unsupervised video denoising methods have emerged in recent years, which do not require labeled data for training. One such approach is based on the "blind-spot" technique, where the CNN is trained to estimate each noisy pixel from its surrounding spatial neighborhood without considering the pixel itself \cite{udvd}. However, most of these self-supervised methods require either training on synthetic datasets by adding random noise to the clean image during training, or need long video sequences to learn the implicit noise model \cite{udvd,noise2void}.
% In applications such as microscopy, obtaining noiseless ground truth videos and estimating the underlying tractable model of the noise can be particularly challenging.
In applications like microscopy, where video sequences might be short, obtaining noiseless ground truth videos and estimating the underlying tractable model of the noise pose significant challenges.
In such cases, the performance of denoising methods based on "blind-spot" CNN can significantly degrade \cite{self-supervised}. Therefore, developing an unsupervised video denoising method that can work directly on raw noisy videos is critical to improve the accuracy and robustness of video denoising in challenging scenarios.
% The overall structure of our network is illustrated in Fig.~\ref{fig:pipeline}. Our network consists of three primary components designed to effectively denoise video data: a feature generator $\mathcal{F}$ rendering feature maps that correspond to the coordinates of the input frames; a Denoise-Net $\mathcal{D}$ generating denoised but slight blurry reference frame given the feature maps of neighboring frames, and a Refine-Net $\mathcal{R}$ adding back the high-frequency details to the blurry denoised frames. By using the input noisy video frames as reference in the loss functions, each component of our network collaboratively contributes to the end goal of effective video denoising, while maintaining the integrity and quality of the original video data. To simplify the network architecture and improve the training efficiency, we adopt coordiante-based networks,\cite{chibane2020implicit,dupont2020equivariant,riegler2020free,fourier,siren}, in $\mathcal{D}$ and $\mathcal{R}$.
The architecture of our network is shown in Fig.~\ref{fig:pipeline}, comprises three key components, each designed to effectively denoise video data. First, the feature generator $\mathcal{F}$ renders feature maps that align with the coordinates of the input frames. Second, the Denoise-Net $\mathcal{D}$ utilizes these feature maps to produce denoised yet slightly blurry reference frames. Finally, the Refine-Net $\mathcal{R}$ restores high-frequency details to the denoised frames, enhancing overall image clarity. The network is trained by minimizing the discrepancy between the generated denoised frame, the input noisy frame, and the refined final output frame, ensuring both efficient denoising and preservation of the original video data's integrity and quality. To streamline the network architecture and enhance training efficiency, we incorporate coordinate-based networks, as referenced in \cite{chibane2020implicit,dupont2020equivariant,riegler2020free,fourier,siren}, into both $\mathcal{D}$ and $\mathcal{R}$. We conduct comprehensive experiments on a diverse range of noisy videos, including both simulated and real-captured footage. In comparison to state-of-the-art method, our approach demonstrates superior performance in effectively correcting noise, highlighting its efficacy and potential for widespread application.
% One critical component in our network is the position encoding operator commonly used in NeRF networks \cite{chibane2020implicit,dupont2020equivariant,riegler2020free,fourier}. By incorporating this operator into the image generator, we can simplify our network structure while maintaining fine spatial details in the reconstructed output images.
% We perform extensive experiments on both simulated and real-captured noisy videos. We compare our method with the state-of-the-art methods and show that our method has better performance in correcting the noise.
\begin{figure*}%[t!]
\centering
\includegraphics[width=1\linewidth]{images/pipeline.png}
\caption{The pipeline consists of three main components: a feature generator $\mathcal{F}$, which renders feature maps matching the input frame coordinates; a Denoise-Net $\mathcal{D}$, which utilizes these feature maps to generate denoised yet slightly blurry reference frames; and a Refine-Net $\mathcal{R}$, which reintroduces high-frequency details to enhance the clarity of the denoised frames. The entire network is trained by optimizing the difference between the generated denoised frame, the input noisy frame, and the final refined output frame, ensuring efficient and accurate video denoising.}
\label{fig:pipeline}
\end{figure*}
% [width=1\textwidth]
% \includegraphics[width=1\textwidth]{images/pipeline.png}
% [scale=1,bb=0 0 30 30]
\section{Related Works}
% \subsection{Supervised denoising.} Supervised CNN approach to denoising has achieved state-of-art performance in a single image and video denoising \cite{dncnn, ffdnet, videnn, rvidenet} and this requires the model to be trained using clean images as ground truth. In \cite{dncnn}, residual learning \cite{residual} and batch normalization \cite{batchnorm} were employed in DnCNN for image denoising by estimating the noise in the input image. Zhang et al. \cite{} tried to develop a faster model by using dilated convolution \cite{dilated} to increase the model's receptive field. FFDNet \cite{ffdnet} built on DnCNN by introducing the downsampling and upsampling layers before and the after the CNN blocks, respectively. The idea is to increase the receptive field by reducing the size of the images. DVDNet \cite{dvdnet} denoised individual frames in a video sequence in the first stage and registers the denoised neighboring frames using DeepFlow \cite{deepflow}. Similar to \cite{dvdnet}, ViDeNN \cite{videnn} and FastDVDnet \cite{fastdvdnet} denoised in two stages (spatial and temporal denoising) but did not require explicit motion estimation. But the difference is that the later \cite{fastdvdnet} used Unet \cite{unet} blocks while ViDeNN \cite{videnn} used only convolution layers. While these models achieved good denoised outputs, getting a dataset with noisy/clean pairs is unrealistic, especially with denoising medical image recordings with no ground truth clean images.
\noindent \textbf{Supervised Video Denoising.}
% \paragraph{Supervised Denoising.}
Supervised CNN approaches to denoising, such as those presented in \cite{dncnn, ffdnet, videnn, rvidenet}, have achieved state-of-the-art results in both single image and video denoising, requiring clean images for model training. Several techniques, like residual learning \cite{residual} and batch normalization \cite{batchnorm} in DnCNN \cite{dncnn}, or the use of dilated convolution \cite{dilated} for a faster model, have been developed. The introduction of downsampling and upsampling layers in FFDNet \cite{ffdnet} aimed to increase the receptive field. Two-stage denoising without explicit motion estimation was explored by DVDNet \cite{dvdnet}, ViDeNN \cite{videnn}, and FastDVDnet \cite{fastdvdnet}. However, these models' reliance on unrealistic noisy/clean pairs is a drawback, especially in medical imaging where clean ground truth images are seldom available.
% \subsection{Unsupervised denoising.} Noise2Noise (N2N) \cite{noise2noise} proposed an unsupervised algorithm using two versions of noisy images as inputs and targets, respectively, with the same underlying clean signals. Frame2Frame (F2F) \cite{frame2frame} used a similar approach to denoise a single image by transforming consecutive frames using TV-L1 optical flow \cite{tvl1} to estimate the motion and use them as the noisy target of the same image to fine-tune pre-trained DnCNN model \cite{dncnn}. This framework was extended to Multi Frame2Frame (MF2F) \cite{mf2f} for video denoising by taking a stack of multiple frames as input to fine-tune a pre-trained FastDVDnet framework. Although their model could denoise a single video sequence, its performance relied on a pre-trained supervised model already trained on a large dataset. Noise2Void (N2V) \cite{noise2void} introduced a blind spot technique. Instead of using two noisy pairs of the image, which is difficult to acquire, they trained a model to estimate the clean signal of each noisy pixel using the neighboring pixels and omitting the center pixel through masking. The idea is that omitting the central pixel prevents the model from learning identity mapping, and the model can estimate the underlying clean signal. Inspired by this idea, \cite{self-supervised} developed convolution blind-spot network architecture, which combined several UNet blocks with restricted receptive fields to omit the center pixel for single-image denoising. UDVD \cite{udvd} used this blind-spot architecture approach for unsupervised video denoising by using consecutive neighboring frames to denoise each central frame.
% Although UDVD achieved the state-of-art result for unsupervised video denoising, their method required noise to be added to the clean image at every model step. This way, the model is exposed to different noise values at every iteration, which makes it difficult for the model to learn an identity noise map, hence could achieve good denoising performance. Also, when trained on a single sequence, several data augmentations were applied, increasing the data size, which helped the model perform very well.
% Lecoq et al. \cite{deepinter} removed independent noise in two-photon imaging using a deep interpolation algorithm that omits the predicted target frame from the input and uses the pre and post-frames of the central target frame as input. While the model showed segmentation results for the denoised recording, training required over 200,000 data samples. Also, the model was trained only on medical imaging, and the authors did not show whether the method can be generalized to natural color videos.
% This paper shows that our method can denoise a single video with a very short sequence without requiring data augmentation. We also show that our method can generalize to noisy medical imaging recordings, color videos, and raw videos. Unlike UDVD and other related approaches, our method does not require adding noise at every model iteration (which changes the noise value) to denoise synthetic noisy videos. We showed that our method can denoise synthetic sequences that the noise had been added before loading the data to the model. Our result also showed that our model could achieve comparable performance with models trained with a large dataset.
\noindent \textbf{Unsupervised Video Denoising.}
% \paragraph{Unsupervised Denoising.}
Unsupervised denoising algorithms like Noise2Noise (N2N) \cite{noise2noise}, Frame2Frame (F2F) \cite{frame2frame}, and Multi Frame2Frame (MF2F) \cite{mf2f} leverage noisy images as both inputs and targets, or consecutive frames as noisy targets for denoising. Techniques like the blind spot technique in Noise2Void (N2V) \cite{noise2void} and the convolution blind-spot network architecture \cite{self-supervised} have been developed to estimate the underlying clean signal. While UDVD \cite{udvd} achieved state-of-the-art results, their methodology necessitates noise addition at every model iteration and substantial data augmentation. Furthermore, the deep interpolation algorithm by Lecoq et al. \cite{deepinter} required over 200,000 data samples and showed limitations in generalizability.
In contrast to these methods, our approach can effectively denoise short video sequences and generalize to different types of noisy videos without requiring excessive data augmentation or iterative noise addition.
% \subsection{Implicit neural representation.} Neural representation is an emerging method that uses neural networks to parameterize the properties of objects or scenes. These methods called coordinate-based representations or coordinate networks, use fully connected neural networks to encode input coordinates to their corresponding signal value. These methods have been used in a wide range of tasks such as view synthesis \cite{nerf}, Image representation \cite{fourier, siren}, and 3D shape representation \cite{neural}. The recent work of RawNeRF \cite{rawnerf} inspires our work which used noisy raw images as input to produce a clean reconstruction of the 3D scene.
\noindent \textbf{Implicit Neural Representation.}
% \subsection{Implicit Neural Representation.}
Implicit neural representations, also known as coordinate-based representations, utilize fully connected neural networks to associate input coordinates with their corresponding signal values. They have shown remarkable utility across a range of tasks, including view synthesis \cite{nerf}, image representation \cite{fourier, siren}, and 3D shape representation \cite{neural}. One significant development in this domain is the Sinusoidal Representation Networks (SIREN) \cite{siren}, which leverage periodic activation functions to encode positional information and model complex natural signals with high precision. This approach allows for rapid training convergence, which significantly improves efficiency, especially in resource-intensive applications. In our framework, we employ coordinate-based networks in the feature generator $\mathcal{F}$ and Refine-Net $\mathcal{R}$ to enhance the denoising performance.
% Specifically, we utilize a Fourier feature mapping network in the feature generator $\mathcal{F}$, which creates feature maps that correspond to the coordinates of the input frames. For the Refine-Net $\mathcal{R}$, which is responsible for adding back the high-frequency details to the blurry denoised frames, we leverage the strengths of the Sinusoidal Representation Networks (SIREN). This strategic application of coordinate-based networks improves our method's robustness and adaptability, making it particularly effective in handling diverse noise patterns and short video sequences.
% \begin{figure}[t!]
% \centering
% \includegraphics[width=0.95\textwidth]{images/model_arc.png}
% \caption{Model Architecture}
% \label{fig:network_all}
% \end{figure}
\section{Unsupervised Coordinate-Based Video Denoising}
\label{sec:method}
Given a sequence of noisy video frames $\{I_t | t = 1, 2, \dots, N\}$ and their corresponding coordinates $\{G_t | t = 1, 2, \dots, N\}$, where $G_t(\textbf{p}_t )$ represents the coordinates of pixel $\textbf{p}_t = (x, y,t)$ in the noisy frame $I_t$, our objective is to recover the noise-free video frames $\{J_t | t = 1, 2, \dots, N\}$. Our approach consists of three primary components: a feature generator $\mathcal{F}_\theta$, a Denoise-Net $\mathcal{D}_\phi$, and a Refine-Net $\mathcal{R}_\eta$, as illustrated in Fig.~\ref{fig:pipeline}.
\subsection{Feature Generator $\mathcal{F}_\theta$}
% \noindent \textbf{Feature Generator $\mathcal{F}_\theta$}
Our Feature Generator processes a batch of uniformly sampled coordinate grids $\{G_t | t = 1,...,B\}$, where each $G_t \in\mathbb{R}^{H \times W \times 3}$, to generate a corresponding batch of feature maps $\{F_t | t = 1,...,B\}$, each being $F_t \in\mathbb{R}^{H \times W \times C}$. Here, $B$ is the batch size, while $C$ is the number of feature channels.
Before the coordinate grid is fed into $\mathcal{F}_\theta$, each coordinate $\textbf{p}_t=(x,y,t)$ is subject to positional encoding \cite{nerf,mipnerf, rawnerf}. This encoding step transforms low-dimensional input coordinates into a higher-dimensional space, enabling the model to better learn and represent high-frequency details inherent in the image data. The positional encoding function we adopt is given as:
\begin{equation}
\gamma(\mathbf{p}_t) = [\mathrm{sin}(2^0 \pi \mathbf{p}_t), \mathrm{cos}(2^0 \pi \mathbf{p}_t), ..., \mathrm{sin}(2^{L-1} \pi \mathbf{p}_t), \mathrm{cos}(2^{L-1} \pi \mathbf{p}_t)]
\label{posenc}
\end{equation}
In this equation, $L$ serves as a hyperparameter controlling the level of detail or high-frequency information in the output. By selecting a smaller value for $L$, we can effectively reduce the level of high-frequency noise in the image data, as noise often manifests as high-frequency information. For our experiments, we set $L=30$. The input coordinates, normalized to the range $[-1, 1]$ using a mesh grid, are passed through the encoding function $\gamma(.)$. The resulting high-dimensional output $\gamma(G_t)\in\mathbb{R}^{H \times W \times 6L}$ is subsequently fed into the feature generator $\mathcal{F}_\theta$:
\begin{equation}
F_t = \mathcal{F}_\theta(\gamma(G_t))
\end{equation}
where $F_t\in\mathbb{R}^{H \times W \times C}$ is the feature maps corresponding to each noisy frame, and $C$ denotes the channel size of the output features.
Our feature generator is composed of 6 convolution layers, each featuring a kernel size of 3 and identical padding to maintain the size throughout the model layers. Each layer has 256 feature channels with batch normalization (BN) applied solely to the first two layers. ReLU activation is employed for all layers, except for the last one.
% Our method has two subnetworks: an image generator ($\mathcal{I}$) which is used to output video frames using coordinate points as input and a denoiser ($\mathcal{D}_\phi$) network that takes the output of ($\mathcal{I}$) as input and generates a denoised version of the noisy video. Our idea is motivated by the ability of RawNeRF \cite{rawnerf} to remove noise from noisy mosaicked images. Standard radiance field networks such as NeRF \cite{nerf} and RawNeRF \cite{rawnerf} take as input 3D and 2D position and viewing direction, respectively, as input in order to generate color and volume density. At the first stage, we use the 3D pixel coordinates of each frame in the video sequence as input and generate the color $C$ at that each pixel position in the frame where C = 3 for color or $C=1$ for grayscale frames respectively. ($\mathcal{I}$) composed of only convolution layers and takes as an input 3D pixel coordinate $(x,y,t) \in \{0...W-1\} \times \{0...H-1\} \times \{0...T-1\} $ where $t$ is the frame index in the video sequence. Unlike \cite{rawnerf} that used multilayer perception (MLP) to approximate the representation, we used CNN since our data is a video sequence containing multiple images, and using MLP would require the model to have $H \times W \times 3$ neurons in the first layer and this might be too much for larger image sizes. Each video sequence $V$ contains $\{v_t\}_{t=1}^T$ frames along the time $(T)$ sequence. The model's input $\mathbf{x}\in\mathbb{R}^{H \times W \times (x,y,t)}$ is the 3D coordinate at frame index $t$ and returns the corresponding image frame ${f_t}\in\mathbb{R}^{H \times W \times C}$ . Therefore to compute the whole video $V$, the model is evaluated at each position $(x,y,t)$ of the coordinate mesh grid.
% \begin{equation}
% V = \{I(x,y,t) \mid \in mgrid(H,W,T)\}
% \end{equation}
% where
% \begin{equation}
% mgrid(H,W,T) = \{I(x,y,t) \mid -1 \le x < 1, -1 \le y < 1, -1 \le t < 1\}
% \end{equation}
% is the set of pixel coordinates. We use $N$ consecutive 3D frame coordinates as input and generate the corresponding image at each location.
% \subsection{Positional encoding}
% Although coordinate-based networks have been used as approximate functions for tasks such as novel view synthesis \cite{nerf, rawnerf} and shape representation \cite{chen2019learning, jiang2020local}. However, passing low-dimensional input to the networks leads to a poor result as shown in \cite{nerf, fourier, rahaman2019spectral, basri2020frequency}. This is because coordinate-based MLPs are unable to learn the high-frequency details in the image data. Experiments done by \cite{nerf, mipnerf, fourier} have shown that mapping input coordinates to a high embedding space allows the model to represent high-frequency content in the image. in this work, we employ the positional encoding function used in \cite{nerf} to map our input coordinates $d$.
% \begin{equation}
% \gamma(d) = [sin(2^0{\pi}{d}),cos(2^0{\pi}{d}),....,sin(2^{L-1}{\pi}{d}),cos(2^{L-1}{\pi}{d})]
% \end{equation}
% where $L$ is a hyperparameter. We chose $L=30$ for our experiments. The input coordinates which are normalized between $[-1, 1]$ using mesh grid are passed to the encoding function $\gamma(.)$, and embedded output is used as input to the Image generator ($\mathcal{I}$).
% \subsection{Image generator}
% The image generator takes the embedded frame coordinate as input and generates the corresponding image at each frame location. The model is composed of 6 convolution layers with a kernel size of 3 and padding set to the same in order to retain the size across the model layers. The number of feature maps was set to 256 with batch normalization (BN) applied to only the first two layers. ReLU activation was used for all the layers except the last layer.
% \noindent \textbf{Denoiser $\mathcal{D}_\phi$}
\subsection{Denoiser $\mathcal{D}_\phi$}
The denoiser network takes the concatenated feature maps output from the feature generator as input, and generates a denoised central frame $\hat{I}_B$:
\begin{equation}
\hat{I}_B = \mathcal{D}_\phi([F_1,F_2,...F_B])
\end{equation}
where the concatenation is applied along the feature channel dimension. This allows $\mathcal{D}_\phi$ to learn spatial-temporal patterns along the neighboring time frames eliminate the noise. Note that the output of $\mathcal{D}_\phi$ may be somewhat blurred, because we've set a low $L$ in the feature generator.
The architecture of $\mathcal{D}_\phi$ includes 6 convolutional layers. ReLU activation is used in the first five layers, and a sigmoid activation function is used in the last layer. Unlike in the feature generator, batch normalization is not applied in this stage. Each layer uses 256 filters with a kernel size of 3, except for the last two layers which have 96 and the number of color channels filters, respectively, with kernel sizes of 1.
% The second stage of our network is the denoiser $\mathcal{D}_\phi$ which takes output from the image generator as input. The denoiser is also composed of CNN layers, however, it takes stacks of $N$ frames output from the image generator $\{I\}_{t-n}^{t+n}\in\mathbb{R}^{N \times H \times W \times C}$ and produces a denoised output of the middle frame $D_t\in\mathbb{R}^{1 \times H \times W \times C}$. Combining the information from the neighboring frames allows the model to learn spatial-temporal differences along the video sequence. Similar to the image generator, the denoiser is also composed of 6 convolution layers with ReLU activation used in the first 5 layers and sigmoid used for the last layer. Unlike the first stage, we did not use batch normalization at this stage. We also used 256 filters and a kernel size of 3 except for the last two layers which had 96 and C \textit{(color channels)} filters respectively with kernel sizes of 1.
% \noindent \textbf{Refine-Net $\mathcal{R}_\eta$}
\subsection{Refine-Net $\mathcal{R}_\eta$}
The refine-net is built upon the backbone of the Sinusoidal Representation Networks (SIREN) \cite{siren}, which is a type of coordinate-based network that uses periodic activation functions, particularly sine functions, in place of traditional activation functions like ReLU. SIREN's unique characteristic lies in its ability to naturally model the high-frequency details of complex patterns by leveraging its intrinsic periodic activation functions. In our context, $\mathcal{R}_\eta$ uses the SIREN network to take the coordinates grid of the central frame as input and generate the refined image $\hat{I}_R$.
\begin{equation}
\hat{I}_R = \mathcal{R}_\eta(G_t^c)
\end{equation}
where $G_t^c$ is the coordinates of the central frames in the input batch.
Note that the SIREN-based refine-net is particularly beneficial in our case, as it helps to further refine the denoised output from $\mathcal{D}_\phi$ by enhancing the finer details and correcting any blurring introduced in the denoising process. The output from $\mathcal{R}_\eta$ represents the final denoised and refined video frames.
\section{Network Optimization}
% To denoise a frame, we take $N$ neighboring frames as input. Our method uses a two-stage approach similar to UDVD, ViDeNN's approach and the idea is to take embedded neighboring frame coordinates as input to the Image generator. This is so that the model can learn the temporal information that exists in the neighbors. The output frames from this model with dimension $N\times C \times H \times \times W$ are then stacked together and passed as the input to the denoiser model with the new dimension of $1 \times N *C \times H \times W$. Both models were trained end-to-end simultaneously. Since our method is unsupervised, clean videos were not used during training. our model was trained to estimate the clean signal in each input using only the noisy video
% Given a set of input frames coordinate $\{{\mathbf{P}}\}$ and a set of noisy frames $\{\tilde{\mathbf{I}}\}$ where $P = (P_{t-N},...,P_t,...,P_{t+N}) $ and $\tilde{\mathbf{I}} = ({\tilde{\mathbf{I}}}_{t-N},...,{\tilde{\mathbf{I}}}_t,...,{\tilde{\mathbf{I}}}_{t+N}) $, our image generator is trained to generate the corresponding set of frames at each position $\{\hat{\mathbf{I}}\}$ and the denoiser generates a clean version $\mathbf{I}$ of the center noisy frame. We trained both models using the $L1$ loss function. The image generator was trained to minimize the absolute difference between the generated neighboring frames and the noisy frames while the denoiser minimized the absolute difference between the denoised middle frame and the noisy middle frame. Both models were trained using ADAM optimizer with the default settings, and the learning rate was set to $1e-4$ throughout the training.
% \begin{equation}
% L = \frac{1}{T} \sum_{t=1}^{T} ||\hat{\mathbf{I}}_t - \tilde{\mathbf{I}}_t|| + ||\mathbf{I} - \tilde{\mathbf{I}}||
% \end{equation}
% \textbf{Spatial information:} At this stage of our network (Image generator), the network learns spatial details of neighboring frames. At this stage, $2K+1$ frame coordinates are passed to the network in a batch. This stage performs two actions; the first is that it generates the corresponding color pixel at each position and the second is that the generated pixel value has lesser noise compared to the target noisy frames. The network depth is 7 plus the output layer. The learning rate was set to 1e-4 throughout the training. For colored videos, we trained this sub-network first for 10 epochs before training simultaneously with the second subnetwork. For the calcium imaging dataset, both the Image generator and Denoiser were trained together from the beginning.
% \textbf{Temporal information.} Many video denoising algorithms \cite{arias2018video, buades2016patch, frame2frame, dvdnet} use explicit motion estimation to compensate for motion in the video sequence. However, recent CNN-based video denoising approaches \cite{udvd, videnn, fastdvdnet} do not perform any explicit motion compensation and have demonstrated that CNN is able to automatically perform motion estimation by exploiting temporal information in neighboring frames. Based on this approach, we pass as input stacked of neighboring frames output from the image generator to denoise the central. Similar to DeepInterpolation \cite{deepinter}, we skipped the central frame from the input and we also omitted N-pre and N-post frames before and after the central frames respectively. Hence, the final denoise input is $2K -2N$ frames stacked together. This is so that the model does not learn an identical noise mapping. The denoiser was trained using $L1$ loss difference between the output and the central frame. The learning rate used was 0.0001 and this was fixed throughout the training. Our architecture has been implemented in PyTorch \cite{paszke2019pytorch}
Given the unsupervised nature of our architecture, the network optimization problem is highly non-convex with a vast parameter search space. To navigate this challenge, we propose a two-step network optimization strategy that exploits the structural similarity between neighboring frames to reconstruct the central frame.
\begin{figure}[t!]
\centering
\includegraphics[width=1\textwidth]{images/LF_illu.png}
\caption{The illustration of $\mathcal{L}_F$}
\label{fig:illu}
\end{figure}
\subsection{First Stage: Joint Training of the Feature Generator $\mathcal{F}_\theta$ and Denoiser $\mathcal{D}_\phi$}
In the first stage, we jointly train the feature generator $\mathcal{F}_\theta$ and the denoiser $\mathcal{D}_\phi$ in an end-to-end fashion. The objective function for this stage is composed of two parts. The first part is the $l_1$ loss, which measures the difference between the denoised central frame $\hat{I}_B$ and the central frame $I^c$ of the input batch of noisy frames $\{I_t | t = 1, 2, \dots, B\}$:
\begin{equation}
\mathcal{L}_{\mathcal{D}} = ||\hat{I}_B - I^c||
\end{equation}
The second part of the objective function ensures that the feature maps generated by $\mathcal{F}_\theta$ capture the image information. This is achieved by enforcing similarity between the central channels of each feature map and the corresponding noisy frame:
\begin{equation}
\mathcal{L}_{\mathcal{F}} = \frac{1}{B}\sum_{t=1}^B|| F_t^{c} - I_t||
\end{equation}
where $F_t^{c}$ is the central channels of each feature map $\mathcal{F}_\theta(\gamma(G_t))$ and $I_t$ is the corresponding noisy frame, as shown in Fig.~\ref{fig:illu} (a).
The final loss function for the first stage is the sum of these two losses:
\begin{equation}
\mathcal{L}_{1} = \mathcal{L}_{\mathcal{D}} + \lambda_1 \mathcal{L}_{\mathcal{F}}
\end{equation}
where $\lambda_1$ is a weight parameter that controls the trade-off between the two terms. To ensure our model doesn't simply learn to reproduce the noise present in the input, we train the network for approximately 2000 epochs.
\subsection{Second Stage: Training the Refine-Net $\mathcal{R}_\eta$}
In the second stage, we focus on training the Refine-Net $\mathcal{R}_\eta$. Unlike in the first stage, where $\mathcal{F}_\theta$ and $\mathcal{D}_\phi$ were trained together, here we fix $\mathcal{F}_\theta$ and $\mathcal{D}_\phi$ and solely train $\mathcal{R}_\eta$.
We use the coordinates of the central frame $G_t^c$ as input to $\mathcal{R}_\eta$, which outputs a refined denoised image $\hat{I}_R$. As this network is designed to further improve the quality of the denoised frames, the loss function for this stage is defined to measure the difference between the output of $\mathcal{R}_\eta$ and both the noisy central frame $I_t^c$ and the denoised central frame $\hat{I}_B$ from the first stage. Specifically, the loss function is defined as:
\begin{equation}
L_{2} = \lambda_2||\mathcal{R}\eta(G_t^c) - I^c|| + \lambda_3||\mathcal{R}\eta(G_t^c) - \hat{I}_B||,
\label{siren_loss}
\end{equation}
where $\lambda_2$ and $\lambda_3$ are weight parameters controlling the contribution of each term. These parameters help balance the network's objectives of reducing noise (by making the output similar to $I_B^c$) and preserving details (by making the output similar to $I_t^c$).
This strategy allows the Refine-Net to leverage the advantages of both the noisy and denoised frames, by enhancing details and suppressing noise. The training of $\mathcal{R}\eta$ is performed until satisfactory results are obtained, typically for about 2000 epochs. It's important to note that the second stage training does not affect the training of $\mathcal{F}_\theta$ and $\mathcal{D}_\phi$, which is crucial for preserving the generalization capability of the whole framework.
Our two-stage optimization can effectively remove the noise in the video frames. The first image on the left represents the initial state, which is typically a noisy video frame. The joint training of the feature generator $\mathcal{F}_\theta$ and the denoiser $\mathcal{D}_\phi$ shows a noticeable reduction in noise (middle frame), but some blurriness might still be present due to the low $L$ we used in our feature generator. The right image showcases the result of the refining stage $\mathcal{R}_\eta$. At this stage, the high-frequency details that might have been lost in the denoising process are recovered, resulting in a crisp, clean frame that retains the original structure and details of the scene, demonstrating the step-by-step improvement of our method.
\section{Experiments}
In this section, we evaluate the performance of our proposed method through extensive experiments. Our method is tested on a variety of video sequences with different types of noise, and the results are compared with state-of-the-art denoising methods to demonstrate its effectiveness.
\subsection{Datasets and Setup}
% A variety of benchmark datasets, including the DAVIS dataset \cite{davis}, were employed to rigorously evaluate our algorithm. These datasets encompass a diverse range of video sequences, each exhibiting unique content and characteristics, thereby offering a comprehensive testing ground for our method under a multitude of conditions. To challenge our method's robustness, we add variety of noise types, such as Gaussian, Salt-and-Pepper, and Poisson noise, to the clean video sequences in Davis dataset and utilize the synthetic noisy sequences as input.
% Furthermore, we applied our algorithm to the significantly noisy calcium imaging data. These were locally sourced recordings from freely behaving transgenic mice involved in cocaine/sucrose self-administration experiments. Single-channel epifluorescent miniscopes were used for the recordings, which were subsequently processed with a motion correction algorithm to compensate for translational motion artifacts.
Our approach is tested on a variety of datasets, encompassing both synthetic and real-world scenarios, to provide a comprehensive evaluation of its performance. Synthetic data are derived from established benchmarks and intentionally corrupted with diverse types of noise, while real-world data are sourced from calcium imaging experiments. Moreover, we outline the specifics of our computational setup and training parameters, detailing the choices that were made to optimize our model's performance. This thorough experimental setup is aimed at providing a robust assessment of our proposed video denoising method and its potential applicability to different types of video data.
\paragraph{Synthetic.}
We employed a variety of benchmark datasets, including the DAVIS dataset \cite{davis}, and SET8\cite{dewil2021self} videos s captured with GoPro camera, to provide a comprehensive evaluation of our algorithm. These datasets include a diverse array of video sequences, each with unique content and characteristics, thus allowing us to test our method under numerous conditions. To challenge our algorithm's robustness, we deliberately introduced various types of noise to the clean video sequences. Gaussian noise was added as per the formula:
\begin{equation}
N(\textbf{c}|\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(\textbf{c}-\mu)^2}{2\sigma^2}}
\end{equation}
where $\textbf{c}$ is a pixel value, and $\mu$ and $\sigma^2$ are the mean and variance of the Gaussian distribution, respectively \cite{laine2019high}. In our experiment, we set $\mu=0$. Poisson noise was added according to the equation:
\begin{equation}
P(\textbf{c}|\lambda) = \frac{\lambda^\textbf{c} e^{-\lambda}}{\textbf{c}!},
\end{equation}
where $\lambda$ is the expected number of occurrences \cite{laine2019high}. Salt-and-pepper noise, also known as impulse noise, was added by randomly selecting pixels and setting their values to the minimum and maximum values representing 'salt' and 'pepper' respectively \cite{laine2019high}.
\paragraph{Real-World.}
In addition to the synthetic datasets, we also applied our algorithm to real-world, highly noisy calcium imaging data, as shown in Fig.~\ref{fig:real}. These were locally sourced recordings from freely behaving transgenic mice engaged in cocaine/sucrose self-administration experiments. The recordings were captured using single-channel epifluorescent miniscopes and were subsequently processed using a motion correction algorithm to adjust for translational motion artifacts. This dataset represents the practical complexity and noise levels that are often present in real-world scenarios, further challenging our algorithm's ability to effectively denoise video sequences.
Our proposed method was implemented using the PyTorch framework and trained on an NVIDIA A100 GPU. We employed the Adam optimizer during the training process, with an initial learning rate of $1e-4$ set for the first stage and $1e-5$ for the second stage. Both learning rates were reduced by a factor of 10 every 100 epochs. In our loss function, we set $\lambda_1=0.1$ and $\lambda_2=1.0$ as the balancing factors for our dual-term loss.
\subsection{Quantitative Evaluation}
% For quantitative evaluation, we use the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) as our metrics. These metrics are widely used in the image and video processing community, and provide a fair basis for comparison.
% The results in Table \ref{tab:results} show that our method outperforms other state-of-the-art methods on all datasets and under all types of noise. The superior performance of our method is attributed to its ability to effectively learn the spatial-temporal information from the video sequence and leverage it to remove the noise, while preserving the high-frequency details in the frames.
\begin{figure}[t!]
\centering
\includegraphics[width=1\textwidth]{images/vis_compare.png}
\caption{\textbf{Visual comparison of denoising results on the DAVIS dataset:} The figure illustrates denoising results from three types of noise: Gaussian, Poisson, and Impulse. Each row presents the denoised results from our method (right) in comparison with UDVD (middle), along with the original noisy frame (left). It can be observed that our method achieves superior noise removal while preserving intricate details, even in the presence of diverse noise types, outperforming UDVD\cite{udvd}.}
\label{fig:syn}
\end{figure}
To facilitate a quantitative evaluation of our approach, we employ two widely-accepted metrics in the image and video processing community: the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). These measures allow us to objectively compare our results with those of the current state-of-the-art unsupervised video denoising algorithm, UDVD\cite{udvd}.
The outcomes, as displayed in Table \ref{tab:results}, highlight that our method consistently outperforms UDVD across all tested datasets and noise types. This superior performance is primarily attributed to our approach's effective utilization of spatial-temporal information from the video sequence. Our algorithm leverages this information to efficiently eliminate noise while simultaneously preserving high-frequency details within the frames.
When examining the visual comparison results, as shown in Fig.~\ref{fig:syn}, it becomes clear that UDVD often struggles to effectively remove noise when dealing with short input sequences. This challenge is particularly evident when confronted with Poisson and Impulse noise types, where UDVD tends to produce noticeable artifacts. Conversely, our method shows remarkable resilience even in these demanding situations. We tested our approach on ten short video sequences, each consisting of ten frames, and the results consistently demonstrated our method's superior noise removal capability and robustness.
Overall, the quantitative and qualitative evaluations underscore the potential of our proposed video denoising method as a robust and effective solution, particularly for applications dealing with short, noisy video sequences.
\newcommand{\SH}{\multicolumn{1}{c}{PSNR} & \multicolumn{1}{c}{SSIM}} % subhead
\begin{table}[htbp]
\label{tab:results}
\small
\centering
\setlength{\abovetopsep}{12pt}
% \setlength{\tabcolsep}{2pt}
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{*{6}{l}}
\toprule
& & \multicolumn{2}{c}{DAVIS} & \multicolumn{2}{c}{SET8} \\
\cmidrule(lr){3-4}
\cmidrule(lr){5-6}
& & \SH & \SH \\
\midrule
\multirow{2}{*}{\parbox{1.2cm}{Gaussian $\sigma=30$}} & UDVD: & 27.9 & \textbf{0.8} & 27.40 & 0.79 \\
& Ours & \textbf{28.49} & 0.78 & \textbf{29.01} & \textbf{0.80} \\
\midrule
\multirow{2}{*}{\parbox{1.2cm}{Gaussian $\sigma=50$}} & UDVD: & 24.3 & 0.63 & 25.27 & 0.73 \\
& Ours & \textbf{28.86} & \textbf{0.74} & \textbf{27.36} & \textbf{0.76} \\
\midrule
\multirow{2}{*}{\parbox{1.2cm}{Poisson $\lambda=30$}} & UDVD: & 27.7 & \textbf{0.82} & 27.84 & \textbf{0.87} \\
& Ours & \textbf{29.81} & \textbf{0.82} & \textbf{29.45} & 0.81 \\
\midrule
\multirow{2}{*}{\parbox{1.2cm}{Poisson $\lambda=50$}} & UDVD: & 24.3 & 0.77 & 26.30 & \textbf{0.85} \\
& Ours & \textbf{27.44} & \textbf{0.79} & \textbf{29.05} & 0.81 \\
\midrule
\multirow{2}{*}{\parbox{1.2cm}{Impulse $\alpha=0.2$}} & UDVD: & 19.1 & 0.23 & 22.06 & 0.67 \\
& Ours & \textbf{22.61} & \textbf{0.66} & \textbf{28.57} & \textbf{0.81} \\
\midrule
\multirow{2}{*}{\parbox{1.2cm}{Impulse $\alpha=0.3$}} & UDVD: & 16.9 & 0.14 & 19.02 & 0.49 \\
& Ours & \textbf{20.66} & \textbf{0.62} & \textbf{28.51} & \textbf{0.78} \\
\bottomrule
\end{tabular}
\vspace{5pt}
\caption{\textbf{Denoising results on synthetic noise.} This table presents the PSNR and SSIM values of our model and UDVD on the DAVIS and SET8 video datasets. While UDVD may achieve higher SSIM score in some cases, it does not effectively eliminate the noise, resulting in a generally lower PSNR value. This underscores the superior noise removal capacity of our model, which consistently delivers higher PSNR values across different noise types and datasets.}
\end{table}
\subsection{Qualitative Evaluation}
A visual comparison between our method and UDVD on the highly noisy calcium imaging sequences further underscores our superior performance, as shown in Fig.~\ref{fig:real}. In the noisy frames, it can be challenging to distinguish individual cells due to high levels of noise. UDVD, while effective in reducing some noise, often blurs the intricate cellular structures, making it difficult to identify individual cells.
In contrast, our approach not only removes the noise effectively but also preserves the intricate cellular structures, allowing for better visualization and identification of individual cells. This difference is particularly notable in regions with a high density of cells, where our method is able to maintain the distinct boundaries between cells, whereas UDVD tends to blur them together. This visual comparison highlights our method's ability to handle real-world data with significant noise, offering promising potential for applications in biological and medical imaging.
\begin{figure}[t!]
\centering
\includegraphics[width=0.8\textwidth]{images/real.png}
\caption{\textbf{Visual comparison on calcium imaging sequences}: The left column shows the original noisy frames, the middle column represents the denoising results from UDVD, and the right column presents the results from our method. It is evident from the comparison that our method is superior at noise reduction, while also maintaining the detailed cellular structures, thereby enhancing the visualization and identification of individual cells.}
\label{fig:real}
\end{figure}
\subsection{Ablation Study}
% We conduct an ablation study to investigate the contribution of each component in our method. When we remove the refining stage $\mathcal{R}_\eta$, the denoised frames are slightly blurry due to the low $L$ in the feature generator. However, with the refining stage, our method can recover the high-frequency details, proving its importance. The results of this ablation study are presented in Table \ref{tab:ablation}.
% \begin{table}[h]
% \small
% \begin{center}
% \begin{tabular}{c|c|c|}
% \hline
% \centering
% Method & PSNR & SSIM\\
% \hline
% Denoised & 30.50 & 0.81 \\
% Denoised+refine & 30.90 & 0.84 \\
% \hline
% \end{tabular}
% \end{center}
% \caption{\textbf{Denoising results on color video.} The result shows the PSNR/SSIM values of each trained on resized single videos (boat, tennis, boat) in the DAVIS dataset corrupted with Gaussian of noise level $\sigma$ = 30. The UDVD was trained using MSE loss as the noise model was unknown. The text in bold shows the model with the highest PSNR values. }
% \label{tab:ablation}
% \end{table}
\begin{figure}[t!]
\centering
\includegraphics[width=1\textwidth]{images/ablation1.png}
\caption{\textbf{Ablation study visual results.}}
\label{fig:ablation}
\end{figure}
We conduct an ablation study to understand the contribution of each component in our method. Notably, when we omit the refining stage $\mathcal{R}_\eta$, the denoised frames tend to be slightly blurry due to the low $L$ in the feature generator $\mathcal{F}_\theta$. However, with the incorporation of the refining stage, our method is able to effectively recover high-frequency details, thereby underscoring its crucial role in enhancing the overall quality of the denoised frames. We show the visual comparison between each of network variants in Fig.~\ref{fig:ablation}. A detailed presentation of the ablation study results is provided in Table \ref{tab:ablation}.
\begin{table}[h]
\small
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
\centering
Method & PSNR & SSIM\\
\hline
$\mathcal{F}_\theta + \mathcal{D}_\phi$ & 30.50 & 0.81 \\
$\mathcal{F}_\theta + \mathcal{D}_\phi + \mathcal{R}_\eta$ & \textbf{30.90} & \textbf{0.84} \\
\hline
\end{tabular}
\end{center}
\vspace{5pt}
\caption{\textbf{Ablation study on color video.} The table presents the PSNR/SSIM values of each model variant when trained on DAVIS dataset corrupted with Gaussian noise of level $\sigma$ = 30. The UDVD was trained using MSE loss as the noise model was unknown. The text in bold shows the model variant with the highest PSNR values. }
\label{tab:ablation}
\end{table}
% \subsection{Datasets}
% We show the generalization of our approach to varying datasets: natural video, raw videos, calcium imaging, and fluorescence microscopy.
% \textbf{Calcium imaging. } We trained and evaluated our approach on calcium imaging recordings collected locally in freely behaving transgenic mice (Drd1-Cre and Drd2-Cre) undergoing cocaine/sucrose self-administration experiments. Calcium imaging was performed using single-channel epifluorescent miniscopes The recording was processed to remove low and high spatial frequency components that do not correspond to cells in focus followed by frame-by-frame motion correction algorithm \cite{thevenaz1998pyramid} to account for translational motion artifacts and the movie was normalized to visualize changes in fluorescence. This data has no ground truth clean images.
% \textbf{Fluorescence microscopy.} Similar to UDVD, we also applied our model to fluorescence microscopy. We used the GOWT1 cells and mesenchymal stem cells of fluorescence microscopy recordings of live cells obtained from \cite{ulman2017objective}. Similar to the calcium imaging, this data has no ground truth clean recordings.
% \textbf{Raw videos.}
% We applied our model using the raw video dataset from \cite{rvidenet}. The dataset was captured using a surveillance camera at different ISO levels, and it contains 11 indoor scenes with 7 frames each. The clean version was estimated by averaging multiple frames obtained from different noise realizations.
% \textbf{Natural videos.} To experiment with our approach to natural videos, we used single video sequences in the DAVIS dataset \cite{pont20172017} for our experiment. To generate synthetic noisy video, we added iid Gaussian noise to each frame in the clean video sequence and saved the noisy videos on file. These noisy frames are used during training as the target and the noise values are fixed throughout the training phase. Unlike most approaches, noise is added at every iteration during training and this means that for each pass, the value of the noise changes and this helps the model not learn an identity mapping to the varying noise values. We used 3 videos from the DAVIS dataset: boat, tennis, and flamingo.
% \subsection{Results}
% \textbf{Denoising on calcium imaging} We applied our method to one-photon calcium imaging recording. We trained with 100 frames from the dataset without any ground-truth clean image and we did not use any data augmentation. For this dataset, we used $2K+1$ frame coordinates as input to the image generator and we set $K=7$. This is because the neuron firing occurs in split seconds and it requires sufficient temporal information from neighboring frames to be able to pick the fired neuron. We skipped the central frame and 3 pre and post frames before and after the central frame respectively. The number of inputs frame to the denoiser was $2K-2N = 8$ frames reshaped to $1 \times 8 \times H \times W $. The denoiser outputs the denoised central frame of shape $1 \times 1 \times H \times W $
% \begin{figure}[t!]
% \centering
% \includegraphics[width=1\textwidth]{images/muscplot.png}
% \caption{\textbf{Denoise result on one-photon calcium imaging}}
% \label{fig:musc_imgs}
% \end{figure}
% \textbf{Natural Video.} We trained our model on three video sequences in the DAVIS training set. Different from the \cite{udvd, fastdvdnet, dvdnet, self-supervised}, we did not add iid Gaussian noise during training, instead, we added the noise to the frames and saved it as JPEG files. We experimented on the JPEG resized image, using a resize factor of 2. For a fair comparison with other approaches, we also used the resized image to run train their models. We compared our approach with UDVD \cite{udvd} and \cite{mf2f}. In Table \ref{ }we show the PSNR of the results obtained on the video sequence with a standard deviation $\sigma$ of 30 using our approach and other approaches.
% \textbf{Denoising on the short video sequence.} We show that our model does not require a very large dataset to train. Since we are using the neural representation approach, our model overfits the data and does not require a very long video sequence to remove the noise. We show that our model can denoise when trained with as little as 10 frames in the natural video without supervision with corresponding clean video frames. To the best of our knowledge, some existing unsupervised approaches either have to employ data augmentation such as flipping and time reversal to increase the data size \cite{udvd} or fine-tunes backbone model pre-trained with supervision as in the case of \cite{mf2f}. Table \ref{} shows the PSNR obtained on 10 frames and comparison with other approaches using the same number of frames during training.
% \begin{table}[h]
% \small
% \begin{center}
% \begin{tabular}{c|c|c|c}
% \hline
% \centering
% Method & Boat & Tennis & Bus\\
% \hline
% UDVD & 33 / 0.81 & 33 / 0.81 & 33 / 0.81\\
% MF2F & 33 / 0.81 & 33 / 0.81 & 33 / 0.81\\
% Ours & \textbf{33} / \textbf{0.81} & \textbf{33} / \textbf{0.81} & \textbf{33} / \textbf{0.81}\\
% \hline
% \end{tabular}
% \end{center}
% \caption{\textbf{Denoising results on color video.} The result shows the PSNR/SSIM values of each trained on resized single videos (boat, tennis, boat) in the DAVIS dataset corrupted with Gaussian of noise level $\sigma$ = 30. The UDVD was trained using MSE loss as the noise model was unknown. The text in bold shows the model with the highest PSNR values. }
% \label{kid}
% \end{table}
% \begin{table}[h]
% \begin{center}
% \begin{tabular}{cllllll}
% \toprule
% Noise type & Method & Boat & Bus & Camel & Snow & Cows \\
% \midrule
% {\parbox{1.5cm}{Gaussian $\sigma=30$}} & UDVD & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8\\
% & Ours & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8 \\
% & Ours+refine & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8 \\
% \hline
% {\parbox{1.5cm}{Gaussian $\sigma=50$}} & UDVD & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8\\
% & Ours & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8 \\
% & Ours+refine & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8 \\
% \hline
% {\parbox{1.5cm}{Poisson $\lambda=30$}} & UDVD & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8\\
% & Ours & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8 \\
% & Ours+refine & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8 \\
% \hline
% {\parbox{1.5cm}{Poisson $\lambda=50$}} & UDVD & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8\\
% & Ours & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8 \\
% & Ours+refine & 30/0.8 & 30/0.8 & 30/0.8 & 30/0.8 &30/0.8 \\
% \bottomrule
% \end{tabular}
% \caption{\textbf{Denoising results on color video.} The result shows the PSNR/SSIM values of each trained on resized single videos (boat, tennis, boat) in the DAVIS dataset corrupted with Gaussian of noise level $\sigma$ = 30. The UDVD was trained using MSE loss as the noise model was unknown. The text in bold shows the model with the highest PSNR values. }
% \end{center}
% \label{davis_result}
% \end{table}
% omlimit{5pt}
% \newcommand{\SH}{\multicolumn{1}{c}{PSNR} & \multicolumn{1}{c}{SSIM}} % subhead
% \begin{table}[htbp]
% \tiny
% \centering
% \setlength{\abovetopsep}{12pt}
% % \setlength{\tabcolsep}{2pt}
% \renewcommand{\arraystretch}{1.2}
% \caption{ARMA order selection}
% \begin{tabular}{*{14}{l}}
% \toprule
% & & \multicolumn{2}{c}{BOAT} & \multicolumn{2}{c}{BUS}
% & \multicolumn{2}{c}{SURF} & \multicolumn{2}{c}{RAFTING} & \multicolumn{2}{c}{HYPERSMOOTH} & \multicolumn{2}{c}{SNOWBOARD} \\
% \cmidrule(lr){3-4}
% \cmidrule(lr){5-6}
% \cmidrule(lr){7-8}
% \cmidrule(lr){9-10}
% \cmidrule(lr){11-12}
% \cmidrule(lr){13-14}
% & & \SH & \SH & \SH & \SH & \SH & \SH \\
% \midrule
% \multirow{2}{*}{\parbox{0.7cm}{Gaussian $\sigma=30$}} & UDVD: & 4936.9 & 4952.8 & 5288.5 & 5304.4 & 4528.1 & 4544.0 & 4733.0 & 4748.9 & 5312.9 & 5328.8 & 5242.3 & 5242.3\\
% & Ours & 30.90 & 0.77 & 26.16 & 0.74 & 4533.6 & 4549.5 & 4733.0 & 4748.9 & 5312.9 & 5328.9 & 5242.4 & 5258.3 \\
% \midrule
% \multirow{2}{*}{\parbox{0.7cm}{Gaussian $\sigma=50$}} & UDVD: & 4936.9 & 4952.8 & 5288.5 & 5304.4 & 4528.1 & 4544.0 & 4733.0 & 4748.9 & 5312.9 & 5328.8 & 5242.3 & 5242.3\\
% & Ours & 28.69 & 0.75 & 24.24 & 0.69 & 4533.6 & 4549.5 & 4733.0 & 4748.9 & 5312.9 & 5328.9 & 5242.4 & 5258.3 \\
% \midrule
% \multirow{2}{*}{\parbox{0.7cm}{Poisson $\lambda=30$}} & UDVD: & 4936.9 & 4952.8 & 5288.5 & 5304.4 & 4528.1 & 4544.0 & 4733.0 & 4748.9 & 5312.9 & 5328.8 & 5242.3 & 5242.3\\
% & Ours & 30.90 & 0.84 & 30.37 & 0.82 & 4533.6 & 4549.5 & 4733.0 & 4748.9 & 5312.9 & 5328.9 & 5242.4 & 5258.3 \\
% \midrule
% \multirow{2}{*}{\parbox{0.7cm}{Poisson $\lambda=50$}} & UDVD: & 4936.9 & 4952.8 & 5288.5 & 5304.4 & 4528.1 & 4544.0 & 4733.0 & 4748.9 & 5312.9 & 5328.8 & 5242.3 & 5242.3\\
% & Ours & 28.72 & 0.80 & 26.90 & 0.77 & 4533.6 & 4549.5 & 4733.0 & 4748.9 & 5312.9 & 5328.9 & 5242.4 & 5258.3 \\
% \midrule
% \multirow{2}{*}{\parbox{0.7cm}{Impulse $\alpha=0.2$}} & UDVD: & 4936.9 & 4952.8 & 5288.5 & 5304.4 & 4528.1 & 4544.0 & 4733.0 & 4748.9 & 5312.9 & 5328.8 & 5242.3 & 5242.3\\
% & Ours & 4944.3 & 4960.3 & 5288.6 & 5304.6 & 4533.6 & 4549.5 & 4733.0 & 4748.9 & 5312.9 & 5328.9 & 5242.4 & 5258.3 \\
% \midrule
% \multirow{2}{*}{\parbox{0.7cm}{Impulse $\alpha=0.3$}} & UDVD: & 4936.9 & 4952.8 & 5288.5 & 5304.4 & 4528.1 & 4544.0 & 4733.0 & 4748.9 & 5312.9 & 5328.8 & 5242.3 & 5242.3\\
% & Ours & 4944.3 & 4960.3 & 5288.6 & 5304.6 & 4533.6 & 4549.5 & 4733.0 & 4748.9 & 5312.9 & 5328.9 & 5242.4 & 5258.3 \\
% \bottomrule
% \end{tabular}
% \label{Selection of ARMA orders}
% \end{table}
\section{Discussion}
In this study, we proposed an innovative unsupervised video denoising framework that leverages a two-stage optimization strategy and a novel refine-net that employs the SIREN architecture as its backbone. Our model has been demonstrated to be highly effective across a range of video datasets, from natural videos to challenging fluorescence microscopy and calcium imaging recordings.
One key strength of our approach is its adaptability to various noise types and levels without needing prior knowledge about the noise characteristics. This flexibility is primarily due to our unsupervised training strategy which enables the model to learn directly from the noisy frames and adapt to the specific noise present in the video.
Moreover, the use of SIREN in our refine-net is another significant advantage, as it excels in representing complex patterns and structures in the data, making it particularly suitable for refining the denoised output.
However, like all models, ours also has certain limitations. The denoising quality could be affected by the value of $L$ in the feature generator. Setting a lower $L$ might result in blurry output frames, whereas a higher $L$ could potentially lead to overfitting. Therefore, the selection of $L$ requires careful tuning based on the specific characteristics of the dataset.
Another challenge arises when dealing with extremely noisy video sequences. Our model might struggle with recovering high-quality frames from these videos as the noise might overwhelm the actual content in the frames. Future research could focus on developing more robust denoising models that can handle higher noise levels.
In conclusion, our proposed video denoising framework presents a promising approach to tackle the video denoising problem, and we hope that our work will inspire future research in this direction.
\appendix
\section{Network Architecture and Training}
\subsection{Feature Generator}
\begin{table}[h]
% \small
\begin{center}
\begin{tabular}{ccc}
\hline
\centering
Name & $N_{out}$ & Function\\
\hline
Input & $2\times C_{coord} \times L$ & $\gamma(p_t)$ \\
conv\_0 & 256 & Convolution 1 x 1 \\
conv\_1 & 256 & Convolution 3 x 3 \\
conv\_2 & 256 & Convolution 3 x 3 \\
conv\_3 & 256 & Convolution 3 x 3 \\
conv\_4 & 256 & Convolution 3 x 3 \\
conv\_5 & $C_{feat}$ & Convolution 3 x 3 \\
\hline
\end{tabular}
\end{center}
\caption{\textbf{Network architecture of the feature generator}}
\label{tab:feat}
\end{table}
The feature generator $\mathcal{F}_\theta$ takes in a batch of sampled coordinated grids $\{G_t | t = 1,...,B\}$ corresponding to a batch of noisy frames $\{I_t | t = 1, 2, \dots, B\}$, where $B$ is the batch size.
% that contains $2K+1$ frames and $K$ is the number of pre and post frames from the central frame at $t$.
In our experiment, we set $B = 5$, and we used 2 consecutive frame coordinate grids before and after the central frame, resulting in 5 frame coordinate grids including the central frame itself. Our goal is to generate a batch of feature map, denoted as $\{F_t | t = 1,...,B\}$, corresponding to each of the input frame.
Initially, a positional encoding $\gamma(.)$ is applied to these batched coordinate grids before they are processed by the feature generator. $\gamma(.)$ expands the low-dimensional coordinates into a high-dimension feature space using the following equation:
\begin{equation}
\gamma(\mathbf{p}_t) = [\mathrm{sin}(2^0 \pi \mathbf{p}_t), \mathrm{cos}(2^0 \pi \mathbf{p}_t), ..., \mathrm{sin}(2^{L-1} \pi \mathbf{p}_t), \mathrm{cos}(2^{L-1} \pi \mathbf{p}_t)]
\label{pos}
\end{equation}
where $L$ signifies the frequency level of output feature map. The output size of $\gamma(.)$ depends on the size of input coordinate $C_{coord}$ and $L$.
% Each pixel $p_t = (x, y,t)$ in the coordinate grid is passed through a positional encoding $\gamma(.)$ in \eqref{pos}, which maps the low-dimensional coordinates into high-dimension of size $2(x*L)$, where $L$ represents the frequency level. We set $L=30$, hence the resulting dimension is 180. We also added the original pixel coordinates to the high-dimension output from the positional encoding. Finally, the input to the feature generator $\mathcal{F}_\theta$ has a dimension of $5 \times 183 \times H \times W$.
Next, the feature generator $\mathcal{F}_\theta$ takes positional encoded grid $\gamma(G_t)$ as input, and output the feature map $F_t\in\mathbb{R}^{H \times W \times C_{feat}}$.
% The feature generator $\mathcal{F}_\theta$ has 6 convolution layers with 256 filters in each layer except for the last layer. The output of the last layer is $C$ a feature map where $C = 3 \times B$. Each layer has a kernel size of 3. For calcium imaging, the output of $\mathcal{F}_\theta$ is $C = 1 \times B$.
\subsection{Denoisier}
\begin{table}[h]
% \small
\begin{center}
\begin{tabular}{ccc}
\hline
\centering
Name & $N_{out}$ & Function\\
\hline
Input & $B*C_{feat}$ & \\
conv\_0 & 256 & Convolution 1 x 1 \\
conv\_1 & 96 & Convolution 1 x 1 \\
conv\_2 & $C_{in}$ & Convolution 1 x 1 \\
\hline
\end{tabular}
\end{center}
\caption{\textbf{Network architecture of the Denoiser $\mathcal{D}_\phi$}}
\label{tab:denoise}
\end{table}
The network architecture for the Denoiser $\mathcal{D}_\phi$ is described in Table \ref{tab:denoise}.
% The output from the feature generator is used as input for $\mathcal{D}_\phi$.
The batched feature map output from $\mathcal{F}_\theta$ is concatenated to form an input dimension of $1 \times B*C_{feat} \times H \times W$, and then fed into $\mathcal{D}_\phi$. The output of the Denoiser network is $C_{in}$, where $C_{in} =3$ for color images and $C_{in}=1$ for grayscale images.
% The output of the feature generator is passed as input to the denoise-net $\mathcal{D}_\phi$. The batched feature map output from $\mathcal{F}_\theta$ is concatenated to form an input dimension of $1 \times B*C \times H \times W$. The output of the denoise-net is $k$ where $k =3$ for RGB color of the middle frame $t$. For calcium imaging, $k=1$ which is the color channel of the grayscale image. The denoise net has 3 convolution layers with the 256, 96, and $k$ filters for the first, second, and third layers, respectively, as shown in Table \ref{tab:denoise}. All the layers of the model have a kernel size of 1 except for the first layer with a kernel size of 3.
\subsection{Refine-Net}
\begin{table}[h]
% \small
\begin{center}
\begin{tabular}{ccc}
\hline
\centering
Name & $H_{out}$ & Function\\
\hline
Input & $C_{coord}$ & \\
mlp\_0 & 256 & Linear \\
mlp\_1 & 256 & Linear \\
mlp\_2 & 256 & Linear \\
mlp\_3 & 256 & Linear \\
mlp\_out & $C_{in}$ & Linear \\
\hline
\end{tabular}
\end{center}
\caption{\textbf{Network architecture of the Refine-Net.}}
\label{tab:ablation}
\end{table}
The network architecture of the Refine-Net $\mathcal{F}_\theta$ is depicted in Table \ref{tab:ablation}. The backbone of this network is a Sinusoidal Representation Networks (SIREN) as described in \cite{siren}. This network takes as its input the coordinates of the denoised output from the denoise-net and produces a refined denoised central frame where each pixel coordinate is denoted by $\textbf{p}_t = (x,y,t)$.
The Refine-Net is composed of fully connected layers, which include 4 hidden layers, with the activation function at each layer being a sine function. Each hidden layer contains 256 units, and the output layer is composed of $C_{in}$ nodes, where $C_{in}=3$ for RGB images and $C_{in}=1$ for grayscale images. The refined denoised output is generated by processing the denoised image's coordinates through this network, resulting in a more precise final image.
% The refine net $\mathcal{F}_\theta$ Sinusoidal Representation Networks (SIREN) \cite{siren} as the backbone. The network takes as input the coordinates of the denoised output from the denoise-net and outputs the refined denoised central frame where each pixel coordinate is $p_t = (x,y,t)$. The network is made up of fully connected layers with 4 hidden layers, and the activation at each layer is a sine function. Each linear has 256 hidden units, while the output layer has $k$ nodes, where $k=3$ for RGB and $k=1$ for grayscale.
\subsection{Training Details}
% Unlike conventional denoising algorithms \cite{udvd, fastdvdnet, self-supervised, dncnn} on synthetic noise, we did not add noise during training. We used fixed noise values throughout the training. We experimented with three different noise types; Gaussian, Poisson, and Impulse types. For Gaussian and Poisson noise types, we used 30 and 50 noise levels and we used noise levels of 0.2 and 0.3 for Impulse noise. To create the noisy version of the clean video sequence, we added noise to frames in the video sequences and saved the noisy frames to a file, this allows us to have a fixed noise value for each frame during training. We simply load these noisy video frames during training instead of adding noise to clean images. We did not use any data augmentation during training.
Our proposed network is trained using a two-stage process to effectively leverage the functionalities of each individual component - the feature generator, denoiser, and refiner. In the first stage, we train the feature generator and denoiser together as a combined model. Here, we optimize for the reduction of noise in the video frames. The aim at this stage is to extract relevant spatio-temporal features from the noisy inputs and then utilize those features to output an initial, denoised estimate of the central frame of the batch.
In the second stage of training, we keep the weights of the feature generator and denoiser fixed, and focus on training the refiner. The refiner uses the denoised output from the previous stage and enhances it further, particularly focusing on the high-frequency details.
% The loss function for this stage is the structural similarity (SSIM) loss, which promotes preservation and enhancement of structural details in the video frames.
This two-stage training approach ensures an effective noise reduction while preserving and enhancing the structural details in the video sequences. It is important to note that each stage of training involves its own data augmentation strategy and the whole network is trained end-to-end, with the weights of each component optimized for their specific tasks. By doing so, our network provides an overall pipeline that delivers high-quality denoised video frames.
% \vspace{5mm}
\textbf{Optimization Details:}
The training of our network leverages the Adam optimizer \cite{kingma2014adam}. Initially, the feature generator and denoise-net are jointly trained, minimizing $l_1$ loss between the central channel of the feature map and corresponding noisy frames, along with the $l_1$ loss between the denoised central frame and its corresponding original frame. The balance between these two loss terms is controlled by $\lambda_1$ in \ref{loss}, which is set to 1.0.
% We trained all the sub-networks using Adam \cite{kingma2014adam} optimizer. We trained the Feature generator and Denoise-net jointly by minimizing $l_1$ loss between the central channel of the feature map and the corresponding noisy frames for the Feature generator and minimizing the $l_1$ loss between the denoised central frame and the corresponding central frame for the denoise-net. $\lambda_1$ in \ref{loss} is set to 1.0, which controls the trade-off between the two loss terms. Where $F_t^{c}$ and $I_t$ are the central channel of the feature map and corresponding noisy frames, while $\hat{I}_B$, and $I^c$ are the denoised central frames and noisy central frame in the input batch respectively
\begin{equation}
\mathcal{L}_{1} = ||\hat{I}_B - I^c||_1 + \lambda_1(\frac{1}{B}\sum_{t=1}^B|| F_t^{c} - I_t)||_1
\label{loss}
\end{equation}
% We trained a refine-net to further improve the quality of the denoised output from the denoise-net. As shown in \eqref{refine_loss}, we minimized $l_1$ loss between the output of the refine-net and the output of the denoise-net and $l_1$ loss between the refine-net output and the corresponding noisy frame. $\lambda_2$ and $\lambda_3$ are used as regularization terms to balance between details preservation and reducing noise, and the regularization terms were set to 0.1 and 1.0, respectively.
Subsequently, we refine the denoised output using the refine-net, minimizing the $l_1$ loss between the refine-net's output and both the denoise-net's output and the corresponding noisy frame. The regularization terms $\lambda_2$ and $\lambda_3$, set to 0.1 and 1.0 respectively, help to strike a balance between detail preservation and noise reduction.
\begin{equation}
L_{2} = \lambda_2||\mathcal{R}_\eta(G_t^c) - I^c||_1 + \lambda_3||\mathcal{R}_\eta(G_t^c) - \hat{I}_B||_1
\label{refine_loss}
\end{equation}
The initial learning rates are set to $0.0001$ for the joint training stage and $0.00001$ for the refine-net training. Both rates are reduced by a factor of 10 after every 1000 epochs. During the refine-net training stage, the weights of the feature generator and denoise-net are frozen.
% We set the initial learning rate to $0.0001$ set for joint training of the feature generator, and the denoise-net and learning rate of $0.00001$ was used for the refine-net training stage. Both learning rates were reduced by a factor of 10 every 1000 epochs. We froze the weights of the feature generator and denoise-net during the training of the refine-net
% \section{Denoising Result}
% In this section, we provide additional visual results of our experiments on DAVIS and Set8 datasets corrupted with Gaussian noise with $\sigma = 30$ and $50$, Poisson noise with $\lambda=30$ and $50$ and Impulse with $\alpha = 0.2$. We show our results for bus, hypersmooth, and snowboard.
\section{More Visual Comparison of Denoising Algorithms}
In this section, we present a comprehensive visual comparison of our denoising method against the UDVD \cite{udvd}, utilizing several test datasets. Our experiments use the DAVIS and Set8 datasets, each corrupted with various noise types: Gaussian noise with $\sigma = 30$ and $50$, Poisson noise with $\lambda=30$ and $50$, and Impulse noise with $\alpha = 0.2$. The denoising performance is evaluated and visually demonstrated on several distinct scenarios, including 'bus', 'hypersmooth', and 'snowboard'.
\begin{figure}[h]
\centering
\includegraphics[width=0.9\textwidth]{images/supp_syn_bus.png}
\label{fig:supp_bus}
\caption{Visual comparison on the denoising result of bus video sequence in the DAVIS dataset}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.9\textwidth]{images/supp_syn_snow.png}
\label{fig:supp_bus}
\caption{Visual comparison of the denoising result of snowboard video sequence in SET8 dataset}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.9\textwidth]{images/supp_syn_set8.png}
\label{fig:supp_bus}
\caption{Visual comparison of the denoising result of hypersmooth video sequence in SET8 dataset}
\end{figure}
\section{Ablation Study}
\textbf{Positional encoding.} We provide ablation studies on the positional encoding hyperparameters in \ref{pos} using video sequences in DAVIS dataset. We experimented with different frequency levels in \ref{pos} to encode the input coordinates. Table \ref{tab:PE} describes the average PSNR and SSIM value using different $L$:
\begin{table}[h]
% \small
\begin{center}
\begin{tabular}{cccccccc}
\hline
\centering
&Metrics & $L=5$ & $L=10$ & $L=20$ & $L=30$ & $L=50$ & $L=100$\\
\hline
\multirow{2}{*}{\parbox{1.2cm}{Poisson $\lambda=30$}} & PSNR & 27.84 & 27.54 & 27.46 & \textbf{29.01} & 27.46 & 27.30 \\
& SSIM & 0.78 & 0.77 & 0.77 & \textbf{0.81} & 0.77 & 0.76 \\
\hline
\multirow{2}{*}{\parbox{1.2cm}{Gaussian $\sigma=30$}} & PSNR & \textbf{27.18} & 26.18 & 26.12 & 26.74 & 25.81 & 26.04 \\
& SSIM & 0.74 & 0.72 & 0.72 & \textbf{0.75} & 0.71 & 0.72 \\
\hline
\end{tabular}
\end{center}
\caption{\textbf{Postional encoding ablation} $L$ means frequency level used to encode the low dimensional coordinate to high dimension. The texts in bold indicate the highest value.}
\label{tab:PE}
\end{table}
% We also show the visual output of each stage of our network. Fig \ref{fig:stages} shows the denoising result of the central RGB channel of the feature map output of the feature generator, we also show the result of the denoise-net and finally the refine-net. The result shows that the addition of refine-net significantly improved the denoising result.
% \begin{table}[h]
% % \small
% \begin{center}
% \begin{tabular}{cccccccc}
% \hline
% \centering
% &Metrics & $L=5$ & $L=10$ & $L=20$ & $L=30$ & $L=50$ & $L=100$\\
% \hline
% \multirow{2}{*}{\parbox{1.2cm}{Poisson $\lambda=30$}} & PSNR & 27.84 & 27.54 & 27.46 & \textbf{29.01} & 27.46 & 27.30 \\
% & SSIM & 0.78 & 0.77 & 0.77 & \textbf{0.81} & 0.77 & 0.76 \\
% \hline
% \multirow{2}{*}{\parbox{1.2cm}{Gaussian $\sigma=30$}} & PSNR & \textbf{27.18} & 26.18 & 26.12 & 26.74 & 25.81 & 26.04 \\
% & SSIM & 0.74 & 0.72 & 0.72 & \textbf{0.75} & 0.71 & 0.72 \\
% \hline
% \end{tabular}
% \end{center}
% \caption{\textbf{Postional encoding ablation} $L$ means frequency level used to encode the low dimensional coordinate to high dimension. The texts in bold indicate the highest value.}
% \label{tab:PE}
% \end{table}
\begin{thebibliography}{10}
\bibitem{mipnerf}
Jonathan~T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo
Martin-Brualla, and Pratul~P Srinivasan.
\newblock Mip-nerf: A multiscale representation for anti-aliasing neural
radiance fields.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 5855--5864, 2021.
\bibitem{chibane2020implicit}
Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll.
\newblock Implicit functions in feature space for 3d shape reconstruction and
completion.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition}, pages 6970--6981, 2020.
\bibitem{videnn}
Michele Claus and Jan Van~Gemert.
\newblock Videnn: Deep blind video denoising.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition workshops}, pages 0--0, 2019.
\bibitem{mf2f}
Val{\'e}ry Dewil, J{\'e}r{\'e}my Anger, Axel Davy, Thibaud Ehret, Gabriele
Facciolo, and Pablo Arias.
\newblock Self-supervised training for blind multi-frame video denoising.
\newblock In {\em Proceedings of the IEEE/CVF winter conference on applications
of computer vision}, pages 2724--2734, 2021.
\bibitem{dewil2021self}
Val{\'e}ry Dewil, J{\'e}r{\'e}my Anger, Axel Davy, Thibaud Ehret, Gabriele
Facciolo, and Pablo Arias.
\newblock Self-supervised training for blind multi-frame video denoising.
\newblock In {\em Proceedings of the IEEE/CVF winter conference on applications
of computer vision}, pages 2724--2734, 2021.
\bibitem{dupont2020equivariant}
Emilien Dupont, Miguel~Bautista Martin, Alex Colburn, Aditya Sankar, Josh
Susskind, and Qi~Shan.
\newblock Equivariant neural rendering.
\newblock In {\em International Conference on Machine Learning}, pages
2761--2770. PMLR, 2020.
\bibitem{frame2frame}
Thibaud Ehret, Axel Davy, Jean-Michel Morel, Gabriele Facciolo, and Pablo
Arias.
\newblock Model-blind video denoising via frame-to-frame training.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition}, pages 11369--11378, 2019.
\bibitem{residual}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
pattern recognition}, pages 770--778, 2016.
\bibitem{batchnorm}
Sergey Ioffe and Christian Szegedy.
\newblock Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
\newblock In {\em International conference on machine learning}, pages
448--456. pmlr, 2015.
\bibitem{kingma2014adam}
Diederik~P Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock {\em arXiv preprint arXiv:1412.6980}, 2014.
\bibitem{noise2void}
Alexander Krull, Tim-Oliver Buchholz, and Florian Jug.
\newblock Noise2void-learning denoising from single noisy images.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition}, pages 2129--2137, 2019.
\bibitem{self-supervised}
Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila.
\newblock High-quality self-supervised deep image denoising.
\newblock {\em Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem{laine2019high}
Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila.
\newblock High-quality self-supervised deep image denoising.
\newblock {\em Advances in Neural Information Processing Systems}, 32, 2019.
\bibitem{deepinter}
J{\'e}r{\^o}me Lecoq, Michael Oliver, Joshua~H Siegle, Natalia Orlova, Peter
Ledochowitsch, and Christof Koch.
\newblock Removing independent noise in systems neuroscience data using
deepinterpolation.
\newblock {\em Nature methods}, 18(11):1401--1408, 2021.
\bibitem{noise2noise}
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras,
Miika Aittala, and Timo Aila.
\newblock Noise2noise: Learning image restoration without clean data.
\newblock {\em arXiv preprint arXiv:1803.04189}, 2018.
\bibitem{deformableattention}
Jingyun Liang, Yuchen Fan, Xiaoyu Xiang, Rakesh Ranjan, Eddy Ilg, Simon Green,
Jiezhang Cao, Kai Zhang, Radu Timofte, and Luc~V Gool.
\newblock Recurrent video restoration transformer with guided deformable
attention.
\newblock {\em Advances in Neural Information Processing Systems}, 35:378--393,
2022.
\bibitem{rawnerf}
Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul~P Srinivasan, and
Jonathan~T Barron.
\newblock Nerf in the dark: High dynamic range view synthesis from noisy raw
images.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 16190--16199, 2022.
\bibitem{nerf}
Ben Mildenhall, Pratul~P Srinivasan, Matthew Tancik, Jonathan~T Barron, Ravi
Ramamoorthi, and Ren Ng.
\newblock Nerf: Representing scenes as neural radiance fields for view
synthesis.
\newblock {\em Communications of the ACM}, 65(1):99--106, 2021.
\bibitem{davis}
Jordi Pont-Tuset, Sergi Caelles, Federico Perazzi, Laura Leal-Taixe, Daniel
Cremers, and Luc Van~Gool.
\newblock The 2017 davis challenge on video object segmentation.
\newblock {\em arXiv preprint arXiv:1704.00675}, 2017.
\bibitem{riegler2020free}
Gernot Riegler and Vladlen Koltun.
\newblock Free view synthesis.
\newblock In {\em Computer Vision--ECCV 2020: 16th European Conference,
Glasgow, UK, August 23--28, 2020, Proceedings, Part XIX 16}, pages 623--640.
Springer, 2020.
\bibitem{udvd}
Dev~Yashpal Sheth, Sreyas Mohan, Joshua~L Vincent, Ramon Manzorro, Peter~A
Crozier, Mitesh~M Khapra, Eero~P Simoncelli, and Carlos Fernandez-Granda.
\newblock Unsupervised deep video denoising.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
Computer Vision}, pages 1759--1768, 2021.
\bibitem{siren}
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon
Wetzstein.
\newblock Implicit neural representations with periodic activation functions.
\newblock {\em Advances in Neural Information Processing Systems},
33:7462--7473, 2020.
\bibitem{neural}
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek
Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler.
\newblock Neural geometric level of detail: Real-time rendering with implicit
3d shapes.
\newblock In {\em Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition}, pages 11358--11367, 2021.
\bibitem{fourier}
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin
Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.
\newblock Fourier features let networks learn high frequency functions in low
dimensional domains.
\newblock {\em Advances in Neural Information Processing Systems},
33:7537--7547, 2020.
\bibitem{dvdnet}
Matias Tassano, Julie Delon, and Thomas Veit.
\newblock Dvdnet: A fast network for deep video denoising.
\newblock In {\em 2019 IEEE International Conference on Image Processing
(ICIP)}, pages 1805--1809. IEEE, 2019.
\bibitem{fastdvdnet}
Matias Tassano, Julie Delon, and Thomas Veit.
\newblock Fastdvdnet: Towards real-time deep video denoising without flow
estimation.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition}, pages 1354--1363, 2020.
\bibitem{dilated}
Fisher Yu and Vladlen Koltun.
\newblock Multi-scale context aggregation by dilated convolutions.
\newblock {\em arXiv preprint arXiv:1511.07122}, 2015.
\bibitem{rvidenet}
Huanjing Yue, Cong Cao, Lei Liao, Ronghe Chu, and Jingyu Yang.
\newblock Supervised raw video denoising with a benchmark dataset on dynamic
scenes.
\newblock In {\em Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition}, pages 2301--2310, 2020.
\bibitem{dncnn}
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang.
\newblock Beyond a gaussian denoiser: Residual learning of deep cnn for image
denoising.
\newblock {\em IEEE transactions on image processing}, 26(7):3142--3155, 2017.
\bibitem{ffdnet}
Kai Zhang, Wangmeng Zuo, and Lei Zhang.
\newblock Ffdnet: Toward a fast and flexible solution for cnn-based image
denoising.
\newblock {\em IEEE Transactions on Image Processing}, 27(9):4608--4622, 2018.
\end{thebibliography}