Stable Single-Pixel Contrastive Learning for Semantic and Geometric Tasks
Extracted source text · arXiv 2512.04970
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
\PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% The authors should use one of these tracks.
% Before accepting by the NeurIPS conference, select one of the options below.
% 0. "default" for submission
\usepackage[dblblindworkshop, final]{neurips_2025}
% the "default" option is equal to the "main" option, which is used for the Main Track with double-blind reviewing.
% 1. "main" option is used for the Main Track
% \usepackage[main]{neurips_2025}
% 2. "position" option is used for the Position Paper Track
% \usepackage[position]{neurips_2025}
% 3. "dandb" option is used for the Datasets & Benchmarks Track
% \usepackage[dandb]{neurips_2025}
% 4. "creativeai" option is used for the Creative AI Track
% \usepackage[creativeai]{neurips_2025}
% 5. "sglblindworkshop" option is used for the Workshop with single-blind reviewing
% \usepackage[sglblindworkshop]{neurips_2025}
% 6. "dblblindworkshop" option is used for the Workshop with double-blind reviewing
% \usepackage[dblblindworkshop]{neurips_2025}
% After being accepted, the authors should add "final" behind the track to compile a camera-ready version.
% 1. Main Track
% \usepackage[main, final]{neurips_2025}
% 2. Position Paper Track
% \usepackage[position, final]{neurips_2025}
% 3. Datasets & Benchmarks Track
% \usepackage[dandb, final]{neurips_2025}
% 4. Creative AI Track
% \usepackage[creativeai, final]{neurips_2025}
% 5. Workshop with single-blind reviewing
% \usepackage[sglblindworkshop, final]{neurips_2025}
% 6. Workshop with double-blind reviewing
% \usepackage[dblblindworkshop, final]{neurips_2025}
% Note. For the workshop paper template, both \title{} and \workshoptitle{} are required, with the former indicating the paper title shown in the title and the latter indicating the workshop title displayed in the footnote.
% For workshops (5., 6.), the authors should add the name of the workshop, "\workshoptitle" command is used to set the workshop title.
% \workshoptitle{WORKSHOP TITLE}
% "preprint" option is used for arXiv or other preprint submissions
% \usepackage[preprint]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsmath}
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{wrapfig}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{comment}
\newcommand*{\m}[1]{\mathbf{#1}}
\DeclareMathOperator*{\argmin}{arg\,min}
% Note. For the workshop paper template, both \title{} and \workshoptitle{} are required, with the former indicating the paper title shown in the title and the latter indicating the workshop title displayed in the footnote.
\title{Stable Single-Pixel Contrastive Learning for Semantic and Geometric Tasks}
\workshoptitle{UniReps: Unifying Representations in Neural Models}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Leonid Pogorelyuk \\
Rensselaer Polytechnic Institute\\
Troy, NY, USA \\
\texttt{pogorl@rpi.edu} \\
\And
Niels Bracher \\
Rensselaer Polytechnic Institute\\
Troy, NY, USA \\
\texttt{brachn@rpi.edu} \\
\And
Aaron Verkleeren \\
Rensselaer Polytechnic Institute\\
Troy, NY, USA \\
\texttt{verkla@rpi.edu} \\
\And
Lars Kühmichel \\
Technical University Dortmund\\
Dortmund, Germany \\
\texttt{lars.kuehmichel@tu-dortmund.de} \\
\And
Stefan T.~Radev \\
Rensselaer Polytechnic Institute\\
Troy, NY, USA \\
\texttt{radevs@rpi.edu} \\
}
\begin{document}
\maketitle
\begin{abstract}
We pilot a family of stable contrastive losses for learning pixel-level representations that jointly capture semantic and geometric information. Our approach maps each pixel of an image to an overcomplete descriptor that is both view-invariant and semantically meaningful. It enables precise point-correspondence across images without requiring momentum-based teacher–student training. Two experiments in synthetic 2D and 3D environments demonstrate the properties of our loss and the resulting overcomplete representations.
\end{abstract}
\section{Introduction}
Many modern computer vision tasks fall into geometric or semantic domains. Geometric tasks focus on the spatial aspects of images, such as matching points between two images \citep{Ma2021ImageMatchingSurvey, XU2024102344}, camera pose estimation \citep{xu2022criticalanalysisimagebasedcamera}, depth estimation \citep{10313067}, simultaneous localization and mapping \citep[SLAM;][]{ABASPURKAZEROUNI2022117734}, and 3D reconstruction \citep{Samavati2023DL3DReconstructionSurvey}. When such frameworks produce pixel-level features useful for spatial purposes, these features are not expected to carry semantic information about the objects in the image. Semantic tasks, on the other hand, focus on the contextual aspects such as object detection and classification \citep{s25010214, Vijayakumar2024YOLOReview}, semantic and instance segmentation \cite{10048555, THISANKE2023106669}, question answering \citep{Huynh2025VQASurvey}, or pose estimation \citep{Gao2025PoseEstimationSurvey}. However, when these frameworks produce features, they are at best at patch-level, not pixel-level precision. Hence, they are often too imprecise for the aforementioned geometric tasks.
% Methods oriented towards geometric tasks can be broadly split into \textit{detector-based} and \textit{detector-free} \cite{Liu2021KeypointsReview}. Most classical approaches \citep{6126544, Lowe2004SIFT} are detector-based and operate by first finding sparse keypoints in images and then matching these keypoints across images to be used in downstream tasks, such as Structure from Motion (SfM)\cite{Schonberger2016SfMRevisited}.
% Keypoints can also be learned by augmenting and affine transforming single images\cite{8575521, Revaud2019R2D2RA} using any large image dataset such as ImageNet\cite{5206848}. When training on augmented images without a downstream task (self-supervised learning), the typical losses have the following elements in common:
% \begin{enumerate}
% \item A loss component that penalizes distance (in feature space) between matching keypoints (i.e., they correspond to the same point in the scene computed by the network from two different views with known correspondence, e.g., homography).
% \item A component that encourages non-zero distance between non-matching keypoints (typically L2 or cosine similarity norm).
% \item Optional components that improve the quality (saliency, repeatability) of the keypoints.
% \end{enumerate}
% Similar principles can also be applied in 3D \citep{Yew_2022_CVPR, Stojanov2022DenseObjectDescriptors}, and keypoints can also be learned while performing the downstream task of aligning two images \citep{Sarlin2021PixLoc, Lindenberger2023LightGlue}. The latter requires datasets with ground-truth correspondences between different views \citep{Dai2017ScanNet, Behley2019SemanticKITTI, Balntas_2017_CVPR} or appearances \cite{Sattler2018Benchmarking6DOF}.
In the following, we develop a novel family of geometric losses that encourage a self-supervised learner to encode semantic and view-invariant information with pixel-level precision.
Encoding features with $D > 3$ dimensions per pixel can be regarded as learning overcomplete representations \cite{dubova2023excess}.
Such representations exhibit favorable properties across different domains and tasks. For instance, they have been applied to unpaired shape transformations in high-dimensional point clouds \citep{yin2019logan}, deep subspace clustering \citep{valanarasu2021overcomplete}, and denoising of high-dimensional images \citep{yasarla2020exploring}.
% They have also been shown to facilitate classification and predict learning speed in settings involving human \cite{dubova2023excess} and machine learning \cite{tang2019effective}.
% Moreover, dimensionality expansion within dynamic systems can promote stronger generalization and adaptability to tasks in recurrent neural networks \citep{farrell2022gradient}.
Building on this idea, our proposed loss family induces networks that can transform images into sets of overcomplete descriptors containing both semantic and geometric information. Such descriptors can be made invariant to view changes and occlusion 3D. In sum, our contributions are:
\begin{itemize}\setlength\itemsep{0.3pt}
\item A new loss family for stable contrastive learning of pixel-level representations without momentum-based update schemes such as teacher-student.
\item A demonstration that the loss encourages learning of semantic representations at the pixel level that are precise enough to match features across images.
\item An extension of pixel-level contrastive learning to 3D environments.
\end{itemize}
\section{Method}
\label{sec:method}
\begin{figure}
\centering
\includegraphics[width=0.99\linewidth,trim={0.51in 0.1in 0.59in 0},clip]{figures/pull_figure.pdf}
\caption{An experiment of matching pixels between different views (top left) of synthetic scenes with objects from ShapeNetCore \citep{chang2015shapenetinformationrich3dmodel} placed in rooms from SceneNN \citep{7785081} (from \textbf{Experiment 2}).
Our loss (with the $\infty$ norm) encourages non-matching features to differ by at least one channel, producing a network that encodes separate objects (instance segmentation; top right channels) and also identifies edges and interiors (bottom right), without explicitly being trained to do so.}
\label{fig:pull_figure}
\end{figure}
\paragraph{Training setup.}
During training, we generate two views of the same image.
Each view has pixel coordinates $\mathbf{p} \in \mathbb{N}^2$, where $\m{I}(\mathbf{p}) \in \mathbb{R}^3$ is the RGB value at $\mathbf{p}$.
A feature extractor $\mathcal{F}$ maps each image $\m{I} \in \mathbb{R}^{H \times W \times 3}$ to an overcomplete feature map in $\mathbb{R}^{H \times W \times D}$ with $D > 3$.
% We use a fully convolutional U-Net \citep{ronneberger2015u} with $D$ output channels (see \textbf{Appendix~A} for details), though transformer-based architectures are also applicable.
We use a U-Net--style encoder--decoder topology~\citep{ronneberger2015u} with residual blocks~\citep{He2016-residual} and $D$ output channels (see \textbf{Appendix~A} for details); self-attention is disabled in all reported runs~\citep{Hoogeboom2023-simplediff}, as it did not improve performance.
The transformation set $\mathbb{T}$ has geometric and photometric components.
In 2D, the geometric part is a perspective transform with random rotation, scaling, and skewing; in 3D, it amounts to random camera repositioning and rotation.
Photometric changes include hue/saturation jitter and, optionally, more complex noise such as motion blur \citep{Pogorelyuk_2024_AlignBlur}.
Since the transformation is known during training, we have a mapping $\mathcal{T}:\mathbb{N}^2 \rightarrow \mathbb{N}^2$ that sends pixel $\mathbf{p}$ in $\mathcal{F}(\m{I}_1)$ to pixel $\mathcal{T}(\mathbf{p})$ in $\mathcal{F}(\m{I}_2)$.
\paragraph{Within-image contrastive loss.} Our loss consists of a \textit{within-image} contrastive and a \textit{between-image} contrastive component. The within-image component decomposes into a \textit{positive} and a \textit{negative} term.
%The positive (or negative) term computes the $p$-norm between the feature map of each original region $\m{I}_1$ at coordinates $\mathbf{p}$ and that of $\m{I}_2$ at the transformed (or random) coordinates $\mathcal{T}(\mathbf{p})$ (or $\mathcal{R}(\mathbf {p})$):
The positive term computes the $p$-norm between the feature map of one view $\mathcal{F}(\m{I}_1)$ at coordinates $\mathbf{p}$ and the feature map $\mathcal{F}(\m{I}_2)$ of the other view at the transformed coordinates $\mathcal{T}(\mathbf{p})$ (random coordinates $\mathcal{R}(\mathbf {p})$ are used instead for the negative term):
\begin{equation}
D_{\text{pos}}(\mathbf{p}) := \left\Vert \mathcal{F}\left(\m{I}_{1}\right)(\mathbf{p}) - \mathcal{F} \left(\m{I}_{2} \right)\left(\mathcal{T}(\mathbf{p})\right) \right\Vert_p;\:\:\:
D_{\text{neg}}(\mathbf{p}) := \left\Vert \mathcal{F}\left(\m{I}_{1}\right)(\mathbf{p}) - \mathcal{F} \left(\m{I}_{2} \right)\left(\mathcal{R}(\mathbf{p})\right) \right\Vert_p.
\end{equation}
%For the negative term, we randomly sample a pixel $\mathcal{R}(\mathbf {p}) \in \mathbb{N}^2$ from image $\mathbf{I}_{2}$ and compute:
We combine the two terms and aggregate them across the set of all pixels $S = P \cup N$:
\begin{equation}\label{eq:loss_func}
\mathcal{L}_{\text{within}} := \frac{1}{\left|P\right|}\sum_{\mathbf{p} \in P}\left[D_{\text{pos}}(\mathbf{p}) + D_{\text{pos}}(\mathbf{p})^2 \right] + \frac{1}{\left|N\right|}\sum_{\mathbf{p} \in N}\left[-D_{\text{neg}}(\mathbf{p}) + D_{\text{neg}}(\mathbf{p})^2 \right],
\end{equation}
where $P$ is the subset of ``positive'' (i.e., transformed) pixels and $N$ is the subset of ``negative'' (i.e., randomly sampled) pixels.
The negative sign on the right is meant to increase the difference (in feature-space) between ``negative'' matches, while the square term is meant to ensure that this difference doesn't grow unbounded.
The fraction of pixels $f = \left|P\right|/\left|S\right|$ designated as positive is a hyperparameter that determines the balance between positive and negative examples.
\paragraph{Between-image contrastive loss.} The second component compares the overcomplete features of two different, unrelated views $\m{I}$ and $\m{I}^\prime$:
\begin{align}
C(\mathbf{p}) := \left\Vert \mathcal{F} \left(\m{I} \right)(\mathbf{p}) - \mathcal{F} \left(\m{I}^{\prime} \right)(\mathbf{p}) \right\Vert _p
\end{align}
The $p$-norms are then summed over the set of all pixels $S$:
\begin{equation}
\mathcal{L}_{\text{between}} := \frac{1}{\left|S\right|} \sum_{\mathbf{p} \in S}\left[-C(\mathbf{p}) + C(\mathbf{p})^2 \right],
\end{equation}
where the views can be either the original images $\m{I}_1$ or their transformed versions $\m{I}_2$. We use the latter, since the feature extractor sees augmented versions of the ``negative examples'' in each epoch.
\paragraph{Total loss.}
The feature extractor minimizes the following combined loss:
\begin{equation}\label{eq:total_loss}
\mathcal{L}_{\text{total}} = \lambda \cdot \mathcal{L}_{\text{within}} + (1 - \lambda) \cdot \mathcal{L}_{\text{between}},
\end{equation}
where $\lambda$ is a tunable hyperparameter, and the loss is taken in expectation over an unlabeled data set, the set of transformations $\mathbb{T}$, and the fraction $f$ of positive pixels in the within-image component.
Ideally, once trained, the feature extractor transforms an image into a set of overcomplete descriptors containing semantic and geometric information at the pixel level.
\section{Related Work}
Large transformer-based vision models, such as ViT \cite{dosovitskiy2021vit} and Swin Transformer \cite{Liu_2021_SwinTransformer}, produce patch-level descriptors. Supervised methods like SAM \cite{Kirillov2023SegmentAnything} generate semantic, localized features via labeled objectives, while self-supervised learning (SSL) approaches \cite{oquab2024dinov,He_2022_CVPR,bao2022beit} achieve similar semantic density without labels \cite{Amir2021DeepViTDescriptors,Pariza2025NeCo}.
However, due to their patch-based design, such models tend to prioritize semantic representation over spatial precision, especially when trained with contrastive losses \cite{Wang2021DenseCL}, leading to insufficient localization for geometric tasks like SfM. This limitation can be mitigated in a task-specific manner by adding spatially consistent objectives during training or fine-tuning \cite{Caron2024LOCA,chen2023vision}, or by adapting ViTs for sparse keypoint detection \cite{Lu_2023_SalViT}.
In 2D, Xie et al.~\cite{Xie_2021_CVPR_PixPro} showed that a geometric contrastive loss as a pretext task can learn pixel-level dense descriptors useful for downstream semantic tasks, using a momentum-based update to stabilize training—a common challenge in contrastive methods.
Stojanov et al.~\cite{Stojanov2022DenseObjectDescriptors} extended this idea to 3D, though at a patch rather than pixel level. The Universal Correspondence Network \citep[UCN;][]{Choy_2016_NIPS_UCN} was notable among early descriptor-based methods for encoding per-pixel keypoints with a 3D loss function, though it remained limited to purely geometric, non-semantic learning objectives.
Our approach avoids momentum-based update rules \citep{Xie_2021_CVPR_PixPro} with a new loss and can use both 2D and 3D data to encode semantic information and view invariance at the pixel level.
\section{Experiments}
\begin{figure}[t]
\centering
\begin{subfigure}{0.55\linewidth}
\centering
\includegraphics[width=\linewidth]{figures/id_ood.pdf}
\end{subfigure}
\hfill
\begin{subfigure}{0.44\linewidth}
\centering
\includegraphics[width=\linewidth]{figures/oxels_3x3_backbones_small.pdf}
\end{subfigure}
%\caption{\textbf{Experiment 1}. Summary of downstream classification performance (\textit{left}) and a sample of overcomplete representations for the three norms (\textit{right}).}
\begin{subfigure}{\linewidth}
\centering
\includegraphics[width=\linewidth]{figures/loss_curves_average.pdf}
\end{subfigure}
\caption{\textbf{Experiment 1}. \textit{Top Left}: ID and OOD accuracy vs. $\lambda$ (relative weight) and $p$ (norm). Wide violins show seed variability: runs either encode color ($\approx$85\% ID, $\approx$10\% OOD) or suppress it (balanced $\approx$75\%). Lower $\lambda$ (more weight on $\mathcal{L}_{\text{between}}$) increases the probability of encoding color; at $\lambda=1$ ($\mathcal{L}_{\text{within}}$ only), features become invariant to color. \textit{Top Right}: Feature maps from frozen backbones at $\lambda=0.5$ for two example inputs: $\ell_\infty$ yields sharp, bit-like encodings; $\ell_1,\ell_2$ yield smoother encodings. Downstream classification on ColoredMNIST revealed color-encoding of all three backbones, which is clearly apparent for $\ell_\infty$ and $\ell_1$; however, this is not the case for $\ell_2$.
% Rows: $\ell_\infty,\ \ell_1,\ \ell_2$. Leftmost: inputs; columns 1–8: first eight normalized feature maps. $\ell_\infty$ yields sharp, bit-like activations; $\ell_1,\ell_2$ give smoother encodings, with additional “bit switches” emerging in later channels.
% See \autoref{sec:appendix-b} for details.
\textit{Bottom}: Shown are the loss curves for each norm averaged over the 10 seeds per $\lambda$. For $\ell_{\infty}$ and $\ell_1$ backbone training is stable and robust across different $\lambda$ in contrast to traditional $\ell_2$ norm. A higher weight of contrastive loss between embeddings (lower $\lambda$) results in a higher overall loss.
}
\label{fig:exp1_results}
\end{figure}
\label{sec:experiments}
\vspace{-1pt}
\subsection{Experiment 1: Domain generalization and color invariance}
\textbf{Motivation and setup.} We first evaluate our loss in a classic domain generalization setting using the Colored MNIST benchmark \citep{arjovsky2019invariant}. The task requires predicting if a digit is $\geq 5$, making it a semantic task designed to separate invariant features (digit identity) from spurious ones (digit color). The training set consists of multiple domains with fixed digit–color correlations, while the test set contains unseen digit–color combinations.
We vary hyperparameters $\lambda$ (trade-off between within- and between-image terms) and $p \in {\ell_1, \ell_2, \ell_\infty}$ (feature distance norm).
\begin{wrapfigure}[18]{r}{0.4\textwidth}
\centering
\includegraphics[width=0.99\linewidth]{figures/mapping.pdf} \caption{Top row: source image, its transformed view (target 1), and a different image (target 2). Bottom row: Each source pixel (colored for visualization only) is mapped onto the location of the most similar (in $ l_2$ distance) encoding.} \label{fig:mapping}
\end{wrapfigure}
After training, we freeze the backbone and fit an attention-based pixel classifier averaging \textit{pixel-level} predictions. This \textit{permutation-invariant} classifier tests whether representations capture semantic information at the pixel level.
%We sweep $D \in \{4, 8, 16, 32\}$ for $\lambda = 1$ and run five independent seeds for $\lambda = 0.1$ to assess stability in color encoding.
We hypothesize that \textbf{(1)} $\lambda = 1$ (within-image loss only) will encourage color invariance, improving OOD performance; \textbf{(2)} lower $\lambda$ values will allow encoding of color, reducing OOD accuracy; and \textbf{(3)} the $\ell_\infty$ norm will produce bit-like, sharp representations, while $\ell_2$ and $\ell_1$ norms will yield smoother, averaged features.
\paragraph{Results.}
The results confirm that \textbf{(a)} semantic information is encoded at the pixel level; \textbf{(b)} $\lambda = 1$ produces color-invariant features while lower $\lambda$ causes the model to encode color (see \autoref{fig:exp1_results}, \textit{left}); \textbf{(c)} the $\ell_\infty$ norm yields sharp features, whereas $\ell_2$ and $\ell_1$ norms produce smoother features (see \autoref{fig:exp1_results}, \textit{right}); and \textbf{(d)} the features are localized: when they match across different images, they correspond to the same part of the object (see \autoref{fig:mapping}).
Further results are available in \autoref{sec:appendix-b}.
\subsection{Experiment 2: 3D invariance in synthetic scenes}
\label{sec:experiment-2}
\paragraph{Motivation and setup.} Overcomplete representations can also be rendered invariant to viewing angle and occlusions. This requires a camera to change its position in 3D, as opposed to applying a perspective transform on the image. We illustrate this concept using synthetic scenes in which 40336 objects from ShapeNetCore \citep{chang2015shapenetinformationrich3dmodel} are placed in 76 rooms from SceneNN \citep{7785081} (see \autoref{fig:pull_figure}, top left).
\begin{wrapfigure}[12]{r}{0.45\textwidth}
\centering
\includegraphics[width=0.99\linewidth]{figures/synthetic_loss_curve.pdf} \caption{Validation loss curve with $\ell_{\infty}$ and $\lambda =1$. The loss demonstrates stable convergence with minimal variance.} \label{fig:mapping}
\end{wrapfigure}
With 3D meshes of all rooms and objects, we can randomly position and orient objects, place and rotate the camera at random, and track the visible pixels in the images.
% For training, we create different views of the same room with some objects that appear in both scenes and some that do not. We randomly select about 1/8 of the pixels in one view, trace them to their closest intersection with the 3D mesh, and find the intersections' corresponding pixels in the second view. To check for occlusions, we trace the pixels from the second view back to their closest intersection with the mesh and compute their distances from the original intersections. A large distance means that the point visible in the first view is occluded in the second view and is not a match. Finally, we select the rest of the non-matching pixels randomly and mask out pixels not rendered due to holes in the mesh.
For training, we generate different views of the same room, with some objects shared across scenes and others not. About 1/8 of the pixels in one view are traced to their nearest 3D mesh intersection and matched to the corresponding pixels in the second view. Occlusions are detected by tracing back from the second view and measuring distances from the original intersections; large discrepancies indicate non-matches. The remaining non-matching pixels are chosen randomly.
\vspace{-1pt}
\paragraph{Results.} Our loss (Eq.~\ref{eq:total_loss}) using the $l_\infty$ norm and $\lambda = 1$ encourages the network to learn at least two modes of information in a bit-like fashion across its channels:
\textit{Semantic}---different objects/parts have encoding different by at least one channel but consistent across views, akin to instance segmentation (\autoref{fig:pull_figure}, top right); \textit{Geometric}---some of the channels flip sign depending on how far a point is from an edge, with some channels effectively detect shape edges and other detect ``inner'' regions in 3D (\autoref{fig:pull_figure}, bottom right). Further results are available in \autoref{sec:appendix-c}.
\vspace{-1pt}
%TODO (Aaron): How we used synthetic images from ShapeNetCore (\cite{chang2015shapenetinformationrich3dmodel}) and SceneNN (\cite{7785081}) environments. How many scenes and objects (76 rooms, 40336 objects)? What was the training/validation split (contrastive dataset 99/1, 80/20 on SceneNN and ShapeNetCore for generating the dataset)? What fraction of matching points (1/8)? Brief description of how objects were placed. Anything else?
\section{Limitations and Conclusion}
\label{sec:limitations}
\vspace{-1pt}
We piloted a new family of losses for stable contrastive learning of pixel-level representations in 2D and 3D environments. They induce semantic information localized at the pixel level. Our approach has several limitations: overcomplete embeddings are less compute-efficient than low-dimensional ones, and artificial perspective transforms can create a simulation gap for real-world viewpoint shifts.
We plan to address the latter in a series of experiments on stereo images with known point matches.
\begin{ack}
% Lars' funding:
This work is partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) Project 528702768.
% TODO: add other acknowledgments
\end{ack}
% \section{References}
\bibliographystyle{unsrtnat}
\bibliography{references}
\newpage
\appendix
\section*{Appendix} \label{sec:appendix}
\section{Source code}
\label{sec:appendix-a}
The source code is available at \url{https://github.com/stefanradev93/oxels}.
\section{Details and additional results for Experiment 1}
\label{sec:appendix-b}
\subsection{Experimental details}
Starting from MNIST (28$\times$28 grayscale), we binarize labels ($<5\!\to\!0$, $\geq\!5\!\to\!1$) and flip with probability $0.25$. Each image’s background is assigned one of two colors, sampled to correlate with the (possibly flipped) label with probability $p_{\text{train}}{=}0.85$ in the training pool and $p_{\text{test}}{=}0.10$ in the OOD domain. The final images are bilinearly resized to $32\times32$ to match the model input. Under this construction, heuristic ceilings are: shape-only $\approx75\%$ across domains; color-only $\approx85\%$ (ID) and $\approx10\%$ (OOD).
\subsection{Training details and hyperparameters}
\begin{comment}
All backbones and classifiers were trained using \textbf{[machine/GPU model, e.g., 1$\times$ NVIDIA A100, 40GB memory]} on \textbf{[framework + version, e.g., PyTorch 2.1]} with CUDA \textbf{[version]}.
We used the \textbf{[optimizer, e.g., Adam]} optimizer with an initial learning rate of \textbf{[value]}, decayed by a factor of \textbf{[value]} every \textbf{[epochs/steps]}. Backbone training ran for \textbf{[number]} epochs with a batch size of \textbf{[number]} for a total of \textbf{[number]} hours per backbone.
Classifier training ran for \textbf{[number]} epochs with a batch size of \textbf{[number]} for a total of \textbf{[number]} hours per classifier.
\end{comment}
All backbone and classifier models were trained on a single NVIDIA GeForce RTX~4090 with 24\,GB memory using PyTorch~2.6 with CUDA~12.4. We used the AdamW optimizer~\citep{Loshchilov2017-adamw} (\(\beta_1{=}0.9\), \(\beta_2{=}0.99\), weight decay \(=10^{-6}\)) with a OneCycleLR schedule~\citep{Smith2017-onecyclelr}. For the backbone trainings, the learning rate was warmed up from \(4\times10^{-5}\) to a peak of \(1\times10^{-3}\) over the first 5\% of training steps, then cosine-annealed to \(1\times10^{-7}\) over the remainder. The classifier trainings followed a similar regime with a peak learning rate of \(2\times10^{-3}\).
Training proceeded sequentially: we first trained each backbone, then trained a classifier with the backbone frozen.
For the $D$-sweep with $D\in\{4,8,16,32\}$ at $\lambda=1.0$, and for the additional five runs per norm at $\lambda=0.1$ with $D=32$, each run used 100{,}000 steps for the backbone training and 30{,}000 steps for the classifier. A 100{,}000-step run took $\sim$1.5 hours on the RTX~4090; scaling this to the 210-run $\lambda$-sweep would have required $\sim$315 GPU-hours for the backbones alone. To make the sweep feasible, we reduced the per-run budget of the backbones to 30{,}000 steps ($\sim$0.45 hours/run) and the classifiers to 12{,}000, accepting a modest loss-convergence degradation for broader coverage. The $\lambda$-sweep (seven values under $\ell_\infty$, $\ell_1$, and $\ell_2$ with 10 seeds each, totalling 210 runs) therefore used 30{,}000/12{,}000 steps per run (backbone/classifier).
\subsection{Backbone architectures}
\begin{comment}
All backbones were implemented as a \textbf{[model type, e.g., U-Net with residual blocks]}, producing \textbf{[D]} output channels. Self-attention was disabled for all layers, as it did not improve performance.
A total of $210$ backbones were trained, each backbone having a total of [\textbf{number}] parameters.
\end{comment}
We use a U-Net variant following \cite{Hoogeboom2023-simplediff} (without time embeddings). An initial \(3\times3\) convolution maps RGB to 32 channels, followed by three encoder stages with 1, 2, and 2 residual blocks at widths 32, 64, and 64. Blocks use SiLU and a LayerNorm-style normalization; residual dropout \(p=0.1\) is applied only in the last two stages. Each stage downsamples by \(\times\frac{1}{2}\) via average pooling with a \(1\times1\) convolution for channel alignment. The bottleneck is a residual block \(\rightarrow\) self-attention \(\rightarrow\) residual block (attention only here). The decoder mirrors the encoder, upsampling by \(\times2\) using a \(1\times1\) convolution followed by nearest-neighbor interpolation, with one extra residual block per stage to fuse skip connections (the last block has no skip). The head is normalization \(\rightarrow\) SiLU \(\rightarrow\) \(3\times3\) convolution projecting to \(D\) output channels (we use \(D{=}32\) unless varied). Convolutions are initialized with Xavier uniform and near-zero biases. Each backbone has a total of 993{,}664 parameters.
\subsection{Classifier architectures}
We use a single-head self-attention pixel classifier (similar to the self-attention block in \citep{Hoogeboom2023-simplediff}) on the backbone features \(\m{F}\in\mathbb{R}^{H\times W \times D}\). Inputs are per-pixel normalized with a LayerNorm-style scheme (one-group normalization; no bias shift). Linear projections produce \(Q,K,V\in\mathbb{R}^{D}\); \(Q\) and \(K\) are normalized and scaled by \(1/\sqrt{D}\). Attention is computed over all pixels (no positional encodings), yielding attended pixel embeddings that a small MLP (hidden widths 150, 100, 50) maps to a \emph{single} logit per pixel (binary task). The image-level prediction is the uniform average of pixel logits. The backbone remains frozen, and a classifier with 28{,}479 parameters is trained for each backbone.
\newpage
\subsection{Additional results}
\begin{figure}[h]
\centering
\includegraphics[width=0.99\linewidth]{figures/neurips_oxels_inf.pdf}
\caption{Example overcomplete representations obtained via an $l_\infty$-based loss from validation samples.}
\label{fig:linf_appendix}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.99\linewidth]{figures/neurips_oxels_l1.pdf}
\caption{Example overcomplete representations obtained via an $l_1$-based loss from validation samples.}
\label{fig:l1_appendix}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.99\linewidth]{figures/neurips_oxels_l2.pdf}
\caption{Example overcomplete representations obtained via an $l_2$-based loss from validation samples.}
\label{fig:l2_appendix}
\end{figure}
\newpage
% \paragraph{Experimental Reproducibility}
% \label{sec:reproducibility}
% TODO: experimental setup details here
% e.g.
% - environmental info (cluster, which/how many GPUs, which OS, etc.)
% - how long training took (wall time)
% - Hyperparameters (available on wandb)
% \begin{table}[h]
% \centering
% \begin{tabular}{rcl}
% Parameter & Value & Notes \\
% \hline
% batch size & 16 & per node \\
% weight decay & 0.0001 & \\
% precision & 16-mixed & \\
% num oxels & 64 & \\
% LR Scheduler & OneCycleLR & \\
% LR Warmup & 2000 steps & \\
% LR Div Factor & 25 & \\
% LR Final Div Factor & 10,000 & \\
% Peak LR & 0.001 & \\
% \end{tabular}
% \caption{Hyperparameters for ImageNet-1k Backbone Training}
% \label{tab:hparams-imagenet}
% \end{table}
% \
% The full details of architectures we used can be found in our code repository. The backbone trained on ImageNet-1K totals around 36.2M parameters.
\section{Details and additional results for Experiment 2}
\label{sec:appendix-c}
\subsection{Training details hyperparameters}
All models were trained using an NVIDIA RTX 4090 GPU with 24\,GB memory on PyTorch 2.6 with CUDA 12.4. We used the AdamW optimizer with a
OneCycle schedule. The initial learning rate starts at 4.0e-5, warms up to 1.0e-3 over the first 5\% of steps, then anneals to 4.0e-9 by the end. Training ran for 6 epochs with a batch size of 32 for a total of 4 hours.
\subsection{Backbone architecture} The backbone was a U-Net with residual blocks, producing 32 output channels. It has five stages with channel widths [16, 16, 32, 32, 64] and residual block counts [1, 1, 1, 2, 4].
Dropout with $p{=}0.1$ is applied on the last two stages. The backbone had a total of $1{,}031{,}344$ parameters.
\subsection{Additional results}
\begin{figure}[h]
\centering
\includegraphics[width=0.99\linewidth]{figures/exp2_loss.pdf}
\caption{\textbf{Training example in 3D:} Training data pairs consisted of two views of a 3D scene, as well as a known transformation, $\cal T$, that maps pixels from view 1 to match their corresponding locations in view 2. Our loss function (\autoref{eq:loss_func} with $\lambda=1$ and $p=\infty$) uses $\cal T$ to compare the network-extracted features (overcomplete representations) of the two views at a fraction of the matching pixels (positive pairs) and at random pixels (negative pairs). For visualization purposes, we show just the first three channels of the features, scaled by 0.5 and shifted by 0.5 to form ``false'' color images.}
\label{fig:exp2_loss}
\end{figure}
\begin{figure}[h]
\centering
\includegraphics[width=0.99\linewidth]{figures/exp2_oxels.pdf}
\caption{Example of overcomplete representations for all the channels encoded by the network, trained with $\lambda=1$ and $p=\infty$ on our synthetic 3D data.}
\label{fig:exp2_oxels}
\end{figure}
\section{Societal Impact}
\label{sec:societal-impact}
Since this paper focuses on foundational research and methodology, we expect that there is no direct path for negative societal impact. In fact, positive impacts may manifest, for instance, in improved performance or robustness of good-faith downstream applications.
Nevertheless, we cannot rule out intentional or unintentional misuse of our methods for harmful (downstream) applications.
\end{document}