Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining
Extracted source text · arXiv 2507.09513
\documentclass{article} % For LaTeX2e
\usepackage{iclr2026_conference,times}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\input{inc/inc_packages}
\input{inc/inc_macros}
\usepackage{hyperref}
\usepackage{url}
\renewcommand{\thefootnote}{\fnsymbol{footnote}}
\title{Animal behavioral analysis and neural encoding with transformer-based self-supervised pretraining}
% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.
\author{Yanchen Wang$^{1*}$, Han Yu$^{1*}$, Ari Blau$^{1}$, Yizi Zhang$^{1}$, International Brain Laboratory,
\\
\textbf{Liam Paninski}$^{1}$, \textbf{Cole Hurwitz}$^{1}$ \& \textbf{Matthew R. Whiteway}$^{1}$\\
$^{1}$Columbia University, New York, NY USA\\
$^{*}$Equal contribution\\
\texttt{\{yw4503,hy2562,bsb2144,yz4123,lmp2107,ch3676,mw3323\}@columbia.edu}
}
% \author{Antiquus S.~Hippocampus, Natalia Cerebro \& Amelie P. Amygdale \thanks{ Use footnote for providing further information
% about author (webpage, alternative address)---\emph{not} for acknowledging
% funding agencies. Funding acknowledgements go at the end of the paper.} \\
% Department of Computer Science\\
% Cranberry-Lemon University\\
% Pittsburgh, PA 15213, USA \\
% \texttt{\{hippo,brain,jen\}@cs.cranberry-lemon.edu} \\
% \And
% Ji Q. Ren \& Yevgeny LeNet \\
% Department of Computational Neuroscience \\
% University of the Witwatersrand \\
% Joburg, South Africa \\
% \texttt{\{robot,net\}@wits.ac.za} \\
% \AND
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email}
% }
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}
\maketitle
% Abstract
\input{contents/c0_abstract.tex}
% Introduction
\input{contents/c1_introduction.tex}
% Related Work
\input{contents/c2_related_work.tex}
% Methods
\input{contents/c3_methods.tex}
% Experiments
\input{contents/c4_experiments.tex}
% Experiments
\input{contents/c5_discussion.tex}
% References
\newpage
\bibliography{iclr2026_conference}
% Appendix
\clearpage
\newpage
\appendix
\input{appendix/main.tex}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\usepackage{iclr2026_conference} % iclr 2026 style
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{adjustbox}
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{xcolor} % colors
\usepackage{amsmath} % for \text in math mode
\usepackage{graphicx}
\usepackage{placeins}
\usepackage{microtype} % microtypography
\usepackage{silence}
\WarningFilter{latex}{Command \showhyphens}
% \bibliographystyle{plain} % references
\bibliographystyle{plainnat} % references no numbers
\usepackage{soul} % \hl
\usepackage{xspace} % xspace
\usepackage{wrapfig}
\usepackage{caption}
\captionsetup{font=footnotesize}
% compress vertical spacing
\preto\figure{\setlength{\belowcaptionskip}{-12pt}}
\makeatother
\usepackage{titlesec}
\titlespacing*{\section}{0pt}{*0.1}{*0.1}
\titlespacing*{\subsection}{0pt}{*0.1}{*0.1}
\titlespacing*{\subsubsection}{0pt}{*0.1}{*0.1}
\setlength{\textfloatsep}{12pt} % Space between floats and text
\setlength{\floatsep}{12pt} % Space between two floats
\setlength{\intextsep}{12pt} % Space between inline floats and text
\renewcommand{\paragraph}[1]{\par\addvspace{0.1\baselineskip}%
\textbf{#1}\ \ignorespaces}
\usepackage{multirow}
\newcommand{\beast}{\textsc{Beast}\xspace}
\newcommand{\vit}{\textsc{ViT}\xspace}
\newcommand{\vitm}{\textsc{ViT-M}\xspace}
\newcommand{\videomae}{\textsc{VideoMAE}\xspace}
\newcommand{\treba}{\textsc{TREBA}\xspace}
%% comments
\definecolor{MyDarkBlue}{rgb}{0,0.08,0.8}
\newcommand{\yanchen}[1]{\textcolor{cyan}{Yanchen: #1}}
\newcommand{\han}[1]{\textcolor{purple}{Han: #1}}
\newcommand{\blue}[1]{\textcolor{MyDarkBlue}{#1}}
\begin{abstract}
The brain can only be fully understood through the lens of the behavior it generates--a guiding principle in modern neuroscience research that nevertheless presents significant technical challenges. Many studies capture behavior with cameras, but video analysis approaches typically rely on specialized models requiring extensive labeled data. We address this limitation with \beast (\textbf{BE}havioral \textbf{A}nalysis via \textbf{S}elf-supervised pretraining of \textbf{T}ransformers), a novel and scalable framework that pretrains experiment-specific vision transformers for diverse neuro-behavior analyses. \beast combines masked autoencoding with temporal contrastive learning to effectively leverage unlabeled video data. Through comprehensive evaluation across multiple species, we demonstrate improved performance in three critical neuro-behavioral tasks: extracting behavioral features that correlate with neural activity, and pose estimation and action segmentation in both the single- and multi-animal settings. Our method establishes a powerful and versatile backbone model that accelerates behavioral analysis in scenarios where labeled data remains scarce.
\end{abstract}
\section{Introduction}
Understanding the relationship between brain and behavior is a fundamental challenge across a wide range of medical and scientific disciplines~\citep{krakauer2017neuroscience, datta2019computational}. Precise methods for extracting meaningful information from behavioral videos are essential for advancing these fields~\citep{pereira2020quantifying}. Self-supervised learning has revolutionized image and video understanding through large-scale foundation models~\citep{chen2020simple, caron2021emerging, he2022masked}, offering powerful tools that are beginning to transform scientific analyses~\citep{huang2023self, lastufka2024vision}. However, these models have yet to be effectively translated to specialized domains like animal behavior analysis, creating a significant opportunity for methods that bridge cutting-edge machine learning with the specific demands of neuroscience and behavioral research.
Animal behavior videos present unique characteristics and challenges distinct from general video understanding. Controlled experiments generate large quantities of videos with static backgrounds and consistent camera angles, where the primary variation arises from animal movements and interactions. These videos enable numerous downstream analyses, and here we focus on three fundamentally different applications that collectively address a large proportion of behavioral neuroscience use cases:
(1) neural activity prediction (or ``neural encoding''), which requires extracting behavioral features that correlate with simultaneously recorded brain activity~\citep{datta2019computational, pereira2020quantifying, urai2022large}; (2) pose estimation, which tracks specific anatomical landmarks for quantitative analysis of movement patterns~\citep{mathis2020deep, pereira2020quantifying}; and (3) action segmentation, which classifies distinct behavioral states like grooming, rearing or social interactions on every frame~\citep{datta2019computational, pereira2020quantifying}. Each task demands different representations of the same underlying behavioral data, and current approaches typically require task-specific models and extensive labeled datasets~\citep{von2021big}. Furthermore, most approaches fail to leverage the vast amounts of unlabeled data generated by behavior experiments, a significant untapped resource that, if harnessed properly, could substantially improve performance on these downstream tasks.
% \textcolor{blue}{look at end of deleted paragraph for other things to add}
% ADD TO DISCUSSION
% \textcolor{blue}{The introduction is not very clear, the first couple of paragraphs seem pretty broad and do not clearly describe what the problem is, why is it important, why are the current approaches insufficient, and what the proposed approach is...}
% The primary barrier to sophisticated behavioral analysis in most laboratories is not the conceptual challenge of creating experiment-specific datasets, but rather the practical constraints of scale and technical expertise~\citep{von2021big}. Individual labs typically lack sufficient data volume to train high-quality models from scratch and face computational limitations that make large-scale model development infeasible~\citep{pereira2020quantifying, von2021big}. This creates substantial adoption barriers across the scientific community. While recent approaches attempt to reduce labeled data requirements~\citep{mathis2018deeplabcut, pereira2019fast, segalin2021mouse, sun2021task, biderman2024lightning, goodwin2024simple, syeda2024facemap} or eliminate them entirely through unsupervised methods~\citep{berman2014mapping, hsu2021b, sun2022self, luxem2022identifying, weinreb2024keypoint, von2024analysis}, each comes with significant limitations. Supervised methods still require tedious and potentially biased human annotation, while unsupervised approaches often require complex post-processing to yield interpretable results suitable for scientific inquiry. Both approaches are typically task-specific and therefore lack versatility.
We address these challenges through a novel self-supervised pretraining framework for raw videos that produces a robust backbone for multiple downstream neuro-behavioral tasks.
\beast (\textbf{BE}havioral \textbf{A}nalysis via \textbf{S}elf-supervised pretraining of \textbf{T}ransformers) leverages the unique properties of experimental videos by combining masked autoencoding~\citep{he2022masked} to capture rich frame-level appearance information with temporal contrastive learning~\citep{hyvarinen2016unsupervised} to model behavioral dynamics. We introduce a novel frame sampling strategy for the contrastive loss, designed to focus on learning representations of animal behavior against static backgrounds.
% We train \beast on videos from a single experimental setup, rather than attempting to create a universal foundation model for all animal videos.
% This targeted approach creates versatile experiment-specific models that can be fine-tuned for multiple analytical needs within that experimental context.
\beast trains on videos from a single experimental setup, creating tailored, versatile models that can be fine-tuned for multiple analytical needs specific to that experimental context.
We demonstrate the value of this approach through comprehensive evaluation on three downstream tasks: (1) neural encoding in three mouse datasets; (2) pose estimation across four datasets spanning two species and single- and multi-view setups; and (3) action segmentation in both single- and multi-animal setups.
\beast achieves competitive or superior performance for the neural encoding and action segmentation tasks--outperforming baselines such as DINOv2~\citep{oquab2023dinov2}--while eliminating the need for the pose estimation step typically required by existing methods, dramatically reducing manual labeling effort.
At the same time, pose estimation remains valuable for producing interpretable features critical to understanding movement dynamics, and \beast enables significantly improved pose estimation for a given labeling budget.
% \beast achieves competitive or superior performance across these tasks while eliminating the need for the pose estimation step typically required by existing neural encoding and action segmentation methods.
These results establish \beast as a simple yet powerful foundation that can accelerate behavioral understanding across disciplines where fine-grained analysis is essential.
% \textcolor{blue}{Notes:}
% \begin{itemize}
% \item \textcolor{blue}{BEAST builds upon established self-supervised objectives–masked autoencoding and temporal contrastive learning–but we believe our contribution lies in how these components are adapted and applied in a domain-specific manner to address longstanding challenges in behavioral neuroscience}
% \item \textcolor{blue}{While the constituent losses are not novel in isolation, our findings show that adapting these techniques to the behavioral neuroscience domain, in combination with a tailored contrastive sampling strategy and careful experimental design, yields meaningful performance improvements and substantial practical benefits.}
% \item \textcolor{blue}{While BEAST builds on existing self-supervised learning objectives, namely masked autoencoding and temporal contrastive learning, the key contribution is not the introduction of new objectives, but rather the domain-specific adaptation of these techniques to behavioral neuroscience. Specifically, BEAST is, to our knowledge, the first framework to perform self-supervised pretraining on animal behavior videos and reuse the resulting backbone across diverse downstream tasks. This design differs meaningfully from prior work in machine learning for behavioral analysis that either (1) relies on pretraining with internet-scale video (e.g., VideoPrism), (2) focuses on a single behavioral task, or (3) depends on pose-based preprocessing pipelines.}
% \item \textcolor{blue}{mention new workflow appendix}
% \end{itemize}
% Code is available at \url{https://github.com/paninski-lab/beast}.
% ----------------------------------------
% previous version
% ----------------------------------------
% Current approaches to behavioral video analysis, particularly pose estimation and action segmentation, typically rely on bespoke models trained on experiment-specific datasets~\citep{pereira2020quantifying}. This creates substantial barriers to adoption across the scientific community, especially as video datasets grow in scale and complexity~\citep{von2021big}. While recent efforts focus on reducing or eliminating labeled data requirements, each approach comes with significant limitations. Methods that reduce labeled data dependencies are typically specialized for particular downstream tasks, limiting their versatility~\citep{mathis2018deeplabcut, pereira2019fast, sun2021task, biderman2024lightning}. Conversely, unsupervised approaches allow training on larger datasets but frequently necessitate complex post-processing to yield interpretable results suitable for scientific inquiry~\citep{berman2014mapping, hsu2021b, sun2022self, luxem2022identifying, weinreb2024keypoint}.
% Connecting animal behavior to simultaneously recorded neural activity presents a different challenge. Video data is inherently high-dimensional, necessitating dimensionality reduction before fitting models--typically accomplished through pose estimation~\citep{ibl2022reproephys, syeda2024facemap} or autoencoders~\citep{batty2019behavenet, whiteway2021partitioning, wang2023not}. Pose estimation provides interpretable behavioral features but may overlook crucial aspects of behavior essential for understanding neural correlates \citep{stringer2019spontaneous, musall2019single}. Unsupervised methods offer the ability to learn from large amounts of unlabeled data and may therefore capture more behavioral detail~\citep{bohnslav2021deepethogram}, but the efficacy of these approaches for neural encoding remains an open question.
% Our work addresses these challenges through a novel self-supervised pretraining framework for vision transformers that serves as a robust backbone for multiple downstream tasks.
% % , reducing labeled data requirements across applications.
% \beast (\textbf{BE}havioral \textbf{A}nalysis via \textbf{S}elf-supervised pretraining of \textbf{T}ransformers) combines a masked autoencoding loss~\citep{he2022masked} that captures rich frame-level appearance information with a temporal contrastive learning objective~\citep{chen2020simple} that captures temporal dependencies, leveraging large quantities of unlabeled video data to produce experiment-specific pretrained models.
% To pretrain \beast, we introduce a novel frame sampling strategy for the contrastive loss, which is particularly effective for neuroscience applications where controlled experiments generate large quantities of videos with static backgrounds, allowing the contrastive loss to focus on learning representations of the animals' behavioral dynamics.
% We demonstrate the versatility of \beast through comprehensive evaluation across three critical tasks: (1) extraction of behavioral features that demonstrate superior correlation with neural activity compared to existing methods across three mouse datasets; (2) pose estimation across four datasets spanning two species and both single- and multi-animal scenarios; and (3) action segmentation on both single- and multi-animal experiments. These results establish \beast as a simple yet powerful foundation that can accelerate animal behavior understanding across numerous disciplines where fine-grained behavioral analysis is essential but labeled data remains scarce.
% \han{I wonder whether we should write ablout our work's contributions in a more explicite way, like: 1) we designed ssl method that is better suited for neuroscience video (novel sampling strategy) 2) we demonstrated that this dynamic-aware pretraining helps to extract neural activity correlated signals from behavior videos (and nobody has explored that yet) 3) on top of that, the same pretrained model can be used for pose-estimation with better quality (still nobody has done pre-training for this yet) and behavior-segmentation requiring no marker labels 4) We are providing this method as an open-source resource for the community so people can train on their own data}
% \han{I believe our method is particularly well-suited for neuroscience applications because these experiments typically follow a controlled protocol, providing large corpora of stereotyped videos. In these recordings, the background remains largely static and identical, while the animal’s behavior accounts for the primary variations across frames over time. The contrastive loss encourages the model to focus on learning representations that discount the static background and instead emphasize the animal’s behavior. [We could further support this by visualizing an attention saliency map, which may improve the interpretability of our model.] As a result, the learned representations capture the behavioral dynamics of the animal in a much richer way than keypoints alone, enabling improved performance in downstream tasks such as neural encoding, pose tracking, and behavior segmentation.}
\section{Related Work}
\label{sec:related_work}
\paragraph{Neural encoding models.} Neural encoding measures how observable signals predict neural activity, and provide a quantitative framework for interrogating neural representations.
Earlier approaches applied generalized linear models to single neurons using controlled stimuli such as visual or auditory inputs~\citep{paninski2004maximum, truccolo2005point, pillow2008spatio, mcfarland2013inferring}. More recently, deep learning methods have shown great promise in predicting neural population responses to sensory stimuli, including visual~\citep{yamins2014performance, schrimpf2018brain, wang2025foundation}, auditory~\citep{kell2018task, li2023dissecting}, and tactile~\citep{zhuang2017toward} inputs. The widespread adoption of video monitoring during experiments has demonstrated that video-based behavioral covariates explain significant neural variability in both spontaneous~\citep{stringer2019spontaneous, syeda2024facemap} and task-driven behaviors~\citep{musall2019single, ibl2023bwm, wang2023not, chen2024brain, zhang2025neural}. For example,~\citet{musall2019single} showed that uninstructed movements explain a substantial fraction of cortical neural variance, and the International Brain Lab leveraged large-scale, region-resolved encoding analyses to chart the distribution of task-related information across the brain~\citep{ibl2023bwm}. However, extracting rich spatiotemporal information from video remains challenging. Most studies rely on either a small set of keypoints~\citep{syeda2024facemap, ibl2023bwm, wang2023not, chen2024brain} or latent dimensions using PCA~\citep{stringer2019spontaneous, musall2019single} or autoencoders~\citep{batty2019behavenet, wang2023not, chen2024brain}, with limited efforts to predict neural activity directly from raw video (but see~\citet{wang2023not}).
% \han{also these video based models only focuses on reconstruction/variance explaining and does not take the temporal dynamics into account, which is critical to extract signals that are temporally correlated with neural activities}
\paragraph{Large-scale models for behavioral video analysis.}
% Recent advances in large-scale models for animal behavior analysis have taken two predominant approaches.
% The first approach utilizes general pretrained backbones for specific behavioral analysis tasks. DeepLabCut~\citep{mathis2018deeplabcut} demonstrated the effectiveness of ImageNet-pretrained backbones for pose estimation when fine-tuning on limited labeled datasets, though it remains specialized to a single task. VideoPrism~\citep{zhao2024videoprism, sun2024video}, while supporting multiple behavioral tasks, relies on a frozen, general-purpose backbone trained on internet data rather than domain-specific pretraining. Neither approach tailors the pretraining to the specific experimental paradigms where behavioral analysis occurs.
% The second approach develops species- or morphology-specific models. SuperAnimal~\citep{ye2024superanimal} pretrains pose estimation models across experimental configurations within a consistent species, while AP-10K~\citep{yu2021ap} and APT-36K~\citep{yang2022apt} provide cross-species pose estimation datasets for quadrupeds with similar morphology. Domain-focused models like Facemap~\citep{syeda2024facemap} for rodent facial analysis and specialized action segmentation systems for resident-intruder assays~\citep{segalin2021mouse, goodwin2024simple} demonstrate the value of targeted pretraining but remain constrained to single tasks.
% Our work bridges these approaches by pretraining experiment-specific backbones that support multiple downstream tasks.
% This strategy captures the benefits of domain-specific pretraining while avoiding the limitations of task-specific architectures.
Large-scale models for animal behavior analysis have predominantly focused on single tasks.
For pose estimation, methods differ in how they balance flexibility and labeling requirements. DeepLabCut~\citep{mathis2018deeplabcut} leverages ImageNet-pretrained backbones for fine-tuning on experiment-specific labeled datasets, offering flexibility but requiring more manual labeling. This work inspired a range of other general-purpose animal pose estimation tools including LEAP~\citep{pereira2019fast}, DeepPoseKit~\citep{graving2019deepposekit}, TRex~\citep{walter2021trex}, SLEAP~\citep{pereira2022sleap}, and Lightning Pose~\citep{biderman2024lightning}.
In contrast, several specialized pose estimation tools provide tailored solutions for common experimental setups, such as top-down views of freely moving mice~\citep{ye2024superanimal} and facial analysis of head-fixed rodents ~\citep{syeda2024facemap, daruwalla2024cheese3d}, significantly reducing labeling requirements.
Similarly, in action segmentation, specialized systems developed for resident-intruder assays~\citep{segalin2021mouse, goodwin2024simple} achieve high performance but remain limited to a specific experimental paradigm. While VideoPrism~\citep{zhao2024videoprism} offers a general foundation model supporting multiple behavioral tasks~\citep{sun2024video}, it relies on a frozen backbone trained on generic internet data rather than domain-specific content.
Despite these advances, no accessible solutions exist for creating general behavior analysis models that leverage unlabeled data across multiple tasks. \beast addresses this gap by enabling individual labs to develop experiment-specific models from their own unlabeled videos for diverse analyses.
% Despite these advances, a critical gap remains: there are currently no accessible solutions for creating general behavior analysis models that leverage abundant unlabeled data to improve performance across multiple tasks. Our work, \beast, addresses this need by introducing a pretraining strategy that enables individual labs to develop experiment-specific models supporting diverse downstream tasks using their own unlabeled videos.
% This approach allows researchers to extract maximum value from their experimental data without extensive manual labeling.
% Our work, \beast, bridges these gaps by introducing an accessible pretraining strategy that enables individual labs to develop experiment-specific models supporting multiple downstream tasks. This approach combines the advantages of domain-specific pretraining while overcoming the constraints of task-specific architectures.
% The application of foundation models to animal behavior analysis represents an emerging frontier with significant potential. DeepLabCut~\citep{mathis2018deeplabcut} pioneered transfer learning for animal pose estimation, demonstrating that ImageNet-pretrained backbones substantially improve performance on limited labeled datasets.
% % Recent efforts are now developing more specialized pretraining approaches targeted at animal behavior analysis.
% Progress towards large, domain-specific models include SuperAnimal~\citep{ye2024superanimal}, a pose estimation model pretrained across diverse experimental configurations with a consistent species or anatomy. Similarly, Facemap~\citep{syeda2024facemap}
% % and Cheese3D~\citep{daruwalla2024cheese3d} have
% is a pose estimation model for rodent facial analysis that transfers effectively to related datasets. More ambitious cross-species initiatives like AP-10K~\citep{yu2021ap} and APT-36K~\citep{yang2022apt} have compiled diverse datasets spanning multiple quadrupeds to enable robust transfer across species with similar morphology. For action segmentation, notable progress has been made in developing models for specific experimental paradigms, particularly in resident-intruder assays~\citep{segalin2021mouse, goodwin2024simple}. These end-to-end systems demonstrate the value of domain-specific pretraining but remain narrowly focused on particular experimental designs.
% The most ambitious use of foundation models for animal behavior analysis to date~\citep{sun2024video} leverages VideoPrism~\citep{zhao2024videoprism}, a video foundation model trained on millions of internet videos. While this approach shows promise for tasks like localization and action segmentation, it does not consider neural encoding or pose estimation as we do here, nor is it publicly available. \han{I think our method also differs from the ones listed in that it uses a pretraining approach based on a foundation model initialization combined with our own data—none of the other models do this.}
% % our work makes several distinct contributions: (1) we explicitly evaluate performance on neural prediction and pose estimation, critical tasks not previously considered; (2) unlike VideoPrism, which remains unavailable for public use, we provide an accessible pretraining framework that researchers can apply to their own data; and (3) our combination of contrastive and masked modeling objectives creates representations specifically optimized for the multi-task demands of animal behavior analysis.
\paragraph{Self-supervised learning for images and videos.}
% The field of self-supervised learning (SSL) has witnessed remarkable advancement over the past decade.
%, fundamentally transforming visual representation learning.
%While multi-modal paradigms like CLIP~\citep{radford2021learning} have demonstrated exceptional zero-shot capabilities through image-text alignment, we focus specifically on single-modality approaches most relevant to available animal behavior datasets.
Contrastive learning has emerged as a powerful self-supervised learning (SSL) framework; among its many predecessors and variants, SimCLR~\citep{chen2020simple} popularized a simple and effective recipe that maximizes agreement between differently augmented views of the same sample via a contrastive loss in latent space.
{The contrastive method has also been extended to the temporal~\citep{hyvarinen2016unsupervised} and video domains~\citep{qian2021spatiotemporal, recasens2021broaden, dave2022tclr}.
Another line of self-supervised approaches uses knowledge distillation, where a student network learns to match the outputs of a teacher network, such as DINO~\citep{caron2021emerging, oquab2023dinov2, simeoni2025dinov3}.
Masked modeling is a complementary approach that has demonstrated remarkable success, particularly masked autoencoding (MAE)~\citep{he2022masked}, which revolutionized visual self-supervised learning by adapting BERT-style masked prediction to images using Vision Transformers~\citep{dosovitskiy2020image}. VideoMAE~\citep{tong2022videomae} and BEVT~\citep{wang2022bevt} extended this approach to video data by leveraging spatiotemporal dependencies.
Various works have combined contrastive and MAE objectives as a more efficient alternative for capturing spatiotemporal dependencies, as video models can require much more compute for training and inference~\citep{mishra2022simple, huang2023contrastive, lu2023cmae, lehner2024contrastive}. Of note is \textsc{ViC-MAE}~\citep{hernandez2024vic}, which uses patch-based features for a masked autoencoding loss. The local features are also pooled into a global feature vector which is used with a contrastive loss computed across frames from multiple videos.
The efficiency of contrastive-based methods compared to native video models is of particular interest in our application domain, where labs often do not have access to extensive compute resources.
SSL techniques from computer vision have increasingly been adapted to the study of animal behavior. These approaches use SSL to extract useful feature representations from pose, image, or video data, which are then applied to downstream tasks such as action segmentation (Appendix~\ref{sec:appendix_ssl_behavior}).
\section{Methods}
\label{sec:methods}
\beast uses a combination of an image-based masked autoencoding (MAE) loss--which excels at capturing per-frame appearance details--and temporal contrastive loss--which captures dependencies across frames (Fig.~\ref{fig:method}A).
% The MAE loss excels at capturing per-frame appearance details, while the temporal contrastive loss captures dependencies across frames.
This integration enables a single backbone to excel across diverse downstream tasks, from precise keypoint localization to predicting complex structure in neural activity (Fig.~\ref{fig:method}B).
\beast builds upon \textsc{ViC-MAE}~\citep{hernandez2024vic}, which combines masked autoencoding and contrastive losses, but introduces key adaptations for neuroscience applications. The most significant modification is how frames are sampled for the contrastive loss.
\textsc{ViC-MAE} allows any two frames from the same video to be a positive pair, and frames from different videos are negative pairs~\citep{xu2021rethinking}. While this may be appropriate for benchmark datasets with short clips,
% the extended duration of animal behavioral videos motivate a modified frame sampling strategy.
animal behavior experiments generate long-duration recordings where behaviors repeat across time.
% The approach of the ViC-MAE is designed for standard vision benchmark datasets that are composed of many short, visually distinct clips: any two frames from the same video can be a positive pair, while frames from different videos serve as negative pairs~\citep{xu2021rethinking}. In contrast, animal behavior experiments generate continuous, long-duration recordings where behaviors repeat across time.
We instead define positive frames within a narrow temporal window around the anchor ($\pm 1$ frame), while allowing negative frames to be either distant and dissimilar frames in the same video, or from different videos. Crucially, this strategy outperforms that of \textsc{ViC-MAE} (Table~\ref{tab:frame_sampling_ablation}). See Appendix~\ref{sec:beast-train} for more details on our frame selection strategy and additional training and architecture simplifications of \textsc{ViC-MAE}.
% \textcolor{blue}{A strength of \beast is its agnostic approach to transformer architecture, enabling adoption of stronger pretrained backbones as they emerge. To demonstrate this flexibility, and validate our selection of an ImageNet-pretrained \vit, we compare performance of various backbones including DINO~\citep{caron2021emerging, oquab2023dinov2}, CLIP~\citep{radford2021learning}, and Segment Anything~\citep{kirillov2023segment} on neural encoding and pose estimation tasks (Tables~\ref{tab:encoding_backbone} and ~\ref{tab:pose_backbone}).
% }
\begin{figure*}[t!]
\begin{center}
\includegraphics[width=\linewidth]{figures/fig1_overview.pdf}
\end{center}
\caption{\textbf{\beast framework.} \textbf{A:} Our self-supervised pretraining framework \beast combines masked autoencoding~\citep{he2022masked} with temporal contrastive learning~\citep{chen2020simple}. An anchor frame at time \textit{t} is paired with a positive frame from $t\pm1$, while more distant frames from the same video, or frames from other videos, serve as negative examples. Frames are divided into patches, with most patches randomly masked. A vision transformer (\vit) processes the remaining patches, which must reconstruct all patches. The \vit \texttt{CLS} tokens, which serve as a global representation of each frame, are nonlinearly projected to a new space where the contrastive loss pulls anchor-positive pairs together and pushes anchor-negative pairs apart.
\textbf{B:} \beast supports various downstream neuro-behavioral tasks including neural encoding, pose estimation, and action segmentation.
}
\label{fig:method}
\end{figure*}
\paragraph{Vision transformer (\vit).} The standard image \vit~\citep{dosovitskiy2020image} data pipeline starts with a 2D image $\mathbf{x} \in \mathbb{R}^{H \times W \times C}$ ($H$, $W$, $C$ are height, width, channels) and splits it into 2D patches, each with shape $(P \times P \times C)$, where the patch size $P$ is typically 16. Each patch is reshaped to a vector of length $P^2C$, and all patches are concatenated into a sequence of the $N$ flattened 2D patches $\mathbf{x}_p \in \mathbb{R}^{N \times (P^2C)}$. Each flattened patch is mapped with a trainable linear projection to a ``patch token,'' a vector of size $D$.
% We show later how this \texttt{CLS} token as well as the patch embeddings at the output layer can be used for downstream analyses.
We add 1D position embeddings to the patch tokens to retain patch location information.
% Following the original \vit procedure,
We add a learnable \texttt{CLS} token to the patch token sequence, which serves as a global representation for the image.
The resulting patch tokens augmented with position embeddings ($\mathbf{t} \in \mathbb{R}^{N \times D}$) and the concatenated \texttt{CLS} token serve as the input to the standard \vit encoder.
\paragraph{Masked autoencoding loss.} The masked autoencoding (MAE) loss randomly masks out a high proportion of the patch tokens (here, 0.75 \citep{he2022masked}). We call the resulting unmasked tokens $\mathbf{t}_{um} \in \mathbb{R}^{L \times D}$, where $L=0.25 \times N$ is the number of unmasked tokens. The unmasked tokens are processed by the \vit to produce embeddings $\mathbf{z}_{um} = \vit(\mathbf{t}_{um})$. The masked embeddings $\mathbf{z}_m \in \mathbf{R}^{(N-L) \times D}$ (consisting of all zeros) are then combined with the unmasked embeddings processed by \vit to form a complete patch sequence $\mathbf{z} \in \mathbb{R}^{N \times D}$, which is passed through a transformer decoder to produce a reconstruction $\hat{\mathbf{x}}_p \in \mathbb{R}^{N \times P^2C}$ trained via mean square error: $\mathcal{L}_\text{MSE} = \frac{1}{N} \sum_{p=1}^{N} (\mathbf{x}_p - \hat{\mathbf{x}_p})^2$. We refer to the model trained only with this MAE loss as \vitm.
\paragraph{Temporal contrastive loss.} The masked autoencoding loss is sufficient for reconstructing low-level features on individual frames. To imbue our embeddings with temporal information (which may be required for certain downstream tasks), we employ a contrastive loss that produces similar embeddings for frames close in time, and distinct embeddings for frames far apart in time or from different videos. To achieve this, each batch with $B$ samples contains $B/2$ anchor frames, and each anchor frame $\mathbf{x}_t^v$ (from time $t$ and video $v$) has a randomly chosen positive frame from $\mathbf{x}_{t\pm1}^v$. All remaining $B-2$ frames are treated as negative frames. Note this approach differs from other temporal contrastive losses that allow any frame from the anchor frame's video to be a positive frame~\citep{xu2021rethinking, hernandez2024vic}, which does not perform well with temporally-extended behavioral videos (Table~\ref{tab:frame_sampling_ablation}). To improve the robustness of this approach, we select the initial set of anchor frames from a given video to be as visually distinct from each other as possible (Table~\ref{tab:frame_selection_ablation}).
We utilize the InfoNCE loss~\citep{oord2018representation} computed on nonlinear projections of the \texttt{CLS} embeddings (which outperform other frame aggregation methods, Table~\ref{tab:pretraining_pooling}) that are output by the \vit.
% Specifically, we take the \texttt{CLS} embeddings $\{\texttt{CLS}_b \in \mathbb{R}^D\}_{b=1}^B$ and apply a nonlinear projector containing in sequence: a linear layer, Decorrelated Batch Norm (necessary for smooth convergence, Appendix), ReLU, final linear layer.
The projector outputs $\{\mathbf{z}^p_b\}$ are used for the contrastive learning, calculated as $\mathcal{L}_\text{InfoNCE}=-\frac{2}{B}\sum_{i \in \mathcal{A}}\log \frac{\exp(\mathbf{z}_i^p \cdot \mathbf{z}_{i'}^p)}{\sum_{j \neq i} \exp(\mathbf{z}_i^p \cdot \mathbf{z}_j^p)}$, where $i'$ is the positive example associated with $i$ and $\mathcal{A}$ is the set of $B / 2$ anchor frames. We refer to the model trained with both the masked autoencoding and contrastive losses as \beast.
\paragraph{Training and finetuning.}
We initialize our models with pretrained ImageNet weights~\citep{deng2009imagenet,he2022masked}. Details of dataset construction, data augmentations, and batch construction are provided in Appendix~\ref{sec:beast-train}.
% We apply minimal data augmentation during training, consisting of a random resized crop to $244 \times 244$ pixels with a crop ratio between 0.2 to 1.0, and a random horizontal flip with a 50\% probability.
We define the loss as $\mathcal{L}_\text{MSE} + \lambda \cdot \mathcal{L}_\text{InfoNCE}$, where $\lambda$ balances the two losses and is selected using the validation sets of the various datasets. Models are trained for 800 epochs using the AdamW optimizer~\citep{loshchilov2017decoupled} with a cosine annealing learning rate scheduler~\citep{loshchilov2016sgdr}, taking approximately 25 hours on 8 Nvidia A40 GPUs.
\section{Results}
\label{sec:results}
We demonstrate the versatility of \beast through comprehensive evaluation across three downstream neuro-behavioral tasks: (1) neural encoding, which challenges the model to extract spatiotemporal features that can predict patterns in neural activity; (2) pose estimation, which assesses the model's ability to extract fine-grained appearance details; and (3) action segmentation, which evaluates the model's capacity to extract spatiotemporal features required for predicting behavioral sequences. Throughout these evaluations, we present systematic ablation experiments that demonstrate the critical importance of our combined loss functions, and explore various adaptation strategies, including the use of \texttt{CLS} tokens or patch embeddings from a frozen backbone, as well as end-to-end fine-tuning.
% The final section provides additional ablation studies that further elucidate the contribution of each component to the overall performance of our framework [add specific details later].
% -----------------------------------------------------
% Neural encoding
% -----------------------------------------------------
\begin{figure*}[t!]
\begin{center}
\includegraphics[width=\linewidth]{figures/fig2_encoding.pdf}
\end{center}
\caption{\textbf{\beast improves neural encoding.}
\textbf{A:} Example video frame from each dataset.
\textbf{B:} Encoding performance is evaluated across multiple baseline features with both linear models (hatched bars; reduced rank regression, RRR) and nonlinear models (solid bars; temporal convolution network, TCN). \textsc{CEBRA} uses a contrastive loss to embed video frames in a latent feature space. ``Motion energy'' for the IBL-whisker dataset is a 1D estimate of movement calculated as the sum of the absolute pixel differences between successive frames. \beast features outperform all baselines in both linear and nonlinear regimes. Error bars show standard error of the mean (S.E.M.) of Bits per Spike (BPS) across $N$=842 neurons from five test sessions (IBL and IBL-whisker) or S.E.M. of variance explained of the principal components of the neural activity across five test sessions (Facemap; see text).
\textbf{C:} Scatterplot comparison of \beast vs keypoint-based model performance in an example session. Each dot corresponds to an individual neuron. The values in the bottom-right corner represent the session-averaged BPS.
\textbf{D:} \textit{Top, middle}: comparison of the predicted trial-averaged firing rates for \beast and keypoints (lines) and single-trial variability obtained by subtracting the neuron's average firing rate on each trial (heatmaps). \textit{Bottom}: comparison of predicted neural principal components for the Facemap dataset.
}
\label{fig:neural_encoding}
\end{figure*}
\subsection{Neural encoding} Predicting neural activity from behavior videos represents a significant challenge with promising implications for understanding the relationship between brain and behavior~\citep{musall2019single, stringer2019spontaneous, wang2023not}. Traditional approaches often rely on keypoints~\citep{ibl2022reproephys, syeda2024facemap}, potentially missing critical behavioral features that are not included in tracking or are obscured by fur or feathers. Studies have employed Principal Component Analysis on frames~\citep{musall2019single, stringer2019spontaneous}, but this linear technique may inadequately capture subtle behavioral nuances.
% \han{is this part a little overlapped with related work?}
Transformer embeddings offer a compelling alternative, potentially outperforming linear approaches without being constrained by predefined keypoints, thereby providing richer representations that could reveal previously undetectable neuro-behavioral correlations.
\paragraph{Datasets.} We present results on three high-quality neuro-behavioral datasets employing diverse neural recording technologies (Fig.~\ref{fig:neural_encoding}). The first dataset is a head-fixed mouse performing a decision-making task from the International Brain Laboratory~\citep{ibl2023bwm}. This dataset, ``IBL,'' features simultaneous behavioral video and neural activity monitoring at single-cell, single-spike resolution using Neuropixels probes~\citep{jun2017fully} spanning multiple brain regions (average of 168 neurons per session). The second dataset, ``IBL-whisker,'' reuses the same sessions but utilizes a cropped area around the whisker pad in the video, a particularly salient behavioral feature for predicting neural activity~\citep{stringer2019spontaneous, whiteway2021partitioning, syeda2024facemap}. The third dataset comes from the Facemap study~\citep{syeda2024facemap}, where neural activity is captured through two-photon calcium imaging, a technique capable of resolving a large number of individual cells, but unable to detect individual spikes. Following the authors' approach, we predict the principal components of the neural activity to capture predominant variance patterns across the large recorded neural populations.
\paragraph{Models.} We first describe various feature representations used for neural encoding, followed by two models (linear and nonlinear) that we fit to each representation. The first representation for IBL and Facemap are keypoints tracked across the face and body (11 for IBL, 12 for Facemap). For the IBL-whisker dataset, which lacks keypoints, we instead utilize a 1D estimate of whisker pad motion energy (Appendix~\ref{sec:appendix_encoding}). The second representation utilizes PCA applied to raw video frames, while the third leverages CEBRA~\citep{schneider2023learnable} (which employs a contrastive loss to embed inputs in a latent feature space) applied to raw video frames. Finally, we present results (using the \texttt{CLS} token) from \beast, which is initialized with ImageNet weights then fine-tuned separately on each test session.
Table~\ref{tab:neural_encoding_results} shows additional baselines that use frozen features from pretrained MAE~\citep{he2022masked}, DINOv2~\citep{oquab2023dinov2}, and CLIP~\citep{radford2021learning} models.
We train two encoders: linear encoders, which reveal how directly accessible information is within the features, and nonlinear encoders, which better determine the upper bounds of information content in the features. The linear encoder is a reduced rank regression model~\citep{zhang2024exploiting}. The nonlinear encoder is the temporal convolution network (TCN) proposed in the Facemap study~\citep{syeda2024facemap}.
\paragraph{Evaluation.} All model hyperparameters are tuned to ensure robust baseline performance (Appendix~\ref{sec:appendix_encoding}). To evaluate our neural encoding approaches, we utilize the Bits Per Spike (BPS) metric~\citep{pei2021neural} on the spike-resolved IBL dataset (higher values better), and the $R^2$ metric on the neural principal components in the Facemap dataset. All models are evaluated on five test sessions.
\paragraph{Results.} We find that nonlinear encoders consistently outperform their linear counterparts across all datasets and feature representations (Fig.~\ref{fig:neural_encoding} and Table~\ref{tab:neural_encoding_results}). Notably, non-keypoint representations surpass keypoint-based approaches in both IBL and Facemap datasets, confirming our hypothesis that behavior videos contain richer information than what pose estimation typically captures.
\beast shows consistent improvements in neural encoding quality across all datasets and a range of dimensionalities (Fig.~\ref{fig:neural_encoding_dims}), indicating that \beast's exceptional performance is not limited to high-dimensional embedding spaces. Interestingly, the comparable BPS values for \beast in both IBL and IBL-whisker datasets suggest that a substantial portion of the neurally-relevant behavior information is captured by the whisker pad activity, at least in the recorded brain regions.
% \subsection{Ablation experiments}
% We can show ablation results on the Zero-shot encoding task: 1. The pretraining mitigate the gap between imagenet and animal domains. 2. Zero-shot rerquires no additional resources or efforts for later analysis, and it's in animal research. 3. ViT-CM is a strong method consistently on Zero-Shot setting.
% Show BPS score of Zero-shot ViT-MAE (ImageNet), Zero-shot ViT-M (IN + IBL), and Zero-Shot ViT-CM (IN + IBL) on two datasets, IBL and ibl-whisker
% The ViT-based models in Fig.~\ref{fig:neural_encoding} are fine-tuned on a per-session basis without any pretraining, to facilitate comparison with the baselines. We now explore the additional benefits of pretraining by considering zero-shot performance in the ViT models (Table~\ref{tab:zero_shot}). Pretraining on ImageNet with the MAE loss and no fine-tuning (IN) still outperforms the baselines, while additional pretraining over 77 IBL sessions improves performance on both IBL datasets.
% \input{tables/tab1_zero_shot.tex}
\begin{wraptable}{r}{0.58\textwidth}
\vspace{-10pt}
% \include{tables/tab1_zero_shot} % \include inside of wraptable breaks referencing
\centering
\footnotesize
\caption{Zero-shot neural encoding performance (BPS$\pm$1 standard error of mean over $N$=842 neurons across 5 test sessions).} \label{tab:zero_shot}
\begin{tabular}{lcc}
\toprule
Method (TCN) & IBL & IBL-whisker\\
\midrule
\vitm (IN) & $0.321 \pm 0.013$ & $0.301 \pm 0.012$ \\
\vitm (IN+PT) & $0.331 \pm 0.013$ & $0.311 \pm 0.013$ \\
\vit-C (IN+PT) & $0.314 \pm 0.013$ & $0.283 \pm 0.011$ \\
\beast (IN+PT) & $0.292 \pm 0.012$ & $0.309 \pm 0.013$ \\
\beast (IN+PT+FT) & $\mathbf{0.347 \pm 0.014}$ & $\mathbf{0.326 \pm 0.013}$ \\
\bottomrule
\end{tabular}
\vspace{-15pt}
\end{wraptable}
The \vit-based models in Fig.~\ref{fig:neural_encoding} are fine-tuned individually for each session to enable direct comparison with baseline approaches. We investigated whether pretraining provides additional benefits (Table~\ref{tab:zero_shot}).
Strikingly, models pretrained on ImageNet using only the MAE loss, ``\vitm (IN)'', outperform previous baselines without any fine-tuning. Further pretraining on 77 IBL sessions, ``\vitm (IN+PT)'', improves performance on both IBL datasets, validating the importance of domain-specific pretraining.
By incorporating the contrastive objective, \beast achieves superior zero-shot performance: \beast (IN+PT) outperforms the MAE-only variant, as well as a contrastive-only variant \vit-C. Session-specific fine-tuning, ``\beast (IN+PT+FT)'', provides additional significant gains, reaching performance levels comparable to models fine-tuned directly from ImageNet weights (Table~\ref{tab:neural_encoding_results}). Notably, even without fine-tuning, domain-specific pretrained models remain highly competitive, offering researchers a practical option when computational resources for fine-tuning are limited. Finally, we experimented with using the patch embeddings as input to the neural encoder, but found superior performance with the \texttt{CLS} tokens (Table~\ref{tab:encoding_patch}).
% Models pretrained on ImageNet with only the MAE loss--\vitm (IN)--remarkably outperform baselines without any fine-tuning. Additional pretraining across 77 IBL sessions--\vitm (IN+PT)--enhances performance on both IBL datasets, demonstrating the value of domain-specific pretraining. By adding the contrastive objective, \beast zero-shot performance--\beast (IN+PT) also outperforms the MAE-only model, and while additional fine-tuning--\beast (IN+PT+FT)--significantly increases performance, it is mostly on par with fine-tuning directly from the ImageNet weights (Fig.~\ref{fig:neural_encoding}B). These results demonstrate that fine-tuning can improve neural prediction performance, but zero-shot performance after dataset-specific pretraining remains a competitive alternative.
% -----------------------------------------------------
% Pose estimation
% -----------------------------------------------------
\subsection{Pose estimation}
Pose estimation is a fundamental technique in animal behavior analysis~\citep{pereira2020quantifying}, enabling precise quantification of posture and movement. Unlike human pose estimation, which benefits from extensive labeled datasets and standardized anatomy, animal pose estimation presents unique challenges such as scarcity of large annotated datasets and significant morphological diversity across species.
% , and variations in body appearance due to differences in coloration, fur patterns, and experimental conditions.
Pretraining models on large volumes of unlabeled behavior videos can potentially reduce the labeled data requirements for accurate keypoint localization in various experimental paradigms.
\paragraph{Datasets.} We present results on four distinct datasets (Fig.~\ref{fig:pose_estimation}): (1) a head-fixed mouse performing a decision-making task~\citep{ibl2023bwm}; (2) a head-fixed mouse running on a treadmill, seen from two views~\citep{warren2021rapid}; (3) the Caltech Resident-Intruder Mouse (CRIM13) dataset, consisting of two freely interacting mice~\citep{burgos2012social}; and (4) a freely moving weakly electric fish, seen from three views~\citep{biderman2024lightning, pedraja2025direct}.
\input{figures/fig3_pose_estimation}
\paragraph{Models.} We implemented pose estimation models using Lightning Pose~\citep{biderman2024lightning}. We established a strong baseline utilizing a ResNet-50 backbone pretrained on AP-10K~\citep{yu2021ap}, which outperforms a DeepLabCut baseline (ImageNet-pretrained ResNet-50) on all but the CRIM13 dataset (Fig.~\ref{fig:litpose_dlc}).
% To evaluate architectural advantages,
Our second baseline is a Vision Transformer (\textsc{ViT-B/16}) pretrained on ImageNet~\citep{he2022masked} using our own implementation of ViTPose~\citep{xu2022vitpose}, enabling assessment of potential improvements when transitioning from convolutional- to transformer-based architectures. Our own \vit-based models utilize this same architecture.
In the Appendix we provide additional baselines that use fine-tuned DINO~\citep{caron2021emerging}, DINOv2~\citep{oquab2023dinov2}, and Segment Anything~\citep{kirillov2023segment} encoders (which \beast consistently outperforms; Fig.~\ref{fig:pose_backbones}).
For consistency across all model variants, we employ an identical pose estimation head that transforms backbone features into keypoint heatmaps. Given the spatial nature of the task, we use patch embeddings rather than \texttt{CLS} tokens in the transformers, and train all models end-to-end.
\paragraph{Evaluation.} To rigorously evaluate our pose estimation models, we designed a challenging limited-data scenario with only 100 labeled training frames, a realistic constraint for many research settings where extensive annotation is impractical. We measured pixel error between predicted keypoints and ground truth on a test set of novel subjects. For each backbone, we fit three models on different 100-frame subsets. Results are presented as pixel error relative to ensemble standard deviation (e.s.d.) across all seeds and backbones following~\citet{biderman2024lightning}, with error curves showing performance at varying difficulty thresholds. Each point corresponds to keypoints with e.s.d. exceeding the threshold value, with the leftmost portion showing error across all keypoints and rightward movement including only increasingly challenging keypoints (those with higher inter-model variability).
\paragraph{Results.} We find robust improvements in pose estimation quality across all datasets when utilizing \beast (Fig.~\ref{fig:pose_estimation}). The ImageNet-pretrained \vit outperforms the AP-10K-pretrained ResNet-50 on all datasets except the fish, demonstrating the effectiveness of transformers even with limited labels. Notably, pretraining the transformer with the MAE objective on experiment-specific data yields substantial performance gains across all datasets, including the challenging fish dataset.
% where standard transformers initially underperformed.
Augmenting MAE with the contrastive objective (\beast) produces additional performance improvements for some datasets, with particularly pronounced benefits observed in the IBL dataset.
However, pretraining only on the contrastive objective leads to significantly worse results, consistent with the different learning objectives: the temporal contrastive loss emphasizes high-level temporal structure, whereas the MAE loss emphasizes low-level, pixel-level features. Consequently, MAE-pretrained representations are better suited for pixel-level prediction tasks like pose estimation, though addition of the contrastive loss in \beast still provides complementary benefits.
We also find \beast's performance advantages persist when scaling to larger training datasets (Fig.~\ref{fig:pose_estimation_1k_train}).
% -----------------------------------------------------
% Action segmentation
% -----------------------------------------------------
\begin{figure*}[t!]
\begin{center}
\includegraphics[width=\linewidth]{figures/pdfs/fig4_action_segmentation.pdf}
\end{center}
\caption{\textbf{\beast improves action segmentation.}
\textbf{A:} Example frame from each dataset; performance evaluated across multiple baseline features with both TCN (solid) and ensembled (hatched) models.
% \beast single-model patch embeddings outperform all baselines in the resident-intruder assay, and perform about as well as keypoints in IBL.
% With ensembling, \beast matches or exceeds all baselines.
Error bars represent standard error of the mean across five random initializations.
\textbf{B:} Confusion matrices for TCN models based on keypoints and \beast patch embeddings.
\textbf{C:} Example behavior sequences with feature traces (single seed shown for \beast models), ensemble probabilities, ensembled model ethograms, ground truth ethograms, and error frames. PCs of SimBA and \beast features are shown for illustration, but the models utilize the full feature set.
}
\label{fig:action_segmentation}
\end{figure*}
\subsection{Action segmentation}
Action segmentation classifies discrete behaviors using spatiotemporal video features~\citep{pereira2020quantifying}. Similar to pose estimation, a central challenge in animal action segmentation is the lack of large annotated datasets, as behaviors of interest often vary across species and experimental contexts. Many current approaches rely on keypoints~\citep{branson2009high, kabra2013jaaba, segalin2021mouse, gabriel2022behaviordepot, goodwin2024simple}, requiring an initial labor-intensive and error-prone preprocessing step. Vision transformer embeddings eliminate this preprocessing requirement and provide an attractive alternative if they match or exceed the performance of keypoint-based methods.
\paragraph{Datasets.}
We present results on two datasets (Fig.~\ref{fig:action_segmentation}): (1) the ``IBL'' dataset~\citep{ibl2023bwm}, which contains four behavior classes for the paw nearest the camera; and (2) the Caltech Mouse Social Interactions (CalMS21) dataset~\citep{sun2021multi}, a resident–intruder assay of interacting mouse pairs that contains four social behavior classes.
% (when there is no social interaction).
\paragraph{Models.}
We implemented models using two types of embeddings from frozen \vit models: (1) \texttt{CLS} tokens and (2) per-patch embeddings. For \texttt{CLS} embeddings, we tested both a linear model and a TCN \citep{lea2016temporal}, enriching the input with inter-frame differences~\citep{blau2024study}. We used a sliding window over this feature sequence to predict the action class of the central frame. For patch embeddings, we applied multi-head attention pooling \citep{lee2019set, yu2022coca, sun2024video} to integrate information across patches, then concatenated the resulting frame-level embeddings with their inter-frame differences before processing them through a TCN (Fig.~\ref{fig:mha_pooling_model}).
For IBL, we compared against three baseline features: (1) a single paw keypoint, obtained using five pose estimation networks (each trained with 7,000 labeled frames) post-processed with an Ensemble Kalman Smoother~\citep{biderman2024lightning}; (2) principal components of the video frames; and (3) the \texttt{CLS} and patch embeddings extracted from a frozen-weight \vit pretrained on ImageNet with MAE loss (which outperformed patch embeddings from DINOv2 in both datasets; Table~\ref{tab:action_segmentation}). For CalMS21, we compared against four baseline features: (1) Trajectory Embedding for Behavior Analysis (TREBA)~\citep{sun2021task}, a self-supervised feature extraction method for keypoint trajectories; (2) Simple Behavioral Analysis (SimBA)~\citep{goodwin2024simple}, which extracts hundreds of hand-crafted features from the keypoint trajectories; (3) principal components of video frames; and (4) the \texttt{CLS} and patch embeddings from a frozen-weight \vit pretrained on ImageNet with MAE loss. The pose estimator used for TREBA and SimBA was trained with 15,000 labeled frames~\citep{sun2021multi}.
For all baselines except SimBA, we also concatenated inter-frame differences.
\paragraph{Evaluation.}
All model hyperparameters are tuned to ensure robust baseline performance (Appendix~\ref{sec:appendix_action}).
We evaluate performance on held-out animals using the macro-averaged F1 score. For the CalMS21 dataset, following~\cite{sun2021multi}, we average the F1 score over the attack, investigation and mount classes. For all models we train five networks using different random initializations. We also report the results of model ensembles by averaging logits across seeds before applying softmax.
\paragraph{Results.} \beast demonstrates strong action segmentation performance across all datasets (Fig.~\ref{fig:action_segmentation}).
Remarkably, ImageNet-pretrained \vitm patch embeddings nearly match keypoint-based methods despite utilizing a frozen, general-purpose backbone. This establishes a competitive baseline without requiring the thousands of labels needed to train pose estimation networks.
On IBL, \beast improves upon ImageNet baselines. The keypoint-based model excels here due to action classes corresponding to paw movements easily captured by pose estimation, but this advantage disappears with ensembling: \beast ensemble F1 matches the keypoint ensemble. CalMS21 better demonstrates \beast's abilities, which surpasses the SimBA baseline and substantially outperforms the TREBA baseline (Table~\ref{tab:action_seg_baselines}). The ensembled F1 score of 0.84 places our result in the top 15 of the Multi-Agent Behavior Challenge on AIcrowd.com (top score of 0.89).
% Ensembling improves performance across all feature types except PCA in CalMS21.
Additional experiments confirm domain-specific pretraining benefits:
\beast \texttt{CLS} tokens consistently outperform their ImageNet-pretrained counterparts (Table~\ref{tab:action_seg_baselines}), though patch-based models perform significantly better due to their enhanced spatial resolution and multi-headed attention pooling. An ablation experiment on \beast's loss terms show that backbones pretrained with a contrastive-only (\vit-C) or MAE-only (\vit-M) loss do not perform as well as their combination (Table~\ref{tab:action_segmentation}). Across all experiments, nonlinear models consistently outperform their linear counterparts, except for PCA features on CalMS21 (Table~\ref{tab:action_segmentation}). All evaluations use frozen backbones with only linear/TCN heads fine-tuned, suggesting further gains may be possible through full backbone fine-tuning.
\begin{wraptable}{r}{0.56\textwidth}
\vspace{-15pt}
\centering
\footnotesize
\caption{Action segmentation performance (F1$\pm$S.E.M.).} \label{tab:action_seg_baselines}
\vspace{-5pt}
\begin{tabular}{lcc}
\toprule
Method (TCN) & IBL & CalMS21\\
\midrule
\treba & -- & $0.72 \pm 0.01$ \\
\vitm (IN) (\texttt{CLS}) & $0.79 \pm 0.00$ & $0.60 \pm 0.00$ \\
% \vitm (IN) (patch) & 0.837 $\pm$ 0.008 & 0.784 $\pm$ 0.004 \\
\vitm (IN) (patch) & $0.84 \pm 0.00$ & $0.74 \pm 0.00$ \\
% \vitm (IN+PT) (CLS) & -- & -- \\
% \vitm (IN+PT) (patch) & -- & -- \\
\beast (IN+PT) (\texttt{CLS}) & $0.81 \pm 0.00$ & $0.63 \pm 0.00$ \\
% \beast (IN+PT) (patch) & 0.871 $\pm$ 0.011 & 0.842 $\pm$ 0.013\\
\beast (IN+PT) (patch) & $\mathbf{0.87 \pm 0.01}$ & $\mathbf{0.81 \pm 0.01}$\\
\bottomrule
\end{tabular}
\vspace{-10pt}
\end{wraptable}
\beast's advantages extend beyond absolute F1 improvements. Pose estimation-based approaches require both extensive labeling and iterative training and validation of pose estimation models before action segmentation, often a months- or even years-long process~\citep{ibl2022video}. \beast eliminates this entire pipeline, achieving competitive or superior performance using only unlabeled video for pretraining.
% -----------------------------------------------------
% Pose estimation
% -----------------------------------------------------
\begin{figure*}[t!]
\begin{center}
\includegraphics[width=\linewidth]{figures/pdfs/fig3_pose_estimation.pdf}
\end{center}
\caption{\textbf{\beast improves pose estimation.}
\textbf{A:} Example frame from each dataset overlaid with ground truth annotations. Green stars indicate the highlighted keypoint in panel B.
\textbf{B:} Example traces from the ResNet-50 (gray) and \beast (green) models for a single keypoint in a held-out video. \beast traces evolve more smoothly in time and do not contain erroneous jumps like the ResNet-50 baseline.
\textbf{C:} Pixel error as a function of keypoint difficulty (see main text; smaller is better): left-hand side shows performance across all keypoints; moving to the right drops the easier keypoints defined by inter-seed and -model prediction variance. Vertical dashed lines indicate the percentage of data used for the pixel error computation. \textsc{ViT-M (IN)} is a \vit backbone pretrained on ImageNet with a masked autoencoding loss; \textsc{ViT-M (IN+PT)} uses the same architecture and loss but is initialized with ImageNet-pretrained weights then further pretrained on experiment-specific unlabeled frames.
\textsc{ViT-C (IN+PT)} performs the experiment-specific pretraining using the temporal contrastive loss only.}
\label{fig:pose_estimation}
\end{figure*}
\section{Discussion}
\label{sec:discussion}
This work introduces \beast, a framework for self-supervised vision transformer pretraining leveraging domain-specific video data. We demonstrated \beast's significant benefits across neural encoding, pose estimation, and action segmentation tasks.
Notably, \beast outperforms DINOv2 and other computer vision foundation models across all tasks (Table~\ref{tab:neural_encoding_results}), demonstrating the power of domain-specific pretraining.
% While \beast requires substantial computational resources (hundreds of GPU hours for large-scale pretraining on a single dataset), this initial investment creates a versatile backbone applicable to multiple downstream tasks. For many laboratories, this computational cost is far outweighed by the dramatic reduction in human labeling time and the improved accuracy achieved without iterative model refinement.
Our frame-based approach is an efficient alternative to native video models like \videomae~\citep{tong2022videomae}, which require significantly more compute for training and inference (Table~\ref{tab:efficiency}); however \beast still outperforms a frozen \videomae on the neural encoding task (Table~\ref{tab:neural_encoding_results}).
Our work establishes a foundation for several promising future directions. Investigation of transformer attention and learned features could clarify how \beast operates across different tasks. The black-box nature of ViT embeddings presents interpretability challenges in scientific contexts where transparent representations like pose estimates are often preferred. Visualization methods provide initial insights (Fig.~\ref{fig:embedding_visualization}), but systematic analysis of what features drive performance on different tasks would strengthen our understanding of when and why \beast succeeds.
While we have demonstrated \beast's performance across diverse experimental contexts--including head-fixed and freely moving animals, single- and multi-view setups, and solitary or social behaviors--validation across more environments and species is needed. The success of masked autoencoding and contrastive losses in general computer vision suggests \beast should adapt well to naturalistic settings (e.g., home cages, zoos, field studies). The primary challenge will be adjusting the frame sampling strategy to accommodate different visual statistics and behavioral distributions in these less controlled environments.
Finally, we see two complementary paths toward making powerful self-supervised models more accessible to individual labs. First, using smaller transformer architectures will reduce pretraining, fine-tuning, and inference costs, but may sacrifice performance. Second, \beast's framework could enable foundation models of animal behavior trained across diverse datasets, rather than the dataset-specific pretraining we present here. Such foundation models would allow labs to finetune already-powerful pretrained models rather than pretraining themselves. Together, these approaches would lower barriers to adoption and enable wider application of self-supervised learning across the neuroscience community.
% \vspace{15pt}
\paragraph{Acknowledgments and Disclosure of Funding}
This work was supported by Gatsby Charitable Foundation GAT3708, NIH U19NS123716 and R50NS145433, NSF 1707398, the National Science Foundation and DoD OUSD (R\&E) under Cooperative Agreement
PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence), Simons Foundation, Wellcome Trust 216324, and funds from Zuckerman Institute Team Science.
\newpage
\section*{Reproducibility statement}
We have attempted to provide sufficient detail to foster reproducibility of all analyses presented in this manuscript.
\paragraph{Data availability.} All datasets are sourced from public repositories, with links and accompanying licenses provided in Appendix A.
\paragraph{Model availability.} Pretrained \beast models for each dataset are available at \url{https://huggingface.co/collections/paninski-lab/beast} under the CC-BY 4.0 license.
\paragraph{Code availability.}
\begin{itemize}
\item \beast frame selection, pretraining, and inference: \url{https://github.com/paninski-lab/beast/releases/tag/v1.0.0}.
\item Neural encoding:
\begin{itemize}
\item Reduced Rank Regression for IBL: \url{https://github.com/realwsq/brainwide-RRR-encoding-model}
\item Reduced Rank Regression for Facemap: \url{https://github.com/MouseLand/facemap/blob/v1.0.7/facemap/neural_prediction/prediction_utils.py#L110}
\item Temporal Convolution Network: \url{https://github.com/MouseLand/facemap/blob/v1.0.7/facemap/neural_prediction/neural_model.py}
\end{itemize}
\item Pose estimation: \url{https://github.com/paninski-lab/lightning-pose/releases/tag/v2.0.2}
\item Action segmentation: \url{https://github.com/paninski-lab/lightning-action/releases/tag/v1.0.1}.
\end{itemize}
\paragraph{Analysis details.}
\begin{itemize}
\item Pretraining (Appendix~\ref{sec:beast-train}): \beast pretraining procedure and frame selection strategy.
\item Neural encoding (Appendix~\ref{sec:appendix_encoding}): Model architectures, training procedures, and hyperparameter tuning details.
\item Pose estimation (Appendix~\ref{sec:appendix_pose}): Training procedures and hyperparameter selection.
\item Action segmentation (Appendix~\ref{sec:appendix_action}): Model architectures, training procedures, and hyperparameter tuning details.
\end{itemize}
\newpage
% \textcolor{blue}{Notes:}
% \textcolor{blue}{How the ML community can build on this work: Our findings reveal several high-impact research opportunities with broad applicability beyond neuroscience.c
% \textbf{Efficient video understanding}: Our frame-based approach leaves room for exploring native video models (VideoMAE, etc.) that could better capture temporal dynamics and lead to better performance. As video models require more compute for training and inference, attention to data efficiency challenges will be a key component of this direction.
% \textbf{Multi-view self-supervision}: Explicitly modeling multi-view relationships could provide several advantages. Most fundamentally, proper modeling could lead to 3D-aware representations that learn explicit scene structure, producing more robust behavioral representations that are invariant to camera placement (a common source of variability in behavioral studies).
% \textbf{Foundation models}: Larger opportunities exist to train models on heterogeneous behavioral datasets spanning multiple experimental setups and species. Successfully developing generalizable backbones (e.g., mouse- or primate-specific models) would eliminate the need for dataset-specific pretraining and enable backbone sharing across experiments and labs, significantly reducing the barriers for adopting these methods in the broader neuroscience community.
% \textbf{Improved architectures and learning objectives for better performance}}
\begin{appendix}
% \section{Appendix}
{\centering\textbf{\Large Appendix}
\vspace{0.25in}}
\input{appendix/datasets}
\input{appendix/beast.tex}
\input{appendix/encoding.tex}
\input{appendix/pose.tex}
\input{appendix/action.tex}
\input{appendix/behavior_ssl_refs.tex}
% \clearpage
\input{appendix/workflow}
\clearpage
\input{appendix/braoder_impact}
% \clearpage
% \input{iclr2026/appendix/rebuttal}
\end{appendix}
\section{Datasets}
All datasets used for this study were collected in compliance with
the relevant ethical regulations (see the references for each dataset).
\subsection{IBL} This dataset \citep{ibl2023bwmdata} from the International Brain Lab (IBL) and consists of head-fixed mice performing a decision-making task \citep{ibl2021behavior, ibl2022reproephys, ibl2023bwm}. Two cameras--`left' (60 Hz) and `right' (150 Hz)--capture roughly orthogonal side views of the mouse’s face and upper trunk during each session. Frames are downsampled to 256 $\times$ 320 pixels for labeling and video storage. We accessed the raw videos and neural activity under the CC-BY 4.0 license using these instructions: \url{https://int-brain-lab.github.io/iblenv/notebooks_external/data_release_brainwidemap.html}. Frames for pretraining are available at \url{https://doi.org/10.6084/m9.figshare.31310143} under the CC-BY 4.0 license.
\paragraph{Pose estimation.} Frames were reshaped during training to 256 $\times$ 256 pixels. Two keypoints were labeled per view, one for each paw. We accessed the initial pose estimation labels from the public repository at \url{https://ibl-brain-wide-map-public.s3.amazonaws.com/aggregates/Tags/2023_Q1_Biderman_Whiteway_et_al/_ibl_videoTracking.trainingDataPaw.7e79e865-f2fc-4709-b203-77dbdac6461f.zip} under the CC-BY 4.0 license. This dataset contains 6,071 labeled train frames from 35 animals and 1,446 labeled test frames from 10 animals, all at 128 $\times$ 102 pixel resolution. Despite the large number of labeled frames, we observed poor performance in sessions with bright lights or other distractors. Additionally, the low resolution often obscured fine details, for example complicating discrimination of individual paws when close together. To address these limitations, we retrieved the full-resolution frames (1280 $\times$ 1024 for left view, 640 $\times$ 512 for right) from the raw videos and downsampled them to 320 $\times$ 256 pixels. This higher resolution revealed occasional labeling errors, which we manually corrected. We then added 1,437 newly labeled frames from 15 additional animals, creating an expanded training set of 7,609 frames from 50 animals. The updated pose estimation dataset is available at \url{https://ibl-brain-wide-map-public.s3.amazonaws.com/aggregates/Tags/2026_Q1_Wang_Yu_et_al/_ibl_videoTracking.trainingDataPaw.zip} under the CC-BY 4.0 license.
\paragraph{Action segmentation.} Four action classes are labeled for the paw closest to the camera: (1) still; (2) grooming; (3) turning the wheel; and (4) fidget (any movement that is not grooming or wheel turning). We accessed the initial action segmentation labels from \url{https://doi.org/10.6084/m9.figshare.27479760.v1} under the CC-BY 4.0 license. This dataset contains 1,000,000 frames from 10 animals, of which 14,107 are labeled.
We expanded this dataset as the original study~\citep{blau2024study} only used five animals each for training and testing. First, we trained an ensemble of five TCN-based action segmentation models on all 10 existing animals. We applied these models to a new batch of 53 sessions and calculated the variance in predicted probabilities across all models for each frame. The 19 sessions with the highest average ensemble variance (indicating where models disagreed most) were selected for further labeling. We then labeled 36,009 additional frames from these sessions. For the analyses in this paper, we selected the subset of all labeled sessions that are included in the \beast pretraining sessions, and split these into train (32,521 frames from 18 animals) and test sets (7,786 frames from 5 animals). The updated action segmentation dataset is available at \url{https://ibl-brain-wide-map-public.s3.amazonaws.com/aggregates/Tags/2026_Q1_Wang_Yu_et_al/_ibl_videoActionSegmentation.trainingDataNearPaw.zip} under CC-BY 4.0 license.
\paragraph{Neural encoding.} For the neural analysis we use a subset of the IBL repeated site dataset \citep{ibl2022reproephys}. This dataset consists of Neuropixels recordings collected from 10 labs with standardized experimental pipelines. The recordings target the same five
brain regions across all mice: VISa (primary visual cortex), CA1 and DG (hippocampus), and LP and PO (thalamic nuclei). We evaluate neural encoding models on five randomly selected sessions. Moreover, we used the trial-aligned neural activity data, taking 2 seconds of activity aligned to wheel movement onset. We binned the spikes every 20 ms to get a total of 100 bins per trial. We also filtered out the low firing rate neurons by setting a minimum threshold of 2 Hz. For each trial we randomly select 100 video frames (out of a possible 120) to fit the downstream neural encoding models, resulting in an effective sampling rate of 50 Hz to match the neural data.
\subsection{IBL-whisker} We localize the whisker pad using anchor keypoints on the nose and eye, following the procedure in \cite{ibl2022video}. We use the same sessions and neural activity as the ``IBL'' dataset.
\subsection{Mirror-mouse} Head-fixed mice ran on a circular treadmill while avoiding a moving obstacle \citep{warren2021rapid}. The treadmill had a transparent floor and a mirror mounted inside at 45°, allowing a single camera to capture two roughly orthogonal views (side view and bottom view via the mirror) at 250 Hz. The camera was positioned at a large distance from the subject ($\sim$1.1 m) to minimize perspective distortion. Frames are 406 $\times$ 396 pixels and reshaped during pose estimation training to 256 $\times$ 256 pixels. Seventeen keypoints were labeled across the two views including seven keypoints on the mouse’s body per view, plus three keypoints on the moving obstacle. The full training dataset consists of 789 labeled frames across 10 animals; the test dataset consists of 253 labeled frames across three animals. We accessed the labeled pose estimation dataset from \url{https://doi.org/10.6084/m9.figshare.24993315.v1} under the CC-BY 4.0 license. Frames for pretraining are available at \url{https://doi.org/10.6084/m9.figshare.31320529} under the CC-BY 4.0 license.
\subsection{Mirror-fish} Mormyrid fish of the species Gnathonemus
petersii swam freely in and out of an experimental tank, capturing
worms from a well \citep{biderman2024lightning, pedraja2025direct}. The tank had a side mirror and a top mirror, both at 45°, providing three different views seen from a single camera at 300 Hz. The camera was placed $\sim$1.7 m away from the center of the fish tank to reduce distortions. Frames are 384 $\times$ 512 pixels and reshaped during training to 256 $\times$ 384 pixels. Seventeen body parts were labeled across each of three views for a total of 51 keypoints. The full training dataset consists of 373 frames across three animals; the test dataset consists of 94 frames across three animals. We accessed the labeled pose estimation dataset from \url{https://doi.org/10.6084/m9.figshare.24993363.v1} under the CC-BY 4.0 license. Frames for pretraining are available at \url{https://doi.org/10.6084/m9.figshare.31320775} under the CC-BY 4.0 license.
\subsection{CRIM13} The Caltech Resident-Intruder Mouse (CRIM13) dataset \citep{burgos2012social} consists of two mice interacting in an enclosed arena, captured by a top-view camera at 30 Hz. Frames are 480 $\times$ 640 pixels and reshaped during pose estimation training to 256 $\times$ 256 pixels. Seven keypoints were labeled on each mouse for a total of 14 keypoints \citep{segalin2021mouse}. The full training dataset consists of 3,986 frames across four resident mice; the test dataset consists of 1,274 frames across the same four resident mice but a different set of intruder mice. The original dataset is available at \url{https://data.caltech.edu/records/4emt5-b0t10}. We accessed the labeled pose estimation dataset from \url{https://doi.org/10.6084/m9.figshare.24993384.v1} under the CC-BY 4.0 license. Frames for pretraining are available at \url{https://doi.org/10.6084/m9.figshare.31321213} under the CC-BY 4.0 license.
\subsection{CalMS21} The Caltech Mouse Social Interactions (CalMS21) dataset \citep{sun2021multi}, like CRIM13, consists of two mice interacting in an enclosed arena, captured by a top-view camera at 30 Hz.
The dataset consists of videos with tracked poses and frame-level behavior annotations. Four behavior classes are labeled: attack, investigation, mount, and other (i.e., none of the above). The full training dataset consists of 506,668 frames across 68 videos; the test dataset consists of 262,107 frames across 19 videos. We accessed the pose estimates, TREBA features \citep{sun2021task}, and behavior annotations from \url{https://doi.org/10.22002/D1.1991} under the CC-BY 4.0 license. Frames for pretraining are available at \url{https://doi.org/10.6084/m9.figshare.31337953} under the CC-BY 4.0 license.
\subsection{Facemap} Head-fixed mice were free to run on an air-floating ball in darkness \citep{syeda2024facemap}. A single infrared camera captured one of several side or front views of the mouse’s face and upper trunk during each session at 50 Hz. Fifteen keypoints were labeled across the face (mouse, nose, whiskers, eyes) and paw. Neural activity was recorded across visual and sensorimotor areas using two-photon calcium imaging at 3 Hz. Approximately 30,000 to 50,000 cells were recorded in a given session. In our encoding task, we predict the 128 neural principal components following \citet{syeda2024facemap}. We evaluate neural encoding models on five randomly selected sessions. The publicly available data did not contain additional videos, so we only fine-tuned neural encoding models with this dataset. We accessed the raw videos, pose estimates, and neural activity from \url{https://doi.org/10.25378/janelia.23712957} under the CC-BY-NC 4.0 license.
\begin{table}[ht]
\centering
\caption{Number of training/validation/test frames utilized across tasks, with number of source videos in parentheses. Pretraining frames are unlabeled, and the number includes both anchor frames and temporal context frames. Pose estimation and action segmentation frames are labeled; these models are trained across multiple videos. Neural encoding frames have matched neural activity (output) for each time point of behavior (input); these models are trained on single videos, since the neural populations change from one session to the next.}
\renewcommand{\arraystretch}{1.3}
% \begin{table}
\begin{adjustbox}{width=\textwidth}
\begin{tabular}{rcrrrrcc}
\toprule
& \multirow{2}{*}{Pretraining} & \multicolumn{2}{c}{Pose estimation} & \multicolumn{2}{c}{Action segmentation} & \multicolumn{2}{c}{Neural encoding} \\
\cmidrule(l){3-4} \cmidrule(l){5-6} \cmidrule(l){7-8}
& & train/val & test & train/val & test & train/val & test \\
\midrule
IBL \citep{ibl2023bwm} & 270,025 (77) & 7,609 (128) & 1,446 (19) & 35,521 (18) & 7,786 (5) & 338,760 (5) & 42,720 (5)\\
Mirror-mouse \citep{warren2021rapid} & 94,272 (17) & 789 (17) & 253 (5) & - & - & - & - \\
Mirror-fish \citep{biderman2024lightning} & 47,943 (28) & 373 (28) & 94 (10) & - & - & - & - \\
CRIM13 \citep{burgos2012social} & 99,926 (36) & 3,986 (37) & 1,274 (19) & - & - & - & - \\
CalMS21 \citep{sun2021multi} & 91,087 (70) & - & - & 506,668 (68) & 262,107 (19) & - & - \\
Facemap \citep{syeda2024facemap} & - & - & - & - & - & 1,790,200 (5) & 447,550 (5) \\
\bottomrule
\end{tabular}
\end{adjustbox}
\label{tab:data}
\end{table}
\section{\beast implementation} \label{sec:beast-train}
\beast utilizes a standard \textsc{ViT-B/16} architecture~\citep{dosovitskiy2020image}, and combines a masked autoencoding and temporal contrastive learning loss. This approach is also taken by \textsc{ViC-MAE}~\citep{hernandez2024vic}, and we introduce key adaptations and simplifications to make \beast suitable for applications in behavioral neuroscience, which we elaborate on more in the following sections.
\subsection{Architecture} \label{sec:beast_arch}
We selected the ``base'' \textsc{ViT-B/16} architecture over other \vit variants because it is expressive enough to capture rich frame-level information while remaining computationally efficient for training and inference on long videos. While \textsc{ViC-MAE} employs a pooled attention layer to transform patch embeddings into a 768-dimensional vector that serves as the global image representation for the temporal contrastive loss, we use the standard \texttt{CLS} token (Table~\ref{tab:pretraining_pooling}). This \texttt{CLS} token is passed through a nonlinear projector consisting of four components: a linear layer, Batch Norm (necessary for stable training, Fig.~\ref{fig:decorr_batch_norm}), ReLU activation, and a final linear layer.
\begin{figure*}[h!]
\begin{center}
\includegraphics[width=0.9\linewidth]{figures/pdfs/appendix_train_contrast_acc.pdf}
\end{center}
\vspace{-10pt}
\caption{\textbf{Effect of Batch Normalization on contrastive training accuracy.}
Training contrastive accuracy improves significantly with the use of Batch Normalization (BatchNorm) in the nonlinear projection head. Models trained with BatchNorm exhibit smoother learning curves and achieve higher final accuracy compared to those without BatchNorm. ``Accuracy'' is defined as the fraction of anchor frames in a batch where the corresponding positive frame has a logit score higher than that of all other negative frames.}
\label{fig:decorr_batch_norm}
\end{figure*}
The \beast backbone is initialized using weights pretrained on ImageNet with a masked autoencoding loss. This model has exceptional zero-shot performance on neural encoding, outperforming all other baselines, and is further improved with domain-specific pretraining (Table~\ref{tab:zero_shot}). We find other pretrained backbones--DINOv2~\citep{oquab2023dinov2} and CLIP~\citep{radford2021learning}--also have strong zero-shot performance, but do not significantly improve upon MAE+ImageNet (Table~\ref{tab:neural_encoding_results}), indicating this is a reasonable pretrained backbone from which to start our own domain-specific pretraining (see Fig.~\ref{fig:pose_backbones} and Table~\ref{tab:action_segmentation} for similar results on pose estimation and action segmentation, respectively).
\beast incorporates time through its temporal contrastive loss, which efficiently captures information across frames. We explored the performance of \videomae~\citep{tong2022videomae}, a related video model pretrained on Kinetics-400~\citep{kay2017kinetics}. We found that the frozen \videomae backbone does outperform the frame-based \vit-MAE on the neural encoding task (Table~\ref{tab:neural_encoding_results}). Interestingly \beast, which applies additional domain-specific pretraining to \vit-MAE, still outperforms \videomae. This raises the intriguing question of whether further domain-specific pretraining of \videomae could surpass \beast performance. We view this as an important direction for future work.
% \begin{table}[ht]
% \centering
% \footnotesize
% \caption{Performance of frozen pretrained backbones. We evaluate the representations of these models using zero-shot performance (except for the pretrained \beast model) on the neural encoding task using the bits per spike (BPS) metric. We report the mean and standard deviation of BPS across five test sessions.}
% \label{tab:encoding_backbone}
% \begin{tabular}{lcccc}
% \toprule
% \multirow{2}{*}{Method} & \multicolumn{2}{c}{IBL} & \multicolumn{2}{c}{IBL-whisker} \\
% \cmidrule(lr){2-3} \cmidrule(lr){4-5}
% & TCN & RRR & TCN & RRR \\
% \midrule
% DINOv2 (IN) & $0.329\pm0.095$ & $0.207\pm0.068$ & $0.302\pm0.072$ & $\mathbf{0.151\pm0.031}$\\
% CLIP & $0.326\pm0.095$ & $0.195\pm0.061$ & $0.300\pm0.076$ & $0.143\pm0.027$\\
% \vit-MAE (IN) & $ 0.325\pm0.091$ & $0.201\pm0.070$ & $0.307\pm0.068$ & $0.142\pm0.051$\\
% \videomae (Kinetics-400)& $0.332 \pm 0.055$ & $-$ & $0.311 \pm 0.087$ & $-$ \\
% \beast & $\mathbf{0.337\pm0.103}$ & $\mathbf{0.277\pm0.076}$ & $\mathbf{0.317\pm0.083}$ & $0.138\pm0.029$ \\
% \bottomrule
% \end{tabular}
% \end{table}
\subsection{Training}
We discuss frame selection and sampling strategies, data augmentations, and global pooling strategies below.
We apply the MAE loss uniformly across all frame types (anchor, positive, negative), whereas \textsc{ViC-MAE} only applies the MAE loss to anchor frames.
The global batch size is set to 2048, distributed across 8 Nvidia A40 GPUs. We use the AdamW optimizer with a weight decay of 0.05. The learning rate is scheduled using PyTorch’s \texttt{OneCycleLR} scheduler, with a base learning rate of $5\times10^{-5}$. The maximum learning rate is computed as $\text{max\_lr} = \text{base\_lr} \times \frac{\text{global\_batch\_size}}{256}$, with \texttt{pct\_start} set to 0.15 and \texttt{div\_factor} set to 10. We train all models for 800 epochs.
\paragraph{Frame selection strategy.}
Animal behavior videos often contain extended periods of inactivity or repetitive behaviors. Pretraining \vit models on all available frames would capture redundant information and increase computation time. Instead, we extract diverse frames exhibiting distinct poses (to optimize the masked autoencoding loss) while preserving local temporal relationships (to leverage the temporal contrastive loss). We first downsample all frames to 32 $\times$ 32 pixels and calculate motion energy, defined as the absolute pixel-wise differences between consecutive frames. We eliminate frames in the bottom 50th percentile of motion energy, retaining only those with significant movement. We then apply $k$-means clustering to the remaining downsampled frames, with the number of clusters matching our target number of anchor frames per video (e.g., 600). For each cluster, we select the frame closest to the cluster center, along with its immediate predecessor and successor in time, which serve as positive examples for the contrastive loss (for a total of, e.g., 1800 frames per video). This methodology ensures a high-quality, diverse dataset for efficient pretraining and outperforms random frame selection in the neural encoding task (Table~\ref{tab:frame_selection_ablation}).
%%%%Table to compare with ViC-MAE negative frame sampling strategy%%%%%%%
\begin{table}[ht]
\centering
\footnotesize
\caption{Frame selection strategy ablation. We pretrain models using either random selection (``Random'') or the PCA+$k$-means strategy described above (``Selected''). We evaluate the representations of these models using zero-shot performance on the neural encoding task using the bits per spike (BPS) metric. We report the mean and standard deviation of BPS across five test sessions.}
\label{tab:frame_selection_ablation}
\begin{tabular}{lcccc}
\toprule
\multirow{2}{*}{Method} & \multicolumn{2}{c}{IBL} & \multicolumn{2}{c}{IBL-whisker} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5}
& TCN & RRR & TCN & RRR \\
\midrule
\vitm (Random) & $0.311 \pm 0.107$ & $0.168 \pm 0.091$ & $0.319 \pm 0.084$ & $\mathbf{0.147 \pm 0.026}$ \\
\beast (Random) & $0.319 \pm 0.104$ & $0.176 \pm 0.094$ & $\mathbf{0.319 \pm 0.081}$ & $0.138 \pm 0.035$ \\
\beast (Selected) & $\mathbf{0.337 \pm 0.103}$ & $\mathbf{0.177 \pm 0.076}$ & $0.317 \pm 0.083$ & $0.138 \pm 0.029$ \\
\bottomrule
\end{tabular}
\end{table}
\paragraph{Frame sampling strategy during training.}
Once we have a diverse set of training frames, we must construct batches during training. As stated in Sec.~\ref{sec:methods}, for a batch of size $B$ we randomly select $B/2$ anchor frames, which can originate from any and all videos. For \beast, each anchor frame $\mathbf{x}_t^v$ is paired with a positive frame randomly selected from $\mathbf{x}_{t\pm1}^v$. All other frames serve as negative frames, including frames from the same video. Due to the frame selection strategy described above, even frames from the same video will be visually distinct and not interfere with the contrastive loss (Fig.~\ref{fig:appendix_sample_strategy}). This batch construction procedure is distinct from \textsc{ViC-MAE}, which allows any two frames from the same video to be a positive pair, while only frames from different videos are negative pairs. We find our approach outperforms the \textsc{ViC-MAE} approach in the neural encoding task (Table~\ref{tab:frame_sampling_ablation}). This sampling strategy only applies to \beast; the \vitm models do not contain the contrastive loss, and we only train them with the anchor frames.
%%%%Table to compare with ViC-MAE negative frame sampling strategy%%%%%%%
\begin{table}[ht]
\centering
\footnotesize
\caption{Frame sampling strategy ablation. We pretrain models using either the \textsc{ViC-MAE} or \beast frame \textit{sampling} strategy; both models use the superior ``Selected'' frame \textit{selection} strategy. We evaluate the representations of these models using zero-shot performance on the neural encoding task using the bits per spike (BPS) metric. We report the mean and standard deviation of BPS across five test sessions.}
\label{tab:frame_sampling_ablation}
\begin{tabular}{lcccc}
\toprule
\multirow{2}{*}{Method} & \multicolumn{2}{c}{IBL} & \multicolumn{2}{c}{IBL-whisker} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5}
& TCN & RRR & TCN & RRR \\
\midrule
ViC-MAE (IN+PT) & $0.331 \pm 0.103$ & $0.141 \pm 0.080$ & $0.289 \pm 0.055$ & $0.127 \pm 0.033$ \\
\beast (IN+PT) & $\mathbf{0.337 \pm 0.103}$ & $\mathbf{0.177 \pm 0.076}$ & $\mathbf{0.317 \pm 0.083}$ & $\mathbf{0.138 \pm 0.029}$ \\
\bottomrule
\end{tabular}
\end{table}
\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=1.0\linewidth]
{figures/pdfs/appendix_sample_strategy.pdf}
\end{center}
\caption{\textbf{Temporal contrastive loss visualization.} UMAP embeddings~\citep{mcinnes2018umap} of the first 32 principal components (PCs) of all anchor frames used for pretraining on the IBL dataset. Each anchor frame is colored by the session it was sampled from. The left and right columns show example frames drawn from the same session and from different sessions, respectively. The sampled frames are largely visually distinct, even frames next to each other in UMAP space (e.g., frames B and C, or D and E, on the left-hand side). }
\label{fig:appendix_sample_strategy}
\end{figure*}
\paragraph{Data augmentation.}
The default data augmentation procedure~\citep{he2022masked} applies a random resized crop to $244 \times 244$ pixels with a crop ratio between 0.2 and 1.0, followed by a random horizontal flip with probability 50\%. We explored an extended augmentation strategy that adds further transformations: rotation up to 45 degrees and color jittering (brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1), in addition to the crop and flip. We compare the performance of \beast using both the default and the extended augmentation strategy. The additional augmentations achieve performance similar to the default setting (Table \ref{tab:data_augmentation}), so we use the default throughout the paper.
\begin{table}[ht]
\centering
\footnotesize
\caption{Data augmentation ablation. We pretrain models using either default or extended data augmentations. We evaluate the representations of these models using zero-shot performance on the neural encoding task using the bits per spike (BPS) metric. We report the mean and standard deviation of BPS across five test sessions.}
\label{tab:data_augmentation}
\begin{tabular}{lcccc}
\toprule
\multirow{2}{*}{Method} & \multicolumn{2}{c}{IBL} & \multicolumn{2}{c}{IBL-whisker} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5}
& TCN & RRR & TCN & RRR \\
\midrule
\beast (default data aug) & $\mathbf{0.337 \pm 0.103}$ & $\mathbf{0.177 \pm 0.076}$ & $\mathbf{0.317 \pm 0.083}$ & $0.138 \pm 0.029$ \\
\beast (extend data aug) & $0.328 \pm 0.102$ & $0.163 \pm 0.081$ & $0.314 \pm 0.077$ & $\mathbf{0.150 \pm 0.042}$ \\
\bottomrule
\end{tabular}
\end{table}
\paragraph{Pooling strategy.} The \texttt{CLS} token serves as a global frame representation, effectively pooling information across all spatial positions into a single latent vector. To explore alternative pooling strategies during pretraining, we conducted additional experiments by pretraining models using mean pooling and attention pooling of the patch embeddings, then evaluating performance on the neural encoding task (Table~\ref{tab:pretraining_pooling}). The ablation results clearly demonstrate that the CLS token is the most effective aggregation method for pretraining.
\begin{table}[ht]
\centering
\footnotesize
\caption{Pooling ablation. We pretrain models using either the \texttt{CLS} token, or mean or attention pooling of the patch embeddings, to aggregate information across the image for the temporal contrastive loss. We evaluate the representations of these models using zero-shot performance on the neural encoding task using the bits per spike (BPS) metric. We report the mean and standard deviation of BPS across five test sessions.}
\label{tab:pretraining_pooling}
\begin{tabular}{lcccc}
\toprule
\multirow{2}{*}{Method} & \multicolumn{2}{c}{IBL} & \multicolumn{2}{c}{IBL-whisker} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5}
& TCN & RRR & TCN & RRR \\
\midrule
\beast (mean pooling) & $0.321\pm0.104$ & $0.159\pm0.080$ & $0.302\pm0.082$ & $0.128\pm0.025$\\
\beast (attention pooling) & $0.323\pm0.100$ & $0.141\pm0.076$ & $0.307\pm0.072$ & $0.130\pm0.031$\\
\beast (\texttt{CLS} token) & $\mathbf{0.337\pm0.103}$ & $\mathbf{0.277\pm0.076}$ & $\mathbf{0.317\pm0.083}$ & $\mathbf{0.138\pm0.029}$ \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Hyperparameter details}
The \beast objective combines reconstruction and contrastive losses. During the early stages of training, it is important for the model to focus on accurately reconstructing the input, so the reconstruction loss should dominate. As the model learns to capture low-level pixel structure, the contrastive loss gradually becomes more important. It acts as a regularizer, encouraging the model to learn higher-level temporal representations rather than overfitting to local pixel patterns.
The weighting factor for the contrastive loss, $\lambda$, plays a crucial role during pretraining; too large and the model will not reconstruct the input well; too small and the model does not reap its regularizing benefits. Through hyperparameter tuning based on neural encoding performance (on the validation set), we set $\lambda = 0.01$ for all datasets, except IBL and mirror-mouse, where we set $\lambda = 0.02$. This choice ensures the reconstruction loss is emphasized in the early phases of training, while the contrastive loss naturally takes over as reconstruction error decreases toward the end, eliminating the need for an annealing schedule.
The masking strategy for the reconstruction loss also significantly impacts performance. We found that an aggressive mask ratio of 0.75 works effectively across various tasks, for both \vitm and \beast. When fine-tuning \beast for neural encoding models (IN+FT or IN+PT+FT), we tested mask ratios of 0.75 and 0.9, with 0.9 performing better on validation data. These 0.9-ratio models are used for the final fine-tuning results presented in our tables and figures.
\subsection{Computational efficiency comparison}
To compare the computational efficiency of image (\vitm, \beast) versus video models (\videomae), we recorded three metrics for each model: runtime (ms per batch), Giga Floating Point Operations per Second (GFLOPS), and memory required for a forward pass with batch size one. All experiments were run using the \texttt{fvcore} library on a single A100 GPU. Since runtime varies across batches while GFLOPS and memory are deterministic, we report mean and standard deviation of runtime across 32 batches.
We benchmark two modes: ``pretrain'', which uses patch masking (0.75 for image models, 0.9 for VideoMAE, following optimal settings from the respective papers); and ``finetune'', which omits patch masking as relevant for our downstream tasks.
\beast and \vitm show comparable performance across all metrics, with \beast having a slightly longer runtime and larger memory footprint due to the nonlinear projector used in the contrastive loss (not substantial enough to affect GFLOPS). \videomae requires substantially more resources due to processing 16 consecutive frames per batch element: during pretraining, it requires 2.5$\times$ runtime, 3.5$\times$ GFLOPS, and $>$2$\times$ memory compared to \beast. These differences are even more pronounced during finetuning when patches are not masked: \videomae requires 11$\times$ runtime, 11$\times$ GFLOPS, and 7$\times$ memory.
\begin{table}[ht]
\centering
\footnotesize
\caption{Computational efficiency comparison. We benchmark runtime (mean and standard deviation across 32 batches), GFLOPS, and memory usage for \beast, \vitm, and \videomae during pretraining (with patch masking) and finetuning (without patch masking). All measurements are for a forward pass with batch size of one on a single A100 GPU.}
\label{tab:efficiency}
\begin{tabular}{lcccccc}
\toprule
\multirow{2}{*}{Model} & \multicolumn{2}{c}{Runtime (ms/batch)} & \multicolumn{2}{c}{GFLOPS} & \multicolumn{2}{c}{Memory (MB)} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7}
& Finetune & Pretrain & Finetune & Pretrain & Finetune & Pretrain \\
\midrule
ViT-M & $5.43\pm0.70$ & $7.6\pm2.43$ & $17.59$ & $9.80$ & $369.0$ & $332.72$ \\
\beast & $6.21\pm0.94$ & $7.9\pm2.26$ & $17.59$ & $9.80$ & $409.1$ & $333.40$ \\
VideoMAE & $71.51\pm3.50$ & $20.16\pm2.47$ & $199.49$ & $35.24$ & $2955.1$ & $831.61$ \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Feature visualizations}
We computed PCA on patch embeddings from 100 randomly sampled frames in the test set and visualized the first three components as RGB channels of two example frames. Compared to DINOv2--whose embeddings emphasize broader semantic structures--the \beast pre-trained model captures finer-grained details that are critical for neural encoding, pose estimation and behavior segmentation. In contrast, the ViT-M model pretrained only on ImageNet produces patch embeddings that appear noisier and less structurally coherent.
\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=1\linewidth]
{figures/pdfs/appendix_visualization.pdf}
\end{center}
\vspace{-10pt}
\caption{\textbf{Visualization of the first three PCA components across models.}}
\label{fig:embedding_visualization}
\end{figure*}
\section{Neural encoding} \label{sec:appendix_encoding}
% For neural encoding we consider a variety of input feature types and modeling approaches.
\subsection{Feature representations}
\paragraph{Keypoints.} For the IBL dataset we use 11 keypoints in the publicly available dataset: left and right paws, two edges of the tongue, two edges of the lick spout, nose, and four edges of the pupil. For the Facemap dataset we use 12 keypoints in the publicly available dataset: three whiskers, four points on the nose, four corners of the eye, and the one visible paw.
\paragraph{Whisker pad motion energy.} We localize the whisker pad in the IBL dataset using anchor keypoints on the nose and eye. We then compute the motion energy of the whisker pad as the absolute pixel-wise differences between consecutive frames, resulting in a one-dimensional representation at each time point~\citep{ibl2022video}.
\paragraph{Principal Component Analysis.}
We compute PCA on a per-session basis using all frames in the video. A subset of the resulting PCs are used for neural encoding. See Sec.~\ref{sec:appendix_encoding_dim_ablation} for information on our dimensionality ablation experiment.
\paragraph{CEBRA.}
CEBRA~\citep{schneider2023learnable} is a contrastive learning approach that provides a baseline which is complementary to the \vitm models pretrained on ImageNet with a masked autoencoding objective. Similar to our PCA approach, we train an individual unsupervised CEBRA model for each session using the convolutional neural network option and the default \texttt{offset10-model}.
\paragraph{DINOv2.}
For the DINOv2 model~\citep{oquab2023dinov2}, we extract the \texttt{CLS} embedding for each frame using a frozen pretrained backbone, and use these as input to the encoding models. We did not pretrain this model ourselves, but rather used the model checkpoint available at \url{https://huggingface.co/facebook/dinov2-base}.
\paragraph{ViT-M variants.}
For the \vitm models, we extract the \texttt{CLS} embedding for each frame using a frozen pretrained backbone, and use these as input to the encoding models. Alternative approaches could include: (1) using patch embeddings with a multi-head attention pooling layer (as in our action segmentation work), or (2) fine-tuning the backbone itself while using either \texttt{CLS} or patch embeddings (similar to our approach for pose estimation). We expect these alternative approaches would improve performance and plan to explore them in future work.
\begin{itemize}
\item \textsc{ViT-M (IN)}: \vitm model pretrained on ImageNet using a masked autoencoding loss. We did not pretrain this model ourselves, but rather used the model checkpoint available at \url{https://huggingface.co/facebook/vit-mae-base}.
\item \textsc{ViT-M (IN+PT)}: Initialized with the ImageNet-pretrained weights, then further pretrained on dataset-specific frames.
\item \textsc{ViT-M (IN+PT+FT)}: Initialized with the dataset-specific pretrained weights (IN+PT), and then further fine-tuned on a single session.
\end{itemize}
All training (except \textsc{ViT-M (IN)}) is performed as described in Appendix~\ref{sec:beast-train}.
\paragraph{\vit-C variants.} For the \vit-C (IN+PT) model, we start with the \textsc{ViT-M (IN)} weights then continue pretraining on dataset-specific frames with the temporal contrastive loss only. Therefore there is no \textsc{ViT-C (IN)} variant. We did not perform additional session-specific fine-tuning, so there is no \textsc{ViT-C (IN+PT+FT)} variant.
\paragraph{\beast variants.} The \beast model variants follow the same naming pattern as \vitm, with one exception: there is no ``\beast (IN)'' variant, as \beast requires video frames rather than just static images for pretraining.
\subsection{Reduced rank regression}
\paragraph{IBL.} For the IBL dataset, we followed the Reduced Rank Regression (RRR) setup described in~\citep{posani2025rarely}. We trained all models using the L-BFGS optimizer and set the rank constraint to 3. To denoise the neural signals, we applied a 1-dimensional smoothing filter to the neural activity. The hyperparameter search (Sec.~\ref{sec:appendix_encoding_hparam}) was conducted over the ranges specified in Table~\ref{tab:ibl_rrr_hyperparams}.
\begin{table}[ht!]
\footnotesize
\centering
\caption{RRR model hyperparameters for IBL dataset.}
\begin{tabular}{lc}
\toprule
Hyperparameter & Value Range \\
\midrule
Output Dim & Number of Neurons \\
Rank & 3 \\
Optimizer & L-BFGS \\
Learning Rate & Log-Uniform(0.1, 2) \\
L2 & 100 \\
\bottomrule
\end{tabular}
\label{tab:ibl_rrr_hyperparams}
\end{table}
\paragraph{Facemap.} For the Facemap dataset, we followed the setup described in~\citep{syeda2024facemap}, using the implementation provided in the \href{https://github.com/MouseLand/facemap/blob/v1.0.7/facemap/neural_prediction/prediction_utils.py#L110}{official Facemap repository}. To deal with different neural and behavioral sampling rates, this model first resamples the behavioral timestamps to match the neural timestamps, and then fits a Reduced Rank Regression model using low-rank SVD. We adopted the default rank of 32 as used in the original implementation. The Lambda parameter refers to the regularization strength, which we set to a relatively low value to avoid over-penalizing the weights. The output dimensionality was set to 128, corresponding to the number of neural principal components used in the model. Model parameters are estimated via a closed-form least squares approach.
The hyperparameters we used are specified in Table~\ref{tab:facemap_rrr_hyperparams}.
\begin{table}[ht!]
\footnotesize
\centering
\caption{RRR model hyperparameters for Facemap dataset.}
\begin{tabular}{lc}
\toprule
Hyperparameter & Value Range \\
\midrule
Output Dim & 128 \\
Rank & 32 \\
Lambda & 1e-6 \\
\bottomrule
\end{tabular}
\label{tab:facemap_rrr_hyperparams}
\end{table}
\subsection{Temporal convolution network}
We used the same implementation of the Temporal Convolution Network (TCN) to process frame embeddings for both the IBL and Facemap datasets, based on the \href{https://github.com/MouseLand/facemap/blob/v1.0.7/facemap/neural_prediction/neural_model.py}{official Facemap repository}. The convolutional kernel operates along the temporal dimension of the input (behavioral) data. To deal with different neural and behavioral sampling rates, this model resamples the resulting latent representation at neural timestamps using nearest-neighbor indexing. The TCN model was trained for 300 epochs using the AdamW optimizer, with learning rate decimation (multiplied by 0.1) applied at epochs 120 and 200. The hyperparameter search (Sec.~\ref{sec:appendix_encoding_hparam}) was conducted over the ranges specified in Table~\ref{tab:facemap_cnn_hyperparams}.
\begin{table}[ht!]
\centering
\footnotesize
\caption{TCN model hyperparameters; $^{*}$IBL dataset; $^{**}$Facemap dataset}
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{lc}
\toprule
Hyperparameter & Value Range \\
\midrule
Output Dim & Number of Neurons$^{*}$, 128$^{**}$ \\
Learning Rate & Log-Uniform(5e-5, 2e-3) \\
Optimizer & AdamW \\
Weight Decay & 1e-4 \\
\bottomrule
\end{tabular}
\label{tab:facemap_cnn_hyperparams}
\end{table}
\subsection{Hyperparameter selection} \label{sec:appendix_encoding_hparam}
For the IBL data, we divided the trials into train (80\%), validation (10\%), and test (10\%) sets. For the Facemap data we followed the experimental setup as described in the original study~\citep{syeda2024facemap}: the session is split into ten blocks; the first 75\% of each block is assigned to the training set; the following 3 seconds are excluded to remove data leakage due to autocorrelation in behavior and neural activity; and the final set of frames from the block are assigned to the test set. There is no validation set.
For both the RRR and TCN models, we conducted hyperparameter searches separately for each feature type to identify the best-performing configurations. Specifically, we performed 30 runs of randomly selected hyperparameters per model type, evaluating performance on the validation set (IBL) or test set (Facemap) using an evaluation metric specific to each dataset: bits per spike (BPS) for IBL and variance explained for Facemap. The only exception was the RRR model for Facemap, where the parameters were fixed according to the original implementation.
We select the model with the best performance on the validation (IBL) or test (Facemap) set, and report results on the test set.
\subsection{Patch embeddings}
The output of the \vit encoder consists of a \texttt{CLS} token embedding $z_{\text{CLS}} \in \mathbb{R}^{D}$ and patch token embeddings $z \in \mathbb{R}^{N \times D}$, where $D=768$ is the embedding dimension and $N$ is the total number of image patches ($N=196$ for a $224 \times 224$ input frame). We compare the \texttt{CLS} token and attention-pooled patch embeddings as inputs for neural encoding (see implementation details in~\ref{sec:appendix_action_mhp}), and find that the \texttt{CLS} token outperforms the patch embeddings (Table~\ref{tab:encoding_patch}). Given its lower dimensionality and superior results, we adopt the \texttt{CLS} token representation for all subsequent encoding tasks.
\begin{table}[ht]
\centering
\footnotesize
\caption{Comparison of \texttt{CLS} and patch embeddings with a TCN encoder. We report the mean and standard deviation of BPS across five test sessions.}
\label{tab:encoding_patch}
\begin{tabular}{lcc}
\toprule
\multirow{1}{*}{Method} & \multicolumn{1}{c}{IBL} & \multicolumn{1}{c}{IBL-whisker} \\
\midrule
\beast (\texttt{CLS}) & $\mathbf{0.337 \pm 0.103}$ & $\mathbf{0.317 \pm 0.083}$ \\
\beast (patch) & $0.278 \pm 0.065$ & $0.288 \pm 0.070$ \\
\bottomrule
\end{tabular}
\end{table}
\subsection{Dimensionality ablation experiments} \label{sec:appendix_encoding_dim_ablation}
One of the most important hyperparameters for the PCA, CEBRA, and \vit models is the latent/embedding dimensionality. To thoroughly explore performance across this parameter, we tested these models using various dimensionality values. For a given dimensionality $k$, we used different approaches: (1) for PCA, we selected the top $k$ principal components; (2) for CEBRA, we retrained the model with $k$ latent dimensions; and (3) for \vit models, due to computational constraints, we first trained the full 768-dimensional models, then applied PCA to the embedding space and selected the top $k$ \vit principal components. For each feature, model type, and dimensionality $k$, we fit downstream neural encoding models using the complete hyperparameter search described previously. Figure~\ref{fig:neural_encoding} reports the best result for each model, though notably \beast outperforms all baselines across all dimensionality values (Fig.~\ref{fig:neural_encoding_dims}).
\begin{figure}[ht!]
\begin{center}
\includegraphics[width=\linewidth]
{figures/pdfs/appendix_encoding_dim.pdf}
\end{center}
\vspace{-20pt}
\caption{\textbf{Encoding performance as a function of embedding dimension.} \beast outperforms all other baselines over embedding dimensions spanning several orders of magnitude, demonstrating the superiority of its representations for any given dimensionality. Results for Keypoints and Motion Energy are included at their respective dimensionalities for each dataset. The Facemap encoding results for 1-dimensional data performed poorly and included NaN values in some sessions, so we excluded them from the figure.}
\vspace{15pt}
\label{fig:neural_encoding_dims}
\end{figure}
\subsection{Extended neural encoding results}
We collect all neural encoding results in Table~\ref{tab:neural_encoding_results}. The values for PCA and CEBRA correspond to the 100-dimensional results in Fig.~\ref{fig:neural_encoding_dims}; the values for \vitm and \beast correspond to the full 768-dimensional models.
\begin{table}[ht!]
\centering
\caption{Neural encoding results across feature types and models. All \vit-based models use \texttt{CLS} tokens from a frozen backbone. ``IN'' refers to a model pretrained with ImageNet weights; ``IN+PT'' refers to models that are initialized with ImageNet-pretrained weights then further pretrained on experiment-specific data; ``+FT'' refers to models that are initialized with pretrained weights based on what comes before ``+'' then fine-tuned on individual sessions.
For IBL and IBL-whisker we report the mean and standard error of the mean of BPS over $N$=842 neurons across five test sessions.
For Facemap we report the mean and standard deviation of Variance Explained of the first 128 Principal Components of neural activity across five test sessions.
}
\label{tab:neural_encoding_results}
\renewcommand{\arraystretch}{1.3}
\begin{adjustbox}{width=\textwidth}
% \begin{tabular}{lcccccc}
% \toprule
% \multirow{2}{*}{Features} & \multicolumn{2}{c}{IBL} & \multicolumn{2}{c}{IBL-whisker} & \multicolumn{2}{c}{Facemap} \\
% \cmidrule(l){2-3} \cmidrule(l){4-5} \cmidrule(l){6-7}
% & RRR & TCN & RRR & TCN & RRR & TCN \\
% \midrule
% Keypoints & $0.173 \pm 0.029$ & $0.271 \pm 0.054$ & -- & -- & $0.224 \pm 0.047$ & $0.403 \pm 0.077$ \\
% Motion energy & -- & -- & $0.087 \pm 0.023$ & $0.117 \pm 0.028$ & -- & -- \\
% PCA & $0.266 \pm 0.054$ & $0.312 \pm 0.078$ & $0.220 \pm 0.038$ & $0.281 \pm 0.065$ & $0.177 \pm 0.064$ & $0.407 \pm 0.090$ \\
% CEBRA & $0.245 \pm 0.036$ & $0.295 \pm 0.049$ & $0.209 \pm 0.265$ & $0.265 \pm 0.034$ & $0.143 \pm 0.046$ & $0.278 \pm 0.064$ \\
% \vitm (IN) & $0.201 \pm 0.070$ & $0.325 \pm 0.091$ & $0.142 \pm 0.051$ & $0.307 \pm 0.068$ & $0.254 \pm 0.061$ & $0.446 \pm 0.100$ \\
% \vitm (IN+PT) & $0.182 \pm 0.071$ & $0.334 \pm 0.098$ & $0.156 \pm 0.032$ & $0.316 \pm 0.073$ & -- & -- \\
% \vitm (IN+FT) & $0.242 \pm 0.089$ & $0.349 \pm 0.106$ & $0.219 \pm 0.048$ & $0.336 \pm 0.075$ & $0.260 \pm 0.051$ & $0.461 \pm 0.099$ \\
% \vitm (IN+PT+FT) & $\mathbf{0.293 \pm 0.082}$ & $0.351 \pm 0.106$ & $0.244 \pm 0.042$ & $0.335 \pm 0.092$ & -- & -- \\
% \beast (IN+PT) & $0.277 \pm 0.076$ & $0.337 \pm 0.103$ & $0.138 \pm 0.029$ & $0.317 \pm 0.083$ & -- & -- \\
% \beast (IN+FT) & $0.276 \pm 0.088$ & $0.351 \pm 0.106$ & $\mathbf{0.251 \pm 0.051}$ & $\mathbf{0.337 \pm 0.088}$ & $\mathbf{0.281 \pm 0.054}$ & $\mathbf{0.464 \pm 0.089}$ \\
% \beast (IN+PT+FT) & $0.291 \pm 0.087$ & $\mathbf{0.352 \pm 0.106}$ & $0.243 \pm 0.048$ & $0.335 \pm 0.079$ & -- & -- \\
% \bottomrule
% \end{tabular}
\begin{tabular}{lcccccc}
\toprule
\multirow{2}{*}{Features} & \multicolumn{2}{c}{IBL (BPS)} & \multicolumn{2}{c}{IBL-whisker (BPS)} & \multicolumn{2}{c}{Facemap (Var Exp)} \\
\cmidrule(l){2-3} \cmidrule(l){4-5} \cmidrule(l){6-7}
& RRR & TCN & RRR & TCN & RRR & TCN \\
\midrule
Keypoints & $0.169 \pm 0.008$ & $0.269 \pm 0.011$ & -- & -- & $0.224 \pm 0.047$ & $0.403 \pm 0.077$ \\
Motion energy & -- & -- & $0.086 \pm 0.006$ & $0.113 \pm 0.007$ & -- & -- \\
PCA & $0.260 \pm 0.010$ & $0.309 \pm 0.012$ & $0.212 \pm 0.009$ & $0.272 \pm 0.011$ & $0.177 \pm 0.064$ & $0.407 \pm 0.090$ \\
CEBRA & $0.239 \pm 0.010$ & $0.293 \pm 0.012$ & $0.204 \pm 0.009$ & $0.260 \pm 0.011$ & $0.143 \pm 0.046$ & $0.278 \pm 0.064$ \\
\midrule
DINOv2 (LVD-142M) & $0.183 \pm 0.008$ & $0.294 \pm 0.013$ & $0.138 \pm 0.007$ & $0.269 \pm 0.012$ & -- & -- \\
CLIP (WIT-400M) & $0.182 \pm 0.008$ & $0.292 \pm 0.012$ & $0.131 \pm 0.006$ & $0.268 \pm 0.012$ & -- & -- \\
VideoMAE (Kinetics-400) & $0.189 \pm 0.010$ & $0.318 \pm 0.013$ & $0.133 \pm 0.007$ & $0.298 \pm 0.013$ & -- & -- \\
\vitm (IN) & $0.192 \pm 0.009$ & $0.321 \pm 0.013$ & $0.129 \pm 0.007$ & $0.301 \pm 0.012$ & $0.254 \pm 0.061$ & $0.446 \pm 0.100$ \\
\midrule
\vitm (IN+PT) & $0.172 \pm 0.008$ & $0.331 \pm 0.013$ & $0.148 \pm 0.007$ & $0.311 \pm 0.013$ & -- & -- \\
\vitm (IN+FT)& $0.233 \pm 0.009$ & $0.345 \pm 0.014$ & $0.211 \pm 0.010$ & $0.330 \pm 0.011$ & $0.260 \pm 0.051$ & $0.461 \pm 0.099$ \\
\vitm (IN+PT+FT) & $\mathbf{0.285 \pm 0.011}$ & $\mathbf{0.347 \pm 0.014}$ & $0.235 \pm 0.010$ & $0.328 \pm 0.013$ & -- & -- \\
\vit-C (IN+PT) & $0.137 \pm 0.008$ & $0.314 \pm 0.013$ & $0.120 \pm 0.006$ & $0.283 \pm 0.011$ & -- & -- \\
\beast (IN+PT) & $0.205 \pm 0.008$ & $0.292 \pm 0.012$ & $0.133 \pm 0.006$ & $0.309 \pm 0.013$ & -- & -- \\
\beast (IN+FT) & $0.264 \pm 0.010$ & $\mathbf{0.347 \pm 0.014}$ & $\mathbf{0.240 \pm 0.010}$ & $\mathbf{0.331 \pm 0.013}$ & $\mathbf{0.281 \pm 0.054}$ & $\mathbf{0.464 \pm 0.089}$ \\
\beast (IN+PT+FT) & $0.282 \pm 0.011$ & $\mathbf{0.347 \pm 0.014}$ & $0.234 \pm 0.010$ & $0.326 \pm 0.013$ & -- & -- \\
\bottomrule
\end{tabular}
\end{adjustbox}
\end{table}
\section{Pose estimation} \label{sec:appendix_pose}
\subsection{Models} The pose estimation models consist of a backbone and a head. The backbone is either a ResNet-50 \citep{he2016deep} or a \textsc{ViT-B/16} \citep{dosovitskiy2020image}, both producing feature maps of shape $[N, H, W]$ for a given image, where $N$ denotes the feature dimension and $H$, $W$ denote the height and width of the feature maps. All models employ an identical linear upsampling head that begins with a \texttt{PixelShuffle} layer, reshaping the feature maps to $[N / 4, 2H, 2W]$. These reshaped features then pass through two consecutive 2D convolutional transpose layers with kernel size $(3, 3)$ and stride $(2, 2)$, doubling the spatial resolution after each layer. The head architecture omits batch normalization and nonlinearities between these layers. The output passes through a 2D softmax function, generating a normalized heatmap for each keypoint.
\subsection{Training} We divided the labeled data into training (95\%) and validation (5\%) sets, with test frames coming from entirely held-out videos. We used a batch size of eight frames. Data augmentations include random crops, rotations, motion blur and histogram equalization. Models were trained for 300 epochs, with validation loss recorded every five epochs. For final evaluation, we selected the model with the lowest validation loss. Training utilized an Adam optimizer \citep{kingma2014adam} with an initial learning rate of 0.001 for the ResNet and $5\mathrm{e}{-5}$ for the transformers, which was halved at epochs 150, 200, and 250. To facilitate feature learning, we kept the backbone frozen during the first 20 epochs of training before allowing for full end-to-end optimization. The loss function is the mean square error between each predicted heatmap and a ground truth heatmap constructed from labeled data.
\subsection{Pretrained backbones}
We test several additional \vit backbones on pose estimation to validate our \beast results:
\begin{itemize}
\item Segment Anything (SAM)~\citep{kirillov2023segment}; checkpoint from \url{https://huggingface.co/facebook/sam-vit-base}
\item DINO~\citep{caron2020unsupervised}; checkpoint from \url{https://huggingface.co/facebook/dino-vitb16}
\item DINOv2~\citep{oquab2023dinov2}; checkpoint from \url{https://huggingface.co/facebook/dinov2-base}
\end{itemize}
These backbones are trained using the same procedure as the ResNet-50 and \beast models (see above). We find other pretrained backbones mostly outperform the ResNet-50 baseline (Fig.~\ref{fig:pose_backbones}).
SAM is generally the least performant backbone. DINOv2 consistently outperforms DINO across all datasets, and \beast achieves the lowest pixel error in most cases (only outperformed by DINOv2 in the Mirror-fish dataset). These results demonstrate \beast's experiment-specific pretraining framework can surpass state-of-the-art general purpose vision foundation models for pose estimation.
\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=1\linewidth]
{figures/pdfs/appendix_litpose_dino.pdf}
\end{center}
\caption{\textbf{Pose estimation of fine-tuned vision foundation model backbones.} We evaluated ResNet-50 (pretrained on AP10K), SAM, DINO, DINOv2, and \beast backbones on pose estimation datasets. We evaluate the these models using pixel error at various ensemble standard deviation thresholds, with values in the table representing the percentage of keypoints at the chosen threshold. Smaller values indicate smaller but more challenging subsets of keypoints (see main text).}
\label{fig:pose_backbones}
\end{figure*}
% \begin{table}[ht]
% \centering
% \footnotesize
% \caption{Performance of fine-tuned pretrained backbones on pose estimation (100 labeled frames). We evaluate the these models using pixel error at various ensemble standard deviation thresholds, with values in the table representing the percentage of keypoints at the chosen threshold. Smaller values indicate smaller but more challenging subsets of keypoints (see main text).}
% \label{tab:pose_backbone}
% \begin{tabular}{lcccc}
% \toprule
% Mirror-mouse & 100\% & 95\% & 50\% & 10\% \\
% \midrule
% ResNet-50 (AP10k)& 6.67 & 7.08 & 10.63 & 36.9 \\
% \vit-MAE (IN) & 5.33 & 5.6 & 8.19 & 25.85\\
% \vit-SAM &5.64 & 5.94 & 8.73 & 28.28\\
% \vit-DINO (IN) & 5.21 & 5.47 & 7.95 & 24.91\\
% \beast & \textbf{4.73} & \textbf{4.94} & \textbf{7.04} & \textbf{19.9}\\
% \bottomrule
% \end{tabular}
% \begin{tabular}{lcccc}
% CRIM13 & 100\% & 95\% & 50\% & 35\% \\
% \midrule
% ResNet-50 (AP10k) & 21.2 & 21.88 & 34.93 & 43.06 \\
% \vit-MAE (IN) & 17.86 & 18.41 & 28.47 & 34.29 \\
% \vit-SAM & 19.58 & 20.2 & 31.92 & 38.94\\
% \vit-DINO (IN) &19.24 & 19.85 & 31.16 & 37.71 \\
% \beast & \textbf{16.33} & \textbf{16.82} & \textbf{25.67} & \textbf{30.82}\\
% \bottomrule
% \end{tabular}
% \end{table}
\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=\linewidth]{figures/pdfs/appendix_litpose.pdf}
\end{center}
\caption{\textbf{Pose estimation performance with more training frames.} The results in Fig.~\ref{fig:pose_estimation} demonstrate pose estimation performance of various models using just 100 labeled frames. To ensure those comparisons hold on larger datasets we trained models (three random seeds per backbone) using the maximum number of frames in the training dataset or 1000 frames, whichever is smaller. We still see consistent gains for both the transformer architecture pretrained on ImageNet (red) and our pretraining strategy (blue, green).
}
\label{fig:pose_estimation_1k_train}
\end{figure*}
\subsection{DeepLabCut baseline}
For the DeepLabCut baseline (version 3.0.0) we trained models using an ImageNet-pretrained ResNet-50 backbone. To properly isolate differences between DeepLabCut and Lightning Pose algorithms, we matched training frames, batch size, learning rate schedule, and number of epochs. For all other hyperparameters we used the DeepLabCut package defaults (e.g., data augmentation). We train models using three different train/val data splits, and ensure these splits exactly match those used for the Lightning Pose models.
Lightning Pose outperforms DeepLabCut on the IBL, Mirror-mouse, and Mirror-fish datasets, while DeepLabCut outperforms Lightning Pose on CRIM13 (Fig.~\ref{fig:litpose_dlc}). Notably, our pretrained \beast backbones outperform both DeepLabCut and Lightning Pose across all four datasets.
\begin{figure*}[ht!]
\begin{center}
\includegraphics[width=1\linewidth]
{figures/pdfs/appendix_litpose_dlc.pdf}
\end{center}
\caption{Pose estimation of DeepLabCut~(DLC). Figure conventions as in Fig.~\ref{fig:pose_backbones}}
\label{fig:litpose_dlc}
\end{figure*}
\section{Action segmentation} \label{sec:appendix_action}
For action segmentation we consider a variety of input feature types and modeling approaches. We first consider a range input features: keypoints, PCA on the raw video frames (fit on the same frames as ViT pretraining; we use 768 PCs for downstream models), and \vit-based \texttt{CLS} tokens. For each of these feature types we fit both linear and nonlinear (temporal convolution network, TCN) models. We also train a TCN model on the \vit patch embeddings, with a multi-head attention pooling layer to reduce the dimensionality of the features before entering the TCN.
\subsection{Models}
\paragraph{Linear model.} The linear action segmentation model uses a 1D temporal convolution layer, followed by a linear layer that maps from the number of features the the number of action classes, followed by a softmax. There are no other forms of nonlinearity in the model.
\paragraph{Temporal convolution network.}
The nonlinear action segmentation model is a dilated TCN \citep{lea2016temporal} with 2 dilation blocks. Each dilation block consists of a sequence of 2 sub-blocks (1D convolution layer → leaky ReLU nonlinearity → dropout with probability=0.10), as well as a residual connection between the input and output of the dilation block. The dilation of the convolutional filters starts with 1 for the first dilation block, then increases by a factor of 2 for each additional dilation block. This results in a larger temporal receptive field as the model gets deeper, allowing for learning of longer range dependencies \citep{yu2015multi}.
Both models utilize a weighted cross entropy loss function, with class weights inversely proportional to the class frequency in the training data.
\subsection{Training} Each video is split into sequences of 500 (IBL) or 100 (CalMS21) frames. We divide the data into training (90\%) and validation (10\%) sets, with test frames coming from entirely held-out videos. We use a batch size of 16 sequences. Models were trained for 500 epochs using the Adam optimizer \citep{kingma2014adam}.
\subsection{Hyperparameter details}
For each model type--linear and nonlinear--and each feature type, we run a hyperparameter search across all combinations of parameters in Table~\ref{tab:action_hyperparams} using three random weight initializations. The hyperparameter combination with the best F1 score on the validation data, averaged across the three seeds, is selected for evaluation on the test set. For this hyperparameter combination, we train with two additional seeds and report results in the figures and tables averaged across five seeds.
\input{tables/tab_action_hyperparams}
\subsection{Temporal convolution network with multi-head attention pooling} \label{sec:appendix_action_mhp}
We aggregate the patch embeddings from the \beast encoder using a multi-head attention pooling layer \citep{lee2019set}, which produces a single pooled embedding per frame as input to the TCN. This layer uses a learnable query $S \in \mathbb{R}^{1\times D}$ with patch embeddings $z \in \mathbb{R}^{N\times D}$ as keys and values. To capture motion-related features, we further concatenate the frame-to-frame difference of the pooled embeddings as additional input to the TCN (Fig.~\ref{fig:mha_pooling_model}). We fixed the number of attention heads in the pooling layer to 8 for all models.
Our previous model utilized \texttt{CLS} embeddings, which allowed for an efficient workflow: we processed videos through the transformer backbone and saved the \texttt{CLS} embedding from each frame as a separate file. These pre-computed embeddings could then be directly loaded to train the downstream TCN classifier without requiring video reading during training. However, patch embeddings present significantly larger memory requirements, making disk storage infeasible. To address this challenge, we developed a data loading pipeline that performs end-to-end processing: it loads video frames, passes them through the transformer backbone, and feeds the resulting patch embeddings directly to the TCN model within the same training loop. This integrated approach, while computationally more intensive, eliminates the need for intermediate storage. Due to these increased computational demands, we modified the training procedure for the multi-head attention pooling models as follows.
The pooling layer and TCN classifier were trained jointly for 200 epochs on CalMS21 and 100 epochs on IBL using the Adam optimizer~\citep{kingma2014adam} with an initial learning rate of $1\mathrm{e}{-3}$. The epoch counts were determined through a separate experiment that withheld a subset of validation videos and monitored the validation F1 score until convergence. Training followed a cosine-annealing schedule with warm restarts \citep{loshchilov2016sgdr} configured with $T_0 = 34$, $T_{mult} = 2$ and $\eta_{min} = 5\mathrm{e}{-5}$. We used 6 or 8 NVIDIA A40 GPUs for the training, each with a batch size of 2, giving an effective batch size of 12 or 16. The sequence length was fixed at 500. Due to the increased compute required to train these models, we fixed the TCN hyperparameters to be those found for the \texttt{CLS}-based model for each dataset.
\begin{figure}[ht!]
\begin{center}
\includegraphics[width=0.9\linewidth]
{figures/pdfs/appendix_behavior_seg_model.pdf}
\end{center}
\caption{\textbf{Multi-head attention pooling TCN head for action segmentation. }
\textbf{A:} Per-patch embeddings from the \beast encoder are pooled using a multi-head attention layer, where a learnable query token attends to the patch embeddings to produce a single pooled embedding for each frame.
\textbf{B:} Temporal differences between consecutive pooled embeddings are concatenated as additional input to the TCN, which predicts the behavior class of the center frame in the sliding window. A window size of 3 is shown for illustration; the actual window size was tuned as a hyperparameter.
}
\label{fig:mha_pooling_model}
\end{figure}
\input{tables/tab_action_results}
\begin{table}[!ht]
\footnotesize
\centering
\caption{Action segmentation hyperparameters. $^{*}$TCN only}
\begin{tabular}{lc}
\toprule
Hyperparameter & Value Range \\
\midrule
Learning rate& 1e-3, 1e-4, 1e-5 \\
Dropout & 0.1 \\
Temporal filter length & 9, 17, 33\\
Number of hidden units$^*$ & 16, 32, 64, 96\\
Number of hidden layers$^*$ & 2\\
\bottomrule
\end{tabular}
\label{tab:action_hyperparams}
\end{table}
\begin{table}[ht]
\centering
\caption{Action segmentation results on IBL and CalMS21 datasets across feature types and models. All \vit-based models use frozen a frozen backbone; ``\texttt{CLS}'' indicates models trained on the global \texttt{CLS} embeddings, while ``patch'' indicates models trained with a multi-head attention pooling layer applied to the patch embeddings. ``IN'' refers to a model pretrained with ImageNet weights, ``IN+PT'' refers to models that are initialized with ImageNet-pretrained weights then further pretrained on experiment-specific data.
We report the mean and standard
deviation of F1 on test data across five random train/val splits.
}
\renewcommand{\arraystretch}{1.3}
\begin{adjustbox}{width=\textwidth}
\begin{tabular}{llcccc}
\toprule
\multirow{2}{*}{Dataset} & \multirow{2}{*}{Features} & \multicolumn{2}{c}{Linear} & \multicolumn{2}{c}{TCN} \\
\cmidrule(l){3-4} \cmidrule(l){5-6}
& & Features & Features, $\Delta$ Features & Features & Features, $\Delta$ Features \\
\midrule
IBL & Keypoints & $0.54 \pm 1.4\mathrm{e}{-3}$ & $0.55 \pm 1.5\mathrm{e}{-3}$ & $0.86 \pm 1.4\mathrm{e}{-3}$ & $\mathbf{0.88 \pm 2.2\mathrm{e}{-3}}$ \\
& PCA & $0.54 \pm 4.6\mathrm{e}{-3}$ & $0.55 \pm 6.3\mathrm{e}{-3}$ & $0.64 \pm 1.0\mathrm{e}{-2}$ & $0.71 \pm 2.8\mathrm{e}{-3}$ \\
& VIT-M (IN) (\texttt{CLS}) & $0.68 \pm 1.4\mathrm{e}{-3}$ & $0.68 \pm 7.0\mathrm{e}{-4}$ & $0.78 \pm 7.4\mathrm{e}{-3}$ & $0.79 \pm 2.7\mathrm{e}{-3}$ \\
& VIT-M (IN+PT) (\texttt{CLS}) & $\mathbf{0.74 \pm 2.5\mathrm{e}{-3}}$ & $\mathbf{0.72 \pm 2.0\mathrm{e}{-3}}$ & $0.78 \pm 4.0\mathrm{e}{-3}$ & $0.80 \pm 4.6\mathrm{e}{-3}$ \\
& \beast (IN+PT) (\texttt{CLS}) & $0.70 \pm 7.9\mathrm{e}{-3}$ & $0.69 \pm 2.5\mathrm{e}{-3}$ & $\mathbf{0.80 \pm 3.0\mathrm{e}{-3}}$ & $0.81 \pm 6.9\mathrm{e}{-4}$ \\
& DINOv2 (patch) & - & - & - & $0.77 \pm 2.7\mathrm{e}{-3}$\\
& VIT-M (IN) (patch) & - & - & - & $0.84 \pm 3.7\mathrm{e}{-3}$\\
& VIT-M (IN+PT) (patch) & - & - & - & $0.85 \pm 3.6\mathrm{e}{-3}$ \\
& \vit-C (IN+PT) (patch) & - & - & - & $0.79 \pm 4.6\mathrm{e}{-3}$\\
& \beast (IN+PT) (patch) & - & - & - & $0.87 \pm 5.1\mathrm{e}{-3}$ \\
\addlinespace[0.5em]
\midrule
CalMS21 & SimBA~\citep{goodwin2024simple} & $\mathbf{0.75 \pm 5.1\mathrm{e}{-4}}$ & $0.53 \pm 8.1\mathrm{e}{-3}$ & $\mathbf{0.78 \pm 3.8\mathrm{e}{-3}}$ & $0.79 \pm 2.9\mathrm{e}{-3}$ \\
& TREBA~\citep{sun2021task} & $0.29 \pm 1.4\mathrm{e}{-3}$ & $0.30 \pm 1.5\mathrm{e}{-3}$ & $0.70 \pm 4.6\mathrm{e}{-3}$ & $0.72 \pm 7.4\mathrm{e}{-3}$ \\
& PCA & $0.10 \pm 3.1\mathrm{e}{-3}$ & $0.10 \pm 3.2\mathrm{e}{-3}$ & $0.16 \pm 5.1\mathrm{e}{-3}$ & $0.18 \pm 4.5\mathrm{e}{-3}$ \\
& VIT-M (IN) (\texttt{CLS}) & $0.50 \pm 2.2\mathrm{e}{-2}$ & $0.52 \pm 1.3\mathrm{e}{-3}$ & $0.53 \pm 1.2\mathrm{e}{-2}$ & $0.60 \pm 2.8\mathrm{e}{-3}$ \\
& VIT-M (IN+PT) (\texttt{CLS}) & $0.60 \pm 5.5\mathrm{e}{-3}$ & $\mathbf{0.60 \pm 1.1\mathrm{e}{-3}}$ & $0.60 \pm 9.1\mathrm{e}{-3}$ & $0.65 \pm 2.2\mathrm{e}{-3}$ \\
& \beast (IN+PT) (\texttt{CLS}) & $0.53 \pm 4.6\mathrm{e}{-3}$ & $0.51 \pm 4.0\mathrm{e}{-2}$ & $0.58 \pm 5.2\mathrm{e}{-3}$ & $0.63 \pm 2.7\mathrm{e}{-3}$ \\
& DINOv2 (patch) & - & - & - & $0.68 \pm 4.4\mathrm{e}{-3}$\\
& VIT-M (IN) (patch) & - & - & - & $0.74 \pm 2.9\mathrm{e}{-3}$ \\
& VIT-M (IN+PT) (patch) & - & - & - & $\mathbf{0.82 \pm 9.5\mathrm{e}{-3}}$ \\
& \vit-C (IN+PT) (patch) & - & - & - & $0.53 \pm 2.9\mathrm{e}{-2}$\\
& \beast (IN+PT) (patch) & - & - & - & $0.81 \pm 7.7\mathrm{e}{-3}$ \\
% CalMS21 & SimBA~\citep{goodwin2024simple} & $0.79 \pm 3.9\mathrm{e}{-4}$ & $0.76 \pm 8.0\mathrm{e}{-3}$ & $0.82 \pm 2.9\mathrm{e}{-3}$ & $0.83 \pm 2.2\mathrm{e}{-3}$ \\
% & TREBA~\citep{sun2021task} & $0.51 \pm 1.4\mathrm{e}{-3}$ & $0.52 \pm 1.5\mathrm{e}{-3}$ & $0.75 \pm 3.3\mathrm{e}{-3}$ & $0.77 \pm 5.5\mathrm{e}{-3}$ \\
% & PCA & $0.25 \pm 1.3\mathrm{e}{-3}$ & $0.25 \pm 1.4\mathrm{e}{-3}$ & $0.30 \pm 4.2\mathrm{e}{-3}$ & $0.32 \pm 3.8\mathrm{e}{-3}$ \\
% & VIT-M (IN) (\texttt{CLS}) & $0.62 \pm 1.5\mathrm{e}{-3}$ & $0.62 \pm 2.0\mathrm{e}{-3}$ & $0.63 \pm 6.4\mathrm{e}{-3}$ & $0.67 \pm 5.9\mathrm{e}{-3}$ \\
% & VIT-M (IN+PT) (\texttt{CLS}) & $0.60 \pm 5.5\mathrm{e}{-3}$ & $0.60 \pm 1.1\mathrm{e}{-3}$ & $0.60 \pm 9.1\mathrm{e}{-3}$ & $0.65 \pm 2.2\mathrm{e}{-3}$ \\
% & \beast (\texttt{CLS}) & $0.61 \pm 3.6\mathrm{e}{-3}$ & $0.63 \pm 8.6\mathrm{e}{-3}$ & $0.65 \pm 4.1\mathrm{e}{-3}$ & $0.68 \pm 2.0\mathrm{e}{-3}$ \\
% & VIT-M (IN) (patch) & - & - & - & $xx \pm xx$ \\
% & VIT-M (IN+PT) (patch) & - & - & - & $xx \pm xx$ \\
% & \beast (IN) (patch) & - & - & - & $xx \pm xx$ \\
\bottomrule
\end{tabular}
\end{adjustbox}
\label{tab:action_segmentation}
\end{table}
\FloatBarrier
\section{Self-supervised learning methods for animal behavior} \label{sec:appendix_ssl_behavior}
Self-supervised learning (SSL) techniques from computer vision have increasingly been adapted to the study of animal behavior. These approaches use SSL to extract useful feature representations from pose, image, or video data, which are then applied to downstream tasks such as action segmentation.
\paragraph{Pose-based approaches.} Trajectory Embedding for Behavior Analysis (TREBA)~\citep{sun2021task} employs a multi-task self-supervised framework that uses trajectory reconstruction as its primary objective through Trajectory Variational Autoencoders~\citep{co2018self, zhan2020learning}. TREBA additionally requires the TVAE embedding to decode various auxiliary tasks consisting of simple data transformations designed by domain experts, with the resulting embeddings serving as input to downstream action segmentation models. Variational Animal Motion Embedding (VAME)~\citep{luxem2022identifying} uses a sequential variational autoencoder to embed pose sequences into a latent space by reconstructing both current and subsequent time steps. Unlike TREBA, VAME applies clustering to the learned embeddings, creating a fully unsupervised action segmentation pipeline. ContrastivePose~\citep{zhou2022constrastivepose} leverages geometric augmentations (flipping, rotation, translation) of pose coordinates to generate positive pairs for contrastive learning, followed by fine-tuning on action segmentation tasks. Bootstrap Across Multiple Scales (BAMS)~\citep{azabou2023relax} employs dual temporal convolutional networks with different receptive field sizes to create complementary short- and long-term embedding spaces. BAMS introduces a novel training objective requiring prediction of future action distributions rather than specific action sequences, with validation on the MABe benchmark~\citep{sun2023mabe22} across multiple tasks including action segmentation and mouse strain classification.
\paragraph{Image and video-based approaches.} Selfee~\citep{jia2022selfee} constructs composite RGB frames from 3-frame grayscale video sequences (assigning each frame to a separate color channel) and applies standard image-based contrastive learning techniques, demonstrating effectiveness on action segmentation and anomaly detection. \cite{mueller2025domain} adapt a pretrained V-JEPA model~\citep{bardes2023v}, an SSL approach specialized for video understanding, through domain-adaptive pretraining on primate behavior datasets, validating their approach on behavior recognition tasks. Similarly, Animal-JEPA~\citep{zheng2024animal} modifies the V-JEPA training strategy with domain-specific masking techniques and validates on mouse behavior classification tasks.
\paragraph{Distinction from prior work.} While these methods share conceptual similarities with \beast, our approach is distinguished by its general frame-based training objectives and comprehensive evaluation across neural activity prediction and pose estimation tasks, in addition to the standard action segmentation task. Furthermore, since the pose-based methods described above rely on pose estimates as input, \beast could potentially enhance their performance by providing higher-quality pose estimation as a preprocessing step.
\section{\beast workflow}
We give an overview of the \beast workflow for a new user, highlighting steps where \beast enhances a traditional workflow.
\textbf{\textsc{Stage 1}: \textsc{Data collection}}
\begin{itemize}
\item Collect behavioral videos
\item Optionally collect simultaneous neural recordings
\end{itemize}
The use of \beast does not affect this step of the workflow.
\textbf{\textsc{Stage 2: \beast pretraining}}
This step will be skipped in a traditional workflow.
\begin{itemize}
\item Extract frames from unlabeled videos (typically $\sim$100K frames)
\item Train \beast transformer backbone on these frames ($\sim$12 hours on 8 GPUs)
\item This step does not require manual annotation
\end{itemize}
\textbf{\textsc{Stage 3: Downstream applications}}
\textit{Option A: Pose estimation}
\begin{itemize}
\item Annotate 100-1000 frames from 5-50 videos with keypoints (can be different from pretraining set)
\item Fine-tune pose estimation model using \beast backbone\\\textcolor{blue}{Key advantage: improved performance over existing backbones (Fig.~\ref{fig:pose_estimation})}
\item Run inference on new videos to extract keypoints
\end{itemize}
\textit{Option B: Action segmentation}
\begin{itemize}
\item Annotate 1000-5000 frames per behavior, ideally across 5-10 videos, with action labels (independent of pose annotation)
\item Fine-tune action segmentation model using \beast backbone \\
\textcolor{blue}{Key advantage 1: skip pose estimation pipeline entirely}\\
\textcolor{blue}{Key advantage 2: equivalent or better performance compared to pose estimates (Fig.~\ref{fig:action_segmentation})}
\item Run inference on new videos to extract frame-by-frame actions
\end{itemize}
\textit{Option C: Neural encoding}
\begin{itemize}
\item No manual annotation (neural activity provides the ``labels'')
\item Fine-tune neural encoding model using \beast backbone (video input, neural activity output)\\
\textcolor{blue}{Key advantage 1: skip pose estimation pipeline entirely}\\
\textcolor{blue}{Key advantage 2: improved performance over existing behavioral features (Fig.~\ref{fig:neural_encoding})}
\item Predict neural activity from video
\end{itemize}
\section{Broader Impacts}
The \beast framework enables more efficient extraction of meaningful information from video data, potentially accelerating behavioral neuroscience research with several beneficial outcomes. By reducing the need for extensive human labeling while improving accuracy, \beast can democratize advanced video analysis capabilities for laboratories with limited resources. This efficiency could accelerate basic science discoveries that underlie advances in biomedical applications, neurological disorder treatments, and improved understanding of brain function.
While \beast is developed primarily for behavioral neuroscience studies using animal subjects, the underlying technology could potentially be repurposed for human video analysis, raising several concerns:
\begin{itemize}
\item Surveillance capabilities: The improved ability to track and categorize behaviors could enhance surveillance technologies, potentially infringing on privacy rights if deployed without appropriate oversight.
\item Bias and fairness: As with any AI system trained on specific datasets, \beast-derived models may perform differently across demographic groups if applied to human subjects, potentially perpetuating biases in downstream applications.
\item Resource inequality: While a pretrained \beast model can improve the efficiency of downstream tasks, the computational requirements for pretraining itself may limit access to this technology for under-resourced institutions, potentially widening existing disparities in research capabilities.
\end{itemize}