Last Layer Hamiltonian Monte Carlo
Extracted source text · arXiv 2507.08905
\documentclass[10pt]{article} % For LaTeX2e
% \usepackage{tmlr}
% If accepted, instead use the following line for the camera-ready submission:
%\usepackage[accepted]{tmlr}
% To de-anonymize and remove mentions to TMLR (for example for posting to preprint servers), instead use the following:
% \usepackage{tmlr}
\usepackage[preprint]{tmlr}
\usepackage{xcolor}
\usepackage{mathtools}
\definecolor{steelblue31119180}{RGB}{31,119,180} % Define a color for the bars
% \usepackage{pgfplots}
% \pgfplotsset{compat=1.18}
% \usepackage{pgfplotstable}
% \usepackage{caption}
% \usepackage{tikz}
% \usetikzlibrary{pgfplots.groupplots}
\definecolor{steelblue31119180}{RGB}{31,119,180} % Define a color for the bars
\usepackage{pifont}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
% Add to your LaTeX preamble:
\usepackage{tikz}
\usetikzlibrary{shapes.geometric, arrows.meta, positioning}
\usepackage{hyperref}
\usepackage{url}
% \usepackage{tcolorbox}
% Recommended, but optional, packages for figures and better typesetting:
\usepackage{microtype}
\usepackage{graphicx}
% \usepackage{subfigure}
\usepackage{booktabs} % for professional tables
\usepackage{algorithm}
\usepackage{multirow}
\usepackage{algpseudocode}
% \usepackage[table]{xcolor}
\definecolor{best}{RGB}{128, 179, 255}
\definecolor{2best}{RGB}{252, 186, 182}
\definecolor{cornflowerblue}{rgb}{0.0, 0.72, 0.92}
\usepackage{appendix}
\usepackage{float}
\usepackage{stfloats}
% \definecolor{best}{RGB}{50, 177, 252}
% \definecolor{2ndbest}{RGB}{200, 244, 245}
\usepackage{subcaption}
% hyperref makes hyperlinks in the resulting PDF.
% If your build breaks (sometimes temporarily if a hyperlink spans a page)
% please comment out the following usepackage line and replace
% \usepackage{icml2024} with \usepackage[nohyperref]{icml2024} above.
% \usepackage{hyperref}
% \usepackage{fontspec} \newfontfamily{\NotoEmoji}
\usepackage{pgfplots}
\usepackage{tikz}
\pgfplotsset{compat=1.18}
\usetikzlibrary{pgfplots.fillbetween}
\usepgfplotslibrary{groupplots}
% Attempt to make hyperref and algorithmic work together better:
% \newcommand{\theHalgorithm}{\arabic{algorithm}}
% Import additional packages in the preamble file, before hyperref
% \input{preamble}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{tikz}
% It is strongly recommended to use hyperref, especially for the review version.
% hyperref with option pagebackref eases the reviewers' job.
% Please disable hyperref *only* if you encounter grave issues,
% e.g. with the file validation for the camera-ready version.
%
% If you comment hyperref and then uncomment it, you should delete *.aux before re-running LaTeX.
% (Or just hit 'q' on the first LaTeX run, let it finish, and you should be clear).
% \definecolor{cvprblue}{rgb}{0.21,0.49,0.74}
% \usepackage[pagebackref,breaklinks]{hyperref}
% For theorems and such
% \usepackage{amsmath}
% \usepackage{amssymb}
% \usepackage{mathtools}
% \usepackage{amsthm}
% if you use cleveref..
\usepackage[capitalize,noabbrev]{cleveref}
\usepackage{array}
\newenvironment{conditions}
{\par\vspace{\abovedisplayskip}\noindent
\begin{tabular}{>{}l<{} @{} c @{} >{}l<{}}}
{\end{tabular}\par\vspace{\belowdisplayskip}}
\title{Last Layer Hamiltonian Monte Carlo}
% Authors must not appear in the submitted version. They should be hidden
% as long as the tmlr package is used without the [accepted] or [preprint] options.
% Non-anonymous submissions will be rejected without review.
\author{\name Koen Vellenga \\
\addr Volvo Car Corporation \\ SAIL, University of Sk\"{o}vde
\AND
\name H. Joe Steinhauer \\
\addr SAIL, University of Sk\"{o}vde
\AND
\name G\"{o}ran Falkman\\
\addr SAIL, University of Sk\"{o}vde
\AND
\name Jonas Andersson \\
\addr Volvo Car Corporation
\AND
\name Anders Sj\"{o}gren\\
\addr Volvo Car Corporation
}
% \addr DeepMind
% \AND
% \name Hugo Larochelle \email hugolarochelle@google.com\\
% \addr Mila, Universit\'e de Montr\'eal \\
% Google Research\\
% CIFAR Fellow}
% The \author macro works with any number of authors. Use \AND
% to separate the names and addresses of multiple authors.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\def\month{MM} % Insert correct month for camera-ready version
\def\year{YYYY} % Insert correct year for camera-ready version
\def\openreview{\url{https://openreview.net/forum?id=XXXX}} % Insert correct link to OpenReview for camera-ready version
\begin{document}
\maketitle
%To preserve some of the benefits of HMC, while also enabling to adapt pre-trained models, we explore the use HMC confined to the last layer of a DNN.
\begin{abstract}
We explore the use of Hamiltonian Monte Carlo (HMC) sampling as a probabilistic last layer approach for deep neural networks (DNNs). While HMC is widely regarded as a gold standard for uncertainty estimation, the computational demands limit its application to large-scale datasets and large DNN architectures. Although the predictions from the sampled DNN parameters can be parallelized, the computational cost still scales linearly with the number of samples (similar to an ensemble). Last layer HMC (LL--HMC) reduces the required computations by restricting the HMC sampling to the final layer of a DNN, making it applicable to more data-intensive scenarios with limited computational resources. In this paper, we compare LL-–HMC against five last layer probabilistic deep learning (LL--PDL) methods across three real-world video datasets for driver action and intention. We evaluate the in-distribution classification performance, calibration, and out-of-distribution (OOD) detection. Due to the stochastic nature of the probabilistic evaluations, we performed five grid searches for different random seeds to avoid being reliant on a single initialization for the hyperparameter configurations. The results show that LL--HMC achieves competitive in-distribution classification and OOD detection performance. Additional sampled last layer parameters do not improve the classification performance, but can improve the OOD detection. Multiple chains or starting positions did not yield consistent improvements. Implementation is available at: \url{https://github.com/koenvellenga/LL-HMC/}.
% After evaluating the top-10 best performing hyperparameter configurations for 100 random seeds, we observe no single best performing LL--PDL method. Furthermore, additional chains, starting positions or samples did not consistently improve the LL--HMC performance.
% To demonstrate such a safety-critical resource constrained real-world application, we compare LL--HMC to five other last layer probabilistic deep learning (LL--PDL) approaches (deep deterministic uncertainty, Bayes-by-backprop, Packed ensembles, Sub-ensembles and Variational Bayes Last Layer) on three video driver action and intention recognition datasets. An extensive grid search is performed to explore the optimal hyperparameter configurations for each dataset.
% We systematically analyze LL–HMC’s sensitivity to hyperparameters, and the influence of additional samples, offering
% The grid search results indicate that LL--HMC can match the other LL--PDL methods on in-distribution performance, calibration, and uncertainty-based out-of-distribution (OOD) detection. After evaluating the top-10 best performing hyperparameter configurations for 100 random seeds, we observe no single best performing LL--PDL method.
%% shows that LL--HMC matches the performance of the other LL--PDL methods () , but no single approach consistently outperformed the other.
% for a single seed indicates that the LL--HMC
% to last layer probabilistic deep learning (LL--PDL) approaches (deep deterministic uncertainty, Bayes-by-backprop, Packed ensembles, Sub-ensembles and Variational Bayes Last Layer). While the grid search indicated top performance for LL--HMC, multiple runs with the top 10 configurations for different independent randoms seeds yielded no single best LL--PDL method.
% % compared to the other LL--PDL methods, evaluating the top 10 hyperparameter configurations for all methods did not yield a definitive best approach in terms of performance or OOD detection.
% We observed that the top-performing LL--HMC hyperparameter configuration can consistently match or outperform other probabilistic last layer approaches. However, the LL--HMC classification and OOD performance is highly sensitive to the optimal hyperparameter and can degrade significantly for suboptimal configurations. Moreover, the average performance for the top-performing configurations for both the classification and OOD detection performance for multiple random seeds does not result in a single best last layer probabilistic deep learning approach.
% Reducing the required compute further by using intermediate latent representations from the model did not improve the performance of the probabilistic last layer approaches.
% (a deep ensemble [N=5], last layer Bayesian-by-backprop, last layer sub-ensembles, last layer Packed-ensembles, Variational Bayes Last Layer, and Deep Deterministic Uncertainty)
% Additionally, while inference can be parallelized, the overall computational demand increases linearly with the number of samples.In safety-critical driving automation scenarios with limited computational resources, it is crucial to efficiently determine when the model cannot make confident predictions. By restricting the HMC sampling to the parameters of the last layer of a pre-trained DNN, we show that last layer HMC (LL--HMC) can easily be applied to video-based driver action and intention recognition. Compared to other probabilistic deep learning (PDL) methods (a deep ensemble [N=5], last layer Bayesian-by-backprop, last layer sub-ensembles, last layer Packed-ensembles, Variational Bayes Last Layer, and Deep Deterministic Uncertainty), we observe that LL--HMC consistently matches or exceeds top performance. From a confidence-performance and uncertainty-performance calibration perspective, none of the included methods consistently yield the best performance and calibration. For the uncertainty-based out-of-distribution detection, all PDL methods improved compared to the baseline, but LL--HMC consistently yielded top performance across all scenarios. Implementation will be made available upon acceptance. %
\end{abstract}
\section{Introduction}
The rapid progress in Artificial Intelligence (AI) has amplified the need for robust and transparent decision making \citep{hendrycks2021unsolved,papamarkou2024position,bengio2024managing,manchingal2025position}. When using deep neural networks (DNN) in safety-critical environments, it is important that the model is able to identify instances it cannot confidently predict. Regular DNNs are unable to express uncertainty about their outputs, suffer from opaque decision-making processes \citep{rudin2019stop}, can be poorly calibrated \citep{guo2017calibration, nixon2019measuring}, struggle with noisy sensor observations \citep{berrada2021make}, require explicit safety verification \citep{wang2024empowering}, and perform poorly on long-tail edge cases \citep{hendrycks2021unsolved}. To address the lack of uncertainty estimation in regular DNNs, various methods have been proposed (e.g., \citealt{neal2012bayesian,shafer2008tutorial,welling2011bayesian,hoffman2013stochastic,gal2016dropout, maddox2019simple, van2020uncertainty, wilson2020bayesian, franchi2024make}). Hamiltonian Monte-Carlo (HMC) sampling is often regarded as the gold standard for uncertainty quantification and can yield top performance \citep{izmailov2021bayesian}. However, HMC can become too computationally expensive to estimate the uncertainty when scaling to bigger datasets and larger DNN architectures.
\begin{figure}[t]
\centering
% First row
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/softmax.pdf} % Add
\captionsetup{font=tiny}
\caption{Softmax}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/ensemble_50.pdf} %
\captionsetup{font=tiny}
\caption{Deep Ensemble (N=50)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/hmc_ll_50.pdf} %
\captionsetup{font=tiny}
\caption{LL--HMC (N=50)}
\end{subfigure}
\caption{Uncertainty estimation example for the two moons toy dataset for a softmax, deep ensemble, and the last layer Hamiltonian Monte-Carlo methods. Yellow indicates certain areas, dark blue indicates uncertain areas (the color maps are normalized.).}
\label{fig:example_sdllhmc}
% \vspace{}
\end{figure}
In machine learning it is common to decompose uncertainty into aleatoric uncertainty (randomness in the process) and epistemic uncertainty (the lack of knowledge) \citep{der2009aleatory}. Distinguishing between these two uncertainty types can help to identify the potentially reducible epistemic uncertainty \citep{hullermeier2021aleatoric}. In practice, it is challenging to decouple these uncertainty types \citep{mucsanyi2024benchmarking}. However, the predictive uncertainty (the aleatoric and epistemic uncertainty combined) can be used to evaluate, for example, whether a model expresses higher uncertainty for out-of-distribution (OOD) instances. The effectiveness of the predictive uncertainty depends on how well a probabilistic approach captures the underlying uncertainty. In classification tasks, the predictive distribution (e.g., softmax output) is sometimes assumed to be sufficient for identifying OOD instances. However, the softmax distribution is commonly only uncertain along the decision boundaries \citep{van2020uncertainty}, which limits its use for detecting unseen or ambiguous inputs (see Figure \ref{fig:example_sdllhmc}). Similarly, an ensemble improves the uncertainty estimations for regions where it has not observed any data compared to a single softmax-based uncertainty estimation. However, the ensemble mostly remains uncertain along the decision boundary. Additionally, the ensemble scales linearly with the number of ensemble members in terms of required computational resources to produce the estimations.
Especially for safety-critical tasks that operate in a resource-constrained environment, we require efficient approaches to produce uncertainty estimates. Driving automation is an example safety-critical application domain with limited onboard computational resources, which requires continuous perception and decision-making under uncertainty. An example of a driving automation task is to accurately recognize the current actions and intentions of a driver to avoid future traffic conflicts. For such a human behavior application, estimating uncertainties and understanding the limitations of a model are essential. Therefore, in this paper we explore confining HMC sampling to the last layer of a DNN as an efficient probabilistic deep learning (PDL) approach. We evaluate last layer HMC (LL--HMC) on three driver action and intention recognition video datasets. We compare the classification performance and uncertainty-based OOD detection of LL--HMC to a deterministic softmax baseline, an ensemble, and five other probabilistic last layer approaches.
% The rapid progress in Artificial Intelligence (AI) has amplified the need for robust and transparent decision making \citep{hendrycks2021unsolved,papamarkou2024position,bengio2024managing,manchingal2025position}. When using deep neural networks (DNN) in safety-critical environments, it is important that the model is able to identify instances it cannot confidently predict. Regular DNNs are unable to express uncertainty about their outputs, suffer from opaque decision-making processes \citep{rudin2019stop}, are poorly calibrated \citep{guo2017calibration, nixon2019measuring}, struggle with noisy sensor observations \citep{berrada2021make}, require explicit safety verification \citep{wang2024empowering}, and perform poorly on long-tail edge cases \citep{hendrycks2021unsolved}. To address the lack of uncertainty estimation in regular DNNs, various methods have been proposed (e.g., \citealt{neal2012bayesian,shafer2008tutorial,welling2011bayesian,hoffman2013stochastic,gal2016dropout, maddox2019simple, van2020uncertainty, wilson2020bayesian, franchi2024make}). Hamiltonian Monte-Carlo (HMC) sampling is often regarded as the gold standard for uncertainty quantification and can yield top performance \citep{izmailov2021bayesian}. However, HMC can become too computationally expensive to estimate the uncertainty when scaling to bigger datasets and larger DNNs.
% In machine learning it is common to decompose uncertainty into aleatoric uncertainty (randomness in the process) and epistemic uncertainty (the lack of knowledge) \citep{der2009aleatory}. Distinguishing between these two uncertainty types can help to identify the potentially reducible epistemic uncertainty \cite{hullermeier2021aleatoric}. In practice, it is challenging to decouple these uncertainty types \citep{mucsanyi2024benchmarking}. However, the predictive uncertainty (the aleatoric and epistemic uncertainty combined) can be used to evaluate, for example, whether a model expresses higher uncertainty for out-of-distribution (OOD) instances. The effectiveness of the predictive uncertainty depends on how well a probabilistic approach captures the underlying uncertainty. In classification tasks, the predictive distribution (e.g., softmax output) is sometimes assumed to be sufficient for identifying OOD instances. However, the softmax distribution is commonly only uncertain along the decision boundaries \citep{van2020uncertainty}, which limits its use for detecting unseen or ambiguous inputs (see Figure \ref{fig:example_sdllhmc}). Similarly, an ensemble improves the uncertainty estimations for regions where it has not observed any data compared to a single softmax-based uncertainty estimation. However, the ensemble mostly remains uncertain along the decision boundary. Additionally, the ensemble scales linearly with the number of ensemble members in terms of required computational resources to produce the estimations.
% Especially for safety-critical tasks that operate in a resource-constrained environment, we require efficient approaches to produce uncertainty estimates. Driving automation is an example safety-critical application domain with limited computational resources, which requires continuous perception and decision-making under uncertainty. An example of a driving automation task is to accurately recognize the current actions and intentions of a driver to avoid future traffic conflicts. For such a human behavior application, estimating uncertainties and understanding the limitations of a model are essential. Therefore, in this paper we explore confining HMC sampling to the last layer of a DNN as an efficient probabilistic deep learning (PDL) approach. We evaluate last layer HMC (LL--HMC) on three driver action and intention recognition video datasets. We compare the action performance and uncertainty-based OOD detection of LL--HMC to a deterministic softmax baseline, an ensemble, and five other probabilistic last layer approaches.
%We also examine how using latent representations from different depths impacts performance and uncertainty estimation across these methods.
% solely relying on the softmax distribution as an uncertainty estimation are commonly only uncertain along the decision boundaries \citep{van2020uncertainty}
% can lead to undesirable results
% and Deep Ensembles are commonly only uncertain along the decision boundaries \citep{van2020uncertainty} (see Figure \ref{fig:example_sdllhmc}).
% Uncertainty estimations are critical for DNN applications in safety-critical enviro
% The increased required computational resources prohibit HMC from easily scaling to safety-critical
% This makes it difficult to yield the benefits from this
% With the rise of large foundation models, where (safety-critical) downstream tasks rely on pre-trained representations \citep{bommasani2021opportunities,moor2023foundation}, it would require lots of computational resources to continuously re-train DNNs from scratch in a probabilistic manner for every additional task.
% \begin{figure}[t]
% \centering
% % First row
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/softmax.pdf} % Add
% \captionsetup{font=tiny}
% \caption{Softmax}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/ensemble_50.pdf} %
% \captionsetup{font=tiny}
% \caption{Deep Ensemble (N=50)}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/hmc_ll_50.pdf} %
% \captionsetup{font=tiny}
% \caption{LL--HMC (N=50)}
% \end{subfigure}
% \caption{Uncertainty estimation example for the two moons toy dataset for a softmax, deep ensemble, and the last layer Hamiltonian Monte-Carlo methods.Yellow indicates certain areas, dark blue indicates uncertain areas (the color maps are normalized.)}
% \label{fig:example_sdllhmc}
% % \vspace{}
% \end{figure}
% In these systems, understanding human drivers' intentions and actions is essential for ensuring safety, yet must be done within the limits of onboard computational resources. Therefore, to test on a more realistic scenario, in this paper we evaluate LL--HMC on for three driver action and intention recognition video datasets. We compare the performance and uncertainty-based OOD detection of LL--HMC to a deterministic softmax-based baseline, a deep ensemble and five probabilistic last layer approaches. Additionally, for each last layer approach, we evaluate the influence of deeper effects of deeper latent representations.
% Both improving performance while efficiently producing uncertainty estimates and dynamically reducing the compute is essential for real-world application.
% in this paper we test the applicability of LL--HMC and
% a driving automation function is , to determine
% Given the continued presence of human driven vehicles on the road, it is important to understand human behavior and to assess whether their intended driving maneuvers are safe to execute given the current driving scene. A central component of such an advanced driver assistance system relies on video-based driving action and intention recognition \cite{vellenga2024evaluation}.
% Therefore, in this paper we explore the combination of optimization-based pre-training and HMC sampling for the last layer of a DNN for three driver action and intention recognition video datasets. We compare the performance and uncertainty-based OOD detection of LL--HMC to a deterministic softmax-based baseline, a deep ensemble (N=5) \citep{lakshminarayanan2017simple}, last layer Bayesian-by-backprop \citep{blei2017variational}, last layer sub-ensembles \citep{lee2015m,valdenegro2023sub}, last layer Packed-ensembles \citep{laurent2023packed}, Variational Bayes Last Layer \citep{harrison2024variational}, and Deep Deterministic Uncertainty \citep{mukhoti2023deep} (on non-spectrally normalized representations).
% and
% To further reduce the required compute to
% Given the limited computational resources of the target operating environment and that the latent representations can be already classified before the final layer of the network (e.g., \citealt{papyan2020prevalence,kothapallineural,qendro2021early,montavon2011kernel,meronen2024fixing}), we evaluate if the intermediate latent representations in combination with the LL--PDL methods can yield similar performance. Using an intermediate latent representation could significantly reduce the processing requirements for incoming videos. For each Transformer block in the ViT architecture, we apply average pooling to extract the intermediate representations. For each intermediate representation, we use the top-performing configurations of the performance experiment to train the LL--PDL methods and evaluate if earlier representations from the pre-trained network can yield similar performance.
% Estimating uncertainties for real-world applications where the computational capabilities of the operating environment are limited and where the problems require more complex DNN architectures remains a persistent challenge \citep{mukhoti2023deep,papamarkou2024position}. To address this, more efficient uncertainty estimation methods that require either a single forward pass or restrict the stochastic component to the last layer of the DNN have been proposed. Such methods offer advantages over ensembles, Markov-Chain Monte Carlo (MCMC), and sequential sampling-based methods in terms of required computational demand \citep{snoek2015scalable,watson2021latent}. Therefore, in this paper we explore the combination of optimization-based pre-training and HMC sampling for the last layer of a DNN for three driver action and intention recognition video datasets. We compare the performance and uncertainty-based OOD detection of LL--HMC to a deterministic softmax-based baseline, a deep ensemble (N=5) \citep{lakshminarayanan2017simple}, last layer Bayesian-by-backprop \citep{blei2017variational}, last layer sub-ensembles \citep{lee2015m,valdenegro2023sub}, last layer Packed-ensembles \citep{laurent2023packed}, Variational Bayes Last Layer \citep{harrison2024variational}, and Deep Deterministic Uncertainty \citep{mukhoti2023deep} (on non-spectrally normalized representations).
\section{Related Work}
% While these DAR and DIR approaches incrementally improved the performance, the robustness and uncertainty-based OOD detection performance is not considered, while this is essential for driving automation tasks.
%Enabling the quantification of uncertainty in DNNs allows for understanding the role of the DNNs parameters. Additionally, uncertainty estimations in predictions aids in risk assessment and enhances safety in decision-making processes.
% \subsection{Probabilistic Deep Learning}
\subsection{Probabilistic Deep Learning}
We provide a high-level explanation of a selection of PDL methods. For a comprehensive overview of Bayesian, PDL or stochastic approaches, refer to one of the following papers: \cite{zhou2021survey,jospin2022hands,arbel2023primer,gawlikowski2023survey,papamarkou2024position,manchingal2025position}. Markov Chain Monte Carlo (MCMC) can be used to estimate the posterior distribution of a DNN's parameters through sampling. Unlike optimization methods, which adjust the model’s parameters by calculating the gradient of the loss function to find a local minimum, MCMC samples from the parameter space using probabilistic inference. Bayes’ theorem ($p(\theta | D) = \frac{p(D | \theta) p(\theta)}{p(D)} $) plays a crucial role in this process, where $\theta$ represents the DNN parameters, $\mathcal{D}$ is the evidence, $p(\theta)$ the prior probability of the parameters before considering the evidence, and $p(\mathcal{D} | \theta)$ the likelihood of the evidence given the parameters. MCMC algorithms begin by selecting an initial value for each parameter and aim to explore the entire parameter space. An example of an informed MCMC sampler is the HMC algorithm \citep{neal1992bayesian, neal2011mcmc}, which leverages gradient information and principles from Hamiltonian dynamics to efficiently navigate the parameter space. The sampler proposes new parameter values and calculates the likelihood of the dataset given both the new and current parameter positions to evaluate the proposal. By dividing these probabilities, we obtain an acceptance ratio, allowing us to compare the full posterior probabilities at two different points without knowing their exact values. This comparison guides the algorithm to sample more frequently from regions with high posterior probability and less frequently from regions with low probability.
For HMC, the No U-Turn Sampler (NUTS) dynamically adjusts the number of steps taken by the integrator to ensure an optimal length for the Hamiltonian trajectory \citep{hoffman2014no}. Consequently, the HMC sampler can explore the parameter space efficiently by concentrating on more probable regions of the posterior. Initially, the samples generated by the Markov Chain may be far from the true target distribution, necessitating a burn-in period to ensure convergence to the posterior distribution $p(\theta | \mathcal{D})$. Once convergence is achieved, the samples produced by the Markov Chain can be stored and used for making predictions on the test data \citep{geyer2011introduction,betancourt2017conceptual,jospin2022hands}. \cite{izmailov2021bayesian} applied HMC to larger datasets and architectures, but required substantial computational resources. While there have been examples of either pre-training \citep{benker2021utilizing,vellenga2024pthmc}, chain-splitting \citep{cobb2021scaling} or stochastic gradient MCMC \citep{chen2014stochastic,zhang2019cyclical} to enable using HMC without significantly scaling the required computational resources, it remains challenging to use HMC sampling for large DNN architectures and big datasets.
Due to the computational demands of MCMC, alternative approaches like stochastic variational inference (SVI) have gained popularity for approximating posterior distributions of DNNs more efficiently. Variational inference (VI) aims to estimate the intractable posterior distribution through replacing the parameters of a DNN with a set of variational distributions $q$. A common approach for VI is to replace each network parameter as a 1-dimensional Gaussian distribution. To approximate the true (intractable) posterior distribution, the evidence lower bound (ELBO) \citep{jordan1999introduction,blei2017variational} is used to minimize the Kullback-Leibler (KL) divergence between the variational distribution and the true posterior. Since backpropagating through randomly sampled variables (the variational distributions $q$) is infeasible, Bayes-by-backprop (BBB, \citealt{blundell2015weight}) employs the reparameterization trick \citep{kingma2013auto}. The introduction of a new variable $\epsilon \sim \mathcal{N}(0,1)$ reparameterizes the random variable $q$, which allows the sampling process to be differentiable and thus facilitates the backpropagation.
Uncertainty can also be estimated by an ensemble of DNNs, which requires no assumptions about the posterior distribution. A Deep Ensemble (DE) \citep{lakshminarayanan2017simple} consists of multiple independently trained DNNs with random initializations, and tend to produce a diverse set of model parameters \citep{ashukhapitfalls,abe2022deep,laurentsymmetry}. However, a key limitation of sequential sampling approaches (e.g., BBB), MCMC or ensembles, is that they scale linearly to produce predictions and thus uncertainty estimations. This requires additional computations, which can be a burden for deployment in a resource-constrained safety-critical operating environment. Given the scalability limitations, more efficient sampling-free PDL approaches have been proposed. Sampling-free PDL methods do not rely on, for example, sampling the DNN parameters when estimating predictive uncertainty. An example of a distance-based sampling-free approach is Deep Deterministic Uncertainty (DDU) \citep{mukhoti2023deep}. DDU applies Gaussian Discriminant Analysis to the spectrally normalized feature space of a pre-trained DNN. DDU has demonstrated effectiveness on image classification OOD benchmarks. Packed-ensembles (PE) \citep{laurent2023packed} combine multiple sub-networks into a deterministic model by using grouped convolutions. This allows training multiple sub-networks simultaneously.
Complementing the more efficient PDL approaches, it is also possible to restrict the stochastic component to the last layer of a DNN \citep{snoek2015scalable,watson2021latent}. For example, Variational Bayes Last Layer (VBLL) \citep{harrison2024variational} performs variational inference over the parameters of the final layer. VBLL uses approximate analytic expression for the variational lower bound of the marginal likelihood, which eliminates the need for stochastic sampling. A Sub-Ensemble (SE) \citep{valdenegro2023sub} is an ensemble of independently trained classifiers instead of entire DNNs. BBB, and PE can also be applied to the final layer of a DNN to produce uncertainty estimations. Altogether, while these previous PDL approaches demonstrated efficient approximation of the posterior, they potentially fall short compared to MCMC-based approaches. An oversimplification of the true posterior can lead to over or underconfident uncertainty estimates, whereas HMC sampling can capture complex multi-modal distributions that are (highly) non-Gaussian.
\subsection{Driver action and intention recognition}
Given the continued presence of human-driven vehicles on the road, an important aspect of driving automation is to understand human behavior in traffic scenarios. An example of an advanced driver assistance system (ADAS) is to assess whether the intended maneuvers of a driver are safe to execute for a particular driving scene. Central components of such an ADAS are driver action and intention recognition. Actions refer to the observable behaviors a driver performs, such as steering, braking, or changing lanes, which can be directly captured through various sensors on a vehicle. Intention recognition focuses on identifying what a driver plans to do in the immediate future, before the actual maneuver takes place \citep{pereira2013state, sadri2011logic}. For example, before changing lanes, a driver might form the intention several seconds before executing any observable actions of that intended driving maneuver. Understanding both actions and intentions is crucial for developing an ADAS that can anticipate and respond to driver behavior \citep{vellenga2022driver}.
Previous studies on driving action recognition (DAR) and driver intention recognition (DIR) have explored various approaches to model the sequential spatio-temporal relationships captured by various sensors. For example, \cite{yang2023aide} demonstrated DAR performance improvements when fusing extracted facial landmarks of the ego-vehicle driver and end-to-end representations from three exterior video streams and one in-cabin video streams. \cite{noguchi2023ego} constructed spatial-temporal graphs based on exterior-based object detection and tracking, employing a semi-supervised contrastive learning framework to train a graph convolutional network. For end-to-end DIR, \cite{gebert2019end} extracted the optical flow of the in-cabin videos, whereas \cite{rong2020driver} extracted and predicted the optical flow for the exterior cameras. \cite{vellenga2024evaluation} fine-tuned video masked auto-encoders and attention fusion to improve DAR and DIR performance. Although these DAR and DIR approaches have incrementally improved performance, they have not considered the uncertainty-based OOD detection, which is a critical aspect for safe driving automation.
% We provide a high-level explanation of a selection of PDL methods. For a comprehensive overview of Bayesian, PDL or stochastic approaches, refer to one of the following papers: \cite{zhou2021survey,jospin2022hands,arbel2023primer,gawlikowski2023survey,papamarkou2024position,manchingal2025position}.
% Markov Chain Monte Carlo (MCMC) can be used to estimate the posterior distribution of a DNN's parameters through sampling. Unlike optimization methods, which adjust the model’s parameters by calculating the gradient of the loss function to find a local minimum, MCMC samples from the parameter space using probabilistic inference. Bayes’ theorem ($p(\theta | D) = \frac{p(D | \theta) p(\theta)}{p(D)} $) plays a crucial role in this process, where $\theta$ represents the DNN parameters, $\mathcal{D}$ is the evidence, $p(\theta)$ the prior probability of the parameters before considering the evidence, and $p(\mathcal{D} | \theta)$ the likelihood of the evidence given the parameters. MCMC algorithms begin by selecting an initial value for each parameter and aim to explore the entire parameter space. An example of an informed MCMC sampler is the HMC algorithm \citep{neal1992bayesian, neal2011mcmc}, which leverages gradient information and principles from Hamiltonian dynamics to efficiently navigate the parameter space. The sampler proposes new parameter values and calculates the likelihood of the dataset given both the new and current parameter positions to evaluate the proposal. By dividing these probabilities, we obtain an acceptance ratio, allowing us to compare the full posterior probabilities at two different points without knowing their exact values. This comparison guides the algorithm to sample more frequently from regions with high posterior probability and less frequently from regions with low probability.
% For HMC, the No U-Turn Sampler (NUTS) dynamically adjusts the number of steps taken by the integrator to ensure an optimal length for the Hamiltonian trajectory \citep{hoffman2014no}. Consequently, the HMC sampler can explore the parameter space efficiently by concentrating on more probable regions of the posterior. Initially, the samples generated by the Markov Chain may be far from the true target distribution, necessitating a burn-in period to ensure convergence to the posterior distribution $p(\theta | \mathcal{D})$. Once convergence is achieved, the samples produced by the Markov Chain can be stored and used for making predictions on the test data \citep{geyer2011introduction,betancourt2017conceptual,jospin2022hands}. \cite{izmailov2021bayesian} applied HMC to larger datasets and architectures, but required substantial computational resources. While there have been examples of either pretraining, chain-splitting or stochastic gradient MCMC to enable using HMC without significantly scaling the required computational resources \citep{chen2014stochastic,zhang2019cyclical,cobb2021scaling,benker2021utilizing,vellenga2024pthmc}, it remains challenging to use HMC sampling for large DNN architectures and big datasets.
% Due to the computational demands of MCMC, alternative approaches like stochastic variational inference (SVI) have gained popularity for approximating posterior distributions of DNNs more efficiently. Variational inference (VI) aims to estimate the intractable posterior distribution through replacing the parameters of a DNN with a set of variational distributions $q$. A common approach for VI is to replace each network parameter as a 1-dimensional Gaussian distribution. To approximate the true (intractable) posterior distribution, the evidence lower bound (ELBO) \citep{jordan1999introduction,blei2017variational} is used to minimize the Kullback-Leibler (KL) divergence between the variational distribution and the true posterior. Since backpropagating through randomly sampled variables (the variational distributions $q$) is infeasible, Bayes-by-backprop (BBB, \citealt{blundell2015weight}) employs the reparameterization trick \citep{kingma2013auto}. The introduction of a new variable $\epsilon \sim \mathcal{N}(0,1)$ reparameterizes the random variable $q$, which allows the sampling process to be differentiable and thus facilitates the backpropagation.
% % \begin{figure}[t]
% % \centering
% % \centering
% % \includegraphics[trim={0cm 8.8cm 16.5cm 0cm},clip, width=0.60\textwidth, keepaspectratio]{llhmc_thesis.pdf}
% % % \vspace{-0.5cm}
% % \caption{Schematic overview of the two last layer HMC phases. Phase 1 covers the regular optimization-based pre-training phase. During the second phase, the frozen pre-trained encoder is responsible for extracting the latent representations $Z$. Then, the NUTS sampler samples the parameters of the last layer $\theta_{LL}$.}\label{fig:concept}
% % \end{figure}
% Uncertainty can also be estimated by an ensemble of DNNs, which requires no assumptions about the posterior distribution. A Deep Ensemble (DE) \citep{lakshminarayanan2017simple} consists of multiple independently trained DNNs with random initializations, and tend to produce a diverse set of model parameters \citep{ashukhapitfalls,abe2022deep,laurentsymmetry}. However, a key limitation of sequential sampling approaches (e.g., BBB), MCMC or ensembles, is that they scale linearly to produce predictions and thus uncertainty estimations. This requires additional computations, which can be a burden for deployment in a resource-constrained safety-critical operating environment. Given the scalability limitations, more efficient sampling-free PDL approaches have been proposed. Sampling-free PDL methods do not rely on, for example, sampling the DNN parameters when estimating predictive uncertainty. An example of a distance-based sampling-free approach is Deep Deterministic Uncertainty (DDU) \citep{mukhoti2023deep}. DDU applies Gaussian Discriminant Analysis to the spectrally normalized feature space of a pre-trained DNN. DDU has demonstrated effectiveness on image classification OOD benchmarks. Packed-ensembles (PE) \citep{laurent2023packed} combine multiple sub-networks into a deterministic model by using grouped convolutions. This allows training multiple sub-networks simultaneously.
% Complementing the more efficient PDL approaches, it is also possible to restrict the stochastic component to the last layer of a DNN \citep{snoek2015scalable,watson2021latent}. For example, Variational Bayes Last Layer (VBLL) \citep{harrison2024variational} performs variational inference over the parameters of the final layer. VBLL uses approximate analytic expression for the variational lower bound of the marginal likelihood, which eliminates the need for stochastic sampling. A Sub-Ensemble (SE) \citep{valdenegro2023sub} is an ensemble of independently trained classifiers instead of entire DNNs. BBB, and PE can also be applied to the final layer of a DNN to produce uncertainty estimations. Altogether, while these previous PDL approaches demonstrated efficient approximation of the posterior, they potentially fall short compared to MCMC-based approaches. An oversimplification of the true posterior can lead to over or underconfident uncertainty estimates, whereas HMC sampling can capture complex multi-modal distributions that are (highly) non-Gaussian.
% \subsection{Driver action and intention recognition}
% Given the continued presence of human-driven vehicles on the road, an important aspect of driving automation is to understand human behavior in traffic scenarios. An example of an advanced driver assistance system (ADAS) is to assess whether the intended maneuvers of a driver are safe to execute for a particular driving scene. Central components of such an ADAS are driver action and intention recognition. Actions refer to the observable behaviors a driver performs, such as steering, braking, or changing lanes, which can be directly captured through various sensors on a vehicle. Intention recognition focuses on identifying what a driver plans to do in the immediate future, before the actual maneuver takes place \citep{pereira2013state, sadri2011logic}. For example, before changing lanes, a driver might form the intention several seconds before executing any observable actions of that intended driving maneuver. Understanding both actions and intentions is crucial for developing an ADAS that can anticipate and respond to driver behavior \citep{vellenga2022driver}.
% Previous studies on driving action recognition (DAR) and driver intention recognition (DIR) have explored various approaches to model the sequential spatio-temporal relationships captured by various sensors. For example, \cite{yang2023aide} demonstrated DAR performance improvements when fusing extracted facial landmarks of the ego-vehicle driver and end-to-end representations from three exterior video streams and one in-cabin video streams. \cite{noguchi2023ego} constructed spatial-temporal graphs based on exterior-based object detection and tracking, employing a semi-supervised contrastive learning framework to train a graph convolutional network. For end-to-end DIR, \cite{gebert2019end} extracted the optical flow of the in-cabin videos, whereas \cite{rong2020driver} extracted and predicted the optical flow for the exterior cameras. \cite{vellenga2024evaluation} fine-tuned video masked auto-encoders and attention fusion to improve DAR and DIR performance. Although these DAR and DIR approaches have incrementally improved performance, they have not considered the uncertainty-based OOD (out-of-distribution) detection, which is a critical aspect for safe driving automation.
%
% \citet{meronen2024fixing} demonstrated that an internal ensemble of multiple probabilistic intermediate last layer predictions can reduce the computational cost and improve the performance. Estimating uncertainty efficiently at intermediate layers supports adaptive computation. This enables more reliable predictions while lowering resource demands. This is especially valuable in safety-critical applications where timely decisions and limited hardware capacity are critical constraints.
% To reduce the required computations further, \cite{qendro2021early} and \cite{meronen2024fixing} showed that using probabilistic last layers trained on the intermediate representations of a single network can improve the performance. Subsequently, these intermediate predictions can be incrementally used to create a model-internal ensemble, which combines the predictions from different depths of a network to improve the performance \citep{meronen2024fixing}.
% where $p(\theta)$ is the prior, $p(\mathcal{D} | \theta)$ the likelihood, and $p(\mathcal{D})$ the marginal likelihood
% It describes how to update the posterior probability density function $p(\theta | \mathcal{D})$ given some evidence $\mathcal{D}$, where $\theta$ represents the DNN parameters, and $p(\theta)$ the prior probability of the parameters before considering the evidence. The prior probability of the DNN parameters $p(\mathcal{D} | \theta) $ represents the likelihood of the evidence given the parameters, and $p(\mathcal{D})$ is the marginal probability of the evidence.
% \citept{chen2014stochastic} introduced stochastic gradient MCMC (SG-MCMC), which does not use the full dataset for posterior estimation, but relies on mini-batches that introduce noise and variability. To address this, a friction term is added as a corrective measure to mitigate excessive fluctuations. To improve the ability of SGHMC to navigate the complex loss landscape of DNNs, \citept{zhang2019cyclical} extended SGHMC with cyclical warm restarts. This extension promotes exploration and sampling from various modes within the multi-modal posterior distributions of the parameter space.
% Practically this means replacing last layer of a network with variational distributions \citep{} or use multiple prediction heads \citep{lee2015m,valdenegro2023sub}, to enable a DNN to produce uncertainty estimations.
% At inference time, such stochastic variational inference (SVI) methods sample the parameters of the DNN from approximated variational distributions $q$. A downside of sampling the network parameters multiple times for the same input data is that it increases the required compute to produce an uncertainty estimation.
% PDL methods that rely on multiple forward passes to produce uncertainty estimations can be unappealing for real-time applications with limited available compute. A more efficient alternative is to only replace the parameters of the last layer of a network with variational distributions \citep{snoek2015scalable,watson2021latent}, sub-ensembles or multiple prediction head \citep{lee2015m,valdenegro2023sub} to enable a DNN to produce uncertainty estimations.
% Deep ensembles \citep{lakshminarayanan2017simple} consist of multiple independently trained DNNs with random initializations, and tend to produce a diverse set of model parameters \citep{ashukhapitfalls,abe2022deep,laurentsymmetry}.
% However, a key limitation of both sequential sampling approaches or ensembles is that they scale linearly at inference time. This requires additional compute, which can be burden for deployment in a resource-constrained safety-critical operating environment.
%ZigZag \citep{durasov2024zigzag} relies on the reconstruction error of an instance to identify OODs \citep{zhou2022rethinking}. ZigZag requires two inferences, the first without any prior information and the second with the initial prediction appended to the input data. To avoid the need to predict multiple times, \citept{vellenga2025latent} proposed latent uncertainty representation (LUR). LUR assumes the same principle as ZigZag, but uses linear projection layers to create multiple latent representations to avoid altering existing layers of a network and to make multiple inferences to estimate the uncertainty.
%Conformal Prediction (CP) \citep{shafer2008tutorial} leverages past data to validate prediction confidence, constructing prediction regions that guarantee the true outcome with a specified confidence level, independent of data distribution assumptions
% Laplace approximations \citep{daxberger2021laplace} use the curvature of the loss landscape around the maximum a posteriori estimate to define the covariance of the Gaussian, thus providing a practical way to estimate predictive uncertainty without extensive sampling.
% \subsection{Driver action and intention recognition}
% Driving actions and intentions are related but distinct concepts. Actions refer to observable driver behaviors (e.g., steering, braking, lane changes) captured via vehicle sensors, while intention recognition predicts a driver’s next move before execution \citep{pereira2013state, sadri2011logic}. For instance, a driver may form the intent to change lanes seconds before any physical maneuver. Understanding both is key to enhancing driver assistance systems \citep{vellenga2022driver}.
% Ramanishka et al. (2018) \citep{ramanishka2018toward} applied a CNN to extract per-frame feature representations and an LSTM to capture sequential patterns for action recognition. Wang et al. (2021) \citep{wang2021oadtr} extended this approach by incorporating a Transformer encoder-decoder setup to better model temporal interactions. Similarly, Noguchi and Tanizawa (2023) \citep{noguchi2023ego} constructed spatial-temporal graphs based on object detection and tracking, employing a semi-supervised contrastive learning framework to train a graph convolutional network for improved recognition of driving actions.
% For Driving Intention Recognition (DIR), multiple end-to-end approaches and multimodal fusion techniques have been explored. Jain et al. (2016) \citep{Jain2016} integrated driver facial landmarks with contextual cues, such as lane position, number of lanes and upcoming intersections, together into an LSTM for intention prediction. Gebert et al. (2019) \citep{gebert2019end} utilized an end-to-end framework where optical flow encoded in-cabin video dynamics, followed by a 3D ResNet for feature extraction. Rong et al. (2020) \citep{rong2020driver} extended this idea by extracting optical flow from the external videos and employed a ConvLSTM encoder-decoder architecture to incorporate future flow prediction, enhancing intention recognition performance. Ma et al. (2023) \citep{ma2023cemformer} introduced the Cross-View Episodic Memory Transformer, which efficiently learns unified memory representations while leveraging a context-consistency loss and Vellenga et al. \citep{vellenga2024evaluation} finetuned video masked auto-encoders and attention fusion to improve the DIR performance. While all of these approaches incrementally improved the performance, most previous works have not evaluated the ability to detect OOD instances or evaluated the uncertainty calibration.
% Driving Action Recognition (DAR) approaches have utilized techniques such as CNNs for frame-wise feature extraction with LSTMs for sequential modeling \citep{ramanishka2018toward}, Transformer encoder-decoder setups for improved temporal interactions \citep{wang2021oadtr}, and graph convolutional networks with contrastive learning for object-based action recognition \citep{noguchi2023ego}. End-to-end Driving Intention Recognition (DIR) approaches range from optical flow with 3D ResNets for in-cabin videos \citep{gebert2019end}, predicting external video flow with ConvLSTMs\citep{rong2020driver}, a Cross-View Episodic Memory Transformer for unified memory learning \citep{ma2023cemformer}, and fine-tuning video masked auto-encoders with attention fusion \citep{vellenga2024evaluation}. However, these all of these previous works do not consider probabilistic approaches to understand the uncertainty of each prediction.
\subsection{Driver action and intention recognition}
Given the continued presence of human-driven vehicles on the road, an important aspect of driving automation is to understand human behavior in traffic scenarios. An example of an advanced driver assistance system (ADAS) is to assess whether the intended maneuvers of a driver are safe to execute for a particular driving scene. Central components of such an ADAS are driver action and intention recognition. Actions refer to the observable behaviors a driver performs, such as steering, braking, or changing lanes, which can be directly captured through various sensors on a vehicle. Intention recognition focuses on identifying what a driver plans to do in the immediate future, before the actual maneuver takes place \citep{pereira2013state, sadri2011logic}. For example, before changing lanes, a driver might form the intention several seconds before executing any observable actions of that intended driving maneuver. Understanding both actions and intentions is crucial for developing an ADAS that can anticipate and respond to driver behavior \citep{vellenga2022driver}.
Previous studies on driving action recognition (DAR) and driver intention recognition (DIR) have explored various approaches to model the sequential spatio-temporal relationships captured by various sensors. For example, Yang et al. (2023) \cite{yang2023aide} demonstrated DAR performance improvements when fusing extracted facial landmarks of the ego-vehicle driver and end-to-end representations from three exterior video streams and one in-cabin video streams. Noguchi and Tanizawa (2023) \cite{noguchi2023ego} constructed spatial-temporal graphs based on exterior-based object detection and tracking, employing a semi-supervised contrastive learning framework to train a graph convolutional network. For end-to-end DIR, Gebert et al. (2019) \cite{gebert2019end} extracted the optical flow of the in-cabin videos, whereas Rong et al. (2020) \cite{rong2020driver} extracted and predicted the optical flow for the exterior cameras. Vellenga et al. (2024) \cite{vellenga2024evaluation} fine-tuned video masked auto-encoders and attention fusion to improve DAR and DIR performance. Although these DAR and DIR approaches have incrementally improved performance, they have not considered the uncertainty-based OOD detection, which is a critical aspect for safe driving automation.
\begin{figure}[t]
% \centering
% First row
% \begin{minipage}[t]{0.48\linewidth}
\centering
\includegraphics[trim={0cm 8.5cm 16.6cm 0cm},clip, width=0.70\textwidth, keepaspectratio]{llhmc_thesis.pdf}
% %\vspace{-0.5cm}
\caption{Schematic overview of the two last layer HMC phases. Phase 1 covers the regular optimization-based training phase. During phase 2, the frozen pre-trained encoder is responsible for extracting the latent representations $z$. Then, the HMC is applied to sample the parameters of the last layer $\theta_{LL}$.}\label{fig:concept}
% \end{minipage}
% \hfill
% \begin{minipage}[t]{0.48\linewidth}
% \centering
% \includegraphics[trim={0.4cm 9cm 18.0cm 0cm},clip, width=0.92\textwidth, keepaspectratio]{intermediate_ens.pdf}% Add
% % \captionsetup{font=tiny}
% \caption{Intermediate representation classification overview. The intermediate latent representation $z_{int}^{i}$ are extracted at different depths from the encoder, which can be used to produce intermediate classifications. }\label{fig:intermed}
% \end{minipage}
\end{figure}
\begin{algorithm}[t]
\caption{Last Layer Hamiltonian Monte Carlo}\label{algo:llhmc}
\begin{flushleft} \small{\textbf{Parameters --} $\theta$: Network parameters, $B$: Burn-in period, $E$:Number of epochs, $C$: Number of chains, $S$: Number of HMC samples, $P$: Parameter storage matrix
}
\end{flushleft}
\begin{algorithmic}[1]
\State Randomly initialize network parameters $\theta$
\State \textbf{Phase 1 -- Optimization-based pre-training:}
\For{$e$ $\leftarrow 1,\ldots, E$}
\State Update $\theta^{(e)}$ using a stochastic optimizer (e.g., SGD/ADAM)
\EndFor
\State Remove the last layer ($\theta_{LL})$ and process the raw input data produce the latent representations $\mathbf{Z}$ for the dataset
\State \textbf{Phase 2 -- HMC Sampling for $\theta_{LL}$:}
\State Initialize the last layer's parameters $\theta_{\text{LL}}$
\For{$c$ $\leftarrow 1,\ldots, C$}
\For{$s$ $\leftarrow 1,\ldots, S$}
\State Perform HMC to generate a new sample $\theta_{\text{LL}}'$
\If{$s > B$} \Comment{After burn-in period}
\State Add $\theta_{\text{LL}}'$ to $P \lbrack c,s \rbrack $
\EndIf
\EndFor
\EndFor
\end{algorithmic}
\end{algorithm}
% \begin{figure}[t]
% % \centering
% % First row
% % \begin{minipage}[t]{0.48\linewidth}
% \centering
% \includegraphics[trim={0cm 8.5cm 16.5cm 0cm},clip, width=0.51\textwidth, keepaspectratio]{llhmc_thesis.pdf}
% % \vspace{-0.5cm}
% \caption{Schematic overview of the two last layer HMC phases. Phase 1 covers the regular optimization-based training phase. During phase 2, the frozen pre-trained encoder is responsible for extracting the latent representations $z$. Then, the HMC is applied to sample the parameters of the last layer $\theta_{LL}$.}\label{fig:concept}
% % \end{minipage}
% % \hfill
% % \begin{minipage}[t]{0.48\linewidth}
% % \centering
% % \includegraphics[trim={0.4cm 9cm 18.0cm 0cm},clip, width=0.92\textwidth, keepaspectratio]{intermediate_ens.pdf}% Add
% % % \captionsetup{font=tiny}
% % \caption{Intermediate representation classification overview. The intermediate latent representation $z_{int}^{i}$ are extracted at different depths from the encoder, which can be used to produce intermediate classifications. }\label{fig:intermed}
% % \end{minipage}
% \end{figure}
\section{Last layer Hamiltonian Monte Carlo}
Given that HMC sampling scales poorly to larger DNNs and bigger datasets, LL–HMC restricts the HMC sampling procedure to the final layer of a DNN. This simplification reduces the computational burden, but introduces a dependency on the fixed trained latent representations. We divide the LL--HMC approach into two phases: optimization-based training and HMC--sampling for the last layer $\theta_{LL}$ (see \cref{algo:llhmc} and \cref{fig:concept}). Phase 1 concerns regular training or fine-tuning of the DNN using a standard gradient-based optimizer (e.g., stochastic gradient descent [SGD] or ADAM) to update all layers of a network $\theta$. Once the training (or fine-tuning) is complete, the last layer $\theta_{LL}$ is removed, and the latent representations $\mathbf{z}$ are extracted from the penultimate layer for the training, test, and OOD datasets. Before starting the HMC sampling, we randomly initialize the parameter values of $\theta_{LL}$. Then, for each chain $C$, HMC sampling is performed for $\theta_{LL}$ to estimate the posterior distribution given the latent representations $\mathbf{z}$ instead of the original raw data. HMC treats the model parameters as positions in a dynamical system to facilitate exploration of the posterior. The HMC algorithm uses the leapfrog integrator to propose new parameter values \(\theta'_{LL}\) by simulating the dynamics of the Hamiltonian system. The NUTS sampler \citep{hoffman2014no} dynamically adjusts the number of leapfrog steps in HMC to avoid inefficient exploration of the parameter space. The acceptance ratio between the current state \((\theta_{LL}, \mathbf{p})\) and the proposed state \((\theta'_{LL}, \mathbf{p}')\) guides the decision to accept or reject the proposed parameters.
% Given that HMC sampling scales poorly to larger DNNs and bigger datasets, LL–HMC restricts the HMC sampling procedure to the final layer of a DNN. This simplification reduces the computational burden, but introduces a dependency on the fixed trained latent representations. We divide the LL--HMC approach into two phases: optimization-based training and HMC--sampling for the last layer $\theta_{LL}$ (see \cref{algo:llhmc} and \cref{fig:concept}). Phase 1 concerns regular training the DNN using a standard gradient-based optimizer (e.g., stochastic gradient descent [SGD] or ADAM) to update all layers of a network $\theta$. Once the training (or fine-tuning) is complete, the last layer $\theta_{LL}$ is removed, and the latent representations $\mathbf{z}$ are extracted from the penultimate layer for the training, test, and OOD datasets. Before starting the HMC sampling, we randomly initialize the parameter values of $\theta_{LL}$. Then, for each chain $C$, HMC sampling is performed for $\theta_{LL}$ to estimate the posterior distribution given the latent representations $\mathbf{z}$ instead of the original raw data. HMC treats the model parameters as positions in a dynamical system to facilitate exploration of the posterior. The HMC algorithm uses the leapfrog integrator to propose new parameter values \(\theta'_{LL}\) by simulating the dynamics of the Hamiltonian system. The NUTS sampler \citep{hoffman2014no} dynamically adjusts the number of leapfrog steps in HMC to avoid inefficient exploration of the parameter space. The acceptance ratio between the current state \((\theta_{LL}, \mathbf{p})\) and the proposed state \((\theta'_{LL}, \mathbf{p}')\) guides the decision to accept or reject the proposed parameters.
% \begin{algorithm}[t]
% \caption{Last Layer Hamiltonian Monte Carlo}\label{algo:llhmc}
% \begin{flushleft} \small{\textbf{Parameters --} $\theta$: Network parameters, $B$: Burn-in period, $E$:Number of epochs, $C$: Number of chains, $S$: Number of HMC samples, $P$: Parameter storage matrix
% }
% \end{flushleft}
% \begin{algorithmic}[1]
% \State Randomly initialize network parameters $\theta$
% \State \textbf{Phase 1 -- Optimization-based pre-training:}
% \For{$e$ $\leftarrow 1,\ldots, E$}
% \State Update $\theta^{(e)}$ using a stochastic optimizer (e.g., SGD/ADAM)
% \EndFor
% \State Remove the last layer ($\theta_{LL})$ and process the raw input data produce the latent representations $\mathbf{Z}$ for the dataset
% \State \textbf{Phase 2 -- HMC Sampling for $\theta_{LL}$:}
% \State Initialize the last layer's parameters $\theta_{\text{LL}}$
% \For{$c$ $\leftarrow 1,\ldots, C$}
% \For{$s$ $\leftarrow 1,\ldots, S$}
% \State Perform HMC to generate a new sample $\theta_{\text{LL}}'$
% \If{$s > B$} \Comment{After burn-in period}
% \State Add $\theta_{\text{LL}}'$ to $P \lbrack c,s \rbrack $
% \EndIf
% \EndFor
% \EndFor
% \end{algorithmic}
% \end{algorithm}
\begin{figure}[t]
% \centering
% First row
\begin{subfigure}[b]{0.30\linewidth}
% \centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/hmc_5.pdf} % Add
\captionsetup{font=tiny}
\caption{HMC NUTS (N=5)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/hmc_50.pdf} %
\captionsetup{font=tiny}
\caption{HMC NUTS (N=50)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/hmc_100.pdf} %
\captionsetup{font=tiny}
\caption{HMC NUTS (N=100)}
\end{subfigure}
% \vskip\baselineskip % Adds vertical space between rows
% Second row
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/hmc_ll_5.pdf} %
\captionsetup{font=tiny}
\caption{LL--HMC NUTS (N=5)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/hmc_ll_50.pdf} % Add your image here
% \caption{Image 5}
\captionsetup{font=tiny}
\caption{LL--HMC NUTS (N=50)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/hmc_ll_100.pdf}
\captionsetup{font=tiny}
\caption{LL--HMC NUTS (N=100)}
\end{subfigure}
% \vskip\baselineskip
% Third row
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/ensemble_5.pdf}
\captionsetup{font=tiny}
\caption{DE (N=5)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/ensemble_50.pdf} % Add your image
\captionsetup{font=tiny}
\caption{DE (N=50)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=1.7cm 1.05cm 0.95cm 1.35cm, clip]{toy_class/ensemble_100.pdf} % Add your image
\captionsetup{font=tiny}
\caption{DE (N=100)}
\end{subfigure}
\caption{Toy two moons classification example results. First row consists of full HMC with 5, 50, and 100 samples, the second row shows the LL--HMC results, and the third row shows the results for a deep ensemble. The yellow area denotes certain predictions, whereas the dark blue area indicates uncertain predictions. The certain (yellow) vs uncertain (dark blue) color map is normalized.}
\label{fig:toy_class}
%% hmc ll burn in 50, prior=70, ta=0.8
\end{figure}
\begin{figure}[t!]
\centering
% First row
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/hmc_5.pdf} % Add your image here
\captionsetup{font=tiny}
\caption{HMC NUTS (N=5)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/hmc_50.pdf} % Add your image here
\captionsetup{font=tiny}
\caption{HMC NUTS (N=50)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/hmc_100.pdf} % Add
\captionsetup{font=tiny}
\caption{HMC NUTS (N=100)}
\end{subfigure}
\vskip\baselineskip % Adds vertical space between rows
% Second row
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ll_hmc_5.pdf} % Add
\captionsetup{font=tiny}
\caption{LL--HMC NUTS (N=5)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ll_hmc_50.pdf} %
\captionsetup{font=tiny}
\caption{LL--HMC NUTS (N=50)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ll_hmc_100.pdf} %
\captionsetup{font=tiny}
\caption{LL--HMC NUTS (N=100)}
\end{subfigure}
\vskip\baselineskip
% Third row
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ensemble_5.pdf} %
\captionsetup{font=tiny}
\caption{DE (N=5)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ensemble_50.pdf} %
\captionsetup{font=tiny}
\caption{DE (N=50)}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.30\linewidth}
\centering
\includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ensemble_100.pdf} %
\captionsetup{font=tiny}
\caption{DE (N=100)}
\end{subfigure}
\caption{Toy Regression Example results. First row consists of full HMC with 5, 50, and 100 samples, the second row shows the LL--HMC results, and the third row shows the results for a deep ensemble. The y-axis is the target variable, the x-axis the observations. The gray line is the original function, the orange dots represent the noisy observations, the blue line the mean prediction, and the light blue area the uncertainty estimation.}
\label{fig:toy_reg}
\end{figure}
\subsection{Toy-examples}
We visualize the uncertainty estimation capabilities of LL–HMC using a classification and a regression toy example. Specifically, we compare LL--HMC to HMC sampling on the entire network and a DE that uses a similar number of samples. We use the two moons dataset \citep{pedregosa2011scikit} with a multi-layer perceptron with two hidden layers of 20 units, following the implementation from van \cite{van2020uncertainty}. Figure \ref{fig:toy_class} shows where the different methods are certain (yellow) and uncertain (dark blue) for the two moons dataset. With more samples, full HMC gradually becomes more uncertain in areas for which no data has been observed. LL--HMC becomes more uncertain in unobserved areas as the number of samples increases from 5 to 50. However, the uncertainty estimations do not improve further with additional samples, indicating the limitations of the LL--HMC approach. In congruence with
\cite{van2020uncertainty}, we observe that the DE mostly remains uncertain along the decision boundary and only improves slightly with more ensemble members.
To evaluate the uncertainty estimations for a regression toy example, we use the implementation of \cite{lippe2022uva} to generate noisy observations from a sinusoidal function. The underlying architecture is an MLP with five hidden layers and 50 hidden units for all methods. Figure \ref{fig:toy_reg} shows the uncertainty estimation results for the toy regression example. When performing HMC on the entire DNN, we observe higher uncertainty in regions without observations. For both the LL--HMC and the deep ensemble, the uncertainty between the two observation areas remains relatively stable, further highlighting the advantage of full-network HMC in capturing the uncertainty.
% \subsection{Classification toy-example}
% We visualize the uncertainty estimation capabilities of LL–HMC using a classification toy example (refer to Appendix \ref{app:reg} for a regression example). Specifically, we compare LL--HMC to HMC sampling on the entire network and a DE that uses a similar number of samples. We use the two moons dataset \citep{pedregosa2011scikit} with a multi-layer perceptron with two hidden layers of 20 units, following the implementation from \cite{van2020uncertainty}. Figure \ref{fig:toy_class} shows where the different methods are certain (yellow) and uncertain (dark blue) for the two moons dataset. With more samples, full HMC gradually becomes more uncertain in areas for which no data has been observed. LL--HMC becomes more uncertain in unobserved areas as the number of samples increases from 5 to 50. However, the uncertainty estimations do not improve further with additional samples, indicating the limitations of the LL--HMC approach. In congruence with \cite{van2020uncertainty}, we observe that the DE mostly remains uncertain along the decision boundary and only improves slightly with more ensemble members. %Thus, while LL--HMC shows limitations in estimating the uncertainty compared to applying HMC to an entire network, it is does improve compared to an ensemble, its ability to deliver principled posterior estimates at a fraction of the computational cost makes it a promising approach for scalable uncertainty quantification.
\section{Experimental setup}
\subsection{Datasets}
For the DAR task, we use the AssIstive Driving pErception (AIDE) dataset \citep{yang2023aide} that consists of 2898 short videos recorded in China, with the following labels: 185 \textit{ backward moving}, 1522 \textit{forward moving}, 138 \textit{lane changing}, 727 \textit{parking}, and 326 \textit{turning}. Additionally, we use annotations from the ROad event Awareness Dataset (ROAD) \citep{singh2022road}, which extend a subset of 22 videos (122.000 frames) of the Oxford RobotCar Dataset \citep{RobotCarDatasetIJRR} with ego-vehicle and road user driving action labels. Since the ROAD data was collected with manual driving, the vehicle action labels reflect human driving maneuvers \citep{RobotCarDatasetIJRR}. To capture temporal patterns, we transform the frame-level annotations into video-level labels by selecting only driving actions with at least 8 consecutive frames. This threshold ensures that each clip provides sufficient temporal context for distinguishing maneuvers while reducing ambiguous or noisy labels. As a result, the offline ROAD DAR dataset consists of 75 \textit{stopping}, 162 \textit{move forward}, 31 \textit{turn right}, {41} \textit{turn left}, 12 \textit{move right}, 12 \textit{move left} and 8 \textit{overtake} maneuvers.
For DIR, we use the open-source Brain4Cars (B4C) dataset \citep{jain2015car} collected in the United States. The B4C dataset consists of 124 \textit{left lane changes}, 123 \textit{right lane changes}, 58 \textit{left turns}, 55 \textit{right turns}, and 234 \textit{driving straight} maneuvers. We retain the natural class imbalance present in all datasets to reflect the real-world distribution of driving behaviors. For all datasets, we use the original train-test splits. For the ROAD and B4C datasets, we use the first fold from the provided splits.
\subsection{Metrics}
\paragraph{\textbf{Performance, and uncertainty.}}We report standard metrics (accuracy $\uparrow$, F1-score $\uparrow$) to measure the in-distribution performance. To understand the differences between the PDL methods, we use two times the standard error of the mean (2SEM, \citealt{barde2012use}, Equation \ref{eq:2sem}) instead of the standard deviation (the variability across random seeds). 2SEM roughly corresponds to the 95\% confidence interval of the mean performance of each PDL method. To estimate uncertainty, we use predictive entropy \citep{smith2018understanding} (Equation \ref{eq:uc_it}), which is maximized when the probabilities are uniformly distributed among all classes. The more confident the model is in its predictions for a particular class, the lower the predictive entropy will be, indicating less uncertainty.
% to understand the mean differences between the methods rather than the variability across individual runs.
\begin{equation}
2SEM = 2 \frac{\sigma}{\sqrt{n}},
\label{eq:2sem}
\end{equation}
where:
\begin{conditions}
$\sigma$& = & standard deviation, and \\
$n$ &=& number of observations.
\end{conditions}
\begin{equation}
H[P(y|x)] = - \sum_{y \in \mathcal{Y}} P(y|x) \log P(y|x),
\label{eq:uc_it}
\end{equation}
where:
\begin{conditions}
$H[P(y|x)]$ & = & the entropy of the predictive distribution,\\
$P(y|x)$ & = & conditional probability distribution, over some discrete set of outcomes $\mathcal{Y}$, and\\
$\mathcal{Y}$& = & set of $N$ stochastic predictions.
\end{conditions}
\paragraph{\textbf{Calibration.}}To measure the alignment between the prediction confidence and the performance, we use the adaptive calibration error (ACE $\downarrow$, \citealt{nixon2019measuring}, Equation \ref{eq:ACE}) with 10 bins . ACE partitions predictions into adaptively-sized bins, comparing the average predicted probability in each bin to the corresponding fraction of correct predictions. Overconfidence occurs when predicted confidence exceeds accuracy; conservativeness occurs when confidence falls short of accuracy. To assess the quality of the uncertainty estimates relative to model performance, we use the relative Area Under the Lifted Curve (rAULC $\uparrow$, \citealt{postels2022practicality}, Equation \ref{eq:raulc}). The rAULC evaluates how well uncertainty scores are calibrated. Ideally, confident and correct predictions should correspond to low uncertainty, while incorrect or low-confidence predictions should yield high uncertainty.
\begin{equation}
ACE = \frac{1}{KR} \sum_{k=1}^{K} \sum_{r=1}^{R} | acc(r,k) - conf(r,k) |,
\label{eq:ACE}
\end{equation}
where:
\begin{conditions}
$k$ & = & class label,\\
$r$ & = & adaptive range for class label,\\
$acc(r,k)$ & = & accuracy within a bin, and\\
$conf(r,k)$ & = & predicted confidence within a bin.\\
\end{conditions}
\begin{equation}
\text{rAULC} = -1 + \sum_{i=1}^{S} \frac{s \cdot A(q_i)}{R(q_i)},
\label{eq:raulc}
\end{equation}
where:
\begin{conditions}
$q_i$ & = & uncertainty quantiles $i\in [1, ..., S]$, \\
$s$ & = & quantile step width based on the number of predictions in the test dataset ($1/{N}$), \\
$A(q_i)$ & = & accuracy of samples with uncertainty below quantile $q_i$, \\
$R(q_i)$ & = & expected accuracy for random guessing at quantile $q_i$, \\
$S$ & = & total number of quantiles. \\
\end{conditions}
\paragraph{\textbf{MCMC diagnostics.}}We evaluate the quality of posterior samples using the Effective Sample Size (ESS, \citealt{geyer1992practical}, Equation \ref{eq:ess}). A higher ESS implies better exploration of the posterior distribution. For LL--HMC with multiple chains, we use Gelman’s $\hat{R}$ statistic \citep{gelman1992inference} to assess convergence (Equation \ref{eq:rhat}). $\hat{R}$ compares the variance between chains to the variance within each chain. Values close to 1 suggest that the chains are exploring the same distribution \citep{izmailov2021bayesian}.
\begin{equation}\label{eq:ess}
ESS(N) = N / \left(1 + 2 \times \sum_{k=1}^{N-1} (1 - \frac{k}{N}) \times R_k \right),
\end{equation}
where:
\begin{conditions}
$N$ &=& total number of samples, and \\
$R_k$ &=& the auto-correlation at lag $k$.
\end{conditions}
\begin{equation}
\label{eq:rhat}
\hat{R} = \frac{E[\text{Var}[X | \text{chain}]] + \text{Var}[E[X | \text{chain}]]}{E[\text{Var}[X | \text{chain}]]},
\end{equation}
where:
\begin{conditions}
$E[\text{Var}[X | \text{chain}]]$&=& the expected value of the variance of X given the chain, and\\
$Var[E[X | \text{chain}]]$& = & the variance of the expected value of X given the chain.\\
\end{conditions}
\paragraph{\textbf{OOD detection.}}To determine whether OOD instances yield high uncertainty, we compute the Area Under the Receiver Operating Characteristic (ROC, $\uparrow$) and Precision-Recall (PR, $\uparrow$) curves \citep{malinin2018predictive,durasov2024zigzag}. Furthermore, we report the False Positive Rate at 95\% Recall (FPR95, $\downarrow$). FPR95 measures the proportion of OOD instances that are incorrectly classified as in-distribution when the model correctly identifies 95\% of in-distribution samples.
\subsection{Underlying model architecture}
Given the relatively small size of the DAR and DIR datasets, we use a Kinetics-400 \citep{kay2017kinetics} self-supervised pre-trained Video Masked Autoencoder (VMAE, \citealt{tong2022videomae}). As a backbone, the VMAE uses a Vision Transformer (ViT) Base architecture \citep{arnab2021vivit} that consists of about 94 million parameters. The ViT-Base expects videos of 16 frames, has a depth of 12 blocks and a hidden latent dimension of 768. For the datasets with multiple video streams, we combine the latent modality representations via attentional feature fusion \citep{dai2021attentional}.
\subsection{Implementation details}
We fine-tune a separate ViT-Base model for 20 epochs per dataset using the AdamW optimizer \citep{loshchilov2018decoupled} with a weight decay of 0.05. We apply early stopping with a patience of 5 epochs and set the learning rate to $5\mathrm{e}{-5}$. We use an Ubuntu 20.04 server with two 16GB NVIDIA Tesla T4 GPUs. To simulate a larger batch size of 8, we use the gradient accumulation module from the Accelerate library (Gugger et al., \citeyear{accelerate}).
The LL--HMC experiments are run on a separate Ubuntu 20.04 server (CPU-only, 60GB RAM) using the NUTS sampler from Pyro \citep{bingham2018pyro}. We perform a grid search for 5 random seeds for different prior standard deviation (0.01, 0.1, 1, 2.5, 5, 10), number of burn-in samples (10, 25, 50, 100, 200), target acceptance probabilities (0.6, 0.7, 0.8), multiple chains (1, 2) and numbers of collected samples (2, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50) for the classification layer $\theta_{LL}$. When we use more than a single chain, we divide the number of collected samples evenly over the chains. For the LL--HMC with multiple starting positions (latent representations from different independently fine-tuned models), we use the top-performing ensemble models to produce the latent representations.
For the remaining last layer PDL (LL--PDL) methods, we also perform a grid search for the same random seeds to evaluate the number of samples/predictors/ensemble members (5, 10, 15, 20, 25, 30, 35, 40, 45, 50), batch size (16, 32, 64), learning rate ($1\mathrm{e}{-2}, 1\mathrm{e}{-3}, 1\mathrm{e}{-4}$) for different numbers of epochs (5, 10, 15, 20, 25). For VBLL \citep{harrison2024variational}, which does not produce multiple predictions, we evaluate modifying the KL regularization (1, 5, 10).
\subsection{Experiments}
\subsubsection{Video-based DAR and DIR performance}
To compare LL--HMC to other PDL methods, we use the fine-tuned ViT-Base models as a regular baseline. The baseline models also produce the latent representations on which the other LL--PDL methods rely. Additionally, we also evaluate an ensemble of 5 independently fine-tuned ViT-Base models. For the LL--PDL methods, we include the following approaches that do not require any form of post-hoc calibration: SE \citep{valdenegro2023sub}, DDU \citep{mukhoti2023deep}, BBB \citep{blundell2015weight}, PE \citep{laurent2023packed}, and VBLL \citep{harrison2024variational}.
\subsubsection{Uncertainty-based out-of-distribution detection}
OOD detection aims to identify inputs that do not belong to the distribution of the training data \citep{yang2024generalized}. To construct realistic OOD instances for DAR and DIR, we remove the most and least prevalent classes from each dataset to simulate OOD instances for common or rare maneuvers. The removed classes from both the train and test datasets serve as the unseen OOD instances. We fine-tune a ViT-Base model on each reduced training set to obtain latent representations to use for the PDL methods. For the ROAD dataset, removing the most occurring maneuver would leave a subset of fewer than 200 videos. Therefore, for the ROAD dataset, we only apply the OOD-min setting, where we remove the maneuvers for which we have less than 20 videos (\textit{Move Right, Move Left, and Overtake}).
\begin{table*}[t]
\centering
\caption{Average performance and two standard errors of the mean of the top-performing hyperparameter configurations per random seed for the AIDE, B4C and ROAD datasets. Each PDL method, except for the deep ensemble, uses the latent representations produced by the \textit{Regular} model. Acc=Accuracy, ACE=Adaptive Calibration Error with 10 bins, rAULC=relative Area under the lifted curve. } %Results are split into the average top (T) performance and the best average hyperparameter (HP) across 5 random seeds.
\vspace{-0.2cm}
\resizebox{\linewidth}{!}{
\begin{tabular}{l|c|c|c|c||c|c|c|c||c|c|c|c}
\toprule
& \multicolumn{4}{c||}{\textbf{AIDE} \citep{yang2023aide}} & \multicolumn{4}{c||}{\textbf{B4C} \citep{jain2015car}} & \multicolumn{4}{c}{\textbf{ROAD} \citep{singh2022road}} \\
& \multicolumn{1}{c|}{\textbf{Acc ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{F1 ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{ACE ($\downarrow$)}} & \multicolumn{1}{c||}{\textbf{rAULC ($\uparrow$)}}
& \multicolumn{1}{c|}{\textbf{Acc ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{F1 ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{ACE ($\downarrow$)}} & \multicolumn{1}{c||}{\textbf{rAULC ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{Acc ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{F1 ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{ACE ($\downarrow$)}} & \multicolumn{1}{c}{\textbf{rAULC ($\uparrow$)}} \\
% & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP}
% & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP} \\
\midrule
\textbf{Regular} & \multicolumn{1}{c|}{87.19} & \multicolumn{1}{c|}{81.88} & \multicolumn{1}{c|}{0.027} & \multicolumn{1}{c||}{0.77} & \multicolumn{1}{c|}{92.31} & \multicolumn{1}{c|}{93.71} & \multicolumn{1}{c|}{0.050} & \multicolumn{1}{c||}{0.53} & \multicolumn{1}{c|}{71.64} & \multicolumn{1}{c|}{45.36} & \multicolumn{1}{c|}{0.063} & \multicolumn{1}{c}{0.61} \\
\textbf{DE (N=5)} & \multicolumn{1}{c|}{89.66} & \multicolumn{1}{c|}{82.24} & \multicolumn{1}{c|}{0.019} & \multicolumn{1}{c||}{0.80} & \multicolumn{1}{c|}{90.88} & \multicolumn{1}{c|}{91.49} & \multicolumn{1}{c|}{0.059} & \multicolumn{1}{c||}{0.73} & \multicolumn{1}{c|}{74.24} & \multicolumn{1}{c|}{44.57} & \multicolumn{1}{c|}{0.066} & \multicolumn{1}{c}{0.70} \\
\textbf{DDU} & \multicolumn{1}{c|}{57.31} & \multicolumn{1}{c|}{19.12} & \multicolumn{1}{c|}{0.088} & \multicolumn{1}{c||}{0.85} & \multicolumn{1}{c|}{76.07} & \multicolumn{1}{c|}{74.72} & \multicolumn{1}{c|}{0.124} & \multicolumn{1}{c||}{0.61} & \multicolumn{1}{c|}{59.70} & \multicolumn{1}{c|}{23.85} & \multicolumn{1}{c|}{0.138} & \multicolumn{1}{c}{0.42} \\ \midrule
\textbf{BBB--LL} & 88.67$\pm$0.36 & 83.13$\pm$0.26 & 0.032$\pm$0.005 & 0.73$\pm$0.02& 93.16$\pm$0.94 &94.28$\pm$0.87 &0.025$\pm$0.007 & 0.65$\pm$0.08 & 71.64$\pm$3.13 & 56.14$\pm$4.37 & 0.068$\pm$0.003 & 0.50$\pm$0.13 \\
\textbf{PE--LL} & 88.64$\pm$0.32 &83.37$\pm$0.02 & 0.030$\pm$0.003 & 0.74$\pm$0.01 & 94.02$\pm$2.24 & 95.02$\pm$0.02 & 0.022$\pm$0.005 & 0.67$\pm$0.03 & 68.66$\pm$2.11 & 59.28$\pm$1.38 & 0.077$\pm$0.008 & 0.36$\pm$0.09 \\
\textbf{SE} & 88.31$\pm$0.19 & 82.69$\pm$0.21 & 0.029$\pm$0.000 & 0.75$\pm$0.01 & 93.16$\pm$0.18 & 94.38$\pm$0.04 & 0.023$\pm$0.001 & 0.72$\pm$0.00 & 68.06$\pm$2.23 & 45.63$\pm$2.45 & 0.065$\pm$0.002 & 0.57$\pm$0.01 \\
\textbf{VBLL} & 88.67$\pm$0.36 & 83.13$\pm$0.26 & 0.032$\pm$0.005 & 0.73$\pm$0.02 & 92.14$\pm$1.26 & 93.47$\pm$1.13 & 0.116$\pm$0.038 & 0.55$\pm$0.10 & 72.54$\pm$2.60 & 47.25$\pm$2.08 & 0.084$\pm$0.004 & 0.48$\pm$0.05 \\
\textbf{LL--HMC} & 89.00$\pm$0.16 & 83.87$\pm$0.08 & 0.029$\pm$0.002 & 0.68$\pm$0.06 & 94.02$\pm$0.42 & 95.00$\pm$0.29 & 0.027$\pm$0.007 & 0.66$\pm$0.10 & 70.15$\pm$5.05 & 54.18$\pm$4.43 & 0.073$\pm$0.014 & 0.41$\pm$0.12 \\
% \textbf{BBB--LL} & 88.47$\pm$0.41 & 82.49$\pm$0.41 & 0.030$\pm$0.001 &0.70$\pm$0.05 & 90.09$\pm$1.16 & 91.26$\pm$1.70 & 0.028$\pm$0.005 & 0.80$\pm$0.02 & 63.58$\pm$5.05 & 34.51$\pm$1.72 & 0.077$\pm$0.007 & 0.62$\pm$0.04 \\
% \textbf{PE--LL} & 87.85$\pm$0.63 & 80.61$\pm$2.12 &0.034$\pm$0.004 & 0.79$\pm$0.02 & 91.97$\pm$0.42 & 91.99$\pm$0.45 & 0.021$\pm$0.002 & 0.84$\pm$0.01 & 65.37$\pm$1.12 & 29.72$\pm$1.12 & 0.092$\pm$0.002 & 0.67$\pm$0.02
% \textbf{SE} & 88.11$\pm$0.13 & 81.61$\pm$0.11 & 0.028$\pm$0.001 & 0.77$\pm$0.01 & 92.14$\pm$0.34 & 92.21$\pm$0.26 & 0.022$\pm$0.002 & 0.82$\pm$0.00 & 65.97$\pm$0.60 & 29.45$\pm$0.32 & 0.103$\pm$0.002 & 0.67$\pm$0.02 \\
% \textbf{VBLL} & 87.95$\pm$0.17 & 78.62$\pm$3.17 & 0.034$\pm$0.003 & 0.78$\pm$0.02 & 86.32$\pm$3.01 & 84.92$\pm$3.17 & 0.081$\pm$0.007 & 0.80$\pm$0.07 66.27$\pm$1.52 &31.29$\pm$1.71 & 0.108$\pm$0.005 & 0.63$\pm$0.03 \\
% \textbf{LL--HMC} & 88.44$\pm$0.22 & 82.69$\pm$0.26 & 0.028$\pm$0.000 & 0.76$\pm$0.01 & 90.26$\pm$1.39 & 90.17$\pm$2.46 & 0.027$\pm$0.004 & 0.85$\pm$0.03 & 61.79$\pm$4.07 & 32.03$\pm$4.00& 0.083$\pm$0.01 & 0.63$\pm$0.15 \\
% \textbf{SE} & 88.11$\pm$0.13 & 81.61$\pm$0.11 & 0.028$\pm$0.001 & 0.77$\pm$0.01 & 65.97$\pm$0.60 & 29.45$\pm$0.32 & 0.065$\pm$0.002 & 0.103$\pm$0.002 & 0.57$\pm$0.01 & 0.67$\pm$0.02 \\
\bottomrule
\end{tabular}
}
\label{tab:performance}
\end{table*}
\begin{table*}[t!]
\centering
\caption{Average grid search results and two standard errors of the mean for the best average hyperparameter configuration across five random seeds. %
}
\vspace{-0.2cm}
\resizebox{\linewidth}{!}{
\begin{tabular}{l|c|c|c|c||c|c|c|c||c|c|c|c}
\toprule
& \multicolumn{4}{c||}{\textbf{AIDE} \citep{yang2023aide}} & \multicolumn{4}{c||}{\textbf{B4C} \citep{jain2015car}} & \multicolumn{4}{c}{\textbf{ROAD} \citep{singh2022road}} \\
& \multicolumn{1}{c|}{\textbf{Acc ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{F1 ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{ACE ($\downarrow$)}} & \multicolumn{1}{c||}{\textbf{rAULC ($\uparrow$)}}
& \multicolumn{1}{c|}{\textbf{Acc ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{F1 ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{ACE ($\downarrow$)}} & \multicolumn{1}{c||}{\textbf{rAULC ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{Acc ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{F1 ($\uparrow$)}} & \multicolumn{1}{c|}{\textbf{ACE ($\downarrow$)}} & \multicolumn{1}{c}{\textbf{rAULC ($\uparrow$)}} \\
% & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP}
% & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP} & \textbf{T} & \textbf{HP} \\
\midrule
% \textbf{Regular} & \multicolumn{1}{c|}{87.19} & \multicolumn{1}{c|}{81.88} & \multicolumn{1}{c|}{0.027} & \multicolumn{1}{c||}{0.77} & \multicolumn{1}{c|}{92.31} & \multicolumn{1}{c|}{93.71} & \multicolumn{1}{c|}{0.050} & \multicolumn{1}{c||}{0.53} & \multicolumn{1}{c|}{71.64} & \multicolumn{1}{c|}{45.36} & \multicolumn{1}{c|}{0.063} & \multicolumn{1}{c}{0.61} \\
% \textbf{DE (N=5)} & \multicolumn{1}{c|}{89.66} & \multicolumn{1}{c|}{82.24} & \multicolumn{1}{c|}{0.019} & \multicolumn{1}{c||}{0.80} & \multicolumn{1}{c|}{90.88} & \multicolumn{1}{c|}{91.49} & \multicolumn{1}{c|}{0.059} & \multicolumn{1}{c||}{0.73} & \multicolumn{1}{c|}{74.24} & \multicolumn{1}{c|}{44.57} & \multicolumn{1}{c|}{0.066} & \multicolumn{1}{c}{0.70} \\
% \textbf{DDU} & \multicolumn{1}{c|}{57.31} & \multicolumn{1}{c|}{19.12} & \multicolumn{1}{c|}{0.088} & \multicolumn{1}{c||}{0.85} & \multicolumn{1}{c|}{76.07} & \multicolumn{1}{c|}{74.72} & \multicolumn{1}{c|}{0.124} & \multicolumn{1}{c||}{0.61} & \multicolumn{1}{c|}{59.70} & \multicolumn{1}{c|}{23.85} & \multicolumn{1}{c|}{0.138} & \multicolumn{1}{c}{0.42} \\ \midrule
% \textbf{BBB--LL} & 88.67$\pm$0.36 & 83.13$\pm$0.26 & 0.032$\pm$0.005 & 0.73$\pm$0.02& 93.16$\pm$0.94 &94.28$\pm$0.87 &0.025$\pm$0.007 & 0.65$\pm$0.08 & 71.64$\pm$3.13 & 56.14$\pm$4.37 & 0.068$\pm$0.003 & 0.50$\pm$0.13 \\
% \textbf{PE--LL} & 88.64$\pm$0.32 &83.37$\pm$0.02 & 0.030$\pm$0.003 & 0.74$\pm$0.01 & 94.02$\pm$2.24 & 95.02$\pm$0.02 & 0.022$\pm$0.005 & 0.67$\pm$0.03 & 68.66$\pm$2.11 & 59.28$\pm$1.38 & 0.077$\pm$0.008 & 0.36$\pm$0.09 \\
% \textbf{SE} & 88.31$\pm$0.19 & 82.69$\pm$0.21 & 0.029$\pm$0.000 & 0.75$\pm$0.01 & 93.16$\pm$0.18 & 94.38$\pm$0.04 &0.023$\pm$0.001 & 0.72$\pm$0.00 & 68.06$\pm$2.23 & 45.63$\pm$2.45 & 0.065$\pm$0.002 & 0.57$\pm$0.01 \\
% \textbf{VBLL} & 88.67$\pm$0.36 & 83.13$\pm$0.26 & 0.032$\pm$0.005 & 0.73$\pm$0.02 & 92.14$\pm$1.26 & 93.47$\pm$1.13 & 0.116$\pm$0.038 & 0.55$\pm$0.10 & 72.54$\pm$2.60 & 47.25$\pm$2.08 & 0.084$\pm$0.004 & 0.48$\pm$0.05 \\
% \textbf{LL--HMC} & 89.00$\pm$0.16 & 83.87$\pm$0.08 & 0.029$\pm$0.002 & 0.68$\pm$0.06 & 94.02$\pm$0.42 & 95.00$\pm$0.29 & 0.027$\pm$0.007 & 0.66$\pm$0.10 & 70.15$\pm$5.05 & 54.18$\pm$4.43 & 0.073$\pm$0.014 & 0.41$\pm$0.12 \\
\textbf{BBB--LL} & 88.47$\pm$0.41 & 82.49$\pm$0.41 & 0.030$\pm$0.001 &0.70$\pm$0.05 & 90.09$\pm$1.16 & 91.26$\pm$1.70 & 0.028$\pm$0.005 & 0.80$\pm$0.02 & 63.58$\pm$5.05 & 34.51$\pm$1.72 & 0.077$\pm$0.007 & 0.62$\pm$0.04 \\
\textbf{PE--LL} & 87.85$\pm$0.63 & 80.61$\pm$2.12 &0.034$\pm$0.004 & 0.79$\pm$0.02 & 91.97$\pm$0.42 & 91.99$\pm$0.45 & 0.021$\pm$0.002 & 0.84$\pm$0.01 & 65.37$\pm$1.12 & 29.72$\pm$1.12 & 0.092$\pm$0.002 & 0.67$\pm$0.02 \\
\textbf{SE} & 88.11$\pm$0.13 & 81.61$\pm$0.11 & 0.028$\pm$0.001 & 0.77$\pm$0.01 & 92.14$\pm$0.34 & 92.21$\pm$0.26 & 0.022$\pm$0.002 & 0.82$\pm$0.00 & 65.97$\pm$0.60 & 29.45$\pm$0.32 & 0.103$\pm$0.002 & 0.67$\pm$0.02 \\
\textbf{VBLL} & 87.95$\pm$0.17 & 78.62$\pm$3.17 & 0.034$\pm$0.003 & 0.78$\pm$0.02 & 86.32$\pm$3.01 & 84.92$\pm$3.17 & 0.081$\pm$0.007 & 0.80$\pm$0.07 & 66.27$\pm$1.52 & 31.29$\pm$1.71 & 0.108$\pm$0.005 & 0.63$\pm$0.03 \\
\textbf{LL--HMC} & 88.44$\pm$0.22 & 82.69$\pm$0.26 & 0.028$\pm$0.000 & 0.76$\pm$0.01 & 90.26$\pm$1.39 & 90.17$\pm$2.46 & 0.027$\pm$0.004 & 0.85$\pm$0.03 & 61.79$\pm$4.07 & 32.03$\pm$4.00 & 0.083$\pm$0.010 & 0.63$\pm$0.15 \\
% \textbf{SE} & 88.11$\pm$0.13 & 81.61$\pm$0.11 & 0.028$\pm$0.001 & 0.77$\pm$0.01 & 65.97$\pm$0.60 & 29.45$\pm$0.32 & 0.065$\pm$0.002 & 0.103$\pm$0.002 & 0.57$\pm$0.01 & 0.67$\pm$0.02 \\
\bottomrule
\end{tabular}
}
\label{tab:performance_avghp}
\end{table*}
\section{Results}
To avoid reliance on a single random initialization \citep{bouthillier2019unreproducible}, we examine classification and OOD detection performance differences across multiple random seeds. For both the in-distribution and OOD experiments, we first analyze the grid search results across five random seeds. We present the top performance averaged across random seeds to illustrate classification and OOD detection results for a single grid search. Next, we report the performance of the best average hyperparameter configuration across the random seeds. To better understand the differences between the included LL--PDL methods, as well as the effects of additional predictions or sampled last-layer parameters, we use the best average hyperparameter configuration and conduct further evaluations with more random seeds. For LL--HMC, we also analyze the influence of the prior, additional chains, additional starting positions, parameter trajectories, and MCMC diagnostics.
\subsection{In-distribution action and intention recognition performance} %#driver action and intention recognition
\paragraph{\textbf{Grid search results.}} Table \ref{tab:performance} reports the average performance of the top hyperparameter configuration per random seed. Table \ref{tab:performance_avghp} shows the performance of the single hyperparameter configuration that achieved the best average across the random seeds. The regular DNN serves as the baseline, and the DE and DDU were only evaluated once per dataset. For the DDU, the lower performance is most likely due to the absence of spectral normalization. The variability in DE performance may be attributed to the small ensemble size (N=5), which limits its robustness. Given the computational cost of adding more members, DE may not be practical for resource-constrained operating environments.
% To assess the stability of the hyperparameter selection, we conducted a grid search across five random seeds.
The top-performing LL--HMC configurations in Table \ref{tab:performance} consistently achieve performance comparable to the best-performing PDL methods. From a confidence-performance calibration perspective (ACE), the BBB--LL, PE--LL, SE and LL--HMC yield a lower ACE for the B4C and ROAD datasets compared to the regular model, but not for the AIDE dataset. Additionally, no single method optimally balances both classification performance and calibration across all datasets. The regular softmax-based model is better calibrated than the LL--HMC approach, except for the B4C dataset. Additionally, we do not observe that the higher classification performance of LL-HMC in Table \ref{tab:performance} also results in an improved rAULC (the ability to distinguish between correctly and incorrectly predicted in-distribution instances based on the uncertainty estimations).
While Table \ref{tab:performance} presents the potential top-performance we could observe when we would perform a single grid search, Table \ref{tab:performance_avghp} reveals that using the average hyperparameter configuration across seeds leads to significantly lower average in-distribution classification performance. More specifically, the F1-scores dropped on average with 2.1, 4.3 and 21.1 for the AIDE, B4C and ROAD datasets respectively. The larger performance drop on the ROAD dataset suggests that LL--PDL methods may be more sensitive to observation noise and initialization instability, particularly in smaller or noisier datasets.
\begin{figure*}[!t]
\centering
\begin{tikzpicture}
\begin{groupplot}[
group style={
group size=3 by 1,
horizontal sep=1.0cm,
vertical sep=1.0cm,
},
width=5.5cm,
height=4.0cm,
grid=major,
xlabel={\tiny{Number of samples/predictions}},
xtick={1,5,10,15,20,25,30,35,40,45,50},
tick label style={font=\tiny},
label style={font=\small},
y tick label style={
/pgf/number format/fixed,
/pgf/number format/precision=0,
/pgf/number format/fixed zerofill=true,
},
tick label style={font=\fontsize{4}{4}\selectfont},
%ylabel style={yshift=-0.2cm}, % Shift y-label down
xlabel style={yshift=0.1cm}, % Shift x-label down
]
% First plot (Dataset A)
\nextgroupplot[
title={\tiny {\textbf{AIDE} \citep{yang2023aide}}},
title style={yshift=-0.2cm},
ylabel={\small{F1 (\%)}},
ymin=79.5,
ymax=84
]
% Plot data for BBB in Dataset A
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=orange}] coordinates {
(5,80.95996846938037) += (0,1.0157) -=(0,1.0157)
(10,80.38051179888186) += (0,0.7448) -=(0,0.7448)
(15,81.2522205585229) += (0,0.4554) -=(0,0.4554)
(20,81.76684349204947) += (0,0.4032) -=(0,0.4032)
(25,81.27572973546255) += (0,0.2846) -=(0,0.2846)
(30,81.20717689275352) += (0,0.6158) -=(0,0.6158)
(35,81.31343389831275) += (0,0.5597) -=(0,0.5597)
(40,81.08118064051627) += (0,0.6506) -=(0,0.6506)
(45,80.94812310733515) += (0,0.6945) -=(0,0.6945)
(50,81.83969173721589) += (0,0.3263) -=(0,0.3263)
};
% Plot data for PE in Dataset B
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=cyan}] coordinates {
(5,80.64427423685513) += (0,1.0801) -=(0,1.0801)
(10,81.39108033133715) += (0,0.5604) -=(0,0.5604)
(15,81.37295306180285) += (0,0.5042) -=(0,0.5042)
(20,80.82737023929269) += (0,1.1425) -=(0,1.1425)
(25,80.91886472940277) += (0,0.6976) -=(0,0.6976)
(30,81.17688066908337) += (0,0.4442) -=(0,0.4442)
(35,81.49046003528815) += (0,0.3197) -=(0,0.3197)
(40,81.41813206619261) += (0,0.6226) -=(0,0.6226)
(45,81.22341562264936) += (0,0.3816) -=(0,0.3816)
(50,81.064988976834) += (0,0.6010) -=(0,0.6010)
};
% Plot data for SE in Dataset A
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=red}] coordinates {
(5,81.72796641439183) += (0,0.2508) -=(0,0.2508)
(10,81.90801885064703) += (0,0.1565) -=(0,0.1565)
(15,81.78739214969264) += (0,0.1620) -=(0,0.1620)
(20,81.87270775771678) += (0,0.1763) -=(0,0.1763)
(25,81.74547366757284) += (0,0.2244) -=(0,0.2244)
(30,81.76892921540839) += (0,0.1918) -=(0,0.1918)
(35,81.71655273815813) += (0,0.2177) -=(0,0.2177)
(40,81.7176956093715) += (0,0.1000) -=(0,0.1000)
(45,81.71353922074951) += (0,0.2220) -=(0,0.2220)
(50,81.60908470361797) += (0,0.1585) -=(0,0.1585)
};
% % Plot data for LL--HMC in Dataset A
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=yellow}] coordinates {
(5,82.24926660396423) += (0,0.6520) -=(0,0.6520)
(10,82.19827942316209) += (0,0.2412) -=(0,0.2412)
(15,82.42064618224457) += (0,0.4252) -=(0,0.4252)
(20,82.68614557647774) += (0,0.2421) -=(0,0.2421)
(25,82.34071008535416) += (0,0.3180) -=(0,0.3180)
(30,82.3229477909996) += (0,0.6125) -=(0,0.6125)
(35,82.91580828836082) += (0,0.1741) -=(0,0.1741)
(40,82.99625162343334) += (0,0.3213) -=(0,0.3213)
(45,82.68709960581347) += (0,0.2738) -=(0,0.2738)
(50,82.23586915620874) += (0,0.3072) -=(0,0.3072)
};
% Single marker for DE at 5 samples
\addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
(5,82.24)
};
% Single marker for Regular Model at 1 sample
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
(1,81.88)
};
% % Single marker for VBLL at 1 sample
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt] coordinates {
% (1,83.00)
% };
% Single marker for VBLL at 1 sample
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=green}] coordinates {
% (1,94.93)
(1,80.478158) += (0,0.8418) -=(0,0.8418)
};
% Second plot (Dataset B)
\nextgroupplot[
title={\tiny {\textbf{B4C} \citep{jain2015car}}},
title style={yshift=-0.2cm},
ylabel={\small{F1 (\%)}},
ymax=95,
ymin=89.5,
]
% Plot data for BBB in Dataset B
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=orange}] coordinates {
(5,91.81428553594436) += (0,1.4334) -=(0,1.4334)
(10,92.08634408387084) += (0,1.2917) -=(0,1.2917)
(15,91.88171836718176) += (0,1.1540) -=(0,1.1540)
(20,92.02217647893363) += (0,1.1959) -=(0,1.1959)
(25,92.4902186633233) += (0,1.3156) -=(0,1.3156)
(30,92.3112903345926) += (0,1.0125) -=(0,1.0125)
(35,90.85696435247613) += (0,0.7344) -=(0,0.7344)
(40,91.84051123373924) += (0,0.8450) -=(0,0.8450)
(45,91.99210892719971) += (0,0.8574) -=(0,0.8574)
(50,92.27768581229043) += (0,1.0191) -=(0,1.0191)
};
% Plot data for PE in Dataset B
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=cyan}] coordinates {
(5,92.7037411936547) += (0,1.2931) -=(0,1.2931)
(10,93.11025366083305) += (0,0.9090) -=(0,0.9090)
(15,93.13511115961856) += (0,1.0287) -=(0,1.0287)
(20,93.30473881052755) += (0,1.2327) -=(0,1.2327)
(25,92.68852537845807) += (0,1.2925) -=(0,1.2925)
(30,93.23527829587042) += (0,0.8315) -=(0,0.8315)
(35,92.33493213991402) += (0,1.0786) -=(0,1.0786)
(40,92.63245413657515) += (0,0.9345) -=(0,0.9345)
(45,94.10309815824014) += (0,0.4679) -=(0,0.4679)
(50,93.05837606672831) += (0,1.1152) -=(0,1.1152)
};
% Plot data for SE in Dataset B
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=red}] coordinates {
(5.0,93.9590185161521) += (0,0.5442) -=(0,0.5442)
(10.0,94.18708528392156) += (0,0.1981) -=(0,0.1981)
(15.0,94.12224595632553) += (0,0.2117) -=(0,0.2117)
(20.0,94.2519246115176) += (0,0.1729) -=(0,0.1729)
(25.0,94.12224595632551) += (0,0.2117) -=(0,0.2117)
(30.0,94.2519246115176) += (0,0.1729) -=(0,0.1729)
(35.0,94.2519246115176) += (0,0.1729) -=(0,0.1729)
(40.0,94.2519246115176) += (0,0.1729) -=(0,0.1729)
(45.0,94.31676393911363) += (0,0.1297) -=(0,0.1297)
(50.0,94.31676393911363) += (0,0.1297) -=(0,0.1297)
};
% Plot data for LL--HMC in Dataset B
\addplot[mark=diamond*, thin, line width=0.25pt,
mark options={fill=yellow},
mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=yellow}] coordinates {
(5,90.99785760859635) += (0,0.6351) -=(0,0.6351)
(10,92.25213285534451) += (0,0.8798) -=(0,0.8798)
(15,92.3731837998395) += (0,0.9760) -=(0,0.9760)
(20,92.71090059658137) += (0,0.7351) -=(0,0.7351)
(25,93.1521381369241) += (0,0.7345) -=(0,0.7345)
(30,92.67648931096207) += (0,1.0303) -=(0,1.0303)
(35,92.33309758977552) += (0,1.0145) -=(0,1.0145)
(40,93.25363425383188) += (0,0.6015) -=(0,0.6015)
(45,93.43567253225955) += (0,0.6504) -=(0,0.6504)
(50,93.64543100334956) += (0,0.8540) -=(0,0.8540)
};
% Single marker for DE at 5 samples
\addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
(5,91.49)
};
% Single marker for Regular Model at 1 sample
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
(1,93.71)
};
% Single marker for VBLL at 1 sample
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=green}] coordinates {
% (1,94.93)
(1,90.7315) += (0,0.9509) -=(0,0.9509)
};
% Third plot (Dataset C)
\nextgroupplot[
title={\tiny {\textbf{ROAD} \citep{singh2022road}}},
title style={yshift=-0.2cm},
ylabel={\small{F1 (\%)}},
ymin=30,
ymax=51
]
% Plot data for BBB in Dataset C
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=orange}] coordinates {
(5,40.03344690170473) += (0,3.2007) -=(0,3.2007)
(10,39.497117850555775) += (0,4.6328) -=(0,4.6328)
(15,41.70045043815289) += (0,2.8485) -=(0,2.8485)
(20,43.95679573963744) += (0,3.6190) -=(0,3.6190)
(25,43.91655890502581) += (0,4.6789) -=(0,4.6789)
(30,40.84849670921552) += (0,4.2924) -=(0,4.2924)
(35,43.46600586280709) += (0,3.1102) -=(0,3.1102)
(40,39.33604574991489) += (0,3.5332) -=(0,3.5332)
(45,41.67154706867009) += (0,3.0697) -=(0,3.0697)
(50,41.170672005887695) += (0,3.4761) -=(0,3.4761)
};
% Plot data for PE in Dataset C
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=cyan}] coordinates {
(5,46.95914301867057) += (0,3.8279) -=(0,3.8279)
(10,44.35103292152759) += (0,1.8950) -=(0,1.8950)
(15,44.21816213450458) += (0,4.0279) -=(0,4.0279)
(20,46.27727705113644) += (0,5.4009) -=(0,5.4009)
(25,46.26570675100138) += (0,4.1432) -=(0,4.1432)
(30,44.06082062870729) += (0,3.2937) -=(0,3.2937)
(35,43.00217105321485) += (0,2.9669) -=(0,2.9669)
(40,44.123030934571204) += (0,1.7077) -=(0,1.7077)
(45,46.803628670707965) += (0,2.3713) -=(0,2.3713)
(50,43.03308469196917) += (0,2.7090) -=(0,2.7090)
};
% Plot data for SE in Dataset C
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=red}] coordinates {
(5,40.73743947791) += (0,2.5367) -=(0,2.5367)
(10,40.056428478757425) += (0,2.6397) -=(0,2.6397)
(15,39.41337466881583) += (0,1.9312) -=(0,1.9312)
(20,37.86591958163472) += (0,0.1186) -=(0,0.1186)
(25,38.723007197626444) += (0,1.4574) -=(0,1.4574)
(30,39.378180178826824) += (0,1.7709) -=(0,1.7709)
(35,38.776521032744974) += (0,1.4436) -=(0,1.4436)
(40,38.91944120501935) += (0,1.9113) -=(0,1.9113)
(45,38.03843460972704) += (0,0.0723) -=(0,0.0723)
(50,38.02956311592404) += (0,0.0777) -=(0,0.0777)
};
% Plot data for LL--HMC in Dataset C
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt, error bars/.cd,
y dir=both, y explicit,
error bar style={line width=0.1pt, color=yellow}] coordinates {
(5,33.887826567916576) += (0,3.8856) -=(0,3.8856)
(10,34.574903713756974) += (0,5.5229) -=(0,5.5229)
(15,39.24376817031906) += (0,2.6870) -=(0,2.6870)
(20,39.30141891898767) += (0,4.2289) -=(0,4.2289)
(25,35.09617883452023) += (0,4.1437) -=(0,4.1437)
(30,37.58546499845445) += (0,4.5910) -=(0,4.5910)
(35,35.363634910564876) += (0,4.3608) -=(0,4.3608)
(40,42.42697302678621) += (0,5.4215) -=(0,5.4215)
(45,32.4746292995812) += (0,3.4674) -=(0,3.4674)
(50,36.93272588550002) += (0,4.5218) -=(0,4.5218)
};
% Single marker for DE at 5 samples
\addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
(5,44.57)
};
% Single marker for Regular Model at 1 sample
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
(1,45.36)
};
% Single marker for VBLL at 1 sample
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=green}] coordinates {
% (1,49.34)
(1,30.63016) += (0,1.4655) -=(0,1.4655)
};
\end{groupplot}
\end{tikzpicture}
% Legend
\begin{tikzpicture}
\vspace{0.3cm} %
% \hspace{3.5cm}
\begin{axis}[
hide axis,
xmin=0, xmax=1, ymin=0, ymax=1,
legend style={at={(0.0,0.0)}, anchor=center, legend columns=7},
legend entries={\tiny{BBB--LL},\tiny{PE--LL},\tiny{SE},\tiny{VBLL}, \tiny{LL--HMC},\tiny{DE},\tiny{Regular},}
]
\addlegendimage{mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=2.0pt}
\addlegendimage{mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2.0pt}
\addlegendimage{mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=2pt}
\addlegendimage{mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=2.0pt}
\addlegendimage{mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=2.0pt}
\addlegendimage{mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=2.0pt}
\addlegendimage{mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=2.0pt}
\end{axis}
\end{tikzpicture}
\caption{Influence of the number of samples or predictions for independent random seeds for the average best-performing hyperparameter configurations of the included PDL methods. The regular model (single prediction), DE (N=5), and VBLL (single prediction) are only included once in each figure. For each number of samples, the whiskers indicate two standard errors of the mean for 10 runs for different random seeds for every number of samples to avoid dependencies.}
\label{fig:method_comparison}
\end{figure*}
% \section{Prior scale influence.}
\begin{figure}[t!]
\centering
\begin{tikzpicture}
\begin{groupplot}[
group style={
group size=3 by 1, % Three figures in one row
horizontal sep=1.2cm,
vertical sep=1.0cm,
},
width=5.5cm,
height=4.0cm,
ybar,
enlargelimits=0.15,
ylabel={\tiny {F1 (\%)}},
symbolic x coords={BBB--LL, SE, PE--LL, VBLL, LL--HMC},
xtick=data,
ymin=0, ymax=100,
% ytick distance=10,
grid=major,
ymajorgrids=true,
tick label style={font=\tiny},
xticklabel style={rotate=45, anchor=east, yshift=-0.2cm},
every node near coord/.append style={
yshift=3pt,
font=\footnotesize,
label style={font=\small},
y tick label style={
/pgf/number format/fixed,
/pgf/number format/precision=0,
/pgf/number format/fixed zerofill=true,
},
tick label style={font=\fontsize{4}{4}\selectfont},
%ylabel style={yshift=-0.2cm}, % Shift y-label down
xlabel style={yshift=0.1cm}, % Shift x-label down
},
]
% First subplot
\nextgroupplot[title={ \tiny \textbf{AIDE} \citep{yang2023aide}},
title style={yshift=-0.2cm},
ylabel style={yshift=-0.2cm},
ymin=77, ymax=83,]
\addplot[
fill=steelblue31119180!60,
draw=steelblue31119180,
error bars/.cd,
y dir=both,
y explicit,
] coordinates {
(SE, 82.1926) +- (0.06622,0.06622)
(BBB--LL, 80.72) +- (0.29784,0.29784)
(PE--LL, 80.91) +- (0.23318,0.23318)
(VBLL, 77.63) +- (0.67146,0.67146)
(LL--HMC, 82.46) +- (0.12536,0.12536)
% (PE--LL, 81.7069) +- (0.20446,0.20446)
% (BBB--LL, 81.594) +- (0.16616,0.16616)
% (VBLL, 79.0825) +- (0.58232,0.58232)
% (LL--HMC, 82.22345) +- (0.17426,0.17426)
% std
% (SE, 82.1926) +-(0.3311,0.3311)
% (PE--LL, 81.7069) +-(1.0223,1.0223)
% (BBB--LL, 81.594) +-(0.8308,0.8308)
% (VBLL, 79.0825) +-(2.9116,2.9116)
% (LL--HMC, 82.22345) +-(0.8713,0.8713)
};
% Second subplot
\nextgroupplot[title={ \tiny \textbf{B4C} \citep{jain2015car}},
title style={yshift=-0.2cm},
ylabel style={yshift=-0.2cm},
ymin=86, ymax=95,]
\addplot[
fill=steelblue31119180!60,
draw=steelblue31119180,
error bars/.cd,
y dir=both,
y explicit,
] coordinates {
% (BBB--LL, 91.5455) +- (0.36998,0.36998)
(SE, 94.3816) +- (0.04,0.04)
(BBB--LL, 91.79) +- (0.41068,0.41068)
(PE--LL, 92.13) +- (0.19792,0.19792)
(VBLL, 87.26) +- (0.92628,0.92628)
(LL--HMC, 90.38) +- (0.70676,0.70676)
% (PE--LL, 93.6392) +- (0.22018,0.22018)
% (VBLL, 89.4546) +- (0.63644,0.63644)
% (LL--HMC, 92.1997) +- (0.63908,0.63908)
% std
% (BBB--LL, 91.5455) +-(1.8499,1.8499)
% (SE, 94.3816) +-(0.2,0.2)
% (PE--LL, 93.6392) +-(1.1009,1.1009)
% (VBLL, 89.4546) +-(3.1822,3.1822)
% (LL--HMC, 92.1997) +-(3.1954,3.1954)
};
\nextgroupplot[title={ \tiny \textbf{ROAD} \citep{singh2022road}},
title style={yshift=-0.2cm}, ymin=29, ymax=45, ylabel style={yshift=-0.2cm},]
\addplot[
fill=steelblue31119180!60,
draw=steelblue31119180,
error bars/.cd,
y dir=both,
y explicit,
] coordinates {
% (BBB--LL, 44.6598) +- (1.01784,1.01784)
(SE, 39.379) +- (0.58896,0.58896)
% (PE--LL, 44.2878) +- (0.91858,0.91858)
% (VBLL, 34.0378) +- (1.0252,1.0252)
% (LL--HMC, 38.3604) +- (0.95442,0.95442)
(BBB--LL, 35.98) +- (0.94728,0.94728)
(PE--LL,35.01) +- (0.38868,0.38868)
(VBLL, 31.31) +- (0.62208,0.62208)
(LL--HMC, 37.5) +- (1.19686,1.19686)
% std
% (BBB--LL, 44.6598) +-(5.0892,5.0892)
% (SE, 39.379) +-(2.9448,2.9448)
% (PE--LL, 44.2878) +-(4.5929,4.5929)
% (VBLL, 34.0378) +-(5.126,5.126)
% (LL--HMC, 38.3604) +-(4.7721,4.7721)
};
\end{groupplot}
\end{tikzpicture}
\caption{Average F1-scores and two standard errors of the mean across the included LL--PDL methods for the best performing grid search hyperparameter configurations across 100 random seeds.}
%\vspace{-0.2cm}
\label{fig:id_performance}
\end{figure}
\paragraph{\textbf{Influence of additional predictions or samples.}}
For the ViT-Base architecture, processing a single 16-frame video requires approximately $361 \times 10^9$ floating point operations (FLOPs). The majority of the FLOPs are required to produce the latent representation on which the LL--PDL methods rely. Every additional last layer sample or prediction requires $7.68$ or $9.22 \times 10^3$ FLOPs for a classifier with 5 or 6 classes respectively. Considering the $361\times 10^9$ FLOPs that are required to produce the latent representation $\mathbf{z}$, the computational impact of the additional predictions are relatively negligible. To examine the influence of additional predictions or sampled last layers, we evaluated the top configurations from Table \ref{tab:performance_avghp} for 10 different random seeds for each number of samples. Figure \ref{fig:method_comparison} illustrates the average performance and two standard errors of the mean for each number of samples. For the LL--PDL methods, no clear trend indicates that additional samples (or predictions) consistently improves the performance. Additionally, we observe that for each dataset another LL--PDL method performs best on average.
\paragraph{\textbf{Comparing the best hyperparameter configurations. }}
To evaluate whether any of the included LL--PDL methods consistently outperforms others, we tested the average best performing hyperparameter configuration from Table \ref{tab:performance_avghp} across 100 different random seeds. Figure \ref{fig:id_performance} illustrates the average DAR and DIR classification performance for each dataset and method. Consistent with the results in Figure \ref{fig:method_comparison}, no method consistently achieves the best average performance across the datasets. VBLL exhibits higher standard errors across datasets, suggesting greater sensitivity to random seed variations. As also observed in Table \ref{tab:performance_avghp}, the performance degradation from top-performing configurations is more pronounced in the smaller ROAD dataset. %, likely due to lower robustness to the random initialization.
\paragraph{\textbf{Prior scale influence.}}
To investigate the impact of prior scale on LL--HMC, Figure \ref{fig:hmc_perf_samples} shows the relationship between F1-score and the number of HMC samples, for multiple prior standard deviations. Because the last layer is randomly initialized (cold start) with uninformative priors, smaller prior scales (i.e., stricter priors) tend to degrade performance. For the AIDE and B4C datasets, performance plateaus after 5–10 samples, suggesting that further samples yield negligible performance gains. This implies that LL--HMC achieves stable predictive performance with relatively few samples, which is beneficial for computational efficiency. The ROAD dataset exhibits greater sensitivity to prior scale and the number of samples, with more variation in F1-score across configurations.
\begin{figure*}[t]
\centering
\begin{tikzpicture}
\begin{groupplot}[
group style={
group size=3 by 1,
horizontal sep=1.2cm,
vertical sep=1.5cm,
},
width=5.5cm,
height=4.0cm,
grid=major,
xlabel={\tiny{Number of samples}},
% xmode=log,
xtick={2,5,10,15,20,25,30,35,40,45,50},
% xticklabels={0.03,0.1,0.3,1,3,10},
tick label style={font=\tiny},
label style={font=\small},
y tick label style={
/pgf/number format/fixed,
/pgf/number format/precision=0,
/pgf/number format/fixed zerofill=true,
},
tick label style={font=\fontsize{4}{4}\selectfont},
%ylabel style={yshift=-0.2cm}, % Shift y-label down
xlabel style={yshift=0.1cm}, % Shift x-label down
]
% Third plot (Accuracy for AIDE Dataset)
\nextgroupplot[
title={\tiny {\textbf{AIDE} \citep{yang2023aide}}},
ylabel={\tiny{F1 (\%)}},
title style={yshift=-0.2cm},
% ytick={50,70,90},
ymin=0,
ymax=100 % Adjusted for AIDE dataset accuracy values
]
% Plot data for each prior value in AIDE dataset
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt] coordinates {
(2,14.0021344717182)
(5,14.017094017094)
(10,14.0021344717182)
(15,14.0021344717182)
(20,14.0021344717182)
(25,14.0021344717182)
(30,14.0021344717182)
(35,14.0021344717182)
(40,14.0021344717182)
(45,14.0021344717182)
(50,14.0021344717182)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1.0pt] coordinates {
(2,44.0785617720902)
(5,45.5579271170791)
(10,44.619027696427)
(15,45.6406287596942)
(20,45.6524267426279)
(25,46.6828965371792)
(30,45.1265108186647)
(35,45.294367914765)
(40,45.294367914765)
(45,45.294367914765)
(50,45.294367914765)
};
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1pt] coordinates {
(2,81.1265996341357)
(5,81.4880528149863)
(10,80.6886033388472)
(15,80.8240769818706)
(20,80.8240769818706)
(25,81.7497542395929)
(30,81.147775318895)
(35,80.6886033388472)
(40,81.2682541885595)
(45,81.2682541885595)
(50,80.8240769818706)
};
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.0pt] coordinates {
(2,81.9988673273403)
(5,81.0725353796292)
(10,83.2210460012779)
(15,82.8653507807344)
(20,82.6611086890146)
(25,82.5995157384987)
(30,82.2820554210384)
(35,82.932999137567)
(40,83.5649615920802)
(45,82.3797680158336)
(50,82.8254570288468)
};
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.0pt] coordinates {
(2,82.5713665407654)
(5,81.3040212030037)
(10,82.3251434729353)
(15,82.3119087170357)
(20,82.6568972522956)
(25,83.0308431010185)
(30,82.5466618076418)
(35,82.2070251182043)
(40,83.3504293628523)
(45,82.5884467876544)
(50,82.7713852772888)
};
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=yellow,}, mark size=1.0pt] coordinates {
(2,80.7062045482558)
(5,81.8924768551738)
(10,81.6863324033022)
(15,81.7573352754438)
(20,82.0510107716231)
(25,81.8958601891935)
(30,82.1463683907081)
(35,81.4699359624406)
(40,81.894579329894)
(45,81.9912178082785)
(50,82.3647639021652)
};
\nextgroupplot[
title={\tiny {\textbf{B4C} \citep{jain2015car}}},
ylabel={\tiny{F1 (\%)}},
title style={yshift=-0.2cm},
% ytick={40,50,60},
ymax=100,
ymin=0,
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange, }, mark size=1.0pt] coordinates {
(2,11.0045871559633)
(2,11.0045871559633)
(5,6.80851063829787)
(10,11.6363636363636)
(15,11.6363636363636)
(20,16.1375661375661)
(25,12.1518987341772)
(30,11.6363636363636)
(35,11.6363636363636)
(40,11.7791411042944)
(45,11.6363636363636)
(50,11.6363636363636)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan, }, mark size=1.0pt] coordinates {
(2,11.6363636363636)
(5,11.6363636363636)
(10,11.6363636363636)
(15,11.6363636363636)
(20,11.6363636363636)
(25,11.6363636363636)
(30,11.6363636363636)
(35,11.6363636363636)
(40,11.6363636363636)
(45,11.6363636363636)
(50,11.6363636363636)
};
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red, }, mark size=1.0pt] coordinates {
(2,92.612915759661)
(5,89.2186741363212)
(10,90.3246376811594)
(15,90.2082047003198)
(20,90.2082047003198)
(25,90.3246376811594)
(30,90.3246376811594)
(35,90.3246376811594)
(40,90.2374331550802)
(45,90.3246376811594)
(50,91.4892504519538)
};
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green, }, mark size=1.0pt] coordinates {
(2,93.1248656780571)
(5,91.9724578203374)
(10,92.0444444444444)
(15,93.5018046485499)
(20,92.882146160962)
(25,91.991304347826)
(30,93.7147086031452)
(35,92.6623462346234)
(40,92.882146160962)
(45,92.0444444444444)
(50,94.3356005165734)
};
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=pink, }, mark size=1.0pt] coordinates {
(2,89.0888888888888)
(5,89.452427184466)
(10,91.0268584403197)
(15,93.546728151076)
(20,94.3356005165734)
(25,94.3356005165734)
(30,92.8344673265824)
(35,92.8777777777777)
(40,95.0027752081406)
(45,94.3816032667096)
(50,92.882146160962)
};
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=yellow, }, mark size=1.0pt] coordinates {
(2,85.2579288323969)
(5,90.2374331550802)
(10,91.7308282160585)
(15,95.0027752081406)
(20,93.7332099907493)
(25,90.8346405228758)
(30,94.3356005165734)
(35,92.8777777777777)
(40,92.7064935064935)
(45,92.8777777777777)
(50,90.7626538987688)
};
\nextgroupplot[
title={\tiny {\textbf{ROAD} \citep{singh2022road}}},
ylabel={\tiny{F1 (\%)}},
title style={yshift=-0.2cm},
% ytick={20,40,60,80},
ymin=0,
ymax=60
]
% Plot data for each prior value in AIDE dataset
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt] coordinates {
(2,8.83652430044182)
(5,4.64285714285714)
(10,13.5233444616074)
(15,8.83652430044182)
(20,20.1298701298701)
(25,19.5238095238095)
(30,8.83652430044182)
(35,8.83652430044182)
(40,8.83652430044182)
(45,8.83652430044182)
(50,10.969387755102)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1.0pt] coordinates {
(2,12.8320802005012)
(5,18.4417086672725)
(10,17.2532781228433)
(15,12.8320802005012)
(20,15.9392789373814)
(25,14.4756838905775)
(30,15.9392789373814)
(35,10.969387755102)
(40,14.4756838905775)
(45,17.2532781228433)
(50,14.4756838905775)
};
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1pt] coordinates {
(2,32.6147576147576)
(5,36.200555737169)
(10,38.2060502426635)
(15,32.9069049435182)
(20,32.4157002417872)
(25,31.4532453297524)
(30,36.4112285230297)
(35,32.7231121281464)
(40,34.1060337178349)
(45,31.4532453297524)
(50,31.3196033562166)
};
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.0pt] coordinates {
(2,32.5569358178053)
(5,41.3946922642574)
(10,39.881051011954)
(15,34.1855520116389)
(20,36.4604292421193)
(25,37.4110223799664)
(30,37.3990175780457)
(35,38.7693309616151)
(40,37.3990175780457)
(45,45.1818230079099)
(50,38.7693309616151)
};
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.0pt] coordinates {
(2,36.1268193195513)
(5,38.6976638921718)
(10,31.7531179138321)
(15,40.382975932848)
(20,41.1269889530759)
(25,37.3990175780457)
(30,40.0163452108532)
(35,36.6398733406406)
(40,38.494528246081)
(45,41.4699324545871)
(50,41.1269889530759)
};
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=yellow,}, mark size=1.0pt] coordinates {
(2,23.8758067329495)
(5,39.196334345588)
(10,47.3923943568408)
(15,37.8867588635307)
(20,45.2957704821058)
(25,40.3683120845592)
(30,39.7268204940839)
(35,38.1645285279289)
(40,39.2487545925163)
(45,39.8514719620189)
(50,39.5121435301809)
};
\end{groupplot}
\end{tikzpicture}
% Legend
\begin{tikzpicture}
\vspace{0.3cm} %
% \hspace{2.1cm}
\begin{axis}[
hide axis,
xmin=0, xmax=1, ymin=0, ymax=1,
legend style={at={(0.0,0.0)}, anchor=center, legend columns=7},
legend entries={\tiny{0.01},\tiny{0.1},\tiny{1},\tiny{2.5},\tiny{5.0},\tiny{10.0},\tiny{50}}
]
\addlegendimage{mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=2.0pt}
\addlegendimage{mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2.0pt}
\addlegendimage{mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=2pt}
\addlegendimage{mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=2.0pt}
\addlegendimage{mark=diamond*, thin,line width=0.25pt, mark options={fill=pink}, mark size=2.0pt}
\addlegendimage{mark=oplus*, thin,line width=0.25pt, mark options={fill=yellow}, mark size=2.0pt}
% \addlegendimage{mark=otimes*, thin, line width=0.25pt, mark options={fill=brown}, mark size=2.0pt}
\end{axis}
\end{tikzpicture}
\caption{Effect of different prior standard deviation values and the number of samples on the LL--HMC in-distribution performance with a target acceptance of 0.7 and 100 burn-in samples for a single random seed.}
%\vspace{-0.1cm}
\label{fig:hmc_perf_samples}
\end{figure*}
\definecolor{steelblue31119180}{RGB}{31,119,180} % Define a color for the bars
% \usepackage{pgfplots}
% \pgfplotsset{compat=1.18}
% \usepackage{pgfplotstable}
% \usepackage{caption}
% \usepackage{tikz}
% \usetikzlibrary{pgfplots.groupplots}
\definecolor{steelblue31119180}{RGB}{31,119,180}
\begin{figure}[t!]
\centering
\begin{tikzpicture}
\begin{groupplot}[
group style={
group size=3 by 1,
horizontal sep=1.2cm,
vertical sep=1.0cm,
},
width=5.5cm,
height=4.0cm,
grid=major,
xlabel={\tiny{Number of samples}},
xtick={1,5,10,15,20,25,30,35,40,45,50},
tick label style={font=\tiny},
label style={font=\small},
y tick label style={
/pgf/number format/fixed,
/pgf/number format/precision=0,
/pgf/number format/fixed zerofill=true,
},
tick label style={font=\fontsize{4}{4}\selectfont},
%ylabel style={yshift=-0.2cm}, % Shift y-label down
xlabel style={yshift=0.1cm}, % Shift x-label down
]
\nextgroupplot[
title={\tiny {\textbf{AIDE} \citep{yang2023aide}}},
ylabel={\tiny{F1 (\%)}},
title style={yshift=-0.2cm},
ymax=100,
ymin=0
]
\path [fill=steelblue31119180, fill opacity=0.2]
(axis cs:1,2.4074074074074)
--(axis cs:1,2.4074074074074)
--(axis cs:2,25.474620560273)
--(axis cs:3,36.3674667925979)
--(axis cs:4,42.51325638209)
--(axis cs:5,46.6554748348895)
--(axis cs:6,49.747791819033)
--(axis cs:7,52.3759854381015)
--(axis cs:8,54.4028424588609)
--(axis cs:9,55.9209589301261)
--(axis cs:10,57.2453624582765)
--(axis cs:11,58.5348946957579)
--(axis cs:12,59.560576578204)
--(axis cs:13,60.4014316685476)
--(axis cs:14,61.2272377480546)
--(axis cs:15,62.0077534585458)
--(axis cs:16,62.5394693599207)
--(axis cs:17,63.2542963942399)
--(axis cs:18,63.788592250409)
--(axis cs:19,64.2778232465613)
--(axis cs:20,64.7268921009604)
--(axis cs:21,64.9933289964728)
--(axis cs:22,65.6312297849877)
--(axis cs:23,66.0002645342677)
--(axis cs:24,66.4217424160643)
--(axis cs:25,66.6605724419808)
--(axis cs:26,66.9852816769292)
--(axis cs:27,67.3434718662061)
--(axis cs:28,67.7023653610644)
--(axis cs:29,67.8897825833578)
--(axis cs:30,68.3337000581644)
--(axis cs:31,68.6067687334276)
--(axis cs:32,68.8531898855806)
--(axis cs:33,69.0005682843235)
--(axis cs:34,69.2553123777573)
--(axis cs:35,69.3835844541345)
--(axis cs:36,69.5052932853387)
--(axis cs:37,69.6413262263697)
--(axis cs:38,69.9221789807699)
--(axis cs:39,70.1410243390448)
--(axis cs:40,70.300281447589)
--(axis cs:41,70.4300286526485)
--(axis cs:42,70.5788842775389)
--(axis cs:43,70.6350495479844)
--(axis cs:44,71.085236983809)
--(axis cs:45,71.0570364024075)
--(axis cs:46,71.2729827035557)
--(axis cs:47,71.3526935006528)
--(axis cs:48,71.3552715499531)
--(axis cs:49,71.4340747546225)
--(axis cs:50,71.4354491242607)
--(axis cs:50,94.1786508585519)
--(axis cs:50,94.1786508585519)
--(axis cs:49,94.4073920109395)
--(axis cs:48,94.5650672343655)
--(axis cs:47,94.8060944357168)
--(axis cs:46,94.9769607166573)
--(axis cs:45,95.0195435497572)
--(axis cs:44,95.3157190345896)
--(axis cs:43,95.1412055075163)
--(axis cs:42,95.3730772181638)
--(axis cs:41,95.5219328430542)
--(axis cs:40,95.7008795433275)
--(axis cs:39,95.8616812758951)
--(axis cs:38,95.9753495283022)
--(axis cs:37,96.040852880551)
--(axis cs:36,96.2665274286223)
--(axis cs:35,96.5212919940758)
--(axis cs:34,96.7851925727096)
--(axis cs:33,96.93919918045)
--(axis cs:32,97.2200153268321)
--(axis cs:31,97.4212690202365)
--(axis cs:30,97.6181646755939)
--(axis cs:29,97.6685872019126)
--(axis cs:28,98.004412171389)
--(axis cs:27,98.1960607900218)
--(axis cs:26,98.4209996436117)
--(axis cs:25,98.7148563481743)
--(axis cs:24,99.133124270034)
--(axis cs:23,99.4089550241137)
--(axis cs:22,99.7855739501644)
--(axis cs:21,99.9452464078881)
--(axis cs:20,100.540954174571)
--(axis cs:19,101.016770640648)
--(axis cs:18,101.527592003416)
--(axis cs:17,102.07948275049)
--(axis cs:16,102.550316303429)
--(axis cs:15,103.324530418874)
--(axis cs:14,103.981307650733)
--(axis cs:13,104.756382415227)
--(axis cs:12,105.712538604354)
--(axis cs:11,106.718015323327)
--(axis cs:10,107.752898669926)
--(axis cs:9,109.136768768901)
--(axis cs:8,110.811668475449)
--(axis cs:7,112.621760156812)
--(axis cs:6,114.750741388895)
--(axis cs:5,117.80588027108)
--(axis cs:4,121.963209752486)
--(axis cs:3,127.876375982594)
--(axis cs:2,136.852978239871)
--(axis cs:1,2.4074074074074)
--cycle;
\addplot [semithick, steelblue31119180]
table {%
1 2.4074074074074
2 81.1637994000723
3 82.1219213875962
4 82.238233067288
5 82.2306775529845
6 82.2492666039642
7 82.4988727974568
8 82.6072554671548
9 82.5288638495136
10 82.4991305641013
11 82.6264550095427
12 82.6365575912792
13 82.5789070418873
14 82.604272699394
15 82.6661419387101
16 82.5448928316749
17 82.6668895723652
18 82.6580921269123
19 82.6472969436046
20 82.6339231377655
21 82.4692877021805
22 82.708401867576
23 82.7046097791907
24 82.7774333430492
25 82.6877143950775
26 82.7031406602705
27 82.769766328114
28 82.8533887662267
29 82.7791848926352
30 82.9759323668791
31 83.014018876832
32 83.0366026062064
33 82.9698837323867
34 83.0202524752334
35 82.9524382241052
36 82.8859103569805
37 82.8410895534604
38 82.948764254536
39 83.0013528074699
40 83.0005804954583
41 82.9759807478514
42 82.9759807478514
43 82.8881275277503
44 83.2004780091993
45 83.0382899760824
46 83.1249717101065
47 83.0793939681848
48 82.9601693921593
49 82.920733382781
50 82.8070499914063
};
% -- INSERT AIDE plot content here (fill + plot commands) --
\nextgroupplot[
title={\tiny {\textbf{B4C} \citep{jain2015car}}},
ylabel={\tiny{F1 (\%)}},
title style={yshift=-0.2cm},
ymin=0,
ymax=100
]
% -- INSERT B4C plot content here (fill + plot commands) --
\path [fill=steelblue31119180, fill opacity=0.2]
(axis cs:1,11.6363636363636)
--(axis cs:1,11.6363636363636)
--(axis cs:2,33.2313373683485)
--(axis cs:3,45.2246665353119)
--(axis cs:4,52.042157305909)
--(axis cs:5,55.8944647155769)
--(axis cs:6,59.3894499308765)
--(axis cs:7,61.5966515078833)
--(axis cs:8,64.5848047719954)
--(axis cs:9,65.8611485395994)
--(axis cs:10,66.5052033211166)
--(axis cs:11,67.7210659063231)
--(axis cs:12,69.081696920578)
--(axis cs:13,70.0248797370222)
--(axis cs:14,70.9859630677586)
--(axis cs:15,71.3825239118339)
--(axis cs:16,72.5432304063399)
--(axis cs:17,73.4558336272192)
--(axis cs:18,73.9464435736174)
--(axis cs:19,74.5250671931756)
--(axis cs:20,74.9495867623111)
--(axis cs:21,75.5987525582482)
--(axis cs:22,75.838532064086)
--(axis cs:23,76.3104282366841)
--(axis cs:24,76.8770199961402)
--(axis cs:25,76.9157529537468)
--(axis cs:26,77.7018668086424)
--(axis cs:27,77.7150922528999)
--(axis cs:28,78.1266069263336)
--(axis cs:29,78.4636466636392)
--(axis cs:30,78.7042764853169)
--(axis cs:31,79.0148008006084)
--(axis cs:32,78.9074912541829)
--(axis cs:33,79.3249514693347)
--(axis cs:34,79.3629089965501)
--(axis cs:35,80.0455279788005)
--(axis cs:36,79.840823848513)
--(axis cs:37,80.3325944151017)
--(axis cs:38,80.3764324238401)
--(axis cs:39,80.6641448835065)
--(axis cs:40,80.776560973099)
--(axis cs:41,80.8457035189497)
--(axis cs:42,81.0764746662266)
--(axis cs:43,81.4398079136612)
--(axis cs:44,81.3706820201598)
--(axis cs:45,81.9669509098113)
--(axis cs:46,81.6148925944012)
--(axis cs:47,81.8247957557394)
--(axis cs:48,81.6396921468075)
--(axis cs:49,81.7369481942302)
--(axis cs:50,82.1366349951082)
--(axis cs:50,105.385915559779)
--(axis cs:50,105.385915559779)
--(axis cs:49,105.21394235306)
--(axis cs:48,105.353723185802)
--(axis cs:47,105.782852517169)
--(axis cs:46,105.821317896431)
--(axis cs:45,106.430512374853)
--(axis cs:44,106.094330764408)
--(axis cs:43,106.438015869588)
--(axis cs:42,106.355641147629)
--(axis cs:41,106.41903074269)
--(axis cs:40,106.655338991235)
--(axis cs:39,106.858202699132)
--(axis cs:38,106.897199697058)
--(axis cs:37,107.194579490296)
--(axis cs:36,107.055163814962)
--(axis cs:35,107.630860061592)
--(axis cs:34,107.329002433587)
--(axis cs:33,107.696083684298)
--(axis cs:32,107.698520919979)
--(axis cs:31,108.248868288834)
--(axis cs:30,108.395513100277)
--(axis cs:29,108.636142921955)
--(axis cs:28,108.806325765594)
--(axis cs:27,108.929548743278)
--(axis cs:26,109.481582940541)
--(axis cs:25,109.287554271572)
--(axis cs:24,109.887528564127)
--(axis cs:23,109.991327117343)
--(axis cs:22,110.238550729354)
--(axis cs:21,110.768829610754)
--(axis cs:20,110.936355158095)
--(axis cs:19,111.400172166044)
--(axis cs:18,111.777903190313)
--(axis cs:17,112.326983877402)
--(axis cs:16,112.541953393202)
--(axis cs:15,112.628025986446)
--(axis cs:14,113.622001019176)
--(axis cs:13,114.191465600902)
--(axis cs:12,114.969112301717)
--(axis cs:11,115.552305306608)
--(axis cs:10,116.584606203721)
--(axis cs:9,118.54222554112)
--(axis cs:8,120.241132877498)
--(axis cs:7,120.74573289365)
--(axis cs:6,122.997534773672)
--(axis cs:5,125.106823462113)
--(axis cs:4,128.805248708613)
--(axis cs:3,132.504587547441)
--(axis cs:2,137.501094294046)
--(axis cs:1,11.6363636363636)
--cycle;
\addplot [semithick, steelblue31119180]
table {%
1 11.6363636363636
2 85.3662158311972
3 88.8646270413765
4 90.4237030072611
5 90.5006440888451
6 91.1934923522742
7 91.1711922007669
8 92.4129688247467
9 92.2016870403596
10 91.544904762419
11 91.6366856064658
12 92.0254046111473
13 92.1081726689622
14 92.3039820434675
15 92.00527494914
16 92.5425918997708
17 92.8914087523107
18 92.8621733819652
19 92.9626196796096
20 92.9429709602032
21 93.1837910845011
22 93.0385413967201
23 93.1508776770136
24 93.3822742801337
25 93.1016536126595
26 93.5917248745918
27 93.3223204980889
28 93.4664663459637
29 93.5498947927971
30 93.5498947927971
31 93.6318345447213
32 93.3030060870809
33 93.5105175768162
34 93.3459557150687
35 93.8381940201965
36 93.4479938317373
37 93.7635869526986
38 93.6368160604492
39 93.7611737913194
40 93.715949982167
41 93.6323671308198
42 93.716057906928
43 93.9389118916245
44 93.7325063922837
45 94.1987316423322
46 93.718105245416
47 93.803824136454
48 93.4967076663045
49 93.4754452736453
50 93.7612752774434
};
\nextgroupplot[
title={\tiny {\textbf{ROAD} \citep{singh2022road}}},
ylabel={\tiny{F1 (\%)}},
title style={yshift=-0.2cm},
ymax=60,
ymin=0
]
\path [fill=steelblue31119180, fill opacity=0.2]
(axis cs:1,4.64285714285714)
--(axis cs:1,4.64285714285714)
--(axis cs:2,13.7444005665719)
--(axis cs:3,17.7905619071176)
--(axis cs:4,19.8908775703713)
--(axis cs:5,20.0353509975758)
--(axis cs:6,21.244676573495)
--(axis cs:7,22.6726533959784)
--(axis cs:8,23.1457818409577)
--(axis cs:9,23.7296709742352)
--(axis cs:10,24.5079919979849)
--(axis cs:11,24.9780474068055)
--(axis cs:12,25.7144547609977)
--(axis cs:13,25.8248159860842)
--(axis cs:14,27.5208373031441)
--(axis cs:15,27.6721986791789)
--(axis cs:16,27.8843029851824)
--(axis cs:17,28.367417821544)
--(axis cs:18,28.6858842086104)
--(axis cs:19,28.6619933248124)
--(axis cs:20,28.8195138523721)
--(axis cs:21,29.1022618506936)
--(axis cs:22,28.4764867004226)
--(axis cs:23,28.3634034525499)
--(axis cs:24,28.4987243284675)
--(axis cs:25,28.6579351109257)
--(axis cs:26,28.7784635847482)
--(axis cs:27,28.8466558797843)
--(axis cs:28,28.9409754815265)
--(axis cs:29,28.828071976118)
--(axis cs:30,28.8861408660453)
--(axis cs:31,28.5756777864016)
--(axis cs:32,28.9299866051721)
--(axis cs:33,29.0413463529665)
--(axis cs:34,29.1223091022094)
--(axis cs:35,29.239503867429)
--(axis cs:36,29.3453299291526)
--(axis cs:37,29.4545014171028)
--(axis cs:38,29.4793736986124)
--(axis cs:39,29.390857157841)
--(axis cs:40,29.4070263695667)
--(axis cs:41,29.5062655170007)
--(axis cs:42,29.8293144138247)
--(axis cs:43,29.8840078379749)
--(axis cs:44,29.7823280779212)
--(axis cs:45,29.5250653096515)
--(axis cs:46,29.8133747473016)
--(axis cs:47,29.9106720548669)
--(axis cs:48,29.9580799256032)
--(axis cs:49,30.0385585583976)
--(axis cs:50,30.2242211935527)
--(axis cs:50,39.5413827066097)
--(axis cs:50,39.5413827066097)
--(axis cs:49,39.4424443308492)
--(axis cs:48,39.4531055629992)
--(axis cs:47,39.5005134337355)
--(axis cs:46,39.5008944673221)
--(axis cs:45,39.3139648583047)
--(axis cs:44,39.6756760500227)
--(axis cs:43,39.8853884524678)
--(axis cs:42,39.940081876618)
--(axis cs:41,39.7300245354178)
--(axis cs:40,39.7498986034867)
--(axis cs:39,39.8571699837793)
--(axis cs:38,40.0731598609598)
--(axis cs:37,40.1847467422023)
--(axis cs:36,40.2180472108852)
--(axis cs:35,40.2603812091167)
--(axis cs:34,40.2976355558866)
--(axis cs:33,40.3785983051295)
--(axis cs:32,40.4363714008718)
--(axis cs:31,40.2597717749872)
--(axis cs:30,40.7605836944945)
--(axis cs:29,40.8974417198148)
--(axis cs:28,41.2155716618331)
--(axis cs:27,41.3389469536479)
--(axis cs:26,41.5003122518907)
--(axis cs:25,41.6208407257132)
--(axis cs:24,41.7167384369853)
--(axis cs:23,41.852059312903)
--(axis cs:22,42.2528224243902)
--(axis cs:21,43.1789833334106)
--(axis cs:20,43.1835316700738)
--(axis cs:19,43.3410521976335)
--(axis cs:18,43.6991322840418)
--(axis cs:17,43.7145477149107)
--(axis cs:16,43.6018761508319)
--(axis cs:15,43.8139804568354)
--(axis cs:14,44.1157851115632)
--(axis cs:13,42.8891815206056)
--(axis cs:12,43.4785505104141)
--(axis cs:11,43.5253937402596)
--(axis cs:10,43.9764283868407)
--(axis cs:9,44.2724539282949)
--(axis cs:8,44.9676334589388)
--(axis cs:7,46.0468979469108)
--(axis cs:6,46.5309765623381)
--(axis cs:5,47.7403021382574)
--(axis cs:4,50.8756180563043)
--(axis cs:3,53.6838854445184)
--(axis cs:2,57.6905397106913)
--(axis cs:1,4.64285714285714)
--cycle;
\addplot [semithick, steelblue31119180]
table {%
1 4.64285714285714
2 35.7174701386316
3 35.737223675818
4 35.3832478133378
5 33.8878265679166
6 33.8878265679166
7 34.3597756714446
8 34.0567076499482
9 34.001062451265
10 34.2422101924128
11 34.2517205735326
12 34.5965026357059
13 34.3569987533449
14 35.8183112073537
15 35.7430895680072
16 35.7430895680072
17 36.0409827682274
18 36.1925082463261
19 36.001522761223
20 36.001522761223
21 36.1406225920521
22 35.3646545624064
23 35.1077313827264
24 35.1077313827264
25 35.1393879183194
26 35.1393879183194
27 35.0928014167161
28 35.0782735716798
29 34.8627568479664
30 34.8233622802699
31 34.4177247806944
32 34.6831790030219
33 34.709972329048
34 34.709972329048
35 34.7499425382729
36 34.7816885700189
37 34.8196240796525
38 34.7762667797861
39 34.6240135708102
40 34.5784624865267
41 34.6181450262092
42 34.8846981452214
43 34.8846981452214
44 34.729002063972
45 34.4195150839781
46 34.6571346073119
47 34.7055927443012
48 34.7055927443012
49 34.7405014446234
50 34.8828019500812
};
\end{groupplot}
\end{tikzpicture}
\caption{Effects of additional dependent samples from the same chain for the top-performing LL--HMC configuration across 100 random seeds. The light blue area illustrates the standard deviation of the cummulative F1-score for the additional samples.}
\label{fig:dependent_samples}
\end{figure}
\paragraph{\textbf{Influence of additional dependent LL--HMC samples.}}
To examine how additional samples from the same LL--HMC chain affect DAR and DIR performance, Figure \ref{fig:dependent_samples} presents the average F1 score and the standard deviation as more samples are cumulatively drawn across 100 random seeds. The results show that standard deviation decreases with more samples, but performance converges quickly. This suggests that beyond a small number of posterior samples, further samples from the LL--HMC chain yield diminishing returns in classification performance.
\definecolor{darkgray176}{RGB}{176,176,176}
\definecolor{steelblue31119180}{RGB}{31,119,180}
\begin{figure}[t]
\centering
\begin{minipage}[t]{0.32\linewidth}
\centering
\includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/ess_aide.pdf}
%\vspace{-0.5cm}
\captionsetup{font=tiny}
\caption*{ESS per last layer parameter for the AIDE dataset with a single chain for 50 samples.}
% \label{fig:nuscenes_rhat_aide1}
\end{minipage}
\hfill
\begin{minipage}[t]{0.32\linewidth}
\centering
\includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/param_1.pdf}
%\vspace{-0.5cm}
\captionsetup{font=tiny}
\caption*{Value for the first weight parameter of the last layer for each sample.}
% \label{fig:nuscenes_rhat_b4c1}
\end{minipage}
\hfill
\begin{minipage}[t]{0.32\linewidth}
\centering
\includegraphics[width=0.99\linewidth, keepaspectratio]{param_51.pdf}
%\vspace{-0.5cm}
\captionsetup{font=tiny}
\caption*{Value for the fifty-first weight parameter of the last layer for each sample.}
% \label{fig:nuscenes_rhat_road1}
\end{minipage}
\caption{Visualization of the ESS and the parameter values for 50 samples for LL--HMC for the AIDE dataset with a single chain, target acceptance of 0.7, prior standard deviation of 1, and 100 burn in samples.}
\label{fig:ess}
\end{figure}
\begin{figure}[t!]
\centering
\begin{minipage}[t]{0.32\linewidth}
\centering
\includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_aide_single.pdf}
\vspace{-0.6cm}
\captionsetup{font=tiny}
\caption*{AIDE - Two Chains, One Starting Position.}
\label{fig:nuscenes_rhat_aide1}
\end{minipage}
\hfill
\begin{minipage}[t]{0.32\linewidth}
\centering
\includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_b4c_single.pdf}
\vspace{-0.6cm}
\captionsetup{font=tiny}
\caption*{B4C - Two Chains, One Starting Position.}
\label{fig:nuscenes_rhat_b4c1}
\end{minipage}
\hfill
\begin{minipage}[t]{0.32\linewidth}
\centering
\includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_oxf_single.pdf}
\vspace{-0.6cm}
\captionsetup{font=tiny}
\caption*{ROAD - Two Chains, One Starting Position.}
\label{fig:nuscenes_rhat_road1}
\end{minipage}
\vskip\baselineskip
% Second row - 3 columns
\begin{minipage}[t]{0.32\linewidth}
\centering
\includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_aide_ens.pdf}
\vspace{-0.6cm}
\captionsetup{font=tiny}
\caption*{AIDE - Two Chains, Two Starting Positions.}
\label{fig:nuscenes_rhat_aide2}
\end{minipage}
\hfill
\begin{minipage}[t]{0.32\linewidth}
\centering
\includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_b4c_ens.pdf}
\vspace{-0.6cm}
\captionsetup{font=tiny}
\caption*{B4C - Two Chains, Two Starting Positions.}
\label{fig:nuscenes_rhat_b4c2}
\end{minipage}
\hfill
\begin{minipage}[t]{0.32\linewidth}
\centering
\includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_oxf_ens.pdf}
\vspace{-0.6cm}
\captionsetup{font=tiny}
% %\vspace{-0.3cm}
\caption*{ROAD - Two Chains, Two Starting Positions.}
\label{fig:nuscenes_rhat_road2}
\end{minipage}
\caption{Log-scale histograms of the $\hat{R}$ values for two chains for single starting position (top) or two starting positions (bottom), for 100 burn-in samples, target acceptance of 0.7, 50 samples and a prior scale of 1.}
\label{fig:rhats}
\end{figure}
\begin{table*}[t!]
\centering
\caption{Average top-performance and two standard errors of the mean across five random seeds for LL--HMC with multiple starting positions or chains. S=Starting Positions, C=Chains.}
\resizebox{\linewidth}{!}{
\begin{tabular}{lcc|c|c|c|c||c|c|c|c||c|c|c|c}
\toprule
& &&\multicolumn{4}{c||}{\textbf{AIDE} \citep{yang2023aide}} & \multicolumn{4}{c||}{\textbf{B4C} \citep{jain2015car}} & \multicolumn{4}{c}{\textbf{ROAD} \citep{singh2022road}} \\
& &&\textbf{Acc ($\uparrow$)} & \textbf{F1 ($\uparrow$)} & \textbf{ACE ($\downarrow$)} & \textbf{rAULC ($\uparrow$)}
& \textbf{Acc ($\uparrow$)} & \textbf{F1 ($\uparrow$)} & \textbf{ACE ($\downarrow$)} & \textbf{rAULC ($\uparrow$)}
& \textbf{Acc ($\uparrow$)} & \textbf{F1 ($\uparrow$)} & \textbf{ACE ($\downarrow$)} & \textbf{rAULC ($\uparrow$)} \\
\midrule
\multirow{4}{*}{\textbf{LL--HMC}} & \multirow{2}{*}{\small{S=1}} & \small{C=1}
& 89.00$\pm$0.16 & 83.87$\pm$0.08 & 0.029$\pm$0.002 & 0.68$\pm$0.06
& 93.73$\pm$0.42 & 94.78$\pm$0.29 & 0.027$\pm$0.007 & 0.66$\pm$0.10
& 70.15$\pm$5.05 & 54.18$\pm$4.43 & 0.073$\pm$0.014 & 0.41$\pm$0.12 \\
& & \small{C=2}
& 88.34$\pm$0.73 & 82.98$\pm$0.25 & 0.045$\pm$0.017 & 0.66$\pm$0.06
& 93.59$\pm$0.42 & 94.74$\pm$0.32 & 0.032$\pm$0.011 & 0.61$\pm$0.03
& 68.16$\pm$5.12 & 53.51$\pm$0.93 & 0.077$\pm$0.017 & 0.16$\pm$0.09 \\
\cmidrule{2-15}
& \multirow{2}{*}{\small{S=2}} & \small{C=1}
& 89.00$\pm$0.17 & 83.66$\pm$0.32 & 0.029$\pm$0.002 & 0.70$\pm$0.03
& 94.02$\pm$0.11 & 95.00$\pm$0.07 & 0.023$\pm$0.001 & 0.70$\pm$0.02
& 69.16$\pm$4.66 & 52.67$\pm$2.45 & 0.072$\pm$0.015 & 0.57$\pm$0.07 \\
& & \small{C=2}
& 88.15$\pm$0.46 & 82.56$\pm$0.60 & 0.042$\pm$0.007 & 0.60$\pm$0.03
& 93.59$\pm$0.42 & 94.72$\pm$0.30 & 0.047$\pm$0.019 & 0.52$\pm$0.07
& 70.90$\pm$4.76 & 53.79$\pm$3.05 & 0.081$\pm$0.017 & 0.27$\pm$0.19 \\
\bottomrule
\end{tabular}
}
\label{tab:performance_multi_hmc}
\end{table*}
\paragraph{\textbf{Effective sample size and parameter trajectories.}}Figure \ref{fig:ess} includes a histogram of effective sample size (ESS) values across parameters (x-axis: ESS; y-axis: number of parameters), along with two example parameter trajectories. Consistent with findings in \cite{vellenga2024pthmc}, LL--HMC yields a low ESS for most parameters, indicating effective sampling only for a small subset. The example parameter trajectories show no discernible trend or drift and fluctuate around zero, suggesting limited exploration.
\paragraph{\textbf{Multiple chains and starting positions.}}
Table \ref{tab:performance_multi_hmc} shows the average top-performing configurations across five random seeds for multiple starting (S) positions and multiple chains (C). The results show that increasing the number of chains or adding an additional starting position does not consistently lead to improved performance. When using multiple chains, it is also important to assess whether the chains have explored distinct regions of the posterior and converged to the target distribution. Figure \ref{fig:rhats} shows the $\hat{R}$ values for the scenarios with multiple chains for both single and multiple starting positions. For all datasets, the $\hat{R}$ values are close to 1 and mostly below 1.1. This indicates that the chains have likely converged to the target distribution but also potentially sample from the same region \citep{izmailov2021bayesian}. %
\begin{table*}[t]
\centering
\caption{Average uncertainty-based OOD detection performance and two standard errors of the mean of the top-performing hyperparameter configurations per random seed. ROC-AUC=Receiver Operating Characteristic - Area Under the Curve, PR=Precision-Recall - Area Under the Curve. } % c=chains, s=starting positions
\resizebox{\textwidth }{!}{
\begin{tabular}{l|c|c|c|c|c|c||c|c|c|c|c|c||c|c|c}
\toprule
\multicolumn{1}{c|}{} & \multicolumn{6}{c||}{\textbf{AIDE} \citep{yang2023aide}} & \multicolumn{6}{c||}{\textbf{B4C} \citep{jain2015car}} & \multicolumn{3}{c}{\textbf{ROAD} \citep{singh2022road}} \\ \cmidrule{2-16}
\multicolumn{1}{c|}{} & \multicolumn{3}{c|}{\textbf{OOD min}} & \multicolumn{3}{c||}{\textbf{OOD max}} & \multicolumn{3}{c|}{\textbf{OOD min}} & \multicolumn{3}{c||}{\textbf{OOD max}} & \multicolumn{3}{c}{\textbf{OOD min}} \\
\multicolumn{1}{c|}{} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} \\
\midrule
\textbf{Regular} & 0.55 & 0.19 & 0.91 & 0.82 & 0.94 & 0.41 & 0.51 & 0.37 & 0.90 & 0.64 & 0.82 & 0.84 & 0.19 & 0.25 & 1.00 \\
\textbf{DE (N=5)} & 0.69 & 0.28 & 0.85 & 0.84 & 0.94 &0.35 & 0.52 & 0.39 & 0.90 & 0.79 & 0.90 & 0.45 & 0.55& 0.41 & 0.88 \\
\textbf{DDU} & 0.73 & 0.31 & 0.49 & 0.80 & 0.90 & 0.45 & 0.63 & 0.44 &0.87 & 0.48 & 0.75 & 0.96 & 0.63 & 0.44 & 0.87 \\ \midrule
\textbf{BBB--LL} & 0.95$\pm$0.02 & 0.72$\pm$0.08 & 0.10$\pm$0.05 & 0.96$\pm$0.01 & 0.99$\pm$0.01 & 0.22$\pm$0.06 & 0.84$\pm$0.03 & 0.70$\pm$0.04 & 0.48$\pm$0.21 & 0.80$\pm$0.02 & 0.92$\pm$0.01 & 0.69$\pm$0.10 & 0.60$\pm$0.10 & 0.49$\pm$0.13 & 0.85$\pm$0.06 \\
% 0.85 & 0.47 & 0.35 & 0.94 &
% 0.98 & 0.23 & 0.69 & 0.44 & 0.58 & 0.69 & 0.87 & 0.83 & 0.69 & 0.44 & 0.58 \\
\textbf{PE--LL} & 0.55$\pm$0.01 & 0.52$\pm$0.01 & 0.83$\pm$0.01 & 0.56$\pm$0.01 & 0.53$\pm$0.01 & 0.72$\pm$0.02 & 0.52$\pm$0.01 & 0.52$\pm$0.01 & 0.91$\pm$0.01 & 0.53$\pm$0.01 & 0.53$\pm$0.00 & 0.94$\pm$0.02 & 0.53$\pm$0.01 & 0.52$\pm$0.01 & 0.92$\pm$0.01 \\
% 0.55 & 0.20 & 0.92 & 0.64 & 0.59 & 0.57 & 0.57 & 0.55 & 0.78 & 0.56 & 0.55 & 0.94 & 0.57 & 0.55 & 0.78 \\
\textbf{SE} & 0.59$\pm$0.01 & 0.21$\pm$0.01 & 0.89$\pm$0.01 & 0.88$\pm$0.01 & 0.96$\pm$0.01 & 0.41$\pm$0.01 & 0.72$\pm$0.02 & 0.54$\pm$0.01 & 0.50$\pm$0.02 & 0.71$\pm$0.01 & 0.88$\pm$0.01 & 0.79$\pm$0.01 & 0.45$\pm$0.05 & 0.33$\pm$0.02 & 0.86$\pm$0.10 \\
%0.55 & 0.20 & 0.92 & 0.88 & 0.96 & 0.41 & 0.71 & 0.52 & 0.50 & 0.72 & 0.88 & 0.78 & 0.71 & 0.52 & 0.50 \\
% \textbf{cSGHMC} & & & & & & & & & & & & \\
\textbf{VBLL} & 0.66$\pm$0.04 & 0.25$\pm$0.03 & 0.79$\pm$0.08 &0.89$\pm$0.01 & 0.97$\pm$0.01 & 0.42$\pm$0.04 & 0.82$\pm$0.05 & 0.64$\pm$0.07 & 0.41$\pm$0.09 & 0.65$\pm$0.02 & 0.84$\pm$0.02 & 0.88$\pm$0.06 & 0.57$\pm$0.09 & 0.44$\pm$0.11 & 0.80$\pm$0.05 \\
% 0.56 & 0.19 & 0.88 & 0.85 & 0.95 & 0.42 & 0.63 & 0.44 & 0.61& 0.56 & 0.79 & 0.91 & 0.63 & 0.44 & 0.61 \\ \midrule
\textbf{LL--HMC} & 0.91$\pm$0.03 & 0.58$\pm$0.10 & 0.18$\pm$0.05 & 0.95$\pm$0.01 & 0.99$\pm$0.01 & 0.14$\pm$0.01 & 0.85$\pm$0.04 & 0.73$\pm$0.05 & 0.36$\pm$0.08& 0.91$\pm$0.01 & 0.97$\pm$0.01 & 0.39$\pm$0.05& 0.87$\pm$0.01 & 0.72$\pm$0.01 & 0.31$\pm$0.04 \\ % 0.86 & 0.47 & 0.34 & 0.95 & 0.98 & 0.16 & 0.73 & 0.53 & 0.50 & 0.90 & 0.96 & 0.33 & 0.73 & 0.53 & 0.50 \\
% \multirow{4}{*}{\textbf{LL--HMC}} & \small{S=1, C=1}& 0.79 & 0.35 & 0.44 & 0.89 & 0.96 & 0.33 & 0.73 & 0.53 & 0.50& 0.90 & 0.96 & 0.33 & 0.73 & 0.53 & 0.50 \\
% & \small{S=1, C=2} & 0.86 & 0.47 & 0.34 & 0.95 & 0.98 & 0.16 & 0.67 & 0.49 & 0.59& 0.88 & 0.95 & 0.38 & 0.67& 0.49 & 0.64 \\
% % \midrule
% & \small{S=2, C=1} & 0.65 & 0.35 & 0.97 & 0.83 & 0.95 & 0.70 & 0.53 & 0.39 & 0.85 & 0.87 & 0.94 & 0.41 & 0.37 & 0.37 & 0.98 \\
% & \small{S=2, C=2} & 0.71 & 0.37 & 0.73 & 0.88& 0.96 & 0.41 & 0.65 & 0.58 & 0.89 & 0.90 & 0.94 & 0.22 & 0.45 & 0.34 & 0.97\\
\bottomrule
\end{tabular}
}
\label{tab:road_ood_exp}
\end{table*}
\begin{table*}[t]
\centering
\caption{Average uncertainty-based OOD detection grid search results and two standard errors of the mean for the best average hyperparameter configuration across five random seeds.} % c=chains, s=starting positions
\resizebox{\textwidth }{!}{
\begin{tabular}{l|c|c|c|c|c|c||c|c|c|c|c|c||c|c|c}
\toprule
\multicolumn{1}{c|}{} & \multicolumn{6}{c||}{\textbf{AIDE} \citep{yang2023aide}} & \multicolumn{6}{c||}{\textbf{B4C} \citep{jain2015car}} & \multicolumn{3}{c}{\textbf{ROAD} \citep{singh2022road}} \\ \cmidrule{2-16}
\multicolumn{1}{c|}{} & \multicolumn{3}{c|}{\textbf{OOD min}} & \multicolumn{3}{c||}{\textbf{OOD max}} & \multicolumn{3}{c|}{\textbf{OOD min}} & \multicolumn{3}{c||}{\textbf{OOD max}} & \multicolumn{3}{c}{\textbf{OOD min}} \\
\multicolumn{1}{c|}{} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} \\
\midrule
\textbf{BBB--LL} & 0.55$\pm$0.11 & 0.20$\pm$0.04 & 0.84$\pm$0.15 & 0.80$\pm$0.06 & 0.93$\pm$0.02 & 0.58$\pm$0.20 & 0.32$\pm$0.05 & 0.30$\pm$0.02 & 0.99$\pm$0.03 & 0.50$\pm$0.11 & 0.76$\pm$0.07 & 0.92$\pm$0.04 & 0.29$\pm$0.06 & 0.28$\pm$0.02 & 0.97$\pm$0.04 \\
\textbf{PE--LL} & 0.50$\pm$0.00& 0.50$\pm$0.00& 0.95$\pm$0.01 & 0.51$\pm$0.00& 0.51$\pm$0.00& 0.93$\pm$0.01 & 0.51$\pm$0.00& 0.51$\pm$0.00& 0.95$\pm$0.01 & 0.52$\pm$0.01 & 0.51$\pm$0.01 & 0.94$\pm$0.01 & 0.51$\pm$0.00& 0.51$\pm$0.00& 0.95$\pm$0.00\\
\textbf{SE} & 0.51$\pm$0.01 & 0.17$\pm$0.00& 0.86$\pm$0.02 & 0.81$\pm$0.00& 0.93$\pm$0.00& 0.41$\pm$0.01 & 0.29$\pm$0.00& 0.29$\pm$0.00& 1.00$\pm$0.00 & 0.32$\pm$0.02 & 0.69$\pm$0.01 & 0.99$\pm$0.01 & 0.17$\pm$0.01 & 0.24$\pm$0.00& 1.00$\pm$0.00\\
\textbf{VBLL} & 0.44$\pm$0.02 & 0.14$\pm$0.01 & 0.96$\pm$0.02 & 0.54$\pm$0.14 & 0.84$\pm$0.07 & 0.88$\pm$0.13 & 0.49$\pm$0.10& 0.36$\pm$0.07 & 0.90$\pm$0.12 & 0.48$\pm$0.03 & 0.42$\pm$0.04 & 0.96$\pm$0.02 & 0.16$\pm$0.03 & 0.24$\pm$0.01 & 0.99$\pm$0.01 \\
\textbf{LL--HMC} & 0.53$\pm$0.02 & 0.18$\pm$0.01 & 0.87$\pm$0.05 & 0.82$\pm$0.06 & 0.94$\pm$0.02 & 0.48$\pm$0.20 & 0.57$\pm$0.06 & 0.42$\pm$0.04 & 0.78$\pm$0.10 & 0.57$\pm$0.06 & 0.74$\pm$0.02 & 0.78$\pm$0.10 & 0.28$\pm$0.08 & 0.27$\pm$0.02 & 0.97$\pm$0.03 \\
\bottomrule
\end{tabular}
}
\label{tab:road_ood_exp_hp}
\end{table*}
%Consistent with the in-distribution evaluation, we performed a grid search across five random seeds for each OOD scenario to assess OOD detection robustness.
\subsection{Uncertainty-based out-of-distribution detection performance}
\paragraph{\textbf{Grid search results.}} Table \ref{tab:road_ood_exp} reports the average OOD detection performance for the top-performing hyperparameter configurations per random seed. Table \ref{tab:road_ood_exp_hp} shows the results of the single hyperparameter configuration that achieved the best average performance across the random seeds. For every OOD scenario, at least one PDL method outperforms the regular softmax baseline. Table \ref{tab:road_ood_exp} shows that the top-performing LL--HMC configuration produces the top or the best uncertainty-based OOD detection results. The other PDL methods also exhibit comparable, and at times superior, performance. However, as observed for in-distribution classification results, Table \ref{tab:road_ood_exp_hp} highlights that the grid searched hyperparameter configurations are in most scenarios not robust to different random seeds and often result in substantial performance degradation. Only the AIDE OOD max scenario (with the most OOD instances) shows comparable performance to the top-performing configurations.
\begin{figure*}[t!]
\centering
\begin{tikzpicture}
\begin{groupplot}[
group style={
group size=3 by 2,
horizontal sep=1.0cm,
vertical sep=0.75cm,
},
width=5.5cm,
height=4.0cm,
grid=major,
xlabel={\tiny{Number of samples/predictions}},
xtick={1,5,10,15,20,25,30,35,40,45,50},
tick label style={font=\tiny},
label style={font=\small},
y tick label style={
/pgf/number format/fixed,
/pgf/number format/precision=2,
/pgf/number format/fixed zerofill=true,
},
tick label style={font=\fontsize{4}{4}\selectfont},
%ylabel style={yshift=-0.2cm},% Shift y-label down
xlabel style={yshift=0.1cm}, % Shift x-label down
]
% First plot (Dataset A)
\nextgroupplot[
title={\tiny {\textbf{AIDE} \citep{yang2023aide}}},
title style={yshift=-0.2cm},
% xlabel=\empty,
ylabel={\tiny{PR-AUC}},
ymin=0.15,
ymax=0.66,
]
% Plot data for BBB in Dataset A
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=orange}] coordinates {
(5.0,0.4099293594955154) += (0,0.0611586526848039) -=(0,0.0611586526848039)
(10.0,0.4708471672582329) += (0,0.06596160100760632) -=(0,0.06596160100760632)
(15.0,0.46847685413525325) += (0,0.06786657155694765) -=(0,0.06786657155694765)
(20.0,0.5961098446503498) += (0,0.06015507569947847) -=(0,0.06015507569947847)
(25.0,0.5619578990892682) += (0,0.0612660126788448) -=(0,0.0612660126788448)
(30.0,0.587875672510149) += (0,0.054593452295624796) -=(0,0.054593452295624796)
(35.0,0.5542912596218069) += (0,0.0583492582662243) -=(0,0.0583492582662243)
(40.0,0.5699888898962179) += (0,0.05678440536488734) -=(0,0.05678440536488734)
(45.0,0.5558766945490639) += (0,0.06268800160241772) -=(0,0.06268800160241772)
(50.0,0.5903266926606767) += (0,0.07500588661606554) -=(0,0.07500588661606554)
};
% Plot data for PE in Dataset B
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=cyan}] coordinates {
(5,0.5162914097457046) += (0,0.0017931393603417774) -=(0,0.0017931393603417774)
(10,0.5192954008000028) += (0,0.002699285090600101) -=(0,0.002699285090600101)
(15,0.5199229997637944) += (0,0.0018445778842600264) -=(0,0.0018445778842600264)
(20,0.5219530311768699) += (0,0.0017208852377501094) -=(0,0.0017208852377501094)
(25,0.5201325378410252) += (0,0.0008048208471415663) -=(0,0.0008048208471415663)
(30,0.5211818105841095) += (0,0.0011579563124847395) -=(0,0.0011579563124847395)
(35,0.520793753455953) += (0,0.000806497184282372) -=(0,0.000806497184282372)
(40,0.5208814466703101) += (0,0.0007430588666324227) -=(0,0.0007430588666324227)
(45,0.5204366515532967) += (0,0.0008858273648187847) -=(0,0.0008858273648187847)
(50,0.5206558062269206) += (0,0.0009148408906930005) -=(0,0.0009148408906930005)
};
% Plot data for SE in Dataset A
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=yellow}] coordinates {
(5,0.20694511344929709) += (0,0.001373024874313264) -=(0,0.001373024874313264)
(10,0.20611171048491678) += (0,0.0009710375647568551) -=(0,0.0009710375647568551)
(15,0.20637550980293162) += (0,0.000658744036075908) -=(0,0.000658744036075908)
(20,0.2063492428356545) += (0,0.0005075677891823795) -=(0,0.0005075677891823795)
(25,0.2067879494966308) += (0,0.0005741695420387532) -=(0,0.0005741695420387532)
(30,0.20682007591505136) += (0,0.0005395029330917639) -=(0,0.0005395029330917639)
(35,0.2064812404309772) += (0,0.000746011032822557) -=(0,0.000746011032822557)
(40,0.20629989745120106) += (0,0.0005379654165682829) -=(0,0.0005379654165682829)
(45,0.20635113018699108) += (0,0.0004337772666133956) -=(0,0.0004337772666133956)
(50,0.20655758244982697) += (0,0.0003972238421868066) -=(0,0.0003972238421868066)
};
% Plot data for LL--HMC in Dataset A
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=yellow}] coordinates {
(5,0.38177465178644876) += (0,0.05353110292857039) -=(0,0.05353110292857039)
(10,0.4214417583457424) += (0,0.01893322197300767) -=(0,0.01893322197300767)
(15,0.4166176479774652) += (0,0.04085449754162002) -=(0,0.04085449754162002)
(20,0.4302313222856375) += (0,0.022941019011776953) -=(0,0.022941019011776953)
(25,0.4538529905132734) += (0,0.02964893808546523) -=(0,0.02964893808546523)
(30,0.45759031304906833) += (0,0.02982390886549429) -=(0,0.02982390886549429)
(35,0.46095961793450313) += (0,0.023537446545199623) -=(0,0.023537446545199623)
(40,0.4617348365783642) += (0,0.04175402035974026) -=(0,0.04175402035974026)
(45,0.46653975967457456) += (0,0.048281145328221614) -=(0,0.048281145328221614)
(50,0.4834894076461963) += (0,0.015383569420799786) -=(0,0.015383569420799786)
};
% Single marker for DE at 5 samples
\addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
(5,82.24)
};
% Single marker for Regular Model at 1 sample
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
(1,0.20)
};
% Single marker for VBLL at 1 sample
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=green}] coordinates {
(1,0.16784053115103267) += (0,0.010862086427357984) -=(0,0.010862086427357984)
};
% Second plot (Dataset B)
\nextgroupplot[
title={\tiny {\textbf{B4C} \citep{jain2015car}}},
title style={yshift=-0.2cm},
% xlabel=\empty,
ylabel={\small{PR-AUC}},
ymax=0.65,
ymin=0.35,
]
% Plot data for BBB in Dataset B
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=orange}] coordinates {
(5.0,0.4638115966944718) += (0,0.06263507963334694) -=(0,0.06263507963334694)
(10.0,0.46837843860822764) += (0,0.03929424611442142) -=(0,0.03929424611442142)
(15.0,0.5383142087951663) += (0,0.0597395982855219) -=(0,0.0597395982855219)
(20.0,0.5015212997930913) += (0,0.02829283734024349) -=(0,0.02829283734024349)
(25.0,0.5367705848030446) += (0,0.022485559864272183) -=(0,0.022485559864272183)
(30.0,0.5518880202900334) += (0,0.03407983637151061) -=(0,0.03407983637151061)
(35.0,0.5709447653908899) += (0,0.0769850406691456) -=(0,0.0769850406691456)
(40.0,0.5831610515156422) += (0,0.0563969018049688) -=(0,0.0563969018049688)
(45.0,0.5558371519949898) += (0,0.05581561722959842) -=(0,0.05581561722959842)
(50.0,0.5695414652006476) += (0,0.049748689551465544) -=(0,0.049748689551465544)
};
% Plot data for PE in Dataset B
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=cyan}] coordinates {
(5,0.5123220248795122) += (0,0.0033621743015949117) -=(0,0.0033621743015949117)
(10,0.5175589443257259) += (0,0.0022718991660126514) -=(0,0.0022718991660126514)
(15,0.5161847669999082) += (0,0.0020600109919502856) -=(0,0.0020600109919502856)
(20,0.5138501479450839) += (0,0.002445889341203009) -=(0,0.002445889341203009)
(25,0.5166221751254654) += (0,0.001693895311244795) -=(0,0.001693895311244795)
(30,0.5174380161209144) += (0,0.001337965376785194) -=(0,0.001337965376785194)
(35,0.5171830021699783) += (0,0.0006439600742188478) -=(0,0.0006439600742188478)
(40,0.5157806828353164) += (0,0.0018848777478051668) -=(0,0.0018848777478051668)
(45,0.5179971876282832) += (0,0.000556637372421376) -=(0,0.000556637372421376)
(50,0.5175012699373359) += (0,0.0007189578131333796) -=(0,0.0007189578131333796)
};
% Plot data for SE in Dataset B
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=red}] coordinates {
(5,0.4206446511658267) += (0,0.00460980486790938) -=(0,0.00460980486790938)
(10,0.4194405869966336) += (0,0.0035345981442183187) -=(0,0.0035345981442183187)
(15,0.4213187852732581) += (0,0.002975979508316335) -=(0,0.002975979508316335)
(20,0.4199438772617269) += (0,0.0016168051918349074) -=(0,0.0016168051918349074)
(25,0.4203765071938733) += (0,0.001856711833077699) -=(0,0.001856711833077699)
(30,0.42074098327383525) += (0,0.0032297818789886475) -=(0,0.0032297818789886475)
(35,0.4233719281478428) += (0,0.001037373570778641) -=(0,0.001037373570778641)
(40,0.4190402100703186) += (0,0.002717013237190011) -=(0,0.002717013237190011)
(45,0.4192048775709722) += (0,0.0017838535081197776) -=(0,0.0017838535081197776)
(50,0.4206813168449681) += (0,0.001979401777005892) -=(0,0.001979401777005892)
};
% Plot data for LL--HMC in Dataset B
\addplot[mark=diamond*, thin, line width=0.25pt,
mark options={fill=yellow},
mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=yellow}] coordinates {
(5,0.49154039080155254) += (0,0.029341496660601556) -=(0,0.029341496660601556)
(10,0.4786783124541153) += (0,0.046594966526135836) -=(0,0.046594966526135836)
(15,0.4858752502869305) += (0,0.03643797444855078) -=(0,0.03643797444855078)
(20,0.49316223276116683) += (0,0.05662706336336582) -=(0,0.05662706336336582)
(25,0.5055797138457091) += (0,0.03157582500057088) -=(0,0.03157582500057088)
(30,0.5000315859462228) += (0,0.03793796850444391) -=(0,0.03793796850444391)
(35,0.5094513789672975) += (0,0.01639906660142857) -=(0,0.01639906660142857)
(40,0.4878801810924019) += (0,0.03814407001476142) -=(0,0.03814407001476142)
(45,0.487995741214408) += (0,0.03930492837330372) -=(0,0.03930492837330372)
(50,0.507522857379817) += (0,0.025232759972828693) -=(0,0.025232759972828693)
};
% Single marker for DE at 5 samples
\addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
(5,0.39)
};
% Single marker for Regular Model at 1 sample
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
(1,0.37)
};
% Single marker for VBLL at 1 sample
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=green}] coordinates {
(1,0.46620109960672795) += (0,0.047805178351586735) -=(0,0.047805178351586735)
};
% Third plot (Dataset C)
\nextgroupplot[
title={\tiny {\textbf{ROAD} \citep{singh2022road}}},
title style={yshift=-0.2cm},
% xlabel=\empty,
ylabel={\tiny{PR-AUC}},
ymin=0.2,
ymax=0.6
]
% Plot data for BBB in Dataset C
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=orange}] coordinates {
(5.0,0.38773302582836655) += (0,0.06927663458620247) -=(0,0.06927663458620247)
(10.0,0.3846003238762753) += (0,0.06946059174627063) -=(0,0.06946059174627063)
(15.0,0.32491859375872256) += (0,0.03606627073289094) -=(0,0.03606627073289094)
(20.0,0.39729878889193204) += (0,0.05093776633649451) -=(0,0.05093776633649451)
(25.0,0.37972716205254164) += (0,0.06648795556275817) -=(0,0.06648795556275817)
(30.0,0.33951794870287716) += (0,0.03920956973347966) -=(0,0.03920956973347966)
(35.0,0.38624654790624013) += (0,0.07775981881775736) -=(0,0.07775981881775736)
(40.0,0.35987718338549146) += (0,0.04119934271810508) -=(0,0.04119934271810508)
(45.0,0.3808373415939105) += (0,0.08027787441865955) -=(0,0.08027787441865955)
(50.0,0.4123227895682818) += (0,0.06473062325494665) -=(0,0.06473062325494665)
};
% Plot data for PE in Dataset C
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=cyan}] coordinates {
(5,0.5306308724747621) += (0,0.002183861218738374) -=(0,0.002183861218738374)
(10,0.5370422497679835) += (0,0.0021791838936996843) -=(0,0.0021791838936996843)
(15,0.5368442517650018) += (0,0.001997232231268516) -=(0,0.001997232231268516)
(20,0.5359424976539415) += (0,0.0015998150337851945) -=(0,0.0015998150337851945)
(25,0.5358339540491588) += (0,0.0011531633534958117) -=(0,0.0011531633534958117)
(30,0.5363968252508218) += (0,0.0014169992520038842) -=(0,0.0014169992520038842)
(35,0.536454165236027) += (0,0.0014441551276023348) -=(0,0.0014441551276023348)
(40,0.5359497122163313) += (0,0.0010680076735959062) -=(0,0.0010680076735959062)
(45,0.5371823826367331) += (0,0.0012376999059711677) -=(0,0.0012376999059711677)
(50,0.53686316284095) += (0,0.0009280963390977259) -=(0,0.0009280963390977259)
};
% Plot data for SE in Dataset C
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=red}] coordinates {
(5,0.3133075629044341) += (0,0.01218731778942207) -=(0,0.01218731778942207)
(10,0.3104261371113608) += (0,0.005470732865910389) -=(0,0.005470732865910389)
(15,0.3138265304819588) += (0,0.005856994784797176) -=(0,0.005856994784797176)
(20,0.3196035307762033) += (0,0.00790886567990158) -=(0,0.00790886567990158)
(25,0.32096080747240985) += (0,0.00833132646399127) -=(0,0.00833132646399127)
(30,0.31792014751682107) += (0,0.004430483838032742) -=(0,0.004430483838032742)
(35,0.3177472908417346) += (0,0.003392476562095908) -=(0,0.003392476562095908)
(40,0.3178155109752583) += (0,0.003980072054699508) -=(0,0.003980072054699508)
(45,0.32215951534707027) += (0,0.006032049968494871) -=(0,0.006032049968494871)
(50,0.3215322735884013) += (0,0.003270997191830635) -=(0,0.003270997191830635)
};
% Plot data for LL--HMC in Dataset C
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt, error bars/.cd,
y dir=both, y explicit,
error bar style={line width=0.1pt, color=yellow}] coordinates {
(5,0.30259105891012805) += (0,0.025997233083679124) -=(0,0.025997233083679124)
(10,0.2913667249900822) += (0,0.016209804868846067) -=(0,0.016209804868846067)
(15,0.31814389406627586) += (0,0.03973041935650117) -=(0,0.03973041935650117)
(20,0.3287216845015036) += (0,0.0503565554763321) -=(0,0.0503565554763321)
(25,0.31062515571996696) += (0,0.044754320959089945) -=(0,0.044754320959089945)
(30,0.28833208941681676) += (0,0.022108741344449857) -=(0,0.022108741344449857)
(35,0.3302366083064697) += (0,0.05503833240035041) -=(0,0.05503833240035041)
(40,0.3045102230648385) += (0,0.02073995834927245) -=(0,0.02073995834927245)
(45,0.3042644592758631) += (0,0.02220499653733075) -=(0,0.02220499653733075)
(50,0.30972900759294825) += (0,0.027734898495034876) -=(0,0.027734898495034876)
};
% Single marker for DE at 5 samples
\addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
(5,0.41)
};
% Single marker for Regular Model at 1 sample
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
(1,0.24955)
};
% Single marker for VBLL at 1 sample
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=green}] coordinates {
% (1,49.34)
(1,0.24497345524411732) +=(0,0.00616422784742784) -=(0,0.00616422784742784)
};
\nextgroupplot[
title=\empty,
% title style={yshift=-0.2cm},
ylabel={\tiny{PR-AUC}},
ymin=0.85,
ymax=0.98
]
% Plot data for BBB in Dataset A
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=orange}] coordinates {
(5.0,0.8890952355627204) += (0,0.02047053526188267) -=(0,0.02047053526188267)
(10.0,0.8953625367929087) += (0,0.01246797746193798) -=(0,0.01246797746193798)
(15.0,0.9067725793580275) += (0,0.013519808461528654) -=(0,0.013519808461528654)
(20.0,0.8969385261953808) += (0,0.01036496620579844) -=(0,0.01036496620579844)
(25.0,0.9130454847262927) += (0,0.014946356767675757) -=(0,0.014946356767675757)
(30.0,0.8938326988036012) += (0,0.01201948574187212) -=(0,0.01201948574187212)
(35.0,0.8970527741214148) += (0,0.015264353856984263) -=(0,0.015264353856984263)
(40.0,0.9083495177397062) += (0,0.014328574100584288) -=(0,0.014328574100584288)
(45.0,0.9127346098767953) += (0,0.00698748386415729) -=(0,0.00698748386415729)
(50.0,0.9005882102907485) += (0,0.01466810260714856) -=(0,0.01466810260714856)
};
% Plot data for PE in Dataset B
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=cyan}] coordinates {
(5,0.5244477566372742) += (0,0.005289947934484055) -=(0,0.005289947934484055)
(10,0.5286537724419726) += (0,0.0017772658828552632) -=(0,0.0017772658828552632)
(15,0.5300709592019601) += (0,0.00249704250284482) -=(0,0.00249704250284482)
(20,0.5304511474279104) += (0,0.002553956877995804) -=(0,0.002553956877995804)
(25,0.5306099762045432) += (0,0.0008398459424151745) -=(0,0.0008398459424151745)
(30,0.5309125311021029) += (0,0.002196645318621404) -=(0,0.002196645318621404)
(35,0.5307407058391697) += (0,0.0018928062862804518) -=(0,0.0018928062862804518)
(40,0.5316459892920249) += (0,0.001427774136696773) -=(0,0.001427774136696773)
(45,0.5315396864551322) += (0,0.0014144342419827878) -=(0,0.0014144342419827878)
(50,0.5316868323701924) += (0,0.0012015886915160867) -=(0,0.0012015886915160867)
};
% Plot data for SE in Dataset A
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=yellow}] coordinates {
(5,0.9555835895870072) += (0,0.002832811409249749) -=(0,0.002832811409249749)
(10,0.9587466540049675) += (0,0.0019056627083049182) -=(0,0.0019056627083049182)
(15,0.9597025633785752) += (0,0.0013939626388902047) -=(0,0.0013939626388902047)
(20,0.9602738656910887) += (0,0.0009153579402506462) -=(0,0.0009153579402506462)
(25,0.960161140329007) += (0,0.0010255319525492982) -=(0,0.0010255319525492982)
(30,0.9601480522823737) += (0,0.0009191390494881075) -=(0,0.0009191390494881075)
(35,0.9608012772991531) += (0,0.000983793732688009) -=(0,0.000983793732688009)
(40,0.9600423600296191) += (0,0.000951478299868778) -=(0,0.000951478299868778)
(45,0.9602279128121924) += (0,0.0007558661646736411) -=(0,0.0007558661646736411)
(50,0.9604300014730894) += (0,0.0010174092592789104) -=(0,0.0010174092592789104)
};
% Plot data for LL--HMC in Dataset A
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=yellow}] coordinates {
(5,0.9617172352105422) += (0,0.0034651347012903784) -=(0,0.0034651347012903784)
(10,0.9646612480139651) += (0,0.002391206411039989) -=(0,0.002391206411039989)
(15,0.9691497394501779) += (0,0.001712411808605333) -=(0,0.001712411808605333)
(20,0.971158174692748) += (0,0.001375620955681129) -=(0,0.001375620955681129)
(25,0.9706999801065308) += (0,0.001210874051052671) -=(0,0.001210874051052671)
(30,0.9720006924024169) += (0,0.001200270967006817) -=(0,0.001200270967006817)
(35,0.9714917236460485) += (0,0.001485601366579603) -=(0,0.001485601366579603)
(40,0.9711898294705111) += (0,0.0022849187140955524) -=(0,0.0022849187140955524)
(45,0.9719698549924789) += (0,0.0009724128543781708) -=(0,0.0009724128543781708)
(50,0.9727679502293537) += (0,0.0007986064197992981) -=(0,0.0007986064197992981)
};
% Single marker for DE at 5 samples
\addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
(5,0.94)
};
% Single marker for Regular Model at 1 sample
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
(1,0.94)
};
% Single marker for VBLL at 1 sample
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=green}] coordinates {
(1,0.9419845204867905) += (0,0.012282229562846944) -=(0,0.012282229562846944)
};
% Second plot (Dataset B)
\nextgroupplot[
title= \empty,
title style={yshift=-0.2cm},
ylabel={\tiny{PR-AUC}},
ymax=0.97,
ymin=0.78,
]
% Plot data for BBB in Dataset B
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=orange}] coordinates {
(5.0,0.8890952355627204) += (0,0.02047053526188267) -=(0,0.02047053526188267)
(10.0,0.8953625367929087) += (0,0.01246797746193798) -=(0,0.01246797746193798)
(15.0,0.9067725793580275) += (0,0.013519808461528654) -=(0,0.013519808461528654)
(20.0,0.8969385261953808) += (0,0.01036496620579844) -=(0,0.01036496620579844)
(25.0,0.9130454847262927) += (0,0.014946356767675757) -=(0,0.014946356767675757)
(30.0,0.8938326988036012) += (0,0.01201948574187212) -=(0,0.01201948574187212)
(35.0,0.8970527741214148) += (0,0.015264353856984263) -=(0,0.015264353856984263)
(40.0,0.9083495177397062) += (0,0.014328574100584288) -=(0,0.014328574100584288)
(45.0,0.9127346098767953) += (0,0.00698748386415729) -=(0,0.00698748386415729)
(50.0,0.9005882102907485) += (0,0.01466810260714856) -=(0,0.01466810260714856)
};
% Plot data for PE in Dataset B
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=cyan}] coordinates {
(5,0.5173564453152024) += (0,0.0034151745426027876) -=(0,0.0034151745426027876)
(10,0.5200962981234224) += (0,0.0035698188151590463) -=(0,0.0035698188151590463)
(15,0.5227057339988891) += (0,0.002515099308197771) -=(0,0.002515099308197771)
(20,0.5219544147257136) += (0,0.003423719114757041) -=(0,0.003423719114757041)
(25,0.5211473657965758) += (0,0.003501700680655848) -=(0,0.003501700680655848)
(30,0.525901883273335) += (0,0.0038507742294101183) -=(0,0.0038507742294101183)
(35,0.5207482599261907) += (0,0.0028456108169123067) -=(0,0.0028456108169123067)
(40,0.5214644177851622) += (0,0.004314115166299102) -=(0,0.004314115166299102)
(45,0.524882574287336) += (0,0.002941551600000084) -=(0,0.002941551600000084)
(50,0.5228059285888904) += (0,0.003947472099933566) -=(0,0.003947472099933566)
};
% Plot data for SE in Dataset B
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=red}] coordinates {
(5,0.8720259503349436) += (0,0.0019019623864402633) -=(0,0.0019019623864402633)
(10,0.8757909638851944) += (0,0.001515904940566373) -=(0,0.001515904940566373)
(15,0.8757736588520493) += (0,0.0013690623348630718) -=(0,0.0013690623348630718)
(20,0.8754886673420307) += (0,0.002493356066266395) -=(0,0.002493356066266395)
(25,0.8748057411117042) += (0,0.0014246995655760824) -=(0,0.0014246995655760824)
(30,0.8758860975688926) += (0,0.0010697960683935824) -=(0,0.0010697960683935824)
(35,0.8745137286778558) += (0,0.000931580980482559) -=(0,0.000931580980482559)
(40,0.8742627982872954) += (0,0.000859664409545465) -=(0,0.000859664409545465)
(45,0.8743424201502226) += (0,0.0005312108151598462) -=(0,0.0005312108151598462)
(50,0.874138561338419) += (0,0.001574229983756852) -=(0,0.001574229983756852)
};
% Plot data for LL--HMC in Dataset B
\addplot[mark=diamond*, thin, line width=0.25pt,
mark options={fill=yellow},
mark size=1.0pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=yellow}] coordinates {
(5,0.934234387079315) += (0,0.007482862024564887) -=(0,0.007482862024564887)
(10,0.9463293321330107) += (0,0.007834248512217627) -=(0,0.007834248512217627)
(15,0.9480493321202473) += (0,0.006899738096230493) -=(0,0.006899738096230493)
(20,0.9510705083455543) += (0,0.004403889758786524) -=(0,0.004403889758786524)
(25,0.9561847616749584) += (0,0.004935406059881822) -=(0,0.004935406059881822)
(30,0.9603580205893574) += (0,0.0027788470535201127) -=(0,0.0027788470535201127)
(35,0.9578881533621375) += (0,0.002279184518381539) -=(0,0.002279184518381539)
(40,0.9560222759258465) += (0,0.0030825319800790515) -=(0,0.0030825319800790515)
(45,0.9601004612890331) += (0,0.001813134265007449) -=(0,0.001813134265007449)
(50,0.9595021218847638) += (0,0.0020104443722019485) -=(0,0.0020104443722019485)
};
% Single marker for DE at 5 samples
\addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
(5,0.90)
};
% Single marker for Regular Model at 1 sample
\addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
(1,0.82)
};
% Single marker for VBLL at 1 sample
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
error bars/.cd,
y dir=both,
y explicit,
error bar style={line width=0.1pt, color=green}] coordinates {
(1,0.7971741487324044) += (0,0.013486337831006126) -=(0,0.013486337831006126)
};
\end{groupplot}
\node[font=\bfseries, rotate=90] at ($(group c1r1.west)+(-1.00cm,0)$) {\tiny {OOD min}};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.00cm,0)$) {OOD};
\node[font=\bfseries, rotate=90] at ($(group c1r1.west)+(-1.00cm,-3.2)$) {\tiny {OOD max}};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.00cm,-1.2)$) {OOD};
\end{tikzpicture}
% Legend
\vspace*{0.3cm}
\begin{tikzpicture}
% \hspace{3.5cm}
\begin{axis}[
hide axis,
xmin=0, xmax=1, ymin=0, ymax=1,
legend style={at={(0.0,0.0)}, anchor=center, legend columns=7},
legend entries={\tiny{BBB--LL},\tiny{PE--LL},\tiny{SE},\tiny{VBLL}, \tiny{LL--HMC},\tiny{DE},\tiny{Regular},}
]
\addlegendimage{mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=2.0pt}
\addlegendimage{mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2.0pt}
\addlegendimage{mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=2pt}
\addlegendimage{mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=2.0pt}
\addlegendimage{mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=2.0pt}
\addlegendimage{mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=2.0pt}
\addlegendimage{mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=2.0pt}
\end{axis}
\end{tikzpicture}
%\vspace{-0.2cm}
\caption{Influence of the number of samples or predictions on the OOD detection across 10 different random seeds for the average best-performing hyperparameter configuration of the included PDL methods. For each number of samples, the whiskers indicate two standard errors of the mean. Markers without visible whiskers indicate small standard errors. For the OOD max figures, the PE--LL is filtered out due to lower OOD performance.} %hyperparameter configurations of the LL--PDL methods for the OOD detection performance. The regular model (single prediction), DE (five predictions), and VBLL (single prediction) are only included once in each figure.
\label{fig:ood_comparison}
\end{figure*}
\definecolor{darkgray176}{RGB}{176,176,176}
\definecolor{steelblue31119180}{RGB}{31,119,180}
\begin{figure}[t!]
\centering
\begin{tikzpicture}
\begin{groupplot}[
group style={
group size=3 by 2, % Three figures in one row
horizontal sep=1.2cm,
vertical sep=0.3cm,
},
width=5.0cm,
height=4.0cm,
ybar,
enlargelimits=0.15,
ylabel={\tiny {PR-AUC}},
symbolic x coords={BBB--LL, SE, PE--LL, VBLL, LL--HMC},
xtick=data,
% ymin=0, ymax=1,
% ytick distance=0.2,
grid=major,
% ymajorgrids=true,
tick label style={font=\tiny},
%ylabel style={yshift=-0.2cm}, % Shift y-label down
xlabel style={yshift=0.1cm, rotate=60}, % Shift x-label down
xticklabel style={rotate=45, anchor=east, yshift=-0.2cm},
every node near coord/.append style={
yshift=3pt,
font=\footnotesize,
label style={font=\tiny},
y tick label style={
/pgf/number format/fixed,
/pgf/number format/precision=2,
/pgf/number format/fixed zerofill=true,
},
tick label style={font=\fontsize{4}{4}\selectfont},
},
]
% First subplot
\nextgroupplot[title={ \tiny \textbf{AIDE} \citep{yang2023aide}},
title style={yshift=-0.2cm}, xlabel=\empty, xticklabels=\empty,
ymin=0.17, ymax=0.52,]
\addplot[
fill=steelblue31119180!60,
draw=steelblue31119180,
error bars/.cd,
y dir=both,
y explicit,
] coordinates {
(SE, 0.18634) +- (0.001151,0.001151)
% (PE--LL, 0.518888) +- (0.001048,0.001048)
% (BBB--LL, 0.429683) +- (0.0241688, 0.0241688)
% (VBLL, 0.169609) +- (0.0050292,0.0050292)
% (LL--HMC, 0.327656) +- (0.0237336,0.0237336)
(BBB--LL, 0.43635) +- (0.01884,0.01884)
(PE--LL, 0.50373) +- (0.00009,0.00009)
(VBLL, 0.17674) +- (0.00578,0.00578)
(LL--HMC, 0.44611) +- (0.00939,0.00939)
% std
% (SE, 0.18634) +-(0.005755,0.005755)
% (PE--LL, 0.518888) +-(0.00524,0.00524)
% (BBB--LL, 0.429683) +-(0.120844, 0.120844)
% (VBLL, 0.169609) +-(0.025146 ,0.025146)
% (LL--HMC, 0.327656 ) +-(0.118668,0.118668)
};
% Second subplot
\nextgroupplot[title={ \tiny \textbf{B4C} \citep{jain2015car}},
title style={yshift=-0.2cm},
xlabel=\empty, xticklabels=\empty, ymin=0.43, ymax=0.55,]
\addplot[
fill=steelblue31119180!60,
draw=steelblue31119180,
error bars/.cd,
y dir=both,
y explicit,
] coordinates {
(BBB--LL, 0.458215) +- (0.0139062,0.0139062)
(SE, 0.517133) +- (0.0049198,0.0049198)
(PE--LL, 0.516577) +- (0.000563,0.000563)
(VBLL, 0.446607) +- (0.013007,0.013007)
(LL--HMC,0.495702) +- (0.0135222,0.0135222)
% std
% (BBB--LL, 0.458215) +-(0.069531,0.069531)
% (SE, 0.517133) +-(0.024599,0.024599)
% (PE--LL, 0.516577) +-(0.002815,0.002815)
% (VBLL, 0.446607) +-(0.065035,0.065035)
% (LL--HMC,0.495702) +-(0.06761,0.067611)
};
\nextgroupplot[title={ \tiny \textbf{ROAD} \citep{singh2022road}},
title style={yshift=-0.2cm}, ymin=0.24, ymax=0.58,]
\addplot[
fill=steelblue31119180!60,
draw=steelblue31119180,
error bars/.cd,
y dir=both,
y explicit,
] coordinates {
% (BBB--LL, 0.33296) +- (0.0089896,0.0089896)
(SE, 0.259985) +- (0.0002554,0.0002554)
% (BBB--LL, 0.43591) +- (0.00739,0.00739)
% (PE--LL, 0.51607) +- (0.00071,0.00071)
% (VBLL, 0.45094) +- (0.01485,0.01485)
% (LL--HMC, 0.52055) +- (0.00439,0.00439)
(BBB--LL, 0.30883) +- (0.00871,0.00871)
(PE--LL, 0.52448) +- (0.00238,0.00238)
(VBLL, 0.2627) +- (0.00814,0.00814)
(LL--HMC, 0.38528) +- (0.00623,0.00623)
% (PE--LL, 0.52891) +- (0.0006278,0.0006278)
% (VBLL, 0.25524) +- (0.0057584,0.0057584)
% (LL--HMC, 0.555987) +- (0.0232446,0.0232446)
%std
% (BBB--LL, 0.33296) +-(0.044948,0.044948)
% (SE, 0.259985) +-(0.001277,0.001277)
% (PE--LL, 0.52891) +-( 0.003139,0.003139)
% (VBLL, 0.25524) +-( 0.028792,0.028792)
% (LL--HMC, 0.555987) +-(0.116223,0.116223)
};
% First subplot
\nextgroupplot[title=\empty,
title style={yshift=-0.2cm},
ymin=0.55, ymax=1,]
\addplot[
fill=steelblue31119180!60,
draw=steelblue31119180,
error bars/.cd,
y dir=both,
y explicit,
] coordinates {
(SE, 0.955661) +- (0.0010662,0.0010662)
% (PE--LL, 0.529377) +- (0.000574, 0.000574)
% (BBB--LL, 0.97792) +- (0.0016466,0.0016466)
% (VBLL, 0.941438) +- (0.0035404, 0.0035404)
% (LL--HMC, 0.961973) +- (0.0029172,0.0029172)
(BBB--LL, 0.93663) +- (0.00173,0.00173)
(PE--LL, 0.50766) +- (0.00021,0.00021)
(VBLL, 0.93656) +- (0.0032,0.0032)
(LL--HMC, 0.96756) +- (0.00072,0.00072)
%std
% (SE, 0.955661) +-(0.005331,0.005331)
% (PE--LL, 0.529377) +-(0.00287, 0.00287)
% (BBB--LL, 0.97792) +-(0.008233,0.008233)
% (VBLL, 0.941438) +-(0.017702, 0.017702)
% (LL--HMC, 0.961973) +-(0.014586,0.014586)
};
% Second subplot
\nextgroupplot[title=\empty,
title style={yshift=-0.2cm},
ymin=0.55, ymax=1,]
\addplot[
fill=steelblue31119180!60,
draw=steelblue31119180,
error bars/.cd,
y dir=both,
y explicit,
] coordinates {
% (BBB--LL, 0.864666) +- (0.0044282,0.0044282)
(SE,0.869604) +- (0.001336,0.001336)
(BBB--LL, 0.86651) +- (0.00418,0.00418)
(PE--LL, 0.51854) +- (0.00046,0.00046)
(VBLL, 0.75418) +- (0.0063,0.0063)
(LL--HMC, 0.84314) +- (0.00818,0.00818)
% (PE--LL, 0.519402) +- (0.0003556,0.0003556)
% (VBLL, 0.783961) +- (0.0061452,0.0061452)
% (LL--HMC, 0.950973) +- (0.00431,0.00431)
% (BBB--LL, 0.864666) +-(0.022141,0.022141)
% (SE,0.869604) +-(0.00668,0.00668)
% (PE--LL, 0.519402) +-(0.001778,0.001778)
% (VBLL, 0.783961) +-(0.030726,0.030726)
% (LL--HMC, 0.950973) +-(0.02155,0.02155)
};
\end{groupplot}
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-0.9cm,0.0)$) {OOD min};
\node[font=\bfseries, rotate=90] at ($(group c1r1.west)+(-1.10cm,0)$) {\tiny {OOD min}};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.00cm,0)$) {OOD};
\node[font=\bfseries, rotate=90] at ($(group c1r1.west)+(-1.10cm,-2.7cm)$) {\tiny {OOD max}};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.00cm,-1.2)$) {OOD};
\end{tikzpicture}
\caption{Average uncertainty-based OOD detection performance (PR-AUC) across the included LL--PDL methods for the best performing hyperparameter configuration across 100 random seeds.}
\label{fig:placeholder_groupplot}
\end{figure}
\begin{figure}[t]
\centering
\begin{tikzpicture}
\begin{groupplot}[
group style={
group size=3 by 1,
horizontal sep=1.2cm,
vertical sep=1.0cm,
},
width=5.5cm,
height=4.0cm,
grid=major,
xlabel={\tiny{Number of samples}},
xtick={2,5,10,15,20,25,30,35,40,45,50},
xmin=2,
xmax=50,
tick label style={font=\tiny},
label style={font=\small},
y tick label style={
/pgf/number format/fixed,
/pgf/number format/precision=2,
/pgf/number format/fixed zerofill=true,
},
% ylabel style={yshift=-0.2cm},
xlabel style={yshift=0.1cm},
]
\nextgroupplot[
title={\tiny {\textbf{AIDE} \citep{yang2023aide}}},
ylabel={\tiny{PR-AUC}},
title style={yshift=-0.2cm},
ymax=0.3,
ymin=0.2
]
\path [fill=steelblue31119180, fill opacity=0.2]
(axis cs:2,0.20082696612975)
--(axis cs:2,0.20082696612975)
--(axis cs:3,0.206269136026063)
--(axis cs:4,0.211883652161684)
--(axis cs:5,0.216906773173809)
--(axis cs:6,0.221433673101807)
--(axis cs:7,0.224979655732907)
--(axis cs:8,0.229382069005079)
--(axis cs:9,0.229767420802002)
--(axis cs:10,0.229355053070612)
--(axis cs:11,0.231222586786899)
--(axis cs:12,0.234677046653289)
--(axis cs:13,0.237397688913769)
--(axis cs:14,0.23862981623995)
--(axis cs:15,0.237755315125637)
--(axis cs:16,0.23780934081883)
--(axis cs:17,0.238106650987553)
--(axis cs:18,0.239101091432751)
--(axis cs:19,0.240559812380572)
--(axis cs:20,0.240419581962129)
--(axis cs:21,0.240742894576274)
--(axis cs:22,0.241704158679544)
--(axis cs:23,0.243020111748291)
--(axis cs:24,0.243840423565301)
--(axis cs:25,0.244065060984747)
--(axis cs:26,0.243177961002371)
--(axis cs:27,0.243099436840492)
--(axis cs:28,0.244526920061491)
--(axis cs:29,0.244624914698632)
--(axis cs:30,0.244895391983439)
--(axis cs:31,0.245602701877174)
--(axis cs:32,0.245082200393341)
--(axis cs:33,0.246141289139533)
--(axis cs:34,0.247118844682639)
--(axis cs:35,0.247634315603031)
--(axis cs:36,0.248008790761979)
--(axis cs:37,0.248823552300667)
--(axis cs:38,0.249464297576278)
--(axis cs:39,0.249881043162904)
--(axis cs:40,0.250754016221559)
--(axis cs:41,0.251222522614405)
--(axis cs:42,0.250764158966271)
--(axis cs:43,0.251362445393215)
--(axis cs:44,0.251262416340393)
--(axis cs:45,0.251625930786636)
--(axis cs:46,0.251370426302328)
--(axis cs:47,0.251217220464025)
--(axis cs:48,0.251526597033637)
--(axis cs:49,0.251361473495114)
--(axis cs:50,0.251747240973589)
--(axis cs:50,0.282507634660928)
--(axis cs:50,0.282507634660928)
--(axis cs:49,0.282275325499589)
--(axis cs:48,0.282602678489762)
--(axis cs:47,0.282452290992835)
--(axis cs:46,0.282762859865336)
--(axis cs:45,0.283171461006649)
--(axis cs:44,0.282945849698275)
--(axis cs:43,0.28318279325004)
--(axis cs:42,0.282710106972456)
--(axis cs:41,0.283309634021306)
--(axis cs:40,0.282957039543784)
--(axis cs:39,0.282197711588964)
--(axis cs:38,0.281931152516512)
--(axis cs:37,0.281461472695229)
--(axis cs:36,0.280840139220452)
--(axis cs:35,0.280673631384622)
--(axis cs:34,0.280360933294929)
--(axis cs:33,0.279597117801791)
--(axis cs:32,0.278808820793172)
--(axis cs:31,0.279623642640749)
--(axis cs:30,0.279187601038192)
--(axis cs:29,0.279236370741613)
--(axis cs:28,0.279467523009777)
--(axis cs:27,0.278357732047506)
--(axis cs:26,0.278784093261736)
--(axis cs:25,0.27999135461144)
--(axis cs:24,0.279995848494187)
--(axis cs:23,0.279420136385134)
--(axis cs:22,0.278264237011443)
--(axis cs:21,0.277432611950079)
--(axis cs:20,0.277301988292157)
--(axis cs:19,0.277601771213806)
--(axis cs:18,0.276088429966025)
--(axis cs:17,0.275126420971812)
--(axis cs:16,0.275002956352188)
--(axis cs:15,0.274993269909988)
--(axis cs:14,0.27578080585786)
--(axis cs:13,0.274095527693655)
--(axis cs:12,0.270925392998617)
--(axis cs:11,0.26680326334707)
--(axis cs:10,0.265692734417512)
--(axis cs:9,0.266897081631096)
--(axis cs:8,0.266827652600438)
--(axis cs:7,0.262152958243399)
--(axis cs:6,0.259282092471034)
--(axis cs:5,0.254636287753316)
--(axis cs:4,0.245409408343973)
--(axis cs:3,0.237948204454689)
--(axis cs:2,0.20082696612975)
--cycle;
\addplot [semithick, steelblue31119180]
table {%
2 0.20082696612975
3 0.222108670240376
4 0.228646530252828
5 0.235771530463563
6 0.240357882786421
7 0.243566306988153
8 0.248104860802758
9 0.248332251216549
10 0.247523893744062
11 0.249012925066984
12 0.252801219825953
13 0.255746608303712
14 0.257205311048905
15 0.256374292517812
16 0.256406148585509
17 0.256616535979683
18 0.257594760699388
19 0.259080791797189
20 0.258860785127143
21 0.259087753263177
22 0.259984197845493
23 0.261220124066713
24 0.261918136029744
25 0.262028207798094
26 0.260981027132053
27 0.260728584443999
28 0.261997221535634
29 0.261930642720123
30 0.262041496510816
31 0.262613172258962
32 0.261945510593256
33 0.262869203470662
34 0.263739888988784
35 0.264153973493827
36 0.264424464991216
37 0.265142512497948
38 0.265697725046395
39 0.266039377375934
40 0.266855527882671
41 0.267266078317855
42 0.266737132969363
43 0.267272619321628
44 0.267104133019334
45 0.267398695896642
46 0.267066643083832
47 0.26683475572843
48 0.2670646377617
49 0.266818399497351
50 0.267127437817259
};
\nextgroupplot[
title={\tiny {\textbf{B4C} \citep{jain2015car}}},
ylabel={\tiny{PR-AUC}},
title style={yshift=-0.2cm},
ymin=0.45,
ymax=0.55
]
\path [fill=steelblue31119180, fill opacity=0.2]
(axis cs:2,0.472049297050022)
--(axis cs:2,0.472049297050022)
--(axis cs:3,0.476261201376686)
--(axis cs:4,0.481034184563371)
--(axis cs:5,0.482228398719696)
--(axis cs:6,0.479344463150629)
--(axis cs:7,0.47854270436869)
--(axis cs:8,0.478588663804916)
--(axis cs:9,0.478332621535614)
--(axis cs:10,0.479846845030431)
--(axis cs:11,0.480268591565185)
--(axis cs:12,0.481720124114031)
--(axis cs:13,0.484502206630871)
--(axis cs:14,0.486774456943587)
--(axis cs:15,0.490105304636759)
--(axis cs:16,0.492052585886778)
--(axis cs:17,0.491745423657405)
--(axis cs:18,0.492734341346875)
--(axis cs:19,0.49509066973402)
--(axis cs:20,0.494116411552949)
--(axis cs:21,0.494621710796874)
--(axis cs:22,0.495533139519828)
--(axis cs:23,0.495936464959795)
--(axis cs:24,0.496708724883093)
--(axis cs:25,0.495988096660825)
--(axis cs:26,0.494508046637702)
--(axis cs:27,0.494589347443993)
--(axis cs:28,0.494180917937972)
--(axis cs:29,0.494288734680759)
--(axis cs:30,0.493670548192374)
--(axis cs:31,0.494991130571111)
--(axis cs:32,0.495288904440143)
--(axis cs:33,0.495142404321849)
--(axis cs:34,0.496047927061433)
--(axis cs:35,0.495276188033123)
--(axis cs:36,0.494932011555301)
--(axis cs:37,0.495791314663435)
--(axis cs:38,0.496083816534049)
--(axis cs:39,0.497194626363594)
--(axis cs:40,0.497564907246973)
--(axis cs:41,0.497710426531636)
--(axis cs:42,0.497396992848516)
--(axis cs:43,0.498208835378659)
--(axis cs:44,0.498626257115228)
--(axis cs:45,0.499002288890598)
--(axis cs:46,0.499173174819543)
--(axis cs:47,0.499628830553708)
--(axis cs:48,0.49963994022018)
--(axis cs:49,0.499858610816221)
--(axis cs:50,0.500258377256011)
--(axis cs:50,0.530517185596643)
--(axis cs:50,0.530517185596643)
--(axis cs:49,0.530223204246891)
--(axis cs:48,0.530110484828306)
--(axis cs:47,0.530222024648719)
--(axis cs:46,0.529896538009646)
--(axis cs:45,0.52985841251074)
--(axis cs:44,0.529586503211321)
--(axis cs:43,0.529283883172318)
--(axis cs:42,0.528579058206734)
--(axis cs:41,0.529020881685359)
--(axis cs:40,0.528995959546056)
--(axis cs:39,0.528739147345399)
--(axis cs:38,0.527737485488303)
--(axis cs:37,0.527557564154529)
--(axis cs:36,0.526780126970904)
--(axis cs:35,0.52715877825787)
--(axis cs:34,0.527962244261834)
--(axis cs:33,0.527055300455652)
--(axis cs:32,0.527140492802301)
--(axis cs:31,0.526712543239267)
--(axis cs:30,0.525161663449809)
--(axis cs:29,0.525427515690935)
--(axis cs:28,0.524815756293175)
--(axis cs:27,0.524726832586691)
--(axis cs:26,0.524112331154937)
--(axis cs:25,0.524925025353184)
--(axis cs:24,0.525048939577477)
--(axis cs:23,0.523732678625799)
--(axis cs:22,0.522806076120896)
--(axis cs:21,0.521467437632591)
--(axis cs:20,0.520436072041672)
--(axis cs:19,0.520920168575401)
--(axis cs:18,0.51802721113951)
--(axis cs:17,0.51660774494)
--(axis cs:16,0.516364032836584)
--(axis cs:15,0.513651244382871)
--(axis cs:14,0.50941663882394)
--(axis cs:13,0.5063141550256)
--(axis cs:12,0.50263181464334)
--(axis cs:11,0.501048421568952)
--(axis cs:10,0.49982715490037)
--(axis cs:9,0.498365678627083)
--(axis cs:8,0.499782573011329)
--(axis cs:7,0.501324797834761)
--(axis cs:6,0.503736318452476)
--(axis cs:5,0.507423315083751)
--(axis cs:4,0.505138595464774)
--(axis cs:3,0.496598074474388)
--(axis cs:2,0.472049297050022)
--cycle;
\addplot [semithick, steelblue31119180]
table {%
2 0.472049297050022
3 0.486429637925537
4 0.493086390014073
5 0.494825856901724
6 0.491540390801552
7 0.489933751101726
8 0.489185618408122
9 0.488349150081348
10 0.489836999965401
11 0.490658506567068
12 0.492175969378685
13 0.495408180828236
14 0.498095547883763
15 0.501878274509815
16 0.504208309361681
17 0.504176584298703
18 0.505380776243192
19 0.508005419154711
20 0.50727624179731
21 0.508044574214732
22 0.509169607820362
23 0.509834571792797
24 0.510878832230285
25 0.510456561007005
26 0.50931018889632
27 0.509658090015342
28 0.509498337115573
29 0.509858125185847
30 0.509416105821092
31 0.510851836905189
32 0.511214698621222
33 0.51109885238875
34 0.512005085661633
35 0.511217483145497
36 0.510856069263103
37 0.511674439408982
38 0.511910651011176
39 0.512966886854497
40 0.513280433396514
41 0.513365654108497
42 0.512988025527625
43 0.513746359275489
44 0.514106380163274
45 0.514430350700669
46 0.514534856414595
47 0.514925427601214
48 0.514875212524243
49 0.515040907531556
50 0.515387781426327
};
\nextgroupplot[
title={\tiny {\textbf{ROAD} \citep{singh2022road}}},
ylabel={\tiny{PR-AUC}},
title style={yshift=-0.2cm},
ymax=0.8,
ymin=0.2
]
\path [fill=steelblue31119180, fill opacity=0.2]
(axis cs:2,0.275117458168383)
--(axis cs:2,0.275117458168383)
--(axis cs:3,0.299131309279073)
--(axis cs:4,0.327027154713565)
--(axis cs:5,0.348331343190482)
--(axis cs:6,0.365331262217934)
--(axis cs:7,0.37701779377515)
--(axis cs:8,0.392452469202735)
--(axis cs:9,0.394536265982295)
--(axis cs:10,0.406819436640144)
--(axis cs:11,0.417704942452028)
--(axis cs:12,0.423955361639726)
--(axis cs:13,0.431453231149226)
--(axis cs:14,0.445770207434816)
--(axis cs:15,0.452742206752329)
--(axis cs:16,0.458391293814999)
--(axis cs:17,0.466381475039342)
--(axis cs:18,0.468769023439449)
--(axis cs:19,0.47527522999373)
--(axis cs:20,0.47496052442839)
--(axis cs:21,0.483302360218524)
--(axis cs:22,0.482551878874507)
--(axis cs:23,0.493938066885996)
--(axis cs:24,0.494879067600715)
--(axis cs:25,0.49722744215485)
--(axis cs:26,0.498928557588683)
--(axis cs:27,0.504679817827822)
--(axis cs:28,0.509337051021471)
--(axis cs:29,0.508729026365266)
--(axis cs:30,0.510887727587358)
--(axis cs:31,0.515269090353243)
--(axis cs:32,0.516193283278065)
--(axis cs:33,0.523002974354718)
--(axis cs:34,0.5294977098895)
--(axis cs:35,0.531325459536842)
--(axis cs:36,0.535831587746147)
--(axis cs:37,0.53759735167599)
--(axis cs:38,0.538815846118453)
--(axis cs:39,0.539126663983706)
--(axis cs:40,0.549960344295658)
--(axis cs:41,0.550235141344492)
--(axis cs:42,0.551503733911316)
--(axis cs:43,0.554787416235092)
--(axis cs:44,0.552842689191243)
--(axis cs:45,0.551202459442872)
--(axis cs:46,0.553795949443208)
--(axis cs:47,0.558824036137439)
--(axis cs:48,0.558321515966149)
--(axis cs:49,0.559992478578359)
--(axis cs:50,0.560753277949136)
--(axis cs:50,0.739484979625226)
--(axis cs:50,0.739484979625226)
--(axis cs:49,0.738445089927039)
--(axis cs:48,0.736430311305528)
--(axis cs:47,0.73654185852547)
--(axis cs:46,0.730993136169602)
--(axis cs:45,0.728010530954271)
--(axis cs:44,0.72925754556456)
--(axis cs:43,0.730599110556421)
--(axis cs:42,0.72644468415384)
--(axis cs:41,0.7243506044773)
--(axis cs:40,0.723116326619058)
--(axis cs:39,0.711199292352092)
--(axis cs:38,0.710384985480261)
--(axis cs:37,0.708539041710656)
--(axis cs:36,0.706043218285911)
--(axis cs:35,0.700726059860097)
--(axis cs:34,0.698193608849744)
--(axis cs:33,0.690866105435825)
--(axis cs:32,0.683476791294466)
--(axis cs:31,0.682195793690637)
--(axis cs:30,0.677323439519928)
--(axis cs:29,0.674701162060829)
--(axis cs:28,0.67466380122864)
--(axis cs:27,0.668954249414169)
--(axis cs:26,0.66225025901629)
--(axis cs:25,0.659797532353257)
--(axis cs:24,0.656286490981316)
--(axis cs:23,0.653939018278663)
--(axis cs:22,0.64063048869585)
--(axis cs:21,0.640480437120543)
--(axis cs:20,0.630585188448819)
--(axis cs:19,0.629782142171948)
--(axis cs:18,0.621359802630562)
--(axis cs:17,0.617290421665991)
--(axis cs:16,0.606991530729663)
--(axis cs:15,0.598921927524825)
--(axis cs:14,0.589264514800586)
--(axis cs:13,0.571840147998813)
--(axis cs:12,0.562818705454511)
--(axis cs:11,0.55581548392746)
--(axis cs:10,0.543613459714645)
--(axis cs:9,0.530493025248699)
--(axis cs:8,0.528564828077903)
--(axis cs:7,0.510183824350839)
--(axis cs:6,0.498245465522197)
--(axis cs:5,0.480065744867161)
--(axis cs:4,0.453490057613887)
--(axis cs:3,0.415080439351548)
--(axis cs:2,0.275117458168383)
--cycle;
\addplot [semithick, steelblue31119180]
table {%
2 0.275117458168383
3 0.357105874315311
4 0.390258606163726
5 0.414198544028822
6 0.431788363870066
7 0.443600809062995
8 0.460508648640319
9 0.462514645615497
10 0.475216448177395
11 0.486760213189744
12 0.493387033547118
13 0.501646689574019
14 0.517517361117701
15 0.525832067138577
16 0.532691412272331
17 0.541835948352666
18 0.545064413035006
19 0.552528686082839
20 0.552772856438605
21 0.561891398669534
22 0.561591183785179
23 0.573938542582329
24 0.575582779291016
25 0.578512487254054
26 0.580589408302486
27 0.586817033620995
28 0.592000426125055
29 0.591715094213048
30 0.594105583553643
31 0.59873244202194
32 0.599835037286266
33 0.606934539895271
34 0.613845659369622
35 0.61602575969847
36 0.620937403016029
37 0.623068196693323
38 0.624600415799357
39 0.625162978167899
40 0.636538335457358
41 0.637292872910896
42 0.638974209032578
43 0.642693263395756
44 0.641050117377902
45 0.639606495198571
46 0.642394542806405
47 0.647682947331454
48 0.647375913635838
49 0.649218784252699
50 0.650119128787181
};
\end{groupplot}
\end{tikzpicture}
\caption{Effects of additional dependent samples from the same chain of the top-performing LL--HMC configuration for the OOD min scenario for different 100 random seeds. The light blue area illustrates the standard deviation of the cumulative of the PR-AUC for each additional sample.}
\label{fig:dependent_samples_ood}
\end{figure}
\begin{table*}[!ht]
\centering
\caption{Average OOD top-performance and two standard errors of the mean across five random seeds for LL--HMC with multiple starting positions or chains. S=Starting Positions, C=Chains.}
%LL--HMC DAR and DIR OOD detection performance for multiple starting positions and chains.
%\vspace{-0.2cm}
\resizebox{\linewidth}{!}{
\begin{tabular}{lcc|c|c|c|c|c|c||c|c|c|c|c|c||c|c|c}
\toprule
\multicolumn{3}{c|}{} & \multicolumn{6}{c||}{\textbf{AIDE} \citep{yang2023aide}} & \multicolumn{6}{c||}{\textbf{B4C} \citep{jain2015car}} & \multicolumn{3}{c}{\textbf{ROAD} \citep{singh2022road}} \\ \cmidrule{4-18}
\multicolumn{3}{c|}{} & \multicolumn{3}{c|}{\textbf{OOD min}} & \multicolumn{3}{c||}{\textbf{OOD max}} & \multicolumn{3}{c|}{\textbf{OOD min}} & \multicolumn{3}{c||}{\textbf{OOD max}} & \multicolumn{3}{c}{\textbf{OOD min}} \\
\multicolumn{3}{c|}{} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
\begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} \\
\midrule
\multirow{4}{*}{\textbf{LL--HMC}}& \multirow{2}{*}{\small{S=1}} & \small{C=1} & 0.90$\pm$0.01 & 0.52$\pm$0.01 & 0.21$\pm$0.02 & 0.95$\pm$0.00 & 0.99$\pm$0.00 & 0.14$\pm$0.01 & 0.83$\pm$0.01 & 0.70$\pm$0.01 & 0.41$\pm$0.03 & 0.91$\pm$0.01 & 0.97$\pm$0.00 & 0.39$\pm$0.05 & 0.86$\pm$0.01 & 0.73$\pm$0.02 & 0.36$\pm$0.05 \\
& & \small{C=2} & 0.91$\pm$0.03 & 0.58$\pm$0.10 & 0.18$\pm$0.05 & 0.95$\pm$0.00 & 0.98$\pm$0.00 & 0.15$\pm$0.00 & 0.85$\pm$0.04 & 0.73$\pm$0.05 & 0.36$\pm$0.08 & 0.85$\pm$0.04 & 0.73$\pm$0.05 & 0.36$\pm$0.08 & 0.87$\pm$0.01 & 0.72$\pm$0.01 & 0.31$\pm$0.04 \\ \cmidrule{2-18}
% \midrule
& \multirow{2}{*}{\small{S=2}} & \small{C=1} & 0.64$\pm$0.03 & 0.35$\pm$0.02 & 0.97$\pm$0.03 & 0.89$\pm$0.03 & 0.97$\pm$0.01 & 0.40$\pm$0.18 & 0.85$\pm$0.01 & 0.70$\pm$0.02 & 0.34$\pm$0.03 & 0.90$\pm$0.01 & 0.96$\pm$0.01 & 0.42$\pm$0.06 & 0.38$\pm$0.05 & 0.39$\pm$0.01 & 1.00$\pm$0.00 \\
& & \small{C=2} & 0.66$\pm$0.07 & 0.35$\pm$0.01 & 0.89$\pm$0.21 & 0.88$\pm$0.01 & 0.97$\pm$0.00 & 0.42$\pm$0.06 & 0.85$\pm$0.00 & 0.70$\pm$0.02 & 0.34$\pm$0.03 & 0.89$\pm$0.01 & 0.96$\pm$0.00 & 0.39$\pm$0.06 & 0.50$\pm$0.06 & 0.42$\pm$0.01 & 0.99$\pm$0.01 \\
\bottomrule
\end{tabular}
}
\label{tab:ood_multi_llhmc}
%\vspace{-0.1cm}
\end{table*}
%This underscores that there is not one single best method among the evaluated PDL approaches.
\paragraph{\textbf{Influence of additional predictions or samples.}} Figure \ref{fig:ood_comparison} visualizes the influence of additional samples or predictions for the best performing hyperparameter configurations from Table \ref{tab:road_ood_exp_hp} for 10 random seeds for the different numbers of samples or predictions. The BBB--LL results for the OOD min scenarios for the AIDE and B4C datasets show a slight improvement with additional samples, but this trend is not visible for the other scenarios or for the ROAD dataset. Opposed to the results in Table \ref{tab:road_ood_exp}, LL--HMC for 10 different random seeds for each number of samples does not yield the highest performance on average for all scenarios. Similar to the in-distribution results, there is no single LL--PDL method consistently performs better in all scenarios.
\paragraph{\textbf{Comparing the best hyperparameter configurations. }} Figure \ref{fig:placeholder_groupplot} shows the average PR-AUC across 100 random seeds for the best hyperparameter configurations from Table \ref{tab:road_ood_exp_hp}. The OOD detection performance of the PE--LL and SE is stable across scenarios, with average PR-AUC values comparable to those reported in Table \ref{tab:road_ood_exp}. In contrast, BBB--LL and LL--HMC exhibit greater variability in average performance across seeds and configurations depending on the OOD scenario. Overall, evaluating the top-performing configurations across multiple seeds suggests that no single LL--PDL method consistently outperforms the others.
\paragraph{\textbf{Influence of additional dependent LL--HMC samples.}}
Figure \ref{fig:dependent_samples_ood} shows how additional samples from the same HMC chain affect PR-AUC performance. The results are based on the best-performing grid search configuration. Compared to the modest F1-score gains in Figure \ref{fig:dependent_samples}, PR-AUC improves slightly more with additional same-chain samples in the OOD min scenario. However, the PR-AUC improvement stabilizes with more samples.
\paragraph{\textbf{Multiple chains and starting positions.}}
Table \ref{tab:ood_multi_llhmc} reports OOD detection performance using the top-performing grid search configuration across five seeds, comparing different numbers of HMC chains (C) and starting positions (S). LL--HMC with a single starting position consistently yields strong OOD detection performance across the datasets, regardless of the number of chains. In contrast, using two starting positions results in less consistent performance. Specifically, in the AIDE OOD min and ROAD OOD min scenarios, LL--HMC with two starting positions produces higher FPR95 values, indicating reduced reliability in distinguishing in- and out-of-distribution instances. This suggests LL--HMC’s performance is sensitive to the learned latent representations, especially in how they guide the initialization and exploration of posterior modes. As with the in-distribution classification performance, increasing the number of chains or starting positions does not consistently improve OOD detection, and in some cases degrades performance.
\section{Discussion and Conclusion}
We evaluated and explored the effectiveness of applying Hamiltonian Monte Carlo (HMC) sampling to the final layer of deep neural networks for video-based driver action and intention recognition. Applying HMC to sample the parameters of the last layer reduces the required computational cost compared to sampling the entire network. While last layer HMC (LL--HMC) benefits from the Markov-Chain Monte Carlo sampling ability to explore complex and potentially multi-modal posterior distributions for the last layer's parameters, its reliance on fixed pre-trained representations comes at the cost of not capturing the uncertainty inherent in the feature extraction layers of a deep neural network (DNN). Furthermore, as already indicated by the toy example, increasing the number of posterior samples improves the uncertainty estimations but saturates after a certain point.
Compared to the softmax-based baseline, the last layer probabilistic deep learning (PDL) approaches demonstrated improved performance under specific hyperparameter settings. However, the classification and OOD detection results across grid searches yielded significantly lower performance dropped compared to the best performance for a single grid search. The effect of additional predictions by evaluating the best performing hyperparameter configuration across 100 random seeds for each last layer PDL method and dataset showed that LL--HMC yields competitive results. Additional samples or predictions did not consistently enhance the in-distribution or OOD performance for the included last layer PDL approaches. %Moreover, the evaluation of the top-10 hyperparameter configurations across random seeds revealed no consistently superior LL--PDL method across the datasets.
While driving action and intention recognition are examples of realistic safety-critical applications, last layer PDL approaches can also be applied in other domains. For example, to support human-robot interaction through action and intention recognition \citep{qi2024computer}, elderly care \citep{pereira2009elder}, predictive maintenance \citep{benker2021utilizing,nguyen2022probabilistic}, medical imaging \citep{zou2023review}, and fraud detection \citep{habibpour2023uncertainty}. These tasks require different modeling approaches, which could provide deeper insights into how the underlying model representations affect the last layer uncertainty estimation performance.
% While the driving action and intention recognition are examples of realistic safety-critical applications, it is also possible to use last layer PDL approaches in other domains. For example, video-based action and intention recognition can support in human-robot interaction \cite{qi2024computer}, elderly care \cite{pereira2009elder}, predictive maintenance \cite{benker2021utilizing,nguyen2022probabilistic}, medical imaging \cite{zou2023review}, or fraud detection \cite{habibpour2023uncertainty}. By considering different problems, it would be interesting to evaluate the last layer PDL methods based on different underlying models responsible for the latent representations.
None of the last layer PDL methods were able to consistently identify all OOD instances when the most or least prevailing driving maneuvers were removed from the datasets. This raises broader considerations for the design of safety-critical systems such as advanced driver assistance and autonomous driving technologies. For instance, when a model predicts two maneuvers (e.g., ‘\textit{left turn}’ vs. ‘\textit{U-turn}’) with similar high confidence and low uncertainty, the system defaults to pick the class with the highest assigned confidence, despite the potential overlap. In such a scenario, it could be interesting to see whether predicting sets \citep{hullermeier2022quantification,wang2024credal} of potential maneuvers improves the performance. More broadly, if PDL methods are unable to reliably flag all OOD scenarios, we should also consider the potential outcome of the recommendations given the current driving scene. By leveraging a more decision-theoretic approach that incorporates generic task knowledge (e.g., penalizing intended maneuvers that potentially harm other road users), we might be able to produce better and safer recommendations \citep{cobb2018loss}.
% \clearpage
\bibliography{tmlr}
\bibliographystyle{tmlr}
\clearpage
\appendix
\section*{Appendices}
\section{Intermediate latent representation performance}
% \subsubsection{Intermediate latent representation performance}
\begin{figure}[h]
% \centering
% First row
% \begin{minipage}[t]{0.48\linewidth}
% \centering
% \includegraphics[trim={0cm 8.5cm 16.5cm 0cm},clip, width=0.58\textwidth, keepaspectratio]{llhmc_thesis.pdf}
% % \vspace{-0.5cm}
% \caption{Schematic overview of the two last layer HMC phases. Phase 1 covers the regular optimization-based training phase. During phase 2, the frozen pre-trained encoder is responsible for extracting the latent representations $z$. Then, the HMC is applied to sample the parameters of the last layer $\theta_{LL}$.}\label{fig:concept}
% \end{minipage}
% \hfill
% \begin{minipage}[t]{0.48\linewidth}
\centering
\includegraphics[trim={0.4cm 9.2cm 18.0cm 0cm},clip, width=0.50\textwidth, keepaspectratio]{intermediate_ens.pdf}% Add
% \captionsetup{font=tiny}
\caption{Schematic intermediate representation classification overview. The intermediate latent representation $z_{int}^{i}$ can be extracted at different depths from the encoder, and used to produce intermediate classifications. }\label{fig:intermed}
% \end{minipage}
\end{figure}
% \begin{tcolorbox}[
% colback=white, % Background color of the box
% colframe=black, % Border color of the box
% arc=4pt, % Rounded corners
% boxrule=1pt, % Border width
% left=6pt, % Left padding
% right=6pt, % Right padding
% top=2pt, % Top padding
% bottom=2pt, % Bottom padding
% width=0.98\linewidth % Width of the box
% ]
% \textit{LL-PDL predictions based on the intermediate representations do not improve the performance.}
% \end{tcolorbox}
\begin{figure*}[ht]
\centering
\begin{tikzpicture}
\begin{groupplot}[
group style={
group size=3 by 5, % Changed to 3 by 3 for three rows
horizontal sep=0.88cm,
vertical sep=0.5cm, % Decreased vertical separation for rows
},
width=5.5cm, % Keep the same width
height=4.00cm, % Decreased height
grid=major,
xlabel={ViT Transformer block number},
xtick={1,2,3,4,5,6,7,8,9,10,11,12(LL)},
tick label style={font=\tiny},
label style={font=\small},
y tick label style={
/pgf/number format/fixed,
/pgf/number format/precision=0,
/pgf/number format/fixed zerofill=true,
},
tick label style={font=\fontsize{4}{4}\selectfont},
ylabel style={yshift=-0.1cm}, % Shift y-label down
xlabel style={yshift=0.1cm}, % Shift x-label down
title style={}, % Empty title style
]
% ROW 1: Single predictions with Row Title
% Add row title for single predictions (rotated like y-axis)
\node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.1cm,-0.0)$) {BBB--LL};
\node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.1cm,-2.9cm)$) {PE--LL};
\node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.1cm,-5.8cm)$) {SE};
\node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.1cm,-8.8cm)$) {VBLL};
\node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.1cm,-11.7cm)$) {LL--HMC};
\node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-0.9cm,-11.7cm)$) {(S=1,C=2)};
% First plot (Accuracy for AIDE Dataset)
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=10,
ymax=90, % Adjusted for AIDE dataset accuracy values
extra y ticks={83.70},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 34.80756)
(2, 41.61931)
(3, 34.48399)
(4, 38.3959)
(5, 43.11464)
(6, 52.05468)
(7, 64.20065)
(8, 68.55294)
(9, 76.66792)
(10, 79.38153)
(11, 78.11334)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 34.80756)
(2, 39.70194)
(3, 35.578)
(4, 37.04696)
(5, 37.90946)
(6, 39.58183)
(7, 43.2181)
(8, 48.90854)
(9, 58.40118)
(10, 63.37307)
(11, 67.82723)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 34.80756)
% (2, 40.08924)
% (3, 35.66923)
% (4, 37.13679)
% (5, 40.16181)
% (6, 40.28478)
% (7, 45.43688)
% (8, 54.18311)
% (9, 62.56105)
% (10, 68.63499)
% (11, 75.64332)
% };
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=0,
ymax=100, % Adjusted for AIDE dataset accuracy values
extra y ticks={95.00},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 14.56977)
(2, 29.41549)
(3, 25.97255)
(4, 31.81229)
(5, 36.04545)
(6, 42.5057)
(7, 50.27513)
(8, 52.52793)
(9, 62.81554)
(10, 79.58001)
(11, 87.68717)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 14.56977)
(2, 22.86667)
(3, 27.24107)
(4, 32.42375)
(5, 37.75556)
(6, 39.47939)
(7, 39.75452)
(8, 42.07074)
(9, 52.39683)
(10, 67.56133)
(11, 78.10365)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 14.56977)
% (2, 22.30396)
% (3, 25.61124)
% (4, 32.86124)
% (5, 40.26828)
% (6, 40.1343)
% (7, 45.13872)
% (8, 44.93333)
% (9, 58.66667)
% (10, 67.76139)
% (11, 82.96405)
% };
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=0,
ymax=65, % Adjusted for AIDE dataset accuracy values
extra y ticks={56.47},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 31.19747)
(2, 40.82603)
(3, 28.35099)
(4, 34.59857)
(5, 38.38608)
(6, 48.86124)
(7, 34.14086)
(8, 25.25275)
(9, 45.05373)
(10, 47.70917)
(11, 44.42916)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 31.19747)
(2, 42.28817)
(3, 42.06203)
(4, 42.11146)
(5, 44.76579)
(6, 51.17573)
(7, 49.48107)
(8, 47.02707)
(9, 47.53458)
(10, 48.68311)
(11, 48.78504)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 31.19747)
% (2, 42.28817)
% (3, 42.06203)
% (4, 40.45721)
% (5, 44.67999)
% (6, 48.7377)
% (7, 49.88394)
% (8, 44.42247)
% (9, 47.28958)
% (10, 48.68311)
% (11, 47.39316)
% };
%%%%%% PE %%%%
% First plot (Accuracy for AIDE Dataset)
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=10,
ymax=90, % Adjusted for AIDE dataset accuracy values
extra y ticks={83.52},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 48.37391)
(2, 50.85772)
(3, 60.40841)
(4, 69.44792)
(5, 73.36362)
(6, 65.7891)
(7, 71.35559)
(8, 72.77942)
(9, 74.89236)
(10, 80.8762)
(11, 77.06119)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 48.37391)
(2, 48.58562)
(3, 63.65119)
(4, 68.31964)
(5, 71.82754)
(6, 69.94979)
(7, 71.74029)
(8, 72.66157)
(9, 74.97421)
(10, 78.57321)
(11, 79.11401)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 48.37391)
% (2, 48.41379)
% (3, 64.8546)
% (4, 69.72777)
% (5, 72.33095)
% (6, 69.98966)
% (7, 71.26843)
% (8, 72.46628)
% (9, 74.966)
% (10, 80.41954)
% (11, 79.74171)
% };
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=0,
ymax=100, % Adjusted for AIDE dataset accuracy values
extra y ticks={95.03},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 13.57317)
(2, 10.06396)
(3, 15.59454)
(4, 33.92625)
(5, 47.02066)
(6, 43.34239)
(7, 38.43244)
(8, 50.50615)
(9, 66.04923)
(10, 76.06624)
(11, 89.69433)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 13.57317)
(2, 12.75251)
(3, 13.62854)
(4, 24.52641)
(5, 29.65018)
(6, 35.17379)
(7, 44.0076)
(8, 46.02523)
(9, 52.62523)
(10, 73.50839)
(11, 81.99567)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 13.57317)
% (2, 12.26199)
% (3, 14.4567)
% (4, 28.13257)
% (5, 31.78908)
% (6, 41.19261)
% (7, 40.96269)
% (8, 45.73325)
% (9, 56.22553)
% (10, 74.18908)
% (11, 83.90472)
% };
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=0,
ymax=65, % Adjusted for AIDE dataset accuracy values
extra y ticks={57.54},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 16.86518)
(2, 38.75458)
(3, 29.26438)
(4, 54.22914)
(5, 55.96379)
(6, 48.70592)
(7, 41.58425)
(8, 42.82384)
(9, 48.70138)
(10, 47.45985)
(11, 38.56271)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 16.86518)
(2, 31.9888)
(3, 25.83333)
(4, 40.77227)
(5, 46.10682)
(6, 49.7619)
(7, 48.08367)
(8, 49.13485)
(9, 51.18355)
(10, 51.00361)
(11, 49.13485)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 16.86518)
% (2, 31.9888)
% (3, 23.34772)
% (4, 42.06683)
% (5, 48.08367)
% (6, 49.7619)
% (7, 48.08367)
% (8, 51.18355)
% (9, 50.70531)
% (10, 50.76791)
% (11, 49.29792)
% };
%%% SE %%%
% First plot (Accuracy for AIDE Dataset)
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=10,
ymax=90, % Adjusted for AIDE dataset accuracy values
extra y ticks={82.97},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 47.92156)
(2, 51.28769)
(3, 60.37662)
(4, 63.59172)
(5, 68.41436)
(6, 75.98272)
(7, 69.79106)
(8, 77.33989)
(9, 77.41162)
(10, 77.86767)
(11, 81.66656)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 47.92156)
(2, 49.54622)
(3, 54.51997)
(4, 58.25786)
(5, 62.79981)
(6, 64.37686)
(7, 67.41963)
(8, 67.84083)
(9, 72.13373)
(10, 74.82318)
(11, 77.42545)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 47.92156)
% (2, 50.4343)
% (3, 55.47859)
% (4, 59.16332)
% (5, 64.23921)
% (6, 66.56982)
% (7, 67.94835)
% (8, 70.39402)
% (9, 71.66394)
% (10, 76.23219)
% (11, 79.58304)
% };
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=0,
ymax=100, % Adjusted for AIDE dataset accuracy values
extra y ticks={94.39},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 13.53434)
(2, 28.70708)
(3, 29.70619)
(4, 40.08207)
(5, 48.9715)
(6, 43.69256)
(7, 47.0734)
(8, 65.07611)
(9, 67.60291)
(10, 83.41823)
(11, 93.741)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 13.53434)
(2, 17.95271)
(3, 22.14991)
(4, 29.53797)
(5, 37.75152)
(6, 39.19328)
(7, 47.39876)
(8, 48.53209)
(9, 59.84462)
(10, 74.27719)
(11, 83.35656)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 13.53434)
% (2, 18.36675)
% (3, 22.75943)
% (4, 32.41586)
% (5, 38.43335)
% (6, 42.95395)
% (7, 47.78806)
% (8, 50.06403)
% (9, 63.70035)
% (10, 76.98813)
% (11, 84.9758)
% };
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=0,
ymax=65, % Adjusted for AIDE dataset accuracy values
extra y ticks={47.65},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 29.44511)
(2, 49.09091)
(3, 48.08367)
(4, 51.52295)
(5, 52.46429)
(6, 50.95238)
(7, 49.40736)
(8, 43.07416)
(9, 35.72022)
(10, 39.8658)
(11, 40.85708)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 29.44511)
(2, 45.7669)
(3, 47.96001)
(4, 52.48299)
(5, 52.48299)
(6, 52.48299)
(7, 52.48299)
(8, 52.48299)
(9, 52.24638)
(10, 49.68223)
(11, 50.08464)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 29.44511)
% (2, 45.7669)
% (3, 49.56457)
% (4, 52.48299)
% (5, 52.48299)
% (6, 52.48299)
% (7, 52.48299)
% (8, 52.0028)
% (9, 49.35675)
% (10, 49.44078)
% (11, 48.10777)
% };
%%% VBLL %%%
% First plot (Accuracy for AIDE Dataset)
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=10,
ymax=90, % Adjusted for AIDE dataset accuracy values
extra y ticks={83.00},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 41.5104)
(2, 33.87967)
(3, 39.28984)
(4, 40.35489)
(5, 47.72237)
(6, 67.93976)
(7, 66.0523)
(8, 69.45391)
(9, 75.93762)
(10, 69.86359)
(11, 77.75914)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 41.5104)
(2, 35.40693)
(3, 41.97037)
(4, 42.33288)
(5, 44.02737)
(6, 44.74939)
(7, 45.7584)
(8, 54.25977)
(9, 61.25627)
(10, 64.09861)
(11, 67.01982)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 41.5104)
% (2, 35.40693)
% (3, 41.97037)
% (4, 42.33288)
% (5, 44.02737)
% (6, 44.74939)
% (7, 45.7584)
% (8, 54.25977)
% (9, 61.25627)
% (10, 64.09861)
% (11, 67.01982)
% };
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=0,
ymax=100, % Adjusted for AIDE dataset accuracy values
extra y ticks={94.93},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 18.11492)
(2, 3.75)
(3, 16.87273)
(4, 15.13342)
(5, 27.0007)
(6, 35.69139)
(7, 45.50221)
(8, 37.16727)
(9, 48.55402)
(10, 72.02053)
(11, 88.08161)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 18.11492)
(2, 13.01723)
(3, 18.25154)
(4, 17.01258)
(5, 22.69958)
(6, 24.00761)
(7, 36.19728)
(8, 37.15934)
(9, 50.6051)
(10, 66.22472)
(11, 74.21875)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 14.56977)
% (2, 22.30396)
% (3, 25.61124)
% (4, 32.86124)
% (5, 40.26828)
% (6, 40.1343)
% (7, 45.13872)
% (8, 44.93333)
% (9, 58.66667)
% (10, 67.76139)
% (11, 82.96405)
% };
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
xlabel=\empty, % Remove x-axis label for the first row
ymin=0,
ymax=65, % Adjusted for AIDE dataset accuracy values
extra y ticks={49.34},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 8.83652)
(2, 14.54545)
(3, 8.83652)
(4, 8.83652)
(5, 8.83652)
(6, 8.83652)
(7, 23.79832)
(8, 14.21246)
(9, 20.51517)
(10, 25.78406)
(11, 33.22437)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 8.83652)
(2, 12.68997)
(3, 8.83652)
(4, 8.83652)
(5, 8.83652)
(6, 8.83652)
(7, 8.83652)
(8, 8.83652)
(9, 8.83652)
(10, 20.51517)
(11, 20.51517)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 8.83652)
% (2, 12.68997)
% (3, 8.83652)
% (4, 8.83652)
% (5, 8.83652)
% (6, 8.83652)
% (7, 8.83652)
% (8, 8.83652)
% (9, 8.83652)
% (10, 20.51517)
% (11, 20.51517)
% };
%%% LL--HMC %%%%
% First plot (Accuracy for AIDE Dataset)
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
% xlabel=\empty, % Remove x-axis label for the first row
ymin=10,
ymax=90, % Adjusted for AIDE dataset accuracy values
extra y ticks={85.24},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 43.82956)
(2, 43.44611)
(3, 14.00213)
(4, 55.81161)
(5, 65.1037)
(6, 67.45548)
(7, 70.78094)
(8, 76.82769)
(9, 75.83238)
(10, 79.27046)
(11, 81.50745)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 43.82956)
(2, 43.94482)
(3, 14.00213)
(4, 14.00213)
(5, 14.00213)
(6, 15.07482)
(7, 24.11139)
(8, 31.65505)
(9, 44.01879)
(10, 60.08508)
(11, 67.23819)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 43.82956)
% (2, 43.94482)
% (3, 14.00213)
% (4, 14.00213)
% (5, 14.00213)
% (6, 20.40029)
% (7, 29.38945)
% (8, 46.07501)
% (9, 59.88256)
% (10, 66.14985)
% (11, 72.11557)
% };
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
% xlabel=\empty, % Remove x-axis label for the first row
ymin=0,
ymax=100, % Adjusted for AIDE dataset accuracy values
extra y ticks={95.00},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 14.94949)
(2, 22.21822)
(3, 27.02882)
(4, 38.91637)
(5, 39.78317)
(6, 45.38431)
(7, 44.95006)
(8, 54.17743)
(9, 68.37018)
(10, 85.88435)
(11, 90.68394)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 14.94949)
(2, 17.75241)
(3, 18.40663)
(4, 25.30447)
(5, 28.79075)
(6, 35.62768)
(7, 38.05286)
(8, 44.64466)
(9, 47.64526)
(10, 65.74323)
(11, 82.30475)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 14.94949)
% (2, 18.19479)
% (3, 17.72981)
% (4, 25.68463)
% (5, 32.30725)
% (6, 38.29502)
% (7, 44.38927)
% (8, 44.68014)
% (9, 53.16883)
% (10, 73.00471)
% (11, 84.5382)
% };
\nextgroupplot[
title={}, % Remove title
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11},
% xlabel=\empty, % Remove x-axis label for the first row
ymin=0,
ymax=65, % Adjusted for AIDE dataset accuracy values
extra y ticks={59.01},
extra y tick labels={},
extra y tick style={grid=major, grid style={thick, dashed, black}}
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1, 37.45106)
(2, 38.84074)
(3, 39.98501)
(4, 52.9289)
(5, 52.15129)
(6, 53.2795)
(7, 44.26237)
(8, 41.9908)
(9, 37.05128)
(10, 39.93982)
(11, 42.04927)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2pt] coordinates {
(1, 37.45106)
(2, 43.25089)
(3, 45.26906)
(4, 48.0142)
(5, 52.64746)
(6, 52.48299)
(7, 53.40875)
(8, 52.48299)
(9, 50.39683)
(10, 50.94785)
(11, 50.94785)
};
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
% (1, 37.45106)
% (2, 43.25089)
% (3, 46.9376)
% (4, 48.0142)
% (5, 51.72257)
% (6, 52.48299)
% (7, 53.40875)
% (8, 52.0028)
% (9, 50.45565)
% (10, 49.17145)
% (11, 49.31457)
% };
\end{groupplot}
% Add dataset labels on top of columns
\node[font=\bfseries\tiny] at ($(group c1r1.north)+(0,0.2cm)$) {\textbf{AIDE} \citep{yang2023aide}};
\node[font=\bfseries\tiny] at ($(group c2r1.north)+(0,0.2cm)$) {\textbf{B4C} \citep{jain2015car}};
\node[font=\bfseries\tiny] at ($(group c3r1.north)+(0,0.2cm)$) {\textbf{ROAD} \citep{singh2022road}};
\end{tikzpicture}
\vspace{0.2cm}
% Legend
\begin{tikzpicture}
% \hspace{5.9cm}
\begin{axis}[
hide axis,
xmin=0, xmax=1, ymin=0, ymax=1,
legend style={at={(0.0,0.0)}, anchor=center, legend columns=3},
legend entries={\tiny{Single predictions},\tiny{Cumulative ensemble}}%,\tiny{Weighted ensembles (exonental decay factor: 0.9)}}
]
\addlegendimage{mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=2.0pt}
\addlegendimage{mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2.0pt}
% % \addlegendimage{mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=2pt}
% % \addlegendimage{mark=diamond*, thin, line width=0.25pt, mark options={fill=lime}, mark size=2pt}
\end{axis}
\end{tikzpicture}
\caption{Performance overview for the single and cumulative ensembles based on the intermediate latent representations. The dashed lines indicate the best performance for each LL--PDL method performed on the final latent representation.}
\label{fig:inter}
% \textcolor{orange!95!white}{\ding{108}} {Single prediction}, \textcolor{cornflowerblue}{\ding{115}} {Cumulative ensemble}.
\end{figure*}
For deeper DNNs, previous studies have indicated that the intermediate latent representations can already be used to classify before the final layer of a network \citep{montavon2011kernel,papyan2020prevalence,kothapallineural} (see Figure \ref{fig:intermed}). To reduce the required computations to produce uncertainty estimations further, \cite{qendro2021early} and \cite{meronen2024fixing} showed that using probabilistic last layers trained on these intermediate representations of a single network can improve the performance. Subsequently, these intermediate predictions can be incrementally used to create a model-internal ensemble, which combines the predictions from different depths of a network to improve the performance \citep{meronen2024fixing}. This potentially enables to not having to compute the entire network but to rely on intermediate representations for confident prediction and uncertainty estimation.
Processing a single 16-frame video requires approximately $361 \times 10^9$ floating point operations (FLOPs) for a ViT-Base model, where each Transformer block accounts for $29.8 \times 10^9$ FLOPs. Given the resource-constrained operating environment of DAR and DIR, and prior work highlighting the utility of intermediate representations for classification \citep{papyan2020prevalence, qendro2021early, kothapallineural, meronen2024fixing}, we evaluate LL--PDL methods' performance when trained on the representations extracted from the intermediate layers. We extract the latent features from each block via average pooling and apply the top-performing configurations to these intermediate features. Following \citep{qendro2021early, meronen2024fixing}, we assess whether predictions, either from individual layers or ensembles in the form of accumulated predictions from previous intermediate layers surpass the full network performance.
% \subsection{Intermediate latent representation performance}
\paragraph{\textbf{Results.}} Figure \ref{fig:inter} illustrates the F1-score of the LL--PDL methods based on the intermediate latent representations of the ViT-Base model. For the AIDE and B4C datasets, the performance starts to saturate when relying on deeper latent representations, but never exceeds the final latent representation performance as reported in Table \ref{tab:performance}. For the ROAD dataset, the performance exhibits more variability, potentially due to the smaller dataset size. Only for the SE on the ROAD dataset, we observe improved performance based on the intermediate representation.
%%
% \section{Experimental setup}
% \subsection{Datasets}
% For the DAR task, we use the AssIstive Driving pErception (AIDE) dataset \citep{yang2023aide} that consists of 2898 short videos recorded in China, with the following labels: 185 \textit{ backward moving}, 1522 \textit{forward moving}, 138 \textit{lane changing}, 727 \textit{parking}, and 326 \textit{turning}. Additionally, we use annotations from the ROad event Awareness Dataset (ROAD) \citep{singh2022road}, which extend a subset of 22 videos (122.000 frames) of the Oxford RobotCar Dataset \citep{RobotCarDatasetIJRR} with ego-vehicle and road user driving action labels. Since the ROAD data was collected with manual driving, the vehicle action labels reflect human driving maneuvers \citep{RobotCarDatasetIJRR}. To capture temporal patterns, we transform the frame-level annotations into video-level labels by selecting only driving actions with at least 8 consecutive frames. This threshold ensures that each clip provides sufficient temporal context for distinguishing maneuvers while reducing ambiguous or noisy labels. As a result, the offline ROAD DAR dataset consists of 75 \textit{stopping}, 162 \textit{move forward}, 31 \textit{turn right}, {41} \textit{turn left}, 12 \textit{move right}, 12 \textit{move left} and 8 \textit{overtake} maneuvers.
% For DIR, we use the open-source Brain4Cars (B4C) dataset \citep{jain2015car} collected in the United States. The B4C dataset consists of 124 \textit{left lane changes}, 123 \textit{right lane changes}, 58 \textit{left turns}, 55 \textit{right turns}, and 234 \textit{driving straight} maneuvers. We retain the natural class imbalance present in all datasets to reflect the real-world distribution of driving behaviors. For all datasets, we use the original train-test splits. For the ROAD and B4C datasets, we use the first fold from the provided splits.
% \section{Last Layer Hamiltonian Monte Carlo Sampling}
% \subsection{Metrics}
% \paragraph{\textbf{Performance, and uncertainty.}}We report standard metrics (accuracy $\uparrow$, F1-score $\uparrow$) to measure the in-distribution performance. To estimate uncertainty, we use predictive entropy \citep{smith2018understanding}, which is maximized when the probabilities are uniformly distributed among all classes. The more confident the model is in its predictions for a particular class, the lower the predictive entropy will be, indicating less uncertainty.
% \paragraph{\textbf{Calibration.}}To measure the alignment between the prediction confidence and the performance, we use the adaptive calibration error (ACE $\downarrow$, \citealt{nixon2019measuring}) with 10 bins. ACE partitions predictions into adaptively-sized bins, comparing the average predicted probability in each bin to the corresponding fraction of correct predictions. Overconfidence occurs when predicted confidence exceeds accuracy; conservativeness occurs when confidence falls short of accuracy. To assess the quality of uncertainty estimates relative to model performance, we use the relative Area Under the Lifted Curve (rAULC $\uparrow$, \citealt{postels2022practicality}). The rAULC evaluates how well uncertainty scores are calibrated. Ideally, confident and correct predictions should correspond to low uncertainty, while incorrect or low-confidence predictions should yield high uncertainty.
% \paragraph{\textbf{MCMC diagnostics.}}We evaluate the quality of posterior samples using the Effective Sample Size (ESS, \citealt{geyer1992practical}). A higher ESS implies better exploration of the posterior distribution. For LL--HMC with multiple chains, we use Gelman’s $\hat{R}$ statistic \citep{gelman1992inference} to assess convergence. $\hat{R}$ compares the variance between chains to the variance within each chain. Values close to 1 suggest that the chains are exploring the same distribution \citep{izmailov2021bayesian}.
% \paragraph{\textbf{OOD detection.}}To determine whether OOD instances yield high uncertainty, we compute the Area Under the Receiver Operating Characteristic (ROC, $\uparrow$) and Precision-Recall (PR, $\uparrow$) curves \citep{malinin2018predictive,durasov2024zigzag}. Furthermore, we report the False Positive Rate at 95\% Recall (FPR95, $\downarrow$). FPR95 measures the proportion of OOD instances that are incorrectly classified as in-distribution when the model correctly identifies 95\% of in-distribution samples.
% .
% For LL--HMC with multiple chains, we report Gelman’s \citep{gelman1992inference} $\hat{R}$ statistic to assess the convergence.
% , we assess convergence across multiple chains,
% with values close to 1 indicating reliable convergence and exploration of the posterior landscape.
% evaluate the quality of posterior samples using the Effective Sample Size (ESS, \citealt{geyer1992practical}). A higher ESS implies better exploration of the posterior distribution. To
% \subsection{Underlying model architecture}
% Given the relatively small size of the DAR and DIR datasets, we use a Kinetics-400 \citep{kay2017kinetics} self-supervised pre-trained Video Masked Autoencoder (VMAE, \citealt{tong2022videomae}). As a backbone, the VMAE uses a Vision Transformer (ViT) Base architecture \citep{arnab2021vivit} that consists of about 94 million parameters. The ViT-Base expects videos of 16 frames, has a depth of 12 blocks and a hidden latent dimension of 768. For the datasets with multiple video streams, we combine the latent modality representations via attentional feature fusion \citep{dai2021attentional}.
% \subsection{Implementation details}
% We fine-tune a separate ViT-Base model for 20 epochs per dataset using the AdamW optimizer \citep{loshchilov2018decoupled} with a weight decay of 0.05. We apply early stopping with a patience of 5 epochs and set the learning rate to $5\mathrm{e}{-5}$. We use an Ubuntu 20.04 server with two 16GB NVIDIA Tesla T4 GPUs. To simulate a larger batch size of 8, we use the gradient accumulation module from the Accelerate library (Gugger et al., \citeyear{accelerate}).
% The LL--HMC experiments are run on a separate Ubuntu 20.04 server (CPU-only, 60GB RAM) using the NUTS sampler from Pyro \citep{bingham2018pyro}. We perform a grid search for different prior standard deviation (0.01, 0.1, 1, 2.5, 5, 10), number of burn-in samples (10, 25, 50, 100, 200), target acceptance probabilities (0.6, 0.7, 0.8), multiple chains (1, 2), and numbers of collected samples (2, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50) for the classification layer $\theta_{LL}$. When we use more than a single chain, we divide the number of collected samples evenly over the chains. For the LL--HMC with multiple starting positions (latent representations from different independently fine-tuned models), we use the top-performing ensemble models to produce the latent representations.
% For the remaining last layer PDL (LL--PDL) methods, we perform a grid search to evaluate the number of samples/predictors/ensemble members (5, 10, 15, 20, 25, 30, 35, 40, 45, 50), batch size (16, 32, 64), learning rate ($1\mathrm{e}{-2}, 1\mathrm{e}{-3}, 1\mathrm{e}{-4}$) for different numbers of epochs (5, 10, 15, 20, 25). For VBLL \citep{harrison2024variational}, which does not produce multiple predictions, we evaluate modifying the KL regularization (1, 5, 10).
% \subsection{Experiments}
% \subsubsection{Video-based DAR and DIR performance}
% To compare LL--HMC to other PDL methods, we use the fine-tuned ViT-Base models as a regular baseline. The baseline models also produce the latent representations on which the other LL--PDL methods rely. Additionally, we also evaluate an ensemble of 5 independently fine-tuned ViT-Base models. For the methods, we include the following approaches that do not require any form of post-hoc calibration: SE \citep{valdenegro2023sub}, DDU \citep{mukhoti2023deep}, BBB \citep{blundell2015weight}, PE \citep{laurent2023packed}, and VBLL \citep{harrison2024variational}.
% Given the increased computational demand required for a DE, we evaluate an ensemble of 5 independently fine-tuned ViT-Base models.
% Following to \citep{qendro2021early,meronen2024fixing}, we evaluate if an ensemble (equally weighted for each prediction or exponentially relying the later predictions) of the cumulative predictions based on the representations at different depths of the ViT progressively improve performance to understand.
% Given the resource-constrained operating environment of DAR and DIR, and that intermediate representations can already be informative for classification \citep{papyan2020prevalence, qendro2021early,kothapallineural,meronen2024fixing}, we evaluate the performance of the methods on these intermediate representations. The Vit-Base architecture requires about $361\mathrm{e}^9$ floating point operations (FLOPs) to process a video of 16 frames, with each of the Transformer blocks accounting for $29.8\mathrm{e}^9$ FLOPs. To extract the intermediate latent representations from each block, we apply average pooling to extract the intermediate latent representations. We train the best-performing hyperparameter configurations for the approaches to perform DAR and DIR based these intermediate latent representations. Similar to \citep{qendro2021early,meronen2024fixing}, we evaluate if an ensemble (equally weighted for each prediction or exponentially relying the later predictions) of the cumulative predictions based on the representations at different depths of the ViT progressively improve performance to understand.
% \subsubsection{Uncertainty-based out-of-distribution detection}
% Uncertainty-based OOD detection aims to identify inputs that do not belong to the distribution of the training data \citep{yang2024generalized}. To construct realistic OOD instances for DAR and DIR, we remove the most and least prevalent classes from each dataset to simulate OOD instances for common or rare maneuvers. The removed classes from both the train and test datasets serve as the unseen OOD instances. We fine-tune a ViT-Base model on each reduced training set to obtain latent representations to use with the methods. For the ROAD dataset, removing the most occurring maneuver would leave a subset of fewer than 200 videos. Therefore, for the ROAD dataset, we only apply the OOD-min setting, where we remove the maneuvers for which we have less than 20 videos (\textit{Move Right, Move Left, and Overtake}).
% For each dataset, we fine-tune a ViT-Base model on the reduced training datasets to produce the latent representations for the LL--PDL methods. For the ROAD dataset, removing the most occurring maneuver would result in a remaining subset of less than 200 videos for the remaining 6 classes. Therefore, we only perform the OOD min setting, where we remove the maneuvers for which we have less than 20 videos (\textit{Move Right, Move Left, and Overtake}).
% To also demonstrate the uncertainty estimation for a regression example, we use the implementation of \cite{lippe2022uva} to generate noisy observations of a sinusoidal function. The underlying architecture is an MLP with five hidden layers and 50 hidden units for all methods. Figure \ref{fig:toy_reg} shows the uncertainty estimation results. When performing HMC on the entire DNN, we observe that the estimations outside of the observed areas directly are more uncertain. For both the LL--HMC and the deep ensemble, the uncertainty between the two observation areas is relatively stable and shows superiority of applying HMC to the entire network.
% Figure \ref{fig:toy_reg} shows the results for the noisy observations from a sinusoidal function. When performing HMC on the entire DNN, we observe that the estimations for the areas with no observations are more uncertain. For both the LL--HMC and the deep ensemble, the uncertainty between the two areas with observation areas is relatively stable and shows the superiority of applying HMC to the entire network. %These toy examples allow us to understand the behavior and the cost in terms of uncertainty estimations. %The performance of LL--HMC is promising, give the required increased compute deep ensembles and HMC require.
% We evaluate the performance and uncertainty-based OOD detections for methods on three video-based driver action and intention recognition datasets. %Action recognition involves determining the current activities of an agent (a driver or pedestrian in our datasets), whereas intention recognition focuses on identifying the agent’s (immediate) future aspirations \citep{pereira2013state, vellenga2022driver}. The rationale to test on such safety critical driving automation datasets is that the aimed operating environment demands efficient uncertainty estimations given the limited available compute, and because actions and intentions can be more ambiguous and overlapping than standard image classification benchmarks.
% For generic video-based action recognition, we use the UCF101 \citep{soomro2012UCF101} and the HMDB51 \citep{kuehne2011hmdb} datasets. UCF101 consists of 13.000 videos and has 101 action classes. The HMDB51 dataset consist of 6.766 video clips with 51 action classes. Both datasets have roughly over a 100 videos for each action class and three pre-defined folds. To test the PDL methods in a more realistic safety-critical environment with limited computational resources, we include a driver action and intention recognition dataset.
% To evaluate LL--HMC and the other methods, we use three DAR and DIR offline video datasets.
% the predictive entropy \cite{smith2018understanding}, which captures both. Intuitively, for an instance $x$ the predictive entropy reaches its maximum value when the predicted probabilities for all classes are exactly the same.
%The higher the predicted probability is for one class, the lower is the predictive entropy, and the more certain is the model about the predictions.
% Intuitively, the KL penalty helps to ensure that the model's learned distribution (variational posterior) does not deviate too much from the initial assumptions (prior distribution). This balance is crucial for maintaining a model that generalizes well to unseen data and provides reliable uncertainty estimates.
% In practical terms, think of the KL penalty as a "tether" that keeps the model's predictions grounded in reality, preventing it from becoming overly confident or overly uncertain based on the training data alone. By adjusting the KL regularization weight, you can control how tightly this tether is held, thereby influencing the model's uncertainty.
% Specifically for the LL--HMC approach, we use the Effective Sample Size (ESS) to measure the quality of a set of samples generated by an MCMC method \cite{geyer1992practical}. When sampling with an MCMC, each sample is not entirely independent because consecutive samples are correlated. ESS provides a way to estimate the number of independent samples that would be equivalent in variance reduction to the correlated samples we have (refer to Equation \ref{eq:ess}). The higher the ESS, the better, as we effectively have more information about the posterior distribution. The ESS is particularly important when we want to make inferences about the distribution of parameters or predictions made by the model. When we use multiple chains to sample DNN parameters, we use Gellman's \cite{gelman1992inference} $\hat{R}$ or potential-scale-reduction diagnostic to analyze the between chain variance The underlying idea of using more than one chain allows (potentially) for exploring different regions.
% sequential or ensemble approaches, we test a similar range of samples or ensemble members to make the comparison as fair as possible (refer to Appendix \ref{app:hyper} for the hyperparameter tuning results of the remaining LL--PDL). We perform an evaluation for each configuration, and report the results of all methods (also for the OOD experiments) based on the highest F1-score.
% (Eq. \ref{eq:raulc})
% (Eq. \ref{eq:uc_it})
% \begin{equation}
% ACE = \frac{1}{KR} \sum_{k=1}^{K} \sum_{r=1}^{R} | acc(r,k) - conf(r,k) |,
% \label{eq:ACE}
% \end{equation}
% where:
% \begin{conditions}
% k & = & class label,\\
% r & = & adaptive range for class label,\\
% acc(r,k) & = & accuracy within a bin, and\\
% conf(r,k) & = & predicted confidence within a bin.\\
% \end{conditions}
% \begin{equation}
% H[P(y|x)] = - \sum_{y \in \mathcal{Y}} P(y|x) \log P(y|x),
% \label{eq:uc_it}
% \end{equation}
% where:
% \begin{conditions}
% H[P(y|x)] & = & the entropy of the predictive distribution,\\
% P(y|x) & = & conditional probability distribution, over a discrete set of outcomes $\mathcal{Y}$, and\\
% \mathcal{Y}& = & set of $N$ stochastic predictions.
% \end{conditions}
% \begin{equation}
% \text{rAULC} = -1 + \sum_{i=1}^{S} \frac{s \cdot A(q_i)}{R(q_i)},
% \label{eq:raulc}
% \end{equation}
% where:
% \begin{conditions}
% q_i & = & uncertainty quantiles $i\in [1, ..., S]$, \\
% s & = & quantile step width based on the number of predictions in the test dataset ($1/{N}$), \\
% A(q_i) & = & accuracy of samples with uncertainty below quantile $q_i$, \\
% R(q_i) & = & expected accuracy for random guessing at quantile $q_i$, and \\
% S & = & total number of quantiles. \\
% \end{conditions}
% To evaluate if teh LL--HMC approach with multiple chains
% Specifically for the LL--HMC approach with multiple chains,
% To evaluate this, \cite{gelman1992inference} proposed the $\hat{R}$ ratio, which quantifies convergence by comparing the variance between chains to the variance within each chain.
% \begin{equation}
% \label{eq:rhat}
% \hat{R} = \frac{E[{Var}[X | \text{Chain}]] + {Var}[E[X | \text{Chain}]]}{E[{Var}[X | \text{Chain}]]},
% \end{equation}
% where:
% \begin{conditions}
% E[{Var}[X | \text{Chain}]]&=& the expected value of the variance \\
% &&of X given the chain, and\\
% Var[E[X | \text{Chain}]] & = & the variance of the expected value \\
% && of $X$ given the chain.\\
% \end{conditions}
%Figure \ref{fig:rhats} presents the $\hat{R}$ values for two chains across different datasets. In total, each chain samples 768 parameter values.
%To evaluate the use of different latent representations $Z$, we use the representations from the ensembles.
% \section{Experiments}
% \subsection{Intermediate latent representations}
% \begin{figure}[t]
% \centering
% \centering
% \includegraphics[trim={0cm 6.5cm 14.5cm 0cm},clip, width=0.48\textwidth, keepaspectratio]{figure_early.pdf}
% \vspace{-0.5cm}
% \caption{Overview of the intermediate latent representation extraction from the pre-trained Vision Transformer. After each block, the representations are extracted, followed by average pooling to extract the one dimensional intermediate latent representations.}
% \label{fig:early}
% \end{figure}
%Figure \ref{fig:early}
% we evaluate the performance for an equal number of samples from each chain. For the datasets with multiple folds, we select the configuration based on the average performance across all folds. To evaluate the use of different latent representations $Z$, we use an ensemble of two or a LUR model.
% we use the NuScenes dataset \citep{caesar2020nuscenes}, which consists of naturalistic driving videos recorded in the USA and Singapore. \citept{vellenga2025latent} annotated 1194 driving scenarios with the following driver intention labels: \textit{right lane change, left lane change, right turn, left turn, forward moving, and waiting}.
% \section{Results}
% For both the in-distribution and OOD experiments, we first present the grid search results, followed by an analysis of the influence of additional predictions or samples for each PDL method. Then, to examine if there are differences between the PDL methods, we present the results of the top 10 hyperparameter configurations for 100 random seeds for each dataset. For LL--HMC, we also explore the influence of the prior, additional chains, starting positions, parameter trajectories and MCMC diagnostics.
% \subsection{In-distribution action and intention recognition performance} %#driver action and intention recognition
% \paragraph{\textbf{Grid search results.}} Table \ref{tab:performance} presents top-performance for the grid search for the different PDL methods for video-based DAR and DIR. In terms of performance, all methods, except for the DDU and ensemble, there is a configuration that outperforms the regular baseline model. For the DDU, the lower performance is most likely due to not using spectral normalization. The fluctuating ensemble performance can possibly be explained by the small number of ensemble members that influence the performance. Given the additional required computations for each additional ensemble member, it might be an undesirable choice for a resource-constrained operating environment. The top-performing LL--HMC configurations consistently rank among the best performing methods overall. From a confidence-performance calibration perspective, the BBB--LL, PE--LL, SE and LL--HMC yield a lower ACE for the B4C and ROAD datasets compared to the regular model, but not for the AIDE dataset. Additionally, no single method optimally balances both classification performance and calibration across all datasets. The regular softmax-based model is better calibrated than the LL--HMC approach, except for the B4C dataset. We do not observe that the improved classification performance of LL-HMC also results in an improved ability to distinguish between correctly and incorrectly predicted in-distribution instances based on the uncertainty estimations.
% \begin{table*}[t]
% \centering
% \caption{Grid search result for PDL methods on the AIDE, B4C and ROAD datasets. Each PDL method, except for the deep ensemble, uses the latent representations produced by the \textit{Regular} model. Acc=Accuracy, ACE=Adaptive Calibration Error with 10 bins, rAULC=relative Area under the lifted curve. }
% \vspace{0.2cm}
% \resizebox{\linewidth}{!}{
% \begin{tabular}{l|c|c|c|c||c|l|c|c||c|c|c|c}
% \toprule
% & \multicolumn{4}{c||}{\textbf{AIDE} \citep{yang2023aide}} & \multicolumn{4}{c||}{\textbf{B4C} \citep{jain2015car}} & \multicolumn{4}{c}{\textbf{ROAD} \citep{singh2022road}} \\
% & \textbf{Acc ($\uparrow$)} & \textbf{F1 ($\uparrow$)} & \textbf{ACE ($\downarrow$)} & \textbf{rAULC ($\uparrow$)} & \textbf{Acc ($\uparrow$)} & \multicolumn{1}{|c|}{\textbf{F1 ($\uparrow$)}} & \textbf{ACE ($\downarrow$)} & \textbf{rAULC ($\uparrow$)} & \textbf{Acc ($\uparrow$)} & \textbf{F1 ($\uparrow$)} & \textbf{ACE ($\downarrow$)} & \textbf{rAULC ($\uparrow$)} \\
% \midrule
% \textbf{Regular} & 87.19 & 81.88 & 0.027 & 0.77 & 92.31 & 93.71 & 0.050 & 0.53 & 71.64 & 45.36 & 0.063 & 0.61 \\
% \textbf{DE (N=5)}& 89.66 & 82.24 & 0.019 & 0.80 & 90.88 & 91.49 & 0.059 & 0.73 & 74.24 & 44.57 & 0.066 &
% 0.70 \\
% \textbf{DDU}& 57.31 & 19.12 & 0.088 & 0.85 & 76.07 & 74.72& 0.124 & 0.61 & 59.70 & 23.85 & 0.138 & 0.42 \\
% \textbf{BBB--LL}& 88.67 & 83.70 & 0.032 & 0.76 &
% 94.02 & 95.00 & 0.019 & 0.78 & 62.87 & 56.47 & 0.094 & 0.51 \\ %aide {'num_epochs': 20, 'learning_rate': 0.0001, 'batch_size': 16, 'n_samples': 50} %b4c {'num_epochs': 20, 'learning_rate': 0.001, 'batch_size': 16, 'n_samples': 50}, OXF{'num_epochs': 20, 'learning_rate': 0.01, 'batch_size': 32, 'n_samples': 50}
% \textbf{PE--LL}& 89.33 & 83.52 & 0.035 & 0.73 & 94.02 & 95.03 & 0.014 & 0.74 & 70.15 & 57.54 & 0.076 & 0.53 \\ % AIDE {'n_est': 10, 'num_epochs': 25, 'learning_rate': 0.01, 'batch_size': 16} %b4c {'n_est': 5, 'num_epochs': 5, 'learning_rate': 0.01, 'batch_size': 32} % OXF {'n_est': 50, 'num_epochs': 5, 'learning_rate': 0.01, 'batch_size': 16}
% \textbf{SE}& 88.83 & 82.97 & 0.031 & 0.73 & 93.16 & 94.39 & 0.023 & 0.72 & 70.15 & 47.65 & 0.068 & 0.54 \\ % AIDE{'n_models': 5, 'max_epochs': 5, 'learning_rate': 0.01, 'batch_size': 32} % B4c {'n_models': 50, 'max_epochs': 10, 'learning_rate': 0.01, 'batch_size': 32} % OXF {'n_models': 15, 'max_epochs': 5, 'learning_rate': 0.01, 'batch_size': 16}
% % \textbf{cSGHMC} & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- & -- \\
% \textbf{VBLL}& 88.18 & 83.00 & 0.037 & 0.66 & 94.02 & 94.93 & 0.085 & 0.34 & 71.64 & 49.34 & 0.151 & 0.20 \\ \midrule %AIDE {'kl_penalty': 1, 'num_epochs': 5, 'learning_rate': 0.001, 'batch_size': 32} %B4C {'kl_penalty': 1, 'num_epochs': 5, 'learning_rate': 0.001, 'batch_size': 16}, %OXF {'kl_penalty': 25, 'num_epochs': 10, 'learning_rate': 0.01, 'batch_size': 16}
% \textbf{LL--HMC} & 89.82 & 85.24 & 0.029 & 0.68 & 94.02 & 95.00 & 0.024 &
% 0.75 & 71.64 & 59.01 & 0.070 & 0.57 \\
% % \multirow{4}{*}{\textbf{LL--HMC}} & \small{S=1, C=1} & 89.16 & 84.29 & 0.027 & 0.73 & 94.02 & 95.00 & 0.025 & 0.69 & 73.13 & 54.73 & 0.054 & 0.57 \\ %AIDE: {'num_sampels': 30, 'burn_in': 50, 'prior': 2,5, 'ta': 0,7} %b4c {'num_sampels': 20, 'burn_in': 100, 'prior': 5, 'ta': 0.7} %OXF {'num_sampels': 40, 'burn_in': 20, 'prior': 5, 'ta': 0.7}
% % & \small{S=1, C=2} & {89.82} & 85.24 & 0.029 & 0.68 & 94.02 & 95.00 & 0.024 & 0.75 & 71.64 & 59.01 & 0.070 & 0.57 \\ %AIDE {'num_samples': 40, 'burn_in': 20, 'prior': 10, 'ta': 0.6} %B4C {'num_samples': 40, 'burn_in': 60, 'prior': 5, 'ta': 0,6} % {'num_samples': 25, 'burn_in': 20, 'prior': 5, 'ta': 0,6}
% % & \small{S=2, C=1} & 88.67 & 82.58 & 0.047 & 0.61 & 94.02 & 95.01 & 0.083 & 0.42 & 73.13 & 58.24 & 0.062 & 0.36 \\
% % & \small{S=2, C=2} & 87.36 & 82.50 & 0.032 & 0.76 & 94.01 & 95.03 & 0.114 & 0.63 & 76.12 & 52.92 & 0.111 & 0.28 \\
% \bottomrule
% \end{tabular}
% }
% \label{tab:performance}
% \end{table*}
% \begin{figure*}[!t]
% \begin{tikzpicture}
% \begin{groupplot}[
% group style={
% group size=3 by 1,
% horizontal sep=1.0cm,
% vertical sep=1.0cm,
% },
% width=6.0cm,
% height=2.5cm,
% grid=major,
% xlabel={\tiny{Number of samples/predictions}},
% xtick={1,5,10,15,20,25,30,35,40,45,50},
% tick label style={font=\tiny},
% label style={font=\tiny},
% y tick label style={
% /pgf/number format/fixed,
% /pgf/number format/precision=0,
% /pgf/number format/fixed zerofill=true,
% },
% tick label style={font=\fontsize{4}{4}\selectfont},
% ylabel style={yshift=-0.2cm}, % Shift y-label down
% xlabel style={yshift=0.1cm}, % Shift x-label down
% ]
% % First plot (Dataset A)
% \nextgroupplot[
% title={\tiny{\textbf{AIDE} \citep{yang2023aide}}},
% title style={yshift=-0.2cm},
% ylabel={\tiny{F1 (\%)}},
% ymin=78,
% ymax=85
% ]
% % Plot data for BBB in Dataset A
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=orange}] coordinates {
% (5,80.95996846938037) += (0,1.6060031992555468) -=(0,1.6060031992555468)
% (10,80.38051179888186) += (0,1.177753117571621) -=(0,1.177753117571621)
% (15,81.2522205585229) += (0,0.7200762617832113) -=(0,0.7200762617832113)
% (20,81.76684349204947) += (0,0.637517608404733) -=(0,0.637517608404733)
% (25,81.27572973546255) += (0,0.449978338223246) -=(0,0.449978338223246)
% (30,81.20717689275352) += (0,0.974315675942861) -=(0,0.974315675942861)
% (35,81.31343389831275) += (0,0.8850200651392026) -=(0,0.8850200651392026)
% (40,81.08118064051627) += (0,1.0287535859538315) -=(0,1.0287535859538315)
% (45,80.94812310733515) += (0,1.0981426914188404) -=(0,1.0981426914188404)
% (50,81.83969173721589) += (0,0.5160174804168965) -=(0,0.5160174804168965)
% };
% % Plot data for PE in Dataset B
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=cyan}] coordinates {
% (5,80.64427423685513) += (0,1.7069168119717488) -=(0,1.7069168119717488)
% (10,81.39108033133715) += (0,0.8861904805266907) -=(0,0.8861904805266907)
% (15,81.37295306180285) += (0,0.7972287538860315) -=(0,0.7972287538860315)
% (20,80.82737023929269) += (0,1.806677462610932) -=(0,1.806677462610932)
% (25,80.91886472940277) += (0,1.1028863017894783) -=(0,1.1028863017894783)
% (30,81.17688066908337) += (0,0.7023957774670868) -=(0,0.7023957774670868)
% (35,81.49046003528815) += (0,0.5055158819696284) -=(0,0.5055158819696284)
% (40,81.41813206619261) += (0,0.9843795217508684) -=(0,0.9843795217508684)
% (45,81.22341562264936) += (0,0.603391592383691) -=(0,0.603391592383691)
% (50,81.064988976834) += (0,0.9503953226386528) -=(0,0.9503953226386528)
% };
% % Plot data for SE in Dataset A
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=red}] coordinates {
% (5,81.72796641439183) += (0,0.39656923786992143) -=(0,0.39656923786992143)
% (10,81.90801885064703) += (0,0.24753237124657146) -=(0,0.24753237124657146)
% (15,81.78739214969264) += (0,0.2561799183705285) -=(0,0.2561799183705285)
% (20,81.87270775771678) += (0,0.27879627212251196) -=(0,0.27879627212251196)
% (25,81.74547366757284) += (0,0.3547532651295543) -=(0,0.3547532651295543)
% (30,81.76892921540839) += (0,0.30325150634648884) -=(0,0.30325150634648884)
% (35,81.71655273815813) += (0,0.3442235108731674) -=(0,0.3442235108731674)
% (40,81.7176956093715) += (0,0.15813499155638713) -=(0,0.15813499155638713)
% (45,81.71353922074951) += (0,0.35106874707325336) -=(0,0.35106874707325336)
% (50,81.60908470361797) += (0,0.2505623494335606) -=(0,0.2505623494335606)
% };
% % % Plot data for LL--HMC in Dataset A
% \addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=yellow}] coordinates {
% (5,82.24926660396423) += (0,1.0320716930812943) -=(0,1.0320716930812943)
% (10,82.19827942316209) += (0,0.38139349674562134) -=(0,0.38139349674562134)
% (15,82.42064618224457) += (0,0.6722662361112617) -=(0,0.6722662361112617)
% (20,82.68614557647774) += (0,0.38298725733413364) -=(0,0.38298725733413364)
% (25,82.34071008535416) += (0,0.5027294819338703) -=(0,0.5027294819338703)
% (30,82.3229477909996) += (0,0.9685785260217125) -=(0,0.9685785260217125)
% (35,82.91580828836082) += (0,0.27537267785279074) -=(0,0.27537267785279074)
% (40,82.99625162343334) += (0,0.5080324588421731) -=(0,0.5080324588421731)
% (45,82.68709960581347) += (0,0.4329118451158503) -=(0,0.4329118451158503)
% (50,82.23586915620874) += (0,0.48580735633640987) -=(0,0.48580735633640987)
% };
% % Single marker for DE at 5 samples
% \addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
% (5,82.24)
% };
% % Single marker for Regular Model at 1 sample
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
% (1,81.88)
% };
% % % Single marker for VBLL at 1 sample
% % \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt] coordinates {
% % (1,83.00)
% % };
% % Single marker for VBLL at 1 sample
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=green}] coordinates {
% % (1,94.93)
% (1,80.478158) += (0,1.330908
% ) -=(0,1.330908)
% };
% % Second plot (Dataset B)
% \nextgroupplot[
% title={\tiny{\textbf{B4C} \citep{jain2015car}}},
% title style={yshift=-0.2cm},
% ylabel={\tiny{F1 (\%)}},
% ymax=96,
% ymin=89,
% ]
% % Plot data for BBB in Dataset B
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=orange}] coordinates {
% (5,91.81428553594436) += (0,2.266230024051118) -=(0,2.266230024051118)
% (10,92.08634408387084) += (0,2.0428920819483207) -=(0,2.0428920819483207)
% (15,91.88171836718176) += (0,1.825182223090597) -=(0,1.825182223090597)
% (20,92.02217647893363) += (0,1.8897040898040445) -=(0,1.8897040898040445)
% (25,92.4902186633233) += (0,2.0800450233186987) -=(0,2.0800450233186987)
% (30,92.3112903345926) += (0,1.6009305831773002) -=(0,1.6009305831773002)
% (35,90.85696435247613) += (0,1.1612547138694027) -=(0,1.1612547138694027)
% (40,91.84051123373924) += (0,1.3356710425592557) -=(0,1.3356710425592557)
% (45,91.99210892719971) += (0,1.3556185716902778) -=(0,1.3556185716902778)
% (50,92.27768581229043) += (0,1.6113916314474797) -=(0,1.6113916314474797)
% };
% % Plot data for PE in Dataset B
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=cyan}] coordinates {
% (5,92.7037411936547) += (0,2.0445640088656867) -=(0,2.0445640088656867)
% (10,93.11025366083305) += (0,1.4371907978670615) -=(0,1.4371907978670615)
% (15,93.13511115961856) += (0,1.6256082714539877) -=(0,1.6256082714539877)
% (20,93.30473881052755) += (0,1.9492536970666108) -=(0,1.9492536970666108)
% (25,92.68852537845807) += (0,2.043757619782589) -=(0,2.043757619782589)
% (30,93.23527829587042) += (0,1.3146639330318657) -=(0,1.3146639330318657)
% (35,92.33493213991402) += (0,1.7055183841701607) -=(0,1.7055183841701607)
% (40,92.63245413657515) += (0,1.477508098998782) -=(0,1.477508098998782)
% (45,94.10309815824014) += (0,0.7398833207078738) -=(0,0.7398833207078738)
% (50,93.05837606672831) += (0,1.7633838185208726) -=(0,1.7633838185208726)
% };
% % Plot data for SE in Dataset B
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=red}] coordinates {
% (5.0,93.9590185161521) += (0,0.8604800033087899) -=(0,0.8604800033087899)
% (10.0,94.18708528392156) += (0,0.31320370805457476) -=(0,0.31320370805457476)
% (15.0,94.12224595632553) += (0,0.3348288479449658) -=(0,0.3348288479449658)
% (20.0,94.2519246115176) += (0,0.2733866095430344) -=(0,0.2733866095430344)
% (25.0,94.12224595632551) += (0,0.3348288479449662) -=(0,0.3348288479449662)
% (30.0,94.2519246115176) += (0,0.27338660954303484) -=(0,0.27338660954303484)
% (35.0,94.2519246115176) += (0,0.273386609543033) -=(0,0.273386609543033)
% (40.0,94.2519246115176) += (0,0.2733866095430342) -=(0,0.2733866095430342)
% (45.0,94.31676393911363) += (0,0.20503995715727527) -=(0,0.20503995715727527)
% (50.0,94.31676393911363) += (0,0.20503995715727527) -=(0,0.20503995715727527)
% };
% % Plot data for LL--HMC in Dataset B
% \addplot[mark=diamond*, thin, line width=0.25pt,
% mark options={fill=yellow},
% mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=yellow}] coordinates {
% (5,90.99785760859635) += (0,1.0042182678453602) -=(0,1.0042182678453602)
% (10,92.25213285534451) += (0,1.3906044090052108) -=(0,1.3906044090052108)
% (15,92.3731837998395) += (0,1.5432087582813103) -=(0,1.5432087582813103)
% (20,92.71090059658137) += (0,1.1624750131352939) -=(0,1.1624750131352939)
% (25,93.1521381369241) += (0,1.1603540794221852) -=(0,1.1603540794221852)
% (30,92.67648931096207) += (0,1.6299949474329307) -=(0,1.6299949474329307)
% (35,92.33309758977552) += (0,1.6049273893598548) -=(0,1.6049273893598548)
% (40,93.25363425383188) += (0,0.9510727848891701) -=(0,0.9510727848891701)
% (45,93.43567253225955) += (0,1.0287222556656954) -=(0,1.0287222556656954)
% (50,93.64543100334956) += (0,1.350460316192456) -=(0,1.350460316192456)
% };
% % Single marker for DE at 5 samples
% \addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
% (5,91.49)
% };
% % Single marker for Regular Model at 1 sample
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
% (1,93.71)
% };
% % Single marker for VBLL at 1 sample
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=green}] coordinates {
% % (1,94.93)
% (1,90.7315) += (0,1.503367
% ) -=(0,1.503367)
% };
% % Third plot (Dataset C)
% \nextgroupplot[
% title={\tiny{\textbf{ROAD} \citep{singh2022road}}},
% title style={yshift=-0.2cm},
% ylabel={\tiny{F1 (\%)}},
% ymin=30,
% ymax=57
% ]
% % Plot data for BBB in Dataset C
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=orange}] coordinates {
% (5,40.03344690170473) += (0,5.056929754523553) -=(0,5.056929754523553)
% (10,39.497117850555775) += (0,7.324741430969779) -=(0,7.324741430969779)
% (15,41.70045043815289) += (0,4.503083114618855) -=(0,4.503083114618855)
% (20,43.95679573963744) += (0,5.724476140854986) -=(0,5.724476140854986)
% (25,43.91655890502581) += (0,7.397736724682226) -=(0,7.397736724682226)
% (30,40.84849670921552) += (0,6.787114836757397) -=(0,6.787114836757397)
% (35,43.46600586280709) += (0,4.918098156383699) -=(0,4.918098156383699)
% (40,39.33604574991489) += (0,5.587082268904128) -=(0,5.587082268904128)
% (45,41.67154706867009) += (0,4.853981201777168) -=(0,4.853981201777168)
% (50,41.170672005887695) += (0,5.497095966312938) -=(0,5.497095966312938)
% };
% % Plot data for PE in Dataset C
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=cyan}] coordinates {
% (5,46.95914301867057) += (0,6.05243372072422) -=(0,6.05243372072422)
% (10,44.35103292152759) += (0,2.9966140290527696) -=(0,2.9966140290527696)
% (15,44.21816213450458) += (0,6.367049452768733) -=(0,6.367049452768733)
% (20,46.27727705113644) += (0,8.537225520140703) -=(0,8.537225520140703)
% (25,46.26570675100138) += (0,6.5510505190307144) -=(0,6.5510505190307144)
% (30,44.06082062870729) += (0,5.206354977019308) -=(0,5.206354977019308)
% (35,43.00217105321485) += (0,4.690638124617283) -=(0,4.690638124617283)
% (40,44.123030934571204) += (0,2.7004686917469254) -=(0,2.7004686917469254)
% (45,46.803628670707965) += (0,3.749361741586067) -=(0,3.749361741586067)
% (50,43.03308469196917) += (0,4.282980414697678) -=(0,4.282980414697678)
% };
% % Plot data for SE in Dataset C
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=red}] coordinates {
% (5,40.73743947791) += (0,4.009901172910407) -=(0,4.009901172910407)
% (10,40.056428478757425) += (0,4.1730650869813655) -=(0,4.1730650869813655)
% (15,39.41337466881583) += (0,3.053884389004715) -=(0,3.053884389004715)
% (20,37.86591958163472) += (0,0.18746914540579074) -=(0,0.18746914540579074)
% (25,38.723007197626444) += (0,2.3043486029802986) -=(0,2.3043486029802986)
% (30,39.378180178826824) += (0,2.798849989574935) -=(0,2.798849989574935)
% (35,38.776521032744974) += (0,2.282984016798466) -=(0,2.282984016798466)
% (40,38.91944120501935) += (0,3.022983449344767) -=(0,3.022983449344767)
% (45,38.03843460972704) += (0,0.11435775350534193) -=(0,0.11435775350534193)
% (50,38.02956311592404) += (0,0.12290633139707) -=(0,0.12290633139707)
% };
% % Plot data for LL--HMC in Dataset C
% \addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt, error bars/.cd,
% y dir=both, y explicit,
% error bar style={line width=0.1pt, color=yellow}] coordinates {
% (5,33.887826567916576) += (0,6.14351969721661) -=(0,6.14351969721661)
% (10,34.574903713756974) += (0,8.73202224498559) -=(0,8.73202224498559)
% (15,39.24376817031906) += (0,4.248698005147985) -=(0,4.248698005147985)
% (20,39.30141891898767) += (0,6.686024326991105) -=(0,6.686024326991105)
% (25,35.09617883452023) += (0,6.55255084074511) -=(0,6.55255084074511)
% (30,37.58546499845445) += (0,7.258541972058792) -=(0,7.258541972058792)
% (35,35.363634910564876) += (0,6.8963686503265595) -=(0,6.8963686503265595)
% (40,42.42697302678621) += (0,8.566257218440315) -=(0,8.566257218440315)
% (45,32.4746292995812) += (0,5.484626427348374) -=(0,5.484626427348374)
% (50,36.93272588550002) += (0,7.148580495119111) -=(0,7.148580495119111)
% };
% % Single marker for DE at 5 samples
% \addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
% (5,44.57)
% };
% % Single marker for Regular Model at 1 sample
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
% (1,45.36)
% };
% % Single marker for VBLL at 1 sample
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=green}] coordinates {
% % (1,49.34)
% (1,30.63016) += (0,2.317158) -=(0,2.317158)
% };
% \end{groupplot}
% \end{tikzpicture}
% % Legend
% \begin{tikzpicture}
% \hspace{3.5cm}
% \begin{axis}[
% hide axis,
% xmin=0, xmax=1, ymin=0, ymax=1,
% legend style={at={(0.5,0.5)}, anchor=center, legend columns=7},
% legend entries={\tiny{BBB--LL},\tiny{PE--LL},\tiny{SE},\tiny{VBLL}, \tiny{LL--HMC},\tiny{DE},\tiny{Regular},}
% ]
% \addlegendimage{mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=2.0pt}
% \addlegendimage{mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2.0pt}
% \addlegendimage{mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=2pt}
% \addlegendimage{mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=2.0pt}
% \addlegendimage{mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=2.0pt}
% \addlegendimage{mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=2.0pt}
% \addlegendimage{mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=2.0pt}
% \end{axis}
% \end{tikzpicture}
% \caption{Influence of the number of samples or predictions for independent random seeds for each top-performing hyperparameter configurations of the included PDL methods. The regular model (single prediction), DE (five predictions), and VBLL (single prediction) are only included once in each figure. For each number of samples, the whiskers indicate the standard deviation for 10 runs for different random seeds for every number of samples to avoid dependence.}
% \label{fig:method_comparison}
% \end{figure*}
% % \paragraph{\textbf{Number of samples or predictions.}}
% % For a ViT-Base model, processing a single 16-frame video requires approximately $361 \times 10^9$ floating point operations (FLOPs).
% \paragraph{\textbf{Influence of additional predictions or samples.}}
% For the ViT-Base architecture, processing a single 16-frame video requires approximately $361 \times 10^9$ floating point operations (FLOPs). The majority of the FLOPs are required to produce the latent representation on which the last layer methods rely. Every additional last layer sample or prediction requires $7.68$ or $9.22 \times 10^3$ FLOPs for a classifier with 5 or 6 classes respectively. Considering the $361\times 10^9$ FLOPs that are required to produce the latent representation $\mathbf{z}$, the computational impact of the additional predictions are relatively negligible. To examine the influence of additional predictions or last layer parameter samples, we evaluated the top configurations from Table \ref{tab:performance} for 10 seeding positions for each number of samples. Figure \ref{fig:method_comparison} illustrates the average performance and standard deviation for each number of samples. For the LL--PDL methods, no clear trend indicates that additional samples (or predictions) consistently improves the performance. Additionally, we observe that for each dataset another LL--PDL method performs best on average.
% \paragraph{\textbf{Top-10 configurations across random seeds. }}The grid search results in Table \ref{tab:performance} rely on a single random seed and only display the best performing hyperparameters configuration. To avoid basing any performance difference on chance and to examine the reproducibility of the grid search results \citep{bouthillier2019unreproducible}, we evaluated the 10 best performing hyperparameter configurations for 100 different random seeds. Figure \ref{fig:id_performance} illustrates the average DAR and DIR performance for each dataset and method. Similar to Figure \ref{fig:method_comparison}, there is no single method consistently yielding top performance. The VBLL and LL--HMC show more deviations across the datasets, indicating more sensitivity to the different random seeds.
% \paragraph{\textbf{Prior influence.}}
% To explore the effects of the hyperparamters on LL--HMC further, Figure \ref{fig:hmc_perf_samples} illustrates the relationship between the F1-score and the number of HMC samples for different prior scales (standard deviation). Given the cold priors of the last layer (randomly initialized), the stricter priors affect the performance negatively. For the AIDE and B4C datasets, the F1-score quickly saturates after a small number of samples (5–10), which suggests that beyond this point, additional samples do not significantly improve the performance. This indicates that LL--HMC can achieve stable predictive performance with relatively few samples. For the ROAD dataset the performance varies slightly more across different priors and sample sizes. The fluctuations suggest that the dataset's complexity or noise may impact the convergence of the sampling process.
% % To better understand if additional samples or predictions for the LL--PDL methods influence the performance, Figure \ref{fig:method_comparison} visualizes the reproducibility of the results \citep{bouthillier2019unreproducible}.
% % the difference between the performance of the LL--PDL methods, we evaluate the influence of additional samples/predictions on each method. We randomly select ten seeding positions to train each method with the top-performing grid search hyperparameter configuration to evaluate the reproducibility of the results \citep{bouthillier2019unreproducible}. For each number of samples, we randomly selected ten random seed positions to train each method with the top-performing hyperparameter configuration. Figure \ref{fig:method_comparison} illustrates the average and standard deviation for each number of samples. For the LL--PDL methods, no clear trend indicates that additional samples (or predictions) consistently improves the performance. Additionally, we observe that for each dataset another LL--PDL method performs best on average.
% \definecolor{steelblue31119180}{RGB}{31,119,180} % Define a color for the bars
% % \usepackage{pgfplots}
% % \pgfplotsset{compat=1.18}
% % \usepackage{pgfplotstable}
% % \usepackage{caption}
% % \usepackage{tikz}
% % \usetikzlibrary{pgfplots.groupplots}
% \definecolor{steelblue31119180}{RGB}{31,119,180} % Define a color for the bars
% \begin{figure}[t!]
% \centering
% \begin{tikzpicture}
% \begin{groupplot}[
% group style={
% group size=3 by 1, % Three figures in one row
% horizontal sep=1.2cm,
% vertical sep=1.0cm,
% },
% width=6.0cm,
% height=2.5cm,
% ybar,
% enlargelimits=0.15,
% ylabel={\tiny {F1 (\%)}},
% symbolic x coords={BBB--LL, SE, PE--LL, VBLL, LL--HMC},
% xtick=data,
% ymin=0, ymax=100,
% ytick distance=10,
% grid=major,
% ymajorgrids=true,
% tick label style={font=\tiny},
% every node near coord/.append style={
% yshift=3pt,
% font=\footnotesize,
% label style={font=\tiny},
% y tick label style={
% /pgf/number format/fixed,
% /pgf/number format/precision=0,
% /pgf/number format/fixed zerofill=true,
% },
% tick label style={font=\fontsize{4}{4}\selectfont},
% ylabel style={yshift=-0.2cm}, % Shift y-label down
% xlabel style={yshift=0.1cm}, % Shift x-label down
% },
% ]
% % First subplot
% \nextgroupplot[title={\tiny \textbf{AIDE} \citep{yang2023aide}},
% title style={yshift=-0.2cm},
% ylabel style={yshift=-0.2cm},
% ymin=70, ymax=100,]
% \addplot[
% fill=steelblue31119180!60,
% draw=steelblue31119180,
% error bars/.cd,
% y dir=both,
% y explicit,
% ] coordinates {
% (SE, 82.1926) +-(0.3311,0.3311)
% (PE--LL, 81.7069) +-(1.0223,1.0223)
% (BBB--LL, 81.594) +-(0.8308,0.8308)
% (VBLL, 79.0825) +-(2.9116,2.9116)
% (LL--HMC, 82.22345) +-(0.8713,0.8713)
% };
% % Second subplot
% \nextgroupplot[title={\tiny \textbf{B4C} \citep{jain2015car}},
% title style={yshift=-0.2cm},
% ylabel style={yshift=-0.2cm},
% ymin=80, ymax=100,]
% \addplot[
% fill=steelblue31119180!60,
% draw=steelblue31119180,
% error bars/.cd,
% y dir=both,
% y explicit,
% ] coordinates {
% (BBB--LL, 91.5455) +-(1.8499,1.8499)
% (SE, 94.3816) +-(0.2,0.2)
% (PE--LL, 93.6392) +-(1.1009,1.1009)
% (VBLL, 89.4546) +-(3.1822,3.1822)
% (LL--HMC, 92.1997) +-(3.1954,3.1954)
% };
% \nextgroupplot[title={\tiny \textbf{ROAD} \citep{singh2022road}},
% title style={yshift=-0.2cm}, ymin=20, ymax=60, ylabel style={yshift=-0.2cm},]
% \addplot[
% fill=steelblue31119180!60,
% draw=steelblue31119180,
% error bars/.cd,
% y dir=both,
% y explicit,
% ] coordinates {
% (BBB--LL, 44.6598) +-(5.0892,5.0892)
% (SE, 39.379) +-(2.9448,2.9448)
% (PE--LL, 44.2878) +-(4.5929,4.5929)
% (VBLL, 34.0378) +-(5.126,5.126)
% (LL--HMC, 38.3604) +-(4.7721,4.7721)
% };
% \end{groupplot}
% \end{tikzpicture}
% \caption{Average performance across the included LL--PDL methods for the top 10 grid search hyperparameter configurations across 100 random seeds.}
% \vspace{-0.2cm}
% \label{fig:id_performance}
% \end{figure}
% % \begin{tcolorbox}[
% % colback=white, % Background color of the box
% % colframe=black, % Border color of the box
% % arc=4pt, % Rounded corners
% % boxrule=1pt, % Border width
% % left=6pt, % Left padding
% % right=6pt, % Right padding
% % top=2pt, % Top padding
% % bottom=2pt, % Bottom padding
% % width=0.98\linewidth % Width of the box
% % ]
% % \textit{LL--HMC matches and occasionally outperforms the other included LL--PDL methods. Compared to the regular softmax-based model, BBB--LL, PE--LL, SE, VBLL and LL--HMC improve the performance but tend to produce a lower ACE. Better classification performance does not result in uncertainty estimates that allow for better discrimination between correctly and incorrectly in-distribution predictions.}
% % \end{tcolorbox}
% \definecolor{darkgray176}{RGB}{176,176,176}
% \definecolor{steelblue31119180}{RGB}{31,119,180}
% \begin{figure*}[!t]
% \begin{tikzpicture}
% \begin{groupplot}[
% group style={
% group size=3 by 1,
% horizontal sep=1.2cm,
% vertical sep=1.5cm,
% },
% width=6.0cm,
% height=2.5cm,
% grid=major,
% xlabel={\tiny{Number of samples}},
% % xmode=log,
% xtick={2,5,10,15,20,25,30,35,40,45,50},
% % xticklabels={0.03,0.1,0.3,1,3,10},
% tick label style={font=\tiny},
% label style={font=\tiny},
% y tick label style={
% /pgf/number format/fixed,
% /pgf/number format/precision=0,
% /pgf/number format/fixed zerofill=true,
% },
% tick label style={font=\fontsize{4}{4}\selectfont},
% ylabel style={yshift=-0.2cm}, % Shift y-label down
% xlabel style={yshift=0.1cm}, % Shift x-label down
% ]
% % Third plot (Accuracy for AIDE Dataset)
% \nextgroupplot[
% title={\tiny{\textbf{AIDE} \citep{yang2023aide}}},
% ylabel={\tiny{F1 (\%)}},
% title style={yshift=-0.2cm},
% % ytick={50,70,90},
% ymin=0,
% ymax=100 % Adjusted for AIDE dataset accuracy values
% ]
% % Plot data for each prior value in AIDE dataset
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt] coordinates {
% (2,14.0021344717182)
% (5,14.017094017094)
% (10,14.0021344717182)
% (15,14.0021344717182)
% (20,14.0021344717182)
% (25,14.0021344717182)
% (30,14.0021344717182)
% (35,14.0021344717182)
% (40,14.0021344717182)
% (45,14.0021344717182)
% (50,14.0021344717182)
% };
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1.0pt] coordinates {
% (2,44.0785617720902)
% (5,45.5579271170791)
% (10,44.619027696427)
% (15,45.6406287596942)
% (20,45.6524267426279)
% (25,46.6828965371792)
% (30,45.1265108186647)
% (35,45.294367914765)
% (40,45.294367914765)
% (45,45.294367914765)
% (50,45.294367914765)
% };
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1pt] coordinates {
% (2,81.1265996341357)
% (5,81.4880528149863)
% (10,80.6886033388472)
% (15,80.8240769818706)
% (20,80.8240769818706)
% (25,81.7497542395929)
% (30,81.147775318895)
% (35,80.6886033388472)
% (40,81.2682541885595)
% (45,81.2682541885595)
% (50,80.8240769818706)
% };
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.0pt] coordinates {
% (2,81.9988673273403)
% (5,81.0725353796292)
% (10,83.2210460012779)
% (15,82.8653507807344)
% (20,82.6611086890146)
% (25,82.5995157384987)
% (30,82.2820554210384)
% (35,82.932999137567)
% (40,83.5649615920802)
% (45,82.3797680158336)
% (50,82.8254570288468)
% };
% \addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.0pt] coordinates {
% (2,82.5713665407654)
% (5,81.3040212030037)
% (10,82.3251434729353)
% (15,82.3119087170357)
% (20,82.6568972522956)
% (25,83.0308431010185)
% (30,82.5466618076418)
% (35,82.2070251182043)
% (40,83.3504293628523)
% (45,82.5884467876544)
% (50,82.7713852772888)
% };
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=yellow,}, mark size=1.0pt] coordinates {
% (2,80.7062045482558)
% (5,81.8924768551738)
% (10,81.6863324033022)
% (15,81.7573352754438)
% (20,82.0510107716231)
% (25,81.8958601891935)
% (30,82.1463683907081)
% (35,81.4699359624406)
% (40,81.894579329894)
% (45,81.9912178082785)
% (50,82.3647639021652)
% };
% \nextgroupplot[
% title={\tiny{\textbf{B4C} \citep{jain2015car}}},
% ylabel={\tiny{F1 (\%)}},
% title style={yshift=-0.2cm},
% % ytick={40,50,60},
% ymax=100,
% ymin=0,
% ]
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange, }, mark size=1.0pt] coordinates {
% (2,11.0045871559633)
% (2,11.0045871559633)
% (5,6.80851063829787)
% (10,11.6363636363636)
% (15,11.6363636363636)
% (20,16.1375661375661)
% (25,12.1518987341772)
% (30,11.6363636363636)
% (35,11.6363636363636)
% (40,11.7791411042944)
% (45,11.6363636363636)
% (50,11.6363636363636)
% };
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan, }, mark size=1.0pt] coordinates {
% (2,11.6363636363636)
% (5,11.6363636363636)
% (10,11.6363636363636)
% (15,11.6363636363636)
% (20,11.6363636363636)
% (25,11.6363636363636)
% (30,11.6363636363636)
% (35,11.6363636363636)
% (40,11.6363636363636)
% (45,11.6363636363636)
% (50,11.6363636363636)
% };
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red, }, mark size=1.0pt] coordinates {
% (2,92.612915759661)
% (5,89.2186741363212)
% (10,90.3246376811594)
% (15,90.2082047003198)
% (20,90.2082047003198)
% (25,90.3246376811594)
% (30,90.3246376811594)
% (35,90.3246376811594)
% (40,90.2374331550802)
% (45,90.3246376811594)
% (50,91.4892504519538)
% };
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green, }, mark size=1.0pt] coordinates {
% (2,93.1248656780571)
% (5,91.9724578203374)
% (10,92.0444444444444)
% (15,93.5018046485499)
% (20,92.882146160962)
% (25,91.991304347826)
% (30,93.7147086031452)
% (35,92.6623462346234)
% (40,92.882146160962)
% (45,92.0444444444444)
% (50,94.3356005165734)
% };
% \addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=pink, }, mark size=1.0pt] coordinates {
% (2,89.0888888888888)
% (5,89.452427184466)
% (10,91.0268584403197)
% (15,93.546728151076)
% (20,94.3356005165734)
% (25,94.3356005165734)
% (30,92.8344673265824)
% (35,92.8777777777777)
% (40,95.0027752081406)
% (45,94.3816032667096)
% (50,92.882146160962)
% };
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=yellow, }, mark size=1.0pt] coordinates {
% (2,85.2579288323969)
% (5,90.2374331550802)
% (10,91.7308282160585)
% (15,95.0027752081406)
% (20,93.7332099907493)
% (25,90.8346405228758)
% (30,94.3356005165734)
% (35,92.8777777777777)
% (40,92.7064935064935)
% (45,92.8777777777777)
% (50,90.7626538987688)
% };
% \nextgroupplot[
% title={\tiny{\textbf{ROAD} \citep{singh2022road}}},
% ylabel={\tiny{F1 (\%)}},
% title style={yshift=-0.2cm},
% % ytick={20,40,60,80},
% ymin=0,
% ymax=60
% ]
% % Plot data for each prior value in AIDE dataset
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt] coordinates {
% (2,8.83652430044182)
% (5,4.64285714285714)
% (10,13.5233444616074)
% (15,8.83652430044182)
% (20,20.1298701298701)
% (25,19.5238095238095)
% (30,8.83652430044182)
% (35,8.83652430044182)
% (40,8.83652430044182)
% (45,8.83652430044182)
% (50,10.969387755102)
% };
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1.0pt] coordinates {
% (2,12.8320802005012)
% (5,18.4417086672725)
% (10,17.2532781228433)
% (15,12.8320802005012)
% (20,15.9392789373814)
% (25,14.4756838905775)
% (30,15.9392789373814)
% (35,10.969387755102)
% (40,14.4756838905775)
% (45,17.2532781228433)
% (50,14.4756838905775)
% };
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1pt] coordinates {
% (2,32.6147576147576)
% (5,36.200555737169)
% (10,38.2060502426635)
% (15,32.9069049435182)
% (20,32.4157002417872)
% (25,31.4532453297524)
% (30,36.4112285230297)
% (35,32.7231121281464)
% (40,34.1060337178349)
% (45,31.4532453297524)
% (50,31.3196033562166)
% };
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.0pt] coordinates {
% (2,32.5569358178053)
% (5,41.3946922642574)
% (10,39.881051011954)
% (15,34.1855520116389)
% (20,36.4604292421193)
% (25,37.4110223799664)
% (30,37.3990175780457)
% (35,38.7693309616151)
% (40,37.3990175780457)
% (45,45.1818230079099)
% (50,38.7693309616151)
% };
% \addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.0pt] coordinates {
% (2,36.1268193195513)
% (5,38.6976638921718)
% (10,31.7531179138321)
% (15,40.382975932848)
% (20,41.1269889530759)
% (25,37.3990175780457)
% (30,40.0163452108532)
% (35,36.6398733406406)
% (40,38.494528246081)
% (45,41.4699324545871)
% (50,41.1269889530759)
% };
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=yellow,}, mark size=1.0pt] coordinates {
% (2,23.8758067329495)
% (5,39.196334345588)
% (10,47.3923943568408)
% (15,37.8867588635307)
% (20,45.2957704821058)
% (25,40.3683120845592)
% (30,39.7268204940839)
% (35,38.1645285279289)
% (40,39.2487545925163)
% (45,39.8514719620189)
% (50,39.5121435301809)
% };
% \end{groupplot}
% \end{tikzpicture}
% % Legend
% \begin{tikzpicture}
% \hspace{5.31cm}
% \begin{axis}[
% hide axis,
% xmin=0, xmax=1, ymin=0, ymax=1,
% legend style={at={(0.5,0.5)}, anchor=center, legend columns=7},
% legend entries={\tiny{0.01},\tiny{0.1},\tiny{1},\tiny{2.5},\tiny{5.0},\tiny{10.0},\tiny{50}}
% ]
% \addlegendimage{mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=2.0pt}
% \addlegendimage{mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2.0pt}
% \addlegendimage{mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=2pt}
% \addlegendimage{mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=2.0pt}
% \addlegendimage{mark=diamond*, thin,line width=0.25pt, mark options={fill=pink}, mark size=2.0pt}
% \addlegendimage{mark=oplus*, thin,line width=0.25pt, mark options={fill=yellow}, mark size=2.0pt}
% % \addlegendimage{mark=otimes*, thin, line width=0.25pt, mark options={fill=brown}, mark size=2.0pt}
% \end{axis}
% \end{tikzpicture}
% \caption{Effect of different prior standard deviation values and the number of samples on the LL--HMC in-distribution performance with a target acceptance of 0.7 and 100 burn-in samples.}
% \vspace{-0.1cm}
% \label{fig:hmc_perf_samples}
% \end{figure*}
% % \begin{tcolorbox}[
% % colback=white, % Background color of the box
% % colframe=black, % Border color of the box
% % arc=4pt, % Rounded corners
% % boxrule=1pt, % Border width
% % left=6pt, % Left padding
% % right=6pt, % Right padding
% % top=2pt, % Top padding
% % bottom=2pt, % Bottom padding
% % width=0.98\linewidth % Width of the box
% % ]
% % \textit{Stricter priors negatively impact the exploration and performance.}
% % \end{tcolorbox}
% % Figure \ref{fig:method_comparison} shows the sensitivity of LL--HMC when using additional sampled classification layers.
% % Similar to the grid-search results, we observe no definitive best method and the addition of additional samples
% % we used the best hyperparameter configuration for each method and dataset.
% % Figure \ref{fig:method_comparison} illustrates the effects of additional samples or predictions on the performance for the PDL methods. For none of the evaluated PDL methods there is a clear trend that indicates that additional samples or predictions consistently improves the performance.
% % \paragraph{\textbf{Number of samples and prior influence.}}
% % % For a ViT-Base model, processing a single 16-frame video requires approximately $361 \times 10^9$ floating point operations (FLOPs).
% % For the ViT-Base architecture, processing a single 16-frame video requires approximately $361 \times 10^9$ floating point operations (FLOPs). The majority of the FLOPs are required to produce the latent representation on which the last layer methods rely. Every additional last layer sample or prediction requires $7.68$ or $9.22 \times 10^3$ FLOPs for a classifier with 5 or 6 classes respectively. Considering the $361\times 10^9$ FLOPs that are required to produce the latent representation $\mathbf{z}$, the computational impact of the additional predictions are relatively negligible. Figure \ref{fig:method_comparison} illustrates the effects of additional samples or predictions on the performance for the PDL methods. For none of the evaluated PDL methods there is a clear trend that indicates that additional samples or predictions consistently improves the performance.
% % \begin{tcolorbox}[
% % colback=white, % Background color of the box
% % colframe=black, % Border color of the box
% % arc=4pt, % Rounded corners
% % boxrule=1pt, % Border width
% % left=6pt, % Left padding
% % right=6pt, % Right padding
% % top=2pt, % Top padding
% % bottom=2pt, % Bottom padding
% % width=0.98\linewidth % Width of the box
% % ]
% % \textit{Using more samples or predictions does not consistently result in improved performance. For the included LL--PDL methods, additional samples or predictions do not LL--HMC, the number of samples saturates quickly.}
% % \end{tcolorbox}
% \paragraph{\textbf{Effective sample size and parameter trajectories.}} Figure \ref{fig:ess} shows a histogram with 20 bins with the ESS values on the x-axis and the number of parameters on the y-axis, and two parameter trajectories. Similar to \citep{vellenga2024pthmc}, we observe that LL--HMC only samples effectively for a few samples. Looking at the individual parameter trajectories, we observe no trend or drift and the values oscillate around 0, which can indicate limited exploration.
% \begin{figure}[t]
% \centering
% \begin{minipage}[t]{0.27\linewidth}
% \centering
% \includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/ess_aide.pdf}
% \vspace{-0.5cm}
% \captionsetup{font=tiny}
% \caption*{ESS per last layer parameter for the AIDE dataset with a single chain for 50 samples.}
% % \label{fig:nuscenes_rhat_aide1}
% \end{minipage}
% \hfill
% \begin{minipage}[t]{0.27\linewidth}
% \centering
% \includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/param_1.pdf}
% \vspace{-0.5cm}
% \captionsetup{font=tiny}
% \caption*{Value for the first weight parameter of the last layer for each sample.}
% % \label{fig:nuscenes_rhat_b4c1}
% \end{minipage}
% \hfill
% \begin{minipage}[t]{0.27\linewidth}
% \centering
% \includegraphics[width=0.99\linewidth, keepaspectratio]{param_51.pdf}
% \vspace{-0.5cm}
% \captionsetup{font=tiny}
% \caption*{Value for the fifty-first weight parameter of the last layer for each sample.}
% % \label{fig:nuscenes_rhat_road1}
% \end{minipage}
% \caption{Visualization of the ESS and the parameter values for 50 samples for LL--HMC for the AIDE dataset with a single chain, target acceptance of 0.7, prior standard deviation of 1, and 100 burn in samples.}
% \label{fig:ess}
% \end{figure}
% \begin{figure}[t!]
% \centering
% \begin{minipage}[t]{0.27\linewidth}
% \centering
% \includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_aide_single.pdf}
% \vspace{-0.6cm}
% \captionsetup{font=tiny}
% \caption*{AIDE - Two Chains, One Starting Position.}
% \label{fig:nuscenes_rhat_aide1}
% \end{minipage}
% \hfill
% \begin{minipage}[t]{0.27\linewidth}
% \centering
% \includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_b4c_single.pdf}
% \vspace{-0.6cm}
% \captionsetup{font=tiny}
% \caption*{B4C - Two Chains, One Starting Position.}
% \label{fig:nuscenes_rhat_b4c1}
% \end{minipage}
% \hfill
% \begin{minipage}[t]{0.27\linewidth}
% \centering
% \includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_oxf_single.pdf}
% \vspace{-0.6cm}
% \captionsetup{font=tiny}
% \caption*{ROAD - Two Chains, One Starting Position.}
% \label{fig:nuscenes_rhat_road1}
% \end{minipage}
% % Second row - 3 columns
% \begin{minipage}[t]{0.27\linewidth}
% \centering
% \includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_aide_ens.pdf}
% \vspace{-0.6cm}
% \captionsetup{font=tiny}
% \caption*{AIDE - Two Chains, Two Starting Positions.}
% \label{fig:nuscenes_rhat_aide2}
% \end{minipage}
% \hfill
% \begin{minipage}[t]{0.27\linewidth}
% \centering
% \includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_b4c_ens.pdf}
% \vspace{-0.6cm}
% \captionsetup{font=tiny}
% \caption*{B4C - Two Chains, Two Starting Positions.}
% \label{fig:nuscenes_rhat_b4c2}
% \end{minipage}
% \hfill
% \begin{minipage}[t]{0.27\linewidth}
% \centering
% \includegraphics[width=0.99\linewidth, keepaspectratio]{rhat/rhat_oxf_ens.pdf}
% \vspace{-0.6cm}
% \captionsetup{font=tiny}
% % \vspace{-0.3cm}
% \caption*{ROAD - Two Chains, Two Starting Positions.}
% \label{fig:nuscenes_rhat_road2}
% \end{minipage}
% \caption{Log-scale histograms of the $\hat{R}$ values for two chains for single starting position (top) or two starting positions (bottom), for 100 burn-in samples, target acceptance of 0.7, 50 samples and a prior scale of 1.}
% \label{fig:rhats}
% \end{figure}
% \begin{table*}[t!]
% \centering
% \caption{LL--HMC DAR and DIR classification performance for multiple starting positions and chains.}
% \vspace{-0.2cm}
% \resizebox{\linewidth}{!}{
% \begin{tabular}{lcc|c|c|c|c||c|c|c|c||c|c|c|c}
% \toprule
% & & & \multicolumn{4}{c||}{\textbf{AIDE} \citep{yang2023aide}} & \multicolumn{4}{c||}{\textbf{B4C} \citep{jain2015car}} & \multicolumn{4}{c}{\textbf{ROAD} \citep{singh2022road}} \\
% & & & \textbf{Acc ($\uparrow$)} & \textbf{F1 ($\uparrow$)} & \textbf{ACE ($\downarrow$)} & \textbf{rAULC ($\uparrow$)} & \textbf{Acc ($\uparrow$)} & \multicolumn{1}{|c|}{\textbf{F1 ($\uparrow$)}} & \textbf{ACE ($\downarrow$)} & \textbf{rAULC ($\uparrow$)} & \textbf{Acc ($\uparrow$)} & \textbf{F1 ($\uparrow$)} & \textbf{ACE ($\downarrow$)} & \textbf{rAULC ($\uparrow$)} \\
% \midrule
% \multirow{4}{*}{\textbf{LL--HMC}} & \multirow{2}{*}{\small{S=1}} & \small{C=1} & 89.16 & 84.29 & 0.027 & 0.73 & 94.02 & 95.00 & 0.025 & 0.69 & 73.13 & 54.73 & 0.054 & 0.57 \\ %AIDE: {'num_sampels': 30, 'burn_in': 50, 'prior': 2,5, 'ta': 0,7} %b4c {'num_sampels': 20, 'burn_in': 100, 'prior': 5, 'ta': 0.7} %OXF {'num_sampels': 40, 'burn_in': 20, 'prior': 5, 'ta': 0.7}
% & & \small{C=2} & {89.82} & 85.24 & 0.029 & 0.68 & 94.02 & 95.00 & 0.024 & 0.75 & 71.64 & 59.01 & 0.070 & 0.57 \\ \cmidrule{2-15} %AIDE {'num_samples': 40, 'burn_in': 20, 'prior': 10, 'ta': 0.6} %B4C {'num_samples': 40, 'burn_in': 60, 'prior': 5, 'ta': 0,6} % {'num_samples': 25, 'burn_in': 20, 'prior': 5, 'ta': 0,6}
% & \multirow{2}{*}{\small{S=2}} & \small{C=1} & 88.67 & 82.58 & 0.047 & 0.61 & 94.02 & 95.01 & 0.083 & 0.42 & 73.13 & 58.24 & 0.062 & 0.36 \\
% & & \small{C=2} & 87.36 & 82.50 & 0.032 & 0.76 & 94.01 & 95.03 & 0.114 & 0.63 & 76.12 & 52.92 & 0.111 & 0.28 \\
% \bottomrule
% \end{tabular}
% }
% \label{tab:performance_llhmc}
% \vspace{-0.1cm}
% \end{table*}
% \paragraph{\textbf{Multiple chains and starting positions.}}
% Table \ref{tab:performance_llhmc} shows the top-performing configurations for multiple starting (S) positions and multiple chains (C). Increasing the number of chains can improve the performance. For example, for a single starting position, the best LL--HMC configuration improved the F1-score from 84.29\% to 85.24\% and from 54.73\% to 59.01\% when adding an additional chain for the AIDE and ROAD datasets. Adding an additional starting position, however, does not consistently improve the performance.
% When using multiple chains, it is also important to assess whether the chains have explored distinct regions of the posterior and converged to the target distribution. Figure \ref{fig:rhats} shows the $\hat{R}$ values for the scenarios with multiple chains for both single and multiple starting positions. For all datasets the $\hat{R}$ values are close to 1 and mostly below 1,1. This indicates that the chains have likely converged to the target distribution but also potentially sample from the same region \citep{izmailov2021bayesian}. %
% % \begin{tcolorbox}[
% % colback=white, % Background color of the box
% % colframe=black, % Border color of the box
% % arc=4pt, % Rounded corners
% % boxrule=1pt, % Border width
% % left=6pt, % Left padding
% % right=6pt, % Right padding
% % top=2pt, % Top padding
% % bottom=2pt, % Bottom padding
% % width=0.98\linewidth % Width of the box
% % ]
% % \textit{Additional starting positions do not yield a consistent performance improvement, whereas another chain can be beneficial for some scenarios. The $\hat{R}$ values indicate that the multiple chains potentially sample from the same region, highlighting the limitation of the reliance on the latent representations.}
% % \end{tcolorbox}
% % \paragraph{\textbf{Additional starting positions and chains.}}
% % For the LL--HMC approach, the best-performing configurations are shown for the number of starting positions (S) and number chains (C). For a single starting position with one or two chains, both best-performing configurations rank among the best performing methods overall. Adding another starting position significantly increases the number FLOPs, and does not consistently lead to improved performance. From a confidence-performance calibration perspective, the BBB--LL, PE--LL, SE and LL--HMC yield a lower ACE for the B4C and ROAD datasets compared to the regular model, but not for the AIDE dataset. Additionally, no single method optimally balances both classification performance and calibration across all datasets. The regular softmax-based model is better calibrated than the LL--HMC approach, except for the B4C dataset. We do not observe that the improved classification performance of LL-HMC in terms of accuracy or F1-score also results in an improved ability to distinguish between correctly and incorrectly predicted in-distribution instances based on the uncertainty estimations.
% %Therefore, similar to the ensemble, the additional required computational resource is not clearly justifiable given these inconsistent results.
% % The LL--HMC approach demonstrates top performance across all datasets. Notably, LL--HMC (S=1, C=2) is most consistent across the datasets, which indicates the stability of the approach. Adding additional chains is not a guarantee for improved performance compared to the single chain.
% % Producing an additional starting position, requires another independently fine-tuned ViT-Base model and thus doubles the number of FLOPs. However, the across the
% % Doubling the computational demand to produce additional starting positions is significantly improves the performance compared to leveraging a single starting position.
% % leads to better uncertainty
% % e improved performance tends to lead to more overconfident predictions. While each dataset has at least one PDL method that outperforms LL--HMC in terms of rAULC, the best-performing method varies, and LL--HMC does not consistently achieve the lowest rAULC value. This may indicate a trade-off between model flexibility and uncertainty calibration. Furthermore, the dataset characteristics may play a role here as well. The ROAD dataset is the smallest in terms of size and yields a lower overall predictive performance for all models.
% % In such cases, methods that impose stronger constraints on uncertainty modeling, can potentially provide more calibrated estimates.
% % \paragraph{\textbf{Intermediate latent representation performance.}}
% % Given the limited computational resources of the target operating environment and that the latent representations can be already classified before the final layer of the network (e.g., \citealt{papyan2020prevalence,kothapallineural,qendro2021early,montavon2011kernel,meronen2024fixing}), we evaluate if the intermediate latent representations in combination with the LL--PDL methods can yield similar performance. Using an intermediate latent representation could significantly reduce the processing requirements for incoming videos. For each Transformer block in the ViT architecture, we apply average pooling to extract the intermediate representations. For each intermediate representation, we use the top-performing configurations of the performance experiment to train the LL--PDL methods and evaluate if earlier representations from the pre-trained network can yield similar performance.
% % It is possible to also consider intermediate latent representations to reduce the computations even further, given that DNNs commonly classify well before the final latent representation \citep{kothapallineural,meronen2024fixing, montavon2011kernel, papyan2020prevalence,qendro2021early}. For scenarios where safety-critical down-stream tasks are based on pre-trained or fine-tuned representations of foundation models, it becomes difficult and computationally expensive to retrain an entire model every time for a new additional task.
% % \begin{figure}[t]
% % \centering
% % % Figure on the left side
% % \begin{minipage}[t]{0.47\linewidth}
% % \centering
% % % \hspace{0.1cm}
% % \includegraphics[width=0.80\linewidth, keepaspectratio]{rhat/rhat_aide.pdf}
% % \vspace{-0.3cm}
% % \caption*{AIDE - Two Chains, One Starting Position.}
% % % \vspace{0.1cm}
% % \label{fig:nuscenes_rhat}
% % \end{minipage}
% % % \vspace{0.2cm}
% % \hfill
% % \begin{minipage}[t]{0.47\linewidth}
% % \centering
% % \includegraphics[width=0.80\linewidth, keepaspectratio]{rhat/rhat_aide_ens.pdf}
% % \vspace{-0.3cm}
% % \caption*{AIDE - Two Chains, Two Starting Positions.}
% % % \vspace{0.1cm}
% % \label{fig:nuscenes_rhat}
% % \end{minipage}
% % \begin{minipage}[t]{0.47\linewidth}
% % \centering
% % % \hspace{0.1cm}
% % \includegraphics[width=0.80\linewidth, keepaspectratio]{rhat/rhat_b4c.pdf}
% % \vspace{-0.3cm}
% % \caption*{B4C - Two Chains, One Starting Position.}
% % \label{fig:nuscenes_rhat}
% % \end{minipage}
% % \hfill
% % \begin{minipage}[t]{0.47\linewidth}
% % \centering
% % \includegraphics[width=0.80\linewidth, keepaspectratio]{rhat/rhat_b4c_ens.pdf}
% % \vspace{-0.3cm}
% % \caption*{B4C - Two Chains, Two Starting Positions.}
% % \label{fig:nuscenes_rhat}
% % \end{minipage}
% % \begin{minipage}[t]{0.47\linewidth}
% % \centering
% % % \hspace{0.1cm}
% % \includegraphics[width=0.80\linewidth, keepaspectratio]{rhat/rhat_oxf.pdf}
% % \vspace{-0.3cm}
% % \caption*{ROAD - Two Chains, One Starting Position.}
% % \label{fig:nuscenes_rhat}
% % % \vspace{0.1cm}
% % \end{minipage}
% % % \vspace{0.1cm}
% % \hfill
% % \begin{minipage}[t]{0.47\linewidth}
% % \centering
% % \includegraphics[width=0.80\linewidth, keepaspectratio]{rhat/rhat_oxf_ens.pdf}
% % \vspace{-0.3cm}
% % \caption*{ROAD - Two Chains, Two Starting Positions.}
% % \label{fig:nuscenes_rhat}
% % % \vspace{0.1cm}
% % \end{minipage}
% % % \vspace{0.1cm}
% % \caption{Log-scale histograms of the $\hat{R}$ values for two chains for with the same starting position (left) or two starting positions (right), 100 burn-in samples, target acceptance of 0.7, 50 samples and a Gaussian prior of 1. }%Total number of weight parameters for each last layer is 768 (the number ViT-Base feature dimensions).
% % % \vspace{-0.3cm}
% % \label{fig:rhats}
% % \end{figure}
% % \paragraph{\textbf{Cold vs. Warm starts.}}
% % \subsection{Intermediate latent representation performance}
% % \begin{table}[!h]
% % \centering
% % \caption{Uncertainty-based out-of-distribution detection results for the ROAD dataset (OOD min).}
% % \resizebox{0.37\textwidth }{!}{
% % \begin{tabular}{lc|c|c|c}
% % \toprule
% % \multicolumn{2}{c|}{} & \multicolumn{3}{c}{\textbf{ROAD (OOD min)}} \\
% % \cmidrule{3-5}
% % \multicolumn{2}{c|}{} & \textbf{ROC-AUC ($\uparrow$)} & \textbf{PR-AUC ($\uparrow$)} & \textbf{FPR95 ($\downarrow$)} \\
% % \midrule
% % \textbf{Regular} & & 0.51 & 0.37 & 0.90 \\
% % \textbf{DE (N=5)} & & 0.55& 0.41 & 0.88 \\
% % \textbf{DDU} & & 0.63 & 0.44 & 0.87 \\
% % \textbf{BBB--LL} & & 0.69 & 0.44 & 0.58 \\
% % \textbf{PE--LL} & & 0.57 & 0.55 & 0.78 \\
% % \textbf{SE} & & 0.71 & 0.52 &
% % 0.50 \\
% % % \textbf{cSGHMC} & & & \\
% % \textbf{VBLL} & & 0.63 & 0.44 & 0.61 \\\midrule
% % \multirow{4}{*}{\textbf{LL--HMC}} & \small{S=1, C=1} & 0.73 & 0.53 & 0.50 \\
% % & \small{S=1, C=2}& 0.67 & 0.49 & 0.64 \\
% % & \small{S=2, C=1}& 0.37 & 0.37 & 0.98 \\
% % & \small{S=2, C=2}& 0.45 & 0.34 & 0.97 \\
% % \bottomrule
% % \end{tabular}
% % }
% % \label{tab:ROAD_ood_min_exp}
% % \end{table}
% % T
% % his suggests that, while LL--HMC frequently can yield the best performance, there is no single best LL--PDL approach.
% %The reliance on the fixed latent representations might this might indicate the equilibrium within the posterior.
% % From a FLOPs perspective, every additional prediction, sample or prediction requires $7.68$ or $9.22 \times 10^3$ FLOPs for a classifier with 5 or 6 classes respectively. Considering $361\times 10^9$ FLOPS to produce the latent representation $z$, the computational impacts of the additional predictions seems negligible.
% %While some configurations appear to stabilize, the fluctuations even at higher numbers of samples suggest that the dataset's complexity or noise may impact the convergence of the sampling process.
% % The choice of prior standard deviation strongly influences performance across all datasets. The smaller priors (0.01 and 0.1) result in poor performance and may overly constrain the last layer weights. This can lead to a limited exploration of the posterior. The remaining priors generally yield better performance, particularly in AIDE and B4C, where the different priors maintain consistently high F1-scores across different sample sizes.
% \begin{table*}[t]
% \centering
% \caption{Uncertainty-based out-of-distribution detection. C=Chains, S=Starting Positions, ROC-AUC=Receiver Operating Characteristic - Area Under the Curve, PR=Precision-Recall - Area Under the Curve. }
% \resizebox{\textwidth }{!}{
% \begin{tabular}{lc|c|c|c|c|c|c||c|c|c|c|c|c||c|c|c}
% \toprule
% \multicolumn{2}{c|}{} & \multicolumn{6}{c||}{\textbf{AIDE} \citep{yang2023aide}} & \multicolumn{6}{c||}{\textbf{B4C} \citep{jain2015car}} & \multicolumn{3}{c}{\textbf{ROAD} \citep{singh2022road}} \\ \cmidrule{3-17}
% \multicolumn{2}{c|}{} & \multicolumn{3}{c|}{\textbf{OOD min}} & \multicolumn{3}{c||}{\textbf{OOD max}} & \multicolumn{3}{c|}{\textbf{OOD min}} & \multicolumn{3}{c||}{\textbf{OOD max}} & \multicolumn{3}{c}{\textbf{OOD min}} \\
% \multicolumn{2}{c|}{} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
% \begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
% \begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
% \begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
% \begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
% \begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
% \begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
% \begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
% \begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} & \begin{tabular}{@{}c@{}}\textbf{ROC-}\\ \textbf{AUC} ($\uparrow$)\end{tabular} &
% \begin{tabular}{@{}c@{}}\textbf{PR-}\\\textbf{AUC} ($\uparrow$)\end{tabular} &
% \begin{tabular}{@{}c@{}}\textbf{FPR95}\\($\downarrow$)\end{tabular} \\
% \midrule
% \textbf{Regular} & & 0.55 & 0.19 & 0.91 & 0.82 & 0.94 & 0.41 & 0.51 & 0.37 & 0.90 & 0.64 & 0.82 & 0.84 & 0.19 & 0.25 & 1.00 \\
% \textbf{DE (N=5)} & & 0.69 & 0.28 & 0.85 & 0.84 & 0.94 &0.35 & 0.52 & 0.39 & 0.90 & 0.79 & 0.90 & 0.45 & 0.55& 0.41 & 0.88 \\
% \textbf{DDU} & &0.73 & 0.31 & 0.49 & 0.80 & 0.90 & 0.45 & 0.63 & 0.44 &0.87 & 0.48 & 0.75 & 0.96 & 0.63 & 0.44 & 0.87 \\
% \textbf{BBB--LL} && 0.85 & 0.47 & 0.35 & 0.94 &
% 0.98 & 0.23 & 0.69 & 0.44 & 0.58 & 0.69 & 0.87 & 0.83 & 0.69 & 0.44 & 0.58 \\
% \textbf{PE--LL} & & 0.55 & 0.20 & 0.92 & 0.64 & 0.59 & 0.57 & 0.57 & 0.55 & 0.78 & 0.56 & 0.55 & 0.94 & 0.57 & 0.55 & 0.78 \\
% \textbf{SE} & & 0.55 & 0.20 & 0.92 & 0.88 & 0.96 & 0.41 & 0.71 & 0.52 & 0.50 & 0.72 & 0.88 & 0.78 & 0.71 & 0.52 & 0.50 \\
% % \textbf{cSGHMC} & & & & & & & & & & & & \\
% \textbf{VBLL} & & 0.56 & 0.19 & 0.88 & 0.85 & 0.95 & 0.42 & 0.63 & 0.44 & 0.61& 0.56 & 0.79 & 0.91 & 0.63 & 0.44 & 0.61 \\ \midrule
% % \textbf{LL--HMC} & 0.86 & 0.47 & 0.34 & 0.95 & 0.98 & 0.16 & 0.73 & 0.53 & 0.50 & 0.90 & 0.96 & 0.33 & 0.73 & 0.53 & 0.50 \\
% \multirow{4}{*}{\textbf{LL--HMC}} & \small{S=1, C=1}& 0.79 & 0.35 & 0.44 & 0.89 & 0.96 & 0.33 & 0.73 & 0.53 & 0.50& 0.90 & 0.96 & 0.33 & 0.73 & 0.53 & 0.50 \\
% & \small{S=1, C=2} & 0.86 & 0.47 & 0.34 & 0.95 & 0.98 & 0.16 & 0.67 & 0.49 & 0.59& 0.88 & 0.95 & 0.38 & 0.67& 0.49 & 0.64 \\
% % \midrule
% & \small{S=2, C=1} & 0.65 & 0.35 & 0.97 & 0.83 & 0.95 & 0.70 & 0.53 & 0.39 & 0.85 & 0.87 & 0.94 & 0.41 & 0.37 & 0.37 & 0.98 \\
% & \small{S=2, C=2} & 0.71 & 0.37 & 0.73 & 0.88& 0.96 & 0.41 & 0.65 & 0.58 & 0.89 & 0.90 & 0.94 & 0.22 & 0.45 & 0.34 & 0.97\\
% \bottomrule
% \end{tabular}
% }
% \label{tab:road_ood_exp}
% \end{table*}
% \begin{figure*}[t!]
% \begin{tikzpicture}
% \begin{groupplot}[
% group style={
% group size=3 by 2,
% horizontal sep=1.0cm,
% vertical sep=0.30cm,
% },
% width=6.0cm,
% height=2.5cm,
% grid=major,
% xlabel={\tiny{Number of samples/predictions}},
% xtick={1,5,10,15,20,25,30,35,40,45,50},
% tick label style={font=\tiny},
% label style={font=\tiny},
% y tick label style={
% /pgf/number format/fixed,
% /pgf/number format/precision=1,
% /pgf/number format/fixed zerofill=true,
% },
% tick label style={font=\fontsize{4}{4}\selectfont},
% ylabel style={yshift=-0.2cm}, % Shift y-label down
% xlabel style={yshift=0.1cm}, % Shift x-label down
% ]
% % First plot (Dataset A)
% \nextgroupplot[
% title={\tiny{\textbf{AIDE} \citep{yang2023aide}}},
% title style={yshift=-0.2cm},
% xlabel=\empty,
% ylabel={\tiny{PR-AUC}},
% ymin=0.15,
% ymax=0.7,
% ]
% % Plot data for BBB in Dataset A
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=orange}] coordinates {
% (5.0,0.4099293594955154) += (0,0.09673169437965605) -=(0,0.09673169437965605)
% (10.0,0.4708471672582329) += (0,0.10431635882073265) -=(0,0.10431635882073265)
% (15.0,0.46847685413525325) += (0,0.10727728380763607) -=(0,0.10727728380763607)
% (20.0,0.5961098446503498) += (0,0.09509619257437361) -=(0,0.09509619257437361)
% (25.0,0.5619578990892682) += (0,0.09690438717451529) -=(0,0.09690438717451529)
% (30.0,0.587875672510149) += (0,0.08628180881219635) -=(0,0.08628180881219635)
% (35.0,0.5542912596218069) += (0,0.09227921962393262) -=(0,0.09227921962393262)
% (40.0,0.5699888898962179) += (0,0.08978315525214099) -=(0,0.08978315525214099)
% (45.0,0.5558766945490639) += (0,0.0991136690446384) -=(0,0.0991136690446384)
% (50.0,0.5903266926606767) += (0,0.11860866662076826) -=(0,0.11860866662076826)
% };
% % Plot data for PE in Dataset B
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=cyan}] coordinates {
% (5,0.5162914097457046) += (0,0.002832952158627647) -=(0,0.002832952158627647)
% (10,0.5192954008000028) += (0,0.0042713665824021555) -=(0,0.0042713665824021555)
% (15,0.5199229997637944) += (0,0.0029151230951760423) -=(0,0.0029151230951760423)
% (20,0.5219530311768699) += (0,0.002723991671759387) -=(0,0.002723991671759387)
% (25,0.5201325378410252) += (0,0.0012726542640489756) -=(0,0.0012726542640489756)
% (30,0.5211818105841095) += (0,0.0018319391068414782) -=(0,0.0018319391068414782)
% (35,0.520793753455953) += (0,0.001275091449974835) -=(0,0.001275091449974835)
% (40,0.5208814466703101) += (0,0.0011747657560132582) -=(0,0.0011747657560132582)
% (45,0.5204366515532967) += (0,0.001401605623117085) -=(0,0.001401605623117085)
% (50,0.5206558062269206) += (0,0.0014470883988708474) -=(0,0.0014470883988708474)
% };
% % Plot data for SE in Dataset A
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=yellow}] coordinates {
% (5,0.20694511344929709) += (0,0.0021716182130065162) -=(0,0.0021716182130065162)
% (10,0.20611171048491678) += (0,0.0015349488300086113) -=(0,0.0015349488300086113)
% (15,0.20637550980293162) += (0,0.0010419906858312716) -=(0,0.0010419906858312716)
% (20,0.2063492428356545) += (0,0.0008025710435382048) -=(0,0.0008025710435382048)
% (25,0.2067879494966308) += (0,0.0009078925305815185) -=(0,0.0009078925305815185)
% (30,0.20682007591505136) += (0,0.0008528173919042681) -=(0,0.0008528173919042681)
% (35,0.2064812404309772) += (0,0.0011806073103746142) -=(0,0.0011806073103746142)
% (40,0.20629989745120106) += (0,0.0008501513619553977) -=(0,0.0008501513619553977)
% (45,0.20635113018699108) += (0,0.0006854879323059255) -=(0,0.0006854879323059255)
% (50,0.20655758244982697) += (0,0.000628357048185816) -=(0,0.000628357048185816)
% };
% % % Plot data for LL--HMC in Dataset A
% \addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=yellow}] coordinates {
% (5,0.38177465178644876) += (0,0.08465266682369238) -=(0,0.08465266682369238)
% (10,0.4214417583457424) += (0,0.029934762215761232) -=(0,0.029934762215761232)
% (15,0.4166176479774652) += (0,0.06457440366539512) -=(0,0.06457440366539512)
% (20,0.4302313222856375) += (0,0.036281591309485856) -=(0,0.036281591309485856)
% (25,0.4538529905132734) += (0,0.04689188747278871) -=(0,0.04689188747278871)
% (30,0.45759031304906833) += (0,0.04716463947606697) -=(0,0.04716463947606697)
% (35,0.46095961793450313) += (0,0.03727024012739902) -=(0,0.03727024012739902)
% (40,0.4617348365783642) += (0,0.06601967866262583) -=(0,0.06601967866262583)
% (45,0.46653975967457456) += (0,0.07638209100229867) -=(0,0.07638209100229867)
% (50,0.4834894076461963) += (0,0.02432877127789978) -=(0,0.02432877127789978)
% };
% % Single marker for DE at 5 samples
% \addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
% (5,82.24)
% };
% % Single marker for Regular Model at 1 sample
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
% (1,0.20)
% };
% % % Single marker for VBLL at 1 sample
% % \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt] coordinates {
% % (1,83.00)
% % };
% % Single marker for VBLL at 1 sample
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=green}] coordinates {
% % (1,94.93)
% (1,0.16784053115103267) += (0,0.01718804751072761) -=(0,0.01718804751072761)
% };
% % Second plot (Dataset B)
% \nextgroupplot[
% title={\tiny{\textbf{B4C} \citep{jain2015car}}},
% title style={yshift=-0.2cm},
% xlabel=\empty,
% ylabel={\tiny{PR-AUC}},
% ymax=0.7,
% ymin=0.3,
% ]
% % Plot data for BBB in Dataset B
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=orange}] coordinates {
% (5.0,0.4638115966944718) += (0,0.0990580906081241) -=(0,0.0990580906081241)
% (10.0,0.46837843860822764) += (0,0.062140427601138304) -=(0,0.062140427601138304)
% (15.0,0.5383142087951663) += (0,0.09442380008148808) -=(0,0.09442380008148808)
% (20.0,0.5015212997930913) += (0,0.044702622699907484) -=(0,0.044702622699907484)
% (25.0,0.5367705848030446) += (0,0.03556880375325622) -=(0,0.03556880375325622)
% (30.0,0.5518880202900334) += (0,0.053877117416156264) -=(0,0.053877117416156264)
% (35.0,0.5709447653908899) += (0,0.12171817113536487) -=(0,0.12171817113536487)
% (40.0,0.5831610515156422) += (0,0.08906224313434735) -=(0,0.08906224313434735)
% (45.0,0.5558371519949898) += (0,0.0882408087523543) -=(0,0.0882408087523543)
% (50.0,0.5695414652006476) += (0,0.07864870233861541) -=(0,0.07864870233861541)
% };
% % Plot data for PE in Dataset B
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=cyan}] coordinates {
% (5,0.5123220248795122) += (0,0.005314710766148739) -=(0,0.005314710766148739)
% (10,0.5175589443257259) += (0,0.003590659409170215) -=(0,0.003590659409170215)
% (15,0.5161847669999082) += (0,0.0032562365266935837) -=(0,0.0032562365266935837)
% (20,0.5138501479450839) += (0,0.003866628624764037) -=(0,0.003866628624764037)
% (25,0.5166221751254654) += (0,0.002678008933556911) -=(0,0.002678008933556911)
% (30,0.5174380161209144) += (0,0.002115990497502903) -=(0,0.002115990497502903)
% (35,0.5171830021699783) += (0,0.0010182632337509948) -=(0,0.0010182632337509948)
% (40,0.5157806828353164) += (0,0.002978738112387499) -=(0,0.002978738112387499)
% (45,0.5179971876282832) += (0,0.0008800929165591667) -=(0,0.0008800929165591667)
% (50,0.5175012699373359) += (0,0.001136316953539339) -=(0,0.001136316953539339)
% };
% % Plot data for SE in Dataset B
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=red}] coordinates {
% (5,0.4206446511658267) += (0,0.007286940312468549) -=(0,0.007286940312468549)
% (10,0.4194405869966336) += (0,0.005590200972095183) -=(0,0.005590200972095183)
% (15,0.4213187852732581) += (0,0.004704726247481645) -=(0,0.004704726247481645)
% (20,0.4199438772617269) += (0,0.0025566291148761463) -=(0,0.0025566291148761463)
% (25,0.4203765071938733) += (0,0.002935886438063627) -=(0,0.002935886438063627)
% (30,0.42074098327383525) += (0,0.0051108937289714345) -=(0,0.0051108937289714345)
% (35,0.4233719281478428) += (0,0.0016400674562432058) -=(0,0.0016400674562432058)
% (40,0.4190402100703186) += (0,0.004298020553824549) -=(0,0.004298020553824549)
% (45,0.4192048775709722) += (0,0.0028227431805547118) -=(0,0.0028227431805547118)
% (50,0.4206813168449681) += (0,0.003132567348319332) -=(0,0.003132567348319332)
% };
% % Plot data for LL--HMC in Dataset B
% \addplot[mark=diamond*, thin, line width=0.25pt,
% mark options={fill=yellow},
% mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=yellow}] coordinates {
% (5,0.49154039080155254) += (0,0.04640405081825597) -=(0,0.04640405081825597)
% (10,0.4786783124541153) += (0,0.07364788549041791) -=(0,0.07364788549041791)
% (15,0.4858752502869305) += (0,0.05754888299896623) -=(0,0.05754888299896623)
% (20,0.49316223276116683) += (0,0.08955126403104446) -=(0,0.08955126403104446)
% (25,0.5055797138457091) += (0,0.04991652539045278) -=(0,0.04991652539045278)
% (30,0.5000315859462228) += (0,0.05995163217047215) -=(0,0.05995163217047215)
% (35,0.5094513789672975) += (0,0.025934137324767178) -=(0,0.025934137324767178)
% (40,0.4878801810924019) += (0,0.060298775241277996) -=(0,0.060298775241277996)
% (45,0.487995741214408) += (0,0.062236536598725356) -=(0,0.062236536598725356)
% (50,0.507522857379817) += (0,0.03989078776064032) -=(0,0.03989078776064032)
% };
% % Single marker for DE at 5 samples
% \addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
% (5,0.39)
% };
% % Single marker for Regular Model at 1 sample
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
% (1,0.37)
% };
% % Single marker for VBLL at 1 sample
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=green}] coordinates {
% % (1,94.93)
% (1,0.46620109960672795) += (0,0.07557387243124529) -=(0,0.07557387243124529)
% };
% % Third plot (Dataset C)
% \nextgroupplot[
% title={\tiny{\textbf{ROAD} \citep{singh2022road}}},
% title style={yshift=-0.2cm},
% % xlabel=\empty,
% ylabel={\tiny{PR-AUC}},
% ymin=0.2,
% ymax=0.6
% ]
% % Plot data for BBB in Dataset C
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=orange}] coordinates {
% (5.0,0.38773302582836655) += (0,0.10954947422666532) -=(0,0.10954947422666532)
% (10.0,0.3846003238762753) += (0,0.10984772132420886) -=(0,0.10984772132420886)
% (15.0,0.32491859375872256) += (0,0.05690989276650245) -=(0,0.05690989276650245)
% (20.0,0.39729878889193204) += (0,0.08055317504079458) -=(0,0.08055317504079458)
% (25.0,0.37972716205254164) += (0,0.1050598624055497) -=(0,0.1050598624055497)
% (30.0,0.33951794870287716) += (0,0.062068491736802664) -=(0,0.062068491736802664)
% (35.0,0.38624654790624013) += (0,0.12299490766087355) -=(0,0.12299490766087355)
% (40.0,0.35987718338549146) += (0,0.06512216526196993) -=(0,0.06512216526196993)
% (45.0,0.3808373415939105) += (0,0.1269082296739477) -=(0,0.1269082296739477)
% (50.0,0.4123227895682818) += (0,0.10236737488512511) -=(0,0.10236737488512511)
% };
% % Plot data for PE in Dataset C
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=cyan}] coordinates {
% (5,0.5306308724747621) += (0,0.003453189399881718) -=(0,0.003453189399881718)
% (10,0.5370422497679835) += (0,0.003444850064075644) -=(0,0.003444850064075644)
% (15,0.5368442517650018) += (0,0.003161043147575295) -=(0,0.003161043147575295)
% (20,0.5359424976539415) += (0,0.0025307499631607605) -=(0,0.0025307499631607605)
% (25,0.5358339540491588) += (0,0.0018238762685481766) -=(0,0.0018238762685481766)
% (30,0.5363968252508218) += (0,0.0022390304486079255) -=(0,0.0022390304486079255)
% (35,0.536454165236027) += (0,0.002281885798973184) -=(0,0.002281885798973184)
% (40,0.5359497122163313) += (0,0.0016882335697935688) -=(0,0.0016882335697935688)
% (45,0.5371823826367331) += (0,0.0019585855466250717) -=(0,0.0019585855466250717)
% (50,0.53686316284095) += (0,0.0014682087734926976) -=(0,0.0014682087734926976)
% };
% % Plot data for SE in Dataset C
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=red}] coordinates {
% (5,0.3133075629044341) += (0,0.019281852806744645) -=(0,0.019281852806744645)
% (10,0.3104261371113608) += (0,0.008651232302635925) -=(0,0.008651232302635925)
% (15,0.3138265304819588) += (0,0.009268747424258171) -=(0,0.009268747424258171)
% (20,0.3196035307762033) += (0,0.012514151437029873) -=(0,0.012514151437029873)
% (25,0.32096080747240985) += (0,0.013174017366005408) -=(0,0.013174017366005408)
% (30,0.31792014751682107) += (0,0.007018807936854558) -=(0,0.007018807936854558)
% (35,0.3177472908417346) += (0,0.005365810548448727) -=(0,0.005365810548448727)
% (40,0.3178155109752583) += (0,0.00629101895609037) -=(0,0.00629101895609037)
% (45,0.32215951534707027) += (0,0.009521724735735884) -=(0,0.009521724735735884)
% (50,0.3215322735884013) += (0,0.0051680001245563625) -=(0,0.0051680001245563625)
% };
% % Plot data for LL--HMC in Dataset C
% \addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt, error bars/.cd,
% y dir=both, y explicit,
% error bar style={line width=0.1pt, color=yellow}] coordinates {
% (5,0.30259105891012805) += (0,0.04111072247208028) -=(0,0.04111072247208028)
% (10,0.2913667249900822) += (0,0.025620912354253227) -=(0,0.025620912354253227)
% (15,0.31814389406627586) += (0,0.06281680100719542) -=(0,0.06281680100719542)
% (20,0.3287216845015036) += (0,0.07961653351602443) -=(0,0.07961653351602443)
% (25,0.31062515571996696) += (0,0.0707956649532733) -=(0,0.0707956649532733)
% (30,0.28833208941681676) += (0,0.03492510051471629) -=(0,0.03492510051471629)
% (35,0.3302366083064697) += (0,0.08704015505502467) -=(0,0.08704015505502467)
% (40,0.3045102230648385) += (0,0.032821559937954506) -=(0,0.032821559937954506)
% (45,0.3042644592758631) += (0,0.035044321253610115) -=(0,0.035044321253610115)
% (50,0.30972900759294825) += (0,0.04386286535644569) -=(0,0.04386286535644569)
% };
% % Single marker for DE at 5 samples
% \addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
% (5,0.41)
% };
% % Single marker for Regular Model at 1 sample
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
% (1,0.24955)
% };
% % Single marker for VBLL at 1 sample
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=green}] coordinates {
% % (1,49.34)
% (1,0.24897345524411732) += (0,0.00975925949403175) -=(0,0.00975925949403175)
% };
% \nextgroupplot[
% title=\empty,
% % title style={yshift=-0.2cm},
% ylabel={\tiny{PR-AUC}},
% ymin=0.5,
% ymax=1
% ]
% % Plot data for BBB in Dataset A
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=orange}] coordinates {
% (5.0,0.8890952355627204) += (0,0.03237682815584933) -=(0,0.03237682815584933)
% (10.0,0.8953625367929087) += (0,0.019708109385618764) -=(0,0.019708109385618764)
% (15.0,0.9067725793580275) += (0,0.021379167110459934) -=(0,0.021379167110459934)
% (20.0,0.8969385261953808) += (0,0.01638647002781579) -=(0,0.01638647002781579)
% (25.0,0.9130454847262927) += (0,0.02363815682341601) -=(0,0.02363815682341601)
% (30.0,0.8938326988036012) += (0,0.019009529519439787) -=(0,0.019009529519439787)
% (35.0,0.8970527741214148) += (0,0.024139127423905466) -=(0,0.024139127423905466)
% (40.0,0.9083495177397062) += (0,0.0226514350727818) -=(0,0.0226514350727818)
% (45.0,0.9127346098767953) += (0,0.011051998302365878) -=(0,0.011051998302365878)
% (50.0,0.9005882102907485) += (0,0.02319698451859789) -=(0,0.02319698451859789)
% };
% % Plot data for PE in Dataset B
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=cyan}] coordinates {
% (5,0.5244477566372742) += (0,0.008363668125944761) -=(0,0.008363668125944761)
% (10,0.5286537724419726) += (0,0.0028127946792281125) -=(0,0.0028127946792281125)
% (15,0.5300709592019601) += (0,0.003948589731375047) -=(0,0.003948589731375047)
% (20,0.5304511474279104) += (0,0.0040355665706302045) -=(0,0.0040355665706302045)
% (25,0.5306099762045432) += (0,0.0013285374644871459) -=(0,0.0013285374644871459)
% (30,0.5309125311021029) += (0,0.0034758903888831186) -=(0,0.0034758903888831186)
% (35,0.5307407058391697) += (0,0.002990517851100531) -=(0,0.002990517851100531)
% (40,0.5316459892920249) += (0,0.002257940069911566) -=(0,0.002257940069911566)
% (45,0.5315396864551322) += (0,0.002236488291376101) -=(0,0.002236488291376101)
% (50,0.5316868323701924) += (0,0.0018976820095359056) -=(0,0.0018976820095359056)
% };
% % Plot data for SE in Dataset A
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=yellow}] coordinates {
% (5,0.9555835895870072) += (0,0.004476051983705348) -=(0,0.004476051983705348)
% (10,0.9587466540049675) += (0,0.0030128507786380816) -=(0,0.0030128507786380816)
% (15,0.9597025633785752) += (0,0.0022045398572342603) -=(0,0.0022045398572342603)
% (20,0.9602738656910887) += (0,0.0014481400466224005) -=(0,0.0014481400466224005)
% (25,0.960161140329007) += (0,0.001623262308215483) -=(0,0.001623262308215483)
% (30,0.9601480522823737) += (0,0.001454570819586515) -=(0,0.001454570819586515)
% (35,0.9608012772991531) += (0,0.0015557016043011326) -=(0,0.0015557016043011326)
% (40,0.9600423600296191) += (0,0.001504799851939997) -=(0,0.001504799851939997)
% (45,0.9602279128121924) += (0,0.0011941244966481928) -=(0,0.0011941244966481928)
% (50,0.9604300014730894) += (0,0.0016097066629510587) -=(0,0.0016097066629510587)
% };
% % % Plot data for LL--HMC in Dataset A
% \addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=yellow}] coordinates {
% (5,0.9617172352105422) += (0,0.00547949550098082) -=(0,0.00547949550098082)
% (10,0.9646612480139651) += (0,0.0037800920063494913) -=(0,0.0037800920063494913)
% (15,0.9691497394501779) += (0,0.0027078851176429502) -=(0,0.0027078851176429502)
% (20,0.971158174692748) += (0,0.002175735918234132) -=(0,0.002175735918234132)
% (25,0.9706999801065308) += (0,0.001914172929528126) -=(0,0.001914172929528126)
% (30,0.9720006924024169) += (0,0.0018975408592584475) -=(0,0.0018975408592584475)
% (35,0.9714917236460485) += (0,0.002349280230643308) -=(0,0.002349280230643308)
% (40,0.9711898294705111) += (0,0.003607969589119205) -=(0,0.003607969589119205)
% (45,0.9719698549924789) += (0,0.0015381376266515926) -=(0,0.0015381376266515926)
% (50,0.9727679502293537) += (0,0.0012616212687732845) -=(0,0.0012616212687732845)
% };
% % Single marker for DE at 5 samples
% \addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
% (5,0.94)
% };
% % Single marker for Regular Model at 1 sample
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
% (1,0.94)
% };
% % % Single marker for VBLL at 1 sample
% % \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt] coordinates {
% % (1,83.00)
% % };
% % Single marker for VBLL at 1 sample
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=green}] coordinates {
% (1,0.9419845204867905) += (0,0.019431116584284674) -=(0,0.019431116584284674)
% };
% % Second plot (Dataset B)
% \nextgroupplot[
% title= \empty,
% title style={yshift=-0.2cm},
% ylabel={\tiny{PR-AUC}},
% ymax=1,
% ymin=0.5,
% ]
% % Plot data for BBB in Dataset B
% \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=orange}] coordinates {
% (5.0,0.8890952355627204) += (0,0.03237682815584933) -=(0,0.03237682815584933)
% (10.0,0.8953625367929087) += (0,0.019708109385618764) -=(0,0.019708109385618764)
% (15.0,0.9067725793580275) += (0,0.021379167110459934) -=(0,0.021379167110459934)
% (20.0,0.8969385261953808) += (0,0.01638647002781579) -=(0,0.01638647002781579)
% (25.0,0.9130454847262927) += (0,0.02363815682341601) -=(0,0.02363815682341601)
% (30.0,0.8938326988036012) += (0,0.019009529519439787) -=(0,0.019009529519439787)
% (35.0,0.8970527741214148) += (0,0.024139127423905466) -=(0,0.024139127423905466)
% (40.0,0.9083495177397062) += (0,0.0226514350727818) -=(0,0.0226514350727818)
% (45.0,0.9127346098767953) += (0,0.011051998302365878) -=(0,0.011051998302365878)
% (50.0,0.9005882102907485) += (0,0.02319698451859789) -=(0,0.02319698451859789)
% };
% % Plot data for PE in Dataset B
% \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=cyan}] coordinates {
% (5,0.5173564453152024) += (0,0.0053989208473570124) -=(0,0.0053989208473570124)
% (10,0.5200962981234224) += (0,0.00563841528006263) -=(0,0.00563841528006263)
% (15,0.5227057339988891) += (0,0.003977407755074143) -=(0,0.003977407755074143)
% (20,0.5219544147257136) += (0,0.005409716726026976) -=(0,0.005409716726026976)
% (25,0.5211473657965758) += (0,0.005538075709958593) -=(0,0.005538075709958593)
% (30,0.525901883273335) += (0,0.006086001337114488) -=(0,0.006086001337114488)
% (35,0.5207482599261907) += (0,0.004488729530551176) -=(0,0.004488729530551176)
% (40,0.5214644177851622) += (0,0.006816659070382818) -=(0,0.006816659070382818)
% (45,0.524882574287336) += (0,0.00465381725159028) -=(0,0.00465381725159028)
% (50,0.5228059285888904) += (0,0.006237186748222989) -=(0,0.006237186748222989)
% };
% % Plot data for SE in Dataset B
% \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=red}] coordinates {
% (5,0.8720259503349436) += (0,0.0030070683128284067) -=(0,0.0030070683128284067)
% (10,0.8757909638851944) += (0,0.0023951604658430615) -=(0,0.0023951604658430615)
% (15,0.8757736588520493) += (0,0.002165155167885959) -=(0,0.002165155167885959)
% (20,0.8754886673420307) += (0,0.0039455682991742964) -=(0,0.0039455682991742964)
% (25,0.8748057411117042) += (0,0.0022534408057138956) -=(0,0.0022534408057138956)
% (30,0.8758860975688926) += (0,0.0016902895606028705) -=(0,0.0016902895606028705)
% (35,0.8745137286778558) += (0,0.0014722667916795417) -=(0,0.0014722667916795417)
% (40,0.8742627982872954) += (0,0.0013590415326470153) -=(0,0.0013590415326470153)
% (45,0.8743424201502226) += (0,0.0008400951530645621) -=(0,0.0008400951530645621)
% (50,0.874138561338419) += (0,0.00248896341861119) -=(0,0.00248896341861119)
% };
% % Plot data for LL--HMC in Dataset B
% \addplot[mark=diamond*, thin, line width=0.25pt,
% mark options={fill=yellow},
% mark size=1.0pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=yellow}] coordinates {
% (5,0.934234387079315) += (0,0.011847309971530726) -=(0,0.011847309971530726)
% (10,0.9463293321330107) += (0,0.012391886422842521) -=(0,0.012391886422842521)
% (15,0.9480493321202473) += (0,0.010910019051946324) -=(0,0.010910019051946324)
% (20,0.9510705083455543) += (0,0.006960193274851655) -=(0,0.006960193274851655)
% (25,0.9561847616749584) += (0,0.007791189218021244) -=(0,0.007791189218021244)
% (30,0.9603580205893574) += (0,0.00439817404556025) -=(0,0.00439817404556025)
% (35,0.9578881533621375) += (0,0.0036049264914185418) -=(0,0.0036049264914185418)
% (40,0.9560222759258465) += (0,0.004877068627372283) -=(0,0.004877068627372283)
% (45,0.9601004612890331) += (0,0.0028671913098146295) -=(0,0.0028671913098146295)
% (50,0.9595021218847638) += (0,0.003178914292906151) -=(0,0.003178914292906151)
% };
% % Single marker for DE at 5 samples
% \addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
% (5,0.90)
% };
% % Single marker for Regular Model at 1 sample
% \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
% (1,0.82)
% };
% % Single marker for VBLL at 1 sample
% \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
% error bars/.cd,
% y dir=both,
% y explicit,
% error bar style={line width=0.1pt, color=green}] coordinates {
% % (1,94.93)
% (1,0.7971741487324044) += (0,0.02130437763910965) -=(0,0.02130437763910965)
% };
% % Third plot (Dataset C)
% % \nextgroupplot[
% % title= \empty,
% % ylabel={\tiny{F1 (\%)}},
% % ymin=30,
% % ymax=57
% % ]
% % % Plot data for BBB in Dataset C
% % \addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.0pt, error bars/.cd,
% % y dir=both,
% % y explicit,
% % error bar style={line width=0.1pt, color=orange}] coordinates {
% % (5,40.03344690170473) += (0,5.056929754523553) -=(0,5.056929754523553)
% % (10,39.497117850555775) += (0,7.324741430969779) -=(0,7.324741430969779)
% % (15,41.70045043815289) += (0,4.503083114618855) -=(0,4.503083114618855)
% % (20,43.95679573963744) += (0,5.724476140854986) -=(0,5.724476140854986)
% % (25,43.91655890502581) += (0,7.397736724682226) -=(0,7.397736724682226)
% % (30,40.84849670921552) += (0,6.787114836757397) -=(0,6.787114836757397)
% % (35,43.46600586280709) += (0,4.918098156383699) -=(0,4.918098156383699)
% % (40,39.33604574991489) += (0,5.587082268904128) -=(0,5.587082268904128)
% % (45,41.67154706867009) += (0,4.853981201777168) -=(0,4.853981201777168)
% % (50,41.170672005887695) += (0,5.497095966312938) -=(0,5.497095966312938)
% % };
% % % Plot data for PE in Dataset C
% % \addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1pt,error bars/.cd,
% % y dir=both,
% % y explicit,
% % error bar style={line width=0.1pt, color=cyan}] coordinates {
% % (5,46.95914301867057) += (0,6.05243372072422) -=(0,6.05243372072422)
% % (10,44.35103292152759) += (0,2.9966140290527696) -=(0,2.9966140290527696)
% % (15,44.21816213450458) += (0,6.367049452768733) -=(0,6.367049452768733)
% % (20,46.27727705113644) += (0,8.537225520140703) -=(0,8.537225520140703)
% % (25,46.26570675100138) += (0,6.5510505190307144) -=(0,6.5510505190307144)
% % (30,44.06082062870729) += (0,5.206354977019308) -=(0,5.206354977019308)
% % (35,43.00217105321485) += (0,4.690638124617283) -=(0,4.690638124617283)
% % (40,44.123030934571204) += (0,2.7004686917469254) -=(0,2.7004686917469254)
% % (45,46.803628670707965) += (0,3.749361741586067) -=(0,3.749361741586067)
% % (50,43.03308469196917) += (0,4.282980414697678) -=(0,4.282980414697678)
% % };
% % % Plot data for SE in Dataset C
% % \addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.0pt,
% % error bars/.cd,
% % y dir=both,
% % y explicit,
% % error bar style={line width=0.1pt, color=red}] coordinates {
% % (5,40.73743947791) += (0,4.009901172910407) -=(0,4.009901172910407)
% % (10,40.056428478757425) += (0,4.1730650869813655) -=(0,4.1730650869813655)
% % (15,39.41337466881583) += (0,3.053884389004715) -=(0,3.053884389004715)
% % (20,37.86591958163472) += (0,0.18746914540579074) -=(0,0.18746914540579074)
% % (25,38.723007197626444) += (0,2.3043486029802986) -=(0,2.3043486029802986)
% % (30,39.378180178826824) += (0,2.798849989574935) -=(0,2.798849989574935)
% % (35,38.776521032744974) += (0,2.282984016798466) -=(0,2.282984016798466)
% % (40,38.91944120501935) += (0,3.022983449344767) -=(0,3.022983449344767)
% % (45,38.03843460972704) += (0,0.11435775350534193) -=(0,0.11435775350534193)
% % (50,38.02956311592404) += (0,0.12290633139707) -=(0,0.12290633139707)
% % };
% % % Plot data for LL--HMC in Dataset C
% % \addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.0pt, error bars/.cd,
% % y dir=both, y explicit,
% % error bar style={line width=0.1pt, color=yellow}] coordinates {
% % (5,33.887826567916576) += (0,6.14351969721661) -=(0,6.14351969721661)
% % (10,34.574903713756974) += (0,8.73202224498559) -=(0,8.73202224498559)
% % (15,39.24376817031906) += (0,4.248698005147985) -=(0,4.248698005147985)
% % (20,39.30141891898767) += (0,6.686024326991105) -=(0,6.686024326991105)
% % (25,35.09617883452023) += (0,6.55255084074511) -=(0,6.55255084074511)
% % (30,37.58546499845445) += (0,7.258541972058792) -=(0,7.258541972058792)
% % (35,35.363634910564876) += (0,6.8963686503265595) -=(0,6.8963686503265595)
% % (40,42.42697302678621) += (0,8.566257218440315) -=(0,8.566257218440315)
% % (45,32.4746292995812) += (0,5.484626427348374) -=(0,5.484626427348374)
% % (50,36.93272588550002) += (0,7.148580495119111) -=(0,7.148580495119111)
% % };
% % % Single marker for DE at 5 samples
% % \addplot[mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=1.5pt] coordinates {
% % (5,44.57)
% % };
% % % Single marker for Regular Model at 1 sample
% % \addplot[mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=1.5pt] coordinates {
% % (1,45.36)
% % };
% % % Single marker for VBLL at 1 sample
% % \addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt,
% % error bars/.cd,
% % y dir=both,
% % y explicit,
% % error bar style={line width=0.1pt, color=green}] coordinates {
% % % (1,49.34)
% % (1,30.63016) += (0,2.317158) -=(0,2.317158)
% % };
% \end{groupplot}
% % \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-0.9cm,0.0)$) {OOD min};
% % \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-0.9cm,-1.5)$) {OOD max};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-0.80cm,0)$) {min};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.00cm,0)$) {OOD};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-0.80cm,-1.2)$) {max};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.00cm,-1.2)$) {OOD};
% \end{tikzpicture}
% % Legend
% \begin{tikzpicture}
% \hspace{3.5cm}
% \begin{axis}[
% hide axis,
% xmin=0, xmax=1, ymin=0, ymax=1,
% legend style={at={(0.5,0.5)}, anchor=center, legend columns=7},
% legend entries={\tiny{BBB--LL},\tiny{PE--LL},\tiny{SE},\tiny{VBLL}, \tiny{LL--HMC},\tiny{DE},\tiny{Regular},}
% ]
% \addlegendimage{mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=2.0pt}
% \addlegendimage{mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2.0pt}
% \addlegendimage{mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=2pt}
% \addlegendimage{mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=2.0pt}
% \addlegendimage{mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=2.0pt}
% \addlegendimage{mark=halfcircle*, thin, line width=0.25pt, mark options={fill=magenta}, mark size=2.0pt}
% \addlegendimage{mark=oplus*, thin, line width=0.25pt, mark options={fill=pink}, mark size=2.0pt}
% \end{axis}
% \end{tikzpicture}
% \vspace{-0.2cm}
% \caption{Influence of the number of samples or predictions for the top-performing Markers without visible whiskers indicate small standard deviation over the different runs.} %hyperparameter configurations of the LL--PDL methods for the OOD detection performance. The regular model (single prediction), DE (five predictions), and VBLL (single prediction) are only included once in each figure.
% \label{fig:ood_comparison}
% \end{figure*}
% \definecolor{darkgray176}{RGB}{176,176,176}
% \definecolor{steelblue31119180}{RGB}{31,119,180}
% \begin{figure}[t!]
% \centering
% \begin{tikzpicture}
% \begin{groupplot}[
% group style={
% group size=3 by 2, % Three figures in one row
% horizontal sep=1.2cm,
% vertical sep=0.3cm,
% },
% width=6.0cm,
% height=2.5cm,
% ybar,
% enlargelimits=0.15,
% ylabel={\tiny {PR-AUC}},
% symbolic x coords={BBB--LL, SE, PE--LL, VBLL, LL--HMC},
% xtick=data,
% ymin=0, ymax=1,
% ytick distance=0.2,
% grid=major,
% ymajorgrids=true,
% tick label style={font=\tiny},
% ylabel style={yshift=-0.2cm}, % Shift y-label down
% xlabel style={yshift=0.1cm}, % Shift x-label down
% every node near coord/.append style={
% yshift=3pt,
% font=\footnotesize,
% label style={font=\tiny},
% y tick label style={
% /pgf/number format/fixed,
% /pgf/number format/precision=1,
% /pgf/number format/fixed zerofill=true,
% },
% tick label style={font=\fontsize{4}{4}\selectfont},
% },
% ]
% % First subplot
% \nextgroupplot[title={\tiny \textbf{AIDE} \citep{yang2023aide}},
% title style={yshift=-0.2cm}, xlabel=\empty, xticklabels=\empty,
% ymin=0, ymax=1,]
% \addplot[
% fill=steelblue31119180!60,
% draw=steelblue31119180,
% error bars/.cd,
% y dir=both,
% y explicit,
% ] coordinates {
% (SE, 0.18634) +-(0.005755,0.005755)
% (PE--LL, 0.518888) +-(0.00524,0.00524)
% (BBB--LL, 0.429683) +-(0.120844, 0.120844)
% (VBLL, 0.169609) +-(0.025146 ,0.025146)
% (LL--HMC, 0.327656 ) +-(0.118668,0.118668)
% };
% % Second subplot
% \nextgroupplot[title={\tiny \textbf{B4C} \citep{jain2015car}},
% title style={yshift=-0.2cm},
% xlabel=\empty, xticklabels=\empty, ymin=0, ymax=1,]
% \addplot[
% fill=steelblue31119180!60,
% draw=steelblue31119180,
% error bars/.cd,
% y dir=both,
% y explicit,
% ] coordinates {
% (BBB--LL, 0.458215) +-(0.069531,0.069531)
% (SE, 0.517133) +-(0.024599,0.024599)
% (PE--LL, 0.516577) +-(0.002815,0.002815)
% (VBLL, 0.446607) +-(0.065035,0.065035)
% (LL--HMC,0.495702) +-(0.06761,0.067611)
% };
% \nextgroupplot[title={\tiny \textbf{ROAD} \citep{singh2022road}},
% title style={yshift=-0.2cm}, ymin=0, ymax=1,]
% \addplot[
% fill=steelblue31119180!60,
% draw=steelblue31119180,
% error bars/.cd,
% y dir=both,
% y explicit,
% ] coordinates {
% (BBB--LL, 0.33296) +-(0.044948,0.044948)
% (SE, 0.259985) +-(0.001277,0.001277)
% (PE--LL, 0.52891) +-( 0.003139,0.003139)
% (VBLL, 0.25524) +-( 0.028792,0.028792)
% (LL--HMC, 0.555987) +-(0.116223,0.116223)
% };
% % First subplot
% \nextgroupplot[title=\empty,
% title style={yshift=-0.2cm},
% ymin=0, ymax=1,]
% \addplot[
% fill=steelblue31119180!60,
% draw=steelblue31119180,
% error bars/.cd,
% y dir=both,
% y explicit,
% ] coordinates {
% (SE, 0.955661) +-(0.005331,0.005331)
% (PE--LL, 0.529377) +-(0.00287, 0.00287)
% (BBB--LL, 0.97792) +-(0.008233,0.008233)
% (VBLL, 0.941438) +-(0.017702, 0.017702)
% (LL--HMC, 0.961973) +-(0.014586,0.014586)
% };
% % Second subplot
% \nextgroupplot[title=\empty,
% title style={yshift=-0.2cm},
% ymin=0, ymax=1,]
% \addplot[
% fill=steelblue31119180!60,
% draw=steelblue31119180,
% error bars/.cd,
% y dir=both,
% y explicit,
% ] coordinates {
% (BBB--LL, 0.864666) +-(0.022141,0.022141)
% (SE,0.869604) +-(0.00668,0.00668)
% (PE--LL, 0.519402) +-(0.001778,0.001778)
% (VBLL, 0.783961) +-(0.030726,0.030726)
% (LL--HMC, 0.950973) +-(0.02155,0.02155)
% };
% \end{groupplot}
% % \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-0.9cm,0.0)$) {OOD min};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-0.80cm,0)$) {min};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.00cm,0)$) {OOD};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-0.80cm,-1.2)$) {max};
% \node[font=\bfseries\tiny, rotate=90] at ($(group c1r1.west)+(-1.00cm,-1.2)$) {OOD};
% \end{tikzpicture}
% \caption{Average uncertainty-based OOD detection performance across the included LL--PDL methods for the top 10 grid search hyperparameter configurations across 100 random seeds.}
% \label{fig:placeholder_groupplot}
% \end{figure}
% \subsection{Uncertainty-based out-of-distribution detection performance}
% \paragraph{Grid search results.} Tables \ref{tab:road_ood_exp} presents the top uncertainty-based OOD detection results for the grid search. For each dataset there is always at least one PDL methods exceeds the regular softmax baseline performance. For nearly all scenarios, LL--HMC with a single starting position performs best for the uncertainty-based OOD detection. However, the best performing LL--HMC configurations with two starting positions (S=2) are less consistent. For the B4C OOD min scenario LL--HMC with two starting positions and two chains resulted in the best PR-AUC, while the FPR95 performance is very low. Similarly, LL--HMC (S=2) produced the worst OOD detection results for the ROAD OOD min scenario. This highlights the sensitivity of the latent representations on which LL--HMC relies. The other PDL methods also exhibit comparable, and at times superior, performance. For example, BBB--LL produces comparable or joint-best results for both AIDE scenarios. Similarly, PE--LL and SE match and outperform the LL--HMC performance for the B4C OOD min and ROAD OOD min scenarios. %This underscores that there is not one single best method among the evaluated PDL approaches.
% \paragraph{\textbf{Influence of additional predictions or samples.}} Figure \ref{fig:ood_comparison} visualizes the influence of additional samples or predictions for the top-performing grid search hyperparameter configurations for 10 random seeds for the different numbers of samples or predictions. The BBB--LL results for the OOD min scenarios for the AIDE and B4C datasets show a slight improvement with additional samples, but this trend is not visible for the other scenarios or for the ROAD dataset. Opposed to the grid-search results, LL--HMC for 10 different random seeds for each number of samples does not yield the highest performance on average for all scenarios. Similar to the in-distribution results, there is no single LL--PDL method consistently performs significantly better for all scenarios.
% %For the AIDE and B4C OOD-min scenarios, the BBB--LL indicate
% \paragraph{\textbf{Top-10 configurations across random seeds. }}Similar to the in-distribution classification performance, Figure \ref{fig:ood_comparison} illustrates the average PR-AUC values for 100 random seeds for the ten best-performing grid search hyperparameter configurations. The average PE and SE OOD detection performance is stable across the scenarios and the performance is similar to Table \ref{tab:road_ood_exp}. For BBB--LL and LL--HMC, the average performance across multiple seeds and configurations varies more and is occasionally lower. In general, evaluation of the top-performing hyperparameters for multiple random seeds also indicates that there is not a single LL--PDL method consistently outperforming.
% % \begin{tcolorbox}[
% % colback=white, % Background color of the box
% % colframe=black, % Border color of the box
% % arc=4pt, % Rounded corners
% % boxrule=1pt, % Border width
% % left=6pt, % Left padding
% % right=6pt, % Right padding
% % top=2pt, % Top padding
% % bottom=2pt, % Bottom padding
% % width=0.98\linewidth % Width of the box
% % ]
% % \textit{While the top-performing LL--HMC frequently demonstrates best uncertainty-based OOD detection performance, BBB--LL, PE--LL, and SE match and occasionally yield top OOD detection performance. The average OOD detection performance over multiple random seeds does not yield a single best approach across the OOD detection scenarios. }
% % \end{tcolorbox}
% \section{Discussion and Conclusion}
% We evaluated the use of Hamiltonian Monte Carlo (HMC) sampling as a last layer probabilistic deep learning approach for video-based driver action and intention recognition. Applying HMC to sample the parameters of the last layer reduces the required computational cost compared to sampling the entire network. While last layer HMC (LL--HMC) benefits from the Markov-Chain Monte Carlo sampling ability to explore complex, potentially multi-modal posterior distributions for the last layer's parameters, its reliance on fixed pre-trained representations comes at the cost of not capturing the uncertainty inherent in the feature extraction layers of a deep neural network (DNN). Furthermore, as illustrated by the toy example and the video-based driver action and intention recognition experiments, using additional samples saturates and does not improve the LL--HMC uncertainty estimations or performance. Compared to the softmax-based regular baseline, the grid search results showed that there is a last layer probabilistic deep learning (PDL) approach that performs better. To analyze the influence of additional samples or predictions, we evaluated the top-performing grid search hyperparameter configuration across multiple random seeds for each LL--PDL method for each dataset. Additional samples or predictions did not consistently improve the average in-distribution and out-of-distribution (OOD) performance. Moreover, when evaluating the 10 best grid search hyperparameter configurations, we observe no single best LL--PDL method across the datasets. For LL--HMC, we observe that suboptimal hyperparameter configuration can result in a significant performance drop, and being sensitive to the random initialization of the last layer parameters.
% %Another limitation for all included last layer approaches is the reliance on a pre-trained model to produce the latent representations.
% %While we explored using intermediate latent representations to further reduce computational demands to produce predictions, we observed that this did not improve or match the in-distribution performance.
% None of the last layer probabilistic deep learning (PDL) methods were able to consistently identify all out-of-distribution (OOD) instances when the most or least prevailing driving maneuvers were removed from the datasets. This brings forward broader questions to consider when designing safety-critical systems, such as an advanced driver assistance or autonomous driving system. For example, if a PDL method predicts two different classes with similar low uncertainty and similar high confidence, we would pick the class with the highest assigned probability. However, driving maneuvers can overlap, for example, a \textit{left turn} and a \textit{u-turn}. In such a scenario, it could be interesting to see whether predicting sets \citep{hullermeier2022quantification,wang2024credal} of potential maneuvers improves the performance. From a broader perspective, if we cannot guarantee that all OOD scenarios are identified by PDL methods, we should also consider the potential outcome of the recommendations given the current driving scene. By leveraging a more decision-theoretic approach that incorporates generic task knowledge (e.g., penalizing intended maneuvers that potentially harm other road users), we might be able to produce better and safer recommendations \citep{cobb2018loss}.
% \section{Regression Toy-example} \label{app:reg}
% Compared to the softmax-based baseline model, Last layer HMC (LL--HMC) consistently improved the performance on our real-world resource-constrained safety-critical tasks (video-based driver action and intention recognition). However, LL--HMC is subject to more hyperparameters than the other probabilistic deep learning (PDL) approaches, and a suboptimal configuration settings can result in a significant performance drops. Another clear limitation for all included last layer approaches is the reliance on pre-trained model to produce the latent representations. Using additional starting positions based on multiple models producing the latent representations did not consistently improve the performance.
% Compared to other last layer probabilistic deep learning methods (Bayesian-by-backprop last layer, sub-ensembles, packed-ensemble last layer, Variational Bayes Last Layer, and Deep Deterministic Uncertainty), LL--HMC consistently ranked among the top-performing methods. Reliance on intermediate latent representations to reduce the compute even further did not improve or match the in-distribution for the PDL methods. The uncertainty-based OOD experiments showed that, while LL--HMC frequently performed best, there was no universal best PDL approach. This suggests it might be interesting to explore beyond a one-size-fits-all approaches for uncertainty-based OOD detection.
% To evaluate the uncertainty estimations for a regression toy example, we use the implementation of \cite{lippe2022uva} to generate noisy observations from a sinusoidal function. The underlying architecture is an MLP with five hidden layers and 50 hidden units for all methods. Figure \ref{fig:toy_reg} shows the uncertainty estimation results for the toy regression example. When performing HMC on the entire DNN, we observe higher uncertainty in regions without observations. For both the LL--HMC and the deep ensemble, the uncertainty between the two observation areas remains relatively stable, further highlighting the advantage of full-network HMC in capturing the uncertainty.
% \begin{figure}[h]
% \centering
% % First row
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/hmc_5.pdf} % Add your image here
% \captionsetup{font=tiny}
% \caption{HMC NUTS (N=5)}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/hmc_50.pdf} % Add your image here
% \captionsetup{font=tiny}
% \caption{HMC NUTS (N=50)}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/hmc_100.pdf} % Add
% \captionsetup{font=tiny}
% \caption{HMC NUTS (N=100)}
% \end{subfigure}
% % \vskip\baselineskip % Adds vertical space between rows
% % Second row
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ll_hmc_5.pdf} % Add
% \captionsetup{font=tiny}
% \caption{LL--HMC NUTS (N=5)}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ll_hmc_50.pdf} %
% \captionsetup{font=tiny}
% \caption{LL--HMC NUTS (N=50)}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ll_hmc_100.pdf} %
% \captionsetup{font=tiny}
% \caption{LL--HMC NUTS (N=100)}
% \end{subfigure}
% % \vskip\baselineskip
% % Third row
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ensemble_5.pdf} %
% \captionsetup{font=tiny}
% \caption{DE (N=5)}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ensemble_50.pdf} %
% \captionsetup{font=tiny}
% \caption{DE (N=50)}
% \end{subfigure}
% \hfill
% \begin{subfigure}[b]{0.25\linewidth}
% \centering
% \includegraphics[width=\linewidth, trim=2.85cm 1.2cm 1.5cm 1.4cm, clip]{toy_reg/ensemble_100.pdf} %
% \captionsetup{font=tiny}
% \caption{DE (N=100)}
% \end{subfigure}
% \caption{Toy Regression Example results. First row consists of full HMC with 5, 50, and 100 samples, the second row shows the LL--HMC results, and the third row shows the results for a deep ensemble. The y-axis is the target variable, the x-axis the observations. The gray line is the original function, the orange dots represent the noisy observations, the blue line the mean prediction, and the light blue area the uncertainty estimation.}
% \label{fig:toy_reg}
% \end{figure}
% \clearpage
% \section{Hyperparameters tuning} \label{app:hyper}
% \subsection{Sub-ensembles}
% Figures \ref{fig:se_per}, \ref{fig:se_oodmax}, and \ref{fig:se_oodmin} show the performance of the Sub-ensemble each with a different number of estimators (5, 10, 15, 20, 25, 30, 35, 40, 45, 50) for different learning rates ($1e^{-2}$, $1e^{-3}$, and $1e^{-4}$) trained for different numbers of epochs (5, 10, 15, 20) with different batch sizes (16, 32, 64). In general, we do not observe that increasing the number of estimators improves the performance or the calibration for the three datasets. A larger learning rate does have a positive effect on the performance metrics.
% \begin{figure}[h]
% \centering
% \includegraphics[height=16cm,keepaspectratio]{hyper/avg_performance_sub_ensemble.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the Sub-ensemble performance results for different hyperparameter settings for the regular DIR and DAR scenarios.}
% % \vspace{-0.5cm}
% \label{fig:se_per}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_max_sub_ensemble.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the Sub-ensemble performance results for different hyperparameter settings for the OOD max scenario.}
% % \vspace{-0.5cm}
% \label{fig:se_oodmax}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_min_sub_ensemble.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the Sub-ensemble performance results for different hyperparameter settings for the OOD min scenario.}
% % \vspace{-0.5cm}
% \label{fig:se_oodmin}
% \end{figure}
% \clearpage
% \subsection{Packed ensembles}
% Figures \ref{fig:pe_per}, \ref{fig:pe_oodmax}, and \ref{fig:pe_oodmin} show the performance of the last layer packed ensemble with different numbers of estimators (5, 10, 15, 20, 25, 30, 35, 40, 45, 50) for different learning rates ($1e^{-2}$, $1e^{-3}$, and $1e^{-4}$) trained for different numbers of epochs (5, 10, 15, 20, 25) with different batch sizes (16, 32, 64). %In general, we do not observe that increasing the the number of estimators improves the performance or the calibration for the three datasets. A larger learning rate does have a positive effect on the performance metrics.
% \begin{figure}[h]
% \centering
% \includegraphics[height=16cm,keepaspectratio]{hyper/avg_performance_pe.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the Packed ensemble performance results for different hyperparameter settings for the regular DIR and DAR scenarios.}
% % \vspace{-0.5cm}
% \label{fig:pe_per}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_max_pe.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the Packed ensemble performance results for different hyperparameter settings for the OOD max scenario.}
% % \vspace{-0.5cm}
% \label{fig:pe_oodmax}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_min_pe.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the Packed ensemble performance results for different hyperparameter settings for the OOD min scenario.}
% % \vspace{-0.5cm}
% \label{fig:pe_oodmin}
% \end{figure}
% \clearpage
% \subsection{Variational Bayes Last Layer}
% Figures \ref{fig:vbll_per}, \ref{fig:vbll_oodmax}, and \ref{fig:vbll_oodmin} show the performance of the VBLL with different values for the KL-penalty (1, 5, 25) for different learning rates ($1e^{-2}$, $1e^{-3}$, and $1e^{-4}$) trained for different numbers of epochs (5, 10, 15, 20, 25) with different batch sizes (16, 32, 64). % In general, we do not observe that increasing the the number of estimators improves the performance or the calibration for the three datasets. A larger learning rate does have a positive effect on the performance metrics.
% \begin{figure}[H]
% \centering
% \includegraphics[height=16cm,keepaspectratio]{hyper/avg_performance_vbll.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the VBLL performance results for different hyperparameter settings for the regular DIR and DAR scenarios.}
% % \vspace{-0.5cm}
% \label{fig:vbll_per}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_max_vbll.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the VBLL performance results for different hyperparameter settings for the OOD max scenario.}
% % \vspace{-0.5cm}
% \label{fig:vbll_oodmax}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_min_vbll.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the VBLL performance results for different hyperparameter settings for the OOD min scenario.}
% % \vspace{-0.5cm}
% \label{fig:vbll_oodmin}
% \end{figure}
% \clearpage
% \subsection{Bayesian-by-backprop}
% Figures \ref{fig:bbb_per}, \ref{fig:bbb_oodmax}, and \ref{fig:bbb_oodmin} show the performance of the last layer BBB with different values for 50 samples for different learning rates ($1e^{-2}$, $1e^{-3}$, and $1e^{-4}$) trained for different numbers of epochs (5, 10, 15, 20) with different batch sizes (16, 32, 64). % In general, we do not observe that increasing the the number of estimators improves the performance or the calibration for the three datasets. A larger learning rate does have a positive effect on the performance metrics.
% \begin{figure}[H]
% \centering
% \includegraphics[height=16cm,keepaspectratio]{hyper/avg_performance_bbb.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the BBB performance results for different hyperparameter settings for the regular DIR and DAR scenarios.}
% % \vspace{-0.5cm}
% \label{fig:bbb_per}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_max_bbb.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the BBB performance results for different hyperparameter settings for the OOD max scenario.}
% % \vspace{-0.5cm}
% \label{fig:bbb_oodmax}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_min_bbb.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the BBB performance results for different hyperparameter settings for the OOD min scenario.}
% % \vspace{-0.5cm}
% \label{fig:bbb_oodmin}
% \end{figure}
% \clearpage
% \subsection{Last Layer HMC - 1 Chain}
% Figures \ref{fig:llhmc_per}, \ref{fig:llhmc_oodmax}, and \ref{fig:llhmc_oodmin} show the performance of the LL--HMC (C=1) with different values for the Gaussian priors (0.01, 0.1, 1, 2.5, 5, 10), number of burn-in samples (10, 25, 50, 100, 200), target acceptance probabilities (0.6, 0.7, 0.8), and number of collected samples (2, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50). % In general, we do not observe that increasing the the number of estimators improves the performance or the calibration for the three datasets. A larger learning rate does have a positive effect on the performance metrics.
% \begin{figure}[H]
% \centering
% \includegraphics[height=16cm,keepaspectratio]{hyper/avg_performance_llhmc.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the LL--HMC (C=1) performance results for different hyperparameter settings for the regular DIR and DAR scenarios.}
% % \vspace{-0.5cm}
% \label{fig:llhmc_per}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_max_llhmc.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the LL--HMC (C=1) performance results for different hyperparameter settings for the OOD max scenario.}
% % \vspace{-0.5cm}
% \label{fig:llhmc_oodmax}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_min_llhmc.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the LL--HMC (C=1) performance results for different hyperparameter settings for the OOD min scenario.}
% % \vspace{-0.5cm}
% \label{fig:llhmc_oodmin}
% \end{figure}
% \clearpage
% \subsection{Last Layer HMC - 2 Chains}
% Figures \ref{fig:llhmc2_per}, \ref{fig:llhmc2_oodmax}, and \ref{fig:llhmc2_oodmin} show the performance of the LL--HMC (C=2) with different values for the Gaussian priors (0.01, 0.1, 1, 2.5, 5, 10), number of burn-in samples (10, 25, 50, 100, 200), target acceptance probabilities (0.6, 0.7, 0.8), and number of collected samples (2, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50). % In general, we do not observe that increasing the the number of estimators improves the performance or the calibration for the three datasets. A larger learning rate does have a positive effect on the performance metrics.
% \begin{figure}[H]
% \centering
% \includegraphics[height=16cm,keepaspectratio]{hyper/avg_performance_llhmc_2chains.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the LL--HMC (C=2) performance results for different hyperparameter settings for the regular DIR and DAR scenarios.}
% % \vspace{-0.5cm}
% \label{fig:llhmc2_per}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_max_llhmc_2chains.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the LL--HMC (C=2) performance results for different hyperparameter settings for the OOD max scenario.}
% % \vspace{-0.5cm}
% \label{fig:llhmc2_oodmax}
% \end{figure}
% \begin{figure}[t]
% \centering
% \includegraphics[height=15cm,keepaspectratio]{hyper/ood_min_llhmc_2chains.png}
% % \vspace{-0.3cm}
% \caption{Masked pairplot for the LL--HMC (C=2) performance results for different hyperparameter settings for the OOD min scenario.}
% % \vspace{-0.5cm}
% \label{fig:llhmc2_oodmin}
% \end{figure}
\end{document}
%
\begin{figure*}[tbp]
\begin{tikzpicture}
\begin{groupplot}[
group style={
group size=3 by 2, % Changed to 3 by 2 for two rows
horizontal sep=1.5cm,
vertical sep=2.0cm, % Increased vertical separation for row titles
},
width=6.0cm,
height=3.8cm,
grid=major,
xlabel={Vision transformer block number},
% xmode=log,
xtick={1,2,3,4,5,6,7,8,9,10,11,12(LL)},
% xticklabels={0.03,0.1,0.3,1,3,10},
tick label style={font=\tiny},
label style={font=\tiny},
y tick label style={
/pgf/number format/fixed,
/pgf/number format/precision=0,
/pgf/number format/fixed zerofill=true,
},
tick label style={font=\fontsize{4}{4}\selectfont},
ylabel style={yshift=-0.2cm}, % Shift y-label down
xlabel style={yshift=0.1cm}, % Shift x-label down
]
% ROW 1: Original Plots with Row Title
\node[font=\bfseries\tiny] at ($(group c2r1.north)+(0,1.0cm)$) {Single intermediate latent representation performance};
% ROW 2: Duplicate plots for now with Row Title
\node[font=\bfseries\tiny] at ($(group c2r1.south)+(0,-1.0cm)$) {Row 2: Additional Experiments};
% First plot (Accuracy for AIDE Dataset)
\nextgroupplot[
title={\tiny{\textbf{AIDE} \citep{yang2023aide}}},
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11,12},
% ytick={50,70,90},
ymin=0,
ymax=100 % Adjusted for AIDE dataset accuracy values
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1,33.62986035646004)
(2,42.981525914077234)
(3,35.203816671725896)
(4,34.47937250299856)
(5,48.625966957940187)
(6,59.12597096882883)
(7,62.14583908938225)
(8,67.80544709124842)
(9,73.71387588718638)
(10,76.86452446648869)
(11,78.72744719209518)
(12,83.70)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1.5pt] coordinates {
(1,52.73290452639203)
(2,55.02681881226503)
(3,62.14862292733013)
(4,60.48207555838631)
(5,66.30087404848333)
(6,47.581844642046517)
(7,74.14812554661229)
(8,70.60382300562675)
(9,70.80096073444461)
(10,75.13629115709701)
(11,80.23062199581839)
(12,83.52)
};
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1pt] coordinates {
(1,47.1100185570691)
(2,60.90393686842245)
(3,63.81982939877677)
(4,64.31501852368)
(5,67.98399530097423)
(6,59.51358976613593)
(7,71.74839562521671)
(8,69.9271523491368)
(9,75.36192468968781)
(10,80.89753851197404)
(11,77.2974306001381)
(12,82.97)
};
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt] coordinates {
(1,33.6155470618008)
(2,42.59924855761078)
(3,43.212005108556834)
(4,43.783319366023593)
(5,50.80479676497246)
(6,55.13477713422311)
(7,48.795389297708097)
(8,70.10072587847654)
(9,69.59258674594464)
(10,77.35510973797343)
(11,79.80739009686378)
(12,83.00)
};
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.5pt] coordinates {
(1,43.114234801465423)
(2,44.7572299927923)
(3,47.90394489433516)
(4,56.6210530822757)
(5,65.01858224875633)
(6,67.14035708331452)
(7,70.81958275847653)
(8,73.10223386653483)
(9,77.7527887359122)
(10,79.58866337030732)
(11,81.29420037135711)
(12, 84.29)
};
% Second plot (B4C dataset)
\nextgroupplot[
title={\tiny{\textbf{B4C} \citep{jain2015car}}},
ylabel={\tiny{F1 (\%)}},
% ytick={40,50,60},
xtick={1,2,3,4,5,6,7,8,9,10,11,12},
ymax=100,
ymin=0,
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange, }, mark size=1.5pt] coordinates {
(1,10.872855528541588)
(2,26.23809523809524)
(3,20.446122168840083)
(4,24.818994958529847)
(5,36.75473416296553)
(6,47.27750721500722)
(7,44.33464483125622)
(8,49.74980574980575)
(9,55.76768006973957)
(10,82.00661910889828)
(11,91.24444444444444)
(12,95.00)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan, }, mark size=1.5pt] coordinates {
(1,19.462641794197966)
(2,18.088401890698855)
(3,21.470588235294122)
(4,32.87426900584795)
(5,40.82635958324392)
(6,41.80623973727422)
(7,37.46655120155878)
(8,56.22351478308926)
(9,64.96405959631766)
(10,82.25837320574163)
(11,93.24848484848486)
(12,95.03)
};
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red, }, mark size=1.5pt] coordinates {
(1,13.81155820500083)
(2,23.284361176164453)
(3,27.55975436939065)
(4,39.28521176305051)
(5,48.523952635014966)
(6,43.692555331991956)
(7,45.54093641314737)
(8,64.93845565374318)
(9,67.60291234248748)
(10,83.41823341163614)
(11,94.32901290129013)
(12,94.39)
};
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green, }, mark size=1.5pt] coordinates {
(1,21.926406926406922)
(2,16.495575221238937)
(3,19.179550608122037)
(4,22.670710412645895)
(5,36.78742608154373)
(6,38.38335907038197)
(7,41.876138433515475)
(8,25.82610372084056)
(9,44.90193955566311)
(10,85.83951734539971)
(11,86.74388312545046)
(12,94.93)
};
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow, }, mark size=1.5pt] coordinates {
(1,14.859671365003604)
(2,16.581900175800657)
(3,24.900693291670736)
(4,32.59324055343956)
(5,40.928433268858794)
(6,42.53738210259949)
(7,44.10249042145594)
(8,57.21845790947958)
(9,65.36742468319987)
(10,86.69349063114444)
(11,90.32267165458656)
(12, 95.00)
};
% Third plot (ROAD dataset)
\nextgroupplot[
title={\tiny{\textbf{ROAD} \citep{singh2022road}}},
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11,12},
% ytick={20,40,60,80},
ymin=0,
ymax=70
]
% Plot data for each prior value in AIDE dataset
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1,40.191369477083766)
(2,50.40816984285577)
(3,47.945823365037127)
(4,55.57985873775347)
(5,41.366503866503873)
(6,27.844664071079167)
(7,44.163003663003664)
(8,29.56521739130435)
(9,38.006133600981373)
(10,44.69954648526077)
(11,49.19193987516969)
(12,56.47)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1.5pt] coordinates {
(1,29.413179413179413)
(2,50.17366444785799)
(3,34.61273666092944)
(4,45.490054924545403)
(5,50.35757594485692)
(6,31.87335540276717)
(7,48.928466169845475)
(8,41.628102746115164)
(9,41.713185926260987)
(10,48.15875038686304)
(11,44.61060211410842)
(12, 57.54)
};
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
(1,30.428571428571427)
(2,50.78970718722272)
(3,46.02453102453103)
(4,54.20188277331135)
(5,52.28571428571429)
(6,50.3763440860215)
(7,48.7833908378373)
(8,45.9679191775966)
(9,35.80368906455863)
(10,40.330859616573905)
(11,43.0851210262975)
(12,47.65)
};
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt] coordinates {
(1,15.0)
(2,8.836524300441827)
(3,8.836524300441827)
(4,5.689548546691404)
(5,4.642857142857143)
(6,8.836524300441827)
(7,8.836524300441827)
(8,8.836524300441827)
(9,18.225200527972385)
(10,26.68267306922769)
(11,35.55249686081329)
(12,49.34)
};
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.5pt] coordinates {
(1,36.84677349405007)
(2,45.11111111111111)
(3,52.64745898359343)
(4,50.42037327751613)
(5,52.45243543641713)
(6,45.65516165516165)
(7,22.764511474188895)
(8,34.724402581545444)
(9,37.960622710622705)
(10,43.422320040976653)
(11,42.906400018201263)
(12,54.73)
};
% First plot in second row (AIDE dataset duplicate for now)
\nextgroupplot[
% title={\tiny{\textbf{AIDE} \citep{yang2023aide}}},
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11,12},
% ytick={50,70,90},
ymin=0,
ymax=100 % Adjusted for AIDE dataset accuracy values
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1,33.62986035646004)
(2,42.981525914077234)
(3,35.203816671725896)
(4,34.47937250299856)
(5,48.625966957940187)
(6,59.12597096882883)
(7,62.14583908938225)
(8,67.80544709124842)
(9,73.71387588718638)
(10,76.86452446648869)
(11,78.72744719209518)
(12,83.70)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1.5pt] coordinates {
(1,52.73290452639203)
(2,55.02681881226503)
(3,62.14862292733013)
(4,60.48207555838631)
(5,66.30087404848333)
(6,47.581844642046517)
(7,74.14812554661229)
(8,70.60382300562675)
(9,70.80096073444461)
(10,75.13629115709701)
(11,80.23062199581839)
(12,83.52)
};
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1pt] coordinates {
(1,47.1100185570691)
(2,60.90393686842245)
(3,63.81982939877677)
(4,64.31501852368)
(5,67.98399530097423)
(6,59.51358976613593)
(7,71.74839562521671)
(8,69.9271523491368)
(9,75.36192468968781)
(10,80.89753851197404)
(11,77.2974306001381)
(12,82.97)
};
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt] coordinates {
(1,33.6155470618008)
(2,42.59924855761078)
(3,43.212005108556834)
(4,43.783319366023593)
(5,50.80479676497246)
(6,55.13477713422311)
(7,48.795389297708097)
(8,70.10072587847654)
(9,69.59258674594464)
(10,77.35510973797343)
(11,79.80739009686378)
(12,83.00)
};
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.5pt] coordinates {
(1,43.114234801465423)
(2,44.7572299927923)
(3,47.90394489433516)
(4,56.6210530822757)
(5,65.01858224875633)
(6,67.14035708331452)
(7,70.81958275847653)
(8,73.10223386653483)
(9,77.7527887359122)
(10,79.58866337030732)
(11,81.29420037135711)
(12, 84.29)
};
% Second plot in second row (B4C dataset duplicate for now)
\nextgroupplot[
% title={\tiny{\textbf{B4C} \citep{jain2015car}}},
ylabel={\tiny{F1 (\%)}},
% ytick={40,50,60},
xtick={1,2,3,4,5,6,7,8,9,10,11,12},
ymax=100,
ymin=0,
]
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange, }, mark size=1.5pt] coordinates {
(1,10.872855528541588)
(2,26.23809523809524)
(3,20.446122168840083)
(4,24.818994958529847)
(5,36.75473416296553)
(6,47.27750721500722)
(7,44.33464483125622)
(8,49.74980574980575)
(9,55.76768006973957)
(10,82.00661910889828)
(11,91.24444444444444)
(12,95.00)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan, }, mark size=1.5pt] coordinates {
(1,19.462641794197966)
(2,18.088401890698855)
(3,21.470588235294122)
(4,32.87426900584795)
(5,40.82635958324392)
(6,41.80623973727422)
(7,37.46655120155878)
(8,56.22351478308926)
(9,64.96405959631766)
(10,82.25837320574163)
(11,93.24848484848486)
(12,95.03)
};
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red, }, mark size=1.5pt] coordinates {
(1,13.81155820500083)
(2,23.284361176164453)
(3,27.55975436939065)
(4,39.28521176305051)
(5,48.523952635014966)
(6,43.692555331991956)
(7,45.54093641314737)
(8,64.93845565374318)
(9,67.60291234248748)
(10,83.41823341163614)
(11,94.32901290129013)
(12,94.39)
};
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green, }, mark size=1.5pt] coordinates {
(1,21.926406926406922)
(2,16.495575221238937)
(3,19.179550608122037)
(4,22.670710412645895)
(5,36.78742608154373)
(6,38.38335907038197)
(7,41.876138433515475)
(8,25.82610372084056)
(9,44.90193955566311)
(10,85.83951734539971)
(11,86.74388312545046)
(12,94.93)
};
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow, }, mark size=1.5pt] coordinates {
(1,14.859671365003604)
(2,16.581900175800657)
(3,24.900693291670736)
(4,32.59324055343956)
(5,40.928433268858794)
(6,42.53738210259949)
(7,44.10249042145594)
(8,57.21845790947958)
(9,65.36742468319987)
(10,86.69349063114444)
(11,90.32267165458656)
(12, 95.00)
};
% Third plot in second row (ROAD dataset duplicate for now)
\nextgroupplot[
% title={\tiny{\textbf{ROAD} \citep{singh2022road}}},
ylabel={\tiny{F1 (\%)}},
xtick={1,2,3,4,5,6,7,8,9,10,11,12},
% ytick={20,40,60,80},
ymin=0,
ymax=70
]
% Plot data for each prior value in AIDE dataset
\addplot[mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=1.5pt] coordinates {
(1,40.191369477083766)
(2,50.40816984285577)
(3,47.945823365037127)
(4,55.57985873775347)
(5,41.366503866503873)
(6,27.844664071079167)
(7,44.163003663003664)
(8,29.56521739130435)
(9,38.006133600981373)
(10,44.69954648526077)
(11,49.19193987516969)
(12,56.47)
};
\addplot[mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=1.5pt] coordinates {
(1,29.413179413179413)
(2,50.17366444785799)
(3,34.61273666092944)
(4,45.490054924545403)
(5,50.35757594485692)
(6,31.87335540276717)
(7,48.928466169845475)
(8,41.628102746115164)
(9,41.713185926260987)
(10,48.15875038686304)
(11,44.61060211410842)
(12, 57.54)
};
\addplot[mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=1.5pt] coordinates {
(1,30.428571428571427)
(2,50.78970718722272)
(3,46.02453102453103)
(4,54.20188277331135)
(5,52.28571428571429)
(6,50.3763440860215)
(7,48.7833908378373)
(8,45.9679191775966)
(9,35.80368906455863)
(10,40.330859616573905)
(11,43.0851210262975)
(12,47.65)
};
\addplot[mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=1.5pt] coordinates {
(1,15.0)
(2,8.836524300441827)
(3,8.836524300441827)
(4,5.689548546691404)
(5,4.642857142857143)
(6,8.836524300441827)
(7,8.836524300441827)
(8,8.836524300441827)
(9,18.225200527972385)
(10,26.68267306922769)
(11,35.55249686081329)
(12,49.34)
};
\addplot[mark=diamond*, thin, line width=0.25pt, mark options={fill=yellow}, mark size=1.5pt] coordinates {
(1,36.84677349405007)
(2,45.11111111111111)
(3,52.64745898359343)
(4,50.42037327751613)
(5,52.45243543641713)
(6,45.65516165516165)
(7,22.764511474188895)
(8,34.724402581545444)
(9,37.960622710622705)
(10,43.422320040976653)
(11,42.906400018201263)
(12,54.73)
};
\end{groupplot}
\end{tikzpicture}
% Legend
\begin{tikzpicture}
\hspace{5.cm}
\begin{axis}[
hide axis,
xmin=0, xmax=1, ymin=0, ymax=1,
legend style={at={(0.5,0.5)}, anchor=center, legend columns=7},
legend entries={\tiny{BBB},\tiny{PE},\tiny{SE},\tiny{VBLL},\tiny{LL--HMC}}
]
\addlegendimage{mark=*, thin, line width=0.25pt, mark options={fill=orange}, mark size=2.0pt}
\addlegendimage{mark=triangle*, thin, line width=0.25pt, mark options={fill=cyan}, mark size=2.0pt}
\addlegendimage{mark=square*, thin, line width=0.25pt, mark options={fill=red}, mark size=2pt}
\addlegendimage{mark=pentagon*, thin, line width=0.25pt, mark options={fill=green}, mark size=2.0pt}
\addlegendimage{mark=diamond*, thin,line width=0.25pt, mark options={fill=yellow}, mark size=2.0pt}
% \addlegendimage{mark=oplus*, thin,line width=0.25pt, mark options={fill=yellow}, mark size=2.0pt}
% \addlegendimage{mark=otimes*, thin, line width=0.25pt, mark options={fill=brown}, mark size=2.0pt}
\end{axis}
\end{tikzpicture}
\caption{Performance overview based on the fine-tuned ViT-Base intermediate latent representations for the methods. The top row shows the original results, while the bottom row presents additional experiments with the same datasets.}
\label{fig:inter}
\end{figure*}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\begin{thebibliography}{92}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Abe et~al.(2022)Abe, Buchanan, Pleiss, Zemel, and Cunningham]{abe2022deep}
Taiga Abe, Estefany~Kelly Buchanan, Geoff Pleiss, Richard Zemel, and John~P Cunningham.
\newblock Deep ensembles work, but are they necessary?
\newblock \emph{Advances in Neural Information Processing Systems}, 35:\penalty0 33646--33660, 2022.
\bibitem[Arbel et~al.(2023)Arbel, Pitas, Vladimirova, and Fortuin]{arbel2023primer}
Julyan Arbel, Konstantinos Pitas, Mariia Vladimirova, and Vincent Fortuin.
\newblock A primer on bayesian neural networks: review and debates.
\newblock \emph{arXiv preprint arXiv:2309.16314}, 2023.
\bibitem[Arnab et~al.(2021)Arnab, Dehghani, Heigold, Sun, Lu{\v{c}}i{\'c}, and Schmid]{arnab2021vivit}
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu{\v{c}}i{\'c}, and Cordelia Schmid.
\newblock Vivit: A video vision transformer.
\newblock In \emph{Proceedings of the IEEE/CVF {I}nternational {C}onference on {C}omputer {V}ision}, pp.\ 6836--6846, 2021.
\bibitem[Ashukha et~al.(2020)Ashukha, Lyzhov, Molchanov, and Vetrov]{ashukhapitfalls}
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov.
\newblock Pitfalls of in-domain uncertainty estimation and ensembling in deep learning.
\newblock In \emph{International Conference on Learning Representations}, 2020.
\bibitem[Barde \& Barde(2012)Barde and Barde]{barde2012use}
Mohini~P Barde and Prajakt~J Barde.
\newblock What to use to express the variability of data: Standard deviation or standard error of mean?
\newblock \emph{Perspectives in clinical research}, 3\penalty0 (3):\penalty0 113--116, 2012.
\bibitem[Bengio et~al.(2024)Bengio, Hinton, Yao, Song, Abbeel, Darrell, Harari, Zhang, Xue, Shalev-Shwartz, et~al.]{bengio2024managing}
Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval~Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et~al.
\newblock Managing extreme ai risks amid rapid progress.
\newblock \emph{Science}, 384\penalty0 (6698):\penalty0 842--845, 2024.
\bibitem[Benker et~al.(2021)Benker, Furtner, Semm, and Zaeh]{benker2021utilizing}
Maximilian Benker, Lukas Furtner, Thomas Semm, and Michael~F Zaeh.
\newblock Utilizing uncertainty information in remaining useful life estimation via bayesian neural networks and hamiltonian monte carlo.
\newblock \emph{Journal of Manufacturing Systems}, 61:\penalty0 799--807, 2021.
\bibitem[Berrada et~al.(2021)Berrada, Dathathri, Dvijotham, Stanforth, Bunel, Uesato, Gowal, and Kumar]{berrada2021make}
Leonard Berrada, Sumanth Dathathri, Krishnamurthy Dvijotham, Robert Stanforth, Rudy~R Bunel, Jonathan Uesato, Sven Gowal, and M~Pawan Kumar.
\newblock Make sure you're unsure: A framework for verifying probabilistic specifications.
\newblock \emph{Advances in Neural Information Processing Systems}, 34:\penalty0 11136--11147, 2021.
\bibitem[Betancourt(2017)]{betancourt2017conceptual}
Michael Betancourt.
\newblock A conceptual introduction to hamiltonian monte carlo.
\newblock \emph{arXiv preprint arXiv:1701.02434}, 2017.
\bibitem[Bingham et~al.(2018)Bingham, Chen, Jankowiak, Obermeyer, Pradhan, Karaletsos, Singh, Szerlip, Horsfall, and Goodman]{bingham2018pyro}
Eli Bingham, Jonathan~P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah~D. Goodman.
\newblock {Pyro: Deep Universal Probabilistic Programming}.
\newblock \emph{Journal of Machine Learning Research}, 2018.
\bibitem[Blei et~al.(2017)Blei, Kucukelbir, and McAuliffe]{blei2017variational}
David~M Blei, Alp Kucukelbir, and Jon~D McAuliffe.
\newblock Variational inference: A review for statisticians.
\newblock \emph{Journal of the American statistical Association}, 112\penalty0 (518):\penalty0 859--877, 2017.
\bibitem[Blundell et~al.(2015)Blundell, Cornebise, Kavukcuoglu, and Wierstra]{blundell2015weight}
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.
\newblock Weight uncertainty in neural network.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 1613--1622. PMLR, 2015.
\bibitem[Bouthillier et~al.(2019)Bouthillier, Laurent, and Vincent]{bouthillier2019unreproducible}
Xavier Bouthillier, C{\'e}sar Laurent, and Pascal Vincent.
\newblock Unreproducible research is reproducible.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 725--734. PMLR, 2019.
\bibitem[Chen et~al.(2014)Chen, Fox, and Guestrin]{chen2014stochastic}
Tianqi Chen, Emily Fox, and Carlos Guestrin.
\newblock Stochastic gradient hamiltonian monte carlo.
\newblock In \emph{International conference on machine learning}, pp.\ 1683--1691. PMLR, 2014.
\bibitem[Cobb \& Jalaian(2021)Cobb and Jalaian]{cobb2021scaling}
Adam~D Cobb and Brian Jalaian.
\newblock Scaling hamiltonian monte carlo inference for bayesian neural networks with symmetric splitting.
\newblock In \emph{Uncertainty in Artificial Intelligence}, pp.\ 675--685. PMLR, 2021.
\bibitem[Cobb et~al.(2018)Cobb, Roberts, and Gal]{cobb2018loss}
Adam~D Cobb, Stephen~J Roberts, and Yarin Gal.
\newblock Loss-calibrated approximate inference in bayesian neural networks.
\newblock \emph{arXiv preprint arXiv:1805.03901}, 2018.
\bibitem[Dai et~al.(2021)Dai, Gieseke, Oehmcke, Wu, and Barnard]{dai2021attentional}
Yimian Dai, Fabian Gieseke, Stefan Oehmcke, Yiquan Wu, and Kobus Barnard.
\newblock Attentional feature fusion.
\newblock In \emph{Proceedings of the IEEE/CVF winter conference on applications of computer vision}, pp.\ 3560--3569, 2021.
\bibitem[Der~Kiureghian \& Ditlevsen(2009)Der~Kiureghian and Ditlevsen]{der2009aleatory}
Armen Der~Kiureghian and Ove Ditlevsen.
\newblock Aleatory or epistemic? {D}oes it matter?
\newblock \emph{Structural safety}, 31\penalty0 (2):\penalty0 105--112, 2009.
\bibitem[Durasov et~al.(2024)Durasov, Dorndorf, Le, and Fua]{durasov2024zigzag}
Nikita Durasov, Nik Dorndorf, Hieu Le, and Pascal Fua.
\newblock Zigzag: Universal sampling-free uncertainty estimation through two-step inference.
\newblock \emph{Transactions on Machine Learning Research}, 2024.
\newblock ISSN 2835-8856.
\bibitem[Franchi et~al.(2024)Franchi, Laurent, Legu{\'e}ry, Bursuc, Pilzer, and Yao]{franchi2024make}
Gianni Franchi, Olivier Laurent, Maxence Legu{\'e}ry, Andrei Bursuc, Andrea Pilzer, and Angela Yao.
\newblock Make me a bnn: A simple strategy for estimating bayesian uncertainty from pre-trained models.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 12194--12204, 2024.
\bibitem[Gal \& Ghahramani(2016)Gal and Ghahramani]{gal2016dropout}
Yarin Gal and Zoubin Ghahramani.
\newblock Dropout as a {B}ayesian approximation: Representing model uncertainty in deep learning.
\newblock \emph{{I}nternational conference on machine learning}, pp.\ 1050--1059, 2016.
\bibitem[Gawlikowski et~al.(2023)Gawlikowski, Tassi, Ali, Lee, Humt, Feng, Kruspe, Triebel, Jung, Roscher, et~al.]{gawlikowski2023survey}
Jakob Gawlikowski, Cedrique Rovile~Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et~al.
\newblock A survey of uncertainty in deep neural networks.
\newblock \emph{Artificial Intelligence Review}, 56\penalty0 (Suppl 1):\penalty0 1513--1589, 2023.
\bibitem[Gebert et~al.(2019)Gebert, Roitberg, Haurilet, and Stiefelhagen]{gebert2019end}
Patrick Gebert, Alina Roitberg, Monica Haurilet, and Rainer Stiefelhagen.
\newblock End-to-end prediction of driver intention using {3D} convolutional neural networks.
\newblock In \emph{2019 IEEE Intelligent Vehicles Symposium (IV)}, pp.\ 969--974. IEEE, 2019.
\bibitem[Gelman \& Rubin(1992)Gelman and Rubin]{gelman1992inference}
Andrew Gelman and Donald~B Rubin.
\newblock Inference from iterative simulation using multiple sequences.
\newblock \emph{Statistical science}, 7\penalty0 (4):\penalty0 457--472, 1992.
\bibitem[Geyer(1992)]{geyer1992practical}
Charles~J Geyer.
\newblock Practical {M}arkov {C}hain {M}onte {C}arlo.
\newblock \emph{Statistical science}, pp.\ 473--483, 1992.
\bibitem[Geyer(2011)]{geyer2011introduction}
Charles~J Geyer.
\newblock Introduction to {Markov chain Monte Carlo}.
\newblock \emph{Handbook of {Markov chain Monte Carlo}}, 2011.
\bibitem[Guo et~al.(2017)Guo, Pleiss, Sun, and Weinberger]{guo2017calibration}
Chuan Guo, Geoff Pleiss, Yu~Sun, and Kilian~Q Weinberger.
\newblock On calibration of modern neural networks.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 1321--1330. PMLR, 2017.
\bibitem[Habibpour et~al.(2023)Habibpour, Gharoun, Mehdipour, Tajally, Asgharnezhad, Shamsi, Khosravi, and Nahavandi]{habibpour2023uncertainty}
Maryam Habibpour, Hassan Gharoun, Mohammadreza Mehdipour, AmirReza Tajally, Hamzeh Asgharnezhad, Afshar Shamsi, Abbas Khosravi, and Saeid Nahavandi.
\newblock Uncertainty-aware credit card fraud detection using deep learning.
\newblock \emph{Engineering Applications of Artificial Intelligence}, 123:\penalty0 106248, 2023.
\bibitem[Harrison et~al.(2024)Harrison, Willes, and Snoek]{harrison2024variational}
James Harrison, John Willes, and Jasper Snoek.
\newblock Variational bayesian last layers.
\newblock \emph{The Twelfth International Conference on Learning Representations}, 2024.
\bibitem[Hendrycks et~al.(2021)Hendrycks, Carlini, Schulman, and Steinhardt]{hendrycks2021unsolved}
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt.
\newblock Unsolved problems in ml safety.
\newblock \emph{arXiv preprint arXiv:2109.13916}, 2021.
\bibitem[Hoffman et~al.(2013)Hoffman, Blei, Wang, and Paisley]{hoffman2013stochastic}
Matthew~D Hoffman, David~M Blei, Chong Wang, and John Paisley.
\newblock Stochastic variational inference.
\newblock \emph{Journal of Machine Learning Research}, 2013.
\bibitem[Hoffman et~al.(2014)Hoffman, Gelman, et~al.]{hoffman2014no}
Matthew~D Hoffman, Andrew Gelman, et~al.
\newblock The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo.
\newblock \emph{J. Mach. Learn. Res.}, 15\penalty0 (1):\penalty0 1593--1623, 2014.
\bibitem[H{\"u}llermeier \& Waegeman(2021)H{\"u}llermeier and Waegeman]{hullermeier2021aleatoric}
Eyke H{\"u}llermeier and Willem Waegeman.
\newblock Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.
\newblock \emph{Machine learning}, 110\penalty0 (3):\penalty0 457--506, 2021.
\bibitem[H{\"u}llermeier et~al.(2022)H{\"u}llermeier, Destercke, and Shaker]{hullermeier2022quantification}
Eyke H{\"u}llermeier, S{\'e}bastien Destercke, and Mohammad~Hossein Shaker.
\newblock Quantification of credal uncertainty in machine learning: A critical analysis and empirical comparison.
\newblock In \emph{Uncertainty in Artificial Intelligence}, pp.\ 548--557. PMLR, 2022.
\bibitem[Izmailov et~al.(2021)Izmailov, Vikram, Hoffman, and Wilson]{izmailov2021bayesian}
Pavel Izmailov, Sharad Vikram, Matthew~D Hoffman, and Andrew Gordon~Gordon Wilson.
\newblock What are bayesian neural network posteriors really like?
\newblock In \emph{International conference on machine learning}, pp.\ 4629--4640. PMLR, 2021.
\bibitem[Jain et~al.(2015)Jain, Koppula, Raghavan, Soh, and Saxena]{jain2015car}
Ashesh Jain, Hema~S Koppula, Bharad Raghavan, Shane Soh, and Ashutosh Saxena.
\newblock Car that knows before you do: Anticipating maneuvers via learning temporal driving models.
\newblock In \emph{Proceedings of the IEEE International Conference on Computer Vision}, pp.\ 3182--3190, 2015.
\bibitem[Jordan et~al.(1999)Jordan, Ghahramani, Jaakkola, and Saul]{jordan1999introduction}
Michael~I Jordan, Zoubin Ghahramani, Tommi~S Jaakkola, and Lawrence~K Saul.
\newblock An introduction to variational methods for graphical models.
\newblock \emph{Machine learning}, 37\penalty0 (2):\penalty0 183--233, 1999.
\bibitem[Jospin et~al.(2022)Jospin, Laga, Boussaid, Buntine, and Bennamoun]{jospin2022hands}
Laurent~Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun.
\newblock Hands-on {B}ayesian neural networks—a tutorial for deep learning users.
\newblock \emph{IEEE Computational Intelligence Magazine}, 17\penalty0 (2):\penalty0 29--48, 2022.
\bibitem[Kay et~al.(2017)Kay, Carreira, Simonyan, Zhang, Hillier, Vijayanarasimhan, Viola, Green, Back, Natsev, et~al.]{kay2017kinetics}
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et~al.
\newblock The kinetics human action video dataset.
\newblock \emph{arXiv preprint arXiv:1705.06950}, 2017.
\bibitem[Kingma \& Welling(2014)Kingma and Welling]{kingma2013auto}
Diederik~P Kingma and Max Welling.
\newblock Auto-encoding variational bayes.
\newblock \emph{International Conference on Learning Representations}, 2014.
\bibitem[Kothapalli(2023)]{kothapallineural}
Vignesh Kothapalli.
\newblock Neural collapse: A review on modelling principles and generalization.
\newblock \emph{Transactions on Machine Learning Research}, 2023.
\bibitem[Lakshminarayanan et~al.(2017)Lakshminarayanan, Pritzel, and Blundell]{lakshminarayanan2017simple}
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.
\newblock Simple and scalable predictive uncertainty estimation using deep ensembles.
\newblock \emph{Advances in neural information processing systems}, 30, 2017.
\bibitem[Laurent et~al.(2023)Laurent, Lafage, Tartaglione, Daniel, Martinez, Bursuc, and Franchi]{laurent2023packed}
Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean-marc Martinez, Andrei Bursuc, and Gianni Franchi.
\newblock Packed ensembles for efficient uncertainty estimation.
\newblock In \emph{The Eleventh International Conference on Learning Representations}, 2023.
\bibitem[Laurent et~al.(2024)Laurent, Aldea, and Franchi]{laurentsymmetry}
Olivier Laurent, Emanuel Aldea, and Gianni Franchi.
\newblock A symmetry-aware exploration of bayesian neural network posteriors.
\newblock In \emph{The Twelfth International Conference on Learning Representations}, 2024.
\bibitem[Lippe(2022)]{lippe2022uva}
Phillip Lippe.
\newblock {UvA} {Deep} {Learning} {Tutorials}.
\newblock \url{https://uvadlc-notebooks.readthedocs.io/en/latest/}, 2022.
\bibitem[Loshchilov \& Hutter(2018)Loshchilov and Hutter]{loshchilov2018decoupled}
Ilya Loshchilov and Frank Hutter.
\newblock Decoupled {W}eight {D}ecay {R}egularization.
\newblock In \emph{International Conference on Learning Representations}, 2018.
\bibitem[Maddern et~al.(2017)Maddern, Pascoe, Linegar, and Newman]{RobotCarDatasetIJRR}
Will Maddern, Geoff Pascoe, Chris Linegar, and Paul Newman.
\newblock {1 Year, 1000km: The Oxford RobotCar Dataset}.
\newblock \emph{The International Journal of Robotics Research (IJRR)}, 36\penalty0 (1):\penalty0 3--15, 2017.
\newblock \doi{10.1177/0278364916679498}.
\bibitem[Maddox et~al.(2019)Maddox, Izmailov, Garipov, Vetrov, and Wilson]{maddox2019simple}
Wesley~J Maddox, Pavel Izmailov, Timur Garipov, Dmitry~P Vetrov, and Andrew~Gordon Wilson.
\newblock A simple baseline for {B}ayesian uncertainty in deep learning.
\newblock \emph{Advances in Neural Information Processing Systems}, 32:\penalty0 13153--13164, 2019.
\bibitem[Malinin \& Gales(2018)Malinin and Gales]{malinin2018predictive}
Andrey Malinin and Mark Gales.
\newblock Predictive uncertainty estimation via prior networks.
\newblock \emph{Advances in neural information processing systems}, 31, 2018.
\bibitem[Manchingal \& Cuzzolin(2025)Manchingal and Cuzzolin]{manchingal2025position}
Shireen~Kudukkil Manchingal and Fabio Cuzzolin.
\newblock Position: Epistemic artificial intelligence is essential for machine learning models to know when they do not know.
\newblock \emph{arXiv preprint arXiv:2505.04950}, 2025.
\bibitem[Meronen et~al.(2024)Meronen, Trapp, Pilzer, Yang, and Solin]{meronen2024fixing}
Lassi Meronen, Martin Trapp, Andrea Pilzer, Le~Yang, and Arno Solin.
\newblock Fixing overconfidence in dynamic neural networks.
\newblock In \emph{Proceedings of the IEEE/CVF winter conference on applications of computer vision}, pp.\ 2680--2690, 2024.
\bibitem[Montavon et~al.(2011)Montavon, Braun, and M{\"u}ller]{montavon2011kernel}
Gr{\'e}goire Montavon, Mikio~L Braun, and Klaus-Robert M{\"u}ller.
\newblock Kernel analysis of deep networks.
\newblock \emph{Journal of Machine Learning Research}, 12\penalty0 (9), 2011.
\bibitem[Mucs{\'a}nyi et~al.(2024)Mucs{\'a}nyi, Kirchhof, and Oh]{mucsanyi2024benchmarking}
B{\'a}lint Mucs{\'a}nyi, Michael Kirchhof, and Seong~Joon Oh.
\newblock Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks.
\newblock \emph{Advances in neural information processing systems}, 37:\penalty0 50972--51038, 2024.
\bibitem[Mukhoti et~al.(2023)Mukhoti, Kirsch, van Amersfoort, Torr, and Gal]{mukhoti2023deep}
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip~HS Torr, and Yarin Gal.
\newblock Deep deterministic uncertainty: A new simple baseline.
\newblock In \emph{Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pp.\ 24384--24394, 2023.
\bibitem[Neal(1992)]{neal1992bayesian}
Radford Neal.
\newblock {B}ayesian learning via stochastic dynamics.
\newblock \emph{Advances in neural information processing systems}, 1992.
\bibitem[Neal et~al.(2011)]{neal2011mcmc}
Radford Neal et~al.
\newblock {MCMC} using hamiltonian dynamics.
\newblock \emph{Handbook of markov chain monte carlo}, 2011.
\bibitem[Neal(1996)]{neal2012bayesian}
Radford~M Neal.
\newblock \emph{Bayesian learning for neural networks}.
\newblock Springer Science \& Business Media, 1996.
\bibitem[Nguyen et~al.(2022)Nguyen, Medjaher, and Gogu]{nguyen2022probabilistic}
Khanh~TP Nguyen, Kamal Medjaher, and Christian Gogu.
\newblock Probabilistic deep learning methodology for uncertainty quantification of remaining useful lifetime of multi-component systems.
\newblock \emph{Reliability Engineering \& System Safety}, 222:\penalty0 108383, 2022.
\bibitem[Nixon et~al.(2019)Nixon, Dusenberry, Zhang, Jerfel, and Tran]{nixon2019measuring}
Jeremy Nixon, Michael~W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran.
\newblock Measuring calibration in deep learning.
\newblock In \emph{CVPR Workshops}, volume~2, 2019.
\bibitem[Noguchi \& Tanizawa(2023)Noguchi and Tanizawa]{noguchi2023ego}
Chihiro Noguchi and Toshihiro Tanizawa.
\newblock Ego-vehicle action recognition based on semi-supervised contrastive learning.
\newblock In \emph{Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision}, pp.\ 5988--5998, 2023.
\bibitem[Papamarkou et~al.(2024)Papamarkou, Skoularidou, Palla, Aitchison, Arbel, Dunson, Filippone, Fortuin, Hennig, et~al.]{papamarkou2024position}
Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, et~al.
\newblock Position: Bayesian deep learning is needed in the age of large-scale ai.
\newblock In \emph{Forty-first International Conference on Machine Learning}, 2024.
\bibitem[Papyan et~al.(2020)Papyan, Han, and Donoho]{papyan2020prevalence}
Vardan Papyan, XY~Han, and David~L Donoho.
\newblock Prevalence of neural collapse during the terminal phase of deep learning training.
\newblock \emph{Proceedings of the National Academy of Sciences}, 117\penalty0 (40):\penalty0 24652--24663, 2020.
\bibitem[Pedregosa et~al.(2011)Pedregosa, Varoquaux, Gramfort, Michel, Thirion, Grisel, Blondel, Prettenhofer, Weiss, Dubourg, et~al.]{pedregosa2011scikit}
Fabian Pedregosa, Ga{\"e}l Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et~al.
\newblock Scikit-learn: Machine learning in python.
\newblock \emph{the Journal of machine Learning research}, 12:\penalty0 2825--2830, 2011.
\bibitem[Pereira \& Han(2009)Pereira and Han]{pereira2009elder}
Lu{\'\i}s~Moniz Pereira and The~Anh Han.
\newblock Elder care via intention recognition and evolution prospection.
\newblock In \emph{International Conference on Applications of Declarative Programming and Knowledge Management}, pp.\ 170--187. Springer, 2009.
\bibitem[Pereira et~al.(2013)]{pereira2013state}
Luis~Moniz Pereira et~al.
\newblock State-of-the-art of intention recognition and its use in decision making.
\newblock \emph{AI Communications}, 26\penalty0 (2):\penalty0 237--246, 2013.
\bibitem[Postels et~al.(2022)Postels, Seg{\`u}, Sun, Sieber, Van~Gool, Yu, and Tombari]{postels2022practicality}
Janis Postels, Mattia Seg{\`u}, Tao Sun, Luca~Daniel Sieber, Luc Van~Gool, Fisher Yu, and Federico Tombari.
\newblock On the practicality of deterministic epistemic uncertainty.
\newblock In \emph{International Conference on Machine Learning}, pp.\ 17870--17909. PMLR, 2022.
\bibitem[Qendro et~al.(2021)Qendro, Campbell, Lio, and Mascolo]{qendro2021early}
Lorena Qendro, Alexander Campbell, Pietro Lio, and Cecilia Mascolo.
\newblock Early exit ensembles for uncertainty quantification.
\newblock In \emph{Machine Learning for Health}, pp.\ 181--195. PMLR, 2021.
\bibitem[Qi et~al.(2024)Qi, Ma, Cui, and Yu]{qi2024computer}
Jing Qi, Li~Ma, Zhenchao Cui, and Yushu Yu.
\newblock Computer vision-based hand gesture recognition for human-robot interaction: a review.
\newblock \emph{Complex \& Intelligent Systems}, 10\penalty0 (1):\penalty0 1581--1606, 2024.
\bibitem[Rong et~al.(2020)Rong, Akata, and Kasneci]{rong2020driver}
Yao Rong, Zeynep Akata, and Enkelejda Kasneci.
\newblock Driver intention anticipation based on in-cabin and driving scene monitoring.
\newblock In \emph{2020 IEEE 23rd International Conference on Intelligent Transportation {S}ystems (ITSC)}, pp.\ 1--8. IEEE, 2020.
\bibitem[Rudin(2019)]{rudin2019stop}
Cynthia Rudin.
\newblock Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.
\newblock \emph{Nature machine intelligence}, 1\penalty0 (5):\penalty0 206--215, 2019.
\bibitem[Sadri(2011)]{sadri2011logic}
Fariba Sadri.
\newblock Logic-based approaches to intention recognition.
\newblock In \emph{Handbook of research on ambient intelligence and smart environments: Trends and perspectives}, pp.\ 346--375. IGI Global, 2011.
\bibitem[Shafer \& Vovk(2008)Shafer and Vovk]{shafer2008tutorial}
Glenn Shafer and Vladimir Vovk.
\newblock A tutorial on conformal prediction.
\newblock \emph{Journal of Machine Learning Research}, 9\penalty0 (3), 2008.
\bibitem[Singh et~al.(2022)Singh, Akrigg, Di~Maio, Fontana, Alitappeh, Saha, Jeddisaravi, Yousefi, Culley, Nicholson, et~al.]{singh2022road}
Gurkirt Singh, Stephen Akrigg, Manuele Di~Maio, Valentina Fontana, Reza~Javanmard Alitappeh, Suman Saha, Kossar Jeddisaravi, Farzad Yousefi, Jacob Culley, Tom Nicholson, et~al.
\newblock Road: The road event awareness dataset for autonomous driving.
\newblock \emph{IEEE Transactions on Pattern Analysis \& Machine Intelligence}, 45\penalty0 (01):\penalty0 1--1, feb 2022.
\newblock ISSN 1939-3539.
\newblock \doi{10.1109/TPAMI.2022.3150906}.
\bibitem[Smith \& Gal(2018)Smith and Gal]{smith2018understanding}
Lewis Smith and Yarin Gal.
\newblock Understanding measures of uncertainty for adversarial example detection.
\newblock \emph{arXiv preprint arXiv:1803.08533}, 2018.
\bibitem[Snoek et~al.(2015)Snoek, Rippel, Swersky, Kiros, Satish, Sundaram, Patwary, Prabhat, and Adams]{snoek2015scalable}
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr~Prabhat, and Ryan Adams.
\newblock Scalable {B}ayesian optimization using deep neural networks.
\newblock In \emph{International conference on machine learning}, pp.\ 2171--2180. PMLR, 2015.
\bibitem[{Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, Benjamin Bossan}(2022)]{accelerate}
{Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, Benjamin Bossan}.
\newblock Accelerate: Training and inference at scale made simple, efficient and adaptable.
\newblock \url{https://github.com/huggingface/accelerate}, 2022.
\bibitem[Tong et~al.(2022)Tong, Song, Wang, and Wang]{tong2022videomae}
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.
\newblock Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.
\newblock \emph{Advances in neural information processing systems}, 35:\penalty0 10078--10093, 2022.
\bibitem[Valdenegro-Toro(2023)]{valdenegro2023sub}
Matias Valdenegro-Toro.
\newblock Sub-ensembles for fast uncertainty estimation in neural networks.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 4119--4127, 2023.
\bibitem[Van~Amersfoort et~al.(2020)Van~Amersfoort, Smith, Teh, and Gal]{van2020uncertainty}
Joost Van~Amersfoort, Lewis Smith, Yee~Whye Teh, and Yarin Gal.
\newblock Uncertainty estimation using a single deep deterministic neural network.
\newblock In \emph{International conference on machine learning}, pp.\ 9690--9700. PMLR, 2020.
\bibitem[Vellenga et~al.(2022)Vellenga, Steinhauer, Karlsson, Falkman, Rhodin, and Koppisetty]{vellenga2022driver}
Koen Vellenga, H~Joe Steinhauer, Alexander Karlsson, G{\"o}ran Falkman, Asli Rhodin, and Ashok~Chaitanya Koppisetty.
\newblock Driver intention recognition: State-of-the-art review.
\newblock \emph{IEEE Open Journal of Intelligent Transportation Systems}, 2022.
\bibitem[Vellenga et~al.(2024{\natexlab{a}})Vellenga, Karlsson, Steinhauer, Falkman, and Sj{\"o}gren]{vellenga2024pthmc}
Koen Vellenga, Alexander Karlsson, H~Joe Steinhauer, G{\"o}ran Falkman, and Anders Sj{\"o}gren.
\newblock {PT-HMC}: Optimization-based pre-training with {H}amiltonian {M}onte-{C}arlo sampling for driver intention recognition.
\newblock \emph{ACM Transactions on Probabilistic Machine Learning}, 2024{\natexlab{a}}.
\bibitem[Vellenga et~al.(2024{\natexlab{b}})Vellenga, Steinhauer, Falkman, and Bj{\"o}rklund]{vellenga2024evaluation}
Koen Vellenga, H~Joe Steinhauer, G{\"o}ran Falkman, and Tomas Bj{\"o}rklund.
\newblock Evaluation of video masked autoencoders' performance and uncertainty estimations for driver action and intention recognition.
\newblock In \emph{Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision}, pp.\ 7429--7437, 2024{\natexlab{b}}.
\bibitem[Wang et~al.(2024{\natexlab{a}})Wang, Cuzzolin, Shariatmadar, Moens, and Hallez]{wang2024credal}
Kaizheng Wang, Fabio Cuzzolin, Keivan Shariatmadar, David Moens, and Hans Hallez.
\newblock Credal wrapper of model averaging for uncertainty estimation in classification.
\newblock In \emph{The Thirteenth International Conference on Learning Representations}, 2024{\natexlab{a}}.
\bibitem[Wang et~al.(2024{\natexlab{b}})Wang, Jiao, Zhan, Lang, Huang, Wang, Yang, and Zhu]{wang2024empowering}
Yixuan Wang, Ruochen Jiao, Simon~Sinong Zhan, Chengtian Lang, Chao Huang, Zhaoran Wang, Zhuoran Yang, and Qi~Zhu.
\newblock Empowering autonomous driving with large language models: A safety perspective.
\newblock In \emph{ICLR 2024 Workshop on Large Language Model (LLM) Agents}, 2024{\natexlab{b}}.
\bibitem[Watson et~al.(2021)Watson, Lin, Klink, Pajarinen, and Peters]{watson2021latent}
Joe Watson, Jihao~Andreas Lin, Pascal Klink, Joni Pajarinen, and Jan Peters.
\newblock Latent derivative {B}ayesian last layer networks.
\newblock In \emph{International Conference on Artificial Intelligence and Statistics}, pp.\ 1198--1206. PMLR, 2021.
\bibitem[Welling \& Teh(2011)Welling and Teh]{welling2011bayesian}
Max Welling and Yee~W Teh.
\newblock Bayesian learning via stochastic gradient langevin dynamics.
\newblock In \emph{Proceedings of the 28th international conference on machine learning (ICML-11)}, pp.\ 681--688. Citeseer, 2011.
\bibitem[Wilson \& Izmailov(2020)Wilson and Izmailov]{wilson2020bayesian}
Andrew~G Wilson and Pavel Izmailov.
\newblock Bayesian deep learning and a probabilistic perspective of generalization.
\newblock \emph{Advances in neural information processing systems}, pp.\ 4697--4708, 2020.
\bibitem[Yang et~al.(2023)Yang, Huang, Xu, Li, Wang, Li, Wang, Liu, Yang, Chen, et~al.]{yang2023aide}
Dingkang Yang, Shuai Huang, Zhi Xu, Zhenpeng Li, Shunli Wang, Mingcheng Li, Yuzheng Wang, Yang Liu, Kun Yang, Zhaoyu Chen, et~al.
\newblock Aide: A vision-driven multi-view, multi-modal, multi-tasking dataset for assistive driving perception.
\newblock In \emph{Proceedings of the IEEE/CVF International Conference on Computer Vision}, pp.\ 20459--20470, 2023.
\bibitem[Yang et~al.(2024)Yang, Zhou, Li, and Liu]{yang2024generalized}
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu.
\newblock Generalized out-of-distribution detection: A survey.
\newblock \emph{International Journal of Computer Vision}, 132\penalty0 (12):\penalty0 5635--5662, 2024.
\bibitem[Zhang et~al.(2020)Zhang, Li, Zhang, Chen, and Wilson]{zhang2019cyclical}
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew~Gordon Wilson.
\newblock Cyclical stochastic gradient mcmc for bayesian deep learning.
\newblock \emph{International Conference on Learning Representations}, 2020.
\bibitem[Zhou et~al.(2022)Zhou, Liu, Pourpanah, Zeng, and Wang]{zhou2021survey}
Xinlei Zhou, Han Liu, Farhad Pourpanah, Tieyong Zeng, and Xizhao Wang.
\newblock A survey on epistemic (model) uncertainty in supervised learning: Recent advances and applications.
\newblock \emph{Neurocomputing}, pp.\ 449--465, 2022.
\bibitem[Zou et~al.(2023)Zou, Chen, Yuan, Shen, Wang, and Fu]{zou2023review}
Ke~Zou, Zhihao Chen, Xuedong Yuan, Xiaojing Shen, Meng Wang, and Huazhu Fu.
\newblock A review of uncertainty estimation and its application in medical imaging.
\newblock \emph{Meta-Radiology}, 1\penalty0 (1):\penalty0 100003, 2023.
\end{thebibliography}