dlordinal: a Python package for deep ordinal classification
Extracted source text · arXiv 2407.17163
%%
%% Copyright 2007-2024 Elsevier Ltd
%%
%% This file is part of the 'Elsarticle Bundle'.
%% ---------------------------------------------
%%
%% It may be distributed under the conditions of the LaTeX Project Public
%% License, either version 1.3 of this license or (at your option) any
%% later version. The latest version of this license is in
%% http://www.latex-project.org/lppl.txt
%% and version 1.3 or later is part of all distributions of LaTeX
%% version 1999/12/01 or later.
%%
%% The list of all files belonging to the 'Elsarticle Bundle' is
%% given in the file `manifest.txt'.
%%
%% Template article for Elsevier's document class `elsarticle'
%% with numbered style bibliographic references
%% SP 2008/03/01
%% $Id: elsarticle-template-num.tex 249 2024-04-06 10:51:24Z rishi $
%%
\documentclass[preprint,12pt]{elsarticle}
%% Use the option review to obtain double line spacing
%% \documentclass[authoryear,preprint,review,12pt]{elsarticle}
%% Use the options 1p,twocolumn; 3p; 3p,twocolumn; 5p; or 5p,twocolumn
%% for a journal layout:
%% \documentclass[final,1p,times]{elsarticle}
%% \documentclass[final,1p,times,twocolumn]{elsarticle}
%% \documentclass[final,3p,times]{elsarticle}
%% \documentclass[final,3p,times,twocolumn]{elsarticle}
%% \documentclass[final,5p,times]{elsarticle}
%% \documentclass[final,5p,times,twocolumn]{elsarticle}
%% For including figures, graphicx.sty has been loaded in
%% elsarticle.cls. If you prefer to use the old commands
%% please give \usepackage{epsfig}
%% The amssymb package provides various useful mathematical symbols
\usepackage{amssymb}
%% The amsmath package provides various useful equation environments.
\usepackage{amsmath}
%% The amsthm package provides extended theorem environments
%% \usepackage{amsthm}
\PassOptionsToPackage{hyphens}{url}\usepackage{hyperref}
%% The lineno packages adds line numbers. Start line numbering with
%% \begin{linenumbers}, end it with \end{linenumbers}. Or switch it on
%% for the whole article with \linenumbers.
%% \usepackage{lineno}
\usepackage{listings}
\usepackage{xcolor}
\usepackage[doublespacing]{setspace}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{makecell}
\usepackage[nameinlink]{cleveref}
\definecolor{codegreen}{rgb}{0,0.6,0}
\definecolor{codegray}{rgb}{0.5,0.5,0.5}
\definecolor{codepurple}{rgb}{0.58,0,0.82}
\definecolor{backcolour}{rgb}{0.97,0.97,0.97}
\lstdefinestyle{python}{
backgroundcolor=\color{backcolour},
commentstyle=\color{codegreen},
keywordstyle=\color{blue},
numberstyle=\tiny\color{codegray},
stringstyle=\color{codepurple},
basicstyle=\ttfamily\footnotesize,
breakatwhitespace=false,
breaklines=true,
captionpos=b,
keepspaces=true,
numbers=left,
numbersep=5pt,
showspaces=false,
showstringspaces=false,
showtabs=false,
tabsize=2
}
\lstset{style=python}
\usepackage{lastpage}
\newcommand{\betac}{Beta}
\newcommand{\binomc}{Binomial}
\newcommand{\clm}{CLM}
\newcommand{\clmwk}{CLMWK}
\newcommand{\expc}{Exponential}
\newcommand{\drop}{Hybrid dropout}
\newcommand{\ecoc}{OBDECOC}
\newcommand{\sbc}{SB}
\newcommand{\triang}{Triangular}
\newcommand{\rev}[1]{{\color{black}#1}}
%\newcommand{\rev}[1]{{#1}}
\journal{Neurocomputing}
\begin{document}
\begin{frontmatter}
\title{\texttt{dlordinal}: a Python package for deep ordinal classification}
\author[UCO]{Francisco Bérchez-Moreno} %% Author name
\author[IMIBIC]{Rafael Ayllón-Gavilán}
\author[UCO]{Víctor M. Vargas\corref{cor1}}
\author[UCO]{David Guijo-Rubio}
\author[UCO]{César Hervás-Martínez}
\author[UCO]{Juan C. Fernández}
\author[UCO]{Pedro A. Gutiérrez}
%% Author affiliation
\cortext[cor1]{Corresponding author\\ E-mail address: \url{vvargas@uco.es}}
\address[UCO]{Department of Computer Science and Numerical Analysis, University of Córdoba, Córdoba, Spain}
\address[IMIBIC]{Department of Clinical-Epidemiological Research in Primary Care, Instituto Maimónides de Investigación Biomédica de Córdoba, Spain}
%% Abstract
\begin{abstract}
\texttt{dlordinal} is a new Python library that unifies many recent deep ordinal classification methodologies available in the literature. Developed using PyTorch as underlying framework, it implements the top performing state-of-the-art deep learning techniques for ordinal classification problems. Ordinal approaches are designed to leverage the ordering information present in the target variable. Specifically, it includes loss functions, various output layers, dropout techniques, soft labelling methodologies, and other classification strategies, all of which are appropriately designed to incorporate the ordinal information. Furthermore, as the performance metrics to assess novel proposals in ordinal classification depend on the distance between target and predicted classes in the ordinal scale, suitable ordinal evaluation metrics are also included. \texttt{dlordinal} is distributed under the BSD-3-Clause license and is available at \url{https://github.com/ayrna/dlordinal}.
\end{abstract}
%%Research highlights
\begin{highlights}
\item Python package with ordinal classification methodologies for deep learning.
\item Ordinal dropout methodologies, output layers, soft labelling, losses and wrappers.
\item Easy integration with third party packages like \texttt{Skorch}.
\item Extensive test coverage.
\item User-friendly interface with intuitive design and comprehensive tutorials.
\end{highlights}
\begin{keyword}
Deep learning \sep Ordinal classification \sep Ordinal regression \sep Python \sep \texttt{PyTorch} \sep Soft labelling
\end{keyword}
\end{frontmatter}
\section{Introduction}
The term ordinal classification (also known as ordinal regression) refers to supervised learning problems characterised by an inherent order relationship between the class labels \cite{gutierrez2016current, perez2014classification}. In real-world scenarios, numerous problems of this nature arise. Specifically, in the healthcare domain, classification problems in which the categories follow a natural order are quite common \cite{tang2023disease}. One example from this domain is classifying the nature of a tumour lesion into varying degrees of severity \cite{arya2021multi,umamaheswari2024cnn}. In this case, the possible labels might be \textit{benign}, \textit{premalignant} (precancerous), \textit{in situ}, and \textit{malignant} (cancerous). Another example is grading the different levels of diabetic retinopathy \cite{pascual2024multivariate}, a complication of diabetes that affects vision. In this instance, the disease is often classified into stages such as \textit{no DR}, \textit{mild}, \textit{moderate}, \textit{severe}, and \textit{proliferative DR}. These examples illustrate two ordinal classification problems, as they involve distinct degrees of a condition and similarities between adjacent categories. Therefore, the primary objective of ordinal classification is to minimise the magnitude of misclassification errors, i.e. the distance between the target and predicted class on the ordinal scale.
This is accomplished by considering the varying costs associated with different types of errors. Specifically, classification errors between adjacent classes are less significant than errors between distant classes. Thus, in the former example, the error committed when misclassifying a \textit{benign} tumour as \textit{malignant} should be greater than the error incurred when misclassifying it as \textit{premalignant} (precancerous). Therefore, ordinal classification approaches are designed to account for the severity of misclassification errors by incorporating the ordinal nature of the labels into both the learning algorithm and the evaluation metrics.
Some existing packages, such as \texttt{ORCA} \cite{sanchez2019orca}, in Matlab, or \texttt{mord} \cite{pedregosa2015feature}, in Python, implement ordinal classification approaches for machine learning. Moreover, some previous works have released their source code\footnote{\url{https://github.com/rpmcruz/unimodal-ordinal-regression}}, as is the case of some unimodal distributions for deep ordinal classification tasks \cite{cardoso2023unimodal}. Nevertheless, these works do not provide a structured Python package, so they can not be easily installed and used. In this way, to the best of the authors' knowledge, there is not a framework that bridges the gap between ordinal classification and Deep Learning (DL), providing the researchers with a single framework integrating numerous ordinal approaches and other functionalities, useful for developing novel methodologies.
In this paper, we introduce \texttt{dlordinal}, a Python toolkit that implements ordinal classification methodologies for DL developed on top of the open-source \texttt{PyTorch} framework. Our software offers a wide number of DL approaches that leverage ordinal information inherent in the problem domain. The documentation and code of the \texttt{dlordinal} toolkit is available at \url{https://github.com/ayrna/dlordinal}.
\section{Code design and features}
The \texttt{dlordinal} package is designed with a modular architecture, facilitating its use and extension. It perfectly integrates with third-party software such as \texttt{skorch} \cite{skorch} for a wide range of classification tasks. The dependency level is kept to a minimum, relying on essential libraries including \texttt{scikit-learn} \cite{pedregosa2011scikit}, \texttt{numpy} \cite{harris2020array}, \texttt{PyTorch} \cite{paszke2019pytorch}, \texttt{pandas} \cite{mckinney2010data}, \texttt{scipy} \cite{virtanen2020scipy}, \texttt{scikit-image} \cite{van2014scikit}, \texttt{tqdm} \cite{da2019tqdm}, and \texttt{pillow}. Additionally, \texttt{dlordinal} supports downloading and loading common ordinal image benchmark datasets, such as FGNet \cite{fu2015robust} or Adience \cite{eidinger2014age}. This facilitates easy benchmarking of novel ordinal approaches.
\begin{figure}[!ht]
\centering
\includegraphics[width=\linewidth]{modules.pdf}
\caption{Representation of the classes and functions included in the various modules of the \texttt{dlordinal} package.}
\label{fig:modules}
\end{figure}
As mentioned above, the \texttt{dlordinal} package features a modular design (see Figure \ref{fig:modules}), divided into several clearly defined modules:
\begin{itemize}
\item \textbf{Datasets}: this module provides methods for loading the FGNet \cite{panis2016overview} and Adience \cite{eidinger2014age}, two ordinal datasets for benchmarking purposes. These datasets are useful for evaluating the performance of models in facial age estimation, a common ordinal classification task. FGNet offers a comprehensive collection of facial images across different ages, providing a valuable resource for studying facial ageing. Similarly, the Adience dataset includes unfiltered images collected from various sources, making it suitable for robust age and gender estimation. The tutorials included in \texttt{dlordinal} demonstrate how to effectively utilise these datasets for model training and evaluation.
\item \textbf{Dropout}: this module includes a new hybrid dropout methodology \cite{berchez2024fusion} modifying the standard dropout approach \cite{Srivastava2014} to consider the order information of the target. In this way, the conventional dropout technique randomly drops neurons during training to prevent overfitting and improve generalisation, while the ordinal version analyses correlation between neurons and targets to promote those neurons that contribute to a better ordering of the classes.
\item \textbf{Output layers}: \texttt{dlordinal} implements the output layers proposed in the current literature. It includes an implementation of the Cumulative Link Models (CLM) output layer \cite{vargas2020clm}, which uses a projection obtained by an arbitrary deep neural network model. It also includes an implementation of the stick-breaking strategy proposed in \cite{liu2020unimodal}, which computes the class probabilities considering the ordinal information. Lastly, it implements an ordinal fully connected layer based on the Ordinal Binary Decomposition (OBD) approach described in \cite{barbero2023error}, which decompose the ordinal problem in several binary classification problems while preserving the ordinal information.
\item \textbf{Soft labelling}: this module implements several functions for generating soft labels using different probability distributions \cite{vargas2024ebano}. This module forms the basis of the soft labelling strategy and is used in the corresponding loss functions. Soft labelling approaches replace one-hot encoded targets with the objective of regularising model predictions. These labels mitigate the effect of noisy labels \cite{liu2024distributed} in ordinal classification problems and lead to better generalisation performance. The underlying distributions generating these soft labels should be unimodal when dealing with ordinal classification. Specifically, for an ordinal classification problem with a given number of categories, the module includes state-of-the-art methodologies for generating soft labels using several distributions, such as Poisson or binomial \cite{liu2020unimodal}, exponential \cite{vargas2023exponential}, beta \cite{vargas2022unimodal}, and triangular \cite{vargas2023soft, vargas2023generalised}.
\item \textbf{Losses}: several state-of-the-art ordinal loss functions are implemented in \texttt{dlordinal}. For example, regularised categorical cross-entropy loss functions based on the soft labelling strategy, considering any of the functions mentioned in the previous module. The weighted kappa loss function proposed in \cite{de2018weighted} is also included. The latter is an ordinal loss function based on a continuous version of the weighted kappa metric. Moreover, for the OBD approach \cite{barbero2023error}, a specific loss function is also implemented. This loss function considers the error of all the binary problems by computing the Mean Square Error (MSE) loss between the output of the model and the ideal output vector for each class.
% \item \textbf{Wrappers}: this module includes wrappers for the different ordinal methodologies implemented. For example, the wrapper for the OBD model \cite{barbero2023error}, which prepares an arbitrary model architecture to work with the Ordinal Error-Correcting Output Codes (ECOC) distance loss \cite{barbero2023error}.
\item \textbf{Wrappers}: this module provides wrappers that enable the adaptation of any existing methodology to ordinal classification. An example can be found for the method introduced in \cite{barbero2023error}. It integrates any type of architecture into an OBD framework, and enhances the learning process for ordinal problems utilising the Ordinal Error Correcting Output Codes (ECOC) distance loss \cite{barbero2023error}.
\item \textbf{Metrics}: in this module, several ordinal classification metrics are included: the 1-off accuracy, which measures the proportion of predictions that are either correct or off by one category. The Average Mean Absolute Error (AMAE) \cite{baccianella2009evaluation} and the Maximum Mean Absolute Error (MMAE) \cite{cruz2014metrics}. Both are robust measures that consider the imbalance degree present in the problem by averaging mean absolute errors or taking the maximum of the mean absolute errors computed for each class, respectively. The Quadratic Weighted Kappa (QWK), based on the weighted kappa index, aims to quantify the inter-rater agreement on classifying elements into a set of ordered categories. The QWK is specially appropriate to measure how well the model is predicting extreme classes \cite{de2018weighted}. The Ranked Probability Score (RPS), presented in \cite{epstein1969scoring}, measures the discrepancy between the predicted cumulative distribution and the actual cumulative distribution of the ordinal target. Finally, the Geometric Mean of the Sensitivities of the Extreme Classes (GMSEC) measures classification performance by focusing solely on the extreme classes, which are often the most important in certain ordinal problems.
\end{itemize}
\section{Sample code}
An example demonstrating the usage of some of the ordinal elements of the \texttt{dlordinal} toolkit, including a dataset, a loss function and some metrics, is as follows:
\begin{lstlisting}[language=Python]
import numpy as np
from dlordinal.datasets import FGNet
from dlordinal.losses import TriangularCrossEntropyLoss
from dlordinal.metrics import amae, mmae
from skorch import NeuralNetClassifier
from torch import nn
from torch.optim import Adam
from torchvision import models
from torchvision.transforms import Compose, ToTensor
# Download the FGNet dataset
fgnet_train = FGNet(
root="./datasets",
train=True,
target_transform=np.array,
transform=Compose([ToTensor()]),
)
fgnet_test = FGNet(
root="./datasets",
train=False,
target_transform=np.array,
transform=Compose([ToTensor()]),
)
num_classes_fgnet = len(fgnet_train.classes)
# Model
model = models.resnet18(weights="IMAGENET1K_V1")
model.fc = nn.Linear(model.fc.in_features, num_classes_fgnet)
# Loss function
loss_fn = TriangularCrossEntropyLoss(num_classes=num_classes_fgnet)
# Skorch estimator
estimator = NeuralNetClassifier(
module=model,
criterion=loss_fn,
optimizer=Adam,
lr=1e-3,
max_epochs=25,
)
estimator.fit(X=fgnet_train, y=fgnet_train.targets)
train_probs = estimator.predict_proba(fgnet_train)
test_probs = estimator.predict_proba(fgnet_test)
# Metrics
amae_metric = amae(np.array(fgnet_test.targets), test_probs)
mmae_metric = mmae(np.array(fgnet_test.targets), test_probs)
print(f"Test AMAE: {amae_metric}, Test MMAE: {mmae_metric}")
\end{lstlisting}
\rev{
\section{Experiments and results}
This section outlines the experimental settings adopted to provide a benchmark of the main methodologies included in \texttt{dlordinal}.
\subsection{Methodologies}\label{Methodologies}
Given that the ordinal methodologies implemented in \texttt{dlordinal} are designed to be built on top of a base architecture, to enable a fair comparison, the same base architecture is selected. In this case, we consider the ResNet18 \cite{zhao2022deep} model pre-trained with ImageNet \cite{ImageNet8978153}. By default, the model implements a softmax output layer and is optimised using the categorical cross-entropy loss function.
The ordinal methodologies listed in \Cref{tab:benchmarked_methods} have been selected for this benchmark. This table shows the name of the methodology along with its type and location within the \texttt{dlordinal} package.
\begin{table}[!ht]
\centering
\rev{
\caption{Methodologies considered for the experiments.}
\label{tab:benchmarked_methods}
\vspace{0.3cm}
{\renewcommand{\arraystretch}{2.0}%
\resizebox{\linewidth}{!}{
\begin{tabular}{lll}
\toprule \toprule
Methodology & Type & Location(s) \\
\midrule
\betac{} \cite{vargas2022unimodal} & Loss & \texttt{dlordinal.losses.BetaCrossEntropyLoss}\\
\binomc{} \cite{liu2020unimodal} & Loss & \texttt{dlordinal.losses.BinomialCrossEntropyLoss}\\
\clm{} \cite{vargas2020clm} & Output layer & \texttt{dlordinal.output\_layers.CLM}\\
\clmwk{} \cite{vargas2020clm} & \makecell[l]{Output layer\\ + loss} & \makecell[l]{\texttt{dlordinal.output\_layers.CLM} \\ \texttt{dlordinal.losses.WKLoss}}\\
\expc{} \cite{vargas2023exponential} & Loss & \texttt{dlordinal.losses.BinomialCrossEntropyLoss}\\
\drop{} \cite{berchez2024fusion} & Dropout & \makecell[l]{\texttt{dlordinal.dropout.HybridDropout} \\ \texttt{dlordinal.dropout.HybridDropoutContainer}}\\
\ecoc{} \cite{barbero2023error} & \makecell[l]{Wrapper\\ + loss} & \makecell[l]{\texttt{dlordinal.wrappers.OBDECOCModel} \\ \texttt{dlordinal.losses.OrdinalECOCDistanceLoss}}\\
\sbc{} \cite{liu2020unimodal} & Output layer & \texttt{dlordinal.output\_layers.StickBreakingLayer}\\
\triang{} \cite{vargas2023soft} & Loss & \texttt{dlordinal.losses.TriangularCrossEntropyLoss}\\
\bottomrule \bottomrule
\end{tabular}}
}
}
\end{table}
\subsection{Training and evaluation procedure}\label{experimental_setup}
For this benchmark, three well-known datasets are considered: 1) Adience\footnote{\rev{\url{https://talhassner.github.io/home/projects/Adience/Adience-data.html}}} \cite{eidinger2014age}, 2) FGNet\footnote{\rev{\url{https://yanweifu.github.io/FG\_NET\_data/index.html}}} \cite{fu2015robust}, and 3) Diabetic Retinopathy\footnote{\rev{\url{https://www.kaggle.com/c/diabetic-retinopathy-detection/data}}} \cite{wang2018diabetic}.
The cross-validation of hyperparameters is conducted using a randomised search strategy using the values shown in Table \ref{crossvalidation}, with a maximum of $15$ iterations. For those hyperparameters not included in that table, the default values in \texttt{dlordinal} are used. Each hyperparameter configuration follows a $3$-fold strategy. Note that a total of $20$ runs, using a different random seed, are considered for robustness and fairness.
For the training procedure, a batch size of $128$ is used. The models are trained for a maximum of $100$ epochs. In addition, an early stopping strategy is adopted so that the training is terminated if no improvement in the validation loss is observed after $40$ epochs. With respect to the data partitioning, an stratified $75\%-25\%$ train-test split is employed. From the $75\%$ designated for training, an stratified $15\%$ partition is reserved for validation purposes.
To evaluate the different models, four metrics are considered, including two ordinal metrics such as the Quadratic Weighted Kappa (QWK) \cite{de2018weighted} and the Mean Absolute Error (MAE) \cite{ayllon2023dictionary}, the Correct Classification Rate (CCR) \cite{rosati2022novel} as a nominal classification measure, and the execution time. Note that both QWK and CCR are maximisation metrics, whereas the MAE has to be minimised.
\begin{table}[!ht]
\centering
\label{tab:parameters}
\rev{
\caption{Considered range of values for tuning the hyperparameters of each methodology.}
\label{crossvalidation}
\vspace{0.3cm}
\resizebox{\textwidth}{!}{
\begin{tabular}{llc}
\toprule \toprule
Classifier(s) & Hyperparameter description & Range of values \\
\midrule
\multirow{2}{*}{\betac, \binomc} & Learning rate & $\{10^{-4},10^{-3},10^{-2}\}$ \\
& Smoothing factor eta & $\{0.8,1.0\}$ \\ [0.3cm]
\sbc, \ecoc, \drop & Learning rate & $\{10^{-4},10^{-3},10^{-2}\}$ \\ [0.3cm]
\multirow{3}{*}{\expc} & Learning rate & $\{10^{-4},10^{-3},10^{-2}\}$ \\
& Exponent & $\{1.0, 1.5, 2.0\}$ \\
& Smoothing factor eta & $\{0.8,1.0\}$ \\ [0.3cm]
\multirow{3}{*}{\triang} & Learning rate & $\{10^{-4},10^{-3},10^{-2}\}$ \\
& Adjacent class probability & $\{0.01, 0.05, 0.10\}$ \\
& Smoothing factor eta & $\{0.8,1.0\}$ \\ [0.3cm]
\multirow{1}{*}{\clm} & Learning rate & $\{10^{-4},10^{-3},10^{-2}\}$ \\[0.3cm]
\multirow{1}{*}{\clmwk} & Learning rate & $\{10^{-4},10^{-2},10^{-3}\}$ \\[0.3cm]
\bottomrule \bottomrule
\end{tabular}
}
}
\end{table}
\subsection{Results}
This section provides the results obtained from the experiments detailed in Section \ref{experimental_setup}. Table \ref{table:resultados} shows the results for QWK, MAE, CCR, and computational time, as the Mean and Standard Deviation (SD) of the $20$ runs (Mean\textsubscript{SD}). For the Adience dataset, \triang{} achieves excellent results in MAE ($0.335$) and CCR ($0.753$), whereas \betac{} is the best in QWK ($0.924$). On the other hand, the \drop{} method is the fastest approach ($74.01$ seconds), achieving competitive results. For the FGNet dataset, \clmwk{} achieves the best results in terms of QWK ($0.846$), while the \triang{} estimator again outperforms the rest of the approaches in MAE ($0.491$) and CCR ($0.594$). Similarly, \drop{} is the most efficient, with an execution time of $10.69$ seconds. For the Diabetic Retinopathy dataset, \clmwk{} emerges as the top-performing method for all the metrics: QWK ($0.624$), MAE ($0.415$) and CCR ($0.687$). In this case, \ecoc{} achieves the shortest execution time ($154.90$ seconds). Overall, these results suggest that performance varies depending on the characteristics of the dataset and the metrics considered (including the one used in the cross-validation stage), with methods such as \betac{}, \triang{} and \clmwk{} achieving strong classification performance. Meanwhile, the \drop{} method consistently stands out in terms of computational time across all datasets, except for Diabetic Retinopathy, where it is outperformed by \ecoc{}.
Overall, these results highlight the effectiveness of the proposed approaches in leveraging ordinal information while offering a trade-off between performance and computational efficiency.
\begin{table}[ht!]
\centering
\rev{
\caption{Results for each metric in the test datasets, expressed as the Mean and Standard Deviation (SD) for the 20 runs, Mean\textsubscript{SD}. The best result for each metric is highlighted in \textbf{bold}, whereas the second best is in \textit{italics}.}
\label{table:resultados}
\vspace{0.3cm}
\resizebox{\textwidth}{!}{
\begin{tabular}{llcccc}
\toprule \toprule
Dataset & Estimator & QWK & MAE & CCR & Time (seg) \\
\midrule
\multirow{9}{*}{Adience} & \betac & \textbf{0.924}$_{0.016}$ & \textit{0.349}$_{0.080}$ & \textit{0.730}$_{0.060}$ & 289.68$_{117.22}$ \\
& \binomc & 0.832$_{0.086}$ & 0.773$_{0.170}$ & 0.442$_{0.052}$ & 353.85$_{82.72}$ \\
& \clm & 0.902$_{0.019}$ & 0.437$_{0.075}$ & 0.670$_{0.055}$ & 1587.81$_{322.26}$ \\
& \clmwk & 0.902$_{0.029}$ & 0.484$_{0.128}$ & 0.617$_{0.087}$ & 1090.42$_{768.00}$ \\
& \expc & 0.918$_{0.026}$ & 0.391$_{0.109}$ & 0.694$_{0.076}$ & 263.45$_{44.55}$ \\
& \drop & 0.812$_{0.037}$ & 0.740$_{0.130}$ & 0.525$_{0.057}$ & \textbf{74.01}$_{20.09}$ \\
& \ecoc & 0.916$_{0.015}$ & 0.376$_{0.077}$ & 0.722$_{0.056}$ & 1009.82$_{704.51}$ \\
%& \poisson & 0.578$_{0.127}$ & 0.533$_{0.065}$ & 0.387$_{0.086}$ & 589.55$_{362.75}$ \\
& \sbc & 0.884$_{0.032}$ & 0.441$_{0.105}$ & 0.701$_{0.065}$ & 874.35$_{730.67}$ \\
& \triang & \textit{0.923}$_{0.019}$ & \textbf{0.335}$_{0.085}$ & \textbf{0.753}$_{0.058}$ & \textit{249.47}$_{140.76}$ \\
%& \wk & 0.619$_{0.211}$ & 0.741$_{0.133}$ & 0.418$_{0.165}$ & 1070.65$_{862.46}$ \\
\midrule
\multirow{9}{*}{FGNet} & \betac & \textit{0.839}$_{0.034}$ & \textit{0.510}$_{0.088}$ & \textit{0.564}$_{0.050}$ & 47.56$_{28.45}$ \\
& \binomc & 0.824$_{0.030}$ & 0.614$_{0.072}$ & 0.468$_{0.054}$ & 49.74$_{22.83}$ \\
& \clm & 0.793$_{0.057}$ & 0.592$_{0.132}$ & 0.514$_{0.071}$ & 65.70$_{22.91}$ \\
& \clmwk & \textbf{0.846}$_{0.027}$ & 0.534$_{0.090}$ & 0.520$_{0.076}$ & 66.49$_{26.48}$ \\
& \expc & 0.829$_{0.046}$ & 0.551$_{0.081}$ & 0.523$_{0.052}$ & 52.49$_{31.08}$ \\
& \drop & 0.706$_{0.160}$ & 0.763$_{0.177}$ & 0.463$_{0.055}$ & \textbf{10.69}$_{2.90}$ \\
& \ecoc & 0.760$_{0.058}$ & 0.639$_{0.104}$ & 0.501$_{0.067}$ & \textit{20.03}$_{14.94}$ \\
%& \poisson & 0.216$_{0.052}$ & 0.332$_{0.030}$ & 0.151$_{0.018}$ & 76.62$_{21.16}$ \\
& \sbc & 0.760$_{0.076}$ & 0.603$_{0.107}$ & 0.517$_{0.063}$ & 29.31$_{21.61}$ \\
& \triang & 0.828$_{0.036}$ & \textbf{0.491}$_{0.057}$ & \textbf{0.594}$_{0.037}$ & 29.09$_{20.44}$ \\
%& \wk & 0.526$_{0.214}$ & 0.780$_{0.098}$ & 0.378$_{0.106}$ & 37.18$_{21.22}$ \\
\midrule
\multirow{9}{*}{Retinopathy} & \betac & 0.374$_{0.038}$ & 1.046$_{0.087}$ & 0.094$_{0.006}$ & \textit{232.64}$_{61.20}$ \\
& \binomc & 0.387$_{0.048}$ & 1.049$_{0.110}$ & 0.089$_{0.006}$ & 356.60$_{538.93}$ \\
& \clm & \textit{0.541}$_{0.058}$ & \textit{0.528}$_{0.132}$ & \textit{0.634}$_{0.096}$ & 2904.94$_{2500.61}$ \\
& \clmwk & \textbf{0.624}$_{0.023}$ & \textbf{0.415}$_{0.038}$ & \textbf{0.687}$_{0.039}$ & 2979.26$_{1374.41}$ \\
& \expc & 0.408$_{0.041}$ & 1.004$_{0.082}$ & 0.092$_{0.005}$ & 1206.61$_{732.51}$ \\
& \drop & 0.428$_{0.084}$ & 0.793$_{0.224}$ & 0.487$_{0.133}$ & 233.98$_{102.20}$ \\
& \ecoc & 0.282$_{0.181}$ & 1.348$_{0.678}$ & 0.220$_{0.117}$ & \textbf{154.90}$_{78.84}$ \\
%& \poisson & -0.003$_{0.015}$ & 0.103$_{0.042}$ & 0.030$_{0.002}$ & 1056.02$_{817.97}$ \\
& \sbc & 0.447$_{0.133}$ & 0.688$_{0.200}$ & 0.554$_{0.116}$ & 269.33$_{79.49}$ \\
& \triang & 0.427$_{0.071}$ & 0.815$_{0.191}$ & 0.414$_{0.145}$ & 250.42$_{140.19}$ \\
%& \wk & 0.262$_{0.237}$ & 0.699$_{0.190}$ & 0.378$_{0.276}$ & 1437.09$_{993.63}$ \\
\bottomrule \bottomrule
\end{tabular}
}
}
\end{table}
}
\section{Conclusions}
\texttt{dlordinal} has been developed to bring the practitioners with a Python package including deep ordinal classification methodologies proposed in recent literature. It is built on top of the well-known \texttt{PyTorch} framework. Up-to-the-knowledge of the authors, there are no previous software packages unifying all deep ordinal classification approaches under a single framework. Consequently, the scientific community should benefit from using \texttt{dlordinal} to develop and test novel methodologies and to apply existing ones to real-world problems with minimal effort.
\section*{Acknowledgments}
The present work has been supported by the ``Agencia Estatal de Investigación (España)'', Spanish Ministry of Research and Innovation (grant ref.: PID2023-150663NB-C22 / AEI / 10.13039 / 501100011033), by the European Commission, project Test and Experiment Facilities for the Agri-Food Domain, AgriFoodTEF (grant ref.: DIGITAL-2022-CLOUD-AI-02, 101100622), and by the ENIA International Chair in Agriculture, University of Córdoba (grant ref.: TSI-100921-2023-3), funded by the Secretary of State for Digitalisation and Artificial Intelligence and by the European Union - Next Generation EU. Recovery, Transformation and Resilience Plan and by the University of Córdoba through competitive grants for Andalusian society challenges (grant ref.: PP2F\_L1\_15). F. Bérchez-Moreno has been supported by ``Plan Propio de Investigación Submodalidad 2.2 Contratos predoctorales'' of the University of Córdoba. R. Ayllón-Gavilán has been supported by the ``Instituto de Salud Carlos III'' (ISCIII) and EU (grant ref.: FI23/00163).
\bibliographystyle{elsarticle-harv}
\bibliography{bibliography}
\end{document}
\endinput
%%
%% End of file `elsarticle-template-num.tex'.
\begin{thebibliography}{40}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\href}[2]{#2}
\providecommand{\path}[1]{#1}
\providecommand{\DOIprefix}{doi:}
\providecommand{\ArXivprefix}{arXiv:}
\providecommand{\URLprefix}{URL: }
\providecommand{\Pubmedprefix}{pmid:}
\providecommand{\doi}[1]{\href{http://dx.doi.org/#1}{\path{#1}}}
\providecommand{\Pubmed}[1]{\href{pmid:#1}{\path{#1}}}
\providecommand{\bibinfo}[2]{#2}
\ifx\xfnm\relax \def\xfnm[#1]{\unskip,\space#1}\fi
%Type = Article
\bibitem[{Arya and Saha(2021)}]{arya2021multi}
\bibinfo{author}{Arya, N.}, \bibinfo{author}{Saha, S.}, \bibinfo{year}{2021}.
\newblock \bibinfo{title}{Multi-modal advanced deep learning architectures for
breast cancer survival prediction}.
\newblock \bibinfo{journal}{Knowledge-Based Systems} \bibinfo{volume}{221},
\bibinfo{pages}{106965}.
\newblock \DOIprefix\doi{10.1016/j.knosys.2021.106965}.
%Type = Inproceedings
\bibitem[{Ayll{\'o}n-Gavil{\'a}n et~al.(2023)Ayll{\'o}n-Gavil{\'a}n,
Guijo-Rubio, Guti{\'e}rrez and
Herv{\'a}s-Mart{\'\i}nez}]{ayllon2023dictionary}
\bibinfo{author}{Ayll{\'o}n-Gavil{\'a}n, R.}, \bibinfo{author}{Guijo-Rubio,
D.}, \bibinfo{author}{Guti{\'e}rrez, P.A.},
\bibinfo{author}{Herv{\'a}s-Mart{\'\i}nez, C.}, \bibinfo{year}{2023}.
\newblock \bibinfo{title}{A dictionary-based approach to time series ordinal
classification}, in: \bibinfo{booktitle}{International Work-Conference on
Artificial Neural Networks}, \bibinfo{organization}{Springer}. pp.
\bibinfo{pages}{541--552}.
%Type = Inproceedings
\bibitem[{Baccianella et~al.(2009)Baccianella, Esuli and
Sebastiani}]{baccianella2009evaluation}
\bibinfo{author}{Baccianella, S.}, \bibinfo{author}{Esuli, A.},
\bibinfo{author}{Sebastiani, F.}, \bibinfo{year}{2009}.
\newblock \bibinfo{title}{Evaluation measures for ordinal regression}, in:
\bibinfo{booktitle}{2009 Ninth international conference on intelligent
systems design and applications}, \bibinfo{organization}{IEEE}. pp.
\bibinfo{pages}{283--287}.
\newblock \DOIprefix\doi{10.1109/ISDA.2009.230}.
%Type = Article
\bibitem[{Barbero-G{\'o}mez et~al.(2023)Barbero-G{\'o}mez, Guti{\'e}rrez and
Herv{\'a}s-Mart{\'\i}nez}]{barbero2023error}
\bibinfo{author}{Barbero-G{\'o}mez, J.}, \bibinfo{author}{Guti{\'e}rrez, P.A.},
\bibinfo{author}{Herv{\'a}s-Mart{\'\i}nez, C.}, \bibinfo{year}{2023}.
\newblock \bibinfo{title}{Error-correcting output codes in the framework of
deep ordinal classification}.
\newblock \bibinfo{journal}{Neural Processing Letters} \bibinfo{volume}{55},
\bibinfo{pages}{5299--5330}.
\newblock \DOIprefix\doi{10.1007/s11063-022-10824-7}.
%Type = Article
\bibitem[{B{\'e}rchez-Moreno et~al.(2024)B{\'e}rchez-Moreno, Fern{\'a}ndez,
Herv{\'a}s-Mart{\'\i}nez and Guti{\'e}rrez}]{berchez2024fusion}
\bibinfo{author}{B{\'e}rchez-Moreno, F.}, \bibinfo{author}{Fern{\'a}ndez,
J.C.}, \bibinfo{author}{Herv{\'a}s-Mart{\'\i}nez, C.},
\bibinfo{author}{Guti{\'e}rrez, P.A.}, \bibinfo{year}{2024}.
\newblock \bibinfo{title}{Fusion of standard and ordinal dropout techniques to
regularise deep models}.
\newblock \bibinfo{journal}{Information Fusion} \bibinfo{volume}{106},
\bibinfo{pages}{102299}.
\newblock \DOIprefix\doi{10.1016/j.inffus.2024.102299}.
%Type = Article
\bibitem[{Cardoso et~al.(2023)Cardoso, Cruz and
Albuquerque}]{cardoso2023unimodal}
\bibinfo{author}{Cardoso, J.S.}, \bibinfo{author}{Cruz, R.},
\bibinfo{author}{Albuquerque, T.}, \bibinfo{year}{2023}.
\newblock \bibinfo{title}{Unimodal distributions for ordinal regression}.
\newblock \bibinfo{journal}{arXiv preprint arXiv:2303.04547} .
%Type = Article
\bibitem[{da~Costa-Luis(2019)}]{da2019tqdm}
\bibinfo{author}{da~Costa-Luis, C.O.}, \bibinfo{year}{2019}.
\newblock \bibinfo{title}{tqdm: A fast, extensible progress meter for python
and cli}.
\newblock \bibinfo{journal}{Journal of Open Source Software}
\bibinfo{volume}{4}, \bibinfo{pages}{1277}.
\newblock \DOIprefix\doi{10.21105/joss.01277}.
%Type = Article
\bibitem[{Cruz-Ram{\'\i}rez et~al.(2014)Cruz-Ram{\'\i}rez,
Herv{\'a}s-Mart{\'\i}nez, S{\'a}nchez-Monedero and
Guti{\'e}rrez}]{cruz2014metrics}
\bibinfo{author}{Cruz-Ram{\'\i}rez, M.},
\bibinfo{author}{Herv{\'a}s-Mart{\'\i}nez, C.},
\bibinfo{author}{S{\'a}nchez-Monedero, J.}, \bibinfo{author}{Guti{\'e}rrez,
P.A.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Metrics to guide a multi-objective evolutionary
algorithm for ordinal classification}.
\newblock \bibinfo{journal}{Neurocomputing} \bibinfo{volume}{135},
\bibinfo{pages}{21--31}.
\newblock \DOIprefix\doi{10.1016/j.neucom.2013.05.058}.
%Type = Article
\bibitem[{Eidinger et~al.(2014)Eidinger, Enbar and Hassner}]{eidinger2014age}
\bibinfo{author}{Eidinger, E.}, \bibinfo{author}{Enbar, R.},
\bibinfo{author}{Hassner, T.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Age and gender estimation of unfiltered faces}.
\newblock \bibinfo{journal}{IEEE Transactions on information forensics and
security} \bibinfo{volume}{9}, \bibinfo{pages}{2170--2179}.
\newblock \DOIprefix\doi{10.1109/TIFS.2014.2359646}.
%Type = Article
\bibitem[{Epstein(1969)}]{epstein1969scoring}
\bibinfo{author}{Epstein, E.S.}, \bibinfo{year}{1969}.
\newblock \bibinfo{title}{A scoring system for probability forecasts of ranked
categories}.
\newblock \bibinfo{journal}{Journal of Applied Meteorology (1962-1982)}
\bibinfo{volume}{8}, \bibinfo{pages}{985--987}.
\newblock \DOIprefix\doi{10.1175/1520-0450(1969)008<0985:ASSFPF>2.0.CO;2}.
%Type = Article
\bibitem[{Fu et~al.(2015)Fu, Hospedales, Xiang, Xiong, Gong, Wang and
Yao}]{fu2015robust}
\bibinfo{author}{Fu, Y.}, \bibinfo{author}{Hospedales, T.M.},
\bibinfo{author}{Xiang, T.}, \bibinfo{author}{Xiong, J.},
\bibinfo{author}{Gong, S.}, \bibinfo{author}{Wang, Y.}, \bibinfo{author}{Yao,
Y.}, \bibinfo{year}{2015}.
\newblock \bibinfo{title}{Robust subjective visual property prediction from
crowdsourced pairwise labels}.
\newblock \bibinfo{journal}{IEEE transactions on pattern analysis and machine
intelligence} \bibinfo{volume}{38}, \bibinfo{pages}{563--577}.
\newblock \DOIprefix\doi{10.1109/TPAMI.2015.2456887}.
%Type = Article
\bibitem[{Guti{\'e}rrez and Garc{\'\i}a(2016)}]{gutierrez2016current}
\bibinfo{author}{Guti{\'e}rrez, P.A.}, \bibinfo{author}{Garc{\'\i}a, S.},
\bibinfo{year}{2016}.
\newblock \bibinfo{title}{Current prospects on ordinal and monotonic
classification}.
\newblock \bibinfo{journal}{Progress in Artificial Intelligence}
\bibinfo{volume}{5}, \bibinfo{pages}{171--179}.
\newblock \DOIprefix\doi{10.1007/s13748-016-0088-y}.
%Type = Article
\bibitem[{Harris et~al.(2020)Harris, Millman, Van Der~Walt, Gommers, Virtanen,
Cournapeau, Wieser, Taylor, Berg, Smith et~al.}]{harris2020array}
\bibinfo{author}{Harris, C.R.}, \bibinfo{author}{Millman, K.J.},
\bibinfo{author}{Van Der~Walt, S.J.}, \bibinfo{author}{Gommers, R.},
\bibinfo{author}{Virtanen, P.}, \bibinfo{author}{Cournapeau, D.},
\bibinfo{author}{Wieser, E.}, \bibinfo{author}{Taylor, J.},
\bibinfo{author}{Berg, S.}, \bibinfo{author}{Smith, N.J.}, et~al.,
\bibinfo{year}{2020}.
\newblock \bibinfo{title}{Array programming with numpy}.
\newblock \bibinfo{journal}{Nature} \bibinfo{volume}{585},
\bibinfo{pages}{357--362}.
\newblock \DOIprefix\doi{10.1038/s41586-020-2649-2}.
%Type = Article
\bibitem[{de~La~Torre et~al.(2018)de~La~Torre, Puig and Valls}]{de2018weighted}
\bibinfo{author}{de~La~Torre, J.}, \bibinfo{author}{Puig, D.},
\bibinfo{author}{Valls, A.}, \bibinfo{year}{2018}.
\newblock \bibinfo{title}{Weighted kappa loss function for multi-class
classification of ordinal data in deep learning}.
\newblock \bibinfo{journal}{Pattern Recognition Letters} \bibinfo{volume}{105},
\bibinfo{pages}{144--154}.
\newblock \DOIprefix\doi{10.1016/j.patrec.2017.05.018}.
%Type = Article
\bibitem[{Liu et~al.(2024)Liu, Tu, Gao and Li}]{liu2024distributed}
\bibinfo{author}{Liu, H.}, \bibinfo{author}{Tu, J.}, \bibinfo{author}{Gao, A.},
\bibinfo{author}{Li, C.}, \bibinfo{year}{2024}.
\newblock \bibinfo{title}{Distributed robust support vector ordinal regression
under label noise}.
\newblock \bibinfo{journal}{Neurocomputing} ,
\bibinfo{pages}{128057}\DOIprefix\doi{10.1016/j.neucom.2024.128057}.
%Type = Article
\bibitem[{Liu et~al.(2020)Liu, Fan, Kong, Diao, Xie, Lu and
You}]{liu2020unimodal}
\bibinfo{author}{Liu, X.}, \bibinfo{author}{Fan, F.}, \bibinfo{author}{Kong,
L.}, \bibinfo{author}{Diao, Z.}, \bibinfo{author}{Xie, W.},
\bibinfo{author}{Lu, J.}, \bibinfo{author}{You, J.}, \bibinfo{year}{2020}.
\newblock \bibinfo{title}{Unimodal regularized neuron stick-breaking for
ordinal classification}.
\newblock \bibinfo{journal}{Neurocomputing} \bibinfo{volume}{388},
\bibinfo{pages}{34--44}.
\newblock \DOIprefix\doi{10.1016/j.neucom.2020.01.025}.
%Type = Inproceedings
\bibitem[{McKinney et~al.(2010)}]{mckinney2010data}
\bibinfo{author}{McKinney, W.}, et~al., \bibinfo{year}{2010}.
\newblock \bibinfo{title}{Data structures for statistical computing in
python.}, in: \bibinfo{booktitle}{SciPy}, pp. \bibinfo{pages}{51--56}.
%Type = Article
\bibitem[{Panis et~al.(2016)Panis, Lanitis, Tsapatsoulis and
Cootes}]{panis2016overview}
\bibinfo{author}{Panis, G.}, \bibinfo{author}{Lanitis, A.},
\bibinfo{author}{Tsapatsoulis, N.}, \bibinfo{author}{Cootes, T.F.},
\bibinfo{year}{2016}.
\newblock \bibinfo{title}{Overview of research on facial ageing using the
fg-net ageing database}.
\newblock \bibinfo{journal}{Iet Biometrics} \bibinfo{volume}{5},
\bibinfo{pages}{37--46}.
\newblock \DOIprefix\doi{10.1007/978-3-319-16181-5_56}.
%Type = Article
\bibitem[{Pascual-Fontanilles et~al.(2024)Pascual-Fontanilles, Valls and
Romero-Aroca}]{pascual2024multivariate}
\bibinfo{author}{Pascual-Fontanilles, J.}, \bibinfo{author}{Valls, A.},
\bibinfo{author}{Romero-Aroca, P.}, \bibinfo{year}{2024}.
\newblock \bibinfo{title}{Multivariate data binning and examples generation to
build a diabetic retinopathy classifier based on temporal clinical and
analytical risk factors}.
\newblock \bibinfo{journal}{Knowledge-Based Systems} \bibinfo{volume}{300},
\bibinfo{pages}{112154}.
\newblock \DOIprefix\doi{10.1016/j.knosys.2024.112154}.
%Type = Article
\bibitem[{Paszke et~al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan,
Killeen, Lin, Gimelshein, Antiga et~al.}]{paszke2019pytorch}
\bibinfo{author}{Paszke, A.}, \bibinfo{author}{Gross, S.},
\bibinfo{author}{Massa, F.}, \bibinfo{author}{Lerer, A.},
\bibinfo{author}{Bradbury, J.}, \bibinfo{author}{Chanan, G.},
\bibinfo{author}{Killeen, T.}, \bibinfo{author}{Lin, Z.},
\bibinfo{author}{Gimelshein, N.}, \bibinfo{author}{Antiga, L.}, et~al.,
\bibinfo{year}{2019}.
\newblock \bibinfo{title}{Pytorch: An imperative style, high-performance deep
learning library}.
\newblock \bibinfo{journal}{Advances in neural information processing systems}
\bibinfo{volume}{32}.
%Type = Article
\bibitem[{Pedregosa et~al.(2011)Pedregosa, Varoquaux, Gramfort, Michel,
Thirion, Grisel, Blondel, Prettenhofer, Weiss, Dubourg
et~al.}]{pedregosa2011scikit}
\bibinfo{author}{Pedregosa, F.}, \bibinfo{author}{Varoquaux, G.},
\bibinfo{author}{Gramfort, A.}, \bibinfo{author}{Michel, V.},
\bibinfo{author}{Thirion, B.}, \bibinfo{author}{Grisel, O.},
\bibinfo{author}{Blondel, M.}, \bibinfo{author}{Prettenhofer, P.},
\bibinfo{author}{Weiss, R.}, \bibinfo{author}{Dubourg, V.}, et~al.,
\bibinfo{year}{2011}.
\newblock \bibinfo{title}{Scikit-learn: Machine learning in python}.
\newblock \bibinfo{journal}{the Journal of machine Learning research}
\bibinfo{volume}{12}, \bibinfo{pages}{2825--2830}.
%Type = Phdthesis
\bibitem[{Pedregosa-Izquierdo(2015)}]{pedregosa2015feature}
\bibinfo{author}{Pedregosa-Izquierdo, F.}, \bibinfo{year}{2015}.
\newblock \bibinfo{title}{Feature extraction and supervised learning on fMRI:
from practice to theory}.
\newblock Ph.D. thesis. Universit{\'e} Pierre et Marie Curie-Paris VI.
%Type = Article
\bibitem[{P{\'e}rez-Ortiz et~al.(2014)P{\'e}rez-Ortiz, de~La~Paz-Mar{\'\i}n,
Guti{\'e}rrez and Herv{\'a}s-Mart{\'\i}nez}]{perez2014classification}
\bibinfo{author}{P{\'e}rez-Ortiz, M.}, \bibinfo{author}{de~La~Paz-Mar{\'\i}n,
M.}, \bibinfo{author}{Guti{\'e}rrez, P.A.},
\bibinfo{author}{Herv{\'a}s-Mart{\'\i}nez, C.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Classification of eu countries’ progress towards
sustainable development based on ordinal regression techniques}.
\newblock \bibinfo{journal}{Knowledge-Based Systems} \bibinfo{volume}{66},
\bibinfo{pages}{178--189}.
\newblock \DOIprefix\doi{10.1016/j.knosys.2014.04.041}.
%Type = Article
\bibitem[{Rosati et~al.(2022)Rosati, Romeo, Vargas, Guti{\'e}rrez,
Herv{\'a}s-Mart{\'\i}nez and Frontoni}]{rosati2022novel}
\bibinfo{author}{Rosati, R.}, \bibinfo{author}{Romeo, L.},
\bibinfo{author}{Vargas, V.M.}, \bibinfo{author}{Guti{\'e}rrez, P.A.},
\bibinfo{author}{Herv{\'a}s-Mart{\'\i}nez, C.}, \bibinfo{author}{Frontoni,
E.}, \bibinfo{year}{2022}.
\newblock \bibinfo{title}{A novel deep ordinal classification approach for
aesthetic quality control classification}.
\newblock \bibinfo{journal}{Neural Computing and Applications} ,
\bibinfo{pages}{1--15}\DOIprefix\doi{10.1007/s00521-022-07050-6}.
%Type = Article
\bibitem[{S{\'a}nchez-Monedero et~al.(2019)S{\'a}nchez-Monedero, Guti{\'e}rrez
and P{\'e}rez-Ortiz}]{sanchez2019orca}
\bibinfo{author}{S{\'a}nchez-Monedero, J.}, \bibinfo{author}{Guti{\'e}rrez,
P.A.}, \bibinfo{author}{P{\'e}rez-Ortiz, M.}, \bibinfo{year}{2019}.
\newblock \bibinfo{title}{Orca: A matlab/octave toolbox for ordinal
regression}.
\newblock \bibinfo{journal}{Journal of Machine Learning Research}
\bibinfo{volume}{20}, \bibinfo{pages}{1--5}.
%Type = Article
\bibitem[{Srivastava et~al.(2014)Srivastava, Hinton, Krizhevsky, Sutskever and
Salakhutdinov}]{Srivastava2014}
\bibinfo{author}{Srivastava, N.}, \bibinfo{author}{Hinton, G.},
\bibinfo{author}{Krizhevsky, A.}, \bibinfo{author}{Sutskever, I.},
\bibinfo{author}{Salakhutdinov, R.}, \bibinfo{year}{2014}.
\newblock \bibinfo{title}{Dropout: A simple way to prevent neural networks from
overfitting}.
\newblock \bibinfo{journal}{Journal of Machine Learning Research}
\bibinfo{volume}{15}, \bibinfo{pages}{1929--1958}.
%Type = Inproceedings
\bibitem[{Studer et~al.(2019)Studer, Alberti, Pondenkandath, Goktepe, Kolonko,
Fischer, Liwicki and Ingold}]{ImageNet8978153}
\bibinfo{author}{Studer, L.}, \bibinfo{author}{Alberti, M.},
\bibinfo{author}{Pondenkandath, V.}, \bibinfo{author}{Goktepe, P.},
\bibinfo{author}{Kolonko, T.}, \bibinfo{author}{Fischer, A.},
\bibinfo{author}{Liwicki, M.}, \bibinfo{author}{Ingold, R.},
\bibinfo{year}{2019}.
\newblock \bibinfo{title}{A comprehensive study of imagenet pre-training for
historical document image analysis}, in: \bibinfo{booktitle}{2019
International Conference on Document Analysis and Recognition (ICDAR)}, pp.
\bibinfo{pages}{720--725}.
\newblock \DOIprefix\doi{10.1109/ICDAR.2019.00120}.
%Type = Article
\bibitem[{Tang et~al.(2023)Tang, Yang and Song}]{tang2023disease}
\bibinfo{author}{Tang, W.}, \bibinfo{author}{Yang, Z.}, \bibinfo{author}{Song,
Y.}, \bibinfo{year}{2023}.
\newblock \bibinfo{title}{Disease-grading networks with ordinal regularization
for medical imaging}.
\newblock \bibinfo{journal}{Neurocomputing} \bibinfo{volume}{545},
\bibinfo{pages}{126245}.
\newblock \DOIprefix\doi{10.1016/j.neucom.2023.126245}.
%Type = Manual
\bibitem[{Tietz et~al.(2017)Tietz, Fan, Nouri, Bossan and {skorch
Developers}}]{skorch}
\bibinfo{author}{Tietz, M.}, \bibinfo{author}{Fan, T.J.},
\bibinfo{author}{Nouri, D.}, \bibinfo{author}{Bossan, B.},
\bibinfo{author}{{skorch Developers}}, \bibinfo{year}{2017}.
\newblock \bibinfo{title}{skorch: A scikit-learn compatible neural network
library that wraps PyTorch}.
\newblock \URLprefix \url{https://skorch.readthedocs.io/en/stable/}.
%Type = Article
\bibitem[{Umamaheswari and Mohanbabu(2024)}]{umamaheswari2024cnn}
\bibinfo{author}{Umamaheswari, T.}, \bibinfo{author}{Mohanbabu, Y.M.},
\bibinfo{year}{2024}.
\newblock \bibinfo{title}{Cnn-fs-ifuzzy: A new enhanced learning model enabled
by adaptive tumor segmentation for breast cancer diagnosis using 3d mammogram
images}.
\newblock \bibinfo{journal}{Knowledge-Based Systems} \bibinfo{volume}{288},
\bibinfo{pages}{111443}.
\newblock \DOIprefix\doi{10.1016/j.knosys.2024.111443}.
%Type = Article
\bibitem[{Vargas et~al.(2023a)Vargas, Dur{\'a}n-Rosal, Guijo-Rubio,
Guti{\'e}rrez and Herv{\'a}s-Mart{\'\i}nez}]{vargas2023generalised}
\bibinfo{author}{Vargas, V.M.}, \bibinfo{author}{Dur{\'a}n-Rosal, A.M.},
\bibinfo{author}{Guijo-Rubio, D.}, \bibinfo{author}{Guti{\'e}rrez, P.A.},
\bibinfo{author}{Herv{\'a}s-Mart{\'\i}nez, C.}, \bibinfo{year}{2023}a.
\newblock \bibinfo{title}{Generalised triangular distributions for ordinal deep
learning: Novel proposal and optimisation}.
\newblock \bibinfo{journal}{Information Sciences} \bibinfo{volume}{648},
\bibinfo{pages}{119606}.
\newblock \DOIprefix\doi{10.1016/j.ins.2023.119606}.
%Type = Article
\bibitem[{Vargas et~al.(2024)Vargas, G{\'o}mez-Orellana, Guti{\'e}rrez,
Herv{\'a}s-Mart{\'\i}nez and Guijo-Rubio}]{vargas2024ebano}
\bibinfo{author}{Vargas, V.M.}, \bibinfo{author}{G{\'o}mez-Orellana, A.M.},
\bibinfo{author}{Guti{\'e}rrez, P.A.},
\bibinfo{author}{Herv{\'a}s-Mart{\'\i}nez, C.}, \bibinfo{author}{Guijo-Rubio,
D.}, \bibinfo{year}{2024}.
\newblock \bibinfo{title}{Ebano: A novel ensemble based on unimodal ordinal
classifiers for the prediction of significant wave height}.
\newblock \bibinfo{journal}{Knowledge-Based Systems} \bibinfo{volume}{300},
\bibinfo{pages}{112223}.
\newblock \DOIprefix\doi{10.1016/j.knosys.2024.112223}.
%Type = Article
\bibitem[{Vargas et~al.(2023b)Vargas, Guti{\'e}rrez, Barbero-G{\'o}mez and
Herv{\'a}s-Martínez}]{vargas2023soft}
\bibinfo{author}{Vargas, V.M.}, \bibinfo{author}{Guti{\'e}rrez, P.A.},
\bibinfo{author}{Barbero-G{\'o}mez, J.},
\bibinfo{author}{Herv{\'a}s-Martínez, C.}, \bibinfo{year}{2023}b.
\newblock \bibinfo{title}{Soft labelling based on triangular distributions for
ordinal classification}.
\newblock \bibinfo{journal}{Information Fusion} \bibinfo{volume}{93},
\bibinfo{pages}{258--267}.
\newblock \DOIprefix\doi{10.1016/j.inffus.2023.01.003}.
%Type = Article
\bibitem[{Vargas et~al.(2022)Vargas, Guti{\'e}rrez and
Herv{\'a}s-Mart{\'\i}nez}]{vargas2022unimodal}
\bibinfo{author}{Vargas, V.M.}, \bibinfo{author}{Guti{\'e}rrez, P.A.},
\bibinfo{author}{Herv{\'a}s-Mart{\'\i}nez, C.}, \bibinfo{year}{2022}.
\newblock \bibinfo{title}{Unimodal regularisation based on beta distribution
for deep ordinal regression}.
\newblock \bibinfo{journal}{Pattern Recognition} \bibinfo{volume}{122},
\bibinfo{pages}{1--10}.
\newblock \DOIprefix\doi{10.1016/j.patcog.2021.108310}.
%Type = Article
\bibitem[{Vargas et~al.(2023c)Vargas, Guti{\'e}rrez, Rosati, Romeo, Frontoni
and Herv{\'a}s-Mart{\'\i}nez}]{vargas2023exponential}
\bibinfo{author}{Vargas, V.M.}, \bibinfo{author}{Guti{\'e}rrez, P.A.},
\bibinfo{author}{Rosati, R.}, \bibinfo{author}{Romeo, L.},
\bibinfo{author}{Frontoni, E.}, \bibinfo{author}{Herv{\'a}s-Mart{\'\i}nez,
C.}, \bibinfo{year}{2023}c.
\newblock \bibinfo{title}{Exponential loss regularisation for encouraging
ordinal constraint to shotgun stocks quality assessment}.
\newblock \bibinfo{journal}{Applied Soft Computing} \bibinfo{volume}{138},
\bibinfo{pages}{110191}.
\newblock \DOIprefix\doi{10.1016/j.asoc.2023.110191}.
%Type = Article
\bibitem[{{Vargas} et~al.(2020){Vargas}, Gutiérrez and
Hervás-Martínez}]{vargas2020clm}
\bibinfo{author}{{Vargas}, V.M.}, \bibinfo{author}{Gutiérrez, P.A.},
\bibinfo{author}{Hervás-Martínez, C.}, \bibinfo{year}{2020}.
\newblock \bibinfo{title}{Cumulative link models for deep ordinal
classification}.
\newblock \bibinfo{journal}{Neurocomputing} \bibinfo{volume}{401},
\bibinfo{pages}{48--58}.
\newblock \DOIprefix\doi{https://doi.org/10.1016/j.neucom.2020.03.034}.
%Type = Article
\bibitem[{Virtanen et~al.(2020)Virtanen, Gommers, Oliphant, Haberland, Reddy,
Cournapeau, Burovski, Peterson, Weckesser, Bright et~al.}]{virtanen2020scipy}
\bibinfo{author}{Virtanen, P.}, \bibinfo{author}{Gommers, R.},
\bibinfo{author}{Oliphant, T.E.}, \bibinfo{author}{Haberland, M.},
\bibinfo{author}{Reddy, T.}, \bibinfo{author}{Cournapeau, D.},
\bibinfo{author}{Burovski, E.}, \bibinfo{author}{Peterson, P.},
\bibinfo{author}{Weckesser, W.}, \bibinfo{author}{Bright, J.}, et~al.,
\bibinfo{year}{2020}.
\newblock \bibinfo{title}{Scipy 1.0: fundamental algorithms for scientific
computing in python}.
\newblock \bibinfo{journal}{Nature methods} \bibinfo{volume}{17},
\bibinfo{pages}{261--272}.
\newblock \DOIprefix\doi{10.1038/s41592-019-0686-2}.
%Type = Article
\bibitem[{Van~der Walt et~al.(2014)Van~der Walt, Sch{\"o}nberger,
Nunez-Iglesias, Boulogne, Warner, Yager, Gouillart and Yu}]{van2014scikit}
\bibinfo{author}{Van~der Walt, S.}, \bibinfo{author}{Sch{\"o}nberger, J.L.},
\bibinfo{author}{Nunez-Iglesias, J.}, \bibinfo{author}{Boulogne, F.},
\bibinfo{author}{Warner, J.D.}, \bibinfo{author}{Yager, N.},
\bibinfo{author}{Gouillart, E.}, \bibinfo{author}{Yu, T.},
\bibinfo{year}{2014}.
\newblock \bibinfo{title}{scikit-image: image processing in python}.
\newblock \bibinfo{journal}{PeerJ} \bibinfo{volume}{2}, \bibinfo{pages}{e453}.
\newblock \DOIprefix\doi{10.7717/peerj.453}.
%Type = Inproceedings
\bibitem[{Wang and Yang(2018)}]{wang2018diabetic}
\bibinfo{author}{Wang, Z.}, \bibinfo{author}{Yang, J.}, \bibinfo{year}{2018}.
\newblock \bibinfo{title}{Diabetic retinopathy detection via deep convolutional
networks for discriminative localization and visual explanation}, in:
\bibinfo{booktitle}{Workshops at the thirty-second AAAI conference on
artificial intelligence}, pp. \bibinfo{pages}{514--521}.
%Type = Article
\bibitem[{Zhao et~al.(2022)Zhao, Zhang, Feng and Xu}]{zhao2022deep}
\bibinfo{author}{Zhao, Y.}, \bibinfo{author}{Zhang, X.}, \bibinfo{author}{Feng,
W.}, \bibinfo{author}{Xu, J.}, \bibinfo{year}{2022}.
\newblock \bibinfo{title}{Deep learning classification by resnet-18 based on
the real spectral dataset from multispectral remote sensing images}.
\newblock \bibinfo{journal}{Remote Sensing} \bibinfo{volume}{14},
\bibinfo{pages}{4883}.
\newblock \DOIprefix\doi{10.3390/rs14194883}.
\end{thebibliography}