Preventing Conflicting Gradients in Neural Marked Temporal Point Processes
Extracted source text · arXiv 2412.08590
\documentclass[10pt]{article} % For LaTeX2e
\usepackage{tmlr}
% If accepted, instead use the following line for the camera-ready submission:
%\usepackage[accepted]{tmlr}
% To de-anonymize and remove mentions to TMLR (for example for posting to preprint servers), instead use the following:
%\usepackage[preprint]{tmlr}
% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{comment}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{graphicx}
\usepackage{subcaption}
\usepackage{algorithmic}
\usepackage{algorithm}
\usepackage{bbm}
\usepackage{wrapfig}
\usepackage{todonotes}
\newcommand{\dobox}[1]{\todo[color=green!40, inline,size=\small]{#1}}
\newtheorem{corollary}{Corollary}
\title{Preventing Conflicting Gradients in Neural Marked Temporal Point Processes}
% Authors must not appear in the submitted version. They should be hidden
% as long as the tmlr package is used without the [accepted] or [preprint] options.
% Non-anonymous submissions will be rejected without review.
\author{\name Tanguy Bosser \email tanguy.bosser@umons.ac.be \\
\addr Department of Computer Science\\
University of Mons
\ANDD
\name Souhaib Ben Taieb \email souhaib.bentaieb@mbzuai.ac.ae \\
\addr MBZUAI \\
\addr University of Mons
}
% The \author macro works with any number of authors. Use \AND
% to separate the names and addresses of multiple authors.
\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}
\def\month{MM} % Insert correct month for camera-ready version
\def\year{YYYY} % Insert correct year for camera-ready version
\def\openreview{\url{https://openreview.net/forum?id=XXXX}} % Insert correct link to OpenReview for camera-ready version
\begin{document}
\maketitle
\begin{abstract}
Neural Marked Temporal Point Processes (MTPP) are flexible models to capture complex temporal inter-dependencies between labeled events. These models inherently learn two predictive distributions: one for the arrival times of events and another for the types of events, also known as marks. In this study, we demonstrate that learning a MTPP model can be framed as a two-task learning problem, where both tasks share a common set of trainable parameters that are optimized jointly. We show that this often leads to the emergence of conflicting gradients during training, where task-specific gradients are pointing in opposite directions. When such conflicts arise, following the average gradient can be detrimental to the learning of each individual tasks, resulting in overall degraded performance. To overcome this issue, we introduce novel parametrizations for neural MTPP models that allow for separate modeling and training of each task, effectively avoiding the problem of conflicting gradients. Through experiments on multiple real-world event sequence datasets, we demonstrate the benefits of our framework compared to the original model formulations.
\end{abstract}
\begin{comment}
Neural Marked Temporal Point Processes (MTPP) are flexible models to capture complex temporal inter-dependencies between labeled events. These models inherently learn two predictive distributions: one for the arrival times of events and another for the types of events, also known as marks. In this study, we demonstrate that learning a MTPP model can be framed as a two-task learning problem, where both tasks share a common set of trainable parameters that are optimized jointly. Furthermore, we show that this often leads to conflicting gradients during training, where task-specific gradients are pointing in opposite directions. When such conflicts arise, following the average gradient can be detrimental to the learning of each individual tasks, resulting in overall degraded performance. To overcome this issue, we introduce novel parametrizations for neural MTPP models that allow for separate modeling and training of each task, effectively avoiding the problem of conflicting gradients. Through experiments on multiple real-world event sequence datasets, we demonstrate the benefits of our framework compared to the original model formulations.
\end{comment}
\section{Introduction}
Sequences of labeled events observed in continuous time at irregular intervals are ubiquitous across various fields such as healthcare \citep{EHR}, finance \citep{bacry2015hawkesprocessesfinance}, social media \citep{farajtabar2015coevolve}, and seismology \citep{Ogata}. In numerous application domains, an important problem involves predicting the timing and types of future events—often called marks—based on historical data. Marked Temporal Point Processes (MTPP) \citep{daley2003pointprocess} provide a mathematical framework for modeling sequences of events, enabling subsequent inferences on the system under study. The Hawkes process, originally introduced by Hawkes \citep{Hawkes}, is a well-known example of a MTPP model that has found successful applications in diverse domains, including finance \citep{hawkes2018finance}, crime analysis \citep{egesdal2010gang}, and user recommendations \citep{du2015recommendation}. However, the strong modeling assumptions of classical MTPP models often limit their ability to capture complex event dynamics \citep{NeuralHawkes}. This limitation has led to the rapid development of a more flexible class of neural MTPP models, incorporating recent advances in deep learning \citep{SchurSurvey}.
This paper argues that learning a neural MTPP model can be interpreted as a two-task learning problem where both tasks share a common set of parameters and are optimized jointly. Specifically, one task focuses on learning the distribution of the next event's arrival time conditional on historical events. The other task involves learning the distribution of the categorical mark conditional on both the event's arrival time and the historical events. We identify these tasks as the time prediction and mark prediction, respectively. While parameter sharing between tasks can sometimes enhance training efficiency \citep{standley2020taskslearnedmultitasklearning}, it may also result in performance degradation when compared to training each task separately. A major challenge in the simultaneous optimization of multi-task objectives is the issue of conflicting gradients \citep{liu2024conflictaverse}. This term describes situations where task-specific gradients point in opposite directions. When such conflicts arise, gradient updates tend to favor tasks with larger gradient magnitudes, thus hindering the learning process of other concurrent tasks and adversely affecting their performance. Although the phenomenon of conflicting gradients has been studied in various fields \citep{chen2018gradnorm, chen2020just, yu2020gradient, shi2023recon}, its impact on the training of neural MTPP models remains unexplored. We propose to address this gap with the following contributions:
(1) We demonstrate that conflicting gradients frequently occur during the training of neural MTPP models. Furthermore, we show that such conflicts can significantly degrade a model's predictive performance on the time and mark prediction tasks.
(2) To prevent the issue of conflicting gradients, we introduce novel parametrizations for existing neural TPP models, allowing for separate modeling and training of the time and mark prediction tasks. Inspired from the success of \citep{shi2023recon}, our framework allows to prevent gradient conflicts from the root while maintaining the flexibility of the original parametrizations.
(3) We want to emphasize that our approach to disjoint parametrizations does not assume the independence of arrival times and marks. Unlike prior studies that assumed conditional independence \citep{IntensityFree, RMTPP}, we propose a simple yet effective parametrization for the mark conditional distribution that relaxes this assumption.
(4) Through a series of experiments with real-world event sequence datasets, we show the advantages of our framework over the original model formulations. Specifically, our framework effectively prevents the emergence of conflicting gradients during training, thereby enhancing the predictive accuracy of the models. Additionally, all our experiments are reproducible and implemented using a common code base available in the supplementary material.
\section{Background and Notations}
\label{section:background}
A \textbf{Marked Temporal Point Process} (MTPP) is a random process whose realization is a sequence of events $\mathcal{S} = \{e_i = (t_i,k_i)\}_{i=1}^n$. Each event $e_i \in \mathcal{S}$ is an ordered pair with an \textit{arrival time} $t_i \in [0, T]$ (with $t_0 = 0$) and a categorical label $k_i \in \mathbb{K} = \{1,...,K\}$ called \textit{mark}. The arrival times form a sequence of strictly increasing random values observed within a specified time interval $[0, T]$, i.e. $0 \leq t_1 < t_2 < \ldots < t_n \leq T$. Equivalently, $\tau_i = t_i - t_{i-1} \in \mathbb{R}_{+}$ is an event \textit{inter-arrival time}. We will use both representations interchangeably throughout the paper. If $e_{i-1}$ is the last observed event, the occurrence of the next event $e_i$ in $(t_{i-1}, \infty[$ can be fully characterized by the \textit{joint PDF} $f(\tau, k|\mathcal{H}_t)$, where ${\mathcal{H}_{t} = \{(t_i,k_i) \in \mathcal{S}~|~t_i < t\}}$ is the observed process history. For clarity of notations, we will use the notation '$\ast$' of \citep{daley2008pointprocess} to indicate dependence on $\mathcal{H}_t$, i.e. $f^\ast(\tau,k) = f(\tau, k|\mathcal{H}_t)$. This joint PDF can be factorized as $f^\ast(\tau,k) = f^\ast(\tau)p^\ast(k|\tau)$, where $f^\ast(\tau)$ is the\textit{ PDF of inter-arrival times}, and $p^\ast(k|\tau)$ is the \textit{conditional PMF of marks}. A MTPP can be equivalently described by its so-called \textit{marked conditional intensity functions} \citep{rasmussen2018lecture}, which are defined for $t_{i-1} < t < t_i$ as $\lambda_k^\ast(t) = f^\ast(t,k)/(1 - F^\ast(t))$, where $F^\ast(t)$ is the \textit{conditional CDF of arrival-times} and $k \in \mathbb{K}$. From the marked intensities, we can also define the \textit{marked compensators} $\Lambda^\ast_k(t) = \int_{t_{i-1}}^t \lambda_k^\ast(s)ds$. Provided that certain modeling constraints are satisfied, each of $f^\ast(\tau, k)$, $\lambda_k^\ast(t)$ and $\Lambda_k^\ast(t)$ fully characterizes a MTPP and can be retrieved from the others \citep{rasmussen2018lecture, EHR}.
\textbf{Neural Temporal Point Processes.} To capture complex dependencies between events, the framework of neural MTPP \citep{SchurSurvey, bosser2023predictive} incorporates neural network components into the model architecture, allowing for more flexible models. A neural MTPP model consists of three main components: (1) An \textit{event encoder} that learns a representation $\boldsymbol{e}_i \in \mathbb{R}^{d_e}$ for each event $e_i \in \mathcal{S}$, (2) a \textit{history encoder} that learns a compact history embedding $\boldsymbol{h}_i \in \mathbb{R}^{d_h}$ of the history $\mathcal{H}_{t_i}$ of event $e_i$, and (3) a \textit{decoder} that defines a function characterizing the MTPP from $\boldsymbol{h}_i$, e.g. $\lambda_k^\ast(t)$. Let $f^\ast(\tau,k;\btheta)$, $\lambda_k^\ast(t;\btheta)$, or $\Lambda_k^\ast(t;\btheta)$ be a valid model of $f^\ast(\tau,k)$, where the set of trainable parameters \( \btheta \) lies within the parameter space \( \Theta \).
To train this model, we use a dataset $\mathcal{S} = \{\mathcal{S}_1,..., \mathcal{S}_L\}$, where each sequence \(\mathcal{S}_l\) comprises \(n_l\) events with arrival times observed within the interval \([0,T]\) and \(l=1,...,L\). The training objective is the average sequence negative log-likelihood (NLL) \citep{rasmussen2018lecture}, given by
\begin{equation}
\mathcal{L}(\btheta; \mathcal{S}) =
-\frac{1}{L} \sum_{l=1}^{L} \left[\left[\sum_{i=1}^{n_l}
\text{log}~f^\ast(\tau_{l,i}, k_{l,i}; \boldsymbol{\theta})\right] - \text{log}(1-F^\ast(T; \btheta))\right],
\label{eq:NLL_before}
\end{equation}
\begin{comment}
\begin{equation}
\mathcal{L}(\btheta; \mathcal{S}) =
-\frac{1}{L} \sum_{l=1}^{L} \left[\sum_{i=1}^{n_l}
\text{log}~p^\ast(k_{l,i}|\tau_{l,i}; \boldsymbol{\theta}) +
\text{log}~f^\ast(\tau_{l,i}; \btheta)\right] - \text{log}(1-F^\ast(T; \btheta)) .
\label{eq:NLL_before}
\end{equation}
\end{comment}
which is optimized using mini-batch stochastic gradient descent \citep{ruder2017overview}.
\section{Conflicting Gradients in Two-Task Learning for Neural MTPP Models}
Consider the factorization of \( f^\ast(\tau_{l,i}, k_{l,i}; \boldsymbol{\theta}) \) into \( f^\ast(\tau_{l,i}; \boldsymbol{\theta}) \) and \( p^\ast(k_{l,i}|\tau_{l,i}; \boldsymbol{\theta}) \), where \( f^\ast(\tau_{l,i}, k_{l,i}; \boldsymbol{\theta}) = f^\ast(\tau_{l,i}; \btheta) \cdot p^\ast(k_{l,i}|\tau_{l,i}; \boldsymbol{\theta}) \). Substituting this decomposition into the NLL in \eqref{eq:NLL_before} and rearranging terms, we obtain:
\begin{equation}
\mathcal{L}(\btheta; \mathcal{S}) =
\underbrace{-\frac{1}{L} \sum_{l=1}^{L} \left[\left[\sum_{i=1}^{n_l}
\text{log}~f^\ast(\tau_{l,i}; \btheta) \right] - \text{log}(1-F^\ast(T; \btheta))\right]}_{\mathcal{L}_T(\btheta, \mathcal{S})} \underbrace{-\frac{1}{L} \sum_{l=1}^{L} \left[\sum_{i=1}^{n_l}
\text{log}~p^\ast(k_{l,i}|\tau_{l,i}; \boldsymbol{\theta}) \right]}_{\mathcal{L}_M(\btheta, \mathcal{S})}.
\label{eq:NLL}
\end{equation}
This shows that the total objective function \( \mathcal{L}(\btheta; \mathcal{S}) \) consists of two sub-objectives: \( \mathcal{L}_T(\btheta; \mathcal{S}) \) and \( \mathcal{L}_M(\btheta; \mathcal{S}) \), revealing that learning an MTPP model can be interpreted as a \emph{two-task} learning problem. The first objective, \( \mathcal{L}_T(\btheta; \mathcal{S}) \), relates to modeling the predictive distribution of (inter-)arrival times, which we refer to as the \emph{time prediction task} \( \mathcal{T}_T \). The second objective, \( \mathcal{L}_M(\btheta; \mathcal{S}) \), concerns modeling the conditional predictive distribution of the marks \( p^\ast(k|\tau; \btheta) \), which we call the \emph{mark prediction task} \( \mathcal{T}_M \).
%This shows that the total objective function \( \mathcal{L}(\btheta; \mathcal{S}) \) consists of two sub-objectives: \( \mathcal{L}_T(\btheta; \mathcal{S}) \) and \( \mathcal{L}_M(\btheta; \mathcal{S}) \). The first objective, \( \mathcal{L}_T(\btheta; \mathcal{S}) \), relates to modeling the predictive distribution of (inter-)arrival times, which we refer to as the time prediction task \( \mathcal{T}_T \). The second objective, \( \mathcal{L}_M(\btheta; \mathcal{S}) \), concerns modeling the predictive distribution of the marks \( p^\ast(k|\tau; \btheta) \), which we call the mark prediction task \( \mathcal{T}_M \).
%We can see that the total objective $\mathcal{L}(\btheta; \mathcal{S})$ in (\ref{eq:NLL}) is composed of two sub-objectives, $\mathcal{L}_T(\btheta; \mathcal{S})$ and $\mathcal{L}_M(\btheta; \mathcal{S})$, revealing that learning a MTPP model can be essentially interpreted as a two-task learning problem.
% Specifically, the time prediction task $\mathcal{T}_T$ aims to learn a predictive distribution of (inter-)arrival times by minimizing $\mathcal{L}_T(\btheta; \mathcal{S})$. Concurrently, the mark prediction task focuses on learning a predictive distribution of marks $p^\ast(k|\tau;\btheta)$ by minimizing \( \mathcal{L}_M(\btheta, \mathcal{S})\).
\textbf{Conflicting gradients.} Assuming that $\mathcal{L}_T(\btheta)$ and $\mathcal{L}_M(\btheta)$ are differentiable, let $\boldsymbol{g}_T = \nabla_{\btheta} \mathcal{L}_T(\btheta)$ and $\boldsymbol{g}_M = \nabla_{\btheta} \mathcal{L}_M(\btheta)$ denote the gradients of $\mathcal{L}_T(\btheta)$ and $\mathcal{L}_M(\btheta)$, respectively, with respect to the shared parameters $\btheta$\footnote{We explicitly omit the dependency of $\mathcal{L}_T$ and $\mathcal{L}_M$ on $\mathcal{S}$ to simplify notations.}. As discussed in \citep{shi2023recon}, when $\boldsymbol{g}_T$ and $\boldsymbol{g}_M$ are pointing in opposite directions, i.e. $\boldsymbol{g}_T \cdot \boldsymbol{g}_M < 0$, an update step in the direction of negative $\boldsymbol{g}_T$ for $\btheta$ will increase the loss for task $\mathcal{T}_M$, and inversely for task $\mathcal{T}_T$ if an update step is taken in the direction of negative $\boldsymbol{g}_M$. Such \textit{conflicting gradients} can be formally defined as follows.
\begin{wrapfigure}{r}{0.35\textwidth}
\vspace{-0.5cm}
\includegraphics[width=\linewidth]{images/drawings/gradients.pdf}
\caption{Conflicting gradients}
\label{fig:conflicting}
\vspace{-0.5cm}
\end{wrapfigure}
\textbf{Definition 1} (Conflicting gradients \citep{shi2023recon}) \textit{Let ${\phi_{TM} \in [0,2\pi]}$ be the angle between the gradients $\boldsymbol{g}_T$ and $\boldsymbol{g}_M$. They are said to be conflicting with each other if} $\text{cos}~\phi_{TM} < 0$.
The smaller the value of $\text{cos }\phi_{TM} \in [-1,0]$, the more severe the conflict between the gradients. Figure (\ref{fig:conflicting}) illustrates these conflicting gradients. Ideally, we want the gradients to align during optimization (i.e. $\text{cos } \phi_{TM} >0$) to encourage positive reinforcement between the two tasks, or to be simply orthogonal (i.e. $\text{cos } \phi_{TM} =0$). Conflicting gradients, especially those with significant differences in magnitude, pose substantial challenges during the optimization of multi-task learning objectives \citep{yu2020gradient}. Specifically, if $\bg_T$ and $\bg_M$ conflict, the update step for \(\btheta\) will likely be dominated by the gradient of whichever task---$\bg_T$ or $\bg_M$---has the greater magnitude, thereby disadvantaging the other task. The degree of similarity between the magnitudes of these two gradients can be quantified using a metric known as \textit{gradient magnitude similarity}, defined as follows:
\textbf{Definition 2} (Gradient Magnitude Similarity \citep{yu2020gradient}) \textit{The gradient magnitude similarity between $\boldsymbol{g}_T$ and $\boldsymbol{g}_M$ is defined as $\text{GMS} = \frac{2 ||\bg_T||_2 ||\bg_M||_2}{||\bg_T||_2^2 + ||\bg_M||_2^2}$, where $||\cdot||_2$ is the $l_2$-norm.}
A GMS value close to 1 indicates that the magnitudes of \(\bg_T\) and \(\bg_M\) are similar, while a GMS value close to 0 suggests a significant difference between them. Ideally, we aim to minimize the number of conflicting gradients and maintain a GMS close to 1 to ensure balanced learning across the two tasks. However, a low GMS value among conflicting gradients does not specify which task is being prioritized. Therefore, we introduce the \textit{time priority index} to address this issue, which is defined as follows:
\textbf{Definition 3} (Time Priority Index) \textit{The time priority index between conflicting gradients $\bg_T$ and $\bg_M$ \textcolor{blue}{with $\text{cos } \phi_{TM} < 0$} is defined as} $\text{TPI} = \mathbbm{1}\left(||\bg_T||_2 > ||\bg_M||_2 \right)$ \textit{where} $\mathbbm{1}(\cdot)$ \textit{is the indicator function.}
If $\bg_T$ and $\bg_M$ are conflicting and optimization prioritizes task $\mathcal{T}_T$, then the TPI takes the value 1. Conversely, if task \(\mathcal{T}_M\) is prioritized, the TPI takes the value 0.
\textbf{Do gradients conflict in neural MTPP models?} To explore this question, we perform a preliminary experiment with common neural MTPP baselines that either learn $f^\ast(t,k;\btheta)$, $\lambda_k^\ast(t;\btheta)$, or $\Lambda_k^\ast(t;\btheta)$: THP \citep{TransformerHawkes}, SAHP \citep{SelfAttentiveHawkesProcesses}, FNN \citep{FullyNN}, LNM \citep{IntensityFree}, and RMTPP \citep{RMTPP}. We aim to determine whether conflicting gradients occur during their training. For this purpose, each model is trained to minimize the NLL defined in (\ref{eq:NLL}) using sequences from two real-world datasets: LastFM \citep{hidasi2012} and MOOC \citep{kumar2019predicting}. For optimization, we rely on the Adam optimizer \citep{adam} used by default in neural MTPP training with learning rate $\alpha= 10^{-3}$. At every gradient update, we calculate the gradients \(\bg_T\) and \(\bg_M\) with respect to the shared parameters \(\btheta\)\footnote{In practice, conflicts are computed layer-wise for each layer of the model (e.g the weights $\mathbf{W}$ of a fully-connected layer).}, recording values of \(\text{cos }\phi_{TM}\), GMS, and TPI. Figure \ref{fig:toy_grad} shows the distribution of \(\text{cos }\phi_{TM}\) across all \textcolor{blue}{$S$} training iterations, along with the \textcolor{blue}{average values of GMS and TPI, and the proportion of conflicting gradients (CG), computed as}
\begin{equation}
\textcolor{blue}{\text{CG} = \frac{\sum_{s=1}^S \mathbbm{1}\left(\text{cos } \phi_{TM}^s < 0\right)}{\sum_{s=1}^S \text{cos } \phi_{TM}^s}},
\label{eq:cg}
\end{equation}
\textcolor{blue}{where $\phi^s_{TM}$ refers to the angle between $\bg_T^s$ and $\bg_M^s$ at training iteration $s$. Gradients that are conflicting during training correspond to the red bars. We observe that some models, such as THP and SAHP on MOOC, frequently exhibit conflicting gradients during training, as indicated by a high value of CG. Conversely, while other models show a more balanced proportion of conflicting gradients, these are generally characterized by low GMS values, which may potentially impair performance on the task with the lowest magnitude gradient. In this context, the data indicates that optimization generally tends to favor \(\mathcal{T}_T\) during optimization, as suggested by an average TPI greater than 0.5. In Section \ref{section:experiments}, our experiments show that the combined influence of a high proportion of conflicts and low GMS values during training can significantly deteriorate model performance on the time and mark prediction tasks. We provide additional visualizations for other datasets in Appendix \ref{sec:neurips_app:distirbution_+}, which confirm the above discussion and further suggest that conflicting gradients emerge frequently during the training of neural MTPP models. }
\begin{figure}[!t]
\vspace{-1cm}
\centering
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_all/conflicting_gradients_all/lastfm_filtered_rmtpp_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_all/conflicting_gradients_all/mooc_filtered_rmtpp_bis.pdf}
\end{subfigure}
\vspace{-0.2cm}
\caption{Distribution of $\text{cos } \phi_{TM}$ during training for the different baselines on MOOC and LastFM. CG refers to the proportion of $\text{cos } \phi_{TM} < 0$ observed during training. \textcolor{blue}{The distribution is obtained by pooling the values of $\phi_{TM}$ over 5 training runs, and gradients that are conflicting correspond to the red bars.}}
\label{fig:toy_grad}
\vspace{-0.3cm}
\end{figure}
\section{A Framework to Prevent Conflicting Gradients in Neural MTPP Models}
\label{section:method}
Given the observations from the previous section, our goal is to prevent the occurrence of conflicting gradients during the training of neural MTPP models with the NLL objective given in (\ref{eq:NLL}). To accomplish this, we first propose in Section \ref{sec:neurips:theoretical} a naive approach that leverages duplicated and disjoint
instances of the same model. Then, to avoid redundancy in model specification, we introduce in Section \ref{sec:existing_TPP} novel parametrizations for neural MTPP models. We finally show in Section \ref{sec:disjoint} how our parametrizations enable disjoint modeling and training of the time and mark prediction tasks.
Existing MTPP models generally fall into three categories based on the chosen parametrization: (1) \textit{Intensity-based} approaches that model the marked intensity function $\lambda_k^\ast(t; \boldsymbol{\theta})$ \citep{TransformerHawkes}, (2) \textit{Density-based} approaches that model the joint density $f^\ast(t, k; \boldsymbol{\theta})$ \citep{IntensityFree}, and (3) \textit{Compensator-based} approaches that model the marked compensators $\Lambda_k^\ast(t; \boldsymbol{\theta})$ \citep{FullyNN}. We denote these different approaches as \emph{joint parametrizations} because they define a function that involves both the arrival time and the mark. To enable disjoint modeling of the time and mark prediction tasks, we consider the factorization of these functions into the products of two components: one involving a function of the arrival-times, and the other involving the conditional PMF of marks:
\begin{align}
\lambda^\ast_k(t; \btheta) &= \lambda^\ast(t; \btheta) p^\ast(k|t;\btheta) = \frac{d}{dt} \left[\Lambda^\ast(t; \btheta)\right] p^\ast(k|t;\btheta), \label{eq:lambda_param} \\
f^\ast(t,k;\boldsymbol{\theta}) &= f^\ast(t;\boldsymbol{\theta})p^\ast(k|t; \boldsymbol{\theta}), \label{eq:joint_param} \\
\Lambda^\ast_k(t; \btheta) &= \int_{t_{i-1}}^t \lambda^\ast(s; \btheta) p^\ast(k|s;\btheta)ds \label{eq:Lambda_param},
\end{align}
where $\lambda^\ast(t;\btheta) = \frac{f^\ast(t;\btheta)}{1 - F^\ast(t;\btheta)}$ and $\Lambda^\ast(t;\btheta) = \int_{t_{i-1}}^t \lambda^\ast(s;\btheta)ds$ are respectively the \textit{ground intensity} and the \textit{ground compensator} of the process. Similarly to (\ref{eq:NLL}), a two-task NLL objective involving the r.h.s of expressions (\ref{eq:lambda_param}) and (\ref{eq:Lambda_param}) can be derived, where task $\mathcal{T}_T$ now consists in learning either $\lambda^\ast(t;\btheta)$ or $\Lambda^\ast(t;\btheta)$. We provide the expression of these two-task NLL objectives in Appendix (\ref{sec:add_nll}). The factorizations presented in expressions (\ref{eq:lambda_param}) and (\ref{eq:joint_param}) show that, to enable disjoint modeling of time and mark prediction functions, we need to specify $p^\ast(k|t; \btheta_M)$ with parameters $\btheta_M$, and either $f^\ast(t; \btheta_T)$, $\lambda^\ast(t;\btheta_T)$, or $\Lambda^\ast(t;\btheta_T)$ with parameters $\btheta_T$.
As a naive approach, we first explore obtaining these functions from duplicated instances of the joint parametrizations $\lambda_k^\ast(t;\btheta)$, $\Lambda_k^\ast(t;\btheta)$ or $f^\ast(t,k;\btheta)$, as presented next.
\subsection{A Naive Approach to Achieve Disjoint Parametrizations}
\label{sec:neurips:theoretical}
Consider a model that parametrizes $\lambda_k^\ast(t; \btheta)$, such as THP \citep{TransformerHawkes}. To obtain a disjoint parametrization of $\lambda^\ast(t;\btheta_T)$ and $p^\ast(k|t;\btheta_M)$, we can parametrize two identical functions $\lambda_k^{\ast}(t; \btheta_T)$ and $\lambda_k^{\ast}(t; \btheta_M)$ from the same model and for all $k \in \mathbb{K}$, where $\btheta_T$ and $\btheta_M$ are disjoint set of trainable parameters. From these two functions, we can finally derive $\lambda^\ast(t;\btheta_T)$ and $p^\ast(k|t;\btheta_M)$ as
\begin{equation}
\lambda^\ast(t;\btheta_T) = \sum_{k=1}^K \lambda_k^{\ast}\left(t; \btheta_T\right) \quad\text{and}\quad p^\ast(k|t;\btheta_M) = \frac{\lambda_k^{\ast}\left(t; \btheta_M\right)}{\sum_{k=1}^K \lambda_k^{\ast}\left(t; \btheta_M\right)},
\label{eq:double}
\end{equation}
effectively defining the desired disjoint parametrization. In the presence of conflicts during training, we can show that a gradient update step for the shared model $\lambda_k^\ast(t;\btheta)$ leads to higher loss compared to the duplicated model in (\ref{eq:double}) with disjoint parameters $\btheta_T$ and $\btheta_M$. Indeed, suppose that $\btheta$, $\btheta_T$ and $\btheta_M$ are all initialized with the same $\btheta^{s}$ at training iteration $s \in \mathbb{N}$. Assuming that $\mathcal{L}_T$ and $\mathcal{L}_M$ are differentiable, let
\begin{comment}
the gradient update steps for $\btheta$, $\btheta_T$ and $\btheta_M$ are
\begin{equation}
\btheta^{s+1} = \btheta^{s} - \alpha (\bg_T + \bg_M), \quad \btheta_T^{s+1} = \btheta^{s} - \alpha \bg_T \quad\text{and}\quad \btheta_M^{s+1} = \btheta^{s} - \alpha \bg_M,
\end{equation}
where $\alpha >0$ is the learning rate and
\end{comment}
\begin{equation}
\bg_T = \nabla_{\btheta} \mathcal{L}_T\left(\btheta^{s}\right) \quad\text{and}\quad \bg_M = \nabla_{\btheta} \mathcal{L}_M\left(\btheta^{s}\right).
\end{equation}
Denoting $\phi_{TM}$ as the angle between $\bg_T$ and $\bg_M$, we have the following corollary of Theorem 4.1. from \citep{shi2023recon}:
\begin{corollary}
\textit{Assume that} $\mathcal{L}_T$ \textit{and} $\mathcal{L}_M$ \textit{are differentiable, and that the learning rate} $\alpha$ \textit{is sufficiently small. If} $\text{cos } \phi_{TM} < 0$, \textit{then} $\mathcal{L}(\{\btheta_T^{s+1}, \btheta_M^{s+1}\}) < \mathcal{L}(\btheta^{s+1})$.
\label{cor:neurips:loss}
\end{corollary}
This result essentially indicates that a model trained with disjoint parameters leads to lower loss after a gradient update if conflicts arise during training, i.e. $\text{cos } \phi_{TM} < 0$. We provide the proof in Appendix \ref{app:proofs}. Naturally, expression (\ref{eq:double}) and Corollary \ref{cor:neurips:loss} remain valid for models that parametrize $\Lambda_k^\ast(t;\btheta)$ or $f^\ast(\tau,k;\btheta)$, as these functions can be uniquely retrieved from $\lambda_k^\ast(t;\btheta)$.
\subsection{Novel Disjoint Parametrizations of Neural MTPP Models}
\label{sec:existing_TPP}
In this section, we introduce an alternative approach to (\ref{eq:double}) to achieve disjoint parametrizations of $\mathcal{T}_T$ and $\mathcal{T}_M$. Specifically, we introduce novel parametrizations of existing neural MTPP model that directly parametrize $p^\ast(k|t; \btheta_M)$ and either $f^\ast(t; \btheta_T)$, $\lambda^\ast(t;\btheta_T)$, or $\Lambda^\ast(t;\btheta_T)$, thereby avoiding the unnecessary redundancy in model specification required by the method in (\ref{eq:double}). For a query time $t \geq t_{i-1}$, we define a history representation $\boldsymbol{h} = \text{ENC}(\{\boldsymbol{e}_1,...,\boldsymbol{e}_{i-1}\}; \btheta_h) \in \mathbb{R}^{d_h}$, where $\text{ENC}(\cdot~;\btheta_h)$ denotes the history encoder with parameters $\btheta_h$. The encoder $\text{ENC}(\cdot~;\btheta_h)$ is general and encompasses any encoder architecture typically found in the neural MTPP literature, such as recurrent neural networks (RNN) \citep{RMTPP, IntensityFree} or self-attention mechanisms \citep{TransformerHawkes, SelfAttentiveHawkesProcesses}.
\paragraph{A general approach to model the distribution of marks.}
Given a query time $t \geq t_{i-1}$ and its corresponding history representation $\boldsymbol{h}$, we propose to define the conditional PMF of marks $p^\ast(k|t;\btheta_M)$ using the following simple model:
\begin{align}
&p^\ast(k|t; \btheta_M) = \sigma_{\text{so}}\left(\mathbf{W}_2 \sigma_R\left(\mathbf{W}_1\left[\boldsymbol{h}||\text{log}(\tau)\right] + \boldsymbol{b}_1)\right) + \boldsymbol{b}_2\right),
\label{eq:pmf_cond_dis}
\end{align}
where $\tau = t-t_{i-1}$, $\sigma_{\text{so}}$ is the softmax activation function, $\mathbf{W}_1 \in \mathbb{R}^{d_1 \times (d_h + 1)}$, $\boldsymbol{b}_1 \in \mathbb{R}^{d_1}$, $\mathbf{W}_2 \in \mathbb{R}^{K \times d_1}$, $\boldsymbol{b}_2 \in \mathbb{R}^{K}$, and $||$ means concatenation. Here, $\btheta_M = \{\mathbf{W}_1, \mathbf{W}_2, \boldsymbol{b_1}, \boldsymbol{b}_2, \btheta_h\}$. Despite its simplicity, this model is flexible and capable of capturing the evolving dynamics of the mark distribution between two events. Note that \eqref{eq:pmf_cond_dis} effectively captures inter-dependencies between arrival times and marks. Moreover, by removing $\text{log}(t-t_{i-1})$ from expression (\ref{eq:pmf_cond_dis}), we obtain a PMF of marks that is independent of time, given $\mathcal{H}_t$.
\textbf{Intensity-based parametrizations.} We first consider MTPP models specified by their marked intensities, namely THP \citep{TransformerHawkes} and SAHP \citep{SelfAttentiveHawkesProcesses}. We propose revising the original model formulations to directly parametrize $\lambda^\ast(t)$, while $p^\ast(k|t)$ is systematically derived from expression (\ref{eq:pmf_cond_dis}). Furthermore, although RMTPP \citep{RMTPP} is originally defined in terms of this decomposition, we extend the model to incorporate the dependence of marks on time.
\textit{SAHP}$^{+}$. While the original model formulation parametrizes $\lambda_k^\ast(t)$, we adapt its formulation to define:
\begin{equation}
\lambda^\ast(t;\btheta_T) = \boldsymbol{1}^\mathsf{T} \left[\sigma_S \left(\boldsymbol{\mu} - (\boldsymbol{\eta}- \boldsymbol{\mu})\text{exp}(-\boldsymbol{\gamma}(t-t_{i-1}))\right)\right],
\label{eq:sahp_1}
\end{equation}
where $\boldsymbol{1} \in \mathbb{R}^{C}$ is a vector of $1$'s allowing to define the ground intensity as a sum over $C$ different representations. In (\ref{eq:sahp_1}), $\boldsymbol{\mu} = \sigma_G(\mathbf{W}_\mu \boldsymbol{h})$, $\boldsymbol{\eta} = \sigma_S(\mathbf{W}_\eta \boldsymbol{h})$ and $\boldsymbol{\gamma} = \sigma_G(\mathbf{W}_\gamma \boldsymbol{h})$ where $\sigma_S$ and $\sigma_G$ are respectively the softplus and GeLU activation functions \citep{hendrycks2023gelu}. $\mathbf{W}_\mu, \mathbf{W}_\eta, \mathbf{W}_\gamma \in \mathbb{R}^{C \times d_h}$ are learnable parameters and $\btheta_T = \{\mathbf{W}_\mu, \mathbf{W}_\eta, \mathbf{W}_\gamma, \btheta_h\}$.
\textit{THP}$^{+}$. Following a similar reasoning, the original formulation of THP is adapted to model $\lambda^\ast(t)$ instead of $\lambda_k^\ast(t)$:
\begin{equation}
\lambda^\ast(t;\btheta_T) = \boldsymbol{1}^\mathsf{T} \left[\sigma_S\left( \boldsymbol{w}_t \frac{t-t_{i-1}}{t_{i-1}} + \mathbf{W} \boldsymbol{h} + \boldsymbol{b}\right)\right],
\label{eq:thp}
\end{equation}
where $\boldsymbol{w}_t \in \mathbb{R}^{C}_+$, $\mathbf{W} \in \mathbb{R}^{C \times d_h}$, $\boldsymbol{b} \in \mathbb{R}^{d_1}$, and $\btheta_T = \{\mathbf{W}, \boldsymbol{w}_t, \boldsymbol{b}, \btheta_h \}$.
\textit{RMTPP}\textbf{+}. We retain the original definition of $\lambda^\ast(t)$ proposed in RMTPP:
\begin{equation}
\lambda^\ast(t;\btheta_T) = \text{exp}\Big(w_t (t-t_{i-1}) + \boldsymbol{w}_h^\mathsf{T} \boldsymbol{h} + b \Big),
\end{equation}
where $w_t \in \mathbb{R}_+$, $\boldsymbol{w}_h \in \mathbb{R}^{d_h}$, $b \in \mathbb{R}$, and $\btheta_T = \{w_t, \boldsymbol{w}_h, b, \btheta_h\}$. A major difference between the original formulation of RMTPP and our approach lies in $p^\ast(k|\tau; \btheta_M)$ being defined by (\ref{eq:pmf_cond_dis}), which alleviates the original model assumption of marks being independent of time given $\mathcal{H}_t$.
\textbf{Density-based parametrizations.} The decomposition of the joint density in expression (\ref{eq:joint_param}) has previously been considered by \citep{IntensityFree} with the LogNormMix (LNM) model. However, similar to RMTPP, this model assumes independence of marks on time, given $\mathcal{H}_t$. In \textit{LNM}$^{+}$, we relax this assumption by using expression (\ref{eq:pmf_cond_dis}) to parametrize $p^\ast(k|t)$, while maintaining a mixture of log-normal distributions for $f^\ast(\tau)$:
\begin{equation}
f^\ast(\tau; \btheta_T) = \sum_{m=1}^M p^\ast(m) \frac{1}{\tau \sigma_m \sqrt{2\pi}}\text{exp}\Big(-\frac{(\text{log}~\tau -\mu_m)^2}{2\sigma_m^2}\Big),
\label{eq:log_norm_dis}
\end{equation}
where ${\mu_m = (\mathbf{W}_\mu \boldsymbol{h} + \boldsymbol{b}_\mu)_m}$, $p^\ast(m)= \sigma_{\text{so}} \big(\mathbf{W}_p \boldsymbol{h} + \boldsymbol{b}_p\big)_m$, ${\sigma_m = \text{exp}(\mathbf{W}_\sigma \boldsymbol{h} + \boldsymbol{b}_\sigma)_m}$ with $\mathbf{W}_p, \mathbf{W}_\mu, \mathbf{W}_\sigma \in \mathbb{R}^{M \times d_h}$ and $\boldsymbol{b}_p, \boldsymbol{b}_\mu, \boldsymbol{b}_\sigma \in \mathbb{R}^M$, $M$ being the number of mixture components. Here $\btheta_T = \{\mathbf{W}_\mu, \mathbf{W}_p, \mathbf{W}_\sigma, \boldsymbol{b}_\mu,\boldsymbol{b}_p, \boldsymbol{b}_\sigma, \btheta_h \}$. Again, parametrizing the mark distribution using (\ref{eq:pmf_cond_dis}) relaxes the original assumption of marks being independent of time given $\mathcal{H}_t$.
\textbf{Compensator-based parametrizations.} Integrating compensator-based neural MTPP models into our framework requires to define $\Lambda^\ast(t;\btheta_T)$, this way retrieving the decomposition in the r.h.s. of (\ref{eq:lambda_param}). Specifically, we extend the improved marked FullyNN model (FNN) \citep{FullyNN,EHR} into \textit{FNN}$^{+}$, that models $\Lambda^\ast(t)$ and $p^\ast(k|t)$, instead of $\Lambda_k^\ast(t)$:
\begin{equation}
\Lambda^\ast(t; \btheta_T) = G^\ast(t) - G^\ast(t_{i-1}),
\end{equation}
\begin{equation}
G^\ast(t) = \boldsymbol{1}^\mathsf{T} \left[\sigma_{S}\big(\mathbf{W}(\sigma_{GS}\big(\boldsymbol{w}_t (t-t_{i-1}) + \mathbf{W}_h \boldsymbol{h} + \boldsymbol{b}_1 \big) + \boldsymbol{b}_2 \big)\right],
\label{fullyNN}
\end{equation}
where $\mathbf{W} \in \mathbb{R}^{C \times d_1}$, $\boldsymbol{w}_t, \boldsymbol{b}_1 \in \mathbb{R}^{d_1}$, $\mathbf{W}_h \in \mathbb{R}^{d_1 \times d_h}$, $\boldsymbol{b}_2 \in \mathbb{R}^{C}$, and $\sigma_{GS}$ is the Gumbel-softplus activation function. Here, $\btheta_T = \{ \mathbf{W}, \mathbf{W}_h, \boldsymbol{w}_t, \boldsymbol{b}_1, \boldsymbol{b}_2, \btheta_h\}$. Similarly to the previous models, we use (\ref{eq:pmf_cond_dis}) to define $p^\ast(k|t)$.
\textbf{Training different history encoders}. The different functions defined in this section, $p^\ast(k|\tau;\btheta_M)$, $f^\ast(\tau; \btheta_T)$, $\lambda^\ast(t;\btheta_T)$, and $\Lambda^\ast(t;\btheta_T)$, share a common set of parameters $\btheta_h$ through a common history representation $\boldsymbol{h}$. To enable fully disjoint modeling and training of the time and mark predictive functions, we define two distinct history representations:
\begin{align}
&\bh_T = \text{ENC}_T\left[\{\boldsymbol{e}_1,...,\boldsymbol{e}_{i-1}\}; \btheta_{T,h}\right] \in \mathbb{R}^{d_h^t} ~~\text{and}~~ \bh_M = \text{ENC}_M\left[\{\boldsymbol{e}_1,...,\boldsymbol{e}_{i-1}\}; \btheta_{M,h}\right] \in \mathbb{R}^{d_h^m},
\end{align}
where $\text{ENC}_T(\cdot~;\btheta_{T,h})$ and $\text{ENC}_M(\cdot~;\btheta_{M,h})$ are the \textit{time} and \textit{mark} history encoders, respectively, while $\btheta_{T,h}$ and $\btheta_{M,h}$ represent the sets of \textit{disjoint} learnable parameters of the two encoders. By using $\bh_T$ for $f^\ast(\tau;\btheta_T)$, $\lambda^\ast(t;\btheta_T)$ and $\Lambda^\ast(t;\btheta_T)$, and $\bh_M$ for $p^\ast(k|\tau;\btheta_M)$\footnote{$\btheta_h$ is now replaced by $\btheta_{T,h}$ in $\btheta_T$, and by $\btheta_{M,h}$ in $\btheta_M$.}, we have defined completely disjoint parametrizations of the decompositions in (\ref{eq:lambda_param}) and (\ref{eq:joint_param}). Using separate history encoders further enables the model to capture information from past event occurrences that are relevant to the time and mark prediction tasks separately. In this paper, without loss of generality, we compute $\bh_T$ and $\bh_M$ by training two GRU encoders that sequentially process the set of event representations in $\{\boldsymbol{e}_1,...,\boldsymbol{e}_{i-1}\}$.
%%%%%%%%%%%%%%%%%
\begin{comment}
Through the common history representation $\boldsymbol{h}$, the parametrizations of $p^\ast(k|\tau;\btheta_M)$ and $f^\ast(\tau; \btheta_T)$/$\lambda^\ast(t;\btheta_T)$/$\Lambda^\ast(t;\btheta_T)$ described in the previous section still share a common set of parameters $\btheta_h$. To define parametrizations for the time and mark prediction tasks that depend on disjoint sets of learnable parameters, we generate two history representations:
\begin{align}
&\bh_T = \text{ENC}_T\left[\{\boldsymbol{e}_1,...,\boldsymbol{e}_{i-1}\}; \btheta_{T,h}\right] \in \mathbb{R}^{d_h^t} \label{eq:h_t}\\
&\bh_M = \text{ENC}_M\left[\{\boldsymbol{e}_1,...,\boldsymbol{e}_{i-1}\}; \btheta_{M,h}\right] \in \mathbb{R}^{d_h^m}, \label{eq:h_m}
\end{align}
where $\text{ENC}_T(\cdot~;\btheta_{T,h})$ and $\text{ENC}_M(\cdot~;\btheta_{M,h})$ are the \textit{time} and \textit{mark} encoders, respectively, while $\btheta_{T,h}$ and $\btheta_{M,h}$ represent the sets of \textit{disjoint} learnable parameters of the two encoders. By using $\bh_T$ to parametrize $f^\ast(\tau;\btheta_T)$/$\lambda^\ast(t;\btheta_T)$/$\Lambda^\ast(t;\btheta_T)$ and $\bh_M$ to parametrize $p^\ast(k|\tau;\btheta_M)$, we now define disjoint parametrizations of the decompositions in (\ref{eq:joint_param}), (\ref{eq:lambda_param}), and (\ref{eq:Lambda_param}). Using separate history encoders further enables the model to capture information from past event occurrences that are relevant to the time and mark prediction tasks separately. In this paper, w.l.o.g., we compute $\bh_T$ and $\bh_M$ by training two GRU encoders that sequentially process the set of event representations in $\{\boldsymbol{e}_1,...,\boldsymbol{e}_{i-1}\}$.
\end{comment}
%%%%%%%%%%%%%%%%%
%Given disjoint parametrizations of the decompositions in (\ref{eq:joint_param}), (\ref{eq:lambda_param}) or (\ref{eq:Lambda_param}), we learn the parameters of the model via Maximum Likelihood Estimation (MLE). Provided $f^\ast(\tau; \btheta_T)$, $p^\ast(k|\tau;\btheta_M)$ and a set of training sequences $\mathcal{S} = \{\mathcal{S}_1,..., \mathcal{S}_L\}$ each containing $n_l$ events observed in $[0,T]$ for $l=1,...,L$, the average sequence negative log-likelihood (NLL) writes \citep{rasmussen2018lecture}:
\subsection{Disjoint training of the time and mark tasks.}
\label{sec:disjoint}
Let us model $f^\ast(\tau;\btheta_T)$ from (\ref{eq:log_norm_dis}) and $p^\ast(k|\tau;\btheta_M)$ from (\ref{eq:pmf_cond_dis}), using distinct history encoders such that $\btheta_T$ and $\btheta_M$ are disjoint set of trainable parameters. By injecting these expressions in (\ref{eq:NLL}), we find that the NLL now is a sum over two disjoint objectives $\mathcal{L}_T(\btheta_T, \mathcal{S})$ and $\mathcal{L}_M(\btheta_M, \mathcal{S})$, i.e.
\begin{align}
\mathcal{L}(\btheta_T,\btheta_M; \mathcal{S}) =
\underbrace{-\frac{1}{L} \sum_{l=1}^{L} \left[\left[\sum_{i=1}^{n_l}
\text{log}~f^\ast(\tau_{l,i}; \btheta_T) \right] - \text{log}(1-F^\ast(T; \btheta_T))\right]}_{\mathcal{L}_T(\btheta_T, \mathcal{S})}
\underbrace{-\frac{1}{L} \sum_{l=1}^{L} \left[\sum_{i=1}^{n_l}
\text{log}~p^\ast(k_{l,i}|\tau_{l,i}; \btheta_M) \right]}_{\mathcal{L}_M(\btheta_M, \mathcal{S})},
\label{eq:NLL_disjoint}
\end{align}
meaning that the associated tasks $\mathcal{T}_T$ and $\mathcal{T}_M$ can be learned separately.
This contrasts with previous works, such as \citep{IntensityFree} and \citep{TransformerHawkes}, in which shared parameters between $f^\ast$ and $p^\ast$ does not allow for disjoint training of (\ref{eq:NLL}). In our implementation, we minimize expression (\ref{eq:NLL_disjoint}) through a single pipeline by specifying different early-stopping criteria for $\mathcal{L}_T(\btheta_T, \mathcal{S})$ and $\mathcal{L}_M(\btheta_T, \mathcal{S})$. Note that a similar decomposition can be obtained from any of the parametrizations presented in Sections \ref{sec:neurips:theoretical} and \ref{sec:existing_TPP}. Finally, we would like to emphasize that disjoint training of tasks $\mathcal{T}_T$ and $\mathcal{T}_M$ through (\ref{eq:NLL_disjoint}) does \textbf{not} imply independence of arrival-times and marks given the history. In fact, in our parametrizations, this dependency remains systematically captured by (\ref{eq:pmf_cond_dis}).
\begin{comment}
A straightforward approach to separately train $f^\ast(\tau; \btheta_T)$ and $p^\ast(k|\tau; \btheta_M)$ would involve minimizing the $\mathcal{L}_T(\btheta_T, \mathcal{S})$ and $\mathcal{L}_T(\btheta_T, \mathcal{S})$ objectives in parallel through two distinct pipelines, combining the two functions after training to assemble the full model.
Neural TPP models are typically trained via Maximum Likelihood Estimation (MLE) \citep{rasmussen2018lecture}. Let denote the time and mark predictive models . For a set of training sequences $\mathcal{S} = \{\mathcal{S}_1,..., \mathcal{S}_L\}$ each containing $n_l$ events where the arrival times are observed in $[0,T]$ for $l=1,...,L$, the average sequence negative log-likelihood (NLL) is expressed as:
\end{comment}
%We show in Appendix \ref{sec:add_nll} how the NLL is formulated when the time component involves $\lambda^\ast(t;\btheta_T)$ or $\Lambda^\ast(t;\btheta_T)$. The NLL objective function involves two terms, the NLL-T and NLL-M terms, each respectively evaluating the time and mark predictive model. However, thanks to the disjoint set of parameters $\btheta_T$ and $\btheta_M$, the objective function is separable, and the two models can be trained independently.
%This objective can be broken down into the sum of two separable objectives, the NLL-T and NLL-M terms, each respectively characterizing a model's performance on the time and mark prediction tasks. Thanks to the disjoint parametrization of $f^\ast(\tau; \btheta_T)$ and $p^\ast(k|\tau, \btheta_M)$, each of these sub-objectives can now be trained independently for varying number of iterations until convergence. This is in contrast to previous works, such as \citep{IntensityFree} and \citep{TransformerHawkes}, in which common parameters shared between $f^\ast(\tau;\btheta_T)$ and $p^\ast(k|\tau;\btheta_M)$ does not allow separate learning.
%A straightforward approach to train $f^\ast(\tau; \btheta_T)$ and $p^\ast(k|\tau, \btheta_M)$ independently would consist in minimizing the NLL-T and NLL-M terms in parallel through two distinct pipelines, and to combine the two functions ad-hoc to retrieve the full model. However, in our implementation, we minimize (\ref{eq:NLL_dijoint}) through a single pipeline by specifying different early-stopping training criteria for the NLL-T and NLL-M terms. We provide additional training details in Section \ref{section:experiments}.
\begin{comment}
In such scenario, one task may start overfitting, while the other would require additional training. Conversely, the separable training objective in (\ref{eq:NLL_dijoint} ensures that each task is trained for an adequate number of iterations.
\end{comment}
\section{Related Work}
\textbf{Neural MTPP models.} To address the limitations of simple parametric MTPP models \citep{Hawkes, SelfCorrecting}, prior studies focused on designing more flexible approaches by leveraging recent advances in deep learning. Based on the parametrization chosen, these neural MTPP models can be generally classified along three main axis: density-based, intensity-based, and compensator-based. Intensity-based approaches propose to model the trajectories of future arrival-times and marks by parametrizing the marked intensities $\lambda_k^\ast(t)$. In this line of work, past event occurrences are usually encoded into a history representation using RNNs \citep{RMTPP, NeuralHawkes, guo2018noisecontrastive, turkmen2019fastpoint, biloš2020uncertainty, zhu2020adversarial} or self-attention (SA) mechanisms \citep{TransformerHawkes, SelfAttentiveHawkesProcesses, zhu2021deepfourierkernelselfattentive, yang2022transformer, li2023smurfthp}. However, parametrizations of the marked intensity functions often come at the cost of being unable to evaluate the log-likelihood in closed-form, requiring Monte Carlo integration. This consideration motivated the design of compensator-based approaches that parametrize $\Lambda_k^\ast(t)$ using fully-connected neural networks \cite{FullyNN}, or SA mechanisms \citep{EHR}, from which $\lambda_k^\ast(t)$ can be retrieved through differentiation. Finally, density-based approaches aim at directly modeling the joint density of (inter-)arrival times and marks $f^\ast(\tau,k)$. Among these, different family of distributions have been considered to model the distribution of inter-arrival times \citep{xiao2017gaussian, lin2021empirical}. Notably \citet{IntensityFree} relies on a mixture of log-normal distributions \citep{IntensityFree} to estimate $f^\ast(\tau)$, a model that then appeared in subsequent works \citep{sharma2021identifyingcoordinatedaccountssocial, gupta2021intermittent}. However, the original work of \citet{IntensityFree} assumes conditional independence of inter-arrival times and marks given the history, which is alleviated in \citep{Waghmare}. Nonetheless, a common thread of these parametrizations is that they explicitly enforce parameter sharing between the time and mark prediction tasks. As we have shown, this often leads to the emergence of conflicting gradient during training, potentially hindering model performance. For an overview of neural MTPP models, we refer the reader to the works of \citep{SchurSurvey}, \citep{LinGenerative} and \citep{bosser2023predictive}.
\textbf{Conflicting gradients in multi-task learning.} Diverse approaches have been investigated in the literature to improve interactions between concurrent tasks in multi-task learning problems, thereby boosting performance for each task individually. In this context, a prominent line of work, called \textit{gradient surgery methods}, focuses on balancing the different tasks at hand through direct manipulation of their gradients. These manipulations either aim at alleviating the differences in gradient magnitudes between tasks \citep{chen2018gradnorm, sener2018multitasklearningmultiobjectiveoptimization, liu2021impartialmultitask}, or the emergence of conflicts \citep{sinha2018gradientadversarialtrainingneural, maninis2019attentivesingletaskingmultipletasks, yu2020gradient, chen2020just,wang2020gradientvaccineinvestigatingimproving, liu2021conflictaversegradientdescentmultitask, javaloy2022rotogradgradienthomogenizationmultitask}. Alternative approaches to task balancing have been explored based on different criteria, such as task prioritization \citep{guo2018dynamictask}, uncertainty \citep{kendall2018multitasklearningusinguncertainty}, or learning pace \citep{liu2019endtoendmultitasklearningattention}.
\begin{comment}
Another line of work refers to task clustering methods, which aim at identifying the tasks that should be learned jointly \citep{thrun1996discoveringstructures, zamir2018taskonomydisentanglingtasktransfer, standley2020taskslearnedmultitasklearning, shen2021variationalmultitasklearninggumbelsoftmax, fifty2021efficientlyidentifyingtaskgroupings}. At the network level, multi-task learning methods can be partitioned into two main groups. Hard parameter sharing methods denotes methods that share common parameters between multiple tasks \citep{kokkinos2017ubernettraininguniversalconvolutional, long2017learningmultipletasksmultilinear, bragman2019stochasticfiltergroupsmultitask}. Conversely, soft sharing methods regroups methods where each task possess its own set of parameters, although a sharing mechanism enables communication across tasks \citep{misra2016crossstitchnetworksmultitasklearning, ruder2019latentmultitaskarchitecturelearning, gao2019nddrcnnlayerwisefeaturefusing, gao2020mtlnastaskagnosticneuralarchitecture}.
\end{comment}
Our methodology relates more to branched architecture search approaches \citep{guo2020learningbranchmultitasklearning, bruggemann2020automatedsearchresourceefficientbranched, shi2023recon}, where the aim is set on dynamically identifying which layers should or should not be shared between tasks based on a chosen criterion, e.g. the proportion of conflicting gradients. Specifically, \citet{shi2023recon} recently showed that gradient surgery approaches for multi-task learning objectives, such as GradDrop \citep{chen2020just}, PCGrad \citep{yu2020gradient}, CAGrad \citep{liu2021conflictaversegradientdescentmultitask}, and MGDA \citep{sener2018multitasklearningmultiobjectiveoptimization}, cannot effectively reduce the occurrence of conflicting gradients during training. Instead, they propose to address task conflicts directly from the root by turning shared layer into task-specific layers if they experience conflicting gradients too frequently. Inspired by their success, our framework follows a similar approach: the time and mark prediction tasks are parametrized on disjoint set of trainable parameters to avoid conflicts from the root during training. We want to emphasize that our goal is not to propose a general-purpose gradient surgery method to mitigate the negative impact of conflicting gradients. Instead, we want to demonstrate that conflicts can be avoided altogether during the training of neural MTPP models by adapting their original parametrizations.
\begin{comment}
These approaches typically enforce the distributions of future arrival-times and marks to depend on a common set of parameters, which may trigger negative transfer learning and precludes separate learning. Our work removes this constraint by proposing disjoint parametrizations of their decompositions, allowing to learn each task separately.
\citep{biloš2020uncertainty} rely on Gaussian processes to model $p^\ast(k|\tau)$ allowing to quantify uncertainty in future mark predictions.
\end{comment}
\section{Experiments}
\label{section:experiments}
\textbf{Datasets and baselines.} We conduct an experimental study to assess the performance of our framework in training the time and mark prediction tasks from datasets composed of multiple event sequences. Specifically, we explore the various novel neural TPP parametrizations enabled by our framework, as detailed in Section \ref{section:method}. These are compared to their original parametrizations\footnote{For the remainder of this paper, these models will be referred to as "base models".}. We use five real-world marked event sequence datasets frequently referenced in the neural MTPP literature: \textbf{LastFM} \citep{hidasi2012}, \textbf{MOOC}, \textbf{Reddit} \citep{kumar2019predicting}, \textbf{Github} \citep{dyrep}, and \textbf{Stack Overflow} \citep{RMTPP}. We provide descriptions and summary statistics for all datasets in Appendix \ref{sec:datasets}. We consider as base models \textbf{THP} \citep{TransformerHawkes}, \textbf{SAHP} \citep{SelfAttentiveHawkesProcesses}, \textbf{RMTPP} \citep{RMTPP}, FullyNN (\textbf{FNN}) \citep{FullyNN}, LogNormMix (\textbf{LNM}) \citep{IntensityFree}, and SMURF-THP (\textbf{STHP}). To highlight the different components of our framework that lead to performance gains, we introduce the following settings:
(1) \textit{Shared History Encoders and Disjoint Decoders}. A common history embedding, denoted as $\boldsymbol{h}$ is used, while the two functional terms from equations (\ref{eq:lambda_param}) and (\ref{eq:joint_param}) are modeled separately as detailed in Section \ref{sec:existing_TPP}. Here, the functions $f^\ast(\tau;\btheta_T)$, $\lambda^\ast(t;\btheta_T)$, $\Lambda^\ast(t;\btheta_T)$ and $p^\ast(k|\tau;\btheta_M)$ share common parameters via $\boldsymbol{h}$. Models trained in this setting are indicated with a "\textbf{+}" sign, e.g. THP\textbf{+}.
(2) \textit{Disjoint History Encoders and Disjoint Decoders}. In contrast to the previous configuration, distinct history embeddings, \(\bh_T\) for time and \(\bh_M\) for marks, are used to define \(f^\ast(\tau; \btheta_T)\), \(\lambda^\ast(t; \btheta_T)\), \(\Lambda^\ast(t; \btheta_T)\), and \(p^\ast(k|\tau; \btheta_M)\) separately. This separation allows for the independent training of the time prediction and mark prediction tasks as described in Section \ref{sec:disjoint}. Models trained within this setting are labeled with a "\textbf{++}" symbol, e.g., THP\textbf{++}.
Compared to the base models, these configurations allow us to assess the impacts of (1) isolating the parameters for the decoders in the time and mark prediction tasks, and (2) using distinct history embeddings for each task, enabling fully disjoint training. A graphical illustration of these configurations is shown in Figure \ref{fig:setups}. We will often refer to these setups as base, base\textbf{+}, and base\textbf{++} throughout the text. The distinction between LNM (RMTPP) and LNM\textbf{+} (RMTPP\textbf{+}) stems from the modeling of the PMF of marks using our model in (\ref{eq:pmf_cond_dis}), which relaxes the conditional independence assumption inherent in the base model. To maintain a fair comparison, we ensure that each configuration controls for the number of parameters, keeping them roughly equivalent across settings to confirm that any observed performance improvements are not merely due to increased model capacity. Finally, all models are trained to minimize the average NLL given in (\ref{eq:NLL_disjoint})\footnote{Note that for the base and base\textbf{+} methods, the two terms in (\ref{eq:NLL_disjoint}) are functions on shared parameters.}. We provide further training details in Appendix \ref{sec:training_details}.
\begin{comment}
In all setups, the optimization process can last for a maximum of 500 epochs, and we revert the model parameters to their state with the lowest validation loss after training. Finally, we evaluate the model by computing test metrics on the test sequences of each split. Our framework is implemented in a unified codebase using PyTorch\footnote{https://pytorch.org/} and builds upon the previous work of \citep{EHR}\footnote{https://github.com/babylonhealth/neuralTPPs} and \citep{bosser2023predictive}\footnote{https://github.com/tanguybosser/ntpp-tmlr2023}. All models were trained on a machine equipped with an AMD Ryzen Threadripper PRO 3975WX CPU running at 4.1 GHz and a Nvidia RTX A500 GPU.
\end{comment}
\begin{comment}
\item \textit{Parametrization of the \textbf{d}ecomposition, with a \textbf{s}eparate \textbf{h}istory} (\textbf{-DSH}). Similar to the previous setup, at the exception that now $\bh_T$ and $\bh_M$ are used to parametrize $f^\ast(\tau)$/$\lambda^\ast(t)$/$\Lambda^\ast(t)$ and $p^\ast(k|\tau)$, respectively.
For performance comparison, we consider several density, intensity, and cumulative intensity based models drawn from the neural TPP literature:
\begin{itemize}
\item The Exponential Constant (\textbf{EC}) decoder \citep{TPPReinf} that parametrizes a constant MCIF between two events using a feed-forward network on $\boldsymbol{h}$. The mark distribution is therefore assumed to be independent of time given the history, i.e. $p^\ast(k|\tau) = p^\ast(k)$.
\item The MLP decoder (\textbf{MLP}) \citep{EHR} that extends the EC decoder by allowing the MCIF to evolve between two events.
\item The RMTPP decoder (\textbf{RMTPP}) \citep{RMTPP} that separately parametrizes the GCIF and a mark distribution independent of time given the history.
\item The LogNormMix decoder (\textbf{LNM}) \citep{IntensityFree} that separately parametrizes the conditional density of inter-arrival times as in (\ref{eq:log_norm_dis}) and the mark distribution similarly to RMTPP.
\item The FullyNN decoder (\textbf{FNN}) \citep{FullyNN} that directly parametrizes the cumulative MCIF $\Lambda_k^\ast(t) = \int_{t_{i-1}}^t \lambda_k^\ast(s)ds$ using a feed-forward network. The MCIF is then retrieved through differentiation.
\item A Hawkes decoder (\textbf{Hawkes}) \citep{Hawkes} with exponential kernels.
\end{itemize}
Note that either explicitly or implicitly through the MCIF, all these baselines parametrize the inter-arrival time and mark distributions using a common history embedding $\boldsymbol{h}$. To highlight the benefits of conditioning the mark distribution on time in DCLNM, we also train a time-independent version of (\ref{eq:pmf_cond_dis}), denoted \textbf{DLNM}:
\begin{equation}
p^\ast(k; \btheta_T, \btheta_{M,h}) = \text{Softmax}\left(\mathbf{W}_2 \text{ReLU}\left(\mathbf{W}_1 \bh_M + \boldsymbol{b}_1\right) + \boldsymbol{b}_2 \right).
\label{eq:pmf_cond_dis_dlnm}
\end{equation}
\end{comment}
\begin{figure*}[!t]
\vspace{-1cm}
\centering
\begin{subfigure}[b]{0.32\textwidth}
\caption{Base}
\includegraphics[width=\textwidth]{images/drawings/base.pdf}
\end{subfigure}
\begin{subfigure}[b]{0.32\textwidth}
\caption{Base\textbf{+}}
\includegraphics[width=\textwidth]{images/drawings/base+.pdf}
\end{subfigure}
\begin{subfigure}[b]{0.32\textwidth}
\caption{Base\textbf{++}}
\includegraphics[width=\textwidth]{images/drawings/base++.pdf}
\end{subfigure}
\caption{Graphical representation of the base, "\textbf{+}", and "\textbf{++}" setups.}
\label{fig:setups}
\vspace{-0.3cm}
\end{figure*}
\paragraph{Metrics.}
To evaluate the performance of the different baselines on the time prediction task, we report the $\mathcal{L}_T$ term in (\ref{eq:NLL_disjoint}) computed over all test sequences. Following \citep{dheur2023largescale}, we also quantify the (unconditional) probabilistic calibration of the fitted models by computing the Probabilistic Calibration Error (PCE). Finally, we evaluate the MAE in event inter-arrival time prediction. To this end, we predict the next $\Tilde{\tau}$ as the median of the predicted distribution of inter-arrival times, i.e. $\Tilde{\tau} = (F^\ast)^{-1}(0.5;\btheta_T)$, where the quantile function $(F^\ast)^{-1}(~\cdot~;\btheta_T)$ is estimated using a binary search algorithm.
Similarly, for the mark prediction task, we report the average $\mathcal{L}_M$ term in (\ref{eq:NLL_disjoint}), and quantify the probabilistic calibration of the mark predictive distribution by computing the Expected Calibration Error (ECE) \citep{naeini2015calibration}, and through reliability diagrams \citep{guo2017calibration, kuleshov2018uncertaintiesdeeplearning}. Additionally, by predicting the mark of the next event as $\Tilde{k} = \underset{k \in \mathbb{K}}{\text{argmax }} p^\ast(k|\tau; \btheta_M)$, we can assess the quality of the point predictions by means of various classification metrics. Specifically, we compute the Accuracy@$n$ for values of $n$ in $\{1,3,5\}$ and the Mean Reciprocal Rank (MRR) \citep{mrr} of mark predictions. Lower $\mathcal{L}_T$, $\mathcal{L}_M$, PCE and ECE is better, while higher Accuracy@$n$ and MRR is better.
\subsection{Results and discussion}
\label{sec:discussion}
\begin{comment}
We report in Table \ref{tabular_results} the NLL-T and NLL-M for the base, \textbf{+} and \textbf{++} setups across all datasets. The metrics are averaged over five splits, and one standard error is reported in parenthesis. The PCE, ECE, MRR, and Accuracy@\{1,3,5\} metrics are deferred to the Appendix. From the NLL-M and NLL-T metrics, we note almost systematic improvement when switching from the base models to the \textbf{+} or \textbf{++} setups. This highlights that both tasks benefit from the three properties enabled by our framework: \textit{direct parametrization of the distribution of marks} (base vs. \textbf{+}), \textit{separate history embeddings}, and \textit{independent training} (\textbf{+} vs. \textbf{++}). In particular, the mark prediction task benefits almost systematically from being trained independently in the \textbf{++} setup. However, for the time prediction task, we note that the \textbf{+} setup sometimes yields a lower NLL-T compared to its \textbf{++} counterpart (e.g. THP on MOOC). This suggests that the time prediction task benefits from being trained jointly with the mark through the shared history representation. On Figure \ref{fig:training_curves}, we show the evolution of the training NLL-T and NLL-M terms for THP\textbf{+} and THP\textbf{++} on LastFM. Thanks to independent learning, the NLL-M term in the \textbf{++} setup can be trained for additional epochs once the NLL-T reached convergence, leading to performance gains on the mark prediction task.
\end{comment}
We present the $\mathcal{L}_T$ and $\mathcal{L}_M$ metrics for the base, base\textbf{+}, and base\textbf{++} configurations across all datasets in Table \ref{tabular_results}. The PCE, MAE, ECE, MRR, and Accuracy@{1,3,5} metrics, which reflect the subsequent discussion, are given in Appendix \ref{sec:add_results}.
\textbf{Distinct decoders mitigate gradient conflicts}. Based on the $\mathcal{L}_T$ and $\mathcal{L}_M$ metrics, we note a consistent improvement when moving from the base to the base\textbf{+} setting for THP, SAHP, and FNN. This underscores the benefits of using two distinct decoders for time and mark prediction tasks with base\textbf{+}, leading to improved predictive accuracy compared to the base models. Figure \ref{fig:grad_jd} shows the distribution of $\text{cos } \phi_{TM}$ during training for THP, SAHP and FNN for both base and base\textbf{+} on the LastFM and MOOC dataset, along with the average GMS and TPI for conflicting gradients. \textcolor{blue}{We would like too emphasize that both base and base\textbf{+} share the \textit{same} encoder architecture, which allows for a direct comparison of the distribution of $\text{cos } \phi_{TM}$ between the two settings during training.} Appendix \ref{sec:add_results} provides detailed visualizations for other baselines and datasets. With the base model, a significant proportion of severe conflicts (as indicated by
$\text{cos } \phi_{TM}$ in [-1, -0.5]) is often observed for the shared parameters of both encoder and decoder heads, typically with low GMS values. Additionally, with the base model, the TPI values suggest that these conflicting gradients at the encoder heads predominantly favor $\mathcal{T}_T$ (i.e., TPI > 0.5). In contrast, base\textbf{+} inherently prevents conflicts at the decoder by separating the parameters for each task. During training with base\textbf{+}, there is also a noticeable reduction in the severity of conflicting gradients for the shared encoder parameters, as evidenced by a more concentrated distribution of $\text{cos } \phi_{TM}$ around 0. \textcolor{blue}{Moreover, the TPI values indicate that base\textbf{+} generally achieves a more balanced training between both tasks, which further contributes to enhancing their individual performance. While we note that the GMS values do not consistently improve between the base and base\textbf{+} settings, improvements with respect to $\mathcal{L}_T$ and $\mathcal{L}_M$ suggest that this effect is offset by a reduction in conflicts during training.} Finally, while LNM and RMTPP already avoid decoder conflicts in their base models by decomposing the parameters, explicitly modeling the dependency of marks on time with base\textbf{+} further enhances mark prediction performance.
\begin{comment}
Although our approach also prevents by design positive gradients at the decoder head (in the sense of $\text{cos } \phi_{TM} \geq 0$), our results in Table \ref{tabular_results} suggest that the negative influence of conflicts during training outweigh the positive reinforcement of task alignment.
\end{comment}
\begin{wrapfigure}{r}{0.4\textwidth}
\vspace{-0.5cm}
\includegraphics[width=\linewidth]{images/drawings/training_curves.pdf}
\caption{Validation curves of the $\mathcal{L}_T$ and $\mathcal{L}_M$ components for SAHP\textbf{++} on MOOC.}
\label{fig:training}
\vspace{-0.3cm}
\end{wrapfigure}
\begin{figure}[!t]
\vspace{-1cm}
\centering
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer/lastfm_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer/mooc_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\caption{\textcolor{blue}{Distribution of $\text{cos } \phi_{TM}$ during training at the encoder (ENC) and decoder (DEC) heads for THP, SAHP and FNN in the base and base\textbf{+} setup on LastFM and MOOC. "B" and "+" refer to the base and base\textbf{+} models, respectively, and the distribution is obtained by pooling the values of $\phi_{TM}$ over 5 training runs. As the decoders are disjoint in the base\textbf{+} setting, note that $\text{cos } \phi_{TM}$ is not defined.}}
\label{fig:grad_jd}
\vspace{-0.3cm}
\end{figure}
\textbf{Disjoint training enhances mark prediction accuracy}. Returning to Table \ref{tabular_results}, moving from the base\textbf{+} to the base\textbf{++} setting often leads to improvements in the $\mathcal{L}_M$ metric for most baselines, while the $\mathcal{L}_T$ metric generally remains comparable between the two configurations. Although conflicting gradients are typically reduced when moving from base to base\textbf{+}, this pattern indicates that the residual conflicts primarily hinder the mark prediction task. In contrast, base\textbf{++} effectively eliminates these conflicts by using distinct history representations for each task. A significant advantage of base\textbf{++} is that it allows one task to continue training after the other has reached convergence. For example, Figure \ref{fig:training} illustrates the validation losses $\mathcal{L}_T$ and $\mathcal{L}_M$ for SAHP++ on MOOC. Thanks to disjoint training, the $\mathcal{L}_M$ metric can be further optimized for additional epochs after training of the $\mathcal{L}_T$ metric ceases due to overfitting, thus achieving gains in mark prediction performance. This feature is absent in base and base\textbf{+}, where both $\mathcal{L}_M$ and $\mathcal{L}_T$ metrics rely on a shared set of parameters. In Figure \ref{fig:training}, $\btheta_T$ is fixed after the vertical red line, resulting in a constant validation $\mathcal{L}_T$ for the remaining training epochs of the model.
\textcolor{blue}{Finally, referring back to the preliminary experiment on Figure \ref{fig:toy_grad}, the results with respect to $\mathcal{L}_T$ and $\mathcal{L}_M$ in Table \ref{tabular_results} suggest that conflicting gradients are mostly detrimental to model performance if (1) they are in great proportion during training (i.e. high CG) and (2) if they are associated to low GMS values. For instance, on Figure \ref{fig:toy_grad}, THP and SAHP exhibit high CG associated to high GMS values, whereas the remaining models conversely show a more balanced CG, but with lower GMS values. We find that model performance often improves in both these scenarios when preventing conflicting gradients altogether in the base\textbf{++} setting.}
\begin{comment}
\begin{figure*}[!t]
\centering
\centering
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/training_curves/training_curves.png}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=0.85\textwidth]{images/tsne/mooc_filtered.pdf}
\end{subfigure}
\vspace{-0.1cm}
\caption{\textbf{Left.} \textbf{Right.} t-SNE embeddings of the history representations learned by THP (left), and by SAHP (right) for the last events of the test sequences in the MOOC dataset. $\bh_T$ and $\bh_M$ are learned in the "\textbf{++}" setup, while $\boldsymbol{h}$ is learned in the base setup.}
\label{fig:training_tsne}
\end{figure*}
\end{comment}
\begin{comment}
Training of the time and mark prediction tasks are interrupted at different epochs (vertical red lines). Once both tasks finished training, the parameters $\btheta_T$ and $\btheta_M$ are resumed to their respective state of lowest validation loss (red cross)
\end{comment}
%This shows that independent training of $f^\ast(t)$ and $p^\ast(k|\tau)$, as enabled by the \textbf{+} setup, lead to better accuracy than training the factorization of $\lambda^\ast_k(t)$ (and likewise, the factorization of $\Lambda_k^\ast(t)$) into $f^\ast(t)$ and $p^\ast(k|\tau)$.
%A similar conclusion can be drawn from the NLL-M improvements between the \textbf{+}I and \textbf{+} setups for THP, SAHP and FNN.
%This highlights that parametrizing and training both tasks independently mostly benefits to the mark prediction task, while little affecting the performance of the model on the time prediction task. When jointly optimizing the factorization as in the \textbf{+} setup, this observation further suggests the appearance of the undesirable effect of \textit{negative transfer} \citep{crawshaw2020multitask} during training, i.e. the shared history representation tends to benefit to the time prediction task, while hurting performance in the mark prediction task. In contrast, models trained in the \textbf{++} setup can leverage history representations relevant to each task separately, hereby mitigating negative transfer. Nonetheless, we note that FNN is little impacted by the \textbf{++} setup, and that the low number of marks in the Retweets dataset ($\mathbb{K}=3$) leads to small differences on the NLL-M between setups.
\begin{comment}
\begin{figure*}[ht!]
\centering
\begin{subfigure}[b]{0.37\textwidth}
\includegraphics[width=\textwidth]{images/distributions/mooc_filtered_gru_sahp_temporal_with_labels_bis.png}
\end{subfigure}
\begin{subfigure}[b]{0.37\textwidth}
\includegraphics[width=\textwidth]{images/distributions/mooc_filtered_gru_temporal_with_labels_gru_temporal_with_labels_sep-sahp-mix_separate_bis.png}
\end{subfigure}
\vspace{-0.2cm}
\caption{Evolution of $f^\ast(t)$ and $p^\ast(k|t)$ as a function of $t$ for SAHP (left) and SAHP\textbf{++} (right) on a test sequence of the MOOC dataset. Events (bottom row) are colored per marks. The upper plot of $p^\ast(k|t)$ is a cumulative plot, i.e. at any given $t$, the \textit{height} of a colored area corresponds to the probability of observing a mark of the associated color. We only show the probability of the observed marks, and $f^\ast(t)$ is scaled by 1000.}
\label{fig:joint_dis_plot}
\end{figure*}
\end{comment}
\begin{figure*}[!t]
\vspace{-1cm}
\centering
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/LastFM_cal1bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/LastFM_1bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/LastFM_cal2bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/LastFM_2bis.pdf}
\end{subfigure}
\vspace{-0.1cm}
\caption{\textcolor{blue}{Reliability diagrams of $p^\ast(k|\tau;\btheta)$ (left) and $f^\ast(\tau;\btheta)$ (right) for the base models and their "\textbf{++}" extensions on the LastFM dataset. Frequency and accuracy aligning with the black diagonal correspond to perfect calibration. The reliability diagrams are averaged over 5 splits, and error bars refer to the standard error.}}
\label{fig:reliability_diag}
\vspace{-0.3cm}
\end{figure*}
\input{results_tabular_main}
\textbf{Reliability Diagrams}. Figure \ref{fig:reliability_diag} presents the reliability diagrams for the predictive distributions of arrival-times and marks for base and base\textbf{++} on LastFM. The diagrams show that the base\textbf{++} models are generally better calibrated than their base counterparts, as evidenced by the bin accuracies aligning more closely with the diagonal. This improvement aligns with the results in Table \ref{tabular_results}, where a lower $\mathcal{L}_M$ indicates better accuracy. However, improvements in the calibration of arrival times between base and base\textbf{++} models are less noticeable, suggesting that conflicting gradients during training predominantly affect the mark prediction task. We provide reliability diagrams for other baselines and datasets in Appendix \ref{sec:add_results}.
\begin{comment}
\textbf{Evolution of the mark distribution.} Figure \ref{fig:joint_dis_plot} shows the evolutions of the predictive $f^\ast(t)$ and $p^\ast(k|t)$ as a function of $t$ for a portion of a test sequence on the MOOC dataset. As observed, SAHP\textbf{++} accurately captures the dynamic of mark occurrences by quickly raising the probability of observing the correct next mark after each event. Conversely, the base SAHP model struggles to catch the complex temporal evolution of the mark distribution.
\end{comment}
\begin{comment}
\textbf{Learned history embeddings}. On Figure \ref{fig:training_tsne}, we compare the distributions of $\bh_T$ and $\bh_M$ learned in the "\textbf{++}" setup to the distribution of $\boldsymbol{h}$ learned in the base setup. The history representations are taken for the last event in the test sequences of the MOOC dataset, and we use t-SNE \citep{tsne} with a perplexity of 5 to reduce dimensionality. In the "\textbf{++}" setup, an interesting observation relates to the models learning to map the history to different representations for the time and mark prediction tasks, allowing each task to capture relevant information from the past of the process separately. Conversely, the history representations $\boldsymbol{h}$ learned in the base setup are forced to be relevant to both tasks simultaneously, creating an overlap with $\bh_M$ and $\bh_T$.
\end{comment}
%Through $\bh_M$, the \textbf{++} models can capture information from the past of a process that is relevant to the mark prediction task only. In contrast, this flexibility is not granted in the \textbf{+} setup, as $\boldsymbol{h}$ is shared between both tasks. We provide additional visualizations on other datasets in Appendix \ref{sec:add_results}. \textbf{Update text}
\textbf{Isolating the impact of conflicts on performance.} Our experiments reveal that the base\textbf{+} and base\textbf{++} settings result in a decrease of conflicting gradients and to enhanced performance with respect to the time and mark prediction tasks. However, for THP, SAHP and FNN, these settings do not enable us to disentangle performance gains brought by a decrease in conflicts from the ones brought by modifications of the decoder architecture. To address this limitation, we introduce the following two settings based on the duplicated model approach of Section \ref{sec:neurips:theoretical}:
(1) \textit{Shared History Encoders and Duplicated Decoders}. The functions $\lambda^\ast(t;\btheta_T)$/$\Lambda^\ast(t; \btheta_T)$ and $p^\ast(k|t;\btheta_M)$ are obtained through (\ref{eq:double}) from two identical parametrizations of the same base decoder.
However, similar to the base\textbf{+} setting, a common history encoder $\bh$ is used, meaning that these functions still share parameters via $\btheta_h$. Models trained in this setting are indicated with a "-D" sign, e.g. THP-D.
(2) \textit{Disjoint History Encoders and Duplicated Decoders}. This setting differs from the previous one in the use of two distinct history embeddings \(\bh_T\) and \(\bh_M\) in (\ref{eq:double}), implying that $\lambda^\ast(t;\btheta_T)$/$\Lambda^\ast(t; \btheta_T)$ and $p^\ast(k|t;\btheta_M)$ are now completely disjoint parametrizations. We use the label "-DD" to denote the models trained in this setting, e.g. THP-DD.
In contrast to base\textbf{+} and base\textbf{++}, the base-D and base-DD settings retain the same architecture as the base model, enabling us to directly evaluate the impact of conflicting gradients on performance. In Table \ref{tab:neurips:d_dd_results}, we report the performance with respect to $\mathcal{L}_T$ and $\mathcal{L}_M$ for THP, SAHP, and FNN trained in the base, base-D and base-DD settings. We follow the same experimental setup as before, and maintain the number of parameters comparable between the different settings for fair comparison. We almost systematically observe improvements on both tasks when moving from the base to the base-D or base-DD settings. Moreover, these performance gains are often associated to a decrease in (severe) conflicts during training, as shown on Figure \ref{fig:neurips:grad_dd}. Furthermore, when comparing the results between Tables \ref{tabular_results} and \ref{tab:neurips:d_dd_results}, we note that our base\textbf{+} and base\textbf{++} parametrizations often show improved performance compared to the base-D and base-DD settings, especially on the mark prediction task. This highlights that the benefits of our parametrizations extend beyond the prevention of conflicts to achieve greater predictive performance. We provide more visualizations in Appendix \ref{sec:neurips_app:distirbution_d}.
\begin{figure}[!t]
\vspace{-1cm}
\centering
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer_double/lastfm_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer_double/mooc_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\caption{\textcolor{blue}{Distribution of $\text{cos } \phi_{TM}$ during training at the encoder (ENC) and decoder (DEC) heads for THP, SAHP and FNN in the base and base-D setup on LastFM and MOOC. "B" and "D" refer to the base and base-D models, respectively, and the distribution is obtained by pooling the values of $\phi_{TM}$ over 5 training runs. As the decoders are disjoint in the base-D setting, note that $\text{cos } \phi_{TM}$ is not defined.}}
%\vspace{-0.2cm}
\label{fig:neurips:grad_dd}
\end{figure}
\input{results_tabular_dd}
\section{Conclusion, Limitations, and Future Work}
\label{sec:conclusion}
Learning a neural MTPP model can be essentially interpreted as a two-task learning problem, in which one task is focused on learning a predictive distribution of arrival times, and the other on learning a predictive distribution of event types, known as marks. Typically, most neural MTPP models implicitly require these two tasks to share a common set of trainable parameters. In this paper, we demonstrate that this parameter sharing leads to the emergence of conflicting gradients during training, often resulting in degraded performance on each individual task. To prevent this issue, we introduce novel parametrizations of neural MTPP models that enable separate modeling and training of each task, effectively preventing the occurrence of conflicting gradients. Through extensive experiments on real-world event sequence datasets, we validate the advantages of our framework over the original model configurations, particularly in the context of mark prediction. However, we acknowledge several limitations in our study. Firstly, our focus was solely on categorical marks. Investigating conflicting gradients in more complex scenarios, such as temporal graphs \citep{dyrep, gracious2023dynamicrepresentationlearningtemporal} or spatio-temporal point processes \citep{zhou2023automaticintegrationspatiotemporalneural, zhang2023integrationfreetrainingspatiotemporalmultimodal}, presents a promising avenue for future research. Secondly, our analysis was limited to neural MTPP models trained using the negative log-likelihood. Extending our framework to other proper scoring rules \citep{brehmer2021comparative} is also a potential area for future exploration.
%Learning a neural MTPP model essentially resumes to a two-task learning problem, in which one task involves learning a predictive distribution over arrival times, while the other involves learning a predictive distribution over marks. By design, most neural MTPP models implicitly enforce these two tasks to rely on a common set of trainable parameters. In this paper, we show that parameter sharing leads to the apparition of conflicting gradients during training, often at the cost of degraded model's performance on each individual task. To address this challenge, we propose novel parametrizations of neural MTPP models enabling disjoint training and modeling of each task, hereby preventing the emergence of conflicting gradients. Through extensive experiments on real-world event sequence datasets, we demonstrate the benefits of our disjoint training framework compared to the original model formulations, especially regarding the mark prediction task. Nonetheless, we highlight several limitations in our work. Firstly, we focused exclusively on categorical marks. Studying conflicting gradients in more complex settings, such as temporal graphs or spatio-temporal point processes, would constitute an interesting research direction. Secondly, our analysis concerned exclusively neural MTPP models trained on the negative log-likelihood. Extending our framework to alternative proper scoring rules is also part of our future prospects.
\begin{comment}
Most neural MTPP models implicitly enforce parameter sharing between time and mark predictive distributions. While parameter sharing allows for strength borrowing between tasks, it may also lead to an unwanted effect known as negative transfer. We show that negative transfer materializes during training of MTPP models by means of conflicting gradients during the time and mark prediction tasks.
To address this challenge, Through extensive experiments on real-world event sequence datasets, we demonstrate the benefits of disjoint training for disjoint training of neural MTPP models, especially for the mark prediction task. However, while we showed the effectiveness of our framework for neural MTPP models trained on the NLL, we cannot affirm that conflicting gradients emerge if alternative scoring rules are employed, nor if our framework would lead to performance gains is such scenarios. Additionally, our study focused only on marked processes, for which only two tasks need to be learned. Extending our study to problems with a greater number of tasks (e.g. marked spatio-temporal point processes) would constitute a promising direction of future research. We plan to address these limitations in our future work.
\end{comment}
%Through their parametrizations, most neural MTPP approaches enforce the predictive distributions of future arrival-times and marks to depend on a common of set of learnable parameters. While sharing parameters between tasks can sometimes facilitate the learning of each task individually, it may also cause the undesirable phenomenon of negative transfer learning. In this work, we design a framework introducing novel parametrizations of neural MTPP models, in which the predictive distributions of arrival times and marks can be trained independently. By conducting a series of experiments, we established the benefits of learning neural TPP models within our framework compared to their original parametrizations. Specifically, we highlighted that significant gains can be achieved on the mark prediction task by training it indepently from the time prediction task. As future work, we propose to further extend our framework by studying alternative scoring rules tailored to each prediction task.
\begin{comment}
\subsubsection*{Broader Impact Statement}
In this optional section, TMLR encourages authors to discuss possible repercussions of their work,
notably any potential negative impact that a user of this research should be aware of.
Authors should consult the TMLR Ethics Guidelines available on the TMLR website
for guidance on how to approach this subject.
\subsubsection*{Author Contributions}
If you'd like to, you may include a section for author contributions as is done
in many journals. This is optional and at the discretion of the authors. Only add
this information once your submission is accepted and deanonymized.
\subsubsection*{Acknowledgments}
Use unnumbered third level headings for the acknowledgments. All
acknowledgments, including those to funding agencies, go at the end of the paper.
Only add this information once your submission is accepted and deanonymized.
\end{comment}
\clearpage
\bibliography{main}
\bibliographystyle{tmlr}
\clearpage
\appendix
\section{Additional Forms of the Negative Log-Likelihood}
\label{sec:add_nll}
Consider a dataset $\mathcal{S} = \{\mathcal{S}_1,..., \mathcal{S}_L\}$, where each sequence \(\mathcal{S}_l\) comprises \(n_l\) events with arrival times observed within the interval \([0,T]\) and \(l=1,...,L\). The average sequence negative log-likelihood (NLL) for these sequences can be expressed as a function of the marked intensity $\lambda^\ast(t;\btheta_T)$ or the compensator $\Lambda^\ast(t;\btheta_T)$ as follows:
\begin{equation}
\mathcal{L}(\btheta_T, \btheta_M; \mathcal{S}) = \underbrace{ -\frac{1}{L} \sum_{l=1}^{L}\sum_{i=1}^{n_l}\text{log}~\lambda^\ast(t_{l,i}; \btheta_T) + \int_{t_n}^T \lambda^\ast(s;\btheta_T)ds}_{\mathcal{L}_T(\btheta_T;\mathcal{S})} \underbrace{-\frac{1}{L} \sum_{l=1}^{L}\sum_{i=1}^{n_l} \text{log}~p^\ast(k_{l,i}|\tau_{l,i}; \btheta_M)}_{\mathcal{L}_M(\btheta_M;\mathcal{S})},
\label{eq:NLL_dijoint_lambda}
\end{equation}
and
\begin{equation}
\mathcal{L}(\btheta_T, \btheta_M; \mathcal{S}) = \underbrace{ -\frac{1}{L} \sum_{l=1}^{L} \left[\sum_{i=1}^{n_l}\text{log}~\left[\frac{d}{dt}\Lambda^\ast(t_{l,i}; \btheta_T)\right] \right] + \Lambda(T; \btheta_T))}_{\mathcal{L}_T(\btheta_T;\mathcal{S})} \underbrace{-\frac{1}{L} \sum_{l=1}^{L}\sum_{i=1}^{n_l} \text{log}~p^\ast(k_{l,i}|\tau_{l,i}; \btheta_M)}_{\mathcal{L}_M(\btheta_M;\mathcal{S})}.
\label{eq:NLL_dijoint_Lambda}
\end{equation}
\section{Alternative Scoring Rules}
\label{sec:scoring_rules}
The NLL in (\ref{eq:NLL_before}) has been largely adopted as the default scoring rule for learning MTPP models \citep{SchurSurvey}. However, our framework can be extended even further by using alternative scoring rules other than the NLL for assessing the mark and time prediction tasks. Let $S^f$, $S^m$ and $S^w$ be (strictly) consistent scoring rules for $f^\ast(\tau; \btheta_T)$, $p^\ast(k|\tau;\btheta_M)$ and $1-F^\ast(T;\btheta_T)$, respectively. Given a sequence $\mathcal{S}$ of $n$ events (i.e. $L=1$), the scoring rule
\begin{equation}
\mathcal{L}(\boldsymbol{\theta};\mathcal{S}) =
\underbrace{\sum_{i=1}^n \left[S^f(f^\ast(\tau_i;\btheta_T))\right] + S^w\left(1-F^\ast(T;\btheta_T)\right) }_{\mathcal{L}_T(\btheta_T, \mathcal{S})} + \underbrace{ \sum_{i=1}^n S^p(p^\ast(k_i|\tau_i;\btheta_M))}_{\mathcal{L}_M(\btheta_M, \mathcal{S})},
\label{eq:scores}
\end{equation}
is (strictly) consistent for the conditional joint density $f^\ast(\tau,k;\btheta_T, \btheta_M)$ restricted to the interval $[0,T]$ \citep{brehmer2021comparative}. Using the LogScore for $S^f$, $S^m$ and $S^w$ in \eqref{eq:scores} reduces to the NLL in (\ref{eq:NLL_disjoint}). One can also use other choices tailored to the specific task. For instance, one can choose to use the continuous ranked probability score (CRPS) \citep{gneiting2007strictlyproper} for $S^f$ to evaluate the predictive distribution of inter-arrival times \citep{souhaib}. Similarly, the Brier score \citep{brier} can be used for $S^p$ to evaluate the predictive distribution of marks. Contrary to the \textit{local} property of the LogScore, both the CRPS and the Brier score are \textit{sensitive to distance}, in the sense that they reward predictive distributions that assign probability mass close to the observed realization \citep{gebetsberger2018estimationmethods}. Nonetheless, the choice between local and non-local proper scoring rules has been generally subjective in the literature. While exploring alternative scoring rules to train neural MTPP models is an exciting research direction, we leave it as future work and train the models exclusively on the NLL in (\ref{eq:NLL_disjoint}).
\section{Proof of Corollary 1}
\label{app:proofs}
Consider a base model $\lambda_k^\ast(t;\btheta)$ with trainable parameters $\btheta$, and a disjoint parametrization of $\lambda^\ast(t;\btheta_T)$ and $p^\ast(k|t;\btheta_M)$ obtained in (\ref{eq:double}) from two identical $\lambda_k^\ast(t;\btheta_T)$ and $\lambda^\ast_k(t;\btheta_M)$ with trainable parameters $\btheta_T$ and $\btheta_M$, respectively. The NLL losses for $\btheta$ and $\{\btheta_T, \btheta_M\}$ respectively write
\begin{equation}
\mathcal{L}(\btheta) = \mathcal{L}_T(\btheta) + \mathcal{L}_M(\btheta),
\end{equation}
\begin{equation}
\mathcal{L}(\{\btheta_T, \btheta_M\}) = \mathcal{L}_T(\btheta_T) + \mathcal{L}_M(\btheta_M).
\end{equation}
Suppose that at training iteration $s \in \mathbb{N}$, $\btheta$, $\btheta_T$ and $\btheta_M$ are all initialized with the same $\btheta^{s}$. This implies that $\lambda_k^\ast(t;\btheta)$, $\lambda_k^\ast(t;\btheta_T)$ and $\lambda_k^\ast(t;\btheta_M)$ are all identical. Assuming that $\mathcal{L}_T$ and $\mathcal{L}_M$ are differentiable, the gradient update steps for $\btheta$, $\btheta_T$ and $\btheta_M$ are
\begin{equation}
\btheta^{s+1} = \btheta^{s} - \alpha (\bg_T + \bg_M), \quad \btheta_T^{s+1} = \btheta^{s} - \alpha \bg_T \quad\text{and}\quad \btheta_M^{s+1} = \btheta^{s} - \alpha \bg_M,
\end{equation}
where $\alpha > 0$ is the learning rate and
\begin{equation}
\bg_T = \nabla_{\btheta} \mathcal{L}_T\left(\btheta^{s}\right) = \nabla_{\btheta_T} \mathcal{L}_T\left(\btheta^{s}\right) \quad\text{and}\quad \bg_M = \nabla_{\btheta} \mathcal{L}_M\left(\btheta^{s}\right) = \nabla_{\btheta_M} \mathcal{L}_M\left(\btheta^{s}\right).
\end{equation}
Denoting $\phi_{TM}$ as the angle between $\bg_T$ and $\bg_M$, we have the following corollary of Theorem 4.1. from \citep{shi2023recon}:
\textbf{Corollary 1.} \textit{Assume that} $\mathcal{L}_T$ \textit{and} $\mathcal{L}_M$ \textit{are differentiable, and that the learning rate} $\alpha$ \textit{is sufficiently small. If} $\text{cos } \phi_{TM} < 0$, \textit{then} $\mathcal{L}(\{\btheta_T^{s+1}, \btheta_M^{s+1}\}) < \mathcal{L}(\btheta^{s+1})$.
\begin{proof}
Let us consider the first order Taylor approximations of the total loss $\mathcal{L} = \mathcal{L}_T + \mathcal{L}_M$ near $\btheta^{s+1}$ and $\{\btheta^{s+1}_T, \btheta^{s+1}_M\}$, respectively:
\begin{equation}
\mathcal{L}(\btheta^{s+1}) = \mathcal{L}(\btheta^s) + (\btheta^{s+1} - \btheta^s)^\intercal \bg_T + (\btheta^{s+1} - \btheta^s)^\intercal \bg_M + o(\alpha).
\end{equation}
\begin{equation}
\mathcal{L}(\{\btheta^{s+1}_T, \btheta^{s+1}_M\}) = \mathcal{L}(\btheta^s) + (\btheta_T^{s+1} - \btheta^s)^\intercal \bg_T + (\btheta_M^{s+1} - \btheta^s)^\intercal \bg_M + o(\alpha).
\end{equation}
Taking the difference between $\mathcal{L}(\{\btheta^{s+1}_T, \btheta^{s+1}_M\})$ and $\mathcal{L}(\btheta^{s+1})$ yields
\begin{align}
\mathcal{L}(\{\btheta^{s+1}_T, \btheta^{s+1}_M\}) - \mathcal{L}(\btheta^{s+1}) &= (\btheta_T^{s+1} - \btheta^{s+1})^\intercal \bg_T + (\btheta_M^{s+1} - \btheta^{s+1})^\intercal \bg_M + o(\alpha) \\
&= \alpha\bg_M^\intercal \bg_T + \alpha\bg_T^\intercal \bg_M + o(\alpha) \\
&= 2 \alpha ||\bg_T|| ||\bg_M|| \text{cos } \phi_{TM} + o(\alpha).
\end{align}
Provided that $\alpha$ is sufficiently small, this difference is negative if $\text{cos } \phi_{TM}^d < 0$, where $\phi_{TM}$ is the angle between $\bg_T$ and $\bg_M$.
\end{proof}
\section{Datasets}
\label{sec:datasets}
We use 5 real-world event sequence datasets for our experiments:
\begin{itemize}
\item \textbf{LastFM} \citep{hidasi2012}: Each sequence corresponds to a user listening to music records over time. The artist of the song is the mark.
\item \textbf{MOOC} \citep{kumar2019predicting}: Records of students' activities on an online course system. Each sequence corresponds to a student, and the mark is the type of activity performed.
\item \textbf{Github} \citep{dyrep}: Actions of software developers on the open-source platform Github. Each sequence corresponds a developer, and the mark is the action performed (e.g. fork, pull request,...).
\item \textbf{Reddit} \citep{kumar2019predicting}: Sequences of posts to sub-reddits that users make on the social website Reddit. Each sequence is a user, and the sub-reddits to which the user posts is considered as the mark.
%\item \textbf{Retweets} \citep{NeuralHawkes}: Each sequence is a stream of retweets following an original tweet on the platform Twitter. The mark is taken as the category (i.e. "small", "medium", "large") to which the tweeter belongs.
\item \textbf{Stack Overflow} \citep{RMTPP}. Sequences of badges that users receive over time on the website Stack Overflow. A sequence is a specific user, and the type of badge received is the mark.
\end{itemize}
We employ the pre-processed version of these datasets as described in \citep{bosser2023predictive} which can be openly accessed at this url: \url{https://www.dropbox.com/sh/maq7nju7v5020kp/AAAFBvzxeNqySRsAm-zgU7s3a/processed/data?dl=0&subfolder_nav_tracking=1} (MIT License). Specifically, each dataset is filtered to contain the 50 most represented marks, and all arrival-times are rescaled in the interval [0,10] to avoid numerical instabilities. To save computational time, the number of sequences in Reddit is reduced by $50\%$. Each dataset is randomly partitioned into 3 train/validation/test splits (60\%/20\%/20\%). The summary statistics for each (filtered) dataset is reported in Table \ref{datastats_marked}.
\begin{table*}[h]
\centering
\footnotesize
\setlength\tabcolsep{4pt}
%\setlength\tabcolsep{4pt}
\caption{Datasets statistics}
\begin{tabular}{ccccccccc}
& \#Seq. & \#Events & Mean Length & Max Length & Min Length & \#Marks \\
\midrule
LastFM & 856 & 193441 & 226.0 & 6396 & 2 & 50 \\
MOOC & 7047 & 351160 & 49.8 & 416 & 2 & 50 \\
Github & 173 & 20656 & 119.4 & 4698 & 3 & 8 \\
Reddit & 4278 & 238734 & 55.8 & 941 & 2 & 50 \\
%Retweets & 12000 & 1309332 & 109.1 & 264 & 50 & 3 \\
Stack Overflow & 7959 & 569688 & 71.6 & 735 & 40 & 22 \\
\bottomrule
\end{tabular}
\label{datastats_marked}
\end{table*}
\section{Training details}
\label{sec:training_details}
\paragraph{Training details.}
For all models, we minimize the average NLL in (\ref{eq:NLL_disjoint}) on the training sequences using mini-batch gradient descent with the Adam optimizer \citep{adam} and a learning rate of $10^{-3}$. For the base models and the base\textbf{+} setup, an early-stopping protocol interrupts training if the model fails to show improvement in the total validation loss (i.e., $\mathcal{L}_T$ + $\mathcal{L}_T$) for 50 consecutive epochs. Conversely, in the base\textbf{++} setup, two distinct early-stopping protocols are implemented for the $\mathcal{L}_T$ and $\mathcal{L}_M$ terms, respectively. If one of these terms does not show improvement for 50 consecutive epochs, we freeze the parameters of the associated functions (e.g. $\btheta_T$ for $f^\ast(\tau;\btheta_T$)) and allow the remaining term to continue training. Training is ultimately interrupted if both early-stopping criteria are met.
\begin{comment}
The pseudo-code for the early-stopping criteria in the base\textbf{++} setup is provided in Algorithm \ref{alg:dshs}. Note that during the back-propagation step, $\btheta_T$ and $\btheta_M$ are only updated if they were not removed from the computational graph at a ulterior step.
\end{comment}
In all setups, the optimization process can last for a maximum of 500 epochs, and we revert the model parameters to their state with the lowest validation loss after training. Finally, we evaluate the model by computing test metrics on the test sequences of each split. Our framework is implemented in a unified codebase using PyTorch\footnote{https://pytorch.org/}. All models were trained on a machine equipped with an AMD Ryzen Threadripper PRO 3975WX CPU running at 4.1 GHz and a Nvidia RTX A4000 GPU.
\textbf{Encoding past events.} To obtain the encoding $\boldsymbol{e}_i \in \mathbb{R}^{d_e}$ of an event $e_i=(t_i,k_i)$ in $\mathcal{H}_t$, we follow the work of \citep{EHR} by first mapping $t_i$ to a vector of sinusoidal functions:
\begin{equation}
\boldsymbol{e}^t_{i} = \underset{j=0}{\overset{d_t/2-1}{\bigoplus}} \text{sin}~(\alpha_j t_i) \oplus \text{cos}~(\alpha_j t_i) \in \mathbb{R}^{d_t},
\label{eq:temporalencoding}
\end{equation}
where $\alpha_j \propto 1000^{\frac{-2j}{d_t}}$ and $\oplus$ is the concatenation operator. Then, a mark embedding $\boldsymbol{e}_i^k \in \mathbb{R}^{d_k}$ for $k_i$ is generated as $\boldsymbol{e}_i^k = \mathbf{E}^k \boldsymbol{k}_i$, where $\mathbf{E}^k \in \mathbb{R}^{d_k \times K}$ is a learnable embedding matrix, and $\boldsymbol{k}_i \in \{0,1\}^K$ is the one-hot encoding of $k_i$. Finally, we obtain $\boldsymbol{e}_i$ through concatenation, i.e. $\boldsymbol{e}_i = [\boldsymbol{e}_i^t||\boldsymbol{e}_i^k]$.
\paragraph{Hyperparameters.} To ensure that changes in performance are solely attributed to the features enabled by our framework, we control the number of parameters such the a baseline's capacity remains equivalent across the base, base\textbf{+}, base\textbf{++}, base-D, and base-DD setups. Notably, since STHP inherently models the decomposition of the marked intensity, the base and base\textbf{+} configurations are equivalent. Also, LNM, RMTPP and STHP are equivalent between the base\textbf{+} and the base-D settings, and between the base\textbf{++} and base-DD settings. Hence, we only consider these models in the base\textbf{+} and base\textbf{++} settings. Furthermore, Table \ref{tab:num_param} provides the total number of trainable parameters for each setup when trained on the LastFM dataset, as well as their distribution across the encoder and decoder heads. For all baselines and setups, we use a single encoder layer, and the dimension $d_e$ of the event encodings is set to 8. Additionally, we chose a value of $M=32$ for the number of mixture components. It is worth noting that \citep{IntensityFree} found LogNormMix to be robust to the choice of $M$. Finally, we set the number of GCIF projections to $C=32$.
\begin{table}[h!]
%\centering
\centering
\scriptsize
\caption{Number of parameters for each baseline when trained on the LastFM dataset. The distribution of parameters between the encoder and decoder heads is reported in parenthesis.}
\begin{tabular}{ccccccc}
\toprule
& THP & STHP & SAHP & FNN & LNM & RMTPP \\
\midrule
Base & 14720 (0.66/0.34) & \textbackslash & 15588 (0.68/0.32) & 15939 (0.65/0.35) & 13930 (0.67/0.33) & 13619 (0.69/0.31) \\
Base\textbf{+} & 14786 (0.64/0.36) & 18586 (0.5/0.5) & 15210 (0.67/0.33) & 16083 (0.65/0.35) & 13946 (0.67/0.33) & 13669 (0.69/0.31) \\
Base\textbf{++} & 14602 (0.63/0.37) & 18402 (0.5/0.5) & 15514 (0.67/0.33) & 14961 (0.67/0.33) & 13340 (0.66/0.34) & 13063 (0.67/0.33) \\
Base-D & 15512 (0.66/0.34) & \textbackslash & 15412 (0.67/0.33) & 16083 (0.65/0.35) & \textbackslash & \textbackslash \\
Base-DD & 15252 (0.66/0.34) & \textbackslash & 15464 (0.68/0.32) & 10446 (0.65/0.35) & \textbackslash & \textbackslash\\
\bottomrule
\end{tabular}
\label{tab:num_param}
\end{table}
\begin{comment}
\begin{algorithm}[th!]
\caption{Pseudo-code for the early-stopping procedure in the base\textbf{++} setup}
\label{alg:dshs}
\begin{algorithmic}
\STATE {\bfseries Input:} Predictive models $f^\ast(\tau;\btheta_T)$, $p^\ast(k|\tau;\btheta_M)$, and $F^\ast(\tau;\btheta_T)$; training sequences $\mathcal{S}$; validation sequences $\mathcal{S}_{\text{val}}$; patience $p > 0$, max epochs $m > 0$.
\STATE $c_m \gets 0$
\STATE $c_t \gets 0$
\STATE $\text{epoch} \gets 0$
\STATE $\mathcal{L}_{T,\text{best}} \gets \infty$
\STATE $\mathcal{L}_{M,\text{best}} \gets \infty$
\WHILE{($c_m <= p$ or $c_t <= p$) and $\text{epoch} <= m$}
\STATE Compute $\mathcal{L}_{T,\text{train}}$ and $\mathcal{L}_{M,\text{train}}$ on $\mathcal{S}$ using (\ref{eq:NLL_disjoint}).
\STATE Update $\btheta_T$ and $\btheta_M$ by back-propagating $\mathcal{L}_{T,\text{train}}$ and $\mathcal{L}_{M,\text{train}}$, respectively.
\STATE Evaluate $\mathcal{L}_{T,\text{val}}$ and $\mathcal{L}_{M,\text{val}}$ on $\mathcal{S}_{\text{val}}$ using (\ref{eq:NLL_disjoint}).
\IF{$\mathcal{L}_{T,\text{val}} < \mathcal{L}_{T,\text{best}}$ }
\STATE $c_t \gets 0$
\STATE $\mathcal{L}_{T,\text{best}} \gets \mathcal{L}_{T,\text{val}}$
\STATE Save $\btheta_T$
\ELSE
\IF{$c_t < p$}
\STATE $c_t += 1$;
\ELSE
\STATE Detach $\btheta_T$ from computational graph.
\ENDIF
\ENDIF
\IF{$\mathcal{L}_{M,\text{val}} < \mathcal{L}_{M,\text{best}}$ }
\STATE $c_m \gets 0$
\STATE $\mathcal{L}_{M,\text{best}} \gets \mathcal{L}_{M,\text{val}}$
\STATE Save $\btheta_M$
\ELSE
\IF{$c_m < p$}
\STATE $c_m += 1$;
\ELSE
\STATE Detach $\btheta_M$ from computational graph.
\ENDIF
\ENDIF
\ENDWHILE
\STATE \textbf{Return:} Last saved $\btheta_T$ and $\btheta_M$.
\end{algorithmic}
\end{algorithm}
\end{comment}
\section{Computational Time}
\label{sec:computational}
We report in Table \eqref{tab:neurips_app:comput_tabular} the average execution time (in seconds) for a single forward and backward pass on all training sequences of all datasets. The results are averaged over 50 epochs. We notice that the computation of two separate embeddings $\boldsymbol{h}_T$ and $\boldsymbol{h}_M$ in the base\textbf{++} setup inevitably leads to an increase in execution time, which appears more pronounced for larger datasets such as Reddit and Stack Overflow. However, the increased computational complexity is generally offset by improved model performance, as detailed in Section \ref{sec:discussion}.
\begin{table}[h]
\caption{Average execution time (in seconds) for a single forward and backward pass on all training sequences for all datasets. Results are averaged over 50 epochs.}
\centering
\scriptsize
\setlength\tabcolsep{1.2pt}
\begin{tabular}{ccccc@{\hspace{10\tabcolsep}}cccc@{\hspace{10\tabcolsep}}cccc@{\hspace{10\tabcolsep}}cccc@{\hspace{10\tabcolsep}}cccc@{\hspace{10\tabcolsep}}ccc}
\toprule \\
& \multicolumn{3}{c}{LNM} && \multicolumn{3}{c}{RMTPP} && \multicolumn{3}{c}{FNN} && \multicolumn{3}{c}{THP} && \multicolumn{3}{c}{SAHP} && \multicolumn{3}{c}{STHP} \\ \\
& Base & \textbf{+} & \textbf{++} && Base & \textbf{+} & \textbf{++} && Base & \textbf{+} & \textbf{++} && Base & \textbf{+} & \textbf{++} && Base & \textbf{+} & \textbf{++} && Base & \textbf{+} & \textbf{++} \\
\midrule
LastFM & 2.2 & 2.23 & 3.26 && 1.91 & 1.94 & 2.94 && 4.29 & 3.66 & 4.74 && 2.96 & 2.57 & 3.62 && 4.82 & 3.42 & 4.48 && 3.08 & \textbackslash & 4.17 \\ \midrule
MOOC & 3.97 & 3.99 & 5.6 && 3.24 & 3.33 & 4.37 && 6.36 & 6.28 & 8.07 && 5.19 & 4.15 & 5.76 && 8.08 & 5.91 & 7.74 && 5.61 & \textbackslash & 6.72 \\ \midrule
Github & 0.34 & 0.39 & 0.51 && 0.34 & 0.34 & 0.5 && 0.76 & 0.65 & 0.8 && 0.38 & 0.43 & 0.58 && 0.51 & 0.55 & 0.69 && 0.49 & \textbackslash & 0.64 \\ \midrule
Reddit & 8.74 & 9.53 & 11.5 && 7.99 & 8.14 & 11.79 && 19.13 & 16.47 & 20.32 && 9.75 & 9.78 & 13.5 && 13.63 & 11.99 & 15.9 && 11.59 & \textbackslash & 15.54 \\ \midrule
Stack O. & 16.02 & 16.63 & 22.98 && 14.08 & 14.54 & 20.31 && 32.83 & 28.15 & 33.95 && 16.32 & 16.81 & 23.1 && 21.11 & 20.77 & 26.45 && 19.84 & \textbackslash & 26.07 \\
\bottomrule
\end{tabular}
\label{tab:neurips_app:comput_tabular}
\end{table}
\begin{comment}
\section{Computational time}
In Table \ref{comput_tabular}, we present the average execution times in seconds for a single forward and backward pass over 10 epochs on all training sequences of the MOOC dataset. In all models, we set the batch size to 32, and the training process is conducted on a Nvidia RTXA5000 GPU. We notice that as the number of learnable parameters increases across the base, \textbf{+}, and \textbf{++} setups, there is a corresponding rise in execution time. Nevertheless, this added complexity is offset by the enhanced performance observed in the \textbf{+} and \textbf{++} setups for both time and mark prediction tasks.
\begin{table}[h]
\centering
\tiny
\setlength\tabcolsep{1.2pt}
\caption{Average execution time (in seconds) for a single forward and backward pass on all training sequences of the MOOC dataset. Results are averaged over 10 epochs, and a standard deviation is reported in parenthesis.}
\begin{tabular}{ccccc@{\hspace{10\tabcolsep}}cccc@{\hspace{10\tabcolsep}}cccc@{\hspace{10\tabcolsep}}ccc}
\toprule \\
& \multicolumn{3}{c}{LNM} && \multicolumn{3}{c}{RMTPP} && \multicolumn{3}{c}{THP} && \multicolumn{3}{c}{SAHP} \\ \\
& Base & DCH & DSHS && Base & DCH & DSHS && Base & DCH & DSHS && Base & DCH & DSHS \\
\midrule \\
&1.63 (0.29) & 1.80 (0.33) &2.21(0.33) && 1.45 (0.32) & 1.53 (0.31) & 1.99 (0.31) && 1.61 (0.40) & 1.95 (0.42) & 2.49 (0.39) && 1.73 (0.35) & 2.02 (0.42) & 2.46 (0.36) \\ \\
\bottomrule
\end{tabular}
\label{comput_tabular}
\end{table}
\end{comment}
\section{An Alternative Approach to Model the Joint Distribution}
\label{sec:lnm_joint}
As detailed in Section (\ref{sec:existing_TPP}), our parametrization of LNM\textbf{+} alleviates the conditional independence of arrival-times and marks made in LNM \citep{IntensityFree}. Relatedly, \citet{Waghmare} also proposed an extension of LNM that relaxes this assumption, although their approach differs from ours in some key aspects. Specifically, their work parametrizes $f^\ast(\tau|k;\btheta_T)$ as a distinct mixture of log-normal distributions for each mark $k$, and $p^\ast(k;\btheta_M)$ is obtained by removing the temporal dependency in (\ref{eq:pmf_cond_dis})\footnote{Both $f^\ast(\tau|k;\btheta_T)$ and $p^\ast(k;\btheta_M)$ rely on a common history embedding $\bh$.}. For further reference, we denote this model as LNM-Joint. Although both LNM\textbf{+} and LNM-Joint aim to model the joint distribution $f^\ast(\tau,k)$, some conceptual differences separate the two approaches:
\begin{enumerate}
\item By design, LNM-Joint cannot be trained in the base\textbf{++} setup as it prevents the decomposition of the NLL into disjoint $\mathcal{L}_T$ and $\mathcal{L}_M$ terms. Indeed, suppose that we use two distinct history representations $\boldsymbol{h}_T$ and $\boldsymbol{h}_M$ to parametrize $f^\ast(\tau|k;\boldsymbol{\theta}_T)$ and $p^\ast(k;\btheta_M)$ respectively, as detailed in Section \ref{sec:existing_TPP}. Here, $\btheta_T$ and $\btheta_M$ are disjoint set of learnable parameters. The NLL of a training sequence $\mathcal{S}=\{(\tau_i, k_i)\}_{i=1}^n$ observed in $[0,T]$ would write
\begin{equation}
\mathcal{L}(\boldsymbol{\theta}_T, \boldsymbol{\theta}_M; \mathcal{S}) = \underbrace{-\sum_{i=1}^n \text{log } f^\ast(\tau_i|k_i;\boldsymbol{\theta}_T) - \text{log }\left(1-F^\ast\left(T;\boldsymbol{\theta}_T, \boldsymbol{\theta}_M\right)\right)}_{\mathcal{L}_T(\btheta_T,\btheta_M;\mathcal{S})} \underbrace{-\sum_{i=1}^n \text{log } p^\ast(k_i;\boldsymbol{\theta}_M)}_{\mathcal{L}_M(\btheta_M; \mathcal{S})},
\end{equation}
where $F^\ast\left(T;\boldsymbol{\theta}_T;\boldsymbol{\theta}_M\right) = \int_0^{T-t_n} \sum_{k=1}^K f^\ast(s|k;\boldsymbol{\theta}_T)p^\ast(k;\boldsymbol{\theta}_M)ds$ depends on both $\boldsymbol{\theta}_T$ and $\boldsymbol{\theta}_M$. Consequently, the NLL cannot be disentangled into disjoint $\mathcal{L}_T$ and $\mathcal{L}_M$ terms, proscribing disjoint training in the base\textbf{++} setup. Conversely, choosing to parametrize $f^\ast(\tau;\boldsymbol{\theta}_T)$ and $p^\ast(k|\tau;\boldsymbol{\theta}_M)$ as done in our framework leads to the decomposition in (\ref{eq:NLL_disjoint}) as $F^\ast(T;\boldsymbol{\theta}_T)$ is solely function of $\boldsymbol{\theta}_T$.
\item For LNM-Joint, $M$ mixtures must be defined for each $k$, leading to $M \times K$ log-normal distributions in total. Conversely, in LNM\textbf{+}, $f^\ast(\tau;\boldsymbol{\theta}_T)$ does not scale with $K$, and $p^\ast(\tau|k;\boldsymbol{\theta}_M)$ requires an equivalent number of parameters as $p^\ast(k;\boldsymbol{\theta}_M)$ in LNM-Joint.
\end{enumerate}
For completeness, we integrate LNM-Joint in our code base using the original implementation as reference. In Table (\ref{tab:tabular_results_lnm_joint}), we compare its performance against LNM\textbf{+} on the time and mark prediction tasks in terms of the $\mathcal{L}_T$, $\mathcal{L}_M$, and accuracy metrics, following the experimental setup detailed previously. We observe improved performance of LNM\textbf{+} compared to LNM-Joint on the mark prediction task ($\mathcal{L}_M$ and accuracy), and competitive results on the time prediction task ($\mathcal{L}_T$). Despite both approaches modelling the joint distribution, our results suggest that the dependency between arrival times and marks is more accurately captured by $p^\ast(k|\tau;\btheta_M)$ than by $f^\ast(\tau|k;\btheta_T)$.
\begin{comment}
\begin{table*}[h!]
\setlength\tabcolsep{1.2pt}
%\centering
\centering
\tiny
\caption{$\mathcal{L}_T$, $\mathcal{L}_M$ and Accuracy results for LNM\textbf{+} and LNM-Joint on all datasets. The values are computed over 3 splits, and the standard error is reported in parenthesis. Best results are highlighted in bold.}
\begin{tabular}{cccccc}
\toprule
& LastFM & MOOC & Github & Reddit & Stack Overflow \\ \\
& \multicolumn{5}{c}{$\mathcal{L}_M$} \\
\midrule
LNM\textbf{+} & \textbf{662.2 (21.3)} & \textbf{77.1 (1.9)} & \textbf{117.9 (24.5)} & \textbf{42.2 (1.3)} & \textbf{102.8 (1.1)} \\
Joint-LNM & 667.2 (16.4) & 123.5 (8.9) & 122.1 (25.4) & 43.1 (1.3) & 106.3 (1.1) \\
\toprule
& \multicolumn{5}{c}{Accuracy} \\
\midrule
LNM\textbf{+} & \textbf{0.25 (0.01)} & \textbf{0.53 (0.0)} & \textbf{0.67 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.49 (0.0)} \\
Joint-LNM & 0.24 (0.01) & 0.23 (0.04) & 0.65 (0.01) & 0.81 (0.0) & 0.46 (0.0) \\
\toprule
& \multicolumn{5}{c}{$\mathcal{L}_T$} \\
\midrule
LNM\textbf{+} & -1315.5 (67.7) & \textbf{-312.0 (4.9)} & -412.7 (93.2) & \textbf{-96.3 (3.1)} & \textbf{-90.1 (2.0)} \\
Joint-LNM & \textbf{-1326.7 (64.0)} & -305.3 (4.7) & \textbf{-415.3 (93.4)} & -93.6 (2.8) & -89.5 (2.1) \\
\bottomrule
\end{tabular}
\label{tab:tabular_results_lnm_joint}
\end{table*}
\end{comment}
\begin{table*}[h!]
\setlength\tabcolsep{1.2pt}
%\centering
\centering
\tiny
\caption{\textcolor{blue}{$\mathcal{L}_T$, $\mathcal{L}_M$ and Accuracy results for LNM\textbf{+} and LNM-Joint on all datasets. The values are computed over 3 splits, and the standard error is reported in parenthesis. Best results are highlighted in bold.}}
\begin{tabular}{cccccc}
\toprule
& LastFM & MOOC & Github & Reddit & Stack O.\\ \midrule
& \multicolumn{5}{c}{$\mathcal{L}_M$} \\
\midrule \midrule
LNM\textbf{+} & \textbf{668.6 (15.8)} & \textbf{77.1 (1.3)} & \textbf{112.3 (15.1)} & \textbf{41.4 (1.1)} & \textbf{103.2 (0.7)} \\
Joint-LNM & 671.3 (17.1) & 127.0 (6.4) & 117.3 (15.5) & 42.9 (0.9) & 106.6 (0.7) \\
\midrule
& \multicolumn{5}{c}{Accuracy} \\
\midrule \midrule
LNM+ & \textbf{0.24 (0.01)} & \textbf{0.52 (0.0)} & \textbf{0.67 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.49 (0.0)} \\
Joint-LNM & 0.23 (0.01) & 0.23 (0.03) & 0.64 (0.01) & \textbf{0.82 (0.0)} & 0.47 (0.0) \\
\midrule
& \multicolumn{5}{c}{$\mathcal{L}_T$} \\
\midrule \midrule
LNM+ & -1320.4 (60.5) & \textbf{-310.6 (3.7)} & -378.7 (59.4) & \textbf{-96.4 (2.0)} & \textbf{-91.1 (1.3)} \\
Joint-LNM & \textbf{-1326.2 (58.3)} & -303.9 (3.3) & \textbf{-381.0 (59.6)} & -94.3 (2.0) & -90.6 (1.4) \\
\bottomrule
\end{tabular}
\label{tab:tabular_results_lnm_joint}
\end{table*}
\section{Additional results}
\label{sec:add_results}
\subsection{Evaluation metrics}
\label{sec:neurips_app:evaluation_metrics}
Tables \ref{tab:pce_ece}, \ref{tab:acc1_acc3} and \ref{tab:acc5_mrr} give the PCE, ECE, MRR, and accuracy@\{1,3,5\} for the base, base\textbf{+} and base\textbf{++} setups across all datasets. The metrics are averaged over 3 splits, and the standard error is given in parenthesis. We note that the results not discussed in the main text are consistent with our previous conclusions. Specifically, we observe general improvement with respect to mark related metrics (i.e. ECE, MRR, accuracy@\{1,3,5\}) when moving from the base models to the base\textbf{+} or base\textbf{++} setups. Finally, the PCE metric does not always improve between the base\textbf{+} and \textbf{++} setups, suggesting that the remaining conflicting gradients at the encoder head in the base\textbf{+} are mostly detrimental to the mark prediction task.
Finally, we report the results with respect to the MAE metric in Table \ref{tab:mae}. We notice that lower MAE values do not systematically match the lower values of $\mathcal{L}_T$ or PCE in Tables \ref{tabular_results} and \ref{tab:pce_ece}, indicating that the MAE may not be entirely appropriate to evaluate the time prediction task. As discussed in \citet{SchurSurvey}, neural MTPP models are probabilistic models that enable the generation of complete distributions over future events. In this context, point prediction metrics, like MAE, are deemed less suitable for evaluating MTPP models because they consider single point predictions into account. In contrast, the NLL and calibration scores directly evaluate the entire predictive distributions, and should be therefore favored compared to point prediction metrics.
\input{results_tabular_appendix}
\subsection{Reliability diagrams}
\label{sec:neurips_app:reliability}
Figures (\ref{fig:reliability_diag_appendix_lastfm}) to (\ref{fig:reliability_diag_appendix_reddit}) show the reliability diagrams of the predictive distribution of arrival-times and marks for all models in the base and base\textbf{++} setups on all datasets. In most cases, we observe improved mark calibration for the base\textbf{++} setup compared to the base models, in accordance to the ECE results of Table \ref{tab:pce_ece}. Additionally, improvements with respect to the calibration of the predictive distribution of arrival times is in general less prevalent, corroborating our discussion in the main text. This observation is also in coherent with the PCE results of Table \ref{tab:pce_ece}. Nonetheless, we observe substantial time calibration improvements for SAHP and FNN when trained in the base\textbf{++} setup on MOOC and Reddit.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{\textwidth}
\caption{LastFM}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/LastFM_cal1s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/LastFM_cal2s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/LastFM_1s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/LastFM_2s.pdf}
\end{subfigure}
\caption{\textcolor{blue}{Reliability diagrams of the predictive $p^\ast(k|\tau;\btheta_M)$ (top) and $f^\ast(\tau,\btheta_T)$ (bottom) on LastFM. Frequency and Accuracy aligning with the black diagonal corresponds to perfect calibration. The results are averaged over 5 splits, and the error bars correspond to the standard error.}}
\label{fig:reliability_diag_appendix_lastfm}
\end{figure}
\begin{figure}
\centering
\begin{subfigure}[b]{\textwidth}
\caption{MOOC}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/MOOC_cal1s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/MOOC_cal2s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/MOOC_1s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/MOOC_2s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\caption{Github}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/Github_cal1s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/Github_cal2s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/Github_1s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/Github_2s.pdf}
\end{subfigure}
\caption{\textcolor{blue}{Reliability diagrams of the predictive $p^\ast(k|\tau;\btheta_M)$ (top) and $f^\ast(\tau,\btheta_T)$ (bottom) on MOOC anf Github. Frequency and Accuracy aligning with the black diagonal corresponds to perfect calibration. The results are averaged over 3 splits, and the error bars correspond to the standard error.}}
\label{fig:reliability_diag_appendix_mooc}
\end{figure}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{\textwidth}
\caption{Reddit}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/Reddit_cal1s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/Reddit_cal2s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/Reddit_1s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/Reddit_2s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\caption{Stack Overflow}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/Stack Overflow_cal1s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/mark/Stack Overflow_cal2s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/Stack Overflow_1s.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/calibration_new/time/Stack Overflow_2s.pdf}
\end{subfigure}
\caption{\textcolor{blue}{Reliability diagrams of the predictive $p^\ast(k|\tau;\btheta_M)$ (top) and $f^\ast(\tau,\btheta_T)$ (bottom) on Reddit and Stack Overflow. Frequency and Accuracy aligning with the black diagonal corresponds to perfect calibration. The results are averaged over 5 splits, and the error bars correspond to the standard error.}}
\label{fig:reliability_diag_appendix_reddit}
\end{figure}
\clearpage
\subsection{Conflicting gradients remain harmful as capacity increases}
To assess whether conflicting gradients remain detrimental to predictive performance for higher-capacity models, we train THP, SAHP and FNN in the base and base-DD settings while progressively increasing the number of trainable parameters. Figure \ref{fig:neurips:evolution_param} shows the evolution of the proportion of CG, GMS and TPI during training for these models on LastFM and MOOC, along with the evolution of their test $\mathcal{L}_T$ and $\mathcal{L}_M$ as a function of number of parameters. For each capacity (25K, 50K, 75K and 100K parameters), we maintain the distribution of parameters between the encoder and decoder heads constant at 0.67/0.33. Note that we only report the CG, GMS, and TPI values for the base models, as the base-DD setting is by definition free of conflicts. We note that increasing a model's capacity has a limited impact on the CG, GMS and TPI values, as well as on model performance with respect to both $\mathcal{L}_T$ and $\mathcal{L}_M$. In contrast, differences in performance between the base and base-DD setups are more significant, suggesting that conflicting gradients remain harmful to performance even with increased model capacity.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{\textwidth}
\caption{LastFM}
\includegraphics[width=\textwidth]{images_tmlr/param/lastfm_filtered.pdf}
\end{subfigure}
\vspace{0.5cm}
\begin{subfigure}[b]{\textwidth}
\caption{MOOC}
\includegraphics[width=\textwidth]{images_tmlr/param/mooc_filtered.pdf}
\end{subfigure}
\caption{Left: Evolution of CG, GMS, TPI with increasing model capacity for the base THP, SAHP and FNN during training on LastFM and MOOC. Right: Evolution of the test $\mathcal{L}_T$ and $\mathcal{L}_M$ in the base and base-DD settings.}
\vspace{-0.3cm}
\label{fig:neurips:evolution_param}
\end{figure}
\subsection{Scaling the loss does not efficiently address conflicts}
Our findings in Figure \ref{fig:grad_jd} suggest that conflicting gradients generally tend to favor $\mathcal{T}_T$ at the encoder heads during optimization, as illustrated by TPI values > 0.5. To better balance tasks during training, a natural approach would consist in scaling the contribution of $\mathcal{T}_T$ in (\ref{eq:NLL}) to reduce its impact on the overall loss, i.e.
\begin{equation}
\mathcal{L}(\theta; \mathcal{S}_{\text{train}}) = \frac{1}{s} \mathcal{L}_T(\theta; \mathcal{S}_{\text{train}}) + \mathcal{L}_M(\theta; \mathcal{S}_{\text{train}}),
\label{eq:neurips:scaled_nll}
\end{equation}
where $s \geq 1$ is a scaling coefficient. To assess the effectiveness of this method, we train the base THP, SAHP and FNN models on the objective in (\ref{eq:neurips:scaled_nll}) following the experimental setup detailed in Section \ref{sec:training_details}. For these models, we report in Figure (\ref{fig:neurips:evolution_scaling}) the evolution of the training CG, GMS and TPI values, along with their unscaled test $\mathcal{L}_T$ and $\mathcal{L}_M$. We observe that the occurrence of conflicting gradients is marginally impacted by larger values of $s$. However, as scaling increases, the magnitude of $\bg_T$ diminishes, which translates into decreased values of GMS and TPI. Consequently, optimization begins to favor $\mathcal{T}_M$, and improvements on $\mathcal{L}_M$ can be observed. Nevertheless, the crashed TPI and GMS values are at the root of significant degradation with respect to $\mathcal{L}_T$, offsetting the gains on $\mathcal{T}_M$. Although a specific value of $s$ could lead to a trade-off between tasks, models trained in our base\textbf{+} or base\textbf{++} settings generally show improved performance with respect to both tasks simultaneously, as shown in Table \ref{tabular_results}.
\begin{figure}[!h]
\centering
\begin{subfigure}[b]{\textwidth}
\caption{LastFM}
\includegraphics[width=\textwidth]{images_tmlr/scaling/lastfm_filtered.pdf}
\end{subfigure}
\vspace{0.5cm}
\begin{subfigure}[b]{\textwidth}
\caption{MOOC}
\includegraphics[width=\textwidth]{images_tmlr/scaling/mooc_filtered.pdf}
\end{subfigure}
\caption{Left: Evolution of CG, GMS, TPI with scaling coefficient $s$ for the base THP, SAHP and FNN during training on LastFM. Right: Evolution of the test $\mathcal{L}_T$ and $\mathcal{L}_M$ in the base and base-DD settings.}
\vspace{-0.3cm}
\label{fig:neurips:evolution_scaling}
\end{figure}
\clearpage
\subsection{Distributions of conflicting gradients between the base and base+ settings}
\label{sec:neurips_app:distirbution_+}
Figures (\ref{fig:conflicting_appendix_lastfm}) to (\ref{fig:conflicting_appendix_stack}) show the distributions of $\text{cos } \phi_{TM}$ during training, as well as the proportion of conflicting gradients (CG), their average GMS, and their average TPI for all models in the base and base+ setups on all datasets. As discussed in the main text, severe conflicts ($\text{cos } \phi_{TM} < -0.5$) often arise when training neural MTPP models in the base setup. For THP, SAHP, and FNN, the base+ setup reduces the occurrence of severe conflicts at the encoder heads, and prevents conflicting gradients to appear at the decoder heads. For LNM and RMTPP, we only show the distribution of $\text{cos } \phi_{TM}$ at the encoder heads as the decoders are by design free of conflicting gradients in the base and base+ setups. For these models, the base+ setup does not change significantly the distribution of conflicting gradients during training, which aligns with expectations. Hence, for LNM\textbf{+} and RMTPP\textbf{+}, we attribute performance gains on the mark prediction task to the relaxation of conditional independence between arrival times and marks via (\ref{eq:pmf_cond_dis}).
\begin{figure}[h]
\centering
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer/lastfm_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=1.1\textwidth]{images_tmlr/gradients_layer/lastfm_filtered_smurf-thp_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer/mooc_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=1.1\textwidth]{images_tmlr/gradients_layer/mooc_filtered_smurf-thp_bis.pdf}
\end{subfigure}
\caption{\textcolor{blue}{Distribution of $\text{cos } \phi_{TM}$ during training at the encoder (ENC) and decoder (DEC) heads for all baselines in the base and base\textbf{+} setup on LastFM and MOOC. "B" and "+" refer to the base and base\textbf{+} models, respectively, and the distribution is obtained by pooling the values of $\phi_{TM}$ over 5 training runs. As the decoders are disjoint in the base\textbf{+} setting, note that $\text{cos } \phi_{TM}$ is not defined.}}
\label{fig:conflicting_appendix_lastfm}
\end{figure}
\begin{figure}[h]
\centering
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer/github_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=1.1\textwidth]{images_tmlr/gradients_layer/github_filtered_smurf-thp_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer/reddit_filtered_short_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=1.1\textwidth]{images_tmlr/gradients_layer/reddit_filtered_short_smurf-thp_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer/stack_overflow_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=1.1\textwidth]{images_tmlr/gradients_layer/stack_overflow_filtered_smurf-thp_bis.pdf}
\end{subfigure}
\caption{\textcolor{blue}{Distribution of $\text{cos } \phi_{TM}$ during training at the encoder (ENC) and decoder (DEC) heads for all baselines in the base and base\textbf{+} setup on Github, Reddit, and Stack Overflow. "B" and "+" refer to the base and base\textbf{+} models, respectively, and the distribution is obtained by pooling the values of $\phi_{TM}$ over 5 training runs. As the decoders are disjoint in the base\textbf{+} setting, note that $\text{cos } \phi_{TM}$ is not defined.}}
\label{fig:conflicting_appendix_stack}
\end{figure}
\clearpage
\subsection{Distributions of conflicting gradients between the base and base-D settings}
\label{sec:neurips_app:distirbution_d}
Figure (\ref{fig:conflicting_appendix_double}) show the distribution of $\text{cos } \phi_{TM}$ during training, as well as the proportion of conflicting gradients (CG), their average GMS, and their average TPI for THP, SAHP, and FNN in the base and base-D setups on all datasets. We observe that using two identical and task-specific instances of the same decoder in the base-D setup for the time and mark prediction tasks mitigates the occurrence of conflicts. The decrease in conflicting gradients during training is in turn associated to increased performance with respect to both tasks. We refer the reader to the discussion in Section \ref{sec:discussion} of the main text for further details.
\begin{figure}[h]
\centering
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer_double/lastfm_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer_double/mooc_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer_double/github_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer_double/reddit_filtered_short_mlp-cm_bis.pdf}
\end{subfigure}
\begin{subfigure}[b]{\textwidth}
\includegraphics[width=\textwidth]{images_tmlr/gradients_layer_double/stack_overflow_filtered_mlp-cm_bis.pdf}
\end{subfigure}
\caption{\textcolor{blue}{Distribution of $\text{cos } \phi_{TM}$ during training at the encoder (ENC) and decoder (DEC) heads for all baselines in the base and base-D setup on all datasets. "B" and "D" refer to the base and base-D models, respectively, and the distribution is obtained by pooling the values of $\phi_{TM}$ over 5 training runs. As the decoders are disjoint in the base-D setting, note that $\text{cos } \phi_{TM}$ is not defined.}}
\label{fig:conflicting_appendix_double}
\end{figure}
\end{document}
%%%%% NEW MATH DEFINITIONS %%%%%
\usepackage{amsmath,amsfonts,bm}
% Mark sections of captions for referring to divisions of figures
\newcommand{\figleft}{{\em (Left)}}
\newcommand{\figcenter}{{\em (Center)}}
\newcommand{\figright}{{\em (Right)}}
\newcommand{\figtop}{{\em (Top)}}
\newcommand{\figbottom}{{\em (Bottom)}}
\newcommand{\captiona}{{\em (a)}}
\newcommand{\captionb}{{\em (b)}}
\newcommand{\captionc}{{\em (c)}}
\newcommand{\captiond}{{\em (d)}}
% Highlight a newly defined term
\newcommand{\newterm}[1]{{\bf #1}}
% Figure reference, lower-case.
\def\figref#1{figure~\ref{#1}}
% Figure reference, capital. For start of sentence
\def\Figref#1{Figure~\ref{#1}}
\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
% Section reference, lower-case.
\def\secref#1{section~\ref{#1}}
% Section reference, capital.
\def\Secref#1{Section~\ref{#1}}
% Reference to two sections.
\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
% Reference to three sections.
\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
% Reference to an equation, lower-case.
\def\eqref#1{equation~\ref{#1}}
% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}
% A raw reference to an equation---avoid using if possible
\def\plaineqref#1{\ref{#1}}
% Reference to a chapter, lower-case.
\def\chapref#1{chapter~\ref{#1}}
% Reference to an equation, upper case.
\def\Chapref#1{Chapter~\ref{#1}}
% Reference to a range of chapters
\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
% Reference to an algorithm, lower-case.
\def\algref#1{algorithm~\ref{#1}}
% Reference to an algorithm, upper case.
\def\Algref#1{Algorithm~\ref{#1}}
\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
% Reference to a part, lower case
\def\partref#1{part~\ref{#1}}
% Reference to a part, upper case
\def\Partref#1{Part~\ref{#1}}
\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
\def\ceil#1{\lceil #1 \rceil}
\def\floor#1{\lfloor #1 \rfloor}
\def\1{\bm{1}}
\newcommand{\train}{\mathcal{D}}
\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
\def\eps{{\epsilon}}
% Random variables
\def\reta{{\textnormal{$\eta$}}}
\def\ra{{\textnormal{a}}}
\def\rb{{\textnormal{b}}}
\def\rc{{\textnormal{c}}}
\def\rd{{\textnormal{d}}}
\def\re{{\textnormal{e}}}
\def\rf{{\textnormal{f}}}
\def\rg{{\textnormal{g}}}
\def\rh{{\textnormal{h}}}
\def\ri{{\textnormal{i}}}
\def\rj{{\textnormal{j}}}
\def\rk{{\textnormal{k}}}
\def\rl{{\textnormal{l}}}
% rm is already a command, just don't name any random variables m
\def\rn{{\textnormal{n}}}
\def\ro{{\textnormal{o}}}
\def\rp{{\textnormal{p}}}
\def\rq{{\textnormal{q}}}
\def\rr{{\textnormal{r}}}
\def\rs{{\textnormal{s}}}
\def\rt{{\textnormal{t}}}
\def\ru{{\textnormal{u}}}
\def\rv{{\textnormal{v}}}
\def\rw{{\textnormal{w}}}
\def\rx{{\textnormal{x}}}
\def\ry{{\textnormal{y}}}
\def\rz{{\textnormal{z}}}
% Random vectors
\def\rvepsilon{{\mathbf{\epsilon}}}
\def\rvtheta{{\mathbf{\theta}}}
\def\rva{{\mathbf{a}}}
\def\rvb{{\mathbf{b}}}
\def\rvc{{\mathbf{c}}}
\def\rvd{{\mathbf{d}}}
\def\rve{{\mathbf{e}}}
\def\rvf{{\mathbf{f}}}
\def\rvg{{\mathbf{g}}}
\def\rvh{{\mathbf{h}}}
\def\rvu{{\mathbf{i}}}
\def\rvj{{\mathbf{j}}}
\def\rvk{{\mathbf{k}}}
\def\rvl{{\mathbf{l}}}
\def\rvm{{\mathbf{m}}}
\def\rvn{{\mathbf{n}}}
\def\rvo{{\mathbf{o}}}
\def\rvp{{\mathbf{p}}}
\def\rvq{{\mathbf{q}}}
\def\rvr{{\mathbf{r}}}
\def\rvs{{\mathbf{s}}}
\def\rvt{{\mathbf{t}}}
\def\rvu{{\mathbf{u}}}
\def\rvv{{\mathbf{v}}}
\def\rvw{{\mathbf{w}}}
\def\rvx{{\mathbf{x}}}
\def\rvy{{\mathbf{y}}}
\def\rvz{{\mathbf{z}}}
% Elements of random vectors
\def\erva{{\textnormal{a}}}
\def\ervb{{\textnormal{b}}}
\def\ervc{{\textnormal{c}}}
\def\ervd{{\textnormal{d}}}
\def\erve{{\textnormal{e}}}
\def\ervf{{\textnormal{f}}}
\def\ervg{{\textnormal{g}}}
\def\ervh{{\textnormal{h}}}
\def\ervi{{\textnormal{i}}}
\def\ervj{{\textnormal{j}}}
\def\ervk{{\textnormal{k}}}
\def\ervl{{\textnormal{l}}}
\def\ervm{{\textnormal{m}}}
\def\ervn{{\textnormal{n}}}
\def\ervo{{\textnormal{o}}}
\def\ervp{{\textnormal{p}}}
\def\ervq{{\textnormal{q}}}
\def\ervr{{\textnormal{r}}}
\def\ervs{{\textnormal{s}}}
\def\ervt{{\textnormal{t}}}
\def\ervu{{\textnormal{u}}}
\def\ervv{{\textnormal{v}}}
\def\ervw{{\textnormal{w}}}
\def\ervx{{\textnormal{x}}}
\def\ervy{{\textnormal{y}}}
\def\ervz{{\textnormal{z}}}
% Random matrices
\def\rmA{{\mathbf{A}}}
\def\rmB{{\mathbf{B}}}
\def\rmC{{\mathbf{C}}}
\def\rmD{{\mathbf{D}}}
\def\rmE{{\mathbf{E}}}
\def\rmF{{\mathbf{F}}}
\def\rmG{{\mathbf{G}}}
\def\rmH{{\mathbf{H}}}
\def\rmI{{\mathbf{I}}}
\def\rmJ{{\mathbf{J}}}
\def\rmK{{\mathbf{K}}}
\def\rmL{{\mathbf{L}}}
\def\rmM{{\mathbf{M}}}
\def\rmN{{\mathbf{N}}}
\def\rmO{{\mathbf{O}}}
\def\rmP{{\mathbf{P}}}
\def\rmQ{{\mathbf{Q}}}
\def\rmR{{\mathbf{R}}}
\def\rmS{{\mathbf{S}}}
\def\rmT{{\mathbf{T}}}
\def\rmU{{\mathbf{U}}}
\def\rmV{{\mathbf{V}}}
\def\rmW{{\mathbf{W}}}
\def\rmX{{\mathbf{X}}}
\def\rmY{{\mathbf{Y}}}
\def\rmZ{{\mathbf{Z}}}
% Elements of random matrices
\def\ermA{{\textnormal{A}}}
\def\ermB{{\textnormal{B}}}
\def\ermC{{\textnormal{C}}}
\def\ermD{{\textnormal{D}}}
\def\ermE{{\textnormal{E}}}
\def\ermF{{\textnormal{F}}}
\def\ermG{{\textnormal{G}}}
\def\ermH{{\textnormal{H}}}
\def\ermI{{\textnormal{I}}}
\def\ermJ{{\textnormal{J}}}
\def\ermK{{\textnormal{K}}}
\def\ermL{{\textnormal{L}}}
\def\ermM{{\textnormal{M}}}
\def\ermN{{\textnormal{N}}}
\def\ermO{{\textnormal{O}}}
\def\ermP{{\textnormal{P}}}
\def\ermQ{{\textnormal{Q}}}
\def\ermR{{\textnormal{R}}}
\def\ermS{{\textnormal{S}}}
\def\ermT{{\textnormal{T}}}
\def\ermU{{\textnormal{U}}}
\def\ermV{{\textnormal{V}}}
\def\ermW{{\textnormal{W}}}
\def\ermX{{\textnormal{X}}}
\def\ermY{{\textnormal{Y}}}
\def\ermZ{{\textnormal{Z}}}
% Vectors
\def\vzero{{\bm{0}}}
\def\vone{{\bm{1}}}
\def\vmu{{\bm{\mu}}}
\def\vtheta{{\bm{\theta}}}
\def\va{{\bm{a}}}
\def\vb{{\bm{b}}}
\def\vc{{\bm{c}}}
\def\vd{{\bm{d}}}
\def\ve{{\bm{e}}}
\def\vf{{\bm{f}}}
\def\vg{{\bm{g}}}
\def\vh{{\bm{h}}}
\def\vi{{\bm{i}}}
\def\vj{{\bm{j}}}
\def\vk{{\bm{k}}}
\def\vl{{\bm{l}}}
\def\vm{{\bm{m}}}
\def\vn{{\bm{n}}}
\def\vo{{\bm{o}}}
\def\vp{{\bm{p}}}
\def\vq{{\bm{q}}}
\def\vr{{\bm{r}}}
\def\vs{{\bm{s}}}
\def\vt{{\bm{t}}}
\def\vu{{\bm{u}}}
\def\vv{{\bm{v}}}
\def\vw{{\bm{w}}}
\def\vx{{\bm{x}}}
\def\vy{{\bm{y}}}
\def\vz{{\bm{z}}}
% Elements of vectors
\def\evalpha{{\alpha}}
\def\evbeta{{\beta}}
\def\evepsilon{{\epsilon}}
\def\evlambda{{\lambda}}
\def\evomega{{\omega}}
\def\evmu{{\mu}}
\def\evpsi{{\psi}}
\def\evsigma{{\sigma}}
\def\evtheta{{\theta}}
\def\eva{{a}}
\def\evb{{b}}
\def\evc{{c}}
\def\evd{{d}}
\def\eve{{e}}
\def\evf{{f}}
\def\evg{{g}}
\def\evh{{h}}
\def\evi{{i}}
\def\evj{{j}}
\def\evk{{k}}
\def\evl{{l}}
\def\evm{{m}}
\def\evn{{n}}
\def\evo{{o}}
\def\evp{{p}}
\def\evq{{q}}
\def\evr{{r}}
\def\evs{{s}}
\def\evt{{t}}
\def\evu{{u}}
\def\evv{{v}}
\def\evw{{w}}
\def\evx{{x}}
\def\evy{{y}}
\def\evz{{z}}
% Matrix
\def\mA{{\bm{A}}}
\def\mB{{\bm{B}}}
\def\mC{{\bm{C}}}
\def\mD{{\bm{D}}}
\def\mE{{\bm{E}}}
\def\mF{{\bm{F}}}
\def\mG{{\bm{G}}}
\def\mH{{\bm{H}}}
\def\mI{{\bm{I}}}
\def\mJ{{\bm{J}}}
\def\mK{{\bm{K}}}
\def\mL{{\bm{L}}}
\def\mM{{\bm{M}}}
\def\mN{{\bm{N}}}
\def\mO{{\bm{O}}}
\def\mP{{\bm{P}}}
\def\mQ{{\bm{Q}}}
\def\mR{{\bm{R}}}
\def\mS{{\bm{S}}}
\def\mT{{\bm{T}}}
\def\mU{{\bm{U}}}
\def\mV{{\bm{V}}}
\def\mW{{\bm{W}}}
\def\mX{{\bm{X}}}
\def\mY{{\bm{Y}}}
\def\mZ{{\bm{Z}}}
\def\mBeta{{\bm{\beta}}}
\def\mPhi{{\bm{\Phi}}}
\def\mLambda{{\bm{\Lambda}}}
\def\mSigma{{\bm{\Sigma}}}
% Tensor
\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
\def\tA{{\tens{A}}}
\def\tB{{\tens{B}}}
\def\tC{{\tens{C}}}
\def\tD{{\tens{D}}}
\def\tE{{\tens{E}}}
\def\tF{{\tens{F}}}
\def\tG{{\tens{G}}}
\def\tH{{\tens{H}}}
\def\tI{{\tens{I}}}
\def\tJ{{\tens{J}}}
\def\tK{{\tens{K}}}
\def\tL{{\tens{L}}}
\def\tM{{\tens{M}}}
\def\tN{{\tens{N}}}
\def\tO{{\tens{O}}}
\def\tP{{\tens{P}}}
\def\tQ{{\tens{Q}}}
\def\tR{{\tens{R}}}
\def\tS{{\tens{S}}}
\def\tT{{\tens{T}}}
\def\tU{{\tens{U}}}
\def\tV{{\tens{V}}}
\def\tW{{\tens{W}}}
\def\tX{{\tens{X}}}
\def\tY{{\tens{Y}}}
\def\tZ{{\tens{Z}}}
% Graph
\def\gA{{\mathcal{A}}}
\def\gB{{\mathcal{B}}}
\def\gC{{\mathcal{C}}}
\def\gD{{\mathcal{D}}}
\def\gE{{\mathcal{E}}}
\def\gF{{\mathcal{F}}}
\def\gG{{\mathcal{G}}}
\def\gH{{\mathcal{H}}}
\def\gI{{\mathcal{I}}}
\def\gJ{{\mathcal{J}}}
\def\gK{{\mathcal{K}}}
\def\gL{{\mathcal{L}}}
\def\gM{{\mathcal{M}}}
\def\gN{{\mathcal{N}}}
\def\gO{{\mathcal{O}}}
\def\gP{{\mathcal{P}}}
\def\gQ{{\mathcal{Q}}}
\def\gR{{\mathcal{R}}}
\def\gS{{\mathcal{S}}}
\def\gT{{\mathcal{T}}}
\def\gU{{\mathcal{U}}}
\def\gV{{\mathcal{V}}}
\def\gW{{\mathcal{W}}}
\def\gX{{\mathcal{X}}}
\def\gY{{\mathcal{Y}}}
\def\gZ{{\mathcal{Z}}}
% Sets
\def\sA{{\mathbb{A}}}
\def\sB{{\mathbb{B}}}
\def\sC{{\mathbb{C}}}
\def\sD{{\mathbb{D}}}
% Don't use a set called E, because this would be the same as our symbol
% for expectation.
\def\sF{{\mathbb{F}}}
\def\sG{{\mathbb{G}}}
\def\sH{{\mathbb{H}}}
\def\sI{{\mathbb{I}}}
\def\sJ{{\mathbb{J}}}
\def\sK{{\mathbb{K}}}
\def\sL{{\mathbb{L}}}
\def\sM{{\mathbb{M}}}
\def\sN{{\mathbb{N}}}
\def\sO{{\mathbb{O}}}
\def\sP{{\mathbb{P}}}
\def\sQ{{\mathbb{Q}}}
\def\sR{{\mathbb{R}}}
\def\sS{{\mathbb{S}}}
\def\sT{{\mathbb{T}}}
\def\sU{{\mathbb{U}}}
\def\sV{{\mathbb{V}}}
\def\sW{{\mathbb{W}}}
\def\sX{{\mathbb{X}}}
\def\sY{{\mathbb{Y}}}
\def\sZ{{\mathbb{Z}}}
% Entries of a matrix
\def\emLambda{{\Lambda}}
\def\emA{{A}}
\def\emB{{B}}
\def\emC{{C}}
\def\emD{{D}}
\def\emE{{E}}
\def\emF{{F}}
\def\emG{{G}}
\def\emH{{H}}
\def\emI{{I}}
\def\emJ{{J}}
\def\emK{{K}}
\def\emL{{L}}
\def\emM{{M}}
\def\emN{{N}}
\def\emO{{O}}
\def\emP{{P}}
\def\emQ{{Q}}
\def\emR{{R}}
\def\emS{{S}}
\def\emT{{T}}
\def\emU{{U}}
\def\emV{{V}}
\def\emW{{W}}
\def\emX{{X}}
\def\emY{{Y}}
\def\emZ{{Z}}
\def\emSigma{{\Sigma}}
% entries of a tensor
% Same font as tensor, without \bm wrapper
\newcommand{\etens}[1]{\mathsfit{#1}}
\def\etLambda{{\etens{\Lambda}}}
\def\etA{{\etens{A}}}
\def\etB{{\etens{B}}}
\def\etC{{\etens{C}}}
\def\etD{{\etens{D}}}
\def\etE{{\etens{E}}}
\def\etF{{\etens{F}}}
\def\etG{{\etens{G}}}
\def\etH{{\etens{H}}}
\def\etI{{\etens{I}}}
\def\etJ{{\etens{J}}}
\def\etK{{\etens{K}}}
\def\etL{{\etens{L}}}
\def\etM{{\etens{M}}}
\def\etN{{\etens{N}}}
\def\etO{{\etens{O}}}
\def\etP{{\etens{P}}}
\def\etQ{{\etens{Q}}}
\def\etR{{\etens{R}}}
\def\etS{{\etens{S}}}
\def\etT{{\etens{T}}}
\def\etU{{\etens{U}}}
\def\etV{{\etens{V}}}
\def\etW{{\etens{W}}}
\def\etX{{\etens{X}}}
\def\etY{{\etens{Y}}}
\def\etZ{{\etens{Z}}}
% The true underlying data generating distribution
\newcommand{\pdata}{p_{\rm{data}}}
% The empirical distribution defined by the training set
\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
% The model distribution
\newcommand{\pmodel}{p_{\rm{model}}}
\newcommand{\Pmodel}{P_{\rm{model}}}
\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
% Stochastic autoencoder distributions
\newcommand{\pencode}{p_{\rm{encoder}}}
\newcommand{\pdecode}{p_{\rm{decoder}}}
\newcommand{\precons}{p_{\rm{reconstruct}}}
\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
\newcommand{\E}{\mathbb{E}}
\newcommand{\Ls}{\mathcal{L}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\emp}{\tilde{p}}
\newcommand{\lr}{\alpha}
\newcommand{\reg}{\lambda}
\newcommand{\rect}{\mathrm{rectifier}}
\newcommand{\softmax}{\mathrm{softmax}}
\newcommand{\sigmoid}{\sigma}
\newcommand{\softplus}{\zeta}
\newcommand{\KL}{D_{\mathrm{KL}}}
\newcommand{\Var}{\mathrm{Var}}
\newcommand{\standarderror}{\mathrm{SE}}
\newcommand{\Cov}{\mathrm{Cov}}
% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
% But then they seem to use $L^2$ for vectors throughout the site, and so does
% wikipedia.
\newcommand{\normlzero}{L^0}
\newcommand{\normlone}{L^1}
\newcommand{\normltwo}{L^2}
\newcommand{\normlp}{L^p}
\newcommand{\normmax}{L^\infty}
\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator{\sign}{sign}
\DeclareMathOperator{\Tr}{Tr}
\let\ab\allowbreak
\newcommand{\btheta}{\boldsymbol{\theta}}
\newcommand{\bg}{\boldsymbol{g}}
\newcommand{\bh}{\boldsymbol{h}}
\newcommand{\bhtheta}{\boldsymbol{\hat{\theta}}}
\begin{comment}
\begin{table*}[!t]
\setlength\tabcolsep{1.4pt}
\centering
\tiny
\caption{$\mathcal{L}_T$ and $\mathcal{L}_M$ results of the different setups across all datasets. The values are computed over 3 splits, and the standard error is reported in parenthesis. Best results are highlighted in bold.}
\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{$\mathcal{L}_M$} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & 714.3 (18.1) & 93.5 (1.8) & 133.2 (28.5) & \textbf{40.7 (1.4)} & 104.9 (1.1) \\
THP\textbf{+} & 691.8 (14.1) & 77.1 (1.7) & 124.7 (26.2) & 43.5 (1.3) & 102.9 (1.1) \\
THP\textbf{++} & \textbf{647.0 (20.2)} & \textbf{71.1 (1.4)} & \textbf{117.6 (25.0)} & 41.2 (1.6) & \textbf{102.7 (1.1)} \\ \midrule
STHP\textbf{+} & 695.4 (19.5) & 83.2 (1.7) & 124.8 (25.8) & \textbf{44.0 (1.5)} & \textbf{103.6 (1.1)} \\
STHP\textbf{++} & \textbf{689.6 (14.5)} & \textbf{82.7 (1.9)} & \textbf{119.3 (24.3)} & 46.4 (0.3) & 104.8 (0.8) \\ \midrule
SAHP & 829.4 (28.3) & 162.8 (3.6) & 144.3 (31.2) & 73.2 (5.7) & 106.9 (0.6) \\
SAHP\textbf{+} & 739.4 (29.7) & 73.9 (1.4) & 121.1 (24.8) & 44.3 (1.6) & \textbf{102.9 (1.1)} \\
SAHP\textbf{++} & \textbf{651.3 (18.7)} & \textbf{71.0 (1.5)} & \textbf{118.5 (25.0)} & \textbf{41.2 (1.0)} & \textbf{102.9 (1.1)} \\ \midrule
LNM & 684.8 (14.4) & 86.3 (1.4) & 121.8 (24.8) & 43.5 (1.3) & 106.1 (1.1) \\
LNM\textbf{+} & 662.2 (21.3) & 77.1 (1.9) & 117.9 (24.5) & 42.2 (1.3) & \textbf{102.8 (1.1)} \\
LNM\textbf{++} & \textbf{630.8 (22.3)} & \textbf{73.7 (1.4)} & \textbf{116.9 (24.8)} & \textbf{41.4 (1.2)} & 102.9 (1.1) \\ \midrule
FNN & 739.3 (30.1) & 79.3 (1.8) & 119.1 (25.1) & 45.7 (1.2) & 107.0 (1.0) \\
FNN\textbf{+} & 666.6 (15.8) & 72.2 (1.4) & 117.2 (24.5) & 41.6 (0.9) & 103.0 (1.2) \\
FNN\textbf{++} & \textbf{647.9 (17.6)} & \textbf{71.8 (1.5)} & \textbf{114.9 (24.4)} & \textbf{40.6 (1.0)} & \textbf{102.8 (1.1)} \\ \midrule
RMTPP & 680.8 (13.3) & 87.3 (1.6) & 133.7 (29.6) & \textbf{41.7 (1.0)} & 106.2 (1.1) \\
RMTPP\textbf{+} & 676.9 (14.4) & 74.9 (1.7) & 123.6 (25.3) & 41.8 (0.9) & \textbf{102.7 (1.1)} \\
RMTPP\textbf{++} & \textbf{651.7 (17.3)} & \textbf{71.7 (1.5)} & \textbf{117.6 (25.0)} & \textbf{41.7 (0.7)} & 102.8 (1.1) \\
\bottomrule
\end{tabular}
\end{minipage}
\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{$\mathcal{L}_T$} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & -961.3 (44.0) & \textbf{-136.7 (1.8)} & -258.8 (62.4) & -71.7 (3.4) & -82.9 (2.1) \\
THP\textbf{+} & -1002.5 (47.7) & -130.5 (1.5) & -274.9 (67.3) & -84.6 (3.4) & \textbf{-83.6 (2.0)} \\
THP\textbf{++} & \textbf{-1052.8 (50.1)} & -135.6 (2.8) & \textbf{-297.3 (80.9)} & \textbf{-87.5 (2.9)} & -83.1 (2.1) \\ \midrule
STHP\textbf{+} & -1003.1 (55.7) & -128.5 (1.3) & -222.2 (51.8) & -76.2 (3.1) & \textbf{-82.4 (2.0)} \\
STHP\textbf{++} & \textbf{-1015.8 (50.8)} & \textbf{-133.0 (2.7)} & \textbf{-255.5 (62.9)} & \textbf{-77.7 (3.1)} & -82.2 (2.1) \\ \midrule
SAHP & -1266.9 (64.7) & -266.5 (4.3) & -380.2 (89.8) & -74.0 (2.6) & \textbf{-88.7 (2.2)} \\
SAHP\textbf{+} & \textbf{-1326.1 (67.9)} & -289.9 (4.6) & -391.7 (90.2) & -94.1 (3.3) & \textbf{-88.7 (2.1)} \\
SAHP\textbf{++} & -1322.6 (67.9) & \textbf{-295.3 (4.8)} & \textbf{-400.8 (92.6)} & \textbf{-94.8 (3.2)} & -76.6 (1.7) \\ \midrule
LNM & -1323.2 (57.1) & -311.6 (4.8) & -414.2 (94.0) & \textbf{-96.6 (2.9)} & -90.0 (2.1) \\
LNM\textbf{+} & -1315.5 (67.7) & \textbf{-312.0 (4.9)} & -412.7 (93.2) & -96.3 (3.1) & \textbf{-90.1 (2.0)} \\
LNM\textbf{++} & \textbf{-1334.7 (63.4)} & -308.8 (5.1) & \textbf{-415.5 (94.1)} & -95.4 (3.1) & -89.4 (2.1) \\ \midrule
FNN & -1271.9 (65.6) & -281.9 (4.1) & -395.4 (89.9) & -76.8 (2.8) & -80.2 (2.0) \\
FNN\textbf{+} & -1319.0 (63.0) & \textbf{-303.3 (4.4)} & \textbf{-396.3 (89.9)} & -93.9 (2.9) & \textbf{-89.0 (2.9)} \\
FNN\textbf{++} & \textbf{-1321.1 (62.8)} & -301.9 (4.7) & -395.5 (89.6) & \textbf{-95.6 (3.3)} & -87.7 (2.1) \\ \midrule
RMTPP & \textbf{-1075.0 (46.7)} & -177.7 (2.5) & -309.0 (80.5) & -87.1 (3.3) & -82.2 (2.1) \\
RMTPP\textbf{+} & -1049.9 (55.5) & \textbf{-190.7 (4.4)} & -301.8 (76.8) & \textbf{-87.3 (3.0)} & \textbf{-82.3 (2.1)} \\
RMTPP\textbf{++} & -1070.4 (53.8) & -181.4 (2.8) & \textbf{-324.1 (79.8)} & -86.2 (3.0) & -81.8 (2.1) \\
\bottomrule
\end{tabular}
\end{minipage}
\label{tabular_results}
\vspace{-0.3cm}
\end{table*}
\end{comment}
\begin{table*}[!t]
\setlength\tabcolsep{1.4pt}
\centering
\tiny
\caption{$\mathcal{L}_T$ and $\mathcal{L}_M$ results of the different setups across all datasets. The values are computed over 5 splits, and the standard error is reported in parenthesis. Best results are highlighted in bold.}
\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{$\mathcal{L}_M$} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & 714.0 (16.5) & 93.3 (1.4) & 128.3 (17.3) & \textbf{39.9 (1.3)} & 105.3 (0.7) \\
THP\textbf{+} & 695.9 (15.7) & 76.9 (1.2) & 120.8 (16.0) & 42.7 (1.2) & 103.3 (0.7) \\
THP\textbf{++} & \textbf{651.1 (18.1)} & \textbf{70.9 (1.0)} & \textbf{112.1 (15.4)} & 40.4 (1.3) & \textbf{103.0 (0.7)} \\ \midrule
STHP\textbf{+} & 700.9 (16.6) & 83.2 (1.3) & 121.4 (15.8) & \textbf{43.7 (1.1)} & \textbf{104.0 (0.7)} \\
STHP\textbf{++} & \textbf{696.5 (15.9)} & \textbf{82.4 (1.4)} & \textbf{114.7 (14.9)} & 46.2 (0.8) & 105.6 (0.6) \\ \midrule
SAHP & 825.5 (25.6) & 163.0 (2.2) & 138.0 (19.1) & 77.9 (4.7) & 108.4 (1.0) \\
SAHP\textbf{+} & 740.0 (24.6) & 73.8 (1.0) & 116.8 (15.2) & 43.4 (1.2) & 103.3 (0.7) \\
SAHP\textbf{++} & \textbf{654.8 (17.3)} & \textbf{71.0 (1.1)} & \textbf{114.1 (15.2)} & \textbf{40.5 (0.8)} & \textbf{103.1 (0.7)} \\ \midrule
LNM & 685.2 (15.8) & 86.6 (1.2) & 116.8 (15.2) & 43.4 (1.2) & 106.5 (0.7) \\
LNM\textbf{+} & 668.6 (15.8) & 77.1 (1.3) & 112.3 (15.1) & 41.4 (1.1) & \textbf{103.2 (0.7)} \\
LNM\textbf{++} & \textbf{637.2 (19.4)} & \textbf{73.8 (1.1)} & \textbf{111.5 (15.2)} & \textbf{40.8 (1.0)} & \textbf{103.2 (0.7)} \\ \midrule
FNN & 739.5 (25.2) & 78.8 (1.3) & 113.5 (15.4) & 47.0 (1.4) & 107.3 (0.6) \\
FNN\textbf{+} & 672.1 (17.9) & 72.3 (1.1) & 111.6 (15.1) & 41.2 (0.9) & 103.2 (0.7) \\
FNN\textbf{++} & \textbf{648.6 (16.2)} & \textbf{71.8 (1.0)} & \textbf{109.5 (15.0)} & \textbf{40.1 (1.0)} & \textbf{103.1 (0.7)} \\ \midrule
RMTPP & 684.6 (15.6) & 87.0 (1.2) & 126.4 (18.3) & 41.4 (0.9) & 106.5 (0.7) \\
RMTPP\textbf{+} & 681.5 (16.3) & 74.9 (1.3) & 118.7 (15.4) & 42.0 (1.0) & \textbf{103.0 (0.7)} \\
RMTPP\textbf{++} & \textbf{654.5 (16.7)} & \textbf{71.4 (1.1)} & \textbf{112.6 (15.3)} & \textbf{40.6 (1.2)} & 103.1 (0.7) \\
\bottomrule
\end{tabular}
\end{minipage}
\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{$\mathcal{L}_T$} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & -945.1 (41.3) & -135.9 (1.3) & -242.5 (38.5) & -72.3 (2.3) & -84.0 (1.4) \\
THP\textbf{+} & -994.1 (48.8) & -130.6 (1.3) & -255.7 (42.0) & -85.5 (2.3) & \textbf{-84.7 (1.4)} \\
THP\textbf{++} & \textbf{-1037.3 (47.0)} & \textbf{-136.0 (2.1)} & \textbf{-271.0 (50.8)} & \textbf{-87.9 (2.0)} & -84.2 (1.4) \\ \midrule
STHP\textbf{+} & -993.3 (44.2) & -128.2 (1.0) & -208.9 (32.2) & -76.6 (2.1) & \textbf{-83.6 (1.4)} \\
STHP\textbf{++} & \textbf{-1014.9 (42.1)} & \textbf{-132.9 (1.8)} & \textbf{-237.0 (39.3)} & \textbf{-78.2 (2.1)} & -83.3 (1.4) \\ \midrule
SAHP & -1263.8 (57.9) & -266.0 (3.5) & -346.5 (57.4) & -72.9 (1.8) & -89.7 (1.4) \\
SAHP\textbf{+} & \textbf{-1320.4 (58.4)} & -288.8 (3.5) & -358.1 (57.6) & -94.8 (2.3) & \textbf{-89.9 (1.4)} \\
SAHP\textbf{++} & -1320.0 (60.5) & \textbf{-293.8 (3.6)} & \textbf{-366.0 (59.3)} & \textbf{-95.4 (2.1)} & -77.4 (1.2) \\ \midrule
LNM & -1326.3 (55.8) & -310.0 (3.8) & -380.4 (59.8) & \textbf{-96.4 (2.0)} & -91.0 (1.4) \\
LNM\textbf{+} & -1320.4 (60.5) & \textbf{-310.6 (3.7)} & -378.7 (59.4) &\textbf{-96.4 (2.0)} & \textbf{-91.1 (1.3)} \\
LNM\textbf{++} & \textbf{-1334.7 (58.5)} & -307.6 (3.9) & \textbf{-381.2 (59.9)} & -96.3 (2.2) & -90.5 (1.4) \\ \midrule
FNN & -1276.2 (58.6) & -280.9 (3.2) & -363.1 (57.2) & -75.3 (2.5) & -81.2 (1.3) \\
FNN\textbf{+} & \textbf{-1324.6 (59.6)} & \textbf{-302.2 (3.6)} & -364.5 (57.0) & -94.4 (2.1) & \textbf{-90.0 (1.8)} \\
FNN\textbf{++} & -1324.2 (58.1) & -300.9 (3.7) & \textbf{-365.0 (56.7)} & \textbf{-96.1 (2.2)} & -88.9 (1.4) \\ \midrule
RMTPP & -1052.9 (46.0) & -178.6 (1.8) & -268.0 (54.3) & \textbf{-88.0 (2.2)} & -83.3 (1.4) \\
RMTPP\textbf{+} & -1040.7 (45.8) & \textbf{-187.8 (3.1)} & -272.2 (49.0) & -87.9 (2.0) & \textbf{-83.4 (1.4)} \\
RMTPP\textbf{++} & \textbf{-1071.1 (50.3)} & -182.2 (1.9) & \textbf{-287.2 (52.6)} & -86.6 (2.0) & -82.9 (1.4) \\
\bottomrule
\end{tabular}
\end{minipage}
\label{tabular_results}
\vspace{-0.3cm}
\end{table*}
\begin{comment}
\begin{table*}[!t]
\setlength\tabcolsep{0.5pt}
\centering
\tiny
\caption{$\mathcal{L}_T$ and $\mathcal{L}_M$ results for the base, base-D, and base-DD settings across all datasets. The values are computed over 3 splits, and the standard error is reported in parenthesis. Best results are highlighted in bold.}
%\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{$\mathcal{L}_M$} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & 714.3 (18.1) & 93.5 (1.8) & 133.2 (28.5) & 40.7 (1.4) & 104.9 (1.1) \\
THP-D & 669.0 (20.6) & 83.0 (1.6) & 126.5 (27.7) & 39.7 (0.8) & 104.4 (1.1) \\
THP-DD & \bf{635.6 (17.3)} & \bf{82.9 (1.4)} & \bf{119.5 (24.9)} & \bf{39.1 (1.0)} & \bf{104.1 (1.1)} \\ \midrule
SAHP & 829.4 (28.3) & 162.8 (3.6) & 144.3 (31.2) & 73.2 (5.7) & 106.9 (0.6) \\
SAHP-D & 814.9 (32.8) & 93.7 (2.4) & 133.0 (30.0) & 54.9 (1.2) & \bf{104.6 (1.1)} \\
SAHP-DD & \bf{709.3 (23.8)} & \bf{89.2 (2.2)} & \bf{122.3 (25.2)} & \bf{51.7 (0.6)} & \bf{104.6 (1.2)} \\ \midrule
FNN & 739.3 (30.1) & 79.3 (1.8) & 119.1 (25.1) & \bf{45.7 (1.2)} & 107.0 (1.0) \\
FNN-D & 731.7 (18.1) & \bf{76.5 (1.7)} & 118.3 (25.0) & 57.6 (1.2) & \bf{103.4 (1.1)} \\
FNN-DD & \bf{719.1 (15.1)} & 78.2 (1.1) & \bf{117.6 (24.4)} & 54.6 (1.5) & 103.7 (1.1) \\
\bottomrule
\end{tabular}
%\end{minipage}
%\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{$\mathcal{L}_T$} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & -961.3 (44.0) & -136.7 (1.8) & -258.8 (62.4) & -71.7 (3.4) & -82.9 (2.1) \\
THP-D & \bf{-1009.4 (42.6)} & \bf{-162.9 (2.4)} & -287.0 (74.0) & -89.4 (2.5) & \bf{-84.2 (2.1)} \\
THP-DD & -1004.3 (49.1) & -151.5 (2.3) & \bf{-313.5 (83.4)} & \bf{-92.0 (3.1)} & -83.9 (2.1) \\ \midrule
SAHP & -1266.9 (64.7) & -266.5 (4.3) & -380.2 (89.8) & -74.0 (2.6) & -88.7 (2.2) \\
SAHP-D & \bf{-1320.7 (65.4)} & -289.6 (4.9) & -386.3 (90.2) & -93.3 (3.3) & \bf{-89.2 (2.0)} \\
SAHP-DD & -1314.4 (67.4) & \bf{-295.7 (4.8)} & \bf{-399.6 (92.6)} & \bf{-95.2 (3.2)} & -88.6 (2.1) \\ \midrule
FNN & -1271.9 (65.6) & -281.9 (4.1) & \bf{-395.4 (89.9)} & -76.8 (2.8) & -80.2 (2.0) \\
FNN-D & \bf{-1309.1 (65.7)} & -287.2 (4.7) & -390.6 (88.1) & -81.1 (2.8) & \bf{-80.6 (2.0)} \\
FNN-DD & -1291.8 (56.2) & \bf{-288.6 (4.2)} & -393.6 (89.2) & \bf{-82.5 (2.8)} & -80.3 (2.0) \\
\bottomrule
\end{tabular}
%\end{minipage}
\label{tab:neurips:d_dd_results}
\vspace{-0.3cm}
\end{table*}
\end{comment}
\begin{table*}[t!]
\setlength\tabcolsep{0.5pt}
\centering
\tiny
\caption{$\mathcal{L}_T$ and $\mathcal{L}_M$ results for the base, base-D, and base-DD settings across all datasets. The values are computed over 5 splits, and the standard error is reported in parenthesis. Best results are highlighted in bold.}
%\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{$\mathcal{L}_M$} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & 714.0 (16.5) & 93.3 (1.4) & 128.3 (17.3) & 39.9 (1.3) & 105.3 (0.7) \\
THP-D & 677.1 (19.4) & 82.8 (1.2) & 121.4 (16.8) & 39.1 (0.9) & 104.7 (0.7) \\
THP-DD & \textbf{607.2 (16.2)} & \textbf{79.8 (1.2)} & \textbf{113.5 (14.9)} & \textbf{38.2 (1.0)} & \textbf{104.4 (0.6)} \\ \midrule
SAHP & 825.5 (25.6) & 163.0 (2.2) & 138.0 (19.1) & 77.9 (4.7) & 108.4 (1.0) \\
SAHP-D & 832.1 (32.0) & 93.3 (1.5) & 128.8 (18.5) & 56.0 (1.0) & 105.1 (0.7) \\
SAHP-DD & \textbf{692.1 (19.7)} & \textbf{89.0 (1.6)} & \textbf{115.4 (15.3)} & \textbf{51.0 (0.5)} & \textbf{104.8 (0.6)} \\ \midrule
FNN & 739.5 (25.2) & 78.8 (1.3) & 113.5 (15.4) & \textbf{47.0 (1.4)} & 107.3 (0.6) \\
FNN-D & 732.8 (19.7) & \textbf{76.8 (1.2)} & 112.9 (15.4) & 56.9 (0.9) & 103.8 (0.7) \\
FNN-DD & \textbf{670.0 (18.1)} & 79.7 (1.4) & \textbf{111.4 (15.3)} & 48.8 (1.6) & \textbf{103.7 (0.7)} \\
\bottomrule
\end{tabular}
%\end{minipage}
%\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{$\mathcal{L}_T$} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & -945.1 (41.3) & -135.9 (1.3) & -242.5 (38.5) & -72.3 (2.3) & -84.0 (1.4) \\
THP-D & -995.1 (50.2) & \textbf{-164.8 (1.8)} & -258.5 (47.2) & -90.3 (1.9) & \textbf{-85.3 (1.4)} \\
THP-DD & \textbf{-1023.8 (53.0)} & -162.1 (4.3) & \textbf{-279.0 (55.2)} & \textbf{-92.6 (2.1)} & -84.8 (1.4) \\ \midrule
SAHP & -1263.8 (57.9) & -266.0 (3.5) & -346.5 (57.4) & -72.9 (1.8) & -89.7 (1.4) \\
SAHP-D & -1319.1 (57.9) & -288.7 (3.7) & -348.7 (58.2) & \textbf{-92.9 (2.3)} & \textbf{-90.1 (1.3)} \\
SAHP-DD & \textbf{-1319.9 (59.3)} & \textbf{-294.0 (3.6)} & \textbf{-367.9 (59.1)} & -87.0 (3.5) & -87.1 (2.6) \\ \midrule
FNN & -1276.2 (58.6) & -280.9 (3.2) & \textbf{-363.1 (57.2)} & -75.3 (2.5) & -81.2 (1.3) \\
FNN-D & \textbf{-1314.1 (59.7)} & \textbf{-286.8 (3.5)} & -360.0 (55.9) & -81.3 (1.7) & \textbf{-81.7 (1.3)} \\
FNN-DD & -1294.4 (57.5) & -286.3 (3.8) & -362.5 (56.4) & \textbf{-82.7 (2.2)} & -81.4 (1.4) \\
\bottomrule
\end{tabular}
%\end{minipage}
\label{tab:neurips:d_dd_results}
\vspace{-0.3cm}
\end{table*}
\begin{table*}[h!]
\setlength\tabcolsep{1pt}
\centering
\tiny
\caption{PCE and ECE results of the different setups across all datasets. The values are computed over 5 splits, and the standard error is reported in parenthesis. Best results are highlighted in bold.}
\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{PCE} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & 0.28 (0.0) & \textbf{0.37 (0.0)} & 0.2 (0.01) & 0.12 (0.0) & \textbf{0.01 (0.0)} \\
THP\textbf{+} & \textbf{0.27 (0.01)} & \textbf{0.37 (0.0)} & \textbf{0.19 (0.01)} & 0.07 (0.0) & \textbf{0.01 (0.0)} \\
THP\textbf{++} & 0.28 (0.01) & \textbf{0.37 (0.0)} & \textbf{0.19 (0.02)} & \textbf{0.05 (0.0)} & \textbf{0.01 (0.0)} \\ \midrule
STHP\textbf{+} & \textbf{0.29 (0.01)} & 0.37 (0.0) & 0.29 (0.02) & \textbf{0.1 (0.0)} & \textbf{0.01 (0.0)} \\
STHP\textbf{++} & \textbf{0.29 (0.01)} & \textbf{0.36 (0.0)} & \textbf{0.25 (0.02)} & \textbf{0.1 (0.0)} & \textbf{0.01 (0.0)} \\ \midrule
SAHP & 0.06 (0.01) & 0.12 (0.0) & 0.07 (0.0) & 0.1 (0.01) & \textbf{0.01 (0.0)} \\
SAHP\textbf{+} & 0.05 (0.01) & \textbf{0.03 (0.0)} & \textbf{0.04 (0.01)} & \textbf{0.01 (0.0)} & \textbf{0.0 (0.0)} \\
SAHP\textbf{++} & \textbf{0.04 (0.01)} & \textbf{0.03 (0.0)} & \textbf{0.04 (0.01)} & 0.01 (0.0) & 0.04 (0.0) \\ \midrule
LNM & \textbf{0.03 (0.01)} & \textbf{0.01 (0.0)} & \textbf{0.02 (0.0)} & \textbf{0.01 (0.0)} & \textbf{0.0 (0.0)} \\
LNM\textbf{+} & 0.05 (0.01) & \textbf{0.01 (0.0)} & 0.03 (0.0) & \textbf{0.01 (0.0)} & \textbf{0.0 (0.0)} \\
LNM\textbf{++} & \textbf{0.03 (0.0)} & \textbf{0.01 (0.0)} & 0.03 (0.0) & \textbf{0.01 (0.0)} & \textbf{0.0 (0.0)} \\ \midrule
FNN & 0.04 (0.0) & 0.09 (0.0) & 0.04 (0.01) & 0.07 (0.0) & 0.05 (0.0) \\
FNN\textbf{+} & 0.03 (0.0) & \textbf{0.01 (0.0)} & 0.03 (0.01) & \textbf{0.01 (0.0)} & \textbf{0.01 (0.0)} \\
FNN\textbf{++} & \textbf{0.02 (0.0)} & \textbf{0.01 (0.0)} & \textbf{0.02 (0.0)} & \textbf{0.01 (0.0)} & \textbf{0.01 (0.0)} \\ \midrule
RMTPP & \textbf{0.25 (0.01)} & 0.29 (0.0) & 0.18 (0.01) & \textbf{0.03 (0.0)} & \textbf{0.01 (0.0)} \\
RMTPP\textbf{+} & 0.26 (0.01) & \textbf{0.27 (0.01)} & 0.18 (0.01) & \textbf{0.03 (0.0)} & \textbf{0.01 (0.0)} \\
RMTPP\textbf{++} & 0.26 (0.0) & 0.29 (0.0) & \textbf{0.17 (0.01)} & 0.04 (0.0) & \textbf{0.01 (0.0)} \\
\bottomrule
\end{tabular}
\end{minipage}
\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{ECE} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & 0.23 (0.03) & 0.07 (0.0) & 0.09 (0.02) & \textbf{0.02 (0.0)} & 0.04 (0.02) \\
THP\textbf{+} & 0.05 (0.01) & \textbf{0.02 (0.0)} & \textbf{0.07 (0.01)} & 0.03 (0.01) & \textbf{0.01 (0.0)} \\
THP\textbf{++} & \textbf{0.03 (0.0)} & \textbf{0.02 (0.0)} & \textbf{0.07 (0.02)} & \textbf{0.02 (0.0)} & \textbf{0.01 (0.0)} \\ \midrule
STHP\textbf{+} & 0.05 (0.0) & \textbf{0.03 (0.0)} & 0.06 (0.02) & \textbf{0.03 (0.0)} & \textbf{0.02 (0.0)} \\
STHP\textbf{++} & \textbf{0.04 (0.0)} & \textbf{0.04 (0.0)} & 0.04 (0.01) & 0.05 (0.01) & 0.03 (0.01) \\ \midrule
SAHP & 0.11 (0.01) & 0.13 (0.0) & 0.08 (0.02) & 0.08 (0.01) & 0.03 (0.0) \\
SAHP\textbf{+} & 0.06 (0.01) & \textbf{0.02 (0.0)} & \textbf{0.07 (0.02)} & 0.03 (0.01) & \textbf{0.01 (0.0)} \\
SAHP\textbf{++} & \textbf{0.03 (0.0)} & \textbf{0.02 (0.01)} & \textbf{0.07 (0.01)} & \textbf{0.02 (0.0)} & \textbf{0.01 (0.0)} \\ \midrule
LNM & 0.09 (0.02) & 0.07 (0.01) & \textbf{0.05 (0.01)} & 0.03 (0.01) & 0.03 (0.01) \\
LNM\textbf{+} & 0.05 (0.01) & 0.04 (0.01) & 0.06 (0.01) & \textbf{0.02 (0.0)} & \textbf{0.01 (0.0)} \\
LNM\textbf{++} & \textbf{0.02 (0.0)} & \textbf{0.02 (0.0)} & \textbf{0.05 (0.01)} & \textbf{0.02 (0.0)} & \textbf{0.01 (0.0)} \\ \midrule
FNN & 0.08 (0.01) & 0.05 (0.0) & \textbf{0.05 (0.0)} & 0.04 (0.01) & 0.02 (0.0) \\
FNN\textbf{+} & 0.04 (0.01) & \textbf{0.02 (0.0)} & 0.06 (0.0) & \textbf{0.02 (0.0)} & \textbf{0.01 (0.0)} \\
FNN\textbf{++} & \textbf{0.03 (0.0)} & \textbf{0.02 (0.0)} & 0.07 (0.02) & \textbf{0.02 (0.0)} & \textbf{0.01 (0.0)} \\ \midrule
RMTPP & 0.05 (0.01) & 0.06 (0.01) & 0.07 (0.0) & 0.03 (0.0) & 0.03 (0.02) \\
RMTPP\textbf{+} & 0.05 (0.01) & 0.03 (0.01) & \textbf{0.06 (0.0)} & 0.03 (0.01) & \textbf{0.01 (0.0)} \\
RMTPP\textbf{++} & \textbf{0.04 (0.01)} & \textbf{0.02 (0.0)} & \textbf{0.06 (0.01)} & \textbf{0.02 (0.0)} & \textbf{0.01 (0.0)} \\
\bottomrule
\end{tabular}
\end{minipage}
\label{tab:pce_ece}
\end{table*}
\begin{table*}[h!]
\setlength\tabcolsep{1pt}
\centering
\tiny
\caption{Accuracy@1 and Accuracy@3 results of the different setups across all datasets. The values are computed over 5 splits, and the standard error is reported in parenthesis. Best results are highlighted in bold.}
\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{Accuracy} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & 0.18 (0.01) & 0.4 (0.0) & 0.59 (0.02) & \textbf{0.83 (0.0)} & 0.48 (0.0) \\
THP\textbf{+} & 0.21 (0.01) & 0.52 (0.0) & 0.64 (0.01) & 0.82 (0.0) & \textbf{0.49 (0.0)} \\
THP\textbf{++} & \textbf{0.25 (0.01)} & \textbf{0.55 (0.0)} & \textbf{0.67 (0.01)} & 0.82 (0.0) & \textbf{0.49 (0.0)} \\ \midrule
STHP\textbf{+} & 0.2 (0.01) & \textbf{0.46 (0.0)} & 0.63 (0.01) & \textbf{0.81 (0.0)} & \textbf{0.48 (0.0)} \\
STHP\textbf{++} & \textbf{0.21 (0.01)} & \textbf{0.46 (0.0)} & \textbf{0.65 (0.01)} & \textbf{0.81 (0.0)} & \textbf{0.48 (0.0)} \\ \midrule
SAHP & 0.05 (0.0) & 0.36 (0.01) & 0.6 (0.01) & 0.69 (0.02) & 0.48 (0.0) \\
SAHP\textbf{+} & 0.14 (0.01) & 0.54 (0.0) & 0.65 (0.01) & \textbf{0.82 (0.0)} & \textbf{0.49 (0.0)} \\
SAHP\textbf{++} & \textbf{0.24 (0.01)} & \textbf{0.56 (0.0)} & \textbf{0.67 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.49 (0.0)} \\ \midrule
LNM & 0.21 (0.01) & 0.43 (0.0) & 0.64 (0.01) & 0.81 (0.0) & 0.47 (0.0) \\
LNM\textbf{+} & 0.24 (0.01) & 0.52 (0.0) & \textbf{0.67 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.49 (0.0)} \\
LNM\textbf{++} & \textbf{0.25 (0.01)} & \textbf{0.54 (0.0)} & \textbf{0.67 (0.01)} & \textbf{0.82 (0.0)} & 0.48 (0.0) \\ \midrule
FNN & 0.13 (0.01) & 0.51 (0.0) & \textbf{0.67 (0.01)} & 0.8 (0.01) & 0.48 (0.0) \\
FNN\textbf{+} & 0.23 (0.01) & \textbf{0.55 (0.0)} & \textbf{0.67 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.49 (0.0)} \\
FNN\textbf{++} & \textbf{0.25 (0.01)} & \textbf{0.55 (0.0)} & \textbf{0.67 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.49 (0.0)} \\ \midrule
RMTPP & 0.23 (0.01) & 0.42 (0.0) & 0.61 (0.02) & \textbf{0.82 (0.0)} & 0.47 (0.0) \\
RMTPP\textbf{+} & 0.23 (0.01) & 0.53 (0.0) & 0.64 (0.01) & \textbf{0.82 (0.0)} & \textbf{0.49 (0.0)} \\
RMTPP\textbf{++} & \textbf{0.24 (0.01)} & \textbf{0.55 (0.0)} & \textbf{0.67 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.49 (0.0)} \\
\bottomrule
\end{tabular}
\end{minipage}
\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{Accuracy@3} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & 0.36 (0.01) & 0.75 (0.0) & 0.86 (0.01) & \textbf{0.91 (0.0)} & 0.82 (0.0) \\
THP+ & 0.37 (0.01) & 0.8 (0.0) & 0.86 (0.0) & 0.9 (0.0) & \textbf{0.83 (0.0)} \\
THP++ & \textbf{0.42 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.89 (0.0)} & 0.9 (0.01) & \textbf{0.83 (0.0)} \\ \midrule
STHP+ & 0.36 (0.01) & \textbf{0.78 (0.0)} & 0.88 (0.01) & \textbf{0.89 (0.0)} & \textbf{0.83 (0.0)} \\
STHP++ & \textbf{0.37 (0.01)} & \textbf{ 0.78 (0.0)} & \textbf{0.89 (0.0)} & \textbf{0.89 (0.0)} & 0.82 (0.0) \\ \midrule
SAHP & 0.14 (0.0) & 0.57 (0.01) & 0.86 (0.01) & 0.79 (0.02) & 0.81 (0.0) \\
SAHP+ & 0.29 (0.01) & 0.81 (0.0) & 0.88 (0.0) & \textbf{0.9 (0.0)} & \textbf{0.83 (0.0)} \\
SAHP++ & \textbf{0.42 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.89 (0.0)} & \textbf{0.9 (0.0)} & \textbf{0.83 (0.0)} \\ \midrule
LNM & 0.38 (0.02) & 0.76 (0.0) & 0.88 (0.0) & 0.89 (0.0) & 0.82 (0.0) \\
LNM+ & 0.41 (0.01) & 0.8 (0.0) & \textbf{0.89 (0.0)} & \textbf{0.9 (0.0)} & \textbf{0.83 (0.0)} \\
LNM++ & \textbf{0.44 (0.01)} & \textbf{0.81 (0.0)} & \textbf{0.89 (0.0)} & \textbf{0.9 (0.0)} & \textbf{0.83 (0.0)} \\ \midrule
FNN & 0.3 (0.01) & 0.79 (0.0) & 0.89 (0.0) & 0.89 (0.0) & 0.81 (0.0) \\
FNN+ & 0.4 (0.01) & 0.81 (0.0) & 0.89 (0.0) & 0.9 (0.0) & \textbf{0.83 (0.0)} \\
FNN++ & \textbf{0.42 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.9 (0.0)} & \textbf{0.91 (0.0)} & \textbf{0.83 (0.0)} \\ \midrule
RMTPP & 0.39 (0.01) & 0.76 (0.0) & 0.86 (0.0) & \textbf{0.9 (0.0)} & 0.82 (0.0) \\
RMTPP+ & 0.39 (0.01) & 0.8 (0.0) & 0.87 (0.01) & \textbf{0.9 (0.0)} & \textbf{0.83 (0.0)} \\
RMTPP++ & \textbf{0.42 (0.01)} & \textbf{0.82 (0.0)} & \textbf{0.89 (0.0)} & \textbf{0.9 (0.0)} & \textbf{0.83 (0.0)} \\
\bottomrule
\end{tabular}
\end{minipage}
\label{tab:acc1_acc3}
\end{table*}
\begin{table*}[h!]
\setlength\tabcolsep{1pt}
\centering
\tiny
\caption{Accuracy@5 and MRR results of the different setups across all datasets. The values are computed over 5 splits, and the standard error is reported in parenthesis. Best results are highlighted in bold.}
\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{Accuracy@5} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & 0.46 (0.01) & 0.87 (0.0) & 0.95 (0.01) & \textbf{0.93 (0.0)} & \textbf{0.93 (0.0)} \\
THP+ & 0.47 (0.01) & 0.89 (0.0) & 0.96 (0.0) & 0.92 (0.0) & \textbf{0.93 (0.0)} \\
THP++ & \textbf{0.52 (0.01)} & \textbf{0.9 (0.0)} & \textbf{0.97 (0.0)} & \textbf{0.93 (0.0)} & \textbf{0.93 (0.0)} \\ \midrule
STHP+ & \textbf{0.46 (0.01)} & 0.88 (0.0) & 0.96 (0.0) & \textbf{0.92 (0.0)} & \textbf{0.93 (0.0)} \\
STHP++ & \textbf{0.46 (0.01)} & \textbf{0.89 (0.0)} & \textbf{0.97 (0.0)} & 0.91 (0.0) & 0.92 (0.0) \\ \midrule
SAHP & 0.22 (0.01) & 0.67 (0.01) & 0.95 (0.0) & 0.84 (0.02) & 0.91 (0.0) \\
SAHP+ & 0.39 (0.01) & 0.89 (0.0) & \textbf{0.97 (0.0)} & 0.92 (0.0) & \textbf{0.93 (0.0)} \\
SAHP++ & \textbf{0.52 (0.01)} & \textbf{0.9 (0.0)} & \textbf{0.97 (0.0)} & \textbf{0.93 (0.0)} & \textbf{0.93 (0.0)} \\ \midrule
LNM & 0.47 (0.02) & 0.88 (0.0) & 0.96 (0.0) & 0.92 (0.0) & 0.92 (0.0) \\
LNM+ & 0.51 (0.01) & 0.89 (0.0) & \textbf{0.97 (0.0)} & \textbf{ 0.93 (0.0)} & \textbf{0.93 (0.0)} \\
LNM++ & \textbf{0.54 (0.01)} & \textbf{0.9 (0.0)} & \textbf{0.97 (0.0)} & \textbf{0.93 (0.0)} & \textbf{0.93 (0.0)} \\ \midrule
FNN & 0.4 (0.01) & 0.88 (0.0) & \textbf{0.97 (0.0)} & 0.92 (0.0) & 0.91 (0.0) \\
FNN+ & 0.5 (0.01) & \textbf{0.9 (0.0)} & \textbf{0.97 (0.0)} & \textbf{0.93 (0.0)} & \textbf{0.93 (0.0)} \\
FNN++ & \textbf{0.53 (0.01)} & \textbf{0.9 (0.0)} & \textbf{0.97 (0.0)} & \textbf{0.93 (0.0)} & \textbf{0.93 (0.0)} \\ \midrule
RMTPP & 0.48 (0.01) & 0.88 (0.0) & 0.96 (0.0) & \textbf{0.93 (0.0)} & 0.92 (0.0) \\
RMTPP+ & 0.49 (0.01) & 0.89 (0.0) & 0.96 (0.0) & \textbf{0.93 (0.0)} & \textbf{0.93 (0.0)} \\
RMTPP++ & \textbf{0.52 (0.01)} & \textbf{0.9 (0.0)} & \textbf{ 0.97 (0.0)} & \textbf{0.93 (0.0)} & \textbf{0.93 (0.0)} \\
\bottomrule
\end{tabular}
\end{minipage}
\begin{minipage}{.49\linewidth}
\centering
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{MRR} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule
\midrule
THP & 0.32 (0.01) & 0.6 (0.0) & 0.74 (0.01) & \textbf{0.87 (0.0)} & \textbf{0.67 (0.0)} \\
THP+ & 0.34 (0.01) & 0.68 (0.0) & 0.77 (0.01) & \textbf{0.87 (0.0)} & \textbf{0.67 (0.0)} \\
THP++ & \textbf{0.38 (0.01)} & \textbf{0.7 (0.0)} & \textbf{0.79 (0.01)} & \textbf{0.87 (0.0)} & \textbf{0.67 (0.0)} \\ \midrule
STHP+ & 0.33 (0.01) & \textbf{0.64 (0.0)} & 0.77 (0.01) & \textbf{0.86 (0.0)} & \textbf{0.67 (0.0)} \\
STHP++ & \textbf{0.34 (0.01)} & \textbf{0.64 (0.0)} & \textbf{0.78 (0.0)} & \textbf{0.86 (0.0)} & \textbf{0.67 (0.0)} \\ \midrule
SAHP & 0.16 (0.0) & 0.5 (0.01) & 0.74 (0.01) & 0.76 (0.02) & \textbf{0.67 (0.0)} \\
SAHP+ & 0.27 (0.01) & 0.69 (0.0) & 0.78 (0.01) & 0.86 (0.0) & \textbf{0.67 (0.0)} \\
SAHP++ & \textbf{0.38 (0.01)} & \textbf{0.7 (0.0)} & \textbf{0.79 (0.01)} & \textbf{0.87 (0.0)} & \textbf{0.67 (0.0)} \\ \midrule
LNM & 0.35 (0.01) & 0.62 (0.0) & 0.77 (0.0) & 0.86 (0.0) & 0.66 (0.0) \\
LNM+ & 0.37 (0.01) & 0.68 (0.0) & \textbf{0.79 (0.0)} & \textbf{0.87 (0.0)} & \textbf{0.67 (0.0)} \\
LNM++ & \textbf{0.4 (0.01)} & \textbf{0.69 (0.0)} & \textbf{0.79 (0.01)} & \textbf{0.87 (0.0)} & \textbf{0.67 (0.0)} \\ \midrule
FNN & 0.27 (0.01) & 0.67 (0.0) & 0.79 (0.01) & 0.85 (0.0) & 0.66 (0.0) \\
FNN+ & 0.36 (0.01) & \textbf{0.7 (0.0)} & 0.79 (0.0) & \textbf{0.87 (0.0)} & \textbf{0.67 (0.0)} \\
FNN++ & \textbf{0.38 (0.01)} & \textbf{0.7 (0.0)} & \textbf{0.8 (0.0)} & \textbf{0.87 (0.0)} & \textbf{0.67 (0.0)} \\ \midrule
RMTPP & 0.36 (0.01) & 0.61 (0.0) & 0.75 (0.01) & \textbf{0.87 (0.0)} & 0.66 (0.0) \\
RMTPP+ & 0.36 (0.01) & 0.69 (0.0) & 0.77 (0.01) & \textbf{0.87 (0.0)} & \textbf{0.67 (0.0)} \\
RMTPP++ & \textbf{0.38 (0.01)} & \textbf{0.7 (0.0)} & \textbf{0.79 (0.01)} & \textbf{0.87 (0.0)} & \textbf{0.67 (0.0)} \\
\bottomrule
\end{tabular}
\end{minipage}
\label{tab:acc5_mrr}
\end{table*}
\begin{table*}[h!]
\setlength\tabcolsep{1pt}
\centering
\tiny
\caption{MAE results of the different setups across all datasets. The values are computed over 5 splits, and the standard error is reported in parenthesis. Best results are highlighted in bold.}
\centering
\begin{adjustbox}{width=\textwidth}
\begin{tabular}{cccccc}
\toprule
& \multicolumn{5}{c}{MAE} \\
& LastFM & MOOC & Github & Reddit & Stack O. \\
\midrule \midrule
THP & \textbf{0.01624 (0.00056)} & \textbf{0.13332 (0.00096)} & 0.0935 (0.01328) & \textbf{0.15257 (0.00322)} & \textbf{0.09885 (0.00097)} \\
THP+ & 0.01925 (0.00165) & 0.13638 (0.0016) & 0.09769 (0.014) & 0.15811 (0.00079) & 0.09972 (0.00077) \\
THP++ & 0.01744 (0.00103) & 0.13365 (0.00134) & \textbf{0.09327 (0.01162)} & 0.17627 (0.00359) & 0.09926 (0.00069) \\ \midrule
STHP+ & \textbf{0.03717 (0.00978)} & 0.17422 (0.01899) & 0.11691 (0.00889) & \textbf{0.15864 (0.00409)} & \textbf{0.09883 (0.0007)} \\
STHP++ & 0.08356 (0.01887) & \textbf{0.17057 (0.02083)} & \textbf{0.1029 (0.01195)} & 0.22542 (0.02801) & 0.09931 (0.00056) \\ \midrule
SAHP & 0.0137 (0.00055) & 0.12319 (0.00174) & 0.08409 (0.01092) & 0.18839 (0.00389) & 0.0992 (0.00066) \\
SAHP+ & 0.01364 (0.00056) & 0.12318 (0.00191) & 0.07491 (0.0117) & 0.13427 (0.00216) & \textbf{0.09853 (0.00071)} \\
SAHP++ & \textbf{0.01358 (0.0006)} & \textbf{0.12138 (0.00197)} & \textbf{0.07127 (0.0102)} & \textbf{0.13168 (0.00206)} & 0.10285 (0.00057) \\ \midrule
FNN & 0.03563 (0.0138) & 0.16602 (0.0487) & 0.41847 (0.19173) & 0.23207 (0.05185) & 0.266 (0.09399) \\
FNN+ & 0.01055 (0.00043) & 0.07732 (0.00034) & \textbf{0.07093 (0.00902)} & 0.13157 (0.00208) & \textbf{0.09299 (0.00072)} \\
FNN++ & \textbf{0.01054 (0.00042)} & \textbf{0.07711 (0.00039)} & 0.07102 (0.0093) & \textbf{0.13148 (0.00231)} & 0.09305 (0.00064) \\ \midrule
LNM & 0.0111 (0.00049) & 0.13516 (0.00078) & 0.08706 (0.01153) & \textbf{0.13494 (0.00202)} & 0.09494 (0.00075) \\
LNM+ & 0.01131 (0.00054) & 0.13548 (0.0011) & 0.08715 (0.0114) & 0.135 (0.00213) & 0.09517 (0.00069) \\
LNM++ & \textbf{0.01098 (0.00049)} & \textbf{0.13083 (0.00096)} & \textbf{0.08694 (0.01135)} & 0.1361 (0.00209) & \textbf{0.09476 (0.00071)} \\ \midrule
RMTPP & \textbf{0.01619 (0.00097)} & 0.1208 (0.00084) & 0.12939 (0.02772) & \textbf{0.13577 (0.00223)} & \textbf{0.09522 (0.00069)} \\
RMTPP+ & 0.01677 (0.00072) & 0.12059 (0.00083) & 0.11416 (0.02624) & 0.1362 (0.00227) & 0.09563 (0.00085) \\
RMTPP++ & 0.01705 (0.00058) & \textbf{0.12011 (0.00079)} & \textbf{0.09183 (0.0106)} & 0.13772 (0.00192) & 0.09656 (0.0008) \\
\bottomrule
\end{tabular}
\end{adjustbox}
\label{tab:mae}
\end{table*}
\begin{thebibliography}{70}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi: #1}\else
\providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi
\bibitem[Bacry et~al.(2015)Bacry, Mastromatteo, and Muzy]{bacry2015hawkesprocessesfinance}
Emmanuel Bacry, Iacopo Mastromatteo, and Jean-François Muzy.
\newblock Hawkes processes in finance.
\newblock \emph{ArXiv preprint}, 2015.
\bibitem[Ben~Taieb(2022)]{souhaib}
Souhaib Ben~Taieb.
\newblock Learning quantile functions for temporal point processes with recurrent neural splines.
\newblock \emph{Proceedings of The 25th International Conference on Artificial Intelligence and Statistics (AISTATS)}, 2022.
\bibitem[Biloš et~al.(2019)Biloš, Charpentier, and Günnemann]{biloš2020uncertainty}
Marin Biloš, Bertrand Charpentier, and Stephan Günnemann.
\newblock Uncertainty on asynchronous time event prediction.
\newblock \emph{Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS)}, 2019.
\bibitem[Bosser \& Ben~Taieb(2023)Bosser and Ben~Taieb]{bosser2023predictive}
Tanguy Bosser and Souhaib Ben~Taieb.
\newblock On the predictive accuracy of neural temporal point process models for continuous-time event data.
\newblock \emph{Transactions of Machine Learning Research (TMLR)}, 2023.
\bibitem[Brehmer et~al.(2023)Brehmer, Gneiting, Herrmann, Marzocchi, Schlather, and Strokorb]{brehmer2021comparative}
Jonas Brehmer, Tilmann Gneiting, Marcus Herrmann, Warner Marzocchi, Martin Schlather, and Kirstin Strokorb.
\newblock Comparative evaluation of point process forecasts.
\newblock \emph{Annals of the Institute of Statistical Mathematics}, 76:\penalty0 47--71, 2023.
\bibitem[Brier(1950)]{brier}
Glenn~W. Brier.
\newblock Verification of forecasts expressed in terms of probability.
\newblock \emph{Monthly Weather Review}, 78\penalty0 (1):\penalty0 1--3, 1950.
\bibitem[Bruggemann et~al.(2020)Bruggemann, Kanakis, Georgoulis, and Gool]{bruggemann2020automatedsearchresourceefficientbranched}
David Bruggemann, Menelaos Kanakis, Stamatios Georgoulis, and Luc~Van Gool.
\newblock Automated search for resource-efficient branched multi-task networks.
\newblock \emph{Proceedings of the 31st British Machine Vision Conference (BMVC)}, 2020.
\bibitem[Chen et~al.(2018)Chen, Badrinarayanan, Lee, and Rabinovich]{chen2018gradnorm}
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich.
\newblock Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.
\newblock \emph{Proceedings of the 35th International Conference on Machine Learning (ICML)}, 2018.
\bibitem[Chen et~al.(2020)Chen, Ngiam, Huang, Luong, Kretzschmar, Chai, and Anguelov]{chen2020just}
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov.
\newblock Just pick a sign: Optimizing deep multitask models with gradient sign dropout.
\newblock \emph{Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS)}, 2020.
\bibitem[Craswell(2009)]{mrr}
Nick Craswell.
\newblock Mean reciprocal rank.
\newblock \emph{Encyclopedia of Database Systems, Springer}, 2009.
\bibitem[Daley \& Vere-Jones(2003)Daley and Vere-Jones]{daley2003pointprocess}
Daryl Daley and David Vere-Jones.
\newblock An introduction to the theory of point processes. volume i: Elementary theory and methods.
\newblock \emph{Springer}, 2003.
\bibitem[Daley \& Vere-Jones(2008)Daley and Vere-Jones]{daley2008pointprocess}
Daryl~J. Daley and David Vere-Jones.
\newblock An introduction to the theory of point processes. volume ii: General theory and structure.
\newblock \emph{Springer}, 2008.
\bibitem[Dheur \& Ben~Taieb(2023)Dheur and Ben~Taieb]{dheur2023largescale}
Victor Dheur and Souhaib Ben~Taieb.
\newblock A large-scale study of probabilistic calibration in neural network regression.
\newblock \emph{Proceedings of the 40th International Conference on Machine Learning (ICML)}, 2023.
\bibitem[Du et~al.(2015)Du, Wang, He, Sun, and Song]{du2015recommendation}
Nan Du, Yichen Wang, Niao He, Jimeng Sun, and Le~Song.
\newblock Time-sensitive recommendation from recurrent user activities.
\newblock \emph{Proceedings of the 28th International Conference on Neural Information Processing Systems (NeurIPS)}, 2015.
\bibitem[Du et~al.(2016)Du, Dai, Trivedi, Upadhyay, Gomez-Rodriguez, and Song]{RMTPP}
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le~Song.
\newblock Recurrent marked temporal point processes: Embedding event history to vector.
\newblock \emph{Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)}, 2016.
\bibitem[Egesdal et~al.(2010)Egesdal, Fathauer, Louie, Neuman, Mohler, and Lewis]{egesdal2010gang}
Mike Egesdal, Chris Fathauer, Kym Louie, Jeremy Neuman, George Mohler, and Erik Lewis.
\newblock Statistical and stochastic modeling of gang rivalries in {Los Angeles}.
\newblock \emph{SIAM Undergraduate Research Online}, 2010.
\bibitem[Enguehard et~al.(2020)Enguehard, Busbridge, Bozson, Woodcock, and Hammerla]{EHR}
Joseph Enguehard, Dan Busbridge, Adam Bozson, Claire Woodcock, and Nils~Y. Hammerla.
\newblock Neural temporal point processes for modelling electronic health records.
\newblock \emph{Proceedings of Machine Learning Research (PMLR)}, 136:\penalty0 85--113, 2020.
\bibitem[Farajtabar et~al.(2017)Farajtabar, Wang, Rodriguez, Li, Zha, and Song]{farajtabar2015coevolve}
Mehrdad Farajtabar, Yichen Wang, Manuel~Gomez Rodriguez, Shuang Li, Hongyuan Zha, and Le~Song.
\newblock Coevolve: A joint point process model for information diffusion and network co-evolution, 2017.
\newblock Journal of Machine Learning Research, 18, 1-49.
\bibitem[Gebetsbergera et~al.(2018)Gebetsbergera, Messner, Mayr, and Zeileis]{gebetsberger2018estimationmethods}
Manuel Gebetsbergera, Jakob~W. Messner, Georg~J. Mayr, and Achim Zeileis.
\newblock Estimation methods for nonhomogeneous regression models: Minimum continuous ranked probability score versus maximum likelihood.
\newblock \emph{Monthly Weather Review}, 146\penalty0 (12):\penalty0 4323--4338, 2018.
\bibitem[Gneiting \& Raftery(2007)Gneiting and Raftery]{gneiting2007strictlyproper}
Tilmann Gneiting and Adrian~E. Raftery.
\newblock Strictly proper scoring rules, prediction, and estimation.
\newblock \emph{Journal of the American Statistical Association}, 102\penalty0 (477):\penalty0 359--378, 2007.
\bibitem[Gracious \& Dukkipati(2023)Gracious and Dukkipati]{gracious2023dynamicrepresentationlearningtemporal}
Tony Gracious and Ambedkar Dukkipati.
\newblock Dynamic representation learning with temporal point processes for higher-order interaction forecasting.
\newblock \emph{Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI)}, 2023.
\bibitem[Guo et~al.(2017)Guo, Pleiss, Sun, and Weinberger]{guo2017calibration}
Chuan Guo, Geoff Pleiss, Yu~Sun, and Kilian~Q. Weinberger.
\newblock On calibration of modern neural networks.
\newblock \emph{Proceedings of the 34th International Conference on Machine Learning (ICML)}, 2017.
\bibitem[Guo et~al.(2018{\natexlab{a}})Guo, Haque, Huang, Yeung, and Fei-Fei]{guo2018dynamictask}
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li~Fei-Fei.
\newblock Dynamic task prioritization for multitask learning.
\newblock \emph{Proceedings of the 2018 European Conference on Computer Vision (ECCV)}, 2018{\natexlab{a}}.
\bibitem[Guo et~al.(2020)Guo, Lee, and Ulbricht]{guo2020learningbranchmultitasklearning}
Pengsheng Guo, Chen-Yu Lee, and Daniel Ulbricht.
\newblock Learning to branch for multi-task learning.
\newblock \emph{Proceedings of the 37th International Conference on Machine Learning (ICML)}, 2020.
\bibitem[Guo et~al.(2018{\natexlab{b}})Guo, Li, and Liu]{guo2018noisecontrastive}
Ruocheng Guo, Jundong Li, and Huan Liu.
\newblock Initiator: Noise-contrastive estimation for marked temporal point process.
\newblock \emph{Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI)}, 2018{\natexlab{b}}.
\bibitem[Gupta et~al.(2021)Gupta, Bedathur, Bhattacharya, and De]{gupta2021intermittent}
Vinayak Gupta, Srikanta~J. Bedathur, Sourangshu Bhattacharya, and A.~De.
\newblock Learning temporal point processes with intermittent observations.
\newblock \emph{Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS)}, 2021.
\bibitem[Hawkes(1971)]{Hawkes}
Alan~G. Hawkes.
\newblock Point spectra of some mutually exciting point processes.
\newblock \emph{Journal of the Royal Statistical Society: Series B (Methodological)}, 33\penalty0 (3):\penalty0 438--443, 1971.
\bibitem[Hawkes(2018)]{hawkes2018finance}
Alan~G. Hawkes.
\newblock Hawkes processes and their applications to finance: A review.
\newblock \emph{Quantitative Finance}, 18\penalty0 (2):\penalty0 193--198, 2018.
\bibitem[Hendrycks \& Gimpel(2023)Hendrycks and Gimpel]{hendrycks2023gelu}
Dan Hendrycks and Kevin Gimpel.
\newblock Gaussian error linear units ({GELUs}).
\newblock \emph{ArXiv preprint}, 2023.
\bibitem[Hidasi \& Tikk(2012)Hidasi and Tikk]{hidasi2012}
Balazs Hidasi and Domonkos Tikk.
\newblock Fast {ALS}-based tensor factorization for context-aware recommendation from implicit feedback.
\newblock \emph{Proceedings of the 2012 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD)}, 2012.
\bibitem[Isham \& Westcott(1979)Isham and Westcott]{SelfCorrecting}
Valerie Isham and Mark Westcott.
\newblock A self-correcting point process.
\newblock \emph{Stochastic Processes and Their Applications}, 8\penalty0 (3):\penalty0 335–347, 1979.
\bibitem[Javaloy \& Valera(2022)Javaloy and Valera]{javaloy2022rotogradgradienthomogenizationmultitask}
Adrián Javaloy and Isabel Valera.
\newblock Rotograd: Gradient homogenization in multitask learning.
\newblock \emph{Proceedings of the 2022 International Conference on Learning Representations (ICLR)}, 2022.
\bibitem[Kendall et~al.(2018)Kendall, Gal, and Cipolla]{kendall2018multitasklearningusinguncertainty}
Alex Kendall, Yarin Gal, and Roberto Cipolla.
\newblock Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.
\newblock \emph{Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 2018.
\bibitem[Kingma \& Ba(2014)Kingma and Ba]{adam}
Diederik~P. Kingma and Jimmy Ba.
\newblock Adam: A method for stochastic optimization.
\newblock \emph{Proceedings of the 2014 International Conference on Learning Representations (ICLR)}, 2014.
\bibitem[Kuleshov et~al.(2018)Kuleshov, Fenner, and Ermon]{kuleshov2018uncertaintiesdeeplearning}
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon.
\newblock Accurate uncertainties for deep learning using calibrated regression.
\newblock \emph{Proceedings of the 35th International Conference on Machine Learning (ICML)}, 2018.
\bibitem[Kumar et~al.(2019)Kumar, Zhang, and Leskovec]{kumar2019predicting}
Srijan Kumar, Xikun Zhang, and Jure Leskovec.
\newblock Predicting dynamic embedding trajectory in temporal interaction networks.
\newblock \emph{Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)}, 2019.
\bibitem[Li et~al.(2023)Li, Xu, Zuo, Jiang, Zhang, Zhao, and Zha]{li2023smurfthp}
Zichong Li, Yanbo Xu, Simiao Zuo, Haoming Jiang, Chao Zhang, Tuo Zhao, and Hongyuan Zha.
\newblock {SMURF-THP}: Score matching-based uncertainty quantification for transformer {Hawkes} process.
\newblock \emph{Proceedings of the 40th International Conference on Machine Learning (ICML)}, 2023.
\bibitem[Lin et~al.(2021)Lin, Tan, Wu, Gao, and Li]{lin2021empirical}
Haitao Lin, Cheng Tan, Lirong Wu, Zhangyang Gao, and Stan.~Z. Li.
\newblock An empirical study: Extensive deep temporal point process.
\newblock \emph{ArXiv preprint}, 2021.
\bibitem[Lin et~al.(2022)Lin, Wu, Zhao, Liu, and Li]{LinGenerative}
Haitao Lin, Lirong Wu, Guojiang Zhao, Pai Liu, and Stan~Z. Li.
\newblock Exploring generative neural temporal point process.
\newblock \emph{Transactions on Machine Learning Research (TMLR)}, 2022.
\bibitem[Liu et~al.(2021{\natexlab{a}})Liu, Liu, Jin, Stone, and Liu]{liu2021conflictaversegradientdescentmultitask}
Bo~Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu.
\newblock Conflict-averse gradient descent for multi-task learning.
\newblock \emph{Proceedings of the 35th International Conference on Neural Information Processing Systems (NeurIPS)}, 2021{\natexlab{a}}.
\bibitem[Liu et~al.(2021{\natexlab{b}})Liu, Liu, Jin, Stone, and Liu]{liu2024conflictaverse}
Bo~Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu.
\newblock Conflict-averse gradient descent for multi-task learning.
\newblock \emph{Proceedings of the 35th International Conference on Neural Information Processing Systems (NeurIPS)}, 2021{\natexlab{b}}.
\bibitem[Liu et~al.(2021{\natexlab{c}})Liu, Li, Kuang, Xue, Chen, Yang, Liao, and Zhang]{liu2021impartialmultitask}
Liyang Liu, Yi~Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Q.~Liao, and Wayne Zhang.
\newblock Towards impartial multi-task learning.
\newblock \emph{Proceedings of the 2021 International Conference on Learning Representations (ICLR)}, 2021{\natexlab{c}}.
\bibitem[Liu et~al.(2019)Liu, Johns, and Davison]{liu2019endtoendmultitasklearningattention}
Shikun Liu, Edward Johns, and Andrew~J. Davison.
\newblock End-to-end multi-task learning with attention.
\newblock \emph{Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 2019.
\bibitem[Maninis et~al.(2019)Maninis, Radosavovic, and Kokkinos]{maninis2019attentivesingletaskingmultipletasks}
Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos.
\newblock Attentive single-tasking of multiple tasks.
\newblock \emph{Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 2019.
\bibitem[Mei \& Eisner(2017)Mei and Eisner]{NeuralHawkes}
Hongyuan Mei and Jason Eisner.
\newblock The neural {Hawkes} process: A neurally self-modulating multivariate point process.
\newblock \emph{Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS)}, 2017.
\bibitem[Naeini et~al.(2015)Naeini, Cooper, and Hauskrecht]{naeini2015calibration}
Mahdi~Pakdaman Naeini, Gregory~F. Cooper, and Milos Hauskrecht.
\newblock Obtaining well calibrated probabilities using bayesian binning.
\newblock \emph{Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI)}, 2015.
\bibitem[Ogata(1998)]{Ogata}
Yosihiko Ogata.
\newblock Space-time point-process models for earthquake occurrences, 1998.
\newblock Annals of the Institute of Statistical Mathematics, 50(2):379–402.
\bibitem[Omi et~al.(2019)Omi, Ueda, and Aihara]{FullyNN}
Takahiro Omi, Naonori Ueda, and Kazuyuki Aihara.
\newblock Fully neural network based model for general temporal point processes.
\newblock \emph{Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS)}, 2019.
\bibitem[Rasmussen(2018)]{rasmussen2018lecture}
Jakob~Gulddahl Rasmussen.
\newblock Lecture notes: Temporal point processes and the conditional intensity function.
\newblock \emph{ArXiv preprint}, 2018.
\bibitem[Ruder(2017)]{ruder2017overview}
Sebastian Ruder.
\newblock An overview of gradient descent optimization algorithms.
\newblock \emph{ArXiv preprint}, 2017.
\bibitem[Sener \& Koltun(2018)Sener and Koltun]{sener2018multitasklearningmultiobjectiveoptimization}
Ozan Sener and Vladlen Koltun.
\newblock Multi-task learning as multi-objective optimization.
\newblock \emph{Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS)}, 2018.
\bibitem[Sharma et~al.(2021)Sharma, Zhang, Ferrara, and Liu]{sharma2021identifyingcoordinatedaccountssocial}
Karishma Sharma, Yizhou Zhang, Emilio Ferrara, and Yan Liu.
\newblock Identifying coordinated accounts on social media through hidden influence and group behaviours.
\newblock \emph{Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)}, 2021.
\bibitem[Shchur et~al.(2020)Shchur, Biloš, and Günnemann]{IntensityFree}
Oleksandr Shchur, Marin Biloš, and Stephan Günnemann.
\newblock Intensity-free learning of temporal point processes.
\newblock \emph{Proceedings of the 2020 International Conference on Learning Representations (ICLR)}, 2020.
\bibitem[Shchur et~al.(2021)Shchur, Türkmen, Januschowski, and Günnemann]{SchurSurvey}
Oleksandr Shchur, Ali~Caner Türkmen, Tim Januschowski, and Stephan Günnemann.
\newblock Neural temporal point processes: A review.
\newblock \emph{Proceedings of 13th Joint Conference on Artificial Intelligence (IJCAI)}, 2021.
\bibitem[Shi et~al.(2023)Shi, Li, Zhang, Chen, and Wu]{shi2023recon}
Guangyuan Shi, Qimai Li, Wenlong Zhang, Jiaxin Chen, and Xiao-Ming Wu.
\newblock Recon: Reducing conflicting gradients from the root for multi-task learning.
\newblock \emph{Proceedings of the 2023 International Conference on Learning Representations (ICLR)}, 2023.
\bibitem[Sinha et~al.(2018)Sinha, Chen, Badrinarayanan, and Rabinovich]{sinha2018gradientadversarialtrainingneural}
Ayan Sinha, Zhao Chen, Vijay Badrinarayanan, and Andrew Rabinovich.
\newblock Gradient adversarial training of neural networks.
\newblock \emph{ArXiv preprint}, 2018.
\bibitem[Standley et~al.(2020)Standley, Zamir, Chen, Guibas, Malik, and Savarese]{standley2020taskslearnedmultitasklearning}
Trevor Standley, Amir~R. Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese.
\newblock Which tasks should be learned together in multi-task learning?
\newblock \emph{Proceedings of the 37th International Conference on Machine Learning (ICML)}, 2020.
\bibitem[Trivedi et~al.(2019)Trivedi, Farajtabar, Biswal, and Zha]{dyrep}
Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha.
\newblock Representation learning over dynamic graphs.
\newblock \emph{Proceedings of the 2019 International Conference on Learning Representations (ICLR)}, 2019.
\bibitem[Türkmen et~al.(2019)Türkmen, Wang, and Smola]{turkmen2019fastpoint}
Ali~Caner Türkmen, Bernie Wang, and Alex Smola.
\newblock Fastpoint: Scalable deep point processes.
\newblock \emph{Proceedings of the 2019 European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD)}, 2019.
\bibitem[Waghmare et~al.(2022)Waghmare, Debnath, Asthana, and Malhotra]{Waghmare}
Govind Waghmare, Ankur Debnath, Siddhartha Asthana, and Aakarsh Malhotra.
\newblock Modeling inter-dependence between time and mark in multivariate temporal point processes.
\newblock \emph{Proceedings of the 31st ACM International Conference on Information \& Knowledge Management (CIKM)}, 2022.
\bibitem[Wang et~al.(2020)Wang, Tsvetkov, Firat, and Cao]{wang2020gradientvaccineinvestigatingimproving}
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao.
\newblock Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models.
\newblock \emph{Proceedings of the 2020 International Conference on Learning Representations (ICLR)}, 2020.
\bibitem[Xiao et~al.(2017)Xiao, Yan, Chu, Yang, and Zha]{xiao2017gaussian}
Shuai Xiao, Junchi Yan, Stephen~M. Chu, Xiaokang Yang, and Hongyuan Zha.
\newblock Modeling the intensity function of point process via recurrent neural networks.
\newblock \emph{Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAI)}, 2017.
\bibitem[Yang et~al.(2022)Yang, Mei, and Eisner]{yang2022transformer}
Chenghao Yang, Hongyuan Mei, and Jason Eisner.
\newblock Transformer embeddings of irregularly spaced events and their participants.
\newblock \emph{Proceedings of the 2022 International Conference on Learning Representations (ICLR)}, 2022.
\bibitem[Yu et~al.(2020)Yu, Kumar, Gupta, Levine, Hausman, and Finn]{yu2020gradient}
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.
\newblock Gradient surgery for multi-task learning.
\newblock \emph{Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS)}, 2020.
\bibitem[Zhang et~al.(2020)Zhang, Lipani, Kirnap, and Yilmaz]{SelfAttentiveHawkesProcesses}
Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz.
\newblock Self-attentive hawkes processes.
\newblock \emph{Proceedings of the 37th International Conference on Machine Learning (ICML)}, 2020.
\bibitem[Zhang et~al.(2023)Zhang, Kong, and Zhou]{zhang2023integrationfreetrainingspatiotemporalmultimodal}
Yixuan Zhang, Quyu Kong, and Feng Zhou.
\newblock Integration-free training for spatio-temporal multimodal covariate deep kernel point processes.
\newblock \emph{Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS)}, 2023.
\bibitem[Zhou \& Yu(2023)Zhou and Yu]{zhou2023automaticintegrationspatiotemporalneural}
Zihao Zhou and Rose Yu.
\newblock Automatic integration for spatio-temporal neural point processes.
\newblock \emph{Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS)}, 2023.
\bibitem[Zhu et~al.(2020)Zhu, Yuchi, and Xie]{zhu2020adversarial}
Shixiang Zhu, Henry~Shaowu Yuchi, and Yao Xie.
\newblock Adversarial anomaly detection for marked spatio-temporal streaming data.
\newblock \emph{Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 2020.
\bibitem[Zhu et~al.(2021)Zhu, Zhang, Ding, and Xie]{zhu2021deepfourierkernelselfattentive}
Shixiang Zhu, Minghe Zhang, Ruyi Ding, and Yao Xie.
\newblock Deep fourier kernel for self-attentive point processes.
\newblock \emph{Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS)}, 2021.
\bibitem[Zuo et~al.(2020)Zuo, Jiang, Li, Zhao, and Zha]{TransformerHawkes}
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha.
\newblock Transformer {Hawkes} process.
\newblock \emph{Proceedings of the 37th International Conference on Machine Learning (ICML)}, 2020.
\end{thebibliography}