Model-based Reinforcement Learning for Predictions and Control for Limit Order Books
Extracted source text · arXiv 1910.03743
%File: formatting-instruction.tex
\def\year{2020}\relax
\documentclass[letterpaper]{article}
\usepackage{aaai}
\usepackage{times}
\usepackage{helvet}
\usepackage{courier}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{subfigure}
\usepackage{color}
\frenchspacing
\graphicspath{figures/}
\setlength{\pdfpagewidth}{8.5in}
\setlength{\pdfpageheight}{11in}
\pdfinfo{
/Title (Insert Your Title Here)
/Author (Put All Your Authors Here, Separated by Commas)}
\setcounter{secnumdepth}{0}
\begin{document}
% The file aaai.sty is the style file for AAAI Press
% proceedings, working notes, and technical reports.
%
\title{Model-based Reinforcement Learning for Predictions and Control for Limit Order Books}
\author{Haoran Wei,\textsuperscript{1*}
Yuanbo Wang,\textsuperscript{2*}, Lidia Mangu,\textsuperscript{2**}, Keith Decker,\textsuperscript{1**}\\
\textsuperscript{1}{University of Delaware}\\
\textsuperscript{2}{J.P. Morgan}\\
nancywhr@udel.edu,
yuanbo.wang@jpmchase.com,
lidia.l.mangu@jpmchase.com
decker@udel.edu}
% \author{Submission \# 6080}
\maketitle
\begin{abstract}
\begin{quote}
We build a profitable electronic trading agent with Reinforcement Learning that places buy and sell orders in the stock market. An environment model is built only with historical observational data, and the RL agent learns the trading policy by interacting with the environment model instead of with the real-market to minimize the risk and potential monetary loss. Trained in unsupervised and self-supervised fashion, our environment model learned a temporal and causal representation of the market in latent space through deep neural networks. We demonstrate that the trading policy trained entirely within the environment model can be transferred back into the real market and maintain its profitability. We believe that this environment model can serve as a robust simulator that predicts market movement as well as trade impact for further studies.
% challenges: no high-fidelity environment model available for RL exploration and no explicit signals to represent the impact of trading activities, especially to the liquid market. To address these challenges, we develop an environment model to address the trading impact as well as predict the future market with the current market and current trading actions. With the environment model, RL can explore completely without intervening the real world and avoid a big loss. We also build an RL agent trained based on the environment model to show this trading framework's feasibility in terms of maximizing the cumulative Profit and Loss (PnL). We test the RL agent with real historical data and compare the performance with two common used trading strategies: momentum-based strategy and classification-based strategy. Our RL agent outperforms other approaches based on the cumulative PnL. Additionally, we also provide an analysis of different agents' trading behaviors. Overall, we have two main contributions in this study: 1) provide a platform for RL to conduct exploration without real-environment interactions in finance; 2) show RL's potential in financial trading systems.
\end{quote}
\end{abstract}
\section{Introduction}
Model-free Reinforcement Learning (MFRL) using deep-learning architecture has achieved robust performance on a variety of complicated tasks, ranging from the classic Atari 2600 video games \cite{mnih2015human}, to locomotion tasks \cite{lillicrap2015continuous}, and have even delivered super-human performance in challenging exploration domains \cite{salimans2018learning}. Learning by trial and error, one of the limitations of such approaches is that they usually require a large number of interactions with the environment for training purposes. This challenge becomes even more significant when these interactions are expensive or even dangerous (e.g., financial trading or self-driving cars).
Building a simulator of the environment could alleviate such problems. By learning a dynamic model that predicts the next state given current state and action, model-based RL (MBRL) enables agents to explore inside the simulator safely. In recent research, methods have proved to be sample efficient in various tasks without compromising policy effectiveness compared to model-free approaches \cite{kaiser2019model,zhang2018solar}. Additionally, an environment model that enables predictions of the future is not only appealing to RL but also has general applications across various domains.
Most of the MBRL work still relies on using some interactive feedback from reality to update the simulator online. This is crucial to avoid mismatching problems: the approximation error of the environment model may lead the RL policy in the wrong direction. With a fixed data set to train policy with, this deviation from optimal strategy becomes challenging to self-correct. Recent studies have started challenging this requirement for real-time interactions with the environment. David Ha et al\cite{ha2018world} experimented with training policies for the VizDoom game completely inside the environment model and it outperformed benchmarks after being transferred back into the original game. Similarly, robust performance was observed in applying model-based RL methods to learn car-driving policies using observational data \cite{henaff2019model}. To advance MBRL systems without real environment interactive data in the more complex real world, we propose a model-based RL framework and test its robustness in electronic trading domains.
Interestingly enough, we observe similarities between the implicit structures of MBRL and the explicit design of an electronic trading system. In the most general and simplistic case, an electronic trading system consists of a market simulator and decision strategies. The simulator predicts market dynamics using quantitative and statistical models, and it has inspired extensive research\cite{ang2006stock,bacchetta2009predictability}. The decision strategies are usually influenced by domain knowledge, heuristics, observations, and sometimes, preferences of the algorithm users. There are two main types of approach to building market simulators: traditional statistical methods and machine learning methods. Traditional statistical approaches try to model linear processes that underlie the generation of time-series. For instance, the Glosten model \cite{rocsu2009dynamic} assumes that the price impact of trades is linear, immediate, and not state-dependent. The Vector Auto-Regressive model (VAR) \cite{zivot2006vector} addresses the non-stationary aspect of financial time-series and can forecast with multiple variances. Similarly, in \cite{beltran2005understanding}, researchers investigate the near-linear dynamics between sequences of orders and the evolution of the market. On the other hand, machine learning approaches, which do not assume linearity and require little prior knowledge about the input data, have also shown promising performance in market forecasting \cite{zhang2019deeplob,ntakaris2018benchmark,tsantekidis2017forecasting,tsantekidis2017using,dixon2017classification}.
Two drawbacks in this explicit simulator-policy framework are a compilation of approximation errors from the simulator and explosion in the number of corner cases in hand-crafted decision policies. There have been efforts to bypass the market-simulating stage and proceed to train an RL agent that takes market conditions as input and directly outputs decisions. \cite{bacoyannis2018idiosyncrasies} applies a model-free technique to solve trade execution problems. \cite{nevmyvaka2006reinforcement} trains an RL policy to replace hand-crafted decision rules while still employing a traditional market simulator. However, the same challenge persists that RL agents are still trained using static, pre-generated time-series data.
Given the structural similarities between MBRL and electronic trading systems, we think a model-based RL agent trained on observational data could potentially solve the problems in electronic trading that we have listed. To our knowledge, no prior work has been done on this topic, and therefore, we decide to bridge the gap. We have two main contributions: (1) In our MBRL framework, we use latent representation learning to model not only the state space but also rewards. We demonstrate the effectiveness of such representation learning in the financial domain, where data is high-dimensional and non-stationary. (2) Using this model of the environment, we show that our model-based agent consistently outperforms commonly used benchmark trading strategies. This approach enables the learning of profitable trading policies using observational data with \textit{no} environment interaction or labeling by human experts. This project code will be released soon for replication.
% \begin{enumerate}
% \item motivations, analysis on challenges in finance trading policy with RL
% \item summary of current work and shortcoming
% \item our proposal: model-based RL
% \item our experiment result and analysis
% \item contributions in industry
% \end{enumerate}
% Oh et al. (2015) and Chiappa et al. (2017) show that learning
% predictive models of Atari 2600 environments is possible
% using appropriately chosen deep learning architectures. Impressively, in some cases the predictions maintain low L2 error over timespans of hundreds of steps. As learned simulators of Atari environments are core ingredients of our approach, in many aspects our work is motivated by Oh et al. (2015) and Chiappa et al. (2017), however we focus on using video prediction in the context of learning how to play the game well and positively verify that learned simulators can be used to train a policy useful in original environments.
% An important step in this direction was made by Leibfried et al. (2016), which extends the work of Oh et al. (2015) by including reward prediction, but does not use the model to learn policies that play the games. Perhaps surprisingly, there
% is virtually no work on model-based RL in video games
% from images. Notable exceptions are the works of Oh et al.
% (2017), Ha & Schmidhuber (2018), Holland et al. (2018)
% and Azizzadenesheli et al. (2018). Oh et al. (2017) use a
% model of rewards to augment model-free learning with good
% results on a number of Atari games. However, this method
% does not actually aim to model or predict future frames, and
% achieves clear but relatively modest gains in efficiency. Ha
% & Schmidhuber (2018) present a way to compose a variational autoencoder with a recurrent neural network into an
% architecture that is successfully evaluated in the VizDoom
% environment and on a 2D racing game. The training procedure is similar to Algorithm 1, but only one iteration of
% the loop is needed as the environments are simple enough
% to be fully explored with random exploration. Holland et al.
% (2018) use a variant of Dyna (Sutton, 1991) to learn a model
% of the environment and generate experience for policy training in the context of Atari games.
\section{Related Work}
Two studies inspired our study. In World Models \cite{ha2018world}, the authors employ generative models to get latent representations of the environment, which enables the RL agent to learn a compact yet effective policy. The other recent study builds a neural network simulated environment $\text{env}_{0}$ that not only shares an action space and reward space with the original environment $\text{env}$ but also produces observations in the same format \cite{kaiser2019model}.
In reality, there are many scenarios where real environment interactions are costly or not feasible, such as autonomous driving \cite{wu2017flow}, recommendation systems\cite{zhao2019model}, and trading systems. These infeasible interaction environments yield a challenge for RL. However, if the next state can be predicted, real environment interactions may not be necessary anymore. Lukasz Kaiser et al. \cite{kaiser2019model} show a complete model-based RL approach to play Atari games where a CNN-based prediction model is used to predict the next game frame given the previous frames and action, and CNN layers are used to extract hidden features autonomously. A variational layer is used as the last layer to learn the posterior of the next frame context so that environment stochasticity is considered, and learning is shown to be improved\cite{ha2018world,oh2015action,leibfried2016deep}. Recently, model-based RL is also used in recommendation systems \cite{zhao2019model} for conducting random exploration without bothering users overwhelmingly. However, few works have been seen in real-world applications compared to the wide application in the gaming domain. This is reasonable because the real world has more complicated, uncertain factors to model.
In the Finance domain, RL has been applied to many different problems \cite{fischer2018reinforcement}, especially designing electronic trading strategies \cite{bacoyannis2018idiosyncrasies,bertoluzzo2012testing}. However, most of the work has been done with model-free RL, such as Deep Q-networks (DQN)\cite{huang2018financial}, that have lower sampling complexity. Alternatively, model-based methods require many fewer training samples; however, there is no existing finance RL model for random exploration. Our work tries to show that 1) A trading model can be built with historical observation data; 2) model-based RL has potential in time-series decision making.
% 1) A trading model can be built with historical observation data and model-based RL can be trained without real-environment interactions; 2) RL has potential on time-series decision making.
% we also find that the policy-based RL works better than the value-based method (e.g. DQN)
% Our video models of Atari environments described in Section 4 are motivated by models developed in the context of
% robotics. Another source of inspiration are discrete autoencoder
% coders proposed by van den Oord et al. (2017) and Kaiser
% & Bengio (2018).
% The structure of the model-based RL algorithm that we employ consists of alternating between learning a model, and
% then using this model to optimize a policy with model-free
% reinforcement learning. Variants of this basic algorithm
% have been proposed in a number of prior works, starting
% from Dyna (Sutton, 1991) to more recent methods that incorporate deep networks (Heess et al., 2015; Feinberg et al.,
% 2018; Kalweit & Boedecker, 2017; Kurutach et al., 2018).
% Related work in Finance:
% \begin{enumerate}
% \item survey \cite{fischer2018reinforcement}
% \item electronic trading \cite{bacoyannis2018idiosyncrasies}
% \item different RL for Financial Trading \cite{bertoluzzo2012testing}
% \item Similar to our work, related work proposed model-free RL method (e.g., DQN) to learning the autonomous trading policy with MDP environment model. However, the state features are handcrafted and no trading impact addressed. \cite{huang2018financial}
% \end{enumerate}
% Our work is similar to their work, one advantage for us that there is large amount of historical trading data available for training a representative environment model. However, we are also facing additional and unique challenges such as long time dependency, uncertain action impacts and etc. Historical trading records are not evenly blended in limit order books data increases the uncertainty of trading impact and how to form efficient state action state transitions is another challenge. Although the model-based RL has brought outstanding results on game playing, few work have be seen in real world applications and this is reasonable because real world has more complicated uncertain factors to model.
% Model-based DynaQ \cite{holland2018effect}.
% Real environment exploration is not feasible: traffic/recommendation/finance.
% Recommendation system \cite{zhao2019model}: users leave system soon with overwhelming random recommendation (random exploration is costly in reality) and imbalanced reward function.
\section{Background}
In this section, we introduce some important elements in electronic trading systems and how they translate to concepts familiar to RL research. We also give a brief overview of the RL methods used in this study.
\subsection{Trading Problem}
\textbf{Limit Order Books} (LOBs) are used by more than half of the exchange markets in the world \cite{rosu2010liquidity}. An LOB has two types of orders: bid and ask. At any given time $t$, a bid (ask) order is to buy (sell) certain quantity (aka size), $\text{bs}(t)$ ($\text{as}(t)$), of a financial instrument at or below (above) the specifies price $\text{bp}(t)$ ($\text{ap}(t)$), as shown in Fig \ref{lob_demo}. Orders submitted at time $t$ are sorted into different levels based on their prices. For instance, the lowest ask price and the highest bid price are grouped into the first level order, followed by the second lowest \text{ap} and the second-highest \text{bp} as the second level, and so on. $\left\{ \text{ap}, \text{bp}, \text{as}, \text{bs}\right\}_{i}$ are vectors of values and quantities at different price levels $i$. The time-series evolution of an LOB can be seen as a 3-dimensional tensor: the first dimension represents time, the second dimension is level, and the third represents prices and order quantities on both the buy and sell sides\cite{gould2013limit}. When submitted orders are executed by an LOB's trade-matching algorithm, the orders' price and quantity with direction (bid or ask) are removed from the LOB and recorded in a historical \textit{\textbf{trade print}}.
\begin{figure}
\centering
\includegraphics[width=0.3\textwidth]{figures/lob.png}
\caption{A slice of limit order books (LOB) with three levels on both ask and bid sides}
\label{lob_demo}
\end{figure}
\textbf{Mid Price} is the mean value of the first-level ask and bid price, as Eq \ref{midprice}
\begin{equation}
\text{mid}_{t} = \frac{\text{ap}_{1} + \text{bp}_{1}}{2}
\label{midprice}
\end{equation}
Take the slice of LOB in Fig \ref{lob_demo} as an example, the mid price at this time $t$ is (408.0+408.3)/2 = 408.15. The movement of the mid price is commonly used to approximate market change. In this study, we use the mid price to calculate reward.
\textbf{Trade prints} are the record of executed trades that contain information of direction (buy or sell), trading price, and quantity. The collection of trade prints may be executed by different agents in the market. In this study, we use historical trade prints as our RL agent's exploration actions. We also include a sequence of trade prints prior to the target action as part of the state. This data provides crucial information for the state transition probabilities model (details are in the next section). We observe a similar problem setup in \cite{henaff2019model}, where the authors use a target car's driving trajectory for RL exploration and surrounding cars' trajectories as part of states.
\subsection{RL for agent}
We test three commonly used RL algorithms for policy training: Double Deep-Q network (Double DQN) \cite{van2016deep}, Policy Gradient \cite{williams1992simple}, and Advantage Actor-Critic (A2C) \cite{mnih2016asynchronous}.
% A2C shows consistent better performance in terms of cumulative reward and stability.
\textbf{Double DQN}
A deep Q-network is a neural network that approximates the Q-value for an input state-action pair. It's optimized by minimizing the square error between the predicted and target Q-value. Double DQN uses two networks: an online network for selecting the actions according to the value of ($Q, \theta$), and another target network to determine its value ($Q, \theta_{-}$). Optimization of the online network ($Q, \theta$) is done through minimizing the prediction error $L(\theta)$ of the target Q-network:
\begin{equation}
\begin{aligned}
& L(\theta) = \mathbb{E}[(Q(s, a;\theta) - Q_{\text{target}})^{2}]\\
& Q_{\text{target}} = r+\gamma Q(s', \arg\max_{a'}Q(s', a'; \theta); \theta_{-})
\end{aligned}
\end{equation}
$Q_{\text{target}}$ is the output of target network ($Q, \theta_{-}$), the weights ($\theta_{-}$) of which remain unchanged except for a periodic copy of weights from the online Q-network ($Q, \theta$). Having a separate target Q-network helps reduce policy variance caused by oscillations of the target value.
% and using the ``argmax" operation to pick the action instead of estimating the Q-value can also avoid the value overestimation issue as well.
\textbf{Policy Gradient} In the policy gradient (PG) algorithm, a policy is directly modeled with a $\theta$-parameterized function $\pi(a|s,\theta)$. Given $\pi(a|s,\theta)$ and environment model $p(s'|s, a)$, we can generate a trajectory $\tau=(s_1, a_1, s_2, a_2, ... s_t, a_t)$ and accumulated reward $r$, see Eq\ref{PG}. The idea is to maximize the accumulated reward $J(\theta)$ by repeatedly optimize $\theta$ with gradient ascent $\nabla J(\theta)$.
\begin{equation}
\begin{aligned}
&J(\theta) = \mathbb{E}_{\tau}[\sum_{t=0}^{H}\log\pi(a|s,\theta)G_{\tau}]\\
& \text{where} \qquad G_{\tau} = \sum_{t=0}^{H}\gamma^{t}r_{t}
\end{aligned}
\label{PG}
\end{equation}
% where $\pi$ is the policy and $\gamma$ is the constant discount factor.
Compared to other value-based RL methods, PG learns a policy directly. It is also less sensitive to value overestimation problems common in Q-learning (caused by the ``max" operation). However, one drawback is that reward accumulation along a trajectory may cause high policy variance.
\textbf{Advantage Actor-Critic (A2C)} A2C is a hybrid RL method combining policy gradient and value-based methods. It consists of two networks: actor and critic. The actor network updates policy $\pi(a|s,\theta)$ by maximizing the objective function:
\begin{equation}
J(\theta) = \mathbb{E}_{\tau}[\sum_{t=0}^{H}\log\pi(a|s,\theta)A(s, a)]
\end{equation}
where $A(s, a)$ is the advantage value representing how much better (worse) a given action performs in state $s$ compared to the average performance over all actions. It updates as:
\begin{equation}
A(s_{t}, a_{t}) = r_{t} + \gamma V(s_{t+1};w) - V(s_{t};w)
\end{equation}
$V(s;w)$ is the state utility representing the average performance over all actions in a given state, computed by the critic network. The critic network's parameter $w$ is updated by gradient descent on the TD-error (parameterized with $w$):
\begin{equation}
J(w) = (r_{t} + \gamma V(s_{t+1};w) - V(s_{t};w_{-}))^{2}
\end{equation}
$w_{-}$ is the parameter from the critic's previous update. The advantage of A2C is twofold: 1) policy variance is reduced due to the advantage value; 2) the policy is directly updated instead of via a value estimation function.
\section{Problem Formulation and Dataset}
\label{Problem Formulation and Data Preparation}
% data summary
Limit Order Book data is time-series with high sampling frequency. We model the environment as a Markov Decision Process (MDP). We use time-series data (LOB + trade prints) from one stock traded on the Hong Kong Stock Exchange. The update frequency of the LOB is $\sim$ 0.17s. We use January 2018 $\sim$ March 2018 (61 days) data for training with 20\% as the validation set, and test model performance on April 2018 (19 days). In all, the dataset contains approximately 6 million transitions for training and 2 million for testing.
\subsection{MDP Forming}
We model the trading problem as a Markov Decision Process (MDP) represented as $\{\mathcal{S}, \mathcal{A}, \mathcal{R}, \mathcal{T}, \rho_{0}\}$. We'll explain how to build the MDP with trading elements in this section.
\textbf{State Space ($\mathcal{S}$)}: $s_{t} = \left\{\text{ae}(\text{ob}_{t-T:t}), u_{t},\text{po}_{t}\right\}$. $z_{t}=\text{ae}(\text{ob}_{t-T:t})$ is a latent representation of the LOB within a time duration $T$. We will discuss the latent representation model more in the following section. $u_{t}$ is a vector of trade prints occurring within the same duration $T$. $\text{po}_{t}$ is the RL agent's position at time $\text{t}$. Position reflects the private inventory held by the agent, which, in our case, is bounded by $(-\text{po}_{\text{max}}, \text{po}_{\text{max}})$.
\textbf{Action Space ($\mathcal{A}$)}: $a_{t} = \pm q$. Each action is the RL agent's decision to trade. The decision includes price, quantity and direction of the trade. In our study, we assume that the trading price is set at mid-price, and can be directly calculated from the LOB update. Therefore the RL agent's action contains $q$ the absolute value of quantity and $\pm$ trading direction (sell/buy).
\textbf{Reward Function ($\mathcal{R}$)}: We use a mark-to-market PnL to calculate agent's reward. It's defined as:
\begin{equation}
\mathcal{R}(t) = \Delta\text{mid}_{s_{t}, s_{t+1}} \times \text{po}_{t}
\label{reward_func}
\end{equation}
where $\text{po}_{t}$ is the RL agent's position at time $t$ and $\Delta\text{mid}_{s_{t}, s_{t+1}}$ is the difference in average LOB mid-price between state $s_{t}$ and $s_{t+1}$.
\textbf{Transitions ($\mathcal{T}(s_{t+1}|s_{t}, a_{t})$)}. We use observed trajectories of state transitions to train the environment model, and one transition is demonstrated in Fig \ref{transition_built}. Specifically, we iterate through trade prints in historical data and treat each trade as a target action in the transition. The same action may become a part of the state $u_{t}$ in the next transition when the next trade becomes the target action.
\textbf{Initial State} ($\rho_{0}$): The initial states are sampled from the first state over all days in the training dataset following a uniform distribution.
\begin{figure}
\centering
\includegraphics[width=0.35\textwidth]{figures/demo_transition_v2.png}
\caption{A demonstration of forming a state-action-state transition with limit order books and trade prints (the subscriptions are the time ticks for the lob and are indices only for the trade prints)}
\label{transition_built}
\end{figure}
\subsection{Data Preprocessing}
% In the state representation, we normalize the raw LOB data to be in the range $[0, 1]$.
For each LOB time-series sequence $\text{ob}_{t-T:t}$ with a length of $T$, we use the feature-level min and max to normalize the data across time. For the trade quantity normalization, we first exclude the outlier trades that either has less than 100 or exceed 1000 of quantity. Then we use 100 and 1000 as the boundary for min-max normalization, and the boundary is empirically determined based on the data distribution. We also implement the min-max normalization followed by a sigmoid transformation on the rewards. Here it's worth noticing that the reward transformation is only done inside the world model during exploration for training purposes.
% Crucially, apart from an Atari 2600 emulator environment env we will use a neural network simulated environment env0 which we call a world model and describe in detail in Section 4. The environment env0 shares the action space and reward space with env and produces visual observations in the same format, as it will be trained to mimic env. Our principal aim is to train a policy π using a simulated environment env0 so that π achieves good performance in the original environment env. In this training process we aim to use as few interactions with env as possible. The initial data to train env0 comes from random rollouts of env. As this is unlikely to capture all aspects of env, we use the iterative data-aggregation method presented in Algorithm 1.
\section{Model}
% Given a set of observational data $\{\text{s, a} \}_{t}$, our modeling work are done in two steps. Firstly, we train a world model that consists of latent representation learning of the LOB, state-action transition, and reward models. Using this environment model, we then train 3 different policy networks.
This work has two parts: a world model that consists of latent representation learning of the LOB, state-action transition, and reward models. A trading agent trained based on the world model with three widely used RL methods.
\begin{figure*}[t]
\centering
\subfigure[]{\label{ave}\includegraphics[width=0.3\textwidth]{figures/ave.png}}
\subfigure[]{\label{rnn_mdn}\includegraphics[width=0.3\textwidth]{figures/RNN.png}}
\subfigure[]{\label{workflow}\includegraphics[width=0.3\textwidth]{figures/workflow.png}}
\caption{(a) Extract latent LOB representation with an auto-encoder (AE) (b)The transition model: a sequence-to-sequence RNN-MDN (c)The RL agent's workflow: exploring with the environment model and exploiting in the real environment.(The dashed lines represent the exploitation and solid lines represent the exploration)}
\end{figure*}
\subsection{World Model}
The world model consists of 3 parts: a latent representation model, transition model, and reward model.
\subsubsection{Latent Representation Model (Auto-encoder)}
The LOB data contains important time-series market information but it's difficult to learn due to its high dimensionality. The role of the Auto-Encoder (AE) (as shown in Fig \ref{ave}) is to find an abstract and low-dimensional representation of the LOB observations for easy learning. The input and output of AE are both of dimension $T \times 4L$ where $L$ is the number of levels in LOB. We use 3 levels in our experiments. After AE is trained, we take the middle layer of dimension $1 \times m$ as the latent representation of high-dimensional LOB data. Here we have $m=16$.
\subsubsection{Transition Model (RNN-MDN)}
We use a RNN-MDN model to learn the state-action transitions $\mathcal{P}(s'|s, a)$. This model also addresses trades' impact on the market. We use a long sequence of \text{s, a} to train RNN and learn both short term and long term impacts. We further combine the RNN with a Mixture Density Network (MDN) \cite{bishop1994mixture}. This approach approximates the output of RNN as a mixture of distributions rather than a deterministic prediction of $s'$. The MAD is written as:
\begin{equation}
p(s'|s, a) = \sum^{K}_{k=1}w_{k}(s, a)\mathcal{D}(s'|\mu_{k}(s, a), \sigma^{2}_{k}(s, a))
\end{equation}
where $\mathcal{D}(\cdot)$ is a presumed distribution, such as Gaussian, Bernoulli, etc. The length of input sequence to the RNN is $N$, and the $(N+1)^{\text{th}}$ state is the prediction based on the previous $N$ sequences, shown as Fig \ref{rnn_mdn}. This approach has been widely applied in the past on sequence-generating tasks \cite{ha2017neural}.
\subsubsection{Reward Model}
Since the PnL depends on the next state that is unseen for the RL agent, a regression model is used to predict the change of mid-price based on the current latent state and the predicted next latent state:
\begin{equation}
r_{t} = \mathcal{R}(z_{t}, z_{t+1};\beta) \times \text{po}_{t}
\end{equation}
where $\beta$ is the reward model's parameter.
In the reward model, position $\text{po}$ depends on the executed actions and a hand-crafted capacity that casts the limit. The next position given current position and action is calculated as:
\begin{equation}
\text{po}_{t+1} = \left\{\begin{matrix}
\min(\text{po}_{t}+\left | a_{t} \right |, \text{po}_{\text{max}})& \text{when} & a_{t}>0\\
\max(\text{po}_{t}-\left | a_{t} \right |, -\text{po}_{\text{max}})& \text{when} & a_{t}<0
\end{matrix}\right.
\end{equation}
where $(-\text{po}_{\text{max}}, \text{po}_{\text{max}})$ defines the position capacity.
The latent representation, transition model, and reward model together are the world model. Then, we train the RL trading agent purely based on this world while maximizing the total reward along a certain time horizon.
\subsection{Agent Model}
The agent learns policy by exploring completely within the pre-trained environment model. Starting with a randomly selected initial state, the RL agent outputs an action, and then the state and action are fed back to the environment model to predict the next state and so on until a stop criteria. The workflow is illustrated in Fig \ref{workflow}. Unlike Atari games that usually have a clear terminal state, termination of trading actions are less well defined. Here we fix time horizon and train RL policy to maximize total reward. Once a policy has converged, we evaluate it with historical data repeatedly following similar steps: 1) from the current state with the latent LOB ($z_{t}$) and the corresponding trade prints ($u_{t}$); 2) take an action by the RL agent; 3) collect a reward; 4) if not time up, return step 1.
% We implement the policy gradient method to our RL agent and it outputs a deterministic action, $\pi(a_{t}|s_{t}; \theta)$ where $\theta$ is the parameter of RL network (weights and bias). Each trajectory has a fixed length and is generated iteratively from the world model with a randomly sampled initial state. The RL agent optimizes its policy by maximizing the average total reward over a batch of trajectories:
% \begin{equation}
% J(\theta) = \mathbb{E}_{\tau \sim \mathcal{B}}[r(\tau)\sum^{H}_{t=0}\log\pi(a_{t}|s_{t})]
% \label{RL_reward_func}
% \end{equation}
% where $\mathcal{B}$ is the trajectory batch, $H$ is a fixed time horizon, and $\tau$ is one trajectory $\tau = (s_{0}, a_{0}, s_{1}, \cdots, s_{H})$. $r(\tau)$ is the discounted cumulative reward along each trajectory shown as Eq \ref{discounted_reward}. With long time horizon $H$, the RL agent can alternatively use other RL method, such as Actor-Critic \cite{konda2000actor} to optimize the policy with smaller variance.
% \begin{equation}
% r(\tau) = \sum_{t=0}^{H}\gamma^{t}r_{t}
% \label{discounted_reward}
% \end{equation}
% Overall, the RL agent learns based on the world model, and the workflow can be summarized as Fig \ref{workflow}. This work also shines the light to integrate different parts of work across finance industry: many works from quantitative researches only focus on small problems, and the environment model provides a platform for integrating those small networks.
% \subsection{Actor-Critic}
% \begin{equation}
% \eta^{\pi}(\theta) = \mathbb{E}[\log\pi(s, a; \theta)A(s, a; w)]
% \end{equation}
% \begin{equation}
% A(s, a; w) = r + \gamma V(s'; w) - V(s' w)
% \end{equation}
\section{Experiment}
We discuss details of training, testing and performance analysis in the experiment.
\subsection{Benchmark}
We use a momentum-driven trading strategy and a classifier-based strategy as our benchmarks. The former one is a well-performing industrial strategy, and the latter one is the state-of-the-art with deep learning networks. We also use a greedy optimal strategy to measure how close the RL policy is to the optimal policy.
\textbf{Momentum-driven Strategy} The momentum is calculated by subtracting the opening mid-price from the closing mid-price. For example, if there are 40 time ticks in one state, the closing mid-price is the mid-price at the $40^{\text{th}}$ time tick, and the opening mid-price is the one at the $1^{\text{st}}$ time tick. It roughly reflects price changes within one state and the change is assumed to be carried over to the next state. The movement can fall into three classes: up, down, and no-change w.r.t a pre-defined threshold. The trading policy is hand-crafted with a fixed quantity for each action in the action space. If the agent always takes the maximum quantity in each action either buy or sell, it is ``aggressive". The aggressive agent may have the largest reward but also has the biggest risk of losing money. The agent could also take the minimum quantity in each action. Such agent is ``conservative", has smaller risks, but it may miss significant gains.
% Momentum means the movement of the current market will be kept in the next state. It delivers excellent performance, especially in a smoothly changing market. Momentum is calculated by The movement can fall into three classes: up, down, and no-change w.r.t a pre-defined threshold. The trading policy is hand-crafted with a fixed quantity in each action. If the agent always takes the maximum quantity in each action either buy or sell, we call it ``progressive". The progressive agent may have the largest reward but also has the biggest risk of losing money. On the other hand, if the agent takes the minimum quantity in each action, we call it ``conservative" and it has smaller risks but may miss significant gains.
\textbf{Classifier-based Strategy}
A classifier is trained with historical LOB data, and it predicts the mid-price movement in the next state based on the current state. Same as the momentum-based strategy, a handcrafted policy is used according to the classification, and it can be anywhere between ``progressive" and ``conservative." In this study, we build this classifier with three CNN layers and one Dense layer (64 neurons) with three output states (same configuration as the work of \cite{zhang2019deeplob}). We use the categorical cross-entropy as the loss function. The performance of the classifier is demonstrated with randomly sampled testing data in Table\ref{classifier_performance}.
\begin{table}
\centering
\caption{Classifier Performance}
\begin{tabular}{|l|l|l|l|l|}
\hline
class & precision & recall & F1 & support \\ \hline
down & 0.66 & 0.62 & 0.64 & 1799 \\ \hline
no change & 0.90 & 0.91 & 0.90 & 6401 \\ \hline
up & 0.63 & 0.63 & 0.63 & 1800 \\ \hline
\end{tabular}
\label{classifier_performance}
\end{table}
\textbf{Greedy Optimal}
This strategy assumes that the agent knows the future LOB in the next state. The requirement for future knowledge makes this strategy unrealistic in the real world. However, it can provide a reference for RL to measure how close the RL policy is to the optimal one. The global optimal strategy can be found with breadth-first search, demonstrated as Fig \ref{real_opt} where actions are discretized into 21 discrete actions. This strategy is computationally intractable for a trading strategy: the complexity can be above $21^{1000}$ for 1000 time ticks, and for a liquid market, a 1000-time-tick length is only $\sim$3 minutes. To overcome this obstacle, we use a greedy optimal strategy: only expand the best action with the maximum cumulative reward at each level (each time step) without visiting backward along time, shown as Fig \ref{greedy_opt}. The greedy optimal policy doesn't guarantee the global optima; however, it reduced the computational complexity from exponential to polynomial.
\begin{figure}[h!]
\centering
\subfigure[]{\label{real_opt}\includegraphics[width=0.45\linewidth]{figures/real_optimal.png}}
\subfigure[]{\label{greedy_opt}\includegraphics[width=0.45\linewidth]{figures/greedy_optimal_2.png}}
\caption{(a)Breadth first search for the optimal trading policy (b) greedy optimal strategy. *Nodes are states, branches are actions and each level is one time step}
\end{figure}
\subsection{Model Architecture}
\begin{figure}[t]
\centering
\includegraphics[width=0.3\textwidth]{figures/AE_architecture.pdf}
\caption{CNN-based auto-encoder architecture.Yellow layers are Upsampling/Downsampling layers, blue layers are convolutional layers}
\label{ae_archi}
\end{figure}
% In this section, we'll provide the details of features and network architectures setting for future replication. We use the HK Tencent limit order books data during Jan, 2018 $\sim$ Mar, 2018 as our training data (20\% as validation dataset), covers 61 unique days, and $\sim 0.17s$ per record. For each state, $T=40$ meaning each state covers 40 time ticks and $L=3$ meaning we use the first 3 levels data with price and size on both bid and ask sides. The testing data has the same setting but from the Apr, 2018 which covers 19 unique days.
We use a CNN-based auto-encoder to represent the original LOB data with latent states, and the network architecture is listed in Fig \ref{ae_archi}. The length of the trading records $u$ for each state is limited to 10 with post zero padding. In the transition model (RNN-MDN), one layer of the RNN has 128 neurons. Its input and output sequence length is $10$. We assume the state distribution is Gaussian and the number of Gaussian distributions is 5. The reward model is composed by one layer of 128 LSTM units and one layer of Dense with 40 units. The actions for greedy optimal policy and RL agent are discretized into 21 classes w.r.t the trade quantity representing $[-1000, -900, \cdots, 0, \cdots, 900, 1000]$. The training time horizon is set as 500 and each state includes 40 time ticks, and thus each training epoch covers $\sim1$h market time. The training horizon can be easily scaled up according to different demands.
\begin{figure*}[t]
\centering
% \subfigure[]{\label{mid-1}\includegraphics[width=0.18\textwidth]{figures/results/4_11_mid.png}}
\subfigure[]{\label{mid-2}\includegraphics[width=0.245\textwidth]{figures/result_2/mid_4_17.pdf}}
\subfigure[]{\label{mid-3}\includegraphics[width=0.245\textwidth]{figures/result_2/mid_4_18.pdf}}
\subfigure[]{\label{mid-4}\includegraphics[width=0.245\textwidth]{figures/result_2/mid_4_20.pdf}}
\subfigure[]{\label{mid-5}\includegraphics[width=0.245\textwidth]{figures/result_2/mid_4_23.pdf}}
\\
\subfigure[]{\label{compare-1}\includegraphics[width=0.245\textwidth]{figures/result_2/momentum_4_17.pdf}}
\subfigure[]{\label{compare-2}\includegraphics[width=0.245\textwidth]{figures/result_2/momentum_4_18.pdf}}
\subfigure[]{\label{compare-3}\includegraphics[width=0.245\textwidth]{figures/result_2/momentum_4_20.pdf}}
\subfigure[]{\label{compare-4}\includegraphics[width=0.245\textwidth]{figures/result_2/momentum_4_23.pdf}}
\caption{Trading agents performance with momentum-based agent, classification-based agent, RL-based agent and greedy optimal based agent tested on 4 random days}
\label{results_day_samples}
\end{figure*}
\begin{figure*}
\centering\subfigure[RL-based (PG)]{\label{explanation-RL}\includegraphics[width=0.24\textwidth]{figures/results/rl.png}}
\subfigure[Momentum-based]{\label{explanation-momentum}\includegraphics[width=0.24\textwidth]{figures/results/momentum.png}}
\subfigure[Classifier-based]{\label{explanation-clf}\includegraphics[width=0.24\textwidth]{figures/results/clf.png}}
\subfigure[Greedy optimal]{\label{explanation-opt}\includegraphics[width=0.24\textwidth]{figures/results/opt.png}}
\caption{Performance explanation in terms of actions selection: red shadows represent ``sell", green shadows represent ``buy", no actions taken in blank area. The blue line is the corresponding mid price.}
\label{action selection}
\end{figure*}
\subsection{Experiment Results}
% We test the RL policy which is trained fully based on the model with real historical data, and compare its performance with two industrial benchmarks and the greedy optimal performance.
\begin{figure}[h!]
\centering
\subfigure[]{
\includegraphics[width=0.45\linewidth]{figures/result_3/variance_down.pdf}
\label{var-descending}}
\subfigure[]{
\includegraphics[width=0.45\linewidth]{figures/result_3/variance_up.pdf}
\label{var-ascending}}
\subfigure[]{
\includegraphics[width=0.45\linewidth]{figures/result_3/variance_osci.pdf}
\label{var-osci}}
\subfigure[]{
\includegraphics[width=0.45\linewidth]{figures/result_3/variance_total.pdf}
\label{var-total}}
% \rule{12.8cm}{7.2cm}
\caption{performance variance in the descending market as \ref{var-descending}, ascending market as \ref{var-ascending}, oscillating market as \ref{var-osci}, and total variance with varied market trending as \ref{var-total}.
}
\label{performance_variance}
\end{figure}
We randomly picked four days in April 2018 with $300 \times 40$ time ticks ($\sim 0.56$hr) each day to demonstrate with the corresponding mid-price shown on top, seen in Fig \ref{results_day_samples}. We considered the transaction fee as 2\% of PnL per quantity unit. The shadow with the momentum-based and the classifier-based strategy represents all possible performances between progressive and conservative policies, and its upper edge shows the best performance at any given time tick, however, the policy may not always be the same. For the sake of reducing the risk and increasing the cumulative rewards, a mixed strategy is more reasonable. However, it requires more complex hand-crafted policies with benchmarks. Alternatively, the RL policy is a straightforward approach to a mixed strategy where actions with various quantities are taken at different states.
As shown in Fig \ref{results_day_samples}, the RL agent's performance is very close to the greedy-optimal solution in general and outperforms the other two benchmarks' average performance by $\sim$ 10\% to 30\%. In some time segments, the progressive momentum-based agent beats our RL agent (e.g, $t=100 \sim t=200$ in fig \ref{compare-2}. This is because the quantity in each RL action during this period is smaller than the momentum-based agent to avoid the future risk predicted by the transition model and that also explains why the cumulative PnL surpasses the momentum-based after $t=200$ when the mid-price starts dropping. A2C overall provides more stable performance compared to PG and DQN. DQN sometimes performance poorly, and this can be improved by having a better state representation (will be addressed in our future work). The classification-based agent performs relatively poorly, and we believe it is caused by the classification bias. The classifier tends to misclassify the movement as ``no change" because this is the dominating class, and thus the classifier-based agent takes fewer actions. This is actually a real-world bottleneck for classification methods in finance because the market doesn't have a big change within a short period in general. One of the improvement solutions is to extend the window length of each state so that each state expands a longer time horizon, and more changes may be captured. However, it will decrease the action granularity because only one action is associated with one state.
\subsection{Explanation and Analysis}
To understand and explain the performance better, we also visualize the action selections in Fig \ref{action selection} with mid-price highlighted in blue. For observation convenience, we only showed the first 100 time ticks. Three agents (RL(PG), momentum-based, and classifier-based) have similar action frequency. The RL agent's strategy can be summarized as always sell around the local peaks and buy around the local valleys. The momentum-based policy has higher action frequency to switch the ``buy" and ``sell" actions, and this may cause lower position so that smaller rewards are collected sometimes. The classifier-based agent has relatively lower action frequency, and it takes irrational actions sometimes, such as ``buy" at the local peaks in the first five time ticks. The greedy optimal has the highest trading frequency, and its main strategy is to maintain a high position and not to take actions when the one-step PnL is negative. We also compared the policy performance variance with different methods in descending markets (Fig \ref{var-descending}), ascending market (Fig \ref{var-ascending}), oscillating market (Fig \ref{var-osci}) and overall (Fig \ref{var-total}). Ascending market means mainly goes up along time (and similarly for the others). RL performs very well in descending and ascending markets, much better compared to the start-of-the-art classifier-based approach. However, it doesn't perform well in the oscillating market, caused by insufficient training data (only $\sim$15.8\% in total training data). We'll address this problem in our future work.
We also show the transferability of RL policy from the world model to the real environment. With five randomly picked days, the same policy is implemented in both world model and real environment on each day, then we compare the cumulative reward. As shown in Fig \ref{transfer}, the asymptotic performance (cumulative reward) of RL policy shows the transferability of RL policy from the environment model to the real world.
% We also proved the transferability of RL policy from the world model to the real environment. The null hypothesis is that a trajectory generated within the world model deviates from the real historical trajectory starting with the same initial state. To disclaim it, we generate five trajectories stating with five random initial states with the length of $500\times40$ time ticks. The historical trajectories starting with the same initial states are used as comparisons. The cumulative reward along each trajectory is used as the metric: with the same initial state, similar trajectories have similar cumulative rewards. As shown in Fig \ref{transfer}, the asymptotic performance of RL policy shows the transferability of RL policy from the environment model to the real world.
\begin{figure}
\centering
\includegraphics[width=0.25\textwidth]{figures/transferability.pdf}
\caption{RL policy (PG) transferability: the policy trained fully based on the environment model has acceptable performance in the real world }
\label{transfer}
\end{figure}
\section{Conclusion and Future Work}
We propose a model-based RL for learning a trading policy in finance. Our work provides the potential of RL applied in the domains where the state space is high dimensional, and real environment interactions are expensive or infeasible. We also contribute a framework for modeling trading markets for future purposes. The trading market is a complex domain, and more dynamic factors should be considered, including broker fees, dynamic transition fees, etc. Some hand-crafted rules could be combined with an RL agent (e.g., setting a risk threshold). System time latency is another concern: a delayed response may influence trading policy's efficacy. Different market data with different liquidity should be tested with our RL approach to demonstrate trading strategy robustness.
% \textbf{Disclaimer}: This paper has been prepared in part by members of the Machine Learning Center of Excellence of JPMorgan Chase & Co. (“JP Morgan Chase”). It is reflective of research only and does not represent the views or direct work of JPMorgan Chase. This paper is not investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction. Past performance is not indicative of future results J.P. Morgan disclaims any responsibility or liability whatsoever for the quality, accuracy or completeness of the information herein, and for any reliance on, or use of this material in any way.
\bibliography{ref}
\bibliographystyle{aaai}
\end{document}
\begin{thebibliography}{}
\bibitem[\protect\citeauthoryear{Ang and Bekaert}{2006}]{ang2006stock}
Ang, A., and Bekaert, G.
\newblock 2006.
\newblock Stock return predictability: Is it there?
\newblock {\em The Review of Financial Studies} 20(3):651--707.
\bibitem[\protect\citeauthoryear{Bacchetta, Mertens, and
Van~Wincoop}{2009}]{bacchetta2009predictability}
Bacchetta, P.; Mertens, E.; and Van~Wincoop, E.
\newblock 2009.
\newblock Predictability in financial markets: What do survey expectations tell
us?
\newblock {\em Journal of International Money and Finance} 28(3):406--426.
\bibitem[\protect\citeauthoryear{Bacoyannis \bgroup et al\mbox.\egroup
}{2018}]{bacoyannis2018idiosyncrasies}
Bacoyannis, V.; Glukhov, V.; Jin, T.; Kochems, J.; and Song, D.~R.
\newblock 2018.
\newblock Idiosyncrasies and challenges of data driven learning in electronic
trading.
\newblock {\em arXiv preprint arXiv:1811.09549}.
\bibitem[\protect\citeauthoryear{Beltran, Grammig, and
Menkveld}{2005}]{beltran2005understanding}
Beltran, H.; Grammig, J.; and Menkveld, A.~J.
\newblock 2005.
\newblock Understanding the limit order book: Conditioning on trade
informativeness.
\newblock Technical report, CFR Working Paper.
\bibitem[\protect\citeauthoryear{Bertoluzzo and
Corazza}{2012}]{bertoluzzo2012testing}
Bertoluzzo, F., and Corazza, M.
\newblock 2012.
\newblock Testing different reinforcement learning configurations for financial
trading: Introduction and applications.
\newblock {\em Procedia Economics and Finance} 3:68--77.
\bibitem[\protect\citeauthoryear{Bishop}{1994}]{bishop1994mixture}
Bishop, C.~M.
\newblock 1994.
\newblock Mixture density networks.
\bibitem[\protect\citeauthoryear{Dixon, Klabjan, and
Bang}{2017}]{dixon2017classification}
Dixon, M.; Klabjan, D.; and Bang, J.~H.
\newblock 2017.
\newblock Classification-based financial markets prediction using deep neural
networks.
\newblock {\em Algorithmic Finance} 6(3-4):67--77.
\bibitem[\protect\citeauthoryear{Fischer}{2018}]{fischer2018reinforcement}
Fischer, T.~G.
\newblock 2018.
\newblock Reinforcement learning in financial markets-a survey.
\newblock Technical report, FAU Discussion Papers in Economics.
\bibitem[\protect\citeauthoryear{Gould \bgroup et al\mbox.\egroup
}{2013}]{gould2013limit}
Gould, M.~D.; Porter, M.~A.; Williams, S.; McDonald, M.; Fenn, D.~J.; and
Howison, S.~D.
\newblock 2013.
\newblock Limit order books.
\newblock {\em Quantitative Finance} 13(11):1709--1742.
\bibitem[\protect\citeauthoryear{Ha and Eck}{2017}]{ha2017neural}
Ha, D., and Eck, D.
\newblock 2017.
\newblock A neural representation of sketch drawings.
\newblock {\em arXiv preprint arXiv:1704.03477}.
\bibitem[\protect\citeauthoryear{Ha and Schmidhuber}{2018}]{ha2018world}
Ha, D., and Schmidhuber, J.
\newblock 2018.
\newblock World models.
\newblock {\em arXiv preprint arXiv:1803.10122}.
\bibitem[\protect\citeauthoryear{Henaff, Canziani, and
LeCun}{2019}]{henaff2019model}
Henaff, M.; Canziani, A.; and LeCun, Y.
\newblock 2019.
\newblock Model-predictive policy learning with uncertainty regularization for
driving in dense traffic.
\newblock {\em arXiv preprint arXiv:1901.02705}.
\bibitem[\protect\citeauthoryear{Huang}{2018}]{huang2018financial}
Huang, C.~Y.
\newblock 2018.
\newblock Financial trading as a game: A deep reinforcement learning approach.
\newblock {\em arXiv preprint arXiv:1807.02787}.
\bibitem[\protect\citeauthoryear{Kaiser \bgroup et al\mbox.\egroup
}{2019}]{kaiser2019model}
Kaiser, L.; Babaeizadeh, M.; Milos, P.; Osinski, B.; Campbell, R.~H.;
Czechowski, K.; Erhan, D.; Finn, C.; Kozakowski, P.; Levine, S.; et~al.
\newblock 2019.
\newblock Model-based reinforcement learning for atari.
\newblock {\em arXiv preprint arXiv:1903.00374}.
\bibitem[\protect\citeauthoryear{Leibfried, Kushman, and
Hofmann}{2016}]{leibfried2016deep}
Leibfried, F.; Kushman, N.; and Hofmann, K.
\newblock 2016.
\newblock A deep learning approach for joint video frame and reward prediction
in atari games.
\newblock {\em arXiv preprint arXiv:1611.07078}.
\bibitem[\protect\citeauthoryear{Lillicrap \bgroup et al\mbox.\egroup
}{2015}]{lillicrap2015continuous}
Lillicrap, T.~P.; Hunt, J.~J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.;
Silver, D.; and Wierstra, D.
\newblock 2015.
\newblock Continuous control with deep reinforcement learning.
\newblock {\em arXiv preprint arXiv:1509.02971}.
\bibitem[\protect\citeauthoryear{Mnih \bgroup et al\mbox.\egroup
}{2015}]{mnih2015human}
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.~A.; Veness, J.; Bellemare,
M.~G.; Graves, A.; Riedmiller, M.; Fidjeland, A.~K.; Ostrovski, G.; et~al.
\newblock 2015.
\newblock Human-level control through deep reinforcement learning.
\newblock {\em Nature} 518(7540):529.
\bibitem[\protect\citeauthoryear{Mnih \bgroup et al\mbox.\egroup
}{2016}]{mnih2016asynchronous}
Mnih, V.; Badia, A.~P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.;
Silver, D.; and Kavukcuoglu, K.
\newblock 2016.
\newblock Asynchronous methods for deep reinforcement learning.
\newblock In {\em International conference on machine learning}, 1928--1937.
\bibitem[\protect\citeauthoryear{Nevmyvaka, Feng, and
Kearns}{2006}]{nevmyvaka2006reinforcement}
Nevmyvaka, Y.; Feng, Y.; and Kearns, M.
\newblock 2006.
\newblock Reinforcement learning for optimized trade execution.
\newblock In {\em Proceedings of the 23rd international conference on Machine
learning}, 673--680.
\newblock ACM.
\bibitem[\protect\citeauthoryear{Ntakaris \bgroup et al\mbox.\egroup
}{2018}]{ntakaris2018benchmark}
Ntakaris, A.; Magris, M.; Kanniainen, J.; Gabbouj, M.; and Iosifidis, A.
\newblock 2018.
\newblock Benchmark dataset for mid-price forecasting of limit order book data
with machine learning methods.
\newblock {\em Journal of Forecasting} 37(8):852--866.
\bibitem[\protect\citeauthoryear{Oh \bgroup et al\mbox.\egroup
}{2015}]{oh2015action}
Oh, J.; Guo, X.; Lee, H.; Lewis, R.~L.; and Singh, S.
\newblock 2015.
\newblock Action-conditional video prediction using deep networks in atari
games.
\newblock In {\em Advances in neural information processing systems},
2863--2871.
\bibitem[\protect\citeauthoryear{Rosu and others}{2010}]{rosu2010liquidity}
Rosu, I., et~al.
\newblock 2010.
\newblock Liquidity and information in order driven markets.
\newblock {\em Chicago Booth School of Business}.
\bibitem[\protect\citeauthoryear{Ro{\c{s}}u}{2009}]{rocsu2009dynamic}
Ro{\c{s}}u, I.
\newblock 2009.
\newblock A dynamic model of the limit order book.
\newblock {\em The Review of Financial Studies} 22(11):4601--4641.
\bibitem[\protect\citeauthoryear{Salimans and
Chen}{2018}]{salimans2018learning}
Salimans, T., and Chen, R.
\newblock 2018.
\newblock Learning montezuma's revenge from a single demonstration.
\newblock {\em arXiv preprint arXiv:1812.03381}.
\bibitem[\protect\citeauthoryear{Tsantekidis \bgroup et al\mbox.\egroup
}{2017a}]{tsantekidis2017forecasting}
Tsantekidis, A.; Passalis, N.; Tefas, A.; Kanniainen, J.; Gabbouj, M.; and
Iosifidis, A.
\newblock 2017a.
\newblock Forecasting stock prices from the limit order book using
convolutional neural networks.
\newblock In {\em 2017 IEEE 19th Conference on Business Informatics (CBI)},
volume~1, 7--12.
\newblock IEEE.
\bibitem[\protect\citeauthoryear{Tsantekidis \bgroup et al\mbox.\egroup
}{2017b}]{tsantekidis2017using}
Tsantekidis, A.; Passalis, N.; Tefas, A.; Kanniainen, J.; Gabbouj, M.; and
Iosifidis, A.
\newblock 2017b.
\newblock Using deep learning to detect price change indications in financial
markets.
\newblock In {\em 2017 25th European Signal Processing Conference (EUSIPCO)},
2511--2515.
\newblock IEEE.
\bibitem[\protect\citeauthoryear{Van~Hasselt, Guez, and
Silver}{2016}]{van2016deep}
Van~Hasselt, H.; Guez, A.; and Silver, D.
\newblock 2016.
\newblock Deep reinforcement learning with double q-learning.
\newblock In {\em Thirtieth AAAI conference on artificial intelligence}.
\bibitem[\protect\citeauthoryear{Williams}{1992}]{williams1992simple}
Williams, R.~J.
\newblock 1992.
\newblock Simple statistical gradient-following algorithms for connectionist
reinforcement learning.
\newblock {\em Machine learning} 8(3-4):229--256.
\bibitem[\protect\citeauthoryear{Wu \bgroup et al\mbox.\egroup
}{2017}]{wu2017flow}
Wu, C.; Kreidieh, A.; Parvate, K.; Vinitsky, E.; and Bayen, A.~M.
\newblock 2017.
\newblock Flow: Architecture and benchmarking for reinforcement learning in
traffic control.
\newblock {\em arXiv preprint arXiv:1710.05465}.
\bibitem[\protect\citeauthoryear{Zhang \bgroup et al\mbox.\egroup
}{2018}]{zhang2018solar}
Zhang, M.; Vikram, S.; Smith, L.; Abbeel, P.; Johnson, M.; and Levine, S.
\newblock 2018.
\newblock Solar: deep structured representations for model-based reinforcement
learning.
\bibitem[\protect\citeauthoryear{Zhang, Zohren, and
Roberts}{2019}]{zhang2019deeplob}
Zhang, Z.; Zohren, S.; and Roberts, S.
\newblock 2019.
\newblock Deeplob: Deep convolutional neural networks for limit order books.
\newblock {\em IEEE Transactions on Signal Processing} 67(11):3001--3012.
\bibitem[\protect\citeauthoryear{Zhao \bgroup et al\mbox.\egroup
}{2019}]{zhao2019model}
Zhao, X.; Xia, L.; Zhao, Y.; Yin, D.; and Tang, J.
\newblock 2019.
\newblock Model-based reinforcement learning for whole-chain recommendations.
\newblock {\em arXiv preprint arXiv:1902.03987}.
\bibitem[\protect\citeauthoryear{Zivot and Wang}{2006}]{zivot2006vector}
Zivot, E., and Wang, J.
\newblock 2006.
\newblock Vector autoregressive models for multivariate time series.
\newblock {\em Modeling Financial Time Series with S-Plus{\textregistered}}
385--429.
\end{thebibliography}