A General Framework on Enhancing Portfolio Management with Reinforcement Learning
Extracted source text · arXiv 1911.11880
\documentclass[11pt]{article}
\usepackage{geometry}
\geometry{margin=1in}
\usepackage{arxiv}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{lipsum}
\usepackage[pdftex]{graphicx}
\title{Reinforcement Learning in Dynamic Portfolio Management}
\author{
Junhao Wang \\
Data Science Institute\\
Columbia University\\
New York, NY 10027 \\
\texttt{jw3668@columbia.edu} \\
%% examples of more authors
\And
Yinhen Li \\
Data Science Institute\\
Columbia University\\
New York, NY 10027 \\
\texttt{yl4039@columbia.edu} \\
\And
Yijie Cao\\
Data Science Institute\\
Columbia University\\
New York, NY 10027 \\
\texttt{yc3544@columbia.edu} \\
}
\begin{document}
\maketitle
\begin{abstract}
Portfolio optimization is the process of managing a collection of assets to achieve optimal return with low risk in some consecutive trading periods. Although there are previous works on automating this process with reinforcement learning, the actions are limited to buy, hold and sell the assets. In this paper, we allow the agents to short the assets and our model accommodates buy/sell transaction costs. We implement two state-of-art reinforcement learning algorithms, Policy Gradient Actor Critic (PGAC) and Evolution Strategy (ES), both are widely-used and successful in game playing and robot control. We compare both algorithms and show that they out perform the benchmark.
\end{abstract}
% keywords can be removed
\keywords{Reinforcement Learning \and Portfolio Management \and Deep Learning \and Policy Gradient \and Evolution Strategy \and Actor Critic}
\section{Introduction}
Portfolio optimization remains one of the most challenging problems in the field of finance. It's the decision making process of continuously reallocating an amount of fund into a number of different financial investment products, aiming to maximize the return while restraining the risk \cite{feldman1992portfolio,haugen2001modern}. Recent years, utilizing deep reinforcement learning in portfolio management has been studied in many researches. The main idea is to train RL agents to trade and get reward as gain or loss from portfolio value. There are many previous works focus on discrete single-asset trading \cite{deng2016deep,cumming2015investigation} or multi-assets management \cite{guo2018robust,jiang2017deep}. However, most of them did not use a realistic setting, for which they did not take transaction cost into account or not allow short on assets, thus, limit their practical usage. In our experiment, we designed an environment that simulates true market behavior in each time step and generates reward for RL agents with transaction cost deducted.Then, we proposed two RL algorithms (PGAC, ES) that can learn to trade by allocating negative (short) and positive (long) weights for each asset with transaction cost. Finally, we fine tuned hyper parameters and compared the result of two models with the benchmarks.
\section{Background}
\subsection{Portfolio optimization}
Dynamic portfolio optimization is the process of optimal assets allocation
in different time with high return in total portfolio value and low risk. There are four main categories in traditional portfolio management methods, ”Follow-the-Winner”, ”Follow-the-Loser”, ”Pattern-Matching”, and ”Meta-Learning”\cite{li2014online}. Classical approaches using dynamic programming and convex optimization, which require discrete action space\cite{li2014online}. In the era of deep machine learning, there are models that proposed focus on ”Pattern-Matching”, and ”Meta-Learning”. ”Pattern-Matching” algorithm aims to predict the market distribution in next time period base on historical data. ”Meta-Learning” method uses combination of other strategies and meta data on financial assets.
\\In each portfolio, there consists $M$ assets where
$W = [w_{1},w_{2}...w_{i}...w_{n}]$ denote the N-dimensional weight vector subject to $\sum_{i=1}^{i=n}{w_{i}}=1$. Each $w_{i}$ is the weight allocated to asset $i$.
$\xi = \left[\xi_{1}, \ldots, \xi_{n}\right]$ denotes the price vector where $\xi_{i}$ is the price of asset $i$. At time $t$, the total value of the portfolio $V_t = W_t \xi_t^T$. In the portfolio, we assume there is a risk-free asset that we can hold, denotes $w_r$ with fixed $\xi_r=1$. The existing machine learning approaches to portfolio optimization problems try to maximize $V_t - V_0$ as well as put constraint on different risk measurements to avoid shortfall.
\subsection{Reinforcement learning in portfolio optimization}
Reinforcement learning has been proposed as a suitable candidate for portfolio optimization by its own nature. The RL algorithm is able to learn by interacting with the environment and get rewards from the environment. In the portfolio optimization setting, the RL agent is trying to get positive reward in each time step $V_t-V_{t-1}$ by allocating weights $w_i$ for each asset in the portfolio. There are existing RL algorithms that manage to deal with this problem \cite{guo2018robust,jiang2017deep,liang2018adversarial}, however, there are several improvement can be made in their approaches. For instance, in the work of \cite{guo2018robust}, the authors did not consider transaction costs as a negative term in their reward function. The RL agent was not penalized when it's more actively trading. Therefore, the algorithm is not guaranteed to perform well in the realistic setting. Other works \cite{jiang2017deep,liang2018adversarial} take transaction costs in to account, yet they only consider long position for the assets in the portfolio. This may limit the performance of the RL agent by cutting of a powerful way of hedging.\\
The general reinforcement learning framework for portfolio management was proposed by \cite{jiang2017deep} and used in \cite{liang2018adversarial}. They used different reinforcement learning algorithms to achieve promising result on different markets. In \cite{jiang2017deep}, the authors introduced a general deterministic policy gradient algorithm to train the agent with different policy network architecture. The experiment was conducted in the crypto-currency market because its high volume and Bitcoin was its risk-free asset. In the work of \cite{liang2018adversarial}, the authors compared three reinforcement learning models, Deep Deterministic Policy Gradient(DDPG), Proximal Policy Optimization(PPO) and Policy Gradient(PG) that were proven successful in many environments.
\section{Environment Design and Data modeling}
The RL environment contains three parts: State, Action, Reward. As the goal of our project is financial portfolio management, we set our action to be the portfolio weights corresponding to each asset for each time period. Assume we have n assets in our portfolio and our total portfolio weight is 1. For $t$'th trading period, we denote the action as a vector of portfolio weights: $W_t = [w_{1,t},w_{2,t}...w_{i,t}...w_{n,t}]$ s.t. $\sum_{i=1}^{i=n}{w_{i,t}}=1$.
For each trading period, the agent submit $W_t$ to the environment(trading market) and get a reward $r_t$, which is the log return of our portfolio from time $t-1$ to $t$. Let $p_t$ denote the portfolio value at time $t$, then we define $r_t = \log{\frac{p_t}{p_{t-1}}}$. If trading cost is considered, for each trading period, $r_t$ needs to be modified to reflect the loss of trading cost. As suggested in Jiang's paper \cite{jiang2017deep} and \cite{ormos2013performance}, a shrinkage vector $\zeta$ can be applied our portfolio. Assume the cost of sell is $c_s$ and the cost of purchase is $c_p$, then $\zeta$ is recursively calculated in the following way suggested in Jiang's paper \cite{jiang2017deep}.
\begin{equation}
\zeta_{t}=\frac{1}{1-c_{\mathrm{p}} w_{ 0,t}}
\left[1-c_{\mathrm{p}} w_{ 0,t}^{\prime}-\left(c_{\mathrm{s}}+c_{\mathrm{p}}-c_{\mathrm{S}} c_{\mathrm{p}}\right) \sum_{i=1}^{m}\left(w_{i,t}^{\prime}-\zeta_{t} w_{i,t}\right)^{+}\right]
\end{equation}
Denote the closing price for asset $i$ at period t to be $v_{i,t}$, and $V_t = [v_{1,t},v_{2,t}...v_{i,t}...v_{n,t}] $ then $w_{i,t}^{\prime} = \frac{v_{i,t}w_{i,t}}{W_{t-1} V_{t-1}^T} $ Hence, the reward after considering transaction cost becomes $R_t = \zeta r_t$ For each trading period, $R_t$ is returned to our agent as reward.
The state is constructed using a 3 dimensional tensor. For each time period t, we collect the most recent $d$ days of information for each feature and each asset. Those $d$ days will be referred as horizon. Features can be the open price, closing price, trading volume and etc. for corresponding asset. As the closing price is used to calculate the portfolio value, it must be included in the tensor. Let $e_{i,t,j}$ denote an entry in the tensor $E$. $e_{i,t,j}$ represents the information (or price or value) of asset $i$ at time $t$ for feature $j$. For $e \in E, i=1,2,...,n \quad t =\tau, \tau-1, ...,\tau-d \quad j = 1,2,...,m $, $m$ is the total number of features. At time t, the environment will return a tensor $E$ to our agent as a state.
\section{Reinforcement Learning Algorithm}
\subsection{Policy Gradient Actor Critic}
\subsubsection{Soft introduction to Policy Gradient Actor Critic}
In this paper, we use a policy gradient actor critic (PGAC) algorithm as our first Reinforcement Learning algorithm. Compared with the vanilla policy gradient algorithm used in the Jiang's paper \cite{jiang2017deep} , PGAC algorithm is more robust and stable. Unlike vanilla policy gradient algorithm, PGAC has a separate neural network for the value function, which reduces its chance in converging into the local optimal\cite{konda2000actor}. Meanwhile, by parameterizing the value in each state, PGAC helps the agent learn better and faster from the previous experience \cite{grondman2012survey}.
A standard PGAC algorithm has the following steps\cite{sutton2000policy}:\
\begin{center}
\begin{tabular}{l}
step 1. sample $\left\{\mathbf{s}_{i}, \mathbf{a}_{i}\right\}$ from $\pi_{\theta}(\mathbf{a} | \mathbf{s})$\\
step 2. fit $\hat{V}_{\phi}^{\pi}(\mathbf{s})$ to sampled reward sums\\
step 3. evaluate $\hat{A}^{\pi}\left(\mathbf{s}_{i}, \mathbf{a}_{i}\right)=r\left(\mathbf{s}_{i}, \mathbf{a}_{i}\right)+\gamma \hat{V}_{\phi}^{\pi}\left(\mathbf{s}_{i}^{\prime}\right)-\hat{V}_{\phi}^{\pi}\left(\mathbf{s}_{i}\right)$\\
step 4. $\nabla_{\theta} J(\theta) \approx \sum_{i} \nabla_{\theta} \log \pi_{\theta}\left(\mathbf{a}_{i} | \mathbf{s}_{i}\right) \hat{A}^{\pi}\left(\mathbf{s}_{i}, \mathbf{a}_{i}\right)$\\
step 5. $\theta \leftarrow \theta+\alpha \nabla_{\theta} J(\theta)$\\
\end{tabular}
\end{center}
A few modifications were made to fit in portfolio management algorithm. First, time value of money is ignored as the training period is relative short. Therefore, we set discount rate $\gamma =1 $. Second, the action $a$ is a vector of portfolio weights, which should be continuous. This will affect the step 4 gradient calculation.
Rewrite step 4 to be
\begin{equation}
\nabla_{\theta} J(\theta)=E_{\tau \sim \pi_{\theta}(\tau)}\left[\left(\sum_{t=1}^{T} \nabla_{\theta} \log \pi_{\theta}\left(\mathbf{a}_{t} | \mathbf{s}_{t}\right)\right)\left(\sum_{t=1}^{T} A\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)\right)\right]
\end{equation}
In the continuous space, summation is replaced with expectation. To generate continuous actions, we use a normal distribution whose mean equals the output of the policy neural net. In this case, we add a Gaussian Noise into our system which helps it explore various states and actions. As we reach the later stage of training, we reduce the variance of the normal distribution to generate more accurate actions.Here is the policy formula:
\begin{equation}
\pi_{\theta}\left(\mathbf{a}_{t} | \mathbf{s}_{t}\right) = \mathcal{N}\left(f_{neural \, net} \left({s}_{t}\right) ; \Sigma\right)
% \mathcal{N}\left(f_{\text { neural network }}\left({s}_{t}\right) ; \Sigma\right)
\end{equation}
Another advantage for adding this Gaussian Noise is that it produces an explicit form of derivative, making it possible for neural network to take gradient. As is show below:
\begin{equation}
\log \pi_{\theta}\left(\mathbf{a}_{t} | \mathbf{s}_{t}\right)=-\frac{1}{2}\left\|f\left(\mathbf{s}_{t}\right)-\mathbf{a}_{t}\right\|_{\Sigma}^{2}+\mathrm{const}
\end{equation}
\begin{equation}
\nabla_{\theta} \log \pi_{\theta}\left(\mathbf{a}_{t} | \mathbf{s}_{t}\right)=-\frac{1}{2} \Sigma^{-1}\left(f\left(\mathbf{s}_{t}\right)-\mathbf{a}_{t}\right) \frac{d f}{d \theta}
\end{equation}
\subsubsection{Neural Network Construction}
\begin{figure}[!htb]
\centering
\includegraphics[width=6in]{arc.png}
\caption{CNN architecture for PGAC model}
\label{picture-4}
\end{figure}
According to in Jiang's Paper\cite{jiang2017deep}, Convolution Neural Network(CNN) works the best in his model. Therefore, we use CNN in both policy network and value function network.
In this work, our training data contains 4 assets (one of them being riskless asset), 7 features and 50 time horizons. Specifically, $n=4,m=7,d=50$. Apply the first convolution filter of size (1,3), we get a tensor of $n=4,m=7,d-2=48$. Then apply the second convolution filter of size (1,48), we get an output vector of shape (4,1). Concatenate this output vector with the previous actions(weights), we get 2 feature maps with size (4,1). At last, add a fully connected layer and use "tanh" as activation function. The output is a vector of length 4. Each value of this vector stands for the mean of weights we should keep in the corresponding asset. The previous actions vector is added to the neural network is to help the agent to learn the trading cost and avoid frequently trading. "tanh" is used in the last activation function in order to produce a continuous value range from -1 to 1. It not only enables shorting any asset, but also keeps the weight of any asset to be within a reasonable range in accordance with the real life scenario.
Similar setting is used in the Value Net. The only difference is that no previous weights is added to the network and the final output is a number representing the value of a state instead of a vector.
In training the RL algorithm, closing prices of 4 traded assets (if one of them is riskless asset, then the price remains constant 1) are selected to be one feature. In addition, other relevant asset price and the 5, 10 day moving average price for selected assets are also selected as features. As the daily price change tends to be very small, the CNN will not be able to tell the difference and produce different results given very similar input tensors. Therefore, an amplification method is used before inputting tensor into the neural network. Denote the original tensor as $E, e_{i,t,j}\in E$ and amplified tensor as $E',e_{i,t,j}' \in E'$. set $e_{i,t,j}' = K \frac{e_{i,t,j}}{e_{i,t-1,j}} $ and when $t=1, e_{i,1,j}' = K$. In this work, K is set to be 100. It can be a hyper-parameter to tune.
In the real trading scenario, the total weights for any portfolio should sum up to 1 but there is no guarantee the sum of actions(weights) will sum up to 1. However, setting the weight of riskless asset to be 1 minus all the other weights solves this problem.
After all the setting, 500 days data is used in training and 10 days right after the training period is used to test.
\subsection{Evolution Strategy}
\subsubsection{Soft Introduction to Evolution Strategy}
There are three general approaches to reinforcement learning: valued based algorithm,policy based algorithm mixed of previous two. First technique include methods such as deep Q learning\cite{mnih2015human}, double Q learning\cite{van2016deep},etc. The second include families of policy gradient method such as , TRPO\cite{schulman2015trust},PPO\cite{schulman2017proximal}, and the last include method that is mixed of both,Policy Gradient Actor Critic\cite{sutton2000policy}, A3C\cite{mnih2016asynchronous}, etc. However, for the actions space that we defined in the context of portfolio management which is high dimensional continuous space, it is nature to choose policy based method. Being a close variation of policy gradient algorithm, evolution strategy is a class of black box optimization technique that is first proposed by Rechenberg and Eigen in 1973 \cite{huning1976evolutionsstrategie} . However, it was not until 2017 when Openai published distributed evolution strategy\cite{salimans2017evolution} can Evolution Strategy vastly grow its popularity. For evolution strategy each iteration, a population of certain number of agents,policies in the context of reinforcement learning, are generated, and each agent’s reward will be evaluated to serve as a way to guide the next generation in the right ‘direction’.
\subsubsection{comparison between ES and Policy Gradient}
Policy gradient and evolution strategy are different mostly in terms of they way gradient are computed and subsequently the sampling procedure.For policy gradient algorithm, we want to directly maximize the objective function
\begin{equation}
\mathbb{E}_{\theta \sim p_{\psi}} F(\theta)
\end{equation}
where theta is the policy of the agent which in this case is just the parameter of the action nerual net. Then F maps the sequence of action,portfolio weights of each asset that are being managed,the agent obey with respect to particular policy of the neural net to the finite horizon undiscounted reward, the percentage portfolio value in compare to what it has at time 0.
The gradient of this objective function is given by the following equation.
\begin{equation}
\nabla_{\psi} \mathbb{E}_{\theta \sim p_{\psi}} F(\theta)=\mathbb{E}_{\theta \sim p_{\psi}}\left\{F(\theta) \nabla_{\psi} \log p_{\psi}(\theta)\right\}
\end{equation}
In evolution strategy, instead of assuming randomized policy, we perform a Gaussian smoothing ,adding a Gaussian noise to the policy, and treat the policy itself as deterministic instead. Then our objective function becomes
\begin{equation}
\mathbb{E}_{\theta \sim p_{\psi}} F(\theta)=\mathbb{E}_{\epsilon \sim N(0, I)} F(\theta+\sigma \epsilon)
\end{equation}
To compute the gradient of this new objective function,one can evaluate it by the R.H.S of the following equation.
\begin{equation}
\nabla_{\theta} \mathbb{E}_{\epsilon \sim N(0, I)} F(\theta+\sigma \epsilon)=\frac{1}{\sigma} \mathbb{E}_{\epsilon \sim N(0, I)}\{F(\theta+\sigma \epsilon) \epsilon\}
\end{equation}
Then the gradient of the objective is evaluated directly using samples from the rewards from query the policy adding their respective Gaussian noise in the environment. In this way, evolution strategy manage to estimate the gradient of original objective function without actually taking the gradient to which people sometimes refer as gradient-free method. To calculate the reward of one particular noise added policy,one can simply query the environment using this policy and get a total rewards at the end of this roll out. Then policy can be updated using samples of this iteration and repeat the procedure until a desired reward is achieved.
\subsubsection{Action Net Modification and miscellaneous}
Since the state space that we designed for the portfolio management problem is a three dimensional tensor, it will be impractical and undesirable to use convolution neural net as policy neural net. Due to the curse of dimensionality, to get a good quality gradient estimate,it requires exponential amount of sample with respect to dimensionality of the policy space. Even though, there are technique to handles this issue, having large state space is unfavorable in general and can results in unstable policy during training. It is always wise to constrain the dimension of the state space. For this portfolio management problem, I truncated the horizon of state tensor to 3, using three days of historical price to make each portfolio weights decision. We choose standard fully connected layer in replace with convolution neural layers which require us to flatten the input state space to 1 dimensional. Then our action net composes of two standard feed forward fully connected layer. We also utilize one of the biggest advantage that evolution strategy offers, ability to easily distributed. Incorporating the random seed technique that proposed by OpenAi on CPU, we manage to train the policy in a single machine in a much shorter time in compare to evolution strategy without distributed computing in our experiment.In addition, we also constrain the maximum number of trade an agent can perform in a roll out to be 50. It’s conducive to restrict the maximum trading period during training to a small window since in reality trading history far back in the past has less importance than those that is close to current time. The trained policy is tested for the next 10 days follow from the training period, and the results can be find in appendix.
\section{results}
The testing results are showed in figure 2, figure 3 each corresponds to PGAC agent and ES agent respectively. Figure 4 shows the price for all the assets that are being managed within the testing window to serve as a baseline comparsion. For the portfolio management agent design we have, ES agent outperforms PGAC agent. However, we notice that ES agent performs really well only till around 5,6th testing days, then almost all the ES agent experience some performance decline. It could be the case that 50 days of testing isn't suffice to train the ES agent to predict 10 days all the way into the future. However, overall ES agent still manage to perform better.
\begin{figure}[!htb] %!t
\centering
\includegraphics[width=6in]{baseline.png}
\caption{Baseline 10-day}
\label{picture-1}
\end{figure}
\begin{figure}[!htb] %!t
\centering
\includegraphics[width=6in]{pgac.png}
\caption{PGAC 10-day}
\label{picture-2}
\end{figure}
\begin{figure}[!htb] %!t
\centering
\includegraphics[width=6in]{es.png}
\caption{ES 10-day}
\label{picture-3}
\end{figure}
\section{future improvement}
\begin{table}
\caption{PGAC 10-day result}
\centering
\begin{tabular}{lllll}
\toprule
\cmidrule(r){1-2}
Trails & Portfolio Value & Sortino Ratio & Sharpe Ratio & MMD \\
\midrule
1st & 1.026 & 112.318 & 1.196 & 0.917 \% \\
2nd & 1.009 & 62.505 & 1.150 & 0.701 \% \\
3rd & 1.004 & 148.957 & 1.249 & 0.242 \% \\
4th & 1.030 & 299.881 & 1.224 & 1.565 \% \\
5th & 0.995 & -0.653 & -0.505 & 1.697 \% \\
\bottomrule
\end{tabular}
\label{tab:table-1}
\end{table}
\begin{table}
\caption{ES 10-day result}
\centering
\begin{tabular}{lllll}
\toprule
\cmidrule(r){1-2}
Trails & Portfolio Value & Sortino Ratio & Sharpe Ratio & MMD \\
\midrule
1st & 1.103 & 6.050E+13 & 1.196 & 0.917 \% \\
2nd & 1.043 & 2.958E+13 & 1.150 & 0.701 \% \\
3rd & 1.006 & 1.220E+13 & 1.249 & 0.242 \% \\
4th & 1.028 & 3.471 & 1.224 & 1.565 \% \\
5th & 1.020 & 5.523E+13 & -0.505 & 1.697 \% \\
\bottomrule
\end{tabular}
\label{tab:table-2}
\end{table}
\begin{table}
\caption{PGAC 10-day result}
\centering
\begin{tabular}{lllll}
\toprule
\cmidrule(r){1-2}
Trails & Portfolio Value & Sortino Ratio & Sharpe Ratio & MMD \\
\midrule
1st & 1.026 & 112.318 & 1.196 & 0.917 \% \\
2nd & 1.009 & 62.505 & 1.150 & 0.701 \% \\
3rd & 1.004 & 148.957 & 1.249 & 0.242 \% \\
4th & 1.030 & 299.881 & 1.224 & 1.565 \% \\
5th & 0.995 & -0.653 & -0.505 & 1.697 \% \\
\bottomrule
\end{tabular}
\label{tab:table-3}
\end{table}
For actor critic policy gradient.
Although actor critic policy gradient has a more stable performance than vanilla policy gradient, it is not stable enough. In 5 tests for the same data, the agent learns different trading algorithms and strategies. One major problem lies in the value net. In financial applications, it is hard to define the value of a state, what does it even mean. It is even harder to evaluate the value of a state and to take actions based on it. Meanwhile, in the convergence assumption of actor critic, it requires the agent to visit all states equivalent often, which is hardly possible in any financial scenario. One possible solution is to prepossess the features and reduce the dimension of the states as suggested [!!!]. In this way, the RL agent will get less noise, have relatively more data in training and become faster to converge to an optimal policy.
For evolution strategy.
One big issue that we find about our approach using evolution strategy is that actions weights we obtained is always -1, +1 despite going through tanh activation in the last layer of action neural net. Our guess is that once the policy is trained, agent become over confident about what is happening in the equity market in the next stet which encourage the agent to behave greedy according to its belief even with trading cost. This is probably due to the fact that we do not have risk factor designed in the environment, and just from the trading cost alone is not enough to let the agent learn to trade conservatively. In addition, We should also include technique that encourage exploration in the policy space to escape local optimal.It is even more crucial under the context of our environment definition where policy space can be huge depending on the length of horizon , number of features, as well as the number of assets we want to manage. Without good exploration, the agent will stuck in local optimal.
\clearpage
\bibliographystyle{plain}
\bibliography{references}
\end{document}
\begin{thebibliography}{10}
\bibitem{cumming2015investigation}
James Cumming, Dr~Dalal Alrajeh, and Luke Dickens.
\newblock {\em An investigation into the use of reinforcement learning
techniques within the algorithmic trading domain}.
\newblock PhD thesis, 2015.
\bibitem{deng2016deep}
Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai.
\newblock Deep direct reinforcement learning for financial signal
representation and trading.
\newblock {\em IEEE transactions on neural networks and learning systems},
28(3):653--664, 2016.
\bibitem{feldman1992portfolio}
Keith Feldman.
\newblock Portfolio selection, efficient diversification of investments. by
harry m. markowitz (basil blackwell, 1991){\pounds} 25.00.
\newblock {\em Journal of the Institute of Actuaries}, 119(1):165--166, 1992.
\bibitem{grondman2012survey}
Ivo Grondman, Lucian Busoniu, Gabriel~AD Lopes, and Robert Babuska.
\newblock A survey of actor-critic reinforcement learning: Standard and natural
policy gradients.
\newblock {\em IEEE Transactions on Systems, Man, and Cybernetics, Part C
(Applications and Reviews)}, 42(6):1291--1307, 2012.
\bibitem{guo2018robust}
Yifeng Guo, Xingyu Fu, Yuyan Shi, and Mingwen Liu.
\newblock Robust log-optimal strategy with reinforcement learning.
\newblock {\em arXiv preprint arXiv:1805.00205}, 2018.
\bibitem{haugen2001modern}
Robert~A Haugen.
\newblock {\em Modern investment theory}, volume~5.
\bibitem{huning1976evolutionsstrategie}
Alois Huning.
\newblock Evolutionsstrategie. optimierung technischer systeme nach prinzipien
der biologischen evolution, 1976.
\bibitem{jiang2017deep}
Zhengyao Jiang, Dixing Xu, and Jinjun Liang.
\newblock A deep reinforcement learning framework for the financial portfolio
management problem.
\newblock {\em arXiv preprint arXiv:1706.10059}, 2017.
\bibitem{konda2000actor}
Vijay~R Konda and John~N Tsitsiklis.
\newblock Actor-critic algorithms.
\newblock In {\em Advances in neural information processing systems}, pages
1008--1014, 2000.
\bibitem{li2014online}
Bin Li and Steven~CH Hoi.
\newblock Online portfolio selection: A survey.
\newblock {\em ACM Computing Surveys (CSUR)}, 46(3):35, 2014.
\bibitem{liang2018adversarial}
Zhipeng Liang, Hao Chen, Junhao Zhu, Kangkang Jiang, and Yanran Li.
\newblock Adversarial deep reinforcement learning in portfolio management.
\newblock {\em arXiv preprint arXiv:1808.09940}, 2018.
\bibitem{mnih2016asynchronous}
Volodymyr Mnih, Adria~Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy
Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.
\newblock Asynchronous methods for deep reinforcement learning.
\newblock In {\em International conference on machine learning}, pages
1928--1937, 2016.
\bibitem{mnih2015human}
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei~A Rusu, Joel Veness,
Marc~G Bellemare, Alex Graves, Martin Riedmiller, Andreas~K Fidjeland, Georg
Ostrovski, et~al.
\newblock Human-level control through deep reinforcement learning.
\newblock {\em Nature}, 518(7540):529, 2015.
\bibitem{ormos2013performance}
Mih{\'a}ly Ormos and Andr{\'a}s Urb{\'a}n.
\newblock Performance analysis of log-optimal portfolio strategies with
transaction costs.
\newblock {\em Quantitative Finance}, 13(10):1587--1597, 2013.
\bibitem{salimans2017evolution}
Tim Salimans, Jonathan Ho, Xi~Chen, Szymon Sidor, and Ilya Sutskever.
\newblock Evolution strategies as a scalable alternative to reinforcement
learning.
\newblock {\em arXiv preprint arXiv:1703.03864}, 2017.
\bibitem{schulman2015trust}
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp
Moritz.
\newblock Trust region policy optimization.
\newblock In {\em International conference on machine learning}, pages
1889--1897, 2015.
\bibitem{schulman2017proximal}
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
\newblock Proximal policy optimization algorithms.
\newblock {\em arXiv preprint arXiv:1707.06347}, 2017.
\bibitem{sutton2000policy}
Richard~S Sutton, David~A McAllester, Satinder~P Singh, and Yishay Mansour.
\newblock Policy gradient methods for reinforcement learning with function
approximation.
\newblock In {\em Advances in neural information processing systems}, pages
1057--1063, 2000.
\bibitem{van2016deep}
Hado Van~Hasselt, Arthur Guez, and David Silver.
\newblock Deep reinforcement learning with double q-learning.
\newblock In {\em Thirtieth AAAI conference on artificial intelligence}, 2016.
\end{thebibliography}