Recurrent Reinforcement Learning

I presented Moody and Saffell's Learning to Trade via Direct Reinforcement (IEEE TNN, 2001) in my Investments class, and reproduced their results along the way. These are short notes on the method.

Moody did some of the earliest work on avoiding the need of forecasting models in financial markets and directly learn an optimal strategy which they call direct reinforcement.

# Trading System

Let ztz_t be the price of the traded asset at time tt, and rt=zt−zt−1r_t = z_t - z_{t-1} its price change over the last period. At each step the trader holds a position Ft∈{1,0,−1}F_t \in \{1, 0, -1\}: long, neutral or short. The position is a function of the trader's parameters, its previous position and everything it has seen so far,
Ft=F(θt; Ft−1, It),It={zt,zt−1,zt−2,… ; yt,yt−1,yt−2,… }\begin{equation} F_t = F(\theta_t;\ F_{t-1},\ I_t), \qquad I_t = \{z_t, z_{t-1}, z_{t-2}, \dots;\ y_t, y_{t-1}, y_{t-2}, \dots\} \end{equation}

where ItI_t is the information set. Basically, a trader is a single recurrent unit that looks at its last position and the last m+1m+1 price changes,

Ft=sign(uFt−1+v0rt+v1rt−1+⋯+vmrt−m+w)\begin{equation} F_t = \mathrm{sign}\big(u F_{t-1} + v_0 r_t + v_1 r_{t-1} + \dots + v_m r_{t-m} + w\big) \end{equation}

We can also write Ft=sign(θ⊤[1,xt,Ft−1])F_t = \mathrm{sign}(\theta^\top [1, x_t, F_{t-1}]), where so θ={u,v0,…,vm,w}\theta = \{u, v_0, \dots, v_m, w\}, and xt=(rt,…,rt−m)x_t = (r_t, \dots, r_{t-m}). Note that during training the sign is smoothed to a tanh⁡\tanh so that FtF_t is differentiable. With position size μ\mu and transaction cost δ\delta, the trading return over period tt is

Rt=μ{Ft−1rt−δ∣Ft−Ft−1∣}\begin{equation} R_t = \mu \big\{ F_{t-1} r_t - \delta \lvert F_t - F_{t-1} \rvert \big\} \end{equation}

which is just the profit from the position held over the period, minus the cost of changing positions.

Instead of forecasting prices or learning a value function, Recurrent Reinforcement Learning (RRL) adjusts θ\theta by gradient ascent directly on a trading performance measure. Moody and Saffell repurposed the sharpe ratio to enable on-line optimization. The differential Sharpe ratio DtD_t is the first-order effect of the latest return on an exponential moving Sharpe ratio. This was their objective for measuring risk adjusted returns. To measure risk, they formulated the differential downside deviation. It is analogous to the sterling ratio (just because it is very hard to minimize drawdowns). The paper also compares this direct reinforcement approach with Q-learning on monthly S&P 500 / T-bill allocation. In my notes, I will just show the differential Sharpe Ratio, since the differential downside deviation follows the same derivation.

# Differential Sharpe Ratio

We want to maximize the Sharpe ratio over TT periods,

ST=Average(Rt)StdDev(Rt)=Rˉ1T∑t=1T(Rt−Rˉ)2,Rˉ=1T∑t=1TRt\begin{equation} S_T = \frac{\mathrm{Average}(R_t)}{\mathrm{StdDev}(R_t)} = \frac{\bar{R}}{\sqrt{\frac{1}{T}\sum_{t=1}^{T} (R_t - \bar{R})^2}}, \qquad \bar{R} = \frac{1}{T}\sum_{t=1}^{T} R_t \end{equation}

But, the Sharpe ratio is a statistic of the whole history, and we want to update after every trade. To solve this, we can keep exponential moving averages of RtR_t and Rt2R_t^2,

At=At−1+η (Rt−At−1),Bt=Bt−1+η (Rt2−Bt−1),\begin{equation} A_t = A_{t-1} + \eta\,(R_t - A_{t-1}), \qquad B_t = B_{t-1} + \eta\,(R_t^2 - B_{t-1}), \end{equation}

and expand the moving Sharpe ratio At/Bt−At2A_t / \sqrt{B_t - A_t^2} to first order in η\eta. The first-order term is the differential Sharpe ratio:

Dt=Bt−1 (Rt−At−1)−12At−1 (Rt2−Bt−1)(Bt−1−At−12)3/2\begin{equation} D_t = \frac{B_{t-1}\,(R_t - A_{t-1}) - \tfrac12 A_{t-1}\,(R_t^2 - B_{t-1})}{(B_{t-1} - A_{t-1}^2)^{3/2}} \end{equation}

DtD_t is the marginal effect of the recent return on the Sharpe ratio.

# The algorithm

Since FtF_t depends on Ft−1F_{t-1}, the gradient has to be carried forward in time, as in real-time recurrent learning.

Algorithm 1: Recurrent Reinforcement Learning (RRL)
Require: learning rate ρ\rho, EMA rate η\eta, transaction cost δ\delta

1:Initialize θ, F0, dF0dθ←0, A0←0, B0←02:for t=1,…,T do3:Ft←tanh⁡(θ⊤[1, xt, Ft−1])▹ policy (recurrent)4:Rt←Ft−1rt−δ ∣Ft−Ft−1∣▹ reward with transaction cost5:dFtdθ←∂Ft∂θ+∂Ft∂Ft−1dFt−1dθ▹ recurrence6:dRtdθ←∂Rt∂FtdFtdθ+∂Rt∂Ft−1dFt−1dθ7:dDtdRt←Bt−1−At−1Rt(Bt−1−At−12)3/2▹ differential sharpe ratio8:θ←θ+ρ dDtdRtdRtdθ▹ online gradient ascent9:At←At−1+η (Rt−At−1)10:Bt←Bt−1+η (Rt2−Bt−1)11:end for\begin{array}{rll} 1: & \text{Initialize } \theta,\ F_0,\ \dfrac{dF_0}{d\theta} \leftarrow 0,\ A_0 \leftarrow 0,\ B_0 \leftarrow 0 & \\[6pt] 2: & \textbf{for } t = 1, \dots, T \textbf{ do} & \\[4pt] 3: & \quad F_t \leftarrow \tanh\big(\theta^\top [1,\ x_t,\ F_{t-1}]\big) & \triangleright \text{ policy (recurrent)} \\[4pt] 4: & \quad R_t \leftarrow F_{t-1} r_t - \delta\, \lvert F_t - F_{t-1} \rvert & \triangleright \text{ reward with transaction cost} \\[6pt] 5: & \quad \dfrac{dF_t}{d\theta} \leftarrow \dfrac{\partial F_t}{\partial \theta} + \dfrac{\partial F_t}{\partial F_{t-1}} \dfrac{dF_{t-1}}{d\theta} & \triangleright \text{ recurrence} \\[10pt] 6: & \quad \dfrac{dR_t}{d\theta} \leftarrow \dfrac{\partial R_t}{\partial F_t} \dfrac{dF_t}{d\theta} + \dfrac{\partial R_t}{\partial F_{t-1}} \dfrac{dF_{t-1}}{d\theta} & \\[10pt] 7: & \quad \dfrac{dD_t}{dR_t} \leftarrow \dfrac{B_{t-1} - A_{t-1} R_t}{\big(B_{t-1} - A_{t-1}^2\big)^{3/2}} & \triangleright \text{ differential sharpe ratio} \\[10pt] 8: & \quad \theta \leftarrow \theta + \rho\, \dfrac{dD_t}{dR_t} \dfrac{dR_t}{d\theta} & \triangleright \text{ online gradient ascent} \\[8pt] 9: & \quad A_t \leftarrow A_{t-1} + \eta\, (R_t - A_{t-1}) & \\[4pt] 10: & \quad B_t \leftarrow B_{t-1} + \eta\, (R_t^2 - B_{t-1}) & \\[4pt] 11: & \textbf{end for} & \end{array}

# Results

Moody and Saffell performed a bunch of experiments. First, the generated synthetic data to see RRL algorithms learns to make optimal trading strategies. The trader in this experiment has 10 weights. Over 100 random markets it had a positive Sharpe ratio every time, and it learned on its own to trade less as costs went up, from about 6% of periods at 0.2% cost to about 3% at 1%.

Box plots from the paper of trading frequency, cumulative profit and Sharpe ratio against transaction costs of 0.2, 0.5 and 1 percent.

Figure 1: Trading frequency, cumulative profit and Sharpe ratio against transaction cost, over 100 trials on artificial data. From Moody and Saffell (2001), Fig. 5, © IEEE.

In the second experiment they trained and evaluated the RRL algorithm on actual US Dollar/British Pound Foreign Exchange Rates from 1996 using the differential downside objective (you just change line 7, in algorithm 1). It performed well, and also interestingly learned to reduce trading frequency with rising transaction costs. The trading system achieved an annualized Sharpe ratio of 2.3 and annualized 15% return over approx 6 months out-of-sample test period. On average, the system made a transaction per 5 hours. However, in my implementations, I tried this with USD/GBP FX data from 2010 and go very poor results. I am currently investigating this as part of my final project for the Investments class. I suspect an obvious expressivity issue that hinders representations because of the the shallow network.

The third experiment was conducted on the S&P 500 against Treasury bills, 1970 to 1994, the RRL trader is also a single tanh⁡\tanh unit. Because one trader's result depends on its random initial weights, the paper trains 30 of them independently and trades their combined position. The Q-learning baseline (30 hidden units) is evaluated the same way. The RRL ensemble ended at about 59 times its starting wealth, against 34 times for the Q-learning ensemble and 13.5 times for buy and hold, with annualized Sharpe ratios of 0.83, 0.63 and 0.34. Most of the edge came from moving into T-bills at the right times including the 1974 oil shock, the early-1980s rate spike and the 1987 crash.

Equity curves from the paper on a log scale for the RRL trader, the Q-trader and buy and hold from 1970 to 1994, with each trader's position below.

Figure 2: Equity of the RRL and Q-trader committees and buy and hold, 1970 to 1994, with each committee's position below. From Moody and Saffell (2001), Fig. 9, © IEEE.

# References

  • J. Moody and M. Saffell, Learning to Trade via Direct Reinforcement, IEEE Transactions on Neural Networks 12(4), 2001;
  • J. Moody, L. Wu, Y. Liao and M. Saffell, Performance Functions and Reinforcement Learning for Trading Systems and Portfolios, Journal of Forecasting 17, 1998.
← All notes

© 2026 by Divin. Design inspired by Goodfire