Evaluate the robustness of an automated trading strategy at first glance

You can find many automated trading systems on the internet that show superhuman performance on paper. But, when you launch it on your real account, you lose immediately your money.

A series of questions is being raised to which I shall refer in this post:

• How know if a strategy is robust or over-fitted?
• How to be sure that my strategy doesn’t lose money immediately after production launching?
• Is there any way to evaluate the robustness of a strategy?

I will explain what you should verify to be sure your automated trading system is over-fitted or not. I created a battery of tests to validate the robustness of a strategy before launching on a real account. In this post, I will show you some tests you can make at a glance.

The global standard deviation of your strategy

The first thing you should watch is the standard deviation over the duration of the strategy. The standard deviation should not be too high and should be regular over the duration of the strategy. This regularity must be independent of market fluctuations.

Too large standard deviation

Here is the result of a strategy reflecting too large standard deviation:

This strategy gives a performance greater than 10% but the standard deviation is greater than 30%.

Acceptable standard deviation

Here is the result of a strategy reflecting a reasonable standard deviation:

This strategy gives a performance lower than 10% but the standard deviation is lower than 2.5%.

The performance of the first strategy is more than double that the second but its standard deviation is very high. That reflects more uncertainty thus more risks.

The performance of the second strategy is lower than the previous one but its standard deviation is much smaller. That means the risk of the second strategy is lower than the first thus it will be more profitable in the long term.

Earnings distribution

The earnings should not be concentrated on a little period of your test. If earnings are concentrated on a little part of the test period, that is evident proof of over-fitting. That means your strategy is over-fitted on a little part of the market, which is unlikely to happen again in the future.

Too concentrated earnings distribution

Here is the result of a too concentrated earnings distribution on a period of the market:

This strategy produces a performance greater than 10% but, earnings are totally concentrated on a little part of the test.

Dispersed earnings distribution

The following backtest shows the strategy generates steady gains throughout the test period:

This strategy produces a performance lower than 2% but, earnings are dispersed on all periods of the test.

The earning distribution can be used to distinguish two strategies. It would be better to choose a less profitable strategy but with more regular profits.

The measure of Tracking Error

The tracking error allows you to measure how a strategy follows or not the traded asset. It is the standard deviation of the difference between strategy performance and asset performance.

Tracking Error = σ(Strategy return – Asset return)

A tracking error of a robust strategy would be as small as possible.

Tracking error of a strategy

The graph below shows an example of a strategy performance with its asset performance. We can see the gains curve follows the asset curve. The difference between the strategy performance and asset performance is rather stable. We can say the tracking error is acceptable. Thus, the strategy is also acceptable.

Some strategies as “scalping” can be uncorrelated with their asset cause they execute itself in a very short term time frame. In this case, it could be correct that this kind of strategy is more impacted by volatility than the global trend.

Personally I prefer to avoid too volatile markets even if there is a bull trend. Thus, in this case, it is normal to observe a slightly higher tracking error.

If the tracking error is very nearly zero, then it might be less risky to buy and hold a tracker of this asset.

However, if the tracking error of your strategy is near zero with leverage lower or equal to 1 but capable of closing its position each day at market closing, it could be exciting. Cause, in this case, your strategy reaches to replicate the performance of traded assets but keep safe your cash each evening. That allows you to increase your capacities of capital allocation management and decrease your exposition to financial risks.

The profit/loss ratio

The average profit would be greater than the average loss. On the other hand, these are to be expected for the highest profit which would also be greater than the lowest loss.

In practice, the profit/loss ratio showed by your backtest is not always relevant. Why? Cause it’s too easy to cheat yourself. I will explain to you why. If you launch a backtest whose number of losing entries is too important for you, you may be tempted to move away stop loss. In this way, you were artificially inflating the success rate of your strategy. Because if a stop loss is placed lower, the likelihood that it touched is lower. In return, risks will increase cause when a stop loss is touched, there will be a bigger loss.

We could push it to its paroxysm removing stop loss. Obviously, if the market is extremely bullish, this way will be very profitable until market backfires on you.

In the following example, I voluntary over-fitted the profit rate of a strategy by increasing stop loss at 1000 points on DAX Index:

This strategy shows a profit rate of 100% on 23 entries. This result appears exceptional, but if you look at the profit curve, something should jump out at you: we can see that an entry creates an enormous latent loss sometime before the win.

Now, look at what would have happened if I had started this strategy on my real account for some weeks:

The strategy takes only one loss but it is a loss of 562 points. And just one loss of 562 points against an average profit of 15 points can devastate all performance of the strategy.

Here we can see consequences of only one big loss on all strategy profit:

You should be very attentive about profit/loss ratio showed by your backtest. Cause that does not reflect every time the reality. If you want to compute the true profit/loss ratio, you need to use stop loss in place of average loss.

If you want to learn more about profit/loss ratio, I invite you to read this post : Simulate the back to heads or tails

Profit securing

For the sake of robustness, a strategy must be equipped with a profit securing mechanism. This mechanism purpose is automatically placing a sell stop order when the latent profits are greater than a certain level.

In this way, if the price felt before reaching the initial target, the opened position will stay a winner. (except if there is a gap or a flash crash)

Therefore, it is indispensable that you verify if your strategy secures profits looking the latent profits do not turn themselves into losses or ridiculous gains. You can easily control that at a glance because if profits are not secured, some pikes will appear on the gains curve. In addition, these pikes become losses. The bellow graph shows you two pikes on the gain curve:

Basically, in the case of trend continuation strategy, the level of profit securing should be greater than or equal to the stop loss level in terms of points. For example, if you placed a stop loss at 50 points, you should begin profit securing at least 50 points of latent profits.

Personally, I prefer to add some points in addition to my stop loss level. Thus if I place a stop loss at 50 points, I will set the beginning of profit securing at 55 points of latent profits.

Time in the market

The time in the market is the duration of an opened trade. And it should be reasonable. If the market duration is greater than a day then you must sub the overnight fees to your backtest performance.

It is imperative to follow the evolution of your strategy market duration. If the time in the market becomes increasingly longer, maybe it reveals your strategy is over-fitted. You should also monitor time in the market carefully while your strategy is running on your real account.

Standard-deviation of time in the market

At the beginning of this post, I showed you how to measure the risks of a strategy thanks to standard deviation. We can do the same with time in the market.

The standard deviation of the time in the market should not be too high and should also be regular over the duration of the strategy. Otherwise, that means there are trades having a short time in the market and others a very long time. This could be indicative of the under-fitted problem of your strategy.

Here is an example of a strategy with too much standard-deviation of its time in the market:

The time in the market showed by this bar graph is too irregular. We can assume that our strategy is certainly under-fitted. The performance of this strategy can be explained more by the market trend than by itself. Thus when the market trend reverses, this strategy will become a loser.

Time in the market increasing

The variation of time in the market could reveal that a strategy is over-fitted or under-fitted. The trade duration should be stable from the beginning to the end of a backtest.

Here is an example of a time in the market increasing:

We can see that the time in the market increases at the end of the backtest. This strategy is increasing difficulties to reach its target price. In this case, the strategy is either under-fitted or becoming obsolete.

If the trade duration increases arise while a trading range or while weakening trend, then we can conclude that the strategy is under-fitted.

Else, it may well be that the strategy beginning obsolete thus the buy signal which gives is henceforth ignored by market operators.

Duration of latent loss

The duration of latent loss is the time while an opening position gives an unrealized loss. We frequently observe episodes of reflux after an entry on the market. In particular with a swing trading or intra-day strategy, and it is normal. However, one must be observant about losses and profits evolution.

The most important point is the variation of latent loss duration printed by your backtest. If the duration of latent loss increases, that would mean an over-fitted risk or that may reveal your strategy becoming obsolete. You should also monitor and compare the evolution of latent loss duration between backtesting and real trading.

If you observe that your entry becomes systematically a latent loss immediately after their opening, then maybe other traders think it is a better idea to go in reversal direction.

Here an example of a wining entry but with latent loss duration overly important:

This position gave latent loss nearly over the duration of the trading session. It would have been impossible to cut this position without losses. This entry was finally winning but it was a bad trade.

Summary of robustness

Learning of each concept that I described in this post cost me something. Please, be careful before launch an automated trading system on your real account. Take the time that you need to obtain sufficient robustness.

I hope you learn something from me and my experiences. If you are any questions, please ask me in a comment and if this article pleased you I would be grateful to see you share it.