Artificall

After having evaluated the robustness of your strategy at first glance, your automated trading system must undergo a battery of stress tests in order to confirm its robustness.

In this post, I will show you how to test the robustness of an automated trading system using, Prorealtime and Python.

I will present to you the more important stress tests that you should make on Probacktest before launching your automated trading system on your real account. And I will also realize some Python simulations that would give you more visibility on the properties of your trading strategy.

Spread increase simulations and tests

Simulate an increase in spread is the first test that I do to verify the robustness of a strategy. It is the easiest and fastest way to know if an automated trading strategy is over-fitted.

This stress-test consists to gradually increase the spread cost invoiced at each entry until the performance of the strategy becomes null or a few losing. This is my interpretation of the spread increase result :

  • If your strategy generates loss after few spread increasing, that means your strategy is certainly over-fitted.
  • If your strategy continues to win money after several spread increases, that maybe indicate that your strategy is under-fitted.

How to conduct this stress test ?

Usually, I divide the spread increasing test into two parts: first, I simulate the stress test thanks to a Python function, and next, I launch this stress test on Probacktest.

Python stress test simulation

I begin to simulate the consequences of the spread increase on my strategy. I use a Python function to proceed this test. This first simulation helps me to visualize on a graph the effect of spread increasing.

Probacktest stress test launching

Then I launch a backtest on the Prorealtime platform increasing spread iteratively. After each iteration, I analyze all entries which impacted by spread rising, in particular entries that became losing too quickly.

Simulation of spread increasing

I will begin by simulating the consequences of spread increasing on a strategy from its properties. By “properties” I mean the started capital, the success rate, the average gain, and the average loss.

The Python function takes in parameters the properties of the strategy. In addition, the function integrates the spread and the number of entries in its parameters.

The function returns the result of the simulation in a matrix. Then the matrix translated into a PNG image.

Source Code of the backtest Python function

# import of libraries
import random
import matplotlib.pyplot as plt

# instanciation of needed objects
rnd = random.Random()

# declaration of  backtest
def backtest(capital, successRate, gain, loss, spread, numberFlips):
	resultat = []
	trade = 0
	startupCapital = capital
	failureRate = 100 - successRate
	i = numberFlips

	while i > 0 and capital > 0:
		trade = rnd.choices([gain, loss], weights=(successRate, failureRate))
		capital = capital + trade[0] - spread
		resultat.append(capital)
		if capital <= 0:
			break
		i-=1

	plt.figure(figsize=(12,5)) 
	plt.axhline(startupCapital, color="gray")
	plt.plot(resultat)
	plt.show()
	plt.close()


# Example of backtest function calling
backtest(10000, 55, 28, -25, 1, 1000)

Now, I will explain you how this function works, what are the used objects, its parameters and the simulation algorithm.

Libraries called by backtest function

The “backtest” function call two libraries : the Random library and the Matplotlib library.

Random

The Random library is necessary to generate random sequences. I use this library to create a sequence of wining and losing trades with a predefined probability.

In order to use this library, you need to instantiate a “Random” object by this way :

rnd = random.Random()

Matplotlib

The Matplotlib library is very famous and largely used in mathematical applications. We will use this library to create a chart from a NumPy matrix.

Parameters of backtest function

The following parameters will allow us to create a personalized simulation :

ParameterDescription
capitalYour started capital
successRateSuccess rate in percent of your strategy
gainAverage gain in points of wining entries
lossAverage loss in points of losing entries
spreadInvoiced spread for each opened entry
numberFlipsMaximal number of entries that the function can open
Parameters of backtest Python function

Algorithm of backtest function

The algorithm of the backtest function is pretty simple. The simulation continues while the capital is greater than zero.

WHILE CAPITAL > 0 DO
    SIMULATE AN ENTRY
    INTEGRATE THE RESULT IN THE CAPITAL
END-WHILE

Stages of the algorithm :

  • The treatment running while capital greater than zero
  • A trade is simulated at each iteration
  • This trade creates profits or losses
  • The result is integrated into capital
  • The value of “capital” variable is added into the “resultat” matrix

Explanation of source code

I will explain you the more important code lines of backtest function.

Generate a random entry:

The following code line will generate an aleatory trade thanks to the “choices” function belonging to the “Random” library. This function called from a Random object named here “rnd”. This “rnd” object was created before we declared the backtest function.

trade = rnd.choices([gain, loss], weights=(successRate, failureRate))

This trade generated with properties passed as parameters of the “backtest” function. The parameters are the average gain (gain), the average loss (loss), the success rate (successRate), and the failure rate (failureRate, deducted from success rate).

  • If the trade is winning, then the variable “trade” will be equal to average gain (gain).
  • If the trade is losing, then the variable “trade” will be equal to average loss (loss).

Recalculate the value of capital :

For each iteration, the value of capital needs to be recalculated depending on the last simulated trade result (trade[0]). Of course, the spread (spread) must be deducted from this result.

capital = capital + trade[0] - spread

Feed the “resultat” matrix :

The new value of capital must be added into the “resultat” matrix. The “resultat” matrix containing the successive evolution of capital following each iteration of simulation.

resultat.append(capital)

Displaying the result :

At the end of processing, the chart created from the « plt » object and printed the matrix “resultat”. The next code line will create an empty chart with a width of 12 and height of 5 thanks to “figure” function :

plt.figure(figsize=(12,5))

Personally, I decided to draw a horizontal line thanks to the “axhline” function which corresponds to started capital for the purpose to visualize the drawdown.

plt.axhline(startupCapital, color="gray")

Finally, the graph plotted with the “plot” function and printed with the “show” function.

plt.plot(resultat)
plt.show()

Simulation running with Python

Now we can simulate the effect of spread increasing on a strategy. The properties of our strategy will be the following :

The started capital is 10000$, the success rate will be 55%, with an average gain of 28 points, an average loss of 25 points, and a spread of 1 point. The strategy will open and close 1000 entries.

The “backtest” function needs to be called by the following way :

backtest(capital, success rate, average gain, average loss, spread, number of entries)

Simulation of the strategy with a spread of 1 point

# Calling up the function backtest
backtest(10000, 55, 28, -25, 1, 1000)

Result of simulation :

Simulation of spread increase
Simulation of spread increase, 1 point

This strategy is globally profitable and it gives a performance of 17.5% with 1000 entries. Let me show you now, what would happen if the spread rises to 2 points.

Simulation of the strategy with a spread of 2 points

# Calling up the function backtest
backtest(10000, 55, 28, -25, 2, 1000)

Result of simulation :

Simulation of spread increase
Simulation of a strategy with a spread of 2 points

When the spread rises by 1 point, we can see an increase in the volatility. In addition, we observe that the performance of the strategy reduced to 7.5% thus a decrease of 50%.

Simulation of the strategy with a spread of 5 points

# Calling up the function backtest
backtest(10000, 55, 28, -25, 5, 1000)

Result of simulation :

Simulation of spread increase
Simulation of a strategy with a spread of 5 points

This strategy does not seem to endure an increasing spread. The performance of the strategy falls down rashly and become losing when the spread reaches 5 points. The reason is that the difference between stop loss and target profit is too few. The difference was 4 points then the spread was 5 points.

Simulation of spread increasing on Probacktest

From now on, we have an idea of the spread increase consequences on the performance of a strategy. We going to submit this stress test on our strategy thanks to Probacktest.

For that, we simply going to increase the invoiced spread on the Probacktest interface and relaunch the backtest as bellow :

Probacktest spread 2 points
Probacktest spread 2 points

We are going to increase the invoiced spread for each transaction until the performance of the strategy becomes neutral or a few losing.

Over-fitted strategy behavior

I will begin to show you what is the reaction of an over-fitted strategy when the spread increases.

Strategy with a spread of 1 point

Backtest over-fitted strategy
Spread increase test on an over-fitted strategy

Strategy with a spread of 2 points

Spread increase test on an over-fitted strategy

Strategy with a spread of 5 points

Spread increase test on an over-fitted strategy

Interpretation of the result

Basically, over-fitted strategies are highly sensitive to spread increasing. They become rapidly losing. The very fact of increase the spread by 1 point on this strategy caused a fall down of the success rate at 55%. This strategy became losing when the spread reached one-third of the average gain.

Under-fitted strategy behavior

Strategy with a spread of 1 point

Spread increase test on an under-fitted strategy

Strategy with a spread of 2 points

Spread increase test on an under-fitted strategy

Strategy with a spread of 5 points

Spread increase test on an under-fitted strategy

Strategy with a spread of 20 points

Spread increase test on an under-fitted strategy

Interpretation of the result

Under-fitted strategies are totally insensitive to the spread increasing. Their results are virtually not impacted. Even with an increase of 20 points, which correspond to one-third of average gain, the success rate is unchanged and the performance of the strategy is still winning.

Summary of spread increase simulations and tests

About simulation

The increase of the spread provokes a strong negative impact on the performance.

This result is easy to understand. If the spread becomes greater than the difference between average gain and average loss, the strategy can not continue to win. The spread was 5 points than the difference between average gain (28 pts) and average loss (25 pts) was 4 points : (28 – 25 = 4).

However, in the reality, the variation of spread causes a variation of the success rate of an entry. That why we need to launch a backtest on the Prorealtime platform to try to observe these variations.

About backtest

Over-fitted strategies

Over-fitted strategies are very sensitives to the spread increasing. They become rapidly losing.

You could try to resolve this problem by widening your stop loss and your target provided that the success rate of your strategy does not too impacted. If the widening of stop loss and target provokes fall down of the success rate, then maybe that the problem comes from your entry points.

In addition, if you decided to widen stop loss and target, please be sure the ratio stop loss/target is unchanged.

Under-fitted strategies

Under-fitted strategies are few perhaps insensitive to spread increasing. Their success rate will be a few impacted by the spread.

You could try to resolve this problem by tightening your stop loss and your target. As considering previously, if the tightening of your stop loss or your target causes a fall down of your success rate then the problem might come from your entry points.

In the real world

In practice, the increase in spread seldom happens by chance. That happens while important events as central bank announcements, market opening or unexpected events.

I already noted that the spread has a tendency to rise when the price of my entry is near its stop loss. This increase is often low and short duration but sufficient to touch the stop loss and close the position.

Thus it is important to add a little gap at yours stop loss positioning ensuring to keep the same stop/target ratio.

Trigger stop loss by a single point

It is a variant of the previous method which allows you to measure the effect of the stop loss hunting on your strategy. While a backtest when the price moves towards a stop loss by a single point your position will be conserved. By contrast, you may rest assured that when this situation happens in the real world, your stop loss will be triggered.

This test consists to voluntarily trigger all stop loss when the price is nearer to it. To perform this test, you have two options : either you subtract one point from the initial stop loss or you add a conditional code line that closes an entry when the market price is too closely its stop loss.

1. Subtract one point from initial stop loss :

myStopLoss = 30 // your initial stop loss
SET STOP pLOSS myStopLoss – 1 // subtract one point

2. Trigger the closing of the entry when the market is too closely :

Long position case :

IF low ≤ POSITIONPRICE - myStopLoss + 1 THEN
    SELL AT MARKET
END-IF

Short position case :

IF high ≥ POSITIONPRICE + myStopLoss - 1 THEN
    SELLSHORT AT MARKET
END-IF

These two alternative tests are important cause the spread increasing test by Probacktest interface has a weakness : when you increase the spread in this way, that provokes effectively a tightening of the stop loss but that also shifts the target. Thus that introduces a negative bias in the final result.

Trigger stop loss summary

The worst way to optimize a strategy would consist to optimize in an optimal manner the stop loss and the target, that is to say by one point or less. You can be confident that your hyper winning strategy on the backtest will be hyper losing on your real account.

This test allows you to detect if the stop loss positioning is over-fitted by one point. It is sufficient to reduce positioning by one point and observe the success rate to be sure of that.

You should process the revert operation on the target increasing its position by one point.

In an automated trading system, when a little variation of the value of a variable provokes a fall down of the performance, that means it is an “orphan value”. I will take about “orphan value” in the next publication.

I will show you how to flush out the over-fitted variables as the stop loss, the target, or all other variables implicated in decision making. But before that, we will make an important simulation about the success rate of a strategy.

Simulate the back to heads or tails

The back to heads or tails test, as well as the spread increase test is one of the most important of this chapter. This test consists to simulate the profitability of a strategy whether the success rate would drop down to 50%.

For a strategy that continues to be profitable with a success rate of 50%, the average gains must be strictly greater than the average losses. It is also important the biggest gain must be greater than the biggest loss.

A strategy that continues to be profitable despite a success rate of 50% guarantees the security of your capital. That gives you the opportunity to test a new strategy with less overall risks. That also protects you against a strategy becoming obsolete.

The example of an asymmetrical strategy

An asymmetrical strategy is a strategy for which the stop loss position is far higher than the target position and the success rate is high. Take the example of a strategy for which the stop loss position is 30 points than the target position is 10 points, and the success rate is 85%. In this case, the stop loss is three times higher than the expected gains.

Hereafter the result of this asymmetrical strategy

Asymmetrical strategy
Asymmetrical strategy

On paper, this strategy appears very winning cause the printed performance is 28% of gains while 1000 entries.

If we zoom in the result of this strategy, it should be noted that there are some big downfalls of the performance when there are losses :

Asymmetrical strategy zoom
Asymmetrical strategy zoom

Now we are going to simulate consequences on the performance whether the success rate of this strategy would go down to 50% :

Asymmetrical strategy crash
Asymmetrical strategy crash

We can see that this strategy losses 10% of the started capital (10000$) after only a little under 50 entries.

Back to heads or tails of an asymmetrical strategy

The most probable comportment of an over-fitted asymmetrical strategy at the time of its launching on the real account is the bellowing :

Back to heads or tails of an asymmetrical strategy
Back to heads or tails of an asymmetrical strategy

For this strategy, the stop loss position was 30 points, the target position was 10 points, and the success rate was 85%.

The 800 firsts entries represent the indicated performance while the backtest then the 200 following entries correspond to the real account result.

The crash force comes from the imbalance in the gap between the average losses and the average gains. Cause this strategy wins 10 when it losses 30.

The example of a symmetrical strategy

This time, we going to test a symmetric strategy, that is to say, the success rate is a few greater than 50% and the average gain is lightly greater than the average loss. We will set 30 points for the target and 25 points for the stop loss with 1 point for the spread. The initial success rate of the strategy will be 65% and the started capital will be 10000$.

Hereafter the result of this symmetrical strategy

Symmetrical strategy
Symmetrical strategy

This strategy seems very winning, the printed performance is 80% of gains while 1000 entries.

Now we are going to simulate consequences on the performance whether the success rate of this strategy would go down to 50% :

Back to head or tails of a symmetrical strategy
Back to head or tails of a symmetrical strategy

There is clearly more volatility but the strategy continues to be a little winning with a profit of 5% while 1000 entries.

Back to heads or tails of a symmetrical strategy

The most probable comportment of an over-fitted symmetrical strategy at the time of its launching on the real account is the bellowing :

Back to head or tails of a symmetrical strategy
Back to head or tails of a symmetrical strategy

The 800 firsts entries represent the indicated performance while the backtest then the 200 following entries correspond to the real account result.

We can see that the strategy has difficulties continuing to give gains when the success rate is 50% but it is still winning.

This approach which consists to define an average gain greater than the average loss ensures more security to your capital. Personally, I chose this configuration to all my automated trading systems.

Make your own simulations thanks to Python

I will show you the Python function that I used in the previous examples. I named this function “successChanger” and I use it to measure and visualize the consequences of the back to heads or tails test on the performance of a strategy. This function is a derivation of the “backtest” function which I showed you in the “spread increase simulation” chapter.

successChanger function code source

def successChanger(capital, successRate1, successRate2, realSet, gain, loss, spread, numberFlips):
	resultat = []
	startedCapital = capital
	successRate = 0
	lossRate = 0

	i = numberFlips
	while i > 0 and capital > 0:
		if i > (numberFlips * realSet):
			successRate = successRate1
			lossRate = 100 - successRate1
		else:
			successRate = successRate2
			lossRate = 100 - successRate2

		a = rnd.choices([gain, loss], weights=(successRate, lossRate))

		capital = capital + a[0] - spread
		resultat.append(capital)
		if capital <= 0:
			break
		i-=1
		
	plt.figure(figsize=(12,5)) 
	plt.axhline(startedCapital, color="gray")
	plt.axvline(500, color="gray")

	plt.plot(resultat)
	plt.show()
	plt.close()
	

Call the function :

# Example of successChanger function calling
successChanger(10000, 65, 50, 0.5, 30, -25, 1, 1000)

Libraries called by backtest function

The “successChanger” function uses the same libraries as the “backtest” function, namely Random and Matplotlib.

Algorithm of successChanger function

The algorithm of the successChanger function is similar to the backtest function. The main difference is the successChanger function divides the simulation into two parts : one backtest part and one real part. These two parts will have its own success rate.

Pseudo-code of the algorithm :

WHILE CAPITAL > 0 DO
    IF FIRST-PART THEN
        SIMULATE AN ENTRY(successRate_1)
        INTEGRATE THE RESULT IN THE CAPITAL
    ELSE
        SIMULATE AN ENTRY(successRate_2)
        INTEGRATE THE RESULT IN THE CAPITAL
END-WHILE

Stages of the algorithm :

  • The treatment running while capital greater than zero
  • Verify if it is the first or the second part of the simulation
  • Success rate assignment of the entry
  • Entry simulation
  • This entry generates a loss or a gain
  • Integrate the result into the capital
  • The capital variable is added into .the “resultat” matrix

Parameters of successChanger function

The following parameters will allow us to create a personalized simulation :

ParameterDescription
capitalYour started capital
successRate1First part success rate in percent of the strategy
successRate2Second part success rate in percent of the strategy
realSetSecond part duration of the simulation (ex : 0.2=20%; 0.5=50 %)
gainAverage gain in points of winning entries
lossAverage loss in points of losing entries
spreadInvoiced spread for each opened entry
numberFlipsMaximal number of entries that the function can open

Example of successChanger function calling

This is a complete example of simulation thanks to the successChanger function

successChanger(10000, 65, 50, 0.5, 30, -25, 1, 1000)

Parameter Analyses

Now I will describe each parameter passed in the previous function example :

ParameterDescription
10000The started capital is 10000$
65The success rate in the first part is of 65%
50The success rate in the first part is of 50%
0.5The two parts have the same size (500 vs 500)
30The average gain is 30 points
-25The average loss is 25 points
1The invoiced spread is 1 point
10001000 entries will be simulated

Result of the previous function calling

Example of successChanger function calling
Example of successChanger function calling

This simulation shows that the strategy gives 40% of profits while the success rate is 65%. And when the success rate is 50%, this same strategy continues to be a little wining with a profit of 2.5%.

Conclusion about back to heads or tails test

The back to heads or tails test is very important cause the majority of trading strategies, frozen in time are intended to become obsolete. And when a trading strategy becomes obsolete, its success rate articulates around 50%.

A way to protect oneself against big loss consists to have the guaranty that the average gains are greater than the average losses.

To do so, the initialization of stop loss, target and begin to gain securing, should be defined by this way :

  • The target should be two times greater than stop loss. For example, if the stop loss is of 30 points then the target should be positioned at 60 points.
  • The beginning of gain securing should be strictly greater than stop loss. Concretely, that means your automated trading system must begin to secure gains when the latent gain is greater than the stop loss level with an addition of some points. For example, if the stop loss is of 30 points, the system should begin to secure gain at 35 points. (I arbitrarily defined 5 points in addition for the example)

The symmetrical trading strategy that I presented you previously corresponds to this example. This style of trading respects the tactical approach that I personally practice. That means I must always win a little more points than I am ready to lose.

Even today, sometime I put in production an automated trading system which promised me a very high success rate while backtesting but which finally wins one of the two calls on my real account.

Thanks to this approach, I can test news strategies with a minimal risk.

Series of losses and worst possible case

Estimate the series of losses cost

Even a good strategy at a time can undergo a series of losses. These series of losses does not have a too great impact on your trading account. A series of ten losses does not be superior to 20% of your capital. That is equivalent to 2% of risk by entry, it is classical in money management.

Please be careful, when I take about a series of losses, I am referring to the biggest loss that your automated trading system can do at a time. That means that if your system shows the greatest loss is 20 points then the stop loss was positioned at 50 points, you should consider 50 points as the biggest possible loss in your calculation.

It is important to do that cause if your strategy is under-fitted, it is possible that the stop loss is never reached while the backtest launched on a too favorable part of the market. In this case, the printed average loss is invalid.

Worst possible case simulation

Simulate the worst possible case consists to compute how many consecutive losing entries needed to lose the whole capital. As before, we consider the stop loss and not the average loss in this calculation.

This test consists to imagine what could happen if the stop loss was immediately reached after the opening of an entry.

In the goal of money management rules respect, an automated trading system must undergo a minimum of 50 losses before losing all the capital. That also corresponds to 2% of risk by entry.

Know the max number of losses provoking a total loss of capital is very important. This number will be determinant for the future when you becoming an automated trading system manager.

Flash crack simulation

I think the flash crack is the event that I fear the most as a trader. Have an opened entry while a flash crack is the worst thing that can happen. The majority of traders are not ready for it. That relates to the previous idea about risk exposure and money management.

An automated trading system must absolutely survive to a fall in the market of 20% would occur under one minute. Thus your automated trading system should never exceed a risk exposition of 500%, which corresponds to leverage below or equal to 5.

USD/CHF Flash crack of 2015/01/15

The following flash crack occurred the 15 January 2015 after the Swiss National Bank re-evaluated the CHF 30% higher against the EUR.

flash crack USD CHF 2015/01/15
flash crack USD CHF 2015/01/15

However, it is recommended that an automated trading system working like a day trader should never exceed 300% of marker exposure and an automated trading system working like a swing trader should never exceed 100% of market exposure.

Stress-tests summary

The following is an overview of the stress tests chapter summarizing the main ideas to withhold that you could apply to your automated trading system :

  1. An automated trading system should accept a spread increase until ¼ of the average gain.
  1. The distance between the stop loss and the low point after the opening entry should always strictly greater than 1 point.
  1. The average gain should be always greater than the average loss in the goal to continue to be a winner in the case of the success rate down to 50%
  1. The market exposure should be reasonable, that is to say, it should be lower than 300% for day trading strategy and lower than 100% for a swing trading strategy.
  1. The gain/loss ratio should be greater than 50% while backtesting, which means your automated trading system should win at least a little more often than one time in two.
  1. A series of 10 losses should not represent a loss of 20% of your capital.
  1. You should know how many losses provokes a total capital loss for each automated trading system running on your real account.
  1. An automated trading system must tolerate a fall in the market of 20% would occur under one minute without causing a total loss of your capital.

Subscribe to our newsletter!


Leave a Reply

Your email address will not be published. Required fields are marked *