After having evaluated the robustness of your strategy at first glance, your automated trading system must undergo a battery of stress tests in order to confirm its robustness.

In this post, I will show you how to test the robustness of an automated trading system using, **Prorealtime** and **Python**.

I will present the **more important stress tests** that you should make on Probacktest before launching your automated trading system on your real account. And I will also realize some Python simulations that would give you more visibility on the properties of your trading strategy.

**Table of contents**

**Spread increase simulations and tests****Trigger stop loss by a single point****Simulate the back to heads or tails**- The example of an asymmetrical strategy
- Back to heads or tails of an asymmetrical strategy
- The example of a symmetrical strategy
- Back to heads or tails of a symmetrical strategy
- Make your own simulations thanks to Python
- Example of successChanger function calling
- Conclusion about back to heads or tails test

**Series of losses and worst possible case****Flash crash simulation****Stress-tests summary**

…

## Spread increase simulations and tests

Simulate an increase in the spread is the first test that I do to verify the robustness of a strategy. It is the easiest and fastest way to know if an automated trading strategy is over-fitted.

This stress-test consists to gradually increase the spread cost invoiced at each entry until the performance of the strategy becomes null or a few losing. This is my interpretation of the spread increase result:

- If your strategy generates loss after few spread increasing, that means your strategy is certainly over-fitted.

- If your strategy continues to win money after several spread increases, that maybe indicate that your strategy is under-fitted.

### How to conduct this stress test ?

Usually, I divide the spread increasing test into two parts: first, I simulate the stress test thanks to a Python function, and next, I launch this stress test on Probacktest.

#### Python stress test simulation

I begin to simulate the consequences of the spread increase on my strategy. I use a Python function to proceed this test. This first simulation helps me to visualize on a graph the effect of spread increasing.

#### Probacktest stress test launching

Then I launch a backtest on the Prorealtime platform increasing spread iteratively. After each iteration, I analyze all entries which impacted by spread rising, in particular entries that became losing too quickly.

### Simulation of spread increasing

I will begin by simulating the consequences of spread increasing on a strategy from its properties. By “properties” I mean the started capital, the success rate, the average gain, and the average loss.

The Python function takes in parameters the properties of the strategy. In addition, the function integrates the spread and the number of entries in its parameters.

The function returns the result of the simulation in a matrix. Then the matrix translated into a PNG image.

#### Source Code of the backtest Python function

# import of libraries import random import matplotlib.pyplot as plt # instanciation of needed objects rnd = random.Random() # declaration of backtest def backtest(capital, successRate, gain, loss, spread, numberFlips): resultat = [] trade = 0 startupCapital = capital failureRate = 100 - successRate i = numberFlips while i > 0 and capital > 0: trade = rnd.choices([gain, loss], weights=(successRate, failureRate)) capital = capital + trade[0] - spread resultat.append(capital) if capital <= 0: break i-=1 plt.figure(figsize=(12,5)) plt.axhline(startupCapital, color="gray") plt.plot(resultat) plt.show() plt.close() # Example of backtest function calling backtest(10000, 55, 28, -25, 1, 1000)

Now, I will explain you how this function works, what are the used objects, its parameters and the simulation algorithm.

#### Libraries called by backtest function

The “backtest” function call two libraries : the **Random library** and the **Matplotlib library**.

**Random**

The Random library is necessary to generate random sequences. I use this library to create a sequence of wining and losing trades with a predefined probability.

In order to use this library, you need to instantiate a “Random” object by this way :

rnd = random.Random()

**Matplotlib**

The Matplotlib library is very famous and largely used in mathematical applications. We will use this library to create a chart from a **NumPy** matrix.

#### Parameters of backtest function

The following parameters will allow us to create a personalized simulation :

Parameter | Description |

capital | Your started capital |

successRate | Success rate in percent of your strategy |

gain | Average gain in points of wining entries |

loss | Average loss in points of losing entries |

spread | Invoiced spread for each opened entry |

numberFlips | Maximal number of entries that the function can open |

#### Algorithm of backtest function

The algorithm of the backtest function is pretty simple. The simulation continues while the capital is greater than zero.

WHILE CAPITAL > 0 DO SIMULATE AN ENTRY INTEGRATE THE RESULT IN THE CAPITAL END-WHILE

**Stages of the algorithm :**

- The treatment running while capital greater than zero
- A trade is simulated at each iteration
- This trade creates profits or losses
- The result is integrated into capital
- The value of “capital” variable is added into the “resultat” matrix

#### Explanation of source code

I will explain you the more important code lines of backtest function.

**Generate a random entry:**

The following code line will generate an aleatory trade thanks to the “choices” function belonging to the “Random” library. This function is called from a Random object named here “rnd”. This “rnd” object was created before we declared the backtest function.

trade = rnd.choices([gain,loss], weights=(successRate,failureRate))

This trade is generated with properties passed as parameters of the “backtest” function. The parameters are the average gain (gain), the average loss (loss), the success rate (successRate), and the failure rate (**failureRate**, deducted from success rate).

- If the trade is winning, then the variable “trade” will be equal to the average gain (
**gain**). - If the trade is losing, then the variable “trade” will be equal to the average loss (
**loss**).

**Recalculate the value of capital :**

For each iteration, the value of capital needs to be recalculated depending on the last simulated trade result (**trade[0]**). Of course, the spread (**spread**) must be deducted from this result.

capital = capital + trade[0] - spread

**Feed the “resultat” matrix :**

The new value of capital must be added into the “resultat” matrix. The “resultat” matrix containing the successive evolution of capital following each iteration of simulation.

resultat.append(capital)

**Displaying the result :**

At the end of processing, the chart created from the « plt » object and printed the matrix “resultat”. The next code line will create an empty chart with a width of 12 and height of 5 thanks to “figure” function:

plt.figure(figsize=(12,5))

Personally, I decided to draw a horizontal line thanks to the “axhline” function which corresponds to started capital for the purpose to visualize the drawdown.

plt.axhline(startupCapital, color="gray")

Finally, the graph plotted with the “plot” function and printed with the “show” function.

plt.plot(resultat) plt.show()

### Simulation running with Python

Now we can simulate the effect of spread increasing on a strategy. The properties of our strategy will be the following:

The started capital is 10000$, the success rate will be 55%, with an average gain of 28 points, an average loss of 25 points, and a spread of 1 point. The strategy will open and close 1000 entries.

**The “backtest” function needs to be called by the following way:**

backtest(*capital, success rate, average gain, average loss, spread, number of entries*)

#### Simulation of the strategy with a spread of 1 point

# Calling up the function backtest backtest(10000, 55, 28, -25,1, 1000)

**Result of simulation :**

This strategy is globally profitable and it gives a performance of 17.5% with 1000 entries. Let me show you now, what would happen if the spread rises to 2 points.

#### Simulation of the strategy with a spread of 2 points

# Calling up the function backtest backtest(10000, 55, 28, -25,2, 1000)

**Result of simulation :**

When the spread rises by 1 point, we can see an increase in the volatility. In addition, we observe that the performance of the strategy reduced to 7.5% thus a decrease of 50%.

#### Simulation of the strategy with a spread of 5 points

# Calling up the function backtest backtest(10000, 55, 28, -25,5, 1000)

**Result of simulation :**

This strategy does not seem to endure an increasing spread. The performance of the strategy falls rashly and becomes losing when the spread reaches 5 points. The reason is that the difference between stop loss and target profit is too few. The difference was 4 points, then the spread was 5 points.

### Simulation of spread increasing on Probacktest

From now on, we have an idea of the spread increase consequences on the performance of a strategy. We going to submit this stress test on our strategy thanks to Probacktest.

For that, we simply going to increase the invoiced spread on the Probacktest interface and relaunch the backtest as bellow:

We are going to increase the invoiced spread for each transaction until the performance of the strategy becomes neutral or a few losing.

### Over-fitted strategy behavior

I will begin to show you what is the reaction of an over-fitted strategy when the spread increases.

#### Strategy with a spread of 1 point

#### Strategy with a spread of 2 points

#### Strategy with a spread of 5 points

#### Interpretation of the result

Basically, over-fitted strategies are highly sensitive to spread increasing. They become rapidly losing. The very fact of increase the spread by 1 point on this strategy caused a fall down of the success rate at 55%. This strategy became losing when the spread reached one-third of the average gain.

### Under-fitted strategy behavior

#### Strategy with a spread of 1 point

#### Strategy with a spread of 2 points

#### Strategy with a spread of 5 points

#### Strategy with a spread of 20 points

#### Interpretation of the result

Under-fitted strategies are totally insensitive to the spread increasing. Their results are virtually not impacted. Even with an increase of 20 points, which corresponds to one-third of the average gain, the success rate is unchanged, and the strategy’s performance is still winning.

### Summary of spread increase simulations and tests

##### About simulation

The increase of the spread provokes a strong negative impact on the performance.

This result is easy to understand. If the spread becomes greater than the difference between average gain and average loss, the strategy can not continue to win. The spread was 5 points, then the difference between average gain (28 pts) and average loss (25 pts) was 4 points : (28 – 25 = 4).

However, in the reality, the variation of spread causes a variation of the success rate of an entry. That why we need to launch a backtest on the Prorealtime platform to try to observe these variations.

##### About the backtest

###### Over-fitted strategies

Over-fitted strategies are very

sensitives to the spread increasing. They become rapidly losing.

You could try to resolve this problem by widening your stop loss and your target. If the widening of stop loss and target provokes a fall down of the success rate, then maybe the problem comes from your entry points.

In addition, if you decided to widen stop loss and target, please be sure the ratio stop loss/target is unchanged.

###### Under-fitted strategies

Under-fitted strategies are few perhaps

insensitive to spread increasing. Their success rate will be a few impacted by the spread.

You could try to resolve this problem by tightening your stop loss and your target. As considering previously, if the tightening of your stop loss or your target causes a fall down of your success rate, then the problem might come from your entry points.

##### In the real world

In practice, the increase in spread seldom happens by chance. That happens while important events as central bank announcements, market opening or unexpected events.

I already noted that the spread has a tendency to rise when the price of my entry is near its stop loss. This increase is often low and short duration but sufficient to touch the stop loss and close the position.

Thus it is important to add a little gap at yours stop loss positioning ensuring to keep the same stop/target ratio.

## Trigger stop loss by a single point

It is a variant of the previous method which allows you to measure the effect of the stop loss hunting on your strategy. While a backtest when the price moves towards a stop loss by a single point, your position will be conserved. By contrast, you may rest assured that your stop loss will be triggered when this situation happens in the real world.

This test consists to voluntarily trigger all stop loss when the price is nearer to it. To perform this test, you have two options: either subtracting one point from the initial stop-loss or adding a conditional code line that closes an entry when the market price is too closely its stop loss.

**1. Subtract one point from initial stop loss :**

myStopLoss = 30 // your initial stop loss SET STOP pLOSS myStopLoss – 1 // subtract one point

**2. Trigger the closing of the entry when the market is too closely :**

Long position case :

IF low ≤ POSITIONPRICE - myStopLoss + 1 THEN SELL AT MARKET END-IF

Short position case :

IF high ≥ POSITIONPRICE + myStopLoss - 1 THEN SELLSHORT AT MARKET END-IF

These two alternative tests are important cause the spread increasing test by Probacktest interface has a weakness. When you increase the spread in this way, that provokes a tightening of the stop loss effectively and shifts the target. Thus that introduces a negative bias in the final result.

### Trigger stop loss summary

The worst way to optimize a strategy would be to optimize the stop loss and the target optimally, that is to say, by one point or less. You can be confident that your hyper winning strategy on the backtest will be hyper losing on your real account.

This test allows you to detect if the stop loss positioning is over-fitted by one point. It is sufficient to reduce positioning by one point and observe the success rate to be sure of that.

You should process the revert operation on the target increasing its position by one point.

In an automated trading system, when a little variation of the value of a variable provokes a fall down of the performance, that means it is an “orphan value”. I will take about “orphan value” in the next publication.

I will show you how to flush out the over-fitted variables as the stop loss, the target, or all other variables implicated in decision making. But before that, we will make an important simulation about the success rate of a strategy.

## Simulate the back to heads or tails

The back to heads or tails test, as well as the spread increase test is one of the most important of this chapter. This test consists to simulate the profitability of a strategy whether the success rate would drop down to 50%.

For a strategy that continues to be profitable with a success rate of 50%, the average gains must be strictly greater than the average losses. It is also important the biggest gain must be greater than the biggest loss.

A strategy that continues to be profitable despite a success rate of 50% guarantees the security of your capital. That allows you to test a new strategy with fewer overall risks. That also protects you against a strategy becoming obsolete.

### The example of an asymmetrical strategy

An asymmetrical strategy is a strategy for which the stop-loss position is far higher than the target position, and the success rate is high. Take the example of a strategy for which the stop-loss position is 30 points, the target position is 10 points, and the success rate is 85%. In this case, the stop loss is three times higher than the expected gains.

#### Hereafter the result of this asymmetrical strategy

On paper, this strategy appears very winning cause the printed performance is 28% of gains while 1000 entries.

If we zoom in the result of this strategy, it should be noted that there are some big downfalls of the performance when there are losses:

Now we are going to simulate consequences on the performance whether the success rate of this strategy would go down to 50%:

We can see that this strategy losses 10% of the started capital (10000$) after only a little under 50 entries.

### Back to heads or tails of an asymmetrical strategy

The most probable comportment of an over-fitted asymmetrical strategy at the time of its launching on the real account is the bellowing:

For this strategy, the stop loss position was 30 points, the target position was 10 points, and the success rate was 85%.

The 800 firsts entries represent the indicated performance while the backtest then the 200 following entries correspond to the real account result.

The crash force comes from the imbalance in the gap between the average losses and the average gains. Cause this strategy wins 10 when it losses 30.

### The example of a symmetrical strategy

This time, we are going to test a symmetric strategy. That is to say, the success rate is a little greater than 50%, and the average gain is slightly greater than the average loss. We will set 30 points for the target and 25 points for the stop loss with 1 point for the spread. The initial success rate of the strategy will be 65%, and the started capital will be 10000$.

#### Hereafter the result of this symmetrical strategy

This strategy seems very winning, the printed performance is 80% of gains while 1000 entries.

Now we are going to simulate consequences on the performance whether the success rate of this strategy would go down to 50%:

There is clearly more volatility but the strategy continues to be a little winning with a profit of 5% while 1000 entries.

### Back to heads or tails of a symmetrical strategy

The most probable comportment of an over-fitted symmetrical strategy at the time of its launching on the real account is the bellowing:

The 800 firsts entries represent the indicated performance while the backtest then the 200 following entries correspond to the real account result.

We can see that the strategy has difficulties continuing to give gains when the success rate is 50% but it is still winning.

This approach which consists to define an average gain greater than the average loss ensures more security to your capital. Personally, I chose this configuration to all my automated trading systems.

### Make your own simulations thanks to Python

I will show you the Python function that I used in the previous examples. I named this function “successChanger” and I use it to measure and visualize the consequences of the back to heads or tails test on the performance of a strategy. This function is a derivation of the “backtest” function, which I showed you in the “spread increase simulation” chapter.

#### successChanger function code source

def successChanger(capital, successRate1, successRate2, realSet, gain, loss, spread, numberFlips): resultat = [] startedCapital = capital successRate = 0 lossRate = 0 i = numberFlips while i > 0 and capital > 0: if i > (numberFlips * realSet): successRate = successRate1 lossRate = 100 - successRate1 else: successRate = successRate2 lossRate = 100 - successRate2 a = rnd.choices([gain, loss], weights=(successRate, lossRate)) capital = capital + a[0] - spread resultat.append(capital) if capital <= 0: break i-=1 plt.figure(figsize=(12,5)) plt.axhline(startedCapital, color="gray") plt.axvline(500, color="gray") plt.plot(resultat) plt.show() plt.close()

Call the function :

# Example of successChanger function calling successChanger(10000, 65, 50, 0.5, 30, -25, 1, 1000)

**Libraries** **called by backtest function**

The “successChanger” function uses the same libraries as the “backtest” function, namely Random and Matplotlib.

#### Algorithm of successChanger function

The algorithm of the successChanger function is similar to the backtest function. The main difference is the successChanger function divides the simulation into two parts: one backtest and one real. These two parts will have their own success rate.

Pseudo-code of the algorithm :

WHILE CAPITAL > 0 DO IF FIRST-PART THEN SIMULATE AN ENTRY(successRate_1) INTEGRATE THE RESULT IN THE CAPITAL ELSE SIMULATE AN ENTRY(successRate_2) INTEGRATE THE RESULT IN THE CAPITAL END-WHILE

Stages of the algorithm :

- The treatment running while capital greater than zero
- Verify if it is the first or the second part of the simulation
- Success rate assignment of the entry
- Entry simulation
- This entry generates a loss or a gain
- Integrate the result into the capital
- The capital variable is added into the “resultat” matrix

#### Parameters of successChanger function

The following parameters will allow us to create a personalized simulation:

Parameter | Description |

capital | Your started capital |

successRate1 | First part success rate in percent of the strategy |

successRate2 | Second part success rate in percent of the strategy |

realSet | Second part duration of the simulation (ex : 0.2=20%; 0.5=50 %) |

gain | Average gain in points of winning entries |

loss | Average loss in points of losing entries |

spread | Invoiced spread for each opened entry |

numberFlips | Maximal number of entries that the function can open |

### Example of successChanger function calling

This is a complete example of simulation thanks to the successChanger function:

successChanger(10000, 65, 50, 0.5, 30, -25, 1, 1000)

#### Parameter Analyses

Now I will describe each parameter passed in the previous function example:

Parameter | Description |

10000 | The started capital is 10000$ |

65 | The success rate in the first part is of 65% |

50 | The success rate in the first part is of 50% |

0.5 | The two parts have the same size (500 vs 500) |

30 | The average gain is 30 points |

-25 | The average loss is 25 points |

1 | The invoiced spread is 1 point |

1000 | 1000 entries will be simulated |

#### Result of the previous function calling

This simulation shows that the strategy gives 40% of profits while the success rate is 65%. And when the success rate is 50%, this same strategy continues to be a little wining with a profit of 2.5%.

### Conclusion about back to heads or tails test

The back to heads or tails test is very important cause the majority of trading strategies, frozen in time are intended to become obsolete. And when a trading strategy becomes obsolete, its success rate articulates around 50%.

A way to protect oneself against big loss consists to have the guaranty that the average gains are greater than the average losses.

To do so, the initialization of stop loss, target and begin to gain securing, should be defined by this way:

- The target should be two times greater than stop loss. For example, if the stop loss is of 30 points then the target should be positioned at 60 points.

- The beginning of gain securing should be strictly greater than stop loss. Concretely, that means your automated trading system must begin to secure gains when the latent gain is greater than the stop loss level with an addition of some points. For example, if the stop loss is 30 points, the system should begin to secure the gain at 35 points. (I arbitrarily defined 5 points in addition to the example)

The symmetrical trading strategy that I presented you previously corresponds to this example. This style of trading respects the tactical approach that I personally practice. That means I must always win a little more points than I am ready to lose.

Even today, sometime I put in production an automated trading system which promised me a very high success rate while backtesting but which finally wins one of the two calls on my real account.

Thanks to this approach, I can test news strategies with a minimal risk.

## Series of losses and worst possible case

### Estimate the series of losses cost

Even a good strategy at a time can undergo a series of losses. This series of losses does not have a too great impact on your trading account. A series of ten losses do not be superior to 20% of your capital. That is equivalent to 2% of risk by entry. It is classical in money management.

Please be careful. When I take about a series of losses, I am referring to the biggest loss that your automated trading system can do at a time. That means that if your system shows the greatest loss is 20 points, and the stop loss is positioned at 50 points. You should consider 50 points as the biggest possible loss in your calculation.

It is important to do that cause if your strategy is under-fitted, it is possible that the stop loss is never reached while the backtest launched on a too favorable part of the market. In this case, the printed average loss is invalid.

### Worst possible case simulation

Simulate the worst possible case consists to compute how many consecutive losing entries needed to lose the whole capital. As before, we consider the stop loss and not the average loss in this calculation.

This test consists to imagine what could happen if the stop loss was immediately reached after the opening of an entry.

In the goal of money management rules respect, an automated trading system must undergo a minimum of 50 losses before losing all the capital. That also corresponds to 2% of risk by entry.

Know the max number of losses provoking a total loss of capital is very important. This number will be determinant for the future when you becoming an automated trading system manager.

## Flash crash simulation

I think the flash crash is the event that I fear the most as a trader. Have an opened entry while a flash crash is the worst thing that can happen. The majority of traders are not ready for it. That relates to the previous idea about risk exposure and money management.

An automated trading system must absolutely survive to a fall in the market of 20% would occur under one minute. Thus your automated trading system should never exceed a risk exposition of 500%, which corresponds to leverage below or equal to 5.

#### USD/CHF Flash crash of 2015/01/15

The following flash crash occurred the 15 January 2015 after the Swiss National Bank re-evaluated the CHF 30% higher against the EUR.

However, it is recommended that an automated trading system working like a day trader should never exceed 300% of marker exposure, and an automated trading system working like a swing trader should never exceed 100% of market exposure.

## Stress-tests summary

The following is an overview of the stress tests chapter summarizing the main ideas to withhold that you could apply to your automated trading system :

- An automated trading system should
**accept a spread increase**until ¼ of the average gain.

- The distance between the stop loss and the low point after the opening entry should always
**strictly greater than 1 point**.

- The average gain should be always greater than the average loss in the goal to
**continue to be a winner**in the case of the success rate down to 50%.

- The
**market exposure should be reasonable**, that is to say, it should be lower than 300% for day trading strategy and lower than 100% for a swing trading strategy.

- The
**gain/loss ratio should be greater than 50%**while backtesting, which means your automated trading system should win at least a little more often than one time in two.

- A series of 10 losses should not represent a loss of
**20% of your capital**.

- You should
**know how many losses provokes a total capital loss**for each automated trading system running on your real account.

- An automated trading system must
**tolerate a fall in the market of 20%**would occur under one minute without causing a total loss of your capital.

In the next post, I will present to you a formidable stress-test calling “**Stochastic modeling**“. This test will give you the possibility to detect if your strategy is over-fitted with a high success rate.

### For further information

**Stress Testing, Investopedia**

https://www.investopedia.com/terms/s/stresstesting.asp

**Stress Testing for Trading Strategy Robustness by Michael R. Bryant**

http://www.adaptrade.com/Newsletter/NL-StressTesting.htm

**CrashMetrics**

http://www.paulwilmott.com/crashm.htm

**Download for free the “SEVEN PILLARS TO BUILD A WINNER PROREALTIME BOT”**

The “SEVEN PILLARS TO BUILD A WINNER PROREALTIME BOT” is a PDF containing **63 pages** allowing you to **improve the success** of your automated trading system.

Great article!! (And the previous about robustness too)

We do this kind of tests manually. In fact never tried to connect by API for now (then we can’t combine the spread change with the BT). Will work on it.