This is the third and last post about the evaluation of automated trading robustness. The first was about evaluating a trading strategy at first glance, the second about the stress tests, and now, I will take about the Stochastic modeling.
I was initially foreseen to present to you the stochastic modeling in my last post about robustness but finally, I preferred to present this test in a separate post to facilitate your reading.
- What about Stochastic modeling
- Application of the test on Prorealtime
- Unitary test of stochastic modeling
- Global Stochastic modeling test
- Generalization of the stochastic modeling
- The limitations of the stochastic modeling
- Stochastic modeling summary
What about Stochastic modeling?
Definition of the stochastic test
The test of the stochastic is very important in robotic and in industry. What is more, when the robot has artificial intelligence like an autonomous car which will have some interaction with human beings.
The stochastic test simply consists to apply little variations on the environmental properties in which the robot will operate. These variations could also be applied to the internal properties of the system.
Purpose of the stochastic test
The stochastic test tries to measure the degree of flexibility of an autonomous system. It is a kind of stress test used to validate the robustness of a system. We want to be sure that the robot will continue to be efficient even if there are some changes in the environment.
To validate the stochastic modeling test, the following condition must be completed :
Application of the test on Prorealtime
Back to the spread increase test
The only environmental property that we can modify on the Prorealtime platform is the spread.
To validate the test, a little variation of the spread shall not entail a big variation of the performance of the automated trading system.
How to process stochastic modeling on Prorealtime?
It is impossible to apply variations on the quotation historical of an asset with Prorealtime apart from the spread. However, it is perfectly possible to launch stochastic modeling only on the internal properties of a system. Thus, we will apply little variations on the values of the variables implicated in the decision-making as the opening, the management, and the closing of an entry.
For example, if your automatic trading system opens an entry when the relative strength index is greater than 30, then the stochastic modeling will consist of replacing 30 with 29 and 31. For the stochastic modeling to be valid, these two values do not provoke a significant change in the performance of the automated trading system.
Validation requirements of the test
There are two steps to validate stochastic modeling. In first, we will apply a little variation on each variable as the previous example about the RSI. This is the unitary test of stochastic modeling.
And secondly, we will test all the combinations of little variations on all the variables in the same sample. This is the global test of stochastic modeling.
What we are trying to find?
We trying to verify that the automated trading system is slightly under-optimized. That means that the system has a sufficient degree of liberty to accept a change in the environment into it will operate in the future.
Unitary test of stochastic modeling
When you launch the optimizer of the variables, it sorts the result from the largest to the smallest gain. And it will naturally choose the value which gives the best performance. It is very tempting to choose this value, but it is often an over-optimized value.
The following chart represents the gain curve and you can see that the best value is a kind of peak :
The value of the variable represented by a red square on the chart is ‘102’ and it generates a gain of 600€. However, its previous value (101 on the chart) would have given a gain of 500€.
That means with only a little variation of 1 point, the total performance of the strategy falls to 20%. The unitary stochastic modeling will help us to avoid this risk on our strategy.
The unitary stochastic modeling procedure
We will apply a little variation on the value of each variable one by one. And a little variation of a variable shall not provoke a large variation of the performance.
Unitary test running
To perform this test, you will need to open the variable optimizer on Prorealtime and add the variable you want to test. But this time, we will define a range of values just around the variable’s initial value. This range will begin from the first previous value to the first next value of the initial value. It is preferable to define an interval so that only are three possible combinations of values. Thus there only are three steps with the previous value, the initial value, and the next value.
In the following example, I want to verify if the period of the relative strength index is over-fitted or not. The initial period is 14 thus, the first value will be 13, the initial value will be 14 and the last value will be 15 :
Interpretation of the result
The initial value of the RSI, here 14 periods, would not have passed the unitary stochastic modeling test. Because its previous and next value provoke a loss on the performance of the system:
Stochastic modeling helps us to know if a chosen value is surrounded by losing values. Now, you will have to process this test on all the variables implicated in the decision-making. This is laborious work, but it is important because neither the Gradient Descent algorithm nor the Walk-Forward optimization can detect this particularity.
I advise you to duplicate your system to create a specific version dedicated to stochastic modeling.
Global Stochastic modeling test
Global Stochastic modeling test goal
The global stochastic modeling test has a different goal than the unitary test. Its main purpose consists of testing the global degree of liberty of the whole automated trading system. If you want that this test makes sense, you have to make sure that all the values of your system have passed the unitary stochastic modeling.
To perform this test, you have to apply little variations on all the variables of the system implicated in the decision making thanks to the optimizer of the variables.
That could be reasonable to run several tests only on a specific part of your system. For example, you could run a global test on all the variables implicated in the entry opening and another on all the variables implicated in closing open entries. Because you can consider that open an entry and close an entry are two distinct activities.
Global test running
For this example, I will launch a test only on three variables to facilitate reading the result. To run a test, I have to open the optimizer of the variables as with the previous test. After that, I added three volatility variables that are based on different periods. The variables are « ecarTypeSmallMax », « ecarTypeMiddleMax » and « ecarTypeLargeMax » :
Initial values of the variables
The initial values of these three variables are the following :
- ecarTypeSmallMax : 7
- ecarTypeMiddleMax : 100
- ecarTypeLargeMax : 130
These are all the combinations that will be tested by the global test :
|Variable||Value -1||Initial value||Value +1|
Result of the global stochastic modeling test
These the table containing the result of all the combinations created by the optimizer of the variables:
Interpretation of the result
You can see that all the combinations are winning. Thus you can consider that the test is validated. To be validated, the test needs that the majority of the combination gives a positive performance. I think it would be good if more than 80% of the combinations are winning.
If the majority of the combination are losing, that indicates that your system is over-optimized. The risk is that your automated trading system will not tolerate a change in the market in the future. However, you probably have remarked that some combinations perform significantly lower than the initial result (3.31% against 7.36%).
It would be better to consider that the test is validated only if most of the combination gives a performance that is not significantly different from the initial performance. In this case, there are only three combinations that are not passed the test.
Generalization of the stochastic modeling
To facilitate your understanding of stochastic modeling, I told you that a little variation of the value of a variable implicated in the decision-making must not provoke a large variation of the system’s performance.
In reality, the performance of the system is not the alone measurable metric. It is recommended to analyze all the properties of the system as drawdown, the volatility of performance, time in the market, etc.
Thus we can generalize the stochastic modeling by the below way :
The limitations of the stochastic modeling
Each mathematical and machine learning model has limitations, and stochastic modeling does not escape the rule. The main limitation often comes from the decision threshold. A decision threshold is a value from which we consider that the test is succeeded or failed. Stochastic modeling needs several metrics that could provoke decision-making problems. The most important thresholds are included in the definition of the model.
The first comes from the definition of what is a “little variation“. This definition is too approximative to be directly used in a decision-making process. A « little variation » could also be 0.1% than 5%. To resolve this problem, I often chose the next lowermost unit variation from the initial value.
For example, I will choose 1 point of variation for a value between 10 and 100, 10 points for a value between 100 and 1000, etc. I am sorry cause I do not have a consistent model to show you for this weakness.
You will have the same difficulty defining what a “large variation is”. You will have to define an adapted acceptance threshold for your strategy.
The third limitation is about the percentage of combinations having a result significantly different against the initial result, also for the same reasons.
This threshold problem concerns most mathematical and machine learning models that make it difficult to use these models in reality. These models can only provide aid to decision-making, and you should ensure that the obtained result is exploitable.
Stochastic modeling summary
- This is a stress test allowing to validate the robustness of a system.
- This test can detect if a strategy is partially or totally over-optimized.
- The test verifies that the system has a sufficient degree of liberty to accept a change in the environment.
- This test can be generalized to all the properties of a system.
- The validity of the result depends on the chosen decision-making thresholds.
Resources about Stochastic modeling
You will find out some other explanations of stochastic modeling by following links :
Corporate Finance Institute
NivedRajeev, CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0, via Wikimedia Commons
Download for free the “SEVEN PILLARS TO BUILD A WINNER PROREALTIME BOT”
The “SEVEN PILLARS TO BUILD A WINNER PROREALTIME BOT” is a PDF containing 63 pages allowing you to improve the success of your automated trading system.