When you backtest an expert advisor in strategy testing, the broker calls the data from its servers. This part is a little bit tricky. What most people are not aware of is that brokers store sort of ‘shaved data’. That is, brokers do not store 100% of live tick data flow in their servers. As every broker has some degree of shaving this live data flow, you will be exposed to different data flows in strategy testers.
When you test an expert advisor in the strategy tester, you backtest the expert with this smoothed data. The expert can generate excellent results with this shaved data but when the same expert is exposed to a live data stream, it can behave unexpectedly from the backtest results. Which in turn can create a pressure on your account.
Lets materialize this situation with a popular trading instrument: EURUSD. For example, when we compare the backtest results of these two brokers, we see that the first broker has 100% history quality and with about 55 million ticks:
For the second broker, we have again 100% history quality and tick number is 78 million!
Both brokers provide 100% data quality, but the second broker provides almost 50% more tick data! In this situation, the tick data of the second broker better represents the reality.
Considering the instrument is the same, the analyzed period is the same, the price movement in that instrument is also unique, both brokers should provide more or less similar tick data numbers. But that is not the situation.
When you test your expert advisor in the first broker your test results will not be as reliable. When you test your expert advisor in the second broker, you will test it with more tick data and therefore the test results are more reliable.
Observe Results
Looking at the results, particularly the risk parameters greatly vary between two brokers. As I stated in the beginning, this situation is expected because the provided data is not unique across brokers.
When you have less tick, the farther you are getting away from the real data flow.
The expert performed better with more tick data in terms of risk but it scarified some gains in return. In this example, you should give more attention to the backtests from the second broker.
In this case, if you are risk sensitive and after you testing the expert in the first broker and change your mind on buying it it would not right decision.
If you are profit oriented and buy this expert because it provides good profit, again you would make wrong decision as with more tick data you have less profit.
What can we do to overcome this situation?
There is a simple rule of thumb you can apply. It is not 100% solution, but it will help you to judge an expert advisor more transparently.
When you are interested in an expert advisor, you firstly need to test it with your own broker. If the results are successful there, you need to test in at least one more and ideally 2 more brokers.
Ideally if an expert advisor is robust to data anomalies, it should not show a big difference in results across different brokers. Usually a difference up to 10% can be accepted.
However, If the expert advisor fails in one of 3 brokers or results are significantly vary, then you should not consider using it.
Thank you for reading and please have a look my other articles.
Evren Çağlar
