Visualize economic indicators with MT5 indicator! Time series display and country comparison
We have not been able to update this blog for a while, but we have finally finished the round of Helios modifications and have started to verify its operation. We are going to check and investigate the basic operation and search for a parameter set with good performance by running backtests. However, this process takes a lot of energy and time, so we decided to automate it this time. I would like to write this article to organize the matters to be addressed.
Contents
Need for automation
However, if only the automatic execution of each part is done, manual work is required in the process of performing individual unit backtests while visually checking the results after optimization, and even if the unit backtest is automated, it is not possible to extract good execution results from a large number of execution results or to immediately extract and utilize the relevant parameter set. Even if unit backtesting is automated, it is not possible to extract good execution results from a large number of execution results or to immediately retrieve the relevant parameter set for further use.
After all, the amount of backtest execution can be increased only after end-to-end automation, so we aimed to automate the entire process this time.
Benefits of Automation
If we could run 100 times more backtests than before, we would realize something different, wouldn't we?
Originally, currency pairs and legs are selected by human minds as execution conditions, but in many cases, we often lead the way based on past experience or our own image of the trade we want to execute, saying, "Perhaps if we verify conditions around this value, it will work out well. This human judgment (or sense, or domain knowledge) is necessary because it takes a great deal of time to run a back-test, and unless you narrow down the range of possible profitable results in advance and start testing, you may not reach a good conclusion no matter how long it takes.
However, such judgments often limit the scope of what should be looked at due to the bias of experience, and this can cause good conditions to be overlooked. This is why we need a system that can collect a wide range of data with the help of automation and visualize the overall results.
Automation requires computing resources to handle a large number of backtests, but it is not difficult even for individuals to obtain a high-spec PC to run backtests at home.
Automation image
This is how I did it.
MT5 performs backtest optimization and unit testing, but the execution conditions and parameter files necessary for automation are generated on the python side, and the MT5 process is started and loaded. The good scores and their parameters are picked up from the optimization results, and a parameter set file is generated for the unit backtest. Based on this, the stand-alone backtest is automatically executed. The results of the stand-alone backtest are output as html files, so the html files are scraped to acquire and organize various indicators, etc., and stored in the DB. At this time, the results are also output as image files at the same time as the html files, and these are uploaded to the wiki for viewing, along with the indicator data of the execution results. By the way, what are the conditions for optimization?
- Issue (USDJPY, EURUSD, ...)
- Period (from ~ to date)
- Legs(M1, M2, M3, ...)
- Brokerage (XM, HF, ...)
- Margin
- Trade Category (Buy, Sell)
- Optimization conditions (maximum profit coefficient, maximum expected gain, ....)
etc., and not a parameter set. Since the number of combinations of parameters is on the order of billions, genetic algorithms are used to control the number of executions. If the number of variables and the number of optimization conditions were 3 for stocks, 1 for periods, 4 for legs, 2 for brokers, 3 for margins, 2 for trading categories, and 4 for optimization conditions, 576 optimization conditions would be generated. This may not seem like much at first glance, but even with a genetic algorithm, 10,000~20,000 backtests are performed for each optimization pattern, so even with an average of 15,000 backtests, 576 = 8,640,000 backtests are required. This is a lot of work for just 3 stocks (e.g. USDJPY, EURUSD, GBPUSD). If we wanted to do it by hand, we would have to hire someone to stay on and do it endlessly, input the next set of parameters into the screen after each backtest, run it, output the report, tabulate it, and send it to ...... This is a mind-boggling amount of time and work. When you think about it, the power of automation is great. By the way, with the EA we are currently testing, it takes about 2~4 hours to complete one optimization pattern. Even with a 64core 128GB PC running at full capacity, it still took about this long. After the optimization is completed, stand-alone backtesting and reporting also take some time, so the number of optimizations that can be done per day is about 5~6. 576 / 5 = 115 days. This still seems daunting, so I'm hoping to start some forward tests as well, while narrowing down the execution conditions a bit more.
Report Image
I use Confluence for my wiki because I like Atlassian products.
We also created an automatic upload and edit process for backtest reports via the Confluence REST API.
Combined with Jira Service Desk, we were able to create a route to share this type of report without significant cost while providing service desk functionality. We have implemented this functionality as a report in the Helios beta version, but have not yet released the program publicly due to the possibility of functional improvements. Please note that we have not yet released this program publicly due to the possibility of functional modifications.
summary page
Single Backtest Report
We have uploaded a summary of the indicators and report images. We will first determine if the indicators can be used at a rough level, and if you are considering using them, we will actually conduct back and forward tests. The back-test parameters can now be downloaded, so you don't have to go through the hassle of recreating the conditions at hand. There were several times when I realized that I had been operating for several days with incorrect parameter settings, and it made me cry, but it seems that such problems can be solved. 
Tableau
The reports on the wiki are limited in the amount of data they can handle, so we will keep them superficial in scope and rely on Tableau for the part that looks at overall backtest trends.
The optimization report that can be viewed in MT5 only shows the breakdown of one pattern, but there is information that can be seen by collecting and comparing optimization reports, such as comparing the overall trend of optimization by stock, how the results change with different legs, how different initial margins appear in the results, and so on. I'd like to see more backtest results accumulated here.
I would like to re-evaluate this point when more back-testing results are accumulated.
impression
- We have actually started to fully automate backtesting and are now able to perform extensive backtesting, which we had not been able to do before.
- Overall, the performance of currency pairs with large negative swaps and trading directions is chronically poor. Although we have not fully analyzed the details, the cases where daily negative swaps accumulated, such as selling USDJPY and buying EURUSD, were much harder to perform than expected. Swaps and spreads seem to affect profits to a greater extent than imagined. Selection of not only stocks but also trading direction is important. *In foreign brokerages, swaps and spreads are often unfavorable, but on the other hand, this is why the zero-cut system is considered to be in place, so it is better to think of it as an expenditure for safety.
- GBPUSD's performance was quite good. I chose it based on its narrow spread and back-tested it, and got good results. It may be a good candidate for operation. I wanted to continue to evaluate it while also forward testing.
- Difficult to evaluate USDCHF Although USDCHF was verified due to its narrow spread and appeared to have good results, the Swiss franc shock on 01/15/2015 caused it to fluctuate by nearly 2,000 pips in an instant. The timing alone was enough to generate nearly JPY30,000,000 in profit.
Whenever you do a backtest on a 10-year scale, you will always see data on the timing of the Swiss franc shock, and this alone can skew the overall results so much that it is difficult to evaluate the results throughout the entire period. - The room is hot due to exhaust heat from the high-spec PC. Since the PCs run 24 hours a day, the air conditioner must also run 24 hours a day, but I am worried about PC failure during the summer.
The monthly electricity bill has not increased drastically, but it is difficult to compare with the previous year due to the change in lifestyle under the influence of Corona. - It is right to buy a physical PC. It will be overwhelmingly expensive if a PC with these specs is built on AWS EC2, etc. and kept running for a long period of time. If your electricity bill is not too high, physical PC is better.
- I would like to try stock indices (Dow, S&P500, Nikkei225, etc.) In many cases, overseas brokerage firms handle various stock indices and individual stock issues, and can apply automated trading to these.
We would like to try these stocks as well, once the rest of the back-testing work is completed.
Crypto assets such as Bitcoin (BTCUSD, etc.) are also handled, but I don't feel that technicals work for those (although I may notice something different if I try them), so they are low on my personal priority list.
See the following article for an overview of HF copying and how to use it. Free EAs are available for HFM (Premium account) or XM (Standard account). Please click the link below to open an account. *Depending on conditions, low spread accounts may be available. The following screen will open. Please fill in the required information and contact us. After you click the "Submit" button, you will receive the following auto-reply e-mail to the e-mail address you entered. If you already have a VPS environment, you can use it as is. Domestic brokerage firms are at risk of additional fees, and MT5 is not available at most brokerage firms. Only foreign brokerage firms are subject to the zero-cut system, which allows you to take no more risk than the amount of your deposit.This will be the case. and many other benefits. When using MT5, please consider opening an account at an overseas brokerage firm at the same time. As a benefit to those who use the EA in their real accounts, we offer the use of our originally developed indicator free of charge. With automated trading tools, the trading itself can be done continuously by the program according to the rules, but it is the role of humans to determine which EAs to start/stop and when to stop them. Since analysis of the market environment is sometimes necessary, we encourage you to use these free indicators to help you realize solid investments. 【PR】 HFM is running an event where you can win great prizes if you reach a target trading volume set according to your trading volume! The event is open until 30/08/2024! Enter as soon as possible! 【Notice】New EA Selene has been released!
EA is available free of charge!
Usage 1: Trade without the need for a VPS using HF Copy
Usage Method 2: Prepare a VPS and deploy EA
Step 1. Create an MT5 account
Please contact us for details. Step 2. Application to use automated trading tools
If you do not receive it, there is a possibility that you typed the address incorrectly or it was sent to your spam mail.
If you have checked your spam mail and still have not received it, please reconfirm your e-mail address and contact us again. 
Step 3. Sign up for a VPS
There are several ways to create a new VPS environment, but if you are not comfortable with building your own VPS (it seems difficult or you do not know how to manage it), we recommend that you use a VPS service. If you want to use MT5, I recommend an overseas brokerage firm!
However, this does not mean that any brokerage firm will do.
by choosing a major international brokerage firm, Special offers available for EA users only!
indicatorVisualize economic indicators with MT5 indicator! Time series display and country comparison
indicatorPrice action determination based on zigzag
indicatorAutomatic generation of trend lines
indicator3x faster analysis! Fibonacci Retracement Auto-Generating Indicator
indicatorUsing RSI Divergence in MT5! Three effective strategies
