Menu
Sister Sites

Options

Bookmark Thread

First Page First Unread Last Page Last Post

Print Thread

Similar Threads

How to achieve 99% modeling quality for the expert adviser 1 reply

M1 timeframe and modeling quality... 3 replies

Backtesting 90% modeling quality? 1 reply

How to achieve 99% Modeling Quality 2 replies

How to improve modeling quality? 6 replies

Attachments: How to get 99.9% Modeling Quality on MT4 (BT)
Exit Attachments

How to get 99.9% Modeling Quality on MT4 (BT)

  • Poll

  • Back testing is.........
  •  
  •  
  •  
  •  
  • Post #1
  • Quote
  • First Post: Edited 5:06am May 24, 2018 3:53am | Edited 5:06am
Hello Folks,

There are many procedures available over internet, how to get 99.9% Modeling Quality Tick Data for back testing on MT4. Here I will share a simple method which is explained here in details. I will copy the same step wise with minor changes.

Step 1: Install Separate MT4 Platform
Install a new separate MT4 platform of your broker and keep it only for back testing purpose and don't use this for trading. You don't need to download history data from MT4 and also no need to load any charts. Just log in using demo or real account but don't change the account later. Keep in mind, you need more or less 10 GB space in your hard drive to download 1 year tick data of one pair.

Step 2: Download Data
Download and install StrategyQuant's Tick Data Downloader. After completion of installation, select configure and uncheck "Remove Weekend Data" and "Automatic Export to CSV" in case these are checked. Chose your time zone and press Ok. Now "Change Download Range" and chose your dates (depends how many years of data you need). After that, select your pair/s and press "Start Download". It will take time based on number of years. (One paid takes approximate 5-8 minutes to download 1 year data).

By default setting, data will be saved at this location; C:\TickDownloader\tickdata\

Step 3: Export Data to MT4
Download and install Tickstory Lite Ver 1.5.3 is free, but if you have extra cash, you can buy latest version. After installation, copy (always chose copy and don't select Cut) your pair folder from C:\TickDownloader\tickdata\ and paste it into Tickstory Lite data location, C:\Users\UserName\AppData\Roaming\Tickstory\Tickstory Lite\Data. For example, for EURUSD, copy complete folder and paste it here.

Now open Tickstory Lite, you will see that same date range is being displayed in front of EURUSD which you selected in StrategyQuant's Tick Data Downloader. Go to File\Setting\MT4 Setting\ and select MT4 Installation and MT4 Data folders of your platform which you installed only for back testing.

Now, right click on EURUSD (or your pair) and chose "Export to MT4", a new window will appear with 3 tabs. Go to last tab "Help" and follow the instructions. Press "Deploy MQ4" and an EA file will be generated in "YourBroker\MQL4\Experts" Open your MT4 Platform, open chart of same pair (e.g. EURUSD) and place this EA on that chart. If market is open and you see a smiley face in top right corner, a new file will be generated in "YourBroker\MQL4\Files"

Now again; go to same setting of "Export to MT4", go to second tab "MetaTrader Info" press Load and select the new generated file from "YourBroker\MQL4\Files". In this file, broker shares important data (spread, leverage, pair digits, commissions, swap, pip value etc) with Tickstory lite to be considered while data export. This file need to be generated for every single pair separately.

Select first tab "Data export", chose the same date range what you selected earlier, check "include weekends". You are almost done. Before pressing OK, make sure, MT4 Output section at the end is correct. (Mt4 Installation address, Server Name, HST and FXT directory)

HST Directory Example: C:\Program Files (x86)\Your Broker MT4\history\Demo 2
FXT Directory Example: C:\Program Files (x86)\Your Broker MT4\tester\history

Press Ok and data will be exported to your MT4 platform.

Step 4: Make Data Files Read Only
Once data exporting process complete, close your all apps (MT4 platform, Tickstory lite etc) and go to your HST and FXT directories (mentioned above), select all files, right click, properties, check "Read Only" and then OK. If read only box is filled with grey box, then you need to double click to check (you must know such basics).

Now you are ready to start back testing. Open MT4 Plat form, open chart of the same pair, press Page Up button and it will easily scroll back chart to your start date. It confirms the data copied successfully. Now you can delete data from below given directories to save space.

C:\TickDownloader\tickdata\ and
C:\Users\UserName\AppData\Roaming\Tickstory\Tickstory Lite\Data

Back test any EA, and you will see full Green bar and 0 mismatched chart errors. In back tests, you will see "Modeling Quality" n/a instead of 99.9% but it doesn't matter as you will see full green line and 0 mismatched chart errors. To get 99.9%, you need to purchase latest version of Tickstory and then open MT4 terminal from inside Tickstory. When you can export data with manual procedure, why should you pay to get this number only when results are same.

Your input:
Kindly let me know, if I missed some important step. Also kindly share what is your method to download reliable tick data. There are some drawbacks in above method (like slippage is not considered). Your feedback will be highly appreciated. Thanks
Grid / Martingale / Hedge Lover
Quoting KashifNawaz
Hello Folks, There are many procedures available over internet, how to get 99.9% Modeling Quality Tick Data for back testing on MT4. Here I will share a simple method which is explained here in details. I will copy the same step wise with minor changes. Step 1: Install Separate MT4 Platform Install a new separate MT4 platform of your broker and keep it only for back testing purpose and don't use this for trading. You don't need to download history data from...
I am aware of the fact that you don't think highly of me so here's one more attempt to share some foundational knowledge for your thread participants. My apologies in advance if you already know this sh*t.

 

  1. Tick Data will always be dependent on your broker's price feed and order book.
  2. Tick Data does not represent actual orders and physical trades you please with your broker. A minor touch to 'order flow' in your broker's book will create a 'fictitious' tick of not meaningful value
  3. Brokers try to fake it by appearance of a global fx market via tools like strategy tester alowing traders to think and feel you strategy will create a perfect curve. There will be a rude awakening after few months of go live
  4. There is no substitute for live forward testing ... even in a demo account.
  5. Optimization of parameters is the industry's biggest scam. This is the reason why 99.9999% of all EAs fail in a live environment. The moment you think of parameter optimization, you have already lost your money. Save it for your kids future!
  6. fixed spread in strategy tester is never going to be the same in live. For this reason also, I rule out all backtest results. No amount of modelling in strategy tester will help you succeed.
  7. Any EA that does not work round the clock 24x5 unattended under all market conditions across all pairs and across all timeframes and across all FX trading sessions is a highly optimized solution (either via strategytester or manually done) based on specific custom market patterns or scenarios. It will never ever last more than 3 yrs.
  8. Any EA based purely on a single chart (price versus time) will never succeed no matter how many indicators you put on it. This is common sense as there aren't many FX Millionaires or Billionaires we see on the Forbes List trading Spot FX.
  9. Don't believe it any StrategyTester balance curve. I can produce a similar 45 degree angle balance curve in a live trading environment in a day or in a week or in a month. I can make 100% realized profit every day with floating loss draining my equity. I can make 9%% win rate systems on the fly. Balance Curve shown in StrategyTester, Demo and Live MT4 reports are meaningless and does NOT reflect an accurate performance measure of folks sharing screenshots. It is the biggest vehicle to promote SCAM on forums. Here's my fancy screenshot of balance curve for the month of May (just in case you feel I am complete BS)

Attached Image (click to enlarge)
Click to Enlarge Name: Image1.png Size: 21 KB
Staying in my lane...
Quoting VEEFX
{quote} I am aware of the fact that you don't think highly of me so here's one more attempt to share some foundational knowledge for your thread participants. My apologies in advance if you already know this sh*t. Tick Data will always be dependent on your broker's price feed and order book. Tick Data does not represent actual orders and physical trades you please with your broker. A minor touch to 'order flow' in your broker's book will create a 'fictitious' tick of not meaningful value

Brokers try to fake it by appearance of a global fx market...
Your post should be made a sticky. Spot on.
  • Post #4
  • Quote
  • Edited 1:15am May 25, 2018 1:04am | Edited 1:15am
Quoting VEEFX
{quote}My apologies in advance if you already know this sh*t.
Nice post mate.

Just on this matter....where there does need to be a general consensus in this decentralised market is in 'control points'. The fidelity of brokers price feeds are matched by the savvy customer against the interbank market and brokers who want to stay on the right side of regulation tend to enforce acceptance of these control points which are the things that can be objectively tested and validated. These control points are *the open, the high, the low and the close* down to the finest granularity allowed for my MT4 which is the M1 timeframe. Nothing lower is meaningful.

The intrabroker variability in the maintenance of de-centralised price management makes tick data valueless between these control points. In fact MT4 in resurrecting price bars from decentralised tick data held by brokers totally avoids this issue entirely in that in the MT4 bar construction process for a historical timeframe data set...the points between control points are randomly generated in the process.

What this means to any strategy developed under the MT4 platform environment and hence the impact on the strategy tester is that the value of ticks between control point holds no value to the outcomes from the test.

So ways around this obstacle in MT4 are as follows.

1. Develop strategies that operate on control points only and where the entries and exits are based on *different bars* from the control points for every timeframe tested. Stuff any conclusions you may draw from activity within the single bar aside from these control points and their sequence.
2. Ensure that your entries and exits occur on different bars...which will force you to the timeframe that is below this level of required granularity to test it;
3. Draw on data from more trusted broker sources of the regulated brokers such as Dukascopy, Pepperstone etc.
4. Recognize that intrabroker variability will exist as their is no centralised exchange to test the fidelity of the broker data sources. The best we can get is consensus around these control points.
5. Step into the world of trading the centralised exchanges to avoid this limitation of forex such as equities and futures;
6. You can still use price feeds from other sources in your data history to perform testing. You just need tools to replace the broker data with your more reliable data sources.
7. MT4 is severely limited in its ability to manage a diversified portfolio, lacks transparency in isolating the frictional costs of trading that are broker dependent such as spread, SWAP and slippage...... and MT4 is a huge time waster in testing any strategy.....and importantly this is exactly what the brokers like.
8. MT4 is the preferred gambling tool of brokers that makes it sufficiently difficult to interpret what a sustainable strategy is.....and hence is ideal for the retail trader who wants to be the net loser in the ultimate transaction with the market makers.
9. There are some more reliable backtesting testing solutions on the market that are not native to MT4 that can address these issues within MT4 and hence can resurrect tick data to a level of granularity below M1...hence you have to pay for the privilege of using them which is worthwhile if you are serious about testing for discretionary and automate solutions; or
10. Move to different platforms that address these issues in testing strategy robustness and across portfolios such as Trading Blox, Ninja Trader, Amibroker etc.
Quoting Copernicus
{quote} Nice post mate. Just on this matter....where there does need to be a general consensus in this decentralised market is in 'control points'. The fidelity of brokers price feeds are matched by the savvy customer against the interbank market and brokers who want to stay on the right side of regulation tend to enforce acceptance of these control points which are the things that can be objectively tested and validated. These control points are *the open, the high, the low and the close* down to the finest granularity allowed for my MT4 which...
Oh man. You just triggered a burning desire in me to piss on CrapT4.. Expect a looong post tonight. My entire (well..for the most part) strategy is designed around the lack of 'features' in MT4. Mt4 is one of the top five reasons why retail traders blow their accounts :-
Staying in my lane...
Quoting VEEFX
{quote} Oh man. You just triggered a burning desire in me to piss on CrapT4.. Expect a looong post tonight. My entire (well..for the most part) strategy is designed around the lack of 'features' in MT4. Mt4 is one of the top five reasons why retail traders blow their accounts :-
Hear ....hear Mr Speaker. MT4 has been a crime in action for many a year....inexcusable. We should lynch them all at Metaquotes :-)

Attached Image
U can get 100%, but, in MT5, with real ticks.
This is the best way, to have a cross-platform EA, for MT4&MT5.
  • Post #8
  • Quote
  • Edited 4:11am May 25, 2018 2:06am | Edited 4:11am
Quoting mql2me
U can get 100%, but, in MT5, with real ticks. This is the best way, to have a cross-platform EA, for MT4&MT5.
If I put my conspiratorial hat on then I could say this.

Conspiracy Assumptions
1. Metaquotes and the market maker industry has had a long boon at the expense of he retail trader for many years as MT4 was promoted as the retail platform of choice by the brokers......no wonder about that given the transparency issues surrounding the machinery inside MT4......probably more of a fact than an assumption.
2. In a more enforceable regime of today where some of the 'issues' about the level playing field of MT4 were progressively exposed by the retail community....we now have seen a *stated but questioned real intent* desire my Metaquotes to force the retail trader onto MT5.....however the 'convenience factor' of MT4 and host of algos developed for it...has actually worked in the favour of MT4....and it still remains very hard to kill off. If it was in the actual *deep dark board room actual* interests of Metaquotes to get the community onto MT5 then with the brokers....this would have been a simple task of simply droppping the MT4 platform 'tout de suite'...to force the change. We have seen a lot of new technologies do just this.....but why the reticence with MT4?
3. The real underlying reason is that the retail trader prefers convenience to the hard work of understanding a new platform which has been very advantageous for those *in the know* who may have been benefiting from this windfall.....but it in reality has been at the expense of the retail trader's PL. You can't blame these drug taking gamblers however. That is just the way life is in the casino.

So is this a double bluff from Metaquotes from the conspiracy theorists?
Quoting VEEFX
{quote} Oh man. You just triggered a burning desire in me to piss on CrapT4.. Expect a looong post tonight. My entire (well..for the most part) strategy is designed around the lack of 'features' in MT4. Mt4 is one of the top five reasons why retail traders blow their accounts :-
Sorry man. I won't share my findings on MT4 design features that are left out on purpose and hidden to retail traders. Last time I did over 3.5 yrs ago, I received tons of PMs asking for more details. Here's what I can share to provoke thoughts for those who are inclined to investigate more...

 

  1. Run the platform thru the weekend and you will notice it will do funky things without your knowledge or approval. An old outdated version 4 of MT4 should not have so many "software updates" from your broker (I am not referring to software builds)
  2. Related to #1, I am almost certain broker can read/upload/access your expert log files or 'see' what you 'see.
  3. Why does MT4 build candles only on Bid price? There is a VERY IMPORTANT reason behind it.
  4. Why does MT4 limit the volume of tick charts? There is a very important reason behind it. FXCM TS2 can zoom out their tick charts and that is the primary reason why CrucialPoint is still with FXCM
  5. Why does broker limit the volume of M1 candles only to approx 30 days in MT4. There is a very important reason behind it
  6. Why does ALL retail brokers prevent updating the order size of submitted limit and stop orders in MT4 or any other retail platform?
  7. Why are sell orders in MT4 less prone to manipulation or scams compared to buy orders?
  8. Why is price feed manipulation via ticks (by design) extremely difficult to pinpoint after M1 candle is closed out?
  9. Across the range of millions of ticks, what is the most accurate price point (or tick) that is 99.9999% reliable across all retail brokers (to ensure there is no cross-broker arbitrage opportunities)?
  10. I actually do not believe there is "cross-broker" collaboration within the retail spot fx industry to keep the price feed 'clean'. That function or responsibility lies within/across the LPs (but not sure... i could be wrong here)
  11. And finally, why would anyone want to trade long term on a platform that is 100% in control of your broker? Is just does not make sense, especially when MT4 is soon going away. MT4 for me is a "research and analysis" tool (i.e. Demo Trader for Life) and a stepping stone to move to API model using FIX down the road. I would say this though... someone here on FF has broken the link between MT4 and broker to make it completely independent of future updates.... I might want to go this route but still not a good idea as it is a 32-bit software dependent on being supported by Microsoft on higher bit computers of the future.

Staying in my lane...
  • Post #10
  • Quote
  • Edited May 26, 2018 1:30am May 25, 2018 3:58am | Edited May 26, 2018 1:30am
Quoting mql2me
U can get 100%, but, in MT5, with real ticks. This is the best way, to have a cross-platform EA, for MT4&MT5.
Are you saying there is a way for same EA to run both on MT4 and MT5? And how long are the tick charts in MT5?
Staying in my lane...
Quoting VEEFX
{quote} AS you saying there is a way for same EA to run both on MT4 and MT5? And how long are the tick charts in MT5?
Yes, u can run one EA on both platforms.
Depends of broker.
Quoting VEEFX
{quote} And finally, why would anyone want to trade long term on a platform that is 100% in control of your broker? Is just does not make sense, especially when MT4 is soon going away. MT4 for me is a "research and analysis" tool (i.e. Demo Trader for Life) and a stepping stone to move to API model using FIX down the road. I would say this though... someone here on FF has broken the link between MT4 and broker to make it completely independent of future updates.... I might want to go this route but still not a good idea as it is a 32-bit software...
What's your thoughts on cTrader?
Quoting VEEFX
{quote} I am aware of the fact that you don't think highly of me so here's one more attempt to share some foundational knowledge for your thread participants. My apologies in advance if you already know this sh*t. Tick Data will always be dependent on your broker's price feed and order book. Tick Data does not represent actual orders and physical trades you please with your broker. A minor touch to 'order flow' in your broker's book will create a 'fictitious' tick of not meaningful value Brokers try to fake it by appearance of a global fx market...
Thanks a lot for such inside knowledge. I need many years, if not decades, to gain such knowledge of forex market. Thanks once again for sharing and just humble request, please don't delete your post so that it will be beneficial for many other users to come. Thanks

Quoting VEEFX
{quote} Sorry man. I won't share my findings on MT4 design features that are left out on purpose and hidden to retail traders. Last time I did over 3.5 yrs ago, I received tons of PMs asking for more details. Here's what I can share to provoke thoughts for those who are inclined to investigate more... Run the platform thru the weekend and you will notice it will do funky things without your knowledge or approval. An old outdated version 4 of MT4 should not have so many "software updates" from your broker (I am not referring to software builds) Related...

VEEFX & Copernicus

After reading your above eye-opening posts (full of facts), everyone will start hating MT4 and even not feel comfortable to go for MT5. So if we cannot use these platforms to not to be a victim of brokers, whats your suggestion for traders with small capital? What will be the genuine and safe way for them to trade? Thanks in advance.

Furthermore, if all back testing techniques in MT4/5 tester are useless then what is the way to test our strategy? As you said, even Demo forward testing will not help, then what option do we have? Live forward test with Cent account?
Grid / Martingale / Hedge Lover
Quoting KashifNawaz
So if we cannot use these platforms to not to be a victim of brokers, whats your suggestion for traders with small capital? What will be the genuine and safe way for them to trade? Thanks in advance. Furthermore, if all back testing techniques in MT4/5 tester are useless then what is the way to test our strategy? As you said, even Demo forward testing will not help, then what option do we have? Live forward test with Cent account?
Hi K

If you understand the shortfalls regarding backtesting then you can plan around them so the backtests from MT4 are much more meaningful. I don't want to be a downer in any way on back testing as a principal. It is the mainstay of my whole reason for being a trader.....but it is essential that your efforts in backtesting are not wasted and there is a level of reality between your backtest and your forward test to ensure that when you go live you are not crippled by simply poor quality standards....otherwise you will throw the principal away with the bath water and make claims about the pointless exercise etc....and find you are left without oars for your leaky boat as a trader. So backtesting is an essential part of the game to design robust systems.

For example you won't find info in the reports generated by MT4 to drill down into the frictional costs such as Spread and SWAP....but with some programming knowledge you can generate your own reports from MT4 that creates a csv file generated from your strategy tester. This is one way to get more useful info such as the impact of SWAP on long hold trades that can easily tip them from profit to loss....not by strategy design but from broker SWAP arrangements.

Having an understanding of how the bars are constructed from tick data by MT4 also helps you design for that deficiency. You will still need to test on high quality data sources to avoid shenigans...but your testing should be about proving the effectiveness of your system....not how well your system responds to your choice of broker. That comes later before you go live.

So you can still use MT4 for testing....which I do provided you are aware of and eliminate the issues to ensure you can interpret the results to help you design better systems......but in reality....there are far better platforms to use for rigurous testing. I am just a lazy bastard quite familiar for MT4....so I use workarounds.

A final obstacle of MT4 is the data range provided. This needs to be overcome to ensure you test with sufficient data sample size...but using Tickstory etc does help in this regard in being able to construct long range time series that have a degree of robustness that is meaningful to your design efforts.
Quoting VEEFX
{quote} Thanks but my question was whether EA written in mql4 will run as-is on MT5 and not both at the same time? I thought MT5 codeset was not downward compatible. I have no desire to learn MT5. I rather move to API and get out of retail spot broker and go institutional.
Why don't inform yourself? CROSS-PLATFORM EXPERT ADVISOR: INTRODUCTION

Who wants to do something, is looking for solutions, who don't want to do, is looking for excuses. You want to trade or to blame?
Is this the only way or we can directly download data from dukascopy/dickdata and import/paste to mt4?
Quoting gharti
Is this the only way or we can directly download data from dukascopy/dickdata and import/paste to mt4?
There are many other ways but they are much complicated than this. Let me know if you find an easy one.
Grid / Martingale / Hedge Lover
Quoting KashifNawaz
{quote} There are many other ways but they are much complicated than this. Let me know if you find an easy one.
I get the message that the files are read only when I start the strategy tester....and then it stops!
Vucking good EA coder...
Quoting reteid2222
{quote} I get the message that the files are read only when I start the strategy tester....and then it stops!
I guess you are using different dates in strategy tester. The disadvantage of above given method is, when you chose date range, you need to use the same in strategy tester for quick results because it make singe file of that range. For example, if you downloaded data from 01.01.2014 to 01.07.2018, it will become one file for each time frame. You need to give same dates in strategy tester for immediate results. If you will use different dates (for example 01.01.2016) still strategy tester will start reading from 01.01.2014 and it will take few minutes (maybe more than 5 minutes) to reach your given date. So better once you downloaded data, use the same dates in strategy tester as well. Hope this will solve your issue.
Grid / Martingale / Hedge Lover
thanks for sharing..tickdownloader and tickstory are very vital softwares every developer has to get hold off...
''If its BULLSHIT when automated trading it manually is STUPIDITY''
1 trader viewing now