Menu
Sister Sites

Options

Bookmark Thread

First Page First Unread Last Page Last Post

Print Thread

Similar Threads

Optimization, statistics, and avoidance of curve fitting 21 replies

Curve Fitting: The Ultimate Conundrum 17 replies

EA Backtesting and Curve Fitting Concern 2 replies

why curve fitting doesn't work? 23 replies

"curve fitting" or just improving?? 5 replies

Attachments: Curve fitting vs optimization
Exit Attachments

Curve fitting vs optimization

Quoting robots4me
@HeyYou -- I really like this topic -- thanks for creating this thread. {quote} I'll be blunt -- this is a complete waste of time -- unless you have a time machine. We trade in the present -- the only data that matters is your broker's most recent data. You only need to go back in history far enough in order to simulate 50 -- 100 trades, which is your "sample size". You need a minimum sample size in order for your stats to be significant. The data patterns from 2000 to 2014 may never occur again -- there are billions and billions of possible data...
this is really really insightful and it really really make sense especially RECALIBRATION.
I have a different view of the markets now thanks to you.

It makes sense because every year & quarter. you have different values, (i.e. who would have guessed that Brexit is a thing? or that Greece is gonna fall?)
thinking of recalibrating my strategy every quarter. WITH DATA FROM MY BROKER. this is EXTREMELY important because fx is not centralized.
Let ur winners run & cut ur losses short.
Quoting HeyYou
changing the parameters because they didn't work well in the past 3 months is crazy. you need at least 10 years of data...
My POV:
Just like the market, the weather is also hard to predict because of turbulent and other butterfly effects. And if you used 10yrs of data to predict the weather tomorrow, it's cynical, considering the humidity,heat,etc. 10yrs ago vs yesterday varies and yes you need to re-calibrate your system from time to time to adjust with the volatility of the market and major news events just like global warming.

Volatility is the key, whether to use a momentum/mean reversion strategy or simply no trades at all.
Let ur winners run & cut ur losses short.
Quoting LloydOz
I wouldn't bother optimising beyond what is generally accepted. Really. I have about a thousand reasons for saying that, probably most of which have been articulated by others, including the late Mr Babcock who was mentioned here and whose books I did read all those years ago. One is that despite having say a 100 trades, what if one of the settings was only responsible for one trade, say a stop that was triggered once, which influenced the entire results, and without which the results would have been forgettable? Optimising everything else may make...
Well said
Let ur winners run & cut ur losses short.
Quoting remonpilip
{quote} My POV: Just like the market, the weather is also hard to predict because of turbulent and other butterfly effects. And if you used 10yrs of data to predict the weather tomorrow, it's cynical, considering the humidity,heat,etc. 10yrs ago vs yesterday varies and yes you need to re-calibrate your system from time to time to adjust with the volatility of the market and major news events just like global warming. Volatility is the key, whether to use a momentum/mean reversion strategy or simply no trades at all.
from my experience, if you backtest in the long term, you don't lose money.

however after a couple months and years of live trading you may realize that the mt4 strategy tester is no good for optimization..... forget it.
Quoting HeyYou
{quote} the mt4 strategy tester is no good for optimization..... forget it.
The price data from MT4 is stupid.
Quoting hanover
{quote} My 2c fwiw: Excellent posts by topherhk88 here and merlin here. Bruce Babcock, a well known futures trader during the 1980s-90s, wrote a lot of good material about optimization. I found this as a starting point; you might be able to find more of his writing using Google....
Hey, thanks for sharing babcock's reviews blog i rwad it all took me 3hrs to finish reading it very infornative and changed the way I see optimizing.

To shorten the 3hrs read from my POV:
To avoid curve fitting you system must be able to trade on at least 10 other markets with a degree of profitability. Start with a core strategy and if it is profitable on 10 other markets then apply a filter to it and test it again on 10 different markets to see if yout filter is over optimized or nah

He also notes that if you have x systems and each of them works on at least 10 different markets, you have a portfolio

x can be ranging from 1 to infinity

But it's best if you have a system that works well on eurusd but also works at 10 other different markets and other system that works well on usdjpy that also works well on 10 other different markets

Diversification:
Bruce notes that, from his experience one's portfolio needs to have 10-20 markets to avoid strategy down days

His style says that he is always in the market and never even think of trying tk predict the market

His Rules:
Entry
Exit when loss
Exit when profit
Let ur winners run & cut ur losses short.
Quoting remonpilip
{quote} Hey, thanks for sharing babcock's reviews blog i rwad it all took me 3hrs to finish reading it very infornative and changed the way I see optimizing. To shorten the 3hrs read from my POV: To avoid curve fitting you system must be able to trade on at least 10 other markets with a degree of profitability. Start with a core strategy and if it is profitable on 10 other markets then apply a filter to it and test it again on 10 different markets to see if yout filter is over optimized or nah He also notes that if you have x systems and each of...
Also he sacrifice return for stability & ultimatrly lowering dd
Let ur winners run & cut ur losses short.
Quoting HeyYou
{quote} from my experience, if you backtest in the long term, you don't lose money. however after a couple months and years of live trading you may realize that the mt4 strategy tester is no good for optimization..... forget it.
Agreed, i moved to mt5 alr

Wayyyyy faster
Let ur winners run & cut ur losses short.
Hi,
My strategy executes entries according to market volatility and volume change and exits are with SL and profits are again according to (fading) volatility.
I have fewest parameters possible to avoid curve fitting.
For long and short trades I have exactly the same entry and exit strategy..
However I have figured out that these both are 2 different realms..
I would like to ask if can tailor my strategy for long and short trades differently.
By doing so would I be definetely get caught by the trap of curve fitting.
  • Post #50
  • Quote
  • Edited 3:35pm May 20, 2024 3:02pm | Edited 3:35pm
I would like the input from traders who are actually math and stats inclined and truly understand my question and workings of MT5 Strategy Tester.

When optimizing ea in MT5 strategy tester for the best results for inputs into ea - MA, RSI, TP, SL, Trailing, etc.for the previous 3 weeks (as an example) on Every Tick Slow Complete algorithm, do optimizations results skewed towards the end of the 3rd(last) week of the optimizations period or results distributed evenly across 3 weeks?

For example, i optimize 5 Min TF for the previous 3 weeks for particular inputs in the ea and trade forward one week with these best inputs, therefore the question is, if best optimized results distributed evenly across 3 optimized weeks.

The initial reaction is "Yes, Evenly" but there are doubts.
Quoting driven18
do optimizations results skewed towards the end of the 3rd(last) week of the optimizations period or results distributed evenly across 3 weeks?
Optimization in MT5 tries to maximize profit, maximize Sharpe Ratio, minimize drawdown, etc. All these KPIs are calculated over the entire equity curve in your backtest interval. There is no way that the profit of the last week is weighted heavier than the profit of the first 2 weeks. Profit is just that: the sum of the profit/loss of all trades in the backtest interval.

You can create your own custom KPI that you use for optimization (see https://www.mql5.com/en/docs/event_handlers/ontester). In that calculation you can do whatever you want. For example add more weight to recent trades in your profit calculation.
Quoting yoriz
{quote} Optimization in MT5 tries to maximize profit, maximize Sharpe Ratio, minimize drawdown, etc. All these KPIs are calculated over the entire equity curve in your backtest interval. There is no way that the profit of the last week is weighted heavier than the profit of the first 2 weeks. Profit is just that: the sum of the profit/loss of all trades in the backtest interval. You can create your own custom KPI that you use for optimization (see https://www.mql5.com/en/docs/event_handlers/ontester). In that calculation you can do whatever...
I agree with above, but when asked MQL5 Forum, here is the answer I received from the person "in the know" .
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 17 KB
Quoting driven18
I agree with above, but when asked MQL5 Forum, here is the answer I received from the person "in the know" .
I think he refers to compounding.

The MT5 strategy tester calculates KPIs assuming fixed risk. Profit is absolute profit. Drawdown is absolute drawdown. Profit factor is absolute profits over absolute losses, etc. Not a problem if your strategy uses fixed lot sizes, or a fixed percentage risk of the starting balance. However, if you scale your lot sizes as a percentage risk of the current equity, your trade volume will increase as your equity increases near the end of the backtest interval. In that case the later trades are weighted heavier than the earlier trades.

You indicated you backtest over a 3 week interval during which your equity hardly changes. Therefore, in your case it is not relevant whether you compound or not.
Quoting yoriz
{quote} I think he refers to compounding. The MT5 strategy tester calculates KPIs assuming fixed risk. Profit is absolute profit. Drawdown is absolute drawdown. Profit factor is absolute profits over absolute losses, etc. Not a problem if your strategy uses fixed lot sizes, or a fixed percentage risk of the starting balance. However, if you scale your lot sizes as a percentage risk of the current equity, your trade volume will increase as your equity increases near the end of the backtest interval. In that case the later trades are weighted heavier...
Yes, it is about compounding based on % of balance traded, but my equity does change for sure, that is the name of this game, get higher and higher equity.
Quoting HeyYou
So guys, many people confuse curve fitting with optimization and I get pissed off when I hear things like "you are just curve fitting!" bla bla bla.. so I wanted to share my views about this interesting subject: is my model actually predictive or plain stupid ? this question drove me nuts until I came up with an answer no it's not plain stupid... My conclusion is that checking if a system works with different parameters (e.g. entry / exit points) is a must before proceeding with optimization.... but this is not enough, it has to work in the...

This is how backtesting usually done. You can also try walk forward test to achieve more robustness for your system. It's simple yet effective
Quoting driven18
Yes, it is about compounding based on % of balance traded, but my equity does change for sure, that is the name of this game, get higher and higher equity.
It doesn't matter. Your equity probably changes a few percent in 3 weeks interval you use in your backtest. That only means the most recent trades weigh a few percent heavier in the optimization, but that is a neglectible effect.

If you are really worried about that small effect, then you can always change your EA so that it does not compound. Calculate your lot size using a fixed risk of the *starting balance*, not the current equity. For example, if your backtest starts at 10000 balance, then keep using 10000 for your risk calculations even if the equity is 10500 after 3 weeks.
  • Post #57
  • Quote
  • Edited May 27, 2024 8:32am May 26, 2024 10:22am | Edited May 27, 2024 8:32am
Here is the optimization equity line with best optimized inputs for previous 50 trades, for approximately 2 months on 1 Hour TF. I optimize every week for the previous 2 months and trade forward one week.

I will trade these inputs next week. Profit per 50K account is $43,680(87%) with DD - 11%(Balance) and 14%.(Equity)

Based on these optimization results, I expect to pull between + 4% and + 9% out of market next week.

Question. Based on your optimization results, what would be your expected results out of your optimized inputs trading forward before next optimisation?

If you can, please expand on the subject of optimized results and expectancy for forward trading interval.

I would also welcome any critiques you have with this approach.(disregard the subject of optimizing everyday for the previous period and subject of not enough trades being optimized).
Attached Image(s) (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 37 KB
Click to Enlarge Name: screenshot.png Size: 17 KB
I was so obsessed with these things...
I remember this was such a stressful period of my life...
I miss it LOL
I live in different reality now (real estate / passive investing..), which works but it's not as fun... so is there any system that actually works ? live stats plz
  • Post #59
  • Quote
  • Edited May 27, 2024 8:07am May 26, 2024 6:33pm | Edited May 27, 2024 8:07am
Quoting HeyYou
I was so obsessed with these things... I remember this was such a stressful period of my life... I miss it LOL I live in different reality now (real estate / passive investing..), which works but it's not as fun... so is there any system that actually works ? live stats plz
You are missing out the best business in the world .
Quoting driven18
Question. Based on your optimization results, what would be your expected results out of your optimized inputs trading forward before next optimisation?
Just based on the equity curve of an optimization there is nothing you can say about the forward test. It is not hard to find beautiful equity curves with optimization, but that says nothing about the profitability of your strategy or whether it even has an "edge" at all. The fastest way to know whether your strategy and your optimization method makes sense is by doing a Walk Forward Optimization (WFO) over the last few years. See here for a diagram.

To do a WFO manually, you should do what you have done in post #57 many times again on historical data. Run the optimizer of a 3-week interval and then use the optimal input parameters you found to backtest the subsequent week. Record its performance in a spreadsheet. Each time shift 1 week forward. For example:
Attached Image (click to enlarge)
Click to Enlarge Name: screenshot.png Size: 16 KB

This is repetitive, tedious work but is much faster than forward trading this approach for many weeks in a demo account. Remember: what is a forward test today is a backtest tomorrow. This assumes MT5 simulates spread, slippage and swap accurately enough, but you can easily try that out by running on a demo account for a few days and then compare that with a backtest over the same days.

Using this approach you might find that your strategy has no edge at all. Or you might find that your strategy does work, but you need to optimize over a larger time period to find better input parameters.
0 traders viewing now