Equity Curve, Lazy Man System Out-of-Sample
As evident from the above report, the system did not perform nearly as well during the out-of-sample testing. There are likely many reasons for this, but here are several that I’m considering:
1. The system is still working well but has just entered a drawdown or phase of under-performance. Or…
2. Due to the lack of more than 6 months worth of data, we do not know whether this type of under-performance is normal or abnormal. (More on this at the end) Or…
3. The system was curve-fit to the in-sample data set.
The system was tested on data from 2/16/2009 until today, 4/29/2009. The same parameters were used:
RSI Long Exit above 85 and RSI Short Cover below 10
Profit Target: $700.00
As I was looking over the trade-by-trade report, I noticed that not one time did the profit target trigger during this test. This means that every exit was triggered by the RSI level. Obviously, something changed between the in-sample and out-of-sample data. My guess is that the out-of-sample data was less volatile, meaning the profit targets were never reached. From December 2008 to the present, we know that volatility has decreased.
And this brings up an important point. When one uses static profit targets or stops, he or she risks that increasing or decreasing levels of volatility will render these static targets and stops ineffective, unless constant readjustment is applied.
Beyond the example demonstrated in this test, anyone using fixed-percentage stops and static profit targets during October and November of last year found out the hard way that both the stops and profit targets could have been doubled.
Why then would I use these stops and profit targets for this system? Primarily because Lazy was curious as to how the system would back test using them. I have to say that I was curious as well. In the end, it seems that this system should adjust with volatility.
These experiments could continue for much longer than most normal people would want to read about in a blog. From here, it is natural to wonder though: Which of the exits (besides RSI), if any, performed well over the out-of- sample results?
I will stop here though, as I think the Lazy Man system needs to marinate for a time, and I need to move on to other ideas. I will soon put all of the posts about this system on one page to make for easy reading / referencing.
It was suggested that a Monte Carlo analysis would help in evaluating whether a system is robust or not. I wanted to run a MC on this system, but had neglected to do so. Bman’s comment reminded me to do so. Below are a few graphs of the data from the first in-sample optimization.
Monte Carlo Simulated Equity Curves
This represents 1000 different equity curves with each curve being made of a random ordering of the actual system trade results.
Looks great, huh?
Max Drawdown Distribution
This graph shows that 95% of the trials had a percentage drawdown of 28% or less.
Looks great too, no?
It does, until these results are contrasted with the results from the out-of-sample tests where the system has experienced a drawdown from starting equity of 28.22%
Simply put, neither I nor the data know what we don’t already know.
The data set looks great, in sample, but the out-of-sample run produced is similar to one of the unlucky red lines in the very bottom-left quadrant of the equity curve graph. Look for the curves that immediately dip and lose money.
When I run the Monte Carlo on the out-of-sample data, the graphs demonstrate truly horrendous performances. The out of sample results show a system where 44% of the trials lost greater than half the starting equity. The Max Drawdown graph shows that 95% of the trials had a drawdown of 64% or less. This is quite a contrast to the Monte Carlo results from the in-sample data.
Without a doubt, more than six months of tick data is required for further development of Lazy Man’s system.
I will continue to monitor this system and post updated results and equity curves from time to time.