In the first post we examined a random stock selection method vs. quantifiable ranking methods. Those tests were run over a 10 year period.
In this post we will drill-down to test from 11/18/09 – 11/19/10. This length of time is important because it covers PDS picks since inception and represents in-sample results. During this time period, SPY posted a gain of 8.12% with a max drawdown of -16.09%.
The model used in the test is the 2% risk, 10% stop model. Commissions are included at .01/share. The model is not allowed to hold more than one open position in the same stock and there are no restrictions put on how many new purchases could be made at one time. De-listed, survivor free data was used.
As with the previous tests, I used the primary ranking method, the inverse of the primary, an alternate ranking method, and the inverse of the alternate method. I then ran 31 tests where stock picks generated by PDS were randomly selected.
A * denotes that the test selected ALKS. ALKS was a very large losing trade for the system in October 2010 after the FDA blindsided the company.
Highlights-
- The biggest affect on performance was whether or not the system was holding ALKS.
- Random28 traded ALKS and still generated an annual return of 24.11%
- The best performance, with a gain of 37.23%, was from the inverse of the alternate ranking method.
- The average of the annual return from the random selection tests was 14.82% with a standard deviation of 8.06%
- 26 (84%) of the random selection tests beat the performance of SPY over the same time period.
- Only 5 (16%) of the random selection tests performed worse than SPY over the same time period and 3 of those 5 underperformed by less than -2%.
- 13 (42%) of the random selection tests doubled (or better) the performance of SPY over the same time period.
Implications of the Results-
Without ALKS, the performance of the primary ranking method would be 16.26%, putting performance slightly better than the average return of the random selection methods. However, 16.26% is considerably less than the inverse ranking results. One possible explanation for this is that trading the system based on the primary ranking method traded out some of the performance. For one year, subscribers have (theoretically) traded the system based on the published rankings. It is certainly possible that this has reduced the edge for the highest ranked stocks.
To reduce this effect in the future, on PDS site, I will be detailing other methods for ranking the picks. It is obvious that the system can outperform the market with a random selection method, but I do not want to encourage subscribers to choose randomly as I believe they will introduce their own biases. Instead I will implement alternative ranking methods. Ultimately we want to spread buying out over a larger number of stocks to avoid a reduction of the edge.
Another explanation for why the primary ranking method has generated an average performance has to do with small sample sizes, both with the number of tests I ran and with the number of trades taken overall. Statistically speaking, we expect that after running more tests with random rankings and having more actual trades taken, eventually the results will approximate the results from the tests generated with 10 years worth of trades.
We must rank stocks, if only to track performance. I have to be able to replicate the strategy. I do not believe it is honest or transparent to use random rankings to demonstrate the performance of the system. From time to time a particular ranking method will be in sync with the market. Other times it may be out-of-sync. Over time, results will likely be smoothed.
The most important element is to follow a ranking method, period. If not, selection bias may become a significant problem.
Reading the Report–
- RAR = Risk Adjusted Return (CAR/Exposure);
- CAR = Compound Annual Return;
- DVR = R² of the Equity Curve * Sharpe Ratio
- SR = Sharpe Ratio
- R2 = A statistical measure of how close an equity curve is to a straight line plotted on a logarithmic graph. A value of 0 indicates a jagged line and a value of 1.0 represents a straight line. A fixed percentage investment that compounded and paid daily would have a straight line equity curve and an R Squared value of 1.0.
- %W = Percentage of Winning Trades
- %Avg = The Average Trade in Percentage Terms
- MaxDD = The Maximum Drawdown
- TT = Total Trades
About the Ranking Mechanism:
- PrimaryRank = The Default Ranking Mechanism used by The Power Dip System.
- Inv_Primary = The Default Ranking Mechanism in reverse. High ranked picks become low ranked and vice versa.
- Alt.Rank = An Alternative Ranking Mechanism
- Inv_Alt.Rank = The Alternative Ranking Mechanism in reverse. High ranked picks become low ranked and vice versa.
- Random = Picks are Selected at Random with no Ranking
Woodshedder,
That’s very interesting that the inverse of the primary ranking system fared the best. Obviously, there’s something about the market this year that’s different than most other years.
I agree with you that you need to use a ranking method. In my view, if you don’t use a ranking method, you don’t really know what the performance metrics of your system actually are (unless you could do some kind of Monte Carlo study where you run thousands of random tests and then get a kind of range of performance results). Even then, I think using a deterministic, non-random method is better. With random selection, there would always be the chance that your results would resemble something like the Random 23 test, above, and you’d have to accept it because that would be within the realm of probability. So it’s more difficult to know when the system is deviating from historical results due to the random noise introduced.
Data, I think it is helpful to know that the inverse selection method has fared better. When I initially saw that the inverse of the primary ranking mechanism did so well, I was worried that we had introduced slippage. Then, when I saw the inverse of the alternative ranking method (which no one is using), I realized it is likely something about the market environment. I am going to run these sorts of tests more often so we have an idea of what is working and can better choose what way to rank our trades. And I’m the King of selection bias, so I definitely need to follow a ranking mechanism!
Also, in the near future, I will be posting on the PDS blog some alternative ranking mechanisms for everyone to consider/use.
Hope you had a good Thanksgiving.
Woodshedder,
Ah yeah, I didn’t even think of us introducing slippage. I wouldn’t think we would getting too much slippage though. I always assumed the open/close would be fairly liquid times to trade (and I know my account certainly isn’t big enough to move the market!) Maybe with nanocap trades like RITT, though, it’s likely – although I suspect I’m the only whose foolish/crazy enough to actually be taking those ones.
But yeah, something about that market… hmm… maybe another blog post on that one would be interesting!
Look forward to the alternative ranking systems. I could see how you might want to use a different ranking system, as the current one does seem to favor the “riskier” picks. I will still probably go with the primary ranking system, though, if it turns out to have the best backtested results.