Category Archives: System Health
Perhaps the most difficult part of mechanical system trading is being disciplined and continuing to trade the system through good times and bad. At times, there can be significant doubt in the mind of the trader, and the urge to quit the system can be overwhelming. However, at some point, the system may indeed be broken, and in need of repair or trashing. How might one know when the system is experiencing a normal period of poor performance versus actually breaking down? Being able to answer this type of question will go a long way towards ensuring the mechanical trader can stick with the system through hard times and adjust or quit trading it when something is out of whack.
I will of course be using the Power Dip system as an example, primarily because I have a great deal of backtested historical data as well as 6 months of real-time trade data. These types of control charts will be available, updated monthly, for subscribers to the Power Dip system.
Historical Backtested vs. Real-Time
Data generated from backtesting is not the same as data created from trading a system in real-time. Perhaps the most significant difference for a system like the Power Dip is going to be with slippage. There are other differences as well, primarily concerning the fact that historical backtested data will rarely match real-time data, trade for trade. Usually the sequence of symbols will not match. This might be an issue for some systems without a lot of trades, but with over 4500 historical trades generated, and over 150 real-time trades, I feel strongly that the Law of Large Numbers will take care of the discrepancies. What I mean by this is that over enough trades, it should not matter that the historical sequence of trades does not match exactly the real-time sequence. Regardless, it is something to keep in mind when comparing historical backtested data to real-time.
Historical Backtested Data
The most important feature of the above chart is the large spike down in the 20TradeAverage, which occurs in March of 2000. Note that the 20TradeAverage approached -8% before recovering.
In the above chart, on the right-most side, is another spike down in the 20TradeAverage, which is approaching -8%.
What can be gleaned from these graphs? The system has experienced a previous period of under-performance similar to what it is experiencing currently. The exact date of the lowest point in the 20TradeAverage was March 17, 2000. What did the market look like in March, 2000, and are there any similarities to the current market?
Lets look at some charts of the Russell 2K index during that time.
First, a zoomed-out view of the Russell 2K from October 1998 to October 2000.
Here we see a market that has climbed ~50% in approximately 16 months. There have been some consolidations and pullbacks along the way, but the last 6 months of the rally, which saw gains of greater than 30%, was parabolic with only a few short and shallow retracements.
Now lets zoom in on this chart.
Above is a closeup of the Russell 2K from October, 1999, to June of 2000.
In the chart above, I’ve drawn a blue arrow pointing to March, 17th, which is the exact day of the low -8% reading on the control chart.
What we see in March of 2000 is a market that has experienced a tremendous rise, with very few pullbacks, in a relatively short amount of time. Our current market has risen even farther in percentage terms, and over a shorter time frame. Hence, I am not surprised that these two market environments, while 9 years apart, both look the same in terms of how they affect the system performance.
There are two important things to note about the above chart of the real-time trades.
First, as I noted earlier, these trades do not exactly match the historical backtested trades.
Secondly, (and of significant importance to me as I consider whether the system is still healthy, but just under-performing, or whether the system is breaking down) the real-time results are actually better than the backtested historical results covering the same period. The real-time results show the 20Trade Average dipping to -6%, which is better than the almost -8% shown by the historical results.
Therefore, the system appears to be healthy, but is having trouble in the current market, due primarily to the huge rise over a short period of time.
Other considerations for the Power Dip
As of Friday evening, there were only 183 stocks eligible to be selected for the Power Dip out of a total universe of 2,376 stocks. To understand how drastically this ratio has changed, on 9/22, there were 1,551 stocks eligible. Basically, the number of eligible stocks has fallen very quickly as the market rolls over. This information is very important, as no one wants to trade a system into Armageddon. If the market continues to roll over, the Power Dip will simply run out of eligible stocks, and will quit making trades, or it will significantly curtail trading until the environment improves.
System traders (and discretionary traders as well) must have a method for looking under the hood of their system/performance. What I have demonstrated is very simple, but it works.
The most important function of such analysis is that it lends confidence to the mechanical trader, ensuring that he doesn’t quit trading the system at the exact worst time.
The secondary function is to ensure the mechanical trader monitors the system’s performance for signs of breaking down. My experience is that systems do not break down all of a sudden. A breakdown generally happens over many months or even years. Using these types of monitoring techniques should give ample warning of when to be cautious about continuing to trade a system that is under-performing.
For the week, the Power Dip was up 0.53% while the [[SPY]] gained 1.52% (from last Friday’s to this Friday’s close).
Discipline is perhaps the most overlooked factor when considering how to successfully trade a mechanical system. For example, what should one do when his or her system is under-performing a benchmark? Should it be tossed to the curb? Does the under-performance mean it is broken? It may or may not be obvious, but I ask these questions more for rhetorical value, as the Power Dip continues to under-perform the indices. It is during these times that it is most important that one possess the discipline to stick with the system.
Control charts can be used to better understand what is happening with a system. Control charts give a “look under the hood,” allowing the mechanical trader a deeper understanding of system performance. If the control charts show everything to be within normal expectations, one may find it easier to stay disciplined when the system has difficult periods.
System Health Control Charts
Below I present two charts.
The first shows two averages: the Total Average percentage gain/loss and the 20 Trade Average percentage gain/loss.
The second also shows two averages: the Total Average win percentage and the 20 Trade Average win percentage.
Note the Law of Large Numbers in action as system takes 80 trades before the average trade approximates the backtest generated average trade. In other words, the TotalAvg is volatile until it is smoothed by more and more data.
The 20TradeAvg remains volatile as it will only average the last 20 trades.
The chart shows that the system has seen a worse period, where 20 trade average dipped to ~2.6%. The system recovered quickly and peaked with an average trade near 3.5%.
The Win Percentage graph is interesting as it shows that the system beat historical expectations of percentage of winning trades for a span of approximately 70 trades (or half the number of trades made since I’ve tracked in on the blog).
How To Interpret the Charts
The charts show that the Power Dip oscillates between over and under-performing. This is normal. Think of it as reversion to the mean. As long as the total average is consistent with historical expectations, and the 20TradeAvg is oscillating above and below the total average, everything is okay.
To get an even better understanding, we would use the data generated from backtesting and then add our real-time data for comparison. The results will not match, due primarily to slippage, but it will give a long history so that we get an idea of what is normal in terms of the oscillation of the short average around the long average.
These types of calculations can also be used to turn systems off and on, but that is beyond the scope of tonight’s post.
Below is the updated spreadsheet, with the metrics for all trades on top and October’s trades below.
The system has open Flowers Foods, Inc. [[FLO]] , DISH Network Corp. [[DISH]] , and Silver Standard Resources Inc. (USA) [[SSRI]] . It will add two more positions on Monday’s open.