You wrote a system that you think is okay, and now you are ready to go. Not so fast. Having a system means nothing unless you "know" it works. There are several ways to figure out whether a trading strategy works. The first is not much fun when it doesn't work: using real money to test it. The second way is to paper trade with the system for a few months before risking money with it. The most effective way, though, is to backtest it on historical data. That means taking your system and checking to see how it would have done if you had used it on past markets. Not too long ago, this meant sitting in front of a chart and doing it by hand, but now, with more sophisticated and powerful software for system writing and testing, there is no excuse for people not to have properly backtested systems.
By backtesting you should get an idea of whether the system can work in the future. Don't fall into the trap of thinking you have a good system but never bothering to test it. If it didn't work on the backtested data, don't expect it to work in real time when you are risking money. By backtesting you give yourself a fighting chance with a system before putting your money at stake. If you find that your system is borderline or find that it doesn't work, throw it out before it costs you anything other than time.
Be careful because backtesting will not tell you how a system will perform in the future. Even a system that performs perfectly when backtested can fail miserably in real time. However, by thoroughly backtesting, you can become aware that it could have a 10-trade losing streak or a drawdown that lasts for 2 months and costs you $10,000. If that happened in the past, there is no reason to expect it to act differently in the future. Knowing the worst that could happen may prevent you from using a costly system altogether or from bailing on a good one after a few bad trades because you are frustrated with it. It is normal for systems to have losing streaks, and so it's important to know how big they can be.
Before I get further into backtesting, I want to spend some time on common mistakes and things that should be avoided when one is backtesting. Sometimes it is easier to learn what to do by learning what not to do. By not doing the wrong things you have no choice but to do the right ones. For example, my cat Sophie doesn't really know she is supposed to scratch only her scratching post, but she has learned she is not supposed to scratch the couch, drapes, carpet, furniture, or my leg. Her options have been narrowed to only the proper ones. Now I just need to figure out how to get her to stop thinking 5 a.m. is the proper time to lick my face good morning and she'll be fine. After I go through the don'ts, I'll explain the proper things to do as well, but for now let's get the bad habits out of the way.
The most basic mistake someone can make after backtesting a system is not knowing how to evaluate the results. Unless one can properly do this, it is impossible to tell how a system stacks up. Some traders look for the highest net return or for a high winning percentage when they backtest, but these things may not mean much if the system also has large drawdowns. You have to factor in a few other things when evaluating systems, including number of trades, profit per trade, consecutive losers, biggest loser, biggest winner, average trade, and distribution of returns. It is a combination of all these things that eventually will tell a trader if a system is good and how it compares to other systems.
An important thing to look for in evaluating a system is whether the results were real or curve-fitted. Curve fitting in a nutshell is matching a system to the data. This is a problem of both system writing and backtesting. If you have a strongly trending chart and you write a system to buy and hold, you've curve-fitted the system to the data. Sometimes traders focus on the area of a chart that confirms their opinions and skip over places where the trades may not have worked as well. They then come up with a system that works well, but only on that portion of the data; if they don't test it on other data, the system shouldn't be used. Overoptimization is another problem for system writers. Those who keep fudging a system's parameters until it works perfectly are guilty of this. These people will try every combination of moving averages until they find the ones that work the best. This doesn't make for a good system but for one that will work on the data on which you tested it. When you backtest a system, you must take this into account. You need to realize that the more you have played around with a system's parameters so that it gives you better results, the less likely it is that it will work in the future.
Not questioning the results is also a mistake traders make. They see decent results and are happy with them. Instead, they should try to find ways to break the system, to find the faults, abnormalities, or curve fitting that may make it seem better than it really is. Maybe the system showed great profitability because of one or two large trades. Will this system perform well if the same condition is not present in the future? Were you realistic about slippage? Were there enough trades to test it properly? If you don't question a system's results and deficiencies, you may be in for a surprise when actually using it. This is also true about a system that does not seem to work. Dig into it to see why it doesn't work and try to find out how you can make it better. You can learn a lot about trading by discovering what doesn't work. Maybe you have a theory that is no good and can abandon it, but you can do that only by questioning it thoroughly first.
Another common mistake in testing a system is not having enough data. The minimum number of trades one should have to make a test statistically acceptable is 30, but the more the better. If you do not have sufficient data you will not know if the results are valid or occur by chance. If you write a system and get only six signals when testing it and five of the signals work, you can't make a judgment about it. This may have been a winning streak that could soon be followed by a larger losing streak. With 30 samples it is fair to say that your system is valid and that the results were not just a matter of luck. You also need enough data to cover all types of market environments, not just the ones you like best. You need to know if your system works in uptrends, downtrends, choppy markets, volatile markets, flat markets, and so forth. Test your system on many markets. If it is good, it should work across the board in every market.
When you are testing on intraday data, don't test just a few months; instead, try to get a few years' worth of data. A year of data is a start, but it is not enough to base a system on. Many systems do great for a year yet act horribly over a 3-year span. Getting intraday data can be expensive and time-consuming when you are testing on futures, as each contract has to be tested individually, but trading is not cheap or easy; putting in the time and money will get the best results.
One problem of not having enough data to test on is that it leaves you with little data to use as an out sample. When you are testing a system, it should be done on a portion of data that has not been used to write or optimize the system. After you think you have a good system and it is ready for the final test, it should be done on fresh, never before seen data. Some systems get optimized to fit the data and seem great, but they are never tested on fresh data to see how they would do. Although testing on data that was used to optimize the system will produce good results, the trick is to get the same results on clean data. This is where the out sample comes into play. Testing on fresh data is the closest you will get to trading in real time without having to risk money.
Many traders ignore slippage and commissions when testing their systems. They may end up thinking they have great systems, but in reality they don't because they forgot to add or underestimated trading costs. Be realistic about how much it will cost you to trade or you will be in for a surprise when it comes to real trading. Every trade will cost you money in commissions and slippage, and so you must consider these costs when testing a system. A lot of the time traders don't allow enough for slippage. They think they will get filled wherever the signal was given, but I can't tell you how many times I put in a market order for a stock and got 30 cents to a dollar worse than I had hoped to get. If you remember my unexpected rate cut disaster, I had slippage close to $5 per share on several stocks. Trades that on paper may look fantastic in reality turn out to be small winners or losers, and systems that seem profitable can easily be losing ones when you consider commissions and slippage. Don't ignore them or your results will not be accurate.
I'll start with curve fitting and then move on to optimization. Earlier, I described curve fitting as matching a system to the data. Curve-fitting a system makes it look great for a given period of time and for specific data. If you eyeball a chart and see that it is choppy, you can write a system that works well over that period. You also could write a filter that lets you be short just before a market collapses. It would have great results but this is deceiving. Will you really catch the next major collapse with the same signal? You probably could keep tweaking the system until it fit the data perfectly, but the problem with this is that the system is made for that data and probably will not work as well over any other data. You probably could come up with a winning system with a 2000 percent return over the data sample for any chart you look at if you tried hard enough, but this is useless if the market behaves differently in the future. As a trader you want to be concerned about how it will work on new data, not about how it worked in the past. This is why after you are happy with the system it is critical to test it on a fresh piece of data called the out sample, as I'll describe soon. The only true test of a system comes when it works on data on which it wasn't written or optimized. Systems that have been curve-fitted look great when backtested over the data they were created on but may give disappointing results in real time. In general, the more complicated and detailed a system is and the better the results are, the more likely it is that it has been curve-fitted.
Optimization is what happens when a system writer keeps searching for the best parameters and indicators for a system over a specific time period. If he is using moving averages, he will keep trying different period ones until he gets the ones that show the best results. If he is happy with a set of moving averages, he may then look to find the best parameters for a filter on his breakout level that will make the system perform even better. A person can go on forever playing around with parameters and indicators, always trying to come up with something better. With TradeStation this is real easy to do as the software does all the optimizations for you. In seconds it spits out the best parameters for any indicator you want. Though this may seem like a great feature, it can mislead you into believing a system is better than it is.
In optimizing, the goal is to increase the profitability of the system, but be careful not to overdo it. As long as the basic concepts of a system are sound, it doesn't matter what values are used for the parameters. What you hope to find when you optimize is a range of parameters that seem to work best. Maybe you are looking to see when the best breakouts of the highest highs occur. By optimizing you may find that as you use a longer lookback period, the system gets more profitable, but that once you get above 20 bars, the improvement is not much. Then you'll know that you want your lookback period to be about 20 bars instead of 5. Trying to pick the ideal one is useless because it will keep changing with different data.
When optimizing, I like to look for a range of parameters that work best and have similar results, and then I'll take the average. If I find that the 12-, 14-, and 17-period moving averages work best compared with all the others, I most likely will stick to the 14- or 15-period one. If I find the 12-, 14-, and 17-period ones worked well but that the 15- and 16-period averages did not have good results, there is something wrong with the system. A system should work whether the moving average is 5, 7, 10, or 15 periods. I don't care that the 14-period average worked best for this data set. I need to know that the overall concept worked, not that one period was a little better than all the rest. If only one or two parameters worked well, the system is not very good and has probably been curve-fitted to the data.
What I find optimizing to be good for is to see if the system works over different parameters or if it was mere luck that it worked on a specific set of parameters. Did the randomness of the market make something work in the past that may not work again? Was a buffer zone used on a breakout system to avoid one trade that was caused by random noise? Should this trade be taken to reflect reality? Some people will look at a chart and then pick the filter that corresponds perfectly to that chart. Try different parameters on the filter to determine if it was a fluke occurrence. Test it over different periods of data to see if it responds the same; you can even test it over different time frames of the same data to get a different perspective. When you are happy, go to your out sample and test the system there. The results should be similar.
Probably the most important part of backtesting is having an out sample on which to test your results. Before going live with a system, make sure you test it on fresh data that was not used in making and optimizing the system. One common mistake made by beginning traders is that they write, test and optimize their system by using all the data they have available. If they have 3 years of data, they use all of it to find the best parameters without understanding the concept of having an out sample. So for those who don't know, if you have 3 years of data, use only 2 of them when working on the system. Don't look at the last year of data, not even a chart of it; this will be your out sample, and you don't want it to influence your thinking in the least. After you are happy with the system and it has been optimized and tested, and then reoptimized and tested again until you are even happier with the results, then and only then go to the data you have ignored. Now take this data (the out sample), which should be about one-third of all your data, and test the system on it. At the very least make sure the out sample is big enough to give the 30 trades necessary to make it statistically valid. If the system is good, it should work on the out sample; if it doesn't, you need to go back to the drawing board to make adjustments or throw it out. Use the out sample to see what went wrong but never try to optimize on the out sample or you will defeat the purpose of having it. I like to look at the chart of the out sample to make sure the system acted as it was supposed to. However, be careful not to take the out sample into consideration if you decide to tweak the system some more.
By using an out sample you are allowing the system to see fresh data, reducing the chance that your results were due to overoptimizing. Since the optimized parameters never saw the new data, if the data works on the system, it is not because the system is curve-fitted to the data but because you may just have a good system on your hands.
First of all make sure you have enough data. Testing a system over 6 months is meaningless, as anything could happen in that short a time period to skew the results. Also, keep in mind that 30 is the minimum number of trades you need in order to get statistically sound results. The fewer trades one has, the less reliable the results are. When you don't have enough trades, one or two extreme moves can throw the numbers out of whack, making a system that seems great only mediocre as more trades are taken into consideration. You also want to see how the system would have done in several different market environments, so make sure you have all the data you need.
The best way to use data is to start by writing your system using the first third of the data. Once you are happy with the basic system, go to the second third of the data to fine-tune it, optimize it a bit, and make the final adjustments. After you are done with that, test it on the last third. Another option could be to write and optimize the system on the middle two-thirds of the data and then use the beginning and ending sixths as the out sample. Whichever method you choose, what you hope to accomplish is to have percentage results in the out sample that are similar to those in the writing stage. This will tell you that the system works consistently over time.
Make sure that the different sets of data aren't all from the same type of trading environment. You also can test on different markets, stocks, and time frames as part of the out sample. If I have a system I wrote using IBM as my data, I will test it on Cisco, Merrill Lynch, Intel, Wal-Mart, the Dow, and the S&P 500 Index. If it works well, it should work on every stock, not just one. Try to make sure you are mixing up the markets and market conditions. Don't test the same system on 10 stocks that have the same chart pattern. Find stocks that are going up, down, sideways, and anything in between, as this will give you a more realistic representation of what can really happen.
Now comes the nitty-gritty part of backtesting: how to read the results and determine if you have a good system with a positive expectancy. A positive expectancy is needed to determine if a trader has an edge; without an edge one should not be trading. Charts 13—1 and 13—2 are typical printouts from TradeStation that evaluate two different canned systems in TradeStation. Though both are profitable, the second one, MACD, is much better not just because it made more money but because, as I'll explain, it met many other, more desirable criteria. I've purposely not included slippage and commissions in these results. I'll demonstrate later how big an impact they can have.
CHART 13—1
Test Results for Stochastic Crossover (System 1)
CHART 13—2
Test Results for System MACD (System 2)
Total net profit is the overall bottom line of a system: Does it make money? These two examples are profitable, showing net profits of $7025 and $32,750. If these results had been negative, it would be back to the drawing board, as you couldn't expect to make money by using the system. Total net profit is something all people will look at when evaluating a system, but by itself it is not that great of an indication. Yes, you want a system that has a net profit and not a loss, but you also need to know how many trades it made, how big the swings were, how large a drawdown there was, what the average trade made, and so on. A trader may be tempted to jump into a system that shows a $50,000 return rather than one that shows only a $10,000 return, but the second may be a better system. Maybe the first one made 1000 trades in a year with a drawdown of $35,000; maybe some months had large losses and others had large wins. Meanwhile the second system made only 50 trades with a drawdown of only $3000 and showed a small profit every month. In that case the second system is a better, safer system even though it returned less. You need to decide what is more important: large potential profits or safe, steady returns. A smart trader will always choose the latter.
It's easy to be misled if one used only total net profit as a criterion for choosing a system. In Chart 13—1 the stochastics system seems okay at first glance, making over $7000 in this 3-month period, but when one digs deeper, it really is not a good system.
In choosing between two systems that have similar results, stick to the one that has a lower number of trades in the same time period. This will reduce the damage done by slippage and commissions, which can be substantial. A system that trades less may be boring for some people, but if you can get the same results, always take the one with fewer trades. Not only does System 1 have a lower net profit, it makes 3 times as many trades as System 2. This means you are working harder to get worse results. Though making fewer trades is desirable, make sure your test has at least 30 trades in it or there is a decent chance that your results were due to chance. If you don't have 30 trades, you will need to get more data to test it properly.
This number means so little, yet people take so much notice of it. The best traders make money only 50 percent of the time, yet in real life people see 50 percent as failure. Any test one took in school required 65 percent for passing. If people see a system with a 40 percent win/loss ratio, they instinctively think it is a loser because they are conditioned to think of it that way. Instead, think of it like a good hitter in baseball; a .400 average is not too shabby. I don't pay much heed to percent profitability, but some traders may not feel comfortable using a system with only a 40 percent win/loss ratio. It doesn't matter if a system is right 30 percent, 40 percent, or 60 percent of the time. What is important is how big the average loser is compared to the average winner. If one has good risk management skills, even a system with a 30 percent win/loss ratio can be quite robust. Both of the systems here have percent profitability numbers in the low 40s, and this is what I typically shoot for.
This is something that I look for to see how valid a system is. First, I look to see if the total profit of the system was due to just one or two trades. In System 1, the total profit was $7025 and the largest winner was $7000. Take away this one trade and the system is profitable by $25 over 182 trades, which is not very good. If a system doesn't perform well after you take one or two of the largest trades away, it is not very reliable. The other thing to look for is that the largest losers are not bigger than the winners. The key to trading is to keep your losers smaller than your winners; the opposite situation can lead to disaster. If the largest losers are too big, you need to work on your exits and stops some more. I look for at least a 2:1 or 3:1 ratio between the largest winner and the largest loser but will settle for a 1.5:1 ratio before I'll consider trading a system. The same ratios hold for average winning trade versus average losing trade; unless the average winning trade is bigger than the average losing trade, I won't use the system. I also look to hold my losers shorter than my winners, so I like to look at the average number of bars in winners versus those in losers to make sure the system fits my trading methodology.
How many losers in a row were there? Many traders can't handle 10 bad trades in a row and will abandon a system before it has a chance to work. By knowing how many losers in a row this system has had, you can decide not to trade the system because it may be more than you can stomach. Or if you are trading it and are going through a losing streak, you may stick with it because you know that it is common to have a six-trade losing streak. If you don't know what the largest consecutive losing streak was, you may abandon a system after four straight losses even though this is considered normal for the system.
This is one of the more important numbers to look at as it can compare two systems to each other or one system to itself when changes are made. The average trade measures how the system does on a per-trade basis. This is the number that tells you how much on average you will make or lose every time you trade with the system. System 1 has an average trade of $38.60 (and this is before commissions), while System 2 has an average trade of $564.66. It doesn't take a genius to realize that it is more profitable on average to take a trade on the second system. If the average trade is negative, don't trade with the system; that's easy enough to figure out. What's harder is determining when the average trade is positive but too small to be worth using a system. Each trader must find an average trade he is comfortable having and then not trade systems with a lower amount.
Probably the most important factor in a system is the drawdown. The drawdown will tell you how much money you will need to start trading with this system in a particular market and give you a basis from which to measure risk. It tells you how much it would have cost you at its worst to use this system. This number will give you an estimate of how much starting capital is needed for each stock or market traded. You may think you have something good, but when you test it over extensive data, you may find that it would have lost $25,000 during its worst period. Don't think it won't happen again; the worst losing streak is always just around the corner. If you can't afford to go through a losing streak twice as long as the biggest drawdown, you shouldn't trade with the system.
Risk-averse traders are more likely to look at the drawdown than at any other statistic. If they can't stomach the drawdown they'll abandon the system or make changes to limit the losses. Money management plays a huge role in trading and should be considered in every aspect of trading. If two systems have similar results and one has a smaller drawdown than the other, it is probably less risky. If a system is too risky, one should avoid it.
The profit factor is total gains divided by total losses. It tells you the amount that will be made for every dollar that is lost. If the profit factor is 1, you are breaking even. To be on the safe side, the profit factor should be at least 1.5. If you have a profit factor of more than 2, you have a very good system. In System 1 the profit factor is barely 1, and so this is a system that should be avoided. The profit factor of 1.64 in System 2 is considered decent, and so one could feel comfortable trading with it.
Finally, one needs to see how volatile a system is. How did the system make money? Was it a steady flow of good returns, or were there wild swings in the system's equity over time? If you have enough data, look for steady monthly performance; if you are testing an intraday system, see how it does on a daily basis. A system with a steady positive return day after day with little variance from the mean is better than one that makes more but has wild swings. If the standard deviation is too wide, this may not be a good system to use, as the drawdowns could be large. If too many trades, days, or months fall more than 2 standard deviations outside the mean on the losing side, I'd be cautious about using the system. Stick to systems that have smaller equity swings, as they are more reliable. TradeStation will give you a month-by-month or trade-by-trade breakdown, and so you can see if the performance was steady. You also can use Excel or another statistical program to get these results. It's not a simple task, but it's one that a serious trader wants to do to make sure his system does not have wild swings.
No one likes to talk about commissions and slippage, but these two items can really make the difference between a winning and a losing system and a winning and a losing trader. One thing to consider when developing a trading style is that every trade that's made–win, lose, or break even–will cost a trader some money. These are costs a trader can't do anything about, and they need to be considered in his system design or his systems will be unrealistic. First, one needs to include every trader's favorite thing, commissions. The second cost, slippage, is something some traders tend to (or would like to) forget about when considering their trading decisions. Basically, slippage is the cost associated with paying more for or selling for less than what a trader intended. It may be caused by the market moving away or may be due to the difference in the spread between the bid and ask prices. Ideally, a trader would like to buy on the bid and sell on the offer. Unfortunately, he tends to do the opposite, putting himself into a losing position from the start. On a per trade basis, commissions and slippage are trivial, but when added up over the course of time, they can be astronomical and can have a huge impact on a trader's profit and loss (P&L) statement. They can easily turn seemingly profitable trades into losing ones.
When designing a system a trader needs to make sure it will cover his trading costs or else it will become a losing system. If I go back to my original two systems and add modest commissions of $15 per round turn and slippage of $100 a trade (again, this is very modest for the S&Ps), you can see the dramatic change in the two systems (Charts 13—3 and 13—4). System 1 ends losing close to $14,000 instead of making $7025, while System 2 drops from over $32,000 to $26,515. System 2 is still a good system and has kept most of its desirable parameters, while System 1 should be avoided at all costs. The large number of trades in that system makes it hard for it to be profitable.
To become a better trader you must backtest your trading ideas and systems properly before you trade them. If you do not do this, you will never know if you are trading with a sound system until you are risking real money. If your system is not profitable, you don't want to find out by losing money. You are better off learning this by spending the time to backtest it. To backtest a system properly you will need to have enough data to give you 30 sample trades and cover all the different market conditions. You don't want to test a system just on a trending market, not knowing how it will react in a choppy market. If you don't test over different conditions, you are not backtesting properly and may end up with a system that was curve-fitted to work in a trending market.
CHART 13—3
System 1 Results with Trading Costs Added
CHART 13—4
System 2 Results with Trading Costs Added
You should use different data when writing a system and when optimizing your parameters. This will lessen the chance of curve fitting around the data. When you are ready for a final test, make sure you do this on an out sample of completely new data, preferably covering different market conditions and long enough to be statistically valid (30 trades). One of the worst mistakes a trader can make is writing, optimizing, and testing the system on the same data. If it's been optimized for a set of data, of course it will work great, but you will never know how it will work in the future. One thing to keep in mind is that no matter how great a system did when it was backtested, it can never predict the future, as markets change. After you are happy with the results you may want to go over a chart visually and see where trades would have been made to get a good feel for how the system worked. You also want to check that your results weren't due to one or two strong wins. You want to trade with a system that is more predictable and steady over time. These are the kinds of systems that end up doing best; ones with wild swings can be unpredictable and dangerous.
The last thing you need to do is to understand the backtesting results and know how to compare different systems. Overall profit and win/loss ratio are not as important as the average win and the profit factor. Look at the drawdown: Can you afford to trade the system? Don't assume that the drawdown won't happen right away, because it might. Make sure to account for commissions and slippage when you are figuring out total profits, as they will make a big difference in the results.
Take your time when backtesting. Don't ignore it or get lazy about doing it, as it is a crucial part of making you a better trader. And never trade a system without backtesting it thoroughly first.
1. Not backtesting
2. Not knowing if your system will make money
3. Trading before you have backtested your system or trading methodology
4. Not knowing how to evaluate backtesting results
5. Not questioning the system and results
6. Getting hung up on the win/loss ratio
7. Putting too much emphasis on total profit
8. Ignoring drawdowns
9. Curve fitting the system to the data
10. Overoptimizing until it is perfect (for the data you optimized on)
11. Not testing on enough data or market environments
12. Not having a proper out sample
13. Not testing on enough markets
14. Ignoring commissions and slippage
Getting the Most Out of Backtesting
1. Use software that will do the testing for you.
2. Never trade with a losing strategy again.
3. If you are not happy with the results, don't trade the system.
4. Make sure you have enough data.
5. Have a sample of at least 30 trades.
6. Do final test on an out sample of fresh data.
7. Save at least one-third of the data for your out sample.
8. Test over different market environments.
9. Test the same system in different time frames.
10. Test the same system in different markets.
11. Don't overoptimize or curve-fit.
12. Know how to evaluate backtesting results.
13. Learn to compare different systems.
14. Don't underestimate commissions and slippage.
15. Make sure you can afford 2 times the largest drawdown.
17. Make sure all the profits didn't come from one or two trades.
Helpful Questions to Ask Yourself
Did I backtest my system or ideas properly ?
Did I overoptimize my system?
Is the system curve-fitted?
Did I test on an out sample?
Does the system have a positive expectancy?
What does the average trade make?
Was I realistic about commissions and slippage?