ply any actual shortage, but rather an upward shift in the
market’s perception of gold’s value. Supplies of virtually any level are still available—at some price. This is not
true for most commodities, in which there is a definite relevant limit in total supplies.
285
SElECTINg THE BEST FuTurES PrICE SErIES For SySTEM TESTINg
far exceeded the net price gain implied by nearest futures, and the subtraction of the cumulative adjust
ment factor from the most recent (2015) prices would result in negative prices for the majority of the
time before 2009. Such an outcome is unavoidable if the continuous futures price series is to refl ect the
net gain in a continually held long position and if the series is shifted by the constant factor necessary to
set the current continuous futures price equal to the current contract actual price.
 Although the fact that a continuous futures price series could include negative prices may sound
disconcerting, it does not present any problems in using the series for testing systems. The reason
for this is that in measuring the profi ts or losses of trades, it is critical that the price series employed
accurately refl ects price changes, not price levels. However, it also will often be useful to generate the
actual prices that correspond to the continuous futures prices in order to facilitate such applications
as checking trading signals against actual contract charts.
 It should also be noted that the transition between contracts need not occur on the last trading
day, as is the conventional assumption in the nearest futures price series. In fact, because physically
delivered contracts are particularly vulnerable to distortions in their fi nal weeks of trading due to
technical concerns regarding delivery, it probably makes sense to avoid these prices in constructing a
continuous series. It follows, then, that one should use a rollover date before the last trading day (e.g.,
20 days prior to the last trading day).
 ■ Comparing the Series
 It is important to understand that a linked futures price series can only accurately refl ect either price
 levels, as do nearest futures, or price moves , as do continuous futures, but not both—much as a coin
can land on either heads or tails but not both. The adjustment process used to construct continuous
 FIGURE  18.1 “Negative” Prices in a Continuous Futures Chart: Soybean Continuous Futures
Chart created using TradeStation. ©TradeStation T echnologies, Inc. All rights reserved.

286
A Complete Guide to the Futures mArket
series means that past prices in a continuous series will not match the actual historical prices that
prevailed at the time. However, the essential point is that the continuous series is the only linked
futures series that will exactly reflect price swings and hence equity fluctuations in an actual trading
account. Consequently, it is the only linked series that can be used to generate accurate simulations in
computer testing of trading systems.
The preceding point is absolutely critical! Mathematics is not a matter of opinion. There is one
right answer and there are many wrong answers. The simple fact is that if a continuous futures price
series is defined so that rollovers occur on days consistent with rollovers in actual trading, results
implied by using this series will precisely match results in actual trading (assuming, of course, accu
rate commission and slippage cost estimates). In other words, the continuous series will exactly paral
lel the fluctuations of a constantly held (i.e., rolled over) long position. All other types of linked series
will not match actual market price movements.
T o illustrate this statement, we compare the implications of various price series using the sideways
gold market example cited earlier in this chapter (i.e., gold hovering near $1,200 and a forward/
nearby contract premium equal to 1 percent per two
month spread). A trader buying a oneyear for
ward futures contract would therefore pay approximately $1,273.82 (1.016 × $1,200 = $1,273.82).
The spot price would reflect a sideways pattern near $1,200. As previously seen, a 60day constant
forward price would reflect a sideways pattern near $1,212 (1.01 × $1,200). A nearest futures
price series would exhibit a general sideways pattern, characterized by extended minor downtrends
(reflecting the gradual evaporation of the carrying charge time premium as each nearby contract
approached expiration), interspersed with upward gaps at rollovers between expiring and subsequent
futures contracts.
Thus the spot, constant
forward, and nearest futures price series would all suggest that a long
position would have resulted in a breakeven trade for the year. In reality, however, the buyer of the
futures contract pays $1,273.82 for a contract that eventually expires at $1,200. Thus, from a trading
or real
world viewpoint, the market actually witnesses a downtrend. The continuous futures price is
the only price series that reflects the market decline—and real dollar loss—a trader would actually
have experienced.
I have often seen comments or articles by industry “experts” arguing for the use of constant

forward (perpetual) series instead of continuous series in order to avoid distortions. This argument
has it exactly backwards. Whether these proponents of constant
forward series adopt their stance
because of naïveté or self interest (i.e., they are vendors of constant forwardtype data), they are
simply wrong. This is not a matter of opinion. If you have any doubts, try matching up fluctuations
in an actual trading account with those that would be implied by constant
forwardtype price series.
you will soon be a believer.
Are there any drawbacks to the continuous futures time series? of course. It may be the best
solution to the linked series problem, but it is not a perfect answer. A perfect alternative simply
does not exist.
one potential drawback, which is a consequence of the fact t


an actual trading account with those that would be implied by constant
forwardtype price series.
you will soon be a believer.
Are there any drawbacks to the continuous futures time series? of course. It may be the best
solution to the linked series problem, but it is not a perfect answer. A perfect alternative simply
does not exist.
one potential drawback, which is a consequence of the fact that continuous futures
accurately reflect only price swings, not price levels, is that continuous futures cannot be used for any
type of percentage calculations. This situation, however, can be easily remedied. If a system requires
the calculation of a percentage change figure, use continuous futures to calculate the nominal price
287
SElECTINg THE BEST FuTurES PrICE SErIES For SySTEM TESTINg
change and nearest futures for the divisor. Also, there is some unavoidable arbitrariness involved in
constructing a continuous series, since one must decide which contracts to use and on what dates
the rollovers should occur. However, this issue is not really a problem since these choices should
merely mirror the contracts and rollover dates used in actual trading. Moreover, there is arbitrariness
involved in the use of any of the price series discussed. Finally, in some markets, the contracts being
linked together may have very different past price patterns (as is often the case in livestock markets).
However, this problem would exist in any kind of linked series.
 ■ Conclusion
For the purpose of computer testing of trading systems, there are only two types of valid price
series: (1) individual contract series and (2) continuous futures series. Individual contract series are
a viable approach only if the methodologies employed do not require looking back more than four or
five months in time (a restriction that rules out a vast number of technical approaches). In addition,
the use of individual contract series is far clumsier. Thus, for most purposes, the continuous futures
price series provides the best alternative. As long as one avoids using continuous prices for percent
age calculations, this type of price series will yield accurate results (i.e., results that parallel actual
trading) as well as provide the efficiency of a single series per market. Again, I would strongly caution
data users to avoid being misled by those who argue for the use of constant
forwardtype series in
computer testing applications. If your goal is a price series that will accurately reflect futures trading,
the constant
forward series will create distortions rather than avoid them.
over the years more—but not all—data vendors and systemtesting platforms have embraced the
continuous futures series described here as the default data type for long term analysis and system
testing. Traders should nonetheless confirm with the vendor that longterm futures data series link
ing different contracts are indeed constructed using the continuous futures (i.e., spread adjusted)
methodology. Traders should also be cognizant of the contracts and rollover dates used by the vendor
so that they can match their contract selection and rollover dates accordingly. V endors should be able
to provide a clear explanation of the methodology they employ for constructing long
term (i.e.,
linkedcontract) futures data series.

289
Chapter 19
Every decade has its characteristic folly, but the basic cause is the same: people persist
in believing that what has happened in the recent past will go on happening into the
indefinite future, even while the ground is shifting under their feet.
—George J. Church
 ■ The Well-Chosen Example1
Y ou’ve plunked down your $895 to attend the 10th annual “Secret of the Millionaires” futures
trading seminar. At that price, you figure the speakers will be revealing some very valuable
information.
The current speaker is explaining the Super-Razzle-Dazzle (SRD) commodity trading system.
The slide on the huge screen reveals a price chart with “B” and “S” symbols representing buy and sell
points. The slide is impressive: All of the buys seem to be lower than the sells.
This point is brought home even more dramatically in the next slide, which reveals the equity
stream that would have been realized trading this system—a near-perfect uptrend. Not only that but
the system is also very easy to keep up.
As the speaker says, “All it takes is 10 minutes a day and a knowledge of simple arithmetic.”
Y ou never realized making money in futures could be so simple. Y ou could kick yourself for not
having attended the first through ninth annual seminars.
T esting and
Optimizing
Trading Systems
1 The following section is adapted from an article that first appeared in Futures magazine in September 1984.
290
A Complete Guide to the Futures mArket
Once you get home, you select 10 diversified markets and begin trading the SRD system. Each
day you plot your equity. As the months go by, you notice a strange development. Although the equity
in your account exhibits a very steady trend, just as the seminar example did, there is one small
difference: The trend on your equity chart is down. What went wrong?
The fact is you can find a favorable illustration for almost any trading system. The mistake is in extrap-
olating probable future performance on the basis of an isolated and well-chosen example from the past.
A true-life example may help illustrate this point. Back in 1983, when I had been working on
trading systems for only a couple of years, I read an article in a trade magazine that presented the
following very simple trading system:
 1. If the six-day moving average is higher than the previous day’s corresponding value, cover short
and go long.
 2. If the six-day moving average is lower than the previous day’s corresponding value, cover long
and go short.
The article used the Swiss franc in 1980 as an illustration. Without going into the details, suffice it
to say that applying this system to the Swiss fr

eBooks\A Complete Guide to the Futures Market Technical Analysis, Trading Systems, Fundamental Analysis, Options, Spreads\A Complete Guide to the Futures Market Technical Analysis, Trading Systems, Fundamental Analysis, Options, Spreads.pdf


e trading system:
 1. If the six-day moving average is higher than the previous day’s corresponding value, cover short
and go long.
 2. If the six-day moving average is lower than the previous day’s corresponding value, cover long
and go short.
The article used the Swiss franc in 1980 as an illustration. Without going into the details, suffice it
to say that applying this system to the Swiss franc in 1980 would have resulted in a profit of $17,235
per contract after transaction costs. Even allowing for a conservative fund allocation of $6,000 per
contract, this implied an annual gain of 287 percent! Not bad for a system that can be summarized in
two sentences. It is easy to see how traders, presented with such an example, might eagerly abandon
their other trading approaches for this apparent money machine.
I couldn’t believe such a simple system could do so well. So I decided to test the system over a
broader period—1976 to mid-1983
2—and a wide group of markets.
Beginning with the Swiss franc, I found that the total profit during this period was $20,473. In
other words, excluding 1980, the system made only $3,238 during the remaining 6½ years. Thus,
assuming that you allocated $6,000 to trade this approach, the average annual percent return for
those years was a meager 8 percent—quite a comedown from 287 percent in 1980.
But wait. It gets worse. Much worse.
When I applied the system to a group of 25 markets from 1976 through mid-1983, the system
lost money in 19 of the 25 markets. In 13 of the markets—more than half of the total survey—the
loss exceeded $22,500, or $3,000 per year, per contract! In five markets, the loss exceeded $45,000,
equivalent to $6,000 per year, per contract! Also, it should be noted that, even in the markets where
the system was profitable, its performance was well below gains exhibited for these markets during
the same period by most other trend-following systems.
There was no question about it. This was truly a bad system. Y et if you looked only at the well-
chosen example, you might think you had stumbled upon the trading system Jesse Livermore used in
his good years. Talk about a gap between perception and reality.
This system witnessed such large, broadly based losses that you may well wonder why fading the
signals of such a system might not provide an attractive trading strategy. The reason is that most of the
2 The start date was chosen to avoid the distortion of the extreme trends witnessed by many commodity markets
during 1973–1975. The end date merely reflected the date on which I tested this particular system.
291
TESTING AND OPTIMIzING TRADING SYSTEMS
losses are the result of the system being so sensitive that it generates large transaction costs. (Trans-
action costs include commission costs plus slippage. The concept of slippage is discussed later in this
chapter.) This sensitivity of the system occasionally is beneficial, as was the case for the Swiss franc in
1980. However, on balance, it is the system’s major weakness.
Losses due to transaction costs would not be realized as gains by fading the system. Moreover,
doing the opposite of all signals would generate equivalent transaction costs. Thus, once transac-
tion costs are incorporated, the apparent attractiveness of a contrarian approach to using the system
evaporates.
Because the related episode and the system testing it inspired occurred many years ago, some
readers might justifiably wonder whether the system has been a viable strategy in more recent years.
T o answer this question, we tested the same system on a portfolio of 31 U.S. futures contracts for the
10 years ending November 30, 2015, and produced similar results: Only 12 of the 31 markets gener-
ated a net gross profit—that is, a profit before accounting for commissions or slippage. Incorporating
a $25 commission and slippage assessment reduced the number of profitable markets to nine, and the
total losses of the unprofitable markets outweighed the profits of the winning markets by a factor of
more than 4 to 1, with a total cumulative loss of −$940,612 for the entire 10-year period (assuming
a trade size of one contract per market).
The moral is simple: Don’t draw any conclusions about a system (or indicator) on the basis of
isolated examples. The only way you can determine if a system has any value is by testing it (without
benefit of hindsight) over an extended time period for a broad range of markets.
 ■ Basic Concepts and Definitions
A trading system is a set of rules that can be used to generate trade signals. A parameter is a value that
can be freely assigned in a trading system in order to vary the timing of signals. For example, in the
basic breakout system, N (the number of prior days whose high or low must be exceeded to indicate
a signal) is a parameter. Although the operation of the rules in the system will be identical whether
N = 7 or N = 40, the timing of the signals will be vastly different. (For an example, see Figure 16.5
in Chapter 16.)
Most trading systems will have more than one parameter. For example, in the crossover moving
average system there are two parameters: the length of the short-term moving average and the length
of the long-term moving average. Any combination of parameter values is called a parameter set. For
example, in a crossover moving average system, moving averages of 10 and 40 would represent a
specific parameter set. Any other combination of moving average values would represent another
parameter set. In systems with only one parameter (e.g., breakout), the parameter set would consist
of only one element.
3
3 Note that the terms parameter set and system variation (the latter was used in Chapter 16) refer to identical con-
cepts. The introduction of the term parameter set was merely deferred until this chapter because doing so allowed
for a more logically ordered presentation of the material.
292
A Complete Guide to the Futures mArket
Most “gener


he parameter set would consist
of only one element.
3
3 Note that the terms parameter set and system variation (the latter was used in Chapter 16) refer to identical con-
cepts. The introduction of the term parameter set was merely deferred until this chapter because doing so allowed
for a more logically ordered presentation of the material.
292
A Complete Guide to the Futures mArket
Most “generic” systems are limited to one or two parameters. However, the design of more
creative and flexible systems, or the addition of modifications to basic systems, will usually imply the
need for three or more parameters. For example, adding a confirmation time delay rule to the cross-
over moving average system would imply a third parameter: the number of days in the time delay.
As a general principle, it is wise to use the simplest form of a system (i.e., the least number of
parameters) that does not imply any substantial deterioration in performance relative to the more
complex versions. However, one should not drop parameters that are deemed important simply to
reduce the number of implied parameter sets. In this case, a more reasonable approach would be
to limit the number of parameter sets actually tested.
It should be noted that even for a simple one- or two-parameter-set system, it is not necessary to
test all possible combinations. For example, in a simple breakout system in which one wishes to test the
performance for values of N = 1 to N = 100, it is not necessary to test each integer in this range. A more
efficient approach would be to first test the system using spaced values for N (e.g., 10, 20, 30, . . . , 100),
and then, if desired, the trader could focus on any areas that appeared to be of particular interest. For
example, if the system exhibited particularly favorable performance for the parameter values N = 40
and N = 50, the trader might want to also test some other values of N in this narrower range. Such an
additional step, however, is probably unnecessary, since, as is discussed later in this chapter, performance
differences in parameter set values—particularly values in such close proximity—are probably a matter
of chance and lack any significance.
As a more practical real-life example, assume we wish to test a crossover moving average system
that includes a time-delay confirmation rule. If we were interested in the performance of the system
for parameter values 1 to 50 for the shorter-term moving average, 2 to 100 for the longer-term mov-
ing average, and 1 to 20 for the time delay, there would be a total of 74,500 parameter sets.
4 Note
that we cannot reduce the number of parameters without severely damaging the basic structure of the
system. However, we can test a far more limited number of parameter sets and still produce a very
good approximation of the system’s overall performance. Specifically, we might use increments of 10
for the shorter-term moving average (10, 20, 30, 40, and 50), increments of 20 for the longer-term
moving average (20, 40, 60, 80, and 100), and three selected values for the time delay (e.g., 5, 10, and
20). This approach would limit the number of parameter sets to be tested to 57.
5 Once these param-
eter sets are tested, the results would be analyzed, and a moderate number of additional parameter
sets might be tested as suggested by this evaluation. For example, if a time delay of 5—the smallest
value tested—seemed to work best for most favorably performing parameter sets, it would also be
reasonable to test smaller values for the time delay.
Conceptually, it might be useful to define four types of parameters:
Continuous parameter. A continuous parameter can assume any value within a given range.
A percentage price penetration would be an example of a continuous parameter. Because a
4 T o avoid double counting, each “short-term” moving average can only be combined with a “long-term” moving
average for a longer period. Thus, the total number of combinations is given by (99 + 98 + 97 + … + 50)
(20) = 74,500.
5 (5 + 4 + 4 + 3 + 3)(3) = 57.
293
TESTING AND OPTIMIzING TRADING SYSTEMS
continuous parameter can assume an infinite number of values, it is necessary to specify some
interval spacing in testing such a parameter. For example, a percent penetration parameter
might be tested over a range of 0.05 percent to 0.50 percent, at intervals of 0.05 (i.e., 0.05,
0.10, . . . , 0.50). It is reasonable to expect performance results to change only moderately for
an incremental change in the parameter value (assuming a sufficiently long test period).
Discrete parameter. A discrete parameter can assume only integer values. For example, the
number of days in a breakout system is a discrete parameter. Although one can test a discrete
parameter for every integer value within the specified range, such detail is often unnecessary,
and wider spacing is frequently employed. As with continuous parameters, it is reasonable to
expect performance results to change only moderately for a small change in the parameter value.
Code parameter. A code parameter is used to represent a definitional classification. Thus, there
is no significance to the cardinal value of a code parameter. For example, assume we wish to
test a simple breakout system using three different definitions of a breakout (buy case): close
above previous N-day high, high above previous N-day high, and close above previous N-day high
close. W e could test each of these systems separately, but it might be more efficient to use a
parameter to specify the intended definition. Thus, a parameter value of 0 would indicate the
first definition, a value of 1 the second definition, and a value of 2 the third definition. Note
that there are only three possible values for this parameter, and that there is no significance to
incremental changes in parameter values.
Fixed or nonoptimized parameter. Normally, any type of parameter will be allowed to assume
different values in


hus, a parameter value of 0 would indicate the
first definition, a value of 1 the second definition, and a value of 2 the third definition. Note
that there are only three possible values for this parameter, and that there is no significance to
incremental changes in parameter values.
Fixed or nonoptimized parameter. Normally, any type of parameter will be allowed to assume
different values in testing a system. However, in systems with a large number of parameters, it
may be necessary to fix some parameter values in order to avoid an excessive number of parame-
ter sets. Such parameters are called nonoptimized parameters. For example, in a nonsensitive (slow)
trend-following system, we might wish to include a backup stop rule to prevent catastrophic
losses. By definition, in this situation, the stop rule would be activated on only a few occasions.
Consequently, any parameters implicit in the stop rule could be fixed, since variation in these
parameter values would not greatly affect the results.
 ■ Choosing the Price Series
The first step in testing a system in a given market is choosing the appropriate price series. The issues
related to this selection were fully detailed in Chapter 18. Generally speaking, a continuous futures series
is the preferred choice, although actual contract data could be used for short-term trading systems.
 ■ Choosing the Time Period
Generally speaking, the longer the test period, the more reliable the results. If the time period is too
short, the test will not reflect the system’s performance for a reasonable range of market situations.
For example, a test of a trend-following system on the Canadian dollar market that used only the three
294A COMPLETE GUIDE TO THE FUTURES MARKET
years of data from roughly October 2012 to October 2015—a period dominated by a sustained bear
market (see Figure 19.1 )—would yield highly misleading results in terms of the system’s probable
long-term performance, as evidenced by the monthly chart inset, which shows the market’s price
action dating back to 2004. Although testing over only the recent past is almost always undesirable,
longer periods are not always necessarily better for testing than shorter ones. In some markets, if too
long a period is used for testing a system, the earlier years in the survey period might be extremely
unrepresentative of current market conditions.
 Although it is impossible to provide a decisive answer as to the optimum number of years to be
used in testing, 10 to 20 years is a reasonable range. For short-term trading systems (average duration
of trades equal to a few weeks or less), a shorter test period (e.g., 5 to 10 years) would probably be
suffi cient. Trading system test results based on time periods signifi cantly shorter than these guidelines
should be suspect. In fact, it is rather incredible that some published studies on trading systems are
based on test periods of two years or less.
 Trading systems that use intraday data do not need to be tested over as long a time period as is
the case for daily data because any time period will contain far more data points. For example, in the
case of fi ve-minute bars, a stock-index futures contract—just during the stock market’s cash trading
session—will generate the equivalent of a year’s worth of daily price bars (252) in a little more than
three days. A year’s worth of this fi ve-minute data would contain approximately as many price bars
as 78 years of daily data.
 However, the far greater amount of data inherent in intraday data does not mean the test period
can be reduced proportionally—not even close. The governing principle will always be to select
 FIGURE  19.1 Major Trending Phase as Unrepresentative Price Sample: Canadian Dollar
Continuous Futures
Chart created using TradeStation. ©TradeStation T echnologies, Inc. All rights reserved.

295
TESTING AND OPTIMIzING TRADING SYSTEMS
enough data to expose the system to a wide range of market conditions. A trader testing a system
based on five-minute bars should run the test on far more than 30 days of data, even though this
data contains more bars than 10 years of daily price bars, since the larger-scale market conditions
can often be relatively static over such brief time periods. For example, the intraday price action
during a very strong 30-day trending period will likely differ dramatically from the typical intraday
price action during a 30-day trading range. The necessity that any meaningful system test span bull,
bear, and sideways markets means that even intraday systems will need to be tested over a period
of at least several years, if not more. In fact, given the current speed of computer processing, if
the data are available, there is no compelling reason to run intraday systems tests for significantly
shorter periods than daily systems. Sure, such tests will include dramatically more data, but that
is a good thing.
Ideally, one should test a system using a longer time period (e.g., 15 years) and then evaluate the
results for the period as a whole and various shorter time intervals (e.g., individual years). Such an
approach is important in determining the system’s degree of time stability—the relative performance
consistency from one period to the next. Time stability is important because it enhances confidence
regarding a system’s potential for maintaining consistently favorable performance in the future. Most
people would be quite hesitant about using a system that generated significant net profits over a
15-year period due to three spectacularly performing years but then witnessed losses or near break-
even results in the remaining 12 years—and rightly so. In contrast, a system that registered moderate
net gains during the 15-year period and was profitable in 14 of the 15 years would undoubtedly be
viewed as more attractive by most traders.
 ■ Realistic Assumptions
System traders often discover that their actual results are subs


cularly performing years but then witnessed losses or near break-
even results in the remaining 12 years—and rightly so. In contrast, a system that registered moderate
net gains during the 15-year period and was profitable in 14 of the 15 years would undoubtedly be
viewed as more attractive by most traders.
 ■ Realistic Assumptions
System traders often discover that their actual results are substantially worse than the paper trad-
ing results implied by the system. In fact, this situation is so common that it even has its own name:
slippage. Assuming that the divergence in the results is not due to errors in the program, slippage is
basically a consequence of a failure to use realistic assumptions in testing the system. Basically, there
are two types of such faulty assumptions:
 1. transaction costs. Most traders don’t realize that merely adjusting for actual commission
costs in testing a system is not a sufficiently rigid assumption. The reason for this is that commis-
sions account for only a portion—and usually a minor portion—of transaction costs. Another
less tangible, but no less real, cost is the difference between the theoretical execution price and
the actual fill price. For example, if one is testing a system assuming order entry on the close,
the use of the midpoint of the closing range might not be a realistic assumption. For some
reason, buys near the upper end of the closing range and sells near the lower end of the clos-
ing range seem to be far more common than their reverse counterparts. There are two ways of
addressing this problem. First, use the worst possible fill price (e.g., high of the closing range
for buys). Second, use a transaction cost per trade assumption much greater than the actual
296A COMPLETE GUIDE TO THE FUTURES MARKET
historical commission costs (e.g., $25 per side, per trade). The latter approach is preferable
because it is more general. For example, how would one decide the worst possible fi ll price for
an intraday stop order?
 2. limit days. Unless it is programmed otherwise, an automated trading system will indicate
executions on the receipt of each signal. However, in the real world, things are not quite so
simple. Occasionally, execution will not be possible because the market is locked at the daily
permissible limit. Or even if execution is possible, it could occur at a much worse level than
the intended price because the market gaps far beyond the signal trigger price. Although
nearly continuous trading hours have made these events less common than in decades past,
they still occur, especially in less liquid markets. If one assumes execution in such a situ-
ation, the paper results may dramatically overstate actual performance. Figure 19.2 illus-
trates the diff erence even a single locked-limit day can have on trade results. September
2011 corn futures closed limit down at 648 cents on June 30. A trader who wanted—
or worse, needed—to sell on this close but did not receive a fi ll would have had to wait
for the next session to execute the trade. The market opened 41.25 cents lower the next
day, representing a $2,062.50 loss per contract, assuming the trade was fi lled exactly at the
opening price.
 The potential systems trader may discover that seemingly attractive trading systems disintegrate
once realistic assumptions are employed. This characteristic is particularly true for very active sys-
tems, which generate very large transaction costs. However, it is far better to make this discovery in
the analytical testing stage than in actual trading.
 FIGURE  19.2 Wide Gap between Signal Price and Actual Entry: Impact of Limit Days
(September 2011 Corn)
Chart created using TradeStation. ©TradeStation T echnologies, Inc. All rights reserved.

297
TESTING AND OPTIMIzING TRADING SYSTEMS
 ■ Optimizing Systems
Optimization refers to the process of finding the best-performing parameter set(s) for a given system
applied to a specific market. The underlying premise of optimization is that parameter sets that worked
best in the past have a greater probability of superior performance in the future. (The question of
whether this assumption is valid is addressed in the next section.)
A basic question that must be considered in optimization is what criteria should be used for defin-
ing best performance. Frequently, best performance is simply interpreted as largest equity gain.
However, such a definition is incomplete. Ideally, four factors should be considered in performance
comparisons:
 1. percent return. Return measured relative to funds needed to trade the system. The impor-
tance of using percent return rather than nominal gain is detailed in Chapter 20.
 2. risk measure. In addition to percent gain, it is also important to employ some measure of
equity fluctuations (e.g., variability in rate of gain, retracements in equity). Besides the obvious
psychological reasons for wishing to avoid parameter sets and systems with high volatility, a
risk measure is particularly significant because one might pick an unfavorable starting date for
trading the system. Chapter 20 discusses several performance measures that incorporate both
percent return and risk.
 3. parameter stability. It is not sufficient to find a parameter set that performs well. It is also
necessary to ascertain the parameter set does not reflect a fluke in the system. In other words,
we wish to determine that similar parameter sets also exhibit favorable performance. In fact, the
goal of optimization should be to find broad regions of good performance rather than the single
best-performing parameter set.
For example, if in testing a simple breakout system one found that the parameter set
N = 7 exhibited the best percent return/risk characteristics but that performance dropped off
very sharply for parameter sets N < 5 and N > 9, while all sets in the range N = 25 to N = 54
performed relatively well, it would make much more sense to choose a param


r than the single
best-performing parameter set.
For example, if in testing a simple breakout system one found that the parameter set
N = 7 exhibited the best percent return/risk characteristics but that performance dropped off
very sharply for parameter sets N < 5 and N > 9, while all sets in the range N = 25 to N = 54
performed relatively well, it would make much more sense to choose a parameter set from the
latter range. Why? Because the exceptional performance of the set N = 7 appears to be a pecu-
liarity of the historical price data, which is not likely to be repeated. The fact that surrounding
parameter sets performed poorly suggests that there is no basis for confidence in trading the
parameter set N = 7. In contrast, the broad range of performance stability for sets in the region
N = 25 to N = 54 suggests that a set drawn from the center of this range would have a better
prospect for success.
 4. time stability. As detailed in a previous section, it is important to ascertain that favorable
performance for the period as a whole is truly representative of the total period rather than a
reflection of a few isolated intervals of extraordinary performance.
For comparisons involving different parameter sets for the same system, the preceding factors tend
to be highly correlated. Generally, the parameter sets with the best gains will also be the sets that
exhibit the smallest equity retracements. Consequently, for the optimization of a single system, the
use of a basic return/risk measure (e.g., the Sharpe ratio or the gain-to-pain ratio) will usually yield
298
A Complete Guide to the Futures mArket
similar results to a complex performance evaluation that incorporates multiple performance mea-
sures. Thus, although the multifactor performance evaluation is theoretically preferable, it is often not
essential. However, if one is comparing parameter sets from completely different systems, the explicit
consideration of risk, parameter stability, and time stability is more important.
The foregoing represents a theoretical discussion of optimization concepts and procedures, and
implicitly assumes that optimization enhances a system’s future performance. As discussed in the next
section, however, the viability of optimization is open to serious question.
 ■ The Optimization Myth
It is ironic that optimization receives so much attention while its underlying premise is rarely considered.
In other words, do the better performing parameter sets of the past continue to exhibit above-average
performance in the future?
As an empirical test of the validity of optimization we examine the historical rankings of a range of
parameter set values for a breakout system: reverse from short to long if today’s close is higher than
the highest close during the past N days; reverse from long to short if today’s close is lower than the
lowest close during the past N days. Nine values of N for this system were tested: 20, 30, 40, 50, 60,
70, 80, 90, and 100.
Tables 19.1 to 19.10 compare the profit/loss rankings of these parameter sets in 10 markets for
three 2-year test periods (2009–2010, 2011–2012, and 2013–2014), with parameter sets listed in
the order of their performance during the respective prior eight-year periods. (All markets were
traded with one contract per signal.) In other words, the top-performing parameter set of the prior
eight-year period (2001–2008, 2003–2010, or 2005–2012) is listed first, the second-best parameter
set of the prior period is listed second, and so on. For example, if the top number in a column is 6, it
means that the best-performing parameter set for that market in the prior eight-year period was the
sixth-ranked parameter set (out of nine) during the given test period.
table 19.1 breakout System (10-Y ear t-Notes): Comparison of parameter Set rankings in two-Y ear test
periods vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 9 9 7
2 8 6 5
3 7 7 3
4 2 8 1
5 5 4 4
6 6 5 6
7 1 3 2
8 3 1 9
9 4 2 8
299
TESTING AND OPTIMIzING TRADING SYSTEMS
table 19.2 breakout System (euro): Comparison of parameter Set rankings in two-Y ear test periods vs.
rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 4 2 1
2 9 1 7
3 5 4 2
4 6 5 5
5 7 6 8
6 3 3 3
7 8 7 9
8 2 8 6
9 1 9 4
table 19.3 breakout System (Japanese Y en): Comparison of parameter Set rankings in two-Y ear test
periods vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 9 5 4
2 2 3 1
3 8 7 6
4 1 6 2
5 3 1 7
6 4 4 8
7 7 9 9
8 6 2 5
9 5 8 3
table 19.4 breakout System (Gold): Comparison of parameter Set rankings in two-Y ear test periods
vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 7 2 2
2 3 4 3
3 4 5 4
4 9 1 9
5 6 6 1
6 8 9 7
7 1 8 5
8 2 7 6
9 5 3 8
300
A Complete Guide to the Futures mArket
table 19.5 breakout System (Natural Gas): Comparison of parameter Set rankings in two-Y ear test
periods vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 8 3 1
2 4 5 4
3 5 1 2
4 1 6 3
5 6 8 9
6 2 9 5
7 9 4 8
8 7 7 6
9 3 2 7
table 19.6 breakout System (WtI Crude Oil): Comparison of parameter Set rankings in two-Y ear test
periods vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank o


iod
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 8 3 1
2 4 5 4
3 5 1 2
4 1 6 3
5 6 8 9
6 2 9 5
7 9 4 8
8 7 7 6
9 3 2 7
table 19.6 breakout System (WtI Crude Oil): Comparison of parameter Set rankings in two-Y ear test
periods vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 3 6 1
2 2 7 6
3 7 9 8
4 4 1 2
5 5 3 5
6 1 5 4
7 9 8 9
8 6 2 3
9 8 4 7
table 19.7 breakout System (Corn): Comparison of parameter Set rankings in two-Y ear test periods
vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 3 7 3
2 4 1 7
3 2 3 5
4 1 8 8
5 9 4 1
6 5 9 6
7 6 2 2
8 8 5 4
9 7 6 9
301
TESTING AND OPTIMIzING TRADING SYSTEMS
table 19.8 breakout System (Soybeans): Comparison of parameter Set rankings in two-Y ear test
periods vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 6 4 5
2 3 5 3
3 4 7 1
4 1 2 4
5 2 3 2
6 8 1 7
7 7 6 6
8 9 8 8
9 5 9 9
table 19.9 breakout System (Coffee): Comparison of parameter Set rankings in two-Y ear test periods
vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 3 1 9
2 8 2 1
3 1 6 6
4 7 8 3
5 9 9 2
6 2 5 4
7 6 7 8
8 5 4 7
9 4 3 5
table 19.10 breakout System (e-Mini Nasdaq 100): Comparison of parameter Set rankings in two-Y ear
test periods vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 2009–2010
rank of Same parameter
Set in 2011–2012
rank of Same parameter
Set in 2013–2014
1 5 3 9
2 7 1 7
3 4 2 8
4 2 8 4
5 6 6 6
6 9 5 1
7 3 9 2
8 8 4 5
9 1 7 3
302
A Complete Guide to the Futures mArket
As a visual aid to help see if there is any consistency between past and future performance, the
two top-performing parameter sets in each test period are denoted by circles and the two bottom
parameter sets by squares. If the basic premise of optimization were valid—that is, that the best-
performing parameter sets of the past were likely to be the best-performing parameter sets in the
future—then Tables 19.1 through 19.10 should reflect a pattern of circles consistently near column
tops and squares consistently near column bottoms. However, this is not the case. Both circles and
squares are sometimes near column tops, sometimes near column bottoms, and sometimes near
column midpoints. The apparent randomness in the vertical placement of the circles and squares in
Tables 19.1 through 19.10 implies the correlation between past and future performance is highly
tenuous.
Table 19.11 further highlights the weakness of the relationship between past and future per-
formance. In addition to showing the average rank of the best-performing parameter sets from the
eight-year sample periods in the subsequent two-year test periods (second column), Table 19.11
also shows how often the best- and worst-performing sets in a prior eight-year period repeated
their positions in the subsequent two-year period versus completely reversing their rank order.
Note the initially best- and worst-performing parameter sets repeated in subsequent two-year
periods a total of eight times, which is only one time more than the number of times the best set
became the worst set or the worst set became the best set. Also notice that the best-performing
parameter set became the worst-performing set one more time (5) than the best-performing set
repeated as the top set.
This instability in the values of the best-performing parameter sets from period to period
means gauging a system’s performance by the best past parameter sets will grossly overstate the
system’s performance potential. T o illustrate this point, Tables 19.12 through 19.15 compare
the performance of the best parameter set in each test period versus the average of all param-
eter sets and the performance of the parameter sets that had the best and worst results in the
table 19.11 Stability of best- and W orst-performing parameter Sets
Market
avg. rank of best
parameter Set
best parameter
Set repeated
W orst parameter
Set repeated
best Set becomes
W orst Set
W orst Set becomes
best Set
10-yr. T -note 4.70 0 0 2 0
Euro 4.30 1 1 0 1
Japanese yen 4.77 0 0 1 0
Gold 4.27 0 0 0 0
Natural gas 5.10 1 0 0 0
WTI crude oil 5.13 1 0 0 0
Corn 6.00 0 1 0 0
Soybeans 5.43 0 2 0 0
Coffee 5.30 1 0 1 0
E-mini Nasdaq 100 4.70 0 0 1 1
total 4 4 5 2
303
TESTING AND OPTIMIzING TRADING SYSTEMS
prior period. In this example, based on the all-market totals, selecting the worst parameter
set in the prior period would have outperformed a strategy of picking the best past parameter
set in one of the three test periods (see Table 19.12), as well as the three-period total (see
Table 19.15). The penultimate column of these tables marks the instances the worst-performing
parameter set in a prior eight-year period outperformed the prior best-performing set in the
subsequent two-year period. The final column shows how often the average parameter set per-
formance in the subsequent two-year period outperformed the best-performing set of the prior
eight-year period.
table 19.12 profit/loss ($) Comparisons for 2009–2010 test period: actual best parameter Set vs.
period average and best and W orst parameter Sets in prior period
Market
best parameter
Set in period
best parameter Set
in prior period
W orst parameter
Set in prior period
avg. of all
parameter Sets
W orst prior
> best prior
avg


performed the best-performing set of the prior
eight-year period.
table 19.12 profit/loss ($) Comparisons for 2009–2010 test period: actual best parameter Set vs.
period average and best and W orst parameter Sets in prior period
Market
best parameter
Set in period
best parameter Set
in prior period
W orst parameter
Set in prior period
avg. of all
parameter Sets
W orst prior
> best prior
avg. >
best prior
10-yr. T -note $7,453 –$7,188 $2,391 $253 X X
Euro $47,575 $18,963 $47,575 $22,511 X X
Japanese yen $5,438 –$23,825 –$9,638 –$8,967 X X
Gold $50,740 $7,420 $19,020 $25,084 X X
Natural gas $46,960 –$7,360 $34,120 $16,522 X X
WTI crude oil –$11,670 –$26,030 –$45,150 –$33,041
Corn $8,875 $6,913 –$338 $3,188
Soybeans $34,188 $11,875 $22,350 $16,944 X X
Coffee $25,650 $12,075 $11,963 $6,713
E-Mini Nasdaq 100 $12,330 $4,820 $12,330 $5,417 X X
total $227,538 –$2,338 $94,623 $54,625 7 7
table 19.13 profit/loss ($) Comparisons for 2011–2012 test period: actual best parameter Set vs.
period average and best and W orst parameter Sets in prior period
Market
best parameter
Set in period
best parameter Set
in prior period
W orst parameter
Set in prior period
avg. of all
parameter Sets
W orst prior
> best prior
avg. >
best prior
10-yr. T -note $13,172 –$3,750 $9,234 $3,516 X X
Euro $10,900 $10,900 –$11,550 $1,938
Japanese yen –$1,538 –$7,963 –$12,913 –$8,157
Gold $16,310 $7,300 $3,170 –$5,672
Natural gas $16,050 $2,590 $10,930 –$712 X
WTI crude oil $12,330 –$30,950 –$11,920 –$19,537 X X
Corn –$963 –$8,563 –$8,538 –$9,138 X
Soybeans $24,013 –$3,113 –$16,413 –$2,590 X
Coffee $48,563 $48,563 $20,963 $8,308
E-Mini Nasdaq 100 $1,540 –$7,630 –$20,870 –$13,506
total $140,377 $7,385 –$37,906 –$45,550 4 3
304
A Complete Guide to the Futures mArket
table 19.14 profit/loss ($) Comparisons for 2013–2014 test period: actual best parameter Set vs.
period average and best and W orst parameter Sets in prior period
Market
best parameter
Set in period
best parameter Set
in prior period
W orst parameter
Set in prior period
avg. of all
parameter Sets
W orst prior
> best prior
avg. >
best prior
10-yr. T -note $2,922 –$2,328 –$3,359 –$1,557 X
Euro $19,963 $19,963 $5,013 $2,568
Japanese yen $39,713 $38,138 $39,713 $27,339 X
Gold $25,840 $21,160 –$4,340 $11,042
Natural gas $6,250 $6,250 –$1,590 –$2,077
WTI crude oil $39,060 $39,060 $18,070 $23,379
Corn $9,750 $3,675 –$1,150 $2,661
Soybeans $8,663 $488 –$12,863 $1,211 X
Coffee $28,313 –$9,113 $2,963 $7,677 X X
E-Mini Nasdaq 100 $29,640 –$8,780 $16,505 $10,635 X X
total $210,112 $108,512 $58,961 $82,878 3 4
table 19.15 profit/loss ($) Comparisons for three test periods Combined: actual best parameter Sets
vs. period averages and best and W orst parameter Sets in prior periods
Market
best parameter
Set in period
total
best parameter
Set in prior
period total
W orst parameter
Set in prior
period total
avg. of all
parameter Sets
total
W orst prior
> best prior
avg. >
best prior
10-yr. T -note $23,547 –$13,266 $8,266 $2,212 X X
Euro $78,438 $49,825 $41,038 $27,017
Japanese yen $43,613 $6,350 $17,163 $10,215 X X
Gold $92,890 $35,880 $17,850 $30,454
Natural gas $69,260 $1,480 $43,460 $13,733 X X
WTI crude oil $39,720 –$17,920 –$39,000 –$29,199
Corn $17,663 $2,025 –$10,025 –$3,289
Soybeans $66,863 $9,250 –$6,925 $15,565 X
Coffee $102,525 $51,525 $35,888 $22,698
E-Mini Nasdaq 100 $43,510 –$11,590 $7,965 $2,546 X X
total $578,027 $113,559 $115,678 $91,953 4 5
Our example used a very small list of only nine parameter sets. Many system developers run opti-
mizations across hundreds or even thousands of parameter sets. Imagine the degree of performance
overstatement that would occur by representing a system’s performance by the best parameter sets
in these cases!
For comparison, Tables 19.16 through 19.19 show the same information as Tables 19.12 through
19.15 except they reflect tests of the same system conducted 20 years earlier on a slightly different
305
TESTING AND OPTIMIzING TRADING SYSTEMS
portfolio (30-year U.S. T -bonds, Deutsche marks, Japanese yen, gold, silver, heating oil, corn,
soybeans, live cattle, and sugar). In this case, the three 8-year sample periods were 1981–1988,
1983–1990, and 1985–1992 and the three 2-year test periods were 1989–1990, 1991–1992, and
1993–1994.
table 19.16 profit/loss ($) Comparisons for 1989–1990 test period: actual best parameter Set vs.
period average and best and W orst parameter Sets in prior period
Market
best parameter
Set in period
best parameter Set
in prior period
W orst parameter
Set in prior period
avg. of all
parameter Sets
W orst prior
> best prior
avg. >
best prior
T -bond 6,670 −9,090 1,420 −2,180 X X
Deutsche mark 7,780 3,020 6,340 5,390 X X
Japanese yen 11,840 9,240 8,420 8,130
Gold 3,390 1,700 −320 1,080
Silver 5,850 5,330 1,630 3,050
Heating oil 7,650 1,760 6,430 3,380 X X
Corn 1,640 −2,190 −2,730 −590 X
Soybeans 4,970 −7,160 4,740 −740 X X
Cattle 2,090 850 −3,290 −20
Sugar 4,240 4,170 −5,560 −840
total 56,120 7,630 17,080 16,030 4 5
table 19.17 profit/loss ($) Comparisons for 1991–1992 test period: actual best parameter Set vs.
period average and best and W orst parameter Sets in prior period
Market
best parameter
Set in period
best parameter Set
in prior period
W orst parameter
Set in prior period
avg. of all
parameter Sets
W orst prior
> best prior
avg. >
best prior
T -bond 3,710 −1,820 −2,920 −420 X
Deutsche mark 9,180 1,680 9,180 4,770 X X
Japanese yen 3,340 −240 −3,620 −1,670
Gold 1,370 90 1,370 −1,050 X
Silver −720 −1,890 −1,780 −1,640 X X
Heating oil 5,510 −980 4,290 1,540 X X
Corn 560 −480 340 −440 X X
Soybeans −2,420 −6,090 −3,190 −4,650 X X
Cattle 1,380 −160 1,380 −340 X
Sugar 810 −1,690 −1,850 −1,410 X
total 22,700 −11,570 3,200 −5,010 7 7
306
A Complete Guide to the Futures mArket
table 19.18 profit/loss ($) Comparisons for 1993–1994 test period: actual best parameter Set vs.
period average and best and W orst parameter Sets in prior period
Market
best parameter


X
Corn 560 −480 340 −440 X X
Soybeans −2,420 −6,090 −3,190 −4,650 X X
Cattle 1,380 −160 1,380 −340 X
Sugar 810 −1,690 −1,850 −1,410 X
total 22,700 −11,570 3,200 −5,010 7 7
306
A Complete Guide to the Futures mArket
table 19.18 profit/loss ($) Comparisons for 1993–1994 test period: actual best parameter Set vs.
period average and best and W orst parameter Sets in prior period
Market
best parameter
Set in period
best parameter Set
in prior period
W orst parameter
Set in prior period
avg. of all
parameter Sets
W orst prior
> best prior
avg. >
best prior
T -bond 11,600 3,500 7,910 7,180 X X
Deutsche mark 6,210 −3,660 −1,410 −3,300 X X
Japanese yen 3,620 2,460 −3,060 260
Gold 490 −1,900 −930 −1,460 X X
Silver 1,600 −3,650 −790 −2,690 X X
Heating oil 2,200 2,200 −890 −1,700
Corn 1,910 1,910 −1,030 640
Soybeans 2,120 1,570 −2,060 −240
Cattle 1,600 950 1,600 500 X
Sugar 880 570 −240 −550
total 32,230 3,950 −900 −1,360 5 4
table 19.19 profit/loss ($) Comparisons for three test periods Combined: actual best parameter Sets
vs. period averages and best and W orst parameter Sets in prior periods
Market
best parameter
Sets in test
periods total
best parameter
Sets in prior
periods total
W orst parameter
Sets in Prior
periods total
period
parameter Set
averages total
W orst prior
> best prior
avg. >
best prior
T -bond 21,980 −7,410 6,410 3,950 X X
Deutsche mark 23,170 1,040 14,110 6,860 X X
Japanese yen 18,800 11,460 1,740 6,720
Gold 5,250 −110 120 −1,430 X
Silver 6,730 −210 −940 −1,280
Heating oil 15,360 2,980 9,830 3,220 X X
Corn 4,110 −760 −3,420 −390 X
Soybeans 4,670 −11,680 −510 −5,330 X X
Cattle 5,070 1,640 −310 140
Sugar 5,930 3,060 −7,650 −2,800
total 111,070 10 19,380 9,660 5 5
Based on the combined three-period, all-market totals from this second set of tests, selecting the
worst parameter set in the prior period actually would have outperformed a strategy of picking the
best past parameter set in two of the three test periods, as well as the three-period total!
This observation is not intended to imply that the prior-period worst-performing parameter set is
likely to outperform the prior-period best-performing set. If similar empirical tests were conducted
for other systems, the prior-period best-performing parameter set would probably outperform the
prior-period worst-performing set more often than the other way around (although the types of results
witnessed in our example are far from uncommon). The key point, however, is that invariably, as was the
307
TESTING AND OPTIMIzING TRADING SYSTEMS
case in Tables 19.12 through 19.15 and 19.16 through 19.19, the prior-period best-performing parame-
ter sets would fall far short of the actual best-performing parameter sets for the given periods and would
often fail to provide any statistically significant improvement over the average of all parameter sets.
Although optimization seemed to have little, if any, value when applied market by market, optimiza-
tion does appear to be a bit more useful if applied to a portfolio. In other words, instead of picking the
best past parameter set for each market, the best past single parameter set applied across all markets is
selected. Table 19.20 shows the two-year test period parameter set rankings for a portfolio consisting
of the 10 markets that provided the results for Tables 19.16 through 19.19.
6 The one striking correla-
tion between past and future performance is that the worst parameter set in the prior eight-year period
is also the worst parameter set in the subsequent two-year period in all three test intervals!
Although the worst past parameter set also seems likely to be the worst future parameter set,
other past ranking placements seem to imply little predictive value. The average ranking for all three
test periods of the remaining eight prior-period ranking placements (i.e., all rankings excluding the
worst one) is 4.5. While the average test period ranking of the best parameter set in the prior eight-
year period (3.3) is somewhat better than this average, the fourth-ranked parameter set in the prior
period has by far the best average ranking in the future test periods (2.3). Also note that the second-
best prior-period parameter set has an average test period rank almost identical to the corresponding
average for the second worst prior-period parameter set (4.7 vs. 5.0).
T o gain some insight as to why the worst prior-period ranking seems to be such an excellent
predictor of future performance (namely, continued poor performance for that parameter set), while
other ranking placements seem to have little predictive value, we examine performance rankings based
on parameter set value. Table 19.21 indicates parameter set rankings in each of the three tests periods
based on parameter set values (as opposed to prior-period rankings as was the case in Table 19.20). The
parameter set values are listed in ascending order.
6 In this case the portfolio consisted of one contract in each market, with the exception of corn, which was traded
with two contracts because of its low volatility.
table 19.20 breakout System (portfolio): Comparison of parameter Set rankings in two-Y ear test
periods vs. rankings in prior eight-Y ear periods
parameter Set rank
prior eight-Y ear period
rank of Same parameter
Set in 1989–1990
rank of Same parameter
Set in 1991–1992
rank of Same parameter
Set in 1993–1994 avg. rank
1 1 7 2 3.3
2 5 1 8 4.7
3 3 6 4 4.3
4 2 4 1 2.3
5 4 8 6 6.0
6 6 3 7 5.3
7 7 5 3 5.0
8 8 2 5 5.0
9 9 9 9 9.0
308
A Complete Guide to the Futures mArket
table 19.21 breakout System (portfolio): Comparison of parameter Set rankings in two-Y ear test
periods based on N-Values
parameter Set N-Value
rank of parameter Set
in 1989–1990
rank of parameter Set
in 1991–1992
rank of parameter Set
in 1993–1994 avg. rank
20 9 9 9 9.0
30 8 2 5 5.0
40 7 5 3 5.0
50 6 3 1 3.3
60 4 6 6 5.3
70 5 7 8 6.7
80 1 1 2 1.3
90 2 4 4 3.3
100 3 8 7 6.0
Table 19.21 reveals that the worst-p


out System (portfolio): Comparison of parameter Set rankings in two-Y ear test
periods based on N-Values
parameter Set N-Value
rank of parameter Set
in 1989–1990
rank of parameter Set
in 1991–1992
rank of parameter Set
in 1993–1994 avg. rank
20 9 9 9 9.0
30 8 2 5 5.0
40 7 5 3 5.0
50 6 3 1 3.3
60 4 6 6 5.3
70 5 7 8 6.7
80 1 1 2 1.3
90 2 4 4 3.3
100 3 8 7 6.0
Table 19.21 reveals that the worst-performing parameter set in each of the test periods was actually
the same parameter set! (Since Table 19.20 indicated the test period worst parameter set was the same
as the prior-period worst parameter set in all three cases, the implication is that this same parameter
set was also the worst-performing parameter set in all three prior eight-year periods.) This consistently
worst-performing parameter set is at one extreme end of the parameter set range tested: N = 20.
Although N = 20—the most sensitive parameter set value tested—is consistently the worst per-
former (when applied across a portfolio), the other values tested (N = 30 to N = 100) show no
consistent pattern. It is true that the parameter set N = 80 was by far the best-performing set with
an incredible average rank of 1.3. However, the average rankings of the two surrounding N-values
(6.7 and 3.3) suggest that the stellar performance of N = 80 was probably a statistical fluke. As was
explained earlier in this chapter, a lack of parameter stability suggests that the past superior perfor-
mance of a parameter set probably reflects a peculiarity in the historical data tested rather than a
pattern that is likely to be repeated in the future.
Tables 19.22 and 19.23 show analogous portfolio optimization statistics for the portfolios in the
more recent test periods that were reviewed in Tables 19.12 through 19.15. Note in Table 19.23 that
the same N = 20 parameter set once again exhibited inferior performance, registering as the worst-
performing set in two of the three periods.
It is instructive to review the observations revealed by the foregoing optimization experiments:
 ■ Optimization appeared to have no value whatsoever when applied on a market-by-market basis.
 ■ When applied to a portfolio, however, optimization in the earlier (1981–1994) example appeared
useful in predicting the parameter set most likely to witness inferior future performance, although
it still showed no reliable pattern in predicting the parameter set most likely to witness superior
future performance.
 ■ Upon closer examination it appeared this pattern of consistent inferior performance was not
so much a consequence of the prior-period ranking as the parameter value. In other words, the
309
TESTING AND OPTIMIzING TRADING SYSTEMS
parameter set range tested began at a value that was clearly suboptimal for the given system: N = 20.
This same parameter value remained suboptimal, on average, in the more recent test period as well.
Although not indicated in the parameter set ranking tables, lower values of N would have shown
even worse performance—in fact, strikingly worse—as the value of N was decreased.
These observations, which are consistent with the results of other similar empirical tests I have
conducted in the past, suggest the following five key conclusions regarding optimization:7
table 19.22 breakout System (portfolio): Comparison of parameter Set rankings in two-Y ear test
periods vs. rankings in prior eight-Y ear periods (2000–2014)
parameter Set rank
prior eight-Y ear period
parameter Set rank
in 2009–2010
parameter Set rank
in 2011–2012
parameter Set rank
in 2013–2014 average rank
1 9 1 1 3.7
2 2 7 5 4.7
3 3 2 3 2.7
4 7 3 9 6.3
5 6 4 4 4.7
6 8 5 8 7.0
7 4 8 2 4.7
8 5 6 7 6.0
9 1 9 6 5.3
table 19.23 breakout System (portfolio): Comparison of parameter Set rankings in two-Y ear test
periods based on N-Values (2000–2014)
parameter Set
N-Value
rank of parameter Set
in 2009–2010
rank of parameter Set
in 2011–2012
rank of parameter Set
in 2013–2014 average rank
20 9 3 9 7.0
30 7 5 8 6.7
40 6 8 7 7.0
50 8 9 6 7.7
60 5 6 2 4.3
70 3 7 3 4.3
80 4 4 4 4.0
90 1 2 5 2.7
100 2 1 1 1.3
7 Although a single empirical experiment cannot be used to draw broad generalizations, I am willing to do so here
because the results of the optimization test just described are fairly typical of many similar tests I have conducted
in the past. In this sense, the optimization tests detailed in the text are not intended as a proof of the severe limita-
tions of optimization, but rather as illustrations of this point.
310
A Complete Guide to the Futures mArket
 1. Any system—repeat, any system—can be made to be very profitable through optimization
(i.e., over its past performance). If you ever find a system that can’t be optimized to show
good profits in the past, congratulations, you have just discovered a money machine (by
doing the opposite, unless transaction costs are exorbitant). Therefore, a wonderful past
performance for a system that has been optimized may be nice to look at, but it doesn’t mean
very much.
 2. Optimization will always, repeat always, overstate the potential future performance of a
system—usually by a wide margin (say, three trailer trucks’ worth). Therefore, optimized
results should never, repeat never, be used to evaluate a system’s merit.
 3. For many if not most systems, optimization will improve future performance only marginally, if
at all.
 4. If optimization has any value, it is usually in defining the broad boundaries for the ranges from
which parameter set values in the system should be chosen. Fine-tuning of optimization is at
best a waste of time and at worst self-delusion.
 5. In view of the preceding items, sophisticated and complex optimization procedures are a waste
of time. The simplest optimization procedure will provide as much meaningful information
(assuming that there is any meaningful information to be derived).
In summary, contrary to widespread belief, there is some reasonable question as to wheth


is at
best a waste of time and at worst self-delusion.
 5. In view of the preceding items, sophisticated and complex optimization procedures are a waste
of time. The simplest optimization procedure will provide as much meaningful information
(assuming that there is any meaningful information to be derived).
In summary, contrary to widespread belief, there is some reasonable question as to whether opti-
mization will yield meaningfully better results over the long run than randomly picking the param-
eter sets to be traded. Lest there be any confusion, let me explicitly state that this statement is not
intended to imply that optimization is never of any value. First, as indicated previously, optimization
can be useful in defining the suboptimal extreme ranges that should be excluded from the selection
of parameter set values (e.g., N ≤ 20 in our breakout system example). Also, it is possible that, for
some systems, optimization may provide some edge in parameter set selection, even after suboptimal
extreme ranges are excluded. However, I do mean to imply that the degree of improvement provided
by optimization is far less than generally perceived and that traders would probably save a lot of
money by first proving any assumptions they are making about optimization rather than taking such
assumptions on blind faith.
 ■ Testing versus Fitting
Perhaps the most critical error made by users of futures trading systems is the assumption the per-
formance of the optimized parameter sets during the test period provides an approximation of the
potential performance of those sets in the future. As was demonstrated in the previous section, such
assumptions will lead to grossly overstated evaluations of a system’s true potential. It must be under-
stood that futures market price fluctuations are subject to a great deal of randomness. Thus, the “ugly
truth” is that the question of which parameter sets will perform best during any given period is largely
a matter of chance. The laws of probability indicate that if enough parameter sets are tested, even a
meaningless trading system will yield some sets with favorable past performance. Evaluating a system
based on the optimized parameter sets (i.e., the best-performing sets during the survey period)
311
TESTING AND OPTIMIzING TRADING SYSTEMS
would be best described as fitting the system to past results rather than testing the system. If optimiza-
tion can’t be used to gauge performance, how then do you evaluate a system? The following sections
describe two meaningful approaches.
blind Simulation
In the blind simulation approach the system is optimized using data for a time period that deliber-
ately excludes the most recent years. The performance of the system is then tested using the selected
parameter sets for subsequent years. Ideally, this process should be repeated several times.
Note that the error of fitting results is avoided because the parameter sets used to measure per-
formance in any given period are selected entirely on the basis of prior rather than concurrent data.
In a sense, this testing approach mimics real life (i.e., one must decide which parameter sets to trade
on the basis of past data).
The optimization tests of the previous section used this type of procedure, stepping through time
in two-year intervals. Specifically, system results for the 2001–2008 period were used to select the
best-performing parameter sets, which were then tested for the 2009–2010 period. Next, the system
results for the 2003–2010 period were used to select the best-performing parameter sets, which were
then tested for the 2011–2012 period. Finally, the system results for the 2005–2012 period were
used to select the best-performing parameter sets, which were then tested for the 2013–2014 period.
The essential point is that simulation and optimization periods should not be allowed to overlap.
Simulations that are run over the same period as the optimization are worthless.
average parameter Set performance
Finding the average parameter set performance requires defining a complete list of all parameter
sets you wish to test before running any simulations. Simulations are then run for all the parameter
sets selected, and the average of all sets tested is used as an indication of the system’s potential per-
formance. This approach is valid because you could always throw a dart to pick a parameter from a
broad range of parameter set values. If you throw enough darts, the net result will be the average. The
important point is that this average should be calculated across all parameter sets, not just those sets
that prove profitable. Note that the trader might still choose to trade the optimized parameter sets
for the future (instead of randomly selected ones), but the evaluation of the system’s performance
should be based on the average of all sets tested (which is equivalent to a random selection process).
The blind simulation approach probably comes closest to duplicating real-life trading circumstances.
However, the average parameter set performance is probably as conservative and has the advantage of
requiring far less calculation. Both approaches represent valid procedures for testing a system.
One important caveat: In the advertised claims for given systems, the term simulated results is often
used loosely as a euphemism for optimized results (instead of implying the results are based on a blind
simulation process). If this is the case, the weight attached to the results should equal the amount of
money invested in the system: zero. The commonplace misuse and distortion of simulated results is
examined in detail in the next section.
312
A Complete Guide to the Futures mArket
 ■ The Truth about Simulated Results
Although the value of optimization in improving a system’s future performance is open to debate,
there is absolutely no question the use of optimized results will greatly distort the implied future
performance o


ro. The commonplace misuse and distortion of simulated results is
examined in detail in the next section.
312
A Complete Guide to the Futures mArket
 ■ The Truth about Simulated Results
Although the value of optimization in improving a system’s future performance is open to debate,
there is absolutely no question the use of optimized results will greatly distort the implied future
performance of a system. As was demonstrated earlier in this chapter, there is very little, if any,
correlation between the best-performing parameters in a system for one period and the best-
performing parameters in a subsequent period. Hence, assuming that the performance implied by
the best-performing parameters could have been achieved in the past is totally unrealistic.
After years of experience, my attitude toward simulated results is summarized by what I call
Schwager’s simulations corollary to Gresham’s law of money. As readers may recall from Economics
101, Gresham’s proposition was that “bad money drives out good.” Gresham’s contention was that
if two types of money were in circulation (e.g., gold and silver) at some arbitrarily defined ratio
(e.g., 16:1), the bad money (i.e., the money overvalued at the fixed rate of exchange) would drive
out the good. Thus, if gold were worth more than 16 ounces of silver, a 16:1 ratio would result in
silver driving gold out of circulation (as people would tend to hoard it).
My corollary is “bad simulations drive out good.” The term bad means simulations derived based
on highly tenuous assumptions, not bad in terms of indicated performance. On the contrary, truly
“bad” simulations will show eye-popping results.
I frequently see ads hawking systems that supposedly make 200 percent, 400 percent, or even
600 percent a year. Let’s be conservative—and I use the term loosely—and assume a return of only
100 percent per year. At this level of return, $100,000 would grow to over $1 billion in just over 13
years! How can such claims possibly be true, then? The answer is they can’t. The point is that, given
enough hindsight, it is possible to construct virtually any type of past-performance results. If anyone
tried to sell a system or a trading program based on truly realistic simulations, the results would
appear laughably puny relative to the normal promotional fare. It is in this sense that I believe that
bad (unrealistic) simulations drive out good (realistic) simulations.
How are simulated results distorted? Let us count the ways:
 1. the well-chosen example (revisited). In constructing a well-chosen example, the system
promoter selects the best market, in the best time period, using the best parameter set. Assum-
ing a system is tested on 25 markets for 15 years and uses 100 parameter set variations, there
would be a total of 37,500 (25 × 15 × 100) one-year results. It would be difficult to construct
a system in which not one of these 37,500 possible outcomes showed superlative results. For
example, if you tossed a group of 10 coins 37,500 times, don’t you think you would get 10 out
of 10 heads sometimes? Absolutely. In fact, you would get 10 out of 10 heads on the average of
one out of 1,024 times.
 2. Kitchen sink approach. By using hindsight to add parameters and create additional system
rules that conveniently take care of past losing periods, it is possible to generate virtually any
level of past performance.
 3. Ignoring risk. Advertised system results frequently calculate return as a percent of margin
or as a percent of an unrealistically low multiple of margin. This return measurement approach
313
TESTING AND OPTIMIzING TRADING SYSTEMS
alone can multiply the implied returns severalfold. Of course, the risk would increase commen-
surately, but the ads don’t provide those details.
 4. Overlooking losing trades. It is hardly uncommon for charts in system websites or adver-
tisements to indicate buy and sell signals at the points at which some specified rules were met,
but fail to indicate other points on the same chart where the same conditions were met and the
resulting trades were losers.
 5. Optimize, optimize, optimize. Optimization (i.e., selecting the best-performing param-
eter sets for the past) can tremendously magnify the past performance of a system. Virtually any
system ever conceived by man would look great if the results were based on the best parameter
set (i.e., the parameter set that had the best past performance) for each market. The more
parameter sets tested, the wider the selection of past results, and the greater the potential simu-
lated return.
 6. Unrealistic transaction costs. Frequently, simulated results only include commissions but
not slippage (the difference between the assumed entry level and the actual fill that would be
realized by using a market or stop order). For short-term systems (e.g., those using intraday
data), ignoring slippage can make a system that would wipe out an account in real life look like
a money machine.
 7. Fabrication. Even though it is remarkably easy to construct system rules with great perfor-
mance for the past, some promoters don’t even bother doing this much. For example, one
infamous individual for years repeatedly promoted $299 systems that were outright frauds.
The preceding is not intended to indict all system promoters or those using simulated results.
Certainly, there are many individuals who construct simulated results in appropriately rigorous
fashion. However, the sad truth is that the extraordinary misuse of simulations over many years has
made simulated results virtually worthless. Advertised simulated results are very much like restaurant
reviews written by the proprietors—you would hardly expect to ever see a bad review . I can assure
you that you will never see any simulated results for a system that show the system long the S&P as of
the close of October 16, 1987, September 10, 2001, or March 5, 2010. Can simulated results ever be
used?
 Y es,


s virtually worthless. Advertised simulated results are very much like restaurant
reviews written by the proprietors—you would hardly expect to ever see a bad review . I can assure
you that you will never see any simulated results for a system that show the system long the S&P as of
the close of October 16, 1987, September 10, 2001, or March 5, 2010. Can simulated results ever be
used?
 Y es, if you are the system developer and you know what you’re doing (e.g., use the simulation
methods detailed in the previous section), or, equivalently, if you have absolute faith in the integrity
and competence of the system developer.
 ■ Multimarket System Testing
Although it is probably unrealistic to expect any single system to work in all markets, generally
speaking, a good system should demonstrate profitability in a large majority of actively traded
markets (e.g., 85 percent or more). There are, of course, some important exceptions. A system
employing fundamental input would, by definition, be applicable to only a single market. In addition,
the behavior of some markets is so atypical (e.g., stock indexes) that systems designed for trading such
markets might well perform poorly over the broad range of markets.
314
A Complete Guide to the Futures mArket
In testing a system for a multimarket portfolio, it is necessary to predetermine the relative
number of contracts to be traded in each market. This problem is frequently handled by simply
assuming the system will trade one contract in each market. However, this is a rather naive
approach, for two reasons. First, some markets are far more volatile than other markets. For
example, a portfolio that included one contract of coffee and one contract of corn would be far
more dependent on the trading results in coffee. Second, it may be desirable to downgrade the
relative weightings of some markets because they are highly correlated with other markets (e.g.,
10-year T -notes and 30-year T -bonds).
8
In any case, the percentage allocation of available funds to each market should be determined prior
to testing a system. These relative weightings can then be used to establish the number of contracts
to be traded in each market.
 ■ Negative Results
One should not overlook the potential value of negative results. Analyzing the conditions under which
a system performs poorly can sometimes reveal important weaknesses in the system that have been
overlooked and thus provide clues as to how the system can be improved. Of course, the fact that
the implied rule changes improve results in the poorly performing case does not prove anything.
However, the validity of any suggested rule changes would be confirmed if such revisions generally
tended to improve the results for other parameter sets and markets as well. The potential value of
negative results as a source of ideas for how a system can be improved cannot be overstated. The con-
cept that disorder is a catalyst for thought is a general truth that was perfectly expressed by the late
novelist John Gardner: “In a perfect world, there would be no need for thought. W e think because
something goes wrong.”
The idea of learning from poor results is basically applicable to a system that works in most mar-
kets and for most parameter sets but performs badly in isolated cases. However, systems that exhibit
disappointing results over a broad range of markets and parameter sets are likely to be lost causes,
unless the results are spectacularly poor. In the latter case, a system that exactly reverses the trade
signals of the original system might be attractive. For example, if tests of a new trend-following
system reveal that the system consistently loses money in most markets, the implication is that one
might have accidently stumbled upon an effective countertrend system. Such discoveries may be
difficult on the ego, but they should not be ignored.
Of course, the fact that a system exhibits stable poor performance does not imply that the
reverse system would perform favorably, since transaction costs may account for a significant por -
tion of losses. Thus, the reverse system might also perform badly once these costs are taken into
account, as was the case for the aforementioned well-chosen example described at the start of
8 For purposes of future trading (as opposed to historical testing), historical performance might be a third
relevant factor in determining contract weightings. However, this factor cannot be included as an input in the
testing procedure because it would bias the results.
315
TESTING AND OPTIMIzING TRADING SYSTEMS
this chapter. As another example, at surface glance, reversing the signals generated by a system that
loses an average of $3,000 per year may appear to be an attractive strategy. If, however, two-thirds
of the loss can be attributed to transaction costs, fading the signals of this system will result in a loss
of $1,000 per year, assuming a continuation of the same performance. (The preceding assumptions
imply that transaction costs equal $2,000 per year and that the trades lose $1,000 per year net of
these costs. Thus, reversing the signals would imply a $l,000-per-year gain on the trades, but the
$2,000-per year transaction costs would imply a net loss of $1,000 per year.) Moral: If you are going
to design a bad system, it should be truly terrible if it is to be of value.
 ■ Ten Steps in Constructing and Testing a Trading System
 1. Obtain all data needed for testing. Again, with the exception of short-term trading systems,
which may be able to use actual contract data, the use of continuous futures (not to be confused
with nearest futures or perpetual prices) is highly recommended.
 2. Define the system concept.
 3. Program rules to generate trades in accordance with this concept.
 4. Select a small subset of markets and a subset of years for these markets.
 5. Generate system trading signals for this subset of markets and time for a given par


al contract data, the use of continuous futures (not to be confused
with nearest futures or perpetual prices) is highly recommended.
 2. Define the system concept.
 3. Program rules to generate trades in accordance with this concept.
 4. Select a small subset of markets and a subset of years for these markets.
 5. Generate system trading signals for this subset of markets and time for a given parameter set.
 6. Check to see that the system is doing what was intended. Almost invariably, a careful check will
reveal some inconsistencies due to either or both of the following reasons:
a.
 There are errors in the program.
b. Rules in program do not anticipate some circumstances, or they create unforeseen
repercussions.
Some examples of the latter might include the system failing to generate a signal, given an
event at which a signal is intended; system generating a signal when no signal is intended; system
rules inadvertently creating a situation in which no new signals can be generated or in which a
position is held indefinitely. In essence, these types of situations arise because there will often
be some missed nuances.
The system rules need to be modified to correct both programming errors as well as
unforeseen inconsistencies. It should be emphasized that corrections of the latter type are only
concerned with making the system operate consistently with the intended concept and should
be made without any regard as to whether the changes help or hurt performance in the sample cases used
in the developmental process.
 7. After making necessary corrections, repeat step 6. Pay particular attention to changes in the
indicated signals versus those from previous runs for two reasons:
a.
 T o check whether the program changes achieved the desired fix.
b. T o make sure the changes did not have unintended effects.
 8. Once the system is working as intended, and all rules and contingencies have been fully defined,
and only after such a point, test the system on the entire defined parameter set list across the full
database. Be sure the intended trading portfolio has been defined before this test is run.
316
A Complete Guide to the Futures mArket
 9. As detailed earlier in this chapter, evaluate performance based on the average of all parameter
sets tested or a blind simulation process. (The former involves far less work.)
 10. Compare these results with the results of a generic system (e.g., breakout, crossover moving
average) for the corresponding portfolio and test period. The return/risk of the system should
be measurably better than that of the generic system if it is to be deemed to have any real value.
The preceding steps represent a rigorous procedure that is designed to avoid generating results
that are upwardly biased by hindsight. As such, expect most system ideas to fail the test of merit
in step 10. Designing a system with a truly superior performance is more difficult than most
people think.
 ■ Observations about Trading Systems
 1. In trend-following systems, the basic method used to identify trends (e.g., breakout, crossover
moving average) may well be the least important component of the system. In a sense, this
contention is merely a restatement of Jim Orcutt’s observation that “There are only two types
of trend-following systems: fast and slow .” Thus, in designing trend-following systems, it may
make more sense to concentrate on modifications (e.g., filters and confirmation rules to reduce
bad trades, market characteristic adjustments, pyramiding rules, stop rules) than on trying to
discover a better method for defining trends.
 2. Complexity for its own sake is no virtue. Use the simplest form of a system that does not imply
a meaningful sacrifice in performance relative to more complex versions.
 3. The well-publicized and very valid reason for trading a broad range of markets is risk control
through diversification. However, there is a very important additional reason for trading as
many markets as possible: insurance against missing any of the sporadic giant price moves in the
futures markets. The importance of catching all such major trends cannot be overstressed—it
can make the difference between mediocre performance and great performance. The 2008–
2011 gold market and the 2007–2009 and 2014–2016 crude oil markets are three spectacular
examples of markets that were critical to portfolio performance.
 4. If trading funds are sufficient, diversification should be extended to systems as well as markets.
Trading several systems rather than a single system could help smooth overall performance.
Ideally, the greatest degree of diversification would be achieved if the mix of systems included
countertrend and pattern-recognition systems as well as trend-following systems. (However,
this goal may be difficult to achieve because countertrend and pattern-recognition systems are
generally significantly harder to design than trend-following systems.)
 5. If sufficient funds are available, it is better to trade a number of diversified parameter sets than
to trade a single optimized set.
 6. Generally speaking, the value of parameter optimization is far overstated.
 7. The previous observation strongly suggests that optimized results should never be used for
evaluating the relative performance of a system. Two meaningful methods for testing systems
were discussed in the text.
317
TESTING AND OPTIMIzING TRADING SYSTEMS
 8. So-called simulated results are frequently optimized results (i.e., derived with the benefi t of hindsight)
and, as such, virtually meaningless. This caveat is particularly pertinent in regard to promotions for
trading systems, which invariably use very well-chosen examples.
 9. An analysis of the results of successful systems will almost invariably reveal the presence of many
markets with one or more years of very large profi ts, but few instances of very large single-year
losses. The implication is that a key reason for the s


ngless. This caveat is particularly pertinent in regard to promotions for
trading systems, which invariably use very well-chosen examples.
 9. An analysis of the results of successful systems will almost invariably reveal the presence of many
markets with one or more years of very large profi ts, but few instances of very large single-year
losses. The implication is that a key reason for the success of these systems is that their rules
adhere to the critical, albeit clichéd principle of letting profi ts run and cutting losses short.
 10. A market should not be avoided because its volatility increases sharply. In fact, the most volatile
markets are often the most profi table.
 11. Isolating negative results for a system that performs well on balance can provide valuable clues
as to how the system can be improved.
 12. A frequently overlooked fact is that trading results may often refl ect more information about the
market than the system. For example, in Figure 19.3 , the fact that a trend-following system that
was short in mid-January 2015 would have witnessed the transformation of a large open profi t
into a large loss before the system provided a liquidation or reversal signal would not necessarily
refl ect inadequate risk control. Virtually any trend-following system would have experienced
the same fate.
 This example illustrates how the value of a system cannot be judged in a vacuum. In some
cases, poor performance may refl ect nothing more than the fact that market conditions would
have resulted in poor results for the vast majority of systems. Similarly, favorable results may
also refl ect the conditions of the market rather than any degree of superiority in the tested
 FIGURE  19.3 Trading Results Refl ect Market, Not System: Short Swiss Franc Continuous
Futures
Chart created using TradeStation. ©TradeStation T echnologies, Inc. All rights reserved.

318
A Complete Guide to the Futures mArket
system. These considerations suggest that a meaningful assessment of a new system’s perfor-
mance should include a comparison to a benchmark (e.g., the corresponding performance of
standard systems, such as a crossover moving average or a simple breakout, during the same
period for the same markets).
 13. Use continuous futures prices for testing systems.
 14. Use only a small portion of the database (i.e., some markets for only a segment of the full time
period) for developing and debugging a system.
 15. Use charts with superimposed signal annotations as an aid to debugging systems.
 16. In checking the accuracy and completeness of the signals generated by a system, make changes
dictated by deviations from the intended operation of the system (due to oversights related to
the full implications of the rules employed or unforeseen situations) with complete disregard
for whether such changes increase or decrease profits in the sample tests.
319
 ■ Why Return Alone Is Meaningless
Y ou are looking for a London hotel room on the Internet. Y ou find the same hotel room at two differ-
ent sites (both including taxes) at two different prices:
 ■ Site A: 300
 ■ Site B: 250
Which is the better deal? The answer may seem obvious, but it’s not. On one occasion, when I
posed this question to a conference audience, one attendee shouted the response, “It depends whether
they both include breakfast.” “That would have to be a very expensive breakfast,” I answered. But at
least he had the right idea. The question I posed contained incomplete information. I didn’t specify
what currency the prices were quoted in. What if the 300 price was in dollars and the 250 price was
in pounds (let’s say when the pound was at $1.40)? Changes everything, doesn’t it?
“W ell,” you are probably thinking, “no rational person will ignore the currency denomination in
comparing two prices, so what’s the point?”
 The point is that investors make this type of error all the
time when selecting investments by focusing only on returns. Comparing returns without risk is as
meaningless as comparing international hotel prices without the currency denomination. Risk is the
denomination of return.
How to Evaluate
Past Performance*
Chapter 20
* This chapter is adapted from Jack D. Schwager, Market Sense and Nonsense: How the Markets Really Work (and How
They Don’t) (Hoboken, NJ: John Wiley & Sons, 2012).
320
A Complete Guide to the Futures mArket
Consider the two managers in Table 20.1. Assuming the two managers are considered qualitatively
equivalent, which is the better-performing manager? 1 Many investors would opt for Manager B,
reasoning, “I am willing to accept the higher risk to get the higher return potential.” But is this
reasoning rational? In Table 20.2 we add a third investment alternative—leveraging an investment
with Manager A at 300 percent.
2 The leveraged investment with Manager A now has both a higher
return and lower risk than Manager B. So even risk-seeking investors should prefer Manager A,
using a leverage factor that raises return to the desired level.
One can picture risk as a hole—the deeper the hole, the greater the risk—and return as a pile of
sand. Leverage is the shovel that, if desired, allows transferring some of the sand from the risk hole
to the return pile, thereby increasing return in exchange for accepting greater risk—a trade-off that
may be preferred if the risk level is lower than desired. Continuing the analogy, by using negative
leverage (i.e., holding more cash), it is also possible to transfer sand from the return pile to the risk
hole, thereby reducing risk in exchange for accepting lower return. In this sense, risk and return are
entirely interchangeable through leverage (that is, through varying exposure).
table 20.1 a Comparison of two Managers
return risk (Standard Deviation) return/risk ratio
Manager A 10% 5 2:1
Manager B 25% 25 1:1
table 20.2 a Comparison of two Managers revisited
return risk (Standard Deviation) return/risk ratio
Manager A 10