CHAPTER 9

The 10 Commandments of Applied Time Series Forecasting for Business and Economics

One of the most important elements of today's decision-making world, in both the public and the private sectors, is the forecasting of macroeconomic and financial variables. For instance, the key driver of borrowing costs is short-term interest rates. The Federal Reserve Board's federal funds target rate, the primary short-term interest rate, has been around 0 to 0.25 percent for more than four years (December 2008 to the time of this writing, July 2013). Decision makers thus ask: How much longer will the Fed leave rates low? Although several factors influence the Fed's decisions on rates, the two main economic factors are inflation and unemployment.¹ The Fed may begin to raise the federal funds target rate if (a) inflation expectations for the next one and two years continuously stay above its inflation target of 2 percent and/or (b) the unemployment rate falls below 6.5 percent.² Accurate forecasts of inflation and unemployment can help a decision maker predict the likelihood of a rate hike by the Fed.

During the past few decades, econometric model-based forecasting has become very popular in the private and the public decision-making process. In this chapter, we present 10 commandments of applied time series forecasting that an analyst should learn. These commandments, in our opinion, help produce accurate forecasts. The commandments are:

1. Know What You Are Forecasting
2. Understand the Purpose of Forecasting
3. Acknowledge the Cost of the Forecast Error
4. Rationalize the Forecast Horizon
5. Understand the Choice of Variables
6. Rationalize the Forecasting Model Used
7. Know How to Present the Results
8. Know How to Decipher the Forecast Results
9. Understand the Importance of Recursive Methods
10. Understand Forecasting Models Evolve over Time

Our discussion is based on “Six Considerations Basic to Successful Forecasting” in Elements of Forecasting by Francis Diebold and A Companion to Economic Forecasting by M. P. Clements and David F. Hendry.³

COMMANDMENT 1: KNOW WHAT YOU ARE FORECASTING

An analyst must know what he or she is going to forecast, because everything depends on it: the appropriate forecast method and the potential cost to forecasting (the forecast error). We divide a forecast objective into four broad groups:

1. Event's Uncertain Outcome Forecast, Certain Timing

   The first category of forecasting is the event's outcome forecast when timing is known. For instance, the U.S. Bureau of Labor Statistics (BLS) releases every month the nation's employment report containing the change to nonfarm payrolls and the unemployment rate. The change in the nonfarm payroll number is a key driver of the financial market. A forecaster knows the timing of the event and the release of nonfarm payroll numbers, but he or she doesn't know the outcome of the event—have payrolls stayed the same, have they increased or decreased, and by how much? This category of forecast objective is common in economic and business forecasting.

2. Timing of the Event Uncertain, Outcome Known

   The second category of forecasting objective is to forecast the timing of the event when the outcome is known. For instance, in business cycle analysis, a forecaster knows the next phase of the event but does not know when it will begin. This type of forecasting is less frequent than the first category of forecasting but is very important. Policy makers and business leaders give considerable weight to business cycle analysis in their decision-making process. For example, government spending usually increases during a recession as they try to fill the gap left by the private sector. In addition, the Fed cuts its target rate.⁴ Another example is the behavior of the Standard & Poor's (S&P) 500 index. During a recession, the S&P 500 index usually trends down, but at some point the S&P 500 index will come back. The timing of such a reversal, however, is unknown. A forecaster may want to forecast the timing of the bull or bear market or the next recession.

3. Bubble Forecast

   The third forecast objective, and the one that we believe is the hardest part of forecasting, is to forecast asset bubbles.⁵ Because of the negative implications, we want to forecast most bubbles and know when (the timing of the event) and where (in which sector) the bubble will take place. We do not know when we will experience the next bubble or in which sector it will occur. Therefore, the timing, the location and the outcome of the event are all unknown.

4. Time Series Forecast

   A common objective in economics and business is to forecast for a particular time period, such as predicting gross domestic product (GDP) over the next eight quarters. Often we know the timing of the event. In this case, we know when the Bureau of Economic Analysis (BEA) will release the next quarterly GDP data and wish to forecast the outcome of the event—the expectations of GDP values for the next eight quarters.

COMMANDMENT 2: UNDERSTAND THE PURPOSE OF FORECASTING

Once we know our forecast objective (what we are going to forecast), then the natural question would be: What is the cost of getting it wrong? What is the forecast loss function? We believe that there is a bridge between forecast objective and forecast error, and the bridge is: Why forecast? Forecast loss functions depend on two elements: the forecast objective and the reasons for forecasting.⁶

Let's pretend there are two forecasters, A and B. Both have the same objective, which is to identify when the recovery phase of a recession will begin. It is an event timing forecast, with the event outcome, the beginning of the recovery phase, known. Let us assume forecaster A is a firm manager. His firm produces consumer goods that have the pro-cyclical behavior of higher demand in a recovery and expansion and less demand in a recession. The firm forecasts for an economic recovery and produces a large number of products, assuming higher-than-normal demand this season. The economy, however, does not recover, and the firm cannot sell all its products, thereby incurring a loss. Forecaster B, in contrast, is an independent forecaster and forecasts frequently. She believes that once she predicts a recovery correctly, she will receive recognition from the media and forecasting peers; if wrong, there will be no tangible consequences. As a result, she will continue to forecast, even if previous forecasts missed the timing. Forecaster A, compared to the Forecaster B, does not have the luxury of missing the forecast often.

COMMANDMENT 3: ACKNOWLEDGE THE COST OF THE FORECAST ERROR

The third commandment stresses that a forecaster must know the cost of forecast error. Forecasters rarely predict the actual number, so there are forecast errors. For example, Forecaster A projects next quarter's real GDP growth rate as 3.5 percent and Forecaster B sees a rate of 1.5 percent. The BEA releases the real GDP growth rate, which is 2.5 percent. Both forecasters are wrong. However, Forecaster A's 3.5 percent growth rate is higher than the actual growth rate; he has over-forecasted. Forecaster B, who predicted a 1.5 percent growth rate, under-forecasted the rate of real GDP growth. These are the two types of forecast errors (the difference between the actual and the forecasted) we may face. Every forecast error is associated with a cost. The question is whether the cost of over-forecasting is the same as the cost of under-forecasting.

Set Y as the actual value of the target variable, GDP growth rate for example, and as the forecast of Y.⁷

where

e = forecast error

e = 0 means that the forecast is equal to the actual

e ≠ 0 indicates the forecast is different from the actual

If both Y and are greater than 0, then e >0 implies under-forecasting and e< 0 implies over-forecasting.

Over time an analyst would produce several forecasts for a variable; that is, one-quarter-ahead GDP growth rate forecasting during the 2006:Q1 to 2012:Q4 period implies 28 forecasts.⁸ The e_t is the forecast error for each period and if we aggregate errors from each period then we can call it L(e_t) which indicates the total cost associated with the forecast errors, e_t, and is known as the loss function.

where

e_t = forecast error for each quarter time series

Y_t = actual values of the GDP growth rate during the 2006:

Q1 to 2012:Q4 period time series

_t = forecasts for the Y_t during that time period

Symmetric versus Asymmetric Loss Function

The loss function, L(e_t), indicates the cost of being wrong. If we rephrase our previous question in terms of a loss function, then if the loss function is identical, it is a symmetrical loss function. A symmetric loss function implies that e_t >0 (under-forecasting) and e_t <0 (over-forecasting) are both associated with an identical total loss. A simple example of a symmetric loss function would be forecasting the experience of a small firm that produces consumer goods. In a regular season, the firm sells 100,000 units, with a per-unit profit of $2.00. Unsold units would have to be stored at the cost of $2.00 per unit. We generate two scenarios to show a symmetric loss.

The symmetric loss function is not the only possibility. The loss function can be asymmetric, implying that the loss from under-forecasting, e_t > 0, is not identical to the loss from over-forecasting, e_t < 0. In reality, loss functions are most often asymmetric. We again follow the above example but this time assume that the loss function is asymmetric. Suppose another storage facility opens in the town and charges only $1.00 per unit. Assuming the same scenarios as above, in the first scenario the firm expects to sell 100,000 units when the actual demand is 125,000 units, missing out on $50,000 in additional profits.

In the second scenario, the firm forecasts for demand of 125,000 units but can sell only 100,000 units. However, with the new storage facility that costs only $1.00 per unit, the firm's over-forecasting cost is only $25,000. Therefore, the cost of under-forecasting (a lost opportunity of additional profit of $50,000) is higher than the cost to over-forecasting (storage cost of $25,000).

Here is another example of an asymmetric loss function. In the leisure and hospitality industry, it is important to forecast the number of visitors. Let us assume the number of visitors a hotel expects to host over the next month is 2,000. The hotel's manager makes arrangements for 2,000 visitors. If the number of visitors exceeds that number, the arrangements will not be enough. The hotel may lose some business, and many visitors may be unhappy with their stay. Under-forecasting may lead to customer dissatisfaction. But over-forecasting will lead to monetary loss as the firm makes unnecessary arrangements. This hotel places greater importance on customer satisfaction than on monetary loss; therefore, the loss function is not symmetric.

Linear versus Nonlinear Loss Function

Up to this point we have discussed the cost associated with being wrong in forecasting. Another important issue would be whether the loss function is linear (the cost of each unit of forecast error is the same) or nonlinear (the cost varies between forecast errors). For instance, from the last example, the hotel predicts that 2,000 people will visit next month, but 500 more people arrive than expected. Now the question is: Among those additional 500 people, does each impose the same loss to the firm? To answer this question, we generate two scenarios. First let us assume that each additional family costs $100. The hotel's loss is thus $50,000 (500 × $100). Since the cost of each forecast error is identical, the loss function illustrates a linear cost function. Now let us assume that the first additional family costs $100, the second family costs $102, the third family $105, and so on, and each unit of forecast error has a higher cost than the previous error. This is an example of a nonlinear cost function.

A further nuance is that we may want to determine if the linearity of the loss function is the same for under-forecasting and over-forecasting, or if the two scenarios have two different forms of a loss function. The simple cases are (1) a linear loss function for both under-forecast and over-forecast and (2) a nonlinear loss function for both under-forecast and over-forecast. The complex case would be, for example, a linear loss function for under-forecast but a nonlinear loss function for over-forecast, and vice versa. For example, the hotel forecasts 2,000 visitors, but only 1,500 visitors show, leading to an over-forecast of 500 families. Each missing visitor costs $50 to the hotel with a total loss of $25,000: a linear loss function. If the hotel forecasts 2,000 visitors but actually there were 2,500, there is an under-forecast of 500 visitors. Each extra visitor costs $100 with a total loss of $50,000, again a linear loss function. Overall, the hotel has an asymmetric loss, where the loss from over-forecasting is different from the loss of under-forecasting.

Another possibility of loss function would be an asymmetric nonlinear loss function. A simple example: The forecast for visitors was 2,000, and only 1,500 families showed (over-forecasting). Each missing family costs $50 for a total loss of $25,000. In the case of under-forecast, 2,500 families visit while the forecast was 2,000 families, and each family costs more than the prior family. This would be a nonlinear asymmetric loss function scenario.

To sum up this section, a forecaster must know whether their loss function is (1) symmetric or asymmetric, (2) linear or nonlinear, (3) symmetric and linear or nonlinear, or (4) asymmetric and linear or nonlinear.

COMMANDMENT 4: RATIONALIZE THE FORECAST HORIZON

The fourth commandment of the economic and business forecasting states that a forecaster must consider the forecast horizon and realize that accuracy depends on that horizon. Simply put, the forecast horizon is how far out we are looking to forecast. Do we want a daily forecast of the S&P 500 index? A weekly forecast of initial jobless claims? Or a monthly forecast of the unemployment rate? The forecast horizon can be one day or up to several years. We can divide the forecast horizon into short-term forecasting (usually up to two years in most macroeconomics applications) and long-term forecasting (typically for utility and energy development application). The reason for this division is that short-term forecasting requires different treatments from long-term forecasting.

Short-Term Forecasting

Broadly speaking, short-term forecasting is for a short time period in which the chance of a big change (structural change, regime shift, policy change, etc.) is very low. For instance, a one-month-ahead forecast of the S&P 500 index or a firm's forecast for next quarter's earnings would not likely be subject to a significant structural change in the model. During short-term forecasting we may see some significant change—a natural disaster, for instance—but chances are less likely when compared to a long-term forecast.

Long-Term Forecasting

When forecasting for a long time period (at least a couple of years ahead), the chances of significant changes are very high. For instance, the forecast of the S&P 500 index beyond the next two years is not easy because of the high probability of a significant change in the model for equity returns. During January 2008, the U.S. economy was in recession and the S&P 500 index remained in bearish territory. Let us assume we were sitting in January 2008 and making a two-year-out forecast for the S&P 500 index. This forecast would be difficult because if the recession not only ended in 2008 but was followed by a strong recovery, then the S&P 500 index may quickly turn bullish.¹⁰ One thing is clear: Longer-term forecasting is associated with higher uncertainty than a shorter-term forecast horizon is.

The challenge for forecasters is that the longer the forecast horizon, the degree of confidence in that forecast declines and the range of possible outcomes rises. The magnitude of the forecast error tends to increase as we lengthen the forecast horizon. A forecaster thus must know about the forecast horizon and whether it is short term or long term. He or she also must be aware that the definition of short-term and long-term is subjective and varies with respect to the subject.

An important point an analyst should keep in mind during the modeling for long-term forecasting is that many macroeconomic variables behave differently during different phases of a business cycle. The U.S. unemployment rate tends to rise during recessions and fall during expansions. A forecaster interested in forecasting the unemployment rate for the next six to eight years should consider business cycle movements because the average business cycle duration, defined as trough to trough post–World War II, is around 70 months. Another important consideration is policy change. Political parties have different tax and spending preferences, and which party is in power can affect long-term forecasting. The possibility of a structural change, due to internal or external shocks, is higher during long-term forecasting than during short-term prediction.¹¹

COMMANDMENT 5: UNDERSTAND THE CHOICE OF VARIABLES

Once a forecaster has determined his or her forecast objective, loss function, and forecast horizon, the next step is the choice of the variables or available information set to build a forecasting model. For instance, what are the appropriate datasets and econometric tools for forecasting? In this section we discuss the data choice and issues related to the dataset. In the next section we focus on the methodology.

The dataset can be divided into two categories: (1) the variable we want to forecast—the dependent variable, and (2) the variables that help us to forecast—the independent variables (the predictors).

First, a forecaster must have a comprehensive understanding about the dependent variable. For instance, what is the source of the data—are you compiling external datasets or internal sources (like revenue and sales)? What is the frequency of the data: weekly, monthly, quarterly, or annual? You should also know the history of the dependent variable and how far the data collection goes back. The longer the history, the better. The next step is to find out what sort of independent variables are available. There are issues related with the choice of independent variables as well. The choice of independent variables depends on the forecast horizon (whether long term or short term), release date, frequency of the variables, and history of the variables.

Some examples show how the choice of the dataset varies with the objective of forecasting. The first example involves a small firm that produces consumer products and sells them only in North Carolina. The firm's manager wants to build a forecasting model to predict next season's demand, a short-term forecast for one period ahead. In addition, the market area consists of only one state, so the chance of economic shocks and uncertainty is small. The forecasting model thus would rely heavily on North Carolina–specific variables, such as state employment and state personal income, which are potential predictors for consumers demand.

Another example involves economists at the Federal Reserve Board interested in building an econometric model to predict the overall U.S. economic performance for the next five years, a long-term forecast. During such a lengthy period, the probability of shocks is high, increasing the forecast uncertainty. This model would contain information from major sectors of the economy (i.e., variables representing consumers, investors, government, housing, and labor). The United States is an open economy, and international trade is also an essential element of the economy. Some measures of the world economy would need to be included in the model. In sum, in long-term forecasting, we often need to include more information in the models and consider the possibility of shocks and uncertainty.

An analyst must also consider how many predictors should be included in a forecast model. The answer, to some extent, depends on the forecasting objective. In short-term forecasting, the most recent information of predictors is a key to successful forecasting. Typically, five to seven variables are included as predictors.¹² In long-term forecasting, however, an analyst may want to include more variables to ensure the model captures the wide range of factors that may influence the target variable(s) over an extended period.

There is a balance. We want to include important information in the model, but, at the same time, we should not include too many variables in a traditional econometric modeling framework because that creates over-fitting and/or degree-of-freedom issues. But excluding essential predictors would lead to under-fitting, which reduces the predictive power of the model.¹³

COMMANDMENT 6: RATIONALIZE THE FORECASTING MODEL USED

The sixth commandment emphasizes selecting an econometric method for the forecasting. There are a number of econometric tools commonly used in today's time series forecasting world. Each econometric method has some advantages and some limitations. We suggest selecting an econometric method consistent with the forecasting objective.¹⁴ Here are a few examples that show different forecasting techniques that can be used for different forecasting objectives.

Suppose an analyst is interested in forecasting the daily closing value of the S&P 500 index. There is not much economic information available with a daily frequency that can be included in a forecasting model, and it is not clear that much of this economic information has an impact on equity prices. One standard econometric tool used in these kinds of scenarios is the ARIMA model.¹⁵ In contrast, to predict a monthly data series, such as a one-month-forward forecast for unemployment, an analyst could use the econometric technique known as the vector autoregressive (VAR) approach, which uses economic variables as potential predictors (e.g., interest rates and inflation).¹⁶ Forecasting the daily close of the S&P and the monthly unemployment rate represent a short-term forecasting approach that uses two different models. The difference between the approaches, however, is that in case of the unemployment rate forecasting, the model includes economic predictors, and the forecast is based on these predictors.

A final example represents the long-term forecasting experience: a five-year-ahead forecast of the U.S. GDP growth rate. Two econometric methods are commonly used for long-term forecasting. First, a macroeconometric model, also known as a structural model, includes hundreds of variables. This model is purely based on economic relationships that reflect economic theory. A system of equations is built based on economic theory and then estimated simultaneously with the help of econometric techniques, such as two-stage least square (2SLS).¹⁷ A benefit of this approach is that much information can be included in the model, but one limitation is that we have to estimate sometimes hundreds of equations. The alternative to the large-scale macroeconometric model is the VAR.¹⁸ Typically in a VAR model, we include eight or more variables representing major sectors of an economy, implicitly assuming that everything depends on everything.¹⁹ VAR modeling, relative to the macroeconometric models, is easy to estimate; that is one reason why it is commonly used in forecasting.

Summing up, there are number of econometric techniques commonly used to forecast economic and financial variables. We suggest selecting a technique that is consistent with the forecasting objective. The approach, in our view, should not be too technical/heavily mathematical and ignores economic theory or practical realities, or too simple and ignores the econometric principles. The appropriate approach is nicely summarized by Diebold (2007) and is known as the KISS principle: “Keep it sophisticatedly simple.”²⁰

COMMANDMENT 7: KNOW HOW TO PRESENT THE RESULTS

An analyst can summarize and present forecasting results in different ways and which way is better depends on the objective of the forecast. That is, the results can be summarized into a single number (known as point forecast), a range of numbers (an interval forecast), probability distribution of the number (density forecast), probability of an event (probability forecast), and conditional forecasting (scenario-based analysis).

A point forecast is a widely used approach in both the private and the public sectors. An example of a point forecast is when analysts predict a 2 percent (a single number) GDP growth rate for the next quarter. The major benefit of this approach is that it is easy to present, understand, and digest. However, it would be better to provide a range of the forecast—in the 1.5 to 2.5 percent range (an interval forecast) since a point forecast suggests a degree of preciseness seldom achieved in forecasting. Typically, a specific level of confidence is attached with the forecast interval, that is, a forecaster is 95 percent confident that the actual GDP growth rate would fall in the 1.5 to 2.5 percent range. There are greater chances that the actual number would fall in the range than that the point forecast would match the actual number. So the interval forecast provides more information than the point forecast, but the interval forecast also requires more calculation (i.e., estimation of the 95 percent confidence interval).

Another way to present forecast results is to provide the entire probability distribution of the forecast, termed a density forecast. We assume that the forecast is normally distributed with a mean μ and standard deviation σ. The 95 percent confidence interval (a forecast interval) can be calculated as μ ± 1.96σ, and the entire probability distribution (different confidence levels, 90 or 99 percent, etc.) of the forecast is called the density forecast.²¹ The difference between an interval forecast and a density forecast is that an interval forecast attaches with one confidence level (in the present case, 95 percent) and a density forecast indicates any desired confidence level. Put differently, the 2 percent GDP growth rate is a point forecast, 2 ± 0.5 (the 1.5–2.5 percent range) is interval forecast (we attach 95 percent confidence interval to that range), and the complete probability distribution (any desired confidence level) of the forecast is the density forecast. The density forecast provides more detail than the point and interval forecast, but it also requires a lot of simulation, which is one reason the density forecast is not commonly used in the forecasting world.

There are a couple of other ways to present forecast results, and one is the probability forecast (i.e., the probability of an event's occurrence). A common example of the probability forecast is the recession probability—for example, the six-month forward probability of a recession is 20 percent. One way to predict a recession is to build a Probit model and produce a recession probability.²² The probability forecast is different from a point forecast in the sense that the probability forecast states the probability of an event's occurrence (the probability of a recession) and the point forecast is a single number (forecast for a GDP growth rate).²³

An increasingly popular way to present forecast results is known as scenario-based analysis—for example, different possible paths of the GDP growth rate for the next three years would be (1) mild growth rates scenario (e.g., less than 1 percent growth rate); (2) possibility of normal growth rates in the range of 2 to 3 percent; and (3) a path of strong (over 3.5 percent) growth rates. Scenario-based forecasting is gaining attention because of its flexibility and ability to provide opportunities for decision makers to consider different possible outcomes of economic growth. Remember, scenario-based forecasting is different from interval forecasts in the sense that scenarios are conditioned on the predictors and represent different paths of the target variable. Interval forecasts, however, state a range of forecasts, which usually represent just one possible path because it conditioned on only one set of predictors' values.

Summing up, point forecasting is good for short-term (one period ahead) forecasting and for medium-term forecasting (one year ahead), while interval and density forecasting provide a wider range of outcomes and better represent the possible range of real-world outcomes. At the same time, point forecasting is relatively easier to estimate than interval forecasting, and interval forecasting is easier to calculate than density forecasting. In the case of an event's timing (we know the outcome but the timing of the event is unknown) or event's outcome (we know the timing of the event but outcome is unknown) forecasting, probability forecasting is a better option. For long-term forecasting (more than a few years ahead) and policy making, scenario-based forecasting is a better approach.

COMMANDMENT 8: KNOW HOW TO DECIPHER THE FORECAST RESULTS

As mentioned, it is extremely difficult to predict the point outcome perfectly every time, and the possibility of a nonzero forecast error is often high. In the case of a time series forecast, where an analyst produces a forecast on a regular basis, forecast errors can be represented by a time series. For example, from 2008 to 2012, we submitted forecasts for the unemployment rate to Bloomberg for every month. If we calculate our forecast errors for that time period, we get a time series that consists of our forecast errors, which we will call e_t. The e_t contains 60 observations (60 months of forecast). Some are zero (in the case of a perfect forecast), positive (under-forecasting), and negative (over-forecasting). Given the fact that we do not perfectly forecast the unemployment rate every time, what we need is to estimate two measures of the forecast error: (1) What is the average forecast error? (2) On average, how many times did we predict the direction of the change correctly (directional accuracy)?

There are several benefits to calculating these two statistics. One major benefit is that the average forecast error indicates the average deviation of the forecast from the actual reported outcome and can be utilized to calculate the forecast interval. For instance, the average forecast error for the unemployment rate model is 0.2 percent. If the model predicts an 8.0 percent unemployment rate for the next month, the forecast interval would be 7.8 to 8.2 percent (forecast ± average forecast error). The average forecast error can also be employed as a benchmark to select or compare different models or two or more subsamples for the model. From the example of the unemployment rate model, at the end of 2009, we can calculate the average forecast error for 2009 and compare it with the average forecast error for 2008. If the model for 2009 produces a smaller average forecast error than the model for 2008, we can say the model performs better in 2009 than it does in 2008, all else being equal. If we want to select one model among competitors, the model with the smallest average forecast error would be our choice.

Directional accuracy provides several benefits as well. Let us assume that the directional accuracy of the unemployment rate model is 80 percent (80 percent of the time, the model correctly predicted the direction of the actual number), and the model predicts an 8.1 percent unemployment rate for the next month, assuming that the current month's unemployment rate is 8.0 percent. The 80 percent directional accuracy shows that there is an 80 percent probability that the unemployment rate would increase during the next month.

So a better way to select a model is to consider both the average forecast error and the directional accuracy measures. A model with the smallest average forecast error and the highest average directional accuracy will be the best among its competitors.

How does an analyst estimate the average forecast error and the average directional accuracy? There are several ways. The out-of-sample root mean square error (RMSE) is utilized as a measure of average forecast error.²⁴ Equation 9.3 is employed to calculate the RMSE:

where

_t+1 = one-period-ahead forecast

Y_t+1 = actual value of the target variable

The magnitude of this statistic indicates the average forecast error over time. Furthermore, a model with a smaller RMSE is the better model among its competitors for a particular variable.

The out-of-sample RMSE is a very good measure of forecast evaluation. However, in practice, and in the financial sector in particular, the direction of the actual variable is also very important since most hedged positions are based on a directional change rather than just the magnitude of the change. To make a financial profit and/or to reduce financial losses, it is imperative to predict the direction of the variable. Since many macroeconomic variables are reported either in percentage change or net change, the sign (positive or negative) of the actual variable is also crucial. Equation 9.4 is used to estimate the directional accuracy:

where

_t+1 = forecast.

Y_t+1 = actual value of the target variable

In addition, if the forecast shares an identical sign (plus or minus) with the actual variable, then the direction is correct. For average directional accuracy, the following equation can be used:

where

X = number of forecasts that have the right direction

N = total number of forecasts

We convert the ratio into a percentage by multiplying by 100.²⁵

There are a few variables that are alternatively reported in level form such as the unemployment rate, and ISM (the Institure for Supply Management) manufacturing index, and so on. We can use equation 9.5 to compute the directional accuracy of the forecast for those variables:

If the difference between the actual current month and the prior month values (Y_t − Y_t+1) has the same sign as the difference between the forecast and the actual prior month value of the time series (Y_t − _t+1), then the direction of the forecast is correct. From the previous example, if the forecast was 8.1 percent and the actual unemployment rate came in at 8.2 percent, then the model was accurate in terms of direction.

In sum, a forecasting approach just based on the RMSE may not provide the most opportunity for a firm to generate financial gains. The first step to a more accurate forecasting approach would be that the forecast should be close to the actual or have a minimum average forecast error (RMSE). The second step should be that the direction of the forecast would also be accurate (directional accuracy), on average.

COMMANDMENT 9: UNDERSTAND THE IMPORTANCE OF RECURSIVE METHODS

At the end of the eighth commandment, a forecaster would know his or her forecasting objective and loss function, have selected variables of interest and econometric methods, and would be familiar with the forecast evaluation measures. While it seems that he or she is ready to forecast the target variable, we propose one more step in the process of building a forecasting model, especially in time series forecasting. The forecaster should conduct a controlled forecasting experiment, also known as the recursive method, before finalizing a forecasting model. For instance, consider an analyst interested in building a forecasting model for the unemployment rate. He or she has selected the potential predictors as well as an econometric methodology. Let us assume that, for the time series model, the analyst has picked a monthly dataset for the 1980 to 2012 time period. The objective is a one-month-ahead forecast for the unemployment rate.

In a controlled forecasting experiment, we suggest running a regression analysis using the dataset for the January 1980 to December 2005 period and generating a forecast for January 2006.²⁶ Then we suggest repeating the regression analysis to include January 2006 values; the forecast will be generated for February 2006. Then the regression analysis is repeated to include February 2006 values, and prediction will be made for March 2006, and so on. This recursive method will be repeated until the analyst reaches the sample end point. In this case, November 2012 will be the end point for regression analysis, and the last forecast will be made for December 2012.

The key benefit of this controlled forecasting experiment is that, at the end of the experiment, the analyst will have seven years (January 2006–December 2012) of out-of-sample forecast data and the corresponding actual unemployment rate for that time period. The analyst can then estimate the out-of-sample RMSE and average directional accuracy from that dataset, figures that can be utilized for several purposes. The RMSE and directional accuracy can be employed to select the final model specification, if that is needed. That is, for instance, there are 10 potential predictors, and the analyst is only interested in including 5 predictors in the final model. The set of 5 predictors that produce the smallest RMSE and the highest directional accuracy would be the best model among competitors. The RMSE and directional accuracy would also shed light on the likely average forecast error as well as directional accuracy for the future time period.

The most important choice is the selection of the time period for the controlled forecasting experiment. In the present case, that would be January 2006 to December 2012. Typically, macroeconomic and financial variables perform differently during different phases of a business cycle. A good model should perform accurately during different phases of a business cycle. In the present example, the unemployment rate tends to rise during recessions and falls during expansions, and we want a model that predicts unemployment rates accurately during recessions as well as expansions. The selected time period (January 2006–December 2012) for the experiment represents different phases of a business cycle. January 2006 to November 2007 indicates the pre-recession era, the December 2007 to June 2009 period is associated with the Great Recession, and the post–June 2009 period shows a recovery/expansionary time period. If the model performed accurately during the 2006 to 2012 time period, then an analyst can expect that the model would perform accurately in the future as well.

In sum, we suggest that the final step of a time series forecasting model should include a controlled forecasting experiment. The time period selected for the controlled experiment should include different phases of a business cycle and be able to forecast accurately during different phases of a business cycle.

COMMANDMENT 10: UNDERSTAND FORECASTING MODELS EVOLVE OVER TIME

The last commandment implies that there is no silver bullet in applied time series forecasting. This is a point that is often neglected in traditional forecasting literature. Once a forecasting model is built, all too often a forecaster assumes mission accomplished. Based on personal experience, the actual job is just beginning: The need to evaluate and maintain accuracy of the forecast model continues over time. If a model has been built and is performing accurately, should a forecaster relax and assume it will always remain accurate? The simple answer is no. Forecasting models are like sailing a boat; they require constant adjustments to the financial and economic winds and the currents. Many analysts look for a best forecasting model, and once it is found they stop searching. This is wrong. The world changes constantly. A finalized model will almost certainly break down over time, as the economy and the relationships among economic variables evolve. One well-known example is that of the Phillips curve, which found that higher wage inflation was associated with a lower unemployment rate. Although the relationship appeared to work very well during in the 1960s, the model fell apart in the 1970s.²⁷

A useful tip to a forecaster, especially in short-term forecasting, is that once a time series forecasting model is finalized, the next procedure should be used to evaluate and maintain its performance. If the forecast missed the direction or the forecast errors are larger than the RMSE for a certain period of time (say, three consecutive months in a monthly dataset), the forecaster should consider revising the model. A best practice is to evaluate all of the models at the end of every year and compare the current year's performance with the previous year's using the real-time out-of-sample RMSE as well as directional accuracy criteria. If a model has a higher RMSE along with a lower directional accuracy compared to the previous year, then the analyst should reconstruct the model using the procedure for controlled forecasting.²⁸ In practice, analysts must build forecasting models then continuously evaluate their performance and, if necessary, revise the models.

The 2007 to 2010 time period was very tough for time series forecasting. There are several reasons for this: the Great Recession, the oil price spike, the housing market crash, the financial crisis/credit crunch, and different types of stimulus packages. These factors made real-time, short-term forecasting harder, and the life span of many models became shorter than normal.²⁹

SUMMARY

In summary, we suggest that time series forecasting has two phases: In the first phase, a forecaster needs to know the forecasting objective and loss function, select the dataset (dependent and independent variables), econometric methodology, and then finalize a model based on the controlled forecasting experiment. In the second phase, a forecaster should continuously monitor and maintain forecasting performance of the models. When the model breaks down, as it eventually will, the forecaster must construct a new model following the approach adopted in the first phase. Because one model specification will not remain accurate forever, and even the best model specification may need to be revised at some point, time series forecasting is an evolving process.