Conclusion

The papers given here make up a large part of the research that lay behind the December 2002 Reserve Bank Bulletin article "The Reserve Bank's forecasting performance". However, they are not the full story. As well as the articles included here there was some work that reached dead ends. For example, running `counter-factual' experiments using the FPS model turned out to be infeasible, and econometric analysis was plagued by technical issues and a lack of data. There was also much verbal discussion around the issues and historical context that cannot be captured in these written documents but which influenced thinking and conclusions.

Nonetheless, these documents provide much of the background research that led us to the conclusions contained within the aforementioned Bulletin article. It has been an interesting and worthwhile task analysing our forecast errors. Although we have not concluded that the results point to a need to immediately address any issues with our current forecasting approach, the findings reiterate the importance of continually reassessing our understanding of the economy.

Definition of forecast error statistics

To assess forecasting performance, three basic measures of accuracy are calculated: the mean error (ME), the mean absolute error (MAE) and the root mean square error (RMSE).

The ME allows us to examine for the presence and direction of bias in the forecasts. When examining forecasts of inflation, for example, a positive ME indicates that on average we tend to over-predict the level of inflation, while a negative value would suggest that on average we under-predict it. We examine whether the bias in the forecast errors is significantly different from zero using t-tests.1 When forecasts from different organisations are compared, F-tests are used to determine whether the mean forecast errors are statistically different from the Reserve Bank's mean forecast errors at each horizon. The mean error is defined as:

where T = number of observations

Ft = forecast of component
At = actual outturn

The MAE allows us to examine the size of our forecast errors. This approach assumes that the seriousness of a forecast error increases in a linear manner (eg a 2 per cent error is twice as serious as a 1 per cent error). The mean absolute error is calculated as follows, with the variables defined as above:

An alternative means of examining the size of our forecast errors is the RMSE. This measure assumes that larger forecast errors are of greater importance than smaller ones; hence they are given a more than proportionate penalty. The root mean square error (RMSE) is defined as:

Editor's note

Regrettably the formulae were given in the incorrect order in the December 2002 Bulletin article.

Glossary of acronyms

¹ Note: in certain cases forecast errors are not normally distributed. This tends to occur due to the limited sample sizes. In such cases we test if the median forecast error is significantly different from zero.