Background research on Reserve Bank forecasting performance: Combined articles
Introduction
This collection of papers makes up the greater part of the background research to the December 2002 (Vol. 65(4)) Reserve Bank Bulletin article, "The Reserve Bank's Forecasting Performance".
This research was not originally intended for an external audience, but rather was an exploration of the issues. The papers provide a road-map of our research as we proceeded, finally reaching the conclusions presented in the Bulletin article. They are not stand-alone documents; nor do they represent the views of the Reserve Bank of New Zealand, but rather the views, at the time, of the authors.
The papers have been adapted slightly for external publication. However, they remain technical background papers and as such may not all be entirely accessible to a non-technical reader.
Our thanks to all of the external forecasters mentioned in this research. Particular thanks must also go to David Archer, Satish Ranchhod, Willy Chetwin, James Twaddle, Alan Boaden, Victor Gaiduch and Tim Hampton, and all who had an interest in this work.
Sharon McCaw
Editor
Contents:
Statistical analysis of the Reserve Bank's forecast errors
The first stage of the project primarily involved basic statistical analysis of our forecast errors for CPI inflation and for other variables.
There are definitions of statistical terms and acronyms used throughout the papers.
Inflation forecast errors: preliminary findings (PDF 123KB)
Year-ahead quarterly inflation forecast errors (PDF 148KB)
Analysis of Bias and RMSE in our forecasts of key variables (PDF 275KB)
GDP forecast errors (PDF 2,978KB)
TWI forecast errors (PDF 135KB)
Looking for causes of our errors
Once we had a good grasp on the statistical properties of our errors, we began more substantial work on analysing potential causes of them. The following papers examine the following:
How the forecasts evolve as the forecasting rounds progress
Our general stories around the inflation outlook
Our numerical forecasts of the exchange rate, the output gap, world growth and world prices for our imports.
What has caused our biased estimates of the output gap?
Do our forecasts improve as we finalise them? (PDF 136KB)
Attributing inflation forecast bias to other variables' forecast bias using FPS (PDF 224KB)
Evaluating starting point output gap estimate errors (PDF 208KB)
Reserve Bank forecast narratives (PDF 133KB)
Comparison of our forecast errors with other forecasters
Having completed basic statistical analysis of our errors, and some investigations into what we believed was driving our errors, the next step was to compare our forecasts to those of others.
Interpreting results from comparison with other forecasters is not straightforward. For one thing, the forecasters at each institution are constantly changing, so that gathering the forecasts under the banner of a single institution may be misleading. Second, we do not know the macroeconomic models being used by each forecaster. We can therefore draw only very tentative conclusions about what may be driving any difference in the forecasts. A third issue is that forecasts are seldom prepared in exactly the same week. Later forecasts may benefit significantly from additional data revealing the current state of the economy. An additional practical issue is that the sample size was very small for several forecasters, making statistical comparison difficult.
With these provisos in mind, the following articles examine our relative forecast performance for CPI inflation, GDP growth, short-term interest rates, the TWI, and world prices for our imports.
Initially we compared our inflation forecasts with a wide range of forecasters. We found that two, the National Bank of New Zealand (NBNZ) and the New Zealand Institute of Economic Research (NZIER), had inflation forecasts that outperformed our own over comparable sample periods. We therefore examined their forecasts of other variables in greater detail. We also stepped back from numerical analysis and considered how our forecast stories compared with those of the NZIER.
Comparison of inflation and GDP forecasts (PDF 200KB)
Comparison of Reserve Bank and NZIER inflation outlook narratives (PDF 126KB)
Comparison of TWI forecast errors: Reserve Bank and other forecasters (PDF 178KB)
Comparison of interest rate forecast errors: the Reserve Bank, NZIER and the National Bank of New Zealand (PDF 149KB)
Comparison of quarterly GDPP forecasts: the Reserve Bank, NZIER and the National Bank of New Zealand (PDF 154KB)
Comparison of world price forecasts: the Reserve Bank, NZIER and the National Bank of New Zealand (PDF 107KB)
Conclusion
The papers given here make up a large part of the research that lay behind the December 2002 Reserve Bank Bulletin article "The Reserve Bank's forecasting performance". However, they are not the full story. As well as the articles included here there was some work that reached dead ends. For example, running `counter-factual' experiments using the FPS model turned out to be infeasible, and econometric analysis was plagued by technical issues and a lack of data. There was also much verbal discussion around the issues and historical context that cannot be captured in these written documents but which influenced thinking and conclusions.
Nonetheless, these documents provide much of the background research that led us to the conclusions contained within the aforementioned Bulletin article. It has been an interesting and worthwhile task analysing our forecast errors. Although we have not concluded that the results point to a need to immediately address any issues with our current forecasting approach, the findings reiterate the importance of continually reassessing our understanding of the economy.
