The value of forecasting in an uncertain world
A speech delivered by Dr John McDermott to the New Zealand Manufacturers and Exporters Association (NZMEA) in Christchurch
"We are all forecasters." Tetlock and Gardner, Superforecasting.1
It has often been said that there is no escaping forecasting. Every decision we make in life involves forming a view about what the future might hold. Likewise, there is no escape for the Bank. We have a Policy Targets Agreement that requires us to keep future inflation between 1 and 3 percent and therefore we need to form a view of where we currently are, how future developments might influence the path of output and inflation, and thus how we should respond.2
We use our forecasting to establish a clear and well-founded plan for what monetary policy settings are necessary to achieve our inflation target, to communicate this plan to the public and to support the effectiveness and transmission of monetary policy. We present our forecasts in the quarterly Monetary Policy Statement. A great deal of effort goes into producing these forecasts.
More often than not, the world does not turn out as we forecast. However, forecasting is still a valuable and necessary part of the monetary policy process. Producing forecasts and allowing them to be publically subject to challenge enables us to build a solid foundation for policy decisions, learn from developments, and improve our policy outcomes. Forecasting is not supposed to be prophecy; rather, it is about being precise about our thinking.
Uncertainty and the challenges for forecasters
A major challenge for forecasters is ‘radical uncertainty’, that is, we fundamentally don’t know what the future holds. Examples of this uncertainty are ‘black swan’ events that fall outside any existing experience. The term is derived from the northern hemisphere belief that all swans were white, based on the empirical evidence that every swan ever observed was white.3 For the first European migrants to Australia, discovering black swans was a completely unexpected event, lying outside of all evidence and experience up until that point.
For forecasters, such unpredictable events are a particular challenge. We have a wide range of information about the past and current state of the economy, and are setting policy to influence the future direction of the economy. But the economy is populated with thousands of households and businesses responding to their own particular circumstances and opportunities, so it is never certain how the economy will evolve over time. Even without black swan events, the chain reactions of economic activity are so complex and so changeable that the range of possible – let alone probable – outcomes is vast.
Tetlock and Gardner argue that the best forecasters understand that the world is uncertain and therefore react to it flexibly.4 Good forecasters are careful, curious, and comfortable with numbers, and, crucially, when the facts change they change their minds. And that is how we strive to be. Conversely, the worst forecasters remain dogmatically attached to their forecasts, unchanging in the face of the evidence.
Forecasting in uncertainty: dynamic programming and conditional forecasts
So how do we make policy for an uncertain future? The approach that the Bank takes is to develop forecasts that are highly conditional on the information currently available. That is, our forecasts are subject to revision when additional information comes to light. These forecasts are then used in what Alan Blinder would describe as a ‘dynamic programming’ plan. 5 This means thinking about what you want to achieve in the future and working backwards to form a plan today that will deliver that goal. In our case the goal is inflation near 2 percent in two or three years’ time, and the plan is given by the forecast of the Official Cash Rate (OCR). If events broadly unfold as envisaged then we carry out that plan, if not, then we adjust the plan.
A feature of our inflation forecasts are that they always end up at the mid-point of the target band. When planning our actions we necessarily plan to succeed in achieving this target. If we thought we would not achieve this we would modify our OCR forecasts.
As such, the forecasts show our judgement of the highest probability path to achieve the goal, but this path is not fixed. As new information becomes available, we assess it and where appropriate incorporate it into our understanding and our plans. Consequently, our policy settings are, and should be, responsive to any developments that might occur, and should move rapidly when required. For example, the sharp fall in inflation expectations in March 2016 (amongst other developments) led to us substantially revise our 90-day interest rate projection6 and move away from a neutral stance to an easing bias (figure 1).7
Figure 1: Inflation expectations and selected 90 day interest rate projectionsSource: RBNZ estimates.
The value of producing quantitative forecasts: internal discipline and the learning process
So why do we produce specific forecasts given that our forecasts are conditional and subject to radical uncertainty? Being numerically precise about our view of the future allows us to test ideas, which in turn accelerates our ability to learn and understand what is going on. Over time, this helps to ensure that our forecasts are as robust as possible and form a solid basis for monetary policy decisions, minimising potentially costly forecast errors and supporting good policy outcomes.
Testing is an integral part of robust analysis. As Nate Silver puts it, "The more eagerly we commit to scrutinising and testing our theories, the more readily we accept that our knowledge of the world is uncertain, the more willingly we acknowledge that perfect prediction is impossible, the less we will live in fear of our failures, and the more freedom we will have to let our minds flow freely." 8
When an outturn is different from that forecast, the Bank seeks to discover why, what it means for the outlook for monetary policy, and what we could do to make a better forecast next time.9 We use our depth of experience to examine the data, distinguish the signal from the noise, and identify where to adapt our approach or where to maintain our existing views.
For example, residential investment at the end of 2016 was considerably weaker than the Bank had forecast. In seeking to understand this forecast error, the Bank considered the composition of residential investment. We found that a sharp fall in transfer costs (based on decreasing house sales) was the cause of weak residential investment, rather than weakness in actual construction activity. Accounting for this information should support better forecasts for this part of the economy in the future (figure 2).
Figure 2: Residential building activity and house salesSource: RBNZ estimates.
Producing forecasts: avoiding biases and supporting robust outcomes
Forecasting can also help reduce human biases that are often present in decision making. Our brains are designed to react to surprises by making sense of them. In seeking to do so, we often rewrite our perceptions from prior to these events.10 Even black swans may feel retrospectively predictable. This is known as ‘hindsight bias’ or the ‘knew it all along effect’.11
As forecasters, we need to be particularly aware of this bias. To learn from our errors, we need to recognise that they are errors. By recording our quantitative forecasts and accompanying reasoning, we prevent ourselves from rewriting our original position and ensure we take on the full insights from any forecast errors we make. Over time the Bank has built an extensive institutional knowledge base of New Zealand economic data and experience of monetary policy formulation. Each time we forecast, we examine the latest data, identify where new information should lead us to change views, and incorporate lessons learned to ensure that our forecasts keep evolving and are as accurate as possible.12 To use Kahneman’s terminology, our forecasts force us to think more slowly.
Forecasting is not the only essential part of monetary policy formulation. The Bank has a well-defined process for monetary policy decisions, based on a Monetary Policy Committee (MPC) that provides advice to the Governing Committee. 13 The MPC includes two external members to add to the Committee’s diversity of backgrounds and views. These external members provide a ‘reality check’ on the Bank’s thinking, and subject us to valuable independent scrutiny.14
The MPC structure is supported by a set of working practices that ensure that colleagues closest to the data and forecasting processes present their work to the MPC, and that subsequent discussions are broad-ranging and unconstrained in order to avoid groupthink or other biases.15 This means that the analytics of our Economics and Financial Markets Departments are blended with the experience of senior staff and Committee members, achieving better results through incorporating judgement in the forecasting process.
How do we create forecasts at the Reserve Bank?
The Bank monitors a wide range of data to gauge the current state of the economy, distilling the underlying trends and signals from the noise and assessing what the emerging picture means for the stance of monetary policy. In addition to formal model-based analysis, monetary policy formulation is supported by the feedback we receive from regular meetings with private businesses, industry associations and government agencies. We also undertake longer-term research projects to improve our understanding of specific features of the economy. 16
At the heart of our forecasting is our structural model, the New Zealand Structural Inflation Model (NZSIM). This is a theoretically rich framework that describes key behavioural relationships in the New Zealand economy and how they influence inflation. NZSIM generates the core forecasts for informing monetary policy decisions, and helps us identify the fundamental economic drivers and empirical relationships to explain the observed economic data.17
The information collated over the forecast round is fed into NZSIM, and the theoretical structure of the model ensures that our forecasts are consistent. For example, the national income accounting discipline is preserved, we can distinguish between domestic and imported inflation which sum up to CPI inflation, and we are able to trace the forces driving economic growth.
During the monetary policy decision-making process, the NZSIM forecasts are cross-checked with other modelling, and where appropriate modified to take into account additional information not directly represented in the model. That is, our forecasts are ‘model-assisted’ rather than ‘model-produced’.18 NZSIM also can consider alternative scenarios, meaning that we can test competing ideas and assess how monetary policy should respond if circumstances develop differently.
NZSIM necessarily abstracts from reality, and (as with any modelling) requires some difficult tradeoffs to be made, balancing the amount of detail that can be incorporated without sacrificing the ease of interpretation and resulting policy recommendations. The combination of tractable model outputs and the expert judgement of Bank colleagues and Committee members forms the basis for building nuanced and detailed policy recommendations.
Over the last few decades, increases in computing power have enabled the Bank to vastly expand its modelling and research capability, including making NZSIM possible. However, one significant limiting factor to forecast improvement is data: macroeconomic statistics have not fundamentally changed over the 30 years we have been using inflation targeting.19 For instance, New Zealand is one of only two advanced countries that use a quarterly consumer price index rather than monthly. This is likely to be a fruitful avenue for future improvement that would greatly improve the Bank’s forecasting ability.
The benefits of publishing our forecasts: improving the effectiveness of monetary policy
The RBNZ Act20 requires the Bank to publically set out its monetary policy stance and underlying reasoning, including a review of previous decisions. This information is published in our quarterly Monetary Policy Statement, along with our economic forecasts and accompanying explanation. The Bank was the first central bank to publish its planned interest rate path, starting in 1997, and we continue to rank as one of the most transparent central banks.21 Publishing these macroeconomic forecasts and our planned OCR path provides a clear ‘line in the sand’ as to the Bank’s thinking. It also helps synthesise our views in a more approachable format, enabling commentators, market participants, and others to understand the Bank’s monetary policy stance.
Communication is a key part of the monetary policy transmission mechanism as it helps people to understand the Bank’s thinking and expected policy direction.22 By building people’s understanding of how the Bank is likely to react to news, the predictability of monetary policy decisions is enhanced and policy uncertainty is reduced. We are a long way from the days when central bankers cultivated their ability to “mumble with great incoherence”.23
To inform the market about the uncertainty inherent in projections, some central banks publish fan charts showing the range of possible outcomes around the highest probability scenario. While this approach shows the extent of uncertainty, it does not help people understand how the central bank’s forecasts may evolve if some of the contingencies change. Consequently, in recent Monetary Policy Statements we have published a range of potential scenarios to help people understand how the forecasts would change should the economy develop differently. We believe that this approach is potentially more helpful.
Figure 3 illustrates how market participants understand the way the Bank is likely to incorporate economic and financial information into its projections. The market revisions (red line) are generally good at anticipating changes to our 1-year ahead 90-day interest rate projection (blue line), which suggests that they understand the conditional nature of the forecasts and the Bank’s likely response (or ‘reaction function’) to new information.
Figure 3: MPS and market revisions
Source: RBNZ estimates.
Case Study: the increase in the OCR in 2014
The increase in the OCR in the first half of 2014 provides a good case study of our conditional approach to forecasting in a ‘radically uncertain’ world. At the start of 2014 the global economy appeared to be recovering from the 2008/9 global financial crises, with the IMF asking whether the “tide was rising”. 24 The New Zealand economy was growing strongly: the terms of trade had reached their highest level in 40 years; construction activity was robust; immigration was boosting housing and consumer demand; and monetary policy was providing substantial stimulus to the economy. Under these circumstances, we judged that there was a high probability that inflationary pressures were rising, contingent on the global and domestic economy developing as forecast and our knowledge of the structure of economic relationships. This view about rising inflation pressures was shared by the private sector forecasters and the IMF. Consequently, the OCR, which was set at an accommodative level of 2.5 percent, was raised by a total of 1.0 percentage points at successive OCR reviews in the first half of 2014.
However, by September 2014 it had become apparent that the forecast contingencies were not developing as expected. Firstly, uncertainty around the central scenario had risen: the global economy was not strengthening as anticipated and New Zealand terms of trade had fallen by about 7 percent since the start of 2014. Secondly, uncertainty about the structure of the domestic economic relationships was becoming apparent as CPI inflation remained moderate despite the inflationary pressures. Under these circumstances the Bank judged it appropriate to put a hold on interest rates, await the flow of new economic data, and investigate the evolution of domestic economic relationships.
In the event, this decision proved correct. Global economic growth continued to be sluggish through 2015 and 2016. The Bank’s analysis of the domestic economic relationships concluded that resource pressures were not as strong as previously estimated, in part due to the record level of migration that the New Zealand economy was experiencing.25 Low inflation had also changed the price-setting behaviour of New Zealand businesses, who were placing a greater weight on recent low inflation (rather than expectations of future inflation) in their wage-setting and pricing decisions.26 Incorporating these lessons into our forecasting framework meant that by mid-2015 the most probable scenario had changed to a continuation of low inflation (figure 4). The Bank commenced a new cycle of reducing the OCR, to an eventual low of 1.75 percent in November 2016. Today, in May 2017, the most likely scenario is for the OCR to remain stable for some time, although uncertainty remains high.
Figure 4: selected 90-day forward rates
Source: RBNZ estimates.
Forecasting is a valuable part of the monetary policy process, helping the Bank to plan for the future, make well-founded policy decisions, communicate its understanding and intentions, and accommodate new information as it appears. It requires the Bank to be rigorous, unbiased, and open to new ideas in formulating and implementing monetary policy.
Because we operate in a world of radical uncertainty, we do not know what might happen in the future; we are not fortune tellers. However, because we want our forecasts to be relevant and useful, we present them numerically, an approach that comes with risks. As Tetlock and Gardner put it, “use the number and you risk being unfairly blamed. Stick with phrases as fuzzy as a puff of smoke and you are safe”.27 We do not want to be safe in this context as we welcome scrutiny, both internally and externally, as it builds understanding and fosters accountability.
Finally, good forecasting happens when the numbers are supported by evidence and reasoning, and subject to public scrutiny. When things – inevitably – do not turn out as projected we seek to learn from those episodes and continually work to improve our forecasts and policy-making.
1 Philip Tetlock and Dab Gardner (2015), Superforecasting: the art and science of prediction, Cornerstone Digital.
2 Our specific mandate is established by the RBNZ Act (1989) and the Policy Targets Agreement (PTA) which state that the Bank should use monetary policy to maintain price stability in the New Zealand economy. We have considerable flexibility around how we do this, reflecting the tradeoffs necessary to protect the overall economy (for example the impact on the volatility of the exchange rate or output), and acknowledging the range of relevant factors outside the Bank’s control. It also recognises that judgements are a necessary part of the task, for example, providing the Bank with discretion around how quickly it seeks to return inflation to target in the future.
3 Nassem Taleb (2007), The Black Swan: The Impact of the Highly Improbable, Random House. Taleb defines these black swan events as outside of previous experience, with an extremely significant impact, and that we have a tendency to try to explain it after the fact. Examples relevant to monetary policy in New Zealand include the 9/11 attacks or the Canterbury earthquakes.
4 Tetlock and Gardner, op.cit. The authors demonstrate that the best forecasters blend technical expertise and mastery of the data with judgement and humility. In addition, teams of forecasters can produce better results when the team is open to sharing and constructive debate.
5 Alan Blinder (1998), Central Banking in Theory and Practice, MIT.
6 Since November 2016 this has been replaced by the OCR projection.
7 For further detail see Rebecca Williams, ‘Business cycle review: 2008 to present day’ RBNZ Bulletin 80(2) March 2017.
8 Nate Silver (2012), The Signal and the Noise: Why So Many Predictions Fail—but Some Don’t, Penguin Books.
9 For example see: Kirdan Lees, ‘Assessing Forecast Performance’, RBNZ Bulletin 79 (10) June 2016; C J McDermott, ‘Understanding low inflation in New Zealand’, a speech delivered to the Bay of Plenty Employers and Manufacturers Association (EMA) in Rotorua, 11 October 2016; and Geordie Reid, ‘Evaluating the Reserve Bank’s forecasting performance’, RBNZ Bulletin 79(13) August 2016.
10 Daniel Kahneman (2011), Thinking, Fast and Slow, Farrar, Straus and Giroux.
12 In essence, this is a Bayesian approach combining priors with new information using judgement. For a readable introduction see James V. Stone (2015), Bayes’ Rule with Matlab: A Tutorial to Bayesian Analysis, Sebtel Press.
13 The Governor retains statutory responsibility for OCR decisions, but the Bank has in practice delegated the decision on the appropriate policy settings to the Governing Committee.
14 The MPC has included external members since 2001. External members are limited to serving a 1 to 3 year term to ensure that they remain truly external. The two current and previous members are: Tony Caughey; Conor English; Richard Townshend; Luke Moriarty; Pip Dunphy; Earl Rattray; Liz Coutts; Bronwyn Monopoli; Terry McFadgen; Malcolm Bailey; Kerrin M Vautier; and Brent Layton.
15 For more detail see C J McDermott, ‘How the Bank formulates and assesses its monetary policy decisions’, a speech delivered to the Manawatu Chamber of Commerce in Palmerston North, 13 July 2016; and Adam Richardson, ‘Behind the scenes of an OCR decision in New Zealand’, RBNZ Bulletin 79(11) July 2016. For example, by convention the Governor speaks last in MPC meetings so as not to unduly influence the direction of the debate before a diverse range of views are heard from staff.
16 For examples of recent papers see the RBNZ’s Analytical Note series, available at http://www.rbnz.govt.nz/research-and-publications/analytical-notes.
17 For more detail on the NZSIM model, see Neroli Austin and Geordie Reid, NZSIM: A model of the New Zealand economy for forecasting and policy analysis, RBNZ Bulletin 80(1) January 2017.
18 An example of how we modified our forecasting framework to capture insights from analysis was when we moved to an adaptive (backward looking) inflation expectations measure from a survey measure of inflation expectations. This adjustment was to reflect the findings of Karagedikli and McDermott (2016), who found that more backward looking measures of inflation expectations were able to explain the low level of inflation since 2011 better than surveyed measures. See Özer Karagedikli and C J McDermott, ‘Inflation expectations and low inflation in New Zealand’ RBNZ Discussion Paper 2016/09.
19 New Zealand still lacks a complete set of national accounts, for example we do not have quarterly income GDP or flow of funds data.
20 Section 15 of the RBNZ Act 1989.
21 Nergiz Dincer and Barry Eichengreen Transparency Index 2014, available at http://eml.berkeley.edu/~eichengr/Dincer-Eichengreen_figures&tables_2014_9-4-15.pdf.
22 C J McDermott, ‘Policy uncertainty from a central bank perspective’, Australian Economic Review, March 2017 discusses how policy uncertainty, or ‘instrinsic uncertainty’, is reduced by having a clear policy framework and structure for decision-making.
23 Alan Greenspan in his often-quoted testimony to Congress in 1987. Another fun quote of Greenspan is: “I know you think you understand what you thought I said but I'm not sure you realize that what you heard is not what I meant”.
24 IMF, World Economic Outlook update, ‘Is the tide rising?’, January 2014.
25 See December 2017 Monetary Policy Statement, Box C.
26 Özer Karagedikli and C J McDermott, “Inflation expectations and low inflation in New Zealand”, RBNZ Discussion paper 2016/09.
27 Tetlock and Garnder, op.sit.