Modelling and Communicating Data Uncertainty

Bank

Modelling and Communicating Data Uncertainty

Jump to

Summary

Due to sampling and non-sampling errors, economic data, like GDP, are uncertain. Despite this, economic data are frequently published without any direct quantitative indication of their uncertainty. This project considered the measurement of data uncertainty and its communication. Experimental surveys were used to provide guidance to statistical agencies on how the public and ‘experts’ interpret data, and how uncertainty information might best be communicated when publishing national account estimates. Our findings reveal that whether uncertainty information is communicated to the public, and how it is communicated, matters. Of particular significance for statistical agencies, we show that communicating uncertainty information alongside the GDP point estimate affects, and improves, public understanding of why data revisions happen.

Overview

This project had two aims: to measure data uncertainty, and then to analyse the reactions of a range of decision makers to communication of this uncertainty, with uncertainty modelled, communicated and visualised in different ways. A key question was whether or not communicating data uncertainty might erode trust in official economic statistics.

Economic data uncertainty is rarely communicated quantitatively. An exception is the ‘fan chart’ for historical GDP growth published at the Bank of England. We used these fan charts, and estimated econometric models of GDP data revisions, to measure, track and evaluate the evolving nature of GDP data uncertainty. We proposed an approach to extract a quantitative measure of unforecastable data uncertainty.

With the aim of assessing empirically if and how the public and more experienced users interpret and understand GDP data uncertainty, we conducted a randomised controlled online experiment with more than 3,000 nationally representative members of the public and a targeted survey of more than 100 experts. To maximise realism, both surveys asked questions about ONS’s latest GDP estimates and headline press release. The specially designed surveys were used to assess perceptions of the uncertainty in single-valued GDP numbers, the public’s interpretation and understanding of uncertainty information communicated in different formats, and how communicating uncertainty affects trust in the data and the producers of the data. By randomly allocating members of the public into one of six groups – a control group presented with no uncertainty information, and five treatment groups presented with different uncertainty information – we identified the causal effects of different ways of communicating uncertainty information. The results can be used to help inform if and how data uncertainty information should best be communicated.

In on-going research we are running an additional wave of our online public survey during the shutdown-induced economic recession due to the coronavirus. This will enable us to identify whether perceptions of data uncertainty and the effectiveness of different communication treatments vary according to the state of the economy and the public’s perceptions of this.

In complementary work we provided an interdisciplinary review of structures and summarised current practice and research in uncertainty communication. This combined statistical and psychological perspectives.

Findings

We provided quantitative estimates of unforecastable data uncertainty for UK GDP growth and found strong evidence of changes over time to the GDP growth data revision process. We also found that data uncertainty rises at the onset of recessions and is positively correlated with popular measures of macroeconomic uncertainty. Methodologically, we defined unforecastable uncertainty as a lack of knowledge about the future or the past and showed how our uncertainty measures can be used as the basis for the construction of tests for calibrating probabilistic forecasts. Our experimental results indicated that the majority of the public, like experts, do not take GDP point estimates at ‘face value’. The majority, whether asked about this qualitatively or quantitatively, expect data uncertainty and are not surprised when data are revised.

Importantly, we found that whether uncertainty information is communicated to the public, and how it is communicated, matters. Communicating uncertainty information alongside the GDP point estimate affects, and improves, public understanding of why data revisions happen. It encourages more of the public, rightly, to view the point estimate as just that: a point within a range of possible outcomes. Our randomised experiments also revealed that these quantitative communications of GDP data uncertainty need not reduce trust in statistical agencies. We recommend that uncertainty information is communicated quantitatively using intervals, density strips and bell curves. These are, in general, preferable to textual descriptions and certainly better than no communication at all beyond referring to GDP as an ‘estimate’. Our experiments suggest this will reduce the chance of the public misinterpreting uncertainty information given to them.

Impact

By showing how communicating data uncertainties can improve the public’s understanding of data without eroding trust, our research might encourage national statistical offices to do more to measure and communicate data uncertainties. It is also hoped that the work of this project will encourage more empirical research into uncertainty measurement and communication, such as the Comunikos project at Eurostat. Our research has generated widespread interest from central banks.

Project Papers and Presentations

Ana Galvão and James Mitchell (2021) ‘Communicating Data Uncertainty: Experimental Evidence for U.K. GDP’ ESCoE Conference on Economic Measurement 2021 Poster Exhibition, 11-13 May 2021. Poster Presentation

Galvão, A.B and Mitchell, J. (2021) ‘Communicating Data Uncertainty: Experimental Evidence for U.K. GDPRES Annual Conference 2021, Special Session ‘Improving the communication of economics and economic statistics’ 12-14th April 2021 (Presentation starts at 19:38)

Ana Galvão, James Mitchell and Johnny Runge ‘Communicating Data Uncertainty: Experimental Evidence for U.K. GDP’ ESCoE Conference on Economic Measurement 2020, Contributed Session G: GDP Measurement and Uncertainty, 16-18 Sep 2020 (Presentation starts at 02:00)

Galvão, A.B and Mitchell, J. (2020) “Real-Time Perceptions of Historical GDP Data Uncertainty” Warwick Business School Finance Group, Macroeconomic Policy and Forecasting, Working Paper Series No. 35 July 2020

Dey-Chowdhury, S., Galvão, A.B., Mahajan, S., Scruton, J. and Walton, A. (2020) “Communicating gross domestic product” 27th April 2020, Office for National Statistics

Galvão, A.B and Mitchell, J. and Runge, J. (2019) ‘Communicating Data Uncertainty: Experimental Evidence for U.K. GDP‘ ESCoE Discussion Paper Series ESCoE DP 2019-20, 5 Dec 2019

Ana Galvão ‘Communicating Data Uncertainty: Experimental Evidence for the UK GDP‘ ESCoE Research Seminar, 15 Oct 2019

Galvão, A.B and Mitchell, J. (2019) ‘Measuring Data Uncertainty: An Application using the Bank of England’s “Fan Charts” for Historical GDP GrowthESCoE Discussion Paper Series ESCoE DP 2019-08, 24 May 2019

van der Bles, A.M., Van der Linden, S., Freeman, A.L.J., Mitchell, J., Galvao, A.B., Zaval, L. and Spiegelhalter, D. (2019) “Communicating uncertainty about facts, numbers and science“, Royal Society Open Science, Vol 6, Issue 5, The Royal Society Publishing. https://doi.org/10.1098/rsos.181870

Ana Galvão ‘Data Uncertainty and Business Cycles: an Evaluation of the Bank of England’s Probabilistic Backcasts‘ ESCoE Research Seminar, 13 Feb 2018.

People

Project partners

Related publications

Events