Using supermarket scanner data to measure inflation

Supermarket scanner data is transforming how National Statistical Institutes (NSIs) measure inflation, offering richer, more timely and detailed insights into consumer prices than ever before.

But with this opportunity comes new challenges. The sheer volume and dynamism of scanner data – with rapidly changing products, fluctuating prices and shifting quantities – can make it difficult to produce stable and representative price indices. Traditional approaches can quickly become outdated or generate unrealistic movements, such as ‘chain drift’.

This project explored how to improve the use of scanner data in practice. It reviewed and evaluated a range of index number methods, examined how they are currently applied by statistical agencies, and developed frameworks to assess their performance. Using UK supermarket scanner data, it provided empirical evidence to compare different approaches and support better measurement of consumer price inflation.

The work also developed new techniques to detect outliers in the data, which can cause price indices to over- or underestimate true inflation.

A second strand of the work empirically assessed different methods using long running UK scanner data. This data enabled us to assess the use the extent of chain drift for different indices, window lengths and splicing methods for a large number of goods and over a long time period.

We argue that with only two quite reasonable changes to the ONS scoring system, Geary-Khamis loses its top ranking to Caves-Christensen-Diewert-Inklaar (“CCDI” or GEKS-Törnqvist) and GEKS-Fisher, both using the mean splice. The selection of properties and the chosen weights can be contested. We found that the Geary-Khamis method is very sensitive to the extension method – a finding is consistent with the emerging empirical literature.

Empirical evidence also suggests that either the regular mean splice or the mean splice on the published series are to be preferred when splicing indices. Indices calculated using the mean splice produced more stable results across index number methods than alternative splicing methods. Long window lengths – of 25 months or more – are required to significantly reduce the degree of chain drift when splicing. Empirically, the CCDI index performs well empirically against the benchmarks we considered.

• This work has helped to inform the Office for National Statistics 2026 transformation of UK consumer price inflation statistics. 
• ESCoE research showed how newer approaches – based on comparing prices across multiple time periods – can produce more accurate results. It also highlighted an important practical issue: how to update these measures each month without introducing instability. 
• The research demonstrated that using a longer time span of data, and carefully linking overlapping periods together, leads to more stable and reliable results. Based on this evidence, the 
• ONS adopted a method called the GEKS-Törnqvist. 
• This is improving the consistency of inflation estimates and makes better use of large, complex datasets, supporting more reliable economic statistics for policymakers and the public. 
• Work on outlier detection also informed the methods decisions made by the ONS for the implementation of rail fares (2023) and second-hand cars (2024).

Office for National Statistics (2026) ‘How multilateral index methods help us understand grocery scanner data’ (Accessed: 24 April 2026).

Fox, K. J., P. Levell, and M. O’Connell. (2025) ‘Inflation Measurement with High Frequency Data.’ Journal of Business & Economic Statistics. DOI:
10.1080/07350015.2025.2537392

Fox, K., Levell, P. and O’Connell, M (2023) ‘Inflation measurement with high frequency data’ Institute for Fiscal Studies Working Paper 23/29, October 2023

Fox, K, P Levell and M O’Connell (2023), ‘DP18539 Inflation measurement with high frequency data‘, CEPR Discussion Paper No. 18539. CEPR Press, Paris & London. https://cepr.org/publications/dp18539

Diewert, E.W. and Fox, K.J. ‘Alternative Approaches to the Treatment of Access Charges in Price Index Construction’ ESCoE Conference on Economic Measurement, Contributed Session Q: Price indexes and rents. King’s College London, 17-19 May 2023.

Levell, P., Fox, K. and O’Connell, M. ‘Multilateral Index Number Methods for Consumer Price Statistics‘ ESCoE Conference on Economic Measurement, Contributed Session Q: Price indexes and rents. King’s College London, 17-19 May 2023.

Diewert, E.W. and Fox, K.J. ‘ Alternative Output, Input and Income Concepts for the Production Accounts’ ESCoE Conference on Economic Measurement, Contributed Session I: Outputs, inputs and productivity, King’s College London,  17-19 May 2023

Diewert, E.W. and Fox, K.J. “Alternative Approaches to the Treatment of Access Charges in Price Index Construction” ESCoE Discussion Paper, ESCoE DP 2023-03

Diewert, E.W. and Fox, K.J.  “Index numbers research for bundled products” Economic Statistics Centre of Excellence: The Next Five Years, poster exhibition, 12 December 2022, One Birdcage Walk, London

Fox, K., Levell, P. and O’Connell, M. “New challenges for Consumer Price Indices: a view from the UK” UNSW BusinessThink, Opinion 25 July 2022

Fox, K., Levell, P. and O’Connell, M. “New data and new challenges for Consumer Price Indices” ESCoE Blog, 11 Apr 2022

Fox, K., Levell, P. and O’Connell, M. (2022) “Multilateral index number methods for Consumer Price Statistics“, ESCoE Discussion Paper Series, ESCoE DP 2022-08

Fox, K. “New index number methods in Consumer Price Statistics” ESCoE Research Seminar, 10 Feb 2022.