By Sylaja Srinivasan and Ryland Thomas, Bank of England
At the 2019 ESCoE conference we launched Historical Data UK, an online repository of historical datasets and statistical publications with its own dedicated section on the ESCoE website. The project is a joint initiative between ESCoE, the Office for National Statistics (ONS) and the Bank of England with an aim to make historical statistics more available to researchers and the general public. The pandemic of the past year has reinforced the need for digital access to historical statistics, given the ability to visit libraries and archives has been limited and many of us are working from home. So we hope the repository will act as a valued resource for a wide variety of statistics users from near and far. Over the past two years we have been working hard with colleagues at the ONS, the Bank, the National Institute of Economic and Social Research (NIESR) and academia to expand the site and maintain the momentum of the initial project. The site now hosts thousands of publications in a readily accessible location and is now attracting hundreds of visits each month. In this blog we set out our recent work and some of our ambitions for the future of the repository.
More coverage
The site originally contained 9 sections covering a range of macroeconomic and financial data. It has now expanded to cover 15 sections additionally covering historical data on climate change, regional data and international data. We also now have dedicated sections on capital and productivity and financial accounts and balances sheets given the recent historical development work by ONS and the Bank in both of these areas.
More publications
The site now contains around 4,000 digitised statistical publications, including complete sets of the core Central Statistics Office (CSO) and ONS publications of yesteryear, such as the Blue Books and Pink Books which began in the 1940s, the Annual Abstract and Monthly Digest of Statistics from 1946 and Financial Statistics from 1962. The publications also cover methodology as well as data including National Accounts Sources and Methods from 1954, Economic Trends articles from 1957, Statistical News and Studies in Official Statistics. In addition to CSO and ONS publications, we have also digitised a number of other statistical publications of government departments such as the Statistical Abstract of the United Kingdom (the forerunner to the Annual Abstract) and the historical abstracts of labour statistics produced by the Board of Trade, which begin in the C19th. A list of publications is available here with links to the relevant area of the repository.
The publications are not just sources of historical data and methodology but also contain reminders of the importance of good practice in the collection of statistics which requires a careful balancing of the costs and benefits. Take this quote from the Minister of Agriculture, Fisheries and Food Frederick Peart in his foreword to A Century of Agricultural Statistics published in 1967:
I take this opportunity to record the gratitude which I and my predecessors owe to the pioneers who established the agricultural census in the face of much indifference and some stubborn opposition; to past and present officials who have collected and analysed the census returns, and used them in developing British agricultural policy; but especially to the farmers who have provided them. For much of its history the census was voluntary, and even today it could not continue without the co-operation of the farming community. All of us, Ministers and farmers, industrialists, traders and consumers, have to take decisions. The choice is between decisions taken in ignorance and those based on knowledge.
Links to other digitisation efforts
Where publications already exist digitally in the public domain we have provided links, most notably to the HathiTrust digital library which provides access to digitised content from a variety of sources, including Google, the Internet Archive, Microsoft, and member-institution initiatives. We have also co-ordinated our digitisation programme with other UK libraries that are carrying out similar programmes to avoid duplication of effort. For example, our colleagues at the LSE digital library have recently digitised the complete set of Ministry of Labour and Employment Gazettes as well as a range of statistical information on the censuses of population and production. We provide links to those documents in the relevant area of the repository.
Academic sources
We have also been busy securing permissions to reproduce important academic publications covering historical UK statistics. This has involved a concerted effort, tracking down copyright holders from publications that were published as far back as the 1920s. A big thank you here goes to the Marshall Library and Economics Faculty at Cambridge and Martin Weale who have kindly helped us to secure permissions for many key statistical publications of the Department of Applied Economics at Cambridge including the “Green Books”, edited by Professor Richard Stone (official title “A Programme for Growth”), which pioneered analysis of economic growth and industrial structure in the UK and the analysis of Aggregated financial accounts for individual British industries 1948-1982 by Andrew Goudie and Geoff Meeks.
Excel transcriptions
We have also begun the task of converting pdf files of the various statistical publications into spreadsheets. Most notably we have now transcribed all the tables of Charles Feinstein’s National Income, Expenditure and Output of the United Kingdom 1855-1965 and his subsequent tables on capital formation and capital stocks between 1760 and 1920. On the financial market side, we have also transcribed the relevant sections of the Gayer, Rostow and Schwartz dataset covering fluctuations in the British economy over the period 1790-1850 and banking system balance sheets from David Sheppard’s The Growth and Role of UK Financial Institutions 1880-1966. Regional employment data by industry covering the period 1841-1971 derived from the census by Professor Clive Lee has also been transcribed. However, we still have many more Excel transcriptions on the way so watch this space!
Other new additions
- The National Accounts Section contains a timeline of UK national accounts developments and Blue Book revision windows put together by Steve Drew at ONS
- The Fiscal Section now contains Budget and Financial Statements going back to 1920 as well as Pember and Boyle’s volumes on British Government Securities
- Bill Allen’s datasets on the Bank’s gilt-edged, foreign exchange and gold operations during the 1950s are available in the Money, Banking and Credit Section
- Work by Robert Calvert Jump at the Institute of Political Economy, University of Greenwich who has constructed provisional estimates of workforce job counts for British local authority districts between the years 1981 and 2018 Regional Data Section
- The International Section now contains the two editions of the International Abstract of Economic Statistics which covered monthly international data for 1919-1930 and 1931-1936 and was co-ordinated by the International Statistical Institute at The Hague. We are grateful to Stephen Penneck and the International Statistical Institute for their permission to reproduce these.
A new version of the Millennium of Macroeconomic Data Spreadsheet
The Bank’s Millennium of Macroeconomic Data Spreadsheet, which is available in the Headline Macro Data Section is largely derived from the digital sources available in the repository. For this reason it has been decided to link the two projects together more formally and the organisation of the spreadsheet will match the structure of the repository. In this way the repository will act as the “Sources and Methods” reference for the Millennium dataset which up until now has not provided additional metadata and information over and above the primary data sources. Version 4 of the Millennium spreadsheet is due shortly and will contain updated data up to and including 2020. It has also been expanded to cover a wide range of additional series thanks in part due to the digitisation of additional statistical material in the data repository.
What next for the repository?
Since the launch of the new ESCoE website in September 2020, we have had over 1,000 visits and 800 document downloads and we have received good feedback from researchers and practitioners alike. We are keen to develop and promote the site further and would like to advertise it more widely to those it would interest most. Feedback is important here and we would particularly welcome any feedback on additional material you think would be useful for us to include. On our ‘to do’ list are the following:
- The Actuaries Share Price Indices from 1928 to 1962 (the forerunner to the modern FTSE share price indices) in conjunction with the Institute and Faculty of Actuaries
- Additional Excel transcriptions of key datasets
- Filling in publication gaps, especially missing Economic Trends Articles
- A Diary of Economic Events
- A new Business Cycles section to cover historical business cycle metrics
Though we are open to ideas for other additions.
We would also value feedback on the functionality of the site. Up until now the resources of the project have been devoted to transcribing and uploading the basic statistical material. The site itself has a simple structure which users can browse through but there is no systematic search facility that would, for example, allow searching through multiple pdf files from a given periodical or publication set.
If you have any thoughts on these ideas or any other issues please feel free to get in touch.
ESCoE blogs are published to further debate. Any views expressed are solely those of the author(s) and so cannot be taken to represent those of the ESCoE, its partner institutions or the Office for National Statistics
