Cost-of-living crisis: ONS to explore web scraping data to track supermarket prices

Written by Sam Trendall on 4 May 2022 in News

Agency’s dedicated data-science facility looks to make a big contribution to government’s response to rising costs

Credit: Jernej Furman/CC BY 2.0 

The Office for National Statistics is to explore the use of web scraping technology to keep tabs on the cost of goods at the UK’s major supermarkets.

The statistical agency has previously experimented with using the tech, in which automated systems trawl the web and extract data from sites. From 2014 to 2017, as part of a wider exploration of the big data sources, the ONS trialled the use of web scraping to collect information on grocery pricing. 

This work largely predated the creation in spring 2017 of the agency’s dedicated Data Science Campus, the remit of which is to explore the use of potential new and innovative information sources. In its work to date, the campus has used data on the quarterly VAT returns and the movement of ships in and out of the UK ports to provide indicators of the country’s economic performance.

As it moves into its second half decade, it has ambitions to use data science to support some of government’s biggest policy objectives and its response to society’s biggest challenges.

Related content

Acting director Arthur Turrell told PublicTechnology: “I want us to make a major contribution to the cost-of-living crisis and levelling up.”

This will partly involve expanding the range of data gathered on the shipping industry, in which worldwide GPS information is used to track the movement of ships. During the pandemic, this helped to identify supply-chain issues, according to Turrell.

“We knew that the ships that were bringing goods into the country were backed up outside ports,” he said.

The ambition is now to derive information from bills of lading – the documents issued by shipping firms to importers and containing information on the cargo being carried.

Turrell says that this could enable better understanding of levels of supply of goods, and help spot patterns and possible shortages.

“We know where the ships are, but what we would like to know is what is on them… and what does it mean if a ship is, for example, 70% trousers and 30% iPhones? And how many units does this equate to?,” he adds.

At the other end of the supply chain, the campus also intends to explore the potential of web scraping to help gather information on consumer prices.

“ONS has a bit of a history of using web-scraped data,” Turrell said. “We are looking at the prices in supermarkets and whether we can use web scraping.”

Keep an eye out in the coming days for PublicTechnology’s full interview with the Data Science Campus chief, including lots of insights into how the unit supported the government’s response to the pandemic, and its ambitions for furthering the use of data in the months and years ahead.

About the author

Sam Trendall is editor of PublicTechnology. He can be reached on

Share this page




Please login to post a comment or register for a free account.

Related Articles

DfT declines review of undigitised DVLA processes for citizens with health conditions
2 June 2023

MPs found that ‘inefficient’ manual processes contributed to a pandemic backlog of driving licence applications from those with notifiable medical needs

Consultation reveals widespread opposition to proposed data-sharing laws for government login system
26 May 2023

Overwhelming majority of respondents voice disapproval but government will press on with plans to bring forward legislation

Interview: CDDO chief Lee Devlin on the ‘move from being disruptive to collaborative’
23 May 2023

In the first of a series of exclusive interviews, the head of government’s ‘Digital HQ’ talks to PublicTechnology about the Central Digital and Data Office’s work to unlock £8bn...

HMRC finds strong support for online Child Benefit claims – but ‘digital by default’ would cause problems for one in five users
17 May 2023

Department publishes findings of study conducted ahead of planned digitisation initiative

Related Sponsored Articles

Proactive defence: A new take on cyber security
16 May 2023

The traditional reactive approach to cybersecurity, which involves responding to attacks after they have occurred, is no longer sufficient. Murielle Gonzalez reports on a webinar looking at...