Much of the digital information that government has amassed over the past two decades is poorly organised and “almost impossible” to search, a Cabinet Office report has said.

Although Whitehall was good at managing paper records, the government said it had not kept up with digital demands – Photo credit: Pixabay

The report, Better Information for Better Government, said the current state of record-keeping creates “real and immediate risks” for operations, information compliance and the government’s reputation, for instance when responding to inquiries and FOI requests.

It said that there was an “immediate need” to address the situation, and recommended that departments assess the level of risk they face, while teams at the Government Digital Service and The National Archives should focus on preparing government for future digital record-keeping.

Related content

Civil service chief John Manzoni urges change as data revolution ‘comes for government’
GOV.UK to take more responsibility for local links as Local Directgov is retired
Unlocking the power of big data

Government information is held in departmental computer systems, databases or paper files, but some of this must be retained long-term in order to maintain the public record, in which case it will usually be transferred to The National Archives.

The government said it had been “generally good” at managing information held on paper – when it resided in only one place and was either destroyed or moved to the archives – but that the shift to digital information “undermined the rigour of information management across much of government”.

This is partly because the move to digital happened at different rates in different departments, meaning that information management was not wholly re-assessed.

Many attempted to use paper practices for digital records, for instance printing out digital records, but this became more difficult as processing power increased and the amount of information generated grew.

Other problems included difficulties in identifying valuable material because emails would also include out-of-office messages and junk mail, and the fact many civil servants began to save information in their own different ways.

“Only now, a decade later, is government beginning to understand the implications of this systemic information management failure,” the report stated.

“While little information has been lost altogether, much of what has accumulated over the past fifteen to twenty years is poorly organised, scattered across different systems and almost impossible to search effectively,” the report said.

This poses problems for government when responding to inquiries – with the report noting that sometimes departments have found important information after an inquiry had finished – and in compliance with data security measures.

It also means that civil servants risk re-inventing the wheel because they can’t access information on previous solutions – the report estimated that this wasted effort could cost the government £500m a year.

Support and culture change

The report – which comes as civil service chief John Manzoni has said Whitehall needs to boost its data expertise to cope with the challenges posed by big data – was published in response to a November 2015 review of government digital records.

That review, carried out by Alex Allan, former chairman of the Joint Intelligence Committee, said that although policies and guidance on capturing and managing information were “sound; the problems come in the implementation”.

He recommended greater co-operation between GDS, The National Archives and departments, and a change in culture across government.

The government report aims to address these concerns, setting out plans to create bespoke support plans for each departments, with the focus being to engage with staff to encourage, rather than mandate, change.

“Our aim should be simply that departments get better at managing their legacy collections and improve current information management by drawing on all the levers at their disposal,” the report said.

Its main recommendations are for departments’ accounting officers and senior leaders to evaluate the risk they face from legacy digital information and “take specific steps” to improve information management.

The most appropriate way for departments to deal with legacy digital collections, it said, would be to embrace data analytics tools to assess which material to get rid of, while making it simpler for staff to save things in the right places in the first place.

Meanwhile, in addition to supporting individual departments’ efforts, the Cabinet Office, The National Archives and GDS will carry out cross-government initiatives to improve information management in the future and increase capability within the civil service.

Over the next six to twelve months, a team from the three organisations will aim to drive change across the whole of government, while retaining a “supportive, collaborative engagement model”.

Commenting on the report, Allan said: “The government has taken the issues I identified very seriously and come forward with a strong set of recommendations which, if put into place by individual departments, should greatly improve digital records management across the civil service.”