A pilot exercise involving 20,000 officials across 12 departments found that, while benefits were fairly consistent across civil service grades, results varied much more widely by profession and use case
A major Whitehall trial of Microsoft’s Copilot has found that the artificial intelligence technology enabled civil servants to save an average of about half an hour a day, with “results [that] were consistent across grades and professions”.
During the last three months of last year, 20,000 employees across 12 departments took part in an exercise intended to help ministers ascertain the potential impact of AI on officials’ working life and satisfaction, as well as on the quality of task outcomes. Microsoft Copilot – which can be used via the M365 suite of programs, including Teams and Outlook – is described by the software vendor as “an AI-powered tool that helps with your work tasks”. The cross-departmental pilot was led by the Government Digital Service, which is part of the Department for Science, Innovation and Technology.
According to a newly published report from GDS and DSIT summarising the findings of the trials, the technology had an adoption rate – equating to participants using the tool to assist with at least one M365 program – of about 80% throughout the experiment.
The AI system was most widely used in Teams, for which adoption peaked at 71%. Usage via Word, Outlook and Copilot Chat topped out, in all cases, near or just above the 50% mark.
The tool was used far less in Powerpoint and in Excel, where adoption hit respective highs of 23% and 24%.
A total 39% of participants reported using Copilot more than once a day, and a further 43% said that they used the AI platform multiple times each week. Only 2% indicated that they did not use the technology whatsoever.
“Policy-focused teams struggled with nuance and when data sources contained multiple contrasting opinions. Concerns were raised on the appropriateness of using M365 Copilot and therefore some users were cautious in its usage.”
Copilot experiment findings report
The average time saved across all participants was 26 minutes each day, the report says – adding that this tots up to 13 days across as whole year.
However, results varied significantly, with 17% reporting no time saved at all, and a further 18% estimating daily savings of no more than 10 minutes.
But 23% pegged the time saved via Copilot at more than half and hour, while one in seven users found that the technology gave them at least an extra hour each day.
The areas where most time was saved included drafting written documents and presentations.
Professional differences
The biggest savings by profession were achieved by project delivery workers, who reported gaining an extra 35 minutes a day, ahead of civil servants in commercial roles with 32 minutes saved.
Four areas – operational delivery, digital and data, communications, and knowledge and information management – were on 31 minutes, while human resources and policy recorded daily savings of 29 minutes, and finance and analysis professionals were each on 25 minutes.
Lower down the list was property on 22 minutes, science and engineering on 21 minutes, and legal on 19 minutes. A long way behind at the bottom the leaderboard was government’s tax profession, where workers saved only 11 minutes per day, on average.
The report says: “Professions with the lowest satisfaction scores were also those that saved the least amount of time. Use of the tool within these roles were impacted by a range of factors. For example, policy-focused teams struggled with nuance and when data sources contained multiple contrasting opinions. Concerns were raised on the appropriateness of using M365 Copilot and therefore some users were cautious in its usage.”
The picture was much more consistent across grades, with everything from entry-level administrative officers up to the ranks of the senior civil service reporting daily time-saving benefits in the range of 24 to 32 minutes.
The report states that “the significant time savings with M365 Copilot meant it was scored highly across the board and users reported high satisfaction scores”.
Teams
M365 program where Copilot was most widely adopted, with 71% peak usage
7.7
Average user satisfaction score
26 minutes
Average daily time saved across all 20,000 participants
Project delivery
Profession with highest reported time savings, of 35 minutes
85%
Proportion of participants that believe Copilot provides good value to their organisation
This is evidenced by the 85% of participants that believed that the technology “provided good value” to their department, as well as 63% – in each case – that reported an uptick in their job satisfaction, and a belief that “their productivity would decline without access to M365 Copilot”.
The average satisfaction score across all users was 7.7 out of 10, while the average recommendation score reached 8.2.
“The most common benefits users highlighted from working with M365 Copilot were around increasing productivity and reducing time searching for information,” the report says. “Many users also signalled benefits from changes in how they spend their time, with productivity gains allowing users to spend time on more strategic and satisfying tasks.”
Concerns and priorities
When given the opportunity to provide detailed qualitative feedback, government found that five major themes emerged, including increased inclusivity and accessibility – particularly for civil servants with disabilities, time savings and other efficiencies, and improved quality of work.
The latter two themes were the challenges of the “learning curve… where feedback indicated a need to learn how to effectively prompt and change usage patterns to get the most out of” the technology, as well “reliability and accuracy, where some users expressed concerns on M365 Copilot’s potential for generating incorrect information… [and] worries about dependency on AI for tasks that involve critical thinking and creativity, as well as the environmental impact”.
In a written statement to the House of Commons, DSIT secretary of state Peter Kyle indicated that “priority should now be given to implementing an AI tool with groups showing the highest and lowest time savings, to better understand the tool’s impact and limitations across diverse user segments and professions”.
“Priority should also be given to exploring how AI tools can improve accessibility, on which we have got some anecdotal evidence, and implementing benefits tracking for deeper insights,” he added. “In line with our commitment to AI adoption in the public sector, we will take these findings forward through the newly established AI adoption unit within the Government Digital Service, which will build and deploy AI into public services, grow AI capacity and capability across government, and ensure trust, responsibility and accountability in all we do. As part of this work, we are exploring a range of AI tools to assess their benefits, recognising that different AI tools and technologies may add more or less value depending on the use case.”
The 12 agencies – each of which deployed at least 1,000 Copilot licences – that took part in the pilot were: the Department for Work and Pensions; HM Revenue and Customs; the Home Office; the Ministry of Justice; the Department for Energy Security and Net Zero; the Department for Science, Innovation and Technology; Foreign Commonwealth and Development Office; the Department for Environment, Food and Rural Affairs; the Department for Business and Trade; Companies House; the Office for National Statistics; and the Welsh Government.
