Recently-created positions differ from ‘traditional’ evaluators who usually operate outside government programme areas and are often contracted from the private sector, according to DBT’s head of digital evaluation Natalia Chivite-Matthews
The Government Digital and Data Profession has created a new “digital evaluator” role as part of its capability framework with a view to better supporting ongoing transformation initiatives with “robust evaluation mechanisms”.
According to a blog by Natalia Chivite-Matthews, head of digital evaluation at the Department for Business and Trade, the move follows a performance review of digital spending by HM Treasury and the Department for Science, Innovation and Technology.
The review, which was published in March last year, found that the way digital projects are funded, managed, and monitored needs to change. It also proprosed a series of evaluation improvements.
Chivite-Matthews’ blog, co-authored with DBT digital evaluator Arianna Rossi, sets out the rationale for the new evaluator roles, which have been shaped by experience at the department.
“Digital evaluators are embedded in agile teams, supporting products through continuous learning,” Chivite-Matthews and Rossi explain. “They assess digital products and services for their impact on society, how efficient they are, and any unexpected results.”
Related content
- GDS launches local government unit and cross-public sector skills hub
- Government reveals plan to overhaul digital funding models
- Whitehall digital profession garlanded for external recruitment
Chivite-Matthews and Rossi say evaluators play a “vital role” in demonstrating the impact of digital tools and contributing insights on progress, adding that digital teams provide “one of the most rewarding environments” for evaluators to work in.
“They actively welcome insights and feedback to support continuous improvement, as this is part of the ethos of agile working,” they say. “They are also very supportive of iterative learning as a necessary step in supporting continuous innovation and ideas. It enables the release of resources to move on to new, hopefully better, projects that best meet the needs of users and provide value for money.”
Digital evaluators use the range of evaluation methods, processes and frameworks set out in HM Treasury’s “Magenta Book” on evaluation, along with a host of quantitative and qualitative research techniques. Evaluators also incorporate the principles of the Treasury’s “Green Book” bible for appraising costs and benefits for policies and projects into their work.
“This comprehensive approach not only ensures that digital products and services are thoroughly evaluated to understand what works, what doesn’t and why,” Chivite-Matthews and Rossi say. “It also looks at their social impact, efficiency gains, and potential unintended consequences.”
Evaluators work closely with performance analysts to develop key performance indicators and support dashboard development for monitoring. They also work with user researchers to share insight and often work together in qualitative and quantitative work to ensure it covers both evaluation and user research needs.
According to Chivite-Matthews and Rossi, digital evaluators are different from traditional evaluators in other parts of government in that the latter category of professionals typically operate outside a programme area – with the role often commissioned from the private sector.
They say that while methods employed by traditional evaluators focus on long-term outcomes after implementation, digital evaluators start their work in the “discovery” or “alpha” stages of a project, ensuring the evaluation is continuous and dynamic.
“They develop evaluation plans in-house and have hands-on experience of applied evaluation methodologies,” Chivite-Matthews and Rossi say. “Digital evaluation is more akin to action evaluation, because digital services are developed in an iterative way, so traditional methods are not always applicable or effective. In digital, evaluation is a crucial role to secure continuous improvement.”
Last year’s Government Digital and Data Profession Capability Framework gave five grades of digital evaluator, ranging from “junior digital evaluator” to “head of digital evaluation”.
According to the framework, the roles share skillsets with data governance managers, performance analysts, data analysts, data ethicists and data scientists.
