Recent incidents brought significant additional attention to the hosting infrastructure of government. PublicTechnology talked to some experts from the civil service and the supplier community to ask what happens next.
“If you’re a technologist, I don’t think it fundamentally changes the thinking – but it’s a really good wake-up call.”
This is government chief technology officer David Knott’s take on the recent series of high-profile outages – encompassing incidents affecting Amazon Web Services, Microsoft, and Cloudflare. All three service shutdowns had at least some impact on government websites and services.
Knott, speaking at a discussion hosted by the Open Cloud Coalition, says that his initial response to the outages – and, indeed, to any such incident was to reach for two words
“So, the first thing is: don’t panic,” he says. “Restore the service, and then do the problem management, figure out what the cause was, and why did that cause have such a widespread impact.”
He adds: “I think the wake up call is [a reminder] to know your clouds, and to know what are the operational characteristics of [those] clouds – down to the physical distribution of assets level – so you can then make a judgment: did we build those things right in the first place so they could survive those kind of incidents? Because those kind of things will happen, and they’ll happen to everybody. So I think, for the technologist, [the response to the outages] is more that we better get on with this quickly and understand our footprint and the dependencies we have, and be able to plan for aggregate risk more effectively.”
“For years, people have seen the market start to coalesce to a very few providers… but were not really prepared to put their heads above the parapet and to speak out publicly.”
Nicky Stewart, OCC
Beyond digital and data specialists, the recent incidents may also be a useful illustrative exercise for “for non-technical people who may have had a slightly rosy view that ‘this cloud stuff never goes wrong, because these are giant globe-spanning [systems’ – it’s another wake up call that this stuff does go wrong”.
“So, when your technologists are saying ‘let’s think about these failure modes and these different scenarios’ – even if they’re unlikely – maybe we better start paying attention to them.”
The need for greater focus on this issue is amplified as an ever-greater proportion of the operations of the state is held in a cloud environment. The State of digital government review released earlier this year by the Government Digital Service, where Knott’s role is based, found that – while adoption varied across organisations – about four in five Whitehall entities now have at least 40% of their estate hosted in the cloud.
More than half of government agencies have passed 60% cloud adoption, and official GDS guidance advises that “public sector organisations should default to public cloud first, using other solutions only where this is not possible”.
Knott says: “If we are serious about cloud being the default deployment strategy, at some point we’ll get to 100% [adoption] – and that’s quite a disconcerting place to be. I think part of our job in the centre is to ask ourselves the question – and ask it to all our partners and to the industry: what would need to be true for us to be confident that us being a 100% cloud-hosted nation in the public sector was a comfortable place to be? And the answer is probably not a configuration that looks like [what we have] today.”
He adds: “As we approach that 100% point, we’re increasing the impact [of outages]. We’re always now going to be high on the impact dimension of the risk score. The only dimension we can control is the risk side of the impact.”
Opening up
Formally launched at the start of this year, the Open Cloud Coalition (OCC) describes itself as an “advocacy group, promoting a diverse, competitive, and resilient cloud services industry across the UK, EU, and global markets”.
The organisation currently has just over 20 members, including major players like Google Cloud, as well as smaller local hosting providers and services firms.
“What binds our members in common is a commitment to a free, open and fair, cloud market – and I would include AI in that as well – that is, above all, founded on competition,” says Nicky Stewart, the OCC’s senior advisor. “Why now? Because, behind closed doors, these conversations have been going on for years, as people have seen that the market started to coalesce to a very few providers… but companies and organisations were not really prepared to put their heads above the parapet and to speak out publicly. I think what’s empowered them now is the degree of regulatory scrutiny that cloud is getting, not just in the UK, but in Europe and beyond.”
This scrutiny includes a major report published earlier this year by the Competition and Markets Authority, which found that across the infrastructure- and platform-as-a-service markets, AWS and Microsoft account for as much as 80% of the collective UK market.
The dominant duo “hold significant unilateral market power in these markets, [and] this harms competition in cloud services in the UK”, the CMA warned.
18%
Baseline public sector discount offered by the first version of government’s OGVA deal with Microsoft
May 2013
Date of introduction of ‘cloud first’ policy across government
£1.3bn
Cumulative annual government budget for cloud computing, according to GDS
55%
Proportion of central government entities that have at least three-fifths of their infrastructure in the cloud
The watchdog proposed designating the two firms with “strategic market status”, which would “allow the CMA… to take a targeted and iterative approach to address… market-wide concerns by directly benefitting the majority of UK customers and producing wider indirect effects by altering the competitive conditions for other providers.”
Stewart tells attendees of the OCC event that one of the report’s most troubling findings was that less than 1% of organisations change their cloud services provider each year. This stasis is caused by issues including restrictive licensing practices and charges for moving data to a rival provider, as well as challenges with technological interoperability, she adds.
But the OCC believes even the measures set out by the CMA are “not going far enough”.
“There are other things in this market which are also damaging competition, such as committed spend agreements – which is where a cloud customer will, in return for a discount, commit to spend a certain amount of money or buy a certain volume of product over a certain term. And that discount will be even greater if you commit to paying some or all of that amount upfront. What that does is lock the customer in – particularly if they’ve paid upfront – if, halfway through, they think: ‘These outages have slightly scared us, and we want to diversify some of our workloads’. Well, they can’t – because they will lose all that money. And that actually forecloses the market for a large period of time for other cloud providers.”
This kind of model is exemplified by the One Government Value Agreement (OGVA) between government and AWS, in which, by committing to three-year contracts, public bodies can achieve significant discounts on the cloud providers’ services.
The first iteration of the agreement, which ran from 2020 to 2023, offered baseline savings of 18%. The OGVA was, effectively, renewed for a second three-year term that runs until 2026 – although neither government nor AWS has confirmed whether the current version offers the same terms and discounts of its predecessor.
“I cannot see… that government’s going to turn around and not have another agreement with AWS – because they can’t afford to lose that discount,” Stewart says. “So, potentially, that’s a market foreclosed for up to decade for some – and that is before you get into the peripheral commercial issues, such as free cloud credits. Which, in the case of the dominant cloud providers, can run into hundreds of thousands of dollars. And they go after new business. They go after startups… and it’s just enough to draw in the customer, just enough to get them completely enmeshed into these proprietary platforms. And, when they come to the end of their credits, and they think ‘we’ll go back to.. the people we were talking to before we were offered this’ – they can’t, because they’re stuck.”
Better together
Stewart notes that – even with the Google put to one side – the OCC’s roster of member companies are “collectively capable of making the kind of inward investments that the big US global tech firms are making into the UK – it’s just that the government is not giving the right signals” as it stands.
Mike Hoy, chief technology officer of CMA member Pulsant, notes that there can also be “a danger that we all trip over each other as well”.
“If we are serious about cloud being the default deployment strategy, at some point we’ll get to 100% adoption – and that’s quite a disconcerting place to be.”
David Knott, government CTO
“We’ve got lots of small cloud providers all trying to [bid] for the same piece of work,” he says. “[We could] actually be working together on that. There’s members of this group that can come together to scale infrastructure. There’s members that can come together to get the standards right. We can start to think about how can we collectively be stronger together, than any [of us would be] as individuals.”
Government, for its part, can also do a better job of reaching out and engaging with the market, according to Knott – who is leaving his post at the end of this year after three years in the role.
“I’m conscious that the big, well-known global hyperscalers have a lot more brand recognition amongst buyers in the public sector than probably many of the companies in this room,” he says. “One of the side effects of the previous procurement rules – in fact, not even the rules, but people’s perception them – is that people have been very nervous about speaking to the markets. I spend a lot of my time reassuring people that, as technologists, it’s fine to talk with your partners – that’s actually what we expect.”
