Andy Grant explores how local authorities can overcome skills shortages to get the most from their Big Data resources.
Big Data has vast potential within local authorities and councils largely because they tend to gather huge volumes of data, from council tax submissions to local voting records to details of road improvement schemes. Unfortunately, at the moment, few of these organisations know how to fulfil the potential of the data they collect.
One of the most significant problems they face is the rapid rate of technological change. The big data field is moving so quickly that local government is struggling to keep up with the latest developments. The market is hugely complex and few IT staff working for councils have the experience and expertise to use it.
Data is growing at such a pace and in so many different formats that many councils and local authorities are investing large sums just to store and manage it. It’s a complicated process in itself to quantify this data and create meaningful metadata that makes it easier to search and analyse. Over and above this, local authorities often struggle to manage the many data sets they deal with that are incomplete or contain inaccurate material.
This overriding emphasis on storing and managing data means that local authorities have little time or resource to invest in data analytics per se, let alone in developing the analytics skills of their internal IT resources.
Training targeted at bringing people up-to-date with the latest techniques offers one potential solution, but local authorities will need a good business case to take the plunge. With ongoing budget cuts, they will need to justify the investment by presenting a compelling argument as to why it will bring the organisation greater value than alternative spending plans.
So how can they prove the case in this scenario? One project that provides a potential solution is the Government-funded Administrative Data Research Network (ADRN), which has been established across the UK over the past year. This pioneering initiative brings together universities, government, national statistics authorities, funders and researchers to maximise the benefits offered by reusing UK administrative data for research in the public benefit.
Typical examples of projects might include urban planning initiatives where, for the sake of argument, council data on bus routes or traffic congestion could be combined with historic weather information to work out the chances of floods occurring and therefore the opportunity to re-route the buses to avoid a major traffic congestion issue.
With these kinds of projects, different ‘what if’ scenarios can be run. Councils and local authorities can benefit from understanding how different data sets can be brought together and analysed and they can gain an insight into how they can better target their services to the public, while saving money at the same time, by using the information gleaned to better manage their resources.
Dealing with concerns
Unfortunately, these organisations are often held back from taking advantage of the big data opportunity by public concerns about data security. People are naturally reluctant to provide personal data, information about healthcare, religion or their political beliefs, for example, but if they feel there is likely to be a benefit in terms of enhanced services offered to them by the local authority, then they will be much more willing to grant access to that information.
There is much for local government to learn here, not just in the value of getting the public involved, but also in terms of how they can best communicate the benefits of information sharing to their target audience. Once again, they may lack the internal skills to do this and may need external guidance in order to make the right choice.
It is also easy for councils and local authorities to make expensive mistakes in buying the wrong technology, or systems that are not capable of delivering the kind of capability they were expecting. To avoid this possibility, they should consider developing proof-of-concepts or technology demonstrators to effectively try before they buy.
Also, because central government is investing heavily in big data, there are a great many training activities and workshops happening around the UK at the moment. IT staff within public sector organisations should be encouraged to attend to get to understand the language of the technology and better appreciate some of the issues in implementation before they take the next step.
Making it simpler
The ease of use that systems and solutions provide is also key in helping to address the public sector skills gap. It is important therefore to be able to provide ease of access mechanisms via the web; put web portals onto big data and to deliver analytical servers in this way also. Similar interfaces are currently being developed to combine different types of data sets to try to extract new knowledge from them.
Further helping drive the uptake of big data within the public sector, we are seeing a growing number of projects that successfully combine high-performance computing (HPC) and Big Data, many of which are both ambitious and future looking and which deliver cloud interfaces that provide ease of access for the non-expert.
It is clear that most councils and local authorities operating across the UK are currently storing and managing vast volumes of data that they have collected in the course of carrying out their day-to-day activities. But with councils often strapped for cash and struggling to find the time, resources or expertise to move to the next level and start extracting added value from the data they hold by running analytics on it, outside help is called for. And that’s exactly what we are now starting to see emerge in the shape of training schemes, government funded initiatives as well as workshops and technology that supports the development of proof-of-concepts or technology demonstrators.
There’s still much work to be done but big data is really starting to happen within the UK’s councils and local authorities.
Andy Grant is director of high-performance computing and Big Data practice at data storage supplier Bull UK & Ireland