Big Data and Analytics in the Public Sector

Posted on: 01/02/2019

Public organisations are sitting on a wealth of data which could be genuinely transformative, increasing efficiency and creating new opportunities. But can they rise to the challenge?


At an individual level, there is not much the public sector does not know about you. It knows when and where you were born, where and how you were educated, your household income, employment history and now, with the advent of smart-listening devices, even what you are talking about in your home. Interaction with data in our everyday lives is evolving in such a way that regular experiences, such as online shopping, are intensely personalised to our individual preferences. There is little doubt that as consumers, we have been empowered to get exactly what we want, when we want it; data has become a part of our life blood. It is unsurprising then how Big Data Analytics has rapidly ascended as a major technique for companies to capitalise on the potential that sits within their data. Large technological institutions (i.e., Google) have built entire business models solely on its exploitation.

Given the profound stream of data that the private sector can access, the insight to be gained by the public sector is equally, if not more compelling. Moreover, in the current economic climate it is especially important that public bodies become more efficient to continue delivering the core services that the public require. However, according to a new study, although nearly half of UK public organisations now apply big data analytics for insights, 70% of these projects fail to realise their full potential. An overall digital skills shortage is repeatedly cited as the chief reason, as many public bodies are still finding their way around Big Data, a concept still in its infancy. Data analytic skills, a mixture of analytical industry knowledge and technical skills, are highly sought after. Despite numerous initiatives to improve these skills in the UK, including the introduction of a new national curriculum and direct investment, demand is still outstripping supply. Furthermore, a host of privacy and legal regulations as well as a strong duty of care over people’s most sensitive information pose additional challenges to the public sector in effectively utilising Big Data and Analytics.

To overcome these challenges, public bodies should ideally have big data strategies in place to determine how they will use the rich amount data that is available to them. A cultural shift may also be required as attitudes take time to change and the provision of information can help people become more willing to use Big Data in their work. Due to the highly personal nature of the data, various ethical issues arise concerning how to use the information about individuals and whether persons should be identifiable. In their analysis of how local governments and institutions are indeed using intelligence from big data, Fola Malomo and Vania Sena suggest that the ambitions around the development of big data capabilities in local government are not reflected in actual use. Indeed, these methods have mostly been employed to develop new digital channels for service delivery, and even if the financial benefits of these initiatives are documented, very little is known about the benefits generated by them for the local communities. While this is changing as councils start to develop their big data capability, the overall impression gained from even a cursory overview is that the full potential of big data is yet to be exploited.

Healthcare is a prime example where big data and analytics can create value and radically improve services for the public. With underfunding reaching crucial levels and average wait-times to see GPs and receive routine operations increasing, there is ongoing debate regarding how the NHS could improve efficiencies. Population health management, which promises to identify patients who are at high risk of developing chronic illnesses, may be the key to these issues. For ‘population analytics’ to deliver true value, comprehensive and thorough patient records are needed in conjunction with information from GPs and local authorities. However, efforts to develop such records within sophisticated IT structures are hampered by the fact that, for many NHS providers, much of the clinical records are still being stored on non-digital platforms; overall digital maturity across the sector remains low.

There is no doubt that Big Data and Analytics will shape the future of the public sector, particularly the NHS. It is also clear that data analytic science, although relatively new, is developing fast in the UK – in 2018, the Department for Work and Pensions saw a 400% rise in the number of data scientists it employs. Unfortunately, despite these efforts, we are still not producing the number of qualified candidates that we should be, especially when we consider the amount of data we are producing daily, at exponential rates. In a recent report, think tank Parliament Street has suggested that the government should establish a national data science strategy together with a cross-department career programme in order to create smarter, more effective strategies to improve the way we handle, process and interpret data – “Big data analytics projects are already saving the private sector billions of pounds and it’s time for the public sector to reap the same benefits. Investing in highly skilled data scientists and the latest BI software will help create a smarter, more efficient government and the sooner this happens, the better.”