THE STATE OF BIG DATA IN 2020

Posted on: 25/08/2020

WHISHWORKS reveals results of Big Data survey

In the past 12-18 months two key developments marked the course of the big data industry: the mergers and acquisitions that took place throughout 2019, and the more recent coronavirus outbreak.

Mergers and Acquisitions in the big data space

2019 was a big year across the big data landscape. After starting the year with the Cloudera and Hortonworks merger, we saw HPE acquiring MapR, Salesforce acquiring Tableau, Google acquiring Looker and the list goes on. Common themes around these mergers and acquisitions have been the gradual shift from on-premise Hadoop-based platforms to cloud data platforms, together with an increasing demand from end-users for real time data streaming and analysis.

Reprioritisation led by Covid 19

The impact of Covid 19 has been felt globally and by everyone. Different industries have been affected in different ways, which in turn has led to a reprioritisation of their technology investment and projects. For some, big data technologies are the way forward to prevent, detect and respond to potential incidents in the future so that their impact is minimised. For others, achieving a leaner model of operation whilst increasing sales is the only sustainable way to remain in business. Regardless of the reason, organisations of all sizes are realising that the ability to access, process and analyse data in time, has more power than ever before and it will be a key driver across industries and functions in the new normal.

The State of Big Data in 2020

So how have these and other developments affected corporate investment in big data? How successful have initiatives been so far? What is the best approach to implementation, and what are the solutions? WHISHWORKS has conducted a survey among big data strategists, architects and users and has consolidated the findings in its latest report: The State of Big Data 2020.

From the findings of this year’s survey, we see that a lot of progress has been made in the big data space, with an increasing number of companies adopting big data. There are three underlying themes shaping the state of big data in 2020: the cloud, real-time streaming and Covid-19.

The cost efficiencies and flexibility offered by cloud service providers have contributed in accelerating the adoption process, taking the reins from the traditional, on-premise data platforms. As the amount of data increases, costs associated with space, power usage, infrastructure and security can become overwhelming and deter further investment to big data projects. Most cloud solutions are highly scalable and help to reduce infrastructure cost and complexity.

Another trend that is highlighted in the report is the increased demand for real-time capabilities across functions and industries. It is true that in the past few years we have seen a massive adoption of real time streaming, and in particular Apache Kafka, across thousands of organizations and today, 60% of Fortune 100 companies are leveraging real time streaming. In our survey, 45% of the respondents selected real time streaming and analytics as one of the key analytical applications used in their organisations, the majority of which are in banking, medical devices and media and entertainment.

Last but not least, the impact of Covid-19 can’t be overlooked. It changed the way organisations prioritise and direct their technology investments and projects, more cognizant of the value of data-driven business insights in sustaining their operation.

Report highlights

The number of companies actively looking into adopting big data has increased to 18% from just 6% in 2018.
There is a clear shift from on-premise Hadoop-based platforms to cloud data platforms (46%), accompanied by an increasing demand from end-users for real time data streaming and analysis (51%).
Investments in big data technologies are still increasing although at a lower rate than before (59% from 81% a year ago).
Cost rationalisation and cost savings have shown the strongest increase as a key driver for initiating investment in big data.Limited understanding across the organisation and lack of experienced resources remain amongst the top reasons preventing big data projects to progress, closely followed by the complexity of the technology landscape.