November 2012

Multiple sponsors

Download pdf

The EMA / 9sight Big Data online survey was comprised of 255 business intelligence and data management professionals, and was designed to identify the key trends surrounding the adoption, expectations and challenges connected to Big Data. This report summarizes the findings. 

Executive Summary

In the Information Technology (IT) industry, 2012 has been the year of Big Data. From a standing start toward the end of the last decade, Big Data has become one of the most talked about topics. There is hardly a vendor who does not have a solution or, at least, a go-to-market strategy. Beyond IT, even the financial and popular press discusses its merits and debates its drawbacks. And yet, the niggling question of exactly how to define Big Data remains. Respondents to this EMA / 9sight survey have clearly indicated that their Big Data solutions range far beyond social media and machine-generated data to include a wide variety of traditional structured and transactional business data.

Still, although the question of definition of “big” may continue to niggle, the answer is becoming increasingly irrelevant. The concept of Big Data has evolved in two key directions. First, was the growing understanding that while size is important, the technology implications of data structure and processing speed are at least as important. Second, what really matters for Big Data is what systemic business cases it supports and what real analytic and operational value can be extracted from it.

Big Data has driven change in our traditional data management strategies and has found a home in an expanding information ecosystem that many companies struggle to manage today. This landscape was once dominated by the enterprise data warehouse (EDW) on the informational side and an array of largely monolithic transaction processing systems on the operational side. This has now given way to an array of data management platforms, including NoSQL platforms like Hadoop. EDWs will continue to play a critical role in this environment but in support of historical and cross-functional consistency to a more sophisticated data management strategy rather than as a central clearing house for all informational needs. This new data management strategy leverages an array of platforms for the highest performance possible and brings together human-sourced information, process-mediated data and machine-generated data as a complete, comprehensive business information resource. At the core of this change is a movement to align data with operational and analytic workloads, each on the best possible platform. This shift in strategy is driven by four significant changes in the data management landscape: Maturing User Community, New Technology, Economic Value and Valuable Data

This new home for Big Data is a muliple node ecosystem of data management platforms. In this ecosystem, each node or platform has an equally critical role in supporting the sophisticated workloads that today’s Big Data requirements demand.

In many cases, it is the combination of the multiple platforms that enables success in addressing of the following requirements: Response, Economics, Workload, Load and Structure.

Each of the nodes involved in this environment delivers a specialized value proposition by addressing the drivers mentioned above and applying appropriate feature sets to meet Big Data requirements.

Key Findings

The EMA / 9sight Big Data online survey was comprised of 255 business intelligence and data management professionals who qualified to participate in this research. The survey instrument was designed to identify the key trends surrounding the adoption, expectations and challenges connected to Big Data. The research identifies trends surrounding Big Data technology, its use, adoption and how it impacts analytics. Below are highlights and key findings from the research:

  • Big Data Strategies are on the Move: Respondents who are already working on Big Data projects are doing so at ambitious rates. Over 36% of our respondents are In Operation with a Big Data oriented project. 35% are following close behind in Serious Planning mode.
  • Enterprise is leading the way: Enterprise-sized companies are the early adopters of Big Data driven analytics. Nearly 40% of Enterprise-sized organizations in the EMA / 9sight survey have indicated that they have implemented Big Data solutions on some scale, as either a production environment or a pilot system.
  • It is not Easy: Most major Industries are embracing Big Data technology at some level. These projects are driven or sponsored by an array of stakeholders within the organization. Big Data sponsors vary by industry. The Finance department is the biggest proponent of Big Data projects in Healthcare representing over 16% of the responses, while Big Data in Leisure industries is primarily driven by CEO and Executive level management 21%.
  • Different Industries, Different Approaches: By industry there are significant differences in how Big Data is used to drive value. The Public Service industry leads all others with 31% of respondents identifying Online Archiving as their primary use case. 31% of Media & PR respondents are primarily focused on staging structured data. 22% of financial services respondents are investing in combining data by structure.
  • It is not the Size of the Data: Big Data isn’t as big as the market buzz indicates. Less than 10% of our respondents are managing 750 terabytes or more within their overall system. The most common enterprise size data environments are 50-100TB. Of that data, most companies have 10-30TB in their Big Data environments indicating that Big Data analytics can be served on a variety of platforms; not just Hadoop.
  • Data Diversity: The data that feeds Big Data systems comes from a diverse set of sources. Our respondents identified structured operational data, human-generated documents and deep operational transaction data as the three most popular for Big Data projects.