Big data is here. The results of this survey clearly indicate a great deal of excitement and activity around planning and implementing big data environments. The IT managers surveyed for this report represent large companies currently involved in big data analytics.2 They come from organizations generating a median of 300 terabytes (TB) of data weekly and represent a mix of verticals. They are being asked to manage a wide range of data sources, and most have already gone beyond structured data to begin processing unstructured and semistructured data. Almost half are implementing or have already implemented big data tools and technologies.
Our group identified a number of challenges and obstacles to successful big data implementations. Security is called out loud and clear, an issue that is most problematic when it comes to worries about data security and privacy related to the use of third-party cloud service providers. However, when it comes to big data analytics, companies face a number of different challenges, including infrastructure and data governance and policy issues. Despite this, IT managers expressed genuine excitement about big data analytics and consider it a top IT priority for their organizations.
Key findings from our study include:
“Highly engaged” IT managers in big data analytics share important characteristics. One-third (33 percent) of companies surveyed are working with very large amounts of data. This group is “highly engaged” with big data analytics, and individuals share a number of important characteristics, including:
• Working with 500 TB or more of data per week
• Prioritizing big data analytics—very high importance is attached to improving these capabilities
• Working from a formalized strategy for big data
• Currently processing unstructured data sources
The top data source continues to be business transactions—but that’s changing. While data comes from a variety of sources—both structured and unstructured—most IT managers cited business transactions in a database as the top source. However, general business documents, e-mail, sensor or device data, and imaging data—all unstructured data—were in the top five.
Most IT managers are processing unstructured data of some type. Some 84 percent are currently analyzing unstructured data, and 44 percent of those that aren’t expect to do so in the next 12 to 18 months.Adoption of big data tools and technologies continues to grow. Slightly less than half are implementing or have already adopted big data tools or technologies, with one-third reporting they are in the evaluation stage.IT managers predict that most big data analytics will be in real time by 2015. While batch versus real-time data analytics is currently split 50-50, respondents predict that by 2015, nearly two-thirds (63 percent) of all analytics will be done in real time.