© Copyright InfoWorld Media Group. All rights reserved.
Big Data AnalyticsDeep Dive
Making sense of big data
New analysis tools and abundant processing power unlock critical insights from unfathomable volumes of corporate and external data
i By David S. Linthicum
THE ABILITY TO DERIVE MEANING quickly from
huge quantities of structured and unstructured data has
been an objective of enterprise IT since the inception of
databases. In the past, this has been a distant dream — or at least a cost-prohibitive one.
Today we have a growing number of technologies that
make that dream a reality. Cloud computing provides
per-drink access to thousands of processor cores and
massive amounts of on-demand data storage. Emerging
technologies apply a divide-and-conquer approach to
big data computing problems, using distributed processing to return results almost instantaneously from huge data sets.
New analytics tools take advantage of all this horsepower. Advanced data visualization technology makes large and complex data sets understandable and enables
domain experts to spot underlying trends and patterns.
Some tools even recognize patterns and alert users about
issues that need attention.
Big data is riding a sharp growth curve. IDC predicts
big data technology and services will grow worldwide
from $3.2 billion in 2010 to $16.9 billion in 2015. This
represents a compound annual growth rate of 40 percent — about seven times that of the overall information and communications technology market.
MAKING THE BUSINESS CASE
The business benefits are clear. On the one hand, we
can derive meaning from data that once merely took up
space — website clickstream data, system event logs, and
so on — and use that information to improve a broad
range of systems. A whole new world of vertical applications opens up. What sort of applications? How about sensor data
collected on jet engine performance that could, when
INFOWORLD.COM DEEP DIVE SERIES
analyzed over time, reduce the incidence of failure. Data
on the vital signs of infants in hospital nurseries could be plumbed for patterns that reveal hazards or new opportunities for progress. The use cases are limited only by the imagination:
• A bank visualizes patterns within data and documents that determine the likelihood of fraud and can take corrective action before the business is
• Doctors determine patterns of treatment of diseases that provide the most desirable outcomes using 20 years of historical data from more than
30 different data sources.
• Auto manufacturers see the core cause of
production delays, and perhaps attach these
analytics to business processes to automatically
take corrective action, such as leveraging different
• The utility industry creates predictive models
of consumer markets by deploying technologies
such as a smart grid.
With the proper data visualization on top, big data can
shortcut the usual process of business analytics, where
stakeholders pass requirements to analytics professionals who create static reports and hand them back over the transom. Big data visualization — even a simple tag
cloud — enables those with domain expertise to develop
a more direct relationship with the data and spot patterns that an analytics professional might miss. Despite all the promise, big data technology and
applications are still very much at an early stage. Hadoop
has become the preferred platform for processing large
quantities of unstructured data, but it’s a distributed processing framework and development environment that typically requires specialized application development
skills to use effectively. And that’s just for unstructured
Big Data AnalyticsDeep Dive
or semi-structured data. Big data can also refer to quantities of structured data so large they can’t...