Big Data: A Profound Influence on Our Lives

Big Data is a topic set to make one of the most profound influences on our lives…We have witnessed several waves of change in the IT landscape: mainframe, mini-computers, PC/microprocessors, client servers, cloud computing and now– Big Data. Though I am sure that most of you are familiar with the term, Big Data relates to the process of capturing and analyzing large amounts of data to uncover hidden patterns, unknown correlations and other useful information. The results are tremendously influential, not only to businesses but also to our society, as I will uncover.

The difference between traditional analytics (which have been around for years) and Big Data lies in three key areas: data volume, data type and processing speed.

1. Data Volume: 12 Zettabytes (1 Trillion GB) of data was created globally in 2011,  it is predicted that data will grow 30x by 2020 (IDC & McKinsey), Wal-Mart generates more than 1M transactions an hour into databases estimated at more than 2.5 petabytes and an average of 294 billion e-mails are sent every day (Economist).
2. Data Type: Applies to structured data (within a database) and unstructured data (that is not within a database), such as Web server logs and Internet click stream data, social media activity reports, mobile phone call detail records and information captured by sensors
3. Data Velocity: Massively Parallel Processing (MPP) databases (such as, Greenplum) allow for the real-time or near real-time analysis of mountains of data, thanks to their high performance database engines that can dramatically improve response times and reduce the amount of data that needs to be stored and transmitted. As an example, just two years ago, it took the CIA 63 days to process a given amount of data, whereas now, it takes them only 27 minutes.

Here are some success stories resulting from the deployment of Big Data:

1. Australia’s Marine department has deployed sensor floats, underwater autonomous vehicles, scientific monitoring stations, remote satellite sensing, and animal tags to collect real-time data about animal migration, ocean salinity, temperature, currents, and carbon storage.
2. European Central Bank is able to sift through volumes of economic data to get an early jump on GDP estimates, which are critical to establishing short-term and long-term fiscal policy.
3. Hungary can now predict consumer behavior in specific regions to gauge major economic activities as car and home purchases.
4. Researchers John Guttag, Collin Stultz, Zeeshan Syed have found that three abnormalities in an EKG are correlated with a 2 -3x higher risk of dying from a second heart attack within a year. They believe their computer model will significantly improve today’s risk-screening techniques, which miss identifying about 70 percent of patients likely to have a repeat heart attack.
5. Researchers at the University of Ontario-Institute of Technology who are using streaming analytics to help neonatal care hospitals predict the onset of potentially fatal infections in premature babies.
6. KTH Royal Institute of Technology in Stockholm, which uses streaming data in their congestion management system, is already reducing traffic in the Swedish capital by 20%, lowering average travel times by almost 50% and decreasing the amount of emissions by 10%.
7. Meteorologists are now able to forecast near-term weather conditions, particularly violent weather approaching a particular region, based on vast amounts of incoming data.

Businesses that exploit Big Data to improve strategy and execution are distancing themselves from competitors. But two challenges do remain, however.  The first has to do with the scarcity of Data Scientists, who are the specialists that run and manage the big data apps. These folks typically don’t already exist within an IT organization, in large part due to the fact that their background is fairly non-IT like: they have skills in quantitative analysis, statistics and mathematics. In fact, the best candidates are PHDs, so it is taking a longer time to source new or train existing resources. The other challenge has to do with the current executive decision-making culture, one that is based more on intuition, as opposed to one that is data-driven or evidence-based. Cultural change normally takes a long time to implement, but the strategic benefits of Big Data will force this change to take place faster.

And finally, there is an opportunity for you to test out the potential of Big Data–  The amazing new project called The Human Face of Big Data, with EMC Corporation as the principal sponsor. The project is aimed at helping the world understand the vital role that Big Data is playing in addressing some of humanity’s biggest challenges. By participating, you can actually discover the hidden secrets about the world you live in. Compare answers about yourself, your family, trust, sleep, sex, dating, and dreams with millions of others around the world. Map your daily footprint, share what brings you luck, and get a glimpse into the one thing people want to experience during their lifetime. Try it!