Blog Series: IT Perspectives by Jaqui Lynch

How to turn big data into bigger insights

We live in the world of the Internet of Things (IOT) where data comes at us from multiple sources. Those sources still include traditional structured data, but now are supplemented by streaming data sources such as wearable devices, social networking sites, web data and intelligent devices, just to name a few. This mass of data is what we refer to as “big data.”

There are many definitions out there for big data, but for the most part it is a buzz-word that is used for large and/or complex sets of data that are difficult to process using traditional databases or software. This difficulty could be due to the amount of data, the arrival rate of the data or even the processing power needed to work with the data. In order to make decisions about how to handle big data it is important to understand what it actually is and the benefits behind working with it.  It is often described using the four V’s of big data:

  • Volume
    • This refers to the size of the dataset which can be petabytes or exabytes in size.
  • Variety
    • This refers to the type of data which typically consists of billions of records from different sources including web, sales, customer contact centers, social media, email, credit cards, sensors, mobile data and other intelligent data. The data can be traditional structured data or large aggregations of unstructured data.
  • Velocity
    • This refers to the speed at which data accumulates. Data can be streamed real-time or it could be received using other methods.
  • Veracity
    • This is a key challenge with unstructured streamed data in particular and refers to how accurate the data actually is.

These four V’s of big data all have an impact on the size of the data as well as storage required and many other technical areas and they need to be evaluated as part of the initial project scoping. Implementing big data involves putting together the environment to gather and analyze data to find relationships that are not obvious and that allow you to discover new opportunities. Gabriel Consulting (2) suggests there is a 5th V – value. They point out that not all data has value but that it may contain individual nuggets or data points, if only you can find them.


There are many challenges to putting together a big data environment. These range from capture and storage to data curation, search, sharing, query/search, analysis, visualization and information privacy.  Some people refer to big data as the new oil, something with great potential value but of little use while in the raw state. A big data environment always starts with raw data of some kind. That data becomes information once it is given meaning, however it is not until we apply context that it becomes real knowledge that can then be used to provide the insight to improve business decisions.

Understanding context is the key to improving customer relationships, improving decision making, reducing risks and finding new opportunities. The role of analytics is to add the necessary context. All data needs to go through these kinds of analytical steps, otherwise it just remains a bunch of data.  The challenge is collecting all the data and providing it with enough context to be valuable. Additionally, the sheer volume and/or velocity of the data makes it challenging to make sense of the data without significant resources.

These challenges have led to new courses being offered in universities. Bioinformatics and data visualization are examples of such courses. They have also given rise to a new career path, that of the data scientist, who is someone who has the following skills:

  • Strong business sense
  • Modelling, statistics, analytics and mathematics skills
  • Ability to communicate findings clearly
  • Ability to understand and interpret data


In their 2011 report (1), McKinsey said “Big data will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus—as long as the right policies and enablers are in place.”

Over time this has turned out to be true.  Some of the early adopters of big data analysis were credit card companies. They use the data to look for fraud by monitoring how people spend money normally. They look for patterns and then flag spending activity that does not conform to the normal pattern for that person. Insurance companies also started with big data early as part of their focus on reducing fraudulent claims. But big data has gone way beyond this now as it incorporates many more devices and pieces of information than what was traditionally being reviewed.

The advantages of big data analysis

  • Finding trends and patterns
  • Providing insights to the business
  • Potential to improve operations and make faster intelligent decisions
  • Ability to focus workers time and other resources where they will yield the most value
  • Can identify relationships that no one knew existed before
  • Ability to make accurate predictions

Big data has also found a home in the medical arena where context aware computing allows healthcare organizations to detect health issues early so that proactive treatments can be recommended. Additionally, the ability to build relationships means that medical records can be analyzed along with current streaming data from intelligent devices, physicians notes and local health outbreak information. This combination leads to better healthcare for clients. There are similar examples in almost all areas of business.


The sheer scale of what is needed to analyze big data has led to some significant software and managed services offerings. There are now SaaS (software-as-a-service) offerings that can take your data and provide analytical insights into your operations. Watson is an example of a custom built environment that takes in data from many sources and processes it to provide expertise that assists with decision making.

SPSS is predictive analytics software that allows you to perform statistical analysis, data and text mining, predictive modeling and decision optimization. Other examples include IBM BigInsights, the Puredata system for Analytics, Cognos, IBM Infosphere Streams, SAP Hana, etc.  There are many other tools out there – these are just a few examples.

Running these on the right hardware is important, as it is critical that the hardware can provide rapid access to data in large in-memory databases. Typically, such systems require massive amounts of both memory and disk space and they can benefit substantially from the addition of technologies such as Flash. POWER8 has been architected to handle the first three V’s of Big Data – volume, variety and velocity. Veracity of the data is something that is handled as part of the analytics.


In a future blog I will look at some of the tools that are of interest in a big data world. As you can see, big data has the ability to drive changes to how IT works. Its ability to analyze data in new ways and to provide information on relationships we never knew existed allows us to explore planning approaches that can lead to making better decisions for IT and the business. This helps us to better align IT with the business and to better optimize resources. big data is being viewed (1) as the next frontier for innovation, competition and productivity. Companies who want to be competitive and flourish should investigate where big data can play a role in their environment, whether it be an internal implementation or one that is hosted and/or managed by a provider. Capturing the value of all the data the company has access to will vastly improve the decision making ability as well as the ability to make predictions as to the future.


  1. McKinsey 2011 report
  1. Gabriel Consulting – Big Data Powers Up
  1. IBM Big Data Resources
  1. IBM Big Data Analytics


Schedule a consultation today to learn how Flagship can help you design and implement an effective big data solution.

If you liked this blog, you also might like:  Forward-thinking Business Intelligence and Analytics

logo-ibmStay connected online:

Facebook | Twitter | LinkedIn | Instagram

IBM Watson

IBM Watson is a technology platform that uses natural language processing and machine learning to reveal insights from large amounts of unstructured data. First Watson learns a new subject:All related materials are loaded into Watson, such as Word documents, PDFs and web pagesQuestions and answers pairs are added to train Watson on the subjectWatson is automatically updated as new information is published  Then Watson answers a question:Watson searches millions of documents to find thousands of possible answersCollects evidence and uses a scoring algorithm to rate the quality of this evidenceRanks all possible answers based on the score of its supporting evidence  What Watson can do for you:Answer your customers' most pressing questionsQuickly extract key information from all documentsReveal insights, patterns and relationships across data   

  • IBM Watson: How it Works

  • Learn how IBM Watson works and has similar thought processes to a human.

  • Video: Welcome to the cognitive era – IBM CEO Ginni Rometty describes a new era in technology and business

  • For the past several years, the converging forces of data, cloud, mobile and social technologies and the Internet of Things have disrupted industries and led to unprecedented transformation across every type of business. Many enterprises have made the shift to become ‘digital.’ However, at IBM we believe that digital is not the destination, but the foundation for a new era of business, which we call cognitive. When digital disruption meets digital intelligence, new possibilities are created by systems that understand, reason and learn.0:00 – Introduction / What is IBM?7:01 – IBM’s Strategic Imperatives15:58 – The IBM Transformation22:44 – The Cognitive Era31:53 – Implications of Cognitive Business38:46 – Lightning Round41:24 – The Future of IBMLearn more at: to the IBM YouTube Channel: to the #CognitiveEra:

  • Video: Announcing the IBM Watson Engagement Advisor

  • Watson can transform the way people interact with companies and other organizations over the lifetime of their relationship. IBM Watson Engagement Advisor is the industry’s first consumer facing cognitive computing breakthrough that will assist a brand’s customer service agents or sit directly in the hands of consumers through mobile devices, cloud-delivered services and online chat sessions.

  • Infographic: Watson Goes To Work For You

  • Consumers are increasingly looking for personalized, efficient interactions with the brands they frequent. To address this need, IBM has unveiled the IBM Watson Engagement Advisor, which uses cloud-delivered mobile and online chat technology to assist businesses’ customers, anytime and anywhere. From answering questions to offering guidance for purchases, IBM Watson will allow a brand’s customer-facing employees to make better recommendations, or sit directly in the hands of consumers, offering helpful advice via mobile engagement. 

  • Infographic: Watson Analytics DIY

  • Do-it-yourself with Watson Analytics. No matter what part of the organization you’re in, find what drives your business–on your own.


  • Welcome to Watson Analytics: Let’s get started

  • Welcome to Watson Analytics! Take a quick tour in this video and see how quickly you can get started analyzing your data.

  • What Watson Analytics Means for the World and You

  • Want to know what people are saying about Watson Analytics? Check out these highlights from the Watson Analytics Signature Moment held in New York on 14 Oct. 2015.