Big Data, Big Risks, Big Opportunities
What Exactly is Big Data?
Big Data refers to large, complex datasets. Today it is often used to describe data that:
- Is significant in Volume (terabytes), has Variety (different types), and is high Velocity (generated in real-time);
- Is technically not “big” enough to be called big data, but reflects an organization’s desire for greater, faster, and more efficient decision-making and insights.
Regardless of the technicalities, Big Data is having a big impact on organizational effectiveness, efficiency, and insights into consumer behavior, supply chain and logistics, financial markets, and more.
Examples of Big Data Analytical Techniques
In the International Journal for Information Management, Gandomi and Haider provide commonly used methods of data analysis in Big Data [PDF], summarized below:
Video analytics, also known as video content analysis (VCA), involves a variety of techniques to monitor, analyze, and extract meaningful information from video streams.
Social Media Analytics
Social media analytics refer to the analysis of structured and unstructured data from social media channels, including social networks (Facebook, LinkedIn), blogs (Blogger, WordPress), microblogs (Twitter, Tumblr) social news (Reddit), review sites (Yelp, TripAdvisor), and more.
Predictive analytics comprise a variety of techniques that predict future outcomes based on historical and current data. In practice, predictive analytics can be applied to almost all disciplines – from predicting the failure of jet engines based on the stream of data from several thousand sensors, to predicting customers’ next moves based on what they buy, when they buy, and even what they say on social media.
Simplified Sample Scenario – Insurance Company Customer Service Call
- Speech analytics using a tool like Vokaturi to detect customer emotions, and then provide live guidance to the customer service agent (e.g., phrases to defuse situation if customer is upset).
- Text analytics using a tool like IBM Watson to identify key concepts in the transcribed speech, and then provide live guidance (e.g., the customer mentioned “boat,” the system can ask the agent to inquire if she needs boat insurance).
- Predictive analytics for management reporting and decision making (e.g., likelihood of hitting specific sales targets, best prospects to pursue, etc.).
Simplified Big Data Opportunity Example – Point-of-Care Intervention Opportunities
Additionally, solutions like SAS can help with fraud prevention and risk adjustment. Other analytics providers, such as NICE Systems, can even help with ensuring broader population well-being with public safety solutions based on audio analytics and more. IBM’s Watson Health initiative spans numerous related technology disciplines including A.I. and machine learning, but at its core relies on Big Data as the source of its insights.
Big Data Technology Landscape
The Big Data landscape is diverse and fragmented, with companies providing a variety of related services:
- User Applications
- Data Sources and Interfaces
Many of the larger companies, such as SAS, Oracle, IBM, Microsoft, Google, and Amazon, are seeking to build a comprehensive ecosystem of data, analytics, machine learning, and business intelligence products.
Matt Turck from Firstmark provides a large overview of the industry players by category, shown here. [PNG]
The companies featured in Turck’s diagram span the spectrum of early-stage and mature companies, both in their overall company existence as well as their development of Big Data solutions and services. Others, such as Knowre in the Education Application space, are on the fringes of Big Data, focusing more on machine learning algorithms with smaller data sets.
The risks of Big Data are manifold, and organizations need to carefully plan for their use of Big Data solutions. These risks include strategic and business risks, such as operational impacts and cost overruns, as well as technical risks, such as data quality and security. Six of the key risks are highlighted in the table below.
Loss of Strategic Focus
Organizations that jump into Big Data without specific goals in mind, or do not know what they are looking for from Big Data are at risk of big investment without commensurate results, information overload and analysis paralysis.
– Create a strategic roadmap that defines the role of big data and data analytics.
– Tie big data projects to specific strategic business goals and outcomes.
Poor Data Quality
Big Data oftens means Lots of Bad Data. This increases the risk of generating outputs and insights from data analysis that are wrong, or even dangerous. While Big Data’s scale and volume sometimes means that idiosyncracies are washed out, there are challenging problems here: in some industries, small discrepancies can mean life or death; and for many, the idiosyncracies are the real business opportunities.
– Design, implement, and maintain an ongoing data quality assurance plan.
– Verify the sources and quality of your data, especially if it has passed through multiple systems.
– Establish a Data Governance plan to ensure accountability and response.
Like many other types of solution implementations, Big Data implementations can be expensive. They often require setup of multiple platforms and systems, including cloud infrastructure, applications, and complex integrations with existing systems. The combination of complexity and relative immaturity of these systems greatly increases the risk of cost overruns.
– Conduct an assessment to determine the ROI and business case for implementing a Big Data solution.
– Establish strong program/project management and governance for Big Data projects.
Big Data provides numerous potential insights from data; in fact, analytics companies are often judged by the novelty and quantity of insights that their algorithms can generate. However, these insights can rapidly change or even contradict others, and should be continuously tested and used as only one factor in decision-making. Companies that chase Big Data insights too aggressively are at risk of making too many high-impact and complex business process changes without being able to accurately predict outcomes.
– Conduct a strategic assessment to test and ensure alignment of project goals with overall business outcomes.
– Set concrete, achievable, and measurable goals for Big Data projects, and evolve projects to change focus from insight-generation to long-term success.
Security and Privacy
The very nature of Big Data increases its security risks: Big Data uses distributed systems, which requires multiple levels of protection; automated data transfers on a very frequent or real-time basis need to be constantly authenticated; data origin is often unknown or untracked because of the scale of the data; Big Data can include sensitive Personal Identifiable Information or Personal Health Information.
– Establish a Data Security Plan and Data Governance Plan.
– Ensure the CSO role provides executive-level accountability and oversight over data security.
– Anonymize data where possible.
– Frequently audit data security, privacy, and compliance; and review and update Data Security plans as needed.
– Use independent, third-party review and oversight as needed.
The Importance of Project Execution in Big Data
The title of this section seems fundamentally contradictory: Big Data is an exciting and young field of interest, especially for enterprise applications, while Project Management is the mundane province of Gantt charts and issue trackers. However, Big Data implementations and data analytics capabilities must fit into the overall organizational strategy, business processes, and business constraints.
With a strong focus on program governance, data governance, and effective project management, Big Data projects can become integrated into the overall business, rather than silos. As with any project, there are several core components of project management required for success:
- Program/Project Governance
- Scope Management
- Schedule Management
- Budget Management
- Risk Management
- Issue Management
- Decision and Change Management
- Data Governance and Management
- Quality Management
- Communications Planning
- Other, as required for the project (e.g., Procurement, Organizational Change Management, etc.)
More information about can be found in Excardo’s feature articles about Project Management.
Want to Learn More?
If we didn’t answer all of your questions, or if you just want to chat about Big Data challenges, feel free to send us a message.