Business Insights of Big Data & Analytics Technology Paradigm

Big Data & Analytics technology, which has been adopted and popularized by core technology companies like Google, Twitter, Facebook, Yahoo etc., is slowly making its way into Fortune 5000 companies. Companies at the forefront of adopting new technologies have moved swiftly from the pilot phase to the production roll-out phase for complex business use cases. Below are some insights that we gained gained from our experience in Big Data & Analytics consulting and implementation:
  • Big data & Analytics (Predictive, Data Science & Machine Learning) has created a new paradigm shift in the business world regarding how we think about data assets, how we collect them, how we process them, and how we monetize the insights from the analysis. The Big Data & Analytics revolution is all about finding new business value by mining the data from both internal and external data sources
  • Big data & Analytics has become disruptive force — enabling levels of insight into a business’s customer demands, market opportunities, and competitive/individual pricing of products or services that are currently difficult to achieve because of technology limitations
  • New Tools ,Technology Frameworks,and Implementation Approaches, are being created to capture, process and manage the new forms of data and generate insights within reasonable time frames
Image
  • Organizations are challenged to re-organize or extend their current Analytics & Information Management (AIM) landscape to the new era of Big Data & Analytics
  • New Enterprise Data Architectures are being centered around re-balancing, coexistence, and cross-pollination, with current Data Architectures in the near-term (next 1-2 years) before a complete new paradigm shift occurs (after 3 years)
Image
  • Organizations are faced with the following dilemmas:
    • Using Open Source vs. Proprietary Software
    • Cloud vs. On Premise
    • Protecting their current investment and extending vs.investing completely in new tools
    • Re-training their existing resources vs. bringing in new talent
  • As more than dozen Software and Hardware vendors are vying for a piece of the big data pie, organizations are evaluating their choices based on the value those vendors bring to the table for their specific needs:
    • Industry heavy weights like Oracle, with its Hadoop-framework-based Big Data Appliance, IBM, with InfoSphere BigInsights that uses Hadoop, SAP with SAP-HANA to harness Hadoop’s with in-memory computing and real-time reporting
    • Specialized players like Teradata with Aster Data Systems, Informatica with Big Insights edition and Microstrategy with Visual Insights on Hadoop distributions
    • Hadoop Open Source Enterprise Support vendors like Cloudera, Hortonworks and Map R
    • Pure play cloud EDW/BI based offerings from Microsoft (Azure HDInsight + Power BI Platform), Amazon (Redshift+ QuickSight) and Snowflake
  • Maximum business benefits are derived from Big Data projects only if both Business & IT takes ownership of those projects with collaborative team structure and proper governance
 
  • Some of the low hanging business use cases that have been implemented using Big Data and Advanced Analytics technologies are:
    • Using Hadoop as Enterprise Data Archive (Active Archive) Solution
      • The growing volume, velocity and range of data are making current traditional data archive mechanisms an expensive undertaking and also the approach of archiving the data in secondary storage devices makes retrieval of this data for analysis painful and cumbersome.
      • Since the solution on Hadoop Platform is Cost Effective, Flexible, Scalable, fault tolerant and provides data on-demand most of the organizations are migrating to this solution
  • Replacing Data Staging / ODS Layer of EDW Platform with Hadoop from RDBMS
    • In the Enterprise Data Warehouse architecture, staging layer and Operational Data Store (ODS) Layer are necessary ingradients but does not add lot of business value. Since most of the Enterprise Data Warehouse are built on platforms which are expensive, moving these layers to Hadoop is helping the organizations to achieve better ROI out of their EDW’s
    • Staging and ODS layers of Hadoop is helping the organizations to perform data discovery and exploratory analysis to understand business patterns
    • Predictive Maintenance
      • For businesses, predicting machine failures early on would help in cutting down the maintenance cost as well as minimizing production downtime
      • Predictive models built on top of historical data patterns are able to forecast which machines would fail in the near future so that maintenance team can proactively fix them
Image

With Artificial Intelligence (AI) and Machine Learning gaining prominence, Hadoop Data Lakes with huge repositories of data are going to play a central role in helping AI applications to learn faster and become intelligent. While mainframes, client-server and web applications automated most of the business processes, these AI applications would automate most of the decision making processes thus helping the organizations to become autonomous.