ICYMI: Your Business Users Don't Care About Big Data ... And they Don't Care Much for MDM Either

The number of companies deploying Big Data software, infrastructure, and analytics tools continues to increase. Wikibon, who began tracking total revenue related to "Big Data" back in 2013, released it's forecast this March that the Big Data market will hit $92.2B by 2026.

Given the opportunity there have also been no shortage of new vendors, all offering some flavor of the latest in Apache Open Source projects (Hadoop, Spark and more), NoSQL, deep learning, and in-memory processing. The landscape continues to expand leading to evaluation paralysis, or Franken-research IT projects that are destined for an unhappy ending. In this never-ending technology treadmill, you can hear business teams continuing to ask IT the grating, but very reasonable question, are we there yet?


But there has always been a nagging problem. For years companies, and their business users have struggled with the reliability of their data, an essential foundation before any analytics and insights can be drawn. Gaining this level of data reliability used to be purely an IT discipline called master data management (MDM). But MDM itself used to cost millions in hardware, on-premises software and consulting services. 

Data holds the key to streamlining processes, optimizing collaboration, improving customer satisfaction and meeting compliance reporting goals, but it also unlocks the formula for competitive advantage. Witness the rise of high paying job titles such as data scientists, and in the c-suite chief data officers (CDOs), are gaining in significance and popularity. 

If anything the 'big data-hype' has just begun. It's true that the term "Big Data" makes people think primarily of size (volume of the data), but it's the variety, velocity and veracity (quality) of the data that has IT tied up in knots. Rather than focusing on the size of the data, their business users are rightly focusing on the size of the 'big problems" that they need to solve. 

But even IT and data savvy users are struggling with getting value out of Big Data projects. According to a report titled "CIO and Big Data - what your IT team wants you to know" 55% of big data projects fail because of the lack of communication between the top managers who had the overall project vision and those who were in charge of actually implementing it. 


Gartner Research, to their credit, was telegraphing this in their July 2015 Enterprise Information Management (EIM) Hype cycle Report.

Big Data had started the descent into the "Trough of Disillusionment". The report summarized the trend as follows:

The hype related to big data continues to subside, as most end-user organizations have figured out that big data opportunities are just more opportunities and that big data is really just the data that an organization should care about.
Perhaps the biggest exception to the subsiding hype around big data is the closely related discipline of data science, which remains heavily hyped and appears in the priority matrix with both a transformational benefit, as well as a short two to five years to mainstream adoption. Lagging behind data science, however, are several key disciplines that will be essential components to the maturity — and thus long-term value — for data science. Analytic governance, data classification, and information valuation and infonomics all fall into this category.
— Gartner EIM Hype Cycle - July 2015

Another Gartner study on Hadoop adoption, 70% of respondents reported having only between 1 and 20 users accessing Hadoop, and 4% reported having no users at all.


If you glance 4 down the curve from Big Data, you'll actually see MDM leading the way into the Trough of Disillusionment. Gartner explains this scenario as follows:

A single version of the truth for master data domains, such as customer, product and asset, remains a central ambition for many organizations in the pursuit of strategic business goals. MDM is a strategic program that can take several years to achieve. The need for business case creation and program management, as well as the ability to deploy information governance and stewardship effectively, restricts MDM success to those organizations that master all these requirements. The technical challenge is to bring together the component technical capabilities of MDM (such as for data integration and data quality) in a fashion that supports real business requirements.
The overall market penetration is still relatively low, due to the complexity of implementation and other organizational and technical barriers, though there is deeper penetration for specific domains such as customer and product data. Although MDM as a whole has yet to reach the bottom of the Trough of Disillusionment, it has moved significantly toward it this year.
— Gartner EIM Hype Cycle - July 2015

For years business has funded standalone MDM initiatives, with very little to show for it. Clearly then, just putting MDM and Big Data together in its current form is the equivalent of combining two boat anchors together.


Being "data-driven" is a business term that has also been hyped to the max over the last few years with many experts encouraging companies to use analytics and visualization to gain the insights they need to succeed. Data mania is reaching a fever pitch with the BI and Analytics market forecast by some to be $18B+, with the enterprise applications market at $32B+. That's a lot of money being invested by business teams and IT, often in uncoordinated efforts. Unquestionably there are successes that can be tied to the investments, but the ROI of software has always been arbitrary. Without a closed loop to correlate back actions to the insights provided, it's a continued guessing game at best.  

The hype and noise shows no signs of abating so buyers of technology want to focus more on time-to-value and accuracy of outcomes. Business users in particular who experience the ease-of-use of products such as LinkedIn and Facebook as consumers, are asking tough questions of their IT counterparts: "Are we there yet?" questions are now morphing into "Why can't I get it and why does it take so long?", and "Why can't my application deliver recommended actions to me like LinkedIn, and why does it make me search for something it already should know?" This powerful voice of the user is putting pressure on IT, already faced with shrinking budgets and legacy systems to maintain, not to mention continuing issues with data quality and security. 


Much of the problems also stem from the fact that despite the volume of data being captured, data is still siloed, and unrelated. So the elusive 360 view of a customer or product has never been further from reality. Individual departments and organizations are still left to making business decisions based solely on their own data. Self service BI is a billion dollar market, but business users looking at poor quality data, without the complete picture across the enterprise is a ticking time bomb.

We are now on the cusp of a revolution in what it really means to be data-driven, and size of data doesn't matter, and traditional/legacy MDM isn't helping with that veracity piece.

Today a holistic “modern” data management strategy is needed, implemented through platforms in the cloud that can easily combine and clean master data across internal and external sources, correlate those profiles with transactions, big data or small, and maintain those relationships to provide a core repository from which predictive analytics can be obtained.

The enterprises that will thrive will be those that can best use those platforms to deliver and share the right information on demand, anytime, anywhere across the enterprise.