From Data Management to Data-driven - Five Trends

Earlier this week, I got an opportunity to present at DBTA (Database Trends and Applications) Data Summit in New York. This is a two-day event where various industry leaders join in to discuss the key industry trends and strategies about how to be more data-driven. The topics of discussion ranged from data warehouses, data lakes, IOT, to data science. I was asked to present in the category, “New Approaches to Data Management,” and track “Moving to a Modern Data Architecture.” I presented five key trends that are helping organizations capitalize on their data and make better-informed, data-driven decisions.

If we see the progression in data management, things have certainly become exciting since the start of this decade. In early to mid 90s, most of the focus was on processes with the rise of ERP, SCM and later CRM applications. At the beginning of the 2000s, businesses wanted more visibility into their performance and processes, and we saw the introduction of many analytics applications based on OLAP databases and star-schema. However, with the advent of mobile, social and cloud, things started to get interesting. Gartner referred to this as the "Nexus of forces," and IDC as the "Third Platform." Organizations began to seek ways to manage the huge volumes of data generated. New roles, like Data Scientist and Chief Data Officer (CDO) emerged. Today, companies are looking for ways to go beyond vanilla data management to become more data-driven. Below are the top five trends I presented:


The first trend involves the use of machine learning to create a reliable data foundation. In the big data world, bringing data together from multiple internal and external sources can be a challenge. We are moving from manual or rules-based matching to matching done via machine learning. However, there is still much distrust on machine learning as a black box "Voodoo." So the initial phase of machine learning is to provide transparency of the actual rules that drive the merge, and then leave it up to the user to evaluate the discovered rule and persist it in the system.


The next significant advancement is the graph, used to help us understand relationships across all real-life data entities. The graph aids to establish and navigate many-to-many relationships among people, products, places and organizations. Uncovering relationships using graph technology helps you solve your problems like householding in retail, or finding the most influential people in your key accounts. 


Trend number three involves intelligent systems that guide users and provide intelligent recommendations, based on data and user behavior. Intelligent recommendations can tell you how to improve your data quality, suggest new relationships in your network (like LinkedIn or Facebook) and offer next-best-actions--suggesting what would be the right time and channel to connect with a customer, or which promotion should be offered next.


Sharing data across all systems and functional groups helps realize the full value of the data collected. Marketing, sales, services and support should all leverage the same reliable, consolidated data. They should be able to collaborate and contribute towards enriching the data. They should also be able to vote on data quality or the business impact of any data entity. New data-driven applications must support this.


The charter of a CDO does not only involve data governance, data integration and management. Increasingly, companies are asking CDOs to turn this data into new revenue streams. With cloud-based, Data as a Service, companies can easily monetize their data and become data brokers. Businesses can now collaborate with each other to create common data resources, and easily share or exchange data.

I had some interesting follow-up discussions with attendees. The idea of building a reliable data foundation, and discovering relationships across all data entities was the most brought up topic, because many companies are struggling with these issues. Attendees from more mature organizations talked about various machine learning algorithms and data monetization opportunities. Surely, these trends were on the top of the mind for many, and are bringing us towards the age of modern data management, where data is considered a strategic asset--not just an exhaust from application systems.