From Data Management to Data Driven - Five Trends

Earlier this week I got an opportunity to present at DBTA (Database Trends and Applications) Data Summit in New York. This is a two-day event where various industry leaders join in and discuss the key industry trends and strategies to be more data-driven. The topics of discussion ranged from data warehouses, data lakes, IOT, to data science. I was asked to present in the category “New Approaches to Data Management” and track “Moving to a Modern Data Architecture.” I presented five key trends that are helping organizations to capitalize on their data and make better-informed data-driven decisions.

If we see the progression in data management, things have certainly become exciting since the start of this decade. In early to mid-nineties most of the focus was on processes with the rise of ERP, SCM, and later CRM applications. At the beginning of the 2000s, businesses wanted more visibility into their performance and processes, and we saw the introduction of many analytics applications based on OLAP databases and star-schema. However, with the advent of mobile, social, cloud things started to get interesting. Gartner referred to this as the nexus of forces and IDC as the 3rd platform. Organizations began to seek ways to manage the huge volumes of data generated. New roles like data scientists and Chief Data Officers(CDO) emerged. Today companies are looking for ways to go beyond vanilla data management and striving to become more data-driven. Here are the top five trends I discussed:


First is the use of Machine Learning to create a reliable data foundation – In big data world bringing data together from multiple internal and external sources can be a challenge. We are moving from manual or rules-based matching to matching done via machine learning. However, there is still much distrust on machine learning as a black box "Voodoo". So the initial phase of machine learning is to provide transparency of the actual rules that drive the merge and then it is up to the user to evaluate the discovered rule and persist it in the system.


Next significant advance is a graph, helping us understand relationships across all real-life data entities. The graph aids to establish and navigate many-to-many relationships across people, products, and locations. Uncovering relationships using graph technology helps you solve your problems like householding in retail or finding most influential people in your key accounts. 


Trend number three is intelligent systems that guide users and provide intelligent recommendations, based on data and user behavior. The recommendations can tell you how to improve your data quality; it can suggest new relationships in your network (like LinkedIn or Facebook) and offer next-best-actions such as right time and channel to connect with a customer or what to offer them next.


Sharing data across all systems and functional groups helps realize the full value of data collected. Marketing, sales, services, and support should all leverage the same reliable, consolidated data. They should be able to collaborate and contribute to enriching the data. They should be able to vote on data quality or the business impact of any data entity. New data-driven applications must support this.


Charter of a CDO is not only about data governance and data integration and management. Increasingly, companies are asking CDOs to turn this data into new revenue streams. With cloud-based data-as-a-service, companies can easily monetize their data and become data brokers. Businesses can now collaborate with each other to create common data resources and easily share or exchange data.

I had some interesting follow-up discussions with attendees. The idea of building reliable data foundation and discovering relationships across all data entities was the most brought up topic since many companies are struggling with these issues. Attendees from more mature organizations discussed various machine learning algorithms and data monetization opportunities. Surely, these trends were on the top of the mind for many and are bringing us to the age of modern data management where data is considered a strategic asset and not just an exhaust from application systems.