TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Trends in Data Management for 2018

Here are the data management trends -- some old, some new -- to watch in 2018

• By Philip Russom
• January 2, 2018

Enterprise data management continues to grow and progress, driven by a number of trends, most of which have been ongoing for years now. Older trends include data warehouse modernization, Hadoop adoption, cloud adoption, and the evolution of traditional tools and practices for data modeling, metadata management, hubs, cloud usage, and self-service data access.

In addition, newer trends have arrived to shake things up, namely the data lake, IoT data, and the use of machine learning in the automation of data management development. We have yet to feel the full impact of the upcoming GDPR, which will soon force many organizations worldwide to modernize their practices in data governance, stewardship and curation.

Let’s take a quick look at the most prominent trends we can expect this year.

Data Warehouse Modernization: This trend exploded on the scene in 2015 and is still wending its way through the warehouse community. There are so many new technologies (Hadoop, NoSQL, hardware advances), user techniques (agile, data lake designs), and business practices (self-service data access) to adopt -- and so many data warehouses that are behind the times! -- that modernization will remain a priority for a few more years.

Hadoop Adoption: Hadoop is a natural fit for mature, multiplatform data warehouses because it provides cheap storage for big data, processing power for advanced analytics algorithms, and the ability to handle non-relational data. Furthermore, Hadoop strengthens the weakest components of data warehouse architecture, namely data staging and analytics sandboxing. A TDWI survey found Hadoop in a fifth of mature data warehouse environments and TDWI expects that adoption to keep rising.

The Data Lake: Most data lakes are deployed atop Hadoop, though a few are on relational databases, cloud storage, or a hybrid mix of these. For Hadoop users, the design pattern and collection of best practices known as the data lake provides much-needed methodology and control. For others, the data lake is a raw data repository that organizes and extends complex, hybrid, and multiplatform data architectures in data warehousing, multichannel marketing, digital supply chain, and other enterprise programs.

Machine Learning Algorithms in Data Management Tools: Many tasks performed by data management developers are recurring and time-consuming, such as mapping sources to targets, categorizing data, remediating data anomalies, and creating metadata to represent new data from new sources. Advancements in machine learning -- first applied in general analytics -- are now being incorporated into data management development tools to provide much needed automation, which in turn leads to greater developer productivity.

Data from the Internet of Things (IoT): Most discussions focus on the sources of IoT or the end-user applications with analytics. Everyone’s ignoring the piece in the middle: the capture, management, and leverage of IoT data. Without that middle mile, the first and last miles will not come together to enable monitoring business processes (as seen in IoT data) and reacting accordingly.

Modern Data Hubs: The data hub is no longer a mere “Roach Motel,” where data checks in but rarely comes back out. The modern data hub keeps data moving -- with solid quality, organization, and governance -- by including advanced tools for orchestration, data curation, and dataset publish-and-subscribe.