Modern Metadata Management
Tool automation and intelligence are replacing manual technical tasks with immediate business results.
- By Philip Russom
- January 11, 2019
Metadata management continues to be a powerful enabler for mission-critical data-driven business activities, including operations, analytics, collaboration, discovery, and compliance. Metadata is the golden thread that stitches together enterprise-wide landscapes, even those that are heavily distributed and heterogeneous, with hybrid mixes of on-premises and cloud systems.
For example, all data-driven actions go through metadata when users are browsing data, running a query, inserting data, creating transactions, updating records, modeling data, making virtual views, or refreshing reports. In addition, metadata enables multiple integration techniques that capture, move, and process data.
Metadata Challenges
Though mission critical, traditional approaches to metadata management confront two challenges.
First, metadata management is notoriously manual. It takes too long to develop new metadata, which makes it slow to start new projects or to introduce new sources. Metadata maintenance is tedious and error-prone, which slows down updates to existing solutions. Thus, metadata management needs to be modernized with better tool automation.
Second, metadata management is likewise notoriously siloed. Users regularly manage metadata on a per-tool or platform basis, with little or no attempt at centralizing metadata for use across the enterprise. This limits sharing or reusing metadata assets. Data governance is hamstrung by physically distributed metadata. It’s no wonder, then, that metadata needs to be modernized with better centralization.
Modern Metadata Management Addresses Challenges, Enables Innovation
The good news is that tools and practices for metadata management are evolving to leverage new technologies and to satisfy new business requirements. Why should your enterprise modernize metadata and its management, and why do it now?
- To address urgent challenges and opportunities with technology and practices that are available today
- To increase developer productivity via better tool automation and intelligence about data, which also help to enable agile and lean development methods
- To spend less time per project, which means completing more projects overall and reducing the time before a business can use a solution
- To enable modern metadata-driven business practices, such as self-service data access, discovery, prep, and visualization
- To enable modern metadata-driven techniques such as data virtualization and cataloging
- To consolidate metadata silos, which facilitates single customer views and enterprise data standards as well as data governance, stewardship, and curation
Key Advancements in Modern Metadata Management
Advancements are occurring eclectically in several areas.
More and better automation for metadata development and maintenance. Many metadata and data management tools now embed smart algorithms that can automate practical tasks by predicting (based on prior developer behavior) how metadata should describe such elements as data structures, mappings, and naming conventions. These algorithms rely on machine learning to create their predictive models and improve them over time automatically. Similarly, modern metadata scanning searches and crawls source and target systems to automatically extract metadata (from BI platforms, apps, databases, ETL jobs, and stored procedures, among others) during development, and it can discover and manage changes during maintenance. Traditional rule-driven approaches to automation are still with us, and these get better all the time.
Adjustments to user best practices. Technical users are applying agile and lean methods to the development and use of metadata and other semantics. They are centralizing metadata instead of keeping it in silos per tool and platform. Business users are adopting new self-service data practices that demand business-friendly metadata.
Metadata is the foundation for new semantics. When we say “metadata,” we usually mean the arcane and nerdy “technical metadata” that powers most data transactions and integration jobs. However, users are applying new tool functions that allow them to create additional semantic layers atop technical metadata. The layers are often for business users (who need friendly business metadata and business glossaries) or for emerging technical practices (e.g., data virtualization and data cataloging).
Metadata centralization. This is harder than it sounds. Consolidating diverse metadata into a single, central metadata repository demands a special, vendor-built metadata platform. Users may also be using their tools for data integration, quality, reporting, analytics, and visualization. Therefore, the central repository must be vendor agnostic while supporting all or most of the vendor and open-source tools, platforms, engines, and interfaces available to IT. Furthermore, users must agree to consolidate and share their metadata, though many are loath to do so.
Why suffer all that pain? A single, central, shared metadata repository has compelling benefits. It delivers or assists with the productive reuse of metadata assets, enterprise data standards, visibility into data across today’s multi-vendor IT landscapes, and the effective governance, stewardship, and curation of data strewn across multiple platforms and tools.
Metadata management on new platforms. For example, modern metadata management tools are now appearing as software-as-a-service (SaaS) platforms. The benefits of SaaS and cloud-based tools apply to metadata management -- namely, minimal tool set up, tool maintenance, and capital investment as well as reduced time to use and elastic scalability in production. Also consider that metadata tools must interface with data management functions on new platforms, such as SaaS operational apps, cloud-based systems and storage, Hadoop, and other open sources. Finally, given the multiplatform, hybrid data environments popular today, it sometimes makes sense to deploy a hybrid metadata repository that stores metadata on diverse platforms, although its interface makes distributed metadata look like a single source.
Final Thoughts
As with all IT disciplines, metadata management needs to keep up with the accelerating pace, broad enterprise integration, and opportunistic agility of the modern digital business. The catch is that traditional metadata management is notoriously manual, time-consuming, and siloed. Modern metadata management is a mix of new tool functionality and user best practice that require metadata via tool-based automation for metadata development, intelligence about data, and centralized metadata.
The result is unprecedented speed for metadata set up and development, comprehensive enterprise visibility for broader data use and governance, and scale for growing data volumes and source/target portfolios. Modern metadata management can also reduce the time it takes before a data-driven solution can be used (so the business is supported sooner) and enable new metadata-driven best practices (such as data self-service, virtualization, and cataloging).
For more information, replay the TDWI webinar “Modern Metadata Management” online here .
About the Author
Philip Russom is director of TDWI Research for data management and oversees many of TDWI’s research-oriented publications, services, and events. He is a well-known figure in data warehousing and business intelligence, having published over 600 research reports, magazine articles, opinion columns, speeches, Webinars, and more. Before joining TDWI in 2005, Russom was an industry analyst covering BI at Forrester Research and Giga Information Group. He also ran his own business as an independent industry analyst and BI consultant and was a contributing editor with leading IT magazines. Before that, Russom worked in technical and marketing positions for various database vendors. You can reach him at prussom@tdwi.org, @prussom on Twitter, and on LinkedIn at linkedin.com/in/philiprussom.