TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Leveraging Data Virtualization for Digital Transformation

Enterprises are modernizing their IT infrastructure as never before. Data virtualization should be a key part of their digital transformations.

• By Ravi Shankar
• May 26, 2020

In troubled times like the current coronavirus pandemic, which is causing unprecedented reduction in budgets and staff, CIOs look to invest in technologies that advance their digital transformation projects while lowering costs. Data integration and management is critical to a successful digital transformation, but some aging, rigid technologies are driving enterprises to look for modern, flexible, efficient, and cost-effective alternatives.

For Further Reading: Data Virtualization Offers a Fresh Approach to Data Lakes Future-Proofing Your Data Storage The Broad New Powers of Modern Data Catalogs

Traditionally, legacy data integration technologies (such as ETL) extracted the data from source systems, moved it into an intermediary system where developers transformed the data format for compatibility with the target system, and then loaded the transformed data into the destination system. Even though ETL is still employed widely, use of this workhorse of data warehousing architecture is waning due to:

• Its rigid programmatic/scripting approach
• The need for a bevy of engineers to create and tweak the scripts whenever the data and sources change
• The high cost of storing the intermediate data when applying transformations

Any changes to the source systems could break the brittle ETL scripts, requiring time to fix them and interrupting business operations.

A Better Approach: Data Virtualization

Data virtualization enables an agile, real-time approach to integrating data without having to physically replicate it. It leaves the data within its source systems and instead creates an abstraction layer on top of them, thereby separating the sources from the data consumers. Such abstraction shields analysts and business users from any changes to the underlying systems and enables IT to independently modernize the underlying systems without interrupting business operations.

Analytics and reporting tools request the data from the data virtualization layer, which tracks the movement of data as IT retires older systems and migrates to newer systems. The virtualization layer takes care of finding the data wherever it is -- the data that was on premises a week ago might be in the cloud today -- and securely and rapidly delivers that data to users in real time, requiring a fraction of the resources and cost. Data virtualization's no- or low-code approach requires fewer developers and reduces storage costs because it does not store the data separately.

Use Cases

Data virtualization is used in projects such as cloud modernization, data science, data discovery, 360-degree views of customers, products, and assets, data warehouse offloading, and integrating data in motion such as IoT data with data at rest.

As a horizontal technology, companies in almost every industry are successfully using data virtualization for modern use cases such as:

Cloud modernization: As companies move away from tying up expensive capital in data centers, they are not just lifting the on-premises applications and shifting them to the cloud. Instead, they are rearchitecting them to take advantage of specific cloud capabilities. As firms retire on-premises applications and implement their cloud equivalents, they use data virtualization as an abstraction layer so a business can continue its operations without worrying about where the data is.

Data science: As organizations mine their data for additional intelligence, data scientists need access to all enterprise data with the flexibility to apply different data models to ensure the correct questions are asked and answered. With its logical data model capability, data virtualization lets data scientists apply various data models in a safe, sandbox environment with all of the available data before deploying them in production.

Data discovery: The enterprise data layer, enabled by data virtualization, knows all the information inside and outside of an organization; data virtualization also catalogs it, providing details such as data location, format, and associations as well as who is accessing it, how often, and for what purpose. Instead of having to search for data in different repositories, business users can query the single data virtualization layer to learn where their data resides, what relationships it might have with other data, and what business definitions are associated with it.

A Foundation for Digital Transformation

Data virtualization is to digital transformation what mechanization was to the Industrial Revolution. They both aim to improve labor productivity and output, reduce costs, and increase revenue and profit.

As enterprises modernize their IT infrastructure with technologies that help lower their ongoing costs, they ensure the availability and integrity of data and enable business operations to continue uninterrupted -- which has never been more important. Data virtualization helps a CIO accomplish both goals: to enable digital transformation while lowering costs.