Data Warehouses and In-Memory Technologies: Myths and Reality
In-memory technology can open doors to a new world of analytics possibilities -- but make sure your information value chain is strong enough to support the investment.
- By Shilpa Yelamaneni
- January 5, 2016
Analytical solutions have transformed organizations across every industry. From aspirational companies that are starting their analytics journey to mature enterprises that provide competitive advantage leveraging analytics, investments in this space are ever growing. Surely, "The price of light is less than the cost of darkness" -- investments in analytics technology, data, and people are critical in transforming businesses. At the same time, to secure these investments, a well-thought out business value case accompanied by a comprehensive data and analytics strategy is required to achieve the business value.
Many software and service providers promise quick results and assure that they will fulfill ambitious long-term goals. One such promise is in-memory technology to analyze big data, improved business performance, decreased time to value of analytics projects, and reduced complexity in layered data architectures. In-memory technology can open doors to a number of business value cases to foster analytics transformation enabled through big data, real-time reporting, etc.
However, in most other business value cases, the technology is not an imperative but only an expensive choice. It is neither a nostrum nor a fundamental requirement for analytics solutions. More importantly, these investments demand a stronger information value chain to leverage the faster processing and optimization capabilities, and a clear enterprise data warehouse architecture and management process to sustain the benefits.
I drew few interesting considerations from my personal experiences. One noteworthy example is a data warehouse implementation on in-memory database technology to provide enterprise reporting and analytics. The initial scope was to provide descriptive reporting and ad hoc analysis for a single business unit followed by an enterprise analytics road map. The initial investment reflected a commendable and ambitious vision of the organization for analytics. Visibility and easier access to information provided indispensable business insights leading to key strategic and operational decisions. However, the solution lacked positioning for long-term success and sustainability. Benefits achieved from the project were limited to faster query response times relative to a traditional data warehouse solution. It is questionable if the value realized justified the high costs of in-memory technology and associated software licenses.
Adding to the costs, the scalability of in-memory analytics platform is expensive and challenging. The situation impaired the enterprise analytics road map progress as the cost of new projects became extremely high.
As a result, the size of the data acted as a major stumbling block for newer business analytics (BA) projects. The organization had been highly selective in choosing analytics and BA projects to effectively leverage the initial investments in data warehouse platform. Business users were left to manually collect and combine distributed data and forced to create offline spread marts and departmental data islands. These local databases and spread marts caused data quality challenges and impaired key business decisions. The existing platform lacked positioning as an enterprise data warehouse and instead became one of the data mart solutions. The organization never obtained the true value of enterprise data warehouse capabilities with integrated information. As the data warehouse was incomplete, business users were unable to leverage the power of online BI tools.
Among many other factors, the expensive in-memory technology in this case hindered their progress on the analytical maturity curve. It was a "chicken and egg" puzzle of investment and value generation. Without seeing a good ROI, it's understandable that the organization hesitates to further scale their analytics platform and data warehouse.
This article provides two key takeaways for those who want to undertake an analytics transformation with an in-memory technology platform. First, having a 360-degree view of all costs will help avoid misconstruing the benefits case. Second, and perhaps more important, poor positioning of technologies can result in greater hurdles in achieving organizational goals.
- Let's drill down into three business needs:
- To beware of hidden costs behind the benefits case
- To evaluate the complete information value chain
- Positioning itself for successfully adopting the platform and realizing its benefits
Hidden Costs Behind the Benefits Case
Organizations should be aware of all costs before embracing in-memory technology for data warehouse solutions. The initial cost savings from simplified architecture, decreased development efforts, and reduced support and maintenance costs are largely a myth.
Increased costs: Initial setup costs are typically high (especially for the infrastructure and licenses) and carry associated hidden costs, such as the need for skilled resources for support, to handle additional governance activities, and to manage platform complexities. For instance, a simple database-related issue might require three workers to solve; a data modeler, in-memory database administrator, and system administrator. If these skills are segregated into separate teams, analyzing a simple performance issue will translate into a project, in turn affecting agility and responsiveness of the IT function. Another possible side effect: increased governance processes.
Increased complexity: Typical in-memory technology configuration has limited capacity for high-performance memory storage, as supposed to the capacity on disk is fairly large and easily scalable. The recommended memory-to-disk ratio must be followed to ensure proper system performance. To utilize the scarce in-memory resources cost-effectively, the data warehouse architecture should incorporate hot, warm, and cold data areas. Adding new data to the environment and regular data loads require memory and disk usage analysis. Moreover, data retention and deletion strategies should be implemented up front.
Evolving product: Product improvements and revisions are frequent in this space. Though the vendors are quite helpful in guiding customers, customers pay the price for continuous improvements and constant upgrades by allocating additional resources to keep up with the platform updates. Forced to be "current and correct" with the solution will cost IT resources and cause business disruptions.
Organizational changes: Unlike traditional analytics platforms, in-memory technology solutions require proper organizational changes to sustain the application and the technology platform. Gray areas of accountability and change management exist between application management and database management. Typically these are divided between two teams. To support the application holistically, business analytics or data warehouse application management will require overlapping knowledge of in-memory technology and database management. The overlapping nature of these skills will require constant and close collaboration to support the application. The magnitude of change management activities increases significantly, causing constant business disruptions.
Strengthen the Information Value Chain
A strong commitment is required from executives to realize the vision behind analytics investments. The broader the goals, the more complex the value-realization equation.
To generate business value, strong components of the entire information value chain will have to be in place: scalable platform, quality data, intuitive BI tools, trained users, and high-value projects covering descriptive, predictive, or prescriptive business cases. Investing in expensive in-memory technology will not reduce the importance of other components. Instead, it will increase the significance of these other components to realize the ROI on costly investments.
The entire information value chain is only as strong as its weakest link. Adding in-memory technology doesn't make the other links stronger. To realize the true business value, good data, skilled resources, and competent business users, right projects, and effective operations are critical.
As shown on the graphic, each component plays an equally important role in generating business benefits
In addition to strengthening the overall information value chain to maximize returns, a technology strategy should be adjusted to properly position new analytics solutions in the overall enterprise analytics framework. This is a critical step in effectively operationalizing new solutions and sustaining the benefits for a long time.
Positioning for Successful Adoption and Benefits Realization
In-memory technologies constitute only a piece of an enterprise data warehouse solution. Whether it is HANA or Hadoop, the new technologies typically augment a data warehouse solution. For organizations that truly want to tap into the power of in-memory technologies for point solutions / use cases and achieve maximum return on their investments (instead of continuously injecting more money as their database grows), consider a mixed technology solution. Enterprises should also consider developing a logical and distributed data warehouse solution that blends:
- Enterprise data-modeling-layer capabilities to provide an enterprise data model and unified data warehouse (considering multiple platforms)
- An architecture that isolates highly used aggregated analysis (separate transactional reporting models)
- Data lake repositories (leveraging smart data access concepts) for staging all data from distinct and distributed data sources
The power of in-memory technology is untapped but it is only a tool, not the ultimate answer to data and analytics challenges that companies face.
A Final Word
This article had provided guidance on better positioning in-memory technology investments for data warehouses. A proper and informed situation analysis will guard against hype cycles and peer pressure. Costs and business disruptions caused by introducing new technology demand a comprehensive benefits case including the hidden operational and organizational costs.
In addition, organizations should be prepared to position any technology within an overall data and analytics strategy framework. They should evaluate the change management necessary to strengthen the value chain and sustain new solutions. This change management process cannot be undermined because it involves more resources, takes more time, and forces reevaluation of architecture and operational aspects that are critical for long-term success.