Emerging Technologies and Methods: Highlights from TDWI’s Forthcoming Best Practices Report
By Fern Halper, TDWI Research Director for Advanced Analytics
Philip Russom, Dave Stodder, and I are in the process of putting together our most recent Best Practices Report: Emerging Technologies for Business Intelligence, Analytics, and Data Warehousing. TDWI refers to new and exciting technologies, vendor tools, team structures, development methods, user best practices, and new sources of big data as emerging technologies and methods (ETMs). For example, tools for data visualization are the most hotly adopted ETM in BI in recent years. In addition to visualization, most of these tools also support other emerging techniques, namely data exploration and discovery, data preparation, analytics, and storytelling. ETMs for analytics involve advanced techniques, including predictive analytics, stream mining, and text analytics, that are progressively applied to emerging data sources, namely social media data, machine data, cloud-generated data, and the Internet of things. A number of emerging data platforms have entered data warehouse (DW) environments, including Hadoop, MapReduce, columnar database management systems (DBMSs), and real-time platforms for event and stream data. The most influential emerging methods are based on agile development or collaborative team structures (e.g., competency centers).
ETMs assist with competitiveness, decisions, business change, and innovation. According to this report’s survey, the leading general benefits of ETMs (in survey order) are improvements in competitiveness, decision making, responses to business change, business performance, and innovation. These benefits are being realized today, because two-thirds of organizations surveyed are already using ETMs and 79 percent consider ETMs an opportunity.
Despite the benefits, a number of barriers stand in the way of adopting ETMs. Many people feel held back by their IT team’s lack of skills, staffing, infrastructure, and buy-in. Others have trouble seeing the business value of leading-edge technologies. Some work in risk-averse organizations that lack a culture of innovation for either IT or the business. Nonetheless, both business and technical respondents report working through these issues to adopt ETMs.
Some ETMs are more like tool features that are emerging in a variety of tool types. The most pervasive is self-service functionality, which is found in tools for reporting, analytics, data prep, and so on. The point is to give certain classes of users tools that are simple, intuitive, and integrated with common data sources, requiring little-to-no setup or assistance from IT. Fifty-four percent of users surveyed consider themselves successful with IT-free self-service.
Open source software (OSS) has become an important wellspring for innovation. Hadoop (whether from Apache or a software vendor), tools associated with it (MapReduce, Spark, Hive, HBase), and other similar data platforms (NoSQL databases) have emerged from their Internet-company roots and are now being adopted by mainstream enterprises. These ETMs are examples of how influential OSS has become for innovative products. Interfaces to these platforms’ data are also common emerging features in vendor-supplied tools for data integration, data prep, data exploration, reporting, and analytics.
DW environments presently include multiple ETMs, many based on open source. All these OSS-based or OSS-inspired ETMs are now entering DW environments, along with slightly older ETMs like DW appliances, analytics DBMSs, and columnar DBMSs. This emergence has driven a trend toward multi-platform DW environments, where the core relational warehouse is joined by a long list of standalone data platforms, most of them ETMs.
Posted by Fern Halper, Ph.D. on July 30, 2015