Harnessing the Power of Diverse Data with Fern Halper
Fern Halper, Ph.D., TDWI vice president and senior research director for advanced analytics, discusses the findings from her new Best Practices Report on harnessing the power of diverse data for business growth.
- By Upside Staff
- January 19, 2024
In this “Speaking of Data” podcast, TDWI’s Fern Halper explores the findings from her latest TDWI Best Practices Report on harnessing the power of diverse data for business growth. Halper is vice president and senior research director for advanced analytics at TDWI. [Editor’s note: Speaker quotations have been edited for length and clarity.]
Halper began with a quick overview of what constitutes diverse data.
“Diverse data is pretty much just what it sounds like -- data in formats other than structured data,” she said. “This includes unstructured and semistructured data (for example, XML and JSON) and data from different sources (such as social media and IoT devices).”
She explained that this diverse data is becoming more important as companies seek any way to compete better in their markets. “For example, a company can use the unstructured or semistructured data from their call center interactions to better predict when a customer might churn.” This diverse data can also be used to uncover hidden insights, make better predictions, and otherwise better respond to what’s happening on the ground, she added.
“This reflects what we saw in the survey, which was that the primary driver for using diverse data, cited by 53% of respondents, was to better understand customers. This was followed by use cases related to operational efficiency, which were cited by 43%.”
The conversation then turned to the subject of how organizations were managing all this data.
“In our survey, no one approach stood out as the leader,” Halper said. “About 20% of respondents said they were managing their diverse data in a combination of cloud data warehouses and data lakes -- the structured data in the warehouse and the other types in the data lake. An additional 20% said their platforms were a hybrid of cloud and on premises.” Lakehouses, data fabrics, data virtualization, and semantic layers were also mentioned as solutions respondents were using, she added.
Survey respondents were also fairly evenly divided on their assessments of how well they were managing their data, Halper said, explaining that about half of respondents reported they were doing either at least a relatively good job and the rest reporting that they could do a much better job.
The reason for this, as Halper sees it, is that managing diverse data is a relatively new mission for most organizations. For example, about 40% of survey respondents said they had either no skills or only limited skills to analyze the data, she said. This raises concerns around data quality and governance, as well as a lack of tools to help resolve them.
She went on to say that there is an expectation that innovations such as generative AI can provide assistance with certain aspects of using this diverse data. “For example, with the example of the call center notes, you can train an LLM to do sentiment analysis to assist in customer retention.”
“When it comes to the key takeaways,” Halper said, “in addition to the fact that making use of diverse data is really important for your success, I think the other is that the market is not nearly as far along as one would hope.” She noted, though, that she had seen evidence that organizations are at least starting to collect this data, even if they haven’t yet begun to use it.
[Editor’s notes: You can replay this podcast on demand. For more information, download the TDWI Best Practices Report: Harnessing the Power of Diverse Data for Business Growth.]