Is Data Quality Stuck in the Dark Ages?
IT decision-makers weren't confident in the quality of their data 20 years ago, and they still aren't hopeful in 2016. Far from it, according to a new survey.
- By Steve Swoyer
- June 8, 2016
Data quality in the enterprise hasn't advanced much in recent years. Survey after survey demonstrates that there's been little progress in improving the perceived or measurable quality of enterprise data.
IT decision-makers weren't confident in the quality of their data 20 years ago. They were no more sanguine a decade ago. In 2016, they still aren't all that hopeful, if the results of a new survey from 451 Research Inc. are any indication.
The State of Enterprise Data Quality: 2016 was written by 451 Research analysts Carl Lehmann, Krishna Roy, and Bob Winter and sponsored by data quality (DQ) management start-up Blazent.
The study found that only 40 percent of IT decision-makers have a high degree of confidence in their data quality management practices and the overall quality of their enterprise data. This aligns with other survey results and is backed up by scads of data points in this report.
For example, 37 percent of respondents say they're managing or integrating between 51 and 100 data sources; more than a quarter are managing or integrating between 101 and 200. That's a prescription for complex data integration and data quality issues -- the kind of complexity that can't be easily managed via traditional approaches, such as data validation scripts or scripted SQL transformations and data quality routines.
Most organizations are using data quality products or services. Nearly two-thirds (62 percent) have a formal master data management program and/or technology. Some respondents must be using more than one kind of solution: three-fifths (60.5 percent) have on-premises data quality management software, and more than half (53 percent) use a cloud data quality management service.
Slightly over 40 percent use applications -- custom-built or third-party -- to validate data, and 37.5 percent say they manually cleanse their data. A surprising number of organizations take a reactive approach to identifying and fixing data quality issues. Just under half (44.5 percent) of respondents say that their companies rely on employees to find and report data quality issues.
Almost one-tenth (8.5 percent) don't attempt to manage data quality at all.
The most common sources of poor data quality are also the most predictable. Respondents cited data-entry errors by employees (57.5 percent), problems introduced during data migration or conversion projects (47 percent), multiple or mixed entries by different users (44 percent), undetected or unmanaged changes to upstream systems (43.5 percent), and data-entry errors by customers (38 percent) as the top causes of data quality problems.
The 451 Research report also flagged a troubling phenomenon: organizations are increasingly apt to ascribe responsibility for poor data quality to IT. They're less likely to recognize the role played by the lines of business in creating and/or amplifying data quality problems.
"A disconnect exists between responsibility and accountability for data quality. While the IT department is mainly held responsible, the originators of the data -- either employees or cross-function teams performing data entry -- don't share in this responsibility," Lehmann, Roy, and Winter write. "IT departments have ... become burdened with the task of employing multiple technologies to compensate for the fact that responsibility for data quality is generally not assigned to those directly involved with its capture."
The State of Enterprise Data Quality: 2016 runs to 16 pages and is chock full of interesting findings. You can download it -- for free -- here.
About the Author
Stephen Swoyer is a technology writer with 20 years of experience. His writing has focused on business intelligence, data warehousing, and analytics for almost 15 years. Swoyer has an abiding interest in tech, but he’s particularly intrigued by the thorny people and process problems technology vendors never, ever want to talk about. You can contact him at evets@alwaysbedisrupting.com.