Modernizing Your Data Team and Its Best Practices
As you modernize your tools and platforms for data management to keep pace with evolving business requirements, take time to modernize your data team.
- By Philip Russom
- October 15, 2020
Being “modern” is important for both business and IT.
Change is rampant in many areas of our lives due to the pandemic, economic fluctuations, evolving political leadership, and new directions for social norms. To adapt to change and to leverage change for business advantage, many enterprises are modernizing their business model, operations, processes, customer relations, and strategic plans. As a consequence, every aspect of IT is also modernizing, from IT infrastructure and operational applications to leading-edge data management and analytics.
In a parallel trend, many new state-of-the-art IT tools, platforms, and best practices have arrived in recent years, and they are fortuitously available for adoption by evolving businesses that seek not merely to survive but to adapt, compete, and thrive.
To support new business directions, data management technology needs to be modern.
A digital enterprise will accelerate the speed and efficiency of operational and analytic tasks so it operates in real time or close to it. IT systems and business processes cannot achieve these business goals without real-time information. This further intensifies the urgency of modern data management.
As another example, the list of new sources of valuable data increases almost daily, as businesses modernize their operational applications by adopting new ones built in the software-as-a-service (SaaS) architecture. Similarly, firms involved with the Internet of Things (IoT) regularly incorporate more data from new sensors and machinery. In turn, these new sources of operational data can provide data for new forms of analytics, and a truly modern digital enterprise is eager to get greater value from data via advanced analytics.
Finally, to handle the diversity of new data sources and advanced analytics, many firms are deploying more data platforms -- both old and new types, both on premises and in the cloud.
Put together all these new sources, targets, and use cases and you can see that capturing and leveraging today’s data is not possible for the digital enterprise without a modern and comprehensive data management infrastructure plus a modern data team that has the skills to create today’s solutions.
Modernize your data team and best practices as you modernize data management.
The modernization of data management can take many forms, but it regularly involves upgrades and replacements of platforms and tools. In fact, modernization via “replatforming” -- where one data platform is replaced or complemented by another -- has garnered considerable attention in the IT press in recent years. However, handling today’s data successfully also requires improvements to the teams that build and maintain solutions for data handling. In fact, TDWI sees many organizations actively modernizing teams that are wholly or partially responsible for data-driven solutions such as teams for data management, data warehousing, data integration, reporting, analytics, and operational data.
A modern data team has many characteristics and relationships. Given the growing amount of data work in enterprises, it is common for midsize and large organizations to have multiple data teams. Each team may report to central IT or to a business unit (e.g., finance, marketing, operations). In addition, many data teams have “dotted line” relationships with multiple departments across an organizational chart. Because of these diverse situations, data teams can be overseen by a variety of managers, including a CDO, IT director, BI director, data architect, or line-of-business manager.
Today’s modern data team must keep using traditional tools and following established best practices while adopting new ones. Consequently, the data team as a whole ends up using a long and broad list of tools and methodologies, ranging from traditional batches, ETL, table dumps, and bulk loads to modern data pipelining, orchestration, ELT, virtualization, and real-time streaming data. In turn, the data team must include people trained in the use of all these tool types, plus new development practices that are agile, lean, self-service, and scrum-based.
TDWI recommends that each team member be assigned specialties and trained in those, but also be cross-trained in other areas so that team members can pinch hit for each other. Cross-training also gives the team manager flexible options for assembling sub-teams for individual projects.
Old and new data management practices must coexist. Older data management and processing practices are not going away because ETL, batch processing, and intense pre-processing are still relevant for high-value use cases, especially in reporting and data warehousing. However, the use cases of a modern digital enterprise require new approaches to data and integration, such as early data ingestion, pipelining, orchestration, real-time integration, and on-the-fly data transformations, especially for advanced analytics and data lakes. Old and new must coexist, and users need to modernize integration tools, teams, and practices to support old and new with equal competency.
New data management skills must be learned. As firms modernize data management and its teams, they struggle to fill the “skills gap.” A recent TDWI survey shows that success comes from a mixture of hiring new team members (73 percent), training existing ones (58 percent), and engaging consultants (45 percent). (See Figure 15 in the 2019 TDWI Best Practices Report: Cloud Data Management.) Teams for data, analytics, and integration typically consist of mostly internal full-time employees (80 percent), augmented by external staff such as consultants (20 percent). (See page 10 in the 2019 TDWI Teams, Skills, and Budgets Report available to TDWI Members.)
The catch with the skills gap is that new gaps open as new technologies and use cases arrive, currently driven by analytics, real-time data, cloud, and hybrid architectures. For example, the average enterprise will soon manage more data on cloud platforms than on premises, and they will manage more unstructured than structured data. Therefore, teams need to plan ahead for the skills, head count, and tools needed to make these innovations successful.
The members of a modern data team continue to evolve. Teams for data warehousing, analytics, and integration are dominated by data engineers (20 percent), architects (18 percent), scientists (13 percent), analysts (13 percent), and managers (13 percent). (See Figure 19 in the 2020 TDWI Best Practices Report: Data Management for Advanced Analytics.) Among these, it is the data engineers who most often create integration solutions but with design guidance from architects and other team members. Thus, keeping a healthy stable of data engineers and architects is a critical success factor for a modern data team.
New data-driven development methods must be adopted. As with everything else in IT, development methodologies have evolved, as used by teams for data, analytics, and integration. Today, the preferences are (in survey priority order) agile (42 percent), waterfall (28 percent), scrum (22 percent), sprint (20 percent), self-service (19 percent), rapid prototyping (16 percent), and lean (5 percent). (See page 20 in the 2019 TDWI Teams, Skills, and Budgets Report.) Teams that are modernizing typically undergo training in agile and similar preferred development methods for data and integration. However, success with agile methods depends on adopting tools that have high ease of use and ample automation functionality.
Modernizing a data team may involve reorganizing its structure and style. Agile methods have taken over but are often coordinated with related methods, namely scrum, sprint, and rapid prototyping. This group of methods has led to the “scrum team,” which is today the most common team structure, slightly more popular than centers of excellence or competency centers. Even so, TDWI is seeing a brisk adoption of DataOps, which is a data-oriented version of DevOps. In a related trend, TDWI is seeing architects rising as team leaders. All these team structure issues should be considered when modernizing a team for data and integration.
For Further Learning
Read the TDWI reports this article draws from: