TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Out of the Loop: Why Agile Alienates Data Teams and What to Do about It

Five ways to reduce risk and find the opportunities in leaner development processes.

• By Cass Brewer
• March 14, 2016

In breaking news, Oxford linguists have discovered three previously unknown meanings for the English word "agile." Scrawled notes in the margins of newly unearthed scrolls reveal the Latin root agilis was also used to indicate "unplanned change," "I don't need to document that," and "I'm in meetings through next Thursday."

…

Of course, this is satire. It's funny only because, although agile is powerful, it's hard to get right. In the absence of enough knowledge, skill, discipline, and especially organizational will, agile can quickly devolve into an excuse for slapdash development without proper communication or control.

In "old school" waterfall environments, uncontrolled development risk was often mitigated by change management requirements, but agile's smaller changes with more customer involvement don't necessarily warrant a formal review and approval process. Most of the time, a product owner, development team, and project or scrum manager can make good choices about development priorities and risks, but sometimes valuable players (notably data managers) are left out of the loop.

However you feel about this scenario -- and as you're reading it on Upside, chances are you don't love it -- it's a common reality of agile. You can try to turn the ship, push for better processes, policies, and practices, but that's likely to be a longer-term effort. In the meantime, you can accept that change will happen and then act accordingly to reduce risk, control costs, and seize new opportunities.

1. Design data systems and processes to anticipate change.

As a rule, people are bad at predicting what they want, but good at explaining how something isn't what they need. Thus, even in the absence of strategic and environmental shifts, end users will come back after you've launched a data system with new ideas and use case "discoveries." This is an opportunity, not (just) an annoyance.

Data teams should manage data warehouses, other centralized data sets, and related software as rolling projects, pursuing frequent, iterative releases that respond to emerging user needs. Technical design should also flavor flexible data modeling, architecture, and infrastructure. (We'll explore these in future articles.)

2. Use developer documentation to keep data teams in the loop.

Changing documentation is usually a snap compared to changing habits, cultural norms, and perceptions. This is particularly true in agile companies, which tend to undervalue (and consequently underestimate the power of) both operational and technical documentation. Although most agile software teams forgo formal change management policies and procedures, they often have release procedures, standard workflows, task management checklists, or scrum how-tos that serve the same purpose. Working with team leaders to add even a single data management/BI touchpoint in these documents can make a world of difference in bridging the data/development gap. It can also act as a fulcrum -- repeated tactical interactions will, over time, forge new cultural norms.

3. Build relationships with project and development managers.

Agile developers prefer lean teams. Particularly in projects with an initial "big" release, data managers might be thanked and shown the door afterward, leaving subsequent iterative changes unchecked. Developing friendly personal relationships with project managers and development leads can help keep data management (and analysis) top of mind, spur informal communication, and ease developers' fears that data managers will add dead weight to decision processes.

4. Pursue new use cases and usage scenarios.

After users get their hands on a system, they'll often get big ideas about other things it "should" do. These ideas do not always trickle through reporting stacks or across teams, which can leave users feeling frustrated. When enough pressure builds, it can manifest as angry, urgent projects. This pattern can be averted. Just as agile software developers are expected to respond to customer needs, data teams should proactively seek opportunities to delight their systems' users. There are many ways to uncover emerging use cases -- for example, automated analysis, user surveys, and interviews. Whichever way you go about it, though, the goal should be to get ahead of change requests and help drive changes that make sense for the entire data-project portfolio.

5. Worry less about the "one truth" than user benefits.

The notion of one centralized data "truth," as reflected in a master data set or data warehouse is a beautiful, but often inflexible and impractical, idea. At the outset, it might not be possible to achieve. If you do pursue it, negotiating across technical and business teams can cause harmful delays and friction. Finally, even if it you can find consensus, those hard-won definitions and models might be dubious (a lose-lose compromise, rather than actual agreement) and fragile.

By focusing on delivering the right data for specific use cases first, you'll deliver faster, simpler data solutions that make allies of your users. This usually doesn't preclude a "one truth" MDM or data warehousing effort down the road.

In the short term, steps such as loosely coupling smaller master "reference" data sets with consuming systems, cataloging entities with universal (surrogate) keys, and letting transactional systems handle most data relationships and transformations as needed might largely prevent you from getting trapped in semantic arguments, inflexible data models, and the need to constantly test whether developers have gone and built "rogue" data repositories to circumvent a rigidly defined "truth."

Conclusion

Most of these steps boil down to recognizing that change management has too often been used as a means of change prevention, to the detriment of business. Agile seeks to avert this problem, among others. Data teams that recognize this causal relationship can take steps to forge stronger relationships and processes in its wake.