Leading Data Driven Software Delivery

Software delivery leaders often desire data driven approaches across programs or product areas.

As an initiative across teams, rather than an emerging team level practice, metrics often provide low value or become destructive.

Can such an initiative be led in a way that enables it to become a valuable practice?

Team level practice

I recall a new team discussing having a lot of work in progress. They decided to implement a limit, to stop working on too much at once. Later, they used this cumulative flow diagram to look back at the impact this had.

An uplift in throughput became visible once levels of work in progress dropped below the limit. Limiting work in progress became a cemented practice because data validated the improvement.

What emerged in this team is not a traditional way to use metrics. Leaders can use this same ethos in wider initiatives to become data driven.

Why Metrics Become Low Value or Destructive

Localised team metrics

Imagine the lifetime of a unit of work in a team. The below visual is overly simplified, but will demonstrate the issue with local metrics.

Once work emerges from the fuzzy front end, where it can spend years going from someone’s brainwave into a real thing, it goes onto a team backlog. Later, it goes into a release queue. We see work occurring for a small percentage of the elapsed time (Shown in blue).

In many teams, once a unit of work is ready for release, it’s accounted for in metrics. Such metrics are only interested in that small percentage of what is happening in the overall system of work.

Velocity is a common metric to suffer from this. It’s a sum of work occurring in this small part of the overall system. It’s focus is too limited to be useful beyond the team.

Traditional Uses of Metrics

Incentivisation and/or fear lead to metrics being gamed, deliberately or subconsciously. Velocity is a common metric to suffer from this too.

In some cases, metrics are exposed in reviews of performance, and comparisons drawn between teams. This is like comparing temperature without knowing if the gauge is Fahrenheit or Celsius. It tells us nothing about performance and destroys trust.

Even for leaders with the best of intentions, subtle interpretations of words and actions create the same effect.

The absence of systemic metrics

Consider everything that makes up a system of work. Process, structure, culture, policies, behaviours.

Systemic metrics draw out deeper insights. One of the easiest system metrics to gather will be release frequency. Consider the impact of an increase to the previous visual.

The illustration is not a view of the entire system, but helps show the benefit of widening our view to be more systemic.

We could increase how long is spent working on each unit, taking time to ensure quality and operability. This would make a team metric like velocity look worse, but improve the overall system.

A Lack of Balance

Another common mistake is using metrics focused on the same thing, usually speed, leading to speed over quality and value.

Quality problems come back to bite, and did we actually realise any benefits? We either have to address that through more work, or accept a lower value product.

Finding a Better Way

Here are a few suggestions to improve the chances of leading a successful data driven approach.

Do it at the right time

Product development may seem to have overtaken traditional approaches. OKRs are becoming a popular way of validating delivery of outcomes. However it’s often surface level. Scratch deeper and we find traditional approaches are still incumbent.

If we optimise a system too early, we risk doubling down on output, rather than outcomes. It takes a lot of work to shift towards delivering value. This may be a better place to invest initially, and get the mindset right.

Do it for the right reasons

Effective leaders approach metrics as a means of understanding and improving the system of work, to enable those within it, and increase its value to those who benefit from it.

Leaders demonstrate how to take a data driven approach at the system level, and help create improvements that people feel the benefits of. This will begin to build the trust required between leadership and teams.

Find the right balance

Henrik Kniberg once drew a diagram like the one below. A balanced approach would include metrics in each circle, to not drive one element at the expense of another.

If we created the focus on value, we have metrics that validate outcomes before we supplement with measures of quality and efficiency of flow. The best metrics live in the areas where the Venn diagram overlaps.

Perhaps overarching are human metrics. Do people feel happy, engaged, supported, trusted, safe, able to have fun at work?

Here’s some more example metrics I recall seeing drive change.

• How frequently do you deploy?

• How long is the gap between dev/test complete and code committed, and the work being deployed to production?

• How frequently do deployments result in an incident effecting live services?

• How long does it take to recover from an incident effecting live services?

• The time between an individual item of work being captured and initiated, and released to production. (It’s important to categorise work into types, and record this for each type).

• The time between an epic or feature being captured and initiated , and being completed.

• The total number of epics or features in progress

• The total number of different teams or groups work must pass through to be able to release value to the customer

• How many individual items were deployed in each release

• How many items are pending deployment

• As a % of our overall work, how much is invested in new functionality vs maintenance / improvement of existing products or platforms.

• Cumulative flow of work items across the program / product area

• Cumulative flow of epics/features across the program / product area

Co-Create a forum to look at systemic improvements

It’s important to share a sense of the system across teams and roles, and a goal to improve the system. To align, you need support from the highest level of leadership you can.

Aggregate across a program or product to create a feeling of a wider team. Aim for actions that improve the system and validate these with the data.

Make sure to involve people who have the authority to carry out actions. Other leaders will be needed to support these actions. Design actions with an owner and explicitly set out how the action will be supported by leadership.

Publish the agenda and the format so that people can see it’s about the system. Publish and outcomes that are achieved, so the value becomes clear.

A quarterly cadence may be sufficient to allow actions to be carried out and impact to be measured.

Co-Create a code of conduct

Make behaviours that enable the effective use of metrics explicit. Examples I recall are:

• Any metric is only the starting point for a conversation.

• Metrics measure the system of work, not the workers or the teams.

• Leadership use data to help understand the system of work, not manage teams.

• Team metrics belong to the team, they are in control of what they share about their team.

• We look at data trends, we do not set specific targets.

• We never compare metrics between teams.

Let teams figure out how to use data

Having demonstrated this at the system level, invite the use of data in teams, but allow teams to figure out the best way to do this.

Offer help where requested, but do not impose. Early adopters will jump on what they have seen. Invite the sharing of success.

Share good practice through lunch and learn sessions. Show how becoming more data driven can enable practices like probabilistic forecasting.

Look for moments where feedback helps identify where data might have helped, but give space to teams to choose how they go forward.

Conclusion

Leading a data driven approach starts with awareness. How could you contribute to either a negative or positive environment?

An awareness of the ways metrics might be harmful doesn’t make the topic a no go area, just one in which to tread very carefully, and work at the right level.