Metrics for assessing the effectiveness of remote teams and beyond

Metrics for assessing the effectiveness of remote teams and beyond

In the good old days, we all worked in an office and team performance was assessed through constant verbal contact. In those days, the involvement of the team was evaluated not so much by digital indicators, but by the time spent by all development participants in the same room.

In 2020, we, like everyone else, switched to distancing. It is logical that after some time the question arose in the management – how effective are we there? And the second, which follows from the first: what do we, as management, do to manage this efficiency?

For answers, business indicators alone are obviously not enough, they do not answer the question of how effectively we are growing in IT. We needed production metrics taking into account the methodologies and processes used in the organization. After all, we want to understand – is the removal effective or not?

We collect and filter metrics

The task was extremely clear: it is necessary to form a list of metrics and develop a tool for working with them.

At first we decided not to limit ourselves. As a result, we got the following set of metrics:

  • Time to market (TTM) – time of delivery to the market.

  • Time to lead (TTL) – production time.

  • Committed vs Delivered (CD) – meeting deadlines.

  • Velocity – speed.

  • Delivery – the number of deliveries to the promotion.

  • Rollback – the number of rollbacks from the promotion.

  • Duration – finding the task in the status.

  • Quantity – in completed tasks.

  • Quality indicator – work with incidents

But when we calculated them, we realized that we simply could not use all the metrics.

Most metrics are not representative due to the wide variety of stacks and platforms we build our products on. In addition, the absolute values ​​of some metrics essentially tell us nothing. For example, Team A’s Velocity is 10 sp (side of points), and Team B’s Velocity is 20 sp. Is it possible to say that team B performed twice as well as team A? Obviously not, because each team has its own 1 sp and they cannot be compared.

It is not possible to work with such indicators only at the level of the organization, so we moved to the adaptation stage.

No. 1. First, we decided on the basis of calculation. Surprisingly, it turned out to be a sprint: it is fixed in terms of time and resources and is repeated regularly (there is a rhythm).

No. 2. Absolute values ​​in the metrics were then avoided due to the problems described above. As a result, some metrics began to be calculated according to the formula of relative deviation from the level calculated empirically or by team or platform.

No. 3. In the end, the obtained metrics were accelerated due to the key, from the point of view, characteristics that determine production efficiency:

Metrics that give ambiguous signals were also excluded. For example, Rollback does not always mean that something went wrong, sometimes it is used to disable an experiment on a feature.

In the end, we settled on the following set:

No. 1. CD – The ratio of the number of tasks completed per sprint to those planned for the sprint.

No. 2. Velocity — the deviation of the sum of completed story points per sprint from the median level of completed sprints in depth per year by team.

No. 3. Delivery (final name “Implementation”) – the deviation of the number of deliveries of changes to the prom per sprint from the median level for completed sprints in depth per year by product/system.

No. 4. Quality indicator. It is calculated according to a matrix with parties: compliance with SLA, reduction of the number of open incidents, where:

  • compliance with SLA is the ratio of non-overdue incidents to the total number for the reporting period in terms of criticality;

  • reducing the number of open incidents is keeping the delta between the number of open incidents at the beginning and end of the reporting period less than or equal to “-1”.

Visualizing metrics: tools

After running most of our metrics in Excel, we thought about where we should keep the metrics so that it is clear.

Not very convenient

First of all, we looked at data sources and thought about the possibilities of building dashboards from them. At that time (already closer to the end of 2020), we had Atlassian Jira and Service manager (SM).

Of course, all kinds of plugins for Jira allow you to build certain reports, but they cannot get data from SM. Plus, we understood that in the future we may need new sources of information to improve the quality of performance evaluation.

BI tools were chosen, especially since in the organization they have long been used in business units to calculate various indicators – support and development expertise was already present. Power BI, at that time, was more profitable in terms of licenses than Tableau, so they settled on it.

Below is how our metrics look in graphs. For example, the first graph on the top left shows how successfully teams manage to meet the deadlines for the tasks scheduled in the sprint. The graph on the right above shows the dynamics of changes in command speed (acceleration/deceleration). And the bottom right shows how successfully teams manage product/system quality.

So how do you use metrics now?

The first thing they proved useful was that introducing a fixed set of metrics allowed us to bring all stakeholders to a common denominator in terms of understanding performance.

You have probably noticed that when there is a discussion about the effectiveness of teams, everyone understands it differently: technicians believe that management processes are ineffective, managers criticize the competences of experts, and customers lack speed. Introducing a fixed set of metrics allowed us to bring all stakeholders to a common denominator in terms of understanding performance. Someone may say that the metrics are few/many, they are incorrect and do not show anything, but despite this, a constructive dialogue focused on specific indicators began to take shape.

Then it’s easier – they built the evaluation of indicators into their work processes with efficiencyand a single set of metrics allowed us to look at different levels: Directorate → Direction → Team.

Currently, we hold regular monthly meetings where we discuss work with efficiency at the department level, analyze current trends, hypotheses of their occurrence/absence, and further activities. Also, involved heads of directorates conduct weekly statuses at their level with IT leaders in their areas, which allows timely tactical decisions to be made.

So what about the efficiency I talked about at the beginning? With the help of metrics, we tested this hypothesis and found that distancing on the majority of teams did not have a negative effect in terms of performance.

If you look at the dynamics from year to year, you can see that most indicators are increasing, although there were also small declines and flats. Despite the increase in speed indicators, the quality remains at a consistently high level.

Conclusion: about the banal and non-obvious

It would seem like trivial things, but it is not so easy to build an objective assessment of the work. And the issue is not a lack of theory, tools or holivars about a set of metrics. The problem is in the minds of people when they perceive what they see in the indicators: there are many of them, then there are few of them, then the method is far from ideal, and in general the picture is not objective.

Most perceive metrics as a threat, despite the fact that the methodology is the same for everyone. And the fact that it gives significant results. For example, the metrics showed us a difference in the frequency of implementations between teams with and without a dedicated support worker.

Of course, we were the first in the organization to engage in performance evaluation. Attempts to us crashed against the wall of lack of understanding of goals or further steps, lack of motivation. But as you might have guessed from the background, in this long-term project we were able to implement a basic set of metrics at the department level (2000+ employees) into the workflow. And this is our main victory.

In the comments, it will be interesting to learn about your experience with efficiency in IT and what tools have helped you succeed.

Related posts