In agile development, we strive to maximize efficient output while maintaining a sustainable pace. It’s the team’s responsibility to constantly assess their capacity, take on work up to, but not exceeding their capacity, and consistently invest in continuous improvement to increase capacity over time.
Unfortunately, a lot of leaders outside of the team like to look at velocity as the core metric of productivity. Velocity is simply the number of estimated story points, hours, or tasks completed in a given cycle. It makes sense on the surface that teams should get better over time, so their velocity should increase. This presents a few major problems:
- It’s trivial to game the system. Want to increase velocity? Simply score stories higher.
- You can’t compare velocity between teams since each team estimates in their own way
- Velocity is simply a measure of “current speed”, and has nothing to do with capacity. Can the team deliver more? Is the team working to increase efficiency? Velocity doesn’t tell us this.
So in looking at team performance, we really want to answer three key questions:
- Is the team working at or near current capacity?
- Is the team continuously improving and increasing capacity over time?
- Can the team maintain the current pace with high levels of quality?
This gives us a true understanding of, not only the team’s performance, but their growth and maturity over time. Let’s dig into each of these questions and look at some better ways to measure the team’s productivity and growth.
Is the team working at or near current capacity?
So to answer the first question, I want to look at these metrics:
- Velocity Variance – This tells us about the stability and consistency of the team. If the velocity varies greatly sprint over sprint, there is some chaotic force that is hurting their ability to estimate and deliver on their commitments. One thing to keep in mind is that velocity can shift if someone is out sick or on vacation, or if someone joins/leaves the team.
- Number of Stories Completed vs. Stories Slipped/Kicked to Backlog – This tells us if the team is able to deliver on their commitments. The team should rarely have to push out work or change priorities mid-sprint. If work starts slipping, the team beyond their capacity.
The idea here is that if the velocity is stable and work isn’t slipping, they’re not yet at capacity, so we should be able to take on more work. If we get to a point where velocity maxes out and work starts slipping, we’ve reached current capacity. Looking at these two metrics allows us to determine the true capacity of the team.
Is the team continuously improving and increasing capacity over time?
Next, we want to see that the team is investing in their growth and their velocity is increasing over time. We want to avoid the team artificially pushing this metric by increasing estimates over time, so we should look at:
- Velocity Growth Over Time
- Continuous Improvement
The second metric is tricky to measure. The idea here is the velocity is growing because the team is working to improve their efficiency and actual throughput. So some ways to measure this are:
- Stories completed that are tagged with “Retro”, “Tech Debt”, “Innovation”, “Automation” or some such initiative that’s tied to continuous improvement.
- Average Time in Status. This measures how long stories sit in various states in the process. The idea is that if these numbers decrease over time, it indicates the team is working to clear bottlenecks. For example, is the team getting better at getting work through code review, QA, UAT, and deployment? You should see the average time in status decrease.
- Kanban metrics like lead-time, cycle time, and the cumulative flow diagram. These metrics measure the flow of work, and you should see that the team is consistently working to surface and clear bottlenecks.
These metrics give you a complete picture of not only the team’s performance, but the team’s investment in getting better over time. There’s one key piece remaining..
Can the team maintain the current pace with high levels of quality?
The thing that ties it all together is Quality. We don’t want to sacrifice quality to move faster, so along with these productivity metrics, we should measure:
- Bug Ratio (# of bugs vs. # of stories completed)
- Number of bugs identified over time, by severity
- Num of bugs open vs. resolved
- Num of defects found in QA vs. bugs found in production
There are many ways to measure quality, but the important thing is that we should see high levels of quality maintained as we increase velocity. Pushing velocity too hard will cause a decrease in quality, so we have to monitor all these metrics to get a complete picture of the team’s performance, maturity, and capabilities.
This collection of KPIs will give the CTO and engineering leads a clear picture of the team’s performance and quality, which can be reasonably compared across teams.
Do you need help?
I’ve been leading agile development teams for over 15 years. If you need help with your team’s performance or agile maturity, I can coach your team and leaders to achieve your highest potential. Contact me at eric@fullcycleproduct.com and let’s connect!