Chapter 22 Ten Key Metrics for Scrum

IN THIS CHAPTER

check Avoiding ineffective metrics

check Making the most of the available data

check Optimizing the value of scrum

With scrum, metrics can be powerful tools for planning, inspecting, adapting, and understanding progress over time. Rates of success or failure can let a scrum team know whether it needs to make positive changes or keep up its good work. Time and cost numbers can highlight the benefits of agile projects and provide support for an organization’s financial activities. Metrics that quantify people’s satisfaction can help a scrum team identify areas for improvement with customers and with the team itself.

Double-work agile is the practice of management expecting to see traditional status reports and meetings in addition to scrum artifacts, events, and appropriate metrics. This practice is one of the top pitfalls of scrum projects. Management is looking for one thing while teams are trying to do another. As a result, decisions are made based on the wrong information, teams burn out from doing double the work, and the benefits of scrum become minimized.

This chapter describes ten key metrics to help guide scrum project teams.

Sprint Goal Success Rates

One way to measure scrum project performance is with the rate of sprint success. The sprint may not need all the requirements and tasks in the sprint backlog to be complete to minimally realize the sprint goal. However, a successful sprint should have a working product increment that fulfills the sprint goal and meets the scrum team’s definition of done: developed, tested, integrated, and documented.

Throughout the project, the scrum team can track how frequently it succeeds in reaching the sprint goal and use success rates to see whether the team is maturing or needs to correct its course. Teams should always be stretching themselves, so 100 percent of the sprint backlog isn’t necessarily the goal. Making sure that individual requirements started get 100 percent done is important, but teams should always be stretching themselves, so success rates less than 100 percent sprint backlog completion should be considered to be opportunities to learn and improve. Scrum masters should always be looking for ways to reduce drag on the team so that the team can set and accomplish higher goals as it continues to accomplish more and more in each sprint. Sprint success rates are a useful launching point for inspection and adaptation.

Velocity isn’t a goal; it’s a postsprint fact. An increasing velocity and an increasing sprint goal completion rate are key indicators that a scrum team is continually improving efficiency.

You can find out more about setting sprint goals in Chapter 5 and reviewing them in Chapter 6.

Defects

To be truly agile, scrum teams need to implement agile practices such as test-driven development and continuous integration (see Chapter 13). Without quality practices such as these, scrum teams are ineffective at delivering quality as fast as the market demands due to the overhead of manual testing before each release and the number of defects introduced that automation could easily catch.

It’s unlikely that any scrum team will be able to accomplish perfection in these areas, so any project is likely to have some defects. Agile techniques combined with the scrum framework help development teams proactively minimize defects.

Tracking defect metrics can let the development team know how well it’s preventing issues and when to refine its processes. To track defects, it helps to look at the following numbers:

Defects during development: If the development team uses practices such as automated testing and continuous integration, it can track the number of defects at the build level in each sprint.

By understanding the number of build defects, the development team can know whether to adjust development processes and environmental factors.
User-acceptance testing defects: The development team can track the number of defects that the product owner finds when accepting requirements in each sprint. By tracking defects, the development team and the product owner can identify the need to refine processes for understanding requirements. The development team can also determine whether adjustments to automated testing tools are necessary.
Defect rates: In addition to seeing how many issues are discovered, it’s useful to look at the frequency per user story (stories delivered/defects found). Two defects out of 20 stories paints a different picture from 2 defects out of 2 stories. As a team increases its work effort, see whether the rate of defects goes up as well.
Release defects: The development team can track the number of defects that make it past the release to the marketplace.

By tracking release defects, the development team and the product owner can know whether changes in the user-acceptance testing process, automated testing, or the development process are necessary. Large numbers of defects at the release level can indicate bigger problems within the scrum team.

The number of defects and whether defects are increasing, decreasing, or staying the same are good metrics to spark discussions about project processes and development techniques at sprint retrospectives.

Time to Market

Time to market is the amount of time that a project takes to provide value by releasing working products and features to users. Organizations may perceive value in a couple of ways:

When a product directly generates income, its value is the money it can make.
When a product is for an organization’s internal use, its value is the employees’ ability to use the product and contains subjective factors based on what the product can do.

When measuring time to market, consider the following values:

Measure the time from the project start until you first show value to the customer.
Some scrum teams deploy new product features for use at the end of each sprint. For scrum teams with a release with every sprint, the time to market is simply the sprint length, measured in days.
Other scrum teams plan releases after multiple sprints and deploy product features in groups. For scrum teams that use longer release times, the time to market is the number of days between the start of development of a feature and the next release.

Time to market helps organizations recognize and quantify the ongoing value of scrum projects. Time to market is especially important for companies with revenue-generating products because it aids in budgeting throughout the year.

Return on Investment

Return on investment (ROI) is income generated by the product, less project costs — money in versus money out. On scrum projects, ROI is fundamentally different from ROI on traditional projects. Scrum projects have the potential to generate income with the first release (which can be as soon as the end of the first sprint) and can increase revenue with each new release.

To fully appreciate the difference between ROI on traditional and scrum projects, consider two scenarios with the same project costs that take the same amount of time to complete. We’re ignoring the additional documentation, meetings, and other expenses of a waterfall project and assuming that they could be kept at the scrum level to simplify the comparison. Both scenarios begin on January 1 and have the potential to generate $100,000 in income every month when all the requirements are finished.

Scenario A is waterfall and releases when all requirements are finished on June 30 of the same year. It enjoys monthly revenue of $100,000 per month for each month thereafter through the end of the year (six months, $600,000 revenue).

Scenario B begins releasing the highest-value and highest-risk features incrementally on January 31 after four one-week sprints, five months earlier than Project A. Monthly revenue is less as it builds up each month from the first release (that is, $50,000 in February, $60,000 in March, $70,000 in April, $80,000 in May, and $90,000 in June) until the entire project is complete.

This extra revenue in each of the five months from February through June gives the project $950,000 in revenue for the year, $350,000 more than scenario A.

Like time to market, ROI metrics are a great way for an organization to appreciate the ongoing value of a scrum project. ROI metrics help justify projects from the start because companies may fund projects based on ROI potential. Organizations may track ROI for individual projects as well as for the organization as a whole.

Total project duration and cost

To calculate ROI, first calculate duration and cost, which can themselves be effective inputs and metrics for scrum projects. Scrum projects should get done faster and be less costly than traditional projects.

A higher ROI as a result of decreasing project durations and costs should be a good indicator that scrum teams are increasing swarming, reducing thrashing, and improving overall efficiency.

New requests within ROI budgets

The ability to quickly generate higher ROI gives organizations that use scrum a unique way to fund additional product development. New product features may translate to higher product income.

Suppose that in the preceding example, the project team identified a new feature at the beginning of June that would take four one-week sprints to complete and would boost the product income from $100,000 a month to $120,000 a month, which results in increased revenue of $120,000 for the year in both scenarios. In scenario B, if the new feature had been identified earlier, ROI would have increased even more.

If a project is already generating income, it can make sense for an organization to roll that income back into new development and see higher revenue. Tracking ROI provides the intelligence required to make that decision.

Capital Redeployment

When the cost of future development is higher than the value of that future development, it’s time for the project to end.

The product owner prioritizes requirements in part by their ability to generate revenue. If only low-revenue requirements remain in the backlog, a project may need to end before the scrum team has used its entire budget. The organization may use the remaining budget from the old project to start a new, more valuable project. The practice of moving a budget from one project to another is called capital redeployment.

To determine a project’s end, you need the following metrics:

The value (V) of the remaining requirements in the product backlog
The actual cost (AC) for the work to complete the requirements in the product backlog
The opportunity cost (OC), or the value of having the scrum team work on a new project

When V < AC + OC (or AC + OC > V, as described in Chapter 5), the project can stop. The cost you sink into the project is more than the value you receive from continuing the project or starting the next project.

Capital redeployment allows an organization to spend efficiently on valuable product development and maximize the organization’s overall ROI.

Satisfaction Surveys

A scrum team’s highest priority is to satisfy the customer, both early and often, by delivering value. At the same time, the scrum team strives to motivate individual team members and promote sustainable development practices.

A scrum team can benefit from digging deeper into customer and team member experiences. One way to get measurable information about how well a scrum team is fulfilling its purpose is through satisfaction surveys.

Customer satisfaction surveys measure the customer’s experience with the project, the process, and the scrum team. The scrum team may want to use customer surveys multiple times during a project, including at the beginning to establish a benchmark for future comparisons. The scrum team can use customer survey results to examine processes, continue positive practices, and adjust behavior as necessary.

Team satisfaction surveys measure the scrum team members’ experience with the organization, the work environment, processes, other project team members, and their work. Everyone on the scrum team can take team surveys. As with the customer survey, the scrum team may choose to give team surveys throughout a project. Scrum team members can use team survey results to regularly fine-tune and adjust personal and team behaviors. The scrum team can also use results to address organizational issues. Customer survey results over time can provide a quantitative look at how the scrum team is maturing as a team.

You can put together informal paper surveys or use one of the many online survey tools. Some companies even have survey software available through their human resources department.

Team Member Turnover

Scrum projects tend to have higher team member morale. One way to quantify morale is to measure turnover. You can look at the following metrics:

Scrum team turnover: Low scrum team turnover can be one sign of a healthy team environment. High scrum team turnover (resulting from things such as burnout, ineffective product owners who force development team commitments, personality incompatibilities, or a scrum master who fails to remove impediments, making the development team look bad in sprint reviews) can indicate problems with the project, the organization, the work, individual scrum team members, or overall team dynamics.

Team members may be quitting, or managers may be stealing team members away to other projects. Either way, the result is increased thrashing that exposes organizational issues that should be addressed.
Company turnover: High company turnover, even if it doesn’t include the scrum team, can affect morale and effectiveness. High company turnover can be a sign of problems within the organization. As a company adopts scrum, it should see turnover decrease.

When the scrum team knows turnover metrics and understands the reasons behind those metrics, it may be able to take actions to maintain morale and improve the work environment.

Project Attrition

Organizations with project portfolios should look at the rate of projects being cut short. Capital redeployment shouldn’t be confused with thrashing teams between projects at the whim of senior managers. Tracking project durations against capital redeployment analyses may expose trends of ending projects prematurely or letting them run longer than needed.

From these trends, portfolio managers can look into reasons why projects are getting cut short. High attrition may indicate such issues as thrashing, planning, prioritization, impediments, and cross-functionality.

For more on how scrum improves portfolio management, see Chapter 13.

Skill Versatility

Strong scrum teams are typically more cross-functional than weaker scrum teams. By eliminating single points of failure on a scrum team, you increase its ability to move faster and produce higher quality. Tracking skill versatility allows scrum teams and functional managers to gauge growth of cross-functionality. Chapter 14 talks about incentivizing and encouraging skill development for scrum teams.

When starting, capture the existing skills and levels contained at each of the following organizational structures:

Per-person skills and levels
Per-team skills and levels
Per-organization skills and levels

Over time, as each person increases the quantity and level of skills, each team and the organization will increase its skill levels. The most important factor isn’t how many managers or directors you have by title in the organization that can deliver quality products and services to your customers. Your focus should be having team members who can contribute to the sprint goal each day without the risk of single points of failure.

Manager:Creator Ratio

Typically, the larger the organization is, the more likely it is to have a heavy middle layer of managers. Many organizations haven’t figured out how to function well without managers to handle personnel, training, and development issues. However, you need to strike the right balance of managers and individuals who produce product.

Every dollar spent on someone who manages organizational processes is a dollar not spent on a product creator.

Track your manager:creator ratio to help you identify bloat and ways to minimize the investment that you’re making in people who don’t create product.