CHAPTER 5

Outcome Measures

Outcome measures show the impact of a program on an organizational goal or need. Ideally, the impact is the isolated impact of learning using the Phillips definition of Level 4, as discussed in chapter 4, to show the difference made by training in achieving the organizational goal or meeting the organizational need. In this chapter we explore these quantitative measures in greater detail and suggest a qualitative approach that represents a middle ground between the Phillips and Kirkpatrick methodologies. We also consider the application rate as a leading indicator when isolated impact is not achievable for outcome measures and the special case of competencies.

The Importance of Outcome Measures

Outcome measures are a critical type of measure. As we discussed in chapter 1, some programs are strategic in nature, meaning they are directly aligned to the goals of the organization. These strategic programs should have outcomes. Other programs are more tactical or operational in nature. They are still very important but will likely not have outcome measures.

The expectations are different for each type of program. A tactical or operational program is expected to be efficient and effective. A strategic program is expected to be efficient and effective as well, but it should also generate an outcome—a contribution toward achieving an organizational goal. The right audience could have taken the training, they could have liked it, learned from it, and applied it, but all this great training still might not have helped the organization achieve its goals or meet its needs. For strategic programs, L&D is not funded or staffed simply to be efficient and effective. The business invests in L&D because it believes that the right learning program can help it succeed if it is properly aligned, designed, delivered, and reinforced. This success is measured in outcomes.

CEOs are particularly focused on outcomes. Recall from chapter 1 that research by Jack Phillips clearly showed CEOs are most interested in seeing Level 4 impact, which is the preferred outcome measure. In fact, almost every CEO who responded wanted to see impact. They want to know what difference L&D is making, specifically how much it is contributing to their goals and meeting key needs.

It is for this reason that Level 4 isolated impact has been elevated to its own category of measure in TDRp. The L&D profession must focus more on this measure and share it with senior leaders. Yes, strategic programs should be run efficiently and effectively (Levels 1–3), but most importantly, these programs must deliver impact and value (Levels 4 and 5). They must contribute to outcomes.

Most Common Outcome Measures

The name of an outcome measure for learning is simply “Impact of learning on ________,” where the business or HR measure to be improved by learning fills in the blank. For example, if content was designed to improve sales, then the outcome measure would be “impact of learning on sales.” If it is designed to increase employee engagement, then the outcome measure is “impact of learning on employee engagement.”

Since learning content can be designed to support any business or HR goal, the list of potential learning outcome measures is long. The most common, however, are included in Figure 5-1.

Figure 5-1. Most Common Learning Outcome Measures

Outcome measures are specific to each organization and will exist whenever learning can support an organizational goal or need. The natural starting point in identifying outcome measures is the CEO’s goals, followed by the head of HR’s goals, followed by goals of other senior leaders. If learning can support these goals, then there should be an outcome measure for it.

The unit of measure for the outcome measure is the same as the unit of measure for the organizational outcome. A sales goal is usually expressed as a percent increase, so the unit of measure for the learning outcome measure (impact of learning on sales) would be percent as well (for example, a 4 percent increase in sales due to learning). Patient and customer satisfaction goals are usually expressed in terms of percentage points of improvement, such as a 2-point increase in customer satisfaction. The outcome measure will be in the same unit of measure (points). Adopt whatever your organization is using as the unit of measure for the organizational goal or outcome for the learning outcome.

Types of Outcome Measures

There are two types of outcome measures: quantitative and qualitative.

Quantitative Outcome Measures

The first type of outcome measure is quantitative—the Phillips Level 4 isolated impact discussed in detail in the last chapter. This type is expressed in numbers and has both a percentage contribution from training and the resulting impact on the business measure. For example, if a sales training program contributed 20 percent (which might come from using a control group or from using the participant estimation method) toward achieving a 10 percent increase in sales, the resulting impact on sales would be 20% × 10% = 2% higher sales due just to training. By formula:

Impact of Learning on Sales:

= Percentage contribution from learning × Actual sales results

= 20% × 10%

= 2% higher sales due to learning.

Our focus will be on the learning outcome measure, but know that an outcome measure for the organization also exists. It is called the organization (or business) outcome measure. Examples include sales, productivity, quality, number of injuries, cost, patient satisfaction, customer satisfaction, employee engagement, and leadership.

Qualitative Outcome Measures

The second type of outcome measure is qualitative. Here, the concept of isolation is still used, but adjectives are used instead of percentages for the isolated contribution, and there is no resulting isolated impact. We recommend using a three-point scale to make this approach manageable. The most common scale is low/medium/high, but any adjectives will work. High is interpreted to mean learning was responsible for contributing at least 50 percent of the result, medium implies a contribution of 20–50 percent, and low implies a contribution of less than 20 percent. If learning were thought to have been the major driver and itself responsible for achieving most of the goal, it would have made a high contribution. If learning was not responsible for achieving most of the goal, but still played a very important role, it would have made a medium contribution. If learning played a relatively minor but still important role, then it would have made a low contribution. Alternatively, you could use adjectives such as important, very important, and significant.

Qualitative outcome measures are particularly useful in the planning stage when L&D is working with a goal owner to agree on the role and expected impact of learning. Drawing on the Kirkpatrick model, L&D and the business goal owner would agree upfront on the expectations for learning (high, medium, or low). Results would be measured against those expectations using the same qualitative outcome measure discussed during planning. Notice that the absolute meaning of the adjective (such as high) is not as important as the agreed-upon meaning before the program is launched. In other words, qualitative outcome measures will work as long as L&D and the goal owner or stakeholder agree on expectations and what their chosen adjective means to them. Last, from a planning point of view, the choice of high/medium/low will directly determine the level of effort by both parties and the completion date. (High impact typically can only be achieved through deployment in the first quarter of the fiscal year.) We’ll discuss this further in chapter 11 on creating plans.

Warning

Levels 0, 1, and 2 are not good leading indicators for Level 4 impact. The number of participants (Level 0), the reaction score (Level 1), and the learning results (Level 2) are not predictive of Level 4 impact. It is true that if no one takes the course there will be no impact. It is also true that if participants did not find the content relevant or did not learn, there will likely be no impact. Unfortunately, however, the reverse does not hold. There are numerous instances where the right audience completed the program, liked it, and learned it, yet there was no application or impact. Senior leaders know this, which is why they want to see application and especially impact. Simply sharing the number of participants and Level 1 scores will never take the place of Level 3 and 4 measures, and should never be used to demonstrate the impact or value of learning.

Application Rate as a Leading Indicator

It may not always be possible to use either a quantitative or qualitative outcome measure in planning. Perhaps the goal owner is not comfortable with either the concept of isolated impact or the methodology to calculate it (quantitative) or characterize it (qualitative). In other cases, the L&D staff is not comfortable having the isolated impact discussion with the goal owner. Rather than abandoning the concept of agreeing on expectations for learning, we recommend using the application rate as a leading indicator of outcome.

In learning, the Level 3 application rate is the best leading indicator for Level 4 isolated impact because we know that if there is no application, there is no impact (outcome). We also know that usually when there is application, there is at least some impact. So, while it is possible to have application without impact, application is in most cases an excellent next-best measure for reaching agreement with the goal owner on expectations for learning. (There is a detailed discussion in chapter 11 about reaching agreement with the goal owner.) Agreement on a Level 3 application rate will facilitate a discussion on roles and responsibilities, timing, and level of effort—just like a discussion to agree on Level 4 impact.

Flexibility in Creating Outcome Measures

Sometimes a situation will arise where a senior leader has goals or needs that are lower level or more transactional in nature. These goals may even involve measures that in the TDRp framework are efficiency or effectiveness measures. For example, a senior leader may have a goal to reduce the duration of a program by 25 percent, which would typically be an efficiency measure. Or, the leader may have a goal to improve the participant reaction by 10 points for a very important program, such as onboarding. Typically, this would be an effectiveness measure.

Our advice is to follow the direction of the leader and adapt TDRp to the needs of the situation. If a leader says reducing duration or improving reaction is one of their top business goals, then treat them as outcomes. Don’t tell them that according to a book you just read their goals are not really outcomes. If it is an important goal or outcome to them, it is an important outcome for you.

This flexibility will be important when we start creating reports showing outcome measures. In this case, we would show “reduction in program duration” and “improvement in participant reaction” as “headline” measures, taking the place of an outcome measure of success, rather than as efficiency or effectiveness measures.

Recommendation

Outcome measures for learning should be calculated and shared with senior leaders (CEO, CFO, governing boards, goal owners, or business unit leaders) whenever learning can support an important organizational goal or need. This is the measure that CEOs most want to see. Sharing it with them shows that L&D is aligned to their goals and is contributing to the success of the organization. No other efficiency or effectiveness (Levels 1–3) measure has this same power. Typically, an L&D organization will only have only a handful of outcome measures, so the task isn’t as daunting as it may appear. And the future of L&D in many organizations may well be determined by whether the senior leaders believe learning is contributing to the organization’s bottom line success.

The Special Case of Competency Measures

Competency measures present a special case in our discussion of outcome measures. In most learning programs, improved competency in itself is not the intended goal or outcome. Rather, based on the needs analysis conducted during the planning stage, L&D identifies gaps in employee competencies, which must be closed if the goal (such as increased sales) is to be achieved. Here, competencies are best viewed as a means to an end rather than the end. Consequently, the outcome measure focuses on the organizational goal or need and not on the competencies.

There are special cases, however, where the goal truly is to improve competency. In these cases, a higher competency level is the end goal. One such case is basic skills training. No one would argue that new employees typically need basic skills before they can start work. The basic skills are not directly aligned to any company goals, but many organizational goals, such as revenue and quality, would nevertheless be adversely affected over time if new employees did not acquire the necessary skills. This is especially true in retail industries like restaurants, department stores, and banking. It is also very true for knowledge-based businesses such as consultancies, where the product provided to clients is knowledge.

For these types of learning programs, then, what should be used for outcome measures? We suggest choosing measures that capture the satisfaction of the senior leaders with the training. This may be considered a “next best measure” for the unmeasurable Level 4 isolated impact. For example, L&D might ask the leaders of new employees who have just received basic skills training about their proficiency on the job. Do they have the skills necessary to complete their first tasks and be a successful member of the team? Of course, they will continue to learn on the job, but the question is how well prepared they are when they arrive at their first duty station. Leaders will have an opinion on this because they are observing the new employees every day.

Another example of this situation is basic training and even the first training after basic training in the military. In this case, the training command provides the training. What is an appropriate outcome measure? Since there is no good way to isolate the impact of basic training on winning wars, the training command could survey the supervisors who receive their trainees and ask about their preparation. The desired outcome, then, is a well-prepared force. This incorporates the voice of the customer, who in this case is the receiving unit.

Note

In the December 2018 standard for human capital reporting, the ISO identified the workforce competency rate as a metric that should be reported internally by large organizations. The purpose of the metric is to understand the competency level for the workforce in total as well as the competency levels for categories of employees. The formula is:

Workforce Competency Rate: Average of employee competency ratings.

Employee competency ratings may be based on an objective assessment (using a test instrument) or a subjective assessment by the employee or supervisor. A five-point scale is recommended, as shown in Figure 5-2.

The most appropriate competency level for each sales representative could be found by asking the employee or the supervisor to select the level that best describes the representative’s level of competency. Once a level has been determined for each representative in the department, they are averaged to determine the workforce competency rate for that category of sales representatives. This competency rate could then be combined with rates from other groups using weighted averages by group size to determine the competency rate of the entire organization.

Figure 5-2. Example of a Five-Point Competency Scale for Sales Representatives

Conclusion

This chapter concludes our discussion of the three types of measures. Throughout this chapter and chapters 3 and 4, we have described more than 100 efficiency measures, more than 10 effectiveness measures, and a suite of outcome measures. Clearly, the L&D profession does not lack measures, and there are more in the CTR measures library if those in this chapter do not meet your needs. These chapters have provided standard names, definitions, and formulas, which are based on the work of the measurement pioneers in our field (Kirkpatrick and Phillips) for the effectiveness measures and ATD, which provides benchmarks for many of the efficiency measures. It is our hope that this work will contribute to the growing standardization of measures, which is important for a mature profession.

Next, we use these measures and the TDRp framework to create a measurement strategy.