CHAPTER 3

Efficiency Measures

Efficiency measures provide answers to the question, “How much?” How much activity has occurred (number of participants, courses, hours, and so forth), how much money has been spent, how much effort has been expended, how much of our target audience are we reaching (reach), how much time has been taken (cycle time), how much staff have we allocated, and how much of our resources (for example, facilities, faculty) are we using (utilization)? Efficiency measures are also called volume or activity measures, inputs, or Level 0 measures.

If the name of the measure starts with “number of,” it is almost always an efficiency measure. In L&D, efficiency measures far outnumber effectiveness and outcome measures. In fact, of the 186 L&D measures in the CTR measures library, 144 are efficiency measures. In this chapter we describe 107 of the most commonly used efficiency measures for formal and informal learning, including all those that are benchmarked by ATD as well as those recommended by ISO. Our list of efficiency measures also includes other measures commonly used by organizations with which we work; measures shared by organizations in presentations, articles, award submissions, and workshops; and measures contained in WLP Scorecard: Why Learning Matters (Rivera 2007).

Most efficiency measures convey a message only when compared to a plan or benchmark. For example, you might consider your program to be efficient if it reaches a very high percentage of the target audience or if it comes very close to budget. Many efficiency measures also are a combination of two individual efficiency measures. For example, cost per learner divides cost by the number of learners and is readily benchmarked. Some efficiency measures, though, like a utilization rate, convey a sense of efficiency without comparison to an external benchmark.

While many of the efficiency measures can be applied broadly across the different types of learning, they vary considerably when measuring formal versus informal learning. Formal learning has distinct events, such as courses or classes. Informal learning is driven by the learner, not only when they choose to learn but also how. Consequently, activity and reach measures will be quite different between these two types of learning. We explore those differences throughout this chapter.

The categories and subcategories of efficiency measures are shown in Figure 3-1, but don’t be overwhelmed by the large number of them. In chapter 7, we provide guidance on how to select the right measures.

Figure 3-1. Categories of Efficiency Measures

Additionally, seven of the eight ISO 30414:2018 standard metrics are found among these more than 100 efficiency measures; they are categorized in Table 3-1.

Table 3-1. Categorization of ISO Efficiency Metrics

ISO Metric Name	TDRp Measure Name	TDRp Category
1. Total development and training cost	Direct expenditure	Cost
2. Average formalized training hours per employee	Learning hours used per employee	Hours
3. Percentage of employees who participate in training compared with the total number of employees	Percentage of employees reached by formal training	Reach
4. Percentage of leaders who have participated in training	Subcategory of measure 3	Reach
5. Percentage of employees who participate in formalized training in different categories	Subcategory of measure 3	Reach
6. Percentage of employees who have completed training in compliance and ethics	Subcategory of measure 3	Reach
7. Percentage of leaders who have participated in leadership training	Subcategory of measure 3	Reach

Five of the seven measures shown are “reach” measures and will be discussed in the section on reach later in this chapter. The ISO measures for cost and hours will be addressed in their respective TDRp sections. The eighth ISO measure is workforce competency, which will be addressed in the outcome measures chapter. ISO-recommended measures are denoted by ISO 30414:2018.

As we list the types of measures in each category, we will denote measures benchmarked annually by ATD in their State of the Industry report using ATD BM. Benchmarking can provide valuable information about averages and the range of values for measures.

State of the Industry Report

By Maria Ho, Manager, ATD Research

ATD publishes its annual industry benchmarking report, State of the Industry, in December of each year. The report allows organizations to compare their talent development activities to those of other organizations, including winners of ATD’s BEST Award (which recognizes organizations that use talent development to achieve enterprise-wide results).

The report analyzes 11 efficiency and expenditure indicators, including direct learning expenditure per employee, learning hours used per employee, and employees per talent development staff member. It also provides benchmarks for the percentage of the learning portfolio dedicated to 12 common content areas, the delivery methods for learning hours used and available, and organizations’ use of 10 types of on-the-job learning. Benchmarking groups include all participating organizations, BEST winners, and industry and company size groupings.

According to the latest State of the Industry, in 2018, at the average organization, each employee used 34 hours of learning, and the average learning spend per employee was $1,299. ATD does note in the report that readers should consider their own organization’s unique circumstances, which may be very different from those of the average participating organization. Therefore, readers should not aim to replicate the averages in the report. Instead, ATD suggests that they use them to better understand their own activities and those of their peers.

The State of the Industry is available at TD.org/research.

Formal Learning Measures

We address formal learning measures first—activity, reach, effort, cycle time, cost, staff, and headcount—followed by informal learning measures.

Activity Measures

Activity measures for formal learning are the natural starting point because they are the most often reported efficiency measures. And of all the activity measures, the most common is the number of participants. We also examine measures for courses, classes, hours, and engagement.

Participants

Number of participants comes in two varieties: Unique Participants and Total Participants. For example, if Tonya takes two courses during the period, she would count as one unique participant with two instances of total participation. In other words, Total Participants equals two while Unique Participants equals one. To make life interesting there will always be multiple participants, so we will be interested in both total Unique Participants and total Total Participants. Since “total Total” is awkward, this measure is just called Total Participants. The formal definitions are:

Total Unique Participants: The unduplicated count of all participants. In other words, the number of unique (by name) participants.

Total Participants: All instances of participation, including multiple instances of participation by the same individual.

An LMS will report both total unique participants and total participants. If an LMS is not available and participant registration is recorded in a spreadsheet, the duplications will have to be removed manually. Table 3-2 shows the calculation of total unique participants and total participants.

When calculated manually, be aware that total participants can be summed over all periods, but unique participants cannot because the same individual may have taken courses in multiple periods. For example, in Table 3-2 notice that the total number of unique participants for the year is 9, not the sum of the unique participants for each quarter, which would be 6 + 5 + 1 + 2 = 14. Total participants, however, may always be easily calculated by summing the periods—7 + 6 + 2 = 2 equals 17, which is the correct total for total participants.

Why should we care about both of these measures? The simple answer is that they tell us different things about the operations of L&D and each one is needed for planning. The measure of unique participants identifies the size of the target audience for a program or curriculum. Larger target audiences will often have more disparate needs resulting from different roles, geographies, or levels in the organization. The number of performance consultants and level of front-end analysis may increase with larger, more diverse target audiences and unique participants. Total participants, on the other hand, has implications for delivery resources. Unless every program is self-paced, increased numbers of total participants will inevitably require larger delivery resource commitments, including instructors, facilities, schedulers, and web-based tools.

Table 3-2. Calculation of Total Unique Participants and Total Participants

Recommendation

Collect and report both total unique participants and total participants for all programs and for the enterprise as a whole at least monthly. Each measure answers a different but important question:

• The measure of total unique participants answers: “How many people were touched or reached by learning?”

• The measure of total participants answers: “How much learning activity took place?” The L&D budget and items like number of instructors, number of classrooms, and amount of development and delivery typically depend on the number of total participants.

Beyond simply counting attendance, L&D and their business counterparts are often interested in knowing the completion rates for specific courses. Many organizations also measure the level of no-shows. Online or virtual instructor-led training (vILT) courses are particularly prone to employees registering but never showing up. When the no-show rate gets high, some L&D functions charge the business for no-shows who don’t cancel at least 24 hours in advance of the course.

To compute these measures, we need to distinguish between the number who registered, the number who attended the course (even in part), and those who completed it. Using these measures, we can then compute:

Target Audience for the Course: The intended audience for the course. For compliance or mandatory courses, it’s the number who are directed to take the course.

Registrations: Count of unique participants registered or scheduled for a course.

Attendees: Count of unique participants who attended the course. These individuals may not complete the course, but they showed up at some point during its delivery.

Attendee Rate: Attendees ÷ Registrations (expressed as a percentage).

No Show Rate: 100% − Attendee rate.

Completions: Count of unique participants who attended and completed a course.

Drop Out Rate: 100% − (Completions ÷ Attendees, expressed as a percentage)

Completion Rate: Completions ÷ Target audience for the course. This measure shows the percentage of the target audience that competed the training.

As with other measures, some practitioners may wish to report by the modality of the learning: instructor-led training (ILT), virtual instructor-led training (vILT), online or e-learning (also called web-based training or WBT), mobile learning, or blended learning (any combination of ILT, vILT, e-learning, or mobile learning). These measures are also often broken out by region, business unit, brand, role, and so forth.

Recommendation

• Measure no-shows for ILT and vILT to pinpoint where issues arise so leaders can develop a mitigation plan. No-show rates help L&D leaders better manage economies of scale. High levels of no-shows affect use of resources, both instructors and facilities.

• Compute completion rates for certain types of learning like compliance training, basic skills training, and advanced skills training, where it is important that the participant be exposed to all the content in the course.

• When the module or course, or especially performance support tool, is designed for the participants to jump in, find what they need, and jump back out, organizations need not compute completion rates.

Courses

There are two important types of course measures. The first addresses availability and usage; the second, development and delivery.

Definitions of Terms

While terms such as program, course, and class are often used interchangeably, we define each specifically, borrowing from the academic terminology.

• Program: A course or series of courses with similar learning objectives designed to accomplish a business or HR goal or meet an organizational need. For example, a program to improve leadership may comprise four related courses over a six-month period. At the university level, a program leading to a degree in economics may require 12 courses.

• Course: A class or series of classes, an online module or series of online modules, prework, post-work, performance support, discussion boards, and other types of learning to convey related and integrated content. For example, a course on leadership may consist of four hours of pre-reading, two online modules, four instructor-led classes, an online discussion board, and performance support. In a corporate environment, each course will have a specific designation in the LMS. At the university level, students will enroll in specific courses each term, such as Economics 101.

• Class: Each physical or virtual meeting of students where content is conveyed by an instructor. A course may consist of just one class if the content can be conveyed in one sitting or it may require multiple classes to convey all the content. At the university level, a semester-long course like Econ 101 might meet for two classes per week for 10 weeks for a total of 20 classes. It is also possible that the number of students enrolled in a course exceeds the optimum class size, which necessitates multiple classes even if the content can be conveyed in a single sitting. So, a one-hour instructor-led course for 150 employees will require six classes of 25 each. The analogy at the university level is 300 students taking Econ 101 where enrollment is limited to 100 per section. In this case, there will be three sections with 100 students in each.

Availability and Use

The number of courses available and the number used are very popular activity measures. There are three key course measures.

Courses Available or Offered: Total courses available = ILT courses available + vILT courses available + e-learning courses available + mobile learning courses available. Total number of unique courses available for enrollment during the period of measurement. For example, a course that is open for registration in December 2020 but not offered until February 2021 would count as a course available in 2021 but not in 2020. A course that is open for enrollment but ultimately canceled due to low or no registration still counts as an available course since it was offered. A course planned but never opened for enrollment does not count as an available course.

Courses Used or Consumed: Total number of unique courses that had participants.

Percentage of Courses Used: Courses used ÷ Courses available (expressed as a percentage)

Another common use of course measures is to determine the mix of learning by modality—ILT only, vILT only, e-learning only, mobile only, and blended learning—as shown in Figure 3-2 and described below

Percentage ILT Only: ILT courses ÷ Total courses (expressed as a percentage)

Percentage vILT Only: vILT courses ÷ Total courses (expressed as a percentage)

Percentage E-Learning Only: E-learning courses ÷ Total courses (expressed as a percentage)

Percentage Mobile Learning Only: Mobile learning courses ÷ Total courses (expressed as a percentage)

Percentage Blended Learning: Blended courses ÷ Total courses (expressed as a percentage)

Figure 3-2. Example of Chart With L&D Mix of Courses by Delivery Modality

Recommendation

Report course measures such as number available, number used, and utilization rate at least annually. These are foundational for an L&D department. Further, if the department has a goal to shift the mix of learning from one modality to another, then the percentage mix by modality should be reported annually and perhaps monthly. For example, many departments are trying to shift from a reliance on ILT to more vILT, e-learning, mobile learning, and blended learning.

Development and Delivery

This second set of course measures addresses the development and delivery of courses, including whether they were on time.

Total Courses Developed: Total number of new and updated courses during a period. The number of new courses and the number of updated courses may be reported separately.

Completion Date for Development: The date development is to be completed. Used for an individual course. Year-to-date (YTD) progress may be expressed as the percentage complete.

Total Courses Developed Meeting Deadline: Total number of new and updated courses completed on or before the deadline established by the stakeholder and learning staff.

Percentage of Courses Developed Meeting Deadline or Percentage of On-Time Completion for Development: Total courses developed meeting deadline ÷ Total courses developed (expressed as a percentage).

Total Courses Delivered: Total number of courses delivered, including e-learning courses made available online.

Completion Date for Delivery: The date delivery is to be completed for an individual course. (YTD progress may be expressed as percentage complete.)

Total Courses Delivered Meeting Deadline or Percentage On-Time Completion for Delivery: Total number of courses delivered on or before the deadline established by the stakeholder or learning staff.

Percentage of Courses Delivered Meeting Deadline: Total courses delivered meeting the deadline ÷ Total courses delivered (expressed as a percentage).

Recommendation

Use on-time development and delivery measures whenever a deadline commitment is made to a goal owner or stakeholder. For example, when developing or delivering just one course for a goal owner or stakeholder, the measure may simply be the completion date. In this case, YTD progress may show percentage progress toward completion and the date once completed. These measures appear in almost all program reports. (See chapter 9 for examples of on-time measures and completion dates.)

Classes

Class measures are less common than course measures but still have a purpose in managing the L&D function. The most common class measures are:

Total Classes: Total number of classes that were held. For example, if course A has five classes, course B has three classes, and course C has one class, the total number of classes for the three courses is 5 + 3 + 1 = 9.

Class Size: Number of participants in an ILT or vILT class.

Average Class Size: Sum of class sizes ÷ Total classes. The simple average of all class sizes.

Classes Canceled: Number of classes that were open for enrollment but later canceled.

Percentage of Classes Canceled: Classes canceled ÷ Total classes (expressed as a percentage).

Recommendation

Consider class-based measures if there is an issue with very small or very large class sizes or if classes are being canceled due to participants not showing up.

Hours

Some common hourly measures include unique hours available, total hours consumed, and learning hours used per employee:

Unique Hours Available (or Offered): Total number of hours available for formal learning. This is typically a one-time count made annually, answering the question, “How many hours of learning were available for people to take last year?” The total is calculated as the sum of the hours for each unique course offered regardless of whether it is subsequently canceled (in other words, the sum of the hours of all classes for all available courses). Typically, a standard duration is set for an e-learning or mobile course and the standard duration is assumed for all participants of that course. (For example, a typical standard duration is 30 minutes, meaning that participants, on average, are expected to engage the e-learning class for 30 minutes.)

Unique Hours Used (or Consumed): Total number of unique hours in courses that had participants. This is typically calculated annually. For example, suppose five two-day ILT courses were available (offered) for the week but two canceled for low enrollment. The hours of formal learning available would be 80 hours (5 courses × 16 hours per course), and the unique hours used would be 48 hours (3 courses × 16 hours per course).

Percentage of Unique Hours Used: Unique hours used ÷ Unique hours available (expressed as a percentage). This measure is usually calculated annually. In the example above, it would be 48 hours ÷ 80 hours = 60%.

Total Hours Used (or Consumed): Total number of hours that participants were engaged in formal learning (accessed or completed courses). This measure is often reported monthly as well as annually. The total is calculated as the sum of each participant’s total number of hours in courses. Using the above example, suppose that there were 20 participants for the first course, 10 for the second, and 25 for the third. The total hours of formal learning used would be (20 participants × 16 hours) + (10 participants × 16 hours) + (25 participants × 16 hours) = 320 hours + 160 hours + 400 hours = 880 hours.

Learning Hours Used per Employee: Total hours of learning used ÷ Number of employees. Alternatively, Total hours of learning used ÷ Number of FTE employees (ATD BM). Many organizations used to set a goal of 40 hours per employee per year. This is typically an annual measure.

Note

This is identical to the ISO 30414:2018 standard measure Average Formalized Training Hours per Employee, which is recommended for internal reporting by all organizations (ISO 2018, 25). ISO recommends using headcount in the denominator but allows for FTE (full-time equivalent) to be used as well. Given the two definitions of employee, a report should always indicate which definition is used. Further, if an organization has a large contingent workforce, consideration should be given to using workforce as the denominator where workforce includes contingent workers in the number of employees. (Contingent workers are temporary and contract workers.) The selection of the denominator depends on how much training contingent workers need (if they need a lot, include them). Regardless of which denominator is chosen for the summary measure, it is recommended the measure also be shown by category of workforce: full-time employees, part-time employees, temporary workers, and contract workers.

Recommendation

These are all foundational measures for an L&D department and should be reported and analyzed at least annually. At a minimum, the ISO recommended measure for average formal training hours per employee should be reported.

Six other useful measures involving hours are:

Learning Hours Used per Participant: Hours of learning used ÷ Number of unique participants.

Learning Hours Available per L&D Staff Member: Unique hours available ÷ FTE learning staff (ATD BM).

Learning Hours Used per L&D Staff Member: Total hours used ÷ FTE learning staff (ATD BM).

Reuse Ratio: Total hours used ÷ Unique hours available. This ratio indicates the intensity of use or how many times one hour of learning is used. Note that this measure is not referring to how much content from the previous year is repurposed or reused for the coming year.

Percentage of Unique Hours Available by Delivery Type: Unique hours available for learning type ÷ Total unique hours available. This measure provides the percentage or share of learning offered by delivery type (modality) and is typically measured annually. For example, ILT may make up 54 percent of all unique learning hours available, e-learning 38 percent, and vILT 8 percent. Note: ATD refers to this measure as Average Percentage of Formal Learning Hours Available via Different Delivery Methods (ATD BM).

Percentage of Total Hours Used by Delivery Type: Total hours used for learning type ÷ Total hours used. This measure provides the percentage or share of learning used by delivery type (modality) and may be reported monthly as well as annually. For example, ILT may make up 60 percent of learning hours used, e-learning 25 percent, and vILT 15 percent. Note: ATD refers to this measure as Average Percentage of Formal Learning Hours Used via Different Delivery Methods (ATD BM).

Recommendation

Report measures for hours used regularly if the department has a goal to shift the mix of learning from one modality to another. For example, many departments are trying to shift from a reliance on ILT to more vILT, e-learning, and blended learning. The hours may tell a different story than the number of courses, so the recommendation is to report both.

Engagement

In addition to the efficiency measures covered under Participants and Courses, there are several measures that are specific to e-learning. These can be collected if the course is developed with xAPI (experience API), a software that provides insight into the learner’s interaction with the content. Some learning platforms also offer the same measurement capability. Common measures include:

Learner Engagement: Number of interactions with the content.

Course Duration: Time spent on the course.

Topic or Task Duration: Time spent on a particular topic, task, or subset of the course.

Learner engagement may also be reported by topic or section, just as duration.

In addition, xAPI enables organizations to aggregate data from different systems (for example, customer relationship management systems, LMSs, learning simulations systems, and data from call-listening sessions) to create the full picture of a learner’s performance.

Recommendation

Become familiar with xAPI and how it is used with e-learning-specific measures to collect data about the user experience.

Reach Measures

Reach shows the number or percentage of people touched by learning. It is a measure of the breadth or dispersion of learning within the organization. Reach answers the question of whether learning is broad based (reaching most employees) or more narrowly focused (reaching only a few).

Employees Reached by L&D: The number of unique employees who participated in formal or informal learning that was sponsored, conducted, or managed by the learning function. For example, any employee who took a course from the learning function, engaged in a community of practice sponsored by the learning function, or used a performance support tool designed by the learning function would count.

Employees Reached by Formal Training: The number of employees who participated in formal training. In practice, it may be difficult to identify employees who used a knowledge-sharing platform or a performance support tool. In this case, the measure might simply reduce to those reached by formal training, which is tracked in the LMS.

Percentage of Employees Reached by L&D: Employees reached by L&D ÷ Number of unique employees (expressed as a percentage).

Percentage of Employees Reached by Formal Training: Employees reached by formal training ÷ Number of unique employees (expressed as a percentage).

Note

This measure is identical to the ISO 30414:2018 standard measure Percentage of Employees Who Participate in Training Compared With Total Number of Employees per Year (ISO 2018, 25; “Technical Specification on Metrics for Skills and Capabilities” forthcoming.)

In addition to reporting this measure at a high level for all employees, we also recommend that L&D generate a report by category of employee. For example, the report might show the percentage of management versus non-management employees reached by learning. The report might also show percentage reached by business unit or region or by age or another common demographic.

ISO 30414:2018 recommends one such demographic in particular for internal reporting by large organizations, namely leaders (ISO 2018, 16)

Percentage of Leaders Who Participate in Training: Number of leaders who participate in formal training ÷ Number of leaders, expressed as a percentage.

This measure may also be reported by category. For example, the percentage taking compliance, basic skills, onboarding, or leadership. Reporting by category is identical to the ISO 30414:2018 measure Percentage of Employees Who Participate in Formal Training in Different Categories, where each organization defines the categories of greatest relevance (ISO 2018, 25; “Technical Specification on Metrics for Skills and Capabilities” forthcoming.)

ISO 30414:2018 recommends two categories for internal reporting by large organizations—compliance and ethics and leadership development:

Percentage of Employees Who Have Completed Training on Compliance and Ethics: Number of employees who have completed training on compliance and ethics ÷ Number of unique employees assigned to the training (expressed as a percentage; ISO 2018, 12).

Percentage of Leaders Who Have Participated in Leadership Development Programs: Number of leaders who have participated in leadership development programs ÷ Total number of leaders (expressed as a percentage; ISO 2018, 16).

There are two other important measures of reach:

Employees With Development Plan: The number of unique employees who have a personal development plan.

Percentage of Employees With Development Plan: Employees with development plan ÷ Number of unique employees, expressed as a percentage.

Recommendation

Report reach at least annually; it is another foundational measure for an L&D department. The CEO at Caterpillar, where David was president of Caterpillar University from 2000 to 2007, was always very interested in the percentage of employees reached by learning because he wanted to ensure that all employees had learning opportunities. Since compliance training may constitute a large percentage of training taken, reach should also be reported separately for compliance and noncompliance-related training, preferably with a breakdown by category as recommended by ISO, including leadership. Per the ISO standard, the percentage of leaders who have completed leadership training should also be reported. Lastly, the percentage of employees with a development plan is also a very important measure and should be reported at least quarterly if not monthly.

Effort Measures

Effort is a measure of the amount of work required to plan, design, develop, deliver, measure, and manage learning, and is usually expressed in hours or days. For example, if one learning staff member spends 200 hours on learning program A and another spends 60 hours on that program, the total effort spent on program A is 260 hours.

Effort to Create New Courses: The effort required to plan, design, and develop a new course, including the performance consulting (expressed in hours or days).

Effort to Update Existing Courses: The effort to plan, redesign, and redevelop an existing course (expressed in hours or days).

Recommendation

Use effort measures to plan the resource requirements for your future programs.

Cycle Time Measures

Cycle time is the time elapsed from the beginning of a defined process to its completion. Tracking cycle times and comparing to internal history and external benchmarks provides an opportunity to improve efficiency.

Cycle Time for Performance Consulting: The time in days from the first meeting with a goal owner to discuss how learning may help meet their need (start of needs analysis) to the completion of the needs analysis and hand off to design and development.

Cycle Time for Handoff to Design: The time in days from the completion of the performance consulting to the start of design and development.

Cycle Time for Design and Development: The time in days from the start of design and development to the completion of design and development (goal owner agrees the course is ready for deployment).

Cycle Time for Handoff to Delivery: The time in days from the completion of design and development to the delivery of the course (first day of ILT or vILT class, or the first day the e-learning is available).

Cycle Time for Delivery: The time in days from the start of delivery to the completion of delivery.

Total Cycle Time: The time in days from the first meeting with the goal owner to completion of delivery. Total cycle time is the sum of the previous five components.

Recommendation

Measure cycle times to determine your opportunities for improvement if you are not meeting promised completion dates or if you want to become more agile.

Cost Measures

There are a number of popular cost measures employed in L&D. We explore 16, including eight that are commonly benchmarked, beginning with the most important cost measure.

When computing costs, expenditures both for formal and informal learning should be included in the totals. We highlight where informal learning costs may show up differently than formal learning within each of the major categories below.

Most L&D departments report a high-level cost measure such as direct expenditure:

Direct Expenditure: Internal costs + External service provider expenditures + Tuition reimbursement expenditures. This includes expenses incurred by the organization for the planning, design, development, delivery, management, and measurement of formal and informal learning. Direct expenditure also includes expenditures by staff in the learning departments for performance consulting, reinforcement of learning, and general management of the learning function. Put another way, direct expenditure includes all the expenses (entire budget) of all learning departments within an organization plus any identifiable expenditures on learning that occur outside the learning departments plus tuition reimbursement expenditures (wherever they are budgeted).

In terms of the financial reports for a learning function, the following types of expenditures are included in direct expenditure:

• Internal cost.

Labor and related expense (salaries plus employer paid taxes, benefits, bonuses, profit sharing).

Overhead expense, including office supplies, dues, subscriptions, printing, mailing, phone, computer, equipment leases, travel and entertainment, consultants, other external service providers, classroom rental, utilities, and training (for the L&D staff).

Internal charges. In many organizations, the L&D department is charged by the organization for occupancy-related expenses. There may also be internal charges for IT, HR, accounting, legal, and so forth. These may appear under “Overhead” in a monthly department expense statement or be shown separately, but they should be included.

• External services and technology platform expenditures. The amount spent on contract services from learning providers or technology vendors outside the company. While this category is part of the internal cost’s overhead expenses, it is often also reported separately. It includes expenses for partners, vendors, and consultants who design, develop, and deliver or administer learning as well as providers of learning systems (LMS, LCMS, and so forth), technology platforms (for example, learning experience platforms), and applications for specific needs (such as a performance support system or coaching program platform).

• Tuition reimbursement expenditure. Reimbursements to employees for degree and non-degree educational programs at educational institutions, including programs for professional education and certification. It is understood that tuition reimbursement may not fall within the scope of all L&D functions.

Direct expenditure may also be calculated by summing the expenditures of all learning functions, all spending on training outside learning functions, and tuition assistance.

Direct Expenditure: All expenses of all learning departments + Expenditures on learning outside learning departments + Tuition reimbursement expenditures.

Note

This measure is identical to the ISO 30414:2018 standard measure Total Development and Training Cost (ISO 2018, 25; “Technical Specification on Metrics for Skills and Capabilities” forthcoming.) ISO recommends this important measure for external reporting by all organizations. See the next section for a discussion on total cost and why the ISO nomenclature may be confusing in this instance.

Conceptually, there are two other important learning-related costs, but both are difficult to measure and consequently seldom reported.

Learner’s Travel-Related and Fee Expenses: The travel-related cost for participants to attend either in-house or external learning programs plus the program-related costs of external programs (for example, the registration fee to attend a three-day leadership program). It is often difficult to obtain this cost data, although some organizations provide a check box on travel expense forms to indicate the expense is related to learning.

Opportunity Cost: The value of what is given up or foregone in choosing one course of action over another. In the context of learning, opportunity cost is the value of what the participants would have been doing if they were not in class. Salespeople would be closing deals, engineers would be designing product, and so forth, which is why they were hired in the first place. So, the organization loses their work contribution when they participate in a learning program.

What Is the Value of the Opportunity Cost?

The opportunity cost should be at least what the organization is paying them, and if it is not, then the employee should be reassigned, retrained, demoted, or fired because it does not make economic sense for the organization to pay the employee more than they are worth.

Thus, at a minimum, opportunity cost can be calculated as the value of the participant’s time in class and in transit. This can be found as hours multiplied by the labor and related hourly rate. For example, the opportunity cost of 100 salespeople (each earning $50 per hour in salary and benefits) in a two-day ILT with 30-minute travel time each way would be:

=100 × [(2 days × 8 hours per day) + (0.5 hour per trip × 4 trips)] × $50 per hour

= 100 × 18 hours × $50 per hour

= $90,000

In some cases, the opportunity cost will exceed the value of the participant’s time and can be more accurately determined. For example, the organization may be able to estimate the value (impact on net income) of the lost sales from having 100 salespeople in training for two days. In this case, and assuming this estimate exceeds the value of their time, the estimate should be used as the opportunity cost. (See Vance [2017] for more on opportunity cost.)

Opportunity cost should also be calculated for organizations that employ a “leaders as teachers” model. In this case, the value of a leader’s time will not be captured in a learning department expense statement but it does represent a cost to the organization, especially if very senior leaders dedicate much time to teaching. In other words, their time is not free.

So, the most comprehensive measure of cost would be total cost, which includes learner’s travel-related expenses and opportunity costs. Total cost for learning in an organization includes all learning related costs except those outside the learning function(s) for management and reinforcement of learning (which generally would be too difficult to obtain). Cost breakdown is shown in Figure 3-3; by formula it would be:

Total Cost: Direct expenditure + Learner’s travel costs and fees + Opportunity cost.

Figure 3-3. Components of Total Cost

Important cost-related measures for L&D that derive from direct expenditure include:

Direct Expenditure per Employee: Direct expenditure ÷ Number of employees. Alternatively, Direct expenditure ÷ Number of FTE employees (ATD BM). Because there are two potential ways to define employee, the report should always specify which definition is used. Like our earlier discussion about hours per employee, this measure should also be calculated for contingent workers if company-provided training is important for their performance on the job.

Since the average decreases with organization size due to economies of scale and since it varies significantly by industry, ATD also reports the following two comparisons:

Direct Expenditure per Employee by Organization Size: Reported for organizations with fewer than 500 employees, 500-9,999 employees, and 10,000 or more employees (ATD BM).

Direct Expenditure per Employee by Industry Grouping: This measure is reported for four different industry groupings: finance, insurance, and real estate; healthcare and pharmaceutical; manufacturing; and software, information, broadcasting, and telecommunications (ATD BM).

Other cost-related measures include:

Direct Expenditure per Learner: Direct expenditure ÷ Number of unique participants.

Direct Expenditure as a Percentage of Revenue: Direct expenditure ÷ Organization (company) revenue (ATD BM).

Direct Expenditure as a Percentage of Payroll: Direct expenditure ÷ Organization (company) payroll (including benefits; ATD BM).

Direct Expenditure as a Percentage of Profit: Direct expenditure ÷ Organization (company) profit (or surplus).

Percentage of Expenditure for Tuition Reimbursement: Direct expenditure for tuition reimbursement ÷ Total direct expenditure (ATD BM).

Percentage of Expenditure for External Services: Direct expenditure for external services ÷ Total direct expenditure (ATD BM).

Cost per Learning Hour Available: Direct expenditure ÷ Unique hours available (ATD BM).

Cost per Learning Hour Used: Direct expenditure ÷ Total hours used (ATD BM).

Average Development Cost of a New ILT or vILT Course: Direct expenditure of developing new ILT or vILT courses ÷ Number of new ILT or vILT courses.

Average Development Cost of a New E-Learning Course: Direct expenditure of developing new e-learning courses ÷ Number of new e-learning courses developed.

Cost Reduction: A reduction in direct expenditure cost or a component of direct expenditure or opportunity cost.

In the next section, we focus on the efficiency measures for informal learning. We will not cover cost measures specifically, not because they do not have associated costs, but because we believe that the taxonomy we outlined in Figure 3-3 applies to informal learning as well. For example, consider the implementation of a mentoring program. The costs associated with the program would include internal resources to design, develop, and implement the program. If L&D purchased software to match mentors and mentees, this cost would be included in the technology expenditures. Finally, if the organization hired external resources to mentor employees in specific roles, these expenditures would be addressed in the external services category. We recommend that organizations incorporate costs for all programs, be they formal or informal in their cost calculations.

It is evident from the list of measures that numerous benchmarks are available for direct expenditure. Some organizations benchmark annually while others only do so periodically. In either case, the following caveats are important to keep in mind when comparing to benchmarks:

1. Compare to organizations of a similar size because economies of scale will almost always drive average costs down for larger organizations.

2. Compare to others within your industry because averages vary significantly by industry.

3. Remember that a benchmark average should not necessarily be your goal. Many organizations with higher than benchmark spending adopt a goal to reduce their own spending toward the benchmark, which is often a mistake. The benchmark, while informative, is nothing more than an average, and the optimum level of L&D expenditure for your company may be above or below the average. It should be above the average if you have very challenging organizational goals or a relatively less experienced and proficient workforce. It should be below the average if you have less challenging organizational goals or a more experienced and proficient workforce. No one outside your organization can tell what the right level of expenditure is for your organization for the coming year, and you should not aspire to be average.

Recommendation

Measure direct expenditure at least annually, even if some assumptions or estimates are required. This is a foundational efficiency measure, and your CEO will expect it. More mature organizations also measure opportunity cost. Typically, learners’ travel-related costs are very difficult to measure because they are not learning department expenses and consequently are seldom measured. It will also be helpful to track direct expenditure per employee and direct expenditure as a percentage of either payroll or revenue through time to gauge whether the investment in learning and development is keeping pace with the growth in employees and with organization size.

L&D Staff and Employee Headcount Measures

There are numerous common measures for staff and headcount. These are the most popular:

FTE L&D Staff: Number of staff in all learning functions expressed as full-time equivalent (FTE). For example, three full-time staff and one part-time staff person who works 20 hours per week (20 hours ÷ 40 hours = 0.5 FTE) would be 3.5 FTEs.

Unique L&D Staff: The unique number of learning staff, including part-time employees. In the example, the number of unique staff would be four.

Total Hours for L&D Staff: The sum of each staff member’s employed hours in a period. A full-time staff member is typically paid for 52 weeks × 40 hours per week = 2,080 hours per year. A part-time employee may be paid for work 20 hours per week or 1,040 hours per year. Typically, an estimate is made of total hours available annually.

Total Hours Used by L&D Staff: The sum of each staff member’s actual hours used to plan, design, develop, deliver, reinforce, measure, and manage learning. Typically, these hours are tracked and recorded on a daily or weekly basis. Time for holidays, vacation, and sick time as well as time for staff meetings, performance management, and training is not included.

Percentage of L&D Staff Hours Used: Total hours used ÷ Total hours, expressed as a percentage. Typically, this comes out to 60–70 percent. This measure is important for calculating the burden rate.

FTE Employees: The size of the organization’s workforce measured as the average number of FTE employees over the year. The count should include employees of consolidated domestic and international subsidiaries, part-time and seasonal employees, and officers. The count should not include contracted workers, consultants, vendors, board members, or employees of unconsolidated subsidiaries. Most organizations report employee counts on a monthly basis, so total employees will be found as the simple average of the counts for the 12 months. Some organizations may not provide the FTE for part-time employees. In this case, estimate the FTE for part-time employees from the total number of part-time employees. (Note: Since ATD uses FTE rather than headcount, this measure for FTE Employees is identical to ATD’s Workforce Size measure.)

Unique Employees: The size of the organization’s workforce measured as the count of all full-time and part-time employees throughout the year. Due to turnover, this count will exceed the number of employees on the payroll at the end of the year. For example, if employees A–D were on the full-time payroll all year, Employee E left in March, part-time Employee F started in June, and full-time Employee G started in December, the unique employee count is seven and the year-end count is six.

Employees per L&D Staff Member: FTE employees ÷ FTE learning staff (expressed as a number to one decimal place; ATD BM). Both the numerator and denominator are best expressed in FTE, but may also be calculated as the number of unique employees divided by the number of unique L&D staff members.

Recommendation

Report the total and unique learning staff at least annually. Many L&D departments also report employees per L&D staff annually. Utilization of staff hours is less common.

Informal Learning Measures

Unlike formal learning with a few modalities but a common set of measures, informal learning comes in many shapes and sizes. Rather than divide measures into categories like activity and reach, for this section, we focus on the measures associated with four common types of informal learning: content, communities of practice, coaching and mentoring, and performance support.

Content Measures

Learning organizations are increasingly creating learner experience portals (LXPs) as a single source of information that may range from whitepapers, tools or job aids, videos, or links to relevant training. LXPs typically track which documents employees access and share, and may identify how much time a user spends on a specific page. The challenge is that LXPs contain thousands of documents that are being accessed each day by employees. It’s impossible to track the details associated with every piece of content, so organizations must make choices about what provides useful insight for decision making. Content efficiency measures focus on a few key areas: users, usage, and time spent.

Users

When organizations develop content portals (or LXPs), they may initially target specific audiences such as sales professionals, client-support personnel, or R&D. Over time, organizations will broaden their reach, and it’s important to understand if the target audience is actually using the resources. These measures are focused on a suite of communities; however, they could be used for specific communities as well.

Target Audience Size: Approximate number of employees in the target audience for the communities overall.

Unique Users: Number of unique employees who have accessed the content within the period.

Percentage Reach: Unique uses ÷ Target audience size.

Total Users: Number of total employees who have accessed content within the period. This allows for a unique employee to access multiple pieces of content and to access the same piece of content more than once.

Usage

Organizations devote significant resources to acquiring, developing, curating, and managing content. Consequently, L&D needs to determine what is being used and what is not being used.

Total Number of Documents Available: Total number of documents (across all types of documents) available in a given period.

Total Number of Documents Accessed: Total number of documents (across all types of documents) accessed in a given period.

Percentage of Documents Accessed: Total number of documents accessed ÷ Total number of documents available (expressed as a percentage).

Average Documents Accessed: Total number of documents accessed ÷ Number of unique users. This measure can be subdivided as shown:

• Average number of documents accessed by type (for example, job aid versus whitepaper).

• Average number of documents accessed by modality (for example, document, blog, or video).

• Average number of documents accessed by community.

Top 25 Most Common Documents Accessed: By document, identify documents with the largest number of hits.

Time Spent

Time spent on a site can mean different things: Users might be engaged and find the content useful, or they might be spending a lot of time searching for content without finding what they need. While time alone doesn’t tell the whole story, it is important to track it to understand the behaviors of employees and how they use the site.

Average Time Spent on the Site: Total time spent on the site ÷ Number of unique users.

Recommendation

For organizations that have created a knowledge portal (or implemented an LXP), track at least the number of users (relative to the target audience) and the usage. Duration data are helpful when combined with effectiveness measures to characterize the behavior of employees when accessing content.

Communities of Practice Measures

Etienne Wenger defined communities of practice (CoPs) as groups of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly (David 2014). In business settings, communities often form to advance knowledge in a specific industry, domain (for example, technical specialty), or practice (such as project management). Communities facilitate knowledge sharing, not only through content but also via people-to-people connections and online or in-person discussions. While CoPs have existed for years, the ability to track them has improved significantly with the use of online platforms. Some CoPs use sophisticated platforms that provide discussion forums, blogging, messaging, content reviews, and news feeds. Others are smaller scale, using messaging apps to enable like-minded employees to interact. For this book, our focus is on a suite of measures that are independent of the platform being used.

The following provide a starter set of efficiency measures for CoPs. If your organization is a heavy user of CoPs, you may want to add to this list to capture the diversity of activity within the communities.

Total Communities of Practice: The number of unique communities of practice in the organization.

Active Communities of Practice: The number of communities where there has been activity within a defined time period (month, quarter, or year); activity would include discussions, requests for help, providing help, identification of experts, or contributions or access to stored knowledge.

Percentage Active Communities of Practice: Active communities of practice ÷ Total communities of practice (expressed as a percentage).

Unique Community of Practice Members: The total number of unique individuals who are members of at least one community of practice.

Unique Active Community of Practice Members: The total number of unique individuals who have engaged in discussions, requested or provided help, served as experts, or contributed or accessed knowledge within a defined time period.

Percentage Active Community of Practice Members: Active community of practice members ÷ Total community of practice members (expressed as a percentage).

Total Community of Practice Members: The total number of individuals who are members of a community of practice. This measure allows for individuals to be members of multiple communities.

Recommendation

All seven measures are highly recommended for monthly reporting if there is a department goal to increase the use of communities of practice. If not, then these measures should be reported at least quarterly if COPs play an important role in the organization’s overall learning strategy.

Coaching and Mentoring Measures

While coaching and mentoring occurs informally in all business settings, we examine measurement in specific programs developed to match mentors or coaches with employees who will benefit from guidance in specific areas (for example, leadership or people management). Structured coaching and mentoring programs usually engage a program coordinator and develop processes to recruit and train mentors, vet mentees, and provide support for employees engaged in the program.

Efficiency measures for coaching or mentoring networks focus on how many individuals are engaged, the ratio of mentors to mentees, the frequency of meetings, and the duration of time in the program. We recognize that coaching and mentoring are very different types of activities and different skills and processes are involved. However, for the sake of measurement, we group these two types of programs together.

Number of Coaches or Mentors: The number of individuals who have been approved to act as a coach or mentor in the program.

Number of Coachees or Mentees: The number of individuals who have been selected to receive coaching or mentoring in the program.

Ratio of Coaches (Mentors) to Coachee (Mentee): Total coachees or mentees ÷ Total coaches or mentors.

Average Time in the Program for Coaches or Mentors: Total months of engagement for all coaches or mentors ÷ Number of coaches or mentors. This is the average amount of time (usually in months) that a coach or mentor has been engaged in the program.

Average Time in the Program for Coachees or Mentees: Total months of engagement for all coachees or mentees ÷ Number of coachees or mentees. This is the average amount of time (usually in months) that a coachee or mentee has been receiving coaching or mentoring in this program.

Frequency of Meetings: Number of meetings per period between the coach or mentor and coachee or mentee.

Duration of Meetings: Average time of meetings (virtual or in person) of coach or mentor and coachee or mentee.

Recommendation

All seven measures are highly recommended for monthly reporting if a coaching or mentoring network has been established to achieve a department goal for building organizational competency. If not, these measures should be reported at least quarterly if coaching or mentoring networks play a role in the organization’s overall learning strategy.

Performance-Support Specific Measures

Performance support refers to the tools that learners use at the time of need to perform a task, which may be as simple as a checklist. Performance support is particularly valuable when a task requires multiple steps (especially if they must be completed in a particular order) or infrequently used instructions.

For example, performance support may be helpful for managers who need to enter performance ratings for their employees into the organization’s HR system annually. While not complex, this involves multiple steps and very specific instructions, all of which occur just once per year. Managers could be sent to a class to learn how to enter the ratings, but it would take time and most would quickly forget the steps. Instead, a performance support tool would supply the information managers need when they need it. Performance support tools are also critical for employees in customer call centers who must answer a wide variety of questions from many customers in a short span of time.

Some measures for performance support include:

Performance Support Tools Available: Number of performance support tools available.

Performance Support Tools Used: Number of performance support tools used.

Percentage of Performance Support Tools Used: Performance support tools used ÷ Performance tools available (expressed as a percentage).

Unique Performance Support Users: Number of people using at least one performance support tool.

In addition to being a standalone tool, performance support can also be included in course design, to be used during the course or for reference afterward. This is a form of blended learning, just like employing both e-learning and instructor-led learning in a program. A measure for this would be:

Percentage of Courses With Performance Support: Number of courses with a performance support tool ÷ Total number of courses (expressed as a percentage).

Recommendation

All five of these measures should be reported if the learning strategy calls for performance support.

Conclusion

This concludes our description of the most common efficiency measures. No organization would ever use all of these or even half. The goal is to identify the right measures to support the learning strategy, which in any year is typically focused on some, but certainly not all, of these measures.

The most common efficiency measures are shown in Table 3-3. These represent the most commonly used efficiency measures and include all the efficiency measures benchmarked by ATD and recommended by the ISO 30414:2018 standard. Additional efficiency measures may be found in the CTR measures library.

Table 3-3. Most Common L&D Efficiency Measures

We’re now finished with efficiency measures, which are the most numerous measures in L&D and the foundation of any measurement strategy. Next, we examine effectiveness measures which are more powerful and form the basis for the evaluation of learning programs.