This chapter discusses different approaches to measuring business innovation objectives and outcomes, extending the measurement of innovation characteristics introduced in Chapter 3. It discusses several qualitative measures of the variety of innovation objectives and outcomes pursued by firms. This is complemented by an evaluation of quantitative measures of innovation outcomes for both product and business process innovations. An overview of the challenges to measuring innovation outcomes is presented before providing a final set of recommendations.
8.1. The planning and development stage for an innovation includes the identification of a set of one or more objectives that the innovation is expected to achieve. The objectives can refer to the characteristics of the innovation itself, such as its specifications, or its market and economic objectives. The outcomes of an innovation can be captured by a similar list of items as the objectives, but consist of the innovation’s realised effects. These can also include unexpected effects that were not identified among the firm’s initial objectives.
8.2. A firm’s economic objectives for its innovations can include the generation of profits, an increase in sales or brand awareness from product innovation, and cost savings or productivity improvements from business process innovation (Crépon, Duguet and Mairesse, 1998). Other objectives include changes to the firm’s capabilities, markets, or the types of customers that buy its products, and the establishment of new external linkages.
8.3. Innovation outcomes include the extent to which a firm’s objectives are met and the broader effects of innovation on other organisations, the economy, society, and the environment. The broader effects may or may not have been identified by a firm as innovation objectives. They include different types of spillovers and externalities that can change the structure of competition in markets and stimulate or hamper the innovation activities of other organisations. Broader effects of innovation can also contribute to or hinder societal goals such as improvements to employment, health and environmental conditions, or help solve or influence other societal challenges.
8.4. Common objectives for many firms are to increase overall profits and growth in terms of sales or market share. Research on the effects of innovation on such outcomes should ideally use administrative data and identify the effect of innovation through econometric analysis (see Chapter 11). However, it is also of value for research to collect data on outcomes that are limited to innovations, such as the sales share or profit margin of innovations.
8.5. This chapter presents different approaches to measuring innovation objectives and outcomes. Section 8.2 discusses qualitative measures of the variety of innovation objectives and outcomes pursued by firms. Section 8.3 includes an evaluation of quantitative measures of innovation outcomes for product and business process innovations. An overview of the challenges for measuring innovation outcomes is presented in section 8.4, before providing a final set of recommendations.
8.6. Innovation objectives consist of a firm’s identifiable goals that reflect its motives and underlying strategies with respect to its innovation efforts (see subsection 5.3.1) Collecting data on innovation objectives is useful for research on the factors that drive a firm’s decision to engage in innovation activities, such as the intensity of competition or the opportunities for entering new markets, and how the firm responds to these drivers, such as improvements to the efficiency of the firm’s operations or enhancements to its innovation capabilities. Data on objectives can also provide insights into the planned characteristics of innovations, for instance if a firm’s objective is to substantially change its business processes or to only make minor adjustments. In addition, data on innovation objectives can be used to construct innovation profiles (see subsection 3.6.2) or other systems for classifying innovative firms.
8.7. Innovation outcomes are the observed effects of innovations. In a survey context, outcome data are based on the perceptions of respondents in innovative firms. Firms may or may not succeed in achieving their innovation objectives, or innovations can entail additional effects that were not part of the firm’s original objectives.
8.8. Many innovation objectives and outcomes can be captured by the same list of items. Table 8.1 lists common objectives that can become outcomes if realised, grouped by areas of influence: markets, production and delivery, firm organisation, and environment and society. Objectives are always intentional, but outcomes can be unintended.
8.9. Objectives and outcomes that influence markets mainly concern product innovations, although some business process innovations can also play an indirect role, such as those that improve the quality or marketing of services thus enhancing the visibility or reputation of these services. The objectives listed under “markets for the firm’s products” capture whether or not the firm planned to change its product portfolio (increase its range of goods or services), enter new markets, target existing markets (increase or maintain market share), or change customer perceptions of the firm’s products (increase its reputation or visibility). Firms may also need to comply with market regulations, for instance by meeting product emissions or recycling standards.
8.10. Objectives and outcomes for production and delivery concern the cost and quality of a firm’s operations. They are mainly related to business process innovation, although some product innovations can contribute. An example is a change in the materials used for a product that reduces the material costs per unit of output.
8.11. The objectives and outcomes that influence the firm’s business organisation capture the effects of business process innovations on a firm’s capabilities. Some of these effects can improve the firm’s capabilities for absorbing, processing and analysing knowledge. Others influence the ability of the firm to adapt to changes or improve working conditions as well as ensuring the continued existence of the firm itself.
8.12. Outcomes that affect an economy, society or the environment are influenced by innovation objectives that target externalities, such as reducing environmental impacts or improving health and safety. Other items refer to the contribution of innovations to wider societal goals such as social inclusion, public security or gender equality. Both product and business process innovations undertaken to comply with standards or regulations can contribute to environmental and societal goals.
8.13. At a minimum, it is recommended to collect data on either the objectives or the outcomes of innovations. As some objectives and outcomes are common, data collection should use an ordinal scale of their importance to the firm. Data on outcomes can only be collected for innovations, while for objectives, data collection should encompass all completed, ongoing, postponed or abandoned innovation activities.
8.14. If data are collected for both innovation objectives and innovation outcomes, then it is recommended to limit both sets of questions to innovations to ensure comparability between objectives and outcomes, and exclude those innovation activities that are ongoing, postponed or ceased.
8.15. A single question can be used to collect data on both objectives and outcomes. In this case, it is recommended to use an importance scale for the objectives. The response options for the outcomes should include whether or not the objective was achieved, if the outcome occurred without a corresponding objective (i.e. it was unintended), and if it is “too early to tell”.
|
Markets for the firm’s products |
|
Upgrade goods or services |
|
Expand the range of goods or services |
|
Create new markets |
|
Enter new markets or adapt existing products to new markets |
|
Increase or maintain market share |
|
Increase the reputation, brand awareness, or visibility of goods or services |
|
Comply with market regulations |
|
Adopt standards and accreditation |
|
Production and delivery |
|
Upgrade outdated process technology or methods |
|
Improve quality of goods or services |
|
Improve flexibility for producing goods or services |
|
Increase speed of producing goods or delivering services |
|
Reduce labour costs per unit of output |
|
Reduce material, energy costs or operating costs per unit of output |
|
Reduce time to market |
|
Business organisation |
|
Improve capabilities for absorbing, processing and analysing knowledge |
|
Improve sharing or transfer of knowledge with other organisations |
|
Improve the efficiency or function of the firm’s value chain |
|
Improve communication within the firm |
|
Improve or develop new relationships with external entities (other firms, universities, etc.) |
|
Increase business resilience and adaptability to change |
|
Improve working conditions, health or safety of the firm’s personnel |
|
Implement a new business model |
|
Contribute to the development of standards |
|
Economy, society or environment |
|
Reduce negative environmental impacts /deliver environmental benefits |
|
Improve public health, safety or security |
|
Improve social inclusion |
|
Improve gender equality |
|
Improve quality of life or well-being |
|
Comply with mandatory regulations |
|
Comply with voluntary standards |
8.16. Outcomes are only observable if they occur within the observation period for data collection; some effects may only occur after this period and consequently will be unobservable. It is not recommended to either extend the length of the observation period to more than three years or to collect outcome data for innovations that occurred before the observation period. Although both approaches could produce a more complete picture of innovation outcomes, they will also decrease data reliability due to a decline in the accuracy of the respondents’ ability to recall past objectives. Furthermore, collecting outcome data for innovations before the observation period could damage the logic of data collection and negatively influence responses to other questions.
8.17. In addition to the basic objectives and outcomes listed in Table 8.1, data can be collected on the relationship between innovation and business strategies, including the contribution of innovation to the firm’s business strategy (see subsection 5.3.1), the extent to which innovations require substantial internal changes in the firm, and the effects of innovation on the market in which a firm operates. Relevant data can be collected for objectives only, or for both objectives and outcomes, as described above. All strategic innovation objectives or outcomes should be measured on an ordinal scale.
8.18. Table 8.2 provides options for data collection on the objectives or outcomes of innovation in relation to a firm’s business strategy. The first set of innovation objectives and outcomes concerns how firms position their product innovations in their market. Relevant strategies include a focus on distinct market segments (specialisation), the diversification or extension of existing offerings (diversification), and solutions for specific customers (customisation). Objectives and outcomes for internal capabilities include improvements in the skill levels of employees, for instance to enhance absorptive capacity (see subsection 5.3.4), more efficient or effective methods for organising innovation activities, and methods to manage risk.
8.19. Innovation objectives can also be part of a firm’s strategy in respect to its competitors (see subsection 5.3.1). For example, a firm can focus on imitation or adoption, first-to-market strategies, or technology, design or cost leadership. A focus on imitation or adoption is a “follower” strategy in which a firm’s innovations lag behind those of its competitors. Conversely, a firm that pursues a leadership strategy seeks to remain ahead of its competitors. Leadership can be based on the design characteristics or technical functions of product innovations, or on quality or cost advantages from business process innovations. A first-to-market strategy can be based on imitating goods or business processes in other markets, or on technology, design or cost leadership.
|
Positioning a firm's products in the market |
|
Strengthen the position in distinct market segments |
|
Diversify or extend existing product offerings |
|
Develop solutions for specific customers |
|
Establish a new business model |
|
Internal capabilities and organisation |
|
Upgrade employee skills |
|
Organisation of innovation activities |
|
Managing risks that can impede innovation (security and cyber risks, etc.) |
|
Positioning a firm in respect to its competitors |
|
Imitate or adapt competitors’ innovations |
|
First-to-market good or service innovations |
|
First in market to use business process innovations |
|
Technology leadership |
|
Design leadership |
|
Cost leadership |
8.20. Innovation can have major impacts on the structure and dynamics of markets, such as driving competitors out of a market or blocking the entry of new competitors, for instance as a result of significant cost advantages, novel product characteristics, or network effects. Other market-transforming outcomes include changes to the business strategies of suppliers or other businesses that use the firm’s innovations. Changes in the business models of other firms can occur when an innovation renders some products or processes obsolete, or when a firm creates a novel online platform that other firms can use.
8.21. Information on the market impacts of a firm’s innovation strategies is of high relevance to policy. However, respondents may be unwilling to comment on the effects of their own firm’s strategies if they have the potential to contravene existing legislation, for example through anti-competitive behaviour. Consequently, it could be preferable to ask basic and neutral questions on the general effects of innovation by all firms active in the respondent’s markets, as shown in Table 8.3.
|
Change in the number of competitors in the firm’s market (increase/decrease/no change) |
|
Change in capital and human resource investments required to enter the firm’s market (increase/decrease/no change) |
|
Change in business strategies of suppliers active in the firm’s market (yes/no) |
|
Change in strategies of business users of the products in the firm’s market1 (yes/no) |
|
1. Only relevant to firms in markets selling to other businesses. |
8.22. Quantitative outcome measures for both product and business process innovations are of interest for three reasons. First, quantitative data are required for research on the economic significance of innovations for the innovative firm and for the markets where the innovations are sold. Second, these data can be used to analyse the effectiveness and efficiency of innovation expenditures and the effects on innovation outcomes, of how firms organise their innovation activities (for example their use of collaboration, information sources, methods to protect their intellectual property and receipt of public funding support). Third, quantitative outcome data are relevant to research on the impacts of innovation on other organisations, the economy, society and the environment.
8.23. The “innovation sales share” indicator can be defined as the share of a firm’s total sales in the reference year that respondents estimate is due to product innovations. It is an indicator of the economic significance of product innovations at the level of the innovative firm (Brouwer and Kleinknecht, 1996). In addition, data on the innovation sales share at the firm level can be aggregated to measure the share of sales from product innovations in the total sales of a specific industry or market. Sales share data can also be used to estimate the share of total demand in an industry that is met by domestic product innovations, if data on total sales from imports and domestic production are also available.
8.24. Innovations can result in very low or no sales if the time between the innovation and the measurement of sales is relatively short. Several factors will influence the time gap between product innovation and sales, including when the innovation occurred during the observation period and the time required to market and sell an innovation. Customised and expensive machinery are likely to be pre-sold (for instance aircraft), while some consumer products could experience a slow, gradual uptake in sales. On average, questions on innovation sales are likely to obtain better results if a three-year observation period is used compared to a one-year period.
8.25. It is recommended to collect data on the innovation sales share as an output measure of product innovation. It is further recommended to collect the sales share of product innovations (both new and improved products combined) for the following three types of markets (the responses should add up to 100%):
product innovations introduced during the observation period that were new to the firm’s market
product innovations introduced during the observation period that were only new to the firm
products that were unchanged or only marginally modified during the observation period.
8.26. Under some conditions it may be possible to disaggregate the innovation sales share by type of product innovation (goods or services), or by the location of sales (domestic or foreign markets). However, disaggregation by type of innovation will be difficult for firms that combine goods and services into a single product, such as when capital equipment manufacturers combine equipment sales with a service maintenance contract.
8.27. A useful disaggregation for research and policy is by the level of novelty, as in the example given above. Other methods of disaggregating by novelty include:
sales from new products or improved products
sales from world-first, market-first, or only first to the firm innovations (see subsection 3.3.2)
sales from innovations that are not available from any of the firm’s competitors, or from innovations that are identical or very similar to products already offered by competitors.
8.28. Respondents may find it difficult to provide an exact figure for the innovation sales share. An alternative is to provide response categories such as “0%,” “more than 0% to less than 5%”, “5% to less than 10%”, etc. The response categories need to be narrowly defined to provide useful data.
8.29. Information on the innovation sales share by type of market is useful for differentiating between the diffusion of product innovations that were previously available in the firm’s markets and product innovations that are market novelties. In addition, accurate interpretation of the share of sales from market novelties requires data on the geographic market where these products were sold. The degree of novelty is likely to differ if the product innovation is only new to a local market compared to a national or international market. Respondents can be asked if any of their new-to-market product innovations were new to their local, regional or national markets, or were a “world-first” product innovation (see subsection 3.3.2). It is also of value for research on capabilities and profiles (see subsection 3.6.2) to collect data on the innovation sales share of “world-first” product innovations.
8.30. The innovation sales share is affected by the speed of change in technology and demand in a firm’s market, with high rates of change resulting in short product life cycles. These and other external factors that can lead to short product life cycles are discussed in subsection 7.4.2.
8.31. A quantitative output indicator for product innovation is the number of product innovations during the observation period. This needs to be measured cautiously because respondents can find it difficult to estimate innovation counts, particularly for large firms with multiple innovations, highly complex products containing several sub-systems, or multiple products that can be subject to substantial or minor variations. To address these issues, data collection for innovation counts should use predefined categories (e.g. 0, 1, 2, 3-5, 6-10, 11-20, more than 20) and instruct respondents not to consider minor variations of the same product as different product innovations.
8.32. Count data on the number of product innovations is useful for interpreting data on the objectives and outcomes of innovation. For instance, the variety of innovation objectives is likely to be positively correlated with the number and diversity of product innovations. Indicators on the share of innovation projects that are completed during the observation period can also be calculated from count data for the number of innovation projects (see subsection 4.5.2).
8.33. Data on the economic significance or market success of product innovations can be collected by asking respondents for their firm’s general performance expectations (in terms of an increase in sales or profits), and the share of product innovations that met these expectations. Questions on performance expectations and outcomes for a change in sales or profits can use predefined response categories (e.g. “0%”, “more than 0% to less than 25%”, “25% to less than 50%”, “50% to less than 75%”, “75% to less than 100%”, “100%”).
8.34. Other quantitative outcome indicators for product innovation include the profit margin of product innovations and the market share of the firm’s product innovations out of all sales in the market for similar products (including the sales of products sold by competitors). Both indicators provide a better measure of the economic and market success of product innovations than the innovation sales share. The profit margin (degree of markup) is a measure of economic success that is positively correlated with the competitive advantage of the firm’s product innovations over other products offered in the same market. Similarly, a high market share indicates that a product innovation is able to outcompete offerings by other firms in the market. In contrast, a high innovation sales share for product innovations can still result in lower economic advantages to the firm, for instance when a firm ceases to sell older products or if a firm sells high volumes of a product innovation at low profit margins.
8.35. Respondents can find it more difficult to provide data on the profit margin or market share of product innovations than for the innovation sales share, particularly if the firm has a large number of product innovations with varying profit margins and market shares that need to be averaged. In addition, respondents can regard data on the profit margin and market share as highly sensitive. Data collection can reduce the response burden by asking for relative measures, such as the difference between the average profit margin for product innovations and the average profit margin for other products. Another option is to only collect data on the profit margin and market share for the firm’s most important product innovation (see Chapter 10).
8.36. In comparison with product innovations, respondents can face greater difficulty in providing estimates of quantitative outcomes for business process innovations. Data on the savings from business process innovations are often not collected by firms. Furthermore, business process innovation can relate to very different areas of operations, requiring different indicators for each type of business process (Davenport, 1993). An alternative is to collect quantitative data on the firm’s most important business process innovation (see Chapter 10).
8.37. A relevant indicator for some types of business process innovations is the percentage of a firm’s personnel who were directly affected by these innovations during the observation period. This indicator is of value for measuring the influence of business process innovations across an organisation. It does not, however, provide any information on whether business process innovations were successful or if they had any positive or negative effects on operations.
8.38. A second indicator is the change in sales that can be attributed to business process innovation. This measure can be driven by efficiency-enhancing business process innovations that reduce costs or that enhance product quality. Respondents can be asked if business process innovations led – directly or indirectly – to an increase in sales, and, if so, the size of the increase using a predefined scale. Useful categories are: “0%”, “more than 0% to less than 1%”, “1% to less than 2%”, “2% to less than 5%”, “5% to less than 10%”, and “10% or more”. This indicator is conceptually similar to the innovation sales share indicator for product innovations.
8.39. Both of these quantitative outcome indicators for business process innovation are likely to be very difficult for respondents from large firms to estimate, or for specific types of business process innovations that are not directly used in production activities, such as in administration and management. The indicators are more suitable for small and medium-sized firms, or for a question that focuses on business process innovations that are directly linked to products. An example is the share of sales affected by business process innovations in production, delivery and logistics.
8.40. Many business process innovations aim to improve the efficiency of a firm’s operations, though it is usually difficult to map individual innovations to specific outcomes. Efficiency-enhancing innovations should, directly or indirectly, result in lower costs compared to the situation before their use or compared to business process innovations that did not improve efficiency. In order to quantify the cost reduction resulting from business process innovations, respondents can be asked if such innovations led – directly or indirectly – to a reduction in operating costs, and, if so, the size of the reduction (Piening and Salge, 2015). Questions on cost reduction should refer to costs per unit of output or per operation, in order to exclude scale-related cost changes from an increase or decrease in production or operations. To reduce response burden, predefined response categories should be used. Experience with this approach in surveys indicates that the response categories should be weighted to small differences, such as “0%”, “more than 0% to less than 2%”, “2% to less than 5%”, “5% to less than 10%”, “10% to less than 20%”, and “20% or more”.
8.41. Other business process innovations aim to improve the quality characteristics of processes, such as flexibility, adaptability, speed, precision, accuracy or customer-friendliness (relevant to many business processes for delivering services). In some cases, quality-enhancing business process innovations can increase unit costs, but these additional costs can be matched or exceeded by an increase in the value of the resulting output.
8.42. Quantitative indicators on quality-enhancing business process innovations have been developed as part of quality management (Powell, 1995). These cover improvements in the timeliness of business processes due to innovations (lead time, processing time, on-time delivery) and improvements in the quality of outputs from business process innovations (customer satisfaction rate, defect rate, accuracy rate, reworking rate, scrap rate). Quantitative indicators for many of these outcomes require individualised scales built into each question, for instance the share of products delivered on time, the share of customers that were satisfied with the process, the share of scrap in total production volume, or the share of products that had to be reworked. Other indicators include improvements to process complexity (the number of steps) and employee satisfaction. Some of these quality indicators are designed for manufacturing processes that produce distinct units of output and are less relevant for business process innovations in continuous manufacturing industries such as chemicals, or in service industries. Other indicators can be applied to all industries, such as the customer satisfaction rate (share of customers that are usually satisfied with the good or service), the accuracy rate (share of operations that produce the intended process result) or the employee satisfaction rate. Many of these indicators are difficult to apply or less relevant (e.g. the scrap rate) to firms in service industries.
8.43. The choice of a subject or object method for data collection will have a substantial effect on the information obtained for innovation objectives and outcomes. The subject approach requires asking firms about the objectives or outcomes of all innovations (or innovation activities) during the observation period. If the objectives or outcomes differ among innovations (or innovation activities), it will be difficult for respondents to derive an average level of importance for each objective or outcome. Conversely, the object approach (see Chapter 10), with a focus on a single innovation, will reduce the response burden and increase the accuracy of the data for specific objectives and outcomes, but at the expense of data for a broader range of objectives.
8.44. The inclusion of questions on outcomes in data collection assumes that respondents are able to assess the consequences of their firm’s innovations. For some outcomes, such as a change in sales, this assumption could be valid, whereas respondents could find it difficult to assess other outcomes, such as a reduction in environmental impacts outside the firm.
8.45. Questions that ask respondents about the performance effects of their firm’s innovations could be subject to biases in favour of positive effects, which can be more visible to respondents than the secondary effects of an innovation. For example, a product innovation could result in the hiring of new employees to develop, produce and market the innovation, but also cause a fall in the demand for other products of the same firm as customers shift to the new or improved product, resulting in the layoff of employees involved in the production and marketing of these other products. Respondents are more likely to recall the positive increase in employment due to the innovation than the negative employment effects from the innovation replacing other product lines. In addition, respondents may find it difficult to assess positive or negative indirect effects, for instance when an innovation reduces the sales of old products with a better safety record than the new product.
8.46. Some of the above issues can be addressed through the use of econometric methods that estimate innovation outcomes while controlling for the effects of possible biases (see subsection 11.5.2). Econometric methods have been developed for analysing productivity performance, employment outcomes, profitability, and measures of competitiveness. These analyses benefit from data on innovation outcomes as described in this chapter, such as sales from product innovations or the effect of business process innovations on sales or costs.
8.47. Recommendations for general data collection are given below. Supplementary data are suitable for specialised data collection exercises.
8.48. Key recommendations for data collection include:
8.49. innovation objectives and outcomes by area of influence (Table 8.1)
innovation objectives and outcomes for business strategies (Table 8.2)
innovation sales share in total business sales.
8.50. At the time of publication, there is a serious lack of quantitative outcome data for business process innovation, which significantly hinders understanding of the role of business process innovation in economic growth. Consequently, a key recommendation is to experiment with one or more of the proposed indicators discussed in subsection 8.3.2 above.
8.51. Supplementary recommendations (given space or resources) include:
counts of product innovations
major impacts of innovations for markets (Table 8.3).
Brouwer, E. and A. Kleinknecht (1996), “Determinants of innovation: A microeconometric analysis of three alternative innovation output indicators”, in Determinants of Innovation: The Message from New Indicators, Palgrave Macmillan, London, pp. 99-124.
Crépon, B., E. Duguet and J. Mairesse (1998), “Research, innovation and productivity: An econometric analysis at the firm level”, Economics of Innovation and New Technology, Vol. 7/2, pp. 115-158.
Davenport, T.H. (1993), Process Innovation: Reengineering Work Through Information Technology, Harvard Business School Press, Boston.
Piening, E.P. and T.O Salge (2015), “Understanding the antecedents, contingencies, and performance implications of process innovation: A dynamic capabilities perspective”, Journal of Product Innovation Management, Vol. 32/1, pp. 80-97.
Powell, T.C. (1995), “Total quality management as competitive advantage: A review and empirical study”, Strategic Management Journal, Vol. 16/1, pp. 15-37.