4
PREDICTION, PREFERENCE, AND IDEA MARKETS: HOW CORPORATIONS CAN USE THE WISDOM OF THEIR EMPLOYEES

Peter Koen

Stevens Institute of Technology

4.1 Introduction

British scientist Francis Galton, in 1906, came upon a weight judging competition at an exhibition in Plymouth (Galton, 1908). For sixpence you could wager a bet on the weight of an ox and receive an award if your guess was closest to the actual weight. A total of 787 guesses were received from butchers and farmers, who presumably were experts, as well as clerks and others who had no expert knowledge. Galton ran a series of statistical tests and found that, on average, the crowd guessed that the weight of the ox was 1197 pounds. The actual weight was 1198 pounds. The crowds' judgment was just about perfect. Francis Galton found that under the right conditions, groups are smarter overall than the smartest individuals in the group.

The first public virtual stock market, which is known as the Iowa Electronic Market (IEM), a not-for-profit organization run by the University of Iowa, predicts the outcome of presidential elections. Berg and colleagues (2008), based on an extensive analysis, found that IEM election predictions more than 100 days from the actual presidential election in 1988, 1992, 1996, 2000, and 2004, were more accurate 76 percent of the time than popular opinion poll organizations such as ABC, CBS, CNN, Gallup, Harris, or NBC. The traders were not even a representative sample of the voters, as they were primarily students and faculty of the University of Iowa. Overall, these results show that the IEM markets may provide a more accurate long-term forecasting tool than polls.

Another well-researched example (Spann and Skiera, 2003) is the Hollywood Stock Exchange (HSX, www.HSX.com). HSX issues MovieStock ahead of the actual release of a movie. The value of a share of stock represents the total of the U.S. box office receipts, in millions of dollars, over the first four weekends after a movie has been released to more than 650 screens. If a share of stock has a current value of $50, it implies that the box office receipts will be $50 million dollars during the first four weekends. Participants who think the share of the stock is undervalued (i.e., box office receipts will be more) buy shares. In contrast, players who believe that the box office receipts will be lower will sell shares. HSX has more than 725,000 registered participants with an average of 15,000 individual visitors each day. There are no financial rewards. However, the participants with the most accurate forecasts are ranked and displayed, creating an intrinsic motivation for them. The percentage error between the actual sales and the HSX predictions was 41 percent based on 140 movies (Spann and Skiera, 2003). This is compared to expert predictions from Box Office Report and Box Office Mojo, whose predictions for the same 140 movies were 53 percent and 35 percent, respectively. Thus, the crowds were smarter than Box Office Report, but not quite as smart as Box Office Mojo. A more recent study by Karniouchina (2011) reveals inefficiencies in the HSX trader algorithm, which would have further improved the actual HSX predictions to the extent that they would have been better than the experts.

In 2004, James Surowiecki published a popular book, Wisdom of the Crowds, to explain why large groups of people are smarter than any individual. If the crowd is big enough and diverse enough, you have access to so much more knowledge than you do if you ask just one expert or even a small team of experts. This is apparent on the popular TV show “Who Wants to be a Millionaire?” When a contestant is stumped, he or she can call an expert, who is usually a smart friend, or he or she can poll the audience. The experts get the answers right about two-thirds of the time, but the audience gets the answer right 91 percent of the time.

Based on these promising results, many companies have begun using virtual stock markets for business forecasting (Cowgill, Wolfers, and Zitzewitz, 2008) and crowdsourcing of new internal ideas (LaComb et al., 2007 Lauto et al., 2013; Hoyt, 2006; Spears et al., 2009; Soukhoroukova et al., 2012; and Taylor, 2006) as a methodology to open the innovation process to more constituents across the company. However, it is not obvious that success in large-scale settings such as IEM or HSX will extend to corporations. The underlying theory is based on having a large volume of participants (Slamka, Skiera, and Spann, 2013). In contrast, businesses require much smaller-scale applications. In addition, incentives for trading are much different in corporations. HSX provides extrinsic bragging rights with accompanying social notoriety to the winners while iPredict (www.ipredict.co.nz), Betfair (www.betfair.com), and InTrade (www.intrade.com) provide financial gain.

This chapter evaluates the use of virtual stock markets in corporations. An overview of the three different types of virtual markets—prediction, preference, and idea—used in corporations is presented in the next section. A conceptual model is then discussed in the subsequent section and is used as a reference for explaining why HSX, IEM, and InTrade achieve high forecast accuracy and the conditions required in corporations to achieve similar results. This chapter concludes with a discussion of the process for setting up a virtual stock market in companies as well as guidance on how to select a virtual stock market platform.

4.2 Virtual Stock Markets in Corporations: Prediction, Preference, and Idea Markets

Prediction, preference, and idea markets are lumped together in many articles and are called prediction markets. However, how they are used in corporations, their outcomes, and measurement accuracy are different.

Prediction markets, in this chapter, are defined as those markets that aggregate employees' information to forecast a specific market event. For example, how many users will Gmail have at the end of a month, three months, six months? Other examples are predicting monthly sales of a product three to six months into the future. Prediction markets forecast a specific market event, such as sales, software release dates, and the like, at some point in the future and have a real-world outcome which can be independently validated later with 100 percent certainty.

Preference markets, which Dahan and colleagues (2011) also refer to as “Securities Trading of Concepts,” use the wisdom of crowds to predict product preferences of future customers. For example, a trader is asked to determine the desirability between 11 different air bike pump products by buying and selling securities for each of the different concepts. Their trading will reveal which concepts are preferred by the market. The difference between preference and prediction markets is that preference markets are focused on determining market preferences on a yet-to-be-released product. In contrast, prediction markets forecast a future actual market event, such as actual product sales, using an actual completed product.

Idea markets represent a virtual platform where each idea is considered a stock in which employees can invest. Participants evaluate each idea and buy and sell the ideas that they regard as having the most value to the corporation. The share price of the idea provides management with the organization's collective view. The outcome of the process is a rank ordering of the ideas, providing management with a filter to select the most promising one to turn into a future project.

Conceptual Model of a Virtual Stock Market

A conceptual model which explains how virtual markets work is shown in Figure 4.1, which provides a guide to understanding the factors that firms need to explicitly address in order to effectively use these tools. The model consists of 8 inter-related factors each of which will be discussed below, and is an adaption of the model originally developed by Kamp and Koen (2009).

Figure 4.1: Conceptual model which explains the key variables and how they relate to the overall accuracy of the virtual stock market. The idea entry restriction and the dashed arrows only apply to the idea market.

Accuracy

The value of the virtual stock market to the corporation is related to its ability to accurately predict the future event. However, the measure of success is different for prediction, preference, and idea markets. For prediction markets, the measure of success is the ability to predict the actual outcome. For example, how accurately can the market predict monthly sales for the new product three months in advance?

Measures of success for preference and idea markets are less straightforward. Dahan and colleagues (2011) used multiple measures to determine the accuracy of preference markets including correlation with conjoint analysis and preference over surveys and actual sales. They found that preference markets were correlated with conjoint analysis, which is a statistical technique to determine how people value different product features, but were not correlated with actual future sales. This is not unexpected as preference markets evaluate the customers' choices without respect to the product price, while in actuality, the buying decision also is constrained by the customer's budget. Accordingly, the most appropriate accuracy measure for preference markets was the correlation with conjoint analysis.

In a similar manner, investigators have developed multiple measures of success for idea markets, including acceptance of the market by the corporation, senior management judgment of the quality of the idea, judgment of the idea by an expert panel, and senior management commitment to fund the idea to the next step (Soukhoroukova et al., 2012). Ultimately, the accuracy of the idea market is evaluated based on senior management's commitment to move the idea forward (Lauto et al., 2013).

Accessible Information

Participants need a sufficient amount of information in order to make accurate predictions (Forsythe et al., 1999). For example, predicting the winner of the presidential election a full year prior to the actual voting will not be very accurate. More information will be revealed as the actual voting day approaches.

Berg and colleagues (2008) found that the most accurate predictions of the 1988, 1992, 1996, and 2000 presidential elections occurred 31 days before the actual voting. More information increases the accuracy. A similar result was found in an extensive analysis of the HSX done by Karniouchina (2011). The study showed that prediction accuracy decreased when there were concurrent movie releases. The investors needed to combine large amounts of information from the concurrent offerings in order to understand the effects of competition.

What this means for firms is different for prediction, preference, and idea markets. For prediction markets, it means that sufficient information and specificity for the item being forecasted needs to be provided in the prediction campaign. For preference markets, it means that product details including embedded attributes for each product need to be provided with sufficient clarity and specificity so that the traders can make clear choices. For idea markets, it means that each of the ideas needs to have an appropriate level of information provided about it so that the traders can judge the potential attractiveness and risk profile of the idea when compared to other ideas.

Truth-Seeking Trading Behavior

A frequent criticism of virtual stock markets is that the stock price can presumably be manipulated by biased traders who are either ill-informed or are motivated to intentionally manipulate the value. However, the election markets, HSX and InTrade, demonstrate that creating accurate markets is possible. Forsythe and colleagues (1999) explain why. Markets, such as IEM, are accurate based on the marginal trader hypothesis. Marginal, or more experienced traders, have more expertise in trading and are more knowledgeable about the market. The experienced trader drives the prices to the correct values and profits from the mistakes of the average or less informed trader. For every dollar an ill-informed trader loses as a result of a mistake, the experienced trader gains a dollar. All traders need not be experienced traders, but there need to be enough to have a personal stake in the outcome to move the share price to its correct value.

What this means for firms is that it is desirable to increase the proficiency of all traders. Training should be provided to all users to increase the knowledge of participants about how the virtual stock market works. Secondly, it is also important to ensure that there are a sufficient number of participants with adequate domain knowledge in the area of trading so that they can adjust for the mistakes of the less knowledgeable trader to assure that the share price moves to its correct value.

Domain Knowledge

The marginal trader hypothesis requires that experienced and knowledgeable traders participate in the market. This may seem obvious, but many of the virtual stock markets run in corporations do not properly incentivize traders with appropriate levels of domain knowledge to participate. Even with experienced traders, some inaccuracies are expected. Karniouchina (2011), in evaluating the accuracy of the HSX, found that movies with stars tend to be overvalued and thrillers are undervalued. Thus, even movie buffs with considerable domain knowledge can be both under- and over-optimistic traders in the market. The dot.com bubble of 1997–2000 and the subprime mortgage recession in 2008 suggest that markets can always have excessive volatility. The need for traders to have adequate levels of domain knowledge in the prediction, preference, or idea markets in corporations represents a necessary factor needed for truth-seeking behavior, as indicated by the arrow in Figure 4.1 linking these two variables.

What this means for firms is that some of the traders in the virtual stock market need to have adequate domain knowledge.

Trading Experience

While many employees are familiar with the traditional stock market, few are familiar with virtual stock markets. James Surowiecki, author of The Wisdom of Crowds (2004), indicates in a McKinsey Quarterly article (Dye, 2008) that one of the shortcomings of predictions markets is “ . . . that a lot of people inside organizations don't find the market mechanism intuitive or easily understood. They find it very challenging to use, which limits the pool of people who participate . . .” (p. 89). In the same article, Best Buy indicates that they continue to teach employees how to use prediction markets. Google has found that new employees and inexperienced traders suffer from overpricing of favorites, optimism, and extreme outcomes (Cowgill et al., 2008). Accordingly, this variable will increase the truth-seeking behavior of the traders as indicated by the arrow in Figure 4.1.

What this means for firms is that some form of training on how to trade is needed for all participants in the virtual stock market.

Number of Participants

Virtual stock markets used in corporations have fewer traders when compared to public ones such as the HSX or IEM markets. Fewer traders leads to inefficient markets since fewer assessments are done regarding the outcome. Lundholm (1991) and Van Bruggen and colleagues (2006) showed in very well-controlled experiments with traders who had domain knowledge, trading experience, and accessible information, that only six active traders per commodity were necessary to achieve accurate results. Markets with six traders averaged 2.6 trades per commodity per period where the trading was done for nine periods (Lundholm, 1991, p. 510). These results indicate that virtual stock markets in corporations may need even more than six participants for each commodity since few of the employees enrolled in the corporation's market do any trading. Thus, the number of traders also directly affects the truth-seeking trading behavior, as shown in Figure 4.1. The more traders, especially those with domain knowledge and trading experience, the more likely they are to drive the price to the correct value and compensate for ill-informed traders. A link is shown between the number of traders and the amount of accessible information. This link plays a more important role in idea markets. Many idea markets allow traders to ask questions and interact with other traders during the trading session. Such interaction will increase the amount of accessible information, which in turn will increase the accuracy of the idea market.

What this means for firms is different for prediction, preference and idea markets. Prediction markets trade around one concept while preference markets trade around 6 to 10 concepts. Accordingly, the number of traders typically exceeds the minimum of 6 traders per commodity required for accuracy. In contrast, idea markets are often populated with too many ideas for the number of traders in the market.

Participation Incentives

Wolfers and Zitzewitz (2004) assert that “Even well designed . . . ” virtual stock markets “ . . . will fail unless a motivation to trade exists” (p. 121). Some corporations assume that financial rewards are necessary for participation. For example, Google rewards traders between $25 to $100 per quarter depending on level of activity (Cowgill et al., 2008, p. 4). General Electric awards $25 gift certificates in random drawings to active traders (LaComb et al., 2007). Play-money exchanges such as IEM and HSX obtain accurate forecasts without a monetary prize. Diemer (2010), in fact, found that play-money predictions were more accurate than real money ones in ipredict (www.ipredict.co.nz), a trading platform where traders can wager real money to predict future political and economic events. The play-money predictions had a forecasting error of 14 percent while the real money error was 19 percent. There is a substantial research stream (Amabile, 1997; Ederer and Manso, 2013) that suggests that tasks that require innovation and creativity may in fact be undermined by financial incentives that reward performance. In a carefully run experiment (Ederer and Manso, 2013) simple incentives focused on pay for performance drove incremental performance in contrast to tasks requiring more innovation and creativity. In addition, many corporations are concerned that monetary incentives may violate gambling laws or even require a disclosure under the regulatory requirements of the Securities and Exchange Commission. These results suggest that motivation to participate in virtual stock markets in corporations should be based on intrinsic motivations. Bo Cowgill, from Google, recounts that he forgot “ . . . to pay out the small cash prizes . . . ” for a recent prediction market and “ . . . nobody noticed. But everyone notices when the T-shirts that show who won one don't come” (Dye, 2008, p. 89). Novozymes (Lauto et al., 2013) gives symbolic awards at an R&D ceremony for participants with the greatest number of shares in the ideas which will be moved forward, as well as the traders who make the most number of comments.

What this means for firms is that the virtual stock market effort needs to be an important corporate initiative that is championed by senior management. Formal recognition of participation by the corporation is far more important than any monetary initiative to participate.

Idea Entry Restriction

As indicated in Figure 4.1, participants need accessible and clear information about the stock they are trading in order to accurately determine how they will trade. This is less of an issue in prediction and preference markets, since the number of stocks in these markets is typically constrained to a small number. In prediction markets, trading occurs around one question. For example, future sales of a product, which allows the manager of the initiative adequate time to describe the opportunity clearly. The number of stocks in preference markets is constrained to a manageable number. For example, the comparison of features in 11 different air bike pump concepts. In contrast, idea markets often contain a much larger number of stocks, each representing a particular idea or new project. For example, Soukoroukova and colleagues (2012) report that 100 ideas were traded in a $3 billion B2B company. Obtaining clear and accurate descriptions for this many ideas represents a challenge for many organizations, despite the fact that Soukoroukova and co-workers (2012) report that the 100 ideas in the market were filtered from the original 252 submitted. In addition, concepts submitted to an idea market need to not only be filtered for clarity, but also need to be evaluated for similarity and/or replication to other ideas submitted so that the traders are able to make clear choices between the different ideas. In addition, several of the ideas may be inappropriate and/or unresponsive to the corporation's initiative. Rite-Solutions, a small company with reported revenue of $23 million, devotes considerable effort to prescreening ideas before they are entered into the market (Hoyt, 2006). Rite-Solutions has had considerable success with the idea market, reporting that one of the ideas for applying three-dimensional visualization technology to help sailors and security personnel make decisions in emergency situations, accounted for 30 percent of sales in 2005 (Taylor, 2006).

What this means for the firm is that ideas need to be pre-screened for relevance and replication before they are placed into the market.

Pricing Algorithm

Stock prices in public trading markets such as the New York Stock Exchange use a double auction mechanism which directly matches the purchase price and sale orders of participants. Both the buyer and seller determine the price and then execute a sale order. Continuous double auctions are complex and labor intensive. The majority of virtual stock markets use a software algorithm generically called an Automated Market Maker (AMM) which provides the ability to sell shares at any point in time and creates unlimited market liquidity. Slamka and colleagues (2013) evaluated four types of AMMs: logarithmic market scoring rules, dynamic pari-mutual markets, dynamic prize adjustments, and the HSX mechanism. It is beyond the scope of this chapter to review the different types of AMMs as they will be embedded in the software platform the corporation uses (see Figure 4.5 and Table 4.5). Nevertheless, choosing the right AMM is not trivial, as the Slamka group's (2013) study indicated that logarithmic scoring rules and dynamic pari-mutual market makers were the most robust and attained the highest forecast accuracy. Corporations may want some ability to adjust the AMM algorithm based on their own experiences. Spann and Skiera (2003) found that they could obtain more accurate predictions if they modified the HSX AMM algorithm to adjust for overestimates for large movies and underestimates for small movies. Accordingly, the number of participants in the model in Figure 4.1 directly affects the pricing algorithm.

What this means for firms is less obvious than in previous sections since the trading algorithm is hard wired into the software and is often proprietary to the vendor. Nevertheless, the user should try out several different software platforms to determine the robustness of the tool, the characteristics of the trading algorithm and the degree to which parameters in the algorithm may be adjusted. The buyer needs to evaluate the pricing algorithm used as many idea software platforms use rudimentary up or down voting schemes.

4.3 How Well Do Prediction, Preference, and Idea Markets Work in Corporations?

Prediction Markets

The documented cases for prediction markets in corporations are shown in Table 4.1. The results from all six companies show high accuracy. The methodology and processes used in each of the cases were compared to the variables illustrated in Figure 4.1. For the most part, all of the factors are congruent with the model, which explains why high accuracy was obtained. Accordingly, these results further confirm the validity of the model and help form the basis of the key factors that corporations need to address when using virtual stock markets.

Table 4.1 Accuracy of Prediction Markets

Company	Error in Predicting the Actual Event	Domain Knowledge	Participation Incentives	Number of Traders	Reference
Best Buy	0.5% error for gift card sales	Random	Corporate initiative	350 people	Dye (2008)
Google	7.2% average error for 100% of the markets	Random	Corporate initiative plus small reward	1463 people across all markets	Google (Cowgill et al., 2008)
German retail company	19% forecast error for the sales of each of the company's 1000 outlets	Knowledgeable participants	Corporate initiative	100 people	Ivanov (2009)
Global Agri-Business	0.5% error in predicting the demand for seeds one year in advance	Knowledgeable participants	Corporate initiative	123 participants	Ivanov (2009)
HP	19.3% error across 8 markets	Knowledgeable participants	Controlled experiment done outside of working time plus a small reward	19 avg/market	Plott and Chen (2002)
Intel	2.7% error for 75% of the markets	Knowledgeable participants	Corporate initiative	20–25 people/market	Hopman (2007)

• Trading Experience. While not shown in Table 4.1, all companies invested time in teaching their traders how to use the virtual stock market platform. For example, participants are trained (Ivanov, 2009) in a two-hour workshop on how to use the system, how profits are made, and the incentive system being used to determine success. The virtual stock market is then open for a two- to three-week training period where participants can observe how the system performs and make comments on the specificity of the questions and performance of the system. The feedback from participants is then collected and incorporated into the system before the stock market is formally opened for trading.
• Domain Knowledge. Except for the studies done at Best Buy and Google, all of the companies carefully selected participants with the right amount of domain knowledge. Ivanov (2009) describes a detailed process used by a German retail company for selecting participants. The process begins by selecting 20 people who have both market and competitor knowledge. An additional 70 people are included from sales managers throughout Germany who have close contact with the customers' tastes and preferences. An additional 10 people from the warehouse are also included. While they have no direct customer contacts, they are familiar with the actual orders received from the different outlets. However, this same detailed process may not be needed at Best Buy and Google since their prediction markets required less sophisticated and more general knowledge than the German retail company. For example, Best Buy was interested in the percentage of gift card sales. Google was interested in the number of people who were going to use the Gmail system.
• Participation Incentives. Except for the HP case, all of the examples were important to the corporation. For example, participation in Best Buy's prediction markets was championed by the CEO. In contrast, the HP case was a laboratory experiment with no involvement from senior management and where the participants received a small amount of money for participating.
• Number of Traders. All of the cases had at least 20 to 25 traders in the market with all of the traders being active, except for Best Buy and Google, where the number of active traders was not discussed. The number of traders always exceeded 6, the recommended minimum number determined by Lundholm (1991) based on a set of carefully controlled laboratory experiments, which helps explain the high accuracy of the results.
• Accessible Information. Sufficient information, while not specifically discussed, was available in all of the cases.
• Pricing Algorithm. No information was given on the pricing algorithm, though this does not become an important issue (Healy et al., 2010) until the number of traders is less than 12. All cases exceeded this number.

Preference Markets

There are no case articles on preference markets being used in corporations. The most extensive analysis was done by Dahan and colleagues (2011), where tests were done on MBA students, as shown in Table 4.2. As discussed earlier, the measure of accuracy was the correlation with conjoint analysis, in contrast to predicting actual sales, as the same study indicated poor correlations. The methodology and processes used in the two cases, where correlation with conjoint analysis was evaluated, was compared to the variables illustrated in Figure 4.1. In the first case, the students chose the most desirable features when comparing 11 different bike pump concepts. In the second case, the students compared six attributes of new Wii game concepts developed by students at the University of California.

Table 4.2 Accuracy of Preference Markets

Correlation with Conjoint Analysis	Number of Attributes Evaluated	Domain Knowledge	Participation Incentives	Number of Traders	Reference
Correlations of 0.75 and 0.83 for Bike Pumps	9	Assumed	None	28 MBA students	Dahan et al., (2011)
Correlations of 0.44, 0.75, and 0.75 for Wii Game	Between 8 and 11	Assumed	None	Between 35 to 58 MBA students	Dahan et al., (2011)

• Trading Experience. While not discussed in the article, some training was presumed.
• Domain Knowledge. The students chosen were executive MBA students. Carefully selecting participants is probably not necessary, in these cases, as most of the students have familiarity with bike pumps and the Wii.
• Participation Incentives. No participation incentives were used.
• Number of Traders. The number of traders varied between 28 and 58 as indicated in Table 4.2. The trader-to-attribute ratio was calculated for all of the five examples and varied between 3.1 and 6.8, which are close to the minimum number of traders per attribute.
• Accessible Information. The article provides evidence that additional information about the product concepts was provided, as the traders became familiar with the concepts by viewing detailed product information.
• Pricing Algorithm. No information was given on the pricing algorithm, though this becomes an issue due to the low trader-to-attribute ratio.

The conceptual model would have suggested problems with accuracy, since the traders were not chosen based on their domain knowledge and the number of traders per attributes was low. Despite these issues, the correlations were excellent except for one of the Wii experiments where a correlation of 0.44 was obtained. The lack of domain knowledge is probably not an issue, as most of the MBA participants understood the product attributes they were trading. The low number of traders (i.e., 5.3 traders/attributes when the correlation was 0.44) could account for the lack of significance.

Idea Markets

There have been five case articles on idea markets being used in corporations as indicated in Table 4.3. As discussed earlier, the measure of accuracy was whether an idea was funded for further development. All of the cases met that standard. However, it was a forgone conclusion that the two cases from GE would meet this standard, as funding was a participation incentive for the top idea. Only one company, Rite-Solutions, has reported an increase in actual sales from one of the ideas. Novozymes results also appear to be robust as they report that “. . . two ideas have been launched as projects,” with one representing an innovation that holds “. . . the promise of opening up an entirely new enzyme application in a new field” (Lauto et al., 2013, p. 21). The results for the other ideation markets were not as robust. The studies were again compared to the variables in Figure 4.1.

Table 4.3 Accuracy of Idea Markets

Company	Number of Ideas Funded for Further Development	Number of Ideas	Idea Entry Restriction	Domain Knowledge	Participation Incentives	Number of Active Traders
B2B company with $3 billion is salesa	“ . . . too early to evaluate commercial success . . . ” p. 110	100	Yes	All employees in division	Monetary (Prizes between $100 to $1500 to best traders)	157
GE Computing and Decision Science Technology Centerb	leadership team “ . . . ranked the winning idea much lower than the market . . . ”(p. 254)	62	No	All employees in division	Monetary (Top trader received Apple iPad and top idea received $50,000. Participants could not trade during work hours)	85 total (Active traders are fewer)
GE Energy sub- businessc	GE was “ . . . extremely pleased with the results . . . ”	32	Yes	All employees	Monetary (Top trader received Apple iPad and top idea received $50,000)	110
Novozymesd	“. . . two ideas launched as projects . . . ”	222	No	Selected within the R&D staff	Non-monetary (Corporate initiative with final ceremony and symbolic awards)	101
Rite-Solutionse	One idea is responsible for 30 percent of sales	50	Yes	All employees	Non-monetary (Corporate initiative)	160

^a Soukhoroukova et al., (2012)

^b LaComb et. al. (2007)

^c Spears et al. (2009)

^d Lauto et al., (2013)

^e Hoyt (2006) and Taylor (2006)

• Trading Experience. Training was provided in all of the cases indicated in Table 4.3.
• Idea Entry Restriction. All of the companies, except Novozymes, provided a filtering mechanism for the initial ideas.
• Domain Knowledge. The idea market was open to all employees, except in the case of Novozymes, who carefully selected the traders participating across the company's eight R&D sites in six countries. Thus one can surmise that many of the traders did not have adequate expertise to properly evaluate the ideas to which they were trading in the other companies.
• Participation Incentives. The idea market was an important company initiative in both the Novozymes and Rite-Solutions cases. For example, the Chief Scientific Officer of Novozymes sent an open letter to the participants to “. . . engage actively” in the idea market. In addition, prizes were rewarded at the end of the idea market at a ceremony involving both the participants and the R&D management (Lauto et al., 2013, p. 20). In contrast, traders at GE were asked to trade before or after work, during lunch, or for only a few minutes during work hours. In addition, providing guaranteed research funding to the best idea in the GE cases may have biased the results as the idea originator bought their own ideas at above-market prices and at high volumes. Novozymes (Lauto et al., 2013) prohibits traders from investing in their own idea.
• Number of Traders. The number of traders varied between 85 and 160 as indicated in Table 4.3. The trader-to-idea ratio was below 3, the minimum suggested value, in all cases except Rite-Solutions and the GE Energy business.
• Accessible Information. The amount of accessible information varied considerably between the studies. Rite-Solutions was an exemplar where each submitter needed to find two champions to support their idea before it could be listed on their market. The submission process also required each idea to answer the following six questions: (1) Is this idea in our path? (2) Does it leverage what we know? (3) Does it allow us to learn new things? (4) Would we use this? (5) Do we know anybody else who would use it? and (6) Will this make us a better company?
• Pricing Algorithm. No information was given on the pricing algorithm. This can become an issue due to the low trader-to-attribute ratio.

Overall, the results, when compared to the prediction and preference markets, appear to be much weaker. This weakness can be attributed to three factors.

• Low Trader-to-Idea Ratios. In order to achieve truth-seeking behavior, more experienced traders will drive the prices to the correct value in order to make larger profits to compensate for the overexuberance of idea submitters who pump up their price or have inadequate domain knowledge and make whimsical trades. However, there were not enough traders for each idea in the market to correct these biases.
• Lack of Adequate Domain Knowledge. The marginal trader hypothesis also requires experienced traders to participate so that they can move the idea price in the correct direction. Only Novozymes carefully selected the R&D staff who participated. In contrast, the participants in the prediction market (see Table 4.1) all had appropriate levels of domain knowledge.
• Inappropriate Participation Incentives. The idea market was a clear corporate initiative in only the Rite-Solutions and Novozymes cases. These were the two cases that yielded the most robust results, despite the relatively low trader-to-idea ratio.

Kickstarter (www.kickstarter.com), a crowdfunding platform, and Threadless (www.threadless.com), which is an online community of artists, are often cited as examples of successful idea platforms. Kickstarter is an Internet platform where the crowd funds new ideas through pledges. The backers receive the product or might get to attend the premiere of a film project if the project is fully funded. Threadless sorts through design ideas created by an online community. About 1000 designs are submitted online every week with 10 selected based on the average score and community feedback. The selected designs eventually end up on articles of clothing such as T-shirts or sweat shirts, which can be purchased on the Internet. Designers whose work is printed receive $2000 in cash and $500 in Threadless gift cards.

Why do these platforms work so effectively when similar ideation markets in companies perform so poorly? The reasons can be directly attributed to the three factors discussed above: trader-to-idea ratios, domain knowledge, and incentives as indicated in Table 4.4. In both Kickstarter and Threadless there are a large number of traders for each idea. In addition, the traders self-select the idea they trade based upon their own domain knowledge of the value of the idea to themselves. And in both cases their incentives are intrinsic. The traders in Kickstarter get the product or attend the premiere of the film. The traders in Threadless see the design they voted on in clothing. In contrast, participants in corporate idea markets are not often carefully selected, so many may have inadequate domain knowledge. In addition, the incentives for trading are weak, such as small monetary incentives. Idea markets in corporations could dramatically improve if they increased the trader-to-idea ratio, assured participants had adequate domain knowledge, and linked participation to compelling corporate initiatives.

Table 4.4 Comparing Idea Market Factors in Corporations to Kickstarter and Threadless

Problems in Idea Markets in Corporations	Trader-to-Idea Ratio	Domain Knowledge	Participation Incentives
Corporations	Low	Participants are not carefully selected	Often no clear corporate initiative.
Kickstarter	Very High	Participants self-select on areas they understand	Participants are rewarded with the product or the ability to attend the premiere of a movie in which they invested
Threadless	Very High	Domain knowledge is not required. People choose designs based on their own artistic sense.	Participants are rewarded by seeing the design they voted on being chosen for printing.

4.4 Implementing a Stock Market in Corporations

Achieving successful virtual stock markets in corporations is both labor intensive and time consuming, as success can only be achieved by ensuring that the activities are supported and encouraged by senior management, combined with the participation and motivation of a sufficient number of employees with the appropriate level of expertise. The overall process for implementing prediction, preference, and idea markets is discussed below. A subsequent section focuses on choosing and using a virtual stock market software platform.

Prediction Markets

The overall procedure for implementing a prediction market is shown in the five-step process illustrated in Figure 4.2 below. The first step in the process is to identify what problem is to be forecasted. The best results are obtained when the boundaries of the problem being forecasted or solved have little ambiguity. The second step involves setting up a task force to manage the process. As with any large project within a corporation, the effort requires senior management, champions, and staff.

Figure 4.2: Implementation Process for a Prediction Market

• Champions. Someone in the organization, preferably in senior management, needs to champion the initiative. For example, the Chief Scientific Officer at Novozymes was the champion.
• Project Team. A project team (Step 2), typically appointed by senior management, has the responsibility for establishing the strategic and management issues associated with the project. This team needs to identify the participants (Step 2a) with the right expertise who need to participate. The prediction market will not work if there is not a sufficient level of participation by domain experts. In addition, the team should reach out to other participants with different backgrounds to ensure a sufficient level of heterogeneity.
• Virtual Stock Market Expert. This is the person responsible for setting up the platform, developing training sessions, running the virtual stock market, and monitoring the results on a continuous basis. He or she also needs to make adjustments and/or council employees when tournament behavior of savvy investors becomes an issue. This becomes more of an issue in markets where the number of traders is low, as is frequently the case in idea markets. It is important to minimize inappropriate trading as the less-experienced traders can become jaded by the results and will criticize the virtual stock market. This virtual stock market expert will be responsible for the overall market design. For example, how much play money each participant should get, how sensitive the market should be to buy and sell orders, what to do if people run out of play money, the length of time that the market is open, and so forth. This person is also responsible for working with IT to ensure smooth operation of the software platform.
• IT Staff. The virtual stock market platform needs to be integrated with the organization's IT infrastructure. Many of the platforms now take advantage of cloud computing platforms, which decreases the amount of effort required. However, security concerns can then become an issue, as the information in the company's virtual stock market is often considered highly confidential.

The participants should have some familiarity with the platform and application (Step 3). The training session should provide participants with information on how to use the system, buy and sell stocks, and how to make a profit. This can be done in a hands-on, one- to two-hour workshop or in a web platform.

The prediction market can then be launched (Step 4). Figure 4.2 shows a two-step process which is often preferable. In the first step (Step 4a), the forecast is done in a small group (i.e., less than 100 people) for a few weeks, to get a better idea of how the system is working and whether the problem is explicitly defined. The trading platform is also assessed to determine if the right amount of play money was given to each participant and the sensitivity of the trading algorithm to the buy and sell orders. The prediction market can then be rolled out (Step 4b) to a much larger group, once further confidence is obtained. In the final step (Step 5), a forecasting report is sent to all participants comparing the actual result with the forecast. Typically such forecasts are quite accurate, which provides incentives for participation in the next prediction market. Overall, the total number of virtual stock market projects needs to be managed as a gradual participation decline will occur when there are too many campaigns. Further details for implementing a prediction market can be found in Ivanov (2009).

THE KEYS TO SUCCESS FOR IMPLEMENTING A PREDICTION MARKET, IN APPROXIMATE ORDER OF PRIORITY, ARE:

• Making the prediction market part of an important corporate initiative that is championed by senior management. Without such a mandate, participation will quickly wane.
• Developing a project team that will support the effort
• Inviting the right set of participants—a combination of domain experts and those with diverse backgrounds
• Developing a set of intrinsic awards that are aligned with the corporate initiative

THE PITFALLS TO AVOID WHEN IMPLEMENTING A PREDICTION MARKET ARE:

• Having too many prediction markets that involve the same participants.

Preference Markets

The overall process for implementing a preference market is shown in the four-step process illustrated in Figure 4.3.

Figure 4.3: Implementation Process for a Preference Market

Overall, it follows the same process as prediction markets discussed previously. An additional conceive stage (Step 2c), which is done by the project team, is needed where detailed product information is provided. For example, illustrations and product information on six different bike pumps can be provided. In addition, each of the bike pumps is characterized by the speed at which they inflate tires, their compactness, ease of operation, and durability. As the overall trading occurs in a few hours, the training session (Step 3) is typically done at the same time with the preference market run shortly thereafter. Further details for implementing a preference market can be found in Dahan and colleagues (2011).

The keys to success for implementing a preference market are the same as in a prediction market. However, the team needs to assure that the detailed product information about the choices has sufficient clarity and specificity so that the traders can make informed choices. Further, the team needs to identify how the results will be utilized to bring value to the corporation.

Idea Markets

The overall process for implementing an idea market is shown in the eight-step process illustrated in Figure 4.4.

Figure 4.4: Implementation Process for an Idea Market

The first three steps are the same as in the prediction and preference markets, although participation incentives (Step 2b) represent a controversial area, as some believe that financial rewards are necessary for success. However, the research discussed previously suggests that financial incentives may actually undermine performance. Accordingly, the project team needs to determine a compelling case of why participation is important. For example, the Chief Scientific Officer at Novozymes (Lauto et al. 2013) sent an open letter to all participants explaining the importance of participation. Further encouragement can come from recognition of the best traders.

The conceive stage (Step 4) is necessary as the number of ideas in the market needs to be restricted to maintain an adequate number of traders and to ensure that the ideas are responsive to the idea campaign and not duplicates of other ideas. In addition, the project team needs to ensure that there is sufficient information around each idea so that the stock traders can make informed choices. The idea market is then run in Step 5. Once the market is closed, the results need to be matured in Steps 6 and 7, since the “winning” ideas frequently are still in a very immature state. The five “winners” are determined from the top 10 ideas with the highest market value and an additional set of ideas chosen by the project team. Not all of the “best ideas” will make it to the top and additional ideas may be included in the review process, as traders sometimes overlook them since they may have lacked critical information. The ideas are further matured in Step 7, with a project team assigned to each. Each idea is then presented to management for further funding at the end of this stage.

Adequate participation in an idea market is typically much harder to achieve and sustain, as the outcome is much less tangible than for a prediction market. Recommended awards (Step 8) which provide adequate incentives to both the “winners” and the traders should consist of symbolic awards given at a recognition ceremony to: (1) the winners, (2) the ideas with the highest market value, (3) the participants with the greatest number of shares, and (4) the traders who made the most number of comments. All of these conditions were met by Novozymes (Lauto et al., 2013) who achieved a successful outcome in their idea market. However, they indicated that “. . . the time required to carry out the campaign exceeded the Innovation Office's expectations: evaluation activities, support to maturation and project management were particularly time consuming” (p. 25). Further details for implementing an idea market can be found in Lauto and colleagues (2013).

The keys to success for the idea market are the same as for a prediction market. However, in addition, the project team needs to spend time carefully filtering the ideas that are entered into the idea market for relevance and potential replication (Step 4 of Figure 4.4). This is a step often skipped, as companies often open the market to all participants with little or no filtering. Further, the market should only be run if there are at least three traders for each idea. Finally, a recognition ceremony should be conducted to reward the traders. The recognition ceremony is more important for the idea market than for the prediction market as the value of the results from the prediction market are typically well recognized. In contrast, the concepts from the idea market are often in a less developed state and not seen as having the same immediate value to the corporation.

4.5 Choosing a Virtual Stock Market Software Platform

Software vendors are divided into those that specialize in prediction markets and those in idea markets, as indicated in Table 4.5, though many of the prediction software vendors try to do both. Currently there are no commercial software vendors for preference markets.

Table 4.5 Software Vendors

Vendors	Websites	Prediction Markets	Idea Markets	Open Source
BrainBank	www.brainbankinc.com		X
BrainJuicer	www.brainjuicer.com	X
Brightidea	www.brightidea.com		X
CogniStreamer	www.cognistreamer.com		X
Consensus Point	www.consensuspoint.com	X
HYPE Innovation	hypeinnovation.com		X
IdeaScale	ideascale.com		X
Imaginatik	www.imaginatik.com		X
Infosurv	www.infosurv.com	X
Inkling	inklingmarkets.com	X
Inno360	www.inno-360.com		X
InnovationCast	innovationcast.com		X
InnovationFactory	www.innovationfactory.eu		X
Kindling	www.kindlingapp.com		X
Lumenogic	www.lumenogic.com	X
MarMix	sourceforge.net/projects/marmix	X		X
Nosco	nosco.co		X
Q markets	www.qmarkets.net	X	X
Serotonin	serotoninpm.sourceforge.net	X		X
Spigit	www.spigit.com	X	X
Zocalo	zocalo.sourceforge.net	X		X

In a 2011 report, Forrester provided an overview of the software platforms for prediction markets (Strohmenger, 2011). They can be broadly divided into two themes, industry focus and the type of service model they provide, as shown in Figure 4.5. Companies such as Inkling and Spigit focused mainly on business-to-business companies, Inforserv and BrainJuicer focused on business-to-consumer companies. Many of the software vendors are moving from a self-service model to an assisted or full-service model. In the full-service model the vendor provides hands-on help in creating, running, analyzing, and interpreting the results. In the self-service model, the company creates and implements the prediction market with the vendor providing a consultative role. If price is an issue there are several platforms that use open-source software, as Table 4.5 indicates.

Figure 4.5: Prediction Market Companies Distinguished Based on Industry Focus and a Service Model Perspective

Reproduced with permission of Forrester Research, Inc.

Software platforms for idea markets are discussed in a different Forrester report on innovation management tools (Gliedman, 2013). The pricing algorithm is important to the ability of idea platforms to successfully generate substantive ideas that have value to the corporation. The most accurate and rigorous platforms, such as Nosco and Q markets, use an economic or automated market makers software (AMM) as discussed previously. Simple “thumbs-up” or “thumbs-down” voting systems lack the sophistication of a virtual stock market with the most popular ideas rising to the top since there is no trading mechanism to adjust for truth-seeking behavior. The idea software platforms are characterized based on their pricing algorithm as shown in Figure 4.6. The less sophisticated ones, such as BrainBank, CongiStreamer, IdeaScale, and Kindling use simple up or down votes. Some companies, such as BrightIdea, Hype, and Imaginatik, use scale voting where the trader can vote from 1 to 5 on how well they like or dislike the idea. Spigit uses a more sophisticated pricing algorithm which is based on the reputation and trading experience of the participant in contrast to traditional automated market makers–based as used by Nosco and Q markets which are based on economic theory.

Figure 4.6: Idea Market Vendors Distinguished on the Basis of their Pricing Algorithm*

*There is no difference in accuracy in any particular area with the vendors listed in alphabetical order. For example, there is no difference in potential accuracy between Nosco or Q markets. Inno360 and InnovationCast were not listed as the pricing algorithm information was unavailable.

As discussed earlier, the most extensive research in prediction markets has been done using economic based AMMs. There has been no academic research to determine the overall accuracy of these less rigorous platforms. One might suspect that the results may be poor using less sophisticated pricing algorithms since thin markets with fewer than six traders per idea strain even the more sophisticated automated market maker pricing algorithms.

It is interesting to note that Novozymes (Lauto et al., 2013), who was able to obtain robust results, used the Nosco software platform which uses an AMM. The ability to adjust the relative weights of the pricing algorithm may also be important in thin markets with low trader-to-idea ratios to compensate for the idiosyncrasies of trading unique to the organization. For example, Google found that stock market prices can increase by over 10 percent when Google's stock increases (Cowgill et al., 2008, p. 8). The more sophisticated pricing algorithms may allow adjustments to be made to the idea market price to compensate for such idiosyncrasies.

Most of the idea software vendors are small, averaging about $4 million in annual sales, with fairly mature software platforms. The overall vendor choice should not only be made based on the robustness of the tool and the pricing algorithm, but in the support which the vendor can provide during the initial training period. If price is an issue, IdeaScale allows access to their platform for free for up to five campaigns with only web support, though the platform only allows up or down voting.

4.6 Conclusions

Predicting the future using the wisdom of the crowds is now being used with increased frequency in corporations. The reported results for companies for prediction markets, such as being able to predict sales of a product three months into the future, have been excellent, often with less than 10 percent error. These robust results occur when companies carefully choose experts to participate who are knowledgeable in the product, when participation is part of an important corporate initiative, and when there is adequate trading volume. A similar result was obtained for preference markets where the results are compared to traditional conjoint analysis; however, there have been no studies documenting their use in corporations.

In contrast, the reported accuracy and outcome of idea markets has been problematic. The reasons are threefold. The majority of the cases have few traders for each idea. Thin markets with low trader-to-idea ratios stress the trading mechanisms making truth-seeking behavior more difficult to occur since there are not enough traders to adjust for incorrect or biased trades. The second factor is that the incentives are not linked to important corporate initiatives. As Google indicated, employees are not bothered when the small cash prizes are forgotten, but notice “. . . when the T-shirts that show who won don't come” (Dye, 2008, p. 89). Without the correct incentives, traders do not actively participate. Finally, many of the idea markets are open to all of the employees in an organization which further increases the noise of the stocks traded since the participants lack a basic understanding of the idea they are trading in. Correcting these deficiencies in idea markets can help companies achieve even higher levels of innovation as both Novozymes (Lauto et al., 2013) and Rite-Solutions (Taylor, 2006) have demonstrated.

References

1. Amabile, T, 1997, Motivating Creativity in organizations: On doing what you love and loving what you do, California Management Review, 40 (1): 39–58.
2. Berg, J. E., F. D. Nelson, and T. A. Rietz, 2008, Prediction market accuracy in the long run, Journal of Forecasting, 24: 285–300.
3. Cowgill, B., J. Wolfers, and E. Zitzewitz, 2008, “Using Prediction Markets to Track Information Flows: Evidence from Google.” Available at http://bocowgill.com/GooglePredictionMarketPaper.pdf.
4. Dahan, E., J. K. Adlar, A. W. Lo, T. Poggio, and N. Chan, 2011, Securities trading of concepts (STOC), Journal of Marketing Research, 48(8): 497–517.
5. Diemer, S., 2010, Real-Money vs. Play Money Forecasting Accuracy in Online Prediction Markets—Empirical Insights from ipredict. (www.predictx.org/mt.pdf).)
6. Dye, R., 2008, The promise of prediction markets: A roundtable, The McKinsey Quarterly, 2: 83–93.
7. Ederer, F., G. Manso, 2013, Is pay for performance detrimental to innovation? Management Science, 59 (7): 1496–1513.
8. Forsythe, R., T. A. Rietz, and T. W. Ross, 1999, Wishes, expectations and actions: A survey on price formation in election stock markets, Journal of Economic Behavior and Organization, 39 (1): 83–110.
9. Galton, F. 1908, Memories of My Life, London: Methuen.
10. Gliedman, C., 2013, The Forrester Wave^TM: Innovation Management Tools, Q3 2013, Forrester Research, Inc. MA.
11. Healy, P. J., S. Linardi, J. R. Lowery, and J. O. Ledyard, 2010, Prediction markets: Alternative mechanisms for complex environments with few traders, Management Science, 56 (11): 1977–1996.
12. Hopman, J. W., 2007, Using forecasting markets to manage demand risk, Intel Technology Journal, 11(2): 127–135.
13. Hoyt, D. 2006, Rite-Solutions: Mavericks Unleashing the Quite Genius of Employees. Stanford Graduate School of Business, Case HR-27.
14. Ivanov, A. 2009, Using predictive markets to harness collective wisdom for forecasting, The Journal of Business Forecasting, Fall, 9–14.
15. Karniouchina, E., 2011, Are virtual markets efficient predictors of new product success? The Case of the Hollywood Stock Exchange, Journal of Product Innovation Management, 28: 470–484.
16. Kamp, G. and Koen, P.A., 2009, Improving The Idea Screening Process Within Organizations Using Prediction Markets: A Theoretical Perspective, The Journal of Prediction Markets, 3(2): 39–64.
17. LaComb, C. A., J. A. Barnett, and Q. Pan, 2007, The imagination market, Information Systems Frontiers, 9: 245–256.
18. Lauto, G., F. Valentin, F. Hatzack, and M. Carlsen, 2013, Managing front-end innovation through idea markets at Novozymes, Research-Technology Management, July–August, 17–26.
19. Lundholm, R. J., 1991, What affects the efficiency of a market? Some answers from the laboratory, Accounting Review, 66(3): 486–515.
20. Plott, C., and K. Y. Chen, 2002, Information Aggregation Mechanisms: Concept, Design and Implementation for a Sales Forecasting Problem, Californian Institute of Technology, Social Science Working Paper 1131 (ideas.repec.org/p/clt/sswopa/1131.html)
21. Slamka, C., B. Skiera, and M. Spann, 2013, Prediction market performance and market liquidity: A comparison of automated market makers, IEEE Transactions on Engineering Management 60(1): 169–185.
22. Soukhoroukova, A., M. Spann, and B. Skiera, 2012, Sourcing, filtering and evaluating new product ideas: An empirical exploration of the performance at idea markets, Journal of Product Innovation Management, 29(1): 100–112.
23. Spann, M., and B. Skiera, 2003, “Internet-Based Virtual Stock Markets for Business Forecasting,” Management Science, 49(10); 1310–1326.
24. Spears, B., J. LaComb, J. Barnett, and S. Deniz, 2009, Examining trader behavior in idea markets: An implementation of GE's imagination markets, The Journal of Prediction Markets, 3(1): 17–39.
25. Strohmenger, R., 2011, How Prediction Markets Help Forecast Consumer Behaviors, Forrester Research, Inc.
26. Surowiecki, J, 2004, The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations, Doubleday, New York
27. Taylor, W., 2006, Here Is an Idea: Let Everyone Have Ideas, New York Times, March 26, 2006.
28. Van Bruggen, G. H., M. Spann, G. L. Lilien, and B. Skiera, 2006, Institutional Forecasting: The Performance of Thin Virtual Stock Markets (http://ideas.repec.org/p/ems/eureri/7840.html.)
29. Wolfers, J., and E. Zitzewitz, 2004, Prediction markets, Journal of Economic Perspectives, 18(2): 107–126.

About the Contributor

Peter Koen is an associate professor in the Wesley J. Howe School of Technology Management at the Stevens Institute of Technology. He is currently the director of the Consortium for Corporate Entrepreneurship (CCE), which he founded in 1998 to increase the number, speed, and success probability of ideas at the front end of innovation. Current members of the CCE include 3M, Corning, Ethicon (a division of Johnson and Johnson), ExxonMobil, P&G, and WL Gore. His academic background includes a BS and MS in mechanical engineering from NYU and a PhD in biomedical engineering from Drexel University. Contact: peter.koen@stevens.edu.