Figure 8.1 Participation Patterns
Sometimes it’s not enough to know what things mean, sometimes you have to know what things don’t mean.
—Bob Dylan
When Wikipedia’s founder, Jimmy Wales, was asked about his active core contributors in 2006, he said that about 500 people were responsible for creating Wikipedia. This group conducted most of the edits (that is, made changes to Wikipedia documents). Since each change is recorded on Wikipedia, it is possible to log and attribute a variety of actions to each of the participants.1
However, a Stanford student named Aaron Swartz challenged this characterization of the community. “Wales was just counting the number of edits—the number of times a user changed something and clicked save. Wouldn’t things be different if he counted the amount of text each user contributed?”2 This question really resonated with your authors. We are responsible for creating the content, but without help from our editors, this book would be much, much worse. We have the old drafts to prove it.
As community manager in chief, it was Wales’ job to understand his contributors. Surely, he knew who his main contributors were. Seems simple, except that we’re asking him to understand hundreds of thousands of participants—this is a very different problem than understanding how to work with the much smaller team on this book.
Swartz’s question about how Wales measured participation was important because it influenced how he would manage the Wikipedia site and community. As he said to Aaron Swartz, “This is really important for me, because I spend a lot of time listening to those four or five hundred and if . . . those people were just a bunch of people talking . . . maybe I can just safely ignore them when setting policy”—and instead worry about “the million people writing a sentence each.”3
Swartz tried to understand the various kinds of different edits. Spelling corrections were edits; but so were the insertions of whole new pages of original content. So how could he count them in the same way? If Wales’ 500 were focusing on spelling and grammar and ensuring certain standards, where was all the content coming from?
When Swartz hit upon the idea of counting letters instead of edits, he uncovered a significantly different story regarding the participants. He found that while most edits were concentrated within a small number of insiders, most of the content (the large paragraph additions, which might take just a few edits) was coming from people outside the core group Wales had described.
Swartz found that when “you put it all together, the story becomes clear: an outsider makes one edit to add a chunk of information, then insiders make several edits [by] tweaking and reformatting it. In addition, insiders rack up thousands of edits doing things like changing a category’s name across the entire site—the kind of thing only insiders deeply care about. As a result, insiders account for the vast majority of the edits. But it’s the outsiders who provide nearly all of the content.”4 So while Wales’ 500 contributors play an essential role organizing and refining content, new content came primarily from a very large group of contributors. And, the number of significant contributors based on Swartz’s definition—people who edited more than 10 times5—skyrocketed to over 150,000, just for the English version of Wikipedia.
This change in the way Wikipedia participation is measured fundamentally changed the understanding of what participants were doing. And, as Swartz concluded, it should have an impact on the way the community needed to be managed. “Wales is right about one thing. . . . This fact does have enormous policy implications. If ‘occasional contributors’ are writing Wikipedia, then growing [the site] requires [that we make] it easier and more rewarding to contribute occasionally. Instead of trying to squeeze more work out of those who spend their life on Wikipedia, we need to broaden the base of those who just add a little bit.”6
Our Chapter 4 discussion gives us the luxury of knowing the rest of this story. The implications of Swartz’s analysis were borne out when a number of researchers started to notice in 2009 that new prospective editors were leaving Wikipedia. In 2011, Wikipedia commissioned its own study to confirm the data: new editors were simply not staying, and old editors were leaving. The potential implications were a wake-up call for management. Yes, Wales’ 500 were critical—but so were the tens of thousands of occasional Swartz contributors. Wikipedia learned an important lesson about understanding contributions and began to rethink how to manage its community.
Wikipedia highlights the complexity of monitoring; at the same time, it also shows how important monitoring is to community management. Wales was concerned about understanding the community in order to properly set policy. But whose needs should he focus on? Who should he advocate for as Wikipedia prioritized the development of new tools? The issues get to the heart of healthy community institutions—collective choice and membership. Shifting the definition of contributions, shifts who is considered part of the core membership. It also reframes who might be involved in making decisions about Wikipedia’s future. The monitoring issue also touches on value exchange. If contributions are undervalued—for example, participants are offered the wrong support or limited incentives—and are based on the wrong metrics, then it is not hard to see how this would have a negative impact on contributors. This would therefore result in fewer contributions and, ultimately, disengagement from the platform.
Finally, understanding contributions also impacts how we think about trust and expertise. As we measure participants we find those who are making valuable contributions in the midst of the noise of hundreds of thousands of contributors. As we discussed in Chapter 7, nesting is necessary to scale by allowing communities to distribute management responsibilities across members of the community—using monitoring to identify contributors offers a way to find new contributors to take on community management roles.
For all crowdstorm patterns, we are interested in finding new ideas, solutions, and talent, but we are also—in the case of collaboration and integration patterns—interested in feedback and filtering as well. When we discuss monitoring on a large scale, we need to account for participants who contribute in one or more ways.
We need to be able to understand which contributions are useful and which are not. In other words, we need to understand which people are adding value and helping to achieve our shared goals for the crowdstorm. Determining who is making valuable contributions helps community managers to better allocate their limited resources to support and offer incentives to the most valuable (or potentially valuable) contributors.
To understand this a bit better, let’s revisit one of our previous cases, the LifeEdited challenge that asked participants to submit designs for a low-footprint apartment. LifeEdited followed the collaboration pattern, enabling contributors to submit ideas and then receive feedback in the form of comments and votes. The task was particularly complex; to encourage quality collaboration, incentives were provided for feedback, in addition to the rewards in prize money for the best ideas. There was a lot of feedback. The winner—who was selected by other participants and who went by the username SungUn—had this to say: “I would like to thank everyone who spent time and effort to vote and give feedback to participants of LifeEdited. Because . . . I know that my design skills would not have become this polished without your helpful advice.”7
This is great feedback to receive at the end of the process. But how do we know how feedback is working during the process? And how can we ensure it is working for all participants?
One approach is to have the community managers read through all the comments and use their judgment to select the most useful. The problem with this approach is that this requires them to sift through thousands of comments. For example, a community manager doing this for the LifeEdited challenge would have to look at the over 7,000 comments submitted on 303 ideas. But that is just the start. The community manager would need to know the context in which each was submitted—which means looking at the ideas and having the expertise to judge on any number of dimensions from aesthetics and materials to engineering, and economics. It is clearly not practical to ask community managers to judge the usefulness of feedback in this way.
Our challenge: how do we understand the large number of contributions from supporting participants? This is not an issue unique to Wikipedia or LifeEdited or more generally the collaboration and integration patterns. In fact, these types of contributions are common far beyond the crowdstorm process—they are core to many of our contributions in our online interactions.
In his book Programming Collective Intelligence,8 author Toby Segaran provides a view into a fast-growing world of algorithms that help evaluate online contributions from large groups of people. Segaran’s examples point the way to how online environments can move beyond counting edits (our Wikipedia example demonstrates this) to providing details about the edits—that is, telling us how we can evaluate more signals to better assess contributions.
A familiar example we take for granted suggests that using algorithms to look at the data differently can yield surprising and very valuable results: search engines. Search is very concerned with understanding content’s value—or, more specifically, content’s relative value in response to a specific search. Segaran points out that Google does this today in a way that is very different from the way they originally did it when the site initially launched.
When Google entered the search business, the industry approach to search was to analyze content directly—that is, to use algorithms to understand a web page’s content. What Google’s founders realized was that there was additional data or signals that could help them rank or value web pages during a search. Nobody was considering the information contained in the links between pages. Initially, this data consisted of information such as who was linking to a given web page or what they were calling the page (in the link text).
Google was extracting meaning from a world of data left behind as people did their work. Links were one of the byproducts of creating content online. People were not purposely trying to help Google; they were focused on other people who might be interested in their content. Today these byproducts have expanded dramatically to include online activities such as content sharing, “liking” on Facebook, or +1 on Google Plus.
If we want to make sense of the growing number of contributions in, for example, the LifeEdited contest, Segaran’s example points the way: we need to consider using algorithms to analyze “signals” that participants leave as they contribute and interact with one another.
As we have already discussed, LifeEdited used the jovoto platform to enable interactions between participants. And jovoto has developed its own approach to understanding the value of feedback. In the beginning, the platform was simply enabled to allow people to comment and vote. But, like the example of Wikipedia edits, they realized that simply counting comments was not the best way to understand the contribution’s value. So jovoto gave participants the opportunity to like comments—a way to indicate whether they found a comment useful. As with Facebook’s like feature or the +1 on Google Plus, jovoto can now also look at how people respond, or who is responding to the comment; for example, did the idea creator respond, or did a senior person in the community say that the comment was useful. So, a number of possible signals quickly emerge.
From these various signals, jovoto has been able to develop algorithms to assign value to comments. This is the feedback that is likely most valuable in helping submitters to improve their ideas. Jovoto has gone a step further and awarded points for commenting behavior—specifically to the comments that the community deems useful (based on how members react to the comment). Jovoto calls these points karma points, which community members can earn not just from good ideas, but also from valuable feedback.
If we use the jovoto approach to understanding the value of comments, we find that it is effective to identify and reward the best commenters. Without an algorithm to analyze these contributions, it would be very difficult to understand or reward this behavior. And this would make it very hard to identify the people who are providing all this good feedback.
The goal of the jovoto karma points system is not just to recognize the most valuable behaviors but, through this, to identify the most valuable members of the jovoto community. The system analyzes the quality of ideas, comments, and also votes (we will discuss ideas and votes in more detail in the next chapter). Karma is designed to get beyond simple counts; while it knows the number of comments, it is trying to determine whether a comment is valuable. If someone is rating an idea, jovoto is less interested in the act of rating; rather, it is trying to understand whether it should trust the rating.
Figure 8.1 shows the core community and the supporting community in the 1 percent and 10 percent levels of participation with the “long tail” of participants whose contributions are low. It shows how monitoring contributions enables the identification of the most valuable contributors.
Figure 8.1 Participation Patterns
As Jimmy Wales pointed out, once you know the most valuable contributors, you are well positioned to serve them by listening and responding to their needs. In other words, a community manager with this knowledge is able to ensure that he can serve his most important stakeholders. Additionally, this provides information that helps us understand the overall health of the community.
For instance, if we look at the jovoto community, we notice that most people tend to contribute in multiple ways. That is, people who submit ideas also offer feedback to their peers via votes and comments. But if we classify people based on where most of their karma comes from, we start to see clear roles emerge. Looking at the top 1,000 jovoto community members ranked by karma score, 56 percent of the top tier earned most of their karma from submitting ideas. But 44 percent received their karma points primarily from feedback (in the form of ratings and comments). Jovoto community managers have long understood that the success of the community is related to feedback, and so they have sought to constantly refine how they measure and recognize those participants who provide it.9
It is useful to have historical view of contributions, but what about high potential participants—that is, people who only recently started contributing, but who might turn out to be highly valuable community members in the future? As we saw with Wikipedia, it is easy to measure historical contributions and focus on current high contributors—something that often comes at the expense of missing the value that new people might add.
So even if we ascribe the right value to contributions, we still have a problem. Let’s say you joined jovoto to participate in the LifeEdited challenge. You may have enjoyed the process and offered great feedback. But you did not return after this challenge. Other participants would continue to earn karma, and your rank would slip since your score remained static. (Actually, in the specific case of jovoto, your karma points decay over time. Recent contributions receive more value—so you would quickly find yourself slipping into the long tail of the power law distribution.) We would want to know that, while your overall rank is low, it is not because you do not have great potential, but because something is stopping you from participating more.
It’s crucial to be aware of this issue. In an effort to figure out why editors who had contributed then left, Wikipedia contacted the editors to ask what happened. Historic tracking provided a way to identify these individuals; their metrics indicated that they were among the core contributors, but they decided for some reason not to continue contributing. Wikipedia interviewed the former editors, determined what the issues were, and used the information in order to keep an eye on the new potential “high performers” rather than waiting for them to encounter problems. They checked in with them to provide support early on as they adjusted to the community. Understanding the changes in contributions over time made this possible.
Under normal circumstances, jovoto participants receive karma points for feedback and ideas. However, the points do not translate into financial compensation. They are simply used to show ranks and recognize significant contributors. The same is true for some of the other communities we have discussed. Local Motors uses metrics to understand contributors—and as a result, the overall health of the community. The Threadless story is similar. As is LEGO Cuusoo. There is no public leaderboard for the community, but Threadless tracks how many designs people have submitted as well as their ranking behavior—giving them a way to understand individual contributions. The same is true for LEGO Cuusoo.
But there are firms that evaluate contributions as a means to pay contributors.
In May 2012, Giffgaff paid out almost $1.8 million to its contributors.10 As we discussed, this company relies on their community for everything from new ideas to sales. Giffgaff participants receive points for their contributions that they can redeem in cash or put toward more credit on the Giffgaff mobile network. Participants can also choose to pay out their points to a community-designated charity.
There are some interesting implications to linking points to rewards.
Some of the Giffgaff points seem easy to calculate. For example, you receive a reward if you help to sell the service to a new subscriber. This is simple enough: Giffgaff provides a number of ways to track sales from specific members and they have a straightforward point-assignment system. But giving points becomes trickier when it comes to contributions in the form ideas and feedback. Giffgaff is wary about connecting these types of contributions to cash rewards. Specifically they believe that if they link points explicitly to cash, they will encourage a number of negative behaviors that will degrade community-wide performance.
What might these negative behaviors look like? If we decide to use a like function to indicate that the comment was useful, we are immediately open to exploitation. Groups could get together and simply like one another’s comments. Very soon, this behavior has the potential to drown out the truly useful comments, thereby rendering this metric useless. In the Giffgaff environment, users can also award one another points if they present a good idea or useful feedback. So they are subject to the same risk; an explicit link to payment will likely result in a variety of creative approaches to rewarding points and gaming the monitoring system.
To solve the problem, Giffgaff does not share real-time points with the community. Instead, their platform enables them to monitor a range of signals like “kudos” received or comments made. During each payout period, the Giffgaff team can analyze and rank participants into contribution bands (a more complex version of the 1–9–90 contribution bands). They avoid sharing specific scores, and they also give themselves an opportunity to identify situations in which gaming might be taking place.
One of the most interesting platforms for contribution measurement is Quirky, whose measurements process directly impacts how much their participants stand to earn. Here is how it works.
Quirky evaluates how much a contributor influences the final result of the product. The more you influence the final product, the more you stand to earn as a percentage of sales when the product is realized. Quirky makes available 30 percent of the retail and 10 percent of the wholesale price to be shared among the community based on their influence. (By comparison, participants in the LifeEdited project earned karma by submitting ideas, commenting, and voting; however, the final payout in terms of financial rewards went primarily to winning ideas.) Quirky takes the next step: their points ultimately result in payouts linked to the volume of sales.
In Chapter 4, we discussed Quirky’s “Pivot Power” challenge—the redesign of the old power strip. The idea submitter gets a minimum of 42 percent of these royalties—which seems like a good deal. But this also means that more than 50 percent of the participants are getting paid for something other than submitting the idea. (And since royalties are paid in perpetuity, this is just the beginning of the rewards for contributors.)
Quirky has devised a number of clever ways to encourage and measure participation. For example, several contributions are winner-takes-all—not very different from many of the prize challenges we have already discussed. If you submit an idea that’s selected for development, you win—and get 42 percent of the available influence.11 You can also submit an idea for the product name or the tagline and receive rewards for the winning ideas. You can win as well if you had cast a vote for the winning design. You feel motivated to participate because you share in the reward of a successful outcome.
Finally, you can earn points during the research phase, since individuals who take part in the focus groups are also rewarded with small shares of the revenue. There does not need to be a best response for this type of activity; there simply need to be enough participants to get Quirky the information they need for research. This is not too different from Giffgaff’s approach, where more participation (of a certain quality, of course) gets contributors more points—which they can then convert into financial rewards.
Quirky has invited contributors to participate in multiple steps of the product design cycle in different ways. By breaking the process down into explicit steps, they take advantage of nesting; that is, they make it easy to measure contributions in a complex process simpler by dividing it into small, focused interactions. Also, by creating discrete phases, they can more easily assign relative value to specific steps and particular roles in the design process. The original idea is worth 42 percent; additional ideas for name and tagline receive another 10 percent. Like jovoto, a little over 50 percent of the available points go to ideas, while the rest go to feedback—from research questions, to pricing and materials.
When we first introduced the idea of different crowdstorm patterns, we referenced the increasing interactions and resulting complexity of the collaboration and integration patterns. We can see from the examples of jovoto, Giffgaff, and Quirky that complexity increases as the crowdstorm process is divided into projects and specific phases. And we can also appreciate the importance of rigorous tracking to help community managers support their crowdstorm communities.
It is also essential for participants to have visibility as part of the activity. Like community managers, they need a way to track what is happening and discern where there are opportunities for them to participate. They need a way to understand things such as: When is a new question posted? When is an idea posted for which they can offer feedback? When has someone responded to their feedback?
All of the organizations and platforms we have discussed have solved this problem in a similar way: they used a social feed similar to what we experience on our social networks like Facebook or Twitter or on corporate feeds like Salesforce’s Chatter or Microsoft’s Yammer. As participants choose to follow or friend people, some of their actions begin to show up in the participants’ feed. The difference for jovoto, Quirky, and Giffgaff is that these actions are focused on the crowdstorm process. They let participants know when new opportunities are announced for participation and when their friends post new ideas (so they can review them and offer feedback) or post feedback that they can respond to. They can see idea ratings and respond by improving their own ideas.
Nesting is an important part of this scaling. These feeds enable participants to evaluate their nesting strategies by helping them determine what interactions to focus on and decide what contributions interest them most. For example, a participant who joins jovoto for LifeEdited can choose to follow activity for architectural challenges. Once a participant submits an idea on Quirky, that action indicates interest in following interactions with that idea and that participant will want to know who has responded to that idea in order to review their feedback and respond if necessary (if only to thank them).
As we consider what shows up in our social feeds, we can get a sense for why it is essential to focus on valuable contributions. For example, we might not be interested in seeing every idea posted, but we do care about the ideas that are generating a lot of discussion or that have received very positive feedback. In crowdstorming spaces, participants want the same capability. They want to focus on what is most important and manage their own feed.
Community managers also need to create useful feeds to determine what is most important—to have a way to decide what is going to be most useful to participants. Their feed becomes an essential way to organize the community. They can pick a sample of participants—from the top contributors through to new arrivals—and directly monitor their activities to get a feel for everything from the rate of idea posting to the quality of feedback.
As most of us have experienced with our social networks, the feeds are a powerful way to monitor and then pick points to engage. They function in the same way for crowdstorming participants, but content is focused on the crowdstorming process.
Beyond social feeds, our examples of collaboration and integration patterns have something else in common: they do not just ask for help with ideas, but also with support. And they all use a variety of monitoring techniques to identify people who need help. Giffgaff monitors Twitter as well as participants’ Facebook pages, in addition to offering a forum space to post longer questions. Jovoto and Quirky also have their own forums where they highlight current open questions or important recent discussions.
Why so much focus on support? All of these organizations understand the same lesson that Wikipedia learned: potentially valuable contributors all have to start somewhere. And since they usually have questions, it’s important that they are answered. After all, they are unlikely to invest the time to share their ideas and knowledge if their first experience is a negative one where they don’t get the help they need. Further, participation rates are linked to the ease with which people can participate—less help means more people who would otherwise be contributing will likely be stuck.
All of these organizations highlight incoming questions so that the community can help to respond, which helps in a few ways: it doesn’t just reduce the workload for community managers, it also ensures that fast and good responses come from generally well-informed people. But these groups go further than simply showing questions. They also highlight the number of responses and views, thereby making community managers aware of situations where people are not receiving timely help. They also ask people to signal when a question has been answered. If questions remain open after they have been answered, participants (including the community managers) might waste time reviewing and responding.
So far, we have discussed monitoring contributions in order to identify and better serve our contributors.
But with lots of interactions in crowdstorming spaces, there is always the potential for abuse. It might be a particular comment’s tone or aggressive negative ratings (which we previously referred to as bashing). Or it might be that an idea is similar to or even a copy of something that exists. These issues can slow down or even prevent participation. As part of monitoring, we always hope for the best but should be prepared for the worst.
After the 9/11 attacks in the United States, law enforcement in New York City’s Metropolitan Transport Authority (MTA) launched a campaign in public spaces. Posters urged people to report any suspicious activity and told them, “If You See Something, Say Something.” The campaign has subsequently been rolled out to other states and, more recently, public spaces like sports stadiums. The idea is to augment police monitoring by reminding the public that we can join in and offer some support.
Community managers can be responsible for policing the community in a similar way—constantly on the lookout for any infringements on agreed-upon policy. But we can also give participants a simple way to say something, if they see something. Often called flagging, it has been a staple of online environments since early message board, when readers could flag messages for review by moderators.
Monitoring is essential to enable community management. It helps community managers identify who needs their support as well as who is making the most valuable contributions. Ultimately, it can provide an important link back to incentives—by establishing who will be rewarded, for what efforts, in particular for efforts that support the community (beyond providing the best ideas). It is easy to find actions to count; the harder issue is determining which of these actions has value. It is therefore often necessary to combine and analyze multiple signals to properly understand contributions.
Beyond the issue of measuring contributions, monitoring can also help participants and community managers to signal when they need help—and also understand where they might help. Feeds have quickly become a mainstay of our social networks—the same is true in these specialty networks, particularly for the collaboration and integration patterns.
We have focused a great deal on evaluating feedback. But this is only half of the issue, of course. How do we evaluate the very ideas that participants offer? In the following chapter, we explore idea evaluation in general and look at the many ways in which monitoring can help to filter and reveal the most promising submissions.
Notes
1. For a detailed discussion of this topic see Anikett Kittur, Ed Chi, Bryan A. Pendleton, Bongwon Suh, and Todd Mytkowitz, “Power of the Few vs. Wisdom of the Crowd: Wikipedia and the Rise of the Bourgeoisie,” Scribd, www.scribd.com/doc/2157257/Power-of-the-Few-vs-Wisdom-of-the-Crowd.
2. Aaron Swartz, “Who Writes Wikipedia,” 2006, www.aaronsw.com/weblog/whowriteswikipedia.
3. Ibid.
4. Ibid.
5. For statistics provided by Wikipedia see, a statistic provided by Wikipedia.
6. Swartz, “Who Writes Wikipedia.”
7. See quote on jovoto platform at www.jovoto.com/blog/2010/11/life-edited-1st-karma-award.
8. Toby Segaran, Programming Collective Intelligence (Sebastopol, CA: O’Reilly Media, 2007).
9. Analysis based on data provided by jovoto to authors in August 2012.
10. See details from Giffgaff site, community.giffgaff.com/t5/Contribute-Innovation-Promotion/Important-Threads-Read-First/td-p/5300682.
11. See details from Quirky site, www.quirky.com/learn.