CHAPTER 6
Experiments in Coproduction
KURT LEWIN, one of the founding fathers of social psychology, attempting to dispense with any argument that theory and practice are distinct and incompatible goals, quipped that there is “nothing as practical as a good theory.” In reality, there is often a gap between efforts to advance theory and the everyday activities of practitioners, but, to echo Lewin, this certainly need not be the case. Theory is simply a framework for organizing and explaining facts, and thereby it offers a basis for extrapolating existing knowledge to new contexts. Its translation into innovations and refinements in policy and practice is its promise to society and, as I have argued throughout this book, the foundation for collaborations between academics and public officials within the context of urban informatics.
Let us illustrate the practical value of a good theory with a tale of two theories—each of which has appeared repeatedly in previous chapters—and the evolution of crime prevention in the United States. In the early 1980s, Wilson and Kelling proposed broken windows theory (BWT). 1 One of the multiple appeals of BWT was its explicit identification of a causal mechanism in the generation of crime: physical and social disorder encourages the escalation and expansion of delinquent behavior. This made for a straightforward translation into new law enforcement policies that targeted low-level misdemeanors in order to preempt violent crime, most famously in the form of New York City’s zero-tolerance policing program. In these regards, BWT seems a highly useful theory. Unfortunately, as summarized in Chapter 2 , further scientific testing has repeatedly cast doubt on its foundational principle, as elegant as it might be, and there is limited if any evidence that disorder actually causes crime. 2 In their seminal evaluation of BWT, Sampson and Raudenbush found that a community’s collective efficacy in the enforcement of social norms has greater influence on local crime rates. 3
The apparent importance of collective efficacy set the stage for another shift in policing strategies, this time to community policing. Though community policing programs have come to vary widely in their implementation, 4 they all seek to implement formal law enforcement in a manner that reinforces the local community’s ability to manage the behavior of residents and visitors. Evaluations of this trend have found that community policing is effective in lowering crime and creates more positive relationships between police and the communities they serve. 5 This latter benefit appears particularly important in light of the tensions we have seen in recent years in places like Ferguson, Missouri, and Baltimore, Maryland. It would be revisionist history to say the proponents of collective efficacy invented community policing, as the concept existed for at least a decade before Sampson and Raudenbush’s work. Nonetheless, it had taken a back seat to broken windows policing in the late 1980s and early 1990s, and the research on collective efficacy provided renewed evidence for its value. I have in fact been in numerous meetings with police departments and other agencies and community groups that serve neighborhoods in which practitioners speak explicitly about “supporting collective efficacy” in the community. A practical theory indeed.
Is the public-as-partner model articulated in Chapter 5 a practical piece of theory? To take a devil’s advocate approach, it uses accessible terms such as “citizen” and “partner” but only as labels for abstracted definitions of the bases of human behavior. Similarly, what does it mean for a behavior to be rooted in territoriality? This would seem to fall outside the realm of the practical. Yet, if we can translate these concepts into applications for a given program, the public-as-partner approach becomes a useful tool for policymakers and practitioners. From this perspective, the lesson might be articulated more simply. Directors of 311 programs and other public officials should think of participation less as a proxy for civic engagement and more as an expression of care and concern for the places that matter to an individual. It is rather intuitive, then, to realize that those places are most often going to be near the home. This framing provides a road map for evaluating the program and developing future refinements in its implementation and outreach, and a similar distillation is possible for any coproduction program.
The goal of this chapter is to make good on the public-as-partner model’s application to 311 and to use this as a case study in the evaluation of civic tech. In doing so, it embodies the virtuous cycle of learning and application offered by urban informatics. The data generated by the 311 system supported novel insights on human behavior, which in turn inspired evidence-based innovations in policy and practice. Specifically, the chapter addresses two main questions. First, does attention to territorial motives improve the effectiveness of the system? Second, to what extent can new interventions engage a civic disposition and construct the hypothesized bridge to citizenship promised by coproduction and civic tech alike? For both of these questions, we will also consider the pressing concern of the “digital divide,” and the extent to which such innovations can be used to increase access to technologically driven city services by disadvantaged groups.
We examine the questions at hand through three experiments surrounding Boston’s 311 system. The first looks at program outreach and whether a message that targets localized sensibilities (“Clean up Dudley Square!”) is more effective than one that appeals to people’s care for the broader city (“Clean up Boston!”). The second is the introduction of BOS:311, a smartphone app that makes it convenient to report issues from anywhere. This increased accessibility raises the possibility that users might more easily break through the invisible boundaries of their “territory” and even form a greater connection with their community and city. The third experiment is an effort to build greater communication with BOS:311 users by sending them thank you notes that include pictures of the work that they instigated, including repaved potholes, fixed streetlights, and the like. This program also carried the hope that it would not only drive continued usage of the system but also elicit greater custodianship for the broader city, both in sentiment and in action.
The Flyer Study: An Experiment in Outreach
The primary goal of outreach for any government program is not only to inform people about it but also do so in a way that encourages participation. Many efforts around outreach focus on whether members of the population are able to participate. For example, Morten Jakobsen recently tested whether a language-learning program for the children of immigrants in Denmark could increase participation by providing parents with resources for practicing at home. 6 In contrast, the public-as-partner model provides a basis for developing interventions that attend to the public’s desire to participate. As noted in Chapter 5 , every human motivation is oriented toward particular tasks and is activated by a characteristic set of cues that are relevant to those tasks. In turn, messaging around a coproduction program would gain the greatest traction if it spoke to those psychological cues.
The lessons of the previous chapters suggest that outreach for the 311 system would be most effective if it spoke to territoriality. Drawing from the modern definition of human territoriality presented in Chapter 3 , this could be treated in practice as appealing to people’s psychological ownership of the space. We have seen repeatedly that custodianship is anchored by the home and the abutting spaces, though for a substantial number of Bostonians their efforts to maintain the neighborhood extend into shared spaces, such as commercial streets and parks. A program of outreach that can speak to one’s identification with this geographical level, at once localized but also collective, would in theory be the most effective in encouraging participation. Boston lends itself well to this challenge thanks to its long history as a “city of neighborhoods” and the well-established names attributed to many regions of the city. Thus, we might ask whether messaging that encourages people to help “Fix Potholes in Eagle Hill!” or “Fix Potholes in Dudley Square!” outperforms more generalized exhortations to “Fix Potholes in Boston!”
In 2013, I conducted this very study in collaboration with the directors of the city of Boston’s 311 system and the Mayor’s Office of New Urban Mechanics. The study was executed as part of a course I was teaching on social psychology methods. The students helped to develop the flyers advertising the 311 system, which reflected the two types of messaging (i.e., the “treatments”), and we distributed those same flyers door-to-door in 10 neighborhoods dispersed throughout the city. In order to maximize the value of the experiment, we concentrated on disadvantaged neighborhoods that appeared to be underutilizing 311. Though the experiment itself was definitively low-tech, its evaluation was made possible by the 311 system’s data. The “before” assessment of custodianship already existed, and the “after” measures, which would inform us as to the effectiveness of the different types of messaging, were forthcoming. Importantly, the value of the experiment sat at the intersection of science and policy: in determining which type of messaging would be more effective for outreach surrounding 311 systems, it also evaluated the practical relevance of the territoriality thesis. 7
Designing a Field Experiment
The field experiment required two main components. First, we needed to design flyers that could accommodate minor adjustments in geographical reference, permitting our two proposed treatments, dubbed Neighborhood and Boston. Second, we had to create an experimental design that could evaluate the relative impacts of these treatments. Because we had no way of knowing whether the specific individuals who received the flyers subsequently used 311, we treated neighborhoods as the unit of analysis. As such, the question was whether neighborhoods receiving one treatment or the other exhibited a greater increase in custodianship.
The Flyers
We constructed the flyers, which are pictured in Figure 6.1 , around three main components, two of which were manipulated for the purposes of the experimental treatment. At the top was a headline stating, “Potholes in [Location]? Get Them Fixed!” Depending on the treatment, this referenced either Boston or the name of a local neighborhood (e.g., Eagle Hill). Second, each flyer included an image of a pothole (drawn from the database of images generated by reports submitted via the smartphone app that is the focus of the next experiment) and a description of where that pothole had been found. The Neighborhood treatment used an image of a pothole from the local neighborhood and gave the nearest intersection in the caption, thereby priming care for neighborhood spaces. The Boston treatment used an image of a pothole from a major street that crosses the city and gave no specific intersection (i.e., “Tremont St.”), emphasizing the generality of the message. Finally, both versions of the flyer contained instructions on how to contact the Mayor’s Hotline.
The Experiment
The goal of the study was to assess the extent to which the two different messaging approaches—one highlighting the local neighborhood, the other highlighting Boston—were effective in eliciting custodianship in a neighborhood. We also compared the Neighborhood and Boston treatments to Control neighborhoods that received no flyers. With finite manpower, we conducted the experiment in only a small subset of the city, utilizing a matching design that created triplets of census tracts with similar characteristics in the same region of the city (based on planning districts, such as Fenway and South Boston). This effectively controlled for not only demographic and socioeconomic characteristics but also regional patterns, as well as previous levels of custodianship. We selected five matched triplets, each in a different planning district. In order to maximize the potential effect of the intervention, we made certain that three of these were located in the city’s three most disadvantaged districts and that they also ranked lowest in custodianship 311. 8 Two-person teams distributed flyers throughout the neighborhoods on a single weekend in April 2012. We chose April because it is a period with a rapid increase in reporting, owing particularly to potholes resulting from the spring thaw. Our goal was to deliver a flyer to every housing unit in the neighborhood. 9 In order to make the task more manageable, we selected a single representative census block group (CBG) for each tract. This had the additional benefit of helping us avoid contagion effects, as we selected CBGs that did not border each other from within tracts that did.
FIGURE 6.1 Example flyers from the Neighborhood treatment (left) and the Boston treatment (right).
The Effect of the Flyers
The main outcome of interest was the volume of reports of public issues in each CBG following the distribution of flyers. To control against the same quantity in the previous year, we analyzed the percentage increase (decrease) in reporting between the six-week period following the distribution of flyers in 2012 and the same period in 2011. Because of the nonnormal distribution of this measure, as well as the very few degrees of freedom, here we use the multinomial test, which evaluates the likelihood of a specific ordering of each of the triplets.
As shown in Figure 6.2a , CBGs receiving the Neighborhood treatment had the greatest percentage increase in reporting in four of the five triplets. In the fifth triplet, the CBG in the control treatment had the greatest percentage increase. This outcome supports the hypothesis that the Neighborhood flyers were more effective in encouraging reporting (p < .05). 10 In contrast, one will note not only that no CBG in the Boston treatment had the greatest percentage increase in its triplet but also that it had a greater percentage increase than the control treatment in only two of five cases.
Importantly, the same pattern was not visible for nonpublic reports (e.g., request for recycling bin; see Figure 6.2b ). The CBG in the Neighborhood treatment had the greatest percentage increase in this class of calls in only two of five triplets and had the second-greatest percentage increase (or the second-lowest percentage decrease) in reports in the three other triplets. In fact, one of the two CBGs with the Neighborhood treatment that led its triplet with the greatest increase in nonpublic reports was the only CBG in the Neighborhood treatment not to lead its triplet with the greatest increase in public reports. This suggests that the increase in custodianship in CBGs with the Neighborhood treatment was not merely an expression of increased knowledge about the system.
Summary: Priming Territoriality to Elicit Custodianship
The “flyer study” found that outreach for the 311 system was most effective when it spoke to people’s attachment to the spaces around their homes rather than a broader opportunity to contribute to the maintenance of Boston. Neighborhoods that received a flyer whose message primed territoriality had a greater increase in reports of public issues than those receiving either a flyer with a Boston-centric message or no flyers at all. This effect was independent of an overall increase in use of the system, suggesting that the flyers were effective specifically in encouraging custodianship rather than merely informing individuals about the Mayor’s Hotline. In stark contrast, the results found that a flyer encouraging people to help clean up Boston was no more effective than distributing no flyer at all. Though the results of the experiment are consistent with the territoriality thesis, one might wonder whether, alternatively, the messaging primed an in-group psychology, thereby stimulating protective behavior. Even if this were the case, however, it would be more of a complement to the territoriality thesis than a true alternative. In situations where there is perceived collective ownership, territorial and in-group motivations are not independent. Rather, the psychological mechanisms underlying territoriality are employed for a space defined by the in group. In other words, if an in-group psychology were active, it would be relying on territoriality to accomplish the goals of maintaining the local space.
FIGURE 6.2 Proportion of change in 311 reporting for the study period in 2012 relative to the same period in 2011 for matched triplets for (a) public issues and (b) personal needs.
Note: Labels indicate planning district. A-B = Allston-Brighton, Dor. = Dorchester, E.B. = East Boston, Matt. = Mattapan, Rox. = Roxbury.
The results provide additional evidence for the territoriality thesis of the urban commons while also demonstrating its practical ability to shape outreach for 311 systems and allied programs. That said, it suffers from some of the limitations of traditional field research. Because of our procedure of distributing flyers door-to-door, we were forced to construct a relatively small sample that also had to conform to the stringent requirements of the matching procedure. Consequently, the analysis consisted only of multinomial tests rather than a stronger, more detailed analysis of the differences between the treatments. The study did benefit from modern technology, but not to the fullest extent possible. The 311 data permitted a straightforward evaluation of the experiment because they were already being collected both before and after the experiment. We did not, however, embed the messages themselves in any web-based communications and applications. The next two experiments explore the potential to expand use of these technologies for the 311 system.
BOS:311: Evaluating an Icon of Civic Tech
To this point, all of the analyses in this book have analyzed 311 reports without differentiating between the various channels by which the city of Boston accepts service requests, including the hotline and online tools. In doing so, we have ignored the original 311 innovation and one of the icons of civic tech: Citizens Connect, now renamed BOS:311, is a smartphone application that allows people to describe and take a picture of an issue and submit it directly to the 311 system without the need to dial the hotline and interact with an operator. When Boston introduced Citizens Connect in 2009, it became the first municipality to successfully integrate a smartphone application into its 311 system. The tool was lauded both in local outlets, such as the Boston Globe , 11 and nationally, as an illustration of how digital technology can make government services more accessible and responsive. Chris Osgood and Nigel Jacob, the co-chairs of the Mayor’s Office of New Urban Mechanics and the initiators of the project, were recognized by Governing magazine as Public Officials of the Year in 2011 for this and related projects. 12 The app has since given rise to a number of daughter apps, including the City Worker app, which enables employees to directly add cases to the agency’s cue; Citizens Connect Text, which permits reporting by text message; Commonwealth Connect, which extends the tool to other municipalities throughout Massachusetts (which will be the focus of Chapter 7 ); and an updated version of the app, rebranded BOS:311, released in 2015.
By comparing the use of Citizens Connect and BOS:311 (hereafter BOS:311) to more traditional channels (e.g., hotline and self-service internet portal), we have a unique opportunity to evaluate the potential of civic tech. Civic tech promises to increase interactions between the government and the public on multiple dimensions—not only diversifying the number of channels for interaction but also expanding the activities these interactions support, thereby inviting in a broader and more representative set of the public. These goals are rather ambitious, and they do not always acknowledge the possibility that the digital divide may result in certain populations having less access to the tools that civic tech offers. Cesar McDowell and Melissa Chinchilla, for example, have argued that civic technologists spend too much time focusing on civic engagement and not enough on creating civic inclusion. 13
Nonetheless, proponents of civic tech argue that it will do more than just expand engagement and will transform the way individual members of the public view their relationship with the government and, more generally, their ability to contribute to society. 14 This, however, is essentially saying that civic tech will succeed in constructing the elusive “bridge to citizenship” originally proposed for coproduction. Some have already questioned whether this is possible. Internet theorist Evgeny Morozov has decried what he calls “slacktivism,” or online activities that convince participants that they are “making a difference” when in reality they have contributed very little. 15 Expressing a similar concern in a more tempered manner, Ethan Zuckerman distinguished between “thin” engagement, which requires relatively simple action on the part of the participant, and “thick” engagement, which requires reflection, problem solving, and full consideration of the societal implications of an action. 16 He does not necessarily describe the latter as universally preferable, but he says that we should understand in any given case which one a particular program needs and which one it is actually eliciting.
The analysis of BOS:311 here will evaluate the two proposed values of civic tech: increasing the effectiveness of public services and further incorporating participants into civic life. In doing so, it will get at an important tension. Suppose that BOS:311 expands usage in critical and valuable ways but there is no evidence that it energizes a civic disposition. Would that undermine the premise of civic tech? Or does it simply mean its proponents must be more precise in that the word “civic” does not necessarily refer to the motivations of the user but more to the public value he or she might contribute?
The following analysis tests how BOS:311 achieves three goals related to civic tech. First, does BOS:311 engage users from new populations or with different motivations? It will include an analysis of demographic differences between traditional users and BOS:311 users, attending especially to the concern that the value of technology-driven innovations might be limited to young, affluent individuals who have the resources and expertise to capitalize on smartphone applications. We will also consider whether the app attracts people with lower or greater levels of territoriality than traditional channels. Second, does BOS:311 increase the frequency and geographic range of custodians? Even though cell phones enable people to call 311 from anywhere in the city, there may be something additional to the level of convenience the app affords. This question also provides an opportunity to further test the robustness of the territoriality thesis, as it is possible that the narrow geographic range found in previous analyses is driven largely by the predominance of hotline calls. Third, does the app elicit thin or thick engagement? Using items from the survey, we can assess how much BOS:311 connects users to both their local community and the city writ large. If we find no support for the hypothesis that BOS:311 engenders greater civic disposition, we may need to reconsider how we discuss civic tech and its operation more generally.
Data for Evaluating BOS:311
Between March 1, 2010, and December 31, 2015, 7,313 users made 96,304 requests for service via BOS:311 that included geographic reference, and 42,887 were for case types that reflected custodianship. Of these, 15,975 were attributed to 4,345 registered accounts. It is worth noting that the app interface prompts users with a small number of popular case types, including graffiti removal, streetlight outages, and potholes, and an option for Other, which generates a custom dialog box. Cases reported this last way are classified as General Requests within the system. As a result, the proportion of General Requests from the app is about six times that of other channels (55 percent vs. 9 percent). Although comments indicate that many of these issues would qualify as being in the public domain, not all are, so the analysis opts for the conservative approach of omitting them. 17 We can also evaluate the effects of BOS:311 through the survey of 311 users, as 188 (28 percent) of the survey respondents were BOS:311 users.
There are two challenges to assessing differences in reporting patterns between BOS:311 users and those who use other 311 channels. First, there is good reason to believe that there are differences between the two populations that lead them to select one medium or the other for reporting issues. Consequently, the implementation of BOS:311 is best described as a quasiexperiment. 18 Two traditional solutions to such a situation are either to match individuals on key factors that might influence the outcome variables of interest or to control for those factors in a regression. 19 In this first step of the analysis, which will analyze the entire 311 database, we pursue the former approach. Unfortunately, 311 accounts do not include demographic information on users, so the only datum available for matching is residential location. This leads again to the use of multilevel models to compare individuals to those residing in the same neighborhood (in this case, CBGs), assuming that this will control for a substantial amount of the preexisting differences between the BOS:311 and traditional user groups. 20 When analyzing the survey sample, we are able to avoid this problem, as we can include demographics.
A second weakness, which bears on the need for multilevel models to compare individuals living in the same neighborhood, is that the vast majority of BOS:311 users provide their e-mail address as their point of contact; only 9 percent of the 4,345 people who made custodial reports through the app had home addresses on file, compared to nearly half of all custodians. To overcome this challenge, we can approximate an individual’s home as lying within the census geography from which he or she most often made reports (see Chapter 2 for more details on the process and validation). 21 It is additionally necessary to estimate the geographical range of custodians lacking a home address. I do so by estimating each individual’s “home” as the centroid of the census block in which he or she is believed to live and then measuring the distance of every report they made from that point. 22 A linear model was used to transform raw results from these estimations so that they might be comparable to the exact distance measures used to this point. As will become apparent, however, these numbers are not completely transferable, as they overestimate the size of ranges at the lower end of the distribution. 23
Question #1: Does BOS:311 Engage New Types of Users?
Demographics
An initial way of thinking about BOS:311’s impact is whether it attracted new and different types of users. Most simply, we observe that 70 percent of BOS:311 users made zero reports through the internet portal and hotline, meaning they have interacted with the system exclusively through this channel. We can further probe who these individuals are in two ways. First, we can test whether certain demographic characteristics of census tracts predict the density of BOS:311 users to be greater (or less) than that of the local density of traditional 311 users. As we have seen before, this approach can be informative in terms of per capita participation but is vulnerable to the ecological fallacy. We can also examine demographic differences between the BOS:311 and traditional users who participated in the survey.
Over the six-year span of the 311 data, BOS:311 users lived in every census tract, though the total number varied broadly, from 2 residents to 236 (median = 34 residents). In terms of proportion of the population, about 1 percent of the average tract’s population were BOS:311 users, and no tract’s population included more than 4.5 percent BOS:311 users. A regression found that this distribution was most strongly associated with the proportion of residents who held a professional degree or greater (i.e., bachelor’s, master’s, or doctorate; β = .61, p < .001). 24 The regression controlled for the number of traditional users in a census tract, meaning that higher levels of education predicted greater usage of BOS:311 over and above that which would be implied by the baseline usage of the hotline and internet self-service portal. Additionally, neighborhoods with greater Asian (β = .20, p < .001) and Hispanic populations (β = .32, p < .001) had more BOS:311 users than expected.
Turning to the survey, we can see clearly in Table 6.1 that BOS:311 users and traditional users had distinct demographic profiles. The former appear to be more likely to be male, white, between ages 25 and 44, and to hold a master’s degree. A regression confirmed these impressions (white: O.R. = 2.86, p < .001; age: O.R. = 0.64, p < .001; education: O.R. = 1.28, p < .001; male: O.R. = 2.23, p < .001). This tells a story similar to that of the earlier tract-level analysis, demonstrating clearly that BOS:311 appeals predominantly to those with higher levels of education. One striking difference between these two analyses is the different conclusion surrounding ethnicity. Two effects may be in play. First, the survey interpretations potentially suffer from the underrepresentation of minorities relative to Boston’s population, causing them to appear particularly uncommon among BOS:311 users as a result of sampling bias. Contrastingly, it could be that some heavily Hispanic and Asian neighborhoods are also popular with young, white renters. Examples include Chinatown and Allston, each of which are majority minority but near downtown centers and major universities. Thus, there is a need for further exploration of this particular question.
Territoriality
In addition to whether BOS:311 invites a new demographic profile, one might also ask whether it engages a different set of motivations. To do this, we can use the survey and multilevel models to examine whether BOS:311 users expressed levels of territoriality different from those of traditional 311 users while also controlling for demographic characteristics and neighborhood effects. The average BOS:311 user reported a greater desire to benefit the community (β = .18, p < .001) but a lower desire to enforce social norms (β = −.13, p < .01). On the one hand, it might seem counterintuitive that the app would appeal to one territorial motivation and not the other, but this is consistent with the general marketing of BOS:311 and the system more generally. It is typically pitched around “[making] Boston more beautiful” and less about defending one’s neighborhood. Importantly, we cannot confirm whether these differences existed before downloading BOS:311 or whether they emerged after, or even in response to, the use of BOS:311. Nonetheless, they would seem to indicate that BOS:311 users express even more caretaking for their neighborhood than traditional users do but are less defensive of it, indicating a moderately different type of user.
|
TABLE 6.1 Demographic comparisons between survey respondents who used BOS:311 and those who exclusively used traditional channels |
||||||||||
|
Trad. users (%) |
BOS:311 users (%) |
Trad. users (%) |
BOS:311 users (%) |
|||||||
|
Gender |
Ethnicity |
|||||||||
|
Male |
215 (44) |
121 (64) |
White |
371 (76) |
166 (88) |
|||||
|
Female |
271 (56) |
67 (36) |
Black |
53 (11) |
7 (4) |
|||||
|
Hispanic |
15 (3) |
2 (1) |
||||||||
|
Asian |
9 (2) |
2 (1) |
||||||||
|
Other |
38 (8) |
11 (6) |
||||||||
|
Education level |
Age |
|||||||||
|
High school or less |
34 (7) |
2 (1) |
18–24 |
9 (2) |
1 (1) |
|||||
|
Some college |
75 (15) |
15 (8) |
25–34 |
67 (14) |
49 (26) |
|||||
|
Professional degree |
11 (2) |
6 (3) |
35–44 |
98 (20) |
62 (33) |
|||||
|
Associate’s degree |
33 (7) |
4 (2) |
45–54 |
126 (26) |
49 (26) |
|||||
|
Bachelor’s degree |
161 (33) |
65 (35) |
55–64 |
108 (22) |
17 (9) |
|||||
|
Master’s degree |
139 (28) |
80 (43) |
65–74 |
67 (14) |
9 (5) |
|||||
|
Doctoral degree |
33 (7) |
16 (9) |
>75 |
11 (2) |
1 (1) |
|||||
Question #2: Are BOS:311 Users More Active Custodians?
At first blush, app users were approximately as active as users of traditional channels, with 68 percent making a single report and 85 percent making three or fewer reports (versus 65 percent and 89 percent for users of traditional channels). This is based on an analysis limited to reports through the app. If, however, we take into account that 30 percent of BOS:311 users used the other channels, they do in fact exhibit greater overall activity, with only 51 percent making a single report and 74 percent making three or fewer. BOS:311 users had an estimated median geographical range of 120.4 m, compared to 74 m for traditional users (Wilcox test: p < .001). This was accompanied by a higher chance of having a range greater than 800 m (24 percent vs. 11 percent). Though these differences are certainly measureable, it is worth noting that the average app user is still acting as a custodian exclusively within a range of fewer than two blocks of home.
To do a more formal analysis of these differences, we can again compare BOS:311 users and traditional users living in the same CBG to each other using multilevel models. 25 Consistent with the initial descriptive statistics, both models found that BOS:311 users were more active reporters than were traditional users living in the same neighborhood (frequency of reporting: β = 1.06, p < .001; geographic range of reporting: β = 2.43, p < .001). Unpacking these parameters, we find that the average custodian using traditional channels living in the average neighborhood was expected to report approximately two public issues (2.05) and have an estimated geographic range of 110 m. 26 In contrast, the average custodian using BOS:311 living in the average neighborhood was expected to report approximately six public issues (5.93) and have an estimated geographic range of 226 m. 27
What Matters: The App or App Users?
Thus far, we have learned three things about BOS:311 users: (1) they are largely individuals who were not already using the 311 hotline; (2) they tend to be more highly educated and express greater caretaking for their neighborhoods than traditional users; and (3) they report public issues more often and over a greater geographic range than traditional users. This raises the question of whether differences in reporting can be attributed to the smartphone application itself or result from the distinctive predispositions of those who have chosen to download and use it in the first place. We can use the survey to pursue this more nuanced analysis. In previous chapters, use of the survey has been limited to those with confirmed addresses, but here for the first time we use estimated census tracts of residence in order to incorporate more BOS:311 users, as very few had addresses on record. 28 The models that follow predict frequency and geographic range of reporting in exactly the same way as those presented in Chapter 3 , except that they include BOS:311 as an additional predictor of custodial activity.
Indeed, we find that BOS:311 users reported more public issues than those using traditional channels (β = 0.25, p < .01), even when controlling for demographics and attitudes (though these effects remained as observed in previous chapters). The overall impact, though, is less dramatic, indicating only about one more report than for a traditional user with the same demographics and level of territoriality. 29 BOS:311 predicted a greater geographic expansion in custodianship than previously seen, amounting to approximately 200 m (β = 0.45, p < .001). Within their different levels of activity, however, BOS:311 users and traditional users reported the same mix of issues. BOS:311 users were no more or less likely to report natural deterioration (β = 0.10, p = ns ) or man-made incivilities (β = −0.15, p = ns ).
Question #3: Does BOS:311 Create a Greater Connection with the Community?
One of the hopes behind BOS:311 and other civic tech is that it will elicit thick engagement from users, thereby constructing the bridge to citizenship anticipated by coproduction. This requires actually helping Bostonians connect with their communities. Our survey also included a series of questions regarding how well 311 facilitated such interactions and sentiments. Two referenced the neighborhood level (“[It’s helpful for] seeing who cares about your community” and “[It’s helpful for] connecting you with others in your community”), and one referenced the level of the city (“[It’s helpful for] connecting you with the city of Boston”). Using models similar to those that were limited to custodians, we see a mixed result. BOS:311 users did report a greater sense of connection with the broader city through the app (β = 0.36, p < .05) but a lesser connection with others in their community (β = −.56, p < .01). This latter result was emphasized by the fact that BOS:311 users averaged 2.10 on connecting with the community, indicating that it was generally seen as not being a component of the system (falling under the Likert-scale neutral point of 3).
Summary: The Nuanced Success of BOS:311
The impact of BOS:311 is characterized by both successes and shortcomings. On the positive side, it attracted new users, as 70 percent had never used the other 311 channels. The overall population of BOS:311 users also reported more public issues and exhibited this custodianship over a greater geographical range. Each of these positive discoveries, however, comes with qualifications. First, the expansion of BOS:311 was limited largely to those with a higher level of education, doing little to narrow any digital divide in access to government services. There is some suggestion that Asian and Hispanic neighborhoods have more BOS:311 users than would be expected based on their use of traditional channels, but this will require further examination. Second, greater reporting by BOS:311 users is largely driven by those using the app in conjunction with traditional channels. This is not problematic on its own, as the city of Boston never intended the app to replace or stand apart from the rest of the system. It does mean, however, that this global tendency toward more frequent reporting is driven only by the 30 percent of app users who utilize multiple channels and not by the app itself. Third, though BOS:311 users have expanded geographical ranges, they are still reporting largely from their home neighborhood. The consensus of the analyses is that app users have approximately a 100–200 m greater range than traditional users, still seating them squarely within a quarter mile of their homes.
Caveats aside, BOS:311 has tangibly increased custodianship in the city. Nonetheless, it appears to fall short of the thick engagement to which civic tech aspires. One of the goals of BOS:311’s founders and managers is to increase custodianship not only in action but in sentiment, and thus stimulate further participation in civic life. We cannot determine from the current study design whether BOS:311 actually leads to other forms of civic and political participation, but we were able to describe the connection it does or does not create between users and their community. The app appears to impart to users a connection to Boston as a whole, which is an important breakthrough. In Chapter 5 , we demonstrated that it is this broader geographical perspective that links a civic disposition to custodianship. However, the behavior itself does not bear this out, as people are still reporting largely in their own neighborhoods. In contrast, the app has engendered less of a connection with users’ local communities, the very locus of their custodianship. Thus, one might quip that BOS:311 users have acknowledged the bridge to citizenship but have not necessarily crossed it, either by extending their custodianship geographically or intensifying it locally.
311 Talks Back: Bringing Public Works Out of the Shadows
Each of the first two interventions successfully increased custodianship by engaging or amplifying territoriality. The flyer experiment made the home neighborhood more salient. BOS:311 made reporting issues more convenient but within the limitations of people’s natural range of custodianship. We have yet to see an innovation, however, that constructs the promised “bridge to citizenship” by engaging a civic disposition and eliciting reporting beyond the bounds of the home neighborhood. The city of Boston conducted a third policy experiment that pursued this in a nearly literal sense, taking advantage of BOS:311’s capacity for two-way communication to increase direct interaction between providers of basic city services and the users of 311. In 2014, the city began sending messages to individual BOS:311 users when their requests were closed. These messages typically included some picture as evidence of the completed job, such as a worker filling the previously cracked sidewalk, or a team standing next to the streetlight they had replaced. The hope was that these messages would emphasize the commitment of the government to public maintenance and thereby validate and reinforce participation by the public, possibly even leading to the persistent engagement anticipated by proponents of civic tech.
The city of Boston conducted this experiment in messaging in collaboration with three professors interested in how such interactions can influence trust in and engagement with government: Ryan Buell and Michael Norton, professors at Harvard Business School, and Ethan Porter at the University of Chicago. Their inspiration stemmed from a marked decline in Americans’ faith in government over the past few decades. 30 Part of the problem appears to be that there are many programs and services provided by government that most people do not understand or even know of. This has been referred to by Suzanne Mettler as the “submerged state.” 31 Take the example throughout this book of the maintenance of public spaces and infrastructure, which traditionally is a task that occurs quietly and without fanfare. When the work is being done most efficiently, it is hard to notice because problems have been fixed so rapidly. When there are issues with the public infrastructure, it is quite easy to blame it on public works employees, whether it is their fault or not. In this way, it falls into that set of thankless tasks that fit the adage: “If you’re being noticed, you’re not doing your job correctly.”
Buell, Porter, and Norton hypothesized that trust in government might be bolstered by bringing the submerged state to the surface through greater transparency. This was rooted in a similar discovery they had gained through studies of the private sector: when people can see evidence of the effort being invested in a product, they rate it as being of higher quality, are more satisfied with it, and are more grateful toward the company that made it. 32 Transparency might hold a similar value for the relationship between government and the public. Importantly, Buell, Porter, and Norton were not advocating for transparency in the sense of the Sunlight Foundation’s push for Open Data or the Freedom of Information Act but rather for an operational transparency that reveals how work is accomplished. This would give government a chance to draw attention to its successes, thereby making the case for its own legitimacy and crucial role in society. The BOS:311 messages were a clear instance of such communication, and they provided a unique opportunity to observe whether operational transparency might not only engender trust but also encourage sustained participation in a government program. Like the other studies in this book, Buell, Porter, and Norton’s study exploited the continuous nature of 311 data for evaluation. They examined specifically whether BOS:311 users who received the messages actually made more reports or reported a greater variety of case types in the following months. 33 I have since worked with them to analyze also whether the messages encouraged a greater geographic range of reporting, as this would be the best evidence that the messages evoked a civic disposition.
The Experiment
In September 2014, the city of Boston released a new version of BOS:311 34 that permitted city departments to send messages to the user who made the request when it was closed. The city leveraged this capacity to send closure messages that included images of the work performed or the city workers who performed it. These acted as a visual confirmation of the work done to fulfill their service request. The new version of BOS:311 was “pushed” to all individuals who already had BOS:311 installed on their phones. It diffused rapidly, achieving more than a 95 percent penetration rate within three months. Reporters received the new closure messages with accompanying images only if the work team responsible for a particular case uploaded one.
As in the analysis of BOS:311 usage itself, this is more appropriately described as a quasiexperiment, because the messages were not randomized across users. Whether and when a 311 user experienced the “treatment” of operational transparency depended on two things: (1) having downloaded the new version of BOS:311 and (2) having made a report that was later responded to with an image of the completed work. For this reason, exposure was staggered across multiple months, with some individuals experiencing it earlier than others. Methodologically, we can best specify the effects of the treatment by analyzing whether and how an individual’s reporting behavior changed after receiving the visual confirmation that their request had been closed. Over the 14 months following the initial implementation of the new version of BOS:311, 21,786 users were exposed to the treatment. 35 The study analyzed each individual’s behavior in each month following their initial installation of the app through panel models that included fixed effects for each user, thereby controlling for an individual’s more general tendencies. The models tested the frequency and geographical range of reporting in a given month while controlling for how many total requests an individual had made before that month, the proportion of those requests that were closed (i.e., to control for objective satisfaction), and the amount of time since downloading BOS:311. This amounted to a final sample of 371,992 person-months. 36 See Appendix E for all model details.
Impacts of Closure Messages
The most apparent and consistent finding was that the messages from the city were successful in eliciting greater use of BOS:311, with those who had received one making more reports (β = 0.81, p < .001). As might be expected, this effect was strongest in the first month after receiving the message (β = 1.63, p < .001) but persisted for as long as a year (all β for 11 months or fewer after exposure > .39, p -values < .001). An intriguing wrinkle to this finding was the way the messaging interacted with satisfaction with the system. Unsurprisingly, if the city had successfully closed more of an individual’s requests, he or she was more likely to use the app more often (β = 1.60, p < .001), but this also modulated the effect of transparency (β = 18.07, p < .001). The effect of messaging was only present if an individual’s requests had been regularly fulfilled by the city. In contrast, if the city had not consistently fulfilled the user’s requests, the message actually depressed reporting. This effect was nonlinear, though, with messaging having the greatest impact on people who had ∼80 percent of their requests fulfilled (more details are available in Appendix E ).
The effect on reporting range was more equivocal. It seemed as though those who had experienced the transparency messages made reports over a narrower range rather than a broader one (β = −0.33, p < .001), though this effect was tempered by whether the message had been received that month (β = 0.18, p < .001). Likewise, those who had received at least one closure message were no more likely to report outside their home cluster (β = −0.17, p = ns ). Before we interpret the transparency messages as causing people to report closer to home, however, we should note that the results translate to reports being about 20 m closer to home than otherwise expected. It is very possible that the statistical significance of the result is an artifact of the large sample size. Even if this is not the case, the actual difference in reporting distance is pretty negligible in practical terms. 37
Summary: Operational Transparency Increases Reporting, but not Range
The success of the BOS:311 closure messages has a certain irony to it. In some ways, the goal of operational transparency is most resonant with the 1990s philosophy of new public management, in which government sought to satisfy its “clients” with techniques developed by the private sector. Indeed, the primary spirit is to effectively communicate the work being done for the public. Nonetheless, within the collaborative context of a coproduction program, encouraging the use of services is actually equivalent to eliciting more participation. This raises the question of how BOS:311 users are interpreting the messages. Do they see them as evidence of a service provider doing its job effectively or as a signal that a collaborator is living up to its end of the bargain? It is not clear which users interpreted it in each way, or whether this would influence their behavior.
One will note that the messages failed to lead to a broader geographical range of reporting. Building on this discovery, I would posit that any hope for the bridge to citizenship would rely on communications that cast the relationship between government and the public as one of collaborative partners rather than service provider and client. If the public sees themselves as clients, using services will be driven by their basic inclinations. We would then expect exactly what we saw here—greater participation but still within the invisible boundaries of one’s own “territory.” But if the public perceives themselves as active collaborators in a joint effort to maintain the urban commons, they might be inclined to expand their custodianship to more spaces. This is speculative at this time but is worth considering in future experiments in operational transparency.
Conclusion
This chapter has described three policy experiments that evaluated innovations to Boston’s 311 system. As a whole, they reiterated the role of territoriality in motivating custodianship while also highlighting the limitations its importance creates for eliciting greater participation in the program. They also made clear just how challenging it is to construct an effective bridge to citizenship in the manner anticipated by proponents of coproduction or, analogously, to instigate the thick engagement so sought after by civic tech. The flyer experiment found that neighborhood-oriented messages, which in theory targeted territorial sentiments, were more effective than messages that reference Boston as a whole. BOS:311 was successful in eliciting more frequent reporting over a greater geographical range, but the magnitude of this latter effect was relatively modest (∼100–200 m), seemingly constrained by territoriality. Furthermore, BOS:311 users reported little connection with the local community, epitomizing thin engagement. Similarly, the messages that provided BOS:311 users with visual confirmation that their requests had been completed led to increased reporting of cases. Again, this elevated activity did not lead to a meaningfully broader geographic range of custodianship.
I take from these results three main lessons for 311 systems and the collaborative model of the maintenance of the urban commons. The first is that territoriality not only matters but should also be a prominent consideration when designing outreach, messaging, and implementation for such programs. The administrators of 311 systems and their colleagues should probably anticipate that even when they succeed in eliciting more participation, the increases will occur near the home of each individual user. This knowledge on its own is powerful because it calls for a more targeted geographical approach. Rather than assuming that generalized increases in participation will be useful, a city’s leaders might focus on encouraging custodianship in neighborhoods or even on streets that have lower levels of engagement with the program. This leads to the second lesson. Although I have spent a lot of time highlighting how BOS:311 and the closure messages did not succeed in their grander vision of constructing a bridge to citizenship, they did increase custodianship in the city. This is no small feat, and it also highlights an important caveat that we have not yet addressed in much detail, that just because an individual engages in custodianship either predominantly or exclusively in spaces adjacent to her home, that does not mean she attends to all of the issues within that radius. For the vast majority of Boston’s 311 users, there are clearly opportunities for them to be more engaged, even in the few blocks surrounding their homes. Programs that make this a reality are offering tangible value. Third, I do not want the reader to conclude that there is no such thing as a “bridge to citizenship.” These are just three initial experiments, and there are certainly other innovations that might successfully connect territoriality with a civic disposition and break the invisible geographic boundary of the “home neighborhood.”
Are these results problematic for coproduction and civic tech? I would argue “No,” but it depends on how you define success in each context. For coproduction, these experiments further indicate the need for the public-as-partner model. Coproduction programs are effectively leveraging a variety of different motivations, but to assume that a civic disposition is either largely or solely responsible for participation creates a substantial blind spot. Similarly, the difficulty for civic tech is semantic, resting on the distinction between civic motivations and civic impacts (i.e., positive impacts on the public good). Personally, I believe that civic tech has done its job if it has tangible impacts, be they via thin or thick engagement, but for those who see the goal as bolstering civic motivations, the results here are underwhelming. Of course, as discussed in Chapter 5 , a civic disposition might play a greater role for other programs, especially when they require a greater time investment or hands-on interaction with public officials and other community members. This adds to the assessment by some that civic tech’s effect on civic life will be more incremental than it is transformative. 38 In the end, it appears that proponents of both coproduction and civic tech need to take a pluralistic approach when considering why people might participate in a given program or activity.
Returning to the broader themes of this book, the progression of Parts II and III has illustrated the virtuous cycle of science and policy made possible by urban informatics. The 311 system generated a data set that was able to advance our understanding of how community members contribute to the maintenance of the commons. These very insights supported a new perspective on how programs such as 311 most effectively encourage public participation, suggesting a series of innovations. These innovations then took the form of experiments that could be evaluated through the 311 system’s data, further assessing and adding nuance to theories about why people report issues to city services departments. Thus the cycle begins anew. Whether this process concluded with more or less support for civic tech was irrelevant to this story, as it would illustrate the opportunity to learn and iterate on policy in a dynamic fashion either way. One particular concern that has arisen, however, is that of the digital divide and whether civic tech is expanding engagement only for those with ready access to smartphones and related tools. This, however, is not the only digital divide that threatens to beset urban informatics. As we will see in Part IV , the rapid emergence of technological tools has created haves and have-nots in other dimensions as well—between cities of different sizes and between the public, private, and nonprofit sectors.