Cambridge Handbook of Experimental Political Science

Chapter 34 Analyzing the Downstream Effects of Randomized Experiments

The work in this volume provides profound evidence of the value of randomized experimentation in political science research. Laboratory and field experiments can open up new fields for exploration, shed light on old debates, and answer questions previously believed to be intractable. Although acknowledgment of the value of experimentation in political science is becoming more commonplace, significant criticisms remain. The oft-repeated shortcomings of experimental research tend to center on the practical and ethical limitations of randomized interventions. I begin this chapter by detailing some of these criticisms and then explore one means of extending the value of randomized interventions beyond their original intent to ameliorate some of these same perceived limitations.

One of the most prominent critiques of this genre is that randomized experiments tend to be overly narrow in scope in terms of time frame and subject matter, as well as high in cost. Although short-term experiments may incur costs similar to observational research, they often focus on a single or just a few variations in an independent variable, seemingly limiting their applicability to a breadth of topics that a survey could cover. The high cost associated with long-term data collection and the necessity of maintaining contact with the subjects involved impedes the likelihood of gathering information on long-term outcomes. There are also few incentives to conduct interventions in which the impacts may only be determined years down the road. Such studies are not amenable to dissertation research unless graduate students extend their tours of duty even longer, nor do they suit junior faculty trying to build their publication records. The isolation of long-term effects necessitates long-term planning, maintenance, and funding, all of which tend to be in short supply for many researchers.

The second practical critique of randomized experiments stems from the difficulty of such studies to capture the influence of variables of interest to many political scientists. We cannot purposefully alter political culture to see its effect on democratization, nor can we alter the family background of a set of individuals to investigate its influence on political socialization. Perhaps, some might argue, experiments are only useful in studying narrow topics of political behavior in highly controlled settings, mitigating their applicability to broad real-world questions.

Randomized interventions also face ethical limitations and critiques. Prohibitions on potentially harmful interventions need little discussion here. We must consider experiments testing political variables of interest that may fall into an ethical gray area. There are more possibilities of these than one might initially assume. As in medical testing, controversy exists concerning the practice of denying a known good to a subject as an evaluative technique. Subjecting participants to varying treatment regimens might indicate an underlying assumption that one such regimen is “better” than another. If this is indeed the case, then practitioners are left open to the charge that they are somehow adversely affecting one group of people over another by failing to provide them with the “better” treatment option. For example, intentionally boosting some individuals’ levels of educational attainment in comparison to others in an effort to examine the ramifications of additional years of schooling on political and behavioral outcomes seems unethical given the widespread belief in the positive externalities associated with schooling.

Also, ethically dubious is the notion that some interventions could have the long-term consequence of influencing social outcomes. Artificially enhancing randomly selected candidates’ campaign coffers to test the effects of money on electoral success could affect electoral outcomes and the drafting and passage of legislation. Randomly assigning differential levels of lobbying on a particular bill could sway the drafting and passage of legislation. Testing the effectiveness of governmental structures through random assignment of different types of constitutions to newly formed states could result in internal strife and international instability.

The list of interventions that, although useful for research and pedagogical purposes, would be simply impractical or unethical is seemingly endless, whereas the universe of interventions that are both feasible and useful appears somewhat limited in scope. Does this mean that experiments will only be useful in answering questions where it is practical and ethical to manipulate variables of interest? The answer is no for myriad reasons discussed in this volume and elsewhere. Here I focus on just one: we can expand the utility of initial interventions beyond their original intent through examination of the long-term, and sometimes unforeseen, consequences of randomized interventions. Using downstream analysis, political scientists can leverage the power of one randomized intervention to examine a host of causal relationships that they might otherwise have never been able to study through means other than observational analysis. Although political scientists may not randomly assign some politicians to receive more money than others, some other intervention, natural or intended, may produce such an outcome. Researchers can exploit this variation achieved through random or near-random assignment to examine the consequences of this resulting discrepancy in campaign finances on other outcomes such as electoral success.

In the next section, I further define and detail the key assumptions associated with downstream analysis of randomized experiments. I then highlight research that uses downstream analysis and outline some potential research areas that may benefit from this outgrowth of randomized interventions. I conclude with a discussion of the methodological, practical, and ethical challenges posed by downstream analysis and offer some suggestions for overcoming these difficulties.

1. Extending the Benefits of Randomized Experiments

For most researchers, a randomized experiment ends once the treatment is applied and the outcome measures are collected. This seeming finality belies the possibility that many, although perhaps not all, treatments produce ramifications that extend well beyond the original time frame and purpose of the study. The unintended consequence of these interventions is that they often provide an exogenous shock to an outcome not normally amenable to classic randomization. This allows researchers to extend the positive externalities of experimentation to different fields of study and interest; this is especially useful for fields in which experiments are often untenable for practical and ethical reasons. Green and Gerber (2002) define these downstream benefits as “knowledge acquired when one examines the indirect effects of a randomized experimental intervention” (394).

Analyzing the second-order consequences of randomized experiments can help justify some of the perceived limitations often associated with such endeavors. Downstream analysis opens up the possibility of extending a narrowly construed topic or outcome to apply to a broader range of fields. Experiments on the utility of direct mail as a mobilization tool can become investigations into the causes of habitual voting (Gerber, Green, and Shacher 2003). Random assignment of a local political office reserved for women can become investigations into the long-term effects of breaking the electoral glass ceiling for women (Bhavnani 2009). This type of application is best seen through an example involving the effects of schooling on participation.

The positive effect of formal schooling on political and civic participation is widely reported in observational research (e.g., Berelson, Lazarsfeld, and McPhee 1954; Campbell et al. 1960; Converse 1972; Verba and Nie 1972; Wolfinger and Rosenstone 1980; Delli Carpini and Keeter 1996; Miller and Shanks 1996; Nie, Junn, and Stehlik-Barry 1996), but some recent research questions the underpinnings of this relationship (Tenn 2007; Kam and Palmer 2008; Sondheimer and Green 2010). Many scholars tend to view this link as causal despite failure to clarify the causal mechanisms that produce this strong relationship and an inability to isolate educational attainment from unobserved factors such as cognitive ability and family background. Observational analysis of the topic is faced with diminishing returns; random assignment of varying levels of educational attainment to otherwise similar subjects would advance current knowledge in this field but is impossible for moral and practical reasons. It is possible to still leverage the benefits of experimental methods in this field. Randomized trials to test different education techniques and policies often produce differential levels of schooling between treatment and control cohorts. The second-order effects of these interventions, if observational analysis is correct, should also produce differential rates of political participation as a result of the boost given to the average years of schooling among the treatment cohort. Although these studies were designed to examine the value of programs such as public preschool and small class size, political scientists can use the exogenous shocks to years of schooling brought about by randomized interventions to examine the effects of high school graduation on voting behavior and other political and civic participation outcomes.

In the next section, I detail how we can estimate such effects, continuing with the example of the downstream possibilities for using randomized educational interventions to untangle the causal relationship between educational attainment and political participation.

2. A Model and Estimation of Downstream Effects of Randomized Experiments

Estimation of Local Average Treatment Effect

The assumptions underlying the Rubin causal model and the use of instrumental variables (IVs) estimation to overcome unobserved heterogeneity are laid out by Sovey and Green (2011) and adapted here for the downstream analysis of experiments. The IV estimator consists of a two-equation model and is written as follows:

(1)

(2)

In Equation (1), we assume that the individual voter turnout (Y_i) is the dependent variable, educational attainment (X₁) is the regressor of interest, Q_1i Q_2i,…Q_Ki are covariates, and u_i is an unobserved disturbance term. Equation (2) holds that the endogenous regressor (X_i), educational attainment in this case, is a linear function of an instrumental variable (Z_i), the covariates, and an unobserved disturbance term (e_i). Random assignment of Z allows us to isolate exogenous variation in X, that is, the piece of X that is independent of other factors that might influence Y, overcoming concerns of omitted variable bias in our estimation of the effect of X on Y.

To see what IV is estimating, it is useful to introduce the Rubin causal model as presented by Angrist, Imbens, and Rubin (1996). I apply this model to our running example of using a randomized educational intervention to isolate the effects of graduation from high school on individual voter turnout. Here, we make use of three dichotomous variables coded as either 0 or 1: 1) individual assignment to the treatment or control cohort of the intervention (Z_i), 2) whether an individual graduated from high school (X_i), and 3) whether an individual voted in a given election (Y_i).

The preceding IV estimator appears to calculate the causal effect of X on Y, but as we see in this chapter, closer examination shows that the estimator isolates the causal effect for X on Y for those subjects affected by the initial intervention – subjects whose X outcome changed due to the introduction of Z. Angrist and Krueger (2001) define this estimand as the local average treatment effect (LATE).¹ Speaking in terms of LATEs alerts us to the fact that IV reveals the causal influence of the intervention for a subset of the population. We need not assume that the initial intervention affected all treatment subjects in the same way or that the intervention determined the outcome for all subjects in the treatment cohort.

The first step in estimating the LATE model is to conceptualize the effects of randomized assignment in the intervention on educational outcomes. In discussing estimation techniques for randomized experiments, Imbens and Rubin (1997) group subjects into four categories based on how the treatments they receive depend on the treatments to which they are assigned. In the case of downstream analysis of randomized interventions, the concept of compliance is applied somewhat differently. In the case of downstream analysis of educational interventions, we define compliance in terms of whether people graduate high school in response to being assigned to the treatment group. In the context of a downstream analysis, Imbens and Rubin's four groups are as follows:

1. Compliers graduate from high school if and only if they are assigned to the treatment group (z_i = 1, x_i = 1) or (z_i = 0, x_i = 0);

2. Never-takers do not graduate from high school regardless of the group to which they are assigned (z_i = 0, x_i = 0) or (z_i = 1, x_i = 0);

3. Always-takers graduate from high school regardless of the group to which they are assigned (z_i = 0, x_i = 1) or (z_i = 1, x_i = 1); and

4. Defiers graduate from high school if and only if they are assigned to the control group (z_i = 0, x_i = 1) or (z_i = 1, x_i = 0).

Note that we cannot look at a given individual and classify him or her into one of these mutually exclusive groups because we are limited to only observing one possible outcome per individual. In other words, if a subject assigned to the treatment group graduates from high school, then we cannot discern whether he or she is a complier who could have only graduated if assigned to the treatment group or an always-taker who would have graduated regardless of random assignment in the intervention.

To draw causal inferences from an instrumental variables regression based on downstream data, one must invoke a series of assumptions. The first is independence: potential outcomes of the dependent variable must be independent of the experimental group to which a person is assigned. This criterion is satisfied by random assignment.

The second is the exclusion restriction. The instrumental variable, Z, only influences Y through its influence on X. In other words, Y changes because of variation in X and not because of something else. In this case, we assume that random assignment in the original experiment has no influence on voting outcomes aside from that mediated by graduation from high school. Evaluating whether a given instrument satisfies this condition rests on understanding the nature of the relationship between an observed variable (Z_i) and an unobserved variable (u_i). Theoretically, random assignment can ensure the statistical independence of Z and u. However, the nature of the randomized intervention may lead to violations of the exclusion restriction – for example, subjects who realize that they are being studied may be influenced by the simple fact that they are part of an experimental treatment group. When evaluating the exclusion restriction, the researcher must consider causal pathways that might produce changes in Y through pathways other than X.

Third, we invoke the stable unit treatment value assumption (SUTVA), which holds that an individual subject's response is only a function of his or her assignment and educational attainment and is not affected by the assignment or outcomes of other subjects. Fourth, we assume monotonicity, that is, the absence of defiers. Finally, we must assume that Z exerts some causal influence on X. Low correlation between Z and X can lead to small sample bias, a problem discussed in Section 4.

We are primarily interested in the relationship between graduation from high school and its subsequent effect on the likelihood of voting. As such, we can say that the dependent variable for each subject i can be expressed as either y_i₁ if the individual graduated from high school or y_i₀ if the subject did not graduate from high school. The mechanics of downstream analysis can be illustrated by classifying subjects’ outcome measures based on assignment and response to the educational intervention, producing four categories of subjects:

1. Individuals who voted regardless of whether they graduated from high school (y_i₁ = 1, y_i₀ = 1);

2. Individuals who voted if they graduated from high school but did not vote if they did not graduate from high school (y_i₁ = 1, y_i₀ = 0);

3. Individuals who did not vote if they graduated from high school but did vote if they did not graduate from high school (y_i₁ = 0, y_i₀ = 1); and

4. Individuals who did not vote regardless of whether they graduated from high school (y_i₁ = 0, y_i₀ = 0).

As detailed in Table 34.1, based on the four compliance possibilities and the four classifications of outcome measures mediated by educational attainment, we can create a total of sixteen mutually exclusive groups into which subjects may fall. The total subject population share of each group is expressed as π_j such that ∑_j¹⁶ π_j = 1. To estimate the causal effect between our variables of interest, we must further assume monotonicity (Angrist et al. 1996) – that the treatment can only increase the possibility of graduating from high school. This assumption holds that there are no defiers or that π₁₃ = π₁₄ = π₁₅ = π₁₆ = 0.

Table 34.1: Classification of Target Population in Downstream Analysis of Educational Intervention

Source: Adapted from Sovey and Green (2011).

^a This share of the population votes if assigned to the treatment group.

^b This share of the population votes if assigned to the control group.

Randomization allows us to leverage the variation brought about by an exogenous shock to an otherwise endogenous measure to isolate the influence of that measure on another outcome of interest. As such, we can only identify the influence of schooling on the outcomes of a limited number of the groups of subjects classified in Table 34.1. We cannot evaluate the effects of schooling on the voting behavior of the never-takers who will not graduate from high school regardless of assignment or the always-takers who will graduate from high school regardless of assignment. These outcomes are determined by factors that might be endogenous to our voting model. To isolate the effects of educational attainment on voting, we can only estimate the schooling's effects on the outcomes of the compliers, who graduate from high school if assigned to the treatment but do not graduate if assigned to the control. This complier average causal effect (CACE) is expressed as

(3)

A randomized experiment and application of the Rubin causal model allows us to estimate the LATE given four previously discussed assumptions: 1) SUTVA, 2) the exclusion restriction, 3) a nonzero effect of assignment on graduation rates, and 4) monotonicity. As shown later in this chapter, if these assumptions are met, then LATE estimation will also provide us with a consistent estimator of the CACE.

The first step is to estimate the voting rates, V, for the treatment and control groups. As the number of observations in the control group approaches infinity, the observed voting rate in the assigned control group

can be expressed as

(4)

(5)

Next, we can find a consistent estimator of the proportion of compliers (α), subjects who only graduated from high school if and only if assigned to the treatment regimen, by subtracting the graduation rate of the control group from the graduation rate of the treatment group:

(6)

Finally, we can combine the estimators presented in Equations (4)–(6) to provide a consistent LATE estimate that is the same as the CACE presented previously:

(7)

The estimator for the LATE also estimates the CACE, the effect of educational attainment among those who would not have graduated save for the intervention (the compliers).

In conclusion, the IV estimator estimates CACE. We are not estimating the effect of high school for everybody, just the effect for those who are influenced by a high school inducing program. Of course, one can generalize beyond compliers, but this must be done cautiously and through replication unless one is willing to make strong assumptions. The downstream analysis of an individual intervention might be best interpreted as a LATE estimate, but, if a persistent pattern emerges through the downstream analysis of multiple interventions with different target populations, we can then begin to extrapolate the results to a more broad population.

3. Downstream Analysis in Practice

“Downstream experimentation” is a term originally coined by Green and Gerber (2002). The concept of using existing randomized and natural experiments to examine second-order effects of interventions was slow to build due, in large part, to the relative dearth of suitable experiments in political science. Now that experiments are becoming more widespread and prominent in political science literature, scholars are beginning to cull the growing number of interventions to test theories seemingly untestable through traditional randomization. In this section, I touch on just a few such examples and offer avenues for further exploration using similar first-order experiments. This discussion is meant to encourage those interested to seek out these and other works to explore the possibilities of downstream analysis.

As I discussed previously, an interesting stockpile of experiments well suited for downstream analysis are interventions designed to test public policy innovations, programs in education in particular. While a key variable of interest to many is the influence of education on a wide array of political and social outcomes, one's level of educational attainment is itself a product of numerous factors, potentially impeding our ability to isolate schooling's causal effect through standard observational analysis (Rosenzweig and Wolpin 2000). At the same time, it is nearly impossible and potentially unethical to randomize the educational attainment of individuals or groups to gauge its effect. Sondheimer and Green (2010) examine two randomized educational interventions, the High/Scope Perry Preschool project examining the value of preschool in the 1960s and the Student-Teacher Achievement Ratio program testing the value of small classes in Tennessee in the 1980s, in which the treatment groups witnessed an increase in years of schooling in comparison control groups. They used these differential levels of schooling produced by the randomized interventions to isolate the effects of educational attainment on likelihood of voting, confirming the strong effect often produced in conventional observational analysis. In addition to examinations of voter turnout, downstream analysis of educational interventions can isolate the effects of years of schooling on a range of outcomes, including views on government, party affiliation, civic engagement, and social networking.

As discussed throughout this volume (see, in particular, Michelson and Nickerson's chapter in this volume), experimentation is proliferating in the field of voter mobilization. Scores of researchers conduct randomized trials to estimate the effectiveness of different techniques aimed at getting people to the polls. In doing so, these studies create the opportunity for subsequent research on the second-order effects of an individual casting a ballot when he or she would not have done so absent some form of intervention. Gerber et al. (2003) use an experiment testing the effects of face-to-face canvassing and direct mail on turnout in a local election in 1998 to examine whether voting in one election increases the likelihood of voting in another election. Previous observational research on the persistence of voting over time is unable to distinguish between the unobserved causes of an individual voting in the first place from the potential of habit formation. Gerber et al. find that the exogenous shock to voting produced within the treatment group by the initial mobilization intervention in 1998 endured somewhat in the 1999 election, indicating a persistence pattern independent of other unobserved causes of voting. Further extension of mobilization experiments could test the second-order effects of casting a ballot on attitudes (e.g., internal and external efficacy), political knowledge, and the likelihood of spillover into other forms of participation.

Laboratory settings and survey manipulation offer fruitful ground for downstream analysis of randomized experiments. Holbrook's chapter in this volume discusses experiments, predominantly performed in laboratories or lablike settings or in survey research, that seek to measure either attitude formation or change as the dependent variable. As she notes, understanding the processes of attitude formation and change is central to research in political science because such attitudes inform democratic decision making at all levels of government and politics. Experiments seeking to understand the causes of attitude formation and change can use downstream analysis to examine the second-order effects of these exogenously induced variations on subsequent beliefs, opinions, and behaviors. For example, Peffley and Hurwitz (2007) use a survey experiment to test the effect of different types of argument framing on support for capital punishment. Individual variation in support for capital punishment brought about by this random assignment could be used to test how views on specific issues influence attitudes on other political and social issues, the purpose and role of government generally, and evaluations of electoral candidates.

Natural experiments provide further opportunity for downstream examination of seemingly intractable questions. In this vein, scholars examine the second- and third-order effects caused by naturally occurring random or near-random assignment into treatment and experimental groups. Looking to legislative research, Kellerman and Shepsle (2009) use the lottery assignment of seniority to multiple new members of congressional committees to explore the effects of future seniority on career outcomes such as passage of sponsored bills in and out of the jurisdiction of the initially assigned committee and reelection outcomes. Bhavnani (2009) uses the random assignment of seats reserved for women in local legislative bodies in India to examine whether the existence of a reserved seat, once removed, increases the likelihood of women being elected to this same seat in the future. The original intent of the reservation system is to increase the proportion of women elected to local office. Bhavnani exploits the random rotation of these reserved seats to examine the “next election” effects of this program once the reserved status of a given seat is removed and the local election is again open to male candidates. He focuses his analysis on the subsequent elections in these treatment and control wards, but one could imagine using this type of natural randomization process to examine a host of second-order effects of the forced election of female candidates ranging from changes in attitudes toward women to shifts in the distribution of public goods in these wards.

As the education experiments indicate, randomized policy interventions used to test new and innovative ideas provide fascinating opportunities to test the long-term ramifications of changes in individual and community level factors on a variety of outcomes. Quasiexperiments and natural experiments that create as-if random placement into treatment and control groups provide additional prospects. Many programs use lotteries to determine which individuals or groups will receive certain new benefits or opportunities. Comparing these recipients to nonrecipients or those placed on a waiting list evokes assumptions and opportunities similar to those of randomized experiments (Green and Gerber 2002). Numerous scholars in a range of disciplines (e.g., Katz, Kling, and Liebman 2001; Ludwig, Duncan, and Hirschfield 2001; Leventhal and Brooks-Gunn 2003; Sanbonmatsu et al. 2006) have examined the Moving to Opportunity (MTO) program that relocates participants from high-poverty public housing to private housing in either near-poor or nonpoor neighborhoods.

Political scientists can and have benefited from this as-if random experiment as well. Observational research on the effects of neighborhoods on political and social outcomes suffers from self-selection of subjects into neighborhoods. The factors that determine where one lives are also likely to influence one's proclivities toward politics, social networking tendencies, and other facets of political and social behavior. Downstream research into a randomly assigned residential voucher program allows political scientists the opportunity to parse out the effects of neighborhood context from individual-level factors that help determine one's choice of residential locale. Political scientists are just beginning to leverage this large social experiment to address such questions. Gay's (2010) work on the MTO allows her to examine how an exogenous shock to one's residential environment affects political engagement in the form of voter registration and turnout. She finds that subjects who received vouchers to move to new neighborhoods voted at lower rates than those who did not receive vouchers, possibly due to the disruption of social networks that may result from relocation. Future research in this vein could leverage this and similar interventions to examine how exogenously induced variations in the residency patterns of individuals and families affect social networks, communality, civic engagement, and other variables of interest to political scientists.

Other opportunities for downstream analysis of interventions exist well beyond those discussed here. As this short review shows, however, finding these downstream possibilities often entails looking beyond literature in political science to other fields of study.

4. Challenges

Downstream analysis of existing randomized interventions provides exciting possibilities for researchers interested in isolating causation. We can expand the universe of relationships capable of study using assumptions generated by randomization and experimental analysis. Although downstream analyses can be used to overcome some of the limitations of randomized interventions, they do pose their own set of challenges. In this section, I discuss the methodological, practical, and ethical challenges faced by those who want to perform downstream analysis.

Methodological Challenges

Two key impediments to downstream analysis of randomized experiments stem from two of the three conditions for instruments to maintain the conditions for instrumental variable estimation, specifically finding instruments that meet the exclusion restriction and provide a strong relationship between assignment and the independent variable of interest. First, a suitable instrument must meet the exclusion restriction in that it should not exert an independent influence on the dependent variable. Randomization of subjects into treatment and control cohorts is not sufficient to meet the exclusion restriction because the specific nature of the treatment regimen can still violate this condition. Returning to our investigation of educational interventions as a means of isolating the influence of educational attainment on political participation, we can imagine myriad situations in which the intervention may influence participation, independent of the effects of schooling. If the intervention works through a mentoring program, then mentors may discuss politics and the importance of political involvement with subjects in the treatment group, increasing the likelihood of voting later in life. Similarly, it is possible that an intervention intended to boost the educational attainment of a treatment group also influences the family dynamics of the subjects’ home lives. If this occurs and the exclusion restriction is violated, then researchers will be unable to isolate the causal influence of variations in years of schooling on political participation independent of family background.

Another example of the potential violation of the exclusion restriction exists concerning Gerber et al.'s (2003) work on voting and habit formation. Recall that Gerber et al. use a randomized mobilization intervention to find that, all else equal, voting in one election increases the likelihood of voting in subsequent elections, indicating that electoral participation is habit forming. This result hinges on the assumption that no other changes took place for individuals in the initial experiment other than assignment to the control or treatment group. A violation of the exclusion restriction would occur if the outcome induced due to assignment influenced other factors believed to influence subsequent voting. For example, if political parties and other campaign operatives tend to reach out to those who already seem likely to vote (Rosenstone and Hansen 1993), then voting in the first election may increase the likelihood of being subject to increased mobilization in subsequent elections, increasing the likelihood of voting in subsequent elections. If this occurs and the exclusion restriction is violated, Gerber et al. (2003) would still be correct in arguing that voting in one election increases the likelihood of voting in subsequent elections, but this result may not be due to habit but to another factor such as outreach to likely voters.

There is no direct way to test whether this condition is met. Instead, we must make theoretical assumptions about the relationship between the instrument and potential unobserved causes of the variation in the dependent variable. In-depth knowledge of the original intervention and its possible effects on numerous factors relating to the outcome variable of interest is necessary to uncover possible violations.

The second methodological difficulty posed by researchers wanting to perform downstream analysis relates to the second condition necessary for consistent estimates using instrumental variables – that the instrument must be correlated with the endogenous independent variable. In downstream analysis, meeting this condition entails finding randomized experiments that “worked” insofar as random assignment produced variation in the treatment group as compared to the control. As Sovey and Green (2011) and Wooldridge (2009) discuss, use of a weak instrument – an exogenous regressor with a low partial correlation between Z_i and X_i – can cause biased IV estimates in finite samples. Fortunately, weak instrumentation can be diagnosed using a first-stage F statistic, as outlined by Stock and Watson (2007).

Finding a suitable instrument to meet this condition is a delicate matter because although researchers need to find an experiment with a strong result, we also need to be wary of interventions with too-strong results, an indication that something might be awry. As Green and Gerber (2002) discuss, such results might indicate a failure of randomization to create similar groups prior to the intervention. Randomized experiments with large numbers of subjects and replicated results provide the best potential pool of studies for downstream analysis.

Practical Challenges

The most straightforward challenge of downstream analysis of randomized experiments is that, in most circumstances, the analyst has no control over the course of the initial intervention. There are some opportunities to improve on the analysis of an original intervention, but, in general, the internal validity of downstream analysis cannot be much, if any, better than that of the original experiment. Downstream analysis cannot recover the value of a botched experiment any more than it can mar the results of a well-executed intervention.

In some cases, scholars can alter the original analysis of results to either narrow down or expand the reach of the intervention, such as by considering the effects of an intended treatment rather than the effects of the treatment on the treated. The ability of downstream analysts to make this type of decision depends on the depth and clarity of the description of the original intervention. In examining the original Perry intervention, Schweinhart, Barnes, and Weikart (1993) detailed their decision to move two subjects from the treatment group to the control group because of the subjects’ unique family concerns. Such documentation allowed Sondheimer and Green (2010) to move these subjects into the original intent-to-treat group for use in their own downstream analysis of the program. This example speaks to the necessity of meticulous record keeping and documentation of decisions made prior to, during, and following a randomized intervention.

A second and related practical challenge is that of subject attrition. Depending on the topic of the investigation, there can be a long lag time between the initial intervention and the collection of new data for downstream analysis. The longer the window between the initial intervention and the downstream analysis, the more likely one is to face high levels of attrition. This loss of data becomes increasingly problematic if it is nonrandom and associated with the initial intervention.

Concerns over attrition can be ameliorated if those performing the original intervention collect and maintain contact and other identifying information on their original subjects. If these data are lost or never collected, then the possibilities for downstream analysis dissipate. Even if researchers do not foresee the necessity of maintaining such information at the time of the initial intervention, the ability to reconnect with subjects cannot be undervalued. In the aftermath of his famous obedience studies, Milgram (1974) followed up with his subjects to evaluate their lingering responses and reactions to the intervention. Davenport et al. (2010) recently studied the enduring effects of randomized experiments, testing the benefits of applying social pressure as a voter mobilization tool by analyzing registration rolls in subsequent elections. Although collecting and keeping track of subject contact information may slightly increase the cost of the original experiment, they have the potential to increase the payoffs in the long term.

Both practical challenges can be overcome through cooperation among researchers. This advice may seem prosaic; however, in the case of experiments, there is usually only one opportunity to perform an intervention, and optimal execution of said experiment will influence future scholars’ ability to build off of the results. Sharing experimental designs prior to implementation can open up important dialogue among scholars, dialogue that can improve initial interventions and heighten prospects for future downstream possibilities. Moreover, the maintenance of subject contact information may not be a first-order priority for researchers, but it may provide long-term benefits to one's own research team and others. The sharing of results among a broad swath of the research community can also increase the likelihood of extending the findings beyond their original intent. As seen, many interventions give way to downstream analysis in entirely different subject areas. Cross-disciplinary collaboration on randomized experiments will help scholars approach previously intractable puzzles. This will be easier to do if researchers cooperate from the outset.

Ethical Challenges

Although downstream analysis of preexisting randomized interventions provides researchers the opportunity to study exogenous changes in independent variables of interest normally deemed off-limits to outside manipulation, this type of analysis does pose some interesting ethical considerations with reference to subject consent and the use of deception. Concerns in both realms derive from the unending nature of experiments subject to future downstream analysis. The issues raised with regard to consent and deception in considering how researchers planning new interventions ought to proceed to ensure that their contemporary research has value beyond its original intent.

In terms of consent, we should consider whether it is problematic for participants to be subjected to tests and data collection post hoc if they did not accede to further examination at the time of the original intervention. Such downstream research might be unobtrusive and not involve interaction with the original subjects, but these concerns remain the same. Moreover, encouraging researchers to share data on experiments to allow for later analysis may violate Institutional Review Board guidelines stipulating that research proposals detail the names of any investigators who will have access to data containing names and other identifiable information of participants in the original study.

Issues raised over deception and full disclosure closely parallel concerns over consent. Although consent focuses on what the original researcher should do prior to the intervention, challenges concerning deception center around behavior following the initial intervention. The American Psychological Association's “Ethical Principles of Psychologists and Code of Conduct” stipulates that researchers using deception provide a full debriefing to participants as soon as possible following the intervention, but “no later than the conclusion of the data collection” (American Psychological Association 2003, section 8.07). Ideally, this disclosure should occur immediately following the conclusion of the intervention, but researchers are permitted to delay the debriefing to maintain the integrity of the experiment. However, this stipulation mandating a debriefing at the conclusion of the data collection is problematic in the face of downstream analysis because researchers might never be sure when, if ever, data collection is completed. Scholars must consider whether a full or even limited debriefing will impede the ability of future researchers to conduct downstream analysis of the original experiment and what the proper course of behavior should be given this potentiality.

5. Conclusion

Researchers conducting randomized experiments ought to consider potential long- and short-term downstream analyses of their initial interventions. Doing so expands our estimates of the costs and benefits associated with randomized experimentation and provides unique research prospects for those who are unable to feasibly conduct such interventions. Awareness of these additional research opportunities can help structure the intervention and data collection process in ways amenable to downstream analysis. Downstream analysis is only possible if data are maintained and made available to others. Moreover, consideration of downstream possibilities prior to implementing a particular protocol will help researchers brainstorm the range of measures collected at the outset of a project. This will expand the value of the experiment to the individual researcher in the short term and to the broader reaches of the research community in the long term.

References

American Psychological Association. 2003. Ethical Principles of Psychologists and Codes of Conduct. Washington, DC: American Psychological Association.

Angrist, Joshua D., Guido W. Imbens, and Donald B. Rubin. 1996. “Identification of Causal Effects Using Instrumental Variables.” Journal of the American Statistical Association 91: 444–55.

Angrist, Joshua D., and Alan B. Krueger. 2001. “Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments.” Journal of Economic Perspectives 15: 69–85.