Troubling Transparency: The History and Future of Freedom of Information

TRANSPARENCY IS a value in the ascendance. Across the globe, the past several decades have witnessed a spectacular explosion of legislative reforms and judicial decisions calling for greater disclosure about the workings of public institutions. Freedom of information laws have proliferated, claims of a constitutional or supra-constitutional “right to know” have become commonplace, and an international transparency lobby has emerged as a civil society powerhouse. Open government is seen today in many quarters as a foundation of, if not synonymous with, good government.

At the same time, a growing number of scholars, advocates, and regulators have begun to raise hard questions about the costs and limits of the transparency movement. Some of these commentators accept the movement’s standard premises and prescriptions but worry that open government measures are not actually delivering the openness they promise due to inadequate legislative funding, bureaucratic resistance, or cramped judicial interpretations. Others wonder whether traditional open records and open meetings laws are well suited to twenty-first-century transparency challenges, or whether these laws need to be reimagined for the digital age. A third group of commentators has thrown a harsh light on transparency’s political and administrative effects, emphasizing its potential to facilitate “neoliberal” agendas or to undermine deliberation, deal-making, and institutional capacity.

These different strains of skepticism are coalescing and have largely been confined to discrete discourses so far. They have not arrested transparency’s ascent in the NGO community or in popular culture. But they have developed to the point where we might say that government transparency, as a democratic ideal, is contested not only in practice but also in theory.

This volume seeks to highlight the richness of these debates and to grapple with some of the complexity and ambivalence that increasingly characterize the best academic writing on transparency. It focuses on the United States Freedom of Information Act (FOIA)—both to contain what might otherwise be an unwieldy inquiry and because FOIA is an especially canonical transparency instrument, one of the ur-texts of the field. The essays collected here ask, in various ways, why FOIA and associated arrangements have come to be seen as troubling; whether that perception is warranted; and, if so, what can be done about it. In asking these questions, the essays themselves trouble the notion that we are reaching durable consensus, or indeed any widespread agreement, with regard to many aspects of open government design.

By 1990, a dozen-odd countries had followed the American example in adopting FOI measures of their own. By 2016, that number had mushroomed to more than one hundred. The FOIA model has been updated and refined many times over during this process of policy diffusion as countries have built on, or departed from, various components of the U.S. law in developing their own versions. Even so, virtually all of the world’s FOI laws replicate FOIA’s basic features, “including the focus on official records; affordance of access rights to any individual or association; reliance on private requests to trigger disclosure obligations; independent or quasi-independent review of denial decisions; and exemptions for the protection of national security, public safety, personal privacy, commercial secrets, and internal deliberations.”1 The FOIA model has become so prevalent that it is difficult to have a conversation about government transparency today without adverting to it.

This book grows out of a conference the two of us organized at Columbia University in the summer of 2016 to honor the fiftieth anniversary of FOIA’s enactment. One goal we had in bringing together dozens of FOIA experts of different stripes—with scholars, journalists, advocates, and administrators all represented—was commemorative in nature. We wished to applaud FOIA’s achievements and to mark an important milestone in U.S. legal and cultural history.

Also, and more important, we saw FOIA’s fiftieth anniversary as an occasion for critical reflection. This anniversary, it seemed to us, created an opportunity and a responsibility to consider the ways in which the statutory regime has evolved and the extent to which it has or has not been serving its founding goals; to consider FOIA’s relationship to other laws and policies within the larger “ecology of transparency,” as Seth Kreimer (chapter 7) calls it; and to consider how FOIA might be improved in the years ahead. These critical aspirations are represented, in a wide range of views, by the essays gathered here, many of them presented in a preliminary form at the conference.

Interpreting FOIA’s past and present is no simple matter. On some dimensions, the FOIA regime appears to be thriving. The law itself has proven highly resilient to legislative retrenchment: most of the amendments Congress has passed since 1966 have sought to make the requesting process easier or more effective. Usage rates continue to climb, with over 700,000 requests submitted to federal agencies in fiscal year 2015.2 There is a substantial FOIA bar and an increasingly professionalized FOIA workforce, as well as a robust coalition of journalists, advocacy groups, and businesses that promote and defend the law. In the broader culture, too, FOIA has assumed a quasi-constitutional if not quasi-sacred status, becoming, in the words of President Obama, a symbol of the United States’ “profound national commitment to ensuring an open Government.”³ And, as already noted, FOIA-style laws can now be found not only in all fifty U.S. states but also in most nation-states. FOIA, in short, has conquered the world, not so much through its specific details as by giving the ideal of transparency practical form and demonstrating the potential of a user-generated process for information disclosure.

Yet, on other dimensions, FOIA appears to be not flourishing but floundering, even in a state of crisis. The U.S. House of Representatives Committee on Oversight and Government Reform issued a scathing report in 2016 titled, simply, “FOIA Is Broken.”⁴ Echoing a set of complaints that have long dogged the act, the report found that FOIA is “systematically broken” on account of severe processing delays, overuse of exemptions, and other barriers to accessing records. To be sure, there may have been an element of partisan grandstanding to this report, which Republican representatives used to criticize the Obama administration. A 2004 report by House Democrats had accused the Bush administration of “not only sucking the spirit out of the FOIA, but shriveling its very heart.”⁵ “FOIA Is Broken” nevertheless tapped into a deep well of frustration, generated not just by response deadlines that are routinely missed (and in some cases patently unrealistic in light of congressionally allocated resource levels) but also by the extraordinary deference many courts seem to afford to withholding agencies, notwithstanding the statutory standard of zero deference (“de novo” review); by the law’s limited reach into the national security state, where millions upon millions of classified documents reside; and by the predominance of commercial requesters and the apparent distortions of FOIA’s public purposes that follow.

Without doubt, FOIA is a tremendously important statute that has done some tremendously important things in its first fifty years. It is also a markedly inefficient, adversarial, and corporate-friendly response to the postwar rise of official secrecy, and one that interacts in complicated ways with the U.S. system of governance. The Columbia conference reinforced our conviction that FOIA defies easy assessment. Students of transparency, we believe, should strive to appreciate what has been working well in this iconic transparency regime while remaining open to reconsidering, and possibly supplementing or even supplanting, parts of the FOIA model that have not been working so well. These sorts of inquiries require, in turn, that we connect our debates on FOIA to broader debates on open government law, policy, and theory, both at home and abroad.

Has the Freedom of Information Act improved the operation of democracy in the United States? One of us has become increasingly skeptical about this. Pozen has argued in recent work that FOIA’s request-and-respond model—identified by Gregory Michener (chapter 13) as an archetype of the Transparency-as-Leverage Paradigm—“empowers opponents of regulation, distributes government goods in a regressive fashion, and contributes to a culture of contempt surrounding the domestic policy bureaucracy while insulating the national security state from similar scrutiny.”⁷ Some of these effects, as Sam Lebovic demonstrates (chapter 1), were anticipated by agencies such as the Department of Health, Education, and Welfare in the early 1960s when the law was still being conceived in Congress. Over time, Pozen suggests, FOIA may have come “to legitimate the lion’s share of government secrecy while delegitimating and debilitating government itself.”⁸ Even if FOIA represented a progressive breakthrough at its creation, the rise of mass communications technologies, statutory reporting requirements, whistleblower protection laws, external watchdog groups, and internal oversight mechanisms, among other developments, has changed the act’s practical and normative meaning.

Schudson, on the other hand, feels more confident that FOIA has improved democracy, especially in concert with other transparency-producing mechanisms (Nadia Hilliard [chapter 9] and Beth Noveck [chapter 10] examine two others, and Kreimer [chapter 7] explores the overall transparency environment). The importance of a law is not necessarily to be measured by who uses it most (in the case of FOIA, the answer Margaret Kwoka establishes is often “corporations” [chapter 4]) but by whether its central purposes are being advanced at reasonable cost. In Schudson’s estimation, FOIA’s benefits in terms of public knowledge and government accountability plausibly outweigh its costs in terms of taxpayer dollars, bureaucratic hassle, and misleading messages that corruption, malfeasance, and mismanagement are especially endemic to government rather than phenomena that are just as likely—or possibly even more likely—to be found in organizations dedicated to private or partisan ends.⁹

Many transparency advocates would take Schudson’s points further. Given the lack of a general “right to know” in the United States, FOIA can be seen as an achievement of constitutional magnitude (the extent to which FOIA is or is not fungible with a constitutional guarantee is explored by Mark Fenster [chapter 3] and Frederick Schauer [chapter 2]). Dating back to the earliest efforts to craft FOIA in the 1950s, some of the statute’s staunchest friends have been members of the news media. In this volume, their faith in a judicially enforceable right to seek agency records may seem vindicated by Kreimer’s (chapter 7) review of the disclosures about the “global war on terror” under President George W. Bush that were made possible by FOIA requests from news organizations and civil liberties groups. It is hard to ignore the important role of FOIA in enhancing public understanding of these secretive operations and the abuses they inflicted. The standard case for FOIA is likewise supported by James Hamilton’s (chapter 6) innovative collection of data on the frequency with which prize-winning investigative journalism has made use of FOI requests at the state or federal level. It is further fleshed out, and complicated, in the argument advanced by Katie Townsend and Adam Marshall (chapter 11) that FOIA should be revised to encourage greater disclosure of information that falls within one of its exemptions. For Townsend and Marshall, FOIA is already an indispensable tool for journalism, but it could be made more valuable still if Congress followed the lead of countries such as Australia, Belgium, India, Ireland, Japan, Mexico, New Zealand, South Africa, and the United Kingdom in creating a “public interest” override.

Other proposals to revise FOIA could have even more far-reaching implications. Cass Sunstein (chapter 9) calls for much more aggressive and systematic disclosure concerning government “outputs,” yet much more reticence about disclosure when it comes to government “inputs.” Noveck (chapter 10) sketches a vision of a future in which “open data” policies, and the public-private collaborations they facilitate, are the major engine of government transparency, with FOIA transitioned into a supporting role. Kyu Ho Youm and Toby Mendel (chapter 12) highlight a number of ways in which the U.S. FOIA falls short of certain foreign counterparts as a transparency tool, and they recommend that U.S. legislators borrow “best practices” on issues such as the scope of public authorities covered (the U.S. FOIA covers only executive branch agencies and does not reach Congress, the courts, or government contractors) and the availability of independent administrative review. Writing in a more critical vein, Irma Sandoval-Ballesteros (chapter 14) offers a cautionary tale about recent experiences in Mexico—whose FOI law is often hailed as the international gold standard—and urges that FOI laws be revamped to reach powerful private entities as well as government bodies.

Despite our disagreements, the two of us draw inspiration and insight from each of these analyses. Along with their authors, we agree that the Freedom of Information Act matters enough to deserve serious scholarly attention, not just gauzy expressions of praise or exasperated anecdotes of delays and denials. Even though we resist romanticizing FOIA, we agree that there is a symbolic nobility to the law and its guarantee that “any person” can demand that the government disclose information. We agree that the tendency of some transparency commentary to pit “the people and their friends in civil society” (the good guys) versus “the government” (the presumed bad guys until proven otherwise) is the wrong way to think about a vibrant democracy. In that spirit, we agree with Sandoval-Ballesteros that transparency advocates should be concerned about a potentially debilitating “anti–public sector bias” to U.S.-style FOI laws, and with Kwoka that they should be concerned about the skew in FOIA usage toward requesters who have no public-regarding purpose. We agree that both the more radical and the more conventional reform proposals articulated in this volume merit consideration. And on all of these issues, we agree that it helps to see FOIA in light of comparable laws developed by the fifty states (as taken up by Katherine Fink [chapter 5]) and by other countries (as taken up in Sandoval-Ballesteros’s assessment of Mexico, in Youm and Mendel’s global survey, and in Michener’s study of Latin American legislation).

We do not resolve any of these issues here, but we air them. We invite readers to examine them collectively as well as individually to see the variety of ways of thinking about open government. With the assistance of organizations such as the American Society of Access Professionals, lively discussions among journalists, media advocacy groups, and FOIA administrators have been taking place for years now. There is a much less unified discussion about transparency in the law schools and among historians and political scientists. We bring together these disparate strands of scholarship in pursuit of a broader understanding of what we have in FOIA, and what we can and should wish for in structuring the place of information in a twenty-first-century democracy.

The volume turns first, in part I, to “FOIA’s Historical and Conceptual Foundations.” In his study of executive branch opposition to FOIA in the 1950s and 1960s, Lebovic draws on original archival research to illustrate how agencies did not see FOIA as a serious threat to national security but did see it as a threat to their ability to develop sound public policy and to their capacity to regulate the economy. These concerns, in Lebovic’s telling, foreshadowed the “uneven effectiveness” of FOIA following its passage and reveal important assumptions about the relationship between the state and the public that helped shape the law’s design.

The next two chapters explore another fundamental feature of FOIA’s original design: that it was created by ordinary legislation rather than a constitutional amendment or judicial interpretation of the First Amendment. Schauer looks at FOIA through the lens of the theory of rights and suggests that it can be seen as “remedying” the absence of a positive right to government information in the U.S. Constitution. Fenster explains that transparency advocates have generally assumed that a constitutional right would be better. Both Schauer and Fenster put pressure on this assumption, although in different ways. Schauer gives reasons to believe that a statutory approach is in fact superior, whereas Fenster gives reasons to doubt that the constitutional/nonconstitutional distinction matters much in this area—or, indeed, that many of the goals of the freedom of information movement are attainable.

Part II considers the relationship between “FOIA and the News Media.” The architects of FOIA hoped and assumed that professional journalists would be the leading acquirers and interpreters of agency records. That is not how things have turned out. Kwoka documents the remarkable scope of commercial requesting under FOIA, as well as the prevalence of “first-person” requesting by individuals seeking information about themselves. Fink documents a similar surfeit of commercial requesters at the state level. Hamilton finds that government records requests have contributed to many significant investigative stories but that the media’s use of FOI laws has been declining over time, especially for local newspapers. Alarmed by these developments, Kwoka, Fink, and Hamilton each propose reforms that might invigorate journalists’ relationship to transparency law.

Part III, on “Theorizing Transparency Tactics,” zooms out to consider FOIA’s relationship to other disclosure policies and to emerging trends in the transparency field. Arguing against critics (including Pozen) who have questioned FOIA’s democratic value, Kreimer draws on case studies from the early 2000s to highlight ways in which FOIA can support other transparency and accountability mechanisms even when a records request is denied or an appeal rejected. Hilliard complicates the place of FOIA in this ecology of transparency by looking at the paradoxical role of bureaucrats and experts in managing, interpreting, and narrating the enormous volumes of information that are generated by both FOIA and the inspector general system.

The remaining chapters in this section are more reform-minded. Sunstein draws a distinction between disclosure about government outputs (regulations, policies, findings, and the like) and government inputs (information about the deliberative process) and proposes a reorientation of transparency law to prioritize the former. Noveck reviews and extols the rise of the “open data” movement—a movement to which she has made significant contributions—as an alternative approach to transparency that is organized around problem-solving rather than accountability per se. Going forward, Noveck suggests, FOIA and open data policies ought to be harmonized much more closely than they are currently. Townsend and Marshall diagnose a problem within FOIA doctrine in the lack of a balancing test that would require disclosure of otherwise exempt records when the public interest in their release strongly outweighs the government’s interest in withholding them. The experience of foreign FOI regimes, according to Townsend and Marshall, suggests that such a balancing test would be workable and beneficial.

Building on this discrete cross-national inquiry, part IV offers several broader “Comparative Perspectives” on FOI law. Youm and Mendel use the global Right to Information (RTI) rating system to investigate the extent to which other countries have or have not followed the U.S. FOIA’s approach; they find that foreign FOI laws frequently incorporate elements that are more advantageous to requesters. After identifying competing paradigms of transparency embodied in the FOI laws of the United States and Finland, Michener applies this framework to help explain the successes and failures of FOI legislation in Latin America. Finally, Sandoval-Ballesteros offers a sobering account of FOI performance in Mexico, the world’s top-ranked RTI regime. The arrival of this regime, Sandoval-Ballesteros argues, has not fundamentally curbed corruption or transformed authoritarian ways of exercising power in Mexico, which suggests that transparency law must be reconstructed in more “democratic-expansive” terms. The chapters in this concluding part vividly convey how far the FOI movement has come in recent decades—and how much work remains to be done.

6. See, for example, Mark E. Warren et al., “Deliberative Negotiation,” in Political Negotiation: A Handbook, ed. Jane Mansbridge and Cathie Jo Martin (Washington, D.C.: Brookings Institution Press, 2016), 141–96. See also Congressional Research Institute, “Transparency Citations,” compiled September 19, 2016, http://www.congressionalresearch.org/citations.html [http://perma.cc/3JY3-N4VG], for a collection of sources that “address the problem of excessive … transparency in legislatures.”