Mismatch

THE SUBSTANTIVE DEBATE

As the politics surrounding “Systemic Analysis” alternatively boiled and simmered, a substantive debate unfolded as well. Several distinguished social scientists teaching at law schools won the competition to write critiques of “Systemic Analysis” for the Stanford Law Review; many other critics jumped into the fray. To date, some two dozen critiques have been published, and many others have been given at talks or posted to websites. I cannot respond to all of them here, but this section discusses those that most engaged the data and that other critics usually point to as the strongest rebuttals of my work.

CLYDESDALE. Two months before “Systemic Analysis” appeared, the journal Law and Social Inquiry published “A Forked River Runs Through Law School” by Rutgers sociologist Tim Clydesdale. Clydesdale’s work was not a direct rebuttal to “Systemic Analysis”—Clydesdale hadn’t seen it when he wrote his article—but it dealt with some overlapping issues and was often cited (as in the letter by Lempert and Chambers quoted above) as evidence that “Systemic Analysis” must be wrong.

Clydesdale, like me, had used the LSAC dataset to evaluate how minority law students did in school and the bar. When he analyzed bar passage outcomes, he had found, like me, that race did not measurably affect bar passage when one controlled for law school grades. But unlike me, he had found that race had a devastating impact upon law school grades. Based on this key finding, Clydesdale constructed a theory in which discrimination and alienation caused blacks and other minorities to do badly in law school and, hence, on the bar.

The problem with Clydesdale’s theory was that his crucial finding—that minorities had much worse law school grades than did whites when one controlled for academic index on entering law school—was based on a subtle but undeniable error. The BPS dataset, readers will recall, collapsed law schools into large tiers, some of them with fifty or more schools. The dataset calculated a “class rank” for each student, which measures and calibrates each student’s grades relative to her fellow students. But the academic index and its components (LSAT and college grades) are not standardized in the BPS (which, as I discussed in Chapter Four, is one of the key reasons it is hard to measure “mismatch” with the BPS directly). This means that Clydesdale’s model would evaluate each student’s academic index relative to all other students in the BPS—including those in much less competitive law schools—not relative to other students at the school attended by that student. This mistake would tend to inflate the academic indices of blacks and deflate those of whites, biasing his regression so severely as to make the results meaningless. In my own analysis (as discussed in Chapter Four) I had used other data sources that did standardize academic indices for each law school, and this, readers will recall, showed that race qua race was a very minor factor in determining law school grades.

Clydesdale’s error is not a matter of opinion; I have discussed it with a number of social scientists who, as soon as they grasp the point, immediately and unanimously agree that the error is fatal to his analysis. Clydesdale more or less conceded the error in subsequent, unpublished work. The error escaped the peer reviewers at Law and Social Inquiry because Clydesdale did not describe the BPS dataset in sufficient detail or point out this limitation. There is no doubt that, had the peer reviewers known of the problem, they would not have allowed the article to be published in anything like its final form. Yet a surprising number of commentators still list “A Forked River” as an important rebuttal of “Systemic Analysis.”

AYRES AND BROOKS. The single-most widely anticipated response to “Systemic Analysis” came from Ian Ayres and Richard Brooks of Yale. Both were sophisticated empiricists with doctorates in economics, and Ayres was one of the most famous social scientists at any law school.

Ayres and Brooks first used the BPS to replicate the main analyses in “Systemic Analysis,” and they publicly confirmed that all the factual claims they checked were correct—a nice thing for them to do, which quelled some of the claims that my article was wrong on its facts. They then tried to use the BPS to create an independent model that could test the mismatch hypothesis. This was an excellent idea. In my article’s discussion of the racial bar passage gap, I had essentially posed a logical puzzle that mismatch could explain and supported it with an array of circumstantial evidence. Ayres and Brooks wanted to try to test the mismatch hypothesis directly by comparing students who had experienced differing degrees of mismatch.

The biggest challenge facing a direct test was something called “the selection effect”—a common analytic challenge that comes up in many types of behavioral research. To give an example unrelated to racial preferences, selection effects pervade health research. When observational studies conclude that some activity—say, eating lots of green vegetables—produces better health outcomes, they often fail to adequately control for all the ways that avid green-vegetable eaters are different from the rest of the population. Those who eat a lot of green vegetables are often health conscious, so they drink less, smoke less, exercise more, floss their teeth, and do all sorts of other things that improve their health. So it is hard to know whether particular outcomes are the result of eating green vegetables, or the result of other health activities that are not included in the scientists’ data.

In many types of mismatch, research selection effects are a major challenge. Suppose one is comparing two black students, whom we’ll call Student X and Student Y. X attended the “most elite” group of BPS law schools, and Y attended a “middle range” law school. We know that the two students have the same academic index. We also know that these credentials suggest that X received a large preference to his school (as most students in that cluster have significantly higher LSATs and grades) and Y probably received a much smaller preference to his middle-range school (as most students in that cluster have LSATs and grades more similar to Y).

But when we simply match students based on academic index, it is likely that we are missing other important characteristics that explain why students are at schools of differing eliteness. Datasets more complete than the BPS show that Student X (at the more elite college) is much more likely, on average, than Student Y to have attended a difficult school, to have majored in a difficult subject, to have secured a master’s degree, or to have distinguished himself academically in any number of ways. That, of course, helps explain why Student X was admitted to and enrolled in the more elite school. But in the BPS we cannot observe (and, therefore, control for) these other achievements, just as in the medical example above, we cannot observe (and, therefore, control for) all the minor ways the green-vegetable eater pursues a healthy lifestyle.

Thus, any time important unobserved characteristics are correlated (i.e., associated) with a characteristic we want to measure, we face the problem of selection bias. Comparisons of the health of green-vegetable eaters with others will be biased toward finding a powerful beneficial effect from green vegetables alone; comparisons of students at elite schools with apparently similar students at nonelite schools will tend to show better outcomes for the elite students by virtue of selection bias alone. Because the mismatch hypothesis contends that, in important ways, more-elite schools can hurt students, the evidence is often buried underneath the many other unobservable strengths that separate students at different schools.

Ayres and Brooks came up with a very clever solution to this problem. They noticed that in a lengthy questionnaire completed by all the BPS students when they entered law school, the students were asked several questions about applying to law school. The BPS asked them (a) how many schools they had applied to, (b) how many schools had admitted them, (c) whether they had gotten into their “first choice” school, (d) if so, whether they enrolled in their first-choice school, and (e) if not, why not. The data showed that about half of all students got into their first-choice school, and about 80 percent of these matriculated there. But the other 20 percent went to a lower-choice school, usually because it was less expensive or because they had geographic reasons to be there (presumably such things as being near a significant other or living at home to save on expenses).

Ayres and Brooks reasoned (and this could be confirmed in the data) that for any given student, their first-choice school was more likely to be elite than their second- or third-choice schools. Therefore, students in groups that usually received preferences, such as blacks, needed larger preference to get into their first-choice schools than to get into their second- or third-choice schools. Mismatch—if real and important—would, therefore, be a bigger problem (on average) for blacks who ended up at their first-choice schools than for those who ended up at their second-choice schools. So, controlling for other characteristics, second-choice-school blacks should have better outcomes if the mismatch effect were real.

This first-choice/second-choice analysis sidestepped the selection-effect problem because here we were comparing students who had all been admitted to their first-choice school. It therefore followed that their unobserved characteristics (like where they went to college and what they majored in) were as impressive as their observed characteristics.

FIGURE 5.1. First Choice, Second Choice

Source: Williams, “Does Affirmative Action Create Educational Mismatches in Law Schools?” (2012).

But Ayres and Brooks did not fully report their results, and they gave readers no context with which to interpret what they did report. I will not speculate on motives, but as far as I have been able reconstruct events, this is what happened.

Ayres and Brooks were notified in early November 2004 that they were among those chosen to write a responsive critique of “Systemic Analysis.” By the beginning of January 2005 they had preliminary results: their first-choice /second-choice analysis showed no meaningful difference in student outcomes. They made presentations of their results to faculty and students at Yale and claimed to have debunked the mismatch theory. A couple weeks later they sent me a copy of their draft. I found pervasive errors in their calculations (apparently made by a research assistant), which accounted for their reversed results. I got in touch with Richard Brooks, who agreed that the numbers were in error and went back to work. In March 2005 Ayres and Brooks sent me another draft, and once again I found errors that they conceded and agreed to fix.

In the final, published version Ayres and Brooks had corrected all of their numbers. But the wording of their conclusion had barely changed: they would at most concede “mixed” but ultimately unconvincing support from the first-choice/second-choice analysis. How was this possible? Ayres and Brooks examined five outcomes:

• First-year law school grades. Ayres and Brooks conceded that second-choice students had a relative advantage here but did not explain that the second-choice students improved their grades in a way that exactly matched the mismatch prediction.

• Third-year law school grades. Ayres and Brooks did not find a statistically significant difference here, but that was because they dropped from their analysis students who left law school altogether (which were disproportionately first-choice students).

• Graduation rates. Ayres and Brooks noted that there was not a statistically significant difference in graduation rates between first- and second-choice students. But they failed to note that this was completely consistent with mismatch theory, which itself predicted that the effects on graduation rates would be too small to be statistically significant, given the sample sizes involved. Nor did Ayres and Brooks note that the improvement in graduation rates among second-choice students was higher than that predicted by mismatch theory.

• First-time bar passage. Ayres and Brooks conceded that second-choice students had a relative advantage the first time they took a bar exam but did not report the enormous magnitude of this advantage (second-choice students were some 40 percent less likely to fail the bar).

• Ultimate bar passage. The only surviving thread of Ayres and Brooks’s argument against mismatch was that after many tries on the bar exam, first-choice students came closer to the bar passage rate of second-choice students, so close that the difference between the two groups was not statistically significant. But “statistical significance” is intended to rule out false positives that occur from random fluctuation. Obviously, the consistently strong performance of second-choice students on many different measures was not random. Moreover, it was not very surprising that students might, after years of further, post-law school study, partially offset the effects of mismatch in law school.

For none of these outcomes did Ayres and Brooks present their results in a way that would let readers evaluate for themselves the consistency and magnitude of the results.

Ayres and Brooks used a second empirical test of the mismatch effect in their paper, one similar to tests used by several other critics. The strategy of all these tests was to compare students with similar characteristics who attended law schools with differing levels of eliteness. Ayres and Brooks called theirs the “relative tier” test.

As I noted earlier, tests of this type suffered from the “selection effect” problem, which was large and serious. The tests were also undermined by the fuzziness of the six tiers in the BPS—the categories of school eliteness that provided the only clue as to just how mismatched students might be. Both of these biases would cause any test to understate the actual mismatch effect.

Despite this handicap, the relative-tier test, when it has been performed by me or (independently) several colleagues, shows substantial mismatch effects for both blacks and Hispanics. But the reported results of Ayres and Brooks showed no such effect. How is this possible?

The only results Ayres and Brooks reported for their relative-tier test were ones that omitted historically black schools from their analysis. In other words, Ayres and Brooks left out all of the law schools where blacks were least likely to be mismatched. When this is done, the mismatch effect goes away.

Now, one might argue that blacks do better at historically black law schools for reasons other than the absence of racial preferences. Perhaps they respond better to black professors or they have tighter social bonds with their classmates. But Ayres and Brooks neither argued this nor presented any evidence that such was the case. Nor did they show what their results looked like with and without black law schools included. They simply excluded from their analysis the data most relevant to examining black mismatch, and they failed to discuss their reported result showing Hispanic mismatch.

The unfolding of the Ayres and Brooks paper was, for me, perhaps the most frustrating part of the mismatch debate. These very smart men had come up with a compelling, independent test of mismatch that, correctly done, produced a stunning confirmation of the theory. But they had first gotten the analysis wrong and then, when corrected by me, had massaged the results in a way that made it appear they had disproven mismatch. Had I given them no feedback at all, I could have easily dismantled their published result by showing its clear mathematical errors. Instead, I had been an unwitting accomplice on what was taken as the most powerful rebuttal of mismatch.

BARNES. Whether I learned anything from the Ayres and Brooks experience may be judged from the resolution of another flawed critique, this one by University of Arizona law professor Katherine Barnes and published in the Northwestern University Law Review (NULR). Barnes had a doctorate in statistics and, using the BPS data, developed a sophisticated model for testing mismatch. Perhaps the distinguishing characteristic of her model was that it did not rely, as many statistical models do, on an assumption that causes and effects must relate to one another in a linear way; her model allowed for more complicated relationships. Her model was subject to selection bias, but even so, her results seemed to rebut the mismatch hypothesis impressively. For example, her model suggested that a black student with low credentials would have a 63 percent chance of passing the bar if she went to an elite law school, but less than an 8 percent chance if she went to a historically black school, where she would be better matched. If true, such results would obviously imply preferences improved outcomes for students.

But the results were not true; indeed, they seemed to me to have been completely made up.

Doug Williams, an old friend of mine and chair of the economics department at Sewanee University, tried to replicate Barnes’s numbers and got radically different results. In the above example he found that the black student at the historically black law school would have a 67 percent chance of passing the bar, compared to a 49 percent chance at an elite law school. This was not only radically different from Barnes’s results; it also fit very closely with the predictions of mismatch theory.

When Williams told me of his findings, my first step was to retain an independent analyst (a psychometrician friend who worked at UCLA), give him the BPS data and Barnes’s article, and ask him to do his own replication. His results, a couple weeks later, almost exactly matched those of Williams.

Our next step was to contact Barnes, explain that we had obtained different results, and ask for a copy of her computer code. Barnes promised to send us the relevant code but never did. After multiple requests stretching over many months, we wrote up a note explaining our results and sent it to the editor-in-chief of NULR, the journal that had published her article. The editor did not respond or even acknowledge our letter. I contacted the Review’s faculty adviser, who seemed nonplussed and suggested that if we obtained different results, we could publish our own article somewhere else.

This level of indifference by the author, the publisher, and the publisher’s academic adviser took Williams and me by surprise. In the scientific world there is no more serious offense than publishing fabricated results. Any scientist would see such a charge as a fatal threat to her career; any journal would take such a charge with the utmost seriousness and spare no effort to establish the facts and correct the record. On reflection, we realized that this situation was different in two respects: First, we were dealing with law reviews and legal academics, who more often published opinions rather than facts and often left the factual details of their papers for law review editors to correct. Second, Barnes’s article was a defense of racial preferences; “Systemic Analysis” was controversial, so if NULR found that it should retract Barnes’s article, that might be seen as taking a political position against affirmative action.

Williams and I had better luck when we contacted a well-known empiricist on Northwestern’s faculty; he saw the seriousness of the issue and made clear to NULR’s editors that they should take it seriously too. In time, a conference call was arranged between Barnes, Williams, NULR’s chief, and me, and Barnes conceded that her results were incorrect. She contended that her original results had been produced on an old computer and had been lost, and that when she redid the analysis, she obtained very different results. A good deal of wrangling and nearly two years of delay followed, but as the editors became immersed in the issue, they became more and more determined to correct the earlier mistake. They secured the input of two academic referees. And in 2011 NULR published a retraction and apology, a restatement of Barnes’s results, and a comment by Williams, two coauthors, and me.

For the substantive mismatch debate, the significance of these events is that when Barnes published her corrected results, she had relatively little wiggle room to pick and choose models and outcomes that fit her side of the argument. Her original piece had specified a detailed model and had presented specific types of results; now she was stuck with that model. For example, her model hypothesized that if schools stopped using racial preferences, the number of blacks entering law school would fall by 21 percent. Her corrected results found that despite this drop, the outcomes of these students would improve so much that 58 percent fewer students would fail to become lawyers, and the overall number of blacks who would became lawyers by graduating from law schools and passing the bar exam would hold steady. Barnes could not bring herself to call these results a demonstration of mismatch, but they plainly were.

THE STATE OF THE LITERATURE. The bottom line of my discussion of the substantive critics is pretty simple: None of these articles could have survived peer review by any expert aware of the facts discussed here. Clydesdale, Ayres and Brooks, and Barnes all used models that, when actually carried out in a reasonable way, provided far more support for than refutation of the mismatch hypothesis. The best of these tests—the first-choice/second-choice analysis, which was both a completely fresh approach and one that avoided selection bias—generated results almost identical to my predictions, to a degree that’s actually quite unusual in something as imprecise as educational theory.

Moreover, these were not three exceptionally weak or vulnerable articles; on the contrary, partisans in the debate often cited them as examples of the best evidence against mismatch. And the Ayres and Brooks article—certainly the most insightful of the three—exemplifies weaknesses that apply to the other empirical critiques of law school mismatch. For example, political scientist and now–Stanford professor Dan Ho used a “matching” model to compare students who had similar academic indices but attended schools of differing eliteness. This approach is similar to the relative-tier analysis in Ayres and Brooks, discussed earlier, but had some methodological advantages. However, in Ho’s analysis he compared students attending schools very close to one another in their eliteness (and, thus, presumably very similar to one another in the level of mismatch facing black students). This is an obvious design flaw in Ho’s approach; when this flaw is removed, the matching test shows levels of mismatch similar to those produced by other tests.

A similar problem affects the analysis of Jesse Rothstein and Albert Yoon; Rothstein was then an economist at Princeton (now at Berkeley), and Yoon is a political scientist and law professor at Northwestern. In a broad review of law school mismatch Rothstein and Yoon offered a candid and fair discussion of the selection-bias problem and conceded that this problem weakened their main evidence against mismatch. But this research, like that of their colleagues, showed evidence of having been carefully massaged. When one made simple improvements in their models—improvements that aimed only to make the measurements more precise—the results were entirely consistent with other mismatch findings. Rothstein and Yoon were aware of the first-choice /second-choice method—the best available solution to the selection-effect problem—but inexplicably did not discuss what that method showed.

At the end of the day the evidence points overwhelmingly toward a large law school mismatch problem, one that affects Hispanics as well as blacks. It is hard to dispute that the first-choice/second-choice model provides the soundest direct empirical test of mismatch with available data, and it strongly confirms the hypothesis. In my view all of the other tests also support mismatch when they are modeled in a reasonable way; at the very least an honest critic would have to concede that they provide mixed support for mismatch. Wherever we have actual data comparing similar students at two specific schools (a point we will explore in more detail in Chapter Fourteen), the comparison suggests very large mismatch effects; that is, black students who had been admitted via large preferences into a more elite school but who had chosen to attend a less elite school that could admit them without such large preferences were failing the bar less than half as often as similarly credentialed black students who had entered the more elite school.

Then there is this fundamental point: Not one of the critics has articulated—let alone tested—an alternative theory to explain the patterns that are plainly there. Blacks in law school get roughly the law school grades their academic indices predict and have the bar passage rates that their law school grades predict. Yet they do dramatically worse on bar exams than do whites who enter law school—usually less elite schools—with the same academic index. Mismatch theory contends that this is because blacks disproportionately receive large preferences and, thus, disproportionately end up with lower grades and less learning. If it takes a theory to beat a theory, as George Stigler once famously observed, then mismatch wins by default.

In the summer of 2006 the US Commission on Civil Rights (USCCR) held a hearing on law school mismatch. Because half of USCCR’s members are appointed by the president, six of the eight commissioners at the time were Republicans and generally affirmative action skeptics; this would necessarily taint whatever they did in the eyes of many academics and reporters. But the USCCR actually had an impressive record of investigating and documenting important civil rights problems, and on the mismatch issue it took pains to proceed in a fair and deliberate way. Testimony came equally from both supporters and critics of mismatch, questioning was not curtailed, and staff made significant efforts to gather a wide range of material on both sides of the issue. The Commission’s background report was authored by an affirmative action critic, University of San Diego law professor Gail Heriot, but Heriot wrote an extraordinarily even-handed analysis of “Systemic Analysis” and its most serious critics. The Commission itself concluded in its 2008 report that some problem was clearly producing unacceptable outcomes for minorities, and mismatch was the most plausible culprit. It urged greater transparency, including a call for Congress to mandate disclosures from law schools that would allow prospective students to assess their likely outcomes across a range of schools.

The legal academic community utterly ignored the USCCR report. At the annual meeting of the American Association of Law Schools (AALS), which sponsors hundreds of panels every year on an array of topics but is most fundamentally responsible for informed deliberation on legal education, not a single panel at its meetings addressed the Commission’s briefing or its finding that legal education was failing minority students. Throughout the official world of law schools, mismatch did not exist. Even a national conference in 2008 that was called to discuss the problem of low minority bar passage rates failed to have any serious discussion of mismatch issues. Self-deception in official circles was seamless and complete.

If the formal law school debate was Orwellian in its resolute disregard of the obvious, “Systemic Analysis” nonetheless stirred enormous interest and reflection in unofficial circles. Dozens of professors and former deans contacted me to share mismatch stories. Frequently these were shared circumspectly; even tenured faculty feared the consequences of any public knowledge of their concerns. One of the best came not long ago to my coauthor. A veteran law professor, whom we will call Kim, recalled a painfully awkward visit from two young African American women when Kim was teaching at a top-ten law school. The women could not understand why Kim had given them such low grades on a first-year exam. It appeared that other professors had also given them low grades. “They were crushed,” reported the professor. “You could just see their self-confidence at an ebb. You could see that they were at sea. It’s a terrible thing to behold.”

Such exams were blind-graded, as is common, so that professors would not know which student wrote which exam until after the grades were in. But these two students wondered whether Kim might somehow have discerned who they were from their penmanship or writing style. The professor assured them that this was not the case. Kim was aware of the likeliest explanation for the low grades—large preferences and mismatch—but thought that sharing such speculations would only further undermine their self-confidence. “I had a very strong intuition,” Kim reflected, “that they would be better off at a less selective law school.” Then they would probably be in the middle of their class or better and emerge with “much more self-confidence and better job offers.” But Kim could not say that, either.

Another lawyer—who has taught at an elite law school and also has practiced at law firms—seethes with anger when explaining how schools like his admit poorly prepared minority students. “They don’t tell them before they enroll,” says this teacher, whom we will call Jason, that based on their grades and scores, they are expected to rank near the bottom of the class and to have trouble passing the bar exam.

Worse, says Jason, most schools provide little remedial help when such students struggle. “The faculty,” he says, “they talk a diversity game, they’ll admit black and Hispanic students with low entering credentials, but they’re just not that into them. They don’t take the time to help them master the skills necessary to pass the bar. You are sacrificing their professional careers. It’s immoral . . . to admit people with much lower credentials and keep them in the dark.”

Jason, who is white, adds that “the law schools lie a lot about all this. They think of it as a ‘noble lie.’ They lie about how well minority students do; they lie about job prospects their law students will have. They are teaching by example.”

Some black law students—after coming to understand how poorly prepared they were to compete with their classmates—have told Jason that they are nonetheless glad to be at an elite law school. But they have also said things like: “You and I both know how I got in here,” and “I’m not competitive here.” In addition, they tend to take the easiest courses to get their GPAs up, as advised by peers. But, says Jason, this is “massively counter to their interests. It does not help to avoid hard courses that give you the competencies to succeed professionally.”

Of course, says Jason, almost all elite law firms use large racial preferences in hiring—like “putting a twenty-pound weight on one side of the scales.” But once the new hires are in place, “What really drives partners’ behavior is you want to keep the young lawyers who make you money.” Those are the ones who, for example, quickly show an ability to draft in a hurry nearly flawless “deal documents.” It doesn’t take the partners long to figure out which new hires can do outstanding work, which cannot—and which ones they want working on the most important cases.

Scores of colleagues have shared similar stories with us. And at the time so did many admissions officers working at law schools. But ultimately more important and interesting were stories from blacks themselves. I heard from and spoke with dozens of African Americans—prospective law students, current law students, law graduates who had failed the bar, bar passers who had gone on to successful careers—all of whom were intensely interested in the idea of mismatch and many of whom felt it intuitively to be true. A senior black partner at a major firm reported that in his broad experience “affirmative action programs do nothing but (1) place certain students in schools which are well above their abilities—which any parent will tell you is a recipe for disaster, and (2) leave the message with many in society (both minority and white) that most minorities can only make it through the help of others rather than their own merit and hard work.”

In the months after “Systemic Analysis” appeared, the audiences who were most eager to invite me to speak—and from whom I learned the most—were predominantly black audiences. I spoke at a panel of the National Bar Association (the national association of black lawyers) and the minority affairs section of the American Bar Association. At one of these panels a black lawyer approached me to report that of the first thirty blacks admitted to his moderately elite law school in the early 1970s, after large preferences had been instituted, he was the only one to pass the state bar exam. I spoke at conferences of black law students and wrote abstracts of “Systemic Analysis” for undergraduate guidance counselors, who wished to pass on the findings in advising undergraduate minorities considering law school. I never experienced at these gatherings any of the hostility that often greeted me at academic events. Black students in particular were serious, interested, and concerned.

Meanwhile, as the law teacher who we call Jason above told coauthor Taylor, “I’ve had two professors say to me, ‘You don’t want to be seen talking to Rick Sander.’”

Did all of the sound and fury over “Systemic Analysis” have any practical effect? There is one very intriguing piece of evidence. Something unusual happened during the 2004–2005 law school admissions season, when my article was being most widely discussed, admissions officers were shaping their incoming classes, and black college seniors applying to law schools were choosing among competing offers. As Figure 5.2 below shows, the number of blacks entering law school did in 2005. The number and academic strength of blacks taking the LSAT did not change noticeably, but the number who ended up choosing to go to law school dropped sharply. Moreover, the black students who started law school in the fall of 2005 appeared to be at least somewhat less mismatched than in other years. Many relatively elite law schools experienced particularly sharp drops in black enrollment that fall, often as much as 50 percent. These data suggest that admission offices in 2005 were extending fewer large racial preferences, and black applicants were choosing schools with an awareness of potential mismatch problems.

I noticed these changes in 2006 and wondered whether blacks entering law school in 2005 might end up having noticeably better outcomes. They did. The matriculants of 2005 went on to have, by far, the lowest rate of attrition from law school during their first year (when academic failure is most common) of any cohort of black law students in the historical record (which goes back several decades). They had the highest graduation rate of any black entering law school cohort on the record. And in California—the only state to publish bar passage rates by race—they had the highest bar passage rate of any black cohort in California’s records. If the California pattern held nationwide (and the fragmentary available evidence suggests that it did), then the smallest cohort of blacks entering law school in many years appears to have produced the largest number of blacks passing the bar on their first attempt—ever. In every measurable respect the black students entering law school in 2005 dramatically narrowed the black-white gap.

Of course, all of this might have been coincidence. It might be coincidence, too, that by 2006, when interest in “Systemic Analysis” had died down and when the official word was that it had been refuted, patterns reverted largely (but not quite) to what they had been before. We do not—and cannot—know for certain until the official bodies of legal education open their records and allow objective research to go forward.

FIGURE 5.2. A Natural Experiment for Law School Mismatch?

Sources: ABA Legal Education Statistics; General Statistics Report on the California Bar Exam (annual series).