CHAPTER 3
Urban education reform is littered with failed ideas. Ideas abandoned too early and ideas abandoned far too late. Ideas that worked for these schools but not those schools. Ideas with mixed effects and unintended consequences. Popular ideas that attracted money and manpower. Polarizing ideas that divided and distracted. New ideas, and bastardized ideas, and musty old ideas repackaged and sold as new.
“If you look from 10,000 feet at education interventions, you can almost count on your hand the number of interventions that have truly scaled and established themselves,” said Jerome D’Agostino, a professor of educational studies at Ohio State University. He led a meta-analysis of the ten thousand studies that have been reviewed by the U.S. Department of Education’s What Works Clearinghouse, which had been designed to help educators select programs and practices that are backed by rigorous evidence. D’Agostino’s analysis found only twenty-nine interventions that had had significant effects, most of them small. This is not to say that the reforms did not work in some places for some students, but rather that the chances of a program or intervention working for many different types of students across many different types of contexts was slim.
This hard truth—that much of school reform in recent years has amounted to so much wheel spinning—is succinctly summarized by the title of Rutgers University professor Charles M. Payne’s book So Much Reform, So Little Change: The Persistence of Failure in Urban Schools. Payne wrote, “Reformers of every stripe got their butts kicked from one end of the 1990s to the other.” Though he was referring to the 1990s, he could just as easily have been writing about any decade since the 1983 release of the Nation at Risk report, which touched off intense waves of reform activity in the United States. Produced by President Ronald Reagan’s National Commission on Excellence in Education, the report employed doomsday language to warn that “the educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a Nation and a people.” American public education systems, particularly in urban areas, have been in perpetual states of reform ever since.
These reforms have tended to follow a common script: under tremendous public pressure to demonstrate turnarounds, leaders go big and fast with a new program or initiative that may previously have worked in another context but has never been tried in that particular school, district, or classroom. Anthony Bryk, a UChicago sociology professor who founded the Consortium and then went on to lead the Carnegie Foundation for the Advancement of Teaching at Stanford University, coined the term “solutionitis” to describe this phenomenon, arguing that “the propensity to jump quickly on a solution before fully understanding the exact problem to be solved” is rampant in education reform and accounts for many of its failures.
Often, in their rush to a solution, reformers fail to get buy-in from teachers or principals, who feel that reform is being done “to” them rather than with them. When the results of the program or policy turn out to be uneven—as is almost inevitably the case when lots of different schools try to implement the same program—policymakers abandon it altogether rather than building on its successes and learning from its failures. Finally, there is a general disavowal of the old and a frantic search for the next “big idea.”
These failures are not just abstractions. They have sunk careers, squandered goodwill, and misspent billions. They have bred a pervading pessimism about the ability of school reform to make lasting, meaningful improvement at scale, an attitude particularly acute among teachers, many of whom long ago adopted a “this too shall pass” attitude to reform. Most damningly, reforms have failed the children they were designed to serve.
Freshman OnTrack is an important exception to this general trend. It is an idea that managed to take root, blossom, and cross-pollinate in ways that have positively impacted tens of thousands of lives in measurable ways: it is the rare education idea that made it at scale.
And yet, it very nearly didn’t. The first time that the district acted on Roderick’s ninth-grade research, it followed the predictable pattern of education reform: many solutions to the problem were generated in short order. Those solutions didn’t account for the particular realities each school was facing on the ground. As a result, many of the solutions were unpopular or untenable to the people (the teachers and principals) charged with carrying them out. Schools implemented the reforms superficially, complying with the letter of the law but not its intent. The reforms failed to have their desired impacts. They were deemed spectacular failures, which helped to usher in a regime change.
Subsequent iterations of Freshman OnTrack would prove much more effective and sustaining. But to understand why Freshman OnTrack eventually beat the odds, we can look to how a similar impetus, based on the same research, with the same goal—to support freshmen during the transition to high school—fell so short. That story begins in the mid-1990s, when a new superintendent named Paul Vallas began shaking up the status quo in Chicago’s public schools and making headlines in Chicago and across the country.
By the early 2000s, Vallas would be the national face of a type of education reform that borrows principles from the corporate world. He would be called a “CEO,” rather than a superintendent, and his system would emphasize accountability, privatization, and free market choice in education. After a six-year stint in Chicago, he would go on to lead the public schools in Philadelphia; Bridgeport, Connecticut; and post–Hurricane Katrina New Orleans. But in 1996, when Roderick’s report was released, Vallas was still an education novice faced with the Sisyphean task of improving graduation rates and student learning in Chicago while also grappling with a gaping budget hole.
How Vallas fared would determine whether he kept his job as head of Chicago schools—and perhaps whether Mayor Richard M. Daley, the son of iconic Chicago mayor Richard J. Daley, would keep his. Daley had named Vallas, his former budget director, the CEO of Chicago’s schools the previous July, just days after the state’s Republican legislature and governor gave him broad authority over the city’s schools. The 1995 overhaul gave Daley and his handpicked school board the power to name a superintendent, intervene in failing schools, and allocate resources in new and creative ways.
Mayoral control represented a break from the long-standing practice across the country of separating mayoral and education politics. An outgrowth of Progressive Era reforms, the separation was meant to buffer schools from the back-scratching that had so often characterized City Hall dealings. But that separation had created an uneasy power structure whereby the mayor had little direct control over the institutions that mattered most to many taxpayers and voters. With mayoral control, city politics would inextricably be linked to school politics. The schools would have to answer to the mayor, and the mayor, in turn, would have to answer to the public about the state of the schools.
In this new era, “accountability” became the new education buzzword. Chronically low-performing schools would need to show measurable improvement or face sanctions. Vallas was threatening to put schools with very low test scores on probation, which would allow him to make personnel changes, including removing principals. Students were also put on notice. Vallas made it clear that he was putting an end to the practice of “social promotion,” whereby students automatically advanced to the next grade. Moving forward, students would need to demonstrate proficiency or risk being held back. Everyone was under the gun in unprecedented ways, from Daley to Vallas, to principals, teachers, and students.
So the timing of the release of Roderick’s bleak portrait of Chicago’s high schools was propitious, coming when the entire educational system and city government were finding themselves under pressure to demonstrate radical change. The report lit a fire under Vallas, who was touched by the stories of the individual students Roderick had chronicled. Vallas talked all the time about Clara, a pseudonym for one of the students whom Roderick had interviewed. “He talked about her like he knew her. And like her actual name was Clara,” Roderick recalled, laughing. “We somehow figured out that by looking at these kids, people could understand. Th se were not bad kids. These kids were getting in trouble for something stupid or small.”
The month that Students Speak was released, Vallas pulled together a task force made up of seven committees totaling 150 members drawn from reform groups, charitable foundations, academia, and CPS administration. He called on them to make “quick and lasting reforms” to the city’s high schools. The task force included “just about every prominent player in education reform,” including Roderick, although notably it did not include any teachers. After the union balked, Vallas added two teachers and seven union representatives. The union was only slightly mollified: “We squeak, and they put a little oil on us,” Chicago Teachers Union vice president Norma White observed.
The seven committees returned after four months with reports that were compiled into a seventy-nine-page draft plan, which was basically a laundry list of concrete recommendations and sketchy aspirations that some dismissed as “pie in the sky.” While Vallas technically opened up the document for public comment, critics complained that the hearings were cursory. He wanted to submit a plan to the Board of Education as quickly as possible, so that high schools would have the balance of the year to implement the ambitious plans. A series of community hearings were crammed into the month of December, including one on Christmas Eve. “It was a joke,” said one teacher who attended the Christmas Eve hearing. “There was no information. They showed us into a conference room, we sat down, we wrote our comments, and that was it.”
Four months later, the Board of Education announced a new “Design for High Schools” based on the recommendations that the committees had submitted. Vallas and his team trumpeted the plan, which came with an estimated $33 million annual price tag, as a game changer.
The plan had two key tenets: increase standards and accountability on the one hand, and increase support for students on the other. To increase support and personalization, the reform called for the creation of freshmen academies at every high school in the city. The board did not dictate what shape the academies needed to take—they could be a separate physical space, or simply a set of common supports and practices for freshmen. “The whole idea is to try,” Roderick said. “If something doesn’t work, try something else.”
The other major initiative to increase student support involved mandatory advisories in every school. Modeled after a successful long-standing program at New Trier High School, which served some of Chicago’s wealthiest suburbs, the advisories program required teachers or counselors to meet daily with a small group of students over the course of their high school careers. The advisor was meant to serve as a mentor/counselor to help students navigate both academic and personal issues.
The other key tenet of the new plan was increased rigor, which was to be achieved through tougher graduation and course-taking requirements. At the time, some Chicago high schools were offering as many as one hundred courses, many of them electives and remedial courses of dubious quality. Students would now be required to take a college-prep course sequence of English, math, science, social studies, and foreign language in order to graduate. Meanwhile, they would be tested regularly to ensure that they were being exposed to suitably rigorous material and that they were actually learning it. The district developed end-of-course exams, which they called the Chicago Academic Standards Exams (CASE), to be administered twice a year and to count toward students’ final grades. Ninth graders who flunked the test would be required to attend summer school.
Expectations were high when schools started the following fall. “I think people’s confidence is being restored in the system,” Vallas declared, the day before the start of the 1997–98 school year. “All the initiatives that we’ve laid out in our education plan have either been implemented or are in the process of being implemented.”
Though Vallas presented the success of his reform agenda as something of a fait accompli, the truth is that the implementation stage marks the beginning—not the middle, certainly not the end—of any reform project. Bryk, who co-authored Students Speak, sounded a more cautious note. “The devil is in the details…. It’s going to take a while—more than one year—to determine if the changes are working.” Bryk knew it is in the implementation stage, when sparkling new ideas are superimposed onto the messy reality of schools, in which details such as whether teachers buy into a common plan and have the capacity to execute it often matter more than the policy itself does.
Advisories and freshmen academies fell apart quickly. High schools had scrambled to put advisories and freshmen academies in place, shuffling around teachers and schedules to comply with the new requirements. But teachers balked at the extra work without extra pay and the expansion of their duties into the non-academic realm, where many felt uncomfortable and out of their depth. Many complained—publicly and privately—that they had been hired to teach subjects like math and science, not to serve as therapist or counselor.
As is often the case in education reform, the one-size-fits-all mandate had not taken into account how the unique context of each high school—even something as simple as student and teacher scheduling logistics—could doom implementation. At Carl Schurz High School, for example, an overcrowded high school on the Northwest Side (and one of the city’s oldest and largest), the school’s staggered bell schedule made implementation of a single advisory period tricky. The way that Schurz’s schedule was set up, it already was a challenge for some students to get the course credits they needed to graduate. The task became more difficult when the board increased math and science requirements and added the mandatory twenty-five-minute daily advisory period.
In order to accommodate the changes, the school day would have to be reconfigured, which required a union vote of approval. Schurz teachers were already demoralized by Vallas’s decision the previous fall to put their school and thirty-seven other high schools on academic probation. That status allowed the administration to intervene in the daily operations of the school, and placed it one step away from being “reconstituted” with new staff Schurz teachers opposed the schedule change on the grounds that they were being asked to pick up an additional class without being compensated for it. The school administration said it did not have the money to pay teachers for an extra period. The school’s union delegate sent a strongly worded letter that urged teachers to reject the schedule change.
Schurz principal Sharon Rae Bender called a meeting to discuss the pending vote. She spent the meeting alternately appealing to teachers’ duty to their students, and apologizing for a Central Office edict that she claimed no responsibility for. She began by talking about how much she had enjoyed getting to know her students personally when she had run an advisory back in her teaching days. “You develop a rapport with them,” she enthused. Then she switched gears. “Sometimes I feel like I’m between a rock and a hard place,” she continued. “As you know, I’m a former union delegate, and now I’m a principal. Twenty-five minutes of so-called advisory was not my idea. I have been instructed by Mr. Vallas and the board to prepare for this.” Then she switched tactics again. “I believe you are professionals, and being professional is different from punching a time clock in a factory…. I’m not asking you to give blood. I really believe that this doesn’t really faze most of you.”
In a vote the following week, the teachers overwhelmingly rejected the reform-related schedule change by a vote of 122–10. Across the district, schools took similar votes. Many ultimately ignored the district’s order to implement advisories. It took another year for the board and the union to resolve the issue. In November 1998, the board negotiated a new four-year deal with the union. Under the new contract, class periods were cut by five minutes each, freeing time for a paid advisory period. But while schools now had the time for advisories, they generally “were conducted out of a sense of compliance, not as a way of fostering closer relationships with and among students in a school.”
Freshmen academies suffered a similar fate. In 1996, shortly after the release of Roderick’s report but a full year before the board approved the high school plan, CPS had offered $100,000 and training for schools interested in launching an academy. These early adopters experienced some successes implementing the academies. But another large group of high schools declined to participate in the original pilot program, only to be drafted into it shortly before the school year began. These schools got a fraction of the money ($35,000) and little time to plan, resulting in programming that seemed “thrown together,” according to some critics.
Not surprisingly, considering its hasty implementation, the freshmen academy plan never did get much traction in Chicago. Only a small number of high schools designated a separate part of the building for freshmen or assigned them to a specific group of teachers—what were supposed to be the defining features of the academies. Most schools simply paid lip service to the idea, referring to their freshman and sophomore classes as the “Junior Academy,” without making any substantive changes.
Finally, there were the controversial CASE exams, which were both unpopular and ineffective. They were initially meant to determine part of students’ final grades and whether ninth graders would be allowed to pass to tenth grade or would be required to attend summer school. However, after the first year few of the freshmen who failed the exam showed up at the “mandatory” summer school, and the board eliminated the promotion gate.
Teachers claimed that the test was a waste of time, poorly constructed, and confusing. They also argued that the multiple-choice questions favored simplistic responses over complex analyses. A question about Romeo and Juliet, for example, asked students to identify the “climax” of the play. To highlight the test’s ambiguity, one reporter polled a high school English teacher, a Shakespearean actor, and a Northwestern University professor about the question, only to receive conflicting answers.
In protest, George Schmidt, an English teacher, printed portions of the test in Substance, his monthly newsletter that was sharply critical of the Vallas administration. School administrators, who had intended to reuse the test the following year, were livid. The board sued Schmidt, claiming he had violated copyright laws and stolen trade secrets. The legal battle dragged on for six years, and in the end the board got to fire Schmidt but had to drop its bid for financial compensation. By the conclusion of the lawsuit, Vallas was long gone and so were the CASE exams.
Even as many of the high school reforms limped along, Vallas managed to maintain a public reputation as reformer-in-chief. In January 1999, President Bill Clinton praised Chicago’s schools as a model for the nation during his State of the Union Address for the second consecutive year. In May of that year, the Chicago Tribune editorial board declared “A Light at the End of the Tunnel,” as high school students made significant gains on state tests. The percentage of freshmen meeting national norms in math had more than doubled, from 20 percent in 1996 to 42 percent in 1999.
But just two years later, in March 2001, Professor G. Alfred Hess Jr. of Northwestern released a damning report that concluded that the high schools had shown “little significant change” despite the administration’s reform efforts. The district had commissioned the $1.8 million study, which had included an extensive review of test scores and attendance and graduation rates, as well as qualitative work that drew on classroom observations and interviews with teachers and principals at high schools on probation. Hess concluded that while high school test scores had indeed improved, the gains were due to students coming into high school more prepared than they had been in the past. He also found that efforts to support students during the transition to high school had generally fallen flat. The Vallas administration had initially attempted to bury the results of Hess’s study.
A week later the UChicago Consortium released its own critical report; it found that graduation rates had not budged since 1985. Then, as before, 43 percent of students tracked from age thirteen left school without a degree. “We still see the same dropout rate,” said Bryk, the Consortium’s leader. “I certainly wouldn’t want to use Chicago as a model for making national policy.” Vallas dismissed Bryk’s criticism as ideological rather than substantive. “Tony [Bryk] has his own agenda,” Vallas charged. “He was the champion of the previous school reform agenda” (a 1988 reform that had decentralized the system and preceded mayoral control). But business leaders, state legislators, and others who had formerly had two feet in the Vallas camp began to defect. “Mayor Daley recently said we ought to think outside of the box,” said Republican state senator Daniel Cronin, who had supported Vallas’s reform agenda as chairman of the State Senate Education Committee. “Maybe we ought to get away from that box entirely.”
Two months later, in May 2001, another round of standardized tests was released. These showed that elementary math scores and high school reading scores had dipped. Three weeks later, on June 6, 2001, Vallas resigned. “Six years is enough,” he was quoted saying in the New York Times. Some speculated that Daley had pushed him out because he had become too independently popular. Others claimed Daley was simply disappointed by flattening scores and wanted to shake things up. Roderick summed up the state of schools: “We don’t stink anymore,” she told the Times. “We’ve now got the problem of how do we really move ourselves up a notch.” Hess, the Northwestern professor who issued the damning report on the high school redesign, added, “The biggest thing that Vallas … did was change the culture of the school system. It is no longer acceptable to say we can’t expect poor children to learn.” High school reforms may not have worked as well as intended, but the accountability Vallas brought to the system would be crucial to the success of the Freshman OnTrack movement down the road.
Chastened by the dismal outcomes of high school reform, Roderick and UChicago Consortium researchers set about redefining the role they played in school improvement in Chicago. In the future, they resolved to focus their research and outreach efforts on helping educators understand and solve their own problems through incremental, steady improvement, rather than on setting out to catalyze large-scale policy shifts. A simple indicator—which they called Freshman OnTrack—would make it possible.
The Freshman OnTrack indicator was developed in the mid-1990s by Consortium researchers to measure whether ninth graders were making basic progress toward graduation. Students who failed no more than one semester of a core course and earned enough credits to advance to sophomore year were considered “on-track.” Preliminary analysis had suggested that the Freshman OnTrack metric was a good predictor of whether freshmen would go on to graduate. That is, those who were on-track were more likely than those who were off track to graduate four years later.
The UChicago Consortium had originally developed the on-track measure at the behest of elementary schools and nonprofits that worked with eighth graders and wanted to know how their students fared when they entered high school. But in 2003, Arne Duncan, who replaced Vallas as CEO, moved to put the metric on the district’s accountability framework, meaning that high schools would be judged in part on whether a large percentage of their freshmen were on-track to graduate. Duncan had been looking for a metric other than test scores to gauge how schools were doing with their underclassmen, and Freshman OnTrack fit the bill. The Consortium’s staff panicked when they learned of Duncan’s intention. They did not yet know enough about the metric to feel comfortable recommending it as something schools should be judged upon. As researcher Elaine Allensworth put it, “If OnTrack was going to matter, then we had to make sure it really mattered.”
Allensworth was assigned to validate the metric. She was not initially enthused about the project; she had more methodologically complex projects she was looking to work on, and this new project seemed like a snore of a detour. Still, the entire purpose of the Consortium was to do research that really mattered to educators, even if it didn’t appear to be the most cutting-edge work for academics.
The UChicago Consortium was founded in 1990 by Bryk, then a professor of urban education in the university’s sociology department. Widely considered to be one of the country’s top education researchers, Bryk bears a striking physical resemblance to Bill Gates and tends to talk in perfectly organized paragraphs. It is easy, therefore, to mistake him for a typical ivory-tower academic. He isn’t. When he arrived at UChicago in the mid-1980s, Bryk was sharply critical of his colleagues who studied educational theory but never engaged with the people working in schools. He believed that the type of research produced by the academy was too often of little practical use to educators, and he was determined to bridge the gap between research and practice.
Bryk’s Consortium was a new kind of education research organization. Its research agenda was driven not by the interests, expertise, or professional concerns of the researchers, but by the needs of educators. Its steering committee was composed of representatives from the teachers union, the principals association, local nonprofits, the school system, parent advocates, local philanthropists, and researchers—a deliberately pluralist group. “The first meeting, I remember thinking, ‘I hope someone doesn’t jump across the table and punch somebody else out,’ because it was Chicago and it was pretty adversarial,” Bryk said in an interview.
The Consortium’s focus on a single district (Chicago) allowed its researchers to build lasting relationships and trust with local educators and policymakers, and deep institutional knowledge about the people and schools they were studying. It also allowed them to research a problem over time, with each study building on the last one. The Consortium’s dropout research exemplified this approach. “There was an accretion that went on for years,” said John Easton, who co-authored the early on-track studies and led the Consortium from 2002 until 2009, at which point Duncan appointed him to serve as the director of the Institute of Education Sciences in the U.S. Department of Education. “On the way to Freshman OnTrack there were little pieces here and there, and it took some time for it to come together.”
Roderick had jump-started the Consortium’s dropout research, focusing the organization on the importance of freshman year, and Allensworth was the perfect candidate to build on Roderick’s initial findings. Roderick was the big-picture thinker, while Allensworth was more measured and precise, content to ask tightly defined questions. Freshman OnTrack cemented Allensworth’s faith in undertaking more modest projects. “I realized,” she said, “that sometimes the simplest analyses also produce the simplest answers.”
Allensworth’s analysis revealed that the on-track indicator was far more predictive of graduation than she had initially guessed. Students who were on-track at the end of freshman year had an 81 percent chance of graduating. Students who were off track had just a 22 percent chance. Thus, being on-track seemed to increase a student’s odds of graduating nearly fourfold. This finding was crucial, because at the time there was a perception that there was no way to predict who would graduate and who would drop out.
Initially, Allensworth had suspected that the on-track indicator might be predictive of graduation only because it was strongly related to test scores. Her analysis debunked this theory. “I ran the numbers and I was like, oh my God, there are a lot of students with really high test scores who are off-track, and they are not graduating even though they test really well,” Allensworth recalled. What she had found was that Freshman OnTrack status was actually more predictive of high school graduation than test scores were. In other words, students’ course performance during freshman year had more to do with their chances of graduating than how they performed on tests, the gold standard of proficiency in most education circles. Allensworth recalled hurrying over to John Easton’s office to share her initial analysis. “I remember saying, ‘Look at this.’”
Allensworth’s findings were completely contrary to the dominant assumptions underlying education reform, then and now—namely, that test scores are the best predictor of future success and that the primary goal of educators and reformers should be test score improvement. The federal No Child Left Behind Act, signed into law in January 2002, both reflected and deepened this preoccupation with test scores. Under the law, schools were required to ensure that all students met standards on state tests by the 2013–14 school year or face sanctions. When Allensworth was conducting her analysis in 2004, schools across the country were frantically working to get as many students as possible over this proficiency bar.
Educators were inevitably taken aback by Allensworth’s findings, often pushing back vehemently against this challenge to the supremacy of test scores. One chart in particular always garnered a reaction (see Figure 1).
What it showed was that off-track students with test scores in the top quartile nationally were far less likely to graduate than on-track students with scores in the bottom quartile. Indeed, the on-track students with the lowest test scores were 31 percentage points more likely to graduate than the off-track students with the highest test scores. And it was not as if just a handful of students with strong test scores were off track. Fully 22 percent of students with eighth-grade test scores in the top quartile and another 35 percent of students in the second quartile were off track. This suggested not only that schools needed to stop focusing exclusively on students with the lowest test scores, but also that they needed to stop blaming high dropout rates on students’ low skills, a common excuse at the time. Accepting these findings would require a paradigm shift or many educators.
Note: Students who dropped or transferred out of CPS before the end of the school year are not included in these calculations.
Figure 1. Source: Elaine M. Allensworth and John Q. Easton, The On-Track Indicator as a Predictor of High School Graduation, Consortium on Chicago School Research, June 2005, 9.
Just as important, Allensworth’s analysis also revealed that Freshman OnTrack was a more targeted, accurate way of thinking about dropout risk than race, class, and other background factors. She performed an analysis that combined all of the background factors that she could measure about a student: race, ethnicity, gender, economic status, neighborhood poverty level, the frequency with which the student transferred schools prior to high school, and whether the student was overage for her grade. When she combined all these factors, she was still only able to identify 65 percent of all graduates. Worse, she was only able to correctly identify 48 percent of all dropouts—less favorable odds than a coin flip. In contrast, the Freshman OnTrack indicator by itself correctly identified 80 percent of all graduates and 72 percent of dropouts.
This one simple metric proved to be more predictive of graduation than everything else measurable about a student combined. And the relationship held for every subgroup of student at every school. Whether you were black, white, or Latino, from the South Side of Chicago or the North, attended a selective enrollment high school or a neighborhood high school, freshman year seemed to have the power to make or break your high school experience. This was an empowering finding for high school educators. They cannot control the race, ethnicity, income, or prior test scores of the students who walk through their doors, but they do have a shot at helping them pass their classes freshman year. At last, Freshman OnTrack offered them something to work on that was squarely within their locus of control.
Nationally, the conversation around dropouts was also shifting, with more emphasis being placed on the role schools had in promoting or thwarting graduation. In 2004, researchers Robert Balfanz and Nettie Legters from Johns Hopkins University released an ambitious study that attempted to determine the size of the national dropout problem and the percentage of high schools with high dropout rates. They found that two thousand schools across the United States produced the majority of the nation’s dropouts; these were schools that had 60 or fewer percent seniors than freshmen. The researchers labeled these schools “dropout factories,” a term that implied that schools were manufacturers of the dropout problem rather than recipients of it, and that placed the burden for fixing the problem on schools rather than on the students they served. The term “dropout factory” went viral, appearing in newspapers and magazines across the country, much as the term “dropout” itself had captured the public’s imagination forty years earlier.
Of course, there was pushback against the idea that schools played a primary role in preventing dropouts. Critics of the “dropout factory” term argued that it ignored the difficult realities of educating students living in poverty and absolved students of their personal responsibility for making it through high school. Skeptics of Allensworth’s on-track research, which was published a year later, argued that students who failed ninth grade were likely to be different from other students in some fundamental way that researchers couldn’t measure. Gangs, drugs, teen pregnancy, a lack of family support: all could affect students’ ability to stay in school, without showing up in any dataset. If this were the case, then schools couldn’t do much to increase students’ odds of passing their courses freshman year. “Principals and some teachers were angry,” Allensworth said, “because there was this very firm belief that ‘Yes our graduation rates are low and yes our on-track rates are low, but it’s because of the kids not us,’” she said.
One figure in the Freshman OnTrack report generally gave these critics pause. It showed the on-track rates of every school in the district, adjusted for the students they served. In other words, it compared what a school’s Freshman OnTrack rates would be if they all served students with similar background characteristics. As it turned out, the adjusted on-track rates varied significantly across the district. Schools serving students with similar background characteristics had widely divergent rates. This suggested that schools could significantly impact their on-track rates, with the right supports. And if they could influence their rates, then they might also be able to influence their graduation rates three years down the line. Thus, the Freshman OnTrack indicator seemed to have the potential to be used in two different and powerful ways: first, as a predictor of who might be at risk of dropping out and in need of extra services and support and, second, as a direct lever for improving students’ likelihood of graduating.
A follow-up report in 2007 provided more actionable information to schools. It named the factors that were likely to move the on-track rate—crucial information if schools were meant to use it as a lever to improve graduation rates in real time. Specifically, it showed that attendance was as predictive of graduation as the on-track indicator was. Indeed, attendance was eight times more predictive of course failure than test scores, and just one additional week of absence per semester could decrease a student’s chance of graduating by 20 percentage points (see Figure 2).
The report also found that schools with strong climates, as measured by surveys of teachers and students, tended to have better freshman attendance and lower course failure. Attendance was significantly higher, and course failure was significantly lower, in schools where students reported that they trusted their teachers and that their teachers provided them with high levels of personal support. This supported Allensworth’s earlier finding that Freshman OnTrack was malleable; that is, it could be influenced by school practices. The Consortium and Duncan had their answer: the Freshman OnTrack indicator did indeed appear to be a metric that mattered, and the way to improve it was to work on improving grades and attendance throughout freshman year, especially by strengthening relationships between teachers and students.
“The stakes are just so extraordinarily high, if we’re trying to stop dropouts in their junior or senior year, we’ve missed the boat,” CEO Arne Duncan said just before the start of the 2007–08 school year. CPS distributed highlights of the study to the homes of thirty thousand high school students before classes began. The handouts called freshman year the “Make It Or Break It Year” for graduation, and emphasized how even small improvements in attendance or grades during this pivotal year could dramatically increase students’ chances of graduating.
Figure 2. Source: Elaine M. Allensworth and John Q. Easton, What Matters for Staying On-Track and Graduating in Chicago Public Schools: A Close Look at Course Grades, Failures, and Attendance in the Freshman Year, Consortium on Chicago School Research, 2007, 7.
Duncan’s enthusiasm for Freshman OnTrack provided much of its early momentum. “Arne loved Freshman OnTrack, and everyone loved Arne,” said Steve Gering, a CPS educational consultant. The responses of administrators and district officials to Freshman OnTrack ran the gamut, but no one at CPS could afford to completely ignore the metric if Duncan was championing it. “He made it really clear that this is what we’re working on. This is what we’re doing,” said Paige Ponder, who led the district’s Freshman OnTrack strategy as the head of its Graduation Pathways department. “If you were a principal or an area officer, you knew you were going to be asked about it, and you were going to be caught with your pants down if you didn’t know your numbers.”
When Duncan replaced Vallas as CEO, most people in Chicago viewed him as amiable, committed, and smart, but also politically inexperienced. A sociology major at Harvard and onetime professional basketball player in Australia, Duncan had returned to Chicago, his hometown, to run the Ariel Education Initiative, a philanthropic endeavor designed to increase educational opportunities for students on Chicago’s South Side. Subsequently he served as deputy chief of staff for Vallas and ran the system’s magnet schools program, positions that came with a fair amount of responsibility but not much direct political fire.
Yet Duncan would demonstrate a steely will and keen political instincts that belied his inexperience. “When Arne saw something that made sense to him, he advocated for it,” Gering said. Duncan managed to survive at the helm of the system for seven years—a year longer than Vallas had—while championing a string of bold, controversial programs that might have sunk someone less politically adept, from closing low-performing schools and opening one hundred new ones, to instituting merit pay for teachers.
Unlike Vallas, whose primary lever for improvement had been turning up the heat on students, teachers, and schools, Duncan often spoke about the need to build the professional capacity of teachers and principals to meet higher standards. He tapped into private funds from the city’s civic community to make large investments in training for early career and veteran teachers. He left Chicago with a reputation as a pragmatist who was more interested in results than ideology.
The Consortium’s reputation also helped Freshman OnTrack gain traction among competing reform factions and maintain a middle ground between the two. By frequently referencing the Consortium’s research, CPS administrators were able to keep those on the ground focused on solving the particular problem of ninth-grade course failure, rather than fighting over the particulars of the district’s approach to Freshman OnTrack. It was a sharp contrast to high school reform under Vallas, where the problem of the ninth-grade transition got lost amidst the rush to find a comprehensive solution to the array of problems plaguing high schools.
Those who became the biggest champions of Freshman OnTrack bought into the research first, before it was a graduation strategy. “When the Consortium brought out the OnTrack data and they talked about the freshman year, it was like a light bulb went off for me,” said Sean Stalling, who was a principal at Manley High School on the city’s West Side when the 2007 Freshman OnTrack report was released. “I started tracking through my experiences before I got to CPS, just as a high school teacher…. I remembered the numbers. We had one hundred and fifty freshmen, and then we had one hundred sophomores, and it was like, where do these kids go? You hear not many of those kids graduated, this kid is locked up, this kid is here, all these stories, but I never tracked each kid.” Stalling would become a key champion of Freshman OnTrack after Duncan left the district and Daley appointed Ron Huberman, the mayor’s former chief of staff, to succeed him.
The research prompted other educators to rethink their previous approach to freshmen. “I feel like the big ‘aha’ was the realization that the old-school, don’t smile until Christmas, scare everyone straight approach—there are real implications to that way of thinking,” Ponder said. “We moved into these deep transition discussions. Before that time there really wasn’t any thought about the freshman year for anything.”
Elizabeth Kirby, a principal at Kenwood High School on the city’s South Side, recalled taping a photographed copy of a Consortium graph showing that one additional week of absences cut a student’s probability of graduating by 20 percentage points to her computer. Every time she considered suspending a student for a week, she would look at that graph and ask herself whether the offe se really warranted the potential long-term harm of the punishment. Usually the answer was no.
Meanwhile, as CPS administrators and Consortium researchers educated teachers and other key staff about the on-track findings, the Parthenon Group, a consulting firm hired to help develop a plan for improving graduation rates, worked with CPS administrators to develop a strategy for how schools could use data to improve on-track rates. For years, CPS’s department of Dropout and Recovery had been a hodgepodge: credit recovery programs, academies for over-age ninth graders, alternative schools for students who had already left high school, etc. But all these programs were on the periphery, separate from the day-to-day work of high schools, and were largely aimed at the district’s lowest-performing students—those who had failed eighth grade or were many credits behind the rest of their classmates. Freshman OnTrack would be a different kind of strategy, and more than an indicator. It would be aimed at helping the average kid, whom the Consortium had demonstrated was still very much at risk of dropping out.
It also differed from extant programs for dropouts because it was explicitly not a program. From the beginning, CPS administrators made it clear that they were not going to tell anyone precisely how to improve their on-track rates. Indeed, no one at that point had the answer anyway. Instead, with funding from the Gates Foundation, they established Freshman OnTrack Labs at six neighborhood high schools, which each had two full-time coordinators. The labs were meant to serve as incubators of new ideas and strategies. The hope was that these strategies would eventually be shared with schools across the district.
Ponder, the district’s project manager for the OnTrack initiative, told the on-track coordinators that they needed to act as if they were researchers observing, analyzing, and experimenting to try to solve the problem of freshman year course failure. She developed a process to help guide their work. First, she said, take an inventory of all the programs and services already in existence at the school that could somehow be utilized to support freshmen. Get the lay of the land and create a strategy that takes into account the current realities on the ground. She also asked them to create strategic plans that listed what they were going to do, what they expected to see, and the data they would use to check whether what they were doing was working. She was trying to strike the right balance between giving on-track coordinators the autonomy to create their own plans and giving them enough structure to avoid spinning their wheels. “Because it’s really easy to go into a school, and just get sucked into just the chaos of the school,” she said.
While only six schools received full-time staff members devoted to improving freshman performance, every district high school began receiving new data reports on their freshmen that were designed to help them identify struggling students early in the semester and then track them throughout the year. A “Watch List,” distributed to schools at the start of the school year, included students’ eighth-grade grades and attendance. The list helped remedy the historic lack of coordination between elementary and high schools in Chicago.
A second report, released every five weeks, listed each student’s grades and attendance. These “Success Reports” were meant to serve as “hot data,” providing up-to-date information on which students were at risk of falling off-track. They allowed schools to craft individualized academic, attendance, or behavioral interventions for struggling students before they failed a course freshman year. Finally, a “Credit Recovery Report” provided information on those students who had failed courses during first or second semester. Schools used the reports to ensure that those students who had failed a course enrolled in summer school or other credit recovery programs, also a point in the system where students historically fell through the cracks.
It is difficult to overstate how revolutionary the five-week data cycles were in 2008. “It was just a totally new concept to be looking at data more than once a year,” Ponder said. Prior to the Freshman OnTrack reports, there was no systematic tracking of students throughout the school year. Teachers might know who was failing their class, but there were no formal mechanisms for sharing that information until the end of the semester—when the failing grade was already in the books.
This was not a problem unique to Chicago. “One thing that became clear with this work was that there were weird fissures in most systems,” recalled Bob Balfanz, the Johns Hopkins researcher. “Up to that point, no one was monitoring student data systematically. Students got report cards, and the data was all there, but that was a private thing. And even teachers never saw the data in totality. They put in their grades and sent in their suspension slips and attendance sheets. But no teacher saw a kid’s complete report card. And no one was looking at data in actionable ways.”
Gering, the education consultant whom Duncan had recruited to work with Area Instructional Officers (AIOs were essentially deputy superintendents overseeing networks of schools), became one of the chief constituents for the reports. He encouraged the AIOs who he worked with to pressure principals to analyze their data, set goals, and develop plans for reaching those goals, and then monitor their progress. It quickly became apparent, however, that the principals were having trouble seeing the forest for the trees. So much data on individual students was sending them in too many directions. They needed a summary measure.
Ponder’s team responded by adding a “point-in-time” on-track rate to every success report. It told principals what their FOT (or Freshman OnTrack) rate would be if school ended that day. AIOs could also be held accountable for their rates. In the meeting area where the AIOs gathered, Gering posted the point-in-time on-track rates of every area, and of every school in the areas. The public displays were a visible reminder of the priority the district was placing on Freshman OnTrack and also a way to monitor who was making progress on the metric and who was not. “You could see at a glance where everyone stood,” Gering said. “We wanted to get everyone’s attention—and we did.” Every five weeks AIOs would receive updated numbers to assess whether each area had met its goals and refine the strategy for the next quarter.
Like the Freshman OnTrack validation research, none of this was particularly thrilling. Watch lists. Data reports. Planning cycles. Teacher teams. Later, when Consortium researchers would try to explain the monumental shift that occurred around Freshman OnTrack, people would look at them skeptically, as if trying to figure out where the big idea was hiding. In fact, it was not that the big idea was hiding, per se, but rather that the big ideas most people tend to think matter most in education don’t really matter all that much.
Education professor John Hattie, director of the Melbourne Education Research Institute at the University of Melbourne, has dedicated his career to studying what really matters for improving student achievement—and to debunking the popular misconceptions that swirl around the topic. Hattie argues that the things we argue about most in education, the hot-button education issues like performance pay, charter schools and school choice, technology, and curricular fixes, for instance, have little effect on improving student learning.
He calls the public’s fixation on these unproven fixes “the politics of distraction,” arguing that “many policy-makers and systems are persistently drawn to the wrong kind of education interventions.” The politics of distraction rests on the faulty assumption that “there is a relatively simple intervention that can be defined and structured from near the top of the political system and that can then work its way through to positive effects for students.” Using Hattie’s criteria, the vast majority of reforms instituted in Chicago during the Vallas and Duncan eras would have fallen under the heading of “politics of distraction.”
These distractions cost lots of money and waste lots of time and effort, but even more crucially, they divert attention away from the factors that do have a large, measurable impact on student learning. These evidence-based factors aren’t headline-grabbing and don’t sound particularly groundbreaking. For the most part, they are hard to make policy around, except indirectly. Many involve getting teachers to work together effectively toward a common goal. They include “working together to evaluate their impact,” “building trust and welcoming errors as opportunities to learn,” and “getting maximum feedback from others about their effect.” Hattie calls this kind of work the “politics of collective action.” In the politics of collective action, educators form communities within and across schools that set a common goal, determine interventions to meet that goal, and use data to evaluate their progress toward that goal and determine where to go next.
The Freshman OnTrack movement in Chicago fit squarely within Hattie’s definition of the politics of collective action. It provided a common, well-defined goal for teachers of freshmen to work toward: preventing course failure. The data reports provided teachers with regular feedback on the progress they were making. At the same time, the central office was encouraging schools to form teams that could make collective decisions about whether their interventions were working or not, and what they might try next. As they did this, they began creating “narratives of impact,” as Hattie called them—stories about what had worked and what had not—that could be shared from teacher to teacher and from school to school.
“It wasn’t this well-defined, sexy thing,” recalled CPS principal Elizabeth Dozier. “It was building relationships with kids, finding ways to gather teachers to collaborate and talk about kids, making small plans, and then checking to see if the plans worked.”
Before Dozier became something of a celebrity principal in Chicago and a leading character in both Paul Tough’s bestselling How Children Succeed and CNN’s docudrama Chicagoland, Dozier was a first-year assistant principal at Harper High School in Englewood, one of the most violent neighborhoods in Chicago. At Harper, her primary charge was to keep all 217 of Harper High’s freshmen on-track to graduate from high school.
During the 2007–08 school year, the year before Dozier arrived at Harper, 57 percent of Harper’s freshmen—142 of them—failed two or more core courses. And just like that, before they were even old enough to drive, they were virtually out of the running for a high school diploma and a spot in the American middle class.
Dozier wanted to draw a bright line for her staff between these dismal freshman outcomes and her students’ futures. At the beginning of the year, she engaged all the freshman teachers in a visioning exercise. Where would the 142 off-track freshmen from the previous year be in three years? In ten years? Would they be in college? Working? Or standing on a street corner in Englewood? She reminded them that black male dropouts—and all of the 142 off-track freshmen were black—have a 70 percent chance of being incarcerated by their mid-thirties. “It was meant to tug at their emotional heartstrings,” Dozier says. “If we’re not careful, we can get desensitized to what this really means. Th se aren’t just data points. This isn’t just about Harper. It’s about lives. It’s about communities.”
Without a specific plan or policy from the district, Dozier’s initial approach to improving Harper’s on-track rates was mostly trial and error. She created a community of freshman teachers who met weekly to review data on each student’s grades, attendance, and discipline patterns. The group strategized about ways to support and motivate students academically and to provide additional help for those whose struggles had nothing to do with academics. To make the work feel more tangible, she created a board with every student’s name on it and moved the student on- or off track throughout the year. The team celebrated every small victory—a kid who went from failing to passing math; a chronic truant who started showing up regularly.
One challenge the group struggled with was how to help students take responsibility for their own learning and achievement. Many students interpreted failing grades as something that was done to them by teachers, not as something they had earned. The group needed a way to help students keep track of where they stood academically. Harper had an online student portal that included grades and assignments, but students rarely checked it. The group decided they needed something more public and easily accessible. Dozier made a giant color-coded board with every freshman’s name on it and hung it in the main hallway for the entire school to see. She placed the students’ names under one of three headers: green for on-track, red for off-track, yellow for “almost there.” Making the names public went against CPS board policy, but she did it anyway. Brain research may suggest that most fourteen-year-olds will not be swayed by the potential long-term consequences of missing school or failing to turn in an assignment, “but what they do understand is social pressure,” Dozier said. “One name would be the size of two napkins. You could read it from down the hallway. It became this thing. No one wanted to be at the end part. They’d say, ‘Ms. Dozier, why am I off track?’ The older you get, it doesn’t matter so much, but with these little ones, this public thing was very big for them.”
Students began making connections for themselves between learning, the grades they were getting in the class, and their status on the “big board,” which was a major breakthrough as far as Dozier was concerned. Dozier added, with a large smile, “I remember one time someone from CPS came and said, ‘Take it down’ and we said, ‘Okay, sure. And, oh, when are you coming back?’”
One freshman student in particular stood out for Dozier that first year. He was a nice kid, she recalled, but something seemed off. One day he got into a verbal argument with a teacher, and she told him, “Okay, we’re going to your house.” The principal advised Dozier to take Harper’s football coach, who was born in Englewood, along with her. “It just completely put some stuff in perspective. We talked to his grandmother. She was completely cracked out. And that was the moment I realized how complex the work really is, and there’s not a formulaic response for every kid. Maybe for a good portion you can use the public stuff, and incentives, and teachers meetings, but it totally changed our approach to the kids who are the most difficult to serve.”
She described going to the next on-track meeting and telling everyone that the kid they had been struggling to reach lived in that house. “Everyone knew that house. It was where all the gangbangers hung out, where all the drugs were sold. Families and gangs used to fight right outside. I remember seeing a mob fight one day. Someone picked up one of those police horses and just threw it into the fight. And so when I said, ‘This is where the kid lives,’ there was a whole new perspective.”
Dozier and her team started conducting regular home visits whenever a student was failing. She would gather a group of teachers, take along the football coach for security, and engage parents in conversations about their child. “Lots of these people had bad experiences with their high school, so they’re not coming to the building,” Dozier said. “So we said, Okay, we will bring school to you. It was really impactful for parents. They saw that the school really cares, and that is not typical. And it was also impactful for teachers because now you’re not just driving down 63rd and going to the parking lot and going home. Now you’re driving through the community. And it made a difference for the kids, because what they want to know is, ‘Do you care about me? Do you have my best interests at heart?’”
At the end of the school year, Harper’s Freshman OnTrack rate had risen 18 percentage points, from 43 percent the previous year, to 61 percent—roughly equivalent to forty additional students being on-track to graduate. One of those on-track students was the kid from “the house,” almost certainly a student who would have fallen off track in the past. Mayor Daley held a press conference at Harper to highlight the improvement there and the modest uptick in district-wide on-track rates (up 4.5 percentage points from the previous year). Dozier remembered a phalanx of city workers descending on the area before Daley arrived, cleaning streets and filling potholes. “It was really big for Harper. We had made this huge jump.”
The press conference was sparsely attended, mostly by the small cadre of City Hall reporters who follow the mayor from one public event to another, asking off-topic questions only relevant to the major stories they are covering that day. A few local outlets briefly noted the Freshman OnTrack story and mentioned that Chicago schools were focusing on freshmen in order to improve graduation rates. And that was all. Just 4.5 percentage points district-wide hardly seemed like something to celebrate, and the on-track metric itself was still obscure. It looked like any other press conference in which data were marshaled to declare some minor victory in the ongoing political war over school reform. Instead, it was a precursor to one of the largest system shifts in Chicago Public Schools’ history, a shift that came not from a high-level policy, trendy national program, or any of the big ideas that typically get discussed during debates around school reform, but from efforts that got teachers working together in new ways to solve problems that most of them had previously assumed they were powerless to affect.
It didn’t happen overnight. Changing the way that teachers related to one another and to their students was critical to the success of Freshman OnTrack, but doing so could be arduous—particularly in very low performing schools, which is what Hancock High School was at the start of the 2008–09 school year.