CHAPTER 1
Schooling Goals versus Education Goals
Everyone has the right to education. Education shall be free, at least in the elementary and fundamental stages. Elementary education shall be compulsory. Technical and professional education shall be made generally available and higher education shall be equally accessible to all on the basis of merit.
UNIVERSAL DECLARATION OF HUMAN RIGHTS, ARTICLE 26
The goal of education is to equip children to flourish as adults—as parents and caregivers to the next generation of youth, as participants in their communities and societies, as active citizens in their polity, and as productive workers in their economy. The challenge of formal education is to supplement what parents can provide and, in a few formative years, build the foundation for a long and successful lifetime. The fundamental measure of success of any system of basic education is whether each successive cohort of children emerges from childhood equipped with the skills and capabilities for the world it will face.
We all know that each child needs schooling in this complex and rapidly changing world. We've seen massive expansions in schooling in nearly every country in the world. Eight Millennium Development Goals for all nations have been identified by the United Nations. One of them is that children “complet[e] a full course of primary schooling” by the target date of 2015. Each new cohort of youths enters adulthood having spent more and more time in a building called a school, so the Millennium Development Goal concerning children's schooling is getting close to being met. But no one has ever really had only a schooling goal. We have education goals for our own and others’ children. Schooling is the means to the goal of education. Are children around the world today emerging from the schooling they get with the education they need? No.
The accumulated body of research on performance in learning—from internationally comparable tests to assessments of curricular mastery to academic studies to civil-society-designed and -implemented tests—shows tragic results among schooled children. In recent studies in rural Andhra Pradesh, India (Muralidharan and Sundararaman 2010b), only around one in twenty fifth-graders could solve this arithmetic problem: 200 + 85 + 400 = 600 + ____. Less than 10 percent of fifth-graders understood that one-fourth of a chocolate bar was less than one-third of a chocolate bar. In a different countrywide assessment in India, 60 percent of the children who had made it all the way to grade eight couldn't use a ruler to measure a pencil (Educational Initiatives 2010). Similar findings of very low levels of conceptual mastery emerge from Pakistan, Tanzania, South Africa, Indonesia, and other countries around the globe. Even in many middle-income countries such as Brazil, internationally comparable assessments reveal that more than three quarters of youths are reaching the age of fifteen without adequate learning achievement and are ill-equipped to participate in their economy and society (Filmer, Hasan, and Pritchett 2006). In educationally advanced countries, educators are rightly worried about twenty-first-century skills. Meanwhile, hundreds of millions of children finish schooling lacking even the basic literacy and numeracy skills of the nineteenth century.
The problem is that the learning achievement profile, the relationship between the number of years children attend school and what they actually learn, is too darn flat. Children learn too little each year, fall behind, and leave school unprepared. In most developing countries schooling goals are not fulfilling even the most modest education goals. Schoolin’ just ain't learnin’.
Schooling: The Success of the Half Century
Some dreams do come true. On December 10, 1948, when the United Nations General Assembly adopted the Universal Declaration of Human Rights, universal free elementary education was a lofty ambition with little chance of fulfillment in the foreseeable future. Soon, international conferences declared not just the goal of universal education but specific target years for the goal to be accomplished. In the early 1960s, conferences declared a target date of 1980. In 1990 the World Conference on Education for All in Jomtien, Thailand, declared 2000 to be the target year for “universal primary education and completion.”1 In 2000, the UN's Millennium Development Goals Report set a target and a date: “Ensure that, by 2015, children everywhere, boys and girls alike, will be able to complete a full course of primary schooling.”2 It did not happen in 1960 or 1980 or 2000, but in 2013 it really is about to happen. The vast majority of countries will meet the Millennium Development Goal target for universal primary school completion, and very few countries will miss it by much.
Figure 1-1. The years of schooling completed by the average adult in the developing world more than tripled from 1950 to 2010.
Source: Data from Barro and Lee (2011, table 3).
This expansion of schooling has been a global transformation, especially in the developing world. The population of labor force age in the developing world has now completed three times more years of schooling than in 1950, when 60 percent of the labor-force-age population had no schooling at all. Figure 1-1 shows that the average completed schooling of adults went from 2.0 years to 7.2 years in just the sixty years from 1950 to 2010. Just think of it! The cumulative schooling in 1950 represented the schooling achievement since the dawn of human civilization, and in the developing world, 6,000 years of recorded human history had led to societies with an average of only 2.0 years of schooling (which implies that most people had none at all). In just sixty years the average schooling of the population increased by 5.2 years. Progress in expanding schooling has been 100 times faster from 1950 to today than from 387 B.C., when Plato's Academy was founded, to 1950.
Figure 1-2. Schooling in poor countries has expanded so rapidly that the average Haitian or Bangladeshi had more years of schooling in 2010 than the average Frenchman or Italian had in 1960.
Source: Data from Barro and Lee (2011, table 3).
The astounding fact is that the average developing-country adult in 2010 had more years of schooling, 7.2, than the average adult in an advanced country in 1960, 6.8. As figure 1-2 shows, levels of grade completion in even very poor countries are higher today than levels in rich countries were even as late as 1960 or 1970. Ghana's 7.8 average years of schooling in 2010 was not attained by the UK until 1970. Countries known as educational laggards, such as Bangladesh and India, have high attainment compared to many European countries in 1960. Even countries often deemed educational basket cases, such as Haiti, have more-schooled populations than France and Italy had in 1960.
It is striking that all types of countries—rich and poor, economically growing and stagnating, democratic and nondemocratic, corrupt and clean—have expanded schooling. While progress on some goals, such as poverty reduction, economic growth, or eliminating corruption, has been spotty, advancing the “human development” part of the UN's Human Development Index has been nearly universal. The Human Development Report 2010, prepared by the UN Development Program, showed massive schooling progress in nearly every country since 1970.3 According to the standard sources,4 the 2010 gross enrollment rate (GER) in primary school in peaceful, democratic, and prosperous Costa Rica was, not surprisingly, 110 percent. In neighboring Guatemala it was 113 percent. How can Guatemala—conflict-ridden, massively unequal, socially stratified, less than fully democratic, and poor—achieve the same enrollment as its much lauded neighbor? Enrollments are high nearly everywhere. Cambodia? 127 percent. In infamously corrupt Nigeria the GER reached 103 percent in 2006 (though it has since fallen). In the borderline “failed” state of Pakistan the GER in 2010 was 95 percent.
Figure 1-3. Schooling increased massively in nearly all countries, including corrupt, nondemocratic, repressive, and slowly growing countries.
Source: Author's calculations based on Barro and Lee (2011) data on schooling, ICRG data on corruption, POLITY 4 data on democracy, and Penn World Tables data on economic growth.
Good governments do schooling, but nearly all bad governments do it too. Figure 1-3 shows the gain in adult years of schooling from 1950 to 2010 for countries demonstrating a range of growth and governance performance. The best third of countries in controlling corruption saw their average schooling increase by 5.1 years and the worst third by 5.0 years. The most democratic third of countries saw schooling go up by 5.0 years and the least democratic third by 5.2 years. The freest third of countries saw schooling increase by 4.9 years and the least free third by 4.9 years. Economic growth generally increases resources available to families and governments, and facilitates expanding schooling. The top third in terms of economic growth saw schooling years increase by 5.4 years and the worst third by 4.9 years. The worst third on average had economic growth at zero percent, and began from a level of schooling of only 1.5 years, but schooling of the population more than tripled, even while their economies stagnated.
The success in expanding schooling was not the result of prosperous economies and democratic and capable regimes. Success happened because the goal of schooling was defined, and redefined, such that it could be consistent with the politics, state capability, and economic resources of every country. For the goal of universal schooling to be reached, the definition of schooling had to be made compatible with universal capabilities.
But there is a big problem with using schooling as the vehicle for achieving education goals. That problem is hidden in plain sight, right in the Millennium Development Goal. The Millennium Development Goal, like the original 1948 goal, is “universal primary education,” but the achievement of this goal is defined as universal completion of primary school. Even malign and autocratic governments wanted to expand schooling, and even weak and corruptible states could handle the logistics of schooling. Focusing solely on measures of schooling assumes that achieving schooling meets the goal of education, yet every person who can spell knows what happens when you assume.
Did reaching the goal of schooling keep faith with the goal of education?
Grade Learning Profiles: The Link between Schooling and Education Goals
Education prepares the young to be adults. The goal of basic education is to equip children with the skills, abilities, knowledge, cultural understandings, and values they will need to adequately participate in their society, their polity, and their economy. The true goals of parents, communities, and societies have always been education goals, and hence a multiplicity of learning goals. Schooling is just one of the many instruments in achieving an education.
An example of one possible education goal is all of a cohort of children at some age having a specified mastery of certain capabilities. This can be broken down into specific learning goals for children to attain by a specific age in a specific domain, such as reading fluently in their mother tongue by age ten, or being able to solve practical problems using arithmetic by age twelve, or having specific critical reasoning skills by age fifteen. An overall education goal would be a collection of learning goals such that a cohort emerges from youth equipped with all the skills, values, competencies, abilities, and dispositions desired by society. Early and intermediate learning goals, and goals in specified domains such as literacy or numeracy or science or history, are part of overall education goals.
The old saying that what gets measured gets done is not quite right, as not everything that gets measured gets done. The converse is more true: what does not get measured does not get done. Today, national governments and development agencies can provide data on myriad aspects of schooling: enrollments, expenditures, grade progression, completion, class sizes, budgets, and so forth. But on the education of a cohort there are next to no data. How many fifteen-year-olds today are ready for their future? No one knows.
Schooling and education goals are unified by the cohort learning profile. It has two elements, a grade attainment profile and a grade learning profile. The latter links years of school and capability. Figure 1-4 shows the learning trajectories of four hypothetical students, indicating their mastery (displayed on the vertical axis) as they persist, or do not persist, through schooling (demarcated on the horizontal axis). A schooling goal is measured as movement along the horizontal axis—another year in school moves the child along no matter what learning progress she has made—whereas dropping out stops schooling progress. A learning goal is measured along the vertical axis.
Schooling goals meet or exceed education goals based not only on whether students stay in school but also on whether the learning trajectory during schooling is steep enough. Suppose that the goal for school completion was a “basic” cycle of nine years. In figure 1-4, Bill drops out in grade four and meets neither an early learning goal, such as “reads fluently in grade three,” nor a final learning goal, nor even a schooling goal. Jack reaches exactly grade nine, and thus meets a schooling goal, but his learning per year was too little to meet either an early or a final education goal. Mary drops out in grade nine and Jill persists past grade ten, and both meet schooling and education goals.
Figure 1-4. The learning trajectories of individual students as they move through school provide the essential link between schooling goals and education goals.
Source: Author's entirely hypothetical trajectories of four students.
A key empirical construct used throughout this book is the grade learning profile, or the distribution of student capability of children enrolled in each grade. This is the aggregate of the individual student learning trajectories for the students enrolled in each grade.5 This distribution of mastery across students can be summarized in variety of ways. The proportion of students who are above a threshold—say, they can read a text fluently—is one summary of the learning profile. The average score on a common instrument that assesses capability in a domain across grades is another summary of the learning profile.
All legitimate schooling goals are reverse-engineered from education goals on the basis of assumptions about learning profiles. How long should the training of a doctor take? Well, start from the capabilities a doctor should have and make assumptions about how long it will take medical students to master those capabilities; this gives the length of medical school training. The training of an orthopedic surgeon takes a long time because the profession and the public rightly expect some level of competence before saw is placed to bone.
How long does it take to train a pilot? Churchill famously remarked, during the Battle of Britain, “Never in the field of human conflict was so much owed by so many to so few.” Why were there so few Royal Air Force pilots? Not for lack of volunteers—those were many—but because it took time to train a pilot before you could put him in the air against the Luftwaffe with any hope of his survival. The Battle of Britain was a race of casualties of existing pilots against the learning curve for pilots. The Japanese practice of kamikaze lowered the training time by cutting out mastery of landing a plane, but at an obvious cost to the pilot.
Schooling goals are based on assumptions about length of schooling and educational attainment, but you cannot fool a learning profile by making assumptions. The RAF might have cut pilot training time and asserted that the shorter training would be enough—except that the Luftwaffe was the final exam, and shorter training meant that exam would be fatally final for too many pilots. You cannot just assume any old learning trajectory and make it true. Conversely, if the learning trajectories of students are flatter than it was assumed they would be, then kids come out of school knowing less than expected, and a schooling goal no longer meets an education goal. Whether learning goals are met hinges on the steepness of the learning trajectories of individual students.
Learning Profiles Are Too Flat: Three Illustrations from India
In this section, I use data from three different studies from India that use grade learning profiles to illustrate troublingly low levels of learning. Why India? First, having lived there for three years, I know the situation well and have confidence in the data. Second, there have been three recent pioneering efforts to measure student learning performance in India, each providing not just the usual reporting of mastery at a single grade but also learning profiles tracking performance across grades: the Annual Status of Education Report (ASER), the Andhra Pradesh Randomized Evaluation Studies, and the Educational Initiatives study. Individually these studies have advantages and disadvantages in coverage and technique, but together they paint a clear and coherent picture of incredibly shallow learning profiles, with at best weak mastery of the fundamentals, and even poorer progress in conceptual understanding. Third, in addition to these three assessments, two Indian states have recently participated in the Program for International Student Assessment (PISA), which is coordinated by the Organization for Economic Cooperation and Development (OECD), so we now have detailed studies showing the learning profile, with internationally comparable figures. Fourth, India is huge, continental in scope, so itself can be used to show variations in learning profiles across states. Fifth, although the data are drawn from Indian samples, I am really using India to illustrate the conceptual points that are applicable to other nations around the world.
Andhra Pradesh Randomized Evaluation Studies: Flat Grade Learning Profiles in the Basics
The researchers Karthik Muralidharan and Venkatesh Sundararaman, working with the Indian state of Andhra Pradesh, the Azim Premji Foundation, and the World Bank, are carrying out one of the most impressive studies of schooling and education ever, the Andhra Pradesh Randomized Evaluation Studies (APRESt). It is striking in several respects. First, it has been carried out on a massive scale with hundreds of schools across different districts in Andhra Pradesh. Second, it has examined not just one possible intervention to raise quality but a whole variety of interventions, from performance pay to increased school grants. Third, the research model uses randomization—assigning schools randomly to the various “treatments,” and then comparing the results with those from a set of “control” schools—so that its findings have powerful claims to have identified the actual causal impacts of the treatments on learning. Fourth, the study developed (together with the organization Educational Initiatives, on which more in the next section) and used a sophisticated test that is able to assess both students’ rote learning and their deeper conceptual understanding. Fifth, by testing students in multiple grades and tracking students over time, the study produced a series of both cross-sectional learning profiles (averages of skill mastery across grades) and actual student learning trajectories (tracking individual students over time). Given the richness of this research, I frequently draw on its findings; but even before talking about the causal impact of the tested policy alternatives, one important set of findings is just what the learning profiles look like (see Muralidharan and Sundararaman 2010a).
Figure 1-5. The learning profiles for children in grades 2–5 in Andhra Pradesh, India, show strikingly little progress in the fraction of children who can answer even simple arithmetic questions, and almost no progress at all for a hard question.
Source: APRESt study (Muralidharan and Sundararaman 2010a).
To delve into the material on the learning profiles, we can start with the simple arithmetic problem of adding two single-digit numbers, say, 9 and 8. The best, though not so good, news is that 35 percent of children in grade two can answer this “grade one” question (the Indian curriculum stipulates that by grade one, students should be able to add two one-digit numbers). The bad news is that 65 percent cannot. The really bad news is that of the 65 percent that did not learn this simple arithmetic operation by second grade, only half learned it in the next three years of schooling. By the time these students finish the five grades of primary school, only 61 percent can do the simplest single-digit addition problems. Figure 1-5 shows grade learning profiles for a variety of learning benchmark questions. Slightly more children can do two-digit addition with no carrying over of digits, and somewhat fewer can do three-digit addition.
Performance on three-digit addition illustrates the stark flatness of the learning profile. The fraction who answered correctly increased very little in three full years of instruction, from just 30 percent in grade two to slightly over 50 percent in grade five, meaning that in three years, only one in five children learned to add three-digit numbers. Somehow roughly four out of five children who could not add in grade two were passed all the way to grade five without gaining this fundamental skill.
This lack of mastery of a rote skill is almost certainly an indication of weaknesses in deeper conceptual issues, such as not grasping the concept of place in a three-digit number. Even the dismal results on addition, unfortunately, exaggerate how many children understand arithmetic operations conceptually, as opposed to having simply memorized responses and a few procedures. When presented with a nonstandard form of addition, such as numbers in horizontal rows rather than columns—say, 200 + 85 + 400 = 600 + ?—which requires some manipulation, there had essentially been no learning at all: only 10 percent of fifth-graders could answer that question. It might look harder in the horizontal arrangement than it actually is, since 200 + 400 = 600, so the arithmetic is easy for a child that can understand the problem.
Mathematics is about concepts and their application, not about doing arithmetic for its own sake. Students gain very little from year to year in fundamental concepts such as weight, definitions, or fractions. In grade two, 30 percent of students could read the weights on two boxes and figure out which was the heaviest, but this percentage had increased to just 50 percent by grade five (table 1-1). So half the students in fifth grade still didn't get this fundamental concept about what numbers represent. As shown in the last column of table 1-1, “Percent who learned,” only 12 percent of students (one in eight) who did not understand the concept one year got it right the next.
Nineteen percent of second-graders could recognize which geometric figure met the definition of a triangle, and by the fifth grade this had risen to only 35 percent. (Granted, this question is more conceptual that it might appear. First, the triangle in the question pointed downward, whereas in most of the examples, children would have seen triangles that pointed upward. So the child had to realize that rotating a figure does not change its classification as a triangle. Second, the test showed figures that are triangular, such as the cone, but not a triangle per se. This required the child to understand that in the context of geometry, “triangle” has a formal definition that does not always correspond to familiar notions of “triangular.”) The total learning gain in three years was only 16 percent of the students. Of the 80 percent of children who could not answer this question in second grade, only 4 percent of those gained the ability in third grade. The progress in the conceptual skill of recognizing formal definitions is painfully slow.
Table 1-1. Students gain little conceptual mastery: only about 12 percent of children (1 in 8) make progress each year in answering even moderately conceptual (nonroutine) questions.
Source: APRESt (2010).
a. “Percent who learned” is the increase in the fraction correct divided by the fraction incorrect in the previous grade (e.g., 39 percent in grade 3 less 30 percent in grade 2, for a net gain of 9 percent of all students, divided by 70 percent who did not know in grade 2 = 13 percent). I assume no retrogress, that is, all of the net gain is due to learning, but since APRESt, the source of the data, does not have individual learning trajectories, I am unsure of the actual gain dynamics.
The ASER Survey: Assessing Flat Learning Profiles in the Basics throughout India
Every year since 2005 the ASER assessment has been carried out by the New Delhi–based NGO Pratham and its partners and, now, the ASER Centre, and it is planned to continue at least until 2016. (ASER is the name of the organization that administers tests and produces results, the name of the test, and the shorthand name of its report presenting test results. In addition, aser is Hindi for “impact.” The assessment is described in the section “Data Sources.”)
ASER uses a simple instrument to assess basic reading and arithmetic skills on a massive scale. The ASER exercise has several advantages. It surveys a sample of in- and out-of-school children. For a variety of reasons, nearly all assessments of learning are done only on in-school children, which gives an overly optimistic picture of an age group's learning achievement, as out-of-school (or behind-grade-for-age) children are not tested. Even if the system is designed so that a child of age fifteen should be in grade ten, in India, about half of the children that age have either dropped out or are in a lower grade. ASER results can be used to generate both grade learning profiles and cohort learning profiles, the latter combining the grade attainment profile and grade learning profile, which in India are very different.
Furthermore, the ASER exercise is massive and repeated regularly, and the raw data are available. More than 600,000 children are tested annually. The sample is drawn from almost every district (an administrative unit with an average population of around 1.5 million) in rural India, using population proportional sampling for the village sample, with thirty villages per district and twenty households per village. (Sampling and surveying in urban areas is much more complex and was done only once, in 2007.) The annual repetition of the exercise allows one to cross-check the reliability and validity of the estimates. Summaries of the data are readily available in the yearly reports; the raw data are available, with a lag, as well.
The ASER instrument is simple—and it takes an enormous amount of intellectual sophistication for a test to be this simple and still be useful. The reading test instrument is one page, with different competency benchmarks: letters, simple words, simple sentences, and a reading passage expected to be understood by grade two students. In the assessment, the child is shown words and progresses to his highest level of competency. (Owing to the great diversity in local languages in India, the test instrument is available in all of them, and children may choose the language.) The numeracy instrument is similarly simple. Children are asked whether they can recognize two-digit numbers, and can progress to the highest level tested, three digits divided by a single digit (say, 824 divided by 6).
Table 1-2. Cohort assessment of skill mastery in India: nearly three-quarters of 10- to 11-year-olds do not master four basic cognitive and practical skills, nor do more than a third of 15- to 16-year-olds.
Source: Author's calculations, based on ASER 2008 survey data.
I once traveled with the ASER test administrators in Uttar Pradesh, the largest state of India. It was emotionally devastating to see children eleven years old, supposedly in class three, who were unable to read simple words. I watched as one boy around ten years old who reported having been in school for three years turned the literacy test card this way and that, not even sure which way was up.
In 2008, ASER covered two other practical skills beside literacy and numeracy: telling time and handling money. For telling time, children were asked to state the time from pictures of two clocks. With money, children were asked questions such as “If this hand has two five-rupee coins and the other two has ten-rupee notes, which hand has more money?”
The cohort results show that less than two-thirds (64.5 percent) of children ages fifteen to sixteen had mastered all four of these skills. Table 1-2 shows three bottom-performing, three middle-performing, and two top-performing states, along with the all-India (rural) average. In the low-achieving states, only about half the youths are competent in all four skills. In the highest-achieving states, these skills are practically universal. (Interestingly, children perform better on the practical skills, which they may have acquired out of school.)
Figure 1-6. Learning achievement profiles by grade attainment in Uttar Pradesh show shockingly low learning levels, even for grade 2–level skills.
Source: Author's calculations based on ASER 2008 survey data.
ASER data for 2008 reveal strikingly flat grade learning profiles. In Uttar Pradesh (see figure 1-6), only 30 percent of children in grade four could read a simple passage. This is shocking, for nearly everything about the organization of the schools, method of teaching, and curriculum assumes that children in grade four can read. By grade five, 41 percent could read. This implies that 10 percent of children—only one in ten—learned to read a simple story in an entire year of schooling.
Table 1-3 shows reading levels for children enrolled in grades four and five in 2008 in Uttar Pradesh. The table may be thought of as a transition matrix (although the data track cohorts, not individual children) in which attending school increases the level of performance from one grade to the next. Nearly 21 percent of fourth-graders could recognize letters but not read words, and if dropping out and repeating the grade are set aside, by grade five nearly 15 percent could still only recognize letters. The combined category of inability to read words or recognize letters fell from 28.3 percent in grade four to 19.2 percent in grade five, so there is some progress, but amazingly little. The result of this cumulative slow progress is that one in five children enrolled in grade five in Uttar Pradesh cannot even read simple words. This lack of any functional literacy means that everything else that is happening in school for these children is unlikely to make sense, as nearly all schoolwork in grade five involves some reading.
Table 1-3. Children in Uttar Pradesh show little progress from grade 4 to grade 5 in reading.
Source: Author's calculations, based on ASER 2008 survey data.
Educational Initiatives: Studying Flat Learning Profiles in Concepts
The problem of low learning levels is even worse than the ASER numbers reveal, for ASER assesses mastery of extremely basic skills in reading and arithmetic. A recent study by the Indian think tank Educational Initiatives (EI) probed not only “mechanical” learning but also conceptual mastery (Educational Initiatives 2010). EI's sample was 101,643 schoolchildren in grades four, six, and eight in 2,399 government schools in forty-eight districts and eighteen states in India, representing about 74 percent of the Indian population, urban and rural.
The EI study paired mechanical questions, asked in exactly the way the textbooks children would have been exposed to would, with conceptual questions covering the same material. Table 1-4 shows an example of this pairing. Multiplying a two-digit number times a three-digit number is computationally complex, but even without multiple choices, 48 percent of grade six students could calculate this multiplication. However, a conceptual understanding that multiplication is repeated addition is rare. Questions probing this understanding produced fewer correct answers by grade six students on a multiple-choice question than random guessing would have produced.
Table 1-4. Performance on conceptual questions is often worse than random guessing, even when students do better on mechanical questions measuring the same skill.
Source: Educational Initiatives (2010, p. 30).
Similarly, when presented with a triangle showing measurements of each side, almost half could calculate the perimeter. However, when presented with a problem in which students had to understand the concept of perimeter (for example, in a rectangle, perimeter is twice length plus twice width, so that if perimeter = 20 and width = 4, then length = 6), students again got this question right much less than random guessing would have yielded.
A second example from EI is related to simple concepts of length and measurement. Nearly all Indian textbooks teach measurement using an example in which an object, such as a pencil, is laid next to a ruler with the base of the object at zero (figure 1-7). However, if students are presented with the object displaced one centimeter from zero, they are thrown off. Typically, measuring length is taught in third grade. But most students appear to learn that “length is the number associated with the tip of the object” rather than actually learning the concept of length and measurement. Even as late as grade eight the most common answer to the question in figure 1-7 about the pencil length is “six centimeters”; only 34.7 percent of eighth-graders get the right answer, and 38.8 percent continue to answer that the pencil is six centimeters long (the rest of the students, 26.5 percent, answer incorrectly, but something other than six centimeters). When children fail to acquire conceptual understanding early, low capability can persist.
Figure 1-7. The majority of students in India do not grasp the concept of measurement with a ruler, even by grade 8—more say the pencil is 6 cm long than the correct answer.
Source: Educational Initiatives (2010).
Overall, these three assessments—ASER, APRESt, and EI—consistently show that the learning progress of Indian students is very slow, as shown in table 1-5. In the EI assessment, the average rate of increase of “percent correct” was only 4.9 percent per year of schooling, meaning fewer than one in twelve students gained the ability to answer a given question in a given year. In the APRESt study, this figure was 6.1 percent per year. One reason these rates are so low is that the EI and APRESt evaluations combined mechanical and conceptual questions, which demonstrated different rates of progress. The ASER results show that progress on even simple basic mechanical skills, such as reading a paragraph or dividing, was still slow. Four out of five students who entered a grade of basic education (grades two to eight) in India unable to read still were unable to do so even after another full year of instruction (see table 1-5).
The LEAPS Study: Following Students across Grades in Punjab, Pakistan
I want to illustrate one feature of learning profiles using data that track individual students over time on the same questions. Such data are available from a study conducted in Punjab, Pakistan. The three learning profiles presented so far are “synthetic”: they track the performance of children in grades but do not track the same children. The Learning and Educational Achievement in Pakistan Schools (LEAPS) study is longitudinal, tracking around 6,000 Pakistani children from grades two to six, so researchers have each child's individual learning trajectory and can build up direct cohort (not grade) learning profiles.
Table 1-5. Grade learning profiles show little progress in mechanical literacy and numeracy mastery from year to year, and even less progress in conceptual mastery.
| Skill/competency by instrument | Average percentage point increase in fraction correct per year of schooling | Of students who didn't know skill, percent who learned in the next grade |
| Education Initiatives, all items asked across grades (4 and 6, 6 and 8, or 4, 6, and 8) in reading and mathematics, 18 states | 4.9 | 8.5 |
| APRESt, 11 common questions in mathematics (mechanical and conceptual), grades 2 to 5, four districts of AP | 6.1 | 9.4 |
| ASER 2011, division (3 digit by 1 digit, mechanical only), grades 2 to 8, all rural India | 9.0 | 12.6 |
| ASER 2011, reading grade 2–level story (mechanical only), grades 2 to 8, all rural India | 11.8 | 21.8 |
Source: Pritchett and Beatty (2012, table 1).
These cohort learning profiles show results similar to those from India. On very simple skills that can be learned by rote, performance is high, but on skills that require even a modest degree of comprehension only about half of students master the skill at age- and grade-appropriate levels, and progress is very slow: only one child in five masters simple multiplication and one in seven masters simple division per year (table 1-6).
One feature that arises when tracking individual students and their answers to specific questions is that there appears to be some amount of learning and forgetting. The LEAPS questions are not multiple choice, so the odds a student gets it right by pure chance or guessing are low. Nevertheless, the increase in the averages from year to year across all students masks considerable “churning” in the individual learning trajectories, as some students get the question right in grade four but miss it when asked in grades five and six.
Table 1-6. Actual cohort learning profiles from tracking the same students over time in Pakistan show the same slow pace of learning as grade profiles.
Source: Data from LEAPS 2007 study; calculations provided to author by LEAPS study authors.
The Consequence of Flat Learning Profiles: Schooled but Uneducated
The grade learning profile can be thought of as a ramp to the door of opportunity, in civil society, in the polity, and in the economy. If the ramp isn't sufficiently steep, even walking it leaves you unable to get a foot in the door. In the first part of this chapter I focused on how shallow the ramp is: children progress little in conceptual mastery and capability as they move through school. Now I turn to the height most children in developing countries reach on the ramp, their cumulative learning at or near the end of schooling, which for most of them is at around age fifteen, or grade eight. Where does school leave children standing relative to opportunity? In Himachal Pradesh, India, 58 percent of students were assessed at level 0 on an international standardized science test, PISA, whose scale of levels runs from 1 to 6. That's right: most could not answer enough questions even to be placed on the scale. Tragically, as seen in results on internationally comparable exams, in this respect India is not so different from many other countries.
International Comparisons of Capabilities
There are many technical, even esoteric, details that go into constructing valid and reliable assessments, which are explained exceptionally well by Koretz (2008). A very small handful of people in the world are expert in assessment, and I am not one of them, nor would I expect most of my readers to be. But a few simple characteristics of assessments are central to understanding reports from two main international comparative assessments, PISA, which is coordinated by the OECD, and the Third International Math and Science Study (TIMSS), a project of the National Center for Education Statistics and the U.S. Department of Education (see Gonzales et al. 2004; OECD 2009):
— First, what is the skill set domain that assessments are trying to measure the mastery of?
— Second, how well do those assessments capture that?
— Third, how are those measures scaled?
Both PISA and TIMSS assess overall performance in large domains. PISA covers language, mathematics, and science and is intended to capture the students’ ability to apply learning from these domains to real-world contexts. TIMSS measures students’ conceptual mastery of the typical mathematics and science curriculum.
I work on the premise that these two assessments provide reliable and valid measures of mastery of the learning domains they cover.
The remaining issue is the scaling of these measures. Any assessment contains questions, gets answers, assigns points to answers for each question, assigns an importance to each question, and comes up with a number. The choice of number is arbitrary (a test in school can be scaled as “percent correct” from 0 to 100; the SAT is scaled from 200 to 800; the ACT test is scaled from 1 to 36). Both TIMSS and PISA have chosen to norm their results so that the average score of students from OECD countries is 500 and the student standard deviation is 100. PISA also classifies students by their “level” of performance on the basis of their score, where each of six levels is described in competencies within each of the three subject areas. A description of the tasks students at the lowest levels should be able to perform is provided in table 1-7.
In 2009, two states in India, Himachal Pradesh and Tamil Nadu, participated in PISA. For the first time in decades there was an internationally comparable benchmark of how an in-school cohort of Indian children performs in language, mathematics, and science.6 The results are consistent with flat learning profiles: fifteen-year-olds, even those still in school, mostly lack even the most basic education (see table 1-7). In Himachal Pradesh, about 60 percent of children were below the bottom category, that is, they did not even reach level 1a in reading or level 1 in math.
Table 1-7. PISA results for two Indian states and OECD comparison figures: half or more of Indian students were in the lowest categories in reading, mathematics, and science.
Source: Compiled from PISA (OECD 2010, tables B.2.2, B.3.2, B.3.4).
The PISA data allow us to compare the distribution of student capability from low to high across countries. Figure 1-8 shows the distribution of student results in mathematics for Tamil Nadu and Denmark, a country with typical OECD results. The huge difference in the average score—351 in Tamil Nadu versus Denmark's 503—also implies that many more students are in the bottom category (55.6 percent in Tamil Nadu versus 4.9 percent in Denmark) and fewer in the top category. In Denmark, 11.6 percent of students are in the highest two categories (levels 5 and 6) of PISA capabilities. In Tamil Nadu, the estimate for these same categories is zero percent, since there were too few students in the categories to measure accurately.
Linking Grade Learning Profiles to Assessment Results
Assessments of cumulative learning at a given age or grade and learning profiles represent the same reality in different ways. A flat learning profile results in low capabilities, and low capabilities means the learning profile was flat. This is because the distribution of scores of any population, such as students in grade eight in Himachal Pradesh or Tamil Nadu, is the result of the starting point plus the cumulative learning profile of the tested population.
This basic, if not definitional, link between learning profiles, end-of-basic-education student outcomes, and, with grade attainment profiles, cohort learning attainment is a core issue of this book. Figure 1-9 shows a pair of three-dimensional figures that link the distribution of results across students with learning trajectories. A grade learning profile—such as progress in an average score or in the percentage of students above a threshold—summarizes the evolution of testing distribution summary statistics. Conversely, each distribution of results is the cumulative result of learning trajectories.
Figure 1-8. A comparison of student 2009 PISA mathematics score distributions in Denmark and in Tamil Nadu, India, shows distinct differences in achievement.
Source: Author's simulations using PISA data (OECD 2009).
Figure 1-9 combines the three elements of a measure of capability in some domain (the y-axis), the progression through school (the x-axis), and the distribution of students (the z-axis). Three-dimensional diagrams can be hard to understand, but they show the whole picture most effectively. Figure 1-9 shows that in each year of schooling, there is a distribution of student capability in any subject. For example, we can start at the beginning of grade three in mathematics. During the school year, some children learn more than others, some learn more in some years than in other years, some have good teachers and learn a lot, others have bad teachers and learn little. There is also forgetting and “depreciation” of learning over time. The net learning of each child across a school year, plus some dynamics of grade progression, results in a new distribution of capability in mathematics of students enrolled in grade four. This process repeats in subsequent grades. If children are tested at age fifteen, in grade nine (82 percent of Danish fifteen-year-olds were in grade nine), the resulting distribution of capability is a result of the cumulative learning trajectories from conception to age fifteen.
Figure 1-9. The (average) grade learning profile tracks the gains in the mean of distribution of student capability.
Source: Author's simulations using PISA data (OECD 2009).
In figure 1-9, I assume the average child in both Denmark and Himachal Pradesh begins at 160 “PISA score equivalents” in grade one.7 We know that Danish children at age fifteen, most of them in grade nine, have an average PISA score of 503. This means the learning profile must have taken them from their initial point (whatever we assume it to be, in this case 160) to that score of 503 at age fifteen in grade nine over the course of their schooling (and out-of-school) learning experiences. Assessments of a cohort are inextricably linked to grade learning profiles, though not necessarily as simply as the graph suggests. The three-dimensional graph illustrates that the grade learning profiles in two dimensions (grade and capability) summarize the distributions across students into a single number (say, the percentage of students who answer a question correctly, or an average score across an index of items, or, as we shall see later, the percentage of students below some threshold). I show three full distributions across students along this trajectory for grades three, six, and nine.
Comparing capabilities across a sample of students of either a certain age or grade in part means comparing learning profiles. For instance, suppose that children in Denmark and Himachal Pradesh begin at 160 “PISA score equivalents” in reading.8 The average score in reading in Denmark for those fifteen years old is 495 (like the math score of 503, very typical of an OECD country), whereas in Himachal Pradesh it is 317. The grade learning profiles must have led to these different distributions. But the main point, illustrated in figure 1-10, is that the initial capability plus the grade-to-grade learning (and forgetting) dynamics over time add up to cumulative acquired capability. If we are concerned that children are leaving school inadequately prepared for adulthood, we must focus on the entire learning profile, both out-of-school learning (including even any relevant preconception issues) and the in-school grade learning profile.
International Performance Comparisons: Developing Countries versus OECD Countries
The 2006 PISA report states, “PISA assesses how far students near the end of compulsory education have acquired some of the knowledge and skills that are essential for full participation in society. In all cycles, the domains of reading, mathematical and scientific literacy are covered not merely in terms of mastery of the school curriculum, but in terms of important knowledge and skills needed in adult life.” In other words, PISA's aim is not only to capture the simple decoding skills of reading or procedural skills of arithmetic but to assess how these skills prepare children for adulthood. PISA's target population for assessment is “15-year-old students attending educational institutions located within the country, in grades 7 or higher.” The sampling is therefore explicitly student-based, not cohort-based (OECD 2009).
Figure 1-10. Comparison of assessed reading capability in grade 9 in Himachal Pradesh and in Denmark in grade 9 illustrating the link between lower capability and a flatter learning profile.
Source: Author's simulations, based on PISA 2009(+) (OECD 2009) averages and distributions.
Table 1-8 compares only selected countries evaluated in the PISA program. Except for using Denmark as a benchmark, as a development expert I focus on the participating developing countries. Even for the mostly middle-income developing countries participating in PISA, it is striking how low the learning levels are relative to Denmark's. One way to illustrate this low learning is to ask where the average student in a developing country would rank if that person were taking the test in Denmark. Brazil is a large middle-income developing country. The average in Brazil was 370, so the typical Brazilian student would be below the seventh percentile (6.8) in the Danish distribution.
Table 1-8. PISA results show that 15-year-old students in developing countries are massively behind the typical OECD country students in mathematics capability.
Sources: For columns 1–4, PISA (OECD 2010, tables B.3.1 and B.3.2). For columns 5–6, author's calculations, based on PISA (OECD 2006, 2010) data and means and standard deviations for Denmark.
a. Estimating the standard deviation from 5th and 95th percentiles under assumption of normal distribution.
b. Assuming a normal distribution for Denmark.
TIMSS's approach is slightly different from PISA's in that TIMSS attempts to assess mastery of the mathematics curriculum in grade eight. Nevertheless, the results for developing countries are similar to PISA's in respect to fifteen-year-olds’ capability and application. In table 1-9 I again compare the developing countries to an OECD country—in this case Australia, chosen because I love kangaroos.
First, the average developing country is at only 386 (on a similar OECD student norm of 500 average, with 100 as the OECD student standard deviation). Australia is at 499, which is very near 500 and a full student standard deviation above 400.
Table 1-9. Results from TIMSS show developing countries are far behind in mathematics capabilities.
Source: TIMS (2008, Exhibit 1.1, 2.2, D.2).
a. Estimating standard deviation from the 5th and 95th percentiles under the assumption of a normal distribution.
b. Assuming a normal distribution for Australia.
Second, more than 50 percent of developing-country students are below the “low” international benchmark of 400, compared to only 11 percent for Australia. Conversely, on average, only 4 percent are above the “high” international benchmark, compared to 24 percent for Australia. Again, if one calculates how far behind the country is relative to its own distribution, we find that developing countries are typically one to two country full OECD student standard deviations behind Australia. Finally, similar to the PISA comparisons to Denmark, the typical developing-country student is only in the tenth percentile of the performance of Australian students.
Eric Hanushek and Ludger Woessmann (2009) have done the most complete compilation of all of the available international tests into a single comparable measure of learning. How far behind on their composite measure are students in lower secondary schools in developing countries? Figure 1-11 scales their overall country averages by the OECD student standard deviation and shows that nearly all developing countries are one or more international assessment student standard deviations below the OECD average.
The Best (and the Richest) Developing Countries’ Performance
In a world of global competition, success, especially in some economic activities, may depend on more than the average skills of the typical worker but also on the number of superstars among the globally best (Pritchett and Viarengo 2009). But even in upper-middle-income countries such as Mexico or Brazil, low average performance on PISA and TIMSS tests, combined with often low variance across students, means that very small fractions of students are in the two top global distribution categories (see the column “Percent at level 5 or above (>607)” in table 1-8 and the column “Percent above high benchmark (550)” in table 1-9).
If we look just at the top category—roughly the global top 10 percent—things are even more dire. Pritchett and Viarengo (2009) calculated that even in Mexico, a country with more than 100 million people, all of the Mexican students who achieved the global top 10 percent yearly in mathematics could fit into one smallish auditorium—there are only around 3,000 to 6,000 total. Hanushek and Woessmann (2009) have also calculated the proportion of students tested who were in the global top 10 percent. As figure 1-12 shows, in most developing countries it is significantly less than 1 percent.
Figure 1-11. Students in most developing countries are at least an OECD student standard deviation behind the OECD level of learning.
Source: Adapted from Hanushek and Woessmann (2009).
That there are very few students at the top levels of performance means that the problems with educational systems do not affect only poor children. Zero percent of students at PISA levels 5 and 6 means zero percent of students from poor families and also zero percent from rich families reach this level because zero is zero.
In figure 1-13, Filmer (2010) compares the scores of children from the bottom and top quintiles of the available countries by socioeconomic status using data from PISA. Not surprisingly, massive gaps—roughly 100 points—separate the learning outcomes of richer and poorer students. Perhaps more surprising, even the rich in developing countries also lag. For example, in Indonesia, the richest quintile has scores around 450—less than the 500 for the poorest quintile in Korea or the same as the poorest quintile in the UK. So in poor countries, the richest are still getting a mediocre education, and the poor cannot be said to be getting any education at all.
Figure 1-12. Developing countries are producing very small proportions of students in the global top 10 percent.
Source: Adapted from Hanushek and Woessmann (2009).
Figure 1-13. Inequalities in PISA 2006 reading test scores show that learning outcomes are the worst for the poorest in poor countries, but are pretty awful for the richest, too.
Source: Filmer calculations with PISA data, provided in private communication with the author.
Meeting a Learning Goal for All Children
So far nearly all the results we have used measure the capabilities of pupils in school. Since, with the exception of ASER, sampling and testing are nearly always school-based, we do not know much about the learning of out-of-school children. This means that the grade learning profiles, as depressing as they are, actually exaggerate the overall progress of a cohort of children, for the results do not account for students who drop out or fail to progress. A genuine learning goal should not be based solely on the learning of those currently enrolled but should reflect the skill set of the entire cohort, including those who never enrolled or who enrolled and later dropped out. How close are developing countries to a cohort-based learning goal: that every child should emerge from childhood into youth and adulthood educationally equipped for life?
The short answer is that no one knows. In the push for schooling, education got pushed aside. The world has reams of data on schooling but almost none on learning. Very few countries can track the learning achievements of its students over time. Very few have measures of learning achievement based on cohort rather than on enrolled students. Deon Filmer, Amer Hasan, and I (2006) attempted to estimate how many fifteen-year-olds in various countries were currently meeting a learning goal (the results of our study are presented in table 1-10). To arrive at these estimates we had to make several assumptions, because existing information was inadequate. We assumed PISA level 1 proficiency as an illustrative “low” learning goal. As a “high” learning goal, we calculated what proportion of fifteen-year-olds was above the OECD mean value of 500.
Second, we needed assumptions to estimate the learning levels of a complete cohort from only school-based tests. PISA tests fifteen-year-olds, who are in different grades, so we used the learning levels across the tested grades and did the simplest possible thing: we extrapolated linearly the distribution of achievement (by extrapolating the mean and keeping the coefficient of variation constant). Fortunately for us, the results are quite robust to the assumptions, and both the ASER and EI data from India suggest that linear extrapolations past the very early grades do not do too much violence to actual learning profiles by grade.
Table 1-10. Even in middle-income countries with high average levels of schooling, and thus meeting the MDG schooling target, between one-third and two-thirds of 15-year-olds do not meet even a low learning goal.
Source: Filmer, Hasan, and Pritchett (2006, tables 2, 3, and 4), Barro and Lee (2011) for average years, Filmer (2010) for cohort completion.
*Note available. Cohort completion data is from Filmer (2010), available only for countries with DHS surveys.
The results of our calculations, as presented in table 1-10, are sobering. Mexico provides a useful example. Mexico in many ways is on the verge of being a developed country (and is now a member of the OECD). Primary schooling is nearly universal, and the average level of education for those age fifteen and above is nearing nine years. But in 2003, according to our calculations, half of fifteen-year-olds were at proficiency level 1 or below for mathematics, 39 percent for reading, and 38 percent for science. In Korea those numbers for 2003 were 2 percent, zero percent, and 2 percent, respectively.
According to our calculations (which were based on the best data we could find and the most plausible assumptions we could make), things are worse in the middle-income countries of Brazil, Turkey, and Indonesia. Averaged across the three PISA subjects, over half of the recent cohort is below a potential learning goal. In mathematics, in each of these three countries two-thirds of fifteen-year-olds are below a standard that is essentially universal in Korea.
TIMSS tests children in grades four and eight, rather than an entire cohort, and includes a variety of developing countries. This means that to calculate a cohort learning goal deficit, one must make assumptions. The first part, the grade attainment of a cohort, is widely available from household surveys. To calculate achievement for each grade, Filmer, Hasan, and I (2006) did the simplest possible simulation: we took the mean score given for grade eight (grade nine for Rajasthan) and then extrapolated it backward and forward using a grade increment, calculated as the linear increment to get from a minimum of 100 on enrolling in school to the observed score. We assumed that the coefficient of variation of student scores was constant across grades. Then, using the assumption that scores follow a normal distribution, we calculated the fraction of students at each grade attainment level who would be above any given threshold. We chose a potential low learning goal in the TIMSS assessment of 420, for three reasons. This is roughly a typical country student standard deviation on the TIMSS (which is around 80) below the OECD normed score of 500. Second, this is near the threshold for proficiency level 1 in PISA (although the two instruments are not comparable). Third, the only country with both a learning goal calculation and a TIMSS 2003 score is Indonesia, and choosing 420 gives an estimate of 56 percent for Indonesia, which is modestly better than the 68 percent estimated from TIMSS, but not wildly off. To be sure, these calculations are weak, yet their very weakness supports my overall contention concerning the lack of information on learning as opposed to schooling. These calculations provide some information about the achievement of cohorts.
Poorer countries are more likely participate in TIMSS than in PISA, which probes mainly middle-income countries, and the TIMSS results are more striking (I report these results in greater depth in chapter 2). In Ghana, 98 percent of a cohort fails to achieve a learning goal of 420; in the Philippines, 67 percent, and in Rajasthan (bearing in mind the lack of TIMSS comparability), 75 percent. Even when schooling is completed the deficits in education are massive.
Despite the failures of schooling, the victory of schooling can now lead to the rebirth of education. Nearly all children in the world start school. Most of them progress. The goal of universal completion of primary schooling has been achieved in nearly all countries. Schooling goals were never based on the notion that the schooling was itself the goal; no one even thought the true and total objective was just coming to a building called a school for a certain length of time. Time served is how we characterize prison terms, not education. Rather, schooling goals were based on the belief that schooling would lead to education, that children who completed the required schooling would be equipped for life. We know now that this is untrue. At the low pace of learning common in the developing world, completion of just primary education provides almost no one with adequate skills, and even completion of a “basic” education of eight or nine years of schooling leaves half the students unprepared for the twenty-first century.
The question is, what is to be done?
1. Clemens (2004) shows that meeting the time-bound enrollment targets nearly always implied expanding schooling systems in the most lagging countries far in excess of what any country had ever achieved.
2. See www.un.org/millenniumgoals/education.shtml.
3. United Nations Development Program (2010).
4. Data from the World Bank reports:http://databank.worldbank.org/data/views/reports/tableview.aspx.
5. The learning profile of those enrolled is a descriptive technique, not an assertion of causality. If only the children with the highest scores in grade three are allowed to progress to grade four, then the average child can know more in grade four than the average child in grade three, even if no child learns anything in grade three. The difference across grades in the learning profile does not imply children “learned” that much in a given grade, as the composition of students might change.
6. Himachal Pradesh and Tamil Nadu are considered among the educationally more progressive states, having achieved high levels of enrollment and grade completion. One drawback of the PISA, though, is that it assesses only fifteen-year-olds in school, not a whole cohort of fifteen-year-olds, so the results are overly optimistic. A footnote warns that PISA could not verify that the sampling in Tamil Nadu and Himachal Pradesh met quality standards as they could not be confident the sampling frames of fifteen-year-olds were complete. No one knows what bias this might have induced (OECD 2009).
7. Actually, controlling for other factors in a multilevel model, the estimated average gain per year in the PISA reading score across OECD countries is 40 (OECD 2009, table A1.2), which implies that if the grade nine score was 500 after eight years of constant gain, the initial grade one value would have been 160.
8. Of course, it is much more complex than this, as children in Himachal Pradesh may have suffered from in vitro malnutrition, diseases, or other early disadvantages, and hence may have been much less reading-ready upon entering grade one than Danish children. Moreover, grade progression is much more irregular in India than in Denmark, so the average score for fifteen-year-olds represents children in many different grades in Himachal Pradesh, which means the actual PISA score is an enrollment-weighted average across the grades.