FIGURE 5.1: The front page of a Web site that makes a clear claim for its product.
He who would do good to another must do it in Minute Particulars
General good is the plea of the scoundrel, hypocrite, and flatterer
For Art and Science cannot exist but in minutely organized Particulars.
—William Blake1
So far, this book has described evidence that we believe things for poor reasons (Chapter One) and that we are especially susceptible to reasons that sound consistent with sensibilities from the Enlightenment or the Romantic era (Chapter Two). I’ve described what constitutes good science (Chapter Three) and how to use it (Chapter Four). It’s time to get specific about how we can differentiate good science from bad.
This chapter and the three following outline a four-step process to help you evaluate the likely scientific soundness of a proposed curriculum, teaching strategy, textbook—anything that is purported to help children learn. Note that I said the likely scientific soundness. I freely admit—no, I emphasize—that what I’m recommending is not a substitute for a thoughtful evaluation by a knowledgeable scientist. Rather, it’s a workaround, a cheat. As such, it’s imperfect. The great advantage is that it doesn’t require a knowledgeable scientist.
When someone approaches you with an education product—whether you’re a parent, a teacher, an administrator, or a policymaker—he is asking you to change something. He wants you to change what you do in your home, classroom, or school in a way that is going to affect the children in your charge. As a shorthand, I’m going to use the term Change to refer to a new curriculum or teaching strategy or software package or school restructuring plan—generically, anything that someone is urging you to try as a way to better educate kids. I will use the term Persuader to refer to any person who is urging you to try the Change, whether it’s a teacher, administrator, salesperson, or the president of the United States.
To get started in your evaluation, you need to be very clear on three points: (1) precisely what Change is being suggested; (2) precisely what outcome is promised as a consequence of that Change; and (3) the probability that the promised outcome will actually happen if you undertake the Change. All other considerations are secondary at this point and should be considered distractions. “Strip it” is a method to get to these minute particulars that Blake refers to in this chapter’s epigraph. (We’ll get to “Flip it” in due course.)
To strip a claim to its essentials, I suggest that you construct a sentence with the form “If I do X, then there is a Y percent chance that Z will happen.” For example, “If my child uses this reading software an hour each day for five weeks, there is a 50 percent chance that she will double her reading speed.” Of course, the agents might vary: the person doing X might be a student, a parent, a teacher, or an administrator, and the person affected by the outcome (Z) might be any of those. Note too that the value of Y (the chance that the desired outcome will actually happen) is often not specified. That’s fine. Right now all you’re trying to do is be clear about the claim made by the Persuader, and if she has left Y out, she’s left Y out.
For example, have a look at the front page of the Web site shown in Figure 5.1.
FIGURE 5.1: The front page of a Web site that makes a clear claim for its product.
This Web site—unlike many—is quite clear in its claim. X is the use of colored overlays when reading. The promised outcome (Z) is improved reading, and the term “correct the processing problem” is used, so the claim seems to be not simply that things get better but that the problem will be corrected. The chances that the colored overlays will work (Y) are less clear from this page, but on another page of the Web site, one sees the claim that 46 percent of people with “reading problems, dyslexia, and learning difficulties” can be helped by the method. So the stripped claim offered on this Web site is “If your dyslexic child uses colored overlays when reading, there is a 46 percent chance that your child’s reading problem will be corrected.” At this point you should suspend judgment as to whether that sounds likely true or false. All you’re trying to do now is to gain a clear idea of exactly what is being claimed.
Exhibit 5.1 is another example. It’s abstracted from a blog, but I could easily imagine its being part of a professional development session on student motivation. It’s titled “Five Ways to Get Students Excited About Learning.”
EXHIBIT 5.1: Summary of Advice for Teachers on the Teaching of Writing, Drawn from a Blog Entry
The Change recommended in Exhibit 5.1 is much harder to strip using the “If I do X, then there’s a Y percent chance that Z will happen” formula, and that’s informative. X is hard to pin down, and Y is unspecified. To be fair, this advice was drawn from a blog entry, so perhaps I shouldn’t take it all that seriously. At the same time, though, you see a lot of advice to teachers that is similarly frothy. Such advice is difficult to disagree with, but it doesn’t help because it is so indefinite.
In Chapter Two, we discussed pat phrases and images meant to call to mind themes from Enlightenment thought (technical jargon, the terms “research-based” and “brain-based”) as well as phrases from Romantic thought (“unleash,” “natural,” “tailored to your child,” and so on). The “If X, then Y percent chance of Z” format ought to rid you of those peripheral cues to belief. What else will it do?
It will also trim emotional appeals, which can be very powerful indeed. Suppose that you’re a speechwriter for the president. He has emphasized the importance of mental health services for returning veterans in his most recent budget proposal, and he asks you and another speechwriter to work something about the topic into his next speech. Each of you spends a few minutes jotting down ideas and then you compare notes. Which of these two options seems more effective for the president’s speech?
1. Point out that 15 to 20 percent of soldiers returning from Iraq show signs of depression or posttraumatic stress disorder. (Note: he’ll need to signal that this disorder is serious.) The figure rises to about 30 percent for soldiers on their third or fourth tour. Rates of suicide among soldiers have not been this high in a quarter of a century.2 Funding has increased, but it is still inadequate, and many soldiers are left untreated.
2. The president had a great uncle who served in World War II and was among the soldiers who liberated a Nazi concentration camp. When he came home, he went up into the attic and was unable to leave the house for six months. At that time, there simply was no mental health support for those who had seen the horrors of war. Have the president tell that story, then segue to this idea: today we have that support, but we must be sure it is available to every one of the soldiers who selflessly put his life on the line to protect our freedom.a
Politicians don’t persuade with statistics. They persuade with emotions. In recent presidential elections, the policy wonk with facts at his fingertips—Al Gore, Michael Dukakis, Bob Dole—has lost to the candidate who can connect with the electorate on an emotional level. This was nicely captured in a political cartoon I recall from the 1984 campaign. There had been a brutal recession in the early 1980s, and the cartoon depicted the average American as a patient in a hospital bed, connected to a monitor labeled “Economy,” which showed a precipitously dropping indicator. Presidential candidates Walter Mondale and Ronald Reagan, dressed as doctors, stand by the bedside. The patient says to a shocked Mondale, “I know you’re a better doctor, but he”—pointing to a smiling, grandfatherly Reagan—“makes me feel better.”
Emotional stories may add personal texture to a problem that we understood only abstractly, or make a problem seem more urgent, but they don’t provide compelling reasons to do any particular thing. Why? Because emotional appeals don’t provide evidence that a particular solution will work. That’s equally true in education.
Persuaders in education seek to rouse different emotions, depending on their audience. For administrators and policymakers, it’s most often fear. For example, consider these quotations from a column written by New York Times columnist Thomas L. Friedman in 2009.3 (I’ve seen similar arguments dozens of times.)
What are we to do?
The Persuader refers to broad economic trends and extrapolates a dark picture to the near future. Foreign, better-educated kids are in America’s rearview mirror, gaining fast, and economic ruin will follow when they pass us. Fear makes us more open to suggestion: “That sounds terrible! Quick—tell me how to fix it!” But in fact, the message mentions a solution only briefly—invest money to take best practices from one school and put them in another—and provides no supporting evidence that this measure will work. In fact, this self-evident solution—take what works one place and implement it elsewhere—is a notorious flop among those who know the history of education policy. Successes depend on many factors that are hard to identify, let alone replicate.
When Persuaders target teachers, they more often use emotional appeals centering on hope, not fear. Most teachers you meet are optimists. They believe that all children can learn and that all children have something to offer the classroom. Teachers are also optimistic about the possibility that they can help children fulfill their potential. An unpublished survey I conducted of several hundred teachers showed that the most frequent response to the question “Why did you become a teacher?” was “I wanted to make a difference in the world.” But teachers are not optimists to the point that they are out of touch with reality. A teacher knows when there is a child with whom she is not connecting. She knows if some aspect of her teaching has become grooved, familiar, and a little stale. When they talk to teachers, Persuaders offer a Change as a way finally to reach that unreachable child or to put the passion back into the teaching.
Administrators often try to sell teachers on an idea by dangling hope before them. Administrators know that “buy-in” is vital—if teachers don’t believe that a Change is a good idea, they won’t implement it in their classroom. Thus administrators see a need not merely to persuade teachers but also to inculcate zeal for the Change. Fear does not encourage zeal. It encourages grudging compliance. Hope breeds zeal. That is why professional development sessions sometimes feel like evangelical revival meetings. But hope, like fear, is not a reason to believe that a Change will work.
When you change a Persuader’s claim to “If I do X, then there’s a Y percent chance that Z will happen,” the emotional language ought to vanish. So too should another set of peripheral cues to persuasion: those primed to make you think that the Persuader is like you. Many Web sites and professional development marketers will claim quite directly, “I know what it’s like . . .” The developer of the product will go to some pains to make clear that he’s a teacher or a parent. Consider this example, from a Web site touting an ADHD treatment: “Your friends think he just needs consistency. Your doctor wants to medicate him. Your husband doesn’t see why you can’t control him. Your mom thinks he just needs a good spanking.” By predicting the reactions of friends and family—reactions that would make a mom feel guilty or inadequate—the author signals, “I know what it’s like to be you.”
But being “like me” doesn’t really increase the chances that you’ve got a solution to the problem I face. Lots of people “know what it’s like” and haven’t found a shortcut to reading comprehension or a way to motivate frustrated kids or a method to help children with autism connect with other kids. And let’s face it: being similar to your audience is an easy credential to inflate. I once attended a professional development seminar in which the speaker told story after story of his experiences in the classroom, all of which were, in turn, funny or poignant, and all of which showed that he “got” teachers. I later learned that he had been a classroom teacher for one year, twenty years earlier. He’d been doing professional development ever since, telling, I suppose, the same set of classroom stories.
Stripping claims also removes the potentially powerful and often misleading role of analogies. When we reason by analogy, we use what we know about a familiar situation to make predictions about an unfamiliar situation. For example, suppose I said to you, “Did you know that robins love onions? It’s true. They don’t often get access to them, but when they do, they just go crazy for them! Now that you know this, let me ask you this question: Do you think that blue jays like shallots?” Admittedly, this would be a strange conversation, but you see the point.
I’m inviting you to draw an analogy. You know that blue jays, like robins, are common American birds that lay eggs, build nests, and share many other features. Shallots, like onions, are pungent, edible roots. So given that robins like onions, it’s at least plausible that blue jays like shallots, even if it’s not guaranteed (Figure 5.2).
FIGURE 5.2: Analogies encourage us to extend knowledge we already have to situations that are new to us.
It’s not guaranteed because the properties between analogous things never overlap completely. Or as Coleridge more eloquently put it, “No simile runs on all four legs.”4
Nevertheless, when analogies are suggested to us, we tend to use them. That’s why politicians so frequently offer analogies to defend their policies. For example, analogies were rampant in the United States during the buildup to the Persian Gulf War. Those who favored intervention drew an analogy between Saddam Hussein and Adolph Hitler: both were dictators of militaristic countries with regional aspirations who invaded weaker neighbors. Most Americans think that earlier action against Hitler could have saved many lives, so if Saddam is like Hitler, military action seems to make sense. But other politicians countered with a different analogy. Iraq is like Vietnam. Both were distant lands that did not directly threaten the United States. Most Americans regret the Vietnam war, so this analogy suggests not undertaking military action.
You would think that people would not be taken in. Surely we make judgments based on the merits of the case, not based on a rather shallow analogy suggested by a politician. But experimental data show otherwise. In one study, subjects read a fictional description of a foreign conflict and were asked how the United States should respond, using a scale from 1 (stay out of it) to 7 (intervene militarily).5 The description they read did not explicitly offer an analogy, but instead dropped hints that were to make subjects associate the scenario with either World War II or Vietnam: for example, the president was said to be “from New York, the same state as Franklin Delano Roosevelt,” or “from Texas, the same state as Lyndon Johnson.” Later, they were asked to judge how similar the fictional scenario was to each of these conflicts.
There were two fascinating results in this study. First, people were influenced by the hints. Those who read the story with the World War II hints favored intervention more than people who read the same story with the Vietnam hints. Second, people thought that they weren’t taken in by the analogy. Both groups said that the story they read was not very similar to World War II and not very similar to Vietnam. In short, people thought, “I see how you’re trying to influence me, but I’m too smart for you. The analogy you’re suggesting doesn’t really apply.” But their judgments of how to respond showed that they were influenced nevertheless.
Analogies are sometimes offered in discussions of education, and that’s another reason to strip claims. Consider this snippet adapted from a talk to a school board, similar to many that I’ve heard in the last five years.b The speaker was there to talk about the role of new technologies in education. Students today carry phones with more computing power than the desktop machines of ten years ago. Many students are in contact with friends via social networking sites and text messages literally during every waking hour. What do those facts imply for education? Here’s the nub of the speaker’s argument:
The speaker’s message was clearly emotional—he was quite literally suggesting that everyone in the audience was going to be as obsolete as a VHS video player, and soon. But this suggestion was by analogy. Obviously he’s right when he says that various industries have been rendered irrelevant by new technology. But it’s not obvious that every industry that delivers information is doomed. Education differs from these other industries in that a personal relationship (between teacher and student) is known to be central.6 I don’t need or want a personal relationship with the person who makes my airline reservation.
Other peripheral cues discussed in Chapter Two will also disappear when you strip a claim. Persuaders naturally want to appear authoritative. They will brag about academic degrees (if they have them). They will claim associations, however tenuous, with universities, especially prestigious ones, or they will claim to have consulted with Fortune 500 companies. They will boast about the authorship of books and articles; they will boast about speaking engagements. These are all indirect ways of saying, “Other people think I’m smart.” They are not claims about the efficacy of the Change, but rather are claims about the Persuader. I’ll have much more to say about how to evaluate the Persuader in Chapter Six, but here’s a preview: characteristics of the Persuader are a very weak indicator of scientific credibility. Stripping the claim will help you ignore them.
Psychologists have long been interested in how people make decisions. We might bet that decision making is a complex cognitive process, but we’d also bet that certain things about that process can be taken for granted—for example, that the particular way you describe the decision I have to make shouldn’t influence what I decide to do, provided that both descriptions are clear. That perfectly reasonable assumption turns out to be incorrect. People are affected by the description of the choice they are to make. That’s why, when you are choosing whether or not to implement a Change, I suggest you Flip it.
To understand the meaning and purpose of the advice to Flip it, have a look at Figure 5.3.
FIGURE 5.3: The two labels shown describe the same information, but they sure don’t sound the same.
One package advertises the ground beef within to be 85 percent lean, and the other as 15 percent fat. The meaning of the two labels is, of course, the same. If beef is 85 percent lean, then the remainder is fat, and if it’s 15 percent fat, then . . . you get the idea. The labels may be interchangeable in their literal meanings, but they sure don’t sound the same, do they? Indeed, in one study, subjects were asked to sample cooked ground beef and were told either that it was “75 percent fat free” or that it was “25 percent fat.” Subjects in the former group rated the beef as better tasting and less greasy.7
This is one example of a large family of phenomena psychologists call framing effects. In framing effects, the way a problem or question is described influences the solution or answer we provide. This is why I suggest that you Flip it. When you hear about an outcome (that’s Z in our Strip it formula) it’s worth thinking about flipping it.
How might this be relevant to education? Just as a grocer would prefer to tell you how lean beef is rather than how fat it is, a Persuader would rather tell you how many children will be reading on grade level if you adopt her Change, and would rather not talk about the converse—how many will not. Although such framing seems like an obvious ruse, experiments show that providing information about success rates rather than failure rates actually makes people rate programs as more successful.8 So when you hear that a curriculum promises that “85 percent of children will be reading on grade level,” flip it. Recognize that 15 percent won’t. This failure rate may still seem acceptable, but it’s worth having it clear in your mind.
Another somewhat obvious framing effect doesn’t concern the outcome (Z in our Strip it formula) but rather concerns what you’re asked to do (X in the Strip it formula). Sometimes a problem is presented as though it is inevitable that we must take action. After all, there’s a problem! Something must be done! But inaction is not always the worst possible choice. Years ago, a dentist told my father that his teeth were in terrible shape. He took about five minutes frightening my dad with all the details, and then another five describing an elaborate set of measures he might take to delay the inevitable, ending with, “Now if I do all that, I think you can keep your teeth for another ten years.” So Dad asked, “Okay, what if I don’t do any of that stuff. How long would my teeth last?” The dentist was taken aback that anyone would consider such a plan, but Dad persevered, and finally squeezed an answer out of him: “I don’t know. Ten years, maybe?”
There are many problems in education with a similar profile: they are real problems, but there is no proven method of dealing with them. Thumping the table and insisting “Something must be done!” misses the point. Yes, lots of kids don’t know as much civics as they ought to.9 That doesn’t mean we should plunge ahead with any civics program that we happen to lay hands on. Do we have some reason to believe that the new program will not make things worse? Is there reason to think that things might get better if we were to take no action? Or perhaps the “cure” being offered will avoid some problems but make others still worse. For example, some critics argue that children with attention deficit hyperactivity disorder (ADHD) should not be given medication. I understand the drawbacks: medications can have side effects, and the child may feel labeled by the diagnosis. Stopping the medication may solve those problems, but it incurs other costs; kids with untreated ADHD are at greater risk for dropping out of school, teen pregnancy, drug abuse, clinical depression, and personality disorders.10 So here’s another way to flip the Persuader’s claim: you should also ask yourself, “What happens if I don’t do X?”
A final framing effect is somewhat less obvious; you need to combine the two we’ve discussed. This won’t seem as complex once we make it concrete, so let’s start with an adapted version of the problem used in the classic experiment on this phenomenon.11
Imagine that an island nation of six hundred people is preparing for the outbreak of a deadly disease. There are two alternative medicines that can be used to fight the disease, but the constraints of time and money mean that the islanders can select only one. The scientific estimates of the medicines are as follows:
Which of the two programs would you favor?
Before you answer, you should know that other subjects saw the same problem, but with a different description of the medicines:
Notice that Medicines A and B have the same consequences in the two versions of the problem. “Two hundred people will be saved” is the same outcome as “Four hundred people will die.” So in this experiment, as in the hamburger situation (lean versus fat), we vary the description of the outcome (people saved versus deaths); but unlike the hamburger situation, this one requires that a choice be made (rather than just rating the appeal of the burger).
The findings were striking. When offered the first description—which emphasizes the people saved—72 percent chose Medicine A. But when offered the second description, which emphasizes deaths, just 28 percent chose Medicine A, even though the choices in each pair are comparable. Why? Most psychologists interpret this as part of a very general bias in how we think about risk and outcomes. We are risk averse for gains, and risk seeking for losses. That means that when we must make a choice between two good outcomes (where we stand to gain something), we like a sure thing. Hence, when the medicines are described in terms of lives saved, we go for the sure thing—the 100 percent chance that two hundred people will be saved. But when losses are salient, suddenly we’re ready to take risks to reduce the loss. Hence, in the second problem description, people are apt to choose Medicine B, hoping for the outcome where no one dies.
Now let’s put this into the Strip it formula. In the first flip, I asked you to think about whether there is another way to describe the outcome (Z)—that’s the lean versus fat hamburger business. In the second flip, I asked you to compare the outcome of adopting the Change (X) to the outcome when you do nothing (not X), as in my dad’s dentistry experience. In the island disease problem, we’ve combined them. Everyone was asked to consider a choice of what to do (X), but the outcome was described positively or negatively (Z).
Let’s put this into an education context. Suppose that you’re a school principal, and the central office in your district closely monitors the percentage of kids who read at grade level, as defined by a state-mandated test. With your current reading program, 34 percent of kids in your school are reading at grade level, and 66 percent are not. If you adopt a new reading program, there is some chance that it will work well, and things will improve. But there is also some chance that things will get worse—teachers will be unfamiliar with the new program and so won’t implement if effectively, or the program just may not be as good as what you’re doing now. We can frame this choice in terms of losses:
Or we can frame the choice in terms of gains:
Naturally, I’ve fabricated the figures in these choices, but I’m sure you get the point. When we think about adopting a Change, we understand that there’s some chance that it will help, but there is also some chance that it will not work or even make things worse. We can frame these possible outcomes either as gains or losses. When things are described as losses, we are more likely to take a risk. So when a Persuader emphasizes again and again that things are really bad, what is she really saying? She’s saying that the current situation means a certain loss! The Persuader is egging you on to take a risk. When the island problem was described in terms of losses (deaths), people were more ready to go for a risky solution to try to minimize the losses. If the Persuader instead emphasized gains, you would be more likely to stick with what you’re doing—where your gains are certain—rather than taking a risk to try to increase your gains.
Whether or not the risk is worth it is, of course, a matter of the odds of the gains and losses, as well as how good the gains seem to you and how bad the losses seem. What I’m emphasizing here is that you should look at these outcomes from all possible angles, because your willingness to try something risky is influenced by whether you think of yourself as trying to get something good or trying to avoid something bad.
So to summarize, Flip it means that you
I’ve emphasized that this first step—Strip it and Flip it—is meant to be devoid of evaluation. You are simply to gain clarity on the claim. We’re not going to go deeply into the process of assessing evidence until Chapter Seven, but some claims are, on their face, unworthy of attention: some are boring, some are unacceptably vague, and some are unlikely to affect students. Let’s look at each of these.
One possibility is that the claim, once stripped of fluff, is revealed as something humdrum because it is already familiar. This phenomenon is especially prevalent in so-called brain-based education. As described in Chapter Two, neuroscientific terms seem so impressive, so unimpeachably scientific that it may not occur to you that the findings, though perfectly true, don’t really change anything. Table 5.1 shows some neuroscientific findings that I have seen emphasized in books and blogs.
TABLE 5.1: Some common neuroscientific claims, stripped.
| Neuroscientific Finding | Stripped |
| Dopamine, a neurotransmitter associated both with learning and with pleasure, is also released during video gaming. Video games may be an ideal vehicle through which to deliver educational content. | Kids like games, so if we could make learning more like games, kids would like learning. |
| Although the brain weighs just three pounds, it commandeers about 20 percent of the body’s glucose—the sugar in the bloodstream that provides energy. When glucose in the brain is depleted, neural firing is compromised, especially in the hippocampus, a structure vital to the formation of new memories. | A hungry child won’t learn very well. |
| The prefrontal cortex of the brain is associated with the highest levels of decision making and rational thought. It is also the last part of the brain to be myelinated—that is, to be coated in the insulation essential to effective neural functioning. The prefrontal cortex may not be fully myelinated until twenty years of age. | Sometimes teenagers do impulsive things. |
| There is massive brain plasticity during the early years of life. Brain plasticity is the process by which the physical structure of the brain changes, based on experience. New networks are formed, and unused networks are “pruned” away—that is, are lost. | Little kids learn a lot. |
Once stripped, some claims are revealed as, well, pretty mundane. But others, though far from mundane, are very hard to size up because they don’t yield to your best efforts to put the claims into the format “If I do X, then there is a Y percent chance that Z will happen.” In other words, you can’t quite figure out either what you’re supposed to do (X) or what is supposed to happen after you do it (Z). That problem ought to strike you as quite serious. You are embarking on this educational Change because you think it’s going to do some good. If you don’t have it clear in your mind what Z is supposed to be, then you can’t know whether or not the Change is working. And if you don’t have X clear in your mind, that means you’re not sure whether you’re doing the right thing to make Z happen.
When I say that X must be clear, I really mean more than that. I don’t just mean that the Persuader should be specific about what you (or the student) is supposed to do; I mean that you should feel confident that you can predict the impact of the Change on the student’s mind. Think about it this way. What prompts students to learn new information, or a new method of analysis, or critical thought? It’s the student’s mental activity that makes these things happen. Listening to a teacher or reading a book or creating an artifact—students learn from these activities only insofar as the listening, reading, or creating gives rise to the right mental activity. I might provide what I think is a brilliantly clear explanation of the relationship of medieval vassals and lords, but if you’re not really paying attention, it won’t do any good. Or consider that sometimes you are paying attention, but the ideas being communicated to you just don’t click. When we say “it’s not clicking,” we mean that the ideas are bumping around in our thoughts (because we are paying attention), but they are not coming together into a coherent whole. Understanding is a mental event. The act of teaching is an effort to precipitate such mental events by shaping the thoughts of students. Students’ mental events can be thought of as the last link in a chain (Figure 5.4).
FIGURE 5.4: The chain of influence in education. This description is not meant to be comprehensive. Many factors are omitted, and there are more mutual influences.
At the far left of Figure 5.4 are the thought processes that will drive learning, understanding, enthusiasm, and so forth. The teacher tries to create an environment that will move the student’s thoughts in particular directions. The school administration tries to support the teacher’s efforts, or the administration tries to get the teacher to teach in ways the administration thinks is most effective. The district does the same, influencing school administrators. The state legislature writes laws in an effort to influence how districts and schools are administered.
The point here is to emphasize that (1) Changes in the educational system are irrelevant if they don’t ultimately lead to changes in student thought; and (2) the further the Change from the student’s mind, the lower the likelihood that it will ultimately change student learning in the way people hope. For example, suppose a state legislature writes a law dictating that contracts between teachers’ unions and school districts cannot forbid the use of student test scores in the evaluation of teachers.c The law doesn’t say that districts must use test score data—it just says that such use can’t be forbidden. The hope, of course, is that some districts will use test score data. An additional hope is that principals, who at most schools have a prominent voice in the evaluation of teachers, will emphasize those test scores rather than ignore them. Further, the hope is that teachers, aware that their principals will base teacher evaluations on test scores, will try to ensure that their students will perform better on state-mandated tests. Finally, the hope is that the Changes teachers make will, in fact, change student mental processes so that they will learn more and score better on the tests.d If anyone in this process does not or cannot act as expected, the law will not have the intended effect. As the chain of influence necessary for the Change gets longer and longer, there is a greater and greater chance that it won’t work.
Take, for example, the Change of placing an interactive whiteboard in a classroom.e It would seem that this tool could be quite useful in a classroom. For starters, the teacher can capitalize on all of the software on the Web. The United Kingdom invested heavily in interactive whiteboards, and today virtually every UK school has at least one. But the impact on student achievement has been minimal. It turns out that the presence of an interactive whiteboard in the classroom does not necessarily change teaching for the better, or even change teaching at all.12 Teachers need not only the whiteboard but also substantive training in its use, expert advice about how to exploit it in lesson plans, and time to gain expertise and confidence.
It’s not just technological Changes that are underspecified. Many Changes that urge project learning or group learning have this characteristic. Just as dropping an interactive whiteboard into a classroom is not enough to ensure that students will learn, assigning group work is not enough to ensure that students will learn how to work well in groups. These pedagogical approaches call for much more independence on the part of students, and therefore they depend on the teacher’s having strong relationships with the students and a good understanding of the existing relationships between students. The teacher uses this knowledge in hundreds of moment-to-moment decisions that guide the groups in the work without micromanaging them. Thus Changes that suggest lots of group work in the classroom are almost always underspecified. The methods are terrific when they work well—in fact, I think that for some types of learning they are probably the ideal—but they are very difficult to implement well, and I seldom see a Persuader acknowledge this difficulty.
There is an inevitable tension here. On the one hand, teachers and administrators are understandably leery of Changes that amount to classroom micromanagement. Classrooms are dynamic places, and teachers need flexibility to change what they are doing based on moment-to-moment evaluations of students’ understanding and motivation. On the other hand, the greater the flexibility, the less one can say with any confidence exactly what the Change amounts to. There is a happy medium. A Change might include considerable detail as to how it is to be implemented without prescribing what must be done moment to moment.
The clarity of the outcome is just as important as the clarity of what you are supposed to change. Readers of this book are interested in evidence and will surely want to know whether or not a Change is actually working. For example, suppose that my first-grader’s teacher has told me that my son is struggling with reading, and I notice that he shows no interest in reading at home. I hear about a technique called Language Experience13 that is supposed to help struggling readers, and I decide to give it a try. Language Experience is quite specific about what you’re supposed to do:
The method is clear enough. The outcome, less so. The technique is supposed to help make reluctant readers more interested in reading. Okay, but how are you to know that’s happening?
At some point you need to evaluate whether or not the Change is working so that you can either continue it or give up and try something else. To know whether it’s working, you’ll need three pieces of information: (1) how to recognize the positive change when it happens; (2) how large the change is supposed to be; and (3) when to expect it.
Knowing what a Change is supposed to do is not quite the same as being able to evaluate whether or not it’s actually happening. When we discussed good science (Chapter Three), we emphasized the need for things to be measurable. The same principle applies here. If a Persuader promises that a Change will make kids like reading more, how will I know that they do? I could just ask them: “Do you like reading more than you did six weeks ago?” But then again, maybe children’s memory for that sort of thing is not that accurate. Then too, if the child says, “Yes, I like reading more,” but then seems just as miserable during reading time at school, should I be persuaded by what she says, or by how she seems to act? If I am to evaluate whether a Change is working, I need something concrete, and something that is well matched to what I was hoping the Change would do. For example, perhaps I was prompted to look for a reading program because my child complained about reading in school and because he seldom read books at home. So I could see whether the Change prompts less complaining and more reading.
I also need some idea of what constitutes “success.” Suppose that in the week before he starts this new reading program, my son doesn’t pick up a book once. If, three weeks into the program, he is looking at books once each week, am I satisfied? Or does that Change seem too small? In addition, I need to know when to expect that the good outcome will have happened. For example, you’d think it pretty odd if I told you that I had been using a reading program for two years with no sign of its helping, but that I was still hopeful that eventually it would do some good. Okay, so two years without results is too long. What’s more reasonable? Two weeks? Two months?
It’s important to define the signs of success before you embark on the Change. Once you’re committed, your judgment of how it’s working is all too likely to be affected by cognitive dissonance. You’ll recall that cognitive dissonance refers to discomfort that is a consequence of holding two conflicting beliefs simultaneously. Dissonance motivates us to change one of the beliefs. (Remember the boring pegboard task that subjects ended up rating as interesting?)
Cognitive dissonance may make it hard to evaluate how an educational Change is working. Once you have been embarked on a Change for a while, you’ve invested your time and that of the students or your child, and you may have a financial investment. Thus, if the Change isn’t really working that well, you will hold two incompatible thoughts in mind: (1) I invested heavily in this program, and (2) this program brings no benefits. It’s hard to rewrite history and pretend that you haven’t invested in the program, so you are likely to seek out reasons to persuade yourself that the program is working, even if you’re grasping at straws.
The best way to protect yourself from this profitless self-delusion is to write down your expectations before you start the program: how big a change you’re expecting, when you expect to see it, and how you’ll know the change is happening. Writing down these expectations makes it difficult for you to persuade yourself that something is working when it’s not, because you have already defined for yourself what it means for the Change to be “working.”
In the Broadway musical (and later, Hollywood movie) The Music Man, con man Harold Hill comes to a small town in Iowa and persuades the townspeople that he is organizing a boy’s marching band. When pressed to describe how he plans to teach the boys to play, Hill describes his “think system.” If a boy thinks the melody and hums the melody, he will, when presented with an instrument (even for the first time), be able to play the melody!
Some claims about Changes are equally extravagant, but are not quite as transparently false. From a cognitive perspective, if a Persuader makes either of two promises, they are very unlikely to be kept: (1) that a Change will help with all school subjects, or (2) that a change will help all kids with a disability. Let’s consider each in turn.
Suppose that instead of being tutored in academic subjects, students performed a set of exercises tapping basic mental processes that underlie all cognition. You don’t just tutor the student in history; instead, you make memory work better, or you improve critical thinking. Many of the “brain games” software packages and cognitive training centers make such claims.
Unsurprisingly, a notion this plausible has been around for a while. In the nineteenth century (and before), it was the rationale for students to study Latin. The thinking was that, although useless for communication, Latin is a logical language requiring logical thought and that this logical habit of mind will transfer to other mental work. In addition, learning Latin is difficult. Kids will learn mental discipline, which will also translate to other intellectual pursuits. As more than one wag has put it, “It doesn’t really matter what kids study, as long as they don’t like it.”
These ideas were tested by Edward Thorndike, an important learning theorist at the turn of the twentieth century, and often considered a founder of educational psychology. Thorndike reported that training in one task led to improvement in that task, but the skill seldom showed much transfer to other tasks. It seemed not to be the case that studying Latin makes you all-around smarter or more mentally disciplined or better at learning new things.14 This finding has more or less held up over the last century.f
A more modern take on this old idea is not to use Latin or some other subject matter to improve a basic mental process but rather to train the basic mental process itself with content-free exercises. Working memory has been the favorite target for this strategy. Working memory is the mental “space” in which you manipulate and combine ideas. You can test the capacity of working memory by asking someone to keep something briefly in mind—a list of letters, say—and then to recount what you asked her to keep in mind. People with larger working memory—those who can keep more stuff in mind simultaneously—tend to do better on standard tests of reasoning.15 It’s long been reported, however, that training working memory, like studying Latin, doesn’t lead to good transfer. Someone who practices keeping letters in mind will get better at keeping letters in mind, but if then asked to remember numbers, shows little or no transfer.16
There has been a new ray of hope in this area of research in just the last few years. Some laboratories are reporting that people trained in certain working memory tasks not only improve on that task but also do show transfer to other working memory tasks; however, they don’t show any advantage on reasoning tasks.17 So at this point, there’s reason to be suspicious of any Change that, once stripped, promises an across-the-board improvement in cognition.
The problem is not just that you can’t train basic cognitive processes like working memory. The problem is that when you practice a cognitive skill—critical thinking, say, or problem solving—the newly acquired skill tends to cling to the domain in which you practiced it. That is, learning how to think critically about science doesn’t give you much of an edge in thinking critically about mathematics.
There are two reasons that critical thinking sticks to subject matter: sometimes you need subject knowledge to recognize what the problem is in the first place, and sometimes you need subject knowledge to know how to use a critical thinking skill.18 Consider this problem:
I am taking a trip on Route 66, all the way from Chicago to Los Angeles. It’s a total of 2,451 miles, and I want to document the trip. I have enough memory in my digital camera to take 150 pictures. If I want them evenly spaced, I should take one picture every how many miles?
Although this problem seems straightforward, many people get it wrong, and they typically make the same mistake. They divide 2,451 by 150 and come up with 16.34 miles. But if I take my first picture in Chicago (mile 0), I’ll snap my 150th picture 16.34 miles short of the end of Route 66 at the Santa Monica Pier. Think of it this way. Suppose I said I wanted to take only two pictures. Now how many miles should separate my shots? If I divide 2,451 by 2, I get 1,225.5. I take my first picture in Chicago, and my second only halfway along the trip. Thus the formula to solve the problem is not (number of miles) ÷ (number of pictures). It’s (number of miles) ÷ (number of pictures −1).
You can see why many people get this wrong. Someone who is very likely to get it right is a person who builds fences or is familiar with iterative loops in computer programming. Buying the right number of fence posts is analogous to my picture-snapping problem, and computer programmers learn about this off-by-one error because many data structures have indices that start at zero rather than one. The fact that these folks are likely to get the problem right shows that sometimes solving a problem is not a matter of critical thinking. It’s a matter of recognizing, “Oh, this is that sort of problem.” This recognition process (and its importance) is familiar to teachers of mathematics. Students may encounter a formula for solving a particular type of problem and can apply the formula with ease. But when later confronted with a word problem, they have a hard time figuring out which of the many formulas they have learned is appropriate. This is not a matter of critical thinking. It’s a matter of having worked enough of these (and other) problems so that the deep structure of the problem is apparent to you, just as a builder of fences would recognize that her knowledge applies to the Route 66 problem.
At other times, you might know exactly which critical thinking strategy you are supposed to use, but lack the knowledge required to apply it. For example, students who have been exposed to the scientific method know that experiments typically have an experimental group and a control group. These two groups are supposed to differ in some important way (for example, the experimental group gets a new math curriculum, and the control group gets the old one); otherwise they are supposed to be comparable. But ensuring that the groups are comparable frequently requires knowing something about the thing you’re testing. For example, if I’m testing the efficacy of a math curriculum, shouldn’t I check to be sure that kids in the experimental and control groups have roughly the same knowledge of math before the experiment begins? Yes, that sounds like a good idea. Are there other things I ought to check because they might affect kids’ learning of math? Here’s a list of some factors. Which ought to be measured before the experiment begins, to ensure that the two groups are equivalent?
Factors That Might (or Might Not) Be Important to Equate When Comparing Math Curricula
Here’s the point. If we plan to test whether the new curriculum is better than the old, we need to make sure that the kids in the two groups are the same. But kids have an endless number of attributes. We could easily add to the list of factors. It’s impractical to test each kid on every attribute. What we really need to do is test kids on the attributes that matter to learning math. Okay, which attributes matter to learning math? Well, knowing the answer to that question is a matter of background knowledge. So here’s a critical thinking skill (evaluating whether or not a study was well done) for which we know what to do (make sure that there is a control group), yet we can’t do it in the absence of the relevant background knowledge. So when I see a Change promise to improve a skill (such as “critical thinking”) and it makes no mention of the need for knowledge to go with it, I’m suspicious.
There’s a second type of across-the-board claim that ought to make you leery. This one does not cut across the cognitive abilities of one child, but rather concerns a single ability in many children. I am suspicious of Changes that promise to remediate a problem in any child. Why? Because each of the outcomes we care about for schooling is complex. Lots of cognitive and noncognitive processes contribute. Put another way, if a child is having problems with reading, there are many possible reasons for that. Thus a Change might help with reading difficulties that are due to a problem in processing sounds, but that’s not going to work for a child who has a problem with visual processing. Hence, when a Persuader claims that a Change will help any reading difficulty, the needle on my nonsense detector flutters close to the red zone.
• • •
In this chapter, we’ve covered the first of three steps in evaluating claims about educational Changes. Table 5.2 summarizes all of the subcomponents of Step One: Strip it and Flip it.
TABLE 5.2: Summary of the steps suggested in this chapter.
| Suggested Action | Why You’re Doing This |
| Strip to the form “If I do X, then there is a Y percent chance that Z will happen.” | To get rid of emotional appeals, peripheral cues, and proffered analogies that may influence your belief. The scientific method is supposed to be evidence based and uninfluenced by these factors. |
| Consider whether the outcome (Z) has an inverse; if so, restate the stripped version of the claim using the inverse. | To be sure that you appreciate all the consequences of the action—for example, that a “85 percent pass rate” implies a “15 percent failure rate.” We are subject to framing effects; we think something is better if the positive aspects are emphasized rather than the negative. |
| Consider the outcome if you fail to take action X. | To ensure that the promised outcome if you do X seems much better than if you don’t do X. When there is a problem, it’s tempting to lunge toward any action because it makes you feel that you are taking some action rather than standing idle. |
| Consider the outcome if you fail to take action, this time using the inverse of Z as the outcome. | To ensure that doing something versus doing nothing looks just as appealing when you think about good outcomes as when you think about bad outcomes. People are generally less willing to take risks to increase their gains—they would rather have a sure thing. But they don’t want a sure thing for losses—they will take a risk to try to minimize them. |
| Evaluate whether the stripped promise is something you already know. | To be sure that what’s being sold to you is something you can’t do yourself. Technical talk—especially neuroscientific talk—can make old ideas seem cutting-edge. |
| Evaluate whether the Change is clear; “clear” means that you feel confident that you know how the Change will affect students’ minds. | To ensure that the Change is implemented as intended. Changes that sound good can go awry if they are not implemented in the classroom as intended or if students don’t do what you’re hoping they will do. |
| Evaluate whether the outcome (Z) is clear; “clear” means that there is some reasonably objective measure of whatever outcome you expect, how big the change in the outcome will be, and when it will happen. | To be sure you will be able to tell whether or not the promised outcome is happening. |
| Check the outcome against this list of frequently claimed but unlikely-to-work promises. | To be sure that claims are not made that are unfeasible, from a cognitive perspective—for example: An improvement in all cognitive processes An improvement in a specific cognitive process (for example, critical thinking) irrespective of material An improvement for all students who struggle with a complex skill such as reading |
I urge you not simply to think about the actions in Table 5.2 but to write down your thoughts about them when you are considering a Change. Forcing yourself to write things down will make you take more time with each action, and articulating your thoughts will increase their precision. It’s well worth the time now, given that a Change usually represents a significant investment of time, money, and energy, not to mention the time and energy of your kids.
Suppose you’ve run through all the evaluations suggested in Table 5.2, and you think the Change is worth pursuing. Now suppose that the Change is promoted by a well-known figure in education; a prominent professor; or a former secretary of education who, in your opinion, has always seemed to have a lot of horse sense. You’ve decided that the Change sounds pretty good; if an authority figure says so too, how persuasive should you find that? That’s the subject of Chapter Six.
a The president actually used this story as part of a Memorial Day speech when on the campaign trail in 2008; he was taken to task for erring in some details when he first told the story. A video of the speech can be seen here: http://www.youtube.com/watch?v=SV1sxq8mqvA.
b This example, like many I’ll use, was inspired by a real talk, but I’ve changed it enough that it’s not clearly attributable to the original speaker.
c In fact, the Race to the Top initiative of 2009 was a federal law, which said, among other things, that states interested in applying for a pot of federal education grant money could not have a law on the books forbidding the use of student test data in teacher evaluations.
d Of course, the motivation for the law was probably not to get teachers to teach better but to fire teachers who were already not teaching well. The chain of action required in this case is different.
e An interactive whiteboard is used as a screen on which one can project an image from a computer. The screen is touch sensitive, so the teacher (or student) can interact with the computer by touching the screen.
f Thorndike, E. L., & Woodworth, R. S. (1901). The influence of improvement in one mental function upon the efficiency of other functions. Psychological Review, 8, 247–261. Note that there must be some process by which learning makes you better at solving unfamiliar problems—after all, adults are better at this than kids are, so their experience must be transferring to the new problem in this situation. But it’s far from obvious how that works.
Notes
1. From Blake, W. (1904). Jerusalem (E.R.D. MaClagan & A.G.B. Russell, Eds.). London: Bullen. Available online at http://books.google.com/books?id=krM8AAAAYAAJ&printsec=frontcover&dq=william+blakepercent27s+jerusalem&hl=en&ei=GKTBTfzxFeX50gHP74m3Cg&sa=X&oi=book_result&ct=result&resnum=1&ved=0CDsQ6AEwAA#v=onepage&q&f=false.
2. Estimates vary, but the figures I’ve offered are in the ballpark. For example: U.S. soldiers experience increased rates of depression, PTSD on third, fourth tours in Iraq, study finds. (2008, March 10). Medical News Today. http://www.medicalnewstoday.com/articles/99981.php.
3. Friedman, T. (2009, April 22). Swimming without a suit. New York Times. http://www.nytimes.com/2009/04/22/opinion/22friedman.html.
4. Coleridge, S. T. (1830). On the constitution of the church and state. London: Hurst, Chance & Co. Available online at http://books.google.com/books?id=_FTM_6q6G3gC&pg=PP15&dq=#v=onepage&q&f=false.
5. Gilovich, T. (1981). Seeing the past in the present: The effect of associations to familiar events on judgments and decisions. Journal of Personality and Social Psychology, 40, 797–808.
6. For example, Hamre, B. K., & Pianta, R. C. (2001). Early teacher-child relationships and the trajectory of children’s school outcomes through eighth grade. Child Development, 72, 625–638.
7. Levin, I. P., & Gaeth, G. J. (1988). Framing of attribute information before and after consuming the product. Journal of Consumer Research, 15, 374–378.
8. For example, Davis, M. A., & Bobko, P. (1986). Contextual effects on escalation processes in public sector decision making. Organizational Behavior and Human Decision Processes, 37, 121–138; Dunegan, K. J. (1995). Image theory: Testing the role of image compatibility in progress decisions. Organizational Behavior and Human Decision Processes, 62, 79–86.
9. National Center for Education Statistics, U.S. Department of Education. (2010). The Nation’s Report Card: Civics 2010. Available online at http://nces.ed.gov/nationsreportcard/pdf/main2010/2011466.pdf.
10. Barkley, R. A. (1998). Attention-deficit hyperactivity disorder (2nd ed.). New York: Guilford Press.
11. Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453–458.
12. Gillen, J., Staarman, J. K., Littleton, K., Mercer, N., & Twiner, A. (2007). A “learning revolution”? Investigating pedagogic practice around interactive whiteboards in British primary classrooms. Learning, Media, and Technology, 32, 243–256.
13. Elements of this technique go back quite far. One of the more influential presentations is Allen, R. V., & Allen, C. (1969). Language experiences in early childhood. Chicago: Encyclopedia Britannica Educational Corporation.
14. McDaniel, M. A. (2007). Transfer: Rediscovering a central concept. In H. L. Roediger, Y. Dudai, & S. M. Fitzpatrick (Eds.), Science of memory: Concepts (pp. 267–270). Oxford: Oxford University Press.
15. For example, Ackerman, P. L., Beier, M. E., & Boyle, M. O. (2005). Working memory and intelligence: The same or different constructs? Psychological Bulletin, 131, 30–60.
16. For example, Ericsson, K. A., Chase, W. G., & Faloon, S. (1980). Acquisition of a memory skill. Science, 208, 1181–1182.
17. Klingberg, T. (2010). Training and plasticity of working memory. Trends in Cognitive Sciences, 14, 317–324.
18. For more on this, see Willingham, D. T. (2007, Summer). Critical thinking: Why is it so hard to teach? American Educator, pp. 8–19.