1. Dahlia Bazzaz, “Dispatches from Parents: Northshore School District’s First Online-Only Day to Prevent Coronavirus Spread,” Seattle Times, March 9, 2020, https://
1. Vanessa Lu and Kristin Rushowy, “Rainbow Loom Bracelet Maker Hot Toy Trend,” Toronto Star, October 4, 2013.
2. Ashley Rivera, “How to Make a Rainbow Loom Starburst Bracelet,” YouTube video, August 1, 2013, https://
3. Cheong Choon Ng, “Experience: I Invented the Loom Band,” Guardian, September 26, 2014, https://
4. Henry Jenkins, Confronting the Challenges of Participatory Culture: Media Education for the Twenty-First Century (Cambridge, MA: MIT Press, 2009).
5. Mizuko Ito, Crystle Martin, Rachel Cody Pfister, Matthew H. Rafalow, Katie Salen, and Amanda Wortman, Affinity Online: How Connection and Shared Interest Fuel Learning (New York: New York University Press, 2018).
6. Clayton M. Christensen, Michael B. Horn, and Curtis W. Johnson, Disrupting Class: How Disruptive Innovation Will Change the Way the World Learns (New York: McGraw-Hill, 2008), 101.
7. Salman Kahn, “Let’s Use Video to Reinvent Education,” TED talk, March 1, 2011, https://
8. Michael Noer, “One Man, One Computer, 10 Million Students: How Khan Academy Is Reinventing Education,” Forbes, November 19, 2012, https://
9. Salman Khan, The One World Schoolhouse: Education Reimagined (Boston: Grand Central, 2012).
10. Laura Pappano, “The Year of the MOOC,” New York Times, November 2, 2012.
11. David Carr, “Udacity CEO Says MOOC ‘Magic Formula’ Emerging,” InformationWeek, August 19, 2013, https://
12. On Thrun’s reversal, see Max Chafkin, “Udacity’s Sebastian Thrun, Godfather of Free Online Education, Changes Course,” Fast Company, November 14, 2013, https://
13. Emily Ann Brown, “Sal Khan Envisions a Future of Active, Mastery-Based Learning,” District Administration, January 31, 3019, https://
14. Kenneth R. Koedinger, John R. Anderson, William H. Hadley, and Mary A. Mark, “Intelligent Tutoring Goes to School in the Big City,” International Journal of Artificial Intelligence in Education 8 (1997): 30–43.
15. Philip Wesley Jackson, Life in Classrooms (New York: Teachers College Press, 1990), 166–167; Larry Cuban, The Flight of a Butterfly or the Path of a Bullet? Using Technology to Transform Teaching and Learning (Cambridge, MA: Harvard Education Press, 2018).
16. Justin Reich and Mizuko Ito, “Three Myths about Education Technology and the Points of Light Beyond,” Connected Learning Alliance Blog, October 30, 2017, https://
17. Morgan G. Ames, The Charisma Machine: The Life, Death, and Legacy of One Laptop per Child (Cambridge, MA: Harvard University Press, 1995).
18. David Tyack and Larry Cuban, Tinkering toward Utopia: A Century of Public School Reform (Cambridge, MA: MIT Press, 2019).
19. For an example of my teaching, see Justin Reich, “Conflict and Identity: Using Contemporary Questions to Inspire the Study of the Past,” World History Connected, last modified February 2007, https://
20. UNESCO Global Education Monitoring Report, “Six Ways to Ensure Higher Education Leaves No One Behind,” Policy Paper 30, April, 2017, https://
21. Victoria Lee and Constance Lindsey, “Access to Calculus Could Hinder Low-Income and Black Students,” Urban Wire (blog), March 6, 2018, https://
1. Audrey Watters, “MOOCs: An Introduction,” Modernlearners.com, n.d., https://
2. “Press Conference: MIT, Harvard Announce edX,” YouTube video, May 3, 2012, https://
3. 1999–2000 online learning numbers are from Anna Sikora and C. Dennis Carroll, A Profile of Participation in Distance Education: 1999–2000 (Washington, DC: National Center for Educational Statistics, 2002), https://
4. “Clayton Christensen Interview with Mark Suster at Startup Grind 2013,” YouTube video, 06:40, “Startup Grind,” February, 20, 2013, https://
5. Daphne Koller, “What We’re Learning from Online Education,” filmed June 2012 at TEDGlobal 2012, Edinburgh, Scotland, video, https://www.ted.com/talks/daphne_koller_what_we_re_learning_from_online_education?language=en.
6. Koller, “What We’re Learning.”
7. The Quote Investigator website has an excellent investigation into this framing, often misattributed to Yeats; see “The Mind Is Not a Vessel That Needs Filling, but Wood That Needs Igniting,” Quote Investigator, last modified March 28, 2013, https://
8. John Dewey, “My Pedagogic Creed,” School Journal 54, no. 3 (1897): 77–80, http://
9. One early entry point into Thorndike’s work is Edward Thorndike, The Psychology of Learning (New York: Teachers College, 1913).
10. For research on MOOC instructional quality, see Anoush Margaryan, Manuela Bianco, and Allison Littlejohn, “Instructional Quality of Massive Open Online Courses (MOOCs),” Computers & Education 80 (2015): 77–83. The literature of MOOC criticism is extensive and thoughtful. An early response was from the faculty of the Philosophy Department of San Jose State University, when the university proposed requiring students to use Harvard’s JusticeX for an introductory course; “An Open Letter to Professor Michael Sandel from the Philosophy Department at San Jose State U,” Chronicle of Higher Education, May 2, 2013, https://
11. James Becker, Toward Automated Learning. A Professional Paper (Pittsburgh: Research for Better Schools, 1968); William Cooley and Robert Glaser, “The Computer and Individualized Instruction,” Science 166, no. 3905 (1969): 574–582; James Becker and Robert Scanlon, Applying Computers and Educational Technology to Individually Prescribed Instruction (Pittsburgh: Research for Better Schools, 1970).
12. On feature convergence, see Carl Straumsheim, “Where Does the LMS Go from Here?” Inside Higher Ed, September 23, 2014, https://
13. For the relationship between MOOC platform and pedagogy, see Shreeharsh Kelkar, “Engineering a Platform: The Construction of Interfaces, Users, Organizational Roles, and the Division of Labor,” New Media & Society 20, no. 7 (2018): 2629–2646.
14. For an early description of the storefront metaphor, see Michael Feldstein, “Is Coursera Facebook, Amazon, or Pets.com?,” e-Literate, November 14, 2012, https://
15. For a recent review, see Draylson M. Souza, Katia R. Felizardo, and Ellen F. Barbosa, “A Systematic Literature Review of Assessment Tools for Programming Assignments,” presentation, International Conference on Software Engineering Education and Training, Dallas, TX, April 2016, IEEE, https://
16. An early iteration of websim, the circuits simulator used in the first MITx course, is online at http://
17. Test developers sometimes argue that multiple-choice questions can test critical thinking or reasoning. Stanford researcher Mark Smith contests this claim, demonstrating that high-performing students on history assessments do not use complex reasoning to take multiple-choice tests, but rather use “three construct-irrelevant processes: factual recall / recognition, reading comprehension, and test-taking strategies.” Mark D. Smith, “Cognitive Validity: Can Multiple-Choice Items Tap Historical Thinking Processes?” American Educational Research Journal 54, no. 6 (2017): 1256–1287.
18. On evaluating writing with peer grading and automated essay scoring, see Stephen P. Balfour, “Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review™,” Research & Practice in Assessment 8 (2013): 40–48. For peer and self-assessment in MOOCs, see Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R. Klemmer, “Peer and Self Assessment in Massive Online Classes,” ACM Transactions on Computer-Human Interaction (TOCHI) 20, no. 6 (2013): 1–31.
19. The literature and commentary on MOOCs’ low completion rates are extensive. One interesting voice making this point was Larry Bacow (along with Michael McPherson), who went on to become Harvard University president; Michael S. McPherson and Lawrence S. Bacow, “Online Higher Education: Beyond the Hype Cycle,” Journal of Economic Perspectives 29, no. 4 (2015): 135–154. The most comprehensive early study of MOOCs and completion rates is Katy Jordan, “Massive Open Online Course Completion Rates Revisited: Assessment, Length and Attrition,” International Review of Research in Open and Distributed Learning 16, no. 3 (2015): 341–358. Early reports on completion rates include Gayle Christensen, Andrew Steinmetz, Brandon Alcorn, Amy Bennett, Deirdre Woods, and Ezekiel Emanuel, “The MOOC Phenomenon: Who Takes Massive Open Online Courses and Why?” SSRN (2013), https://
20. Zach Lam, Kathy Mirzae, Andreas Paepcke, Krystal Smith, and Mitchell Stevens, “Doing Things with MOOCs: Utilization Strategies of Stanford’s California MOOC Learners,” MIT Office of Digital Learning x-Talks, October 16, 2018, https://
21. “Open edX Conference 2018 with Zvi Galil Keynote: Georgia Tech’s Online MOOC-based Master Program,” YouTube video, “Open edX,” June 13, 2018, https://
22. Joshua Littenberg-Tobias and Justin Reich, Evaluating Access, Quality, and Equity in Online Learning: A Case Study of a MOOC-Based Blended Professional Degree Program, pre-print retrieved from doi:10.31235/osf.io/ 8nbsz.
23. Littenberg-Tobias and Reich, “Evaluating Access.”
24. NanoDegrees have much in common with earlier forms of non-degree technical certificates, such as the Microsoft Certified Technician programs. On the 1990s history of Microsoft and related information technology non-degree certifications, see Clifford Adelman, “A Parallel Universe: Certification in the Information Technology Guild,” Change: The Magazine of Higher Learning 32, no. 3 (2000): 20–29.
25. For more on for-profit higher education and the role of credentials in higher education, see Tressie McMillan Cottom, Lower Ed: The Troubling Rise of For-Profit Colleges in the New Economy (New York: New Press, 2017).
26. Phil Hill, “Coursera CEO Interview: Betting on OPM Market and Shift to Low-Cost Masters Degrees,” E-literate, December 6, 2018, https://
27. Laura Pappano, “The Boy Genius of Ulan Bator,” New York Times, September 13, 2013, https://
28. Justin Reich and Ruipérez-Valiente, “MOOC Pivot”; John D. Hansen and Justin Reich, “Democratizing Education? Examining Access and Usage Patterns in Massive Open Online Courses,” Science 350, no. 6265 (2015): 1245–1248; René F. Kizilcec, Andrew J. Saltarelli, Justin Reich, and Geoffrey L. Cohen, “Closing Global Achievement Gaps in MOOCs,” Science 355, no. 6322 (2017): 251–252; Ezekiel J. Emanuel, “Online Education: MOOCs Taken by Educated Few,” Nature 503, no. 7476 (2013): 342. For more recent data, see Isaac Chuang and Andrew Ho, “HarvardX and MITx: Four Years of Open Online Courses—Fall 2012–Summer 2016,” December 23, 2016, https://
29. The initial report on the SJSU Plus experiment was Ellaine D. Collins, “SJSU Plus Augmented Online Learning Environment Pilot Project Report,” Research and Planning Group for California Community Colleges 38 (2013): 45. A subsequent analysis put results in a somewhat more favorable light, especially for courses run twice with refinements between iterations. Erin L. Woodhead, Preston Tim Brown, Susan Snycerski, Sean Laraway, Nicholas G. Bathurst, Greg Feist, and Ronald F. Rogers, “An Examination of the Outcomes of a Brief and Innovative Partnership: SJSU and Udacity,” Innovative Higher Education 42, no. 5–6 (2017): 463–476, DOI: 10.1007/s10755-017-9400-4; Lindsay McKenzie, “Arizona State Moves on from Global Freshman Academy,” Inside Higher Ed, September 17, 2019, https://
30. For self-regulated learning and MOOCs, see Allison Littlejohn, Nina Hood, Colin Milligan, and Paige Mustain, “Learning in MOOCs: Motivations and Self-Regulated Learning in MOOCs,” Internet and Higher Education 29 (2016): 40–48; René F. Kizilcec, Mar Pérez-Sanagustín, and Jorge J. Maldonado, “Self-Regulated Learning Strategies Predict Learner Behavior and Goal Attainment in Massive Open Online Courses,” Computers and Education 104 (2017): 18–33; and M. Elena Alonso-Mencía, Carlos Alario-Hoyos, Jorge Maldonado-Mahauad, Iria Estévez-Ayres, Mar Pérez-Sanagustín, and Carlos Delgado Kloos, “Self-Regulated Learning in MOOCs: Lessons Learned from a Literature Review,” Educational Review (2019): 1–27. For how people develop self-regulated learning skills, see Scott G. Paris and Alison H. Paris, “Classroom Applications of Research on Self-Regulated Learning,” Educational Psychologist 36, no. 2 (2001): 89–101; and Barry J. Zimmerman, “Becoming a Self-Regulated Learner: An Overview,” Theory into Practice 41, no. 2 (2002): 64–70.
31. Reich and Ruipérez-Valiente, “MOOC Pivot.”
32. Reich, “Rebooting MOOC Research.”
33. Justin Reich, “Big Data MOOC Research Breakthrough: Learning Activities Lead to Achievement,” EdTech Researcher (blog), March 30, 2014, http://
34. For a case study of OLI development, see Candace Thille, Emily Schneider, René F. Kizilcec, Christopher Piech, Sherif A. Halawa, and Daniel K. Greene, “The Future of Data-Enriched Assessment,” Research & Practice in Assessment 9 (2014): 5–16.
35. For the OLI statistics study, see William G. Bowen, Matthew M. Chingos, Kelly A. Lack, and Thomas I. Nygren, “Interactive Learning Online at Public Universities: Evidence from a Six-Campus Randomized Trial,” Journal of Policy Analysis and Management 33, no. 1 (2014): 94–111. David Pritchard’s introductory physics MOOC at MIT is probably the best studied xMOOC; Kimberly F. Colvin, John Champaign, Alwina R. Liu, Qian Zhou, Colin Fredericks, and David E. Pritchard, “Learning in an Introductory Physics MOOC: All Cohorts Learn Equally, Including an On-Campus Class,” International Review of Research in Open and Distributed Learning 15, no. 4 (2014). On the high costs of effective online learning, also see McPherson and Bacow, “Beyond the Hype Cycle.”
36. Justin Reich and Elizabeth Huttner-Loan, Teaching Systems Lab MOOCs in Review: 2017–2019 (Cambridge, MA: Teaching Systems Lab, 2019), doi:10.35542/osf.io/c3bhw.
37. The total number of MOOC learners has increased at a lower rate than the total number of MOOC courses, so that each course now has many fewer people in it. See Chuang and Ho “HarvardX and MITx Year 4”; and Reich and Ruiperez-Valiente, “MOOC Pivot.”
38. On credit recovery, see Carolyn J. Heinrich, Jennifer Darling-Aduana, Annalee Good, and Huiping Cheng, “A Look Inside Online Educational Settings in High School: Promise and Pitfalls for Improving Educational Opportunities and Outcomes,” American Educational Research Journal 56, no. 6 (2019): 2147–2188. For state-level studies of virtual schools, see June Ahn and Andrew McEachin, “Student Enrollment Patterns and Achievement in Ohio’s Online Charter Schools,” Educational Researcher 46, no. 1 (2017): 44–57; Brian R. Fitzpatrick, Mark Berends, Joseph J. Ferrare, and R. Joseph Waddington, “Virtual Illusion: Comparing Student Achievement and Teacher and Classroom Characteristics in Online and Brick-and-Mortar Charter Schools,” Educational Researcher 49, no. 3 (2020): 161–175, https://
39. Chris Dede and John Richards proposed a name for LMSs that are prepopulated with content but intended for use by teachers in small classes: digital teaching platforms; Christopher Dede and John Richards, eds., Digital Teaching Platforms: Customizing Classroom Learning for Each Student (New York: Teachers College Press, 2012). On Summit Learning, see Joanne Jacobs, “Pacesetter in Personalized Learning: Summit Charter Network Shares Its Model Nationwide,” Education Next 17, no. 4 (2017): 16–25; and Matt Barnum, “Summit Learning, the Zuckerberg-Backed Platform, Says 10% of Schools Quit Using It Each Year. The Real Figure is Higher,” Chalkbeat, May 23, 2019, https://
40. John Daniel, Asha Kanwar, and Stamenka Uvalić-Trumbić, “Breaking Higher Education’s Iron Triangle: Access, Cost, and Quality,” Change: The Magazine of Higher Learning 41, no. 2 (2009): 30–35. The metaphor comes from the project management field; Dennis Lock, Project Management Handbook (Aldershot, Hants, England: Gower Technical Press, 1987).
41. Patrick McAndrew and Eileen Scanlon, “Open Learning at a Distance: Lessons for Struggling MOOCs,” Science 342, no. 6165 (2013): 1450–1451.
1. In the late 2000s and early 2010s, some other common reform efforts in schools involved alignment with the new Common Core curriculum, using data from standardized tests to improve instruction, adopting differentiated instruction approaches like “response to intervention” and addressing issues of social and emotional learning.
2. Benjamin S. Bloom, “The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring,” Educational Researcher 13, no. 6 (1984): 4–16. Bloom’s thought-piece was inspired by two doctoral dissertations. Subsequent investigations of human tutoring have demonstrated less than two standard deviation effects; see Kurt VanLehn, “The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems,” Educational Psychologist 46, no. 4 (2011): 197–221.
3. R. C. Atkinson and H. A. Wilson, “Computer-Assisted Instruction,” Science 162, no. 3849 (1968): 73–77.
4. Brian Dear, The Friendly Orange Glow: The Untold Story of the PLATO System and the Dawn of Cyberculture (New York: Pantheon, 2017). R. A. Avner and Paul Tenczar, “The TUTOR Manual” (Washington, DC: Education Resources Information Center, 1970), https://
5. Michael Horn and Heather Staker, Blended: Using Disruptive Innovation to Improve Schools (New York: Wiley, 2014).
6. Eric Westervelt, “Meet the Mind Reading Robo Tutor in the Sky,” NPRed, October 13, 2015, https://
7. Anya Kamenetz, “5 Doubts about Data Driven Schools,” NPRed, June 3, 2016, https://
8. “The Mismeasure of Students: Using Item Response Theory instead of Traditional Grading to Assess Student Proficiency,” Knewton Blog, June 7, 2012, https://
9. For a history and introduction to item response theory, see W. J. van der Linden, “Item Response Theory,” in International Encyclopedia of Education, 3rd ed., eds. Penelope Peterson, Eva Baker, and Barry McGaw (Oxford, UK: Elsevier, 2010), 81–89.
10. Among the most important variants to IRT are a set of algorithms developed at Carnegie Mellon University called “knowledge tracing.” For an introduction, see John R. Anderson, Albert T. Corbett, Kenneth R. Koedinger, and Ray Pelletier, “Cognitive Tutors: Lessons Learned,” The Journal of the Learning Sciences 4, no. 2 (1995): 167–207; Kenneth R. Koedinger and Albert T. Corbett, “Cognitive Tutors: Technology Bringing Learning Sciences to the Classroom,” in The Cambridge Handbook of the Learning Sciences, ed. R. K. Sawyer (New York: Cambridge University Press, 2006), 61–77.
11. Tony Wan, “Jose Ferreira Steps Down as Knewton CEO, Eyes Next Education Startup,” EdSurge, December 21, 2016, https://
12. Maciej Cegłowski. “The Internet with a Human Face,” presentation at Beyond Tellerrand, Düsseldorf, Germany, May 20, 2014, https://
13. For further discussion, see Justin Reich, “Personalized Learning, Backpacks Full of Cash, Rockstar Teachers, and MOOC Madness: The Intersection of Technology, Free-Market Ideology, and Media Hype in U.S. Education Reform,” May 7, 2013, presentation at Berkman Klein Center, Harvard University, https://
14. Emily Ann Brown, “Sal Khan Envisions a Future of Active, Mastery-Based Learning,” District Administration, January 31, 3019, https://
15. Audrey Watters, Teaching Machines (Cambridge, MA: MIT Press, forthcoming), citing Simon Ramo, “A New Technique of Education,” Engineering and Science 21 (October 1975): 372.
16. Clayton M. Christensen, Curtis W. Johnson, and Michael B. Horn, Disrupting Class: How Disruptive Innovation Will Change the Way the World Learns (New York: McGraw-Hill, 2008).
17. Jill Lepore, “The Disruption Machine,” New Yorker 23 (2014): 30–36. Audrey Watters, “The Myth and Millennialism of ‘Disruptive Innovation,’ ” Hack Education, May 24, 2013, http://
18. Alex Molnar, Gary Miron, Najat Elgeberi, Michael K. Barbour, Luis Huerta, Sheryl Rankin Shafer, and Jennifer King Rice, “Virtual Schools in the U.S., 2019,” National Education Policy Center, May 28, 2019, https://
19. Mark Dynarski, Roberto Agodini, Sheila Heaviside, Timothy Novak, Nancy Carey, Larissa Campuzano, Barbara Means, et al., “Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort,” National Center for Education Evaluation and Regional Assistance (2007); Saiying Steenbergen-Hu and Harris Cooper, “A Meta-analysis of the Effectiveness of Intelligent Tutoring Systems on K–12 Students’ Mathematical Learning,” Journal of Educational Psychology 105, no. 4 (2013): 970.
20. Eric Taylor, “New Technology and Teacher Productivity” (unpublished manuscript, January 2018), Cambridge, MA, Harvard Graduate School of Education, available at https://
21. John F. Pane, Beth Ann Griffin, Daniel F. McCaffrey, and Rita Karam, “Effectiveness of Cognitive Tutor Algebra I at Scale,” Educational Evaluation and Policy Analysis 36, no. 2 (2014): 127–144.
22. Interpreting effect sizes is the topic of vigorous discussion in education research. Cohen proposed a set of guidelines whereby a 0.2 standard deviation improvement would be considered small, 0.5 medium, and 0.8 large; in Jacob Cohen, Statistical Power Analysis for the Behavioral Sciences (New York: Academic Press, 1977). Since then, it has been more widely agreed in education research that these large effect sizes are almost never found in well-conducted randomized control trials. Lipsey argues for evaluating the magnitude of effect size based on prior research about gains in typical conditions; Carolyn J. Hill, Howard S. Bloom, Alison Rebeck Black, and Mark W. Lipsey, “Empirical Benchmarks for Interpreting Effect Sizes in Research,” Child Development Perspectives 2, no. 3 (2008): 172–177. Kraft suggests that based on more recent evidence, appropriate guidelines might be 0.05 standard deviation as small, 0.1 as medium, and 0.2 as large; Matthew A. Kraft, “Interpreting Effect Sizes of Education Interventions,” Educational Researcher (forthcoming), available at https://
23. Steve Ritter, Michael Yudelson, Stephen E. Fancsali, and Susan R. Berman, “How Mastery Learning Works at Scale,” in Proceedings of the Third (2016) ACM Conference on Learning at Scale (Association of Computing Machinery Digital Library, 2016), 71–79.
24. Neil T. Heffernan and Cristina Lindquist Heffernan, “The ASSISTments Ecosystem: Building a Platform that Brings Scientists and Teachers Together for Minimally Invasive Research on Human Learning and Teaching,” International Journal of Artificial Intelligence in Education 24, no. 4 (2014): 470–497; Jeremy Roschelle, Mingyu Feng, Robert F. Murphy, and Craig A. Mason, “Online Mathematics Homework Increases Student Achievement,” AERA Open 2, no. 4 (2016), https://
25. Early evidence of Teach to One was more positive; Douglas D. Ready, Katherine Conn, Elizabeth Park, and David Nitkin, “Year-One Impact and Process Results from the I3 Implementation of Teach to One: Math” (New York: Consortium for Policy Research Education, Teachers College, Columbia University, 2016), https://
1. For an example of where lines blur between algorithmic personalization and “whole child” learning, see Tom Vander Ark, “Chan Zuckerberg Backs Personalized Learning R&D Agenda,” Getting Smart, November 17, 2017, https://
2. Antonio Fini, “The Technological Dimension of a Massive Open Online Course: The Case of the CCK08 Course Tools,” International Review of Research in Open and Distributed Learning 10, no. 5 (2009); David Cormier, “What Is a MOOC?,” YouTube video, December 8, 2010, https://
3. George Siemens, “Connectivism: A Learning Theory for the Digital Age,” International Journal of Instructional Technology and Distance Learning 2, no. 1 (2005), http://
4. Detailed technical plans for developing these kinds of learning environments can be found at Kim Jaxon and Alan Levine, “Staking Your Claim: How the Open Web Is Won for Teaching and Learning,” University of California Irvine, 2017, http://
5. Downes, “Places to Go”; Fini “The Technological Dimension.”
6. Stephen Downes, “The Rise of MOOCs,” April 23, 2012, https://
7. Jean Lave and Etienne Wenger, Situated Learning: Legitimate Peripheral Participation (Cambridge, UK: Cambridge University Press, 1991). For other connections from connectivism to prior pedagogical theory, see Rita Kop and Adrian Hill, “Connectivism: Learning Theory of the Future or Vestige of the Past?,” The International Review of Research in Open and Distributed Learning 9, no. 3 (2008), http://
8. On DS106, see Howard Rheingold, “DS106: Enabling Open, Public, Participatory Learning,” Connected Learning Alliance, https://
9. A discussion of the Daily Create is in Abram Anders, “Theories and Applications of Massive Online Open Courses (MOOCs): The Case for Hybrid Design,” International Review of Research in Open and Distributed Learning 16, no. 6 (2015): 39–61.
10. W. Ian O’Byrne and Kristine E. Pytash, “Becoming Literate Digitally in a Digitally Literate Environment of Their Own,” Journal of Adolescent & Adult Literacy 60, no. 5 (2017): 499–504, http://
11. I discuss some of my own efforts along these lines at Justin Reich, “Techniques for Unleashing Student Work from Learning Management Systems,” KQED Mindshift, February 13, 2015, https://
12. Alexandra Juhasz and Anne Balsamo, “An Idea Whose Time Is Here: FemTechNet–A Distributed Online Collaborative Course (DOCC),” Ada: A Journal of Gender, New Media, and Technology 1, no. 1 (2012); Robinson Meyer, “5 Ways of Understanding the New, Feminist MOOC That’s Not a MOOC,” Atlantic, August 20, 2013, https://
13. For the shift from the open web to walled gardens maintained by oligopolist technology companies see David Weinberger, “The Internet That Was (and Still Could Be),” Atlantic, June 22, 2015, https://
14. Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Evelyn Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian Silverman, and Yasmin Kafai, “Scratch: Programming for All,” Communications of the ACM 52, no. 11 (2009): 60–67.
15. Mitchel Resnick, Lifelong Kindergarten: Cultivating Creativity through Projects, Passion, Peers, and Play (Cambridge, MA: MIT Press, 2017).
16. Seymour Papert, Mindstorms: Children, Computers, and Powerful Ideas (New York: Basic Books, 1980).
17. For Scratch usage statistics, see https://
18. Mizuko Ito, Kris Gutiérrez, Sonia Livingstone, Bill Penuel, Jean Rhodes, Katie Salen, Juliet Schor, Julian Sefton-Green, and S. Craig Watkins, Connected Learning: An Agenda for Research and Design (Irvine, CA: Digital Media and Learning Research Hub, 2013), https://
19. Mitchel Resnick, “ Let’s Teach Kids to Code,” TEDx Beacon Street, November 2012, https://
20. Mitchel Resnick, “The Next Generation of Scratch Teaches Much More Than Coding,” EdSurge, January 3, 2019, https://
21. Jal Mehta and Sarah Fine, In Search of Deeper Learning (Cambridge, MA: Harvard University Press, 2019).
22. Ito et al., Connected Learning.
23. Resnick et al., “Scratch”; Papert, “Mindstorms.”
24. Paul A. Kirschner, John Sweller, and Richard E. Clark, “Why Minimal Guidance during Instruction Does Not Work: An Analysis of the Failure of Constructivist, Discovery, Problem-Based, Experiential, and Inquiry-Based Teaching,” Educational Psychologist 41, no. 2 (2006): 75–86.
25. Sigal Samuel, “Canada’s ‘Incel Attack’ and Its Gender-Based Violence Problem,” Atlantic, April 28, 2018, https://
26. danah boyd, “Media Manipulation, Strategic Amplification, and Responsible Journalism,” Points, September 14, 2018, https://
27. Alice Marwick and Rebecca Lewis, “Media Manipulation and Disinformation Online,” Data and Society, May 5, 2017, https://
28. Zeynep Tufekci, “YouTube: The Great Radicalizer,” New York Times, March 10, 2018.
29. Richard Hofstadter, The Paranoid Style in American Politics (New York: Vintage, 2012). On Lewin, see MIT News Office, “MIT Indefinitely Removes Online Physics Lectures and Courses by Walter Lewin,” December 8, 2014, http://
30. On confusion in cMOOCs, see Rita Kop, “The Challenges to Connectivist Learning on Open Online Networks: Learning Experiences during a Massive Open Online Course,” The International Review of Research in Open and Distributed Learning 12, no. 3 (2011): 19–38; Colin Milligan, Allison Littlejohn, and Anoush Margaryan, “Patterns of Engagement in Connectivist MOOCs,” Journal of Online Learning and Teaching 9, no. 2 (2013): 149–159. For “getting stuck” in classrooms in Scratch and solutions for educators, see Paulina Haduong and Karen Brennan, “Helping K–12 Teachers Get Unstuck with Scratch: The Design of an Online Professional Learning Experience,” in Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Association for Computing Machinery Digital Library, 2019), 1095–1101; Karen Brennan, “Beyond Right or Wrong: Challenges of Including Creative Design Activities in the Classroom,” Journal of Technology and Teacher Education 23, no. 3 (2015): 279–299.
31. Anton Barua, Stephen W. Thomas, and Ahmed E. Hassan, “What Are Developers Talking About? An Analysis of Topics and Trends in Stack Overflow,” Empirical Software Engineering 19, no. 3 (2014): 619–654.
1. Games researcher Jane McGonigal estimated in 2010 that people across the world played 3 billion hours of video games per week. Jane McGonigal, “Gaming Can Make a Better World,” TED talk, February 2010, https://
2. See, for instance, Larry Johnson, Samantha Adams Becker, Victoria Estrada, and Alex Freeman, NMC Horizon Report: 2014 K–12 Edition (Austin, TX: New Media Consortium, 2014), https://
3. For games as powerful sites of learning, see James Gee, What Video Games Have to Teach Us about Learning and Literacy (New York: St. Martin’s Press, 2007).
4. Federal Trade Commission, Lumosity to Pay $2 Million to Settle FTC Deceptive Advertising Charges for Its “Brain Training” Program, January 5, 2016, https://
5. Daniel J. Simons, Walter R. Boot, Neil Charness, Susan E. Gathercole, Christopher F. Chabris, David Z. Hambrick, and Elizabeth A. L. Stine-Morrow, “Do ‘Brain-Training’ Programs Work?,” Psychological Science in the Public Interest 17, no. 3 (2016): 103–186. Thomas S. Redick, Zach Shipstead, Elizabeth A. Wiemers, Monica Melby-Lervåg, and Charles Hulme, “What’s Working in Working Memory Training? An Educational Perspective,” Educational Psychology Review 27, no. 4 (2015): 617–633.
6. Robert S. Woodworth and E. L. Thorndike, “The Influence of Improvement in One Mental Function upon the Efficiency of Other Functions (I),” Psychological Review 8, no. 3 (1901): 247. For a recent piece on the origins of transfer research, see Daniel Willingham, “Critical Thinking: Why Is It So Hard to Teach?,” American Educator, Summer 2007, https://
7. Giovanni Sala and Fernand Gobet, “Does Far Transfer Exist? Negative Evidence from Chess, Music, and Working Memory Training,” Current Directions in Psychological Science 26, no. 6 (2017): 515–520. The research on memory and chess is quite extensive. An important early contribution is William G. Chase and Herbert A. Simon, “Perception in Chess,” Cognitive Psychology 4, no. 1 (1973): 55–81. A more recent study is Yanfei Gong, K. Anders Ericsson, and Jerad H. Moxley, “Recall of Briefly Presented Chess Positions and Its Relation to Chess Skill,” PloS one 10, no. 3 (2015): https://
8. Douglas B. Clark, Emily E. Tanner-Smith, and Stephen S. Killingsworth, “Digital Games, Design, and Learning: A Systematic Review and Meta-analysis,” Review of Educational Research 86, no. 1 (2016): 79–122. Pieter Wouters, Christof van Nimwegen, Herre van Oostendorp, and Erik D. van der Spek, “A Meta-analysis of the Cognitive and Motivational Effects of Serious Games,” Journal of Educational Psychology 105, no. 2 (2013): 249–265, https://
9. Brenda Laurel, Utopian Entrepreneur (Cambridge, MA: MIT Press, 2001).
10. David B. Tyack and Larry Cuban, Tinkering toward Utopia (Cambridge, MA: Harvard University Press, 1995).
11. Frederic Lardinois, “Duolingo Hires Its First Chief Marketing Officer as Active User Numbers Stagnate but Revenue Grows,” Techcrunch, August 1, 2018, https://
12. Hermann Ebbinghaus, Memory, trans. H. A. Ruger and C. E. Bussenius (New York: Teachers College, 1913); Nicholas Cepeda, Harold Pashler, Edward Vul, John T. Wixted, and Doug Rohrer, “Distributed Practice in Verbal Recall Tasks: A Review and Quantitative Synthesis,” Psychological Bulletin 132, no. 3 (2006): 354.
13. Burr Settles and Brendan Meeder, “A Trainable Spaced Repetition Model for Language Learning,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, vol. 1, Long Papers (Stroudsburg, PA: Association for Computational Linguistics, 2016), 1848–1858; Roumen Vesselinov and John Grego, “Duolingo Effectiveness Study” (City University of New York, 2012), https://
14. Eric Klopfer, Jason Haas, Scot Osterweil, and Louisa Rosenheck, Resonant Games (Cambridge, MA: MIT Press, 2018).
15. Alex Calhoun, “Vanished Helps Kids Save the Future with Science,” Wired, November 28, 2011, https://
16. For an introduction to the game and an optimistic take on educational possibilities, see Steve Nebel, Sascha Schneider, and Günter Daniel Rey, “Mining Learning and Crafting Scientific Experiments: A Literature Review on the Use of Minecraft in Education and Research,” Journal of Educational Technology & Society 19, no. 2 (2016): 355. Sales and user participation data at https://
17. Stampylonghead’s Minecraft introductions can be found at https://
18. Katie Salen, “10 Life Skills Parents Can Nurture through Minecraft,” Connected Camps Blog, September 16, 2017, https://
19. Samantha Jamison, “Computer Game a Building Block for Engineers,” Carnegie Mellon University News, July 26, 2017, https://
20. For some recent research on Zoombinis gameplay, see Elizabeth Rowe, Jodi Asbell-Clarke, Santiago Gasca, and Kathryn Cunningham, “Assessing Implicit Computational Thinking in Zoombinis Gameplay,” in Proceedings of the 12th International Conference on the Foundations of Digital Games (Association for Computing Machinery, 2017), https://
1. Judith Haymore Sandholtz, Cathy Ringstaff, and David C. Dwyer, Teaching with Technology: Creating Student-Centered Classrooms (New York: Teachers College Press, 1997).
2. A comment on Class Central, a review site for MOOCs, captures some of the critiques of the Neuroscience course: “Material & topic seemed interesting. Kind of silly way of presenting it, in my opinion, but quality of the videos was frankly good. Instructor tried to be funny too hard. I dropped; too much posh design & annoying music as background distraction for me in a neuroscience course,” https://
3. Tom Page, “Skeuomorphism or Flat Design: Future Directions in Mobile Device User Interface (UI) Design Education,” International Journal of Mobile Learning and Organisation 8, no. 2: 130–142; David Oswald and Steffen Kolb, “Flat Design vs. Skeuomorphism–Effects on Learnability and Image Attributions in Digital Product Interfaces,” in DS 78: Proceedings of the 16th International Conference on Engineering and Product Design Education (Design Education and Human Technology Relations, University of Twente, The Netherlands, May 4, 2014), 402–407.
4. Matthew Glotzbach, “A New Monthly Milestone for Quizlet: 50 Million Monthly Learners,” Quizlet Blog, October 29, 2018, https://
5. One useful source for understanding why structural elements of schools are so inimical to new innovations is David K. Cohen and Jal D. Mehta, “Why Reform Sometimes Succeeds: Understanding the Conditions That Produce Reforms That Last,” American Educational Research Journal 54, no. 4 (2017): 644–690.
6. From 2012 to 2018, Audrey Watters tracked and commented on venture investments in the edtech sector. For the 2018 example, see Audrey Watters, “The Business of ‘EdTech Trends,’” Hack Education, December 31, 2018, http://
7. John B. Diamond, “Where the Rubber Meets the Road: Rethinking the Connection between High-Stakes Testing Policy and Classroom Instruction,” Sociology of Education 80, no. 4 (2007): 285–313; Christopher Lynnly Hovey, Lecia Barker, and Vaughan Nagy, “Survey Results on Why CS Faculty Adopt New Teaching Practices,” in Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Association for Computing Machinery Digital Library, 2019), 483–489.
8. Seymour Papert, Mindstorms: Children, Computers, and Powerful Ideas (New York: Basic Books, 1980); Mitchel Resnick, Brad Myers, Kumiyo Nakakoji, Ben Schneiderman, Randy Pausch, Ted Selker, and Mike Eisenberg, “Design Principles for Tools to Support Creative Thinking,” National Science Foundation workshop on Creativity Support Tools (Washington, DC, 2005), http://
9. Natalie Rusk and Massachusetts Institute of Technology Media Laboratory, Scratch Coding Cards: Creative Coding Activities for Kids (San Francisco: No Starch Press, 2017).
10. Tsur and Rusk, “Scratch Microworlds”; Phillip Schmidtt, Mitchel Resnick, and Natalie Rusk, “Learning Creative Learning: How We Tinkered with MOOCs,” P2PU, http://
11. Karen Brennan, Sarah Blum-Smith, and Maxwell M. Yurkofsky, “From Checklists to Heuristics: Designing MOOCs to Support Teacher Learning,” Teachers College Record 120, no. 9 (2018): n9; Paulina Haduong and Karen Brennan, “Getting Unstuck: New Resources for Teaching Debugging Strategies in Scratch,” in Proceedings of the 49th ACM Technical Symposium on Computer Science Education, 1092 (Association for Computing Machinery Digital Library, 2018), https://
12. For an example of Desmos integration into College Board tests, see https://
13. Dan Meyer, “Math Class Needs a Makeover,” TED talk (2010), https://www.ted.com/talks/dan_meyer_math_class_needs_a_makeover?language=en; “The Three Acts of a Mathematical Story,” Mr. Meyer (blog), May 11, 2011, http://
14. Dan Meyer, “Video and Multiple Choice: What Took Us So Long?” Desmos (blog), October 27, 2016, https://
15. Ibid.
16. Larry Cuban. Hugging the Middle: How Teachers Teach in an Era of Testing and Accountability (New York: Teachers College, Columbia University, 2009); Larry Cuban, Inside the Black Box of Classroom Practice: Change without Reform in American Education (Cambridge, MA: Harvard Education Press, 2013).
17. Cohen and Mehta, “Why Reform.”
18. For the connections between Dewey’s pragmatism and Web 2.0, see Michael Glassman and Min Ju Kang “The Logic of Wikis: The Possibilities of the Web 2.0 Classroom,” International Journal of Computer-Supported Collaborative Learning 6, no. 1 (2011): 93–112, https://
1. This chapter is adapted from Justin Reich and Mizuko Ito, From Good Intentions to Real Outcomes: Equity by Design in Learning Technologies (Irvine, CA: Digital Media and Learning Research Hub, 2017), https://
2. Matthew 25:29 (New International Version).
3. Larry Cuban, Teachers and Machines: The Classroom Use of Technology since 1920 (New York: Teachers College Press, 1986), 23.
4. Jeannie Oakes, Keeping Track: How Schools Structure Inequality (New Haven: Yale University Press, 2005).
5. Paul Attewell, “Comment: The First and Second Digital Divides,” Sociology of Education 74, no. 3 (2001): 252–259.
6. Harold Wenglinsky, “Does It Compute? The Relationship between Educational Technology and Student Achievement in Mathematics” (Princeton, NJ: Educational Testing Services, 1998), https://
7. Ulrich Boser, “Are Schools Getting a Big Enough Bang for Their Education Buck?,” Center for American Progress Blog, June 14, 2013, https://
8. I describe the hopes for education in the Web 2.0 era in “Reworking the Web, Reworking the World: How Web 2.0 Is Changing Our Society,” Beyond Current Horizons (2008), https://
9. Justin Reich, Richard Murnane, and John Willett, “The State of Wiki Usage in US K–12 Schools: Leveraging Web 2.0 Data Warehouses to Assess Quality and Equity in Online Learning Environments,” Educational Researcher 41, no. 1 (2012): 7–15.
10. John D. Hansen and Justin Reich, “Democratizing Education? Examining Access and Usage Patterns in Massive Open Online Courses,” Science 350, no. 6265 (2015): 1245–1248.
11. Justin Reich, “The Digital Fault Line: Background,” EdTech Researcher, May 3, 2013, https://
12. René F. Kizilcec, Andrew J. Saltarelli, Justin Reich, and Geoffrey L. Cohen, “Closing Global Achievement Gaps in MOOCs,” Science 355, no. 6322 (2017): 251–252.
13. Kizilcec et al., “Closing.”
14. Rene Kizilcec, Justin Reich, Michael Yeomans, Christoph Dann, Emma Brunskill, Glenn Lopez, Selen Turkay, Joseph Williams, and Dustin Tingley, “Scaling Up Behavioral Science Interventions in Online Education,” in Proceedings of the National Academy of Science (forthcoming).
15. Kizilcec et al. “Closing.”
16. Tressie McMillan Cottom, “Intersectionality and Critical Engagement with the Internet” (February 10, 2015). Available at SSRN, https://
17. Reich and Ito, “From Good intentions.”
18. Betsy James DiSalvo, “Glitch Game Testers: The Design and Study of a Learning Environment for Computational Production with Young African American Males,” PhD diss., Georgia Institute of Technology, 2012; Betsy James DiSalvo, Mark Guzdail, Tom Mcklin, Charles Meadows, Kenneth Perry, Corey Steward, and Amy Bruckman, “Glitch Game Testers: African American Men Breaking Open the Console,” in Proceedings of the 2009 DiGRA International Conference: Breaking New Ground: Innovation in Games, Play, Practice and Theory, http://
19. Digital Promise, IT Best Practices Toolkits: Student Tech Teams, 2018, https://
20. Ricarose Roque, “Family Creative Learning,” Makeology: Makerspaces as Learning Environments 1 (2016): 47. For research on Scratchers and their parents, see Karen Brennan and Mitchel Resnick, “Imagining, Creating, Playing, Sharing, Reflecting: How Online Community Supports Young People as Designers of Interactive Media,” in Emerging Technologies for the Classroom, eds. Chrystalla Mouza and Nancy Lavigne (New York: Springer, 2013), 253–268. In the One Laptop Per Child deployment in Paraguay, similar evidence emerged that constructionist tools like Scratch were primarily taken up in homes where parents could support learning about computing; see Morgan Ames, The Charisma Machine: The Life, Death, and Legacy of One Laptop per Child (Cambridge, MA: MIT Press, 2019).
21. See, Tech Goes Home 2018 Annual Report, https://
22. Sarah Kessler, “How Jim McKelvey’s Launchcode Is Helping Unconventional Tech Talent,” Fast Company, April 18, 2016, https://
23. Mizuko Ito, Kris Gutiérrez, Sonia Livingstone, Bill Penuel, Jean Rhodes, Katie Salen, Juliet Schor, Julian Sefton-Green, and S. Craig Watkins, “Connected Learning: An Agenda for Research and Design” (Irvine, CA: Digital Media and Learning Research Hub, 2013), https://
24. Moran Tsur and Natalie Rusk, “Scratch Microworlds: Designing Project-Based Introductions to Coding,” in Proceedings of the 49th ACM Technical Symposium on Computer Science Education (Association for Computing Machinery Digital Library, 2018), 894–899.
25. Nichole Pinkard, Sheena Erete, Caitlin K. Martin, and Maxine McKinney de Royston, “Digital Youth Divas: Exploring Narrative-Driven Curriculum to Spark Middle School Girls’ Interest in Computational Activities,” Journal of the Learning Sciences 26, no. 3 (2017): 477–516.
26. Rebecca Pitt, “Mainstreaming Open Textbooks: Educator Perspectives on the Impact of Openstax College Open Textbooks,” International Review of Research in Open and Distributed Learning 16, no. 4 (2015); David Ruth, “OpenStax Announces Top 10 Schools That Have Adopted Free College Textbooks,” OpenStax, February 21, 2019, https://
27. Benjamin L. Castleman, and Lindsay C. Page, Summer Melt: Supporting Low-Income Students through the Transition to College (Cambridge, MA: Harvard Education Press, 2014).
28. Benjamin L. Castleman and Lindsay C. Page, “Summer Nudging: Can Personalized Text Messages and Peer Mentor Outreach Increase College Going among Low-Income High School Graduates?,” Journal of Economic Behavior & Organization 115 (2015): 144–160; Benjamin L. Castleman and Lindsay C. Page, “A Response to ‘Texting Nudges Harm Degree Completion,’ ” EducationNext, January, 28, 2019, https://
1. An early version of the argument in this chapter is in Justin Reich, “Will Computers Ever Replace Teachers?,” New Yorker: Elements, July 8, 2014, https://
2. One of the most readable summaries of how computers are changing labor market demands is Frank Levy and Richard J. Murnane, Dancing with Robots: Human Skills for Computerized Work (Washington, DC: Third Way NEXT, 2013). Two of the researchers who have continued research along similar lines are David Autor and David Deming; see Autor’s Work of the Past, Work of the Future (Cambridge, MA: National Bureau of Economic Research, 2019); and Deming’s “The Growing Importance of Social Skills in the Labor Market,” Quarterly Journal of Economics 132, no. 4 (2017): 1593–1640. See also Morgan R. Frank, David Autor, James E. Bessen, Erik Brynjolfsson, Manuel Cebrian, David J. Deming, Maryann Feldman, et al., “Toward Understanding the Impact of Artificial Intelligence on Labor,” Proceedings of the National Academy of Sciences 116, no. 4 (2019): 6531–6539.
3. Levy and Murnane “Dancing with Robots.”
4. Frank Levy and Richard J. Murnane, The New Division of Labor: How Computers Are Creating the Next Job Market (Princeton, NJ: Princeton University Press, 2005).
5. David H. Autor, Frank Levy, and Richard J. Murnane, “The Skill Content of Recent Technological Change: An Empirical Exploration,” The Quarterly Journal of Economics 118, no. 4 (2003): 1279–1333; Deming, “Social Skills.”
6. On various skill frameworks, see Chris Dede, “Comparing Frameworks for 21st Century Skills,” in 21st Century Skills: Rethinking How Students Learn, eds. James Bellanca and Ron Brandt (Bloomington, IN: Solution Tree Press, 2010), 51–76.
7. Dana Remus and Frank S. Levy, “Can Robots Be Lawyers?,” Computers, Lawyers, and the Practice of Law (November 27, 2016), available at SSRN: https://
8. Levy and Murnane, “New Division.”
9. Henry I. Braun and Robert Mislevy, “Intuitive Test Theory,” Phi Delta Kappan 86, no. 7 (2005): 488–497.
10. Daniel M. Koretz, Measuring Up (Cambridge, MA: Harvard University Press, 2008).
11. Brian Dear, The Friendly Orange Glow: The Untold Story of the PLATO System and the Dawn of Cyberculture (New York: Pantheon, 2017).
12. R. A. Avner and Paul Tenczar, “The TUTOR Manual” (US Department of Education, Education Resources Information Center [ERIC], 1970), https://
13. See Common Core State Standards Initiative, http://
14. For another argument in favor of focusing education on what computers can’t do, see Conrad Wolfram, “Teaching Kids Real Math with Computers,” TED talk (2010), https://
15. Unsupervised machine learning, where algorithms classify documents into clusters based on feature similarity, is another branch of machine learning, which thus far has limited applications to assessment. With colleagues, I’ve proposed ways of using natural language processing and unsupervised machine learning to aid human assessment; see Justin Reich, Dustin Tingley, Jetson Leder-Luis, Margaret E. Roberts, and Brandon Stewart, “Computer-Assisted Reading and Discovery for Student Generated Text in Massive Open Online Courses,” Journal of Learning Analytics 2, no. 1 (2015): 156–184; Researchers at the University of Illinois at Urbana-Champaign have proposed using unsupervised learning models as inputs to supervised algorithms for assessment; see Saar Kuzi, William Cope, Duncan Ferguson, Chase Geigle, and ChengXiang Zhai, “Automatic Assessment of Complex Assignments Using Topic Models,” Proceedings of the 2019 ACM Learning at Scale Conference (Association for Computing Machinery Digital Library, 2019), 1–10.
16. For an overview of machine-evaluated pronunciation, see Silke M. Witt, “Automatic Error Detection in Pronunciation Training: Where We Are and Where We Need to Go,” Proceedings of the International Symposium on Automatic Detection on Errors in Pronunciation Training (2012): 1–8.
17. For a paper that offers some sense of the amount of data required for various pronunciation training tasks, see Wenping Hu, Yao Qian, and Frank K. Soong, “A New DNN-based High Quality Pronunciation Evaluation for Computer-Aided Language Learning (CALL),” Interspeech (2013): 1886–1890, https://
18. For an optimistic argument for the state of automated essay grading, see Mark D. Shermis, “State-of-the-Art Automated Essay Scoring: Competition, Results, and Future Directions from a United States Demonstration,” Assessing Writing 20 (2014): 53–76. For a more pessimistic view, see Les Perelman, “When ‘the State of the Art’ Is Counting Words,” Assessing Writing 21 (2014): 104–111.
19. Shermis, “State-of-the-Art.”
20. Perelman, “State”; Randy Elliot Bennett, “The Changing Nature of Educational Assessment,” Review of Research in Education 39, no. 1 (2015): 370–407, https://
21. Quotations from https://
22. Steven Kolowich, “Writing Instructor, Skeptical of Automated Grading, Pits Machine vs. Machine,” Chronicle of Higher Education 28 (2014), https://
23. The excerpt is found in Audrey Watters Teaching Machines (Cambridge, MA: MIT Press, forthcoming), and the quotation is from “Exams by Machinery,” Ohio State University Monthly (May 1931): 339. The quotation is also cited in Stephen Petrina, “Sidney Pressey and the Automation of Education, 1924–1934,” Technology and Culture 45, no. 2 (2004): 305–330.
24. Harold Abelson, Gerald Jay Sussman, and Julie Sussman, Structure and Interpretation of Computer Programs (Cambridge, MA: MIT Press, 1996), xxii.
25. Jennifer French, Martin A. Segado, and Phillip Z. Ai, “Sketching Graphs in a Calculus MOOC: Preliminary Results,” in Frontiers in Pen and Touch: Impact of Pen and Touch Technology on Education (Cham, Switzerland: Springer, 2017), 93–102.
26. Valerie Jean Shute and Matthew Ventura, Stealth Assessment: Measuring and Supporting Learning in Video Games (Cambridge, MA: MIT Press, 2013). Jennifer S. Groff, “The Potentials of Game-Based Environments for Integrated, Immersive Learning Data,” European Journal of Education 53, no. 2 (2018): 188–201.
27. Groff, “The Potentials of Game-Based Environments for Integrated, Immersive Learning Data.”
28. For one example of a rapidly developing AI / machine-learning-based technological system in the game of Go, see David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, et al., “Mastering the Game of Go with Deep Neural Networks and Tree Search,” Nature 529, no. 7587 (2016): 484.
29. David Pogue, “I’ll Have My AI Call Your AI,” Scientific American 319, no. 2 (2018): 26, https://
30. An early description of edX efforts at autograding is Piotr Mitros, Vikas Paruchuri, John Rogosic, and Diana Huang, “An Integrated Framework for the Grading of Freeform Responses,” in The Sixth Conference of MIT’s Learning International Networks Consortium, 2013, https://
1. On the analogies between online learning platforms and experimentation in consumer technology, see Shreeharsh Kelkar, “Engineering a Platform: The Construction of Interfaces, Users, Organizational Roles, and the Division of Labor,” New Media & Society 20, no. 7 (2018): 2629–2646.
2. Bruce Schneier, “Data Is a Toxic Asset,” Schneier on Security Blog, March 4, 2016, https://
3. A detailed description of a publicly available dataset with five years of Scratch data is Benjamin Mako Hill and Andrés Monroy-Hernández, “A Longitudinal Dataset of Five Years of Public Activity in the Scratch Online Community,” Scientific Data 4 (2017): 170002, https://
4. For an example of treating learning records like text documents, see Cody A. Coleman, Daniel T. Seaton, and Isaac Chuang, “Probabilistic Use Cases: Discovering Behavioral Patterns for Predicting Certification,” in Proceedings of the Second (2015) ACM Conference on Learning at Scale (Association for Computing Machinery Digital Library, 2015), 141–148.
5. Guanliang Chen, Dan Davis, Claudia Hauff, and Geert-Jan Houben, “Learning Transfer: Does It Take Place in MOOCs? An Investigation into the Uptake of Functional Programming in Practice,” in Proceedings of the Third (2016) ACM Conference on Learning at Scale (Association for Computing Machinery Digital Library, 2016), 409–418. In a related study that connected MOOC learning to new behaviors, researchers investigated whether participation in a MOOC about learning analytics led to greater involvement in the scholarly society for learning analytics or submissions to learning analytics conferences; see Yuan Wang, Luc Paquette, and Ryan Baker, “A Longitudinal Study on Learner Career Advancement in MOOCs,” Journal of Learning Analytics 1, no. 3 (2014): 203–206.
6. Tommy Mullaney, “Making Sense of MOOCs: A Reconceptualization of HarvardX Courses and Their Students,” SSRN (2014), https://
7. For another take on using large-scale data to zoom in and zoom out on learner behavior, see Jennifer DeBoer, Andrew D. Ho, Glenda S. Stump, and Lori Breslow, “Changing ‘Course’: Reconceptualizing Educational Variables for Massive Open Online Courses,” Educational Researcher 43, no. 2 (2014): 74–84. For research on how changes in the Scratch environment lead to changes in behavior, see Sayamindu Dasgupta and Benjamin Mako Hill, “How ‘Wide Walls’ Can Increase Engagement: Evidence from a Natural Experiment in Scratch,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Association for Computing Machinery Digital Library, 2018), 1–11.
8. Ryan Baker, Sidney D’Mello, Merecedes Rodrigo, and Arthur Graesser, “Better to be Frustrated Than Bored: The Incidence, Persistence, and Impact of Learners’ Cognitive-Affective States during Interactions with Three Different Computer-Based Learning Environments,” International Journal of Human-Computer Studies 68, no. 4 (2010), 223–241, https://
9. Boston School Committee, Reports of the Annual Visiting Committees of the Public Schools of the City of Boston, (1845), City Document no. 26 (Boston: J. H. Eastburn), 12.
10. Justin Reich, “ ‘Compass and Chart’: Millenarian Textbooks and World History Instruction in Boston, 1821–1923,” SSRN (2009), https://
11. Lorrie Shepard, “A Brief History of Accountability Testing, 1965–2007,” in The Future of Test-Based Educational Accountability, eds. Katherine Ryan and Lorrie Shepard (New York: Routledge. 2008), 25–46.
12. For two studies with promising results for education technology at scale, see John F. Pane, Beth Ann Griffin, Daniel F. McCaffrey, and Rita Karam, “Effectiveness of Cognitive Tutor Algebra I at Scale,” Educational Evaluation and Policy Analysis 36, no. 2 (2014): 127–144; and William G. Bowen, Matthew M. Chingos, Kelly A. Lack, and Thomas I. Nygren, “Interactive Learning Online at Public Universities: Evidence from a Six-Campus Randomized Trial,” Journal of Policy Analysis and Management 33, no. 1 (2014): 94–111. On the risk / benefit calculation from data-intensive educational research, see Rebecca Ferguson and Doug Clow, “Where Is the Evidence? A Call to Action for Learning Analytics,” Proceedings of the Seventh International Learning Analytics and Knowledge Conference (2017), 56–65.
13. For the official statement from the meeting, see Asilomar Convention for Learning Research in Higher Education, 2014, http://
14. For an introduction to some of these concerns, see Elana Zeide, “Education Technology and Student Privacy,” in The Cambridge Handbook of Consumer Privacy, eds. Evan Selinger, Jules Polonetsky, and Omer Tene, 70–84, SSRN (2018), https://
15. A landmark text of how surveillance and data capture is reshaping the world is Shoshana Zuboff, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power (New York: Public Affairs, 2019).
16. For an overview of K–12 legal issues related to student data and privacy, see Leah Plunkett, Alicia Solow-Niederman, and Urs Gasser, “Framing the Law and Policy Picture: A Snapshot of K–12 Cloud-Based Ed Tech and Student Privacy in Early 2014,” Berkman Center Research Publication 2014-10 (2014). For the yawning gaps between current law and much needed ethical guidelines, see Elana Zeide, “Unpacking Student Privacy,” in Handbook of Learning Analytics, eds. Charles Lang, George Siemens, Alyssa Wise, and Dragan Gasevic (Society for Learning Analytics Research, 2017), 327–335.
17. Natasha Singer, “Online Test-Takers Feel Anti-cheating Software’s Uneasy Glare,” New York Times, April 5, 2015, https://
18. Helen Nissenbaum, Privacy in Context: Technology, Policy, and the Integrity of Social Life (Stanford, CA: Stanford University Press, 2009). For the theory of contextual integrity applied to MOOCs and virtual learning environments, see Elana Zeide and Helen F. Nissenbaum, “Learner Privacy in MOOCs and Virtual Education,” Theory and Research in Education 16, no. 3 (2018): 280–307, available at https://
19. John D. Hansen and Justin Reich, “Democratizing Education? Examining Access and Usage Patterns in Massive Open Online Courses,” Science 350, no. 6265 (2015): 1245–1248.
20. Two efforts to share the role of research in the edX consortium are the edX page on research, https://
21. Audrey Watters, “The Weaponization of Education Data,” Hack Education, December 11, 2017, http://
22. On school cooperation with Immigration and Customs Enforcement, see Erica Green, “For Immigrant Students, a New Worry: Call to ICE,” New York Times, May 30, 2018, https://
23. James Murphy, “The Undervaluing of School Counselors,” Atlantic, September 16, 2016, https://
24. David Christian, Amy Lawrence, and Nicole Dampman, “Increasing College Access through the Implementation of Naviance: An Exploratory Study,” Journal of College Access 3, no. 2 (2017): 28–44; Christine Mulhern, “Changing College Choices with Personalized Admissions Information at Scale: Evidence on Naviance,” April 2019, https://
25. A Learning at Scale keynote, unrecorded but with a short published abstract, provides one example of this position: Peter Norvig, “Machine Learning for Learning at Scale,” in Proceedings of the Second (2015) ACM Conference on Learning at Scale (Association for Computing Machinery Digital Library, 2015), 215. See also Daphne Koller, “What We’re Learning from Online Education,” filmed June 2012 at TEDGlobal 2012, Edinburgh, Scotland, video, https://
26. Justin Reich “Engineering the Science of Learning,” Bridge 46, no. 3 (2016), https://
27. Rene Kizilcec, Justin Reich, Michael Yeomans, Christoph Dann, Emma Brunskill, Glenn Lopez, Selen Turkay, Joseph Williams, and Dustin Tingley, “Scaling Up Behavioral Science Interventions in Online Education,” in Proceedings of the National Academy of Science (forthcoming).
28. Benjamin Harold, “Pearson Tested ‘Social-Psychological’ Messages in Learning Software, with Mixed Results,” Education Week: Digital Education, April 17, 2018, https://
29. Michelle N. Meyer, Patrick R. Heck, Geoffrey S. Holtzman, Stephen M. Anderson, William Cai, Duncan J. Watts, and Christopher F. Chabris, “Objecting to Experiments That Compare Two Unobjectionable Policies or Treatments,” Proceedings of the National Academy of Sciences 116, no. 22 (2019): 10723–10728.
30. For Pearson’s response, see Valerie Strauss, “Pearson Conducts Experiment on Thousands of College Students without Their Knowledge,” Washington Post: Answer Sheet Blog, April 23, 2018, https://
31. For more thoughts on my cautious optimism on these experiments, see Justin Reich, “Can Text Messages and Interventions Nudge Students through School?” KQED Mindshift, June 3, 2015, https://
32. Zeide, “Unpacking.”
33. Monica Bulger, Patrick McCormick, and Mikaela Pitcan, “The Legacy of inBloom,” Data & Society Research Institute, 2017, https://
34. Bulger, McCormick, and Pitcan “The Legacy of inBloom.”
35. Leah Plunkett, Alicia Solow-Niederman, and Urs Gasser, “Framing the Law and Policy Picture: A Snapshot of K–12 Cloud-Based Ed Tech and Student Privacy in Early 2014,” presentation at Harvard Law School, June 3, 2014, available at https://
36. Kenneth R. Koedinger, Ryan S.J.d. Baker, Kyle Cunningham, Alida Skogsholm, Brett Leber, and John Stamper, “A Data Repository for the EDM Community: The PSLC DataShop,” Handbook of Educational Data Mining 43 (2010): 43–56.
1. Frederick James Smith, “The Evolution of the Motion Picture: VI—Looking into the Future with Thomas A. Edison,” New York Dramatic Mirror, July 9, 1913, 24, column 3, as analyzed in https://
2. Phil Hill, “Instructure: Plans to Expand Beyond Canvas LMS into Machine Learning and AI,” e-Literate, March, 2019, https://
3. “Roy Amara: 1925–2007, American Futurologist,” Oxford Essential Quotations, 4th ed., ed. Susan Ratcliffe, 2016, https://
4. John F. Pane, Beth Ann Griffin, Daniel F. McCaffrey, and Rita Karam, “Effectiveness of Cognitive Tutor Algebra I at Scale,” Educational Evaluation and Policy Analysis 36, no. 2 (2014): 127–144; Jeremy Roschelle, Mingyu Feng, Robert F. Murphy, and Craig A. Mason, “Online Mathematics Homework Increases Student Achievement,” AERA Open 2, no. 4 (2016), https://
5. For statistics on Wikipedia size, see https://