11 The Future

On a service, every investigation is like that. Suddenly, bugs become an immediate concern for developers. A widely used strategy is to give the developers pagers and put them on call to look into issues on a rotating basis, transforming developers into living exceptions handlers. This forces programmers to directly feel the pain of bugs, and it also inspires them to make sure the monitoring and alerting is accurate; nobody wants to miss a real issue, but nobody wants to be woken up in the middle of the night by a false alarm. And if your telemetry data isn’t rich enough to debug the failures, then you have lots of motivation to improve it, and quickly.

On the plus side, because the bugs are happening on machines that companies own, it is easier to keep track of them and figure out which ones cause the worst disruptions, and then go back and figure out how the problem could have been avoided—and how future problems of the same sort could be avoided. Was it poor design choices? Was it an API with undocumented side effects? Was it a unit test that should have been written? Was it a test that was written, but not run at the right time? Was it a step on a deployment checklist that was missed because somebody was not being careful or it wasn’t written down? It is now much more obvious if people are skipping out on these engineering-like behaviors that programmers had previously been able to ignore.

At some point, you could even conceive of tying this back to the so-called religious debates that have befuddled programmers through the years. Do you prefer your particular coding style? Do you think your favorite language leads to fewer bugs or more rapid development? Do you like Hungarian prefixes on your variable names? It’s unlikely that one company is going to produce enough data to individually come up with an answer, but if you analyze this across the industry, it’s possible that actual answers could emerge from the murk. Even if companies are not able to answer the questions themselves, they would at least be motivated to care about the answers (and sometimes to impose whatever mitigation was discovered onto their teams) because services have a nice way of converting these decisions into dollar amounts. If you have more bugs than another company that has made a different choice of coding style, if your service runs more slowly, if it is harder to deploy … that all translates to more money you need to spend somewhere to keep it running, which makes it harder to compete in the market.

Essentially, running a service imposes an automatic “no bullshit” filter on the old wives’ tales that permeate software engineering.

As an added benefit, the way data is transferred between components of a service, or between a client machine and service in the cloud, is much closer to the ideal of object-oriented programming than is typically achieved with software running on a single computer. In the early days of computer networking, when machines were all on the same local area network, the network was low bandwidth but also low latency; sending a lot of data through the network was slow, but a small network packet could reach the other machine fairly quickly. As a result, early network protocols focus on packing as much data into as little space as possible, which made writing the code to handle them tricky; it was processing this sort of incoming network packet that led to the Heartbleed worm in 2014.

Modern networks connecting a client to the cloud have much higher bandwidth. The computer network at Microsoft back in 1990 ran at 10 megabits per second shared by the entire floor of a building, while today it is typical for a home broadband connection to support 100 megabits per second directly to a single house, with the backbone connections that get you to the Internet running much faster than that. The connections have a higher latency, however, since a packet has to make multiple hops between computers. As a result, the amount of data in a packet doesn’t matter as much—a large packet doesn’t take noticeably longer than a short one to reach its destination—and communication is usually encoded in a more verbose format known as Extensible Markup Language (XML).

XML is a slightly more categorized version of the text strings used to communicate between pipelined UNIX commands, which previously had been the most effective implementation of “object-oriented” software building blocks. In older network protocols containing tightly packed binary data, the meaning of any given byte depended on its exact location in the packet, and it was hard for a human to parse the data to figure out what it meant; entire separate programs known as packet sniffers existed to assist in debugging network traffic (the sniffers we used in my early days at Microsoft were entirely separate machines, quite expensive to replace if you happened to drop a network cable into the back of one while it was turned on—not that I would know anything about that). XML, by contrast, uses human-readable tags to identify data, so a programmer can scan the XML request itself to figure out its meaning, and code to parse XML can be written more generically and therefore more safely (Microsoft Office uses XML in its newer file formats to avoid the problem of exploits hidden inside the old binary format). It’s similar to the difference between reading raw bytes of machine language and reading code in a higher-level language.

A second benefit is that XML communication is much more forgiving of version mismatches than earlier protocols. In particular, if a client sends data in a request that the server doesn’t expect, it can simply ignore it, which makes it easier to expand a network protocol without breaking interoperability with older clients. If early network protocols were like the tightly bound API calls in a performance-focused language like C++, where a single added API parameter could break the code unless all callers were also updated, XML is more like the Smalltalk messages-with-named-parameters approach—a little slower, but also less fragile in the face of changes.

That said, there is an interesting bifurcation going on. At the same time that programmers are working on cloud services with almost infinite computing power, they are writing programs to run on smaller devices. Just as a generation of programmers came of age on the resource-limited mainframes and minicomputers of the 1960s, and a new generation came of age on the resource-limited personal computers of the 1980s, it’s possible that another generation will learn to program on resource-limited phones and tablets. What remains to be seen is whether the Third Great Software Resource Bottleneck will teach programmers all the same bad habits of the previous generations, favoring performance and convenience, and less typing of course, at the expense of clean design.

The services approach can be difficult for programmers experienced in packaged software to adapt to. Services require a mind shift away from performance toward maintainability and reliability, and a further mind shift away from concentrating on shipping software toward focusing on keeping it running. Microsoft famously has a Ship-It Award where everybody who contributes to a project that ships to customers receives a small commemorative metal sticker to attach to a plaque, with the resulting accretion forming a record of each employee’s time at Microsoft.² The award was created in the early 1990s after a few large projects visibly flamed out and were canceled, with nothing to show for years of work; it was meant to emphasize the importance of actually finishing software and delivering it to customers. Despite a legendary stumble at the launch of the program,³ people do care about their Ship-It Awards, and many Microsoft employees proudly display their plaques in their offices.

Flash-forward fifteen years and your humble narrator, as part of the duties managing my team in Engineering Excellence, was solely responsible for determining whether a piece of software written at Microsoft was significant enough to deserve a Ship-It Award. The process by which this authority came to be vested in my position was somewhat convoluted, but I did my best to dispense justice as I saw fit (new versions of Office and Windows: yes; update to the point-of-sale software used in the Microsoft Store: no). The Ship-It program had been designed for packaged software, but around this time the large services teams were realizing that shipping is only the beginning; the real trick is to keep a service running. They began agitating for an equivalent recognition of this to adorn their plaques (the working title was Run-It Award), yet in the end their request was shot down—a legacy of the “it’s just sustained engineering” mentality.

In 2009, a lot of programmers at Microsoft were beginning to work on services, so we in Engineering Excellence went around the company asking those who had already been through the transition for their advice (besides “give us a Run-It Award”). The most insightful comment came from an engineer on the Exchange e-mail service who had previously worked on packaged software. After walking through a list of all the things he had needed to relearn in order to be effective working on a service, he ruefully said, “The problem is, if I went back in time two years and told myself all of that, I wouldn’t have believed it.” Brooks at one point quotes Benjamin Franklin, writing in Poor Richard’s Almanac: “Experience is a dear teacher, but fools will learn at no other.”⁴

The move to services is a step in the right direction, but more needs to be done.

Gawande’s book The Checklist Manifesto is subtitled How to Get Things Right. At one point he talks to a structural engineer about how the construction industry has changed:

Today’s software engineers are master builders, and they are in danger of being overwhelmed.

Gawande also discusses the hero test pilots from the early days of high-speed aircraft, as documented in Tom Wolfe’s book The Right Stuff. They succeeded through improvisation and daring, but eventually “values of safety and conscientiousness prevailed, and the rock star status of the test pilots was gone.”⁶ The universe applies a natural corrective to test pilots who fail to adapt, but no such effect exists in software; the middle and upper management of software companies are full of successful self-taught programmers. Hopefully, more recent graduates have heard about design patterns and unit tests; while those are a long way from providing a scientific basis for software engineering, at least they have opened people’s eyes to the fact that accepted knowledge is possible in the software industry, which in turn should make them more amenable to learning new ways if we can discover them. But of course they do have to be discovered first.

Are programmers inept? I don’t think they are in the exact way Gawande indicts medicine, the prime focus of his book. He is talking about situations in which the knowledge of how to deliver proper medical care is unquestionably available and agreed on by all involved, but for various reasons is not applied properly. Gawande sees parallels to this struggle in “almost any endeavor requiring mastery of complexity and of large amounts of knowledge”—a list that to him includes foreign intelligence failures, tottering banks, and (little does he know) flawed software design.⁸

The argument against labeling programmers inept is that they have not even reached the stage of knowing the right way to do their jobs. That’s because they have stopped trying to figure this out, which isn’t a great defense: we are still inept, just in a slightly different way.

In 1986, Brooks wrote an essay titled, “No Silver Bullet”—a reference to the ammunition reputedly needed to kill werewolves. “We hear desperate cries for a silver bullet, something to make software costs drop as rapidly as computer hardware costs do,” he lamented,

Brooks describes what he feels is the essence of the problem, the inherent difficulties of software: complexity, conformity (the need to fit new code into an existing API), changeability, and invisibility (the difficulty of visualizing the internals). He lists various developments that may help, including object-oriented programming (which at that time was just beginning its transformation into Object-Oriented Silver Bullet Amalgamated Holdings), but doubts that any of them will supply the magic needed.

Nine years later, almost at the end of the time period he was considering in the original paper, Brooks produced a follow-up essay “‘No Silver Bullet’ Refired,” discussing reactions to the original. He wrote, “Most of these attack the central argument that there is no magical solution, and my clear opinion that there cannot be one. Most agree with the arguments in ‘NSB,’ but then go on to assert that there is indeed a silver bullet for the software beast, which the author has invented.”¹⁰ The italics are mine, because I can think of no more succinct summary of the ongoing procession of alleged software panaceas.

Brooks ends his second essay with a short section titled “Net on Bullets—Position Unchanged.” He quotes noted software observer Robert Glass, in an essay in System Development magazine: “At last, we can focus on something a little more viable than pie in the sky. Now, perhaps, we can get on with the incremental improvements to software productivity that are possible, rather than waiting for the breakthroughs that are not likely to come.”¹¹

The history of software engineering has been a search for the silver bullet. Structured programming, formal testing, object-oriented languages, design patterns, Agile methodologies—all are useful, but none alone can slay the werewolf. I personally lived through all these transitions, even structured programming; due to spending my early years in a self-taught bubble of Fortran and BASIC, I was able to experience the twilight of GOTOs firsthand, ten years later than the rest of the industry. Each of these started out small, gained supporters, and wound up being hyped as the solution to all programming problems, leading to inevitable disappointment and disillusionment. I can understand why programmers hope beyond hope for the silver bullet and have eagerly glommed onto a succession of shiny objects. “Any port in a storm,” as the saying goes. Unfortunately, as Parnas put it, “[Programmers] have been fed so many ‘silver bullets’ that they don’t believe anything anymore.”¹²

Moreover, the software industry has gotten in the habit of abandoning old silver bullets once they get tarnished. It’s a binary view of the world: a programming technique is either going to solve every problem or it’s useless. When Microsoft began hiring software test engineers to test the software, the developers, who previously had been responsible for this, quickly segued into “throw it over the wall to test” mode. In the mid-2000s, Microsoft replaced software test engineers with software design engineers in test, responsible for writing automated tests, and stopped relying on manual testing. With the later move to unit tests, which are written by developers, the company got rid of a lot of the software design engineers in test and the user-interface-level tests that they provided. Now with the transition to the cloud, there is an emphasis on “test in production,” where updates to the software are quickly deployed to a small percentage of real customers, with checks in place to quickly detect problems and roll back the update. Each new technique is useful, but it should be viewed as another arrow in the quiver, not the be-all and end-all that obviates the need for what went before.

In 1978, Harlan Mills predicted that “The next generation of programmers will be much more competent than the first ones. They will have to be. Just as it was easier to get into college in the ‘good old days,’ it was also easier to get by as a programmer in the ‘good old days.’ For this new generation, a programmer will need to be capable of a level of precision and productivity never dreamed of before.”¹³ As with almost everything Mills wrote, this is as true today as it was back then.

Most of my recommendations involve changing how students are taught software engineering in universities. I am not entirely blaming universities; one of the main reasons they do not know what to teach is because industry is too self-satisfied and complacent to interact with academia. If you ask companies what they would like colleges to teach, they will likely start talking about so-called soft skills: communication, being on time, and working well with others. That’s nice, but it’s not surprising that a bunch of adults, who now have families, mortgages, and other responsibilities, are able to recognize that college seniors aren’t quite as mature as they are. It’s more difficult to determine what technical skills may be lacking in college graduates, particularly when those same skills may be lacking in the experienced employees as well.

And in the spirit of The Checklist Manifesto, I am not going to be too prescriptive. As long as industry and academia have talked and agreed on what is to be done, I will trust their judgment. But by “agreed,” I don’t mean the ACM has a conference where professors and industry researchers show up; there has been enough of that already. The ACM, to its credit, has over thirty special interest groups, including ones focused on computer science education, programming languages, and software engineering, but they do not get enough traction among working software professionals. I want software engineering programs to change their curriculum, and companies to care.

In any case, it’s a question of timing: future programmers usually are exposed to software education before they are exposed to software jobs, so let’s begin there. Through a combination of academia, industry, and divine providence, the following areas need to be addressed.

Force Students to Learn Something New

When the independent software industry emerged in the 1980s, it reinvented everything about managing software projects that had been written in the 1960s and 1970s. It is hard to fathom how much time and energy was wasted because of this. Yet there is no reason why it won’t happen again with the next generation of programmers.

Some students arrive in college with no knowledge of how to program, but many have significant experience. The sooner students are disabused of the notion that they know everything they need to know, the sooner they will become receptive to new ideas. I made it through my Princeton computer science classes, and to some extent my early years at Microsoft, on the basis of the skills I had taught myself in high school, writing BASIC games for the IBM PC. The classes weren’t easy; I had to write a lot of code and spent a lot of late nights in von Neumann getting my programs to work. But this did nothing to open my mind to the wide world of software engineering.

Forcing students to learn something new rather than get by on their existing skills will make them humble. And as jazz trumpeter Wynton Marsalis said, “The humble improve.”¹⁴ One example is the details of different algorithms for sorting arrays, which have names like bubble sort and selection sort. Different sorting algorithms perform better with certain data, which Knuth (in his landmark series of books The Art of Computer Programming) and others worked out a long time ago. When I asked programmers to write a sort algorithm on the whiteboard during a Microsoft interview, I didn’t care how they did it or if they knew the name of the algorithm. I was looking for that hard-to-capture ability to write code and get it to work, summarized in the statement, “They can code.” But why not expect that students can rattle off the intricacies of different sorting algorithms? Ironically, information floating around the Internet claims that Microsoft is looking for this level of detail during an interview, but I never saw it, although if you look over the slides from Stanford University’s class Problem-Solving for the CS Technical Interview or scan discussion threads on the website Reddit, it seems that some companies are.¹⁵

Even if students don’t remember all the details of algorithms, at least they will take away the awareness that there is a lot of knowledge out there, available when they need it. There are situations where your sorting algorithm does matter a lot, and you want students to develop the instinct of recognizing them.

While you’re at it, have students do up-front estimates of how much time they think it will take them to complete various phases of a project and then look back to compare that to how long they actually took. These will be smaller projects, so the estimates won’t be as far off as in the real world, but it can’t hurt to show them that even for smaller projects, estimating software is a perilous endeavor.

Certainly a new graduate may encounter a situation where deep engineering knowledge is not needed. Coding camps, which claim to turn anybody into a programmer in a few months, have a mixed reputation, but they do indicate that the basic knowledge required for certain programming jobs can be taught relatively quickly.

If students wind up working in an Agile Eden, on a small team that stays together for a long period of time, with a single customer who is engaged in overseeing their software, calling well-documented APIs in a well-understood environment—then great, they can dial back their application of software engineering principles and relive their halcyon days. But if not, they need to have the core engineering knowledge. It’s a lot easier to know the underlying principles and choose not to apply them than it is to not know them and be in over your head.

Students should also study their history. In June 2001, a conference was held in Bonn, Germany, featuring presentations by sixteen influential software pioneers; the results (including videos) were collected in the book Software Pioneers.¹⁶ It’s an impressive list: Wirth and Brooks; Dahl and Nygaard, who designed Simula; Friedrich Bauer, inventor of the stack; Kay, who came up with Smalltalk and the graphical user interface; Hoare, who did early work on program correctness, and his fellow Algol 68 denouncer Dijkstra; Parnas, who wrote some of the earliest papers on modularizing software; and others whom I have not mentioned only due to lack of room in this book.

The fact is, these luminaries won’t be around forever. Dijkstra, Dahl, and Nygaard all died the following year, within six weeks of each other; Bauer died in 2015. John Backus, inventor of Fortran, who was invited but could not attend, died in 2007. It would be a tremendous disservice to them if all the programmers who are benefiting from their insight did not acknowledge their contributions, and a tremendous waste of time if their wisdom was not dispensed to the next generation of programmers.

Work to Level the Playing Field

We need programmers; we can’t afford to exclude anybody before they leave the starting gate. You want to attract anybody who is interested in programming, high school geek or not, whatever their race and gender. Even if the sum total of what motivates a person to major in computer science is a single Hour of Code tutorial or the fact that they like to play video games, they should still feel welcome. Hopefully this will produce a double benefit: making other new programmers feel more welcome at school, and producing software engineers who are more teachable and open to lifelong learning. The alleged rise of the “brogrammer,” that annoying fraternity kid now reborn as a web developer, is in some ways an encouraging sign of software careers opening up to a wider audience.

Some universities have split their introductory classes into multiple tracks, so that students who have no experience programming are not intimidated by the propeller-heads.¹⁷ There is also a push to organize first-year programming classes around projects that are more interesting to students than pure algorithmic noodling, such as robotics or game programming.¹⁸

Another tactic is ensuring that having been in the computer club in high school is unlikely to provide a big advantage. Carnegie Mellon University teaches a language called ML in its introductory class. ML is an elegant but nonmainstream language, belonging to a class of languages known as functional languages—that is, not a procedural or object-oriented language. It’s something that programmers may never use in their professional career (although not surprisingly some programmers today claim that functional programming is, at last, the silver bullet that will cure all programming ills); the big benefit of using ML in an introductory class is the likelihood that none of the incoming students has used it in high school (they are unlikely to have learned any functional languages), so it places everybody on an equal footing at the start.

Related to this, there has been a fair bit of discussion about how to increase the percentage of women majoring in computer science, sometimes expanded to include having underrepresented minorities reach proportional representation. Harvey Mudd College has gotten a lot of press recently for reaching a proportion of almost 50 percent women in its computer science program, with a multitrack approach to introductory classes being one of its main strategies. Carnegie Mellon also has close to 50 percent women in its computer science program.

Computer science enrollment waxes and wanes, depending on whether the news is focusing on stock option millionaires or tech companies going bankrupt. A 2017 report by the Computing Research Association shows that since 2006, when computer science enrollment was at a low point following the bursting of the dot-com bubble in the early 2000s, the number of computer science majors has tripled, reaching an all-time high that is almost twice the number at the peak of the dot-com boom.¹⁹

The percentage of female computer science majors was 14 percent in 2006, dropped to 11 percent in 2009, and had climbed back to 16 percent by 2015, the last year covered by the report. Meanwhile, the percentage of underrepresented minorities hovered around 10 percent the entire time, ticking up slightly to 13 percent in 2015.²⁰ Given the overall rise in enrollment, these still represent large increases in the absolute numbers of women and underrepresented minorities majoring in computer science, but they remain much lower percentages than in the overall population (those underrepresented minorities, as defined for the report, constituted around 30 percent of the US population in the 2010 census).²¹

Female programmers have a secret weapon here. Both academia and industry need to do more to connect students and employees with the IEEE and ACM; these are the relevant professional societies, and people working in software engineering should know and care about them. The secret weapon for women is an annual conference known as the Grace Hopper Celebration of Women in Computing, named after the author of the first compiler who was one of the guiding forces behind COBOL (which, despite being the butt of jokes today, was a big step forward in its time). The conference has both career-related and technical talks. Such material is available in a lot of places, but the key to the Grace Hopper Celebration is that people, both students and professionals, attend it in large numbers; it is the conference that Microsoft, by a large margin, was the most interested in sending employees to. (The conference does allow men to attend and in fact actively encourages it.)

Teach Students to Work with Larger Pieces of Software

Back in 1980, Mills wrote,

It is characteristic in software engineering that the problems to be solved by advanced practitioners require sustained efforts over months or years from many people, often in the tens or hundreds. This kind of mass problem-solving effort requires a radically different kind of precision and scope in techniques than are required for individual problem solvers. If the precision and scope are not gained in university education, it is difficult to acquire them later, no matter how well motivated or adept a person might be at individual, intuitive approaches to problem solving.²²

It is common nowadays for students to do class programming projects in teams of two to four, to expose them to the issues that arise when you are not the only person working on a program. Even back in my day, I did several class projects working with a partner. This is a laudable attempt to expose students to some of the situations they will encounter working in industry.

Unfortunately, working with two to four people for one semester is not a large enough step up from a one-person project. Yes, you will see somebody else’s coding style, have to divide up the code and work out an API between each person’s contribution, and practice your interpersonal skills—all valid and possibly eye-opening experiences. But fundamentally, you can engineer a project like that using the programming skills you have taught yourself; if the API interface you are calling was worked out with a fellow student, it is unlikely that there will be confusion about the functionality, and in a semester you are not going to forget how the code works, so future maintainability won’t be an issue. Recalling Brooks’s description of the difference between a program and a programming system product—requiring both a move from a single author to multiple authors, and a move from a single component to a program that is built up from connections across API boundaries—working on small group projects gives you a slight nudge in both dimensions, but you still remain firmly in the starting quadrant.²³

Larger pieces of software will also let students practice reading code, and teach them the benefits of clear variable names, code comments, and the like. To get the benefits here, they need software that is much more complicated than what students could create on their own. It’s not like civil engineering, where if you can model the stress on one steel beam, you can extrapolate to an entire bridge. Perhaps some companies with large codebases would volunteer their code, viewing it as an opportunity to get name recognition among potential future employees. Alternatively, if companies won’t fork over their code, there are large open-source projects that would provide fruitful training grounds for students too. Care should also be taken to choose code written in a variety of languages, to help students get a better understanding of what languages are most suitable to what sorts of problems.

Debugging is also vastly easier on small programs, especially since they tend to run on small amounts of data. In college, I made do with what was termed printf debugging, after the printf() API in C that is used to print to the console; if my program was not working as expected, I would litter it with temporary printf() statements to display the contents of my variables at different points and then look through the output trying to find the point of the first fault. This works great on college-scale projects, but falls apart rapidly in the corporate world—at least for most of us. Thompson, who designed and implemented the first UNIX system, has stated that he much prefers printf debugging.²⁴ But as Baird put it, “If you are a genius like Ken Thompson, you are going to write good code.”²⁵ The rest of us need to move beyond printf debugging to get our code working, as I rapidly discovered once I started working on the internals of Windows NT at Microsoft (the fundamental skill is learning to use a specialized piece of software called a debugger, which can examine the memory of another program).

Debugging can in ways be compared to a doctor diagnosing a patient—a skill that medical schools certainly spend a lot of time teaching. Of course, the doctor doesn’t first have to learn how to build a human being, which leaves more room for other topics, but debugging is rarely taught to students. There are tools and techniques that can be used for debugging large programs that could be taught in college if large enough codebases were used to practice on, but I arrived in industry completely unaware of them.

Parnas expressed hope that internships would expose students to these sorts of issues.²⁶ The problem is that internships are primarily extended job interviews as well as advertisements for the company. For both reasons, companies tend to smooth the path for interns and avoid exposing them to too much code, no matter how well organized it might be. Requiring interns to assimilate a large body of code would be discouraging and also soak up a significant percentage of the internship. But they could certainly be required to do it for a college course.

Emphasize Writing Readable Code

Code is usually written once and read many times, but historically the emphasis has been on making the writing easier, at the expense of the reading. Requiring that students participate in inspections—formal inspections, not ad hoc code reviews—would be extremely valuable for everyone, not necessarily to find defects, but to ensure that students were writing readable code because they would need to read each other’s code in order to participate in the inspections. Inspections can be daunting not only for the person whose code is being inspected (who can feel like they are suffering through an inquisition) but also for the inspectors (who can feel pressure to find issues). But these can be mitigated, partly by having an experienced person leading the inspection, and partly by having everybody in turn volunteer code to be inspected. And if students complain to their professors that inspections are unfair because they haven’t been taught what their code should look like—well, that would be a great forcing function for professors to go figure that out.

There are other techniques that have attempted to produce more readable code. In 1984, Knuth published the paper “Literate Programming” about writing programs with a primary focus on making them comprehensible to others. He enthused, “The impact of this new approach on my own style has been profound, and my excitement has continued unabated for more than two years. I enjoy the new methodology so much that it is hard for me to refrain from going back to every program that I’ve ever written and recasting it in ‘literate’ form.”²⁷

Literate programming involves the programmer authoring a file that contains both the code and an explanation of it, mixed together, with the explanation as the primary focus, and the code appearing in small fragments following the relevant explanation; fragments of code can reference other fragments of code, similar to an API call. A separate program then parses this original file to produce two pieces of output: a file containing all the fragments pulled together so they can be compiled, and a formatted document containing all of the explanations of the fragments (Knuth named the first implementation of this system “WEB,” explaining—this was in 1984, remember—“I chose the name WEB partly because it was one of the few three-letter words of English that hadn’t already been applied to computers”).²⁸ When my kids took drivers’ education, they were taught to do “commentary driving,” in which they explain their thoughts out loud as they are driving. Literate programming is sort of like that for coding, except you write your commentary down alongside the source code.

Knuth did an experiment with seven undergraduates one summer, teaching them literate programming; six of them loved it because, as he put it, “it blended well with their psyche.”²⁹ The methodology doesn’t have broad adoption; Knuth commented that if one in fifty people are good at programming, and one in fifty people are good at writing, it is hard to find people who are good at both. Knuth successfully used it for the popular text-formatting program known as T_EX, although for much of it he was the only author.

Literate programming’s greatest success may be a computer graphics renderer—software that takes a 3-D representation and turns it into an image—written by Matt Pharr and Greg Humphreys, which they documented in the book Physically Based Rendering.³⁰ They were inspired by the book A Retargetable C Compiler, which was also created using literate programming (in fact, as a collaboration between a Bell Labs researcher, Christopher Fraser, and a Princeton professor, David Hanson).³¹ In both cases, the actual book was generated from the original file that had the code fragments mixed in; the book therefore served as complete documentation of the algorithms used for the program and was easy to keep in sync with the code.

Pharr and Humphreys call literate programming “a new programming methodology based on the simple but revolutionary idea that programs should be written more for people’s consumption than for computers’ consumption.”³² Along with their Stanford professor Pat Hanrahan, they won a Scientific and Technical Academy Award for the book; if you watch videos of the presentation, not only do you see Kristen Bell and Michael B. Jordan do a surprisingly credible job of explaining recent advances in rendering technology, you get to hear Knuth be thanked in an Oscar acceptance speech.

Is literate programming a great idea? I honestly don’t know; even if Bell does have a copy of Physically Based Rendering on her Kindle, as she jokingly claimed, it’s just one example of a successful program. Knuth himself wrote, “My enthusiasm is so great that I must warn the reader to discount much of what I shall say as the ravings of a fanatic who thinks he has just seen a great light.”³³ In the foreword to Physically Based Rendering, Hanrahan observes, “Writing a literate program is a lot more work than writing a normal program. After all, who ever documents their programs in the first place!? Moreover, who documents them in a pedagogical style that is easy to understand? And finally, who ever provides commentary on the theory and design issues behind the code as they write the documentation?”³⁴ Maybe it is more work—but maybe that is work that programmers need to do.

Literate programming is an interesting example of how hard it is for new ideas in programming to emerge from academia. It was invented by Knuth, who has impeccable credentials and broad name recognition; it has an example in T_EX that is widely used and unusually large for a piece of software produced by an academic; and it had positive results from the informal study that Knuth performed. Nonetheless, it has a hard time getting airplay when competing with trendier initiatives such as Agile, whose proponents’ livelihoods are based on its success.

Whether it is a literate program or something else, if the provider of an API created more formal documentation on what the API did, rather than just the method name and parameter list, it would mitigate API confusion. In Java, a method has to list every exception that it throws as well as every exception thrown by a method it calls that it does not itself catch—in other words, the complete set of exceptions that a caller of this method can expect to have thrown.³⁵ This requires more thinking and typing by the programmer, but it does clarify some of the potential side effects of the method, allowing the calling code to be more robust, since unclear side effects inside an API are a source of errors. Parnas, in a 1994 paper with Jan Madey and Michal Iglewski, proposed a symbolic notation for specifying the side effects of API calls.³⁶ Such things make programmers blanch, but if it helps clear up confusion, we need to consider biting the bullet and having programmers learn to read the notation.

Relocate Certain Well-Understood Topics

The university majors computer science and software engineering are used inconsistently although often interchangeably. But certainly, you have people who undertake a course of study in college that is more theoretical than what they apply on the job. As McConnell wrote about the issue of computer science majors becoming software engineering professionals, “This puts computer scientists into a technological no-man’s land. They are called scientists, but they are performing job functions that are traditionally performed by engineers without the benefit of engineering training.”³⁷

We need a real software engineering major that focuses on the engineering practices. Yet undergraduate curriculums are already jam-packed. How will they make room to teach more fundamentals?

One answer is to push a few time-honored subjects out of the software engineering major. There are areas of computer science where the theory and practice have been fleshed out reasonably well over the years, including graphics, compilers, and databases. These are often taught in undergraduate courses, but the reality is that unless a student goes on to work in that specific area, these courses don’t add much value, except as a chance to write more code. And if students do wind up working in that area, they can look up that knowledge. The problems with software engineering are generally not in finding the correct algorithm; they involve translating that algorithm into code that works correctly. So while it’s good for all students to learn fundamental algorithms for sorting and such, the more advanced ones aren’t necessary.

Of course, these are frequently the areas that professors specialize in, so teaching such classes has appeal to them. I’m not saying they shouldn’t be taught at all; I’m saying they should be moved over to a real computer science major that does not claim to teach software engineering or, instead, offered to graduate students.

Pushing these topics to graduate school would allow a student who was interested to concentrate on them, which would allow students to differentiate themselves more. Currently, software engineers coming out of college are viewed as fungible; it is expected that any programmer, if found competent by whatever hiring procedure is used, can go work on any part of a program. As software becomes more and more complicated, however, it makes more sense for people to specialize in different areas.

Mills wrote about specialization on a surgical team:

A surgical team represents a good example of work structuring, with different roles predefined by the profession and previous education. Surgery, anesthesiology, radiology, nursing, etc. are dimensions of work structuring in a surgical team. The communication between these roles is crisp and clean—with a low bandwidth at their interface, e.g., at the “sponge and scalpel” level, not the whole bandwidth of medical knowledge.³⁸

Gawande also talks about specialization in his book on checklists. Following his guidance, the way to ensure that software was secure, fast, reliable, or whatever other aspect you were concerned about, would not be to make a long checklist that had to be followed by every programmer. It would be to hire programmers who were specialized in security, performance, reliability, or whatever, and then have a checklist item that simply read, “Have you checked with the security/performance/reliability/whatever expert that they are happy with this software?” In an undergraduate education, not all these areas can be covered. Making them graduate specialties would solve that problem and give a graduate degree higher status in industry. Companies, in turn, would learn to favor them when hiring people to fill these specialized roles.

Pay Attention to Empirical Studies

In 1995, Brooks wrote, “In preparing my retrospective and update of The Mythical Man-Month, I was struck by how few of the propositions asserted in it have been critiqued, proven, or disproven by ongoing software engineering research and experience.”³⁹

Empirical studies never went away; the Empirical Studies of Programmers workshops continued, and the journal Empirical Software Engineering is still published. A well-known (within the field of empirical research) conference was held at Dagstuhl Castle in Germany in 2006. The book Making Software, from 2011, has thirty essays on empirical software engineering, including “How Effective Is Test-Driven Development” and “How Usable Are Your APIs.”⁴⁰ The Psychology of Programming Interest Group in England continues to put on an annual conference.

What changed was the appetite of programmers for consuming this material. With the PC explosion in the late 1970s and 1980s came the emergence of an independent software industry, filled with self-taught programmers who quickly realized they could make lots of money by continuing the practices they had taught themselves. They felt no need to look at research on how to improve, so they didn’t. Whatever methodology turned out to be critical for their success, such as tracking bugs or scheduling, they gradually and painfully reinvented, unaware that the prior generation had already written extensively on how to address these problems.

The irony is that programmers generally like to get uncommon insight into the way the world works. In 1984, when I was a senior in high school, I attended a weeklong camp at the University of Waterloo for students who had done well in an annual math competition. At this meeting, essentially the annual gathering of the Future Programmers of Canada Society, a fellow attendee was reading a book by a baseball writer I had never heard of named Bill James. The book was all about his attempts to bring mathematical rigor to the game: taking long-unquestioned “rules” of baseball, such as using wins as a measure of pitcher quality or the value of a stolen base, and trying to answer them by mining the rich trove of statistical data that had been built up over the previous century. I immediately loved this idea and became a James fan, as many other programmers are. But somehow this never translated into thinking about whether the long-unquestioned “rules” of software could also be studied and analyzed.

Glass, in his collection of essays Software Conflict, asks, “So why don’t we have an experimental component to our field’s science and our field’s engineering?” and continues,

There are two reasons that I can think of:

1. Experiments that are properly controlled and conducted are hard and expensive to conduct. It’s not enough to rope three undergraduate students into writing 50 lines of Basic and then compare notes. If an experiment is going to be meaningful, it ought to involve real software developers solving real software problems in a carefully pre-defined and measured setting.

2. The engineers and scientists in our field are neither motivated nor prepared to conduct meaningful experiments. Advocacy [by which Glass means writing that promotes a methodology without any supporting experimental evidence] has been with us for so long that is just doesn’t seem to occur to anyone that there’s a component missing from our research. And without motivation to supply the missing component, no one is getting the proper intellectual tools to know how to conduct experimental research.

Glass does allow, “Perhaps the words no one in the preceding paragraph is an example of going too far. For example, the folks working on the intersection of software and psychology, the ‘empirical studies’ folks, are doing fairly interesting experimental work.”⁴¹

Just because empirical studies of programming are hard doesn’t mean that researchers should not be doing them—and it certainly isn’t an excuse for companies to ignore what has already been studied. The name empirical studies and alternative term software psychology need an upgrade. What we are talking about is programming science. Normally I would put this one on academia and say that it needed to do more of this sort of research. But industry is also not conditioned to care about this sort of research because it has been so successful without it.

When I worked at Microsoft, particularly when I was in Engineering Excellence, I observed a resistance to advice if it was gleaned from experience within Microsoft itself. There was a team in Microsoft Research that examined software engineering, generally using Microsoft as its population of interest, but the results, although considered to be thought-provoking tidbits, rarely had any uptake back in the product groups. It was a variation of the Gell-Mann Amnesia effect, where people realize that news stories about their areas of expertise are simplistic or inaccurate, but completely trust news stories about topics they know nothing about. If you told members of one Microsoft team about the engineering experience of another team, they would immediately be able to identify—because of their knowledge of the internals of Microsoft—the ways in which that other team was different from their team, and therefore dismiss the guidance as not relevant. Meanwhile, they would happily slurp up guidance on Scrum, even if it was completely inapplicable to their team, because they weren’t aware of the details of the environment in which it had been successful.

Set a Goal of Eventual Certification and Licensing

McConnell’s 1999 book After the Gold Rush: Creating a True Profession of Software Engineering addresses a theme similar to that of this book: How do we fix software engineering? He references an earlier iteration of SWEBOK as a guide for what universities should teach students, and proposes the certification (voluntary) or licensing (mandatory) of software engineers.⁴²

I concur with McConnell’s ultimate goal: software engineering should be taught from an agreed-on body of knowledge, which could then lead to the certification and licensing of software engineers based on that body of knowledge. This would benefit everybody—colleges would know what to teach, companies would know what to interview for, and software developers would know the gaps in their skills, and of course, in the end, they would do a better job and we would have more reliable software.

Unfortunately, SWEBOK is still not yet ready for prime time; it includes curriculum guidelines, but I agree with the SEMAT book’s dismissive summary of those: “guidelines at a very high level, leaving (too) much to be defined by individual universities and professors.”⁴³ In 2013, the Texas Board of Professional Engineers, as part of the licensing of software engineers, adopted the Principles and Practice of Engineering exam, which exists for a wide variety of disciplines, in the area of software engineering.⁴⁴ The exam, which covers a broad but shallow SWEBOK-ish range of knowledge, is being viewed with a “wait-and-see” attitude by other jurisdictions.⁴⁵

Even with a well-defined body of knowledge, companies’ attitude toward certification and licensing would likely still be indifferent at best, and actively oppositional at worst, because they view them as having no tangible benefits. Companies care about employees knowing the specific languages and tools they need for their job, but those vary widely across the industry. In addition, they are concerned that licensing would open up the issue of liability for bugs.

Parnas provided a version of a standard line (attributing it to a friend): “We live in countries where you need a license to cut hair, but you don’t need anything to write code for safety critical software or other mission critical software.”⁴⁶ McConnell helpfully supplied a list of over thirty professions that require licensing in the state of California, including custom upholsterer and mule jockey.⁴⁷ It’s an easy punch to land, but it’s still true.

Certification and licensing should happen, but not right away; what is important is that industry and academia agree that they are a goal to strive for. Maybe it’s not required for everybody, and maybe you need a master’s degree, but we should set it as a long-term goal. In 1999, McConnell wrote, “We need to continue working on several fronts—instituting widespread undergraduate education, licensing professional software engineers, establishing software engineering certification programs, and thoroughly diffusing best practices into the industry.”⁴⁸ Twenty years later, there has been almost zero progress in those areas.

In the meantime, what do we have? What we have is the situation I found myself in, twenty-plus years into my career as a software developer: I realized that there was a curious void in my professional life because there was no craftsperson-apprentice relationship that I could develop with recent college graduates. I could mentor people on how to navigate the waters of corporate life, but that was generic advice that they could get from anybody. Like others, my guidance was vague: “Well, in this one case I remember this sort of thing worked OK, so why not try that?”

I saw the author Michael Lewis give a talk at Microsoft about his book The Big Short, which covered the financial meltdown of 2008. Lewis’s first book, Liar’s Poker, chronicled his time working on Wall Street, the headquarters of the US financial services industry. He mentioned that over the years, many people told him they went to work on Wall Street after reading Liar’s Poker. He found this curious because his intent had been to demonstrate that Wall Street was a silly place to work, but somehow it came across to college students as an interesting place to work. He realized that an author doesn’t have control over readers, but he did feel a little funny because he felt that Wall Street was a huge waste of talent.

Is the software industry a huge waste of talent? I am sure that most Wall Street workers feel that they provide a valuable service, notwithstanding Lewis’s opinion. And I certainly don’t feel that my career has been a waste. I’ve worked on software products that are extremely useful to many people. A few employees have told me that reading one of my earlier books inspired them to want to work for Microsoft, which made me proud, not guilty. Still, I have felt the effect of a problem that a Microsoft executive once lamented: “We hire a lot of smart people at Microsoft, but they tend to cancel each other out.” Expending time and energy, when I was working on product teams at Microsoft, convincing peers of the benefits of a certain approach to the software development process (or worse, failing to convince them because their unexamined beliefs were as entrenched as mine), when the positives or negatives of such a process should have been worked out long ago, was extremely wasteful, not to mention frustrating. I think of the first line of Allen Ginsberg’s poem “Howl,” “I saw the best minds of my generation destroyed by madness,” and I feel some of his frustration.⁴⁹

Steve Lohr, in his history of software titled Go To, stated the proposition that “Americans typically brought an engineering mentality to the task of designing programming languages—compromises were made to solve the computing problem at hand. The Europeans, by contrast, often took a more theoretical academic approach to language design.”⁵⁰ This claim, which he admits is a “broad generalization,” is not borne out by the facts. You can look back at the history of programming languages and find similar ideas being propagated on both sides of the Atlantic. Simula and Pascal came from Europe, and Smalltalk and Algol came from the United States. The two early salespeople of object-oriented design, Meyer and Cox, were French and American, respectively. Dijkstra was European, and Knuth is American. A Dane working in the United States designed C++, the ultimate compromise. If the availability of peanut butter has an effect on language design, I haven’t observed it.

What is undoubtedly true, however, is that as the needs of industry outstripped the ability of college professors to keep up, the nexus of software engineering moved firmly to the United States, because that’s where almost all the largest software companies are headquartered.

Is the current state of software a reflection of US values that favor individuality over community, breaking the rules over following them, MacGyver over Hercule Poirot? It’s not that US companies institutionally ignore quality; the hardware companies that initially fostered an engineering-focused approach to software, when they were the places where software was being written back in the 1960s, were also primarily American. Kurt Andersen, in his recent book Fantasyland, discusses how starting in the 1960s Americans began to drift away from rationalism, as individuals increasingly felt that “your beliefs about anything are equally as true as anyone else’s … What I believe is true because I want and feel it to be true.”⁵¹ This would make software the quintessential American industry, and certainly there is something about the speed with which software has arrived on the scene, coupled with the amount of money being made today, that seems to uniquely bind software to foundational US legends—the self-made person, the genius in the garage, the myth of the West. It may be impossible to convince US software leaders that they aren’t experiencing manifest destiny replayed in the twenty-first century, and that the software industry should be anything other than exactly what it is now. If you use the number of millionaires as your metric for determining the success of an industry, software engineering appears to be doing great.

But that doesn’t mean we don’t need to change it. And the change doesn’t have to originate in the United States. There are plenty of technologies, from cars to skyscrapers to solar panels, which initially flowered in the United States but whose center of innovation later shifted elsewhere. Currently, the United States is the predominant source of wisdom, as it were, on software development, but there is no innate reason that it has to stay that way, particularly if other countries are more willing to invest in fundamental engineering research.

Mills fretted about this back in 1980, referring to the missile gap that existed with the Soviet Union in the early days of the Cold War: “The inertia of several hundred thousand undisciplined programmers in the United States is a real reason for concern. … Unless we address this problem with exceptional measures, we are on the way to a ‘software gap’ much more serious and persistent than the famous ‘missile gap’ which helped fuel the very growth of our electronics industry.”⁵² So far this has not manifested itself: the mass of undisciplined programmers has grown, but they are not uniquely concentrated in the United States.

Meanwhile, noted skeptic Edward Yourdon (whose early warnings about the impending Y2K catastrophe, depending on whom you ask, either vastly overstated the problem or gave the world time to avoid it) published a book in 1992 titled Decline and Fall of the American Programmer, in which he blithely predicted, “The American programmer is about to share the fate of the dodo bird. By the end of this decade, I foresee massive unemployment among the ranks of American programmers, systems analysts, and software engineers.” His premise was that “American software is developed at a higher cost, less productively, and with less quality than that of several other countries.” Therefore, “if your slovenly bunch of software engineers doesn’t want to play at the world-class level of performance, trade them in for a new bunch from Ireland or Singapore.”⁵³

This has not come to pass. US code is currently no worse than other code. Yourdon himself, a mere four years later, published the book Rise and Resurrection of the American Programmer, in which he walked back some of his earlier warnings, inspired by, among other things, the growth of the Internet and success of Microsoft. Referring to the strawperson COBOL programmer from his first book, Yourdon wrote, “That American programmer is indeed dead, or at least in grave peril. But there’s a new generation of American programmers, doing exciting new things.”⁵⁴ But of course that doesn’t mean it can’t happen in the future. Personally I don’t care; I want software to become a real engineering discipline, but this is unrelated to whether the center remains in the United States.

Coming back to the original question I asked in the introduction, Is software development really hard, or are software developers not that good at it? Certainly there are parts of software that are hard, but software developers seem to do everything in their power to make even the easy parts harder by wasting an inordinate amount of time on reinvention and inefficient approaches. A lot of mistakes are in fundamental areas that should be understood by now, and so the software industry needs to push to figure them out and then teach people about them, so we can devote our energy to the parts that really are hard.

In the 1982 coming-of-age movie Fast Times at Ridgemont High, the character of Jeff Spicoli is visited on the evening of his high school graduation dance by his history teacher, the inimitable Mr. Hand, and forced to atone for his previously lackluster commitment to scholarship by demonstrating his knowledge of the American Revolution. At the end, Spicoli produces this summary: “What Jefferson was saying was, hey, we left this England place because it was bogus. So if we don’t get some cool rules ourselves, pronto, we’ll just be bogus too.”⁵⁵

This is what software engineering is facing today. We left behind the other jobs we could have taken because they didn’t appeal to us, and headed to the land of software where we could be clever and creative. We had fun for a while, but now the whole world is depending on us, and if we don’t get some cool rules ourselves, pronto, … well, you know.

Notes