8

Engineering a Solution

We build software like the Wright brothers built airplanes: build the whole thing, push it off a cliff, and start over again.

—Ronald Graham, NATO Conference on Software Engineering¹

Industrializing Software Development

In the collective memory of the programming community, the years between 1968 and 1972 mark a major turning point in the history of its industry and profession. It is during this period that the rhetoric of the crisis became firmly entrenched in the vernacular of commercial computing. Although there had been earlier concerns expressed about “software turmoil” and the “software gap,” it was not until 1968 that the word “crisis” began to be applied to the challenges facing the software industry. Within a few short years, the existence of a looming software crisis had been widely and enthusiastically embraced within the popular and industry literature. The discourse of crisis became one of the defining features of the software industry; since the late 1960s, almost every new computer-science curriculum proposal, programming technology, or development methodology has positioned itself relative to this perception of widespread crisis. Even those who deny the very existence of the crisis are continually forced to engage with its pervasive discursive legacy.²

To a certain degree the emergence of the software crisis of the late 1960s represents the culmination of a long series of concerns about software: the seemingly perpetual shortage of programming personnel; the burgeoning complexity of both application and systems software; the apparent failure of automatic programming technologies to make the process of programming less mysterious or more cost-effective; the professional and political tensions inherent in management information systems and other organizationally disruptive technologies; and a growing sense at all levels of society that the changes associated with the computer revolution were more fundamental and pervasive—and at times intrusive—than had previously been anticipated.³

What is novel and significant about the software crisis discourse, therefore, was not in its identification of a series of problems but rather in the nature of its proposed solutions. For most historians as well as most contemporary observers, the software crisis of the late 1960s was defined by the emergence of new software-engineering approaches to the problems of software development.

The phrase “software engineering” appears to have first been used by the hardware engineer J. Presper Eckert in an address to the Eastern Joint Computer Conference in 1965 in reference to the growing conflict between computer programmers and their corporate employers. Computer programming “would only be manageable,” he claimed, “when we could refer to it as ‘software engineering.’”⁴ But it was the 1968 NATO Conference on Software Engineering that marks the moment that software engineering dramatically entered the public consciousness.

In October 1968, a diverse group of academic computer scientists, corporate managers, and military officials gathered in Garmisch, Germany, for the first-ever NATO Conference on Software Engineering. The conference was intended to address what many industry observers believed to be an impending crisis in software production. Large software development projects had acquired a reputation for being behind schedule, over budget, and bug ridden. The solution to the so-called software crisis, suggested the conference organizers, was for software developers to adopt a more methodical and industrial approach. The phrase “software engineering” was “deliberately chosen as being provocative,” suggested the conference organizers, “in implying a need for software manufacturing to be based on the types of theoretical foundations and practical disciplines that are traditional in the established branches of engineering.”⁵ In the interest of efficient software manufacturing, the black art of programming had to make way for the science of software engineering.

By defining the software crisis in terms of the discipline of software engineering, the conference set an agenda that influenced many of the technological, managerial, and professional developments in commercial computing for the next several decades. The general consensus among historians and practitioners alike is that the Garmisch meeting marked a major cultural shift in the perception of programming. In the aftermath of Garmisch, “software writing started to make the transition from being a craft for a long-haired programming priesthood to becoming a real engineering discipline. It was the transformation from an art to a science.”⁶ The call to integrate “good engineering principles” into the software development process has been the rallying cry of software developers from the late 1960s to the present.⁷

The fundamental problem with software, according to the NATO conference organizers, was not personnel or technology but rather technique. Software development was difficult because computer programmers had failed to follow an appropriate methodology. They persisted in their craft-based mentality when what was demanded was clearly an industrial system of manufacturing. “We undoubtedly produce software by backward techniques, argued M. Douglas McIlroy of Bell Telephone Laboratories: “We undoubtedly get the short end of the stick in confrontations with hardware people because they are the industrialists and we are the crofters.”⁸ Like many of his fellow participants, McIlroy rejected the notion that large software projects were inherently unmanageable. The imposition of engineering management methods had enabled efficient manufacturing in myriad other industries, and would not fail to do the same for computer programming. Software engineering promised to bring control and predictability to the traditionally undisciplined practices of software development.

For a number of conference participants, the key word in the provocative NATO manifesto was “discipline.” For example, in his widely quoted paper on “mass-produced software components,” McIlroy proposed applying mass-production techniques to software.⁹ His vision of a software “components factory” invokes familiar images of industrialization and proletariatization. According to his proposal, an elite corps of “software engineers” would serve as the Frederick Taylors of the software industry, carefully orchestrating every action of a highly stratified programmer labor force. And like the engineers in more traditional manufacturing organizations, these software engineers would identify themselves more as corporate citizens than as independent professionals.¹⁰

Not every proposed solution to the software crisis suggested at Garmisch was as blatantly management oriented as McIlroy’s. Nevertheless, the theme of transformation from a craft-based black art of programming to the industrial discipline of software engineering dominated many of the presentations and discussions. The focus on management solutions reflected—and reinforced—a larger groundswell of popular opinion that extended far beyond the confines of the actual conference. The industry literature of the period is replete with examples of this changing attitude toward software management. Even those proposals that seemed to be most explicitly technical, such as those advocating structured programming techniques or high-level language developments, contained a strong managerial component. Most required a rigid division of labor and the adoption of tight management controls over worker autonomy. When a prominent adherent of object-oriented programming techniques spoke of “transforming programming from a solitary cut-to-fit craft, like the cottage industries of colonial America, into an organizational enterprise like manufacturing is today,” he was referring not so much to the adoption of a specific technology but rather to the imposition of established and traditional forms of labor organization and workplace relationships.¹¹ The solutions to the software crisis most frequently recommended by managers—among them the elimination of rule-of-thumb methods (i.e., the black art of programming), the scientific selection and training of programmers, the development of new forms of management, and the efficient division of labor—were not fundamentally different from the four principles of scientific management espoused by Frederick Taylor in an earlier era.¹²

Aristocracy, Democracy, and Systems Design

In practice, software engineering was more an expression of ideals than a well-defined agenda. At best it was a loose collection of techniques, technologies, institutions, and practices.¹³ As Stuart Shapiro has suggested, the essence of the software-engineering movement was control: control over complexity, control over budgets and scheduling, and, perhaps most significantly, control over a recalcitrant workforce.¹⁴ Although a number of technological or procedural innovations were developed to facilitate software engineering—structure programming techniques, the ADA programming language, Computer-Aided Software Engineering (CASE) environments—the focus of most software-engineering efforts were managerial. In this sense, software engineering represents the culmination of the turn toward managerial solutions to the software crisis that characterized the late 1960s.

Unhappy with the ballooning costs of software development, threatened by the growing power of the computer people, and frustrated by the apparent inability of either academic computer science or the professional societies to institute more formal methods for regulating the industry, corporate managers attempted to construct development methodologies that would eliminate the uncertainty and expense associated with computerization projects.

It would be impossible to describe all of the numerous approaches to programmer management that were developed in this and subsequent periods. The remainder of this chapter will focus on the defining characteristics of a few of the most prominent development methodologies that emerged in response to the declaration in 1968 of the software crisis: the hierarchical system, or software factory; the superprogrammer, or chief programmer team (CPT) approach; and the adaptive programmer team (or “egoless” programming) model. The hierarchical systems approach—originally developed for large, government-sponsored programming projects at the SDC and the IBM Federal Systems Division—resembles the highly stratified, top-down organizational structure familiar to most conventional corporate employees. The CPT, although it was also developed at the IBM Federal Systems Division, reflects an entirely different approach to programmer management oriented around the leadership of a single managerially minded superprogrammer. The adaptive team approach was popularized as egoless programming by the iconoclastic Gerald Weinberg in his classic The Psychology of Computer Programming from 1971.¹⁵ Weinberg proposed an open, “democratic” style of management that emphasized teamwork and rotating leadership.

Although it is possible to arrange these approaches into a roughly chronological order, it is not my intention to suggest that they represent any simple evolution toward increasing managerial control or economic efficiency. Each of these management methodologies captures separate but interrelated visions about how computer programming as an economic activity, and computer programmers as aspiring professionals, could best be integrated into the established social and technological systems of the traditional corporation. Each of these approaches built on, and responded to, the innovations and shortcomings of the others. They also reflected the backgrounds and aspirations of their advocates and developers. By studying carefully the salient features of each of these three methodologies, we will be better able to situate them in their particular social and historical context, and hence to understand more fully their contribution to contemporary debates about the nature and causes of the software crisis.

Armies of Programmers

The first concerted attempts to manage software development projects using established management techniques occurred at the government- and military-sponsored SAGE air-defense project. The SAGE project was the heart of an ambitious early warning radar network intended to provide an immediate and centralized response to sneak attacks from enemy aircraft. The plan was to develop a series of computerized tracking and communications centers that would coordinate observation and response data from a widely dispersed system of interconnected perimeter warning stations. First authorized by Congress in 1954, by 1961 the SAGE system had cost more than $8 billion to develop and operate, and required the services of over two hundred thousand employees. The software that connected the specially designed, real-time SAGE computers was the largest programming development then under way. SDC, a RAND Corporation spin-off company responsible for developing this software, had to train and hire almost two thousand programmers. In the space of a few short years the personnel management department at SDC effectively doubled the number of trained programmers in the United States.

In order to effectively organize an unprecedented number of software developers, SDC experimented with a number of different techniques for managing the programming process. For the most part, however, SDC relied on a hierarchical structure that located most programmers at the lowest levels of a vast organizational pyramid built with layer on layer of managers.¹⁶ The top of this hierarchy was occupied by nontechnical administrators. The middle layers were peopled by those EDP personnel who had exhibited a desire or aptitude for management. In other words, the managers in the SDC hierarchy were self-selected as being either uninterested or uncommitted to a long-term programming career. The management style in this hierarchical structure was generally autocratic. Managers made all of the important decisions. They assigned tasks, monitored the progress of subordinates, and determined when and what corrective actions needed to be taken.

This hierarchical approach to management was attractive to SDC executives for a number of reasons. First of all, it was a familiar model for government and military subcontractors. Second, it was often easier to justify billing for a large number of mediocre low-wage employees than a smaller number of excellent but expensive contractors. Finally, and perhaps most important, the “Mongolian horde” approach to software development corresponded nicely with contemporary constructions of the root causes of the burgeoning software turmoil. This was also known as the “Chinese Army” approach, at least until the phrase became unpopular in the early 1950s.

Faced with a shortage of experienced programmers, SDC embarked on an extensive programming of internal training and development. Most of its trainees had little or no experience with computers; in fact, many managers at SDC preferred it that way. Like many corporations in the 1950s, SDC believed that “it is much easier to teach our personnel to program than it is to teach outside experienced programmers the details of our business.”¹⁷ In any case, in the period between 1956 and 1961 the company trained seven thousand programmers and systems analysts. At a time when all the computer manufacturers combined could only provide twenty-five hundred student weeks of instruction annually, SDC devoted more than ten thousand student weeks to instructing its own personnel how to program.¹⁸

The apparent success that SDC achieved in mass-producing programming talent reinforced the notion that a hierarchical approach was the suitable model for large-scale software development. If large quantities of programmers could be produced on demand, then individual programmers were effectively anonymous and replaceable. A complex system like SAGE could be broken down into simple, modular components that could be easily understood by any programmer with the appropriate training and experience. The principles behind the approach were essentially those that had proven so successful in traditional manufacturing: replaceable parts, simple and repetitive tasks, and a strict division of labor.

The hierarchical model of software development was adopted by a number of other major software manufacturers, particularly those involved in similarly large military or government projects. It is not clear how direct the connection was between SDC and these other manufacturers. SDC certainly had a role in training a large number of programmers and EDP managers. “We trained the industry!” boasted SDC veterans: “Whatever company I visit, I meet two or three SDC alumni.”¹⁹ The labor historian Philip Kraft attributes much of what he refers to as the “routinization” of programming labor to the “degrading” influence of military-industrial organizations such as SDC. He describes the SDC so-called software factories as “the first systematic, large-scale effort on the part of EDP users to transform the highly idiosyncratic, artisan-like occupation” of computer programming into “one which more closely resembled conventional industrial work.”²⁰ He argues that SDC played a significant role in diffusing and popularizing the hierarchical approach to software engineering management.

Whether the claim that SDC policies and SDC personnel played a direct role in diffusing the hierarchical system of management throughout the computer industry was valid, similar top-down methodologies were widely adopted. In the IBM Federal Systems Division, a multilevel organizational structure was used on all large government projects. IBM manager Philip Metzger provided a detailed description of the Federal Systems approach in his highly popular textbook Managing a Programming Project, which went through three editions in the period between 1973 and 1996.²¹ An article titled “Issues in Programming Management” that appeared in 1974 in the respected industry newsletter EDP Analyzer listed the hierarchical systems approach as one of the most commonly implemented software management methodologies.²² Joel Aron, another IBM Federal Systems veteran, used the hierarchical model as the basis for his series of books on the Program Development Process.²³ The hierarchical approach to software development was attractive to managers because it corresponded nicely with the contemporary management theories. In the first half of the twentieth century, corporate management became a professional activity dominated by specialists and experts. These professional managers developed a shared culture and value system reinforced by an increasingly formalized program of training and education. They exerted a high degree of control over the work practices of their subordinates, scientifically managing all aspects of the business and manufacturing process. EDP managers assumed that the techniques and structures that appeared to work so efficiently in traditional industries would translate naturally into the software development department. It was only a matter of identifying and implementing the one best way to develop software components.

Embedded in the hierarchical model of management were a series of assumptions about the essential character of programming as an occupational activity. Implied in the suggestion that the structures and procedures of a traditional manufacturing organization could be seamlessly mapped onto the EDP department was a belief that the skills and experience required to program a computer were, in effect, not all that different from those required to assemble an automobile. Managers could define, in the minutest detail, the specifications that the programmers would follow. In turn, the programmers need only be trained to perform a limited and specialized function. Individual programmers were looked on as interchangeable units.²⁴ They lacked a distinct professional identity. The path to advancement in the hierarchical system (if indeed there actually was one available to mere programmers) was through management. Certification programs were desirable in order to ensure a minimum level of competence, but only as a means for assuring a standard degree of performance and product.²⁵ Programmers were encouraged to be professionals only to the extent that being a professional meant self- discipline, a willingness to work long hours with no overtime pay, and loyalty to the corporation and obedience to supervisors.²⁶

The notion that programmers could be treated as unskilled clerical workers was reinforced by a series of technical developments intended to allow managers to mechanically translate high-level systems designs into the low-level machine code required by a computer. For example, one of the alleged advantages of the COBOL programming language frequently touted in the literature was its ability to be read and understood—and perhaps even written—by informed managers.²⁷ More than a fashionable management technique, the hierarchical organizational model was a philosophy about what programming was and where programmers stood in relation to other corporate professionals. It embodied—in a complex of interrelated cultural, technical, and political systems—a particular social construction of the nature and causes of the software crisis.

Despite the obvious appeal that the theory of hierarchical systems held for conventional managers, it rarely worked as intended in actual practice. Although managers would have preferred to think of programming as routine clerical work and programmers as interchangeable laborers, experience suggested that in reality the situation was quite different. I have already described how, in the late 1950s and early 1960s, programming had acquired a reputation as being a uniquely creative activity requiring “real intellectual ability and above average personal characteristics.”²⁸ “To ‘teach’ the equipment, as is amply evident from experience to date, requires considerable skill, ingenuity, perseverance, organizing ability, etc. The human element is crucial in programming.”²⁹ The anecdotal evidence that suggested skilled programmers were essential elements of software development was supported by numerous empirical studies produced by industrial psychologists and personnel experts.

The realization that computer programming was a more intellectually challenging activity than was originally anticipated threw a monkey wrench into the elaborate hierarchical systems that managers had constructed. Whereas the software turmoil of the 1950s was attributed largely to numerical shortages of programmers, the “programmer quality” problems of the 1960s demanded a subtly different construction of the root causes of the software crisis. The problem could still be defined as a management problem requiring a management-driven solution. What had changed was the prevailing conception of what programmers were and what they did. “The massive attack on systems software poses difficult management problems,” concluded Gene Bylinsky in the pages of Fortune magazine. “On the one hand, a good programmer, like a writer or composer, works best independently. But the pressure to turn out operating systems and other programs within a limited time make it necessary to deploy huge task forces whose coordination becomes a monstrous task.” Echoing conventional wisdom about the creative nature of programming, Bylinsky maintained that the problem was “further complicated” by the fact that there is no “best way” to write computer programs. “Programming has nowhere near the discipline of physics, for example, so intuition plays a large part. Yet individual programmers differ in their creative and intuitive abilities.”³⁰

Companies that implemented hierarchical systems methodologies also discovered that programmers were not content with the professional identity that these systems imposed on them. Programmers voted with their feet by leaving for other firms, and salaries inflated dramatically.³¹ One large employer experienced a sustained turnover rate of 10 percent per month.³² The problem, according to one SDC survey of termination interviews, was that programmers working in hierarchical organizations “did not foresee for themselves the opportunities they want for professional growth and development . . . or for promotion and advancement.”³³ The career aspirations of the programmers conflicted with the occupational role they had been assigned by the managers. Many preferred to pursue professional advancement within programming, rather than away from programming. In the hierarchical system, the higher that individuals advanced, the more they worked as administrators rather than technologists.

Superprogrammer to the Rescue

The IBM System/360 has been called “the computer that IBM made, that made IBM.”³⁴ The System/360 systems solved a number of problems for IBM and its customers. It filled in the gaps in the IBM line of product offerings by providing an entire range of hardware- and software- compatible computers ranging from the low-end model 360/20 (intended to compete directly with the Honeywell H-200) to the model 360/90 supercomputer, which compared favorably to the CDC-6600. By making all of these machines software compatible (theoretically, at least), IBM supplied an inexpensive upgrade path for its customers. The client could purchase just the amount of computing power that they needed, knowing that if their needs changed in the future they could simply transfer their existing applications and data to the next level of System/360 hardware. They could also make use of their existing peripherals, such as tape readers and printers, without requiring an expensive upgrade.

The System/360 was an enormously risky and expensive undertaking. The Fortune journalist Tom Wise referred to it as “IBM’s $5,000,000,000 Gamble.” He quoted one senior IBM manager as calling it the “we bet the company” project.³⁵ The riskiest and most expensive component of System/360 development was the OS/360 operating system. As mentioned earlier, in the years between 1963 and 1966, over five thousand staff years of effort went into the design, construction, and documentation of OS/360. When OS/360 was finally delivered in 1967, nine months late and riddled with errors, it had cost the IBM Corporation half a billion dollars—four times the original budget, or “the single largest expenditure in company history.”³⁶

Although the System/360 project turned out to be a tremendous success for IBM, sealing its position of leadership in the commercial computer industry for the next several decades, the OS/360 project was generally considered to be a financial and technological disaster. The costs of the OS/360 debacle were human as well as material; according to Frederick Brooks, they were “best reckoned in terms of the toll it took on people: the managers who struggled to make and keep commitments to top management and to customers, and the programmers who worked long hours over a period of years, against obstacles of every sort, to deliver working programs of unprecedented complexity.” Many in both groups left, victims of a variety of stresses ranging from technological to physical.³⁷

The highly publicized failure of the OS/360 project served as a dramatic illustration of the shortcomings of the hierarchical management method. Techniques that had worked well on an application requiring ten thousand lines of code failed miserably when applied to a million lines of code project. Faced with serious schedule slippages, quality problems, and unanticipated changes in scope, the OS/360 managers did what traditional manufacturing managers were accustomed to doing: they added more resources. The only noticeable result was that the project fell more and more behind schedule.

The Mythical Man-Month was OS/360 project leader Frederick Brooks’s postmortem analysis of the failures of traditional hierarchical management. It is one of the most widely read and oft-quoted references on the practice of software engineering. The mythical man-month in the title refers to the commonly held notion that progress in software development projects occurs as a function of time spent times the number of workers allocated—the implication being that more workers equals faster production. Brooks dismissed this assumption with the now-famous Brooks’s law, one of the most memorable aphorisms in the lore of software development: adding personnel to a late software project makes it later. Or to use one of Brooks’s more earthy metaphors, “the bearing of a child takes nine months, no matter how many women are assigned.”³⁸

The highly quotable Brooks’s law was neither the only nor even the most significant of the insights provided in The Mythical Man-Month. Brooks did more than criticize existing methodologies; he provided an entirely new model for understanding software development management. He was firmly convinced that there was a wide disparity in performance among individual programmers. Brooks believed that small teams of sharp programmers were substantially more productive than much larger groups of merely mediocre performers. But he also recognized that even the best small team could only accomplish so much in any given period of time. The small team approach simply did not scale well to larger projects. The problem of scalability was the heart of the “cruel dilemma” facing project managers: “For efficiency and conceptual integrity, one prefers a few good minds doing design and construction. Yet for large systems one wants a way to bring considerable manpower to bear, so that the product can make a timely appearance.”³⁹ And yet the Mongolian horde model of throwing programming resources—so-called man-months—at projects was also obviously insufficient. What was needed was a way to apply the efficiency and elegance of the small team approach to the problems of large-project management.

Brooks proposed the adoption of what he called the “surgical team” model of software development. In doing so, he borrowed heavily from the work of IBM manager and researcher Harlan Mills, who had earlier developed the CPT concept. This notion was first introduced as one of two experimental superprogrammer projects by Aron in a paper given at a second NATO Software Engineering conference held in Rome in 1969. The first experiment involved a thirty-man-year project requiring fifty thousand instructions. Mills attempted to complete the project himself (using a prototype surgical team) in only six months. The project eventually required about six man-years of effort to complete, and was considered a moderate success. The second experiment mentioned by Aron at the Rome conference turned out to be the famous New York Times project, which established the reputation of the CPT approach when it was publicized by F. Terry Baker in 1971. In both versions of the CPT approach, a single, expert programmer was responsible for all major design and implementation decisions involved with system development. The chief programmer (or surgeon) defined the program specifications, designed the program, coded it, tested it, and wrote the documentation. The chief was assisted in their tasks by an operating team of support staff. Their immediate assistant (or copilot) was only slightly less expert than the chief programmer. The copilot was the chief programmer’s mirror and alter ego, serving not only as an emergency backup or stand-in but also as an adviser, discussant, and evaluator. Although the assistant knew the program code intimately and may even have written some of it, it was the chief programmer who was ultimately responsible for it.

Other members of the Brooks’s surgical team included an administrator, who handled schedules, money, personnel issues, and hardware resources; an editor, who provided the finishing touches to the chief programmer’s documentation; two secretaries, who dealt with correspondence and filing; a program clerk, who maintained all the technical records for the project; a “toolsmith,” who built, constructed, and maintained the interactive tools used by the rest of the team for programming, debugging, and testing; a tester, who served as the chief programmer’s adversary and assistant, developed test plans to challenge the integrity of the program design, and devised test data for day-to-day debugging; and finally, the “language lawyer,” who delighted in the mastery of the intricacies of a programming language. The language lawyer, unlike the chief programmer, was not involved in big-picture issues or system design; the lawyer’s responsibility was finding “neat and efficient ways to use the language to do difficult, obscure, or tricky things.” Language lawyers were usually called in only for special, short-term assignments.⁴⁰

The advantage to the CPT approach, according to Mills and Brooks, was that it dramatically simplified communications between team members. Whereas a large, hierarchical organization of X number of employees could require as many as (X² - X)/2 independent paths of communication, in the CPT model all essential information passed through the person of the chief programmer. All team members reported to the chief directly and did not communicate with each other directly.

By centralizing all decision making in the person of the chief programmer, this approach assured the maintenance of the program’s structural integrity. Brooks compared the conceptual architecture of the typical large software project to the haphazard design of many European cathedrals; the patchwork structure of these cathedrals revealed an unpleasant lack of continuity, reflecting the different styles and techniques of different builders in different generations. Brooks preferred the architectural unity of the cathedral at Reims, which derived “as much from the integrity of design as from any particular excellences.” This integrity was achieved only through the “self-abnegation of eight generations of builders,” each of whom “sacrificed some of his ideas so that the whole might be of pure design.” Using wonderfully evocative biblical language, Brooks extolled the virtues of a unified conceptual design: “As the child delights in his mud pie, so the adult enjoys building things, especially things of his own design. I believe that this delight must be an image of God’s delight in making things, a delight shown in the distinctiveness and newness of each leaf and each snowflake.”⁴¹ Only the CPT approach could guarantee such a degree of uncompromised architectural integrity.

The CPT approach differed from hierarchical systems methodologies in a number of essential characteristics. Whereas the hierarchical model allowed for (and in fact encouraged) the use of novice programmers, the CPT was built entirely around skilled, experienced professionals. This implied a radically different approach to professional development. Each member of the team was encouraged to develop within their own particular disciplinary competency; that is, it wasn’t necessary to become a surgeon to advance one’s career. For example, an aspiring language lawyer could continue to focus on their technical specialty without feeling pressure to transfer into management. The CPT approach embodied the belief that computer programming was a legitimate, respectable profession.

The CPT also reflected changing contemporary notions about the nature of programming ability. The primary justification for using small teams of experienced programmers rather than large hordes of novices was the belief that one good programmer was worth at least ten of their average colleagues. In the person of the chief programmer, the innate technical abilities of the superprogrammer were merged with the organizational authority of the traditional manager. The chief programmer was both a technical genius and expert administrator. Programming aptitude could not be abstracted from its embodiment in particular individuals; skilled programmers were anything but replaceable components of an automated software factory. In the elite surgical team model, the contributions of talented professionals far outweighed those provided by traditional management techniques or development methodologies.

Besides endowing computer programmers with considerable institutional power, The Mythical Man-Month reinforced the notion that programming was an exceptional activity, unlike any other engineering or manufacturing discipline. Brooks’s suggestion that programming was akin to poetry strongly implied that programming was not an activity that could be readily systematized. What Brooks proposed was the adoption of useful tools and techniques, not some overarching methodology. As he later declared in a famous article titled “No Silver Bullet,” although the management of large programming projects could be improved incrementally, there were no easy solutions to be derived from the lessons of traditional manufacturing.⁴²

Like the hierarchical systems model, the CPT was intimately linked to specific techniques and technologies. Since all major decisions relating to both design and implementation had to be made by a single superprogrammer, the CPT approach effectively demanded the adoption of top-down development techniques. Top-down programming was one of the foundational principles of the structured programming approach to software engineering advocated by many academic computer scientists in this period. The essence of top-down programming was the concept of abstraction: by proceeding step by step from general design goals to the specific implementation details, a systems architect could individually manage the otherwise-unmanageable complexity of a large software development project. The use of top-down programming techniques enabled the authoritarian chief programmer to maintain the architectural integrity that Brooks believed was so central to the design of useful and beautiful software programs. The heyday of the structured programming movement was coincident with the publication of The Mythical Man-Month, and the attractiveness of the surgical team approach to management was reinforced by, and helped to reinforce, the popularity of structured programming as a development technology.

In addition to borrowing heavily from the established techniques and technologies of structured programming, the CPT model also helped to define technological innovations of its own. The development support library (DSL) was a system of documents and procedures that provided for the “isolation and delegation” of secretarial, clerical, and machine operations.⁴³ In earlier accounts the DSL is referred to as the programming production library. Basically, the DSL was a set of technologies (including coding sheets, project notebooks, and computer control cards) that facilitated communications within the development team. It was envisioned as a means of further centralizing control in the hands of the chief programmer. “The DSL permits a chief programmer to exercise a wider span of control over the programming, resulting in fewer programmers doing the same job. This reduces communications requirements and allows still more control in the programming. With structured programming, this span of detailed control over code can be greatly expanded beyond present practice; the DSL plays a crucial role in this expansion.”⁴⁴

By providing a core set of public programs and documents that were highly visible to all members of the surgical team, the DSL was supposed to discourage the “traditional ad hoc mystique” associated with conventional craft-oriented programming.⁴⁵ The chief programmer could read, understand, and validate all of the work done by their subordinates. The technology of the DSL was clearly intended to reinforce a conventional management agenda: the transfer of control over the work practices of programmers into the hands of the managerial superprogrammer. In language remarkably reminiscent of the “head versus hand” dialectic emphasized by Karl Marx and his disciples, one proponent of the CPT approach described the DSL as having been “designed to separate the clerical and intellectual tasks of programming.”⁴⁶

Although the CPT received much attention in the industry literature, it does not seem to have been widely or successfully implemented.⁴⁷ The original concept had been popularized by Baker in a series of articles documenting the successful implementation of the approach by Mills. Mills had been the chief programmer in a team that developed a computerized information bank application for the New York Times. He claimed to accomplished in twenty-two months what a traditionally, hierarchically managed group would have required at least several more years of calendar time to develop. Baker’s favorable reports on the New York Times project, which involved eighty-three thousand lines of code and eleven man-years of effort, convinced many computer professionals of the scalability of the CPT approach. The project was portrayed as having high productivity and low error rates, although questions later arose about the accuracy of Baker’s assessment; Mills’s system eventually proved unsatisfactory and was replaced with a less ambitious system.⁴⁸ For the time being, however, the New York Times system was considered to be proof positive of the efficiency of the CPT approach.

Several objections to the CPT approach were raised in the contemporary industry literature, though. The first is that it was difficult to find individuals with enough talent and energy to fulfill all of the functions required of the chief programmer.⁴⁹ The few who did exist were expensive, and were not interested in working on small computers and mundane applications. A second problem was a perceived overdependence on key individuals implied in the CPT approach: “What happens if [our superprogrammer] snaps up a more lucrative offer elsewhere? He’ll likely take our back-up programmer with him, leaving us high-and-dry.”⁵⁰ A number of observers suggested that the surgical team model led to excessive specialization.⁵¹ The computer scientist C.A.R. Hoare derided the small-team approach as a retreat “to the age of the master craftsman—more fashionably known as a chief programmer.”⁵² There were widespread doubts about the ability of the small-team approach to scale up to the needs of large development efforts.

The most revealing criticisms of the CPT system, however, had to do with the ways in which the presence of an elite administrator/programmer disrupted existing patterns of managerial authority: “The CPT perpetuates the prima donna image of the programmer. Instead of bringing the programmer into the organization’s fold, it isolates and alienates him by encouraging the programmer to strive for a superhero image.”⁵³ The CPT allowed for little participation by nontechnical administrators. A textbook, Managing Software Development and Maintenance, from 1981 corrected this perceived overdependence on technical personnel by proposing a revised chief programmer team (RCPT) in which “the project leader is viewed as a leader rather than a ‘super-programmer.” Whereas the chief programmer was clearly a technical specialist, the project leader was “an expert conceptualizer, designer, and project manager”—but not necessarily a superprogrammer. Because the project leader possessed both project management and technical skills, they were “able to direct, oversee, and review all technical functions.”⁵⁴

The RCPT approach was clearly intended to address a concern faced by many traditionally trained department-level managers—namely, that top executives had “abdicated their responsibility and let the ‘computer boys’ take over.”⁵⁵ As was described in chapter 7, it was this fear of the loss of control over valuable occupational territory that most determined contemporary reactions to proposed managerial solutions to the software crisis.

Computer Programming as a Human Activity

The hierarchical model unapologetically attempted to make programmers’ work as routine and mechanical as possible; the CPT provided a real creative outlet for a single superprogrammer only. For moderately skilled programmers attempting to establish for themselves a legitimate professional identity that would provide them with autonomy and status, both models were equally uninviting. What was needed was an alternative organizational model that could simultaneously support two seemingly contradictory agendas: increased managerial control over the “irrational” programming process, and ongoing support for the independent professional authority of programmers.

In 1969, the programmer and computing consultant Weinberg published The Psychology of Computer Programming. The book claimed to present the first detailed empirical study of computer programming as a complex human activity, and indeed, although Weinberg was neither a psychologist nor an ethnographer, his observations appear to be remarkably accurate and insightful. At the very least his work was well received by practitioners, whose personal experiences seem to have resonated with the anecdotes provided by Weinberg. The Psychology of Computer Programming has been widely cited as an accurate description of what really went on in actual programming projects.

Weinberg’s book did more than simply portray existing attitudes and practices, though. It also proposed a new method for organizing and managing teams of software developers. The problem with existing hierarchical methods of software production, according to Weinberg, was that they encouraged programmers to become “detached” from the social environment—and overly possessive of their software. When programmers invest so much of themselves in their programs, Weinberg suggested, they lose the ability to evaluate their creations objectively. The immediate result was bad software—and ultimately a software crisis. “Programmers, if left to their own devices, will ignore the most glaring errors in their output—errors that anyone else can see in an instant.” The solution to the crisis provoked by “property-oriented” programming, argued Weinberg, was the adoption of the “egoless programming team,” in which every programmer is equal, and where all of the code is “attached” to the team rather than to the individual.⁵⁶ By opening up the programming process to self-reflection and criticism, the egoless (or adaptive) programming model would increase efficiency, eliminate errors, and enhance communication—all without inhibiting the creative abilities of programmers.

Although egoless programming represented a relatively radical departure from traditional software development methodologies, it was predicated on fairly conventional notions about the nature of programming ability. For Weinberg, there was little doubt that the majority of people in programming were detached personality types who preferred to be left to themselves. This tendency toward detachment was reinforced “both by personal choice and because hiring policies for programmers are often directed toward finding such people.”⁵⁷ This detachment from people often led programmers to become excessively attached to their products. The “abominable practice” of attaching their names to their software (as in Jules’ Own Version of the International Algebraic Language, better known as the JOVIAL programming language) offered evidence of the programmer’s inability to disassociate themselves from their creations. The JOVIAL programming language was created for the U.S. Air Force in the late 1950s by the SDC. As it was to be a variant of the International Algebraic Language (eventually renamed ALGOL), it was suggested that it be called OVIAL (Our Own Version of the International Algebraic Language), but since OVIAL apparently had “a connotation relative to the birth process that did not seem acceptable to some people,” the name was soon changed to JOVIAL. It was later decided that the J in JOVIAL would stand for Jules Schwartz, one of the programmers involved in the project. Hence, Jules’ Own Version of the International Algebraic Language. This proprietary sense of ownership on the part of the creator was not necessarily an unusual or even undesirable tendency; after all, artists “owned” paintings, authors “owned books,” and architects “owned” buildings. In many cases these attributions led to the admiration and emulation of good workers by lesser ones. What was different about computer programs, however, was that they were owned exclusively by their creators. Good programs, unlike good literature, were never read by anyone other than the author. Thus, according to Weinberg, “the admiration of individual programmers cannot lead to an emulation of their work, but only to an affectation of their mannerisms.”⁵⁸ Junior programmers were unable to benefit from the wisdom and experience of their superiors. The only thing available to emulate was their mannerisms. The result was the perpetuation of bad work habits and personal eccentricities—“the same phenomenon we see in ‘art colonies,’ where everyone knows how to look like an artist, but few, if any, know how to paint like one.”⁵⁹

Weinberg believed that the use of small, unstructured programming teams and regular code reviews would alleviate the problem of programmer attachment. Each of the programmers in the group would be responsible for reading and reviewing all of the application code. Errors that were identified during the process were simply “facts to be exposed to investigation” with an eye toward future improvement, rather than personal attacks on an individual programmer.⁶⁰ By restructuring the social environment of the workplace and thereby restructuring the value system of the programmers, the ideal of egoless programming would be achieved. The result would be an academic style, peer-review system that would encourage high standards, open communication, and ongoing professional development. Junior programmers would be exposed to good examples of programming practice, and more senior developers could exchange subtle tricks and techniques. A piece of completed code would not be considered the product of an individual team member but rather of the team as a whole. The openness of this process would also encourage the development of proper documentation.

There were a number of other salient features of the egoless (or adaptive) programming team that differed from conventional team-oriented approaches. The most unusual and significant was that all major design and implementation decisions were to be determined by consensus instead of decree. There were no assigned team leaders, at least not in the conventional sense. Leadership shifted between team members based on the needs of the moment and the strength of the individual team members (hence the term adaptive). For example, if a particular phase of the project involved a lot of debugging, one of the team members especially skilled at debugging might assume the temporary role of team leader during that period. Even then, all of the important decisions would be made democratically. Work was assigned based on the strengths—and preferences—of the individual team members.

The democratic approach to software project management, in Weinberg’s view, offered a number of advantages. It encouraged communication and flexibility. Schedule and design changes could be more readily accommodated, and resources could be allocated efficiently. Second, the lack of a formal hierarchy made the adaptive team significantly more robust than more structured alternatives. For instance, the adaptive team could readily adjust to the addition or removal of members. The success of the project would no longer hinge on the presence of any one particular individual. In an era in which the performance of programmers was believed to vary dramatically from programmer to programmer, and when turnover in the software industry averaged upward of 25 percent annually, this was an appealing benefit. Last but not least, the social dynamics of the democratically managed adaptive team appeared to correspond well with the actual experiences and expectations of the average working programmer.⁶¹ Weinberg provided a great deal of anecdotal evidence suggesting that programmers worked best in environments in which they participated in all aspects of project development, from design to implementation to testing. By eliminating the things that caused programmers to become dissatisfied, turnover could be reduced significantly. The adaptive team approach to programming, argued Weinberg, was not only cost-effective and efficient; it kept the programmers happy. And of course, happy programmers were productive programmers.

Like the CPT and the hierarchical system of management, egoless programming constituted a solution to a specific conception of the burgeoning software crisis. The advocates of the adaptive team approach shared with many of their contemporaries certain basic assumptions about the nature of programming as a skill and activity: that programming was an essentially creative undertaking; that individual programmers varied enormously in terms of style and productivity; and that current programming practices resembled craft more than they did science. They also believed that despite these exceptional characteristics, software development was an activity that could, to a certain extent, be managed and controlled. What was unusual about the adaptive team solution was the degree to which it offered computer programmers a legitimate career path and an attractive professional identity.

In the hierarchical system of management, programmers were generally regarded as technicians rather than professionals. The few programmers who rose through the hierarchy did so by abandoning their technical interests in favor of managerial careers. The CPT offered status and authority only to a small corps of elite superprogrammers. All but the most talented individuals served as much less privileged support personnel. As will be seen, many programmers were extremely concerned with issues of professional development, both as they related to themselves as individuals and to their larger disciplinary community. The journal articles, job advertisements, and letters to the editor from this period show that many programmers were worried about becoming dead-ended in purely technical positions. Hierarchical organizations and CPTs did not offer them an attractive model of professionalization.

The adaptive team approach, in comparison, offered promising career opportunities to a wide range of software workers. The goal of the adaptive team was to foster a family atmosphere in which every member’s contributions were important. Team members were anything but interchangeable units. Programmers could cultivate their technical skills and advance their careers without feeling pressure to transfer into administration. As one knowledgeable observer suggested, in the adaptive team approach “a good programmer does not get further and further away from programmers, as occurs in a hierarchical structure when he moves up the management ladder. Instead, he stays with programming and gravitates toward what he does best.”⁶²

Judging from the response it received in the industry literature, The Psychology of Computer Programming appealed to a broad popular audience.⁶³ Weinberg’s anecdotes about the real-life work habits of programmers rang true to many practitioners. His descriptions of the mischievous pranks that programmers played on their managers, for example, or the social significance of a strategically located Coca-Cola dispenser, captured for many of his readers the essential character of the programming profession. The book has remained in continuous publication since 1969, and was widely celebrated as one of the few classic texts in the programming literature.⁶⁴ Weinberg presented a romantic portrait of software development that emphasized the quiet professionalism of skilled, dedicated programmer-craftspeople. Of the many models for software engineering that were proposed in the late 1960s and early 1970s, the egoless programmer was by far the most attractive to the average practitioner.

Yet the popularity of egoless programming extended beyond the community of practitioners. Weinberg’s theories about the efficiency of small family work groups and bottom-up, consensus decision making resonated with certain popular contemporary management theories. In 1971, Antony Jay’s Corporation Man provided an ethological analysis of “tribal behavior” in modern corporations that reinforced Weinberg’s conclusion that six- to ten-member teams were a “natural” organizational unit.⁶⁵ Douglas McGregor’s The Human Side of Enterprise discriminated between the Theory X approach to management, which assumed that because of their innate distaste for regimented labor, most employees must be controlled and threatened before they would work hard enough, and the Theory Y belief that the expenditure of physical and mental effort in work is as natural as play or rest, and that the average person learns, under proper conditions, not only to accept but to seek responsibility.⁶⁶ For the supporters of Theory Y management, Weinberg’s adaptive team represented an exemplary model of the participative problem-solving approach.⁶⁷

The concept of egoless programming was rarely adopted in toto, though. In later descriptions of the chief programming team, Baker and Mills claimed that their system represented a form of egoless programming, in the sense that the code produced by the chief programmer was open for inspection by other members of the surgical team. By this point, egoless programming was interpreted by many managers in terms very favorable to management: it meant that programmers should not be defensive about code reviews, task assignments, and other management-imposed structures. The adaptive team terminology in this case seems to have been adopted for public relations purposes only. The whole point of the chief programming team was to consolidate all aspects of design and implementation into the hands of a single superprogrammer. It would have been impossible to maintain the level of architectural integrity desired by Brooks if the chief programmer were not heavily invested in their own individual conceptual structure.

Indeed, by the mid-1970s the language of egoless programming appears to have been almost entirely transformed and co-opted by conventional managers. These managers picked up on the idea that requiring programmers to develop open, nonpropriety code allowed for increased administrative oversight. To them, egoless programming meant that “all programmers were to adhere to rules that would make their products understandable to others and make the individual programmer replaceable.”⁶⁸ Weinberg’s original intention that egoless programming would enable programmers to develop as autonomous professionals appears to have gone entirely by the wayside. One management consultant reminded his audience that managers should “stress the non-punitive nature of the new approaches. Egoless programming is designed to help the programmer, not point out his faults.”⁶⁹ The not-so-subtle subtext of this reminder is that by this period, egoless programming had acquired a reputation for being worker-hostile management jargon.

Although The Psychology of Computer Programming received a great deal of popular attention for its descriptive verisimilitude, it was less successful in its prescriptive capacity. Weinberg’s recommendations do not appear to have been taken seriously by many academic or industry leaders. It may be that his adaptive teams did not scale well to large development efforts, and were used in nothing but small local projects. They may have proven inefficient or difficult to implement, although there is evidence that the use of informal, unstructured programming teams was standard practice in the industry. At least one author rejected the adaptive team approach because it failed to provide adequate mechanisms for formal managerial control.⁷⁰ It seems likely that this last objection was what ultimately proved fatal to Weinberg’s proposal. The adaptive team approach reinforced the notion that programmers were independent professionals. It shifted organizational control and authority away from managers. It ceded valuable occupational territory to a group whose institutional power base had not yet been firmly established. Weinberg’s adaptive teams were unappealing to everyone but programmers, and programmers did not have the leverage to push through such an unpopular agenda.

From Exhilaration to Disillusionment

The 1968 NATO Conference on Software Engineering was, according to contemporary accounts, an exhilarating experience for many participants. The public acknowledgment of a perceived software crisis was a cathartic moment for the industry. As one prominent computer scientist described it, “The general admission of the software failure in this group of responsible people is the most refreshing experience that I have had in a number of years, because the admission of shortcomings is the primary condition for improvement.”⁷¹ Despite the general recognition of impending crisis, the spirit of the conference was “positive, even liberatory.”⁷² Attendees rallied behind the organizers’ call for “a switch from home-made software to manufactured software, from tinkering to engineering.”⁷³ Software engineering emerged as the dominant rhetorical paradigm for discussing the future of software development. By adopting the “types of theoretical foundations and practical disciplines that are traditional in the established branches of engineering,” computer programming could be successfully transformed from a black art into an industrial discipline. Software workers from a wide variety of disciplines and backgrounds adopted the rhetoric of software engineering as a shared discourse within which to discuss their mutual professional aspirations.

In order to capitalize on the enthusiasm generated in the wake of the Garmisch meeting, the NATO Science Committee quickly organized a second conference to be held the following year in Rome. The Rome conference in 1969 was intended to have an explicitly practical focus: the goal was to develop specific techniques of software engineering. As with the Garmisch meeting, a deliberate and successful attempt was made to attract a wide range of participants. The resulting conference, however, bore little resemblance to its predecessor. Whereas the Garmisch participants had coalesced around a shared sense of urgency, the Rome conference was characterized by conflict. According to the same observer who had referred glowingly to the Garmisch conference as a “most refreshing experience,” the discussions at the Rome meeting were “sterile,” the various groups of attendees “never clicked,” and “most participants” left feeling “an enormous sense of disillusionment.”⁷⁴ A prolonged debate about the establishment of an international software engineering institute proved so acrimonious and divisive that it was omitted from the conference proceedings: “All I remember is that it ended up being a lot of time wasted, and no argument ever turned up to make something happen—which is probably just as well.”⁷⁵

Why was the Rome conference considered to be such a disappointment relative to Garmisch? Many of the same participants had attended both meetings; there had been no significant changes in terms of demographic makeup or organizational structure. Neither were there any major new issues or technologies introduced or discussed. Many of the Rome presentations covered material that had previously been addressed, albeit at a less detailed and technical level, at Garmisch. And yet while the Garmisch conference is widely considered to have marked a pivotal moment in the history of software development—“a major cultural shift in the perception of programming”—the Rome one seems to have been deliberately forgotten.⁷⁶

One obvious difference between the two events is that the earlier conference had encouraged participants to focus their attention on a commonly perceived but vaguely defined emergency, while the latter forced them to deal with specific controversial issues. Software engineering had emerged as a compelling solution to the software crisis in part because it was flexible enough to appeal to a wide variety of computing practitioners. The ambiguity of concepts such as professionalism, engineering discipline, and efficiency allowed competing interests to participate in a shared discourse that nevertheless enabled them to pursue vastly different personal and professional agendas. Industry managers adopted a definition of professionalism that provided for educational and certification standards, a tightly disciplined workforce, and increased corporate loyalty. Computer manufacturers looked to engineering discipline as a means of countering charges of incompetence and cost inefficiency. Academic computer scientists preferred a highly formalized approach to software engineering that was both intellectually respectable and theoretically rigorous. Working programmers tended to concentrate on the more personal aspects of professional accomplishment, including autonomy, status, and career longevity. The software engineering model seemed to offer something for everyone: standards, quality, academic respectability, status, and autonomy. As Michael Mahoney has suggested, software engineering “was not coined to characterize an ongoing activity but rather to express a desire for one. By 1967, when the computer industry was less than twenty years old, people felt the need for software engineering, even if they were not sure what it was.”⁷⁷

Yet the rhetorical flexibility that had served the consensus-seeking Garmisch participants proved unwieldy when it came to establishing specific standards and practices. The Rome conference illuminated in sharp relief the vast differences that existed between competing visions for the software engineering discipline. Unlike the conflict between workers and managers described in the previous chapter, these divisions were largely internal to the programming community. The primary split was between academic computer scientists and commercial software developers. The industry programmers resented being invited to Rome “like a lot of monkeys to be looked at by theoreticians”; the theoreticians complained of feeling isolated, of “not being allowed to say anything.”⁷⁸ As the editors of the conference proceedings have pointed out, the “lack of communication between different sections of the participants” became the “dominant feature” of the meeting.⁷⁹ “The seriousness of this communications gap,” and the realization that it “was but a reflection of the situation in the real world,” caused the gap itself to become a major topic of discussion.⁸⁰ It was to remain an issue of central concern to the programming community for the next several decades.

Indeed, in the years after 1968 the rhetoric of the software crisis became even more heated. In 1987 the editors of Computerworld complained that “the average software project is often one year behind plan and 100% over budget.”⁸¹ In 1989 the House Committee on Science, Space, and Technology released a report highly critical of the “shoot-from-the-hip” practices of the software industry. Among other things, the report called for a professional certification program for programmers. The thirty-three-page report, “Bugs in the Program: Problems in Federal Government Computer Software Development and Regulation,” was written by staff members James H. Paul and Gregory C. Simon of the Subcommittee on Investigations and Oversight of the House Committee on Science, Space, and Technology.⁸² Later that same year the Pentagon launched a broad campaign to “lick its software problems” that included funds for a Software Engineering Institute and the widespread adoption of the ADA programming language. ADA was touted by Department of Defense officials as “a means of replacing the idiosyncratic ‘artistic’ ethos that has long governed software writing with a more efficient, cost-effective engineering mind-set.”⁸³ The list of critical reports, denunciations of current practices, and proposed silver-bullet solutions goes on and on. In his 1996 summary of the legacy the first NATO Conference, W. Wayt Gibbs suggested that “a quarter of a century later software engineering remains a term of aspiration,” rather than a real accomplishment: “The vast majority of computer code is still handcrafted from raw programming languages by artisans using techniques they neither measure nor are able to repeat consistently.”⁸⁴