From the outset I have emphasized that the underlying philosophical framework that has guided many of the arguments presented in this book is based on a paradigm inspired by a physics perspective. Consequently, a major theme has been to explore the extent to which a quantitative, predictive understanding based on underlying generic principles that transcend the details of any particular system can be developed. A fundamental tenet of science is that the world around us is ultimately governed by universal principles, and it is in this context that scaling laws for highly complex systems such as organisms, cities, or companies should be viewed. As I have tried to demonstrate, scaling laws reflect systematic regularities that reveal underlying geometric and dynamical behaviors, suggesting that a possible quantitative science of such systems might be attainable. At the very least, they allow us to explore just how far such a paradigm can be pushed.
The quest for grand syntheses, for commonalities, regularities, ideas, and concepts that transcend the narrow confines of specific problems or disciplines, is one of the great inspirational drivers of science and scientists. Arguably, it is also a defining characteristic of Homo sapiens, manifested in our multitudinous creeds, religions, and mythologies that help us come to terms with the awesome mysteries of the universe. This search for synthesis and unification has been a major theme in science since its origins in early Greek thinking, which introduced concepts such as atoms and elements as fundamental building blocks from which everything else is constructed.
Among the classic grand syntheses in modern science are Newton’s laws, which taught us that heavenly laws are no different from those on Earth; Maxwell’s unification of electricity and magnetism, which brought the ephemeral ether into our lives and gave us electromagnetic waves; Darwin’s theory of natural selection, which reminded us that we’re just animals and plants after all; and the laws of thermodynamics, which suggest that we can’t go on forever. Each of these has had profound consequences not only in changing the way we think about the world, but also in laying the foundations for technological advancements that have led to the standard of living many of us are privileged to enjoy. Nevertheless, they are all to varying degrees incomplete. Indeed, understanding the boundaries of their applicability, the limits to their predictive power, and the ongoing search for exceptions, violations, and failures have provoked even deeper questions and challenges, stimulating the continued progress of science and the unfolding of new ideas, techniques, and concepts.
One of the great ongoing scientific challenges that has dominated modern physics is the search for a grand unified theory of the elementary particles and their interactions, including its extension to understanding the cosmos and even the origin of space-time itself. Such an ambitious theory would be conceptually based on a parsimonious set of underlying mathematizable universal principles that integrate and explain all of the fundamental forces of nature from gravity and electromagnetism to the weak and strong nuclear forces, incorporating Newton’s laws, quantum mechanics, and Einstein’s general theory of relativity. Fundamental quantities like the speed of light, the four dimensions of space-time, and the masses of all of the elementary particles would all be explained, and the equations governing the origin and evolution of the universe through to the formation of galaxies and down to the planetary level including life itself would be derived. It is a truly remarkable and enormously ambitious quest that has occupied thousands of researchers for almost one hundred years at a cost of billions of dollars. Measured by almost any metric this ongoing quest, which is still far from its ultimate goal, has been enormously successful, leading, for example, to the discovery of quarks as the fundamental building blocks of matter, the Higgs particle as the origin of mass in the universe, and to black holes and the Big Bang . . . and to many Nobel Prizes.1
Emboldened by its great success, physicists endowed this fantastic vision with the grand title of the Theory of Everything. Demanding mathematical consistency between quantum mechanics and general relativity suggested that the basic building blocks of this universal theory might be microscopic vibrating strings rather than the traditional elementary point particles upon which Newton and all subsequent theoretical developments were based. Consequently this vision took on the more prosaic subtitle “string theory.” Like the invention of gods and God, the concept of a Theory of Everything connotes the grandest vision of all, the inspiration of all inspirations, namely that we can encapsulate and understand the entirety of the universe in a small set of precepts, in this case, a concise set of mathematical equations from which literally everything follows. Like the concept of God, however, it is potentially misleading and intellectually dangerous.
Referring somewhat hyperbolically to a field of study as the Theory of Everything connotes a certain degree of intellectual arrogance. Is it really conceivable that there is a master equation that encapsulates everything about the universe? Everything? Where’s life, where are animals and cells, brains and consciousness, cities and corporations, love and hate, your mortgage payments, this year’s presidential elections, et cetera? How in fact does the extraordinary diversity, complexity, and messiness that we all participate in here on Earth arise? The simplistic answer is that these are inevitable outcomes of the interactions and dynamics encapsulated in this grand Theory of Everything. Even time itself is presumed to have emerged from the geometry and dynamics of these vibrating strings. Following the Big Bang the universe expanded and cooled and this resulted in the sequential hierarchy from quarks to nucleons, thence to atoms and molecules, and ultimately to the complexity of cells, brains, and emotions and all the rest of life and the cosmos to come tumbling out, a sort of deus ex machina. All of this metaphorically as a consequence of turning the crank of increasingly complicated equations and computations presumed, at least in principle, to be soluble to any sufficient degree of accuracy. Qualitatively, this extreme version of reductionism may indeed have some partial validity, though I’m not sure to what extent anyone actually believes it—but, in any case, something is missing.
The “something” includes many of the concepts and ideas implicit in a lot of the problems and questions considered in this book: concepts like information, emergence, accidents, historical contingency, adaptation, and selection, all characteristics of complex adaptive systems whether organisms, societies, ecosystems, or economies. These are composed of myriad individual constituents or agents that take on collective characteristics that are generally unpredictable in detail from their underlying components even if the dynamics of their interactions are known. Unlike the Newtonian paradigm upon which the Theory of Everything is based, the complete dynamics and structure of complex adaptive systems cannot be encoded in a small number of equations. Indeed, in most cases, probably not even in an infinite number. Furthermore, predictions to arbitrary degrees of accuracy are not possible, even in principle.
On the other hand, as I have tried to show throughout the book, scaling theory provides a powerful tool for forging a middle ground in which a quantitative framework can be developed for understanding and predicting the coarse-grained behavior of many broad aspects of such systems.
Perhaps, then, the most surprising consequence of a visionary Theory of Everything is that it implies that on the grand scale the universe, including its origins and evolution, though extremely complicated, is not complex but in fact is surprisingly simple because it can be encoded in a limited number of equations, conceivably even just a single master equation. This is in stark contrast to the situation here on Earth, where we are integral to some of the most diverse, complex, and messy phenomena that occur anywhere in the universe, and which require additional, possibly nonmathematizable concepts to understand. So while applauding and admiring the search for a grand unified theory of all the basic forces of nature, we should recognize that it cannot literally explain and predict everything.
Consequently, in parallel with the quest for the Theory of Everything, we need to embark on a similar quest for a grand unified theory of complexity. The challenge of developing a quantitative, analytic, principled, predictive framework for understanding complex adaptive systems is surely one of the grand challenges for twenty-first-century science. As a vital corollary to this and of greater urgency is the need to develop a grand unified theory of sustainability in order to come to terms with the extraordinary threats we now face. Like all grand syntheses, these will almost certainly remain incomplete, and very likely unattainable, but they will nevertheless inspire significant, possibly revolutionary new ideas, concepts, and techniques with implications for how we move forward and whether what we have thus far achieved can survive.
Although such a vision may not be explicitly articulated in such grandiose terms, it does encapsulate what the Santa Fe Institute was founded to address. It’s a remarkable place. Maybe not everybody’s cup of tea, but for many of us who still harbor a naive, possibly romantic image of wanting to be part of an eclectic community of scholars searching for “truth and beauty”—and having been disappointed that we didn’t find it in a classic university setting—SFI comes the nearest we’re likely to get to realizing it. I feel extraordinarily fortunate and privileged to have been able to spend a number of very productive years in such a marvelous place, stimulated by like-minded colleagues from every possible corner of academia.
The ambience and character of SFI are perhaps best captured by the British science writer John Whitfield, who in 2007 wrote:
The institute was intended to be truly multidisciplinary—it has no departments, only researchers. . . . Santa Fe and complexity theory have become almost synonymous . . . the institute, now situated on a hill on the town’s outskirts, must be one of the most fun places to be a scientist. The researchers’ offices, and the communal areas they spill into for lunch and impromptu seminars, have picture windows looking out across the mountains and desert. Hiking trails lead out of the car park. In the institute’s kitchen, you can eavesdrop on a conversation between a paleontologist, an expert on quantum computing, and a physicist who works on financial markets. A cat and a dog amble down the corridors and in and out of offices. The atmosphere is like a cross between the senior common room of a Cambridge college and one of the West Coast temples of geekdom, such as Google or Pixar.
The italics in the last sentence are mine; I added them because I think that Whitfield really got this singular combination of characteristics right: on the one hand, the ivory tower image of an Oxford or Cambridge college, a community of scholars devoted to the pursuit of knowledge and understanding for “its own sake” by following their proverbial noses wherever it might take them; on the other, the cutting-edge image of Silicon Valley grappling with problems of the “real” world, seeking innovative solutions and new ways of tackling the complexities of life. Although SFI is a classic basic research institute in that it is not driven by programmatic or applied agendas, the very nature of the problems it addresses necessarily bring many of us face-to-face with major societal issues. As a consequence, in addition to its academic network of scholars, the institute also has a very active business network (called the Applied Complexity Network) comprising diverse companies, some small and incipient, but many of them large and well-known corporations spanning the spectrum of business activities.
SFI occupies a unique place in the academic landscape. Its mission is to address fundamental problems and big questions across all scales at the cutting edge of science with a bias toward quantitative, analytic, mathematical, and computational thinking. There are no departments or formal groups, but rather a culture dedicated to facilitating long-term, creative, transdisciplinary research across all fields, ranging from the mathematical, physical, and biomedical sciences to the social and economic. It has a small resident faculty (but no tenure) and about one hundred external faculty whose major appointment is elsewhere and who spend varying periods of time ranging from a day or two to several weeks in residence. In addition, there are postdoctoral fellows, students, journalism fellows, and even writers. It supports lots of working groups and workshops, seminars and colloquia, and hosts a huge flux of visitors (several hundred a year). What a fantastic melting pot. There is almost no hierarchy, and its size is sufficiently small that everyone on-site can easily get to know everyone else; the archaeologist, economist, social scientist, ecologist, and physicist all freely interact on a daily basis to talk, speculate, bullshit, and seriously collaborate on questions big and small.
The philosophy of the institute stems from the underlying assumption that if you bring smart people together in a supportive, facilitative, dynamic environment that lets them freely interact, good things will inevitably result. The SFI culture is designed to create an open catalytic atmosphere where interactions and collaborations that are often difficult to promote within the traditional departmental structure of universities are strongly encouraged. Bringing together highly diverse minds prepared to engage in substantive, in-depth collaboration in the search for underlying principles, commonalities, simplicity, and order in highly complex phenomena is the hallmark of SFI science. In a curious sense the institute is an instantiation of the very thing it studies: a complex adaptive system.
The institute has been internationally recognized as “the formal birthplace of the interdisciplinary study of complex systems” and has played a central role in recognizing that many of the most challenging, exciting, and profound questions facing science and society lie at the boundaries between traditional disciplines. Among these are the origins of life; the generic principles of innovation, growth, evolution, and resilience whether of organisms, ecosystems, pandemics, or societies; network dynamics in nature and society; biologically inspired paradigms in medicine and computation; the interrelationship between information processing, energy, and dynamics in biology and society; the sustainability and fate of social organizations; and the dynamics of financial markets and political conflicts.
Having had the great privilege of serving as president of SFI for a few years, I obviously have a somewhat biased view regarding its philosophy, its standing, and its successes. So lest you think this is entirely hyperbole on my part, let me give you a couple of other comments and opinions regarding its character. Rogers Hollingsworth is a distinguished social scientist and historian at the University of Wisconsin, well known for his in-depth investigations into what the essential ingredients are that make research groups successful. In addressing a subcommittee of the National Science Board (it oversees the National Science Foundation) charged with reviewing “transformational” science, he remarked:
My colleagues and I have studied approximately 175 research organizations on both sides of the Atlantic, and in many respects the Santa Fe Institute is the ideal type of organization which facilitates creative thinking.
And here’s a quote from Wired magazine:
Since its founding in 1984, the nonprofit research center has united top minds from diverse fields to study cellular biology, computer networks, and other systems that underlie our lives. The patterns they’ve discovered have illuminated some of the most pressing issues of our time and, along the way, served as the basis for what’s now called the science of complexity.
The institute was originally conceived by a small group of distinguished scientists, including several Nobel laureates, most of whom had some association with Los Alamos National Laboratory. They were concerned that the academic landscape had become so dominated by disciplinary stovepiping and specialization that many of the big questions, and especially those that transcend disciplines or were perhaps of a societal nature, were being ignored. The reward system for obtaining an academic position, for gaining promotion or tenure, for securing grants from federal agencies or private foundations, and even for being elected to a national academy, was becoming more and more tied to demonstrating that you were the expert in some tiny corner of some narrow subdiscipline. The freedom to think or speculate about some of the bigger questions and broader issues, to take a risk or be a maverick, was not a luxury many could afford. It was not just “publish or perish,” but increasingly it was also becoming “bring in the big bucks or perish.” The process of the corporatization of universities had begun. Long gone were the halcyon days of polymaths and broad thinkers like Thomas Young or D’Arcy Thompson. Indeed, there were now scant few broad intradisciplinary thinkers, let alone interdisciplinary ones, who were comfortable articulating ideas and concepts that transcended their own fields and potentially reach across to foreign territory. It was to combat this perceived trend that SFI was created.
The early discussions of what the actual scientific agenda of the institute might be centered on the burgeoning fields of computer science, computation, and nonlinear dynamics, areas where Los Alamos had played a seminal role. Enter the theoretical physicist Murray Gell-Mann. He realized that all of these suggestions revolved more around techniques rather than ideas and concepts and that if such an institute was to have a major impact on the course of science, its agenda would have to be broader and bolder and address some of the big questions. Whence arose the idea of complexity and complex adaptive systems as overarching themes, since they encompass almost all of the major challenges and big questions facing science and society today—and, furthermore, they invariably cross traditional disciplinary boundaries.
An interesting sign of the times, and I would argue a significant indicator of the impact that SFI has had, is that many institutions nowadays promote themselves as being multidisciplinary, transdisciplinary, cross-disciplinary, or interdisciplinary. Although such designations have to some extent been co-opted as buzzwords to describe any collaborative interaction across subfields within traditional disciplines, rather than being reserved for bold leaps across the vast divides that separate them, it does represent a significant change in image and attitude. This is infecting all of academia and is now almost taken for granted despite the reality that universities are to varying degrees as stovepiped as ever. Here’s a quote from the Web pages of Stanford University as it rebrands itself in this image, even claiming that it has always operated in this mode:
Since its founding, Stanford University has been a pioneer in cross-disciplinary collaboration . . . producing innovative basic and applied research in all fields. . . . This naturally facilitates multidisciplinary collaboration.
To give you a sense of the extraordinary shift in perception that has occurred over just the last twenty years, here’s an anecdote from the early days of SFI.
Among its founding fathers were two other major figures of twentieth-century academia, both Nobel laureates: Philip Anderson, a condensed matter physicist from Princeton University who had worked on superconductivity and was an inventor, among many other things, of the mechanism of symmetry breaking that underlies the prediction of the Higgs particle; and Kenneth Arrow from Stanford University, whose many contributions to the fundamental underpinnings of economics, from social choice to endogenous growth theory, have been hugely influential. He was the youngest person ever to have been awarded the Nobel Memorial Prize for economics, which five of his students have also received. Anderson and Arrow together with David Pines, also a distinguished condensed matter physicist and founder of SFI, initiated the first major program that put SFI on the map. It was designed to address foundational questions in economics from this new complex systems perspective, asking, for instance, how ideas from nonlinear dynamics, statistical physics, and chaos theory might provide new insights into economic theory. After one of the early workshops in 1989, Science magazine wrote an article about the meeting titled “Strange Bedfellows.”2 It began:
They make an odd couple, these two Nobel laureates. . . . Over the past 2 years, Anderson and Arrow have worked together in a venture that is one of the oddest couplings in the history of science—a marriage, or at least a serious affair, between economics and the physical sciences. . . . This ground-breaking venture is taking place under the auspices of the Santa Fe Institute.
How times have changed! These days collaborations between physicists and economists are hardly rare occurrences—witness the huge influx onto Wall Street of physicists and mathematicians, many of whom have since become absurdly rich—but just twenty-five years ago it was almost unheard of, especially between two such distinguished thinkers. It’s still hard to believe that this was considered to be so rare and bizarre that it would be characterized as “one of the oddest couplings in the history of science.” Maybe horizons are indeed expanding.
When I became president of SFI I came across some words of wisdom that strongly resonated with me from a man who had helped found and run an extraordinarily successful institute more than fifty years earlier. This was the crystallographer Max Perutz, who shared the Nobel Prize in Chemistry for discovering the structure of hemoglobin. X-ray crystallography, the technique used by Perutz, had been pioneered in the early twentieth century by the unique father-and-son team of William and Lawrence Bragg, who were jointly awarded the Nobel Prize in physics in 1915 when Lawrence, the son, was only twenty-five years old. He remains the youngest person ever to have received the prize in the sciences.
Lawrence Bragg had the great foresight to see that these techniques, which he had helped develop for exploring the crystalline structure of ordinary matter, could potentially prove to be a powerful tool for revealing the structure of the complex molecules that are the building blocks of life, such as hemoglobin and DNA. He strongly encouraged Perutz, who had been his student, to start a research program along these lines entirely devoted to unraveling the structural mysteries of life. Thus in 1947 was born one of the most successful enterprises in all of science, the Medical Research Council Unit (MRCU) within the famed Cavendish Laboratory at Cambridge, whose director was Lawrence Bragg. While under Perutz’s guidance the MRCU produced in just a few short years no fewer than nine Nobel prizes, one of which was the famous discovery of the double-helix structure of DNA by James Watson and Francis Crick.
What was the secret to Perutz’s extraordinary success? Is there some magic formula he had discovered for optimizing how research should be carried out? If so, how could we exploit it to ensure the future success of the Santa Fe Institute? These were questions that I naturally asked myself when I assumed the leadership of SFI. I learned that Perutz, while maintaining his own research program, gave his researchers independence and treated everyone equally, even turning down a knighthood because he thought it would separate him from younger researchers. He stayed fully conversant with everyone’s work and made a point of sitting with different colleagues at coffee, lunch, or tea. Well, at least in spirit, though maybe not always in action, these were all things I was aspiring to do—except for the opportunity of turning down a knighthood in the highly unlikely event that it would have been offered to me.
But what really inspired me about Perutz was something I read about him in his obituary in the Guardian.3 It read:
Impishly, whenever he was asked whether there are simple guidelines along which to organize research so that it would be highly creative, he would say: no politics, no committees, no reports, no referees, no interviews; just gifted, highly motivated people picked by a few men of good judgment. Certainly not the way research is usually run in our fuzzy democracy but, from a man of great gifts and of extremely good judgment, such a reply is not elitist. It is simply to be expected, for Max had practiced it and shown that this recipe is right for those who, in science, want to beat the world by getting the best in the world to beat a path to their door.
So he did have a formula—and it had worked brilliantly. These days it’s hard to believe it was for real: no politics, no committees, no reports, no referees, no interviews, “just” focus on excellence and use extremely good judgment. Well, at least in principle, that’s what we were trying to do at SFI and, indeed, still are: find the best people, trust them, give them support, and don’t hamper them with bullshit . . . and good things will happen. This was the spirit in which SFI had been founded and what all of its presidents from the visionary George Cowan to our wonderful current president, David Krakauer, have enthusiastically championed. It seemed so simple, so why wasn’t everyone following Max’s magic formula? Well, try suggesting this recipe to the funding agencies, the NSF, the DOE, the NIH, to the philanthropic foundations, or to the provosts and deans at universities or to your local congressman and you’ll quickly discover the answer. The formula is, of course, simplistic, somewhat unrealistic, and easier said than done, harking back to an image of support for science and scholarship that probably never actually existed in its naive form. But perhaps that’s its power. Aspiring to such lofty ideals and trying to create a spirit and culture where the development of ideas and the search for knowledge are unencumbered by the hegemony of quarterly reports, continual proposal writing and oversight committees, political intrigue and petty bureaucracy should supersede all other considerations. By example, Perutz had shown this to be a critical component of success. So every year at the conclusion of my annual report to our board of trustees, after boasting of our successes and bemoaning our financial situation and the difficulties of raising funds for our research activities, I would read that magic formula out loud as a mantra or aspiration to remind us to keep our priorities straight.
Beginning with the quantitative observations of planetary motion by the Danish astronomer Tycho Brahe in the sixteenth century, measurement has played a central role in the development of our understanding of the entire universe around us. Data provide the basis for constructing, testing, and refining our theories and models whether they seek to explain the origins of the universe, the nature of evolutionary processes, or the growth of the economy.
Data are the very lifeblood of science, technology, and engineering, and in more recent years have begun to play an increasingly central role in economics, finance, politics, and business. Almost none of the problems I have addressed in this book could be analyzed without recourse to enormous amounts of data. Furthermore, we could not seriously think about developing anything approaching a theory of complex adaptive systems or a science of cities, companies, or sustainability without having access to the sorts of data that I have relied upon in earlier chapters. A good example is the billions of cell phone calls we used in our work to test predictions for the role of social networks and the movement of people in cities.
Critical in these more recent developments has been the IT revolution, not just in assembling and gathering data, but in analyzing and organizing the massive amounts being generated into a manageable form from which insights can be gained, regularities deduced, or predictions made and verified. The speed and capacity of even this thirteen-inch MacBook Air, which I am using to type this manuscript, is awesome and its power for analyzing and retrieving data, storing information, and making complicated calculations is truly extraordinary. My little iPad is more powerful than the Cray-2, the world’s most powerful supercomputer just twenty-five years ago, which would have cost you around $15 million to buy. The amount of data now being amassed through the multiple devices that monitor almost everything around us from our bodies, social interactions, movements, and preferences to the weather and traffic conditions is mind-boggling.
The number of networked devices in the world is now more than double that of the entire global population and the total screen area of all such devices is now larger than one square foot per person. We have truly entered the era of big data. The amount of information currently being stored and exchanged continues to grow exponentially. And all of this has happened in just the last decade or so, yet another impressive manifestation of the accelerating pace of life. The advent of big data has been heralded with a fanfare of promises and hyperbole, suggesting that it will provide the panacea for solving any number of our impending challenges from health care to urbanization while simultaneously improving even further the quality of life. As long as we measure and monitor everything and shovel the data into mammoth computers that will magically produce all of the answers and solutions, then all of our problems and challenges will be overcome and life will be good for all. This evolving paradigm is aptly encapsulated in the flood of “smart” devices and methodologies that are increasingly dominating our lives. “Smart” has become a mandatory designation for almost any new product, whether it’s smart cities, smart health care, smart thermostats, smart phones, smart cards, or even smart parcel boxes.
Data are good and more data are even better—this is the creed that most of us take for granted, especially those of us who are scientists. But this belief is implicitly based on the idea that more data lead to a deeper understanding of underlying mechanisms and principles so that credible predictions and further progress in constructing models and theories can be built upon a firm foundation subject to continual testing and refinement. Data for data’s sake, or the mindless gathering of big data, without any conceptual framework for organizing and understanding it, may actually be bad or even dangerous. Just relying on data alone, or even mathematical fits to data, without having some deeper understanding of the underlying mechanism is potentially deceiving and may well lead to erroneous conclusions and unintended consequences.
This admonition is closely related to the classic warning that “correlation does not imply causation.” Just because two sets of data are closely correlated does not imply that one is the cause of the other. There are many bizarre examples that illustrate this point.4 For instance, over the eleven-year period from 1999 to 2010 the variation in the total spending on science, space, and technology in the United States almost exactly followed the variation in the number of suicides by hanging, strangulation, and suffocation. It’s extremely unlikely that there is any causal connection between these two phenomena: the decrease in spending in science was surely not the cause of the decrease in how many people hanged themselves. However, in many situations such a clear-cut conclusion is not so clear. More generally, correlation is in fact often an important indication of a causal connection but usually it can only be established after further investigation and the development of a mechanistic model.
This is particularly important in medicine. For example, the level of high-density lipoproteins (HDL) in blood—often referred to as “good” cholesterol—is negatively correlated with the incidence of heart attacks, suggesting that taking medications to increase HDL should lower the probability of having a heart attack. However, the evidence supporting this strategy is inconclusive: cardiovascular health does not seem to be improved by artificially raising levels of HDL. This might be because other factors such as genes, diet, and exercise simultaneously affect both HDL levels and the incidence of heart attacks without there being any direct causal link between them. It’s even possible that the causation is inverted so that that good cardiovascular health induces higher HDL levels. To determine what the predominant causes of heart attacks are clearly requires a broad research program involving the gathering of huge amounts of data coupled with the development of mechanistic models for how each factor—whether genetic, biochemical, diet, or the environment—contributes. And indeed, huge resources are devoted across the medical profession to carry out this strategy.
Big data should primarily be viewed within this context: the classic scientific method involving painstaking analyses, the development of models and concepts whose predictions can be tested and used for devising new therapies and strategies, can now be augmented with the additional power of “smart” devices for gathering huge amounts of relevant data. Central to this paradigm is that continual refinement guides what data are most important to measure, how much is needed, and how accurate it needs to be. The variables we choose to focus on and measure in order to obtain data are not arbitrary—they are guided by previous success and failure within the context of an evolving conceptual framework. Doing science is much more than a fishing expedition.
With the advent of big data this classic view is being challenged. In a highly provocative article published in Wired magazine in 2008 titled “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete,” its then editor, Chris Anderson, wrote:
The new availability of huge amounts of data, along with the statistical tools to crunch these numbers, offers a whole new way of understanding the world. Correlation supersedes causation, and science can advance even without coherent models, unified theories, or really any mechanistic explanation at all . . . faced with massive data, this approach to science—hypothesize, model, test—is becoming obsolete. . . . Out with every theory of human behavior, from linguistics to sociology. Forget taxonomy, ontology, and psychology. Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves. . . . Today companies like Google, which have grown up in an era of massively abundant data, don’t have to settle for wrong models. Indeed, they don’t have to settle for models at all. . . . There’s no reason to cling to our old ways. It’s time to ask: What can science learn from Google?
Well, I won’t answer that question other than to say that this radical view is becoming fairly prevalent across Silicon Valley, the IT industry, and increasingly in the business community. In a less extreme version it is also rapidly gaining traction in academia. In the last few years almost every university has opened up a well-funded center or institute devoted to big data while at the same time paying due obeisance to the other buzzword, interdisciplinary. For example, Oxford University has just launched its Big Data Institute (BDI) in a new, sexy, “state-of-the-art building.” Here’s what they say: “this interdisciplinary research centre will focus on the analysis of large, complex, heterogeneous data sets for research into the causes and consequences, prevention and treatment of disease.” Obviously an extremely worthy cause, despite there being no emphasis on theory or concept development.
A contrary view to this trend was forcibly expressed by the Nobel Prize–winning geneticist Sydney Brenner, whom I quoted in chapter 3 and who was coincidentally director of the famous institute in Cambridge founded by Max Perutz that I mentioned earlier: “Biological research is in crisis. . . . Technology gives us the tools to analyse organisms at all scales, but we are drowning in a sea of data and thirsting for some theoretical framework with which to understand it. Although many believe that ‘more is better,’ history tells us that ‘least is best.’ We need theory and a firm grasp on the nature of the objects we study to predict the rest.”
Not long after the publication of Chris Anderson’s article, Microsoft published a fascinating series of essays in a book titled The Fourth Paradigm: Data-Intensive Scientific Discovery. It was inspired by Jim Gray, a computer scientist at Microsoft who was sadly lost at sea in 2007. He envisioned the data revolution as a major paradigm shift in how science would advance in the twenty-first century and called it the fourth paradigm. He identified the first three as (1) empirical observation (pre-Galileo), (2) theory based on models and mathematics (post-Newtonian), and (3) computation and simulation. My impression is that, in contrast to Chris Anderson, Gray viewed this fourth paradigm as an integration of the previous three, namely as a unification of theory, experiment, and simulation, but with an added emphasis on data gathering and analysis. In that sense it’s hard to disagree with him because this is pretty much the way science has progressed for the last couple of hundred years—the difference being primarily quantitative: the “data revolution” provides us with a much greater possibility for exploiting and enabling strategies we have been using for a very long time. In this sense this is more like paradigm 3.1 than paradigm 4.0.
But there is a new kid on the block that many feel promises more and, like Anderson, potentially subverts the need for the traditional scientific method. This invokes techniques and strategies with names like machine learning, artificial intelligence, and data analytics. There are many versions of these, but all of them are based on the idea that we can design and program computers and algorithms to evolve and adapt based on data input to solve problems, reveal insights, and make predictions. They all rely on iterative procedures for finding and building upon correlations in data without concern for why such relationships exist and implicitly presume that “correlation supersedes causation.” This approach has become a huge area of interest and has already had a big impact on our lives. For instance, it is central to how search engines like Google operate, how strategies for investment or operating an organization are devised, and it provides the foundational basis for driverless cars.
It also brings up the classic philosophical question as to what extent these machines are “thinking.” What, in fact, do we mean by that? Are they already smarter than we are? Will superintelligent robots eventually replace us? The specter of such science fiction fantasies seems to be rapidly encroaching on us. Indeed, we can readily appreciate why some like Ray Kurzweil believe that the next paradigm shift will involve the integration of humans with machines, or eventually lead to a world dominated by intelligent robots. As I expressed earlier I have a fairly jaundiced view of such futurist thinking, though the questions raised are fascinating and very challenging and need to be addressed. But the discussion needs to engage with another potential paradigm shift, driven by an impending finite time singularity associated with the accelerating pace of life and involves the challenge of global sustainability and the addition of four to five billion people who will shortly be joining us on our planet.
There is no question that big data will have a major influence across all aspects of life and, in addition, will be a huge aid in the scientific enterprise. Its success in terms of major discoveries and new ways in which we view the world will depend on the extent to which it is integrated with deeper conceptual thinking and traditional development of theory. The vision proposed by Anderson, and to a lesser extent by Gray, is the computer scientists’ and statisticians’ version of the Theory of Everything. It carries with it a similar arrogance and narcissism that this is the singular way to understand everything. How far it will truly reveal new science remains open to question. But when combined with the traditional scientific method, it surely will.
The discovery of the Higgs particle is a fascinating example of how Big Data can lead to important scientific discovery when integrated with traditional scientific methodology. First, I remind you that the Higgs is a crucial linchpin in the basic laws of physics. It permeates the universe, giving rise to the mass of all of the elementary particles of matter from electrons to quarks. Its existence was brilliantly predicted more than sixty years ago by a group of six theoretical physicists. This prediction didn’t come out of the blue but was the end result of a traditional scientific process involving analyses of thousands of experiments carried out over many years iterated with mathematical theories and concepts that were developed to parsimoniously explain these observations, and thereby stimulate further experimentation to test predictions.
It took more than fifty years before the technology was sufficiently developed for a serious search for this elusive but crucial element of our unified theory of the fundamental forces of nature to be undertaken. Central to this was the construction of a giant particle accelerator in which protons move in opposite directions in a circular beam at almost the speed of light and collide with each other in a highly controlled interaction region. This machine, coined the Large Hadron Collider (LHC), was built at CERN in Geneva, Switzerland, at a cost of more than $6 billion. The scale of this gargantuan scientific instrument is huge: its circumference is about seventeen miles long and each of the two major detectors that actually observe and measure the collisions are about 150 feet long, 75 feet high, and 75 feet wide.
The entire project represents an unprecedented engineering achievement whose output is the mother of all big data—nothing comes close. There are about 600 million collisions per second monitored by about 150 million individual sensors in each detector. This produces about 150 million petabytes of data a year or 150 exabytes a day (a byte being the basic unit of information). Let me give you a sense of what the scale of this means. The Word document containing this entire book, including all of its illustrations, is less than 20 megabytes (20MB, meaning 20 million bytes). This MacBook Air can store 8 gigabytes of data (8GB, meaning 8 billion bytes). All of the films stored by Netflix amount to less than 4 petabytes—which is 4 million GB, or about half a million times larger than the capacity of this laptop. Now to the big one: each day the total amount of data produced by all of the world’s computers and other IT devices taken together amounts to about 2.5 exabytes; an exabyte is 1018 bytes, or a billion gigabytes (GB).
This is awesome and is often touted as a measure of the big data revolution. But here’s what’s truly awesome: it pales in comparison to the amount of data produced by the LHC. If every one of the 600 million collisions occurring each second were recorded it would amount to about 150 exabytes a day, which is about sixty times greater than the entire amount of data produced by all of the computational devices in the world added together. Obviously this means that the strategy of naively letting the data speak for themselves by devising machine-learning algorithms to search for correlations that would eventually lead to the discovery of the Higgs mechanism is futile. Even if the machine produced a million times less data it is extremely unlikely this strategy could succeed. How, then, did physicists discover the proverbial needle in this mammoth haystack?
The point is that we have a well-developed, well-understood, well-tested conceptual framework and mathematical theory that guides us in where to look. It tells us that almost all of the debris resulting from almost all of the collisions are actually uninteresting or irrelevant as far as searching for the Higgs particle is concerned. In fact, it tells us that out of the approximately 600 million collisions occurring every second, only about 100 are of interest, representing only about 0.00001 percent of the entire data stream. It was by devising sophisticated algorithms for focusing on only this very special tiny subset of the data that the Higgs was eventually discovered.
The lesson is clear: neither science nor data are democratic. Science is meritocratic and not all data are equal. Depending on what you are looking for or investigating, theory resulting from the traditional methodology of scientific investigation, whether highly developed and quantitative as in the case of fundamental physics, or relatively undeveloped and qualitative as in the case of much of social science, is an essential guide. It is a hugely powerful constraint in limiting search space, sharpening questions, and understanding answers. The more one can bring big data into the enterprise the better, provided it is constrained by a bigger-picture conceptual framework that, in particular, can be used to judge the relevance of correlations and their relationship to mechanistic causation. If we are not to “drown in a sea of data” we need a “theoretical framework with which to understand it . . . and a firm grasp on the nature of the objects we study to predict the rest.”
One final point: The IT revolution is our most recent great paradigm shift, and like all previous ones it is driving us toward a “finite time singularity” whose nature I speculated about in chapter 9. It was enabled by the invention of a startling assortment of extraordinarily “smart” devices that are producing enormous amounts of data. And, like previous major paradigm shifts, it has predictably resulted in an increase in the pace of life. In addition, it has metaphorically brought the world closer together with instant communication anywhere across the globe at any time. It has also led to the possibility that we no longer need to live in an urban environment to participate in and benefit from the fruits of urban social networks and the dynamics of agglomeration, which are the very origin of superlinear scaling and open-ended growth. We can devolve to develop smaller, or even rural, communities that are just as plugged in as living in the heart of a great metropolis. Does this mean that we can avoid the pitfalls that lead to an ever-accelerating pace of life, finite time singularities, and the prospect of collapse? Have we somehow stumbled upon a way to avoid the ironic quandary that the very system that led to our great socioeconomic expansion of the past two hundred years may be leading to our ultimate demise, and that we can have our cake and eat it too?
This is clearly an open question. There are indeed signs that such a dynamic is beginning to develop but so far on an extremely small scale. In fact, the vast majority of people who could in principle de-urbanize and yet remain connected to the center of things choose not to. Even Silicon Valley, which was primarily suburban, has invaded downtown San Francisco, leading to tension between traditional commerce and the excesses of the high-tech lifestyle. I know of no high-tech geeks who are operating from high up in the mountain ranges of the California Sierra. The vast majority seem to prefer traditional urban living. Rather than depopulating, cities seem to be reviving and growing, partially because of the social attractiveness of real-time social contact.
Furthermore, we tend to think that nothing can compare to the changes wrought by the IT revolution with our iPhones, e-mail, text messages, Facebook, Twitter, and so forth. But think of what the railway brought in the nineteenth century or the telephone in the early twentieth. Before the coming of the railway most people didn’t travel more than twenty miles from their home during their entire lifetime: suddenly Brighton was in relatively easy reach of London, and Chicago in reach of New York. Messages that took days, weeks, or even months to be communicated before the invention of the telephone could now be communicated instantaneously. The changes were fantastic. Relatively speaking, these had a greater impact on our lives and, in particular, in speeding up life and changing our visceral perception of space and time than our present IT revolution. But these didn’t result in a de-urbanizing phenomenon or a contraction of our cities. On the contrary, they led to their exponential expansion and to the development of suburbs as an integral part of urban living. Whether the present paradigm continues this trend is open to question, though I suspect that life will continue to speed up and urbanization remain the dominant force as we head toward an impending singularity. How this plays itself out will determine much about the sustainability of the planet.