“I can calculate the movement of the heavenly bodies,” said Sir Isaac Newton, “but not the madness of men”—a humbling confession for a man universally considered the greatest mind of his generation.1 What on earth could have prompted such a remark? Turns out, intellectual giants are human too.
In February of 1720, Newton invested a modest sum of his substantial wealth in shares of the South Sea Company. This British joint-stock company, founded in 1711, was granted a monopoly to trade in Spain’s South American colonies as a part of a treaty from the War of the Spanish Succession.
In three months, Newton’s shares had tripled in value, and he decided to sell. Had the story ended here, all would have been well. But Newton could not stay away from the South Sea Company. He watched anxiously as friends who still held their shares continued to get rich. By July, Newton could no longer resist the temptation. He reinvested in the company, paying £700 for each share he had sold earlier for £300. However, this time he did not invest a modest amount—it was a substantial chunk of his entire net worth.
By November, it was all over. The “South Sea Bubble” had popped. Like a raging fever, speculation in shares of the company had quickly come and gone. Newton scrambled to sell his investment, eventually exiting just north of £100 per share. Had he not held the post of Master of the Royal Mint, with its guaranteed salary, Newton’s remaining life would have been a financial struggle.
It’s unfortunate that Charles Mackay’s Extraordinary Popular Delusions and the Madness of the Crowds was not available for Newton. But that masterpiece on crowd psychology was not to be written for another 120 years. Still, Newton could have studied Joseph de la Vega, a successful Jewish merchant and philanthropist who had written the first book on the stock market titled Confusion of Confusions ([1688] 1996). In it, Vega presents the art of speculation as a dialogue between different market participants. It was a brilliant narrative tool, which helped the reader better understand speculation and trading.
Vega’s Confusion of Confusions is easily summarized. In the Second Dialogue, Vega lists four basic principles of trading—as relevant today as they were 325 years ago:
The first principle: Never advise anyone to buy or sell shares. Where perspicacity is weakened, the most benevolent piece of advice can turn out badly.
The second principle: Take every gain without showing remorse about missed profits. It is wise to enjoy what is possible without hoping for the continuance of a favorable conjuncture and the persistence of good luck.
The third principle: Profits on the exchange are the treasures of goblins. At one time they may be carbuncle stones, then coals, then diamonds, then flint-stones, then morning dew, then tears.
The fourth principle: Whoever wishes to win in this game must have patience and money, since values are so little constant and the rumors so little founded on truth. He who knows how to endure blows without being terrified by the misfortune resembles the lion who answers the thunder with a roar and is unlike the hind who, stunned by the thunder, tries to flee.
Together, Joseph de la Vega, Isaac Newton, and Charles McKay are telling us something very important: The relationship between the individual investor and the stock market, which in itself is nothing more than a collection of individuals, is a profound puzzle. For over four hundred years, it has perplexed the rich and the poor as well as the genius and the dimwitted, and it is the story of our current chapter on social systems.
Sociology is the study of how people function in society, with the ultimate hope of understanding group behavior. When we stop to consider that all the participants in a market constitute a group, it is obvious that until we understand group behavior, we can never fully understand why markets and economies behave as they do.
Throughout history, poets, novelists, philosophers, political leaders, and theologians have all submitted ideas about how societies work, but the distinction for social scientists is their recognition of the scientific process. This process, in essence, involves developing a theory (a hypothesis), then testing that theory through controlled, repeatable experiments. This is the same approach used by chemists, physicists, biologists, and all other scientists in their search for answers.
Social scientists, as they work to uncover and explain how human beings form collectives, organize themselves, and interact, have adopted the scientific process, developing theories that lead to the construction of models that can be compared with data collected and then testing and verifying those theories. However, because their investigation by definition involves the subjective and unpredictable behavior of human beings, in the social sciences the process is less precise than in the natural sciences, and in many circles the social sciences have not yet reached the same level of scientific acceptance.
Indeed, some have suggested that the lack of maturity in social sciences is directly attributable to the absence of hard, quantitative results commonly associated with the natural sciences. This is now changing, as immense computer power makes possible the collection of vast amounts of data, but nonetheless, there are those who question the validity of attaching the term “science” to the study of social systems. We might say that the social sciences are still waiting for their Isaac Newton.
The development of the social sciences has followed two distinct paths: a drive for a unified theory and a move toward narrower specializations. The first approach was urged by the French philosopher Auguste Comte, who in the mid-nineteenth century called for a new science to take its place alongside astronomy, physics, chemistry, and biology. This new science, which he named “sociology,” would explain social organization and help guide social planning. Comte saw the study of society as a unified pursuit; society is an indivisible thing, he argued, and so too must be the study of it. But despite Comte’s efforts at synthesis, the nineteenth century did not end with a single unified theory of social science but rather with the promotion of several distinct specialties, including economics, political science, and anthropology.
Economics was the first discipline to attain the status of a separate study within social science. Some trace the history of modern economics back to 1776, the year that the Scottish economist Adam Smith published his most famous work, Wealth of Nations. Considered the founder of economics, Smith was also one of the first to describe its effect on society. He is best known to today’s economists for his advocacy of a laissez-faire capitalist system—that is, one free of government interference, including industry regulation and protective tariffs. Smith argued that an economic system works best when it is based solely on its own natural mechanism, what he called the “invisible hand.”
Smith believed that division of labor is the cause of increased productivity and, ultimately, wealth for the owners of capital. He was not, however, unaware of the social consequences produced by division of labor: the decline of general skills and craftsmanship, the likely incorporation of women and children into the workforce, and the tendency to divide society into economic classes with opposing interests. He acknowledged that, over time, the owners of capital would seek to limit the wages of labor. Thus was set into motion a countervailing view of economics propounded by Karl Marx and other socialists: that capitalism was but a passing stage of development that would soon be replaced by a more humane economic system based on cooperation, planning, and the common ownership of the means of production.
Given the debate over the interplay of economics and society, it is not surprising that at the same time there was an increasing investigation into the behavior of governments. By the nineteenth century, the role of the state held the same fascination for a group of social scientists, soon called political scientists, as did the impact of capital for economists. These new political scientists were soon investigating the political consequences of Adam Smith’s laissez-faire economics. How should the government respond to the new democratic rights of working people while at the same time protecting the private property rights of the owners of capital? Deciding who gets what, when, where, and how became the essence of the new field called political science.
Soon another science took its place alongside economics and politics as a separate discipline within the social sciences: anthropology. From the beginning, anthropology was divided into two classes: physical and cultural. Physical anthropology was chiefly concerned with the evolution of man as a species and with genetic systems such as different races in the world. Cultural anthropology, on the other hand, investigated the social aspects of the many different human institutions, as found in both primitive and contemporary societies. It was here that the science of sociology came into its own. At first it was difficult to separate the identities of cultural anthropologists from the new sociologists, but the distinction was made clearer when sociologists began to limit their inquiry to contemporary societies, leaving the investigation of primitive societies to the anthropologists.
By the twentieth century, sociology had been further separated into social psychology and social biology. Social psychologists studied the ways in which the individual human mind, as well as the collective mind, relates to the social order. They sought to explain how culture affects psychology and, conversely, how collective psychology influences culture. We shall see more about this in the next chapter.
Social biologists, for their part, owe much to Charles Darwin. The increasing academic acceptance and scientific maturity of Darwin’s theory of evolution caused several scientists to make considerable advances promoting a biological view of society. There was no greater champion of this approach than the Yale sociologist William Graham Sumner, who founded an intellectual movement known as Social Darwinism, in which he sought to connect Adam Smith’s principle of laissez-faire economics to Charles Darwin’s concept of natural selection.
In Sumner’s mind, there was a strong connection between the struggle of existence within nature and the struggle for existence within society. He believed the market, just like nature, is in a constant struggle for scarce resources, and that the process of natural selection in humans would inevitably lead to social, political, and moral progress.
After World War II, Social Darwinism all but disappeared from academic debate. Only recently has the biological concept resurfaced. Several scientists, most notably Edward O. Wilson, have reintroduced the connections between social science and biology into a field of inquiry now called sociobiology. However, most have sought to distance themselves from the implication that natural selection can be a justification for social inequality, which they consider a gross distortion of Darwin’s message. Instead, the new sociobiologists are now focusing their energies on the more scientific principles associated with evolution and its connection to social development.
All of these areas of social science—sociology, political science, economics, and the several subdisciplines within each—are, in one sense, only different platforms from which to think about one large question: how human beings form themselves into groups, or societies, and how those groups behave. The study of political science gives us insight into how people create governments; the study of economics helps us understand how they produce and exchange goods; and so on. Of course each individual simultaneously participates in several groups, and so the larger concern for those who wish to understand behavior is how the pieces fit together and influence one another.
Although the idea of a unified theory of social science faded in the late nineteenth century, here at the beginning of the twenty-first century there has been a growing interest in what we might think of as a new unified approach. Scientists have now begun to study the behavior of whole systems—not only the behavior of individuals and groups but the interactions between them and the ways in which this interaction may in turn influence subsequent behavior. Because of this reciprocal influence, our social system is constantly engaged in a socialization process the consequence of which not only alters our individual behavior but often leads to unexpected group behavior.
Granted, this is a complicated perspective from which to investigate humankind. But man is a complex being, and those who would understand human behavior must find a way to work within the complexity. Fortunately, guidance is at hand in the scientific area of inquiry known as complexity theory.
In earlier chapters, we have identified economies and stock markets as complex systems. The term complexity is derived etymologically from the Latin word plexus, which means interwoven. When we think about complexity we intuitively understand the difficulty of separating the individual from the whole. Furthermore, separating individuals in order to study them singularly negates the observation, for we know individual behavior is highly influenced by its interactions with other individuals in the collective. We have come to understand that economies and stock markets are adaptive systems. As such, their behavior constantly changes as individuals in the system interact with other individuals and within the system itself.
Many social scientists now start with the same assumption. They recognize that human systems, whether economic, political, or social, are complex systems. Furthermore, sociologists now recognize that a universal trait of all social systems is their adaptability.
From these pioneering scientists now studying complex adaptive systems, we can gain insights into that great social system called humankind and, by extension, into the functioning of specific systems like the stock market.
One aspect of these systems is the formation process. How do people come together to form complex systems (social units) and then further organize themselves into some sense of order? This question has led to a new hypothesis that may provide a common framework to describe the behavior of all social systems. It is called the theory of self-organization.
The term “self-organization” refers to a process whereby structure appears in a system that does not have a central authority or some other element that imposes its will by preplanning. We can observe self-organization in chemistry, biology, mathematics, and computer science. It also occurs in human networks or societies.
The term was first used by Immanuel Kant in his Critique of Judgment (1790). Kant referred to an entity whose parts or “organs” are able to behave as if it had a mind of its own and was capable of governing itself. He writes, “Every part is thought as owning its presence to the agency of all the remaining parts, and also for existing for the sake of others … only under these conditions and upon these terms can such a product be an organized or a self-organized being [italics original].”
Self-organization as a theory, although associated with general systems theory in the 1960s, did not become a part of the mainstream academic literature until the late 1970s and early 1980s when physicists began to explore complex systems. Ilya Prigogine, the Russian chemist, is credited with popularizing self-organization theory. He was awarded the Nobel Prize in 1977 for his thermodynamic concept of self-organization.
The economist Paul Krugman, author of twenty books, over two hundred scholarly articles, and winner of the 2008 Nobel Prize for Economics, began a systematic inquiry into the theory of self-organization, particularly as it related to the economy (The Self-Organizing Economy, 1996). To illustrate how it works, Krugman asks us to imagine the city of Los Angeles. Today, we know that Los Angeles is not one homogeneous landscape but a collection of different socioeconomic, racial, and ethnic neighborhoods including Koreatown, Watts, and Beverly Hills. Surrounding the city is a further collection of many business districts. Now each of these distinct spaces was formed not by urban planners drawing lines on a map but by the spontaneous process of self-organization. Koreans moved to Koreatown to be closer to other Koreans. As the population increased, still more Koreans were drawn to the neighborhood, and thus a self-organized community also became self-reinforcing. No central controller made this decision for everyone, explains Krugman; the city just spontaneously evolved and organized itself in this fashion.
The evolution of a large city is a relatively simple example of self-organizing and self-reinforcing systems, but we can observe similar behavior in economic systems. Setting aside for the moment the occasional recessions and recoveries caused by exogenous events such as oil shocks or military conflicts, Krugman believes that economic cycles are in large part caused by self-reinforcing effects. During a prosperous period, a self-reinforcing process leads to greater construction and manufacturing until the return on investment begins to decline, at which point an economic slump begins. The slump in itself becomes a self-reinforcing effect, leading to lower production; lower production, in turn, will eventually cause return on investment to increase, which starts the process all over again. Some might argue that the Federal Reserve, by altering interest rates and making open market purchases and sales, acts as the central controller for the economy, but as we all know the Fed is not omnipotent. If we stop and think, we realize that the equity and debt markets have no central controller, and both are excellent examples of self-organizing, self-reinforcing systems.
It is important for us to keep in mind that the theory of self-organization is just that—a theory. Although it appears to be a plausible explanation of how social systems work, there are no models yet built that can test the theory, much less predict its future behavior. In search for unified theories of how social systems behave, however, the self-organizing theory appears to be a legitimate candidate.
The second characteristic of complex adaptive systems—their adaptivity—is embedded within what is known as the theory of emergence. This refers to the way individual units—be they cells, neurons, or consumers—combine to create something greater than the sum of the parts. Paul Krugman suggests that Adam Smith’s “invisible hand” is a perfect example of emergent behavior. Many individuals, all of them trying to satisfy their own material needs, engage in buying and selling with other individuals, thereby creating an emergent structure called the market. The mutual accommodation of its individual units coupled with the self-organizing behavior of the system creates a behavioral whole, an emergent property that transcends its individual units.
Just like the concept of self-organization, emergence is also a theory. However, it appears to be a thoughtful explanation of what actually occurs when individual units come together and organize. Although scientists have had difficulty modeling the phenomenon of self-organization, they have made excellent progress modeling emergent behavior.
The Los Alamos National Laboratory (LANL) is the largest U.S. Department of Energy laboratory in the country and one of the biggest multidisciplinary research institutions in the world. It covers forty-three square miles and employs almost ten thousand people, including physicists, engineers, chemists, biologists, and geoscientists.
Most people know Los Alamos as the facility that developed the first atomic bomb, but today the laboratory’s vision has widened and now includes several scientific programs that are directed at preserving and improving the quality of life on earth. The research projects underway at Los Alamos are too numerous to list here. But to you give you a sense of the breadth of research, LANL includes the Center for Integrated Nanotechnologies; the Energy Security Center, which is exploring reliable, secure, and sustainable carbon-neutral energy solutions; the Institute of Geophysics and Planetary Physics; the Neutron Scattering Center; and a High Magnetic Field Laboratory.
At the top of the list is the Center for Bio-Security Science (CBSS). Founded in 2008, the CBSS works to achieve science and technology breakthroughs in understanding and mitigating threats to national security, public health, and agriculture from natural, emerging, and engineered infectious agents. Inside the CBSS resides the Biological Threat Reduction Program led by Dr. I. Gary Resnick and Dr. Norman L. Johnson, the assistant director.
Johnson studied chemical engineering at the University of Wisconsin, where he soon gained a reputation for tackling problems most people put in the “too hard to do” box. Johnson’s success, he claimed, came from assembling heterogeneous teams that were able to break through the intellectual barriers by fostering synergistic solutions developed from diverse contributions.
After joining Los Alamos National Laboratory, Johnson founded the Symbiotic Intelligence Project (SIP). Its purpose was to study the unique abilities of information systems, such as the Internet, as well as human problem-solving teams to create a capability that is greater that the sum of the parts. This newly created knowledge is an emergent property of the collective. Although the term “emergent” may be new to laypeople, Johnson points out that the experience is commonplace. For thousands of years, societal structures have been able to collectively solve problems that have threatened their very existence.
Self-organized systems, explains Johnson, have three distinct characteristics. First, the complex global behavior occurs by simple connected local processors. In a social system, the local processors are individuals. Second, a solution arises from the diversity of the individual inputs. Third, the functionality of the system, its robustness, is far greater than any one of the individual processors. Johnson believes that the symbiotic combination of humans and networks (Internet) will generate, in a collective, far better results that any one individual can do acting alone. He envisions an “unprecedented capability in organizational and societal problem solving will result from increased human activity on smart distributed information systems.”2
One of the great advantages of the Internet is how it helps us manage information; in this, explains Johnson, the Internet has three significant advantages over prior systems. First, it is able to integrate a wide breadth of knowledge compared to other systems whose information was often physically separated. Second, the Internet is able to capture and display depth of information. With digitization, systems are able to produce volumes of data on a single topic without significant additional cost. Third, the Internet is able to process information correctly. As we will learn in the next chapter on psychology, communication missteps between individuals sometimes result in the loss of vital information. Information exchanged via the Internet is delivered accurately, in much the same way that books and documents are able to transmit information. It is Johnson’s belief that these three advantages, along with the interconnectivity of millions of individuals, will greatly enhance the collective problem-solving ability of self-organized systems.
To illustrate the phenomenon of emergence, let’s look in on a familiar social system: an ant colony. Because ants are social insects (they live in colonies, and their behavior is directed to the survival of the colony rather than the survival of any one individual ant), social scientists have long been fascinated by their decision-making process.
One of the ant’s most interesting behaviors is the process of foraging for food and then determining the shortest path between the food source and the nest.3 While walking between the two, ants lay down a pheromone trail that allows them to trace the path and also show other ants the location of the new food source.
At the beginning, the search for food is a random process, with ants starting out in many different directions. Once they locate food, they return to the nest, laying down the pheromone trail as they go. But now comes the very sophisticated aspect to collective problem solving: the colony, acting as a whole, is able to select the shortest path. If one ant randomly finds a shorter path between the food source and the nest, its quicker return to the nest intensifies the concentration of pheromone along the path. Other ants tend to choose the path with the strongest concentration of pheromone and hence set off on this newly discovered short path. This increased number of ants along the trail deposits even more pheromone, which further attracts more ants until this path becomes the preferred line. Scientists have been able to demonstrate experimentally that the pheromone-trail behavior of the ant colony solves for the shortest path. In other words, this optimal solution is an emergent property of the collective behavior of the ant colony.
Norman Johnson, who like many is fascinated by ant behavior, set out to test humans’ ability to solve collective problems. He constructed a computer version of a maze with countless paths but only a few that are short. The computer simulation consists of two phases: a learning phase and an application phase. In the learning phase, a person explores the maze with no specific knowledge of how to solve the maze until the goal is found. This is identical to the process an ant follows when it begins to look for food. In the application phase, people simply apply what they learned. Johnson discovered that people need an average of 34.3 steps to solve the maze in the first phase and 12.8 steps in the second phase. Then, to find the collective solution, Johnson combined all the individual solutions and applied the application phase. He found that if at least five people were considered, their collective solution was better than the average individual solution. It took a collective of only twenty to find the very shortest path through the maze, even though they had no global sense of the problem. This collective solution, argues Johnson, is an emergent property of the system.
Although Johnson’s maze is a simple problem-solving computer simulation, it does demonstrate emergent behavior. It also leads us to better understand the essential characteristic a self-organizing system must contain in order to produce emergent behavior. That characteristic is diversity. The collective solution, Johnson explains, is robust if the individual contributions to the solution represent a broad diversity of experience in the problem at hand. Interestingly, Johnson discovered that the collective solution is actually degraded if the system is limited to only high-performing people. It appears that the diverse collective is better at adapting to unexpected changes in the structure.4
To put this in perspective, Johnson’s research suggests that the stock market, theoretically, is more robust when it is composed of a diverse group of agents—some of average intelligence, some of below-average intelligence, and some very smart—than a market singularly composed of smart agents. At first, this discovery appears counterintuitive. Today, we are quick to blame the amateur behavior of uninformed individual investors and day traders for the volatile nature of the market. But if Johnson is correct, the diverse participation of all investors, traders and speculators—smart and dumb alike—should make the markets stronger, not weaker.
Another important insight from Norman Johnson was his discovery that the system, as long as it is adequately diverse, is relatively insensitive to moderate amounts of noise (by which he means any sort of discordant, disruptive activity). To prove the point, Johnson intentionally degraded an individual contribution; he learned his action had no effect on participants’ finding the shortest path out of the maze. Even at the highest levels of disruption, the collective behavior, after a brief postponement, was able to discover the minimal path. Not until the system reached its highest noise level did the collective decision-making process break down.
The work of Norman Johnson appears to contradict the classical views of crowd behavior. From Henry David Thoreau to Thomas Carlyle to Friedrich Nietzsche, the nineteenth century’s great intellectuals were highly suspicious of collective judgment. It was Thoreau who said “as a member of a crowd, he at once becomes a blockhead.” Nietzsche tells us “the mass never comes up to the standard of its best member,” and Carlyle wrote, “I do not believe in the collective wisdom of individual ignorance.”5 But no one was a more vocal critic of the intellect of crowds than Gustave Le Bon.
A French sociologist and psychologist, Le Bon spent his career studying herding behavior and crowd psychology. The culmination of his work was published in 1895 under the title La psychologie des foules (The Psychology of Crowds); the English version was titled The Crowd: A Study of the Popular Mind. On first reading, it appears Le Bon has foreseen Norman Johnson. He writes that the crowd is an independent organism greater than the sum of its parts. It has the ability to operate independently and as such forms its own identity and will. But whereas Johnson tells us the emergent property of the crowd is superior reasoning, Le Bon reached the opposite conclusion. Like Thoreau, Carlyle, Nietzsche, and Mackay, Le Bon believed that crowds “can never accomplish acts demanding a high degree of intelligence” and “they are always intellectually inferior to the isolated individual.”6
Who is right?
The answer lies in an outstanding book titled, The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations. Written by James Surowiecki, the business columnist for The New Yorker, it purposefully takes aim at Mackay’s idea of “the madness of crowds” with a simple, powerful thesis: “under the right circumstances [italics mine], groups are remarkably intelligent and are often smarter than the smartest people in them.”7
Surowiecki begins by telling the story of Francis Galton, the English Victorian-era polymath. In a 1907 article in Nature, Galton describes a contest he promoted at the West of England Fat Stock and Poultry Exhibition. In that contest, 787 people paid sixpence for the opportunity to guess the weight of a rather large ox at the exhibition. A few of the guessers were farmers and butchers, people who might be classified as experts, but a far greater number had no specialized knowledge of farm animals. Based on this information, Galton surmised the mix of participants contained a few very smart guessers, a few who were totally clueless, and the rest—the largest number—being mediocre guessers at best. Based on that formula, he anticipated his 787 participants would most likely end up with a dumb answer. He was wrong.
The ox actually weighed 1,198 pounds. Galton took all the guesses and plotted a distribution curve. He found that the median guess was within 0.8 percent of the correct weight and the mean guess was within 0.1 percent. Put differently, the average guess was 1,197 pounds. What Galton had discovered was the errors in the left and right tail cancelled each other out and what remained was the distilled information.
According to Surowiecki, the two critical variables necessary for a collective to make superior decisions are diversity and independence. If a collective is able to tabulate decisions from a diverse group of individuals who have different ideas or opinions on how to solve a problem, the results will be superior to a decision made by a group of like-minded thinkers.
Independence, the second critical variable, does not mean each member of the group must remain in isolation but rather each member of the group is basically free from the influence of other members. Independence is important to the collective decision-making process for two reasons, explains Surowiecki. “First, it keeps the mistakes that people make from becoming correlated. Errors in individual judgment won’t wreck the group’s collective judgment as long as those errors aren’t systematically pointing in the same direction. Second, independent individuals are more likely to have new information rather than the same old data everyone is already familiar with.”8
Building on the work of Surowiecki and the science of Norman Johnson, Scott Page at the University of Michigan is working to continually press forward the theory of smart collectives.9 Page is the Leonid Hurwicz Collegiate Professor of Complex Systems, Political Science and Economics, and also the current director for the Study of Complex Systems at the university.
Like Johnson, Page set up a series of computer-simulated problem-solving agents to demonstrate the emergent outcome of a diverse group attempting to solve a problem. For example, Page put together groups of ten to twenty agents, each with a different set of skills, and then had each group solve relatively difficult problems. In each group there were some who were excellent at solving the specific problem and others that were less effective. What Page discovered was a group composed of very smart agents and less-smart agents always did better at solving the problem than a segregated group of smart agents only. Furthermore, you could do just as well at solving the problem by randomly selecting any combination of agents as you could if you spent time isolating which were the smart agents and putting them to work on the problem.
In his book, The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies, Page firmly says, “Diverse perspectives and tools enable collections of people to find more and better solutions.” He goes further: “Diverse predictive models enable crowds of people to predict values accurately.”10
What does he mean by “predictive models”? Examples would include the Hollywood Stock Exchange (future predictions of ticket sales of movies), Iowa Electronics Market (future predictions on political contests), and Intrade (which claims to be the world’s leading prediction market and will let you bet on pretty much anything you can imagine). Each of these predictive markets is composed of a diverse group of agents acting independently to make decisions. There are incentives to make the correct decision, and each of these markets aggregates the collective decisions.
How efficient are these predictive markets? In other words, how successful are they at correctly predicting outcomes? Remarkably successful, the evidence shows.
There is another predictive market we can observe. It is called the stock market.
So now we come to the crossroads. Is the stock market Charles Mackay’s unruly mob of irrational investors who constantly unleash booms and busts or is it Francis Galton’s county fair attendees who can miraculously make the right prediction? The answer is context dependent. In other words, it depends.
We know the stock market is an incentive-based system that can aggregate investor decisions. What we need to understand is the market’s level of diversity and the independence of its participants. If the stock market is adequately diversified and, most importantly, if the decisions of its participants have been reached independently, then it is likely the market is efficient. Surowiecki reminds us that just because we can observe some irrational investors, that does not necessarily mean the market is inefficient. Indeed, proponents of the efficient market hypothesis have latched onto the “wisdom of the crowds” as a plausible explanation for market efficiency.11
But what if independence is lost? What if the decisions of the market’s participants are not independent but are now coalesced into one opinion? When this occurs, the system has effectively lost its diversity and along with it any chance of generating an optimal solution. If diversity is the key to how collectives can best reach solutions, then diversity breakdowns are the cause of suboptimal outcomes—or in the case of the stock market, diversity breakdowns cause the market to become inefficient.
Scientists are now turning their attention to understanding what causes diversity breakdowns. Michael Mauboussin, author of two very important books, More Than You Know: Finding Financial Wisdom in Unconventional Places and Think Twice: Harnessing the Power of Counterintuition, tells us “information cascades (which can lead to diversity breakdowns) occur when people make decisions based on the actions of others rather than on their own private information. These cascades help explain booms, fads, fashions, and crashes.”12 Social network theorists, who view social relationships in terms of nodes and ties, whereby nodes are the individual actors and ties are the relationships between the actors, consider this to be the proper framework for understanding how information cascades can sweep across large populations.
Mauboussin also reminds us diversity breakdowns are not just a large group phenomenon but can also occur in smaller groups. Whether it be a committee, jury, or small working team, information cascades, which lead to diversity breakdowns, are often the result of a dominant leader operating with limited facts, sometimes even with no facts.
To illustrate his point, Mauboussin cites the work of Cass Sunstein, a professor at Harvard Law School. Sunstein first separated liberals and conservatives into like-minded groups and then asked them to debate controversial issues ranging from same-sex marriage to affirmative action. Sunstein then rearranged the groups so that each was an equal mix of liberal and conservative and asked them to repeat the same debates. One might think the new heterogeneous group would reach a more moderate conclusion. But in fact, because a strong leader emerged in all of these diverse groups, the groups ultimately settled on one view—the leader’s—that was more extreme than the opinions held before the debate began. The strong leader, whether liberal or conservative, influenced the rest of the group to move completely to his position.
Much has been written about social conformity of groups over the years. Perhaps the most famous social psychology experiments were Solomon Asch’s 1940s studies of individual conformity under group pressure, also described by Mauboussin.
Asch first assembled several groups of eight individuals. Each group was then asked to complete a very easy task. Several poster boards were divided in half. On the left-hand side was a single line. On the right-hand side were three unequal lines of which one was identical in length to the line on the left. The groups then had to match the length of the single line to one of the three unequal lines. The first few experiments went smoothly. Then on cue, seven of the eight members (who had been told about the experiment beforehand) purposely matched a noticeably shorter line from the right side to the test line on the left side. Asch wanted to gauge the response of the lone true subject.
What happened? Although several of the subjects did hold fast to their initial decision—they remained independent—about one-third of the test subjects altered their decisions so they would conform to the group’s decision. What Asch discovered was that group decisions, even noticeably poor ones, have a profound influence over individual decisions.13
When catastrophes occur, we naturally seek to identify the principal cause so we can avoid another disaster or at least derive some comfort from knowing what happened. We like it best when we can point to one specific, easily identifiable cause, but that is not always possible. Many scientists believe that large-scale events in biology, geology, and economics are not necessarily the result of a single large event but rather of the unfolding of many smaller events that create an avalanche-like effect. Per Bak, a Danish theoretical physicist (1948–2002), developed a holistic theory of how systems behave called “self-organized criticality.”
According to Bak, large complex systems composed of millions of interacting parts can break down not only because of a single catastrophic event but also because of a chain reaction of smaller events. To illustrate the concept of self-criticality, Bak often used the metaphor of a sand pile. Imagine an apparatus that drops one single grain of sand on a large flat table. Initially, the sand spreads across the table and then begins to form a slight pile. As one grain rests on top of another grain, the pile of sand rises until it forms a gentle slope on each side. Eventually, the pile of sand cannot grow any higher. At this point, sand trickles down the slope as fast as the grains are added to the top. In Bak’s analogy, the sand pile is self-organized in the sense that it has formed without anyone placing the individual grains. Each grain of sand is interlocked in countless combinations. When the pile has reached its highest level, we can say the sand is in a state of criticality. It is just on the verge of becoming unstable.
When one more grain of sand is added to the pile at that point, that single grain of sand can start an avalanche, with sand rolling down the side slope of the pile. Each rolling grain of sand will stop if it happens to fall into a stable position; otherwise, it continues to fall and possibly hits other grains of sand that may also be unstable, knocking even more grains farther down the side. The avalanche ceases when all unstable grains have fallen as far as they are going to fall. If the shape of the pile of sand has flattened from the avalanche, we can say the pile is in a subcritical phase and will remain there until more sand is added, once again raising the sides of the slope.
Per Bak’s sand pile metaphor is a powerful tool that helps us understand the behavior of many different systems. In both natural and social systems, we can see the dynamic: the systems become a class of interlocking subsystems that organize themselves to the edge of criticality and, in some cases, break apart violently only to reorganize themselves at a later point. Is the stock market such a system? Absolutely, said Per Bak.
In a joint paper written with two colleagues titled, “Price Variations in a Stock Market with Many Agents,” Bak defended his thesis.14 The three scientists constructed a very simple model that sought to capture the behavior of two types of agents operating in a stock market. They called the two types noise traders and rational agents. With apologies to the authors, I will instead use the more familiar terms of fundamentalists and trend followers. Trend followers seek to profit from changes in the market by either buying when prices go up or selling when prices go down. Fundamentalists buy and sell based not on the direction of the price changes but rather because of the difference between the price of a security and its underlying value. If the value of the stock is higher than the current price, fundamentalists buy shares; if the value is lower than the current price, they sell.
Most of the time, the interplay between trend followers and fundamentalists is somewhat balanced. Buying and selling continue with no discernible change in the overall behavior of the market. We might say the sand pile is growing without any corresponding avalanche effects. Put differently, diversification is present in the market.
But when stock prices climb, the ratio of trend followers to fundamentalists begins to grow. This makes sense. As prices increase, a larger number of fundamentalists decide to sell and leave the market and are replaced by a growing number of trend followers who are attracted to rising prices. When the relative number of fundamentalists is small, stock market bubbles occur, explained Bak, because prices have moved far above the fair price a fundamentalist would pay. Extending the sand pile metaphor further, as the number of fundamentalists in the market declines, and the relative number of trend followers increases, the slope of the sand pile becomes ever steeper, increasing the possibility of an avalanche. Once again, we can put this differently by saying that when the mix of fundamentalists and trend followers becomes unbalanced, we are heading toward a diversity breakdown.
It is important for us to remember at this point that while Per Bak’s self-organizing criticality explains the overall behavior of avalanches, it does nothing to explain any one particular avalanche. When we ultimately are able to predict the behavior of individual avalanches, it will not be because of self-organized criticality but because of some other science yet to be discovered.
That in no way diminishes the significance of Bak’s ideas. Indeed, several notable economists have acknowledged Per Bak’s work on self-organized criticality as a credible explanation for how complex adaptive systems behave, including the Nobel physics laureate Phil Anderson and the Santa Fe Institute’s Brian Arthur. Both recognize that self-organizing systems tend to be dominated by unstable fluctuations and that instability has become an unavoidable property of economic systems.
Instability in the stock market is, of course, painfully familiar to everyone involved. It is the treacherous threshold upon which we all too often stub our toes. Surely it would ease our frustration if we understood it better. To get a better fix on the dynamics of instability, we will need to venture back into the social sciences.
Diana Richards, a political scientist, is investigating what causes a complex system of interacting agents to become unstable. Or, in Per Bak’s terms, she is trying to determine how a complex system of individuals reaches self-organized criticality.
According to Richards, a complex system necessarily involves aggregation of a wide number of choices made by the individuals in the system.15 She calls this “collective choice.” Of course, combining all the individuals’ choices does not always result in a straightforward collective choice; nor should we assume the aggregate choice, which is the sum of individual choices, always leads to stable outcomes. Collective choice, says Richards, occurs when all the agents in the system aggregate information in a way that allows the system to reach a single collective decision. To reach this collective decision, it is not necessary that all the agents hold identical information but that they share a common interpretation of the different choices. Richards believes that this common interpretation, which she calls mutual knowledge, plays a critical role in the stability of all complex systems. The lower the level of this mutual knowledge, the greater the likelihood of instability.
An obvious question at this point is how people select from a collection of choices. According to Richards, if there is no clear favorite, the tendency of the system is to continually cycle over the possibilities. You might think this cyclical outcome would lead to instability, but according to Richards, it need not if the agents share similar mental concepts (that is, mutual knowledge) about the various choices. It is when the agents in the system do not have similar concepts about the possible choices that the system is in danger of becoming unstable. And that is clearly the case in the stock market.
If we step back and think about the market, we can readily identify a number of groups that exhibit different meta-models. We already know that fundamentalists and trend followers possess different meta-models. What about macro-traders who are not interested in individual companies but are interested only in directional changes in the overall market? What about long-short hedge funds? What about statistical arbitrageurs versus entrepreneurs? What about quantitatively driven strategists that seek low volatility-absolute return strategies? Each of these groups works from a different reality, a different sense of how the market operates and how they should operate within it. In reality, there are many different meta-models at work in the stock market, and if Richards’s theory is correct, this all but guarantees periodic instability.
The value of this way of looking at complex systems is that if we know why they become unstable, then we have a clear pathway to a solution, to finding ways to reduce overall instability. One implication, Richards says, is that we should be considering the belief structures underlying various mental concepts and not the specifics of the choices. Another is to acknowledge that if mutual knowledge fails, the problem may center on how knowledge is transferred in the system. In the next chapter on psychology, we will turn to our attention to those two points: how individuals form belief structures and how information is exchanged in the stock market.
At this point, we have a fixed compass on how to analyze social systems. Whether they are economic, political, or social, we can say these systems are complex (they have a large number of individual units), and they are adaptive (the individual units adapt their behavior on the basis of interactions with other units as well as with the overall system). We also recognize that these systems have self-organizing properties and that, once organized, they generate emergent behavior. Finally, we realize that complex adaptive systems are constantly unstable and periodically reach a state of self-organized criticality.
We come to these conclusions by studying a large number of complex adaptive systems across a wide variety of fields in both the natural and the social sciences. In all our study, we are currently limited to understanding how the systems have behaved so far. We have not made the scientific leap that will enable us to predict the future behavior, particularly in complex social systems involving the highly unpredictable units known as human beings. But we may be on the track of something even more valuable.
What separates the study of complex natural systems from complex social systems is the possibility that in social systems we can alter the behavior of their individual units. Whereas we cannot as of yet change the trajectory of hurricanes, where groups of people are concerned we may be able to affect the outcome by influencing how individuals respond in various situations. To say this another way, although self-organized criticality is an inherent property of all complex adaptive systems, including economic systems, and although some degree of instability is unavoidable, we may be able to alter potential landslides by better understanding what makes criticality inevitable.