In the final decades of the twelfth century, the Societas Mercatorum, the organization of merchants that had presided over the commercial culture of Florence for nearly a hundred years, began to break apart into splinter groups: guilds with names like the Arte di Por Santa Maria and the Arte di Calimala, structured around specific trades—blacksmiths, moneylenders, wine merchants. A few guilds incorporated diverse groups under one umbrella. One such guild, the Arte di Por Santa Maria, included both silk weavers and goldsmiths.
The creation of the guild system, by all accounts, proved to be a reorganization that literally changed the world. Historians like to talk up the aesthetic accomplishments of the Renaissance, but the guild system pioneered in Florence had as much of an impact on Western civilization as anything dreamed up by da Vinci or Brunelleschi. The gold florin, the local coin minted by the Florentine guilds, was for a long stretch the standard currency of Europe, and one of the first since Roman days to be honored so widely. A number of inventions that turned out to be essential to modern commercial life—double-entry accounting, to name one—date back to the golden age of the guilds. If the engine of history restarted in Italy during the twelfth and thirteenth centuries, as the canonical story goes, the guilds were its turbines.
The guild of Por Santa Maria took its name from a central street that leads directly to the ancient Ponte Vecchio, the much-photographed bridge spanning the River Arno, overloaded with shops and a secret corridor built for the Florentine duke Cosimo I in 1565. There are records of silk weavers setting up shop along the Por Santa Maria as early as 1100, a century before joining forces with the goldsmiths to form their own guild. Merchants who were in the silk trade and other wealthy Florentines could stroll down to the Por Santa Maria comparison shopping, while their servants combed the Ponte Vecchio for the meat sold by the butchers who populated the bridge for the first centuries of the millennium.
They are still there today. Walk north of the Ponte Vecchio on a weekday morning, and you’ll still find stores selling fine silks, some of them hawking processed items such as blouses and scarves, others selling the raw goods directly, as they did nearly a millennium ago.
Do cities learn? Not the individuals who populate cities, not the institutions they foster, but the cities themselves. I think the answer is yes. And the silk weavers of Florence can help explain why.
* * *
Learning is one of those activities that we habitually associate with conscious awareness—such as falling in love or mourning the loss of a relative. But learning is a complicated phenomenon that exists on a number of levels simultaneously. When we say we “learn someone’s face,” there’s a strong implication of consciousness in the statement—you feel something different when you see someone you know, and that feeling of recognition is part of what it means to learn, so much so that it can sometimes seem interchangeable with the experience itself.
But learning is not always contingent on consciousness. Our immune systems learn throughout our lifetimes, building vocabularies of antibodies that evolve in response to the threat posed by invading microorganisms. Most of us have developed immunity to the varicella-zoster virus—also known as the chicken pox—based on our exposure to it early in childhood. That immunity is a learning process: the antibodies of our immune system learn to neutralize the antigens of the virus, and they remember those neutralization strategies for the rest of our lives. We don’t come into the world predisposed to ward off the chicken pox virus—our bodies learn how to do it on the fly, without any specific training. Those antibodies function as a “recognition system,” in Gerald Edelman’s phrase, successfully attacking the virus and storing the information about it, then recalling that information the next time the virus comes across the radar.
Like a six-month-old infant, the immune system first learns to recognize things that differ from itself, then sets out to control those things. It is only part of the wonder of this process that it works as well as it does. What’s equally amazing is the fact that the recognition unfolds purely on a cellular level: we are not aware of the varicella-zoster virus in any sense of the word, and while our minds may remember what it was like to have chicken pox as a child, our conscious memory has nothing to do with our resistance to the disease.
The body learns without consciousness, and so do cities, because learning is not just about being aware of information; it’s also about storing information and knowing where to find it. It’s about being able to recognize and respond to changing patterns—the way Oliver Selfridge’s Pandemonium software does or Deborah Gordon’s harvester ants. It’s about altering a system’s behavior in response to those patterns in ways that make the system more successful at whatever goal it’s pursuing. The system need not be conscious to be capable of that kind of learning, just as your immune system need not be conscious to learn how to protect you from the chicken pox.
Imagine a contemporary citizen of Florence who time-travels back eight hundred years, to the golden age of the guilds. What would that experience—the “shock of the old”—be like? Most of it would be utterly baffling: few of modern Florence’s landmarks would exist—the Uffizi, say, or the church of San Lorenzo. Only the baptistery of the Duomo would be recognizable, as would the ancient city hall, the Bargello. The broad outline of most streets would look familiar, but in many cases their names would have changed, and our time traveler would find almost nothing recognizable in the buildings lining those streets. The cultural life of the city would be even more disconcerting: the systems of trade and governance would look nothing like those of present-day Florence. Our time traveler might catch some familiar words in the spoken tongue, since the Italian language is a product of Florentine culture, dating back to the turn of the millennium. But if he traveled anywhere else in Italy, he would face serious linguistic hurdles—until the late thirteenth century, Latin was the only language common to all Italians.
And yet, despite that abject confusion, one extraordinary thing remains constant: our time traveler would still know where to buy a yard of silk. Fast-forward a few hundred years, and he’d know where to pick up a gold bracelet as well. And where to buy leather gloves, or borrow money. He wouldn’t be equipped to buy any of these things, or even to communicate intelligibly to the salesmen—but he’d know where to find the goods all the same.
Like any emergent system, a city is a pattern in time. Dozens of generations come and go, conquerors rise and fall, the printing press appears, then the steam engine, then radio, television, the Web—and beneath all that turbulence, a pattern retains its shape: silk weavers clustered along Florence’s Por Santa Maria, the Venetian glassblowers on Murano, the Parisian traders gathered in Les Halles. The world convulses, sheds its skin a thousand times, and yet the silk weavers stay in place. We have a tendency to relegate these cross-generational patterns to the ossified nostalgia of “tradition,” admiring for purely sentimental reasons the blacksmith who works in the same shop as his late-medieval predecessors. But that continuity has much more than sentimental value, and indeed it is more of an achievement than we might initially think. That pattern in time is one of the small miracles of emergence.
Why do cities keep their shapes? Certain elements of urban life get passed on from generation to generation because they’re associated with a physical structure that has its own durability. (Cathedrals and universities are the best examples of this phenomenon—St. Peter’s Basilica has fostered a religious-themed neighborhood west of the Tiber for a thousand years, and the Left Bank has been a hotbed of student types since the Sorbonne was founded in 1257.) But because those neighborhoods are anchored by specific structures, their persistence has as much to do with the laws of physics as anything else: as long as the cathedral doesn’t burn down or disintegrate, there’s likely to be a religious flavor to the streets around it. But the Florentine silk weavers are a different matter. There’s nothing in the physical structure of the shops that mandates that they be occupied by silk weavers. (Indeed, many of the buildings along the Por Santa Maria have been rebuilt several times over the past thousand years.) They could just as easily house bankers or wine merchants or countless other craftsmen. And yet the silk weavers remain, held in place by the laws of emergence, by the city’s gift for self-organization.
You could argue that the silk weavers stay put not because they are part of an emergent system, but because they are subject to the laws of inertia. They remain clustered along the Por Santa Maria because staying put is easier than moving. (In other words, it’s not emergence we’re seeing here—it’s laziness.) The objection might make some sense if we were talking about a fifty-year span, or even a century. But on a thousand-year scale, the force of cultural drift becomes far more powerful. Technological and geopolitical changes obviously have a tremendous impact—killing off entire industries, triggering mass migrations, launching wars, or precipitating epidemics. Neighborhood clusters are extremely vulnerable to those dramatic forces of change, but they are also vulnerable to the slower, mostly invisible drift that all culture undergoes. Over twenty or thirty generations, even something as fundamental as the name of a common item can be transformed beyond recognition, and the steady but imperceptible shifts in pronunciation can make a spoken language unintelligible to listeners. However difficult it is to read Chaucer’s Canterbury Tales in the original, it would be even more disorienting to hear it read aloud by an inhabitant of fourteenth-century Britain. And if words can transform themselves over time, the changes in social mores, etiquette, and fashion are so profound as to be almost unimaginable. (Parsing the complex sexual codes of thirteenth-century Florence from a modern perspective would be a daunting task indeed.) Viewed on the scale of the millennium, the values of Florentine society look more like a hurricane than a stable social order: all turbulence and change. And yet against all those disruptive forces, the silk weavers hold their own.
Cities are blessed with an opposing force that keeps the drift and tumult of history at bay: a kind of self-organizing stickiness that allows the silk weavers to stay huddled together along the same road for a thousand years, while the rest of the world reinvents itself again and again. These clusters are like magnets planted in the city’s fabric, keeping like minds together, even as the forces of history try to break them apart. They are not limited to Italian cities, though Florence’s clusters are some of the most ancient. Think of London’s Savile Row or Fleet Street, clusters that date back hundreds of years. In Beijing, street names still echo the pockets of related businesses: Silk-Brocade Hat Alley, Dry-Noodle Street. In Manhattan today you can see the early stirrings of clusters, some of them only a few decades old: the diamond row of West Forty-seventh Street, the button district, even a block downtown devoted solely to restaurant supply stores. The jewelry merchants on West Forty-seventh don’t have quite the pedigree of their colleagues on the Ponte Vecchio, but then New York is a young city by Italian standards. Look at those Manhattan streets from the thousand-year view, the scale of the superorganism, and what comes to mind is an embryo self-organizing into recognizable shapes, forming patterns that will last a lifetime.
* * *
“From its origins onward,” Lewis Mumford writes in his classic work The City in History, “the city may be described as a structure specially equipped to store and transmit the goods of civilization.” Preeminent among the “goods” stored and transmitted by the city is the invaluable material of information: current prices in the marketplace; laborsaving devices dreamed up by craftsmen; new remedies for disease. This knack for capturing information, and for bringing related pockets of information together, defines how cities learn. Like-minded businesses cluster together because there are financial incentives to do so—what academics call economies of agglomeration—enabling craftsmen to share techniques and services that they wouldn’t necessarily be able to enjoy on their own. That clustering becomes a self-perpetuating cycle: potential consumers and employees have an easier time finding the goods and jobs they’re looking for; the shared information makes the clustered businesses more competitive than the isolated ones.
There are manifest purposes to a city—reasons for being that its citizens are usually aware of: they come for the protection of the walled city, or the open trade of the marketplace. But cities have a latent purpose as well: to function as information storage and retrieval devices. Cities were creating user-friendly interfaces thousands of years before anyone even dreamed of digital computers. Cities bring minds together and put them into coherent slots. Cobblers gather near other cobblers, and button makers near other button makers. Ideas and goods flow readily within these clusters, leading to productive cross-pollination, ensuring that good ideas don’t die out in rural isolation. The power unleashed by this data storage is evident in the earliest large-scale human settlements, located on the Sumerian coast and in the Indus Valley, which date back to 3500 B.C. By some accounts, grain cultivation, the plow, the potter’s wheel, the sailboat, the draw loom, copper metallurgy, abstract mathematics, exact astronomical observation, the calendar—all of these inventions appeared within centuries of the original urban populations. It’s possible, even likely, that more isolated groups or individuals had stumbled upon some of those technologies at an earlier date, but they didn’t become part of the collective intelligence of civilization until there were cities to store and transmit them.
The neighborhood system of the city functions as a kind of user interface for the same reason that traditional computer interfaces do: there are limits to how much information our brains can handle at any given time. We need visual interfaces on our desktop computers because the sheer quantity of information stored on our hard drives—not to mention on the Net itself—greatly exceeds the carrying capacity of the human mind. Cities are a solution to a comparable problem, both on the level of the collective and the individual. Cities store and transmit useful new ideas to the wider population, ensuring that powerful new technologies don’t disappear once they’ve been invented. But the self-organizing clusters of neighborhoods also serve to make cities more intelligible to the individuals who inhabit them—as we saw in the case of our time-traveling Florentine. The specialization of the city makes it smarter, more useful for its inhabitants. And the extraordinary thing again is that this learning emerges without anyone even being aware of it. Information management—subduing the complexity of a large-scale human settlement—is the latent purpose of a city, because when cities come into being, their inhabitants are driven by other motives, such as safety or trade. No one founds a city with the explicit intent of storing information more efficiently, or making its social organization more palatable for the limited bandwidth of the human mind. That data management only happens later, as a kind of collective afterthought: yet another macrobehavior that can’t be predicted from the micromotives. Cities may function like libraries and interfaces, but they are not built with that explicit aim.
Indeed, traditional cities—like the ones that sprouted across Europe between the twelfth and fourteenth centuries—are rarely built with any aim at all: they just happen. There are exceptions of course: imperial cities, such as St. Petersburg or Washington, D.C., laid out by master planners in the image of the state. But organic cities—Florence or Istanbul or downtown Manhattan—are more an imprint of collective behavior than the work of master planners. They are the sum of thousands of local interactions: clustering, sharing, crowding, trading—all the disparate activities that coalesce into the totality of urban living.
All of which raises the question of why—if they are so useful—cities took so long to emerge, and why history includes such long stretches of urban decline. Consider the state of Europe after the fall of the Roman Empire: for nearly a thousand years, European cities retreated back into castles and fortresses, or scattered their populations across the countryside. Imagine a time-lapse film of western Europe, as seen by a satellite, with each decade compressed down to a single second. Start the film at A.D. 100 and the continent is a hundred points of lights, humming with activity. Rome itself glows far brighter than anything else on the map, but the rest of the continent is dotted with thriving provincial capitals: Córdoba, Marseilles, even Paris is large enough to span the Left Bank. As the tape plays, though, the lights begin to dim: cities sacked by invading nomads from the East, or withered away by the declining trade lines of the Empire itself. The Parisians retreat back to their island fortress and remain there for five hundred years. When the Visigoths finally conquer Rome in 476, the satellite image suggests that the power grid of Europe has lost its primary generator: all the lights fade dramatically, and some go out altogether. The system of Europe shifts from a network of cities and towns to a scattered, unstable mix of hamlets and migrants, with the largest towns holding no more than a thousand inhabitants. It stays that way for five hundred years.
And then, suddenly, just after the turn of the millennium, the picture changes dramatically: the continent sprouts dozens of sizable towns, with populations in the tens of thousands. There are pockets on the map—at Venice or Trieste—that glow almost as brightly as ancient Rome had at the start of the tape, nascent cities supporting more than a hundred thousand citizens. The effect is not unlike watching a time-lapse film of an open field, lying dormant through the winter months, then in one sudden shift bursting with wildflowers. There is nothing gradual or linear about the change; it is as sudden, and as emphatic, as turning on a light switch. As the physicist Arthur Iberall once described the process, Europe underwent a transition not unlike that between H2O molecules changing from the fluid state of water to the crystallized state of ice: for centuries the population is liquid and unsettled—and then, suddenly, a network of towns comes into existence, possessing a stable structure that would persist more or less intact until the next great transformation in the nineteenth century, during the rise of the industrial metropolis.
How can that sudden takeoff be explained? Cities aren’t ideas that spread, viruslike, through larger populations; the town system of the Middle Ages didn’t reproduce by spores, the way the city-states of ancient Greece did. And of course, Europe was no longer united by an empire, so there was no command center to decree that a hundred cities should be built in the span of two centuries. How then can we account for the strikingly coordinated urban blossoming of the Middle Ages?
Start by taking the analogies literally. Why does a field of wildflowers suddenly bloom in the spring? Why does water turn to ice? Both systems undergo “phase transitions”—changing from one defined state to another at a critical juncture—in response to changing levels of energy flowing through them. Leave a kettle of water sitting at room temperature in your kitchen, and it will retain its liquid form for weeks. But increase the flow of energy through the kettle by putting it on a hot stove, and within minutes you’ll induce a phase transition in the water, transforming it into a gas. Take a field of tall meadow buttercups accustomed to nightly frost and ten hours of sun, then raise the temperature thirty degrees and add four hours of sunlight. After a month or two, your field will be golden yellow with buttercups. A linear increase in energy can produce a nonlinear change in the system that conducts that energy, a change that would be difficult to predict in advance—assuming, that is, you’d never seen a flowering plant before, or a steam room.
The urban explosion of the Middle Ages is an example of the same phenomenon. We saw before that the idea of building cities didn’t spread through Europe via word of mouth, but what did spread through Europe, starting around A.D. 1000, were a series of technological advances that combined to produce a dramatic change in the human capacity for harnessing energy flows. As the historian Lynn White Jr. writes, “These innovations . . . consolidated to form a remarkably efficient new way of exploiting the soil.” First, the heavy wheeled plow, which tapped the muscular energy of domesticated animals, arrived with the German invaders, then swept through the river valleys north of the Loire; at roughly the same time, European farmers adopted triennial field rotation, which increased land productivity by at least a third. Capturing more energy from the soil meant that larger population densities could be maintained. As larger towns began to form, another soil-based technology became commonplace, one that was even more environmentally friendly: recycling the waste products generated by town residents in the form of crop fertilizer. As Mumford writes, “Wooded areas in Germany, a wilderness in the ninth century, gave way to plowland; the boggy Low Countries, which had supported only a handful of hardy fishermen, were transformed into one of the most productive soils in Europe.” The result is a positive feedback loop: the plow and the crop rotation makes better soil, which supplies enough energy to sustain towns, which generate enough fertilizer to make better soil, which generates enough energy to sustain even larger towns.
We sometimes talk about emergent systems “bootstrapping” themselves into existence, but in the case of the Middle Ages, we can safely say that the early village residents shat themselves into full-fledged towns. But those residents aren’t setting out to build bigger settlements; they’re all solving local problems, such as how to make their fields more productive, or what to do with all the human waste of a busy town. And yet those local decisions combine to form the macrobehavior of the urban explosion. “This acceleration in urban development,” writes philosopher-historian Manuel De Landa, “would not be matched for another five hundred years, when a new intensification of the flow of energy—this time arising from the exploitation of fossil fuels—propelled another great spurt of city birth and growth in the 1800s.” And with that new flow of energy, new kinds of cities emerged: the factory towns of Manchester and Leeds, and the great metropolitan superorganisms of London, Paris, and New York.
* * *
We are, by all accounts, in the midst of another technological revolution—an information age, a time of near-infinite connectedness. If information storage and retrieval was the latent purpose of the urban explosion of the Middle Ages, it is the manifest purpose of the digital revolution. All of which raises the question, is the Web learning as well? If cities can generate emergent intelligence, a macrobehavior spawned by a million micromotives, what higher-level form is currently taking shape among the routers and fiber-optic lines of the Internet?
I first started thinking about this question a few years ago, during the promotional tour for my last book, Interface Culture. As it happened, my book’s publisher also specialized in “contemporary spiritual” titles, and so the in-house publicist sent galleys of what I thought was a decidedly un–New Agey book to every New Age radio station, print zine, and ashram in the country. What’s more, some of them ended up taking the bait, and so the tour assumed a slightly schizophrenic air: NPR in the morning, followed by a Q&A with alternative magazines like San Francisco’s Magical Blend in the afternoon.
The questions from the Harmonic Convergence set turned out to be as consistently smart and forward-thinking and technologically adept as any I’d encountered on the rest of the tour. The New Agers were sensitive to the nuances of my argument, and refreshingly indifferent to the latest IPO pricing. (Contrast that with the TV reporters, who seemed incapable of asking me anything other than “What’s your take on Yahoo’s market cap?”) But just when I’d start kicking myself for embarking on the interview with such prejudice, my interlocutors would roll out a Final Question. “You’ve written a great deal about the Web and its influence on modern society,” they’d say. “Do you think, in the long term, that the rise of the Web is leading towards a single, global, holistic consciousness that will unite us all in godhead?” I’d find myself stammering into the microphone, looking for exit signs.
It’s a question with only one responsible answer: “I’m not qualified to answer that.” And each time I said this, I thought to myself that something was fundamentally flawed about the concept, something close to a category mistake. For there to be a single, global consciousness, the Web itself would have to be getting smarter, and the Web wasn’t a single, unified thing—it was just a vast sum of interlinked data. You could debate whether the Web was making us smarter, but that the Web itself might be slouching toward consciousness seemed ludicrous.
But as the years passed, I found that the question kept bouncing around in my head, and slowly I started to warm up to it, in a roundabout way. Some critics, such as Robert Wright, talk about a “global brain” uniting all the world’s disparate pools of information, while other visionaries—such as Bill Joy and Ray Kurzweil—believe that the computational powers of digital technology are accelerating at such a rate that large networks of computers may actually become self-aware sometime in the next century.
Did Arthur C. Clarke and The Matrix have it right all along? Is the Web itself becoming a giant brain? I still think the answer is no. But now I think it’s worth asking why not.
* * *
Begin by jettisoning two habitual ways of thinking about what a brain is. First, forget about gray matter and synapses. When someone like Wright says “giant brain,” he means a device for processing and storing information, like the clustered neighborhoods of Florence. Second, accept the premise that brains can be a collective enterprise. Being individual organisms ourselves, we’re inclined to think of brains as discrete things, possessed by individual organisms. But both categories turn out to be little more than useful fictions. As we’ve seen, ants do their “learning” at the colony level—growing less aggressive with age, or rerouting a food assembly line around a disturbance—while the individual ants remain blissfully ignorant of the larger project. The “colony brain” is the sum of thousands and thousands of simple decisions executed by individual ants. The individual ants don’t have anything like a personality, but the colonies do.
Replace ants with neurons, and pheromones with neurotransmitters, and you might just as well be talking about the human brain. So if neurons can swarm their way into sentient brains, is it so inconceivable that the process might ratchet itself up one more level? Couldn’t individual brains connect with one another, this time via the digital language of the Web, and form something greater than the sum of their parts—what the trendy philosopher/priest Teilhard de Chardin called the noosphere? Wright’s not exactly convinced that the answer is yes, but he’s willing to go on the record that the question is, as he puts it, “noncrazy”:
Today’s talk of a giant global brain is cheap. But there’s a difference. These days, most people who talk this way are speaking loosely. Tim Berners-Lee, who invented the World Wide Web, has noted parallels between the Web and the structure of the brain, but he insists that “global brain” is mere metaphor. Teilhard de Chardin, in contrast, seems to have been speaking literally: humankind was coming to constitute an actual brain—like the one in your head, except bigger. Certainly there are more people today than in Teilhard’s day who take the idea of a global brain literally Are they crazy? Was Teilhard crazy? Not as crazy as you might think.
Part of Wright’s evidence here is that Homo sapiens brains already have a long history of forming higher-level intelligence. Individual human minds have coalesced into “group brains” many times in modern history, most powerfully in the communal gatherings of cities. In Wright’s view, the city functions as a kind of smaller-scale trial run for the Web’s worldwide extravaganza, like an Andrew Lloyd Webber musical that gets the kinks out in Toronto before opening on Broadway. As in the urban explosion of the Middle Ages, a city is not just an accidental offshoot of growing population density—it’s a kind of technological breakthrough in its own right. Sustainable city life ranks high on the list of modern inventions—as world-transforming as the alphabet (which it helped engender) or the Internet (which may well be its undoing). It’s no coincidence that the great majority of the last millennium’s inventions blossomed in urban settings. Like the folders and file directories of some oversize hard drive, the group brain of city life endowed information with far more structure and durability than it had previously possessed. Wright’s position is that the Web looks to be the digital heir to that proud tradition, uniting the world’s intellects in a way that would have astonished the early networkers of Florence or Amsterdam. Macrointelligence emerged out of the bottom-up organization of city life, he argues, and it will do the same on the Web.
I’m obviously sympathetic to Wright’s argument, but I think it needs clarifying. Emergence isn’t some mystical force that comes into being when agents collaborate; as in the freeways vs. sidewalks debate, there are environments that facilitate higher-level intelligence, and environments that suppress it. To the extent that the Web has connected more sentient beings together than any technology before it, you can see it as a kind of global brain. But both brains and cities do more than just connect, because intelligence requires both connectedness and organization. Plenty of decentralized systems in the real world spontaneously generate structure as they increase in size: cities organize into neighborhoods or satellites; the neural connections of our brains develop extraordinarily specialized regions. Has the Web followed a comparable path of development over the past few years? Is the Web becoming more organized as it grows?
You need only take a quick look at the NASDAQ most active list to see that the answer is an unequivocal no. The portals and the search engines exist in the first place because the Web is a tremendously disorganized space, a system where the disorder grows right alongside the overall volume. Yahoo and Google function, in a way, as man-made antidotes to the Web’s natural chaos—an engineered attempt to restore structure to a system that is incapable of generating structure on its own. This is the oft-noted paradox of the Web: the more information that flows into its reservoirs, the harder it becomes to find any single piece of information in that sea.
Imagine the universe of HTML documents as a kind of city spread out across a vast landscape, with each document representing a building in that space. The Web’s city would be more anarchic than any real-world city on the planet—no patches of related shops and businesses; no meatpacking or theater districts; no bohemian communities or upscale brownstones; not even the much-lamented “edge city” clusters of Los Angeles or Tyson’s Corner. The Web’s city would simply be an undifferentiated mass of data growing more confusing with each new “building” that’s erected—so confusing, in fact, that the mapmakers (the Yahoos and Googles of the world) would generate almost as much interest as the city itself.
And if the Web would make a miserable city, it would do even worse as a brain. Here’s Steven Pinker, the author of How the Mind Works, in a Slate dialogue with Wright:
The Internet is in some ways like a brain, but in important ways not. The brain doesn’t just let information ricochet around the skull. It is organized to do something: to move the muscles in ways that allow the whole body to attain the goals set by the emotions. The anatomy of the brain reflects that: it is not a uniform web or net, but has a specific organization in which emotional circuits interconnect with the frontal lobes, which receive information from perceptual systems and send commands to the motor system. This goal-directed organization comes from an important property of organisms you discuss: their cells are in the same reproductive boat, and thus have no “incentive” to act against the interests of the whole body. But the Internet, not being a cohesive replicating system, has no such organization.
Again, the point here is that intelligent systems depend on structure and organization as much as they do on pure connectedness—and that intelligent systems are guided toward particular types of structure by the laws of natural selection. A latter-day Maxwell’s Demon who somehow manages to superglue a billion neurons to each other wouldn’t build anything like the human brain, because the brain relies on specific clusters to make sense of the world, and those clusters only emerge out of a complex interplay among neurons, the external world, and our genes (not to mention a few thousand other factors). Some systems, such as the Web, are geniuses at making connections but lousy with structure. The technologies behind the Internet—everything from the microprocessors in each Web server to the open-ended protocols that govern the data itself—have been brilliantly engineered to handle dramatic increases in scale, but they are indifferent, if not downright hostile, to the task of creating higher-level order. There is, of course, a neurological equivalent of the Web’s ratio of growth to order, but it’s nothing you’d want to emulate. It’s called a brain tumor.
Still, in the midst of all that networked chaos, a few observers have begun to detect macropatterns in the Web’s development, patterns that are invisible to anyone using the Web, and thus mostly useless. The distribution of Web sites and their audiences appears to follow what is called a power law: the top ten most popular sites are ten times larger than the next hundred more popular sites, which are themselves ten times more popular than the next thousand sites. Other online cartographers have detected “hub” and “spoke” patterns in traffic flows. But none of these macroshapes, even if they do exist, actually makes the Web a more navigable or informative system. These patterns may be self-organizing, but they are not adaptive in any way. The patterns are closer to a snowflake’s intricacy than a brain’s neural net: the snowflake self-organizes into miraculously complicated shapes, but it’s incapable of becoming a smarter snowflake, or a more effective one. It’s simply a frozen pattern. Compare that to the living, dynamic patterns of a city neighborhood or the human brain: both shapes have evolved into useful structures because they have been pushed in that direction by the forces of biological or cultural evolution: our brains are masterpieces of emergence because large-brained primates were, on the whole, more likely to reproduce than their smaller-brained competitors; the trade clusters of the modern city proliferated because their inhabitants prospered more than isolated rural craftsmen. There is great power and creative energy in self-organization, to be sure, but it needs to be channeled toward specific forms for it to blossom into something like intelligence.
But the fact that the Web as we know it tends toward chaotic connections over emergent intelligence is not something intrinsic to all computer networks. By tweaking some of the underlying assumptions behind today’s Web, you could design an alternative version that could potentially mimic the self-organizing neighborhoods of cities or the differentiated lobes of the human brain—and could definitely reproduce the simpler collective problem-solving of ant colonies. The Web’s not inherently disorganized, it’s just built that way. Modify its underlying architecture, and the Web might very well be capable of the group-think that Teilhard envisioned.
How could such a change be brought about? Think about Deborah Gordon’s harvester ants, or Paul Krugman’s model for edge-city growth. In both systems, the interaction between neighbors is two-way: the foraging ant that stumbles across the nest-building ant registers something from the encounter, and vice versa; the new store that opens up next to an existing store influences the behavior of that store, which in turn influences the behavior of the newcomer. Relationships in these systems are mutual: you influence your neighbors, and your neighbors influence you. All emergent systems are built out of this kind of feedback, the two-way connections that foster higher-level learning.
Ironically, it is precisely this feedback that the Web lacks, because HTML-based links are one-directional. You can point to ten other sites from your home page, but there’s no way for those pages to know that you’re pointing to them, short of you taking the time to fire off an e-mail to their respective webmasters. Every page on the Web contains precise information about the other addresses it points to, and yet, by definition, no page on the Web knows who’s pointing back. It’s a limitation that would be unimaginable in any of the other systems that we’ve looked at. It’s like a Gap outlet that doesn’t realize that J.Crew just moved in across the street, or an ant that remains oblivious to the other ants it stumbles across in its daily wanderings. The intelligence of a harvester ant colony derives from the densely interconnected feedback between ants that encounter each other and change their behavior according to preordained rules. Without that feedback, they’d be a random assemblage of creatures butting heads and moving on, incapable of displaying the complex behavior that we’ve come to expect from the social insects. (The neural networks of the brain are also heavily dependent on feedback loops.) Self-organizing systems use feedback to bootstrap themselves into a more orderly structure. And given the Web’s feedback-intolerant, one-way linking, there’s no way for the network to learn as it grows, which is why it’s now so dependent on search engines to rein in its natural chaos.
Is there a way around this limitation? In fact, a solution exists already, although it does nothing to modify the protocols of the Web, but rather ingeniously works around the shortcomings of HTML to create a true learning network that sits on top of the Web, a network that exists on a global scale. Appropriately enough, the first attempt to nurture emergent intelligence online began with the desire to keep the Web from being so forgetful.
* * *
You can’t really, truly understand Brewster Kahle until you’ve had him show you the server farm in Alexa Internet’s basement. Walk down a flight of outdoor steps at the side of an old military personnel-processing building in San Francisco’s Presidio, and you’ll see an entire universe of data—or at least a bank of dark-toned Linux servers arrayed along a twenty-foot wall. The room itself—moldy concrete, with a few spare windows gazing out at foot level—might have held a lawn mower and some spare file cabinets a few decades ago. Now it houses what may well be the most accurate snapshot of The Collective Intelligence anywhere in the world: thirty terabytes of data, archiving both the Web itself and the patterns of traffic flowing through it.
As the creator of the WAIS (Wide Area Information Server) system, Kahle was already an Internet legend when he launched Alexa in 1996. The Alexa software used collaborative-filtering-like technology to build connections between sites based on user traffic. The results from its technology are showcased in the “related sites” menu option found in most browsers today. Amazon.com acquired Alexa Internet in 1999, but the company remains happily ensconced in its low-tech Presidio offices, World War II temporary structures filled with the smell of the nearby eucalyptus trees. “In just three years we got bigger than the Library of Congress, the biggest library on the planet,” Kahle says, arms outstretched in his basement server farm. “So the question is, what do we do now?”
Obsessed with the impermanence of today’s datastreams, Kahle (and his partner, Bruce Gilliat) founded Alexa with the idea of taking “snapshots” of the Web and indexing them permanently on huge storage devices for the benefit of future historians. As they developed that project, it occurred to them that they could easily open up that massive database to casual Web surfers, supplementing their Web browsing experience with relevant pages from the archive. Anytime a surfer encountered a “404 Page Not Found” error—meaning that an old page had been deleted or moved—he or she could swiftly consult the Alexa archive and pull up the original page.
To make this possible, Kahle and Gilliat created a small toolbar that launches alongside your Web browser. Once the application detects a URL request, it scurries off to the Alexa servers, where it queries the database for information about the page you’re visiting. If the URL request ends in a File Not Found message, the Alexa application trolls through the archives for an earlier version of the page. Kahle dubbed his toolbar a “surf engine”—a tool that accompanies you as you browse—and he quickly realized that he’d stumbled across a program that could do far more than just resuscitate old Web pages. By tracking the surfing patterns of its users, the software could also make connections between Web sites, connections that might otherwise have been invisible, both to the creators of those sites and the people browsing them.
Two months after starting work on Alexa, Kahle added a new button to his toolbar, with the simple but provocative tag “What’s Next?” Click on the button while visiting a Marilyn Monroe tribute site, and you’ll find a set of links to other Marilyn shrines online; click while you’re visiting a community site for cancer survivors, and you’ll find a host of other like-minded sites listed in the pull-down menu. How are these connections formed? By watching traffic patterns, and looking for neighbors. The software learns by watching the behavior of Alexa’s users: if a hundred users visit FEED and then hop over to Salon, then the software starts to perceive a connection between the two Web sites, a connection that can be weakened or strengthened as more behavior is tracked. In other words, the associations are not the work of an individual consciousness, but rather the sum total of thousands and thousands of individual decisions, a guide to the Web created by following an unimaginable number of footprints.
It’s an intoxicating idea, and strangely fitting. After all, a guide to the entire Web should be more than just a collection of handcrafted ratings. As Kahle says, “Learning from users is the only thing that scales to the size of the Web.” And that learning echoes the clustered neighborhoods of Florence or London. Alexa’s power of association—this site is like these other sites—emerges out of the desultory travels of the Alexa user base; none of those users are deliberately setting out to create clusters of related sites, to endow the Web with much-needed structure. They simply go about their business, and the system itself learns by watching. Like Gordon’s harvester ants, the software gets smarter, grows more organized, the more individual surfing histories it tracks. If only a thousand people fire up Alexa alongside their browsers, the recommendations simply won’t have enough data behind them to be accurate. But add another ten thousand users to the mix, and the site associations gain resolution dramatically. The system starts to learn.
Let’s be clear about what that learning entails, because it differs significantly from the traditional sci-fi portraits of computerized intelligence, both utopian and dystopian. Alexa makes no attempt to simulate human intelligence or consciousness directly. In other words, you don’t teach the computer to read or appreciate Web site design. The software simply looks for patterns in numbers, like the foraging ants counting the number of fellow foragers they encounter per hour. In fact, the “intelligence” of Alexa is really the aggregated wisdom of the thousands—or millions—of people who use the system. The computer churns through the millions of ratings in its database, looks for patterns of likes and dislikes, then reports back to the user with its findings.
It’s worth noting here that Alexa is not truly a “recommendation agent”; it is not telling you that you’ll like the five sites that it suggests. It’s saying that there’s a relationship between the site you’re currently visiting and the sites listed on the pull-down menu. The clusters that form via Alexa are clusters of association, and the links between them are not unlike the traditional links of hypertext. Think about the semantics of a hypertext link embedded in an online article: when you see that link, you don’t translate it as “If you like this sentence, you’ll like this page as well.” The link isn’t recommending another page; it’s pointing out that there’s a relationship between the sentence you’re reading and the page at the other end of the link. It’s still up to you to decide if you’re interested in the other sites, just as it’s up to you to decide which silk merchant you prefer on the Por Santa Maria. Alexa’s simply there to show you where the clusters are.
Outside of the video-game world, Alexa may be the most high-profile piece of emergent software to date: the tool was integrated into the Netscape browser shortly after its release, and the company is now applying its technology to the world of consumer goods. But the genre is certainly diversifying. An East Coast start-up called Abuzz, recently acquired by the New York Times digital unit, offers a filtering service that enables people searching for particular information or expertise to track down individuals who might have the knowledge they’re looking for. A brilliant site called Everything2 employs a neural-net-like program to create a user-authored encyclopedia, with related entries grouped together, Alexa-style, based on user traffic patterns. Indeed, the Web industry is teeming with start-ups promising to bring like minds together, whether they’re searching for entertainment or more utilitarian forms of information. These are the digital-age heirs to the Por Santa Maria.
Old-school humanists, of course, tend to find something alarming in the idea of turning to computers for expert wisdom and cultural sensibility. In most cases, the critics’ objections sound like a strangely inverted version of the old morality tales that once warned us against animating machines: Goethe’s (and Disney’s) sorcerer’s apprentice, Hoffmann’s sandman, Shelley’s Frankenstein. In the contemporary rendition, it’s not that the slave technology grows stronger than us and learns to disobey our commands—it’s that we deteriorate to the level of the machines. Smart technology makes us dumber.
The critique certainly has its merits, and even among the Net community—if it’s still possible to speak of a single Net community—intelligent software remains much villified in some quarters. Decades ago, in a curiously brilliant book, God and Golem, Inc., Norbert Wiener argued that “in poems, in novels, in painting, the brain seems to find itself able to work very well with material that any computer would have to reject as formless.” For many people the distinction persists to this day: we look to our computers for number crunching; when we want cultural advice, we’re already blessed with plenty of humans to consult. Other critics fear a narrowing of our aesthetic bandwidth, with agents numbly recommending the sites that everyone else is surfing, all the while dressing their recommendations up in the sheep’s clothing of custom-fit culture.
But it does seem a little silly to resist the urge to experiment with the current cultural system, where musical taste is usually determined by the marketing departments at Sony and Dreamworks, and expert wisdom comes in the form of Ann Landers columns and the Psychic Hotline. If the computer is, in the end, merely making connections between different cultural sensibilities, sensibilities that were originally developed by humans and not by machines, then surely the emergent software model is preferable to the way most Westerners consume entertainment: by obeying the dictates of advertising. Software like Alexa isn’t trying to replicate the all-knowing authoritarianism of Big Brother or HAL, after all—it’s trying to replicate the folksy, communal practice of neighbors sharing information on a crowded sidewalk, even if the neighbors at issue are total strangers, communicating to each other over the distributed network of the Web.
* * *
The pattern-seeking algorithms of emergent software are already on their way to becoming one of the primary mechanisms in the great Goldberg contraption of modern social life—as familiar to us as more traditional devices like supply and demand, representational democracy, snap polls. Intelligent software already scans the wires for constellations of book lovers or potential mates. In the future, our networks will be caressed by a million invisible hands, seeking patterns in the digital soup, looking for neighbors in a land where everyone is by definition a stranger.
Perhaps this is only fitting. Our brains got to where they are today by bootstrapping out of a primitive form of pattern-matching. As the futurist Ray Kurzweil writes, “Humans are far more skilled at recognizing patterns than in thinking through logical combinations, so we rely on this aptitude for almost all of our mental processes. Indeed, pattern recognition comprises the bulk of our neural circuitry. These faculties make up for the extremely slow speed of human neurons.” The human mind is poorly equipped to deal with problems that need to be solved serially—one calculation after another—given that neurons require a “reset time” of about five milliseconds, meaning that neurons are capable of only two hundred calculations per second. (A modern PC can do millions of calculations per second, which is why we let them do the heavy lifting for anything that requires math skills.) But unlike most computers, the brain is a massively parallel system, with 100 billion neurons all working away at the same time. That parallelism allows the brain to perform amazing feats of pattern recognition, feats that continue to confound digital computers—such as remembering faces or creating metaphors. Because each individual neuron is so slow, Kurzweil explains, “we don’t have time . . . to think too many new thoughts when we are pressed to make a decision. The human brain relies on precomputing its analyses and storing them for future reference. We then use our pattern-recognition capability to recognize a situation as compatible to one we have thought about and then draw upon our previously considered conclusions.”
It’s conceivable that the software of today lies at the evolutionary foothills of some larger, distributed consciousness to come, like the SKYNET network from the Terminator films that “became self-aware on August 15, 1997.” Certainly the evidence suggests that genuinely cognizant machines are still on the distant technological horizon, and there’s plenty of reason to suspect they may never arrive. But the problem with the debate over machine learning and intelligence is that it has too readily been divided between the mindless software of today and the sentient code of the near future. The Web may never become self-aware in any way that resembles human self-awareness, but that doesn’t mean the Web isn’t capable of learning. Our networks will grow smarter in the coming years, but smarter in the way that an immune system or a city grows smarter, not the way a child does. That’s nothing to apologize for—an adaptive information network capable of complex pattern recognition could prove to be one of the most important inventions in all of human history. Who cares if it never actually learns how to think for itself?
An emergent software program that tracks associations between Web sites or audio CDs doesn’t listen to music; it follows purchase patterns or listening habits that we supply and lets us deal with the air guitar and the off-key warbling. On some basic human level, that feels like a difference worth preserving. And maybe even one that we won’t ever be able to transcend, a hundred years from now or more. But is it truly a difference in kind, or is it just a difference in degree? This is the question that has haunted the artificial intelligence community for decades now, and it hits close to home in any serious discussion of emergent software. Yes, the computer doesn’t listen to music or browse the Web; it looks for patterns in data and converts those patterns into information that is useful—or at least aims to be useful—to human beings. Surely this process is miles away from luxuriating in “The Goldberg Variations,” or reading Slate.
But what is listening to music if not the search for patterns—for harmonic resonance, stereo repetition, octaves, chord progressions—in the otherwise dissonant sound field that surrounds us every day? One tool scans the zeros and ones on a magnetic disc. The other scans the frequency spectrum. What drives each process is a hunger for patterns, equivalencies, likenesses; in each the art emerges out of perceived symmetry. (Bach, our most mathematical composer, understood this better than anyone else.) Will computers ever learn to appreciate the patterns they detect? It’s too early to tell. But in a world where the information accessible online is doubling every six months, it is clear that some form of pattern-matching—all those software programs scouring the Net for signs of common behavior, relevant ideas, shared sensibilities—will eventually influence much of our mediated lives, maybe even to the extent that the pattern-seekers are no longer completely dependent on the commands of the masters, just as city neighborhoods grow and evolve beyond the direct control of their inhabitants. And where will that leave the software then? What makes music different from noise is that music has patterns, and our ears are trained to detect them. A software application—no matter how intelligent—can’t literally hear the sound of all those patterns clicking into place. But does that make its music any less sweet?