Think of the ten ideas presented here as concentric circles. We start small and direct, close to the technology, focusing on specific mechanisms for imposing constraints by design. From there each idea gets progressively broader, ascending a ladder of interventions further away from the hard technical specifics, the raw code and materials, and moving up and out toward the nontechnical but no less important actions, the kinds that add up to new business incentives, reformed government, international treaties, a healthier technological culture, and a popular global movement.
It’s the way all these layers of the onion build that makes them powerful; each alone is insufficient. Each necessitates very different kinds of interventions, with different skills, competencies, and people; each is generally its own vast and specialized subfield. Collectively, I believe, they could add up to something that works.
Let’s start at the beginning, with the technology itself.
A few years ago, many large language models had a problem. They were, to put it bluntly, racist. Users could quite easily find ways of making them regurgitate racist material, or hold racist opinions they had gleaned in scanning the vast corpus of texts on which they’d been trained. Toxic bias was, it seemed, ingrained in human writing and then amplified by AI. This led many to conclude the whole setup was ethically broken, morally nonviable; there was no way LLMs could be controlled well enough to be released to the public given the obvious harms.
But then LLMs, as we have seen, took off. In 2023 it’s now clear that, compared with the early systems, it is extremely difficult to goad something like ChatGPT into racist comments. Is it a solved problem? Absolutely not. There are still multiple examples of biased, even overtly racist, LLMs, as well as serious problems with everything from inaccurate information to gaslighting. But for those of us who have worked in the field from the beginning, the exponential progress at eliminating bad outputs has been incredible, undeniable. It’s easy to overlook quite how far and fast we’ve come.
A key driver behind this progress is called reinforcement learning from human feedback. To fix their bias-prone LLMs, researchers set up cunningly constructed multi-turn conversations with the model, prompting it to say obnoxious, harmful, or offensive things, seeing where and how it goes wrong. Flagging these missteps, researchers then reintegrate these human insights into the model, eventually teaching it a more desirable worldview, in a way not wholly dissimilar from how we try to teach children not to say inappropriate things at the dinner table. As engineers became more aware of their systems’ inherent ethical problems, they became more open to finding technical innovations to help address them.
Addressing the racism and bias in LLMs is an example of how careful and responsible deployment is necessary to advance the safety of these models. Contact with reality helps developers learn, correct, and improve their safety.
While it’s wrong to say technical fixes alone can solve the social and ethical problems engendered by AI, it does show how they will be a part of it. Technical safety, up close, in the code, in the lab, is the first item on any containment agenda.
Hear the word “containment” and, assuming you’re not an international relations scholar, chances are you think of the physical sense of keeping something in. To be sure, physically containing technology is important. We’ve seen, for example, how even BSL-4 labs can leak. What kind of environment might make that fully impossible? What does a BSL-7 or -n look like?
Although I argued in the last chapter that containment shouldn’t be reduced to a kind of magic box, it doesn’t mean we don’t want to figure out ways of building one as part of it. The ultimate control is hard physical control, of servers, microbes, drones, robots, and algorithms. “Boxing” an AI is the original and basic form of technological containment. This would involve no internet connections, limited human contact, a small, constricted external interface. It would, literally, contain it in physical boxes with a definite location. A system like this—called an air gap—could, in theory, stop an AI from engaging with the wider world or somehow “escaping.”
Physical segregation is just one aspect of transforming technical safety architecture to meet the challenge of the next wave. Taking the best of what’s out there is a start. Nuclear power, for instance, gets a bad rep thanks to well-known disasters like Chernobyl and Fukushima. But it’s actually remarkably safe. The International Atomic Energy Agency has published more than a hundred safety reports tackling specific technical standards for given situations, from the classification of radioactive waste to preparedness in cases of emergency. Bodies like the Institute of Electrical and Electronics Engineers maintain more than two thousand technical safety standards on technologies ranging from autonomous robot development to machine learning. Biotech and pharma have operated under safety standards far beyond those of most software businesses for decades. It’s worth remembering just how safe years of effort have made many existing technologies—and building on it.
Frontier AI safety research is still an undeveloped, nascent field focusing on keeping ever more autonomous systems from superseding our ability to understand or control them. I see these questions around control or value alignment as subsets of the wider containment problem. While billions are plowed into robotics, biotech, and AI, comparatively tiny amounts get spent on a technical safety framework equal to keeping them functionally contained. The main monitor of bioweapons, for example, the Biological Weapons Convention, has a budget of just $1.4 million and only four full-time employees—fewer than the average McDonald’s.
The number of AI safety researchers is still minuscule: up from around a hundred at top labs worldwide in 2021 to three or four hundred in 2022. Given there are around thirty to forty thousand AI researchers today (and a similar number of people capable of piecing together DNA), it’s shockingly small. Even a tenfold hiring spree—unlikely given talent bottlenecks—wouldn’t address the scale of the challenge. Compared with the magnitude of what could go wrong, safety and ethics research on AI is marginal. Only a handful of institutions, owing to the challenges of resources, take technical safety issues seriously. And yet safety decisions made today will alter the future course of technology and humanity.
There’s a clear must-do here: encourage, incentivize, and directly fund much more work in this area. It’s time for an Apollo program on AI safety and biosafety. Hundreds of thousands should be working on it. Concretely, a good proposal for legislation would be to require that a fixed portion—say, a minimum of 20 percent—of frontier corporate research and development budgets should be directed toward safety efforts, with an obligation to publish material findings to a government working group so that progress can be tracked and shared. The original Apollo missions were expensive and onerous, but they showed the right immense level of ambition, and their can-do attitude in the face of daunting odds catalyzed the development of technologies from semiconductors and software to quartz clocks and solar panels. This could do something similar for safety.
Although numbers are currently small, I know from experience that a groundswell of interest is emerging around these questions. Students and other young people I meet are buzzing about issues like AI alignment and pandemic preparedness. Talk to them and it’s clear the intellectual challenge appeals, but they’re also drawn to the moral imperative. They want to help, and feel a duty to do better. I’m confident that if the jobs and research programs are there, the talent will follow.
For the technical safety experts of tomorrow, there are plenty of promising directions to explore. Pandemic preparedness could, for example, be greatly enhanced by using low-wavelength lightbulbs that kill viruses. Giving off light with a wavelength between 200 and 230 nanometers, close to the ultraviolet spectrum, they can kill viruses while not penetrating the outer layer of the skin: a powerful weapon against pandemics and the spread of disease more widely. And if the COVID-19 pandemic taught us one thing, it’s the value of an integrated, accelerated approach across research, rollout, and regulation for novel vaccines.
In AI, technical safety also means sandboxes and secure simulations to create provably secure air gaps so that advanced AIs can be rigorously tested before they are given access to the real world. It means much more work on uncertainty, a major focus right now—that is, how does an AI communicate when it might be wrong? One of the issues with LLMs is that they still suffer from the hallucination problem, whereby they often confidently claim wildly wrong information as accurate. This is doubly dangerous given they often are right, to an expert level. As a user, it’s all too easy to be lulled into a false sense of security and assume anything coming out of the system is true.
At Inflection, for example, we are finding ways to encourage our AI called Pi—for personal intelligence—to be cautious and uncertain by default, and to encourage users to remain critical. We’re designing Pi to express self-doubt, solicit feedback frequently and constructively, and quickly give way assuming the human, not the machine, is right. We and others are also working on an important track of research that aims to fact-check a statement by an AI using third-party knowledge bases we know to be credible. Here it’s about making sure AI outputs provide citations, sources, and interrogable evidence that a user can further investigate when a dubious claim arises.
Explanation is another huge technical safety frontier. Recall that at present no one can explain why, precisely, a model produces the outputs it does. Devising ways for models to comprehensively explain their decisions or open them to scrutiny has become a critical technical puzzle for safety researchers. It’s still early days for this research, but there are some promising signs that AI models might be able to provide justifications for their outputs, if not yet causal reasoning for them, although it is still unclear how reliable these will be.
There’s also great work being done in using simplified architectures to explore more complex ones, even on automating the process of alignment research itself: building AIs to help us contain AI. Researchers are working on a generation of “critic AIs” that can monitor and give feedback on other AI outputs with the goal of improving them at speeds and scales that humans cannot match—speeds and scales that we see in the coming wave. Managing powerful tools itself requires powerful tools.
The computer scientist Stuart Russell proposes using the kind of built-in systematic doubt we are exploring at Inflection to create what he calls “provably beneficial AI.” Rather than give an AI a set of fixed external objectives contained in what’s known as a written constitution, he recommends that systems gingerly infer our preferences and ends. They should carefully watch and learn. In theory, this should leave more room for doubt within systems and avoid perverse outcomes.
Many key challenges remain: How can you build secure values into a powerful AI system potentially capable of overriding its own instructions? How might AIs infer these values from humans? Another ongoing question is how to crack the problem of “corrigibility,” ensuring that it is always possible to access and correct systems. If you think all of this sounds like fairly fundamental must-have safety features of advanced AI, you’d be right. Progress here needs to keep up.
We should also build robust technical constraints into the development and production process. Think of how all modern photocopiers and printers are built with technology preventing you from copying or printing money, with some even shutting down if you try. For example, resource caps on the amount of training compute used to create models could place limits on the rate of progress (across that dimension at least). Performance might be throttled so that a model can run only on certain tightly controlled hardware. AI systems could be built with cryptographic protections ensuring model weights—the most valuable IP in the system—can be copied only a limited number of times or only in certain circumstances.
The highest-level challenge, whether in synthetic biology, robotics, or AI, is building a bulletproof off switch, a means of closing down any technology threatening to run out of control. It’s raw common sense to always ensure there is an off switch in any autonomous or powerful system. How to do this with technologies that are as distributed, protean, and far-reaching as in the coming wave—technologies whose precise form isn’t yet clear, technologies that in some cases might actively resist—is an open question. It’s a huge challenge. Do I think it’s possible? Yes—but no one should downplay for a second the scale of how hard it will be.
Too much safety work is incremental, focused on narrow impact assessments, small technical issues, or fixing problems that flare up postlaunch rather than working on foundational issues ahead of time. Instead, we should identify problems early and then invest more time and resources in the fundamentals. Think big. Create common standards. Safety features should not be afterthoughts but inherent design properties of all these new technologies, the ground state of everything that comes next. Despite the fierce challenges, I’m genuinely excited by the range and ingenuity of ideas here. Let’s give them the intellectual oxygen and material support to succeed, recognizing that while engineering is never the whole answer, it’s a fundamental part of it.
Audits sound boring. Necessary, maybe—but deadly dull. But they are critical to containment. Creating secure physical and virtual containers—the kind of work we just saw—is foundational. But alone, it’s insufficient. Actually having meaningful oversight and enforceable rules and reviewing technical implementations are vital. Technical safety advances and regulation will struggle to be effective if you can’t verify that they are working as intended. How can you be sure what’s really happening and check that you’re in control? It’s an immense technical and social challenge.
Trust comes from transparency. We absolutely need to be able to verify, at every level, the safety, integrity, or uncompromised nature of a system. That in turn is about access rights and audit capacity, about adversarially testing systems, having teams of white hat hackers or even AIs probing weaknesses, flaws, and biases. It’s about building technology in an entirely different way, with tools and techniques that don’t exist yet.
External scrutiny is essential. Right now there’s no global, formal, or routine effort to test deployed systems. There’s no early warning apparatus for technological risks and no uniform or rigorous way of knowing if they abide by regulations or even adhere to commonly agreed benchmarks. There are neither the institutions nor the standardized assessments nor the tools necessary. As a starting point, then, having companies and researchers working at the cutting edge, where there is a real risk of harm, proactively collaborating with trusted experts in government-led audits of their work, is basic common sense. If any such body existed, I would happily cooperate with it at Inflection.
A few years ago I co-founded a cross-industry and civil society organization called the Partnership on AI to help with this kind of work. We launched it with the support of all the major technology companies, including DeepMind, Google, Facebook, Apple, Microsoft, IBM, and OpenAI, along with scores of expert civil society groups, including the ACLU, the EFF, Oxfam, UNDP, and twenty others. Shortly after, it kick-started an AI Incidents Database, designed for confidentially reporting on safety events to share lessons with other developers. It has now collected more than twelve hundred reports. With more than a hundred partners from nonprofit, academic, and media groups, the partnership offers critical, neutral windows for interdisciplinary discussion and collaboration. There’s scope for more organizations like this, and programs of audit within them.
Another interesting example is “red teaming”—that is, proactively hunting for flaws in AI models or software systems. This means attacking your systems in controlled ways to probe for weaknesses and other failure modes. Those thrown up today are likely to be magnified in the future, and so understanding them allows for safeguards to be built in as systems grow more powerful. The more this is done publicly and collectively, the better, enabling all developers to learn from one another. Again, it’s high time that all big tech companies proactively collaborate here, quickly sharing insights about novel risks, just like the cybersecurity industry has long shared knowledge of new zero-day attacks.
It’s also time to create government-funded red teams that would rigorously attack and stress test every system, ensuring that insights discovered along the way are shared widely across the industry. Eventually, this work could be scaled and automated, with publicly mandated AI systems designed specifically to audit and spot problems in others, while also allowing themselves to be audited.
Systems implemented to keep track of new technologies need to recognize anomalies, unforeseen jumps in capability, hidden failure modes. They must spot Trojan attacks that look legitimate but conceal unwelcome surprises. To do this, they will have to monitor a huge range of metrics without falling into the ever-tempting trap of the panopticon. Keeping close tabs on significant data sets that are used to train models, particularly open-source data sets, bibliometrics from research, and publicly available harmful incidents, would be a fruitful and noninvasive place to start. APIs that let others use foundational AI services should not be blindly open, but rather come with “know your customer” checks, as with, say, portions of the banking industry.
On the technical side, there’s scope for targeted oversight mechanisms, what some researchers have called “scalable supervision” of “systems that potentially outperform us on most skills relevant to the task at hand.” This proposal is about mathematically verifying the non-harmful nature of algorithms, requiring strict proofs from the model that mean actions or outputs are demonstrably constrained. Essentially, guaranteed records of activity and limits around capabilities are built in. Verifying and validating a model’s behavior in this way can potentially provide an objective, formal means for guiding and tracking a system.
Another promising example of a new oversight mechanism is SecureDNA, a not-for-profit program started by a group of scientists and security specialists. At present only a fraction of synthesized DNA is screened for potentially dangerous elements, but a global effort like the SecureDNA program to plug every synthesizer—benchtop at home or large and remote—into a centralized, secure, and encrypted system that can scan for pathogenic sequences is a great start. If people are printing potentially harmful sequences, they’re flagged. Cloud based, free, cryptographically secure, it updates in real time.
Screening all DNA synthesis would be a major bio-risk reduction exercise and would not, in my view, unduly curb civil liberties. This wouldn’t stop a black market in the long term, but building noncompliant synthesizers or hacking an existing system introduces a nontrivial hurdle. Pre-vetting DNA synthesis or data inputs to AI models would front-load audits before systems were deployed, reducing risk.
Right now approaches to surveillance of the emergence of new technologies, or their misuse by hostile states and other actors, differ across the globe. It’s an uneven picture: a mix of often opaque open-source information, academic research, and, in some cases, clandestine surveillance. It’s a legal and political minefield, where the thresholds for intrusion are very mixed and, at worst, deliberately obscured. We can do better. Transparency cannot be optional. There has to be a well-defined, legal route to checking any new technology under the hood, in the code, in the lab, in the factory, or out in the wild.
Most of this should be carried out voluntarily, in collaboration with the technology producers. Where it can’t be done that way, legislation must enforce cooperation. And if that does not work, there could be consideration of alternative approaches, such as the development of technical safeguards—including in some cases encrypted back doors—to provide a verifiable entry system controlled by the judiciary or an equivalent publicly sanctioned independent body.
Where a case was made to access any public or private system by law enforcement or regulators, this would be decided based on the merits of the case. Likewise, cryptographic ledgers that record any copying or sharing of a model, system, or knowledge would help track its proliferation and use. Melding social and technological containment mechanisms like this is critical. The details need new research and public debate. We will need to find a new, secure, and difficult-to-abuse balance between surveillance and safety that works for the coming wave.
Laws and treaties and brilliant technical solutions are all very well. But they still need aligning and checking, and doing so without resorting to draconian means of control. Building technologies like these initiatives is far from boring; it’s one of the twenty-first century’s most galvanizing technical and social challenges. Getting both technical safety features and audit measures in place is vital, but it takes something we don’t have. Time.
Xi Jinping was worried. “We rely on imports for some critical devices, components, and raw materials,” the Chinese president told a group of the country’s scientists in September 2020. Ominously, the “key and core technologies” he believed so vital to China’s future and geopolitical security were “controlled by others.” Indeed, China spends more on importing chips than it does on oil. Not much publicly rattles the Chinese leadership, but having pinned its long-term strategy on dominance of the coming wave, it was admitting an acute vulnerability.
Some years earlier a government-run newspaper had used a more graphic image to describe the same problem: Chinese technology was, it said, limited by a series of “choke points.” If someone was to pressure those choke points, well, the implication was clear.
Xi’s fears came to pass on October 7, 2022. America declared war on China, attacking one of those choke points. This didn’t involve missiles shooting over the Taiwan Strait. There wasn’t a naval blockade of the South China Sea or marines storming the Fujian coastline. It came instead from an unlikely source: the Commerce Department. The shots fired were export controls on advanced semiconductors, the chips that underwrite computing and so artificial intelligence.
The new export controls have made it illegal for U.S. companies to sell high-performance computing chips to China and for any company to share the tools to manufacture these chips, or provide the know-how to repair existing chips. The most advanced semiconductors (generally involving processes under fourteen nanometers, that is, fourteen-billionths of a meter, distances representing as few as twenty atoms)—including IP, manufacturing equipment, parts, design, software, services—for use in areas like artificial intelligence and supercomputing are now subject to stringent licensing. Leading American chip companies like NVIDIA and AMD can no longer supply Chinese customers with the means and know-how to produce the world’s most advanced chips. U.S. citizens working on semiconductors with Chinese companies are faced with a choice: keep their jobs and lose American citizenship, or immediately quit.
It was a bolt from the blue, designed to annihilate China’s grip on the single most important building block of twenty-first-century technology. This isn’t just an arcane trade dispute. This declaration was an almighty Klaxon in Zhongnanhai, the Chinese leadership compound, coming just as the Communist Party Congress effectively installed Xi as ruler for life. One technology executive, speaking anonymously, outlined the move’s scope: “They are not just targeting military applications, they are trying to block the development of China’s technology power by any means.”
In the short to medium term, the consensus is that this is going to hurt. The challenges of building this infrastructure are immense, especially in the sophisticated machines and techniques that produce the world’s most advanced chips, an area in which China lags. In the long term, though, that probably won’t stop it. Instead, it is pushing a difficult and hugely expensive but still plausible path toward domestic semiconductor capacity. If it takes hundreds of billions of dollars (and it will), they’ll spend it.
Chinese companies are already finding ways to bypass the controls, using networks of shell and front companies and cloud computing services in third-party countries. NVIDIA, the American manufacturer of the world’s most advanced AI chips, recently retroactively tweaked its most advanced chips to evade the sanctions. Nonetheless, it shows us something vital: there is at least one undeniable lever. The wave can be slowed, at least for some period of time and in some areas.
Buying time in an era of hyper-evolution is invaluable. Time to develop further containment strategies. Time to build in additional safety measures. Time to test that off switch. Time to build improved defensive technologies. Time to shore up the nation-state, regulate better, or even just get that bill passed. Time to knit together international alliances.
Right now technology is driven by the power of incentives rather than the pace of containment. Export controls like the United States’ semiconductor gambit have all kinds of uncertain implications for great power competition, arms races, and the future, but almost everyone agrees on one thing: this will slow down at least some technological development in China, and by extension the world.
Recent history suggests that for all its global proliferation, technology rests on a few critical R&D and commercialization hubs: choke points. Consider these points of remarkable concentration: Xerox and Apple for interfaces, say, or DARPA and MIT, or Genentech, Monsanto, Stanford, and UCSF for genetic engineering. It’s remarkable how this legacy is only slowly disappearing.
In AI, the lion’s share of the most advanced GPUs essential to the latest models are designed by one company, the American firm NVIDIA. Most of its chips are manufactured by one company, TSMC, in Taiwan, the most advanced in just a single building, the world’s most sophisticated and expensive factory. TSMC’s machinery to make these chips comes from a single supplier, the Dutch firm ASML, by far Europe’s most valuable and important tech company. ASML’s machines, which use a technique known as extreme ultraviolet lithography and produce chips at levels of astonishing atomic precision, are among the most complex manufactured goods in history. These three companies have a choke hold on cutting-edge chips, a technology so physically constrained that one estimate argues they cost up to $10 billion per kilogram.
Chips aren’t the only choke point. Industrial-scale cloud computing, too, is dominated by six major companies. For now, AGI is realistically pursued by a handful of well-resourced groups, most notably DeepMind and OpenAI. Global data traffic travels through a limited number of fiber-optic cables bunched in key pinch points (off the coast of southwest England or Singapore, for example). A crunch on the rare earth elements cobalt, niobium, and tungsten could topple entire industries. Some 80 percent of the high-quality quartz essential to things like photovoltaic panels and silicon chips comes from a single mine in North Carolina. DNA synthesizers and quantum computers are not commonplace consumer goods. Skills, too, are a choke point: the number of people working on all the frontier technologies discussed in this book is probably no more than 150,000.
So, as negative impacts become clear, we must use these choke points to create sensible rate-limiting factors, checks on the speed of development, to better ensure that good sense is implemented as fast as the science evolves. In practice, then, choke holds should apply not just to China; they could be widely applied to regulate the pace of development or rollout. Export controls are then not just a geostrategic play but a live experiment, a possible map for how technology can be contained but not strangled altogether. Eventually, all these technologies will be widely diffused. Before then, the next five or so years are absolutely critical, a tight window when certain pressure points can still slow technology down. While the option is there, let’s take it and buy time.
The fact that technology’s incentives are unstoppable does not mean that those building it bear no responsibility for their creations. On the contrary, they, we, I, do; the responsibility is crystal clear. No one is compelled to experiment with genetic modification or build large language models. Technology’s inevitable spread and development are not a get-out-of-jail-free card, a license to build what you want and see what happens. They are rather a hammering reminder of the need to get things right and the awful consequences of not doing so.
More than anyone else, those working on technology need to be actively working to solve the problems described in this book. The burden of proof and the burden of solutions rest on them, on us. People often ask me, given all this, why work in AI and build AI companies and tools? Aside from the huge positive contribution they can make, my answer is that I don’t just want to talk about and debate containment. I want to proactively help make it happen, on the front foot, ahead of where the technology is going. Containment needs technologists utterly focused on making it a reality.
Technology’s critics also have a vital role here. Standing on the sidelines and shouting, getting angry on Twitter, and writing long and obscure articles outlining the problems are all very well. But such actions won’t stop the coming wave, and in truth they won’t change it significantly either. When I first began working professionally, the outside view of technology was almost wholly benign, rapturous even. These were cool, friendly companies building a shiny future. That has changed. Yet as the voices of critique have grown much louder, it’s notable how few and far between their successes are.
In their own way, tech’s critics fall into a form of the pessimism-aversion trap that is hardwired into techno/political/business elites. Many who ridicule overly optimistic technologists stick to writing theoretical oversight frameworks or op-eds calling for regulation. If you believe technology is important and powerful, and you follow the implications of these critiques, such responses are clearly inadequate. Even the critics duck the true reality in front of them. Indeed, at times shrill criticism just becomes part of the same hype cycle as technology itself.
Credible critics must be practitioners. Building the right technology, having the practical means to change its course, not just observing and commenting, but actively showing the way, making the change, effecting the necessary actions at source, means critics need to be involved. They cannot stand shouting from the sidelines. This is in no way an argument against critics, quite the opposite. It’s a recognition that technology deeply needs critics—at every level but especially on the front lines, building and making, grappling with the tangible everyday reality of creation. If you’re reading this and are critical, then there’s a clear response: get involved.
I fully acknowledge this doesn’t make for an easy life. There’s no comfortable place here. It’s impossible not to recognize some of the paradoxes. It means people like me have to face the prospect that alongside trying to build positive tools and forestall bad outcomes, we may inadvertently accelerate the very things we’re trying to avoid, just like gain-of-function researchers with their viral experiments. Technologies I develop may well cause some harm. I will personally continue to make mistakes, despite my best efforts to learn and improve. I’ve wrestled with this point for years—hang back or get involved? The closer you are to a technology’s beating heart, the more you can affect outcomes, steer it in more positive directions, and block harmful applications. But this means also being part of what makes it a reality—for all the good and for all the harm it may do.
I don’t have all the answers. I constantly question my choices. But the only other option is to relinquish the task of building altogether. Technologists cannot be distant, disconnected architects of the future, listening only to themselves. Without critics on the outside and within, the dilemma hurtles toward us inexorably. With them, there’s a better shot of building technology that does not further damage the nation-state, is less prone to catastrophic failures, does not help increase the chances of authoritarian dystopias. Ten years ago, the tech industry was also monocultural, in every sense of the word. That’s started to change, and there’s now more intellectual diversity than ever before, including more critical, ethical, humanistic voices in the development process itself.
When I co-founded DeepMind, building safety and ethics concerns into the core fabric of a tech company felt novel. Simply using the word “ethics” in this context got me universally strange looks; today in contrast it’s sadly in danger of being another overused buzzword. Nevertheless, it has led to real change, opening up meaningful opportunities for discussion and contestation. Promisingly, research on ethical AI has ballooned—a fivefold increase in publications since 2014. On the industry side this growth is even faster; ethical AI research with industry affiliations is up 70 percent year on year. Once, it would have been strange to find moral philosophers, political scientists, and cultural anthropologists working in tech, now less so. Major shortfalls in bringing nontechnical perspectives and diverse voices into the discussion are still all too commonplace, however: contained technology is a project requiring all kinds of disciplines and perspectives. Proactively hiring to that effect is a must.
In a world of entrenched incentives and failing regulation, technology needs critics not just on the outside but at its beating heart.
Profit drives the coming wave. There’s no pathway to safety that doesn’t recognize and grapple with this fact. When it comes to exponential technologies like AI and synthetic biology, we must find new accountable and inclusive commercial models that incentivize safety and profit alike. It should be possible to create companies better adapted to containing technology by default. I and others have long been experimenting with this challenge, but to date results have been mixed.
Corporations traditionally have a single, unequivocal goal: shareholder returns. For the most part, that means the unimpeded development of new technologies. While this has been a powerful engine of progress in history, it’s poorly suited to containment of the coming wave. I believe that figuring out ways to reconcile profit and social purpose in hybrid organizational structures is the best way to navigate the challenges that lie ahead, but making it work in practice is incredibly hard.
From the beginning of DeepMind, it was important to me that we factored in governance models equal to our end goal. When we were acquired by Google in 2014, I designed an “ethics and safety board” to oversee our technologies, and we made this a condition of the acquisition. Even back then we realized that if we were to be successful in achieving our mission of building true AGI, it would unleash a force far beyond what could reasonably be expected to be owned and controlled by a single corporation. We wanted to ensure that Google understood this and put in place a commitment to broaden our governance beyond us technologists. Ultimately, I wanted to create a global, multi-stakeholder forum for deciding what would happen with AGI when or if it was achieved, a kind of democratic world institute for AI. The more powerful a technology, it seemed to me, the more important it was to have multiple perspectives controlling and gaining access to it.
After our acquisition by Google, my co-founders and I spent years trying to build an ethics charter into the legal fabric of the company, endlessly arguing about how much of this charter could be public, how much of DeepMind’s work could be further subject to independent oversight and scrutiny. Our goal in these discussions was always to ensure that unprecedented technology was matched by unprecedented governance. Our proposal was to spin DeepMind out as a new form of “global interest company,” with a fully independent board of trustees separate from and in addition to the board of directors tasked with operationally running the company. Membership, decision-making, and even some of the board’s reasoning would be more public. Transparency, accountability, ethics—these would be not just corporate PR but foundational, legally binding, and built into everything the company did. We felt this would let us work in an open way, proactively learning how companies could be resilient and modern long-term stewards of exponential technologies.
We established a plausible way profits from AI could be reinvested in an ethical and social mission. The spun-out company would be “limited by guarantee,” without shareholders but with an obligation to provide Alphabet, the main funder, with an exclusive technology license. As part of its social and scientific mission DeepMind would use a large portion of its profits to work on public service technologies that might only be valuable years down the line: things like carbon capture and storage, ocean cleaning, plastic-eating robots, or nuclear fusion. The deal was that we’d be able to make some of our major breakthroughs open-source, much like an academic lab. IP core to Google’s search business would stay with Google, but the rest would be available for us to advance DeepMind’s social mission, working on new drugs, better health care, climate change, and so on. It would mean investors could be rewarded, but also ensured that social purpose was in the company’s legal DNA.
In hindsight it was just too much for Google at the time. Lawyers were retained, years of intense negotiations took place, but there didn’t seem to be a way to square the circle. In the end we couldn’t find an answer that would satisfy everyone. DeepMind continued as a normal unit within Google with no formal legal independence, operating just as a separate brand. It was a foundational lesson for me: shareholder capitalism works because it is simple and clear, and governance models too have a tendency to default to the simple and clear. In the shareholder model, lines of accountability and performance tracking are quantified and very transparent. It may be possible to design more modern structures in theory, but operating them in practice is another story.
During my time at Google, I continued working on experimental efforts to create innovative governance structures. I drafted Google’s AI Principles and was part of the team that launched the AI ethics advisory council, made up of eminent independent legal, technology, and ethics experts. The goal of both was to take the first steps toward establishing a charter around how Google handles cutting-edge technologies like AI and quantum computing. Our ambition was to invite a diverse group of external stakeholders to gain privileged access to the technical frontier, give feedback, and provide much-needed external perspectives from those far away from the excitement and optimism of building new technologies.
However, the council fell apart days after it was announced. Some employees at Google objected to the appointment of Kay Coles James, the president of the Heritage Foundation, a Washington-based conservative think tank. She had been appointed alongside a range of figures from the left and the center, but a campaign was quickly launched inside Google to get her removed. Forming a coalition with Twitter employees, the activists pointed out that she had made a number of anti-trans and anti-LGBTQ remarks over the years, including most recently arguing, “If they can change the definition of women to include men, they can erase efforts to empower women economically, socially, and politically.” While I personally disagreed with her remarks and political positions, I defended our choice to ask her to join the board, arguing that the full range of values and perspectives deserved to be heard. After all, Google is a global company with global users, some of whom might share this view.
Many Google employees and external activists disagreed and within days of the announcement published an open letter demanding James’s removal from the council. Staffers and others were actively lobbying university campuses to remove academic funding from other board members who refused to step down, arguing that their ongoing participation could only be understood as condoning transphobia. In the end three members resigned, and the effort was scrapped entirely in less than a week. The political atmosphere was, unfortunately, too much both for public figures and for a public company.
Once again, my attempts to rethink the corporate mandate failed, even as they spurred conversation and helped put some difficult discussions on the table, both at Alphabet and in wider policy, academic, and industry circles. What teams and what research are funded, how products are tested, what internal controls and reviews are in place, how much outside scrutiny is appropriate, what stakeholders need to be included—senior leaders at Alphabet and elsewhere started having these conversations on a regular basis.
Across tech companies the kinds of AI safety discussions that felt fringe a decade ago are now becoming routine. The need to balance profits with a positive contribution and cutting-edge safety is accepted in principle by all the major U.S. tech groups. Despite the awesome scale of the rewards on offer, entrepreneurs, execs, and employees alike should keep pushing and exploring corporate forms that can better accommodate the challenge of containment.
Encouraging experiments are underway. Facebook created its independent Oversight Board—staffed with ex-judges, campaigners, and expert academics to advise on governing the platform. It has come in for criticism from all quarters and clearly doesn’t “solve” the problem alone. But it’s important to begin by praising the effort, and encouraging Facebook and others to keep experimenting. Another example is the growing movement of public benefit corporations and B Corps, which are still for-profit companies but have a social mission inscribed into their legally defined goals. Technology companies that have strong containment mechanisms and goals written in as a fiduciary duty are a next step. There’s a good chance of positive change here, given the growth of these alternative corporate structures (more than ten thousand companies now use the B Corp structure). While economic goals do not always align well with contained technology, innovative corporate forms make it more likely. This is the kind of experimentation that’s needed.
Containment needs a new generation of corporations. It needs founders and those working in tech to contribute positively to society. It also needs something altogether more difficult. It needs politics.
Technological problems require technological solutions, as we’ve seen, but alone they are never sufficient. We also need the state to flourish. Every effort to buttress liberal democratic states and steel them against the stressors must be supported. Nation-states still control many fundamental elements of civilization: law, the money supply, taxation, the military, and so on. That helps with the task ahead, where they will need to create and maintain resilient social systems, welfare nets, security architectures, and governance mechanisms capable of surviving severe stress. But they also need to know, in detail, what is happening: right now they’re operating blind in a hurricane.
The physicist Richard Feynman famously said, “What I cannot create, I do not understand.” Today this couldn’t be more true of governments and technology. I think the government needs to get way more involved, back to building real technology, setting standards, and nurturing in-house capability. It needs to compete for talent and hardware in the open market. There’s no two ways about it: this is expensive and will come with wasteful mistakes. But proactive governments will exert far greater control than if they just commission services and live off outsourced expertise and tech owned and operated elsewhere.
Accountability is enabled by deep understanding. Ownership gives control. Both require governments to get their hands dirty. Although today companies have taken the lead, much of the most speculative fundamental research is still funded by governments. U.S. federal government expenditure on R&D is at an all-time-low share of the total—just 20 percent—but still amounts to a not inconsiderable $179 billion per year.
This is good news. Investing in science and technology education and research and supporting domestic tech businesses create a positive feedback loop where governments have a direct stake in state-of-the-art technology, poised to capitalize on benefits and stamp down harms. Put simply, as an equal partner in the creation of the coming wave, governments stand a better chance of steering it toward the overall public interest. Having much more in-house technical expertise, even at considerable cost, is money well spent. Governments should not rely on management consultants, contractors, or other third-party suppliers. Full-time, well-respected staffers who are properly compensated, competitively with the private sector, should be a core part of the solution. Instead, private sector salaries can be ten times their public sector equivalents in national critical roles: it’s unsustainable.
Their first task should be to better monitor and understand developments in technology. Countries need to understand in detail, for example, what data their populations supply, how and where it is used, and what it means; administrations should have a strong sense of the latest research, where the frontier is, where it’s going, how their country can maximize upsides. Above all they need to log all the ways technology causes harm—tabulate every lab leak, every cyberattack, every language model bias, every privacy breach—in a publicly transparent way so everyone can learn from failures and improve.
This information then needs to be used effectively by the state, responding in real time to emerging problems. Bodies close to executive power, like the White House’s Office of Science and Technology Policy, are growing more influential. More is still needed: in the twenty-first century it doesn’t make sense to have cabinet positions addressing matters like the economy, education, security, and defense without a similarly empowered and democratically accountable position in technology. The secretary or minister for emerging technology is still a governmental rarity. It shouldn’t be; every country should have one in the era of the coming wave.
Regulation alone doesn’t get us to containment, but any discussion that doesn’t involve regulation is doomed. Regulation should focus on those incentives, better aligning individuals, states, companies, and the public as a whole with safety and security while building in the possibility of a hard brake. Certain use cases, like AI for electioneering, should be prohibited by law as part of the package.
Legislatures are beginning to act. In 2015 there was virtually no legislation around AI. But no fewer than seventy-two bills with the phrase “artificial intelligence” have been passed worldwide since 2019. The OECD AI Policy Observatory counts no fewer than eight hundred AI policies from sixty countries in its database. The EU’s AI Act is bedeviled with problems, sure, but there is much to be praised in its provisions, and it represents the right focus and ambition.
In 2022 the White House released a blueprint for an AI Bill of Rights with five core principles “to help guide the design, development, and deployment of artificial intelligence and other automated systems so that they protect the rights of the American public.” Citizens should, it says, be protected from unsafe and ineffective systems and algorithmic bias. No one should be forced to subject themselves to AI. Everyone has the right to say no. Efforts like this should be widely supported and quickly implemented.
However, policy makers’ imaginations will need to match the scope of technology. Government needs to go further. For understandable reasons, we don’t let any business build or operate nuclear reactors in any way they see fit. In practice, the state is intimately involved in—and closely watching, licensing, and governing—every aspect of their existence. Over time this will and should become more true of technology in general. Today anyone can build AI. Anyone can set up a lab. We should instead move to a more licensed environment. This would produce a clearer set of responsibilities and harder mechanisms for revoking access and remedying harms around advanced technologies. The most sophisticated AI systems or synthesizers or quantum computers should be produced only by responsible certified developers. As part of their license, they would need to subscribe to clear, binding security and safety standards, following rules, running risk assessments, keeping records, closely monitoring live deployments. Just as you cannot simply launch a rocket into space without FAA approval, so tomorrow you shouldn’t simply be able to release a state-of-the-art AI.
Different licensing regimes could apply according to model size or capability: the bigger and more capable the model, the more stringent the licensing requirements. The more general a model, the more likely it is to pose a serious threat. This means that AI labs working on the most fundamental capabilities will require special attention. Moreover, this creates scope for more granular licensing if need be to home in on the specifics of development: training runs of models, chip clusters above a given size, certain kinds of organisms.
Taxation also needs to be completely overhauled to fund security and welfare as we undergo the largest transition of value creation—from labor to capital—in history. If technology creates losers, they need material compensation. Today U.S. labor is taxed at an average rate of 25 percent, equipment and software at just 5 percent. The system is designed to let capital frictionlessly reproduce itself in the name of creating flourishing businesses. In the future, taxation needs to switch emphasis toward capital, not only funding a redistribution toward those adversely affected, but creating a slower and fairer transition in the process. Fiscal policy is an important valve in controlling this transition, a means of exercising control over those choke points and building state resilience at the same time.
This should include a greater tax on older forms of capital like land, property, company shares, and other high-value, less liquid assets, as well as a new tax on automation and autonomous systems. This is sometimes called a “tax on robots”; MIT economists have argued that even a moderate tax of just 1 to 4 percent of their value could have a big impact. A carefully calibrated shift in the tax burden away from labor would incentivize continued hiring and cushion disruptions in household life. Tax credits topping up the lowest incomes could be an immediate buffer in the face of stagnating or even collapsing incomes. At the same time, a massive re-skilling program and education effort should prepare vulnerable populations, raise awareness of risks, and increase opportunities for engagement with the capabilities of the wave. A universal basic income (UBI)—that is, an income paid by the state for every citizen irrespective of circumstances—has often been floated as the answer to the economic disruptions of the coming wave. In the future, there will likely be a place for UBI-like initiatives; however, before one even gets to that, there are plenty of good ideas.
In an era of hyper-scaling corporate AIs, we should start to think of capital taxes like this applying to the largest corporations themselves, not just the assets or profits in question. Moreover, mechanisms must be found for cross-border taxation of those giant businesses, ensuring they pay their fair share in maintaining functioning societies. Experiments are encouraged here: a fixed portion of company value, for example, paid as a public dividend would keep value transferring back to the population in an age of extreme concentration. At the limit there is a core question about who owns the capital of the coming wave; a genuine AGI cannot be privately owned in the same manner as, say, a building or a fleet of trucks. When it comes to technology that could radically extend human life span or capabilities, there clearly has to be a big debate from the get-go about its distribution.
Who is able to design, develop, and deploy technologies like this is ultimately a matter for governments to decide. Their levers, institutions, and domains of expertise will all have to evolve as rapidly as technology, a generational challenge for everyone involved. An age of contained technology is, then, an age of extensively and intelligently regulated technology; no ifs or buts. But of course, regulation in one country has an inevitable flaw. No national government can do this alone.
Laser weapons sound like science fiction. Unfortunately, they’re not. As laser technology developed, it was clear they could cause blindness. Weaponized, this could incapacitate adversary forces or, indeed, anyone targeted. An exciting new civilian technology was again opening up the prospect of horrible modes of attack (although not to date in the manner of Star Wars). No one wants armies or gangs roaming around with blinding lasers.
Luckily, it didn’t happen. Use of blinding laser weapons was outlawed under the 1995 Protocol on Blinding Laser Weapons, an update to the Convention on Certain Conventional Weapons that prohibited the use of “laser weapons specifically designed, as their sole combat function or as one of their combat functions, to cause permanent blindness to unenhanced vision.” A hundred and twenty-six countries signed up. Laser weapons are, as a result, neither a major part of military hardware nor common weapons on the streets.
Sure, blinding lasers are not the kinds of omni-use technologies we’re talking about in this book. But they are evidence that it can be done; a strong ban can work. Delicate alliances and international cooperation can be pulled off, and they can change history.
Consider these examples, some of which we discussed earlier: the Treaty on the Non-proliferation of Nuclear Weapons; the Montreal Protocol outlawing CFCs; the invention, trialing, and rollout of a polio vaccine across a Cold War divide; the Biological Weapons Convention, a disarmament treaty effectively banning biological weapons; bans on cluster munitions, land mines, genetic editing of human beings, and eugenics policies; the Paris Agreement, aiming to limit carbon emissions and the worst impacts of climate change; the global effort to eradicate smallpox; phasing out lead in gasoline; and putting an end to asbestos.
Countries no more like giving up power than companies like missing out on profit, and yet these are precedents to learn from, shards of hope in a landscape riven with resurgent techno-competition. Each had specific conditions and challenges that both helped it come about and hindered perfect compliance. But each, crucially, is a precious example of the world’s nations uniting and compromising to face a major challenge, offering hints and frameworks for tackling the coming wave. If a government wanted to ban synthetic biology or AI applications, could it? No, clearly not in anything but a partial, fragile sense. But a powerful, motivated alliance? Maybe.
Faced with the abyss, geopolitics can change fast. In the teeth of World War II, peace must have felt like a dream. As the Allies wearily fought on, few on the ground could have imagined that just a few years later their governments would pump billions into rebuilding their enemies. That, despite horrific and genocidal war crimes, Germany and Japan would soon become critical parts of a stable worldwide alliance. In hindsight it seems dizzying. Just a few short years separate the bullets, bitterness, and beaches of Normandy and Iwo Jima from a rock-solid military and commercial partnership, a deep friendship that lasts to this day, and the biggest foreign aid program ever attempted.
At the height of the Cold War high-level contacts were maintained in spite of severe tensions. In the event of something like a rogue AGI or major biohazard being released, this kind of high-level coordination will be critical, yet as the new Cold War takes shape, divides are growing. Catastrophic threats are innately global and should be a matter of international consensus. Rules that stop at national borders are obviously insufficient. While every country has a stake in advancing these technologies, they also have a good cause to curtail their worst consequences. So what do the nonproliferation treaty, the Montreal Protocol, the Paris Agreement look like for the coming wave?
Nuclear weapons are an exception partly but not only because they are so difficult to build: the long, patient hours of discussion, the decades of painstaking treaty negotiations at the UN, the international collaboration even at times of extreme tension, it all matters when it comes to keeping them in check. There are both moral and strategic components to nuclear containment. Reaching and enforcing such agreements has never been easy, doubly so in an era of great power competition. Diplomats hence play an underrated role in containing technology. A golden age of techno-diplomacy needs to emerge from the era of arms races. Many I’ve spoken with in the diplomatic community are acutely aware of this.
Alliances, however, can also work at the level of technologists or subnational bodies, collectively deciding what to fund, what to turn away from. A good example here comes from germ-line gene editing. A study of 106 countries found that regulation of germ-line gene editing is patchy. Most countries have some kind of regulation or policy guidelines, but there are considerable divergences and gaps. It doesn’t add up to a global framework on a technology with global scope. More effective to date is the international collaboration of scientists on the front line. In the aftermath of the first gene editing of human beings, a letter signed by luminaries like Eric Lander, Emmanuelle Charpentier, and Feng Zhang called for “a global moratorium on all clinical uses of human germline editing—that is, changing heritable DNA (in sperm, eggs or embryos) to make genetically modified children” and “an international framework in which nations, while retaining the right to make their own decisions, voluntarily commit to not approve any use of clinical germline editing unless certain conditions are met.”
They’re not calling for a permanent ban, they’re not banning germ-line editing for research purposes, and they’re not saying every nation should follow the same path. But they are asking that practitioners take time to harmonize and make the right decisions. Enough people at the leading edge can still make a difference, allowing room for a pause, helping to create space and a foundation for nations and international bodies to come together and find a way through.
Earlier in the chapter I discussed the frictions between the United States and China. Despite their differences there are still obvious places for collaboration between these vying powers. Synthetic biology is a better starting point than AI here, thanks to lower existing competition and the obvious mutually assured destruction of novel biothreats. The SecureDNA project is a good example, laying out a path for governing synthetic biology similarly to how chemical weapons have been curtailed. If China and the United States could create, say, a shared bio-risk observatory, encompassing everything from advanced R&D to deployed commercial applications, it would be a precious area of collaboration to build on.
China and the United States also share an interest in restraining the long tail of bad actors. Given that an Aum Shinrikyo could come from anywhere, both countries will be keen to restrain the uncontrolled spread of the world’s most powerful technologies. Currently China and the United States are in a struggle to set technological standards. But a shared approach is a clear win-win; splintered standards make things harder for everyone. Another point of commonality might be maintaining cryptographic systems in the face of advances in quantum computing or machine learning that could undermine them. Each could pave the way for wider compromise. As the century wears on, the lesson of the Cold War will have to be relearned: there is no path to technological safety without working with your adversaries.
Beyond encouraging bilateral initiatives, the obvious thing at this stage is to propose creating some new kind of global institution devoted to technology. I’ve heard it said many, many times: What does a World Bank for biotech or a UN for AI look like? Could a secure international collaboration be the way to approach an issue as daunting and complex as AGI? Who is the ultimate arbiter, the lender of last resort as it were, the body that when asked “Who contains technology?” can put its hand up?
We need our generation’s equivalent of the nuclear treaty to shape a common worldwide approach—in this case not curbing proliferation altogether but setting limits and building frameworks for management and mitigation that, like the wave, cross borders. This would put clear limits on what work is undertaken, mediate among national licensing efforts, and create a framework for reviewing both.
Where there is a clear scope for a new body or bodies is with technical concerns. A dedicated regulator that navigates contentious geo-politics (as much as possible), avoids overreach, and performs a pragmatic monitoring function on broadly objective criteria is urgently needed. Think of something like the International Atomic Energy Agency or even a trade body like the International Air Transport Association. Rather than having an organization that itself directly regulates, builds, or controls technology, I would start with something like an AI Audit Authority—the AAA. Focused on fact-finding and auditing model scale and when capability thresholds are crossed, the AAA would increase global transparency at the frontier, asking questions like: Does the system show signs of being able to self-improve capabilities? Can it specify its own goals? Can it acquire more resources without human oversight? Is it deliberately trained in deception or manipulation? Similar audit commissions could operate in almost every area of the wave, and would, again, offer a foundation for government licensing efforts while also helping the push for a nonproliferation treaty.
Hard realism has a much better chance of success than vague and unlikely proposals. We don’t need to totally reinvent the institutional wheel, creating more opportunities for rivalry and grandstanding. We should just find every possible means of improving it—and fast.
The common thread here is governance: of software systems, of microchips, of businesses and research institutes, of countries, and of the international community. At each level is a thicket of incentives, sunk costs, institutional inertia, conflicting fiefdoms and worldviews that must be cut through. Make no mistake. Ethics, safety, containment—these will be products of good governance above all. But good governance doesn’t just come from well-defined rules and effective institutional frameworks.
In the early days of jet engines, the 1950s, crashes—and fatalities—were worryingly common. By the early 2010s they were at just one death per 7.4 million passenger boardings. Years now go by with no fatal accidents whatsoever involving American commercial aircraft. Flying is just about the safest mode of transport there is: sitting thirty-five thousand feet in the sky is safer than sitting at home on your couch.
Airlines’ impressive safety record comes down to numerous incremental technical and operational improvements over the years. But behind them is something just as important: culture. The aviation industry takes a vigorous approach to learning from mistakes at every level. Crashes are not just tragic accidents to mourn; they’re foundational learning experiences in determining how systems fail, opportunities for diagnosing problems, fixing them, and sharing that knowledge across the entire industry. Best practices are hence not corporate secrets, an edge over rival airlines: they’re enthusiastically implemented by competitors in the common interests of collective industry trust and safety.
That’s what’s needed for the coming wave: real, gut-level buy-in from everyone involved in frontier technologies. It’s all very well devising and promoting initiatives and policies for ethics and safety, but you need the people delivering to actually believe in them.
While the tech industry talks a big game when it comes to “embracing failure,” it rarely does so when it comes to privacy or safety or technical breaches. Launching a product that doesn’t catch on is one thing, but owning a language model that causes a misinformation apocalypse or a drug that causes adverse reactions is far more uncomfortable. Criticism of tech is, not without good reason, unrelentingly fierce. Competition likewise. One consequence is that as soon as a new technology or product goes awry, a culture of secrecy takes over. The openness and mutual trust that characterize portions of the development process get lost. Opportunities for learning, and then broadcasting that learning, disappear. Even admitting to mistakes, opening the floodgates, is seen as a risk, a corporate no-no.
Fear of failure and public opprobrium is leading to stasis. Immediate self-reporting of problems should be a baseline for individuals and organizations alike. But rather than being commended for experimentation, companies and teams are hung out to dry. Doing the right thing only triggers a backlash of cynicism, Twitter flaming, and vicious public point scoring. Why would anyone actually admit their mistakes in this context? This has got to stop if we want to produce better, more responsible, more containable technologies.
Embracing failure must be real, not a sound bite. For a start, being utterly open about failures even on uncomfortable topics should be met with praise, not insults. The first thing a technology company should do when encountering any kind of risk, downside, or failure mode is to safely communicate to the wider world. When a lab leaks, the first thing it should do is advertise the fact, not cover it up. The first things other actors in the space—other companies, research groups, governments—need to then do are listen, reflect, offer support, and most crucially learn and actively implement that learning. This attitude saved many thousands of lives in the sky. It could save millions more in years to come.
Containment can’t just be about this or that policy, checklist, or initiative, but needs to ensure that there is a self-critical culture that actively wants to implement them, that welcomes having regulators in the room, in the lab, a culture where regulators want to learn from technologists and vice versa. It needs everyone to want in, own it, love it. Otherwise safety remains an afterthought. Among many, and not only in AI, there’s a sense we are “just” researchers, “just” exploring and experimenting. That’s not been the case for years, and is a prime example of where we need a culture shift. Researchers must be encouraged to step back from the constant rush toward publication. Knowledge is a public good, but it should no longer be the default. Those actively conducting frontier research need to be the first to recognize this, as their peers in areas of nuclear physics and virology already have. In AI, capabilities like recursive self-improvement and autonomy are, I think, boundaries we should not cross. This will have technical and legal components, but also needs moral, emotional, cultural buy-in from the people and organizations closest to it.
In 1973, one of the inventors of genetic engineering, Paul Berg, gathered a group of scientists on the Monterey Peninsula in California. He’d begun to worry about what his invention might unleash and wanted to set some ground rules and moral foundations for going forward. At the Asilomar conference center, they asked the difficult questions thrown up by this new discipline: Should we start genetically engineering humans? If so, what traits might be permissible? Two years later they returned in even larger numbers for the Asilomar Conference on Recombinant DNA. The stakes in that sea-lapped hotel were high. It was a turning point in the biosciences, establishing durable principles for governing genetic research and technology that set guidelines and moral limits on what experiments could take place.
I attended a conference in Puerto Rico in 2015 that aimed to do something similar for AI. With a mixed group, it wanted to raise the profile of AI safety, start building a culture of caution, and sketch real answers. We met again in 2017, at the symbolic venue of Asilomar, to draft a set of AI principles that I along with many others in the field signed on to. They were about building an explicitly responsible culture of AI research and inspired a raft of further initiatives. As the wave keeps building, we will need to self-consciously return again and again to the spirit—and letter—of Asilomar.
For millennia, the Hippocratic oath has been a moral lodestar for the medical profession. In Latin, Primum non nocere. First, do no harm. The Nobel Peace Prize winner and British-Polish scientist Joseph Rotblat, a man who left Los Alamos on the grounds of conscience, argued that scientists need something similar. Social and moral responsibility was, he believed, not something any scientist could ever set aside. I agree, and we should consider a contemporary version for technologists: ask not just what doing no harm means in an age of globe-spanning algorithms and edited genomes but how that can be enacted daily in what are often morally ambiguous circumstances.
Precautionary principles like this are a good first step. Pause before building, pause before publishing, review everything, sit down and hammer out the second-, third-, nth-order impacts. Find all the evidence and look at it coldly. Relentlessly course correct. Be willing to stop. Do all this not just because it says so in some form, but because it’s what’s right, it’s what technologists do.
Actions like this can’t just operate as laws or corporate mantras. Laws are only national, corporate mantras transitory, too often cosmetic. They must instead operate at a deeper level whereby the culture of technology is not that just-go-for-it “engineering mindset” but something more wary, more curious about what might happen. A healthy culture is one happy to leave fruit on the tree, say no, delay benefits for however long it takes to be safe, one where technologists remember that technology is just a means to an end, not the end itself.
Throughout this book the word “we” has featured. It might have referred to “we” the author and co-author, “we” AI researchers and entrepreneurs, “we” the scientific and technology community more widely, “we” in the global West, or “we” the sum total of humanity. (Facing fully global- and species-altering technology is one of the few places where talking about a human “we” actually is warranted.)
When people talk about technology—myself included—they often make an argument like the following. Because we build technology, we can fix the problems it creates. This is true in the broadest sense. But, the problem is, there is no functional “we” here. There is no consensus and no agreed mechanism for forming a consensus. There actually is no “we,” and there is certainly no lever any “we” can pull. This should be obvious, but it bears repeating. Even the president of the United States has remarkably limited powers to alter the course of, say, the internet.
Instead, countless distributed actors work sometimes together and sometimes at cross-purposes. Companies and nations, as we have seen, have divergent priorities, fractured, conflicting incentives. For the most part concerns over technology like those outlined in this book are elite pursuits, nice talking points for the business-class lounge, op-eds for bien-pensant publications, or topics for the presentation halls at Davos or TED. Most of humanity doesn’t yet worry about these things in any kind of systematic way. Off Twitter, out of the bubble, most people have very different concerns, other problems demanding attention in a fragile world. Communication around AI hasn’t always helped, tending to fall into simplistic narratives.
So, if the invocation of the grand “we” is at present meaningless, it prompts an obvious follow-up: let’s build one. Throughout history change came about because people self-consciously worked for it. Popular pressure created new norms. The abolition of slavery, women’s suffrage, civil rights—these are huge moral achievements that happened because people fought hard, building broad-based coalitions that took a big claim seriously and then effected change based on it. Climate wasn’t just put on the map because people noticed the weather getting more extreme. They noticed because grassroots activists and scientists and then later (some) writers, celebrities, CEOs, and politicians agitated for meaningful change. And they acted on it out of a desire to do the right thing.
Research shows that when introduced to the topic of emerging technologies and their risks, people really do care and want to find solutions. Although many of the harms are still a way off, I believe people are perfectly capable of reading the runes here. I’ve yet to find anyone who’s watched a Boston Dynamics video of a robot dog or considered the prospect of another pandemic without a shudder of dread.
Here is a huge role for popular movements. Over the last five or so years, a burgeoning civil society movement has begun to highlight these problems. The media, trade unions, philanthropic organizations, grassroots campaigns—all are getting involved, proactively looking at ways to create contained technology. I hope that my generation of founders and builders energizes these movements rather than stands in the way. Meanwhile, citizen assemblies offer a mechanism for bringing a wider group into the conversation. One proposal is to host a lottery to choose a representative sample of the population to intensively debate and come up with proposals for how to manage these technologies. Given access to tools and advice, this would be one way of making containment a more collective, attentive, grounded process.
Change happens when people demand it. The “we” that builds technology is scattered, subject to a mass of competing and different national, commercial, and research incentives. The more the “we” that is subject to it speaks clearly in one voice, a critical public mass agitating for change, demanding an alignment of approaches, the better chance of good outcomes. Anyone anywhere can make a difference. Fundamentally, neither technologists nor governments will solve this problem alone. But together “we” all might.
Just a few days after the release of GPT-4, thousands of AI scientists signed an open letter calling for a six-month moratorium on researching the most powerful AI models. Referencing the Asilomar principles, they cited reasons familiar to those reading this book: “Recent months have seen AI labs locked in an out-of-control race to develop and deploy ever more powerful digital minds that no one—not even their creators—can understand, predict, or reliably control.” Shortly after, Italy banned ChatGPT. A complaint against LLMs was filed with the Federal Trade Commission aiming for much tighter regulatory control. Questions about AI risk were asked at the White House press briefing. Millions of people discussed the impacts of technology—at work, at the dinner table.
Something is building. Containment it is not, but for the first time the questions of the coming wave are being treated with the urgency they deserve.
Each of the ideas outlined so far represents the beginning of a seawall, a tentative tidal barrier starting with the specifics of the technology itself and expanding outward to the imperative of forming a massive global movement for positive change. None of them works alone. Knit measures like this together, however, and an outline of containment comes into view.
One good example comes from the MIT biotechnologist Kevin Esvelt. Few people have considered biosecurity threats in more detail. Those bespoke pathogens designed to cause maximum fatalities? Kevin is determined to use every tool to stop them from happening. His program is one of the most holistic containment strategies around. It’s built around three pillars: delay, detect, and defend.
To delay, he echoes the language of nuclear technology, proposing a “pandemic test-ban treaty,” an international agreement to stop experimentation on the most pathogenic materials. Any experiments that would seriously raise the risk of a pandemic event, including gain-of-function research, would be banned. He also advocates an entirely new regime of insurance and liability for anyone working with viruses or other potentially harmful biomaterials. It would amp up the costs of responsibility in an immediately tangible way by literally factoring low-probability but catastrophic consequences—currently negative externalities borne by everyone else—into the price of the research. Not only would institutions conducting potentially dangerous research have to take out additional insurance, but a trigger law would mean anyone shown to be responsible for a major biohazard or catastrophic event would become liable.
DNA screening on all synthesizers is an absolute must, and moreover the whole system should be cloud based so that it could update in real time according to newly understood and emerging threats. Swiftly detecting an outbreak is just as important in this schema, especially for subtle pathogens with long incubation periods. Think of a disease dormant for years. If you aren’t aware of what’s happening, you can’t contain it.
Then, if the worst happens, defend. Resilient and prepared countries are vital: the most extreme pandemics would make even maintaining food, power, water supplies, law and order, and health care difficult. Having stockpiles of state-of-the-art pandemic-proof PPE equipment ready for all essential workers would make a massive difference. So would strong supply lines of medical equipment capable of withstanding a serious shock. Those low-wavelength lightbulbs that can destroy viruses? They need to be everywhere, before the pandemic starts, or at the very least ready to get rolled out.
Put all the elements here together and there is an outline of what will meet and match the coming wave.
1. Technical safety: Concrete technical measures to alleviate possible harms and maintain control.
2. Audits: A means of ensuring the transparency and accountability of technology.
3. Choke points: Levers to slow development and buy time for regulators and defensive technologies.
4. Makers: Ensuring responsible developers build appropriate controls into technology from the start.
5. Businesses: Aligning the incentives of the organizations behind technology with its containment.
6. Government: Supporting governments, allowing them to build technology, regulate technology, and implement mitigation measures.
7. Alliances: Creating a system of international cooperation to harmonize laws and programs.
8. Culture: A culture of sharing learning and failures to quickly disseminate means of addressing them.
9. Movements: All of this needs public input at every level, including to put pressure on each component and make it accountable.
Step 10 is about coherence, ensuring that each element works in harmony with the others, that containment is a virtuous circle of mutually reinforcing measures and not a gap-filled cacophony of competing programs. In this sense, containment isn’t about this or that specific suggestion but is an emergent phenomenon of their collective interplay, a by-product of societies that learn to manage and mitigate the risks thrown up by Homo technologicus. One move alone isn’t going to work, whether with pathogens or quantum computers or AI, but a scheme like this gains force from the careful accretion of interlocking countermeasures, guardrail layered on guardrail from international treaties to supply chain reinforcement of protective new technologies. Proposals like “delay, detect, and defend” are, moreover, not end states, destinations. Safety in the context of the coming wave is not somewhere we arrive but something that must be continually enacted.
Containment is not a resting place. It’s a narrow and never-ending path.
The economist Daron Acemoglu and the political scientist James Robinson share the view that liberal democracies are much less secure than they might look. They see the state as an inherently unstable “shackled Leviathan”: vast and powerful, but held in check by persistent civil societies and norms. Over time, countries like the United States entered what they call a “narrow corridor” that kept them in this precarious balance. On either side of this corridor lie traps. On the one hand, the power of the state breaks that of wider society and completely dominates it, creating despotic Leviathans like China. On the other, the state falls apart, producing absent Leviathans, zombies, where the state has no real control over society, as in places like Somalia or Lebanon. Both have terrible consequences for their populations.
Acemoglu and Robinson’s point is that states constantly walk this corridor. At any moment, they could fall. For every increase in state capacity there needs to be a corresponding increase in social capacity to counterbalance it. There’s a constant pressure toward despotic Leviathans that needs constant weight to stop. There is no final destination, no happy, safe, and continual existence at the corridor’s end; rather, it’s a dynamic, unstable space where elites and citizens contest outcomes and at any time shackled Leviathans can either disappear or grow despotic. Safety is a matter of inching forward and carefully maintaining balance.
I think this metaphor holds for how we approach technology, and not just because the argument here is that technology now makes that balance so much more precarious. Safe, contained technology is, like liberal democracy, not a final end state; rather, it is an ongoing process, a delicate equilibrium that must be actively maintained, constantly fought for and protected. There’s no moment when we say, aha, we’ve solved the problem of proliferating technology! Instead, it’s about finding a way through, ensuring sufficient numbers of people are committed to keeping the unending balance between openness and closure.
Rather than a corridor, which implies a clear direction of travel, I imagine containment as a narrow and treacherous path, wreathed in fog, a plunging precipice on either side, catastrophe or dystopia just a small slip away; you can’t see far ahead, and as you tread, the path twists and turns, throws up unexpected obstacles.
On the one hand, total openness to all experimentation and development is a straightforward recipe for catastrophe. If everyone in the world can play with nuclear bombs, at some stage you have a nuclear war. Open-source has been a boon to technological development and a major spur to progress more widely. But it’s not an appropriate philosophy for powerful AI models or synthetic organisms; here it should be banned. They should not be shared, let alone deployed or developed, without rigorous due process.
Safety relies on things not failing, not getting into the wrong hands, forever. Some level of policing the internet, DNA synthesizers, AGI research programs, and so on is going to be essential. It’s painful to write. As a young twentysomething, I started out from a privacy maximalist position, believing spaces of communication and work completely free from oversight were foundational rights and important parts of healthy democracy. Over the years, though, as the arguments became clearer and the technology more and more developed, I’ve updated that view. It’s just not acceptable to create situations where the threat of catastrophic outcomes is ever present. Intelligence, life, raw power—these are not playthings, and should be treated with the respect, care, and control they deserve. Technologists and the general public alike will have to accept greater levels of oversight and regulation than have ever been the case before. Just as most of us wouldn’t want to live in societies without laws and police, most of us wouldn’t want to live in a world of unrestricted technology either.
Some measure of anti-proliferation is necessary. And yes, let’s not shy away from the facts; that means real censorship, possibly well beyond national borders. There are times when this will be seen—perhaps rightly—as unbridled U.S. hegemony, Western arrogance, and selfishness. Quite honestly, I’m not always sure where the right balance is, but I now firmly believe that complete openness will push humanity off the narrow path. On the other side of the ledger, though, as should also be clear, complete surveillance and complete closure are inconceivable, wrong, and disastrous. Overreach on control is a fast track to dystopia. It too has to be resisted.
In this framework countries are always at risk. And yet some have managed to keep going for centuries, working hard to stay ahead, stay balanced, stay just shackled enough. Every single aspect of containment, all of what we’ve described, will have to tread this excruciating tightrope. Every measure discussed here or in the future needs to be seen on this spectrum—pushed far enough to offer meaningful protection and yet prevented from going too far.
Is containment of the coming wave possible?
Looking at the myriad paths forward, all the possible directions where technology will take human experience, the capabilities unleashed, the capacity to transform our world, it seems containment fails in many of them. The narrow path must be walked forever from here on out, and all it takes is one misstep to tumble into the abyss.
History suggests this pattern of diffusion and development is locked in. Immense incentives appear entrenched. Technologies surprise even their creators with the speed and power of their development. Every day seems to herald a new breakthrough, product, or company. The cutting edge diffuses in a matter of months. Nation-states charged with regulating this revolution are flailing because of it.
And yet, while there is compelling evidence that containment is not possible, temperamentally I remain an optimist. The ideas presented here help give us the tools and means to keep walking, step by step, down that path, the lamps and ropes and maps for wandering the tortuous route forward. The blunt challenge of containment is not a reason to turn away; it is a call to action, a generational mission we all need to face.
If we—we humanity—can change the context with a surge of committed new movements, businesses, and governments, with revised incentives, with boosted technical capacities, knowledge, and safeguards, then we can create conditions for setting off down that teetering path with a spark of hope. And while the sheer scale of the challenge is huge, each section here drills down into plenty of smaller areas where any individual can still make a difference. It will require an awesome effort to fundamentally change our societies, our human instincts, and the patterns of history. It’s far from certain. It looks impossible. But meeting the great dilemma of the twenty-first century must be possible.
We should all get comfortable with living with contradictions in this era of exponential change and unfurling powers. Assume the worst, plan for it, give it everything. Stick doggedly to the narrow path. Get a world beyond the elites engaged and pushing. If enough people start building that elusive “we,” those glimmers of hope will become raging fires of change.