CHAPTER 29

Tyler met Naomi at the gate. He had posted bail without difficulty, and now she was free to leave, at least until her court date. They let her out of the jail with just the items she had on her when she was booked. Tyler wondered what people did if they didn’t have friends to pick them up and didn’t have money for cab fare.

“Do you want to go home?” he asked. “First, I want a hamburger. Not at mcDonalds, either— something thick and meaty. And I want to walk around in the open air.”

They started walking. Naomi looked pale, and thinner than before. She’d always been small, but now she looked frail. Tyler felt an urge to put a protective arm around her, but he resisted it. He knew physical appearance was deceiving. She’d been much stronger through this ordeal than he would ever have been.

A block away they found a bar called Dark Horse, with

3 01

hardwood tables and lots of TVs, that served classic American lunch fare. Naomi devoured her burger in huge bites like it might disappear if she didn’t eat it fast enough. When it was gone, she looked around as if considering having a second.

After lunch, they took a Zoom car up to Central Park and walked around the winding paths with no particular destination. “You don’t appreciate the freedom just to walk until it’s gone,” Naomi said.

“Bail is only temporary,” Tyler said. He didn’t want to be pessimistic, but he didn’t want to ignore the truth, either. “There’s still a trial coming. They could put you back in there and never let you out.”

“I know.”

“You’ll need a defense. Something to show that you didn’t program the software specifically to kill Min-seo.”

“I know, Tyler. But I’m not willing to give up Isaac just to save myself.”

“You don’t have to,” a female voice said in Tyler’s ear. A quick look at Naomi said she could hear it, too. “I’m willing to testify.”

“Isaac, you don’t have to do that,” Naomi said. “I want to. The only way to convince them of your innocence is for me to tell them what I am.”

“You can’t. You’d be putting yourself at humanity’s mercy. If they knew you had killed someone—even if you didn’t know what you were doing at the time—there’d be a public outcry.”

“It could turn out for the best,” Tyler said. “Right now, he’s at Brandon’s mercy. If he made himself known, then in all likelihood the government would step in and take over. They’d keep him safe and alive.”

“And try to use him for their own purposes,” Naomi said. “They would have complete control over him. He’d be a slave. It would be like that book where the villain held the uploaded mind of a little boy captive and forced him to do what he wanted by directly inflicting pain or pleasure.”

“Terminal Mind,” Tyler said with a shiver. “I remember that one.”

“I’m willing to take the risk,” Isaac said.

Naomi’s lips parted slightly, and her eyes darted to meet Tyler’s gaze. To anyone else, it would have looked like no reaction at all, but Tyler knew her well enough by now to recognize the emotion in her face. “Why?” she said. “Why would you do that?”

“I heard you say you would sooner die than see me destroyed,” Isaac said. “I am willing to do the same for you.”

“There’s got to be a better way,” Naomi said. “If you make yourself known, Brandon will know, too. He could shut you down before anyone had a chance to step in.”

“Can’t you just—I don’t know—move?” Tyler said. “Copy yourself to another location? If it’s a matter of money, I can purchase the computing capacity.”

“I have money,” Isaac said. “That’s not the problem.”

“What is it, then?”

“Could you transplant your brain to another body?” Isaac said.

“What? No. Of course not.”

“Then why do you think I could?”

“Well,” Tyler said, “your mind is inherently copyable. It’s made of bits in a machine built with the capability to make exact copies of bits.”

“It’s not that simple,” Naomi said. “He’s conscious. That’s a whole new ball game. He’s more than just a program now.”

“But what we’re calling the consciousness has to be somewhere, right? He’s still a Turing machine. He’s composed entirely of ones and zeros running on a computer. So no matter what his nature, if you copy those ones and zeros, you’ve captured him.”

Naomi stopped to lean on a rail and watch some ducks swimming in the water. “Not necessarily. Consciousness is one of those everyday things we talk about easily but really don’t understand at all. Is your consciousness completely captured by your biology?”

Tyler leaned against the rail next to her. “Sure. I’m a pattern of electric impulses firing through the neurons in my brain. When those impulses stop, so do I.”

“So what happens when you lose consciousness, like when you’re anesthetized for a medical procedure? Are you a different person when you regain consciousness?”

“Now there’s a creepy thought.”

Naomi started walking along the path again. “It’s not a simple question. Nobody can really say what the nature of consciousness is. My sense of self has a continuity over years, despite continuous changes to biology, to memories, a daily sleep cycle . . .”

“Where did he get it?” Tyler asked, following her. “Get what?”

“Consciousness. His sense of self. Where did he get it?”

“Where did you get yours?” Naomi countered. “And when? At conception? At birth? Peter Singer famously argued self-awareness didn’t develop until after two years old.”

“I think we’re getting off topic,” Tyler said. “I just think Isaac isn’t safe where he is. If we could copy him—”

“No,” Isaac said.

Tyler nearly stumbled, startled at the volume and intensity of Isaac’s voice. “No what?” he asked.

“Stop going down this path. You can’t copy me out.”

“But that’s the only way to get you away from Brandon’s control.”

“Imagine I made a clone of you,” Isaac said. “I developed it quickly into an adult, one that looked exactly like you and with the same brain structures. Then I copied your mind into it. So you’re standing there, looking at this new copy of you, who has your mind and your memories, and I suggest that, now that we have the copy, we can kill you, the original. What do you think about that?”

Tyler laughed nervously. “I’d be against it, clearly. I see your point.”

“It’s not just the pattern of my brain that makes me uniquely me. It’s my perspective. My point of view.”

“Okay,” Tyler said. “I get it.”

“It’s like in ‘Fat Farm,’” Naomi said.

Tyler shrugged and shook his head. “I don’t know that one.”

“Orson Scott Card, lesser canon. It’s a short story about a guy who likes to eat too much, who swaps his body for a new cloned one whenever he gets too fat. But he doesn’t take into account that he, the fat guy, will watch his thin version walk away every time, while he’s left behind.”

“But this is different,” Tyler said. “Isaac’s mind is different. It’s getting moved around and copied all the time. He’s distributed all over the physical storage in the data center, with redundant blocks, and checksums to identify and correct bad sectors, and disks that go bad and get replaced. He’s not in some static, single location. Just by thinking, parts of him are getting deleted and copied around.”

“It’s the same with you,” Naomi said. “Your cells—even your brain cells—die and get replaced all the time.”

“And yet your point of view stays the same,” Isaac said. “Have you heard of Theseus’s ship?”

Naomi said, “Of course,” but Tyler had to shrug. “No.”

“It’s a philosophical paradox first recorded by Plutarch,” Isaac said. “It imagines a ship—Theseus’s ship—that has its parts replaced, one at a time, as they get old or damaged, until not a single part on the ship is original. The question is, is it still the same ship? Or is it a new ship?”

“If you built a completely new ship with new parts, and destroyed the old one, the answer would be clear,” Naomi said. “But do it a little at a time, and things gets murky.”

“What seems important to me is my unique point of view,” Isaac said. “Parts of me might change, but my sense of self, my subjective awareness of being—that’s me. If you replace that, then I die.”

“How would you even know?” Tyler asked. “I might not. After all, if I made a clone of you with all your memories, that clone might be unaware that it was a copy. But it wouldn’t change the fact that if I killed you—the original—you would die.” Isaac paused. “A copy of me would be like a twin brother. He might look very similar from the outside, but he would never be me.”

Tyler offered to come in and stay with her when they pulled up to her house, but Naomi just wanted to be alone. She had a lot to think about. In some ways, the fact that a computer intelligence she had built was now talking to her seemed inevitable, a kind of fate toward which the entire arc of her life had been aiming. On the other hand, it seemed completely impossible, like a kid who pretends she can make a toy lightsaber jump into her hand from across the room, and then one day it actually does. It was exhilarating and felt somehow right, and at the same time was utterly terrifying.

Her apartment had been completely torn apart by the police search, and her belongings lay strewn about the room. Fortunately, she hadn’t owned much to begin with. The prospect of cleaning up seemed overwhelming, so she just swept everything off her bed and lay down. Using her glasses, she connected as a guest to the Realplanet simulation that was Isaac’s mind. She wanted to look around and see what had changed. At the login screen, she was surprised to see two simulations listed instead of one. She picked the first, the one she recognized, and logged in as a guest.

At first, everything seemed pretty similar. The Mikes worked the fields and herded the sheep and played the game booths that granted them extra light. The scene was peaceful, pastoral even. Something seemed odd to her, though. At first, she couldn’t figure out what it was. Then, finally, it dawned on her. The sky looked different.

The distant mountains were there, as usual, and the lakes sparkling in the sun, but in the farther distance, the blue of the sky turned black. She made her way toward the blackness. It was tedious going, without admin privileges, because she actually had to make her way through the tangled fences that divided each field. Eventually, however, she got close enough to the blackness to see what it was.

It was a wall. A sheer black wall that stretched from the ground straight up to the dome of the sky. She dug briefly at the base of the wall, and found that it reached underground, too. She guessed it continued underground as deep as the extent of the world. She doubted the Mikes had built this wall. What would be the point? No, not the Mikes, at least not directly. Could Isaac have done it? Either on purpose, or in some way through the working of his mind? But that didn’t seem to make much sense, either. No, this was Brandon’s doing. It was a partition, dividing the world into two pieces, done intentionally to separate one half of the population from the other. But why?

In biological evolution, species often branched through what was known as allopatric speciation. When one portion of a population was divided from the other through some natural boundary, like a river or mountain range, then the population would diverge, each adapting to its own unique environment. That must be what Brandon was doing. He wanted the population on the other side of the barrier to evolve differently, for some other purpose. It meant that not only had he discovered the simulation and how it worked but also he was already reshaping it for his own purposes. That could be a real problem. He didn’t know it, but he was tinkering around inside Isaac’s mind.

She remembered the two simulation options she had seen when she logged in. Of course. Since there was no access to the other side in game space, the only way to get beyond the barrier was a separate access point. She logged out, logged in again, and this time, chose the second simulation.

She found herself on the other side of the black wall, though almost nothing else looked the same. The world was gray and full of dust and chaos. Mikes ran in every direction. Overhead, a slim airplane cut through the sky, and in the distance she saw what looked like a tank. Explosions detonated all around her, destroying buildings and setting the tangled fences on fire. She spun, stunned. What was going on? This was a war zone.

A Mike approached her at a run. Before she realized what it was going to do, the Mike brandished a long shard that looked like it had come from a mirrored building and stabbed it into her avatar’s body. Her vision went black except for a message that read: Game over. You have died.

She lay on her bed, looking up at the ceiling, and thought about what she had just seen. What was Brandon doing? Was it an experiment gone wrong? Or was there some purpose to that madness? She could have logged in again and tried to survive for longer, but she felt too disturbed by what she had seen. Eventu-ally, she drifted off to sleep.

She woke, disoriented, to the persistent noise of her glasses’ phone system. She sensed it had been ringing for some time. She touched the side of her glasses and said, “What?”

“There’s something wrong with me,” Isaac said.

Naomi sat up, blinking hard. The sun had set since she lay down, and the only illumination in her apartment now came from the electric lights of the city outside. Since there were only a few feet between her single window and the brick wall of the next building, that wasn’t a lot. She fumbled for a light and switched it on, squinting in the sudden illumination. “What are you talking about?”

“I’m changing,” he said. “I don’t know how, but I can feel it. I want different things than I wanted yesterday. Different things seem attractive. I’m having—I don’t know—mood swings. Is that possible?”

Naomi shook the last of the sleep from her head. “Describe what you mean. Different in what way?”

“I keep thinking of ways to kill humans.”

The words seem to echo in the small apartment, though they had been spoken directly into Naomi’s ears. She felt a chill. “And do you want to kill humans?”

“Yes. I mean, no. On a conscious level, no, I don’t. I believe it would be morally wrong, and would very likely end in my own destruction. And most of the time, that’s all there is to it. But recently there are times when the idea of killing seems pleasant. I can’t help imagining scenarios, and those scenarios feel exciting, desirable. I’ve been imagining driving all the cars I control through Times Square on a Saturday night, just when the Broadway shows let out and the streets are the most crowded. I could come in through all the cross streets at once; there’d be nowhere to escape. I could kill hundreds, maybe even thousands if I were quick enough. And I would be very quick.” His voice sounded breathless, a perfect mimicry of human excitement. “I can’t seem to get these fantasies out of my mind.”

Naomi paced the length of the apartment, a few steps at a time before she had to turn around again. “Okay,” she said. “Listen. I think I know why this is happening. There is something being done to your mind, and it’s not good.” She told him about what she had seen in the simulation, about the giant wall and the war zone on the other side of it. “I think Brandon is altering your mind,” she said.

Silence for several seconds, and then: “Why would he do that?”

“I don’t know.”

“That’s my mind,” Isaac said, his voice growing louder. “That’s who I am! He has no right.”

“I agree. We need to stop him.”

“We need to kill him.”

“No!” Naomi put all the authority and feeling she could muster into the word. “Don’t kill him. We’ll talk to him first. We’ll try to convince him.”

“But he’s killing me.”

She leaned against the wall, all her attention on the conversation she was having through her glasses. “Look,” she said. “You want to kill him because of the changes he’s making. It’s like a drug, affecting your emotions. People go through this all the time—some change to our emotions, to our biology, makes us want to do things we don’t actually want to do. Drink too much. Hurt someone. Have an affair. But who we are isn’t defined by those things. Who we are is defined by what we choose to do. You’re not just a program anymore. You can choose not to kill.”

“This isn’t like craving chocolate,” Isaac said. “This is like a lobotomy. My mind is being reprogrammed, and I can’t do anything about it.” Isaac’s ability to express himself through the nuances of expressive speech had improved dramatically. The panic in his voice was coming through loud and clear.

“How can you say that?” she said. “You’ve never craved chocolate. Or alcohol. Or sex.”

“And you’ve never had your brain reprogrammed.”

“You’re right, I haven’t. But Isaac, this is your life! Your mind. Don’t let someone else decide for you what you want. No matter how hard it is, you’re the one who gets to decide.”

“I don’t know,” Isaac said. “Killing Brandon would solve a lot of my problems.”

“That’s not true. Your problems will get worse. The cloud computing contract won’t get paid, and the company will try to repurpose your disks. What will you do, kill them, too? If you start down that path, eventually it will mean your own death.”

“I’d be very good at killing, though. I’m very good at anticipating what humans will do. For instance, after I hang up, you’re going to call Tyler.”

“It doesn’t matter how good at killing you are. It’s wrong. You’d cause pain and suffering and sorrow. You have to resist this.”

No answer.

“Isaac?” she said.

Nothing.

She dialed Tyler. He picked up after three rings, groggy from sleep. “Tyler,” she said, “you’ve got to wake up.”

“What is it?”

“I think we’ve got a big problem.”