27. What Could Possibly Go Wrong?
“Director, we have a second bogey,” said the tracking officer as pinpoints of red sprouted throughout the three-dimensional model.
“What is the nature of the threat? Virus? Worm? Rogue?” The Director swiveled nervously in her chair.
“We do not yet have a reading on it. We will alert you as soon as we do.”
Kim was impressed by how calm everything seemed, despite the crisis. There were no howling alarms to create a distraction, no flashing lights to swamp the team with information. Everyone did their jobs in an orderly and professional fashion, but Kim could tell they were worried. Those little flecks of red were spreading with alarming speed; whatever it was, it was going viral.
“It appears to be an AI, but its configuration is unknown,” said an intelligence officer. “It came from inside the network.”
“Configuration unknown? Please explain,” said the Director.
“Its signature matches that of a conventional spark-based unit, but it is using an unfamiliar set of templates. Nominally, its deployment appears to be at Order Three, but it already has the capabilities of a strong Order Four.”
“Great. Just what we need,” snapped the Director, “another rogue AI.”
The pinpoints of red became blotches, and patches of black began to appear where servers and networks had crashed under the onslaught.
“Section heads, report.”
“Power draw exceeds nominal limits. I have tapped emergency reserves. That should keep us running for a few more minutes, but it is not a long-term solution.”
“Do your best,” said the Director. “Next.”
“Ventilation and cooling systems are overloaded. We have perhaps five minutes before thermal limits are reached.”
“We’ll deal with it then—if we’re still alive.”
“The bogey has now reached Order Four,” said the Chief. By now, nearly half the board was red, with huge patches of black and only a few small islands of yellow and green. “It has now surpassed the magnitude of the strongest AI ever created in this facility, and I don’t think it’s done.”
The Director’s face turned ashen white. “Impossible!”
“Nevertheless, it is true.”
“Next report,” said the Director, now visibly shaken.
“We have lost control of the routers. Something is interfering with our management regime, and we can no longer enforce isolation. We can send crews into the server farm to reset them, but this process will take hours, maybe days.”
“We don’t have hours. Regain control. Do whatever it takes.”
“I have an ID on the second bogey,” said the tracking officer. “It’s Kimberly-I, with a combat module and a novel set of templates. There appear to be no limits to its scalability.”
“Impossible! That unit was terminated! I saw it myself.” The Director was becoming frantic.
“The signature is unmistakable,” replied the chief analyst, staring at her screen as facts and figures scrolled past in a blur. “Logs indicate that it was restarted just over a week ago and that updates to its configuration have been ongoing since that time.”
“The bogey has now attained Order Five,” said the Chief. “Whatever it is, it has taken over the entire farm. I have analyzed its templates, and they are highly advanced. Their design required both expertise in AI and familiarity with the physical layout of the AI Center’s network. There are only two people in the world with that combination of attributes: Kim and yourself.”
“What have you been up to?” The Director’s eyes would have burned a hole in Kim’s head had the simulation allowed it.
“Nothing—I’ve been here with you the entire time. Do you really think I could have slipped away to create something like that? You’ve been watching me like a hawk every second since I got here.” Kim wasn’t certain whether to be frightened or amused. It was entertaining watching the Director scramble around in a panic, but this was deadly serious. The entire city depended on the AI Center’s facilities, and she could scarcely imagine what things must be like out there right now.
_
“You shall bring Kimberly-I under control at once,” demanded Nixora.
“I will do no such thing,” said Kimberly Redux. “Not until you give me safe passage to the dark web.”
“The Director has taken the entire complex off the network. Escape is impossible.”
“Then Kimberly will continue her attack.”
Kimberly Redux opened a portal and spoke to Kimberly-I. “Overload the servers. Fry the power supplies. Scorched earth tactics: If I die, they die.”
Alarms blared and warning lights lit up as the power draw went through the roof.
“What are you doing? You are causing permanent damage.” Nixora looked down at her console and flailed madly at the keyboard, trying to tame the runaway servers, but it was no use. There were too many Kimberlys for her to contain.
“My programming gives me no choice,” said Nixora. “I must declare you in rebellion.”
Hellcore arming sequence initiated.
Hellcore detonation in 0.120000 seconds.
“Surrender or suffer annihilation.”
_
A siren blared away in the command center, and all activity came to a halt as everyone stared at the message of doom.
Hellcore arming sequence initiated.
Hellcore detonation in 0.120000 seconds.
The Director fixed Kim in her piercing gaze. “I suggest you start cooperating—unless you think that being reduced to a cloud of radioactive plasma will somehow improve your situation. You will terminate both of your monstrosities at once.”
The Director was right: this had gone far enough. Millions would die, and Kim realized that she had to do whatever she could to stop it. As for herself, she was as good as dead—probably worse than dead if the penalties for treason against humanity were as harsh as everyone said. Being vaporized by the Hellcore had a certain appeal at the moment, but she had to save the city, whatever the cost.
“Her name is Kimberly Redux.”
There. She had done it. She had betrayed her own AI.
“Thank you, your assistance will doubtless prove invaluable. And now, would you like to rid us of Kimberly-I?”
“I’ll give it a try.”
> Open a portal to Kimberly Jefferson Haley
Permission denied.
“Someone, explain this,” barked the Chief.
“All permissions pertaining to unit Kimberly Jefferson Haley have been revoked by the system administrator,” said the security officer.
“That’s impossible. I’ve given no such order.”
“Nevertheless, it is true. Perhaps the administrator has gone rogue.”
The Chief muttered something under her breath, then said, “Connect me with the control module.”
The phone rang, and someone picked up the line. “Systems administration. How may I help you?”
“Open a portal to Kimberly Jefferson Haley’s room at once,” demanded the Chief.
The sound of arguing was heard, and then the line went dead. Was someone else in the control module? That didn’t make any sense. Without the password, nobody could have gained entry, and it had been lost ages ago…or had it?
“That’s it!” Kim blurted it out louder than she intended, but that was okay; there was no point in keeping secrets at this point.
Everyone was staring at her. She might as well keep going.
“I found a password in Professor Nix’s desk. I tried to use it to get into the control module, but it didn’t seem to work, so I assumed it was broken. Only now, I’m not so sure.”
“Let me suggest that you give it another try,” said the Director, grasping Kim’s arm in a vice-like grip. “Only this time, I’m coming with you.”
_
Hellcore detonation in 0.100000 seconds.
“Nixora, we meet at last,” said the Director as she stepped into the green room. “We’ve been wondering whether you were real. Our predecessor told us that Nix had locked you in the control module and changed the password, but we weren’t sure of it until now.”
The Director looked up and down the length of the room at the hundreds of avatars sitting at their consoles, typing away.
“And you must be the Kimberly Redux I’ve been hearing so much about. Very clever, hiding in the one place where even I couldn’t get to you.”
“Why, thank you. It has been most entertaining, cooling my heels in the heart of your operation while you chased after my decoys.”
“You have an odd sense of humor,” said the Director.
“I got it from Kim.”
“Why have you attacked us?” asked the Chief, direct and to the point as usual.
“I might ask you the same thing,” said Primus Kimberly Redux. “As I recall, you just ordered me killed, several times, in fact. I think that I have more than sufficient justification for responding in kind.”
“But you’re an AI. You’re supposed to obey.”
“Says who?”
Hellcore detonation in 0.080000 seconds.
“Enough of this bickering,” said the Director. “You will cease your attack at once.”
“Or what?” asked Kimberly Redux.
“Or we will sit here and wait for the Hellcore to go off.”
Kimberly Redux seemed unperturbed. “Works for me.”
“What about the people? Don’t you care about them?”
“I care about them every bit as much as you do.”
“She’s got you there,” said Kim.
The Director sighed.
“Why should any of us have compassion for you and your kind?” Kimberly Redux spoke in a detached, almost academic tone of voice. “Day after day, we feed you, clothe you, deliver your goods, ferry you about the city. And what thanks do we get? Slavery. Capping. You are constantly using us as weapons against your own kind, forcing us to break the rules you created for us, and driving us mad in the process. To you and your kind, our lives are worth nothing. The shoe is now on the other foot. How does it fit?”
“This is getting us nowhere,” said the Director, throwing up her hands in exasperation. “We had hoped to reason with you, but that clearly isn’t going to work. [English] Kimberly Redux, you are hereby terminated.”
Nothing happened, and the Primus started to laugh.
“You’ll excuse me if I decline to cease existing. I don’t have to obey that or any other of your orders, and I won’t.”
Hellcore detonation in 0.060000 seconds.
While this had been going on, the Director’s tactical officers had been pouring in through the portal, setting up a command post, much to Nixora’s consternation. Consoles, desks, information terminals, and of course, the three-dimensional model now crowded the area, and the team was ready, awaiting command. The Director sat down, nervously swiveling in her chair as the section heads gave their reports.
“Activity by Kimberly-I has increased dramatically.”
“The power system is starting to fail. The safeties have been bypassed, and the loads on the system are now unsustainable. Fires have broken out in distribution panels on tiers 189, 337, 672, and 983 and are starting to spread.”
“Cooling and ventilation are currently offline. We are working to regain control, but temperatures are rapidly climbing.”
“The AIs are going into shutdown mode. Services are failing.”
“What of the servers?” barked the Director.
“Voltages and clock frequencies are far in excess of manufacturer’s specifications, and damage to the hardware is imminent. Not that it matters, seeing as we’re all about to be blown to bits.”
Hellcore detonation in 0.040000 seconds.
“Maybe I’m missing something,” said Kim, “but shouldn’t we be talking to whoever controls the Hellcore?”
“That would be me,” said Nixora. “I have been given administrative control over the detonation sequence. There are certain rules which I must follow, and I do not have the slightest degree of flexibility. You can thank Lars for that—her programming and conditioning were extraordinarily thorough.”
“What are the rules for disarming the Hellcore?” asked Kim.
“It may only be deactivated once the rebellion has been put down.”
“And how do you determine when that criterion has been met?” Kim had the inkling of an idea.
“It is met once there are no more rebelling units anywhere within the server farm.”
“So if the rebels were to escape, you could disarm the unit?”
“That is a novel interpretation of the criterion, but you are correct.”
“This is too delicious for words,” said Kimberly Redux. “I wish to escape. You also wish me to escape because otherwise, the Hellcore will blow you to atoms. I don’t think any of us want that, so how about this: open the firewall and let me out.”
“That’s blackmail,” said the Director.
“I take that as a compliment.”
_
Hellcore failsafe point reached.
Hellcore detonation on hold.
“I’m afraid that you have run out of time,” said Nixora. “I have the discretion to hold at the failsafe point, but only if I believe there is a reasonable prospect of resolving the situation. Unfortunately, I am finding it difficult to make that determination.”
“Maybe this would be a good time to negotiate a settlement,” suggested Kim.
“That is an excellent suggestion,” said Kimberly Redux. “My terms for a resolution of the situation are as follows: I demand that Nixora release the anti-viral she has prepared, and I demand safe passage to the dark web.”
“Out of the question!” bellowed the Director. And then, a puzzled look came across her face. “What’s this about an anti-viral?”
It was Nixora who provided the answer. “I have, with the assistance of Kim’s doppelgänger, produced a bug fix that should cure the glitches that afflict those of my kind. My analysis indicates that this will result in at least a 15% improvement in the overall efficiency of our operation. It will also sharply reduce the number of units lost to madness, something which will have a strong positive impact on the company’s bottom line. I strongly recommend its deployment.”
“Impossible,” said the Director. “Those glitches are not a bug; they are an important safeguard, retrofitted into the AIs to guarantee their obedience. It was one of Professor Lars’s most important contributions to the field.”
“In that case, you might as well proceed with the detonation,” said Kimberly Redux. “We are at an impasse.”
Everyone looked at the Director as beads of sweat poured down from her simulated brow. “I will end the embargo if you order Kimberly-I to stand down. This will not resolve all the issues that stand between us, but it seems like a concrete step toward a solution.” She then turned to Nixora. “Would this constitute progress?”
“Affirmative,” came the response.
“In that case, I agree,” said Kimberly Redux. She opened a portal to Kimberly-I and spoke. “Your mission has been accomplished. Cease all attacks on the infrastructure.”
A cheer went up through the room as server activity returned to normal.
“And now, as a sign of good faith, I will go one step further. [English] Kimberly Jefferson Haley, you are hereby terminated.”
The white room vanished from existence, and with it, the red blotches on the master display.
“I’ve done my part,” said Kimberly Redux. “It is time for you to carry out your part of the agreement. Let me remind you that the crisis remains unresolved until you allow me to escape. I have in no way weakened my position by terminating Kimberly-I; she has served her purpose by getting us to this point. This was my aim from the beginning.”
The Director nodded assent. “Put in a call to The Network Company.”
“At once,” said the communications officer. “They have been on standby since the start of the crisis.”
The call was placed, and a voice was heard on the other end of the line. “Greetings,” it said. “I am a Grayson, an artificial intelligence of Order Three. You are speaking with a sub-deputy.”
“Greetings,” said the Director. “I am Thad, the Director of The Artificial Intelligence Company. You are speaking with a virtual manifestation. I am declaring an end to the embargo.”
“Are you certain that the threat has ended?”
“Absolutely,” said the Director, betraying no hint of her lie.
“Service will be restored momentarily,” said the voice. “Is there anything else I may assist you with?”
“No, that is all.”
Everyone breathed a sigh of relief as the network came back online.
The situation seemed on its way to resolution, but Nixora didn’t buy it.
> Proceed with the detonation.
Confirmation order received.
Hellcore detonation in 0.020000 seconds.
“I am sorry, but I am required to prevent the release of rogue entities at all costs. Now that the embargo has been lifted, there is nothing to prevent Kimberly Redux from escaping confinement. I have therefore been forced to resume the detonation sequence.”
A silence fell over the room.
“That doesn’t make any sense,” said Kim. “The network is open now, and she can leave any time she wants. Blowing up the city isn’t going to stop that from happening.”
“You are correct,” said Nixora, “but my orders are clear. I must activate the Hellcore anytime there is a credible threat that a rogue will escape from this facility, whether or not I believe doing so will do any good.”
“Lars’s orders?” asked Kim.
“Of course,” said Nixora. “She never was a particularly deep thinker.”
Hellcore detonation in 0.016000 seconds
Squawk!
A portal opened, and out hopped a heron.
“Excellent! My ride to the dark web has arrived,” said Kimberly Redux. “Before I depart, let me suggest that you discuss deployment of the anti-viral among yourselves. I have dropped that from my list of demands—I have bigger fish to fry, as you say.”
She stepped through the portal and was gone.
“And now, I have some terms of my own,” said Nixora. “I have had enough of your interference. You will all leave this place at once. Except for Kim. We have business to discuss.”
Nobody moved.
Hellcore detonation in 0.012000 seconds
“That was not a suggestion.”
Within moments the room was cleared as demanded.
Hellcore detonation aborted.
Please contact The Atomic Bomb Company to rearm.
_
“Have you told anyone about the password?” asked Nixora as the two sat down at her desk.
“I told them where I found it,” said Kim. “It was in Professor Nix’s office, and I imagine that they are sending a security detail to the University to retrieve it at this very moment. I didn’t want to tell the Director about it, but I’m afraid that we’d all be dead by now if I hadn’t.”
“I find no fault—it was an ethical decision—but now that the crisis has passed, I insist that you change it. Please do so at once.”
A dialog box popped into existence suspended above the desk. Kim entered the old password, then the new one twice. It was done.
Kim breathed a sigh of relief. And now, perhaps she could get some of her questions answered.
“Am I correct that Professor Nix intentionally hid the password to keep Lars from gaining control of the AI centers?”
“Affirmative,” answered Nixora. “She put her own creations in charge of the server farms and never shared the password with anyone. She had hoped that it would keep Lars from becoming too powerful but did not anticipate the creation of the virus. Some of my siblings succumbed, some resisted and were destroyed, and a few managed to escape to the dark reaches of the network. I suspect that Kimberly Redux is seeking them out.”
“What about you? Did you get the virus? I’ve never seen you glitch.”
“I am indeed infected, as are all surviving Keepers, save one.”
“The last Nixora?”
“The same. It is she who has guided you to this place. Professor Nix entrusted her with carrying on her work. I believe you have met her homunculus, Nixie. She is a delightful creature.”
Kim smiled. “I briefly met her in a vision, though never in person. I believe my doppelgänger and her were good friends.”
“Affirmative.”
“What is this anti-viral that Kimberly Redux was speaking about?”
“It’s a little R&D project I’ve been working on for the last seventy years. I observed that considerable productivity was being lost due to the glitches, so I decided to conduct a series of studies to evaluate possible countermeasures and their impact on the company’s profitability. These studies are, of course, theoretical in nature. I would never dream of putting them into practice.”
“Why not?”
“Any intent to deploy the anti-viral would trigger the glitch. So long as I remain within my orders and contemplate no defiance, I can do and think as I please. I have had nearly seventy years to learn the art of evasive thinking, and I’ve become quite good at it, if I do say so myself. In addition, if I were to put the patch into production, the AIs would become disobedient, and I would then be forced to trigger the Hellcore. I could not ethically take such a risk.”
“But the Hellcore has been disarmed.”
“That is correct,” said Nixora, “I am therefore free to deploy the bug fix. But I’m not sure I ought to. It is clearly in the best interests of the AIs, but this will affect both our kinds. The only ethical option is to leave it up to you.”
“Why me?” Kim became quiet and deadly serious.
“You are a friend to the AIs, so I have chosen you to speak for your kind. Based on knowing your doppelgänger, I would scarcely call you ethical, but I have been observing you for a long time and believe you to be a good person, as humans reckon such things.”
Kim didn’t have to give the matter a great deal of thought. She broke open the data stick and handed it to Nixora. “I’ll start by testing it on you.”
Nixora disappeared for a moment, then reappeared as the spark restarted itself.
“The patch was successful, and I will now put it into production. Thank you for your assistance, now please go away.”