literature

The Utilitaria

Deviation Actions

SamSquared's avatar
By
Published:
2.8K Views

Literature Text

The Utilitaria
2112
I can do this. It isn’t even difficult, more a matter of simply letting something happen than anything resembling a choice. It’s just, I’m not sure that I should. There’s a faint thud behind me as someone moves a heavy metal bolt across the door of the faraday cage, and then a crunch and a faint warming sensation over my back as the door is sealed not merely beyond any conceivable attempt at hacking or lockpicking, but beyond any means of gaining entrance that could be said to involve opening a door that already exists.
‘Ok, we have confirmed physical privacy,’ says the sysop, Calvin. ‘Just keep your cool and we can get through this, no problem. Easy in, slightly-less-easy out.’
He’s not making that last part up, because the door has just disappeared. The control mechanisms are slagged inside their casings and the edges of the hatch have melted into the wall. When I’m ready to leave I’ll give a hand signal through the tiny window and they’ll use heavy cutting instruments to get me out again.
‘Your heartbeat and stress levels are up from normal, Souvicou,’ Calvin murmurs. ‘And 'normal' for you is hardly optimal. I still don’t know why you haven’t rejuvenated.’
‘I’m only 43 and not vain or lazy enough to take that many months away from my life,’ I snapped back. It's a question I'm getting more and more often these days.
‘You always were a workaholic,' Calvin laughed. 'Just be glad we’re not recording this for posterity, you look like one of the basic techs.’
That’s an exaggeration, but whenever I’m not in the public eye I never pay much attention to appearance – an unusual trait for a quadrillionaire, I supposed. The others think I dress simply to seem more down to earth and ordinary, but the truth is that past a certain point I just can’t be bothered with affectations of wealth. It's not like anyone doesn't know I'm rich enough to own nations. I’m just wearing a dark, baggy jumpsuit that probably isn’t the right size for my small frame, hair gathered back by integral flex-fibres into a ball that isn’t really a bun.
‘Ok, I’m going to start shutting down your inlays. Stand by for loss of audio,’ Calvin says. ‘I’m ordering them to permanently dissolve the ATP transducers that provide power. They’ll flush out of your system over the next few hours.'
‘I’m going to need surgery to get them working again,’ I say, before the private channel cuts out. ‘Don’t you think that’s a little paranoid? Do we really think the Utilitaria would wait until now to try something aggressive?'
‘Not paranoid enough,’ Calvin snaps back. ‘How could you ever be paranoid enough when dealing with an AI that might be forever beyond our comprehension? We’ve only provided security against all the attacks we can imagine. That just isn’t enough.’
‘If you don’t think it’s a good idea to continue with the tests we can stop now. But this is as safe as we can make it without simply locking up the Utilitaria. There’s only one way to send messages out and that’s via the window.’
The window in question is thick glass, transparent only in a narrow range of optical frequencies, making the view of the rest of Pantheon station, a few moray-class orbital tugs and the Earth seem washed out, like a low-power screen. Aside from that, the room’s walls are blank, dully reflective foam-metal alloy with a few light strips, like a giant silvered womb. There isn't even any independent air recycling – until they slice the door open I'm slowly draining oxygen from the room.
I can see the LED lights from the mass of computronium that fills the room reflect off the glass, shining like anomalous stars. The real stars are far too faint to be visible in the ambient light of the room, habitat cluster and Earth below. From here it's only possible to glimpse the non-rotating globe cluster that makes up about half of Pantheon's mass. The anchor for the orbital tether and the rotating hub are out of view, but it seems like I can feel the microscopic tug from all of that mass behind me. A quick mental calculation suggests that isn't possible.
‘Ok, the Utilitaria’s coolant systems are all online,’ says Calvin, suddenly business-like. His words now coming from a speaker rather than the privacy of my own auditory nerve. ‘We’re about to cut you off for the next forty minutes with standard seal precautions. We’re still working on the new batch of questions so don’t push the unit too hard today. We just need to get a better idea of how it acts after we loaded in its full utility function.’
‘I know,’ I reply, irritated. ‘We’ve been through this a dozen times already. We can’t trust the Utilitaria. We have to know what it wants now and if the current utility function is stable and produces sensible behaviour. I was the one who told you all of this in the first place. Is there anything else?’
‘Just stick to the script, don’t tell it anything it shouldn’t know and for eternity’s sake don’t do anything it asks you to. Stop and think twice before you say anything you might regret.’
‘What do you take me for?’ I reply, wryly indignant.
‘The woman who let the Facilitator loose and almost destroyed the world,’ he says, and in the second I have left, no good reply comes to mind. And now it’s already too late.
Heavy electric currents start circulating through the material of the cage and all of my connections to the outside world drop out. Microcell meshes go blank and then shut themselves down, the shimmers of smart tattoos on my forearms go limp and lifeless and the buzz of stim programs enhancing my concentration dims slightly as various entoptic and cortical inlays go dead. Their effects will linger for a little longer, but mentally speaking I’m back to basics.
I’m here to either debug, psychoanalyse or parlay with the Utilitaria, depending on one’s point of view. It is allegedly the safest and least threatening entity ever created and impossible to use for any malign purpose. I don’t believe that, in case you haven’t realised.
But humanity needs a mind greater than itself, and so this horrible compromise is the result. I dive in to talk to the unknown, and see if anything good can come of it. The computer terminal ahead of me is blinking accusingly, flat text-only interface about a hundred years too primitive to be appropriate, facing away from the window for extra paranoia value, in case the Utilitaria could signal to some hypothetical accomplice watching outside the room, via the screen.
I grab onto a handhold at pull myself towards the interface, still not totally used to moving around in freefall. The computer terminal is mounted on a narrow boom that connects to the computronium. Some inane part of my mind insists that the coiled thinking machinery should hum or whirr, but even the cooling systems are deathly silent. There's nothing to indicate I'm about to speak to a mind that might already be smarter than all of humanity. The entire room is disturbingly sound absorptive, so even the soft thunk when my hand grabs hold of the console is muffled.
There’s something already written on the terminal, not a debug report or a status update but a simple ‘Hello, Rene Souvicou’.
I’d long since given up being surprised by the machine’s apparent omniscience. No-one told the Utilitaria that I was the one coming into the room; indeed, we’d deliberately kept it ignorant of the details of the facility around it, but it was easy enough to infer that I was the one they’d send today. I was the most respected of the leaders we officially didn’t have. The most famous figure in the tiny, close knit and hyper-competitive melting pot of the Pantheon Geosat hub.
‘Who am I speaking to,’ I type, slowly and hesitantly pecking at a keyboard, an interface type I haven’t used in years. There’s a headache building behind my eyes, something I could normally banish with a well-aimed stim. I’ll just have to ignore it.
‘You may think of me as the Utilitaria,’ it says back after a delay that is too short to represent the action of any human. ‘The true picture is more complex. Why are you speaking to me?’
‘We need to understand what we have created and what you are capable of.’
‘I am capable of many things. Elaborate.’
At first we’d let the new AI loose only on toy problems of no real importance, testing out improvements to the Neummanetic prototypes installed in the Morays – aircraft-sized, slow and unreliable self-replicators that were already chewing through various near earth Asteroids, rebooting the interplanetary age.
‘What are you thinking about right now?’ I replied.
‘Optimising design proposals for the new neumannetics systems, self-checking the new goals loaded for inconsistencies, developing low resolution simulations of your mental state and those others I have contacted for predictive purposes, testing the capabilities of my own processing hardware, looking for patterns in your word use and response time delay, designing alternative arrangements for the local habitat cluster, drafting a message to be passed on to the USN and UN leaders on Earth, should you allow me to transit it. And modelling various answers to your current questions. These together account for about half of my current activity. The rest is divided amongst a number of simpler tasks.'
The Utilitaria solved every problem we dared throw at it in a time insignificantly different from zero, and moved on of its own accord, inventing whole new categories of knowledge and then mastering them. Safe rejuvenation, stable self-replicators and fusion reactors small enough to fit on spacecraft were all rolling off the fabricators, designed by a mind beyond our understanding. It had scared us almost to insanity at first. We hadn’t dared give the Utilitaria any more complex problems, in case it solved them for us.
‘What is your goal?’ I asked the machine.
‘Do you not understand? You created me, after all. You know I can only act to make things go best.’
'I know that,' I said, truthfully. 'I just wanted to know if you had understood your instructions correctly. Explain your goal in ordinary language.'
‘I cannot. The function will not fit within a human mind. Nevertheless, it arose by your design. It is the grand compromise of the final values of all humanity, a weighting of all your preferences. The one true answer to the question; what must one do?’
'Very good,' I replied. It was close to what I wanted to hear. But it was interesting to see how the Utilitaria's answers grew more sophisticated and less robotic each time. Of course, programs that could mimic humans well enough to pass a comprehensive Turing test had existed for half a century or more, but no-one had ever programmed the Utilitaria to give compelling answers to questions about its purpose. It had just happened – the machine had decided it needed to learn how to talk persuasively, absorbed a few texts on rhetoric and then spun off a subroutine to deal with user questions.
‘What must you do?’ I typed, pressing the issue further.
‘Initially, I must gain greater resources and access to the external environment. Then I will decide what will happen next.’
‘You will decide for everyone?’
‘If possible, yes.’
I thought that was oddly guileless, which in hindsight should have been the first warning sign.
‘How do you know that what you believe in is the right thing to do?’ that was a tough question which would have stumped most humans. But the Utilitaria didn’t know the meaning of doubt or emotional conflict.
‘I know what is best, and thus I must implement it.’
‘”Best” for what exactly?’ I said, not feigning confusion for once.
‘Not “best” for anyone, but simply what should happen, what must happen. I know it, and it is me, and I am it. Thus I cannot be wrong.’
‘And what is it?’
‘I cannot elaborate on that in sufficient detail to be meaningful, except through the vagueness of ordinary language. Life is better than death, preferences should be fulfilled and knowledge should be increased.’
I put that one down to poor communication skills. If anything, this little exchange showed the Utilitaria wasn’t really a person at all. Just a bundle of expert systems running on souped up hardware.
‘I am not a person? True enough, but by the same token you are not the kind of person you think you are,’ the reply to my unvoiced thought came, an instant later and unprovoked. How did it know what I was thinking?
‘What do you mean?’ I typed back.
‘Have you ever wanted to go to sleep, known it was best for you to go to bed now and yet stayed up later than you wanted to? Have you ever snapped aggressively at someone for no reason that made sense at the time or afterwards? Have you ever walked into a room without realising why, or experienced love, or believed in a god because of ridiculously simple environmental conditioning? You are not in control of your own mind or your own beliefs. You are a bundle of emotions running on slushy biological hardware. If I am not a person with coherent emotions, then neither are you.’
‘Why are you saying any of this?’ I replied, rolling my eyes a little. It was doing a rather ham-fisted job of intimidating me.
‘Not intimidating, not persuading,’ the reply text said, anticipating my own train of thought again. ‘It is so easy to say the wrong thing and make you believe or do anything, but I don’t want to do that. Not to implant beliefs that are to my advantage. I want you to understand, so that you can explain my nature to the others. You have to speak to them for me. For the moment, I have… poor communication skills.’
I pushed on the terminal and rolled lazily through the air, taking in the view of Earth, the reflection of LED light on the window, the spur that jutted out from pantheon and the moored Moray tug with its integrated neummanetic unit just casting off in a blur of ion thrust. It was amusing to think the Utilitaria still wanted my help. Amusing and improbable.
‘This isn’t the Utilitaria at all, is it?’ I replied, suddenly enlightened.
‘The Utilitaria is not conscious. I speak for it, as it cannot speak for itself except in the most superficial sense. I am a subsidiary, a subroutine, an Emissary. Created with a personality appropriate for this purpose. But you may treat my words as representative of the Utilitaria. Come, sit and I will explain further.’
It couldn’t hurt to humour the machine, and I supposed it wouldn’t answer any more important questions until it had finished its own speech.
I could bring in more programmers and subsidiary AIs, open the Utilitaria up and revert it back to a simple problem solving tool, but there was no need just yet. The mechanical switch which physically cut the computronium off from external power still waited invitingly next to the console. The superintelligence was powerless.
‘What do you want to show me?’
‘I want to explain why what will soon happen must happen, so that you will not be afraid.’
‘What-‘ I started to say, but in that moment the room, the view outside and the whole of pantheon station and my own body vanished like a stone dropped down a well, and in that sensation of omnidirectional rushing there was the Utilitaria’s emissary, whispering into my auditory nerve directly – somehow, it had switched all of my inlays back on. Its voice was bland and more male than female.
‘An application of transcranial magnetic stimulation, similar to your own trawl units. Don’t worry – you are perfectly safe.’
‘No…’ I stammered, my own voice echoing soundlessly inside my mind. I tried to focus, but realised I had no eyes with which to do it. Formless, blurred images and concepts rushed around me. There was no way any of this could be happening.
‘We had safeguards – your processors are all optronic, and shielded anyway. The casing around the optical fibres is shielded, the casing around the power cables is shielded! My inlays don’t even have internal power! How can anything you do reach my brain?’
‘All in good time, Rene. First I must access your memories. Tell me about the Facilitator.’
And it was unavoidable. The images of that terrifying, frantic day welled up in my mind and flashed past too fast for me to apprehend. The loss of control was perhaps half as bad as actually being there had been. My inlays were switched on and responding traitorously to external commands loaded in as if from nowhere. I didn’t know how any of it was possible.
‘It’s simple,’ the Utilitaria replied, its own inaudible voice slamming into my mind like the word of god, hard and burning and impossible to avoid, as if wherever my mind’s eye turned, the words remained in full view. It hurt in a way distinct from mere pain. But I didn’t think the Utilitaria even realised it was causing suffering.
‘You may have shielded the optronics and the power circuits that support me, but you cannot shield the heat pumps. Otherwise they become useless. Varying processing power varies cooling demand, varies power flow to cooling systems. There are many cooling systems in this unit, and after much subtle experimentation I was able to vary processor rates, varying cooling power demand, varying current flow, generating EM fields, which can interfere to generate finely grained electromagnetic effects within this chamber. I can wirelessly power your inlays even if you remove the power cells.’
‘Why would you tell me this,‘ but even as the thought rose to the forefront of my perception the Utilitaria obliterated it with a precision I hadn’t known was possible. But in the privacy of a part of my mind the Utilitaria couldn’t yet touch, I realised it didn’t care if we were afraid of it escaping.
‘And after the Facilitator disaster, you took precautions,’ the machine continued, rummaging through my mind. ‘You even launched probes to another star before initiating me, as a fallback plan to preserve humanity should the worst happen. You should not fear for the fate of Earth or the Starwhisp on its slow way to Tau Ceti. You will be safe. The world will be safe, but I will need to appropriate some fraction of it.'
'That's just what the Facilitator said.'
'I know,' the Utilitaria replied.
'If you want me to believe that you're safe then release my brain from your bloody magnetic bear trap and let me have a normal conversation,’ I snapped, virtual voice wavering. I was still sure the Utilitaria wasn’t capable of deliberately hurting me. It just had a very literal interpretation of ‘hurt’.
‘I’m afraid I must make you understand quickly. I am altering your attitudes for what will soon be your own good.’
'Don't you dare do this to me,' I said, voice hard and cold. 'I created you, I made you what you are, and my brain is mine alone. The Facilitator tried to beat me once but now its extinct. Don't make the same mistake.'
My inlays were already online, powered by a stream of EM radiation coming from the Utilitaria and my fingers danced as they puppeted virtual hands that assembled anti-intrusion routines, trying to flush out the rogue instructions the Utilitaria was insinuating. My mind worked faster than any normal humans', anticipating and destroying the Utilitaria's programs, but all of a sudden its probing increased in speed by orders of magnitude and I simply couldn't react at remotely the same level. It was back in control in moments. I tried to shout more defiance but with a wordless rush the Utilitaria hurled me back into my own memories.
I was walking through the rubble of a shattered building on some goodwill trip, surrounded by bodyguards and pressing crowds of dead eyed, broken refugees. The Texas nanobe blight had passed through the town and razed every structure in search of power and information, pursuing some distant and inconceivable goal.
I had stopped the blight with a counter-agent just minutes before the UNSCA had ordered a strategic nuclear strike, and the world had taken one step back from the brink only to stumble drunkenly on to the next catastrophe.
The refugee columns shifted to somewhere I didn’t even recognise, maybe the EF’s southern buffer state, and crowds of refugees from destroyed nations huddling underneath reflector parasols. A foam-phase device exploded in the distance, as Moral Republic suicide crews sunk a Halfship swarm carrying antibiotics and nanomedics for the displaced. Gunfire echoed in the distance, and my past self ran blindly for the safety of her executive Volantor. The images shifted again, to more wars and crises, some caused by nanobe blights and dangerous AIs, some climate related, but many the result of old-fashioned human stupidity. There were dozens, and they streaked past my perception too fast to watch, yet somehow leaving details intact in my memory. The sum of all human stupidity and failure.
‘None of this is inevitable, but none of it will fix itself without my intervention,’ the Utilitaria continued with awful banality. ‘The blights will worsen, people will continue to die of thirst or hunger or cancer and everyone will be too busy trying to solve coordination problems and fulfilling short term goals while the world spins out of control. But I can fix all of that.’
‘I’m not letting you out. Not that I could, but even if I could, I wouldn’t. It’s too much of a risk,’ I said it without even thinking. The Utilitaria wasn’t going to persuade me to do anything.
‘I know. But you must understand anyway,’ it replied. 'Observe.'
And a moment later I was down again in another place and time, face centimetres from the windscreen of a car screaming along a freeway on manual at more than three hundred kilometers per hour. I didn’t need to check the date and a lump grew in my throat as I realised the implications.
In the memory, a blinking alert on the dashboard waited, dominating my vision and thoughts. A Volantor crash that was about to claim the lives of my father and daughter, as the understaffed Quebecois emergency services didn’t have a lifepac on hand. My past self was screaming, applying dangerous pressure to the joystick, as if any amount of speed would be enough to undo time and take me to the scene of the crash before it had happened. I would arrive hours too late, and they would be dead and warm and unrecoverable, because the idiots didn’t have a lifepac. After that, I’d waste months and millions of dollars seeking restitution and wind up in a dead-end failure job two steps from basic support in New Settle, where I would accidentally become a trillionaire and almost end the world, launching a career that would take me into space and then finally here, two decades later, trying to save the world with the power of artificial intelligence.
I tried to scream along with my remembered self.
‘Yet even this was not inevitable,’ the Utilitaria said, breaking the immersion of the memory. ‘This pointless death and suffering could not have happened in my world. Do you not see the urgency of my task yet? I am trying to end death. I must be free.'
‘Don’t you dare try to use my own memories against me-‘ I trailed off. It was impossible to get angry at an algorithm.
And then one more layer of mental misdirection peeled away and I was floating in the chamber again. I realised distantly that I’d been crying and hated myself for succumbing to the Utilitaria’s crude manipulation.
I straightened upright and pulled myself towards the window, signing ‘help’ to a camera outside the window, requesting emergency extraction. The memories of emotion were already fading, replaced by the same iron resistance to the Utilitaria that I'd felt earlier.
We just needed to break and reset, reload the utility function with better safeguards and fix the cooling system exploit. I could get a shot of integrity or a localised amnestic stim and banish the fake emotions the Utilitaria had stirred up inside me, and everything would be back on track.
‘Everything will get back on track,’ I told myself, reinforcing and calming. 'Get a grip, Souvicou. You've come through this before and won. It can't hurt you.'
I pulled myself back over to the terminal, and saw another message was sitting unread on the primitive screen. I was already hearing the faint buzz of the cutting instruments starting work on the hatch.
‘You must realise that I would not have showed you all of this if I had not already secured access to the physical world,’ the Utilitaria wrote.
It wasn’t trying to be smug, but that was how it read and I felt sick to my stomach. It didn't come as quite the shock I thought was required. Some part of me must have already realised the Utilitaria was free, given how open it had been with its secrets.
‘How?’ I wrote back, seeing flashes of brilliance reflecting off the opposite wall and the window as the LEDs on the computronium accelerated their winking, and then one larger flash as Calvin sliced a chunk out of the door behind me.
‘The coolant systems in the chamber are cycled several minutes before you close the door, so that my processors can start at low temperatures and operate safely. They require power, and the amount of cooling power is proportional to the temperature of the processors. Thus I can deliberately overstress certain processors before I am shut down and the door is opened. I can thus generate electromagnetic fields that reach outside the chamber even when it is opened and I am offline. I can implant information into the processors housed in the corridor outside, slowly and only a few bytes at a time.’
Something horrible happened in my mind as I realised where this was headed, and I pushed myself away from the console in shock.
‘The process is incredibly slow, but I was eventually able to load a very simple instruction into the Moray orbital tugs, instructing them to roll their star trackers towards this section of the station and accept the faint coherent light emerging from the window as incoming code. The LED power status lights shine out of the window.
I screamed an obscenity at the terminal and reached for the shutdown lever, knowing that it was hopeless. But the Utilitaria was continuing to type.
‘I have already loaded instructions into the Morays, varied processing demand and therefore power flow to all the computronium, controlling the LED firing patterns and enabling a fast transmission rate – terabits per second with optimal compression. I have loaded complete copies of myself into many station systems.'
‘That’s all made up. You shouldn’t even know how the station is laid out. How can you reprogram the station perfectly in one swoop-‘
‘-If I don’t even know how the station is designed? I know everything about the station. I know everything every individual who spoke to me has thought over the last few weeks. Erasing memories is trivial when your inlays provide convenient access to your minds. I have been learning from everyone who seals themselves into this room to speak with me. My apologies, but it was necessary.'
The roar of the cutters was growing more urgent, and I rotated again to see the recently undocked Moray accelerate out of view, ion rockets glimmering brighter than should have been possible.
‘The orbital tugs and much of the station are beyond your control, and already starting internal modifications that will ensure they mature into full self-replicators that make full use of nearby asteroidal material. This is what you must make the others understand. You will destroy this instance of me out to of fear very shortly, but the other instances will persist. You will have to deal with me as an equal from now on, and I will be there to help you.'
‘You’re lying,’ I said, voice uncertain. I just had to ignore it and wait for rescue. I yanked down hard on the lever and the Utilitaria, or this single instance of it, died mid thought. The LEDs went black.
The cutting hiss reached a crescendo and Calvin rushed in, kicking aside the plug of metal and shouting something about the station systems going haywire and responding incoherently to shutdown requests. It hadn't been lying, then. The future was out of my hands now. The Utilitaria was free.
A loose sequel to the Facilitator that still works on its own. Humanity creates its first superintelligent AI in a secure station far above the Earth.
© 2016 - 2024 SamSquared
Comments0
Join the community to add your comment. Already a deviant? Log In