“Letter from Utopia” and Other Triple-S Transhumanist Media

by Nick Bostrom (2010)

Dear Human,

Greetings, and may this letter find you at peace and in prosperity! Forgive
my writing to you out of the blue. Though you and I have never met, we are
not strangers. We are, in a certain sense, the closest of kin. I am one of your
possible futures.

I hope you will become me. Should fortune grant this wish, then I am not
just a possible future of yours, but your actual future: a coming phase of you,
like the full moon that follows a waxing crescent, or like the flower that
follows a seed.

I am writing to tell you about my life – how good it is – that you may choose
it for yourself.

Although this letter uses the singular, I am really writing on behalf of all
my contemporaries, and we are addressing ourselves to all of your
contemporaries. Amongst us are many who are possible futures of your
people. Some of us are possible futures of children you have not yet given
birth to. Still others are possible artificial persons that you might one day
create. What unites us is that we are all dependent on you to make us real.
You can think of this note as if it were an invitation to a ball that will take
place only if folks turn up.

We call the lives we lead here “Utopia”.

*

How can I tell you about Utopia and not leave you mystified? What words
could convey the wonder? What inflections express our happiness? What
points overcome your skepticism? My pen, I fear, is as unequal to the task as
if I had tried to use it against a charging elephant.

But the matter is so important that we must try even against long odds.
Maybe you will see through the inadequacies of my exposition.

Have you ever known a moment of bliss? On the rapids of inspiration,
maybe, where your hands were guided by a greater force to trace the shapes
of truth and beauty? Or perhaps you found such a moment in the ecstasy of
love? Or in a glorious success achieved with good friends? Or in splendid
conversation on a vine-overhung terrace one star-appointed night? Or
perhaps there was a song or a melody that smuggled itself into your heart,
setting it alight with kaleidoscopic emotion? Or during worship?

If you have experienced such a moment, experienced the best type of such a
moment, then a certain idle but sincere thought may have presented itself to
you: “Oh Heaven! I didn’t realize it could feel like this. This is on a
different level, so very much more real and worthwhile. Why can’t it be like
this always? Why must good times end? I was sleeping; now I am awake.”

Yet behold, only a little later, scarcely an hour gone by, and the softly-falling
soot of ordinary life is already piling up. The silver and gold of exuberance
lose their shine. The marble becomes dirty.

Every way you turn it’s the same: soot, casting its veil over all glamours and
revelries, despoiling your epiphany, sodding up your white pressed collar and
shirt. And once again that familiar beat is audible, the beat of numbing
routine rolling along its tracks. The commuter trains loading and unloading
their passengers… sleepwalkers, shoppers, solicitors, the ambitious and the
hopeless, the contented and the wretched… like human electrons shuffling
through the circuitry of civilization.

We do so easily forget how good life can be at its best – and how bad at its
worst. The most outstanding occasion: barely is it over before the sweepers
move in to clean up the rice. Yellowing photos remain.

And this is as it should be. We are in the business of living, and life must go
on. Special moments are out-of-equilibrium experiences in which our
puddles are stirred up and splashed about; yet when normalcy returns we are
usually relieved. We are built for mundane functionality, not lasting bliss.

So the door that was ajar begins to close, disappearing hope’s sliver behind
an insensate scab.

Quick, stop that door! Look again at your yellowing photos, search for a
clue. Do you not see it? Do you not feel it, the touch of the possible? You
have witnessed the potential for a higher life, and you hold the fading proof
in your hands. Don’t throw it away. In the attic of your mind, reserve a
drawer for the notion of a higher state of being. In the furnace of your heart,
keep an aspiring ember alive.

I am summoning this memory of your best experience – to what end? In the
hope of kindling in you a desire to share my happiness.

And yet, what you had in your best moment is not close to what I have now
– a beckoning scintilla at most. If the distance between base and apex for
you is eight kilometers, then to reach my dwellings requires a million light-year
ascent. The altitude is outside moon and planets and all the stars your
eyes can see. Beyond dreams. Beyond imagination.

My consciousness is wide and deep, my life long. I have read all your
authors – and much more. I have experienced life in many forms and from
many angles: jungle and desert, gutter and palace, heath and suburban creek
and city back alley. I have sailed the high seas of cultures, and swum, and
dived. Quite some marvelous edifice builds up over a million years by the
efforts of homunculi, just as the humble polyps amass a reef in time. And
I’ve seen the shoals of colored biography fishes, each one a life story,
scintillate under heaving ocean waters.

The whole exceeds the sum of its parts. What I have is not merely more of
what is available to you now. It isn’t just the particular things, the paintings
and toothpaste-tube designs, the record covers and books, the epochs, lives,
leaves, rivers, and random encounters, the satellite images and the hadron
collider data – it is also the complex relationships between these particulars
that make up my mind. There are ideas that can be formed only on top of
such a wide experience base. There are depths that can be fathomed only
with such ideas.

You could say I am happy, that I feel good. You could say that I feel
surpassing bliss. But these are words invented to describe human
experience. What I feel is as far beyond human feeling as my thoughts are
beyond human thought. I wish I could show you what I have in mind. If
only I could share one second of my conscious life with you!

But you don’t have to understand what I think and feel. If only you bear in
mind what is possible within the present human realm, you will have enough
to get started in the right direction, one step at a time. At no point will you
encounter a wall of blinding light. At no point will you have to jettison
yourself over an end-of-the-world precipice. As you advance, the horizon
will recede. The transformation is profound, but it can be as gradual as the
growth that made the baby you were into the adult you think you are.

You will not achieve this through any magic trick or hokum, nor by the
power of wishful thinking, nor by semantic acrobatics, meditation,
affirmation, or incantation. And I do not presume to advise you on matters
theological. I urge on you nothing more, nothing less, than reconfigured
physical situation.

*

The challenge before you: to become fully what you are now only in hope
and potential. New capacities are needed if you wish to live and play on my
level.

To reach Utopia, you must first discover the means to three fundamental
transformations.

The First Transformation: Secure life!

Your body is a deathtrap. This vital machine and mortal vehicle, unless it
jams first or crashes, is sure to rust anon. You are lucky to get seven decades
of mobility; eight if you be Fortuna’s darling. That is not sufficient to get
started in a serious way, much less to complete the journey. Maturity of the
soul takes longer. Why, even a tree-life takes longer!

Death is not one but a multitude of assassins. Do you not see them? They
are coming at you from every angle. Take aim at the causes of early death –
infection, violence, malnutrition, heart attack, cancer. Turn your biggest
gun on aging, and fire. You must seize control of the biochemical processes
in your body in order to vanquish, by and by, illness and senescence. In
time, you will discover ways to move your mind to more durable media.
Then continue to improve the system, so that the risk of death and disease
continues to decline. Any death prior to the heat death of the universe is
premature if your life is good.

Oh, it is not well to live in a self-combusting paper hut! Keep the flames at
bay and be prepared with liquid nitrogen, while you construct yourself a
better habitation. One day you or your children should have a secure home.
Research, build, redouble your effort!

The Second Transformation: Upgrade cognition!

Your brain’s special faculties: music, humor, spirituality, mathematics,
eroticism, art, nurturing, narration, gossip! These are fine spirits to pour
into the cup of life. Blessed you are if you have a vintage bottle of any of
these. Better yet, a cask! Better yet, a vineyard!

Be not afraid to grow. The mind’s cellars have no ceilings!

What other capacities are possible? Imagine a world with all the music dried
up: what poverty, what loss. Give your thanks, not to the lyre, but to your
ears for the music. And ask yourself, what other harmonies are there in the
air, that you lack the ears to hear? What vaults of value are you witlessly
debarred from, lacking the key sensibility?

Had you but an inkling, your nails would be clawing at the padlock in sacred
frenzy.

Your brain must grow beyond the bounds of any genius of humankind, in its
special faculties as well as its general intelligence, so that you may better
learn, remember, and understand, and so that you may apprehend your own
beatitude.

Mind is a means: for without insight you will get bogged down or lose your
way, and your journey will fail.

Mind is also an end: for it is in the spacetime of awareness that Utopia will
exist. May the measure of your mind be vast and expanding.

Oh, stupidity is a loathsome corral! Gnaw and tug at the posts, and you will
slowly loosen them up. One day you’ll break the fence that held your
forebears captive. Gnaw and tug, redouble your effort!

The Third Transformation: Elevate well-being!

What is the difference between indifference and interest, boredom and thrill,
despair and bliss?

Pleasure! A few grains of this magic ingredient are dearer than a king’s
treasure, and we have it aplenty here in Utopia. It pervades into everything
we do and everything we experience. We sprinkle it in our tea.

The universe is cold. Fun is the fire that melts the blocks of hardship and
creates a bubbling celebration of life.

It is the birth right of every creature, a right no less sacred for having been
trampled upon since the beginning of time.

There is a beauty and joy here that you cannot fathom. It feels so good that
if the sensation were translated into tears of gratitude, rivers would overflow.

I reach in vain for words to convey to you what it all amounts to… It’s like a
rain of the most wonderful feeling, where every raindrop has its own unique
and indescribable meaning – or rather a scent or essence that evokes a whole
world… And each such evoked world is subtler, richer, deeper, more
palpable than the totality of what you have experienced in your entire life.

I will not speak of the worst pain and misery that is to be got rid of; it is too
horrible to dwell upon, and you are already aware of the urgency of
palliation. My point is that in addition to the removal of the negative, there
is also an upside imperative: to enable the full flourishing of enjoyments that
are currently out of reach.

The roots of suffering are planted deep in your brain. Weeding them out
and replacing them with nutritious crops of well-being will require advanced
skills and instruments for the cultivation of your neuronal soil. But take
heed, the problem is multiplex! All emotions have a natural function. Prune
carefully lest you reduce the fertility of your plot.

Sustainable yields are possible. Yet fools will build fools’ paradises. I
recommend you go easy on your paradise-engineering until you have the
wisdom to do it right.

Oh, what a gruesome knot suffering is! Pull and tug on those loops, and you
will gradually loosen them up. One day the coils will fall, and you will
stretch out in delight. Pull and tug, and be patient in your effort!

May there come a time when rising suns are greeted with joy by all the living
creatures they shine upon.

*

How do you find this place? How long will it take to get here?

I can pass you no blueprint for Utopia, no timetable, no roadmap. All I can
give you is my assurance that there is something here, the potential for a
better life.

If you could visit me here for but a day, you would henceforth call this place
your home. This is the place where you belong. Ever since one hairy
creature picked up two flints and began knocking them together to make a
tool, this has been the direction of your unknown aspiration. Like Odysseus
you must journey, and never cease journeying, until you arrive upon this
shore.

“Arrive?” you say; “But isn’t the journey the destination? Isn’t Utopia a
place that doesn’t exist? And isn’t the quest for Utopia, as witnessed
historically, a dangerous folly and an incitement to mischief?”

Friend, that is not such a bad way for you to think about it. To be sure,
Utopia is not a location or a form of social organization.

The blush of health on a convalescent’s cheek. The twinkling of the eye in a
moment of wit. The smile of a loving thought… Utopia is the hope that the
scattered fragments of good that we come across from time to time in our
lives can be put together, one day, to reveal the shape of a new kind of life.
The kind of life that yours should have been.

I fear that the pursuit of Utopia will bring out the worst in you. Many a
moth has been incinerated in its pursuit of a brighter future.

Seek the light! But approach with care – swerve if you smell your wingtips
singeing. Light is for seeing, not dying.

When you embark on this quest, you will encounter rough seas and hard
problems. To prevail will take your best science, your best technology, and
your best politics. Yet each problem has a solution. My existence breaks no
law of nature. The materials are all there. Your people must become
master builders, and then you must use these skills to build yourselves up
without crushing your cores.

*

What is Suffering in Utopia? Suffering is the salt trace left on the cheeks of those
who were around before.

What is Tragedy in Utopia? There is tragedy in Snowman’s melting. Mass
murders are not required.

What is Imperfection in Utopia? Imperfection is the measure of our respect for
things as they are and for their history.

What is Body in Utopia? Body is a pair of legs, a pair of arms, a trunk and a
head, all made of flesh. Or not, as the case may be.

What is Society in Utopia? Society is a never-finished tapestry, its weavers equal
to its threads – the parts and patterns an inexhaustible bourne of beauty.

What is Death in Utopia? Death is the darkness that ultimately surrounds all
life.

What is Guilt in Utopia? Guilt is our knowledge that we could have created
Utopia sooner.

*

We love life here every instant. Every second is so good that it would blow
our minds had their amperage not been previously increased. My
contemporaries and I bear witness, and we request your aid. Please, help us
come into existence! Please, join us! Whether this tremendous possibility
becomes reality depends on your actions. If your empathy can perceive at
least the outlines of the vision I am describing, then your ingenuity will find a
way to make it real.

Human life, at its best, is fantastic. I’m asking you to create something even
greater. Life that is truly humane.

Yours sincerely,
Your Possible Future Self


See also a musicalized video rendition of this piece by Mario Montano: Letter From Utopia


Analysis

Nick Bostrom is a prominent transhumanist philosopher and academic who works at Oxford’s Future of Humanity Institute. An incredibly prolific writer, Nick has a very wide and comprehensive worldview. I find his work extremely valuable and worth diving into. Letter From Utopia is one of my favorite works of his, as it encompasses what David Pearce called “The Three Supers of Transhumanism“: Super-Intelligence, Super-Longevity, and Super-Happiness (cf. Triple-S Genetic Counseling). Bostrom also has other amazing stories and essays (such as The Fable of the Dragon-Tyrant, cf. story video rendition by CGP Grey: video), but Letter From Utopia takes the cake for not leaving behind anything of crucial importance.

Alas, despite Bostrom’s far-reaching contributions, many argue that Nick’s most important impact has been in the field of AI Alignment (cf. Superintelligence: Paths, Dangers, Strategies). In 2009 AI safety research was perceived to be a fringe concern of sci-fi aficionados and weirdos. Today, partly thanks to Bostrom (along with Yudkowsky, Chalmers, and others), many top journals publish serious work in this area.

I worry that this is not as good as it may seem. Nick Bostrom’s name is first and foremost associated with AI safety, followed by the Simulation Argument and Existential Risk, and only later by his extensive work on all other areas of transhumanism. For example, if you search Youtube for his name, you will see that of the top 20 results, 15 concern AI safety/digital superintelligence. Of the remaining 5, 3 are about the Simulation Argument, 1 is about agnosticism, and 1 is CGP Grey’s Dragon-Tyrant video. Where are the Triple-S videos?

nick_bostrom_top_results

I remembered that I encountered the work of both David and Nick when I was 16, googling terms like consciousness, AI, psychedelics, and far future. I was drawn to both of them, though I particularly liked David’s focus on ending suffering as a priority and his acknowledgment of the scientific significance of altered states of consciousness. I thought that their work was complementary rather than redundant. Alas, Bostrom is far more well known than Pearce, perhaps due to his success as both a fringe philosopher and a mainstream academic. In contrast, David dropped out of Oxford out of frustration with the academic community; the analytic philosophy of the time was not empirical, and it focused on language use rather than real philosophical questions, including the nature of suffering, psychedelics, and physical causality (e.g. “Philosophy may in no way interfere with the actual use of language, it can in the end only describe it. For it cannot give it any foundation either. It leaves everything as it is. It also leaves mathematics as it is, and no mathematical discovery can advance it.” – Wittgenstein). Bostrom, unlike Pearce, has the blessing of Ra, the God of optionality, superlativity, status legibility, and groundless prestige. And yet, it was David’s conversation with Nick that gave rise to the creation of the World Transhumanist Association, and provided one of the most important memetic Schelling points of the early 2000s. Alas, David is not focused on AI Safety. Why?

People in the transhumanist community accuse David of not getting it. David, after all, is not a mathematician, computer scientist, or physicist; he is merely a philosopher. I must confess that the very first time I met David Pearce in person I got the sense that (1) he was an incredibly well-read and creative genius in most areas of philosophy, and yet (2) naïve and unenlightened in the field of AI. As a fan of his work, and having co-founded the Stanford Transhumanist Association a couple of months earlier, I thought to invite him to give a talk at Stanford (see: David Pearce at Stanford – 2011).

325714_184732704952228_485224132_o

David Pearce and the officers of the Stanford Transhumanist Association (December 1st 2011) at Palo Alto’s Chinese Vegan restaurant Garden Fresh, before David’s talk.

We had a lively conversation while eating dinner at a Chinese vegan restaurant before the event along with other members of the Association. I recall that he checked all of the right boxes when it came to personal identity (Open/Empty Individualism), ethics (consequentialism), physics (Everettian multiverse), psychedelia (they disclose new varieties of qualia), evolution (modern synthesis; selfish gene), social signaling theory (Mating Mind and sexual selection theory), and more (see his Reddit AMA). And yet, how could he dare to say that a digital computer would never be conscious? Meeting a brilliant thinker who had a better grasp of my favorite topics than I did and yet would try to hit on one of my core load-bearing beliefs was uncomfortable and unexpected. I dismissed his take on AI as that of a fuzzy thinker (at least in this area); I reassured myself by recalling that it was me who was studying AI academically at a top US institution and not him. Little did I know that over the next few years, and after hanging out with him in person for over 20 cumulative hours, he would finally change my mind- and worldview- concerning this whole field. If it wasn’t for him, I suspect I would have jumped on the bandwagon of AI-as-the-top-priority (cf. Altruists Should Prioritize Artificial Intelligence). Thankfully, I was already extremely interested in consciousness and didn’t have it in me to dismiss it. Additionally, my interest in personal identity reduced my (relative) interest in longevity research (at least as the top priority), for if we are all one consciousness, dying is more akin to forgetting a timeline than a true ontological death. The instrumental value of intelligence, however, ought not to be taken for granted, which is why I now advocate for a twin approach of improving subjective wellbeing while retaining critical insight. Figuring out that consciousness required more than digital computation utterly transformed my approach to transhumanism, and I largely credit this change to my conversations with David.

This slideshow requires JavaScript.

Later on I met Mike Johnson, Romeo Stevens, and a number of other top thinkers in the field of consciousness who could both understand the genuine problems consciousness poses and at the same time grasp the broader transhumanist meme-plex, transcend it, and include it (cf. Why I think the Foundational Research Institute should rethink its approach). Thus we founded the Qualia Research Institute, in order to bring a new full-stack meme-plex where consciousness – and valence – are front and center. Alas, we have experienced some resistance…

AI safety is sexy. If you are a smart, industrious, open-minded, and systematizing undergraduate, studying AI gives you access to a wide circle of really fun people to hang out in. It also signals intelligence, sober-mindedness, and stoicism. It gives you both an in into smart cool kid circles, and a profitable career in Silicon Valley. It allows you to straddle the world of normies and cutting-edge thinkers.

But, crucially, you have to consider the opportunity cost that comes from directing such a large fraction of hyper-intelligent young altruistic systematizers to this problem. The field is plagued with misconceptions about pleasure and value; Eliezer Yudkowsky’s Fun Theory suffers from the severe delusion that value comes from the intentional object of experience, rather than from its phenomenal character. Brian Tomasik’s (admittedly tongue-in-cheek) People for the Ethical Treatment of Reinforcement Learners is seemingly unaware of the fact that neuroscience has found that pleasure/suffering and reinforcement learning are doubly dissociated. Pleasure is not reinforcement, and until you grasp this, your ethical models will output nonsense.

Tongue-in-cheek, perhaps AI risk is a real threat. Not because of the usual reasons, but because it siphons out top brain power into a relatively sterile field, leaving vast amounts of unclaimed marginal value in the fields of rejuvenation research and valence technologies by the wayside.

In light of all of this, I would want to advocate for the reinvigoration of the broader transhumanist meme-plex, now updated with a post-Galilean understanding of consciousness. Writers, animators, Youtubers, and philosophers ought to collaborate in creating more balanced Triple-S Transhumanist outreach in the form of widely consumable media. This, I think, is the path forward.

Would Maximally Efficient Work Be Fun?

Excerpt from Superintelligence: Paths, Dangers, Strategies (2014) by Nick Bostrom (pg. 207-210).

Would Maximally Efficient Work Be Fun?

One important variable in assessing the desirability of a hypothetical condition like this* is the hedonic state of the average emulation**. Would a typical emulation worker be suffering or would he be enjoying the experience of working hard on the task at hand?

We must resist the temptation to project our own sentiments onto the imaginary emulation worker. The question is not whether you would feel happy if you had to work constantly and never again spend time with your loved ones–a terrible fate, most would agree.

It is moderately more relevant to consider the current human average hedonic experience during working hours. Worldwide studies asking respondents how happy they are find that most rate themselves as “quite happy” or “very happy” (averaging 3.1 on a scale from 1 to 4)***. Studies on average affect, asking respondents how frequently they have recently experienced various positive or negative affective states, tend to get a similar result (producing a net affect of about 0.52 on a scale from -1 to 1). There is a modest positive effect of a country’s per capita income on average subjective well-being.**** However, it is hazardous to extrapolate from these findings to the hedonic state of future emulation workers. One reason that could be given for this is that their condition would be so different: on the one hand, they might be working much harder; on the other hand, they might be free from diseases, aches, hunger, noxious odors, and so forth. Yet such considerations largely miss the mark. The much more important consideration here is that hedonic tone would be easy to adjust through the digital equivalent of drugs or neurosurgery. This means that it would be a mistake to infer the hedonic state of future emulations from the external conditions of their lives by imagining how we ourselves and other people like us would feel in those circumstances. Hedonic state would be a matter of choice. In the model we are currently considering, the choice would be made by capital-owners seeking to maximize returns on their investment in emulation-workers. Consequently, this question of how happy emulations would feel boils down to the question of which hedonic states would be most productive (in the various jobs that emulations would be employed to do). [Emphasis mine]

Here, again, one might seek to draw an inference from observations about human happiness. If it is the case, across most times, places, and occupations, that people are typically at least moderately happy, this would create some presumption in favor of the same holding in a post-transition scenario like the one we are considering. To be clear, the argument in this case would not be that human minds have a predisposition towards happiness so they would probably find satisfaction under these novel conditions; but rather that a certain average level of happiness has proved adaptive for human minds in the past so maybe a similar level of happiness will prove adaptive from human-like minds in the future. Yet this formulations also reveals the weakness of the inference: to wit, that the mental dispositions that were adaptive for hunter-gatherer hominids roaming the African savanna may not necessarily be adaptive for modified emulations living in post-transition virtual realities. We can certainly hope that the future emulation-workers would be as happy as, or happier than, typical workers were in human history; but we have yet to see any compelling reason for supposing it would be so (in the laissez-faire multipolar scenario currently under examination).

Consider the possibility that the reason happiness is prevalent among humans (to whatever limited extent it is prevalent) is that cheerful mood served a signaling function in the environment of evolutionary adaptedness. Conveying the impression to other members of the social group of being in flourishing condition–in good health, in good standing with one’s peers, and in confident expectation of continued good fortune–may have boosted an individual’s popularity. A bias toward cheerfulness could thus have been selected for, with the result that human neurochemistry is now biased toward positive affect compared to what would have been maximally efficient according to simpler materialistic criteria. If this were the case, then the future of joie de vivre might depend on cheer retaining its social signaling function unaltered in the post-transition world: an issue to which we will return shortly. 

What if glad souls dissipate more energy than glum ones? Perhaps the joyful are more prone to creative leaps and flights of fancy–behaviors that future employers might disprize in most of their workers. Perhaps a sullen or anxious fixation on simply getting on with the job without making mistakes will be the productivity-maximizing attitude in most lines of work. The claim here is not that this is so, but that we do not know that it is not so. Yet we should consider just how bad it could be if some such pessimistic hypothesis about a future Malthusian state turned out to be true: not only because of the opportunity cost of having failed to create something better–which would be enormous–but also because the state could be bad in itself, possibly far worse that the original Malthusian state.

We seldom put forth full effort. When we do, it is sometimes painful. Imagine running on a treadmill at a steep incline–heart pounding, muscles aching, lungs gasping for air. A glance at the timer: your next break, which will will also be your death, is due in 49 years, 3 months, 20 days, 4 hours, 56 minutes, and 12 seconds. You wish you had not been born.

Again the claim is not that this is how it would be, but that we do not know that it is not. One could certainly make a more optimistic case. For example, there is no obvious reason that emulations would need to suffer bodily injury and sickness: the elimination of physical wretchedness would be a great improvement over the present state of affairs. Furthermore, since such stuff as virtual reality is made of can be fairly cheap, emulations may work in sumptuous surroundings–in splendid mountaintop palaces, on terraces set in a budding spring forest, or on the beaches of azure lagoon–with just the right illumination, temperature, scenery and décor; free from annoying fumes, noises, drafts, and buzzing insects; dressed in comfortable clothing, feeling clean and focused, and well nourished. More significantly, if–as seems perfectly possible–the optimum human mental state for productivity in most jobs is one of joyful eagerness, then the era of the emulation economy could be quite paradisiacal.

There would, in any case, be a great option value in arranging matters in such a manner that somebody or something could intervene to set things right if the default trajectory should happen to veer toward dystopia. It could also be desirable to have some sort of escape hatch that would permit bailout into death and oblivion if the quality of life were to sink permanently below the level at which annihilation becomes preferable to continued existence.

Unconscious outsourcers?

In the long run, as the emulation era gives way to an artificial intelligence era (or if machine intelligence is attained directly via AI without a preceding whole brain emulation stage) pain and pleasure might possibility disappear entirely in a multipolar outcome, since a hedonic reward mechanism may not be the most effective motivation system for a complex artificial agent (one that, unlike the human mind, is not burdened with the legacy of animal wetware). Perhaps a more advanced motivation system would be based on an explicit representation of a utility function or some other architecture that has not exact functional analogs to pleasure and pain.

A related but slightly more radical multipolar outcome–one that could involve the elimination of almost all value from the future–is that the universal proletariat would not even be conscious. This possibility is most salient with respect to AI, which might be structured very differently than human intelligence. But even if machine intelligence were initially achieved through whole brain emulation, resulting in conscious digital minds, the competitive forces unleashed in a post-transition economy could easily lead to the emergence of progressively less neuromorphic forms of machine intelligence, either because synthetic AI is created de novo or because the emulations would, through successive modifications and enhancements, increasingly depart their original human form.


* Scenarios where sentient emulations are being used to do maximally efficient work.

** Footnote: “An ethical evaluation might take into account many other factors as well. Even if all the workers were constantly well pleased with their condition, the outcome might still be deeply morally objectionable on other grounds–though which other grounds is a matter of dispute between rival moral theories. But any plausible assessment would consider subjective well-being to be one important factor. See also Bostrom and Yudkowsky (2015).”

*** World Values Survey (2008).

**** Helliwell and Sachs (2012).