From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Mon Dec 23 2002 - 18:04:02 MST
I'm not going to write my own thoughts on this subject. Talking about
post-Singularity life tends to devour people's souls and I was always
taught that was impolite. Yes, sometimes I dream. But I know I'm
dreaming. Some people seem under the impression they're awake.
What is it that you want?
What is it that you are trying to accomplish with all these technological
tricks and toys?
Whatever it is you're really wishing for, you can be sure that the kind of
gadgetry dreamed up by human science fiction writers - even the very best
science fiction writers like Greg Egan - is not the best way to go about
it. We notice our technology because it's clumsy. Because the technology
is so awkward that it is visible, and has visible side effects, and
imposes visible constraints on what can be accomplished by it.
Nanotechnology and utility fog may be able to build a flying carpet;
current technology leaves us with airplanes; an upload just is where they
want to be. To a medieval scholar the only understandable difference
between nanotechnology and femtotechnology is that one can turn lead into
gold, and the other can't; and there's no distinction at all between
femtotechnology and chromotechnology because the medieval scholar doesn't
know enough to invent a task that chromotechnology can perform but
femtotechnology can't. Sufficiently advanced technology is *very*
distinguishable from magic. Magic is just as awkward as insufficiently
advanced technology.
And so we have the odd concept of cyborgs. Of neural links to computer
systems. Of humanoid robot bodies.
The future will not be science fiction because it will have no buttons to
press.
And faced with that possibility, people tend to panic, and fall back on
comforting habits learned from the cozy books of childhood. It can be
dizzying to confront the question "But what, exactly, do you actually
want?" once the crutch of wishing for gadgetry is stripped away.
But let's ask anyway. Forget about the toys. What is it that you want?
Forget about the technology. What are the patterns in reality that you
are trying to alter?
The pattern of yourself.
You are not your body. But you can't see the inside of your mind. You
can't see the kind of improvements that are really worthwhile. You can't
imagine changing yourself, so you imagine changing your body instead.
Perhaps the first thing that changes will be your ability to understand
change, and the changes you want to make.
The SF plots that have been posted to this thread don't look to me like
changes at all, just running in the same place, because they don't alter
any of the things I have learned to see as important. It seems like
displacement activities for deeper wishes that are harder to verbalize.
Has anyone here said "I wish to be happy", or "I wish that it didn't take
so much effort just to exist", or "I wish I understood reality and my
place in it", in among the dreams of cloning and robot bodies?
I don't know if what matters is the journey or the destination. I don't
know if the philosophy of the trip is to stop and smell the flowers, or
race for the light as fast as possible. In the first case, I might have
some small idea of who I'll be immediately afterward, though I consider it
unhealthy to spend too much time thinking about it. In the second case, I
think little or nothing can be said at all. To spend too much time
thinking about slow paths is to forget that you don't know whether the
path will be fast or slow.
People contemplating the slow path lose their objectivity and begin
spinning fantasy scenarios in which the entire course of technological
progress ends up stretched over a thousand years or whatever. If humans
end up walking down the slow path, and smelling the flowers, it will be
because some Friendly AI blazed a trail all the way to the end, just to
make sure that there are no dangers that slowly blossoming humans wouldn't
see. No gamma-ray bursts, exploding galaxies, decaying laws of physics,
hostile superintelligences, what-have-you. Even if the greatest amount of
fun comes from stopping to smell the flowers, there has to be at least one
sentience who sacrifices leisure and burns for the light as fast as
possible. Otherwise we might end up prey for anything that we weren't
smart enough to know about. If we end up doing this the slow way, it'll
be because whichever sentience blazed the trail (a Friendly seed AI or its
heir, I expect) comes back and says: "It's safe, and desirable, to stop
and smell the flowers."
To summarize the parts of this that I consider it polite to talk about:
1) We don't know whether it's the journey or the destination;
2) Even if it is the journey that counts, and not the destination, I
expect at least one sentient mind will blaze a trail for the rest;
3) And as a result of that superintelligence being around, anyone making
a slow journey will do so without any visible gadgetry, because only
clumsy technology is visible.
So if you're thinking that what you want involves chrome and steel, lasers
and shiny buttons to press, neural interfaces, nanotechnology, or whatever
great groaning steam engine has a place in your heart, you need to stop
writing a science fiction novel with yourself as the main character, and
ask yourself who you want to be.
-- Eliezer S. Yudkowsky http://intelligence.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:41 MDT