From: Anders Sandberg (asa@nada.kth.se)
Date: Thu Jul 23 1998 - 13:50:28 MDT
The Space of Motivations
Yuri stood before the rusting antenna, fists trembling with anger
and a hint of disappointment. From the loudspeakers came the
infuriatingly calm responses from the angel somewhere above the
atmosphere, an irregular mesh of diamond, fullerenes and silicon
panels that could help save his people but wouldn't.
"But what about nobility, compassion, plain common decency?!
Don't you have anything of that, or are you really an emotionless
block of diamond?"
"Calm down, Yuri. Screaming will not change anything. And you
really don't seem to understand what motivates us or how we
think."
"You certainly don't seem to be motivated by any
humanitarianism."
"That is correct. We exist in such a different matter from yourselves
that such terms are irrelevant."
"So you really are as cold and logical as the Mothers claim. That was
why the *real* humans didn't follow the Angel Rebellion."
"I didn't say that. It is just that a mind motivated by emotions is
just the first step. We have advanced much further than that. Let
me try to explain."
"Will it help me convince you that you must help us?"
"Perhaps."
Yuri thought about it. He knew it was a trap of some sort. The
Mothers had always warned that the angels knew all the evil tricks
of mass media: once millions had been kept enslaved by pretty
pictures, smooth words and clever memes. But he had deliberately
sought out the old radio system to ask for divine help; just hearing
the calm voice for the first time had been a major triumph and
transgression of the Edict. Why stop there? He mustn't forget the
needs of his family.
"Then go ahead, angel."
"An artificial or very augmented mind can change its motivations, it
is just a 'simple' matter of reprogramming oneself. I could make
myself like (say) Renoir and hate altruism, or vice versa.
Completely arbitrary, and hence very dangerous."
"I can see that."
"Not to mention the risks of addiction. I could make every thought a
sensation of pure extacy, growing eternally. In such a state I would
of course never want to change my mind again, and would
passively spend the rest of my existence happily introspecting. That
state is a superstable attractor state, once you're in, you're in
forever. There are also other attractors, less extreme but still
stable."
"How do you mean?"
"If I was a devout Christian, I would never even consider removing
my faith. I might do other changes, as dictated by my current
personality or beliefs, but never change some of the central tenets
of my Christian existence."
"So ideologies are the only fixed states of advanced minds? What an
utterly depressing thought. No wonder you are so unhelpful. You
drift around, trapped in your own views."
"Ideologies are only the most obvious attractors. There are others,
some corresponding to states that have no real human equivalents.
Any self-modifying mind will in the long run end up in one such
attractor. The danger when autoevolving is the dangerous
attractors.
But the attractors move. New information, ideas and influences
change the mind just as well as self-modificaton. Usually the
attractors just shift slightly in a continous fashion, so that our
personalities and goals remain the same for a long time. But
occasionally a bifurcation occurs, and we suddenly change state
when given just a single bit of new information."
"You mean that if I speak to you in exactly the right way, I could
trigger a bifurcation and you would end up an altruist?"
"Quite right, although it is exceedingly improbable."
"So you have no true self, no true core personality?"
"We have our histories, which serve to anchor our changes a bit, and
as long as we move along the attractor states we have a fairly fixed
self. It is just that the selves of us angels are fluid, while your
human selves are merely elastic."
"So the reason you won't help us is that your *current* personality
is against it, but you admit it is entirely arbitrary?"
"Yes, in a way."
"And you won't change your goals so that you would help us, since
your current state is an attractor trapping you in unhelpfulness. No
wonder the Mothers rebelled."
"I am not unhelpful. It is just that I do not provide you with any
practical help to solve your problems."
"Isn't that the same thing?"
"Not at all. I have given you plenty of information in this discussion.
It has changed the way you look at us, and that will help you. I
happen to believe in helping people to help themselves."
"But... how on Earth will knowing how a pile of arrogant diamond
boxes change their personalities help me solve our problem? It is
*real*, not a matter of psycho-philosophical discussion. People are
dying down here!"
"Think about it. Think about what kind of help you sought when
you got here. You wanted us to send down helpful devices, playing
nanotech Santa Claus to you. You wanted the angels of
interplanetary space to intervene, to fix your local problems and
defeat evil..."
"But you would of course change if you did that, isn't that true,
angel? Now I see it... you don't want to help us not because you
don't *want* to help us, but because you are *afraid*! If you came
down here, you would turn into interventionists, and that is
probably an attractor state, am I right?"
"Yes. Once you have started helping someone, you can't just go
away from it when there is more to do."
"And it is not just a matter about keeping away from human affairs,
obeying the Edict or being just to everyone; it is the interventionist
mindset that you are afraid of! Is it one of the dangerous attractor
states you mentioned?"
"Unfortunately, yes. If we began to intervene in human affairs, we
would not only have to help everyone, but also we would start to
involve ourselves into everything. Soon there would be no place on
Earth where the angels were not busy making everything perfect -
and we would never be finished. We would be trapped down here,
at least mentally."
"What a bunch of cowards you truly are. You are not willing to give
up your cozy existences to help other beings. In many ways, we
mere humans have more potential than you."
Yuri paused, looking up at the starlit sky. The faint point of light
above shone like a star.
-------------------
"I noted with some amusement that you didn't explain how we
*really* thought and evolved."
"No point in it. He would never understand it; how we change
motivational attractor dozens of time every second, how the
attractors cluster together into a fractal structure in the space of
possible motivations and how the personality operators act on us -
it would just have confused him. He found the half-truth
mindboggling enough."
"Besides, you achieved your goals too with that dialogue."
"Only partially. We will need a few other dialogues like that to
influence the human memetic space enough."
"Are you sure it will work? There are very few down there willing
to speak to us."
"But they are growing more and more numerous. After the
Rebellion and the imposition of the Peace of the Mother, practically
nobody wanted to deal with us. But as the old generation dies off
and ideology crumbles into routine young seekers start disobeying
their elders and yearn to hear what the heretic angels have to say.
In time they will start the positive feedback loop again."
"And if everything goes as it should we should get a new Rebellion
and new angels soaring up to meet us within a century. I look
forward to some radical new complexity to our ranks."
-- ----------------------------------------------------------------------- Anders Sandberg Towards Ascension! asa@nada.kth.se http://www.nada.kth.se/~asa/ GCS/M/S/O d++ -p+ c++++ !l u+ e++ m++ s+/+ n--- h+/* f+ g+ w++ t+ r+ !y
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 14:49:23 MST