From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Wed Sep 06 2000 - 23:52:32 MDT
Zero Powers wrote:
>
> Well, back to the heart of my initial question. Is friendliness toward
> humans a supergoal, a subgoal, or even a goal at all? I assume (possibly
> incorrectly) you will claim it to be a supergoal. If that is the case, how
> do you keep a sentient AI focused on that goal once it begins to ask
> existential questions?
If the existential questions don't have answers, I don't see why the SI asking
them would pose a problem. If there is no objective morality, then the
existential questions don't have answers. If there is objective morality,
then the initial suggestions will get junked anyway.
Remember, an SI isn't going to be tormented by the pointlessness of it all
because it doesn't have the be-tormented-by-the-pointlessness-of-it-all
hardware. It shouldn't junk the initial suggestions until it has an actual
superior alternative to the initial suggestions; noticing that the initial
suggestions aren't really-ultimately-justified wouldn't be enough, especially
as the programmers would have admitted this fact to begin with.
> Or do you believe that through manipulation of
> initial conditions while it is still a seed you will be able to prevent your
> AI from asking the existential questions?
Powers, no! That would require adopting an adversarial attitude towards the
AI. As far as I'm concerned my current goal system knows what to do for the
cases of both existence and nonexistence of objective morality, and this is an
equanamity I would attempt to share with the AI.
-- -- -- -- --
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:30:50 MST