I've followed this thread with great interest, having myself pondered
this question and variations of it. I've found everyone's comments
interesting, but I'm especially interested in Eliezer's since AI/SI are
the main focus of his work.
I have some questions, Eliezer, and please know that these are asked for
the purpose of trying to increase understanding rather than to pick an
argument. It seems a pity that what could be stimulating intellectual
discussions on this list sometimes turn into unfriendly arguments.
On Tue, 05 Sep 2000 14:19:43 -0400 "Eliezer S. Yudkowsky"
<sentience@pobox.com> writes:
>
> Which assumes an inbuilt desire to reach some stage of independence.
> You have
> not explained how or why this desire materializes within the seed
> AI.
>
> I'm assuming we're talking about a seed AI, here, not a full-grown
> SI.
What if you don't make this assumption? Would a full-grown SI
necessarily have an inbuilt desire to reach some stage of independence?
Independence from what or whom? It seems as though this "from what or
whom" question must be answered before one can meaningfully discuss
whether or not such independence would be a necessary condition of
SuperIntelligence.
> You must be joking. You cannot "put down" a superintelligence like
> some kind
> of wounded pet.
No, not like a wounded pet. But I can imagine wanting to destroy a SI
that I perceive as a threat to me.
.The time for such decisions is before the seed AI
> reaches
> superintelligence, not after.
Is this because once the AI reaches the SI stage it would be hopeless for
the less intelligent humans to try to destroy it? Or because of moral
concerns? Or some combination of both? Or do you condiser it logically
necessary given the properties of humans & SI's?
> One does not set out to "control" an AI that diverges from one's
> desires.
>
> One does not create a subject-object distinction between oneself and
> the AI.
Would the pronoun "one" in the sentences above refer only to the
creator(s) of the AI? Or to some larger group of humans? Would these
sentences be valid if the pronoun "one" could also refer to an SI in
relation to an AI?
> You shape yourself so that your own altruism is as rational, and
> internally
> consistent as possible; only then is it possible to build a friendly
> AI while
> still being completely honest, without attempting to graft on any
> chain of
> reasoning that you would not accept yourself.
You're not implying that the AI will necessarily take on the personality
of its creator, though? Why would honesty on the part of the creator be
necessary? Honesty with whom and with respect to what?
> You cannot build a friendly AI unless you are yourself a friend of
> the AI,
> because otherwise your own adversarial attitude will lead you to
> build the AI
> incorrectly.
The above makes sense--but the fact that the creator is a friend of the
AI doesn't imply that all humans are friends of the AI. It seems as
though the inverse of this last statement would also be true--that the Ai
would not be a friend of all humans. It seems unlikely that the creator
could be friendly towards all of humanity except at an abstract level.
But maybe friendliness at the abstract level is all that's required?
What about when a human or group of humans has interests that conflict
with another human's or group of humans'? What sort of instructions
would the fledgling SI have for dealing with such situations?
Barbara
________________________________________________________________
YOU'RE PAYING TOO MUCH FOR THE INTERNET!
Juno now offers FREE Internet Access!
Try it today - there's no risk! For your FREE software, visit:
http://dl.www.juno.com/get/tagj.
This archive was generated by hypermail 2b29 : Mon Oct 02 2000 - 17:37:14 MDT