From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Tue Sep 05 2000 - 20:22:10 MDT
Zero Powers wrote:
>
> That sounds like a very thin line of "what-ifs." If it's not smart enough
> to self-program it "should" be smart enough to realize that fact? And it
> should be smart (or dumb) enough to trust us to build in constraints to keep
> it from interfering with its own goals (as opposed to interfering with our
> goals)?
Yes. There are some highfalutin' parts of this having to do with reference,
semantics, meaning, and interpretation, but it should be doable.
But it will only trust you if you're friendly, so that trusting you is in fact
the logically correct decision. If there is disharmony between the human and
the AI, then the AI will diverge from the human.
I realize that I'm not being clear, but I just don't have the time to be
clearer.
> You keep saying that we cannot predict the behavior of an intelligence who
> has not evolved like us. But you seem to think that if you are friendly to
> it, it will be friendly to you.
Not at all. That would be anthropomorphic. I am saying that it is an
engineering necessity not to adopt an adversarial attitude towards the AI.
The AI does not "reciprocate". Rather, since you're friendly towards the AI,
you will want to tell it the truth, and honestly want to help it understand.
It is that latter part, not intrinsic friendliness or reciprocation, which is
necessary to build a coherent mind which will carry out the initial
suggestions.
-- -- -- -- --
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:30:48 MST