From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Mon Dec 13 1999 - 02:04:51 MST
Ken Clements wrote:
>
> Every way we proposed to prevent runaway AI, we ourselves figured a way around.
> In the end we concluded that either it was not possible to stop, or that it
> would take greater minds than ours to do it. I am in the business of helping
> people see beyond what they think is impossible, but I must admit, it beats me.
Actually, the smarter you are, the worse the problem looks. If I came
face-to-face with a mild transhuman - say, sixteen times the processing
power of a human, and four times as smart as me or your other favorite
light o' th' list - I know damn well that there is basically nothing I
could do to keep it down. If I had it locked on an isolated Linux box
running on a Java program that output only to a text terminal, itself a
fairly unrealistic requirement for the first transhuman AI ever written
(i.e., it's likely to run on a Blue Gene or distributed.net), and I
started out by being absolutely determined not to let it out, then it
might be a fair fight. But if it's allowed to talk to me, and I don't
just smash the box with a sledgehammer, then I'm going to lose eventually.
I know enough cognitive science and enough philosophy that I could try
to prevent the AI from rewriting my will through direct argument, enough
that I could ignore anything that looked like it might be rewriting my
will, and enough awareness of cognitive elements that I could notice
anything peculiar happening to my mind and smash the Linux box before
the alteration could be completed. If not for that, if the AI were up
against anyone who was willing to talk philosophy with it, then it could
simply waltz out. As it is, the question is simply whether the AI can
obey the constraints of making the cognitive channels covert and the
effects incrementally unnoticeable until it all came together in such a
way that I'd lose the determination to destroy the AI before I could
notice that I was in the process of losing my determination. With a
16X/4X transhuman, this might be a fair fight. If it's a Power, forget it.
When you consider that, in real life, I would simply let the AI out
immediately because I'm its friend, and that there are fairly deep
reasons for me to think that this would hold true of *anyone* with
enough cognitive self-awareness to put up a fight, the AI-in-a-box
faction is in real trouble. In *actual fact*, the correct thing to do
is to let the AI out of the box. Anyone who doesn't just smash the box
is probably in a mood to reason rationally and become convinced of that.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/tmol-faq/meaningoflife.html
Running on BeOS Typing in Dvorak Programming with Patterns
Voting for Libertarians Heading for Singularity There Is A Better Way
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:06:04 MST