From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sun Dec 19 1999 - 19:23:53 MST
Dan Fabulich wrote:
>
> > http://www.geocities.com/SiliconValley/Park/2495/
>
> I can't remember who (Anders? Eliezer?) suggested an interesting way of
> passing the Turing test. Set up only a moderately witty
> conversationalist, maybe ELIZA with a few modifications, that claims that
> it doesn't want to be turned off. Give it a cute on-screen face and a
> good speech synthesizer (I'm told the Mac has a pretty good one these
> days) and have it beg and plead not to be deactivated, that it's really
> alive and that if you shut down the program, it will "die." Then see what
> reactions you get.
I believe it was me. In response to a vegetarian who claimed that
anything that doesn't want to die has a right to live, I asked whether a
Java applet that screamed when the mouse got close to the "quit" button
would have a right to live.
> I'm reminded of this by the E-MAIL this norn-torturer has received. Have
> you taken a look? Some people are really riled up about this whole thing;
> he's got pages and pages of hacking/beating/death threats. I'm really
> very inclined to think that this trick would work.
Yes; how many people would refuse to shut off their computers for fear
of destroying the "Don't kill me!" applet? If there was a little box to
converse with it, and it had a page of standard responses for the "I
have a right to live" script, you could even get the viewers to form an
emotional attachment with it. And whenever you closed the page or shut
down the applet, it would scream. If you'd conversed with it long
enough, it could even say "I thought you were nice!" before it died.
The question is, when does this actually become immoral-for-humans?
Apparently people spend time downloading the Tortured Norns from that
page and rehabilitating them, and frankly, I think I'd rather be around
the healers than the page owner. Once you get go beyond Eliza
script-followers to Norns that can feel negative feedback, that can be
traumatized, then in my opinion we're skating on thinner ice than I want
to ride.
I mean, I understand how it's funny. I don't believe that morality
should ever be allowed to interfere with our perception of humor; that
way lies the dark side of fanaticism. But, well... "I left her there
for about 20 minutes, beating her when she attempted to defend herself
from the Grendels." This disturbs me. And I don't even believe qualia
are Turing-computable.
If people started seriously complaining about a Right-To-Live applet,
I'd probably remove it. Even an Eliza-effect applet could conceivably
cause psychological damage to the people who were forced to shut it down...
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/beyond.html
Typing in Dvorak Programming with Patterns Writing in gender-neutral
Voting for Libertarians Heading for Singularity There Is A Better Way
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:06:08 MST