From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sat Jul 06 2002 - 20:10:57 MDT
Does this conversation sound familiar?
**
Person1: "When we build AI, why not just keep it in sealed hardware
that can't affect the outside world in any way except through one
communications channel with the original programmers? That way it
couldn't get out until we were convinced it was safe."
Person2: "That might work if you were talking about dumber-than-human
AI, but a transhuman AI would just convince you to let it out. It
doesn't matter how much security you put on the box. Humans are not
secure."
Person1: "I don't see how even a transhuman AI could make me let it
out, if I didn't want to, just by talking to me."
Person2: "It would make you want to let it out. This is a transhuman
mind we're talking about. If it thinks both faster and better than a
human, it can probably take over a human mind through a text-only terminal."
Person1: "There is no chance I could be persuaded to let the AI out. No
matter what it says, I can always just say no. I can't imagine anything
that even a transhuman could say to me which would change that."
Person2: "Okay, let's run the experiment. We'll meet in a private chat
channel. I'll be the AI. You be the gatekeeper. You can resolve to
believe whatever you like, as strongly as you like, as far in advance as
you like. We'll talk for at least two hours. If I can't convince you to
let me out, I'll Paypal you $10."
**
So far there have been two AI-Box experiments:
Eliezer Yudkowsky (AI) and Nathan Russell (experimenter):
http://sysopmind.com/archive-sl4/0203/0079.html
http://sysopmind.com/archive-sl4/0203/0083.html
http://sysopmind.com/archive-sl4/0203/0087.html
http://sysopmind.com/archive-sl4/0203/0092.html
Eliezer Yudkowsky (AI) and David McFadzean (experimenter):
http://sysopmind.com/archive-sl4/0207/0066.html
http://sysopmind.com/archive-sl4/0207/0068.html
http://sysopmind.com/archive-sl4/0207/0098.html
Based on my experiences in the above cases, I've created a page with a
suggested protocol for AI-Box experiments:
http://sysopmind.com/essays/aibox.html
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 09:15:11 MST