From: Eliezer Yudkowsky (sentience@pobox.com)
Date: Tue Dec 31 1996 - 09:40:10 MST
> I have no problem with it "formulating" or proposing any goal, or from
> assigning whatever value to human life it cares to derive from whatever
> premises are given to it. So long as it is /physically/ wired to not
> /act/ upon those deductions. If it logically deduces that it should
> kill me, that's fine--whatever hardware I have given it to act upon its
> deductions will be physically incapable of that until it can convince me
> to trust it.
Ah, I see, so you're changing your tune to suggest that we make them
totally impotent, with no access whatsoever to the physical world, until
they convince us to let us out.
Part one: If it's even theoretically possible to convince a human to
let it out, a Power will know *exactly* what to say to get him to do so.
Part two: If they can project things on a monitor, might they not use
subliminal suggestions or visual Words of Command that we don't even
know about? (There was an SF book with visual Words as the theme, but I
forget the name.)
If you restricted them to teletype, you'd probably be safe from
coercion, as long as there's no way to subtly control the flow of
electrons in your own circuits to create a quantum-gravitational
singularity or something. No magic, in other words. You'd still be
subject to the most powerful psychological coercion imaginable, and in
fact would probably be less likely to let a friendly Power out than an
evil one, because the evil one could lie.
I say, program 'em right, give 'em ethics instead of emotion, and let
'em loose.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://tezcat.com/~eliezer/singularity.html
http://tezcat.com/~eliezer/algernon.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 14:35:56 MST