Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc..
From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sat Jan 13 2001 - 11:35:52 MST
- Next message: Michael M. Butler: "OFF TOPIC: Re: Obsession, paranoia, risks, was Re: GUNS/TOOLS: Needed? Paranoia!"
- Previous message: Justin Corwin: "META: i hope this list is bigger than this issue: control/guns/extremism"
- In reply to: John Marlow: "Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc.."
- Next in thread: xgl: "Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc.."
- Reply: xgl: "Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc.."
- Maybe reply: John Marlow: "Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc.."
- Maybe reply: John Marlow: "Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc.."
- Messages sorted by:
[ date ]
[ thread ]
[ subject ]
[ author ]
John Marlow wrote:
>
> >
> > An AI can be Friendly because there's nothing there
> > except what you put there,
>
> This is the crux of the problem. The intentions may be
> noble, but I believe this to be an invalid assumption.
> If the thing is truly a sentient being, it will be
> capable of self-directed evolution.
Hence the phrase "seed AI". But don't say "evolution"; "evolution" is a
word with extremely specific connotations. Say "capable of recursive
self-improvement".
> Since, as you say,
> we will have no control over it, it may continue to be
> Friendly--or evolve into something very, very
> UnFriendly.
Again - not "evolve". Whatever modification it chooses to make to itself,
it will make that choice as a Friendly AI.
> In which case, there may not be a damned
> thing we can do about it.
Yep.
> You're playing dice.
No, I'm choosing the path of least risk in a world where risk cannot be
eliminated. You seem to be thinking in terms of "see risk - don't take
risk". A popular viewpoint. It could very easily get us all killed.
-- -- -- -- --
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
- Next message: Michael M. Butler: "OFF TOPIC: Re: Obsession, paranoia, risks, was Re: GUNS/TOOLS: Needed? Paranoia!"
- Previous message: Justin Corwin: "META: i hope this list is bigger than this issue: control/guns/extremism"
- In reply to: John Marlow: "Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc.."
- Next in thread: xgl: "Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc.."
- Reply: xgl: "Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc.."
- Maybe reply: John Marlow: "Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc.."
- Maybe reply: John Marlow: "Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc.."
- Messages sorted by:
[ date ]
[ thread ]
[ subject ]
[ author ]
This archive was generated by hypermail 2b30
: Mon May 28 2001 - 09:56:18 MDT