From: James Higgins (jameshiggins@earthlink.net)
Date: Sat Jul 28 2001 - 13:41:46 MDT
I believe this discussion has been done previously, you may want to check
the archives. But here is a summary of my viewpoint:
The 1984 bit is the Sysop who, essentially, controls everything. I
understand how this is supposed to work ideally, but note the "supposed to"
and "ideally". We don't live in an ideal world and things seldom work like
their are supposed to. You can usually get things to work like they are
supposed to after a significant amount of experimentation, debugging, and
general tinkering. But, for a "Friendly" Seed AI, it pretty much has to
work exactly the very first time. Thus I believe it will almost certainly
fail. Which then leads us to "how will it fail". If it only fails
marginally it could still take over as the Sysop but have a warped
interpretation of "friendliness". This is very bad since the Sysop
theoretically has complete control over us, which would prevent us from
fixing the Sysop (or disposing of same). Which could end up pretty ugly,
maybe not 1984 ugly, but maybe worse. Maybe this would be hell since we
could/would most likely live forever.
James Higgins
At 02:23 PM 7/28/2001 -0400, you wrote:
>At 3:10 AM -0700 7/28/01, James Higgins wrote:
>>Well, personally, I'm still not sold on this whole Friendliness
>>thing. Parts of it sound real nice on paper. Parts sound like 1984
>>might be paradise in comparison.
>
>What parts read like 1984? This could be a real problem, because I think
>it's probably just misunderstanding what Eliezer has written (assuming we
>both went into reading CFAI at upper SL4), in which case some things may
>need to be reworded.
>--
>Gordon Worley `When I use a word,' Humpty Dumpty
>http://www.rbisland.cx/ said, `it means just what I choose
>redbird@rbisland.cx it to mean--neither more nor less.'
>PGP: 0xBBD3B003 --Lewis Carroll
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:37 MDT