From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sun Jan 10 1999 - 12:25:21 MST
KPJ wrote:
>
> When you select a meme system you should take into account the ramifications
> of adopting the meme system. After you have done it, you will find it much
> harder to change from it.
>
> As a hypothetical example of why we select the rule of avoiding killing:
>
> "If we adopt the meme system that we should avoid killing then we need not
> put lots of resources on avoid being killed (cf. Bosnia). Therefore, we
> should avoid killing."
I see. So, in other words, you're walking a totally predetermined
trajectory where your current meme system determines the meme system
you'll adopt next. (As adjusted for your innate instincts, of course.)
Your walk takes into account no sort of objective right or wrong
whatsoever, just your current set of prejudices. If your meme system
told you to adopt a meme system that told you to nuke Cincinatti, you'd
basically have no defense. If you'd been born a worshipper of Great
Moloch, you'd stay that way until the end of time. You don't claim that
the truth has anything to do with what you believe, so why should I lend
credence to your beliefs, why should I think your beliefs are covariant
with the truth?
You could do anything, kill me or kiss your worst enemy, and deny that
reasoning has anything to do with it. It appears as if your meme system
makes you a sentience to avoid, maybe even to terminate with extreme
prejudice. Thank you for the warning.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:02:46 MST