Eliezer S. Yudkowsky wrote:
> Damien Broderick wrote:
> >
> > At 09:06 AM 9/11/98 -0500, Eliezer wrote:
> >
> > [Broderick wrote that what Eliezer wrote:]
> > >> < Never allow arbitrary, illogical, or untruthful goals to enter the AI. >
> > >> reflects a touching faith in human powers of understanding and consistency.
> >
> > >I'm not quite sure what you mean by this. [Eliezer]
> > [Broderick again]
> > Isn't it obvious? How can any limited mortal know in advance what another
> > intelligence, or itself at a different time and in other circumstances,
> > might regard as `arbitrary, illogical, or untruthful'? Popper spank.
>
> So let's throw in all the coercions we want, since nobody can really know
> anything anyhow? That's suicidal! I didn't say the Prime Directive was easy
> or even achievable; I said we should try, and never ever violate it deliberately.
>
> Perhaps the interpretation of the Prime Directive is too dependent on context,
> and it should be amended to read:
>
> "No damn coercions and no damn lies; triple-check all the goal reasoning and
> make sure the AI knows it's fallible, but aside from that let the AI make up
> its own bloody mind."
How about: Thou shalt model any decision first to determine choice most beneficial to one's own long term ration self interest.
I think that given such a rule, any AI will come to its own conclusions as to moral behavior without needing hardwired rules, as it will find that choices most beneficial to one's own long term self interest are also those choices which are least harmful to others.
Mike Lorrey