From: Nick Bostrom (bostrom@ndirect.co.uk)
Date: Wed Feb 24 1999 - 13:34:53 MST
Eliezer S. Yudkowsky wrote:
> Your posthumans will find their own goals. In any formal goal system
> that uses first-order probabilistic logic, there are lines of logic that
> will crank them out, totally independent of what goals they start with.
> I'm not talking theory; I'm talking a specific formal result I've
> produced by manipulating a formal system.
What?!?
> It's like a heat engine. Choices are powered by differential
> desirabilities. If you think the real, factual landscape is flat, you
> can impose a set of arbitrary (or even inconsistent) choices without
> objection. But we don't *know* what the real landscape is, and the
> probabilistic landscape *isn't flat*. The qualia of joy have a higher
> probability of being "good" than the qualia of pain. Higher
> intelligence is more likely to lead to an optimal future.
Well, if we allow the SIs to have completely unfettered intellects,
then it should be all right with you if we require that they have
respect for human rights as a fundamental value, right? For if there
is some "objective morality" then they should discover that and
change their fundamental values despite the default values we have
given them. Would you be happy as long as we allow them full capacity
to think about moral issues and (once we think they are intelligent
enough not to make stupid mistakes) even allow them full control over
their internal structure and motivations (i.e. make them autopotent)?
> Can we at least agree that you won't hedge the initial goals with
> forty-seven coercions, or put in any safeguards against changing the
> goals?
As indicated, yes, we could allow them to change their goals (though
only after they are intelligent and knowledgeable enough that they
know precisely what they are doing -- just as you wouldn't allow a
small child to experiment with dangerous drugs).
The "coercions" would only be necessary for early generations of AI
that don't have a full understanding of what they are doing. A robot
that is cleaning your house -- it could be useful if it has an
internal program that monitors its actions and "coerces" it to pull
back and shut down if it perceives that it is about to fall down a
staircase or if it hears a human distress dry. (In fact, this kind of
internal "coercion" is even useful in humans. Instinctive fear
sometimes saves us from the consequences of some really stupid
decisions.)
Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk
Department of Philosophy, Logic and Scientific Method
London School of Economics
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:03:08 MST