From: Nick Bostrom (bostrom@ndirect.co.uk)
Date: Thu Feb 25 1999 - 14:48:56 MST
> Nick Bostrom wrote:
> >
> > Eliezer S. Yudkowsky wrote:
> >
> > > Your posthumans will find their own goals. In any formal goal system
> > > that uses first-order probabilistic logic, there are lines of logic that
> > > will crank them out, totally independent of what goals they start with.
> > > I'm not talking theory; I'm talking a specific formal result I've
> > > produced by manipulating a formal system.
> >
> > What?!?
>
> Sigh... here we go again. Take a look at "Coding a Transhuman AI";
> there's a section in there about goal systems. That's got the full
> cognitive architecture.
There was a similar discussion some time back. Your reasoning is
that: either there are objective values, or there aren't any. In the
latter case it doesn't matter what we do -- the expected utility is
zero for all possible actions. In the former case it does matter what
we do, but we don't yet know exactly how. To maximize expected
utility you thus set out on a search for what the objective values
are. Nothing to lose, but you might have something to win.
Well, note first of all that you can't derive any recommended
action unless you already assume that we have some (probabilistic)
knowledge of objective values. If we know absolutely nothing about
them, then, for all we know, it might have a horribly negative
utility that somebody sets out on a search for objective values. We
would thus have no reason to adopt that as an interim goal: it would
lead to the worst possible outcome.
Second, an agent is rational (I maintain) iff she chooses actions
that maximize her expected utility, where her utility function is
determined by her present preferences. If one does not have a
preference for spending one's life ransacking one's soul (or building
a seed SI) in order to find out what the "objective values" are, then
it would be irrational to do that. If you prefer to spend your life
wanking or playing Duke Nuke'm, then that's what is rational for you
to do (given that you have taken into account the long term
consequences of your actions). It may not be very nobel, but it
would not be irrational either, if that's what you really want to do
most of all.
Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk
Department of Philosophy, Logic and Scientific Method
London School of Economics
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:03:08 MST