> Nick Bostrom wrote:
> >
> > Eliezer S. Yudkowsky wrote:
> >
> > > Your posthumans will find their own goals. In any formal goal system
> > > that uses first-order probabilistic logic, there are lines of logic that
> > > will crank them out, totally independent of what goals they start with.
> > > I'm not talking theory; I'm talking a specific formal result I've
> > > produced by manipulating a formal system.
> >
> > What?!?
>
> Sigh... here we go again. Take a look at "Coding a Transhuman AI";
> there's a section in there about goal systems. That's got the full
> cognitive architecture.
There was a similar discussion some time back. Your reasoning is that: either there are objective values, or there aren't any. In the latter case it doesn't matter what we do -- the expected utility is zero for all possible actions. In the former case it does matter what we do, but we don't yet know exactly how. To maximize expected utility you thus set out on a search for what the objective values are. Nothing to lose, but you might have something to win.
Well, note first of all that you can't derive any recommended action unless you already assume that we have some (probabilistic) knowledge of objective values. If we know absolutely nothing about them, then, for all we know, it might have a horribly negative utility that somebody sets out on a search for objective values. We would thus have no reason to adopt that as an interim goal: it would lead to the worst possible outcome.
Second, an agent is rational (I maintain) iff she chooses actions that maximize her expected utility, where her utility function is determined by her present preferences. If one does not have a preference for spending one's life ransacking one's soul (or building a seed SI) in order to find out what the "objective values" are, then it would be irrational to do that. If you prefer to spend your life wanking or playing Duke Nuke'm, then that's what is rational for you to do (given that you have taken into account the long term consequences of your actions). It may not be very nobel, but it would not be irrational either, if that's what you really want to do most of all.
Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk
Department of Philosophy, Logic and Scientific Method
London School of Economics