Re: Posthuman mind control (was RE: FAQ Additions)

Nick Bostrom (bostrom@ndirect.co.uk)
Wed, 3 Mar 1999 17:48:51 +0000

Eliezer S. Yudkowsky wrpte:

> Nick Bostrom wrote:
> >
> > My answer to this is a superposition of three points: (1) I
> > explicitly allowed that fundamental values could change, only (except
> > for the mind-scan) the change wouldn't be rationally brought about.
> > For example, at puberty peoples' values may change, but it's not
> > because of a rational choice they made.
>
> Are you arguing that, say, someone who was brought up as a New Age
> believer and switches to being an agnostic is not making a rational choice?

Believing in the healing powers of crystals is not a value, it's a mistaken factual opinion. The New Ager, provided he has the same data as we do, will be rational to give up his New Agey beliefs.

> > (2) Just because somebody
> > calls a certain value fundamental doesn't mean it actually is
> > fundamental.
>
> So fundamental values are... whatever values don't change?

Not exactly. They can change in a non-rational way (i.e. not as a result of a well-informed deliberate choice) or in a rational way (for example in the mind-scan scenario).

>Please
> clarify by defining the cognitive elements constituting "fundamental
> values". I make the following assertion: "There are no cognitive
> elements which are both invariant and given the highest priority in
> choosing goals."

It's hard to give a precise definition of fundamental value, just as it is hard to give a precise definition of what it means to believe in a proposition. But let me try to explain by giving a simplified example. Suppose RatioBot is a robot that moves aroung in a finite two-dimensional universe (a computer screen). RatioBot contains two components: (1) a long list, where each line contains a description of a possible state of the universe together with a real number (that state's "value"); (2) a module that simulates all possible sequences of actions the RatioBot can make, 10 moves ahead. The simulated state of the world is compared to the list, and RatioBot then performs the sequence of actions leading to the state with the highest achievable value.

Imagine RatioBotII, which is slightly smarter than RatioBot. In order to use work more efficiently, RatioBotII has a third module, that tries to plan ahead. It looks at possibilities more than 10 moves ahead, and figures out suitable subgoals, that module (2) then tries to approximate. The goals that module (3) gives to module (2) are intermediary goals, and they represent one type of non-fundamental values (emotions would be another type). On the other hand, the values expressed by the list (1) could be said to be fundamental.

> Another question: What are *your* "fundamental values" and at what age
> did you discover them?

I don't know (in the reflective, abstract sense) exactly what my fundamental values are; the human mind is, as we know, far from transparent to itself. To the extent that I do know them, I have discovered them gradually. My values may also have changed somewhat on some occasions, mostly in a non-rational (not "irrational") way.

I think I know approximately what my fundamental values are: I want everybody to have the chance to prosper, to be healty and happy, to develop and mature, and to live as long as they want in a physically youthful and vigorous state, free to experience states of consciousness deeper, clearer and more sublime and blissful than anything heard of before; to transform themselves into new kinds of entities and to explore new real and artificial realities, equipt with intellects incommensurably more encompassing than any human brain, and with much richer emotional sensibilities. I want very much that everybody or as many as possible get a chance to do this. However, if I absolutely had to make a choice I would rather give this to my friends and those I love (and myself of course) than to people I haven't met, and I would (other things equal) prefer to give it to people now existing than only to potential future people.

> > (3) With
> > imperfectly rational beings (such as humans) their might be conflicts
> > between what they think are their fundamental values. When they
> > discover that that is the case, they have to redefine their
> > fundamental values as the preferred weighted sum of the conflicting
> > values (which thereby turned out not to be truely fundamental after
> > all).
>
> Why wouldn't this happen to one of your AIs?

With human-level AIs, unless they have a very clear and unambigous value-structure, it could perhaps happen. That's why we need to be on our guard against unexpected consequences.

Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk Department of Philosophy, Logic and Scientific Method London School of Economics