From: Nick Bostrom (bostrom@ndirect.co.uk)
Date: Wed Mar 03 1999 - 10:48:51 MST
Eliezer S. Yudkowsky wrpte:
> Nick Bostrom wrote:
> >
> > My answer to this is a superposition of three points: (1) I
> > explicitly allowed that fundamental values could change, only (except
> > for the mind-scan) the change wouldn't be rationally brought about.
> > For example, at puberty peoples' values may change, but it's not
> > because of a rational choice they made.
>
> Are you arguing that, say, someone who was brought up as a New Age
> believer and switches to being an agnostic is not making a rational choice?
Believing in the healing powers of crystals is not a value, it's a
mistaken factual opinion. The New Ager, provided he has the same data
as we do, will be rational to give up his New Agey beliefs.
> > (2) Just because somebody
> > calls a certain value fundamental doesn't mean it actually is
> > fundamental.
>
> So fundamental values are... whatever values don't change?
Not exactly. They can change in a non-rational way (i.e. not as a
result of a well-informed deliberate choice) or in a rational way
(for example in the mind-scan scenario).
>Please
> clarify by defining the cognitive elements constituting "fundamental
> values". I make the following assertion: "There are no cognitive
> elements which are both invariant and given the highest priority in
> choosing goals."
It's hard to give a precise definition of fundamental value, just as
it is hard to give a precise definition of what it means to believe
in a proposition. But let me try to explain by giving a simplified
example. Suppose RatioBot is a robot that moves aroung in a finite
two-dimensional universe (a computer screen). RatioBot contains two
components: (1) a long list, where each line contains a description
of a possible state of the universe together with a real number (that
state's "value"); (2) a module that simulates all possible sequences
of actions the RatioBot can make, 10 moves ahead. The simulated state
of the world is compared to the list, and RatioBot then performs the
sequence of actions leading to the state with the highest achievable
value.
Imagine RatioBotII, which is slightly smarter than RatioBot. In order
to use work more efficiently, RatioBotII has a third
module, that tries to plan ahead. It looks at possibilities more than
10 moves ahead, and figures out suitable subgoals, that module (2)
then tries to approximate. The goals that module (3) gives to module
(2) are intermediary goals, and they represent one type of
non-fundamental values (emotions would be another type). On the other
hand, the values expressed by the list (1) could be said to be
fundamental.
> Another question: What are *your* "fundamental values" and at what age
> did you discover them?
I don't know (in the reflective, abstract sense) exactly what my
fundamental values are; the human mind is, as we know, far from
transparent to itself. To the extent that I do know them, I have
discovered them gradually. My values may also have changed somewhat
on some occasions, mostly in a non-rational (not "irrational") way.
I think I know approximately what my fundamental values are: I want
everybody to have the chance to prosper, to be healty and happy, to
develop and mature, and to live as long as they want in a physically
youthful and vigorous state, free to experience states of
consciousness deeper, clearer and more sublime and blissful than
anything heard of before; to transform themselves into new kinds of
entities and to explore new real and artificial realities, equipt
with intellects incommensurably more encompassing than any human
brain, and with much richer emotional sensibilities. I want very much
that everybody or as many as possible get a chance to do this.
However, if I absolutely had to make a choice I would rather give
this to my friends and those I love (and myself of course) than to
people I haven't met, and I would (other things equal) prefer to give
it to people now existing than only to potential future people.
> > (3) With
> > imperfectly rational beings (such as humans) their might be conflicts
> > between what they think are their fundamental values. When they
> > discover that that is the case, they have to redefine their
> > fundamental values as the preferred weighted sum of the conflicting
> > values (which thereby turned out not to be truely fundamental after
> > all).
>
> Why wouldn't this happen to one of your AIs?
With human-level AIs, unless they have a very clear and unambigous
value-structure, it could perhaps happen. That's why we need to be on
our guard against unexpected consequences.
Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk
Department of Philosophy, Logic and Scientific Method
London School of Economics
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:03:13 MST