From: Lee Corbin (lcorbin@ricochet.net)
Date: Fri Jun 15 2001 - 21:27:38 MDT
Hal Finney asks
> Why is it that when something evolves a mind, suddenly
> some goals have "inherent" desirability and others are
> only a means to an end?
Eliezer Yudkowsky replies
>Because, when something becomes a mind, it can formulate
>additional goals beyond the built-in ones and treat those
>new goals as a means to the "inherently" desirable ones.
I would reply, instead, as follows: When something evolves
a mind, it is capable for the first time of abstracting a
goal, and this can be a "supergoal". Now a paramecium may
have parent and child goals---which as Eliezer explains below
---are only relative. For example, a certain enzyme may be
catalyzed as a subgoal towards another metabolic activity.
It is, by the way, proper to regard such activities as goals
because of the feedback mechanisms involved; it is obvious that
certain machinery is trying to achieve some effect or other.
But something with a mind can abstract a supergoal. For example,
Feynman in his later years formulated the supergoal "Tuva or
bust". In this he was deliberately confusing himself by creating
a supergoal. Of course, if you'd confronted him about it, he
would have come round and admitted that this wasn't really a
supergoal---yes, he really wanted to do or learn certain things in
Tuva, and even those were perhaps only towards some (perhaps
rather unarticulated) supergoal of enlightenment or something.
But until you forced him to admit otherwise, his nervous
system would maintain "Tuva or bust" as a supergoal.
Eliezer writes:
> Okay, by request, definitions. See also "Creating Friendly AI" (of
> course).
>
> "Child goal": A relation between two goals. "C is a child goal of
B".
> "B is a child goal of A".
>
> "Parent goal": A relation between two goals. "B is a parent goal of
C".
> "A is a parent goal of B."
>
> "Supergoal": A category of cognitive object. The set of events that
a
> given mind views as having inherent desirability.
>
> "Subgoal": A category of cognitive object. The set of all events
that
> are desirable only as a means to an end.
In other words, says I, in mindless creatures there exist only
parent goals and child goals. They're all relative. The enzyme
A catalyzed for reaction B; reaction B takes place to facilitate
the formation of a folicle; the folicle's formation takes place
to accomplish motion; the motion takes place to facilitate the
ingestion of food; the ingestion of food takes place in order
to... catalyze enzyme A, and so on. No supergoals.
But humans (and perhaps a few others here on Earth) formulate
supergoals. "Joy" and "Progress", for example, are Extropian
ones.
Lee Corbin
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 08:08:11 MST