From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Jun 08 2001 - 11:28:26 MDT
hal@finney.org wrote:
>
> Eliezer writes:
> > Okay, by request, definitions. See also "Creating Friendly AI" (of
> > course).
> >
> > "Child goal": A relation between two goals. "C is a child goal of B".
> > "B is a child goal of A".
> >
> > "Parent goal": A relation between two goals. "B is a parent goal of C".
> > "A is a parent goal of B."
> >
> > "Supergoal": A category of cognitive object. The set of events that a
> > given mind views as having inherent desirability.
> >
> > "Subgoal": A category of cognitive object. The set of all events that
> > are desirable only as a means to an end.
>
> Aren't you giving idiosyncratic meaning to relatively widely used
> terms within AI? Supergoal and subgoal traditionally mean roughly what
> you are here calling parent and child goals. I think it is going to
> foster confusion to take existing terms and give them your own peculiar
> definitions.
Supergoal and subgoal traditionally have *two* *distinct* meanings in
classical AI, the one I have assigned to "supergoal" and "subgoal", and
the one I have assigned to "parent goal" and "child goal".
> Philosophically, it's not clear that the distinction you have drawn
> between super- and subgoals is well defined or even meaningful.
> One problem is that you describe them as being relative to a given mind.
> But goals are a property of systems without minds as well. Animals,
> even very primitive ones, plants, bacteria can all be said to have goals.
No, they can be said to exhibit goal-oriented behaviors, as opposed to
goal-oriented cognition. Thermostats exhibit goal-oriented behavior but
do not contain an internal model of reality. See CFAI again.
> Do you mean to claim that the goal structure for mindless creatures does
> not have super and subgoals?
Yes.
> Why is it that when something evolves a
> mind, suddenly some goals have "inherent" desirability and others are
> only a means to an end?
Because, when something becomes a mind, it can formulate additional goals
beyond the built-in ones and treat those new goals as a means to the
"inherently" desirable ones. Also, the way you're using "inherent" makes
me think that you may be dragging in the issue of objectivity versus
relativism again, which, again, is orthogonal to the cognitive
distinction...
> Another problem is that it's not clear that there are any supergoals
> as you describe them, or at least that there are any beyond the trivial
> goals of survival and propagation of genes (which, again, apply to lower
> life forms just as much as to humans). Ultimately everything comes down
> to that, we just have different strategies to achieve these goals.
The fact that evolution exhibits goal-oriented behavior does not mean that
the "goals" of evolution are declaratively represented in our own minds.
A human being has the emotional pseudo-supergoal of having sex, not of
reproducing, because in the ancestral environment there was no selection
pressure to distinguish between the two. Hence, condoms.
> There is also the issue that we are not always consciously aware of our
> true goal structure.
Yes, that's quite right. What of it?
> We think we are doing something for one reason
> when the actual purpose is something else. Look at the results in
> sociobiology, like Robin's recent article on why we believe even the
> poor should have good health care. If these analyses are right then
> what we believe about our goals is not very accurate.
Are you talking about "rationalization", in which the line of reasoning
from consciously held goals is distorted such that the child goals are
more likely be evolutionarily advantageous actions? Or are you talking
about "puppeting", i.e. the way that some human supergoals can be viewed
as goal-oriented behaviors of evolution with the purpose of reproduction?
> If we don't know what our true goal hierarchy is, then we may be
> completely mistaken about whether something is a subgoal or a supergoal
> in your sense. We may think that a given goal has inherent desirability
> while we are actually following that goal in order to attract members
> of the opposite sex. Does that make it a supergoal or not?
That depends on whether the "in order to attract MOTOS" motive is
declaratively represented as a repressed subconscious urge, or whether it
is simply a historical fact about selection pressures in the ancestral
environment.
> I think if you stick to the more traditional notion of supergoals and
> subgoals that you can still achieve many of your rhetorical aims.
My aim is to untangle the two completely different meanings conflated by
the usual usage of the term. "Child goal" and "parent goal" have meanings
that are immediately obvious to any computer programmer. I agree that
"supergoal" and "subgoal" are more subtle, but the meaning I use is a
traditional one, i.e. "What are the supergoals of the system?" (Note that
asking "What are the parent goals of the system?" makes no sense. "Parent
goal" is a relation, not a category.)
> It is
> still valid to criticize an argument which supposes that a subgoal will
> inherently override a supergoal. You don't have to give supergoals this
> exalted and somewhat mystical state of inherent validity, they just have
> to form the justification and purpose of the subgoal.
Again, I think you're dragging in objective morality. If you like, drop
the word "inherent".
-- -- -- -- --
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 08:08:01 MST