From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Thu Jun 07 2001 - 23:06:52 MDT
Okay, by request, definitions. See also "Creating Friendly AI" (of
course).
"Child goal": A relation between two goals. "C is a child goal of B".
"B is a child goal of A".
"Parent goal": A relation between two goals. "B is a parent goal of C".
"A is a parent goal of B."
"Supergoal": A category of cognitive object. The set of events that a
given mind views as having inherent desirability.
"Subgoal": A category of cognitive object. The set of all events that
are desirable only as a means to an end.
It is always possible to view a normative goal system in such a way that
there are *no* subgoals, *only* individual decisions taken on the basis of
whether they are predicted, in the ultimate long run, to result in the
achievement of supergoals. Subgoals are *way stations* between decisions
and supergoals that save on computing power at the expense of introducing
additional cognitive complexity.
*Where* the supergoals come from is another issue entirely. Objective
morality versus moral relativism versus renormalization versus Kant versus
Ayn Rand or whoever is an issue entirely orthogonal to the cognitive
distinction between subgoal content and supergoal content.
Why am I annoyed? Because of the following Frequently Heard Fallacy:
"Ethics is knowable! *Everyone* has to do *X*!"
"Why?"
"Look, X is a child goal of Y! So everyone has to do X! See, ethics is
knowable after all!"
Of course, this only holds true if Y is desirable. Nothing has been
proven except a particular local relation - if that.
This is why it is important to distinguish between subgoals and
supergoals. As soon as you see that X is strictly a child goal of Y, you
know that X is a subgoal. (Y, however, may be a subgoal, or a supergoal.)
-- -- -- -- --
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 08:08:01 MST