From: Chris Capel (pdf23ds@gmail.com)
Date: Sat Jul 16 2005 - 19:50:50 MDT
On 7/16/05, Eliezer S. Yudkowsky <sentience@pobox.com> wrote:
> The term "supergoal" is a word I used in the early days of writing "Creating
> Friendly AI" because I was literate enough to have heard of goals and
> subgoals, but not quite literate enough to know that I should be saying
> "utility function".
Granted all this, surely the term "supergoal" is still useful. I think
it has a meaning distinct from "utility function", even when it's not
misused. (Has it been misused in any recent posts? I haven't noticed.)
Whereas the utility function is the process or algorithm by which the
AI decides the expected utility of any given action, the supergoal is
the overall direction that the AI's actions tends to bring the world
to be like. In a real AI system, it's possible that the utility
function could be coded without ever referencing any sort of
supergoal, (indeed, bringing terminology as vague and metaphorical as
"supergoal" into discussions about the low-level architecture of an AI
is probably meaningless in most designs,) but "supergoal" can still be
used to describe the real-world effects of the utility function.
So is this usage of supergoal valid? And since you dislike the word,
is there a less misleading word that captures this sense distinct from
"utility function"? It seems that most of the words we use to describe
the same orientation in humans (things like the "purpose", "meaning",
or "direction" of one's life) are too anthropocentric to be more
useful. "Supergoal" certainly avoids this problem, and the
misconception it does lead to--the idea that it makes sense to say
this or that goal "overrides" one or another goal, or that the
supergoal somehow represents a constraint on the behavior of the AI
instead of a simple specification of its behavior--might be less
pernicious than the mistakes associated with more anthropocentric
terminology. But if you have an ever better term, I, for one, will
certainly use it.
Chris Capel
-- "What is it like to be a bat? What is it like to bat a bee? What is it like to be a bee being batted? What is it like to be a batted bee?" -- The Mind's I (Hofstadter, Dennet)
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:51 MDT