From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sun Jun 02 2002 - 17:45:19 MDT
Lee Corbin wrote:
>
> Eliezer writes
>
> > Since, under my best understanding of FAI, the same architecture
> > that makes the goal system stable and what we would regard as commonsensical
> > is the same architecture that gives an FAI the power to conduct its own
> > moral reasoning, a deliberate attempt to exert undue influence seems to me
> > to indicate a profound misunderstanding of what FAI is about and, more
> > importantly, how to build it, and hence doomed to end in disaster.
>
> I guess that "undue influence" means either the insertion of
> bias so that the FAI builders will unduly prosper, or that
> the FAI will reflect just their own narrow views of what
> moral behavior is.
If you're building the first transhuman mind into the Singularity, which may
quite possibly, under some scenarios, need to build itself into the Sysop,
or which for any other reason has the potential to influence the entire
future course of human history, then it seems pretty obvious to me that ANY
personal influence is "undue influence".
> > One good start lies in studying the evolutionary psychology which reveals
> > why it is that, once your stomach is full, you experience an urge to meddle
> > in other's affairs.
>
> Well, I thought that the origins of this urge to "meddle in
> other people's affairs" was quite simple. Humans have an
> urge to control their surroundings. It's why we create
> comfortable dwellings, convert forests to farmland, and (in
> part) colonize the universe. But this urge to control, fine
> as it is in most ways, should wane a little when that control
> starts to affect other sentients. In other words, people
> should think twice before rushing off to cure the poverty in
> a third world country or fighting injustice in a remote county.
No, people have a specific urge to meddle in the affairs of other sentients
because with meddling comes power and with power comes inclusive
reproductive fitness.
> > Figuring out how to build an FAI that can solve this moral problem
> >> (where exactly, or how, to draw the line between true charity that
> >> actually improves the lot of life in the universe, and that which
> >> only makes the giver feel good and incidentally extends his power)
> > - at least as well as the civilization that built it could have - is not
> > as satisfying to our political instincts as directly arguing about morality,
> > but it is also far more useful, since I tend to take it as given that any
> > morality arrived at by human intelligence will not be optimal[!]. Of course,
> > this is a far deeper question than most moral arguments, in the same way
> > that building an AI is more difficult than solving a problem yourself; but
> > unlike moral argument, there is a hope of arriving at an adequate answer.
>
> Your last two sentences appear to say: (1) arguing about morality
> is easier that determining how the moral problem above can be solved
Precisely the reverse. Arguing about morality is much more natural to us
humans, much easier, and much shallower a problem than building a mind that
can argue about morality.
> (2) there is hope at arriving at an objective optimal answer, a
> *truly* moral solution.
>
> I don't agree with the last point, because I doubt whether
> morality can ever be objective. Hence an absolutely rational
> FAI machine can never derive what true morality would dictate
> from some platonic heaven. (Sorry if I've mischaracterized
> your position.)
Objective morality could exist. Failing that, I'll settle for building an
AI that becomes, not "the AI Eliezer built", but "the best AI anyone could
have possibly built". Objective morality satisfies this definition but is
not the only possible outcome that satisfies this definition.
> If morality is more empirical than rational,
> and I think it is, then the FAI needs a lot of data points
> about what moral behavior is, and it's up to the FAI to
> determine the most consistent principles based on that data.
> Curve fitting, in other words.
You curve fit the elements of the model, not the model itself. Human
morality is a process that contains many perfectible elements; moral
argument is a multistep process that includes, for example, "rational
reasoning" steps. These steps can be mistaken and can be corrected and the
result is a "better" morality. An FAI curve fits (experientially learns)
the steps in the process, not the final outputs.
> But then this probably crosses the line into what you would
> consider "undue influence".
It's certainly undue influence if the learned steps include, as a basic
element, something that says "Preferentially favor moral arguments leading
to the conclusion that Eliezer Yudkowsky has higher moral value than Lee
Corbin". I think so. You think so. And teaching an FAI how to stamp out
these kind of biases, should any be accidentally transferred over by the
experiential learning set, does not IMO count as "undue influence". One
good way to handle your responsibility of "no undue influence" is to teach
an FAI what undue influences are.
> I don't see how any machine,
> including a human, can derive, deduce, infer moral principles
> without consulting a table of values. A pointer, if you have
> one, or an explanation if you don't?
At least some moral beliefs are the results of complex thought processes.
You transfer the process, not the output.
-- -- -- -- --
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 09:14:33 MST