From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Dec 04 1998 - 14:44:31 MST
Hal Finney wrote:
>
> How do you define a moral system? Then, how do you define "absolute
> morality"?
>
> Consider an ALife creature. Faced with a situation, it has a choice of
> actions, and an algorithm to choose an action.
Actually, this is almost exactly the way I define moral systems. I even
considered writing an ALife called "HappyApplet" to put everything down in
code, but put it off once I realized just how much basic architecture was
needed to capture the Singularity logic. (At the very least, I would need a
tightly integrated theorem prover and goal hierarchy.) You can find more on
HappyApplet in the section on "Interim Goal Systems" in "Coding a Transhuman AI".
http://www.tezcat.com/~eliezer/AI_design.temp.html#det_igs
It doesn't answer your question, but it amplifies your definition of the problem.
> There are obviously a multitude of possible moral systems. If we think
> of the moral system as an algorithm which, given a situation and a list
> of actions, produces a rank ordering of the actions, then there would
> probably be potentially an infinite number of moral systems, since there
> are an infinite number of computer programs.
Agreed.
> How do we single out the one or ones which represent "absolute morality"?
(The phrase I use is "objective morality" or "External morality".)
> What does this concept mean? I can't get a grip on it. Is there
> really such a thing?
Is there really such a thing? The only honest answer is "I don't know". I
could tell you how the "objective morality" is singled out, however.
Unfortunately, this gets into some pretty deep issues, down into the qualia
again, and I'm running out of time. Again, however, the quick explanation is
by the analogy between "morality" and "reality".
Out of all the possible descriptions of the world that the ALife might choose,
which one is "true"? The one that corresponds to his perceptions? But we
know from our own investigation that perceptions are often wrong. The problem
that Hughes points out is that there's a basic disconnect between perceptions
and reality; we can know the perceptions exist, but the idea of reality is a
wild speculation. And why this strange idea of "reality" in the first place?
Why, exactly, does anything exist at all? Why this peculiar delusion that one
Turing machine is "real" while the rest are not? What distinguishes "true"
perceptions from "false" perceptions? How is one set of opinions, out of all
the opinions there are, singled out as "correct"? And, considering that any
statement requires another statement to prove it, how can we believe anything
without wandering into an infinite regress? And how do we get from there to
our absolute belief in qualia, the "I think therefore I am", that apparently
ties the whole meta-infinite regress into a single knot?
I don't know. As far as reason can tell me, there is no good reason why
objective morality *or* objective reality should exist. My hope is that there
is a sensible explanation for it all, somewhere on the level from which you
and I look like goldfish, and that there really is something called "objective
reality" that determines the "correct" set of opinions, and that somewhere out
there is something called "objective morality" that determines the "correct"
set of choices.
In short, when I said it would take superintelligence to figure this all out,
I meant it.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 14:49:54 MST