From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Dec 04 1998 - 15:07:04 MST
Hal Finney wrote:
>
> There are obviously a multitude of possible moral systems. If we think
> of the moral system as an algorithm which, given a situation and a list
> of actions, produces a rank ordering of the actions, then there would
> probably be potentially an infinite number of moral systems, since there
> are an infinite number of computer programs.
>
> How do we single out the one or ones which represent "absolute morality"?
> What does this concept mean? I can't get a grip on it. Is there
> really such a thing?
Incidentally, after producing a rather speculative post along the lines of
"Dunno, but I hope so", I should like to remind everyone why I hope so:
There's an infinite set of possible choices.
If all choices are equally right, why do you get up in the morning?
In short, "objective morality" is the *correct* choice.
If there's no distinction between a "correct" moral system and an "incorrect"
moral system, no way of singling out the best, if they're all
indistinguishable from each other - well, why do you make any choice at all?
I acknowledge this possibility, but I also acknowledge that there may be a
method of distinguishing between choices, and it's the second possibility that
yields meaningful guidance.
Don't tell me that the "correct" choice depends what system you're in. The
system *is* the set of correct-to-the-system choices; that's the *definition*
of "system". Which *system* is "correct"? If all the systems are
indistinguishable, why not choose the system where I jump off a building?
Again, I act on the assumption that systems are distinguishable, because the
alternative doesn't yield meaningful directions.
You have some people who say, "The choices are all the same; I'll do what
seems best to me." At a higher level of self-awareness, you have: "I'll
stick with the evolutionary system I was born in, they're all the same." At a
higher level of self-awareness, you say: "Which new system you choose depends
on how your current system evaluates that choice." Choices, systems,
trajectories... but I want to jump out of the system and choose the real answer.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 14:49:54 MST