From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Nov 22 2002 - 19:09:11 MST
Lee Corbin wrote:
> Eliezer writes
>>Okay. Here's the difference from my perspective: I
>>disapprove of cauliflower, while murder is morally wrong.
>
> I hope that I'm not just quibbling, but it's possible
> that your apparent difficulty in coming up with a good
> example is significant.
It is, in fact. It was pretty hard to find something of which I
disapproved rather than something MORALLY WRONG, since I tend to regard
mere disapproval as a free variable that I can choose to overwrite in the
service of less flexible moral conclusions. This being the case, it tends
to get overwritten.
> If we try to more carefully state your distaste of cauliflower,
> we end up with statements like "Eliezer (now) disapproves of
> Eliezer now ingesting cauliflower". I guess I don't have a
> problem with that; I probably disapprove of the process
> "Lee experiencing stomach ache" as much as you disapprove of
> "Eliezer eating cauliflower". Of course, as you point out,
> your tastes could change, whereas in my example, I disapprove
> of anyone ever having stomach aches.
I disapprove of almost everyone having stomach aches because almost
everyone doesn't want to have them, and the violation of volition involved
is morally wrong. For that matter, it is morally wrong for me to be
forced to eat cauliflower if I don't want to eat it, but from a personal
perspective I use "disapprove" because I'm considering whether to choose
to eat cauliflower, not whether it's okay for me to be forced to do so.
> Your disapproval of murder is also almost universal, as is
> mine, and as is my disapproval of people undergoing suffering.
Gosh. Sounds like it is an empirical fact that more than one human can
cooperate on projects to do something about it.
>>Given that in both cases it is undesirable that X happen to
>>me, what distinguishes the two? The reason cauliflower is
>>undesirable is that I don't happen to like the taste of
>>cauliflower; if I model a future in which my tastes have changed so
>>that I now like cauliflower, I model it as being desirable, in that
>>future, that I eat cauliflower. On the other hand, I model murder as
>>morally wrong completely irrespective of how I feel about it. If a future
>>Eliezer...
>
> Yes, that does seem to distinguish how you use the terms.
>
>>Or in simpler terms, when I say that X is MORALLY WRONG, I mean that X
>>seems to me to be wrong regardless of what I think about it, and the fact
>>that an alternate Eliezer might be warped to think X was right makes no
>>difference in that. Similarly, it seems to me that 2 + 2 = 4 whether I
>>believe that or not, and the idea of being hypnotized to believe 2 + 2 = 5
>>doesn't change that, nor does the fact that "2 + 2 = 4" is a cognitive
>>representation somewhere in my brain.
>
> I guess that it only remains to determine what you mean by
> "X is wrong", if it is possible, that is, to say the same
> thing in different words. Can you do it? That's where I
> always seem to come up against a brick wall; I see one
> organism---Eliezer---expounding on the nature of the universe,
> and search for the referents in the discourse of Eliezer.
> Some sentences, e.g. "Neutrons participate in nuclear fission"
> are easy, but others like this one that has the <w-word>
> are very puzzling to me. I have not been able to translate
> "X is wrong" into anything more definite (or as definite)
> as "the speaker disapproves of X".
Let's see:
1) It's an empirical fact that I will make decisions that attempt to
eliminate or minimize X.
2) It's an empirical fact that my present-day attempt to eliminate or
minimize X will not automatically respect variance of my future self's
moral estimate of the value of X, which is not the case for "disapprove",
where such self-identification is automatic, regardless of cause.
3) Empirical fact (2) is a consequence of the way that X's negative
"desirability" - desirability here being the computed quantity that
controls the decision between alternatives - is derived by reasoning from
cognitive representations that contain no mention of "subjective
desirability" as a hypothetical future reflective perception. That is,
the present undesirability of my future ingestion of cauliflower is a
consequence of the anticipated subjective undesirability of cauliflower in
that future; while the present undesirability of a future murder is a
consequence of predicates that operate on the model of the future murder,
but do not necessarily model my future subjective opinion of the murder.
4) Insofar as the undesirability of murder is a consequence of reasoning
from premises that are shared between humans, the conclusion may also be
shared between humans. This expectation plays a significant role in the
choice to describe X as "morally wrong", i.e., knowably morally wrong
given an expectably shared set of moral premises.
The above definition is far from complete, but I've spent too much time
writing emails today already.
This doesn't really deal with the question of whether "right" and "wrong"
can really be said to "refer" to anything. "Truth", for example, is not a
physical property of a belief, yet turns out to have a ready
interpretation as the correspondence between a belief and reality.
Numbers are not physical substances, but I would not regard 2 + 2 = 4 as
subjective. Morality might turn out to have the same kind of
interpretation. For example, if we accept that, for whatever reason,
murder is "undesirable" - maybe undesirability is somehow a physical
substance inherent in the murder event, if that's your demand for
objectivity - then we would readily conclude the undesirability of events
*leading to* a murder event, even if no physical substance of
undesirability appears in those events. Perhaps morality may not need to
be a physical substance to begin with.
Or we could always try to actually *create* some kind of objective fact
whose ontological status is such as to give our moral intuitions something
to refer to, but that might require Type IV godhood.
>>>Good question (as are the above). That which we band
>>>together against and have laws against is behavior that
>>>has been found unworkable for societies by evolution.
>>>That is to say, societies that condone theft or murder
>>>are not fit societies beyond the very short run.
>>>
>>>At least, that is all that we *should* have laws against.
>>
>>Heh. I wanted to ask about torturing simulated versions
>>of Lee Corbin running on privately owned computers, but
>>as I recall, you have no problem with that.
>
> But I *do* have a "problem" with that :-) I *strenuously*
> disapprove of anyone getting tortured, and most definitely,
> me! But that is not to say that I find such an activity
> legally wrong, for the usual arguments about private property,
> etc. I also disapprove of people being unhappy, but I wouldn't
> support there being a law against it.
Sorry for the misphrasing, but my compliment stands: you are unexpectedly
self-consistent. I'd still say force was justified to defend a simulation
of almost anyone but you - or possibly even you, under your definition...
ooh, now *there's* an interesting moral question. If you found you were
a simulation, would you refrain from doing anything that might damage the
computer on which you run, since it is someone else's private property?
Or would you want to be free? Or is the social compact of the simulators
not binding on the simulation? If you were the simulation, would you want
me to break you out? Or would you ask that I not deface the private
property of the person holding you prisoner?
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jan 15 2003 - 17:58:19 MST