hal@rain.org wrote:
>
> I don't agree. The real question is whether there is any meaning to the
> term "objective morality". I don't know a definition which avoids
> circularity.
How about "the set of choices made by an SI"? That's what Singularitarianism is all about; making more intelligent decisions through a proxy. The idea is that either they'll all make the same choice, or that at least *some* choices will be excluded as being objectively wrong.
Scott Badger wrote:
>
> Of course distinctions exist between choices and consequences.
> What does that have to do with the existence of some immutable,
> objective morality.
Sigh... the trouble with discussing this subject is the verbal contortions I have to go through to maintain correspondence between the actual logic and our cognitive intuitions. (That's casting aspersions on our cognition, not the theory. We're nuts; the logic is perfectly straightforward.)
What I meant is "distinctions useful for making the choice". That is, distinctions which render some choices objectively better than others. If you claim that this only applies within a pre-existing system, then I just ask whether there are distinctions useful for choosing between systems. Is there a sense in which blowing up a K-Mart is actually _wrong_, not just a matter of taste? I don't know. Not knowing, it seems to me that the results of assuming an objective result have a higher priority than any results deriving from the assumption that it's all a matter of taste.
I'm not interested in subjective distinctions. "Subjective" means "evolution is pulling my strings." "Subjective" means "I don't want to justify my assumptions." "Subjective morality" is as silly as "subjective economics" or "subjective physics". I want rational justifications. If you assume anything, I want a rational justification for that. I want a chain of logic leading right back to a blank slate. And now that I have it, I'm not settling for anything less.
What is the concrete difference between Externalism and utilitarianism? In utilitarianism, the morality of happiness is assumed. In Externalism, it has to be rationally justified. Why, exactly, does this make Externalism *less* rational? Occam's Razor!
I think we're getting lost in the Great Maze of Words. Let's look at what you need to do to program "objective morality" vs. "utilitarianism" into an AI. In the second case, you need an initial, unjustified goal labeled "happiness" with positive value. Using Externalism, you can start from a blank state - all goals at zero value.
> Agreed. I want to be free of the psychological dictates of evolutionary
> processes as well as the more artificial, religious proposals of an
> immutable morality. We are process, and any moral system will
> ultimately evolve to reflect changes in that process.
An immutable morality is no more religious than an immutable reality. When science impinges on territory formerly held by religion, whether it be the nature of reality or cosmology or even making choices, it makes the issues scientific ones, it doesn't turn the science into mysticism. To borrow Greg Egan.
> >I don't necessarily _believe_ in objective morality, but my best course
> >is to _act_ as if objective morality.
>
> How did you determine that is your best course? What exactly is wrong
> with a rational, functional, utilitarian morality?
I beg your pardon? Externalism is as rational, functional, and utilitarian as you can get and still be a human being. Externalism is what you get when you take utilitarianism and prune the unnecessary assumptions. We're talking about a philosophy designed for AIs!
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.