Hal wrote:
>
> Is this notion coherent? Does it make sense to speak of a proof that
> a moral system is correct?
Does it make sense to speak of a proof that _any_ object is real?
> Earlier I proposed that a "moral system" was a method of ranking possible
> actions. Given an organism, a situation, and a set of possible actions,
> the moral system is an algorithm for ranking the actions in order (or at
> least selecting the highest-ranked action). We interpret the ranking as
> being in order of most-moral to least-moral, but mathematically it's just
> a ranking.
Let's say that a "reality system" is a method of sorting the space of all objects into "real" and "unreal" objects. Thus the Sun would be real, while unicorns are unreal*. How can you possibly prove something is real without reference to a cause or perception previously known to be real? The causal regress winds up with the Hard Problem of the First Cause, while the perception regress winds up at the Hard Problem of Consciousness. And yet despite this, our Universe does exist, and we do have conscious experiences.
If you wish an example of a goal made solid, consider the qualia of pleasure. I don't know if that's an objective morality, but it's a good argument that the probability of objective morality is nonzero.
> Expressed in these abstract terms, there is no way to distinguish a
> "good" moral system from a "bad" one. Every ranking algorithm is a
> moral system, and they are all on equal footing. You can then
> introduce a "meta-moral system" which ranks moral systems. Given all
> possible algorithms (moral systems), it puts them into a rank order.
> Again, we would interpret this ranking as most-moral moral system to
> least-moral moral system, but mathematically it is just a ranking.
You could argue that an opinion system assigns rankings to reality systems, and a meme system assigns rankings to opinion systems, and so on and so on... but this is building castles on top of air when we should be digging down. To solve the problem of consciousness, to determine what it is about qualia that allows us to be absolutely sure of their reality, we need to dissect cognition and the brain, not invent observers of observers of observers. To solve the First Cause, we need to probe the laws of physics, not invent Creators to create gods to create worlds. To solve objective morality, we probably need to solve the other two problems, just to grab a handhold on what morality is made of.
> I don't see how to ground this regress. It doesn't even seem to me that
> it makes sense to say that a particular ranking is objectively selected.
I don't see how to ground any of the regresses. The world doesn't make sense to me at all. I think I'm on the wrong cognitive level to solve these kinds of problems. You wouldn't expect a Neanderthal to solve them; and I wouldn't expect anyone but me to solve them; and I don't expect me to solve them.
> I'd like to see an example of an objectively-best moral system for a
> simple system.
So would I!
> Does it really seem that the problem is that we are not smart enough to
> solve this? It seems to me that the problem is simply that the question
> is meaningless.
Maybe it is meaningless. In fact, it probably is meaningless. We've proven not only that everything is meaningless but also that we don't exist and neither does reality. As far as we can tell the space of choices is an indistinguishable blur, so is the space of truths, and there's nobody to decide. So how can we choose anything, even to walk across the room?
We're probably asking the wrong question from beginning to end. We're lost. We don't know what the hell we're doing. We don't know what it's all for, or why, or whether we're doing anything right, or if we're asking the right questions, or even if we're capable of asking the right questions. We are in a fog. We are looking through mud. We are blind. We don't even have a visual cortex. Under the circumstances, there are two possibilities:
That's the logic.
Anything more complicated is just a translation.
--
(*) = Unicorns probably are real, somewhere. Even if you specify
magical unicorns attracted to human virgins, we could probably cook some
up in a few years. The mere act of mentioning something on the
Internet's permanent record makes it more likely that some descendant
will eventually create it (even fire-breathing ducks), especially if
it's in the context of an impossibility proof.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.