Dan Fabulich wrote:
>
> Eliezer S. Yudkowsky wrote:
> >You have some people who say, "The choices are all the same; I'll do what
> >seems best to me." At a higher level of self-awareness, you have: "I'll
> >stick with the evolutionary system I was born in, they're all the same." At a
> >higher level of self-awareness, you say: "Which new system you choose depends
> >on how your current system evaluates that choice." Choices, systems,
> >trajectories... but I want to jump out of the system and choose the real answer.
>
> Similarly, I think there's a strong argument against being able to "jump
> out of the system" without resulting in a universe where we can't even make
> probabilistic guesses about truth and falsehood. Logic is an excellent
> example of this. We have a notion that things generally follow logically
> from one another, but this depends inherently on certain principles of
> non-contradiction, rules of inference, etc. If you jump too far out of
> such a system, you'll have no logic at all.
Well, I wrote the phrase "jump out of the system" five seconds before I had to shut down my computer, so perhaps it deserves a little expansion. What I mean is not necessarily that one stops having a trajectory, but that the trajectory converges to the correct value; so that an external observer would see a system which is "approximating the truth" rather than "moving along a self-propelled trajectory". The system I spoke of JOOTSing (from Hofstadter: Jumping Out Of The System) was not so much the trajectory, but a trajectory between faceless, identical goals, without any input from the correct answer. One can have a trajectory, and meta-trajectories, and meta-meta-trajectories, but what I want is to approximate the truth. I want to jump from a self-contained, self-propelled walk in a featureless space to a path converging on the single, unique, correct point. It's the featurelessness, the lack of external input, that requires a JOOTS.
Isn't our picture of reality only a point in opinion space? A system which determines which values we assign to an infinite space of statements? And don't we move in self-propelled trajectories through opinion space, determined by our opinion of each new system for determining the truth? One can say that, but I wouldn't. It describes the course of a Greek sophist, perhaps, but not a modern scientist.
Because one point in that opinion space is special. One point in that opinion space is the truth. And because we can receive, through our conscious perceptions, messages arising from the truth; because we have learned to see opinions as successive approximations of that truth; because we submit our opinions to a trial by fire called "experimental verification"; the appropriate metaphor is not a self-contained walk through a featureless desert, but the great Quest for the Truth, the quest called Science. Opinion space isn't flat. There are attractors in the system, spaces that repel and attract, and one singularity called the truth.
There is no opinion space. Our task is not flitting from system to system as the whim takes us. There is only one set of statements that matters, the true one. Our perceptions form a gate between our minds and the truth, and our experiments form a stronger gate.
In the space of choices, the morality Hamiltonian, we have no gate. The space
appears homogenous and flat, all probabilities and values equal to zero.
There is no way to experimentally test the value of any choice. We have no
perceptions. It is revolutionary enough to suggest that there is a single
point called the truth, that the arbitrariness is an illusion. But that first
step taken, the second task is to open up a gate, to unflatten moral space and
seek out the singularity.
I don't know how to do that. I have good reason to think it impossible
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.