Alejandro Dubrovsky wrote:
>
> On Wed, 2 Dec 1998, Eliezer S. Yudkowsky wrote:
>
> [snipped lots]
> > functioning cohesively to produce correct answers. I don't allow myself to be
> > directed by wishful logic. Even my root supergoal, the Singularity, is
> > produced by a tight logic chain of minimal length, and I defy anyone to
> > produce a shorter chain of logic without assuming what they're trying to prove.
>
> so, what is this chain of logic that leads you to seek the Singularity?
> I'm sorry if you've stated it before, i may have missed it.
> chau
Okay. First, the actual Interim logic:
Step 1: Start with no supergoals, no initial values.
Result: Blank slate.
Step 2: Establish a goal with nonzero value.
Either all goals have zero value (~P) or at least one goal has nonzero value
(P). Assign a probability of Unknown to P and 1-Unknown to ~P. Observe that
~P (all goals have zero value) cancels out of any choice in which it is
present; regardless of the value of Unknown, the grand total contribution to
any choice of ~P is zero. All choices can be made as if Unknown=1, that is,
as if it were certain that "at least one goal has nonzero value". (This
doesn't _prove_ P, only that we can _act_ as if P.)
Result: A goal with nonzero renormalized value and Unknown content.
Step 3: Link Unknown supergoal to specified subgoal.
This uses two statements about the physical world which are too complex to
be justified here, but are nevertheless very probable: First,
Result: "Superintelligence" is subgoal with positive value.
Or to summarize: "Either life has meaning, or it doesn't. I can act as if it does - or at least, the alternative doesn't influence choices. Now I'm not dumb enough to think I have the vaguest idea what it's all for, but I think that a superintelligence could figure it out - or at least, I don't see any way to figure it out without superintelligence. Likewise, I think that a superintelligence would do what's right - or at least, I don't see anything else for a superintelligence to do."
--
There are some unspoken grounding assumptions here about the nature of goals,
but they are not part of what I think of as the "Singularity logic".
Logical Assumptions:
LA1. Questions of morality have real answers - that is, unique,
observer-independent answers external from our opinions.
Justification: If ~LA1, I can do whatever I want and there will be no true
reason why I am wrong; and what I want is to behave as if LA1 - thus making my
behavior rational regardless of the probability assigned to LA1.
Expansion:
By "questions of morality", I mean the cognitive structures known as "goals"
or "purposes", whose function is to choose between actions by assigning
relative values to different states of the Universe.
By "real answers", I mean that there are logically provable theorems,
experimentally provable laws, or physically real objects that correspond to
these functions sufficiently that it would be perverse not to regard our
cognitive models as imperfect approximations of reality.
By "unique, observer-independent" I mean that the objective morality is not
different from each intelligent perspective, although of course different
intelligences may have different pictures.
By "external from our opinions" I mean that the objective morality is not
influenced by our own opinions about it, and is likely to be as strange and
alien as quantum physics or general relativity.
Note: This does not say anything about whether the correct value is nonzero.
Rational Assumptions:
RA1. I don't know what the objective morality is and neither do you.
This distinguishes from past philosophies which have attempted to "prove"
their arguments using elaborate and spurious logic. One does not "prove" a
morality; one assigns probabilities.
LA1 and RA1, called "Externalism", are not part of the Singularity logic per
se; these are simply the assumptions required to rationally debate
morality.(1) The rules for debating morality do not differ fundamentally from
the rules for debating reality. If you substitute "reality" for "morality",
"statements" and "opinions" for "goals" and "purposes", and "models" for
"actions", in the assumptions above, you will have true statements with the
same interpretation.
(1) This is not to say that the assumptions are necessarily true, simply that
I do not see a plausible or useful alternative.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.