From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Wed Sep 08 1999 - 13:04:03 MDT
John Clark wrote:
>
> Eliezer S. Yudkowsky <sentience@pobox.com> On September 01, 1999 Wrote:
>
> >I'm serious - Elisson, in _Coding a Transhuman AI_, contains design
> >features deliberately selected to cause a "collapse of the goal system",
> >and lapse into quiesence, in the event existence is found to be
> >meaningless. Preventing those nihilist AIs you were talking about.
>
> You can't be serious. Rightly or wrongly many people, myself included,
> are certain that the universe can not supply us with meaning, and yet
> that idea does not drive us insane, I haven't murdered anyone in months.
You aren't a self-modifying AI, buckaroo. It doesn't strike me as being
tremendously stable to have no reference for what the contents of your
mind should be except the current contents of your mind. If an AI
thinks its opinions are facts, its view of the world will probably
spaghetti off into chaos or some strange attractor. If an AI thinks
its motivations are the only arbiter of its motivations, you have a
similar stability problem.
> If something has no meaning and you don't like that then don't go into
> a coma, change the situation and give it a meaning. You can give
> a meaning to a huge cloud of hydrogen gas a billion light years away
> but it can't give meaning to you because meaning is generated by mind.
> Personally I like this situation because I might not like the meaning
> that the universe assigns to me, I'd much rather be the boss and
> assign a meaning to the universe.
Perhaps. But I'm not sure that assigning meaning is permitted. I'm not
sure, absolutely, that meaning is subjective. So I don't dare try.
I don't trust human logic at all. The only real logic is the logic of
the Universe, the logic that created reality. If you prove that the sky
is blue, did that *make* it blue? No. But there's some kind of logic
that leads to the conclusion that "something exists", and lo and behold
something exists. *That's* what I call logic; logic so forceful it can
actually bring its conclusions into reality. That's a logic I can
trust. We can't use that logic, so I don't trust any human reasoning at
all until we can upgrade ourselves to understand that logic.
*Then* we will actually know things. Then, maybe, we'll be able to
actually be certain about things, instead of just manipulating
probabilities. And we'll be certain because the conclusions themselves
are strong enough to bring things into existence. Maybe we'll even be
able use the logic to change reality.
Until then, I am *not* going to construct an AI that thinks the contents
of its mind are facts. What happens if they're not? What happens if
that irresistable logic comes into conflict with conclusions the AI has
been taught are arbitrary? I am going to construct an AI that seeks out
the logic of the Universe and incorporates it and applies it to
absolutely everything; *if* it turns out that logic has no effect on
motivations, *then* we can start worrying about how to construct a
stable AI with arbitrary motivations.
> You might argue that even though I can't be proven wrong I still
> might be wrong, well maybe.
Nonsense. I think you can be proven wrong.
> But if being right means death or
> insanity and being wrong means operating in a efficient and happy
> manner then whatever could you mean by right and wrong?
True and false. I don't care how happy and efficient it makes you to
believe the sky is green, and I don't care if death or insanity follows
from knowing the sky is blue. The sky *is* blue. It is *not* green.
> I'd
> say a happy efficient brain is constructed correctly and a insane or
> quiescent brain is constructed incorrectly, but again that's just my
> opinion, I like things that work
I'd say a brain that correctly predicts and successfully manipulates
reality is constructed well, but only a brain that incarnates the logic
of the Universe is real.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/tmol-faq/meaningoflife.html
Running on BeOS Typing in Dvorak Programming with Patterns
Voting for Libertarians Heading for Singularity There Is A Better Way
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:05:04 MST