From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Mon Aug 02 1999 - 22:10:57 MDT
Xiaoguang Li wrote:
>
> fascinated by the recent exchanges between Eliezer S. Yudkowsky and den
> Otter regarding the feasibility of AI vs. IA and the inescapable mystery
> of SI motivations, i visited Eliezer's Singularity Analysis. since i find
> Eliezer's view that objective goals are possible at all extremely
> refreshing in this postmodernist, existential zeitgeist, Eliezer's section
> on Superintelligence Motivations catch my attention especially.
Why, thank you.
> if the most stable system of goals is the most rational by Occam's
> Razor, then might not death be a candidate? it seems intuitively sound
> that if an entity were to commit a random action, that action would most
> likely bring the entity closer to destruction than to empowerment; in
> other words, is not entropy (cursed be that word) the default state of the
> universe and therefore the most stable by Occam's Razor? thus if a SI
> decides to pursue the goal of suicide, it may find that by and large any
> action most convenient at the moment would almost certainly advance its
> goal and thus possess a positive valuation in its goal system. could it be
> that only us petty slaves of evolution are blinded to the irrevocable
> course of the universe and choose to traverse it upstream?
Okay, a couple of points here. First of all, I'm not arguing that
stable systems are rational; rather, I'm arguing that rational systems
are stable - or at least more stable than irrational systems. Occam's
Razor isn't what makes rational systems stable so much as KISS, which in
American engineering slang stands for "Keep It Simple, Stupid".
Irrational systems of the type that are commonly proposed around here -
that is, systems which attempt to force some particular set of goals as
output - are more complex than rational systems because of all the
special cases and the special-purpose code. This, in turn, makes them
less stable.
My visualization of an objective, or "external", morality doesn't
involve an ideally rational means of evaluating goals. Rather, I think
that some goals are "true", objective facts about the Universe, which
are there even when nobody's looking; and any rational system will
arrive at those goals, just as any rational system will conclude that
186,000 miles/second is the speed of light.
The goal system you hypothesize is *too* simple, like a "rational"
system for adding numbers that says, for all X and Y, that X + Y = 4.
Certainly this system is much simpler than an actual calculator, but
even with Occam's Razor it still isn't true.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/tmol-faq/meaningoflife.html
Running on BeOS Typing in Dvorak Programming with Patterns
Voting for Libertarians Heading for Singularity There Is A Better Way
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:04:38 MST