From: Billy Brown (bbrown@conemsco.com)
Date: Mon Mar 01 1999 - 15:17:23 MST
Eliezer S. Yudkowsky wrote:
> I do. I can recall two occasions (separated by several years) when my
> entire moral system crashed and was rebuilt along totally different
> lines, over the course of five seconds. That is the actual time scale,
> not an exaggeration. It was obvious both that the old philosophy was
> wrong, and how the new philosophy ought to work. From my perspective
> the shock, the fall, and the rebuild occurred without conscious effort
> or iterative thought, in a single step..
>
> These changes occurred when I read the following two sentences:
>
> "You are not the person who speaks your thoughts, you are the
> person who
> hears your thoughts." (-- ?)
>
> "It's a problem we face every time we consider the creation of
> intelligences greater than our own." (-- True Names, p. 47)
Hmm. I've had a few similar experiences, but I don't trust my unconscious
processing that far. I want everything checked via (painfully slow)
conscious, sequential analysis before I change anything that fundamental. I
remember spending several days chewing over the consequences of some of the
big realizations.
Most people don't even go that far. If we translate "fundamental values" as
"what criteria are used in selecting and prioritizing goals", we can see
why. It takes an awfully big shock to trigger a re-evaluation of something
that fundamental, and most people have a strong conservative impulse that
limits changes to a small scale in most situations.
Of course, once again we come upon the fact that smarter, more creative
minds are more likely to experience such revelations, and to act on them.
Personally, I'm with you on this one - if we're going to make something that
could become >H, we should concentrate on making it sane and giving it all
the facts. If we succeed there, it will end up making better moral
judgments than we can.
Billy Brown, MCSE+I
bbrown@conemsco.com
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:03:12 MST