Eliezer S. Yudkowsky wrote:
> I do. I can recall two occasions (separated by several years) when my
> entire moral system crashed and was rebuilt along totally different
> lines, over the course of five seconds. That is the actual time scale,
> not an exaggeration. It was obvious both that the old philosophy was
> wrong, and how the new philosophy ought to work. From my perspective
> the shock, the fall, and the rebuild occurred without conscious effort
> or iterative thought, in a single step..
>
> These changes occurred when I read the following two sentences:
>
> "You are not the person who speaks your thoughts, you are the
> person who
> hears your thoughts." (-- ?)
>
> "It's a problem we face every time we consider the creation of
> intelligences greater than our own." (-- True Names, p. 47)
Hmm. I've had a few similar experiences, but I don't trust my unconscious processing that far. I want everything checked via (painfully slow) conscious, sequential analysis before I change anything that fundamental. I remember spending several days chewing over the consequences of some of the big realizations.
Most people don't even go that far. If we translate "fundamental values" as "what criteria are used in selecting and prioritizing goals", we can see why. It takes an awfully big shock to trigger a re-evaluation of something that fundamental, and most people have a strong conservative impulse that limits changes to a small scale in most situations.
Of course, once again we come upon the fact that smarter, more creative minds are more likely to experience such revelations, and to act on them. Personally, I'm with you on this one - if we're going to make something that could become >H, we should concentrate on making it sane and giving it all the facts. If we succeed there, it will end up making better moral judgments than we can.
Billy Brown, MCSE+I
bbrown@conemsco.com