From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sat Dec 08 2001 - 16:21:09 MST
Ben Goertzel wrote:
>
> For what it's worth, my intuition agrees with Eli's on this. We have yet
> explored only a very small part of the "known universe", and there are big
> aspects of particle physics that we don't yet come close to understanding
> (e.g. quantum gravity, quantum measurement -- yes, there are claims of
> understanding, but nothing well-substantiated). To assume that the Big Bang
> /Big Crunch model or any other fragment of modern science is going to
> survive untouched 1000 years from now is just plain silly.
>
> The same intuition leads me to believe that the notion of "designing
> Friendly AI" is only slightly relevant to the final outcome of the grand
> "superhuman AI engineering" experiment. I agree that it's a worthwhile
> pursuit, because it has a clearly > 0 chance of making a difference. But as
> with fundamental physics, there's a hell of a lot we don't understand about
> minds...
Well, the CFAI model doesn't require that the creators know everything
about the human mind. It requires a certain bounded amount of complexity
which is used to construct an unambiguous pointer to unknown facts about
the human mind, facts which may not be known now, but which are expected
to be accessible to a transhuman intelligence.
In other words, under the CFAI model, you can say: "I have this vague
feeling that liberty and life and love and laughter are important, but I'm
not sure about it, and I don't know where the feeling comes from. Count
that in, okay?" The physical causation behind this statement - in your
accumulated experience, in your brainware, in your genes - is in principle
accessible to a transhuman intelligence, even one that has to extrapolate
the causes after the event. The Friendly AI can then intelligently use
existing philosophical complexity to decide which of these causes are
valid and should be absorbed. The Friendly AI can then "repeat" the above
statement at a higher level of intelligence - that is, having absorbed the
moral baseline behind the statement, it can re-produce the statement as
you would have produced it at a higher intelligence level.
So what's needed is the threshold level of moral complexity to understand
how to correctly use pointers like the one described above - not a
complete diagram of human moral complexity, or a complete understanding of
transhuman philosophy. That threshold level of complexity - which is big,
but bounded, and hopefully accessible to merely human understanding - is
what CFAI attempts to describe.
-- -- -- -- --
Eliezer S. Yudkowsky http://intelligence.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:37 MDT