From: hal@finney.org
Date: Fri Mar 17 2000 - 09:14:00 MST
Xiaoguang Li, <xli03@emory.edu>, writes:
> the debate revolves around the creation of a _power_ -- an entity that
> has no further charactrization than near infinite power and incrutability.
I like this definition.
> to
> den otter, an inscrutable power, at the very best, is neutral to human beings.
> that is, we can only regard its potential actions as random. from this
> perspective, a _power_ is little different from an detonating fusion bomb. at
> the worst, den otter fears that the continuance of existence is an universal
> goal of all existent entities, and as such, may direct the _power_ to pursue
> the extermination of human beings as an explicit subgoal.
The "detonating fusion bomb" isn't a good analogy (not sure whether it
is yours or den Otter's), because that is guaranteed to be destructive.
If part of the definition of the Power is that its actions cannot be
predicted or understood, then the problem is we don't know if it will
be destructive or not.
> yudkowsky, on the other hand, contends that the potential actions of a
> _power_, while uncertain, is not random even from the present human
> perspective. his intuition is that something resembling morality is embeded in
> the deep structure of reality; therefore, a sufficiently intelligent being
> would have access to that knowledge and entertain a chance of preserving
> humanity.
I think, instead, that Eliezer believes that there is such a system of
morality, but not that it necessarily has any relation to our own ideas.
A moral Power may wipe us out, but if so, that was the right thing to do.
I haven't heard of anyone besides Eliezer who finds this comforting.
> every agrees that the reasoning of a _power_ would be
> incomprehensible to any human being -- one would have no way of verifying its
> motives -- only through a priori knowledge of the fabric of reality could one
> retain some peace of mind in the presence of a _power_.
Again, the problem is that virtually no one here beyond Eliezer believes
in this notion of absolute morality, in fact most of us think the idea is
incoherent (unable to be defined consistently). So it does not appear
that any empirical knowledge will help resolve the quandary, certainly
not in the relevant time frame.
>From an outsider's perspective, the argument between Eliezer and den
Otter is not very important. Both of them share assumptions that seem
questionable, namely, that they have any significant say in the outcome.
Bill Joy wrote his jeremiad calling for a halt to progress, but that's
a feather to dam the flood. Likewise for Elizer and den Otter's little
tempest.
Economic growth and technical progress is driving us forward regardless
of what these two people decide. Given the complexity of the world and
the range of talents and abilities that exist, the notions that one or
the other of them is going to become or create the first Power is simply
a fantasy. Our future is not in the hands of these two, but depends
on the industrial, academic and military research being performed in
thousands of labs all over the world.
Hal
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:27:28 MST