From: Eugene Leitl (eugene.leitl@lrz.uni-muenchen.de)
Date: Wed Sep 06 2000 - 09:14:18 MDT
Barbara Lamar writes:
> In an earlier email in this thread, Eliezer wrote the following:
>
> >>If, as seems to be the default scenario, all supergoals are ultimately
> arbitrary, then the superintelligence should do what we ask it to, for
> lack of
> anything better to do.<<
Sorry, I don't believe in Santa Claus. This assumes a single spatially
distributed thing which is brittle as hell. I do not see a viable
trajectory towards such a world state, and I do not see how such state
can be sustainable.
The whole of evolutionary biology and a lot of computational physics
would go out of the window if that thing will become reality. Since I
have a lot of faith in evolutionary biology and computational physics,
I'd rather assume that thing is a yet another red herring, one of far
too many.
> In another email, Eliezer wrote this:
> > If it's smart enough, then it won't make mistakes. If it's not
"If it's smart enough, then it won't make mistakes".
Sounds rather transcendent. Can anything in the physical world be that
smart, especially since the world contains that supersmart thing?
Wouldn't I have to have full knowledge of my state at t, while I'm
myself at t, and have not enough bits to represent myself and the
state of the world I'm supposed to be in full knowledge of?
> > smart enough,
> > then it should be able to appreciate this fact, and help us add
> > safeguards to
> > prevent itself from making mistakes that would interfere with its
> > own goals.
>
> I'm having a difficult time reconciling these 2 statements. If the SI
> has no preferences and would do whatever asked for lack of anything
> better to do, then how could it be said to have its own goals?
We're not smart. But we can make something much smarter than us in a
fully controlled fashion, and can predict all properties of this
hypothetical future entitity reliably before we even have attempted to
build it. Yeah, right.
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:30:49 MST