From: Charlie Stross (charlie@antipope.org)
Date: Mon Nov 29 1999 - 02:58:03 MST
On Sun, Nov 28, 1999 at 03:00:48PM -0600, Eliezer S. Yudkowsky wrote:
> > and you have a pretty strong argument
> > for extinction. Now, where does that 70% figure come from??
>
> So why are we still here? This has to have happened at least once
> before in the Universe.
So postulate that it happened but it's still outside our light
cone. (C.f. collapse of the vacuum state.)
> > > Trust me: I don't think I'm infallible.
> >
> > But nonetheless you are prepared to act as if you *were*
> > infallible...The moment you activate your ASI Golem, you
> > and you alone will have passed judgement on the world,
> > using your finite wisdom.
>
> Like I said in the other post, the SI passes judgement. I'm not the
> smartest being in this chain of causality.
Hate to but in so late, but I find the whole framework of this
debate somewhat disturbing.
Even without invoking silliness like cartesian dualism or Roger Penrose's
theories it is important to bear in mind that we don't *yet* know enough
to know if uploading is even possible. F'r'ex, suppose significant
computational work is being done by the microtubule skeleton within each
neuron; all we get to see (and base our estimates of the complexity of
the brain on) is a signal emerging from a much more complex system,
while critically important things like long-term memory are encoded
internally. _That_ sort of mistake could jack up the computational
complexity of the human brain by, say, ten or twelve orders of
magnitude, and I wouldn't rule it out until we _know_ how long-term
memory is encoded. (I'm not current with cutting-edge neurobiology, to
say the least, but I'm under the distinct impression that this isn't a
cut and dried issue.) More to the point, such an "oops" mis-estimate
jacks _down_ the intelligence of an SI in relation to ourselves to an
equivalent degree.
Then we have the question of _why_ an uploaded SI might take action
detrimental to not-yet-uploaded people. If Moravec (in Robot) is correct
about the computational complexity of a human mind, there isn't going to
be any immediate resource contention issue; there'll be more than enough
computronium to go round until the SI(s) have elaborated themselves to
make use of resources far beyond anything one human mind might need.
This _will_ take time. I'd therefore expect there to be a window of
opportunity between uploading becoming possible and competition among
SI's for computational resources impacting un-uploaded people. (A window
that may well last for many years -- consider the timescale for converting
the whole solar system into an MB, for example.)
Where's this paranoia coming from?
-- Charlie
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:05:52 MST