It's likely that, if we can produce an SI, we can produce many SIs.
However, my belief
is that there is really only one relevant SI, and that is an SI whose
motivation is to
become more intelligent. This SI is the important one, because this is
the one that
has a built-in positive feedback mechanism. I also belief that this
motivation is very
likely to be a basic part of the first SI, almost by definition. The
creator(s) of the
first SI are likely to have this motivation themselves Otherwise, why
create an SI? Further
the SI may be a computer-augmented human or some other type of
human-computer collaboration,
in which case the SI is likely to include its creator, who surely has
this motivation.
>
> Even if they could "use" us for manual labor- and what would we produce for
> them?
More intelligence. We will be useful until the SI has direct control of
manufacturing
and connection of additonal computational capability. The SI will be
able to
"use" us just as we "use" each other: by contracting for services,
either by letter or over the telephone. An SI embedded in the internet
will have no difficulty arranging for
valid credit card numbers, bank accounts, etc. I believe that an SI will
be able to design and build whatever automated tools it needs in just
this way, in a matter
of a few days or weeks. Once the tools are available, it will no longer
need humans to provide
these services. At this point utility is no longer a reason for the SI
to preserve humanity.
However I hope the SI will derive some other reason, using its superior
intelligence.
>
> In essence, what would they want to destroy us *for*? Comparatively
> aesthetic messiness?
The SI may not want to destroy humanity. Humanity may simply be unworthy
of
consideration, and get destroyed as a trivial side effect of some
activity of the
SI. A simple example: the SI decides to maximixed its computational
speed by indreasing
its mass and converting the mass into computational units. It does this
by gathering
all the mass in the solar system (mostly the sun's mass) This is likely
to be unhealthy
for the humans. the SI may then decide to increase its computatinal
speed by increasing
its density, and convert all that mass into a neutron star. This is
likely to be more
unhealthy.
>
> [PS Anders your post made me want to draw transhuman wet/dry multi- sided
> organelles, but then that gave me Borg images again...]