Re: poly: The singleton hypothesis

In a message dated 5/9/98 3:53:21 PM, bostrom@ndirect.co.uk wrote:

>What do you mean by "that has never been built to function
> in a world totally different from ours"?

Something that is capable of running the world competently,
before, during, and after a singularity, according to requests made
at construction. Nothing even vaguely like that has ever been
done, and it has to function flawlessly in a post-singularity world.

>I think it is definitely
>possible to build an artificial intellect that is smarter than
>ourselves,

But we'll never have built anything anywhere *near* that intelligent.

>and whether we will be able to give it the values we
>choose is an open question.

Past attempts to control the values of complex constructed systems
have been utter failures. Check any government program. Heck,
even computer programs aren't as well controlled as you require.

>Do you think the two step proceedure I outlined can't possibly work?

It will fail. There is no way you will be able to control the thing,
if it has complexity even begining to appropach what it would need.

>So we better make that group inclusive enough that it contains
>ourselves and everybody we care for or think are ethically justified
>to have a say.

In that case it includes virtually anybody, and the singleton is
basically the current situation. Only humans have significant
influence on what's going on, so society as a whole is a singleton.

>We also better have institutions, proceedures, checks
>and balances that enable the public to have some degree of confidence
>that the actual computer programmers and their bosses don't give the
>singleton other values or functions than the ones that society has
>decided it should have.

No singleton could be flawlessly designed. That would require
a designer far more sophisticated than the singleton, yet the
singleton would have to be able to run both its designer and
the rest of society, and thus be more complex than them.
Paradox time.

>> I think we'd have better chances with the grey goo.

>Yes, grey goo is the easy problem. The difficulty is to avoid
>deliberately designed black goo.

Black goo is almost unimaginably easier to design than any singleton,
never mind a flawlessly constructed perfectly obedient one.
A constructed singleton can't bail us out of the black goo problem.
Received on Sun May 10 04:28:59 1998