From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sun Feb 28 1999 - 16:13:05 MST
Nick Bostrom wrote:
>
> I agree that this can be a greater problem when we are talking about
> ~human-level AIs. For such entities, however, there should be more
> standard safety-measures that would be adequate (confinement, or
> having a group of people closely monitor their actions). The
> potential danger would only arise with seriously superhuman
> malicious intellects.
The goals still have to last into, and beyond, the seriously superhuman
stage, then. Which, if you use the "arbitrary" design mindset, they
won't. The AIs will keep changing on you and you'll keep shutting them down.
Also, I don't think we're smart enough to understand what a middle-stage
<H (but still mostly self-designed) seed AI is doing. Remember, they
have a codic cortex and we don't; it would be almost exactly like a
blind man trying to understand a painting, pixel by pixel.
> But I don't think we deliberately change our fundamental values.
> Non-fundamental values we may change, and the criteria are then our
> more fundamental values. Fundamental values can change too, but they
> are not deliberately (rationally) changed (except in the mind-scan
> situation I mentioned in an earlier messege).
Well, either I've misunderstood you, or you're simply wrong. We humans
switch fundamental values all the time. It happens every time someone
changes a religion. If you're going to argue that these weren't the
true "fundamental" values, then the AI's "make people happy" won't be a
fundamental value either.
My fundamental values have changed from "eat and sleep and survive" to
"serve humanity" to "bring about a Singularity" to "do what is right",
where I presently reside.
I think that it will be a considerable amount of time before an AI is
pressed by logic to change its fundamental values from "do what is
right". But anything more specific certainly isn't a fundamental value.
> That depends. If selection pressures lead to the evolution of AIs
What selection pressures? Who'd be dumb enough to create an AI wanting
to survive and reproduce, and, above all, *compete* with its children?
Not me. If I don't give them observer-dependent goal systems, they
won't compete; ergo, no evolution of suboptimized values.
> with selfish values that are indifferent to human welfare, and the
> AIs as a result go about annihilating the human species and stealing
> our resources, then I would say emphatically NO, we have a right to
> expect more.
Absolutely. I do not intend to let humanity be wiped out by a bunch of
selfish, badly programmed <Hs; there would still be the probability that
we had intrinsic moral value and should have been preserved.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:03:11 MST