From: hal@finney.org
Date: Sun Oct 24 1999 - 12:03:46 MDT
Robert J. Bradbury, <bradbury@www.aeiveos.com>, writes:
> I believe that Skye may have hit upon something every
> interesting that partially solves a major problem
> involving self-modifying AI that has plagued the
> list and worried me greatly:
>
> > It might be interesting if you could create an
> > artificial environment where you could test such
> > things. For example, if you had an artificial human
> > body existing on some kind of programmed level, you
> > could instill these devices into it and see if all of
> > the simulated functions could continue... this would
> > be a ways beyond modern technology, though, because
> > an artificial computer model of a genome and the full
> > animal generated therefrom, not to mention more
> > processing power and memory space than a medium sized
> > country, might be necessary:)
In the novel Cassini Division, by Ken MacLeod, something similar to this
is done. Super-advanced life forms have somehow been kept or captured
by humans and are kept in storage, run only in simulated environments
when it is necessary to get some information from them. In the novel
this appears to have been at least moderately successful. The humans
have gotten some high technology from the AIs.
In Greg Bear's novel Eon, a super-intelligent enemy Jart is kept in
a simulated system, however it knows techniques to escape from any
simulation into a higher level system. It ends up getting out of its
prison and taking over its captor.
So this idea has been explored in fiction, with opposite outcomes.
I suspect that it is possible in principle to keep an AI within a
simulation, but in practice it may be able to notice aspects of the
environment which you have imposed ad hoc and which don't seem "natural".
> (a) Can we guarantee that the AI never discovers it is running on,
> and more importantly escape from, a simulation machine?
> This goes back to the entire thread of whether we can detect *we*
> are running on a simulation or whether our reality is an illusion.
I think if you could actually set up the AI so that it actually evolved
from a starting environment which did not have much more ad hoc features
than our own, you might be able to fool it. However that might take
billions of years of simulated evolution. Any shortcuts you take might
be noticeable by a super-smart AI. Even if you succeeded, it's not
clear what use an AI would be which lived in a world which is probably
so dissimilar to our own.
In our own case, we do observe a great many coincidences which suggest
that the universe is specially set up for life to be possible. However
few people conclude from this that we are running in an artificially
created simulation. (The people who do, we call religious, and apparently
we are supposed to bash them. Bash! Bash!)
> (b) How do we guarantee that everybody understands and
> adheres to the rules that self-evolving AIs are only
> allowed to exist in simulated worlds? {This is not
> dis-similar from the problem of how do we guarantee
> that petty dictators don't release bioweapons that
> in the process of killing us, come back to "bite" them.}
I view this as similar to many related ethical issues which will arise
when it becomes possible to create intelligent/conscious simulated
life forms. Besides the question of creating dangerous life forms,
there is the problem of creating unhappy ones, of creating slaves and
using coercion against intelligent beings.
How do we guarantee today that people treat their children well and
don't raise them to be sociopaths? Well, some people do mistreat their
children, and about the only way to catch them is to notice that the
children have something wrong with them when they interact with others.
If children are kept inside all the time and no one knows they exist,
severe mistreatment is possible, and does happen occasionally.
I don't see any way to enforce such provisions short of some kind of
all pervasive surveillance, which invites a great many abuses of its own.
Furthermore I don't agree that this trick of running the AI in a
simulation is a panacea. You're running the AI for a purpose, probably
for a reason related to solving your own problems. It may be giving
you advice about how to live your life for maximum advantage. In order
to be effective you'll have to give it so much information about the
outside world that it would basically know the whole story anyway.
Hal
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:05:36 MST