Re: Preventing AI Breakout [was Genetics, nannotechnology, and , programming]

hal@finney.org
Sun, 24 Oct 1999 11:03:46 -0700

Robert J. Bradbury, <bradbury@www.aeiveos.com>, writes:
> I believe that Skye may have hit upon something every
> interesting that partially solves a major problem
> involving self-modifying AI that has plagued the
> list and worried me greatly:
>
> > It might be interesting if you could create an
> > artificial environment where you could test such
> > things. For example, if you had an artificial human
> > body existing on some kind of programmed level, you
> > could instill these devices into it and see if all of
> > the simulated functions could continue... this would
> > be a ways beyond modern technology, though, because
> > an artificial computer model of a genome and the full
> > animal generated therefrom, not to mention more
> > processing power and memory space than a medium sized
> > country, might be necessary:)

In the novel Cassini Division, by Ken MacLeod, something similar to this is done. Super-advanced life forms have somehow been kept or captured by humans and are kept in storage, run only in simulated environments when it is necessary to get some information from them. In the novel this appears to have been at least moderately successful. The humans have gotten some high technology from the AIs.

In Greg Bear's novel Eon, a super-intelligent enemy Jart is kept in a simulated system, however it knows techniques to escape from any simulation into a higher level system. It ends up getting out of its prison and taking over its captor.

So this idea has been explored in fiction, with opposite outcomes. I suspect that it is possible in principle to keep an AI within a simulation, but in practice it may be able to notice aspects of the environment which you have imposed ad hoc and which don't seem "natural".

> (a) Can we guarantee that the AI never discovers it is running on,
> and more importantly escape from, a simulation machine?
> This goes back to the entire thread of whether we can detect *we*
> are running on a simulation or whether our reality is an illusion.

I think if you could actually set up the AI so that it actually evolved from a starting environment which did not have much more ad hoc features than our own, you might be able to fool it. However that might take billions of years of simulated evolution. Any shortcuts you take might be noticeable by a super-smart AI. Even if you succeeded, it's not clear what use an AI would be which lived in a world which is probably so dissimilar to our own.

In our own case, we do observe a great many coincidences which suggest that the universe is specially set up for life to be possible. However few people conclude from this that we are running in an artificially created simulation. (The people who do, we call religious, and apparently we are supposed to bash them. Bash! Bash!)

> (b) How do we guarantee that everybody understands and
> adheres to the rules that self-evolving AIs are only
> allowed to exist in simulated worlds? {This is not
> dis-similar from the problem of how do we guarantee
> that petty dictators don't release bioweapons that
> in the process of killing us, come back to "bite" them.}

I view this as similar to many related ethical issues which will arise when it becomes possible to create intelligent/conscious simulated life forms. Besides the question of creating dangerous life forms, there is the problem of creating unhappy ones, of creating slaves and using coercion against intelligent beings.

How do we guarantee today that people treat their children well and don't raise them to be sociopaths? Well, some people do mistreat their children, and about the only way to catch them is to notice that the children have something wrong with them when they interact with others. If children are kept inside all the time and no one knows they exist, severe mistreatment is possible, and does happen occasionally.

I don't see any way to enforce such provisions short of some kind of all pervasive surveillance, which invites a great many abuses of its own.

Furthermore I don't agree that this trick of running the AI in a simulation is a panacea. You're running the AI for a purpose, probably for a reason related to solving your own problems. It may be giving you advice about how to live your life for maximum advantage. In order to be effective you'll have to give it so much information about the outside world that it would basically know the whole story anyway.

Hal