Re: [SL4] AI flight recorder
From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Jul 21 2000 - 00:11:04 MDT
Peter Voss wrote:
>
> This flight recorder idea will really only work if:
> a) we actually become aware of undesirable goals before it's too late
> b) what is recorded isn't too complex for us to analyze
> c) that the AI isn't largely connectionist (bootstrapping data rather
> than code)
The idea is that we can go back later and analyze it, when we have
better tools (hopefully not ones designed by untrusted AIs!) or more
resources. Analyzing the data in realtime is another issue entirely.
The flight recorder is intended to make sure that some traces are left
if something does go wrong; to ensure that the possibility exists of
recovering from the error; to answer, or at least make it clear that a
definite answer is possible, when people say "What if - ?"
Specifically, the risk being reduced is as follows:
Old scenario: The Singularity Institute's team of five people,
working with ad-hoc tools, creates a prototype AI that winds up with
some random self-justifying and self-preserving goal. Five years later,
the AI has grown enough to outwit the Singularity Institute's three
trained AI watcher-analysts and the other fifty people on the research
team, even armed with their new tools.
New scenario: The Singularity Institute's team of five people,
working with ad-hoc tools, creates a prototype AI that winds up with
some random self-justifying and self-preserving goal. Five years later,
the Singularity Institute's three trained watcher-analysts go over the
flight-recorder data and catch the problem.
Actually, this is a very slight risk reduction in the quantitative sense
- both scenarios are themselves improbable. Most of the risk-prevention
should come from avoiding the problem in the first place. I don't
expect the flight-recorder to ever actually turn something up. If it
does, it really will be time to panic.
In some sense, most of what we're avoiding is the sheer anxiety of
working with an AI whose earliest moments are lost. If, after the
initial stages, we think of even one thing we missed - no matter how
unlikely - and we haven't recorded the AI's development, then we must
either be extremely nervous, or start all over from scratch.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://intelligence.org/beyond.html
This archive was generated by hypermail 2.1.5
: Wed Jul 17 2013 - 04:00:35 MDT