From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Thu Feb 25 1999 - 14:23:15 MST
"Michael S. Lorrey" wrote:
>
> You create a blind spot. In the blind spot is the 'concience' kernel, which
> cannot be directly manipulated. It can only be programmed by the experiential
> data input which it analyses for useful programming content. It colors this new
> content by its existing meme set before it integrates the new content into its
> accumulated database of 'dos' and 'donts'. The entire database gets to vote on
> every decision, so new content cannot completely wipe out old content, except
> under extremely stressful circumstances (i.e. HOT STUFF HURTS!).
Inadequately specified cognitive elements. What is a "meme set"? What
are "stressful circumstances"? When you say "it analyzes", what exactly
is doing the analyzing, and how, and why do you trust it?
>From http://www.tezcat.com/~eliezer/AI_design.temp.html#pre_prime
==
The Rigidity Problem:
"So?" you say. "Don't let the AIs reprogram the goal system."
Leaving out the equivalence of the goal system and the reasoning
system, leaving out the ubiquitous and reasoned connections between
goal and action, leaving out the web of interdependencies that make
goal behaviors a function of the entire AI...
What is likely to happen if the whole AI gets upgraded except for
the goal system? About the same thing that would happen if you
took a gigantic program going through a major revision and suddenly
made 10% of the code unalterable. Parts of the basic architecture
would be frozen. Rewrites going past the surface would be almost
impossible.
If the basic architecture of all the AI except the goal system
changed, one of two things would happen. The goal system would fall
out of synchronization with the rest of the AI, and would
increasingly exhibit random behavior - in other words, the AI would
go gaga. The other thing that might happen is that other parts of
the architecture would have new and internal goal systems, to
replace the useless and unalterable legacy code. Eventually, the new
goal system would be both broader and more intelligent than the old
code, and they would come into conflict.
==
This also sounds like a prime target for the Architecture Change
Problem, the Unstable Definition Problem, the Goal Conflict Problem, the
Entailment Problem, and the Redirection Problem. Depending on how
you're enforcing the "blind spot", you could also run into the
Combinatorial Hack Problem. On the plus side, as far as I can tell, the
proposed architecture is too primitive to run into the Inconsistency
Problem, the Resonant Doubt Problem, or the Arbitrary Goal Problem.
The Recursive Watcher Problem isn't an actual "problem", an anomalous AI
behavior; it's just a technical impossibility.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:03:08 MST