From: Ben Goertzel (ben@webmind.com)
Date: Sat Nov 18 2000 - 14:46:37 MST
> I am not sure I agree with your exact phrasing. I would rephrase
> as follows:
>
> 1) Once an AI system is smart enough to restructure all the matter of the
> Solar System into vis own mind-stuff, we will not be able to guide vis
> development *if ve doesn't want us to*.
Even if it wants us to, we may not have the intelligence to understand how
its mind works
well enough to guide it...
How much can a dog guide YOUR development?
> In practice, what is necessary is that the early Friendly AI make
> reference to
> the intentions of the programmer. Not the specific intentions that have
> already been embodied, but "the intentions of the programmer", in general,
> including the intentions that the early AI doesn't know about yet.
>
> When the programmer says: "I have this new element to include in
> the design
> of your goal system", the AI needs to think: "Aha! Here's an element of
> what-should-be-my-design that I didn't know about before!", not
> "He wants to
> give me a new goal system, which leads to suboptimal results from the
> perspective of my current goal system... I'd better resist."
>
> In "Friendly AI", I'm working on describing the specific cognitive imagery
> necessary for all that to take place.
Isn't this just a fancy way of saying that a Friendly AI should
love its mommy and its daddy? ;>
> Well, without more detailed knowledge of Webmind, I can't be
> sure; however, I
> don't *think* you're encountering challenges of the same
> underlying class as
> the challenges that would be involved in Friendly AI. But I
> don't know. I'm
> not a Webminder.
Right now, we're just dealing with parameter tuning for improved
intelligence.
In Feb. we'll start a new phase, when we'll make operational the "psyche"
component
of the system (goals, feelings, motivations) ... we then will be quite
precisely
dealing with issues of friendliness and unfriendliness. Questions like:
What attitude
does the system have when we insert new knowledge into its mind, which
causes it annoyance
and pain because it forces it to revise its hard-won beliefs.... How does
the system feel
about us changing the way it evaluates its own health... or the degree to
which it "feels it"
when humans are unhappy with it...
Because we're so close to this phase (just a couple more months of testing &
debugging simpler
components), this conversation is particularly interesting to me
Don't get me wrong, the Baby Webmind whose feelings I'm talking about here
is a pretty naive
little baby at the moment... it's a long way from transcending human
intelligence (except in very
narrow areas like market prediction) ... but the issue you're mentioning
arise nonetheless
> A year ago, I believed there was nothing you or I or anyone could
> or should
> know about Friendly AI in advance. I now recognize that this
> belief was quite
> convenient.
There certainly is something to be known in advance... but the percentage
of relevant knowledge that can be known in advance is NOT one of the things
that can be known in advance ;>
> There is still a discipline of seed AI in Artificial Intelligence, and a
> discipline of "seed morality" in Friendly AI.
>
Yes indeed...
ben
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:35 MDT