From: Ben Goertzel (ben@goertzel.org)
Date: Fri Feb 25 2005 - 23:09:01 MST
Hi,
Having absorbed a bunch of commentary on my ITSSIM and GoalGeneralization
draft papers, here are my current thoughts. Part of this is summary, but
there are some new ideas at the end.
1)
I continue to think ITSSIM is a valid method for providing a reasonable
degree of assurance that an AI will continue to be careful as it
self-modifies and self-improves. Of course, there are no guarantees with
such things, as proved by the possibility that when an AI gets smarter than
some particular value, superintelligent aliens will contact it and modify
it. An interesting question to explore is how stable ITSSIM is if hardware
errors or the like occur; I think the answer will be "quite stable."
2)
I also think that it makes sense to impose a criterion of "Careful Supergoal
Generalization" (which I now see is a better name than Goal Generalization).
The issue here is, if an AI has an overarching supergoal, and then modifies
itself to as to change its supergoal, what restrictions can we make as to
how the new supergoal relates to the prior supergoal? My suggestion is that
one could (and perhaps should) intentionally restrict this relationship to
one of "careful generalization", in a certain sense. My definition of
generalization in this context seems sound but may end up needing some
refinement.
3)
These two ideas can be combined naturally, if one assumes that the initial
supergoal includes ITSSIM. I didn't quite say this in either of the papers
but I now think this is a good way to phrase things.
4)
I now agree with several commentators that the inclusion of explicit
"emergency measures", as I suggested in the Goal Generalization essay, is a
bad hack. Rather, I think it's better to use either
a) pure ITSSIM, or
b) careful supergoal generalization, with an initial supergoal that includes
ITSSIM, and also includes emergency measures covering all known existential
risks
In case b, we are counting on the AI to generalize ITSSIM as seems
appropriate, and also to carefully generalize its initial emergency measures
as seems appropriate.
5)
After reading Pei Wang's commentary, I see that the use of "theorem proving"
in the context of proving theorems about empirical reality needs more
elaboration. Specifically, it needs to be emphasized that the axioms of
one's formal system must include an assumptive solution to the Humean
"problem of induction". This is implicit in Schmidhuber's Godel Machine
work but he doesn't elaborate on the point at all, so far as I can recall.
I will product a new version of the paper (with the best parts of the goal
generalization paper fused into the ITSSIM paper), when I get a chance,
probably sometime late next week.
Further constructive criticism is appreciated.
I have also put a little thought into how these ideas could be explored
using current AI technologies, prior to the advent of amazingly powerful
theorem-provers.
It seems to me that one could implement preliminary versions of both ITSSIM
and careful supergoal generalization, by substituting "guessing" for
"proving" in both algorithms. Of course this results in rather different
algorithms -- but the qualitative nature of the dynamics of the evolving AI
system might not be so different. At very least it would be an interesting
sort of experiment.
This kind of experiment could be done using an only moderately improved
version of the current Novamente AI codebase. But of course, the Novamente
project is in a pretty severe financial situation at this moment, so we have
no resources to play with this sort of thing. Maybe next year ;-p
-- Ben G
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:50 MDT