From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sat Jul 28 2001 - 12:28:27 MDT
Ben Goertzel wrote:
>
> Eli may have some ideas for simple tests involving program modification?
>
> In the context of "schema learning" in Webmind we created some tests
> involving goal-oriented program creation: some maze-running stuff, some
> stuff like "learn to solve the Tower of Hanoi problem", etc.
Actually, my thoughts have more focused on milestones than on benchmarks.
I have a pretty strong feeling that there will be no good way to compare
benchmarks.
Here are some of the milestones I've been thinking about for the GISAI
architecture, in no particular order (some of them are early, some of them
are really advanced).
Milestone: Teaching an AI to play tic-tac-toe using a purely verbal,
communicational description and no programming whatsoever.
Milestone: Getting an AI to play humanstyle chess - chess without
internally representing a search tree larger than humans use - at credible
amateur levels.
Milestone: An AI being able to successfully determine when two pieces of
code do "the same thing" even when they have different graphs. The
submilestones here are the various levels of "the same thing" - i.e.,
recursive vs. iterative implementations of the Fibonacchi sequence; the
different ways of computing Pascal's Triangle; the same piece of code
written in Java and in Perl; an array container class written in Java and
a linked-list container class written in C++; assembly language and C++
source; a high-level verbal description of an algorithm... and so on.
Milestone: As above, but being able to translate on demand between
substrates. (Quite a different problem - not necessarily a more advanced
or less advanced version of the above.)
Milestone: Inventing a complex tool, through abstract reasoning rather
than blind search or even heuristic search, within a complex toy world or
the world of code. Note that by "abstract reasoning" I mean something
quite different than the use of blind symbols in classical AI!
Milestone: Inventing a complex plan to solve a complex solution, again
through the use of abstract reasoning rather than blind search.
Milestone: Successfully linking events observed in a billiard-ball
modality image to higher-level symbols, such that a generic event (nine
balls in three groups of three) can be translated into a symbolic
structure and reconstructed in rough detail using only that structure.
Milestone: As above, but communicated to a human in English, and then
reconstructed from the human's description in English. Both this and the
above also come in gradiated versions depending on the complexity of the
image.
Milestone: Storing a billiard-ball image in memory and retrieving it from
memory.
... and so on ...
-- -- -- -- --
Eliezer S. Yudkowsky http://intelligence.org/
Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:37 MDT