Robin Hanson wrote:
>
> Eliezer S. Yudkowsky writes:
> >You can't draw conclusions from one system to the other. The
Basically, "I designed the thing and this is how I think
> >genes give rise to an algorithm that optimizes itself and then programs
> >the brain according to genetically determined architectures ...
>
> But where *do* you draw your conclusions from, if not by analogy with
> other intelligence growth processes? Saying that "superintelligence is
> nothing like anything we've ever known, so my superfast growth estimates
> are as well founded as any other" would be a very weak argument. Do you
> have any stronger argument?
Statement: A seed AI trajectory consists of a series of sharp snaps and bottlenecks. Slow improvements only occur due to human intervention or added power.
Reason: Either each added increment of intelligence yields an increment of efficiency that can sustain the reaction, or it doesn't. While the seed AI might be slow enough that we could watch the "snap" in slow motion, either the going is easy or the going is very hard - the function is inherently unbalanced, and also this is the behavior exhibited by all current AIs.
After that, it's just guessing where the first bottleneck will be. After
optimization, after the level of the neural-level programmer, and before human intelligence, have been established in various parts of the argument. Then I make a guess, like Lenat with EURISKO, that the key is adding new domains; andthen I moreover guess the key ability for _that_ is "architecture".
Certain? No, of course not. If I may be permitted to toot my own horn, the argument is far too rational to be certain. But I still think it's a step above "does not"/"does too" debate of superintelligent trajectories. At least it's based on a mental model you can sink your teeth into.
> We humans have been improving ourselves in a great many ways for a long time.
> By a six year old's definition of intelligence ("she's so smart; look at all
> the things she knows and can do"), we are vastly more intelligent that our
> ancestors of a hundred thousand years ago. Much of that intelligence is
> embodied in our social organization, but even when people try their hardest
> to measure individual intelligence, divorced from social supports, they
> still find that such intelligence has been increasing dramatically with time.
The structure is still a lot different. What you have is humans being optimized by evolution. "A" being optimized by "B". This is a lot different than a seed AI, which is "C" being optimized by "C". Even if humans take control of genetics, "A" being optimized by "B" being optimized by "A" is still vastly different from "C" being optimized by "C", in terms of trajectory.
> This experience with intelligence growth seems highly relevant to me.
> First, we see that the effect of smarter creatures being better able to
> implement any one improvement is counteracted by the fact that one tries the
> easy big win improvements first. Second, we see that growth is social; it
> is the whole world economy that is improving together, not any one creature
> improving itself. Third, we see that easy big win improvements are very rare;
> growth is mainly due to the accumulation of many small improvements.
> (Similar lessons come from our experience trying to write AI programs.)
With respect to human genetic evolution, I agree fully, but only for the past 50,000 years. On any larger scale, punctuated equilibrium seems to be the rule; slow stability for eons, then a sudden leap. The rise of the Cro-Magnons was a very sharp event. A fundamental breakthrough leads to a series of big wins, after _that_ it's slow optimization until the next big win opens up a new vista. A series of breakthroughs and bottlenecks.
> Now it is true that AIs should be able to more easily modify certain
> aspects of their cognitive architectures. But it is also true that human
> economic growth is partly due to slowly accumulating more ways to more
> easily modify aspects of our society and ourselves. The big question is:
> why should we believe that an isolated "seed AI" will find a very long stream
> of easy big win improvements in its cognitive architecture, when this seems
> contrary to our experience with similar intelligence growth processes?
It isn't contrary to our experience. Just the opposite. Oh, it might not find the Motherlode Breakthrough right away; I fully expect a long period while we add more computers and fiddle continuously with the code. But once it does - self-enhancing rise of the Cro-Magnons.
If it's breakthroughs and bottlenecks in cases of *no* positive feedback; and even smooth functions turn sharp when positive feedback is added; and every technological aid to intelligence (such as writing or the printing press) produces sharp decreases in time-scale - well, what godforsaken reason is there to suppose that the trajectory will be slow and smooth? It goes against everything I know about complex systems.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.