Anders Sandberg wrote:
>
> OK, this is the standard SI apotheosis scenario. But note that it is
> based on a lot of unsaid assumptions: that it is just hardware
> resources that distinguish a human level AI from an SI (i.e, the
> software development is fairly trivial for the AI and can be done very
That's not what Billy Brown is assuming. He's assuming software development is _harder_ than hardware development, so that by the time we have a working seed AI the available processing power is considerably larger than human-equivalent. I happen to agree with this, by the way.
> fast, and adding more processor power will make the AI *smarter*),
I'll speak for this one. If we could figure out how to add neurons to the human brain and integrate them, people would get smarter. (I deduce this from the existence of Specialists; adding cognitive resources does produce an improvement.) Similarly, I expect that substantial amounts of the AI will be working on sufficiently guided search trees that additional processing power can produce results of substantially higher quality, without requiring exponential amounts of power. Anyway, I think we did this during the Singularity debate.
> that this process has a time constant shorter than days (why just this
> figure? why not milliseconds or centuries?),
Actually, I think milliseconds. I just say "hours or days" because otherwise my argument gets tagged as hyperbole by anyone who lives on the human timescale.
Centuries is equally plausible. It's called a "bottleneck".
> that there will be no
> systems able to interfere with it - note that one of your original
> assumptions was the existence of human-level AI; if AI can get faster
> (not even smarter) by adding more processor power, "tame" AI could
> keep the growing AI under control
> - that this SI is able to invent
> anything it needs to (where does it get the skills?)
If you dropped back a hundred thousand years, how long would it take you to out-invent hunters who had been using spears all their life? Skill is a poor substitute for smartness.
And if it's stumped, it can get the answers off the Internet, just like I do.
> and will have
> easy access to somebody's automated lab equipment (how many labs have
> their equipment online, accessible through the Net?
There's some lab with a Scanning Tunneling Probe that's made "selectively" available to students on the Internet. I thought that was hysterical when I read it. All along, I'd been assuming the fast infrastructure would start with hacking automated DNA sequencers and protein synthesis machines. But this way is probably faster.
> why are you
There are bloody _humans_ who can hack any system.
> assuming the AI is able to hack any system,
> especially given the
> presence of other AI?).
In a supersaturated solution, the first crystal wins.
> And finally, we have the assumption that the
> SI will be able to outwit any human in all respects - which is based
> on the idea that intelligence is completely general and the same kind
> of mind that can design a better AI can fool a human into (say)
> connecting an experimental computer to the net or disable other
> security features.
If the intelligence is roughly human-equivalent, then there will be specialties at which it excels and gaping blind spots. If the intelligence is far transhuman, it will still have specialties and blind spots, but not that we can perceive. So yes, I make that assumption: An SI will be able to outwit any human in all respects. I'll go farther: An SI will correctly view humans as manipulable, deterministic processes rather than opponents.
> As you can tell, I don't quite buy this scenario. To me, it sounds
> more like a Hollywood meme.
Not really. Hollywood is assuming that conflicts occur on a humanly understandable level - not to mention that the hero always wins. Remember the Luring Lottery, when Hofstadter received a large collection of Vast numbers, all using different specification schemes? Determining which was largest would be a serious mathematical problem, but I feel quite confident in announcing that one of the numbers was far larger than all the competitors raised to the power of each other.
Likewise, the first SI timescale that isn't too slow to notice will be too fast to notice.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.