Damien R. Sullivan writes:
>I really have trouble believing in SI we can't understand at all.
>Things that move too fast, sure. Things that are Einstein+Feynman+Bach
>+Turing+Darwin rolled up, sure. But not the magical SI. In fact,
>magical SI through AI seems a bit incoherent; you exploit the
>Church-Turing thesis, then assume the result does something beyond
>Turing-completeness...
This whole image of a super intelligence as incomprehensible to us as we are to ants suggests a terrible failing of our sciences of mind (AI, cogsci, neurosci, phil of mind, etc.) Such sciences have elaborated a great many useful concepts for categorizing and otherwise understanding minds of humans, animals, computers, etc.
We can use these concepts to imagine a zoo of possible minds, each with their advantages and disadvantages. With some effort, we could envision when different types will be possible and desirable for someone to construct, and how various selection processes might prefer some types over others.
The incomprehensible god idea seems to me to suggest that such analysis would be futile because it would miss the big important concepts. While older researchers in the sciences of mind lament that each new generation seems to reinvent the same concepts under new names, this claim suggests that is just because the important concepts just cannot be understood by minds like ours.
I can certainly understand the claim that it is very hard to figure out all the details of all the possible interactions we can conceive of between all the possible minds our current concepts can express. But this would just suggest that, as usual, our ability to predict fades out into a fog of possibilities. It wouldn't suggest that we not try to shine as bright a light as we can find into that fog.
In contrast, the incomprehensible god idea suggests that before the world gets very far filling out the zoo of possibilities we can now imagine, it will start to create creatures we cannot now imagine, and these creatures will be the important ones.
I have to admit I have been seduced by the apparent generality of human language and computation abilities. We may be slow, with limited and unreliable memory and processing. But it seems to me that given enough of us and enough time, we can compute and understand just about anything important. Like Damien, I have trouble believing in "the magical SI."
Robin Hanson
hanson@econ.berkeley.edu http://hanson.berkeley.edu/
RWJF Health Policy Scholar, Sch. of Public Health 510-643-1884
140 Warren Hall, UC Berkeley, CA 94720-7360 FAX: 510-643-2627