Geniebusters (was: RE: Yudkowsky's AI (again))

From: Lyle Burkhead (lybrhed@earthlink.net)
Date: Fri Apr 02 1999 - 18:41:01 MST


There are three questions at issue here. In decreasing order of importance,
they are:

1. Whether my method of argument is valid and useful.
2. Whether my conclusions are true.
3. Who I am; whether I know anything about computer science, or anything
else; whether I did a good job of applying the principle of calibration;
etc.

Question #3 is of no relevance. Maybe I'm an idiot. That has nothing to do
with it. As I said on the home page of geniebusters, Eric Drexler is not
the issue here. Neither am I.

Question #2 is more important. Is someone going to design the entire
"universal assembler," in advance, in such a way that once set in motion
the whole thing suddenly comes into existence with explosive force? Are
robots going to build skyscrapers for free? Is there going to be an event
that is incommensurable with everything else? or not?

Question #1 is the most important one. How can we get a handle on question
#2? Is there a general procedure for approaching such questions, so that
they become matters of fact and proof rather than matters of opinion and
speculation?

After writing the above, I finally looked at Billy Brown's comments and
found, to my surprise, that they are rational. If he had made a rational
comment in the first place, matters would have taken a different course.

> Ah. So you do still read the list. I wondered.

You wondered? In other words you were on the list two years ago? Anyway I
don't read the list. Twirlip of Greymist pointed out to me that Anders had
mentioned geniebusters on the list, so I looked at his comment and your
reply. Then I read the earlier posts in that thread, and sampled some other
threads in the archive. Apart from that, I haven't read the list since I
retired from it. However, this afternoon I find myself writing yet another
post to the list, the fourth one this week, in spite of my repeated efforts
to stop. I may have to join some kind of
12-step program for postaholics. Maybe a Higher Power will help me do what
I can't do myself.

> Certain sections of your site make it seem as if
> your method is to assume that any revolutionary claim
> must be false, and then look for an excuse to back up
> your assumption. This is just as wrong-headed as
> assuming that all such claims are true.

You don't say which sections you are referring to, or what "makes it seem"
that way. I am concerned here with certain specific claims made by Eric
Drexler in Engines of Creation. I don't even dispute all of the claims made
in that book, let alone "any" revolutionary claim. If someone wants to
claim that we will, in the foreseeable future, be able to make our bodies
much more self-correcting than they are now, and that our cells and organs
will become (asymptotically) perfect, I would not have a problem with that
claim, even though it could be considered "revolutionary." If someone wants
to claim that it is possible (and practicable) to make artificial
organisms, I don't have a problem with that either. The assumption that I
do make, and defend, is that new technologies, however revolutionary they
may appear to be, will occur within a certain unchanging framework.

> Your assertion that any software capable of replacing a human
> would demand a salary

I'm not asserting this for software capable of replacing *a* human, just
for software that replaces humans who make unsupervised decisions requiring

human judgment. Some humans are not on "our" side of the line. Many
employees are, for all practical purposes, automatons, and can be replaced
by machines.
This happens all the time and will continue. My assertion is that the realm
of automation will always exist within a larger realm that is not
automated. The boundary between agents and automatons will always be there,
and agents will always have to be dealt with as agents. They will always
expect to get paid (unless you can find some other way to motivate them). I
think what I'm going to have to do is illustrate this point with an
animated diagram, where you can see something changing within a framework
that does not change. I also need to put in a lot more interstitial
material explaining what I'm doing. (In other words the revised version
will be longer, not shorter. It isn't going to be an article, it's going to
be a book. Sorry, Robin ;-)

> The comparison of a truly general-purpose nanomanufacturing system
> to a modern industrial nation is such an oversimplification that it has
> no predictive value. There are too many differences between the behaviors

> of the two systems, and the implications of these differences need to be
> assessed individually and in detail if you want to make a meaningful
prediction.

That section isn't supposed to be read in isolation. It is part of the
argument leading up to Exercise 5 in Section 7, and it also has to be read
in conjunction with sections 12 and 13. The question is whether the
Assembler Breakthrough is going to happen. To get a handle on that
question, I'm trying to establish the level of complexity involved in "a
system that can make anything, including copies of itself." If a nanosystem
can make anything, in the sense required for the Assembler Breakthrough,
then it will amount to the same thing as an industrial economy.

It would of course be possible to have a nanosystem that could make
anything in a certain domain, maybe an extensive domain, without being able
to make anything in general. But that's not what you need for the Assembler
Breakthrough. If anyone has a proof (i.e. a structured argument, not some
impressionistic remarks) that the Breakthrough can happen with a less
general system, I'd like to see it.

Lyle



This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:03:28 MST