From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Mon Mar 20 2000 - 01:00:15 MST
There's a common saying to the effect that worrying about nanotechnology
"escaping" to the "wild" is like worrying about cars going rogue and
learning to run on tree sap instead of gasoline.
But consider this: Build a car that can reproduce, and the time will
have arrived to worry about it learning to run on tree sap.
After reading _Nanomachines_, it seems to me that triquintuple
error-checking should suffice to prevent evolution by means of errors in
the genetic code; however, it occurs to me to worry about the
nanomechanical equivalent of prions; that is, some particular defect in
a rod structure which results in the same defect being present in the
rods manufactured.
The one thought that kept running through my mind while reading
_Nanosystems_ is that assuming that a single dislocated atom knocks out
a mechanism is not a "conservative" assumption. It is vastly
optimistic. Even assuming that a single dislocated atom causes a
complete system crash is optimistic. Speaking as a programmer, if every
error produced an immediate and simple crash, life would be a lot easier
than it is. Assuming that a single dislocated atom causes the
production of subtly wrong moieties that break or subtly corrupt the
rest of the system; now that's realistic. Nanodebugging the declarative
design errors will be nightmare enough; debugging radiation damage will
be beyond imagining.
There's no way - that I know of, anyway, correct me if I'm wrong - to
read out the atomic positions inside a complex molecular structure. We
have enough trouble designing computer systems that will operate in
spite of programmer errors. Imagine the problem of designing a computer
system that has to operate in spite of random bitflips, when there's no
way to examine outputs directly, just take a few simple measurements.
It seems to me that one of the major problems of nanotechnology will be
designing systems that fail reliably (or systems in which any error not
"measurable" to a checking mechanism must automatically maintain all
design characteristics; after all, any error which shows up externally
must be "measurable" in some sense).
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/beyond.html
Member, Extropy Institute
Senior Associate, Foresight Institute
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 15:27:32 MST