From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Thu Aug 28 1997 - 23:25:37 MDT
[Discussion broken up due to enormous letters. See NANO posts.]
[Terminology: "Black goo" is deliberately designed attack goo.]
> > > = Sandberg
> > = Yudkowsky
> = Sandberg
Anders Sandberg wrote:
>
> "Eliezer S. Yudkowsky" <sentience@pobox.com> writes:
>
> > Anders Sandberg wrote:
> >
> > > Ever thought about the fact that the immune system isn't mutually supporting?
> > > It is rather a balance of predators which do not predate on each other since
> > > that destroys their fitness.
> >
> > Interrogative? Expand, please.
>
> I would like to get some expert in on this, since my medical knowledge is
> more neuroscientific than immunological (I'm not an immunologist, I just
> pretend on the net :-). Basically, the immune system is programmed to
> destroy, destroy, destroy - everything. But during the complex process
> where white blood cells are bred the cells that would kill unsuitable
> things (like each other, the body etc) are weeded out, as are the "meek"
> cells. The rest are just barely tamed, and will with no hesitation destroy
> any cell that doesn't show the right antigens, and everything in the
> vicinity.
Okay, so our immune system uses directed evolution to create attack nanites.
Just because the attack nanites are unsophisticated does not mean the immune
system is unsophisticated. There's strategy, tactics, and dynamic evolution
of new attack nanites. The actual battles may be between idiots, but the
supporting operations and design of the idiots is where victory will be
decided, and where relative complexity is calculated.
> > > Good guys will be motivated to contemplate directed evolution since the bad guys
> > > do it; and they tend to have more brainpower and money on their side. I
> > > think the weapon of openness is important here.
> >
> > It could go either way. Maybe the bad guys kidnap all the scientists and then
> > use pain-center stimulation to achieve faster results. Idealism may win
> > battles and break ties, but it is not a defense to be trusted.
>
> Or maybe the good guys spread a nanite which give all bad guys migraine
> when they think evil thoughts. Get real, this sort of unrealistic Hollywood
> speculations is the last thing we need ("... if you do that, I'll send in
> my *dragons*!" "Ha! I have an army of invisible pink unicorns that will
> eat your dragons!").
That was my point, although it was directed more at the dangers of idealistic
optimism than at fantasy complexification.
> I never said I would sketch an
> *invulnerable* system, just a sufficiently strong system. For any immense
> shielding I can come up with you could always invoke an even greater
> cosmological disaster ("But your immune system can't stand a supernova!").
> This exercise is trying to look at defenses against gray goo, which
> is the real problem, not macro-level warfare.
But you *do* need an invulnerable system, or at least one that is invulnerable
to goo. I can reasonably invoke any forces modern technology is capable of
wielding, up to and including nuclear weapons. If black goo reduces your city
to radioactive ash, you lose! It doesn't matter how sophisticated your
defenses are! I am darn well entitled to demand that your immune system stand
up against nuclear weapons, because in practice, that's what going to be used!
Not impossible. Not at all impossible. If you have a layered defense system,
a diamond shell, and Fog seat belts, the city might be perfectly capable of
withstanding a nuclear attack. It would lose a layer of defense, but might
well be capable of rebuilding it before more black goo crossed the radioactive
zone. In addition, as I pointed out, the city might gain more than it lost.
Even so, it's entirely possible that, on any planet, the black goo wins. All
the time. Every time. The Universe is under no obligation to make things
easy for us. Hence the proposed Directive of Evacuation: "Get everyone off
the Earth, into partitioned space colonies, before releasing nanotechnology to
the world."
> Your claim that destruction is
> fundamentally more efficient than creation suggests that if I know about
> your goo I can destroy it.
There is, of course, something to that. The problem is preventing it from
being mutual destruction. The surface area of conflict, and strategic
compartmentalization, mean that if destruction is easier than creation, then
attack is easier than defense, and both parties lose equal amounts of
resources until only one is left. Don't forget that your resources may be
devoted to supporting humans. Also, nanobattle might end in stalemate, but
even so eliminate all higher-than-nano structures such as humans.
My mental picture of these conflicts is partially drawn from Conway's Game of
Life, in which a single particle can destroy an enormous, complex structure.
Things on the molecular level will almost certainly be different. Even so, I
know of no better image.
Though molecules are FAR more robust than any Life structure I have ever been
privileged to look at, the actual "front" might still resemble a game of Life.
Energy pulses through and breaks all bonds, creating a no-man's-land
reminiscent of randomized Life. Both sides grab at the atoms and volume
created. Tactically, the situation appears symmetrical. Strategically, it's
hard to tell who has the advantage.
Still, let's dwell on the tactics for a few moments. The goo can send a
shaped explosion aimed at the city, so that the no-man's-land is composed of
formerly city material, rather than equal amounts of city and goo. Can the
city send shaped charges back at the goo? Yes. Can the city "grab" the
no-man's-land thus created? Locally, yes - although there is one *minor*
tactical advantage, which is that the goo is independent and the city is
structured, so that the city has to expend more effort to build into a given
amount of volume. But I'm claiming a lot of battle intelligence for the black
goo, while the city's first line of defense, rather than being diamondoid, may
be independent 'munes. I think that, tactically, there's a match.
Strategically, I'm assuming the goo has unlimited material, while the city has
limited material. I'm assuming that nuclear power is cheap enough that both
have unlimited power. (I'm also assuming the city is padded, so nukes can't
kill it outright.) Also, the goo surrounds the city.
Try this tactic. First, the goo hits the city with a vaporize-one-layer
attack, even though this also vaporizes a layer of goo. Then it detonates a
nuclear weapon next to the city. This pushes the city into the goo,
dissipating the no-man's-land and replacing it with the soft goo. The goo is
using, as a strategic/tactical advantage, the city's rigidity. If the city is
surrounded by a layer of soft goo, it's difficult to tell what happens.
The city is a steel ball surrounded by 'munes surrounded by goo. A nuke stirs
things up. Since goo outnumbers 'mune, most of the city is now surrounded by
goo. The 'mune stirred into the goo can be treated as isolated spores and
obliterated with overwhelming force. The city has lost some mass. Moreover,
this is a one-way tactic; it cannot be used against the goo.
> > How much of an insulator is [diamond] for temperature? What about gamma rays?
>
> It conducts heat well, and as far as I know is transparent for gamma rays.
> But you are not seriously suggesting grey goo radiating gamma rays?
> (that would BTW be very bad for the goo too, since nanites are radiation
> sensitive; and I could have radiation shielding in the skin while the
> goo would be unprotected (since grey goo carrying around lead is easy
> to find and exterminate by taking it away)).
Sigh. Gray goo, yes, a sea of black goo, no. I am not proposing that goo
spores will radiate gamma rays. War goo might.
> Diamond does quite nicely here, although even more inert materials would
> do better. The breaches can be of two kinds, a strong
> external force making its way in, and in that case we are talking about
> macro level defense and offense again, or nanites forcing their way through
> the walls. In the later case, they would need energy to breach the
> walls if they are inert, and it would be exothermic for the defenders to
> thicken their walls. If they turned the surrounding compartments into
> a "diamondoid scar" the energy in the infected compartment would not
> be enough to breach the barrier, and the whole infection would require
> a lot of external support (which could be broken).
I am mainly worried about macro attack and defense. If it is easier to
destroy than create, this translates into a strategic advantage for the human
in spore attacks. You just find and destroy the spore, using nuclear weapons
if you feel like it.
I'd be more inclined to trap the spore in something solid, heat it to a few
thousand degrees, pass a few thousand amperes, and hurl into the nuclear
reactor that powers the city. If you can make something the spore can't get
into, you can make something it can't get out of, and will be greatly aided by
the fact that you don't give a damn about living conditions inside the trap.
Diamond scar? Diamond cyst!
> > What detector mechanism are you using? Can I falsely trigger it over the
> > entire body, thus making the "detector" useless and redundant?
>
> I have been thinking of having nanometer "pipes" criss-crossing the barrier,
> containing linear molecules which break or vibrate when interfered with. You
> cannot detect them from the outside (just diamond, and then an opening -
> oops!), and the signal into the compartment will give information about
> where something is going on.
Okay, sounds to me like they're perfectly vulnerable to spoofing. And
regardless of whether I can "detect" them from the outside, why shouldn't I
simply assume they're there? Can they be fooled if I know they're there
(although not exactly where)?
> > If sealed-off
> > areas are easier to attack, wouldn't the false alarm let me peel off an entire
> > layer? And why couldn't I just repeat the action until you were the size of
> > an apple core?
>
> Sounds like something that could be designed away. OK, I haven't yet
> tried to sketch how the control systems ought to look (I haven't got the
> time even for doing this discussion properly! :-( ). The software
> part is of course even more important than the hardware part.
>
> But basically, if you trigger all the surface cells, don't you think the
> system will immediately (assuming some intelligence somewhere) guess
> the strategy?
Of course. But even so, the detectors are useless. You offered an advantage;
that advantage has been negated.
> > > The rest of the organism (could be a transhuman, factory
> > > or a city) is compartmentalized with similar walls; infected sections
> > > can be isolated.
> >
> > Okay, so you've made the problem of immunity recursive on immunity for a
> > single section. This may actually be an excellent plan of defense, allowing
> > for partial breaches and reserves. But a design for an immune system it's not.
>
> It is a partial solution, and makes the problem more manageable. The body
> uses something fairly similar with its internal barriers (it is surprising
> how strong the blood-brain barrier is, even adrenaline can't get through!).
I don't deny it's an effective partial solution. Like the principle of proof
by induction, it's a very useful solution. But you still need to write the
recursive function, not just decide to use LISP as a language.
> > Okay, I see a trend developing in your plan. Detection is the important
> > thing; it's assumed that pockets of infection, once detected, can be
> > obliterated. Very much like the human immune system. But if viruses could
> > outfight white blood cells, we'd be dead. I'll conceed that fooling detection
> > might be made difficult, almost impossible, through the use of digital
> > signatures. But again - how do you keep a single cell from being penetrated?
> > Once a cell is penetrated, you destroy it - but does the destruction keep the
> > infection from spreading? A single spore might be defeated by this strategy.
> > A sea of gray goo won't be.
>
> A sea of gray goo is also a macro-level problem. It is obvious, it can
> be fought on the macro level (meaning that human or transhuman intelligence
> have the time and space to implement defenses not just based on what their
> ordinary immune systems can do, but also their other available resources).
> For example, it wouldn't appear impossible to nanofacture "antibodies",
> very simple nanites which disrupt the goo nanites, en masse once the goo
> is understood.
Which is more complex, more subject to disruption? You, or the goo? I won't
discuss transhuman defense against the goo; that's their problem. But human
intelligence might NOT have the time to implement defenses in nanowar. Not if
things move too quickly; not if evolution controls the battlefield. Anyway, I
think that nanowar is the problem. If nanowar is easy enough, destructive
enough, the Evacuation becomes necessary.
The gray goo problem is overrated; it's black goo we need to worry about. We
can get along just fine without any self-reproducing nanites, ever. We can
make 10 capable of constructing 1, given plans broadcasted by remote signal.
If we can't use remote signals, we can use super-redundant "queens" to carry
the information that directs 10 to make 1, or 20 to make a queen. Will this
slow down reproduction? You bet. Who cares? It's not the speed that makes
nano reproduction so powerful, it's that it happens at all.
> I have noticed that your approach seems to be that when subtle attacks
> doesn't work, you invoke maximum overkill instead. But that takes a lot
> of resources, and can often be disrupted more easily than a subtle attack
> (compare "AAARGH! The goo has my password!" to "Ouch, this goo is corrosive,
> let's wash it away in a shower of hydrogen peroxide").
Subtle attacks are operating under enormous strategic disadvantages. Not that
they won't be possible. I've noticed and informed authorities of my share of
analogies between computer security programs and toilet tissue. But we can't
discuss subtle attacks, not really. Subtle attacks will mature slowly as the
power of nanotech unfolds, as with computer security in our times. What could
a 1950s programmer say about computer security?
> Let's go down to the real battle - dismantling nanites.
>
> > As far as I can tell, your strategy simply lets
> > me destroy your entire complex cell by cell. Penetrate a cell, watch it go up
> > in flame... repeat.
>
> Hmm, like Stalingrad? ;-) You assume that the destroyed cell will no
> longer be a problem. But what if it turned into a cube of inert diamondoid?
> Then it would also be a hinder, and give me even more time to develop
> countermeasures.
If it is inert diamondoid, is that more of a hindrance than the outer shell?
Besides, this whole discussion is looking obsolete anyway. Any rigid outer
defense is toast. It has to be surrounded by explosive-equipped soft 'mune.
At most, you can have a rigid outer defense as a pretty shell.
> I see that I have been a bit too defensive (I usually am in strategy),
> since you are right in that just slowing the advance of the goo won't
> win the fight. So, to get back to how the body works, I'll better
> develop antibodies and macrophages.
>
> An antibody would be a very simple device that sabotages a certain nanite.
> It could be a molecule that binds and then tends to stick to other
> antibodies, trapping the goo nanites as dimers or polymers. A macrophage is
> simply a selective disassebler, which destroys a certain kind of nanite
> or structure. Assuming I have detected the infection the next step would
> be to make antibodies and macrophages against it, and start spreading them.
> Note that they would be produced faster than the goo since the goo would
> have to both reproduce, breach security and defend itself while the
> antibodies and macrophages would just be produced (although a goo-like
> macrophage is an interesting and dangerous concept) and sent on their way.
It has often struck me that homogeneity is a tremendous tactical advantage in
battle. Well, not really, only in this past thread. But even so, homogeneity
means you have no delay time to bring up specialized warriors; you can replace
all fallen comrades simply by stirring things up.
But even granting your case, if the goo has so much as a cubic millimeter to
call its own, it can keep its queen reproducers in the center while
surrounding itself with warrior goo, just like the city.
Why can't the goo use antibodies and macrophages against the 'mune?
The tactical symmetry remains unbroken.
> > The real obvious one is this. Your immune system - any immune system
> > conceived along these lines - requires that two nanosystems fight each other
> > to a standstill. If two nanosystems fight and destroy each other, then any
> > city in an island of gray goo is doomed.
>
> I think you are thinking too much of ants fighting here. I see it more
> as chemistry, especially protein interactions. It is much easier to make
> something which binds goo into an inert mass than to build a goo eater.
Aha! A tactical advantage for defense over offense! It is easier to keep a
mass inert than it is to keep it active! ... Still, one has to wonder about explosives.
> > The goo simply makes repeated
> > attacks, and each time, the city shrinks a little. We won't even speak of
> > such horrors as cutting off that city's solar power.
>
> An isolated immune system in a world where nobody else has immune systems
> is weaker than a system in a world where nanodefenses are common. The ultimate
> defense would of course be to have immune systems everywhere, defending
> not just the transhumans but the himalayas, squirrels and grass. A bit like
> the current state in biology, really.
If nanotechnology was released on Earth, the 'munes would have to be
EVERYWHERE. The air. The water. Earth's molten core. Otherwise, the goo
could gain a foothold. At all costs, the goo can *never* be allowed to become
a sea. While it is spores it can be obliterated in mutual annihilation. As a
sea, it obliterates you. It's not quite a molecular game of Go. You have to
outnumber and outdeepen your enemy, not just surround it. But even so...
> > In short, the immune system requires that *defense* have an advantage over
> > *offense*. I can conceive of any number of ways that this could happen. I
> > shall now proceed to list them.
>
> Yes, this is the crux. I contend that defense is easier.
>
> [Some good examples deleted for brevity; I get the impression we are
> discussing nano-sized knights fighting on the battlements of a transhuman
> city :-)]
>
[Examples of exotic tech deleted by author.]
>
> OK, but at this level I simply escape into my own basement universe (Mitch
> suggested that you could even hide backdoor wormholes inside atoms for
> unexpected ambushes). Dragons vs. invisible pink unicorns again?
I agree, and pretty much said so. But the ideas might have sparked something.
Couldn't hurt to put it down.
> > Then there's the simple possibility, dependent on molecular details of which I
> > know nothing, that defense on the molecular level has an innate advantage over
> > offense. Certain molecules or nano-screens might simply be easier to form
> > than to break, just like atoms or quarks.
>
> Good point. There are certainly some molecules which are much easier to
> break than to form, and hence allows me to make good detection systems or
> nanites whose programming is erased if they are opened. I think there might
> be some structures which are very troublesome to destroy on the nano level.
Yeah, but the problem with rigid structures is that they can be destroyed by
explosives. There are no invulnerable shells. A blast creates a free area
which both sides try to gobble. Sit there in your shell and you're dead meat;
the goo will send a shaped charge to create an asymmetrical dead zone, eat the
whole thing, and repeat until you're the size of an apple core.
> > Nanosystems are always faced with "destruction by induction". That is, you
> > can always destroy one cell; therefore you can destroy the whole thing. To
> > defend against this, it is required that the system expand faster than the
> > destruction OR that it be impossible to destroy one cell.
>
> Or that you can make the loss of a finite number of cells bearable. If their
> loss removes the threat (for example by forming a nanoscar), then the
> defense side will win.
I disagree with the whole concept of a nanoscar. If there is such a thing,
you turn it into your first line of defense, not wait until after you've been invaded.
> > Ergo, it is easier to destroy than create.
>
> Om mani padme hum... :-)
You aren't serious, but even so... after all that? The whole discussion? I'm
debating it, not asserting it, although considering the existence of TNT and
nukes, destruction is close to axiomatic on the molecular level.
> > (1) "Harvard doesn't publish science fiction." New Destinies III, Spring
> > 1988. Edited by Jim Baen.
>
> What is this? Sounds interesting.
A nice piece by Moravec. One of those Little Bits of Everything. Superdense
computers (hence Higgsium and Monopolium), uploading identity, and an odd
theory about quantum mechanics. The book is by Baen Books, ISBN 9-671-65385-7.
> (now I have to get to bed - why do Elizer have to start a good argument
> 2 in the morning? :-)
That's when I think fastest. Besides, in my time zone, it was 11.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://tezcat.com/~eliezer/singularity.html
http://tezcat.com/~eliezer/algernon.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 14:44:47 MST