From: Anders Sandberg (asa@nada.kth.se)
Date: Sun Aug 18 2002 - 03:44:03 MDT
On Sat, Aug 17, 2002 at 11:15:51PM -0400, Brian Atkins wrote:
> transparent society (and I DO mean 100%... it has to be perfect, and
> it has to be backed up with the ability to take near-instant action in
> crises), superintelligence. I mean, 10 years from now when technology
> puts designer pathogens into the reach of terrorists, the options become
> quite limited. It gets to the point where you absolutely 100% have to
> stop an attack before it starts UNLESS you have some magic way of
> creating perfect defense (for instance a cure to a disease) after you
> are already under attack. Despite what Anders and Kurzweil want to
> think, this is not equivalent to a computer virus where you have a
> backup of your data somewhere to fall back on.
OK, I perhaps should explain a bit more clearly why I would trust
layered but imperfect defenses (although Bruce Schneier does a better
job at that).
Imagine the enemy having to overcome one super defense (forcefields, SI,
whatever) to wreak havoc. No defense is ever perfect, so there is a
finite probability P of it failing. The cost of a perfect defense is
also enormous and grows fast; I would expect it to scale at least as
1/P. A 99.9% safe defense would be ten times as expensive as a 99% safe
system.
Compare this super-defense with having N layers of less powerful
defenses, each with a probability Q of failing. If the threat got through
defense 1, it would still have a 1-Q chance of being caught in the next.
The risk of it passing all the layers is Q^N, which shrinks very fast as
N increases. The total cost would be N/Q. So if we have ten layers of 90%
safe defenses the risk of failure is 10^-10 and the cost is 100. A single
super-defense with the same risk would cost 10^10.
Clearly we can get enormously safer and cheaper systems by layering
imperfect systems rather than strive for the 100% solution. Having
multiple layers also allow you to handle more varied threats, while a
single strong layer could have a vulnerability or simply not be designed
to handle new threats. 100% perfection is also infeasible, since you
could always have quantum tunneling and thermodynamic miracles help
bring about the threat. The best we can do in principle is what is
allowed by physics, and even then the expected cost of the disaster
might be less than the costs of the system. We have no backup of
humanity, but if a proposed solution to preventing an existential risk
would cause being human to be intolerable the solution is too costly.
-- ----------------------------------------------------------------------- Anders Sandberg Towards Ascension! asa@nada.kth.se http://www.nada.kth.se/~asa/ GCS/M/S/O d++ -p+ c++++ !l u+ e++ m++ s+/+ n--- h+/* f+ g+ w++ t+ r+ !y
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 09:16:12 MST