From: Anders Sandberg (asa@nada.kth.se)
Date: Sun Aug 18 2002 - 09:46:59 MDT
On Sun, Aug 18, 2002 at 10:35:11AM -0400, Eliezer S. Yudkowsky wrote:
>
> This is a rather specious argument, Anders. As far as I can tell the
> entire reasoning consists of calling a multicomponent shielding system
> proposed by humans a "multi-layer" defense; while any system proposed by
> an SI, no matter how intricately optimized to maximize safety while
> minimizing cost, counts as a "single-layer" defense. Why? Because you,
> as a human, don't comprehend what the layers are and how they work, and in
> fact the optimal solution doesn't consist of these chunky discrete "layer"
> things at all, so to you it's a "single thing". Your human-level
> solution, of course, has modularity that you can understand, even though
> it's charmingly inadequate as a system design, so you consider that to be
> "multi-layered" and hence fitting your heuristic "don't put all your eggs
> in one basket", even though your system has far sparser coverage... being
> squarely in the tiny "human basket".
I am quite convinced that a superintelligence would use a multilayer
defense too. It might not be easily understandable for us, but the
argument I gave is general and would be true for all forms of defenses.
Whether it would be compressible enough so that we could understand it
or not is an interesting issue; I can see arguments for either position.
However, I don't think the issue here is what a hypothetical SI could
do; we want protections good enough so that we can get into the SI era
reasonaby safely, and that requires systems that can be built by humans,
maintained by humans and upgraded by humans. As we get better our
systems might get better.
-- ----------------------------------------------------------------------- Anders Sandberg Towards Ascension! asa@nada.kth.se http://www.nada.kth.se/~asa/ GCS/M/S/O d++ -p+ c++++ !l u+ e++ m++ s+/+ n--- h+/* f+ g+ w++ t+ r+ !y
This archive was generated by hypermail 2.1.5 : Sat Nov 02 2002 - 09:16:13 MST